May 4, 2011

How to pull large amount of posts? | CyberSEO Pro | Support Forum

Avatar

Lost password?
Advanced Search

— Forum Scope —




— Match —





— Forum Options —





Minimum search word length is 3 characters - maximum search word length is 84 characters

sp_TopicIcon
How to pull large amount of posts?
Topic Rating: 0 Topic Rating: 0 Topic Rating: 0 Topic Rating: 0 Topic Rating: 0 Topic Rating: 0 (0 votes) 
December 13, 2014
7:09 am
Avatar
jtt200
Guest
Guests

I would like to test CyberSeo plugin (full version), and I am pulling, lets say 50-70 (in reality, it may be around 25 or so per blog), posts from 700 separate Wordpress blogs (domain1.com/feed, domain2.com/feed, and so on). I’ve set up a Wordpress blog on a domain-name.com, and created 70 categories, something like domain1, domain2, and so on.

So: 700 blogs -> to 1 blog with 700 categories (can be a total of 35,000 Wordpress blogs, or more).

I’ve been working on this for close to 24 hours, and as of right now I have around 170 posts, which are published on the target site. I’ve added around 35 urls to XML Syndicator section of the program, and I would say, that only around 9 are active at any time / at the same time.

I am using cheap / shared hosting, the lowest type of plan that’s out there, so server “resources” may be limited (and that’s why it is going so slow), but I am not sure.

I would like to find out:

1) What is the best way to pull up to, or more than 35,000 posts from 700 different blogs to one domain? I am assuming it depends on the hosting plan also, but I am not sure. How would I set the XML Syndicator, in order to get the best possible results (meaning, do it correctly in the shortest amount of time).

2) Is there any way to check / measure how the resources are used by the program, and optimize it, based on that.

3) Would the program work better on a VPS plan, and even better on a dedicated hosting plan. Would it be also like that, that the more RAM and processing power I have the better everything would work.

In general, how to do it on shared, how would I “measure” what is used by the plugin, and optimize based on that, and would it be all a lot easier and faster on, lets say “strong” dedicated plan.

Also, how many domains should I work on at the same time, how many posts should I pull per domains, and how often. And how to figure it out?

Thanks.

December 13, 2014
7:22 am
Avatar
CyberSEO
Admin
Forum Posts: 3678
Member Since:
July 2, 2009
sp_UserOfflineSmall Offline

1) The number of simultaneously pulled posts fully depends on your host capacity (can be obtained experimentally only). However I would suggest you to avoid of pulling all feeds at the same time.

2) There are many tools to measure the resource usage. E.g.: Login to see this link

3) Ye of course. More server resources -> better performance of the script.

> Also, how many domains should I work on at the same time, how many posts should I pull per domains, and how often. And how to figure it out?

I can’t answer to this question because everything is very individual. Some have 2-3 sites, others run multiple networks with 100’s of domains.

December 13, 2014
7:35 am
Avatar
jtt200
Guest
Guests

I am working on a network of 128 CyberSeo blogs, and I am coming across some issues here. I think that I set the plugin to pull too many feeds at the same time, which is causing MySQL and Apache to stop (and websites don’t work). What would be the best way of pulling feeds, and how often, what would you recommend?

– pull 1 feed at a time and 1 post every 60 minutes (again, a total of 128 sites, total)

– pull 2 feeds at the same time, and 1 post every 60 minutes

– pull 3 feeds at the same time with the above settings…

– or maybe pull 3 feeds at the same time and 3 posts every 30 minutes (or lets say 120 or 180 minutes)

I understand that it is a matter of figuring it out over time, but what would be your advice?

How would you do the settings, if these were your sites (128 CyberSeo blogs, not bad / good hosting; it is something like vps or dedicated, I think).

Thanks.

Forum Timezone: Europe/Amsterdam

Most Users Ever Online: 541

Currently Online:
10 Guest(s)

Currently Browsing this Page:
1 Guest(s)

Top Posters:

ninja321: 84

s.baryshev.aoasp: 64

Freedom: 61

MediFormatica: 49

B8europe: 47

saviulisse67: 45

Member Stats:

Guest Posters: 338

Members: 2636

Moderators: 0

Admins: 1

Forum Stats:

Groups: 1

Forums: 5

Topics: 1537

Posts: 7758

Newest Members:

kevin.crywolf, kevin.cooper.boonenewsmedia, FloSti, t.schleiser, worldnetsites, janodamek

Administrators: CyberSEO: 3678