May 4, 2011

'Limit' RSS Feeds When Set to Run via Cron Job/Manually | CyberSEO Pro | Support Forums

Avatar

Lost password?
Advanced Search

— Forum Scope —




— Match —





— Forum Options —





Minimum search word length is 3 characters - maximum search word length is 84 characters

Related Topics sp_topicicon
'Limit' RSS Feeds When Set to Run via Cron Job/Manually
Topic Rating: 0 Topic Rating: 0 Topic Rating: 0 Topic Rating: 0 Topic Rating: 0 Topic Rating: 0 (0 votes) 
December 27, 2011
9:33 am
Avatar
cyber1
Guest
Guests

Hello Admin,

Can you please provide an option to run just “One” feed a time once the next update shows as ‘ASAP’ ? So if we set the ‘Cron’ to run once/hour then only 1 of ‘x’ RSS feeds at the ‘ASAP’ stage runs.

The RSS feeds “bunch up” at ASAP and then tend to all run at the same time taking alot of resources at once. (for example, others software let you ‘stager’ the feeds by setting the exact time of day for each.) If the ‘cron/manual’ can be set to run just “x” RSS feeds per run (for instance 1) of all the feeds at ‘ASAP’ that would suffice.

Then the next time the Cron runs it picks up the next feed at ‘ASAP’ therebye staggering the run times. If the number was set to ‘2’ then 2 RSS at ASAP would run.

Hope I explained this properly, it is an improvement to run RSS feeds ‘staggered’ , so not all feeds run at once.

Sincerely,
cyber1

January 8, 2012
1:30 am
Avatar
cyber1
Guest
Guests

hello admin,

was just wondering if you noticed this post?

January 8, 2012
8:58 pm
Avatar
Admin
Forum Posts: 3097
Member Since:
July 2, 2009
sp_userofflinesmall Offline

So let's consider the following situation:

  1. You have 10 feeds that have to be pulled say every 30 minutes
  2. When the cron runs every hour, all 10 feeds have the “ASAP” status
  3. According to your suggestion, the 1st feed marked “ASAP” will be pulled, others will be ignored.
  4. Next time when the cron runs again, the situation will be the same: all feeds will be marked as “ASAP” and only the first one will be pulled.

In such a case, the rest 9 feeds will never be pulled. Thus I think it's not a good idea to implement a feature suggested by you above.

January 17, 2012
6:32 pm
Avatar
cyber1
Guest
Guests

Hello Admin,

I understand what you are saying thanks. In my set up the CRON checks every 15 minutes so the next ASAP would be checked, but your point is valid as eventually all feeds would go to ASAP. A much better idea is as follows:

Would it be possible to set a different ‘run time’ for each feed? In that way it would ‘distribute’ loads and posts over time having immediate benefits.

So the 10 feed scenario ‘staggered’ 30min, 60 min, 90 min.. 300 min.. would look like

Feed#      Next Run      Last Run

feed1        asap            30 min

feed2        60 min         60 min

feed3        90 min         120 min

feed10      300 min       330 min

The CRON would check every 30 minutes and pick up the ASAP job. I believe this is a valuable option all users would agree has many benefits: evenly distribute load and distribute posts over time.

Hope you will consider.

Regards,
cyber1

January 18, 2012
4:53 pm
Avatar
Admin
Forum Posts: 3097
Member Since:
July 2, 2009
sp_userofflinesmall Offline

With your example:

1) First run:

Feed#      Next Run      Last Run

feed1        asap            30 min

feed2        60 min         60 min

2) Second run (after 30 minutes):

Feed#      Next Run      Last Run

feed1        asap            30 min

feed2        asap            90 min

3) Third run (after another 30 minutes):

Feed#      Next Run      Last Run

feed1        asap            30 min

feed2        asap            120 min

First feed will be always updated. Feed #2 and others – never.

January 19, 2012
8:12 am
Avatar
cyber1
Guest
Guests

Hello Admin,

 

Of course you know this product, but I don't understand your logic. Even if the 1st Feed gets updated at some point all the others will also. (what is Next Run, ive never seen this?)

 

As I can see It will be like this, here is a more detailed example showing 4 feeds over 6 iterations. Each feed runs at a different rate frequency([email protected], Login to see this link.

All feeds start together for illustration:

 

1) First run:

Feed#      Next Run      Last Run

feed1        30 min         0 min

feed2        60 min         0 min

feed3        90 min         0 min

feed4       120 min        0 min

*all feeds are run in this scenario starting out.

 

2) Second run (after 30 minutes):

Feed#      Next Run       Last Run

feed1       asap             30 min

feed2       30 min          30 min

feed3       60 min         30 min

feed4       90 min         30 min

*feed1 will run next

 

3) Third run (after another 30 minutes):

Feed#      Next Run      Last Run

feed1        30 min         0 min

feed2        asap            60 min

feed3        30 min         60 min

feed4       60 min          60 min

*feed 1 runs,feed2 is next

 

4) 4th run (after another 30 minutes):

Feed#      Next Run      Last Run

feed1        asap            30 min

feed2        60 min         0 min

feed3        asap            60 min

feed4        30 min         60 min

*feed2 runs, feeds1/3 are next

 

5) 5th run (after another 30 minutes):

Feed#      Next Run      Last Run

feed1        30 min          0 min

feed2        30 min         30 min

feed3        90 min         0 min

feed4        asap            90 min

*feed1 and 3 runs, feed4 is next

 

6) 6th run (after another 30 minutes):

Feed#      Next Run      Last Run

feed1        asap            30 min

feed2        asap            60 min

feed3        60 min         30 min

feed4        120 min       0 min

*feed4 runs, feeds1 and 2 are next.

 

7) and so on.. repeating the process.

 

Over the course of 6 iterations at least 1 of the 4 feeds will have run, and having run through all 4 feeds. If the interval of the feeds is spread out further this could avoid/limit running 2 at once.

 

Note: I'm assuming that when ASAP is reached, the next iteration runs the feed but the effect is the same regardless of what it is called.

 

Sincerely,

Cyber1

January 19, 2012
1:28 pm
Avatar
Admin
Forum Posts: 3097
Member Since:
July 2, 2009
sp_userofflinesmall Offline

Your points (2) – (7). Are not correct. Try to emulate them mentally like if you were a computer. The wrong behaviour starts from here:

2) Second run (after 30 minutes):

Feed#      Next Run       Last Run
feed2       30 min        30 min
The “Last Run” for feed2 will be zero because it hasn’t been updated in at the 1st run.

and here:

3) Third run (after another 30 minutes):

Feed#      Next Run      Last Run
feed1        30 min         0 min
The “Next Run” for feed1 will be “asap”, and the “Last Run” will be 30 min, because exactly 30 minutes passed since the 2st run.

This is not related to my product. It’s just a simple logic.

January 19, 2012
7:18 pm
Avatar
cyber1
Guest
Guests

Hello Admin,

Doesn’t the code pick up the ‘next’ item(s) in the list that say’s ‘asap’? I really I don’t understand all the subtleties of how the computer functions as I am not the developer. You’d really need to show me through the 6 iterations for me to fully understand this.

My point here is simply having indepedent run times for each feed to ‘stagger’ so all don’t run/clump up and run at the same time. This would have many benefits not the least of which is minimize server resources for any single execution.

For myself having up to 12 rss feeds i’m finding lots of performance/timing related issues running all the jobs together. Having things ‘spread out’ so let’s say 1 or 2 of the 12 feeds run at a time would be a great difference in usability of  the product.

I’m sure this is a reasonable request as it is just a benefit to running the feeds which everyone would benefit from.

I hope you can consider this.

Sincerely,
cyber1

January 19, 2012
9:04 pm
Avatar
Admin
Forum Posts: 3097
Member Since:
July 2, 2009
sp_userofflinesmall Offline

cyber1 said:

Hello Admin,

Doesn't the code pick up the 'next' item(s) in the list that say's 'asap'? I really I don't understand all the subtleties of how the computer functions as I am not the developer. You'd really need to show me through the 6 iterations for me to fully understand this.

You don't have to be a developer to understand this. When scripts checks the statues of every feed, it does update it (the feed) in case if status is “ASAP”. Isn't it obvious?

January 20, 2012
8:56 am
Avatar
cyber1
Guest
Guests
10sp_permalink sp_print
0

Hello Admin,

 

No this is not obvious to me.. why would it be? I am just your customer.. not your developer or programer of the logic of the application.

 

If as you say it updates when 'ASAP' then we are saying the same thing. My point is to allow different update intervals for each feed, so not all the feeds run at once. I think this is a reasonable request considering the critical importance of server performance, SEO etc.

 

If you take the time to review my sample above for all 4 feeds I think you'll see i make a valid point.. having different frequency of the feeds will avoid running all the feeds at the same time. No solution is perfect, the CRON interval will always impact this.. but at least this method gives users a way to 'spread out' the run time of the feeds.

“Is there some reason you are opposed to improving the distribution of the feed run times? I'd much rather have a solution than back and forth as to my knowledge of the product which is limited to layman's knowledge..”

 

The sample you gave needs to show the 4 feeds over 6 iterations.. I think you'll see the feeds are distributed to some extend.. if not please show me a way this can be done, I'm only looking for a solution.. not a test of my intellect.

 

Sincerely,

cyber1

Forum Timezone: Europe/Moscow

Most Users Ever Online: 105

Currently Online:
3 Guest(s)

Currently Browsing this Page:
1 Guest(s)

Top Posters:

ninja321: 81

Freedom: 61

B8europe: 54

saviulisse67: 45

adultdenim: 44

allgoodies: 38

Member Stats:

Guest Posters: 350

Members: 1927

Moderators: 0

Admins: 1

Forum Stats:

Groups: 1

Forums: 2

Topics: 1328

Posts: 6627

Administrators: CyberSEO: 3097