May 4, 2011

[Wish] for future release: NEW SCHEDULER. | Freeware Plugins | Support Forum

Avatar

Lost password?
Advanced Search

— Forum Scope —




— Match —





— Forum Options —





Minimum search word length is 3 characters - maximum search word length is 84 characters

sp_Related Related Topics sp_TopicIcon
[Wish] for future release: NEW SCHEDULER.
Topic Rating: 0 Topic Rating: 0 Topic Rating: 0 Topic Rating: 0 Topic Rating: 0 Topic Rating: 0 (0 votes) 
July 31, 2017
8:58 am
Avatar
s.popadiin@gmail.com
Member
Members
Forum Posts: 32
Member Since:
May 11, 2017
sp_UserOfflineSmall Offline

Hello,

I have been using CyberSEO and CyberSYN for few months with 30+ feeds to pull.

I also have tested few other plugins serving the same purpose.

I came to conclusion that most of Cyber problems like php overload, missed spins, missed translations, lost connection with DB, and maybe few more errors, come from the way scheduling of pulling feeds is done.

All problem come from two facts:
1. Feed pulling takes some time more than a minute and more.
2. Feed pulling takes considerable amount of resources.

Hence, feed pulls should be evenly distributed in time to allow every pull to finish before new starts.
This is not possible with current scheduling, because it has no fixed time schedule rather it is relative. For example it schedules a feed pull every 5 minutes not every exact 5 minutes of every hour. More about the scheduling problem has been discussed already here.
Login to see this link

I would like to suggest new scheduling that is completely CRON LIKE. If anybody still wants to use old scheduling, she could. For example the old each 5 minutes feed pull would result in 12 lines of new scheduling, ex. */5, */10 ….
But still will keep the order that is initially set of all feeds. More about why the order breaks you can read on the link above. So the new scheduling will not break any legacy configs, but will fix order of all feeds pull timing.

I think with proper integration of new scheduling nothing could be lost and much could be won.

For everybody convenience I am attaching a screenshot of another plugin with example of pull feed each 10 minutes. By the way the old scheduling options to pull feeds each 1,2 minutes are useless, maybe 3,4 as well. Anyway, could be left for legacy configs.
Login to see this link

 

The advantages of new scheduling are
1. that it will be in coherence with server cron times
2. this will not overlap feed pulls
3. that will not but stress on PHP or DB.
4. It will provide exact pulling time with enough time period between pulls so they can finish in a nicely fashion.

July 31, 2017
9:52 am
Avatar
CyberSEO
Admin
Forum Posts: 3678
Member Since:
July 2, 2009
sp_UserOfflineSmall Offline

The cron-like scheme won’t work for PHP. You should understand the difference between cron and PHP script (like CyberSEO or WordPress engine itself). Cron is a daemon which works 24/7 on your server and is waiting for its time every millisecond. If the time comes, cron triggers and does its task.

The PHP code is a way different thing. It’s a text script which does not work till it’s started by someone else, like if somebody has opened your site in the browser, a search engine has visited it or it was opened… by cron. Actually all that cron does, it just emulates opening of your WordPress site in the browser and nothing else.

So to be short, the CyberSEO scheduler can not schedule anything if it won’t be started by cron. It will work for a time which is needed to pull all the feeds and after that ill will literally die for the next time when someone or something will start it again. It doesn’t live in memory of your server like cron, because it’s just a PHP script. Thus it can’t run the scheduler in the specified time (just like a human can’t lift himself by his hair). And if you want to be able to pull some feed accurate to one minute (say in 4:32PM +/- 1 minute), you have to run cron every minute as well. Of course your server will be overused, because every minute the PHP script will be awaken to check if it’s time (like if it’s 4:32PM right now?) to pull the feed.

I see what you say, but it looks like you don’t understand how PHP scripts work and why it’s just impossible to make them work like cron, which is a resident executable program that listens to every tick of the system clock. The only way you can do it is to tell your cron to kick CyberSEO every 10 minutes or so to tell it: “hey buddy, go check your feeds, maybe it’s time to update one or a couple?”

July 31, 2017
1:39 pm
Avatar
s.popadiin@gmail.com
Member
Members
Forum Posts: 32
Member Since:
May 11, 2017
sp_UserOfflineSmall Offline

What you explain I understand completely. I know very well that cyberseo/syn cron just simulates site visit.

However, there is nothing impossible here.

Server Cron kicks php, than php checks time, if returned value is time, php continues with heavy feed pulling.

Currently it works exactly like this, BUT cyberseo checks if it is time by minutes back counter (if backcounting has finished, or overdue what is the ASAP stalled feeds problem). It rather should check … What is the time, is it 10:05, is it 11:05, is it 12:05 …., if yes run feed pulls which are scheduled with each hour past 5.

This way no matter if you use server cron or only WP cron, PHP would know exactly when to execute. Currently CyberSEO php runs ambiguously on no exact time. The practice shows it messes it all up.

Scheduling in its nature means knowing exact time. With CyberSEO/SYN you can make it work in exact time only if you run cron each minute.

I dont want to say CyberSEO is not god or anything else is better. CyberSEO has its own advantages, but you can check WPeMatico free version.
Their scheduler works and makes order in all timings.

August 1, 2017
3:31 am
Avatar
CyberSEO
Admin
Forum Posts: 3678
Member Since:
July 2, 2009
sp_UserOfflineSmall Offline

Login to see this link said

This way no matter if you use server cron or only WP cron, PHP would know exactly when to execute. Currently CyberSEO php runs ambiguously on no exact time. The practice shows it messes it all up.

I see you still don’t understand it. Currently CyberSEO runs the exact time, as it defined by the cron tab. Your cron it runs it the exact second as it supposed to.

Once again, you can not make CyberSEO to run at some exact time if you don’t run it at that exact time via cron. PHP script is not a resident program and it can not launch itself.

If you think you know something I don’t know, please show me an example of a PHP code which runs say once a day exactly at 11:10AM and prints “Hello Word!” and works w/o sever cron. Like this:

Login to see the code

No theory please, just one simple example. Once you try to do it, you will understand what I’m talking about.

August 2, 2017
1:48 am
Avatar
s.popadiin@gmail.com
Member
Members
Forum Posts: 32
Member Since:
May 11, 2017
sp_UserOfflineSmall Offline

I did not say PHP executes itself.
I did say:
In 10:05 Cron or a user visit wakes up PHP. PHP asks is time equal to pull feed time, if Yes – pull, if not Die. If scheduling options are at 5 minute increments (who would need less!?!) and a server cron is done every 5 minutes each hour. PHP waking up and scheduling will coincide.

Your script works like 

In 10:05 Cron or a user visit wakes up PHP. PHP asks is time expired (pull time equal or less server time), if Yes pull feed, if No die. And it receives Yes on 20 feeds, so 20 feeds start to be pulled.

This is a huge difference (exactly this pull time LESS server time). The first case PHP executes only when it is told to. Second case PHP executes under condition that is always true (period has long time ago expired) until it succeeds in order to start next back counting period. So if it never manage to succeed, it will keep trying every time cron wakes PHP. This is very bad logic that progressively loads the server. And you know this already. The forum is full with problem posts.

You need to limit feed pulls outside any other time than the EXACT specified time. That has nothing to do with who and how triggers the action.

The condition LESS limits your script from any scalability. Lets say I have 120 feeds to pull. In correct scenario I would spread them across 10 hours. Lets say second feed has bigger latency and huge images. In correct scenario I would not care. In your case that feed will ruin all feeds, because it will be expired, it will expire all other feeds and all will end up in a mess. 

August 2, 2017
2:43 am
Avatar
CyberSEO
Admin
Forum Posts: 3678
Member Since:
July 2, 2009
sp_UserOfflineSmall Offline

So in your case cron should be triggered once per every 5 minutes. Right? What about those who would like to update a few feeds, say onece per 8 minutes and someone else – every 13 minutes and so on?

Now what if cron server time and WordPress site time are different (usually they always different if those are located at two different hosts)? Say it’s 10:05 for cron and 16:03 for the host where your WordPress site is located. Should the feed which is scheduled at 10:05 be pulled? How CyberSEO will know that if it was started by cron at 16:03 has to process the feed which is set to be pulled at 10:05?

It sounds easy in theory, but not IRL.

August 2, 2017
4:26 am
Avatar
s.popadiin@gmail.com
Member
Members
Forum Posts: 32
Member Since:
May 11, 2017
sp_UserOfflineSmall Offline

Simplicity and durability is achieved when main use case scenario is perceived. Hence, there will be no 8 or 13 minutes. Only 5 minutes increments. By this logic you have to make the ability for those who would like to pull at 3 minutes 55 seconds. This non sense.

Yes, Cron should be setup to at least 5 minutes. A user can decide depending on his feed pulls preferences.
One user can have 120 feeds and setup a 5 minute cron.
Another user can have 12 feeds and setup 1 hour cron.
This is the order conditions, nothing bad about it. In the case right now the only real working scenario is setting cron to 1 minute, or having under 10 feeds. I think majority of your user base has more than 10 feeds.

 

Second – the time difference. You are right, usually they are different. But again this is out of the box logic. It is an external system that has its own setup.

The case where an alien cron (cron not local to WP install) is used, is very exotic scenario and this should also remain out of the logic. 

Usually time difference comes in hours, not in minutes. Although a case with minutes is possible. I check my server, WP and server time minutes are 1:1.

If you are that concerned with this time differential, you can check this once and create a variable to compare with pull time instead server time. If you want to cover exotic scenarios, write a cron testing page where cron beacon will be recorded and it will show the user if cron is always late and give user possibility to insert the time differential. However, I think this is not necessary.

Once again I advise you to check how your rivals have achieved it. There is nothing to feel bad about it. SyberCEO is a great plugin with many nice tools, but needs a little fix up. Otherwise, pity, non of the great tools can not be used. For example translation. I turned off 29 feeds and left only one for translation. Now I get all posts translated without random non translated. So it is not Google API problem, its the server time our and peak overloading, because of feed pull congestion.

Forum Timezone: Europe/Amsterdam

Most Users Ever Online: 541

Currently Online:
15 Guest(s)

Currently Browsing this Page:
1 Guest(s)

Top Posters:

ninja321: 84

s.baryshev.aoasp: 64

Freedom: 61

MediFormatica: 49

B8europe: 47

saviulisse67: 45

Member Stats:

Guest Posters: 338

Members: 2636

Moderators: 0

Admins: 1

Forum Stats:

Groups: 1

Forums: 5

Topics: 1537

Posts: 7758

Newest Members:

kevin.crywolf, kevin.cooper.boonenewsmedia, FloSti, t.schleiser, worldnetsites, janodamek

Administrators: CyberSEO: 3678