(Migrated) APScheduller datetime without timezone bug

forum-admin · June 20, 2016, 4:17pm

(This message has been automatically imported from the retired mailing list)

Hi again
I just notice a bug in APScheduller use of datetime without timezone, after
a search I found this:

Basically APScheduller used by Zato is not timezone aware, affecting the
schedulling of task that have start_date bellow of the UTC time, for
example, for those servers who have their timezone datetime in UTC + 5
can’t schedule task that are bellow the range of now plus 5 hours, any task
in that range of time is not getting scheduled by APScheduller because it
says that the task will never run.

That link says that this will be fixed in APScheduller 3.0, but until then
what will be a possible solution in conjuntion with Zato??

I will continue digging, but any lights will be good

Cheers

forum-admin · June 20, 2016, 4:17pm

to the list

First of all, this errors are shown when I use the webservice client API,

When I tried to create a job using a datetime with timezone this is the
error log

File “/opt/zato/zato-1.1/zato-server/src/zato/server/scheduler.py”, line
96, in create_edit
handler(job_data, broker_msg_type)
File “/opt/zato/zato-1.1/zato-server/src/zato/server/scheduler.py”, line
107, in create_one_time
SCHEDULER_JOB_TYPE.ONE_TIME], name=job_data.name)
File
"/opt/zato/zato-1.1/eggs/APScheduler-2.0.3-py2.7.egg/apscheduler/scheduler.py",
line 300, in add_date_job
return self.add_job(trigger, func, args, kwargs, **options)
File
"/opt/zato/zato-1.1/eggs/APScheduler-2.0.3-py2.7.egg/apscheduler/scheduler.py",
line 274, in add_job
self._real_add_job(job, jobstore, True)
File
"/opt/zato/zato-1.1/eggs/APScheduler-2.0.3-py2.7.egg/apscheduler/scheduler.py",
line 230, in _real_add_job
job.compute_next_run_time(datetime.utcnow())
File
"/opt/zato/zato-1.1/eggs/APScheduler-2.0.3-py2.7.egg/apscheduler/job.py",
line 74, in compute_next_run_time
self.next_run_time = self.trigger.get_next_fire_time(now)
File
"/opt/zato/zato-1.1/eggs/APScheduler-2.0.3-py2.7.egg/apscheduler/triggers/simple.py",
line 9, in get_next_fire_time
if self.run_date >= start_date:
TypeError: can’t compare offset-naive and offset-aware datetimes

and after formating the datetime to “%Y-%m-%d %H:%M:%S” to obtain a normal
datetime without timezone, this is the error log

2014-05-30 23:02:47,928 - INFO - 11517:Dummy-9 -
zato.scheduler.job.create:22 -
cid:[K011397624108052907815571501002669803050],
name:[zato.scheduler.job.create], SIO r$
2014-05-30 23:02:47,945 - ERROR - 11516:Dummy-93 - zato.server.scheduler:22

Caught exception [Traceback (most recent call last):
File “/opt/zato/zato-1.1/zato-server/src/zato/server/scheduler.py”, line
96, in create_edit
handler(job_data, broker_msg_type)
File “/opt/zato/zato-1.1/zato-server/src/zato/server/scheduler.py”, line
107, in create_one_time
SCHEDULER_JOB_TYPE.ONE_TIME], name=job_data.name)
File
"/opt/zato/zato-1.1/eggs/APScheduler-2.0.3-py2.7.egg/apscheduler/scheduler.py",
line 300, in add_date_job
return self.add_job(trigger, func, args, kwargs, **options)
File
"/opt/zato/zato-1.1/eggs/APScheduler-2.0.3-py2.7.egg/apscheduler/scheduler.py",
line 274, in add_job
self._real_add_job(job, jobstore, True)
File
"/opt/zato/zato-1.1/eggs/APScheduler-2.0.3-py2.7.egg/apscheduler/scheduler.py",
line 232, in _real_add_job
raise ValueError(‘Not adding job since it would never be run’)
ValueError: Not adding job since it would never be run

so I think that zato is passing my start_date to APScheduler as is, and
APScheduler is running his time without timezone, saying that my timezone
is UTC -5, APScheduler is running in localtime +5, I also check this
showing datetime.now() used to add the job in APScheduler and it confirm
that

On Sat, May 31, 2014 at 2:52 AM, Dariusz Suchojad dsuch@zato.io wrote:

On 05/31/2014 06:47 AM, Axel Mendoza Pupo wrote:

Basically APScheduller used by Zato is not timezone aware, affecting the
schedulling of task that have start_date bellow of the UTC time, for
example, for those servers who have their timezone datetime in UTC + 5
can’t schedule task that are bellow the range of now plus 5 hours, any
task
in that range of time is not getting scheduled by APScheduller because it
says that the task will never run.

That link says that this will be fixed in APScheduller 3.0, but until
then
what will be a possible solution in conjuntion with Zato??

Hi Axel,

APScheduler isn’t but Zato is, that’s the goal of these patches that get
applied when you install Zato binaries:

https://github.com/zatosource/zato/blob/master/code/patches/apscheduler/triggers/interval.py.diff

https://github.com/zatosource/zato/blob/master/code/patches/apscheduler/scheduler.py.diff

They make Zato servers always compute time of the next job’s run in UTC
rather than in local time.

Internally, all Zato’s own services operate in UTC only and that
includes the admin API.

Web-admin on the other hand always accepts input in user’s local timezone
…

https://zato.io/docs/web-admin/settings/main.html

… and it’s translated into UTC internally before a service to create a
job is invoked.

In other words, it’s UTC everywhere except for the user interface.

But if you say it isn’t, can you please post an excerpt from server.log
showing:

Input/output of zato.scheduler.job.create - the service that is used
to schedule a job

An exact warning message

thanks a lot,

–
Dariusz Suchojad

https://zato.io
ESB, SOA, REST, APIs and cloud integrations in Python

forum-admin · June 20, 2016, 4:17pm

On 05/31/2014 06:47 AM, Axel Mendoza Pupo wrote:

Basically APScheduller used by Zato is not timezone aware, affecting the
schedulling of task that have start_date bellow of the UTC time, for
example, for those servers who have their timezone datetime in UTC + 5
can’t schedule task that are bellow the range of now plus 5 hours, any task
in that range of time is not getting scheduled by APScheduller because it
says that the task will never run.

That link says that this will be fixed in APScheduller 3.0, but until then
what will be a possible solution in conjuntion with Zato??

Hi Axel,

APScheduler isn’t but Zato is, that’s the goal of these patches that get
applied when you install Zato binaries:

https://github.com/zatosource/zato/blob/master/code/patches/apscheduler/triggers/interval.py.diff

https://github.com/zatosource/zato/blob/master/code/patches/apscheduler/scheduler.py.diff

They make Zato servers always compute time of the next job’s run in UTC
rather than in local time.

Internally, all Zato’s own services operate in UTC only and that
includes the admin API.

Web-admin on the other hand always accepts input in user’s local timezone …

https://zato.io/docs/web-admin/settings/main.html

… and it’s translated into UTC internally before a service to create a
job is invoked.

In other words, it’s UTC everywhere except for the user interface.

But if you say it isn’t, can you please post an excerpt from server.log
showing:

Input/output of zato.scheduler.job.create - the service that is used
to schedule a job
An exact warning message

thanks a lot,

forum-admin · June 20, 2016, 4:17pm

On 05/31/2014 06:47 AM, Axel Mendoza Pupo wrote:

Basically APScheduller used by Zato is not timezone aware, affecting the
schedulling of task that have start_date bellow of the UTC time, for
example, for those servers who have their timezone datetime in UTC + 5
can’t schedule task that are bellow the range of now plus 5 hours, any task
in that range of time is not getting scheduled by APScheduller because it
says that the task will never run.

That link says that this will be fixed in APScheduller 3.0, but until then
what will be a possible solution in conjuntion with Zato??

Hi Axel,

APScheduler isn’t but Zato is, that’s the goal of these patches that get
applied when you install Zato binaries:

https://github.com/zatosource/zato/blob/master/code/patches/apscheduler/triggers/interval.py.diff

https://github.com/zatosource/zato/blob/master/code/patches/apscheduler/scheduler.py.diff

They make Zato servers always compute time of the next job’s run in UTC
rather than in local time.

Internally, all Zato’s own services operate in UTC only and that
includes the admin API.

Web-admin on the other hand always accepts input in user’s local timezone …

https://zato.io/docs/web-admin/settings/main.html

… and it’s translated into UTC internally before a service to create a
job is invoked.

In other words, it’s UTC everywhere except for the user interface.

But if you say it isn’t, can you please post an excerpt from server.log
showing:

Input/output of zato.scheduler.job.create - the service that is used
to schedule a job
An exact warning message

thanks a lot,

forum-admin · June 20, 2016, 4:17pm

example of my payload request generated at today at 10:06 am to be executed
today at 10:09:48, this is without timezone, the one that produce
"ValueError: Not adding job since it would never be run"

{‘cluster_id’: 1, ‘name’: u’Batch queries for BANCO DE LA PRODUCCION S.A.
at 2014-05-31 10:05:27.187315-05:00’, ‘service’:
u’openerp-job-generic-web-service-3.job_generic_web_service’, ‘extra’:
"{‘ftp_dir’: u’BANCO DE LA PRODUCCION S.A.’, ‘file_url’: u’
http://192.168.16.41:8069?db=rdc_test2#login=produccionreferencia&model=ir.attachment&id=23’,
‘filename’: u’Resultado-Procesos-Masivos-23-BANCO DE LA PRODUCCION
S.A.-2014-05-31 10:09:48.648989-05:00.zip’, ‘service_id’: 17, ‘date’:
‘2014-05-31T10:09:48.648989-05:00’, ‘partner’: {‘name’: u’BANCO DE LA
PRODUCCION S.A.’, ‘email’: False}, ‘to_process’:
[u’0101762748|3801110018902011’, u’0101956324|3801130002352012’, u’0101997401|3805060000772012’, u’0102069465|3802060024572011’, u’0102224052|3802110007112011’, u’0102845450|3802110004722011’, u’0102856978|3801130004462012’, u’0103284246|3801110024292012’, u’0103927554|3804060006202012’, u’0105965602|3801130003382012’]}",
‘job_type’: ‘one_time’, ‘repeats’: 1, ‘is_active’: True, ‘start_date’:
‘2014-05-31 10:09:48’}

the APScheduler compare with 2014-05-31 15:09:48.166245

On Sat, May 31, 2014 at 9:33 AM, Dariusz Suchojad dsuch@zato.io wrote:

On 05/31/2014 04:31 PM, Axel Mendoza Pupo wrote:

First of all, this errors are shown when I use the webservice client API,

When I tried to create a job using a datetime with timezone this is the
error log

Can you please attach a request you’re invoking the service with?

thanks,

–
Dariusz Suchojad

https://zato.io
ESB, SOA, REST, APIs and cloud integrations in Python

forum-admin · June 20, 2016, 4:17pm

By the way the job task is created in zato but not scheduled in APScheduler
to be run. Also if I go through the webadmin interface and try to execute
it manually, the job is not executed neither, here is the log:
2014-05-31 10:17:44,584 - WARNING - 11796:Dummy-2899 -
zato.server.scheduler:22 - Job [Batch queries for BANCO DE LA PRODUCCION
S.A. at 2014-05-31 10:05:27.187315-05:00] is not scheduled, could not
execute it

On Sat, May 31, 2014 at 10:11 AM, Axel Mendoza Pupo aekroft@gmail.com
wrote:

example of my payload request generated at today at 10:06 am to be
executed today at 10:09:48, this is without timezone, the one that produce
"ValueError: Not adding job since it would never be run"

{‘cluster_id’: 1, ‘name’: u’Batch queries for BANCO DE LA PRODUCCION S.A.
at 2014-05-31 10:05:27.187315-05:00’, ‘service’:
u’openerp-job-generic-web-service-3.job_generic_web_service’, ‘extra’:
"{‘ftp_dir’: u’BANCO DE LA PRODUCCION S.A.’, ‘file_url’: u’
http://192.168.16.41:8069?db=rdc_test2#login=produccionreferencia&model=ir.attachment&id=23’,
‘filename’: u’Resultado-Procesos-Masivos-23-BANCO DE LA PRODUCCION
S.A.-2014-05-31 10:09:48.648989-05:00.zip’, ‘service_id’: 17, ‘date’:
‘2014-05-31T10:09:48.648989-05:00’, ‘partner’: {‘name’: u’BANCO DE LA
PRODUCCION S.A.’, ‘email’: False}, ‘to_process’:
[u’0101762748|3801110018902011’, u’0101956324|3801130002352012’, u’0101997401|3805060000772012’, u’0102069465|3802060024572011’, u’0102224052|3802110007112011’, u’0102845450|3802110004722011’, u’0102856978|3801130004462012’, u’0103284246|3801110024292012’, u’0103927554|3804060006202012’, u’0105965602|3801130003382012’]}",
‘job_type’: ‘one_time’, ‘repeats’: 1, ‘is_active’: True, ‘start_date’:
‘2014-05-31 10:09:48’}

the APScheduler compare with 2014-05-31 15:09:48.166245

On Sat, May 31, 2014 at 9:33 AM, Dariusz Suchojad dsuch@zato.io wrote:

On 05/31/2014 04:31 PM, Axel Mendoza Pupo wrote:

First of all, this errors are shown when I use the webservice client
API,

When I tried to create a job using a datetime with timezone this is the
error log

Can you please attach a request you’re invoking the service with?

thanks,

–
Dariusz Suchojad

https://zato.io
ESB, SOA, REST, APIs and cloud integrations in Python

forum-admin · June 20, 2016, 4:17pm

here is another request payload example with start_date datetime with
timezone, the one that produce “TypeError: can’t compare offset-naive and
offset-aware datetimes”.

All of this request are executed
agaist ‘/zato/json/zato.scheduler.job.create’

{‘cluster_id’: 1, ‘name’: u’Batch queries for BANCO DE LA PRODUCCION S.A.
at 2014-05-31 10:27:
22.986778-05:00’, ‘service’:
u’openerp-job-generic-web-service-3.job_generic_web_service’, ‘extra’:
"{‘ftp_dir’: u’BANCO DE LA PRODUCCION S.A.’, ‘file_url’: u’
http://192.168.16.41:8069?
db=rdc_test2#login=produccionreferencia&model=ir.attachment&id=24’,
‘filename’: u’Resultado-
Procesos-Masivos-24-BANCO DE LA PRODUCCION S.A.-2014-05-31
10:32:11.556816-05:00.zip’,
‘service_id’: 17, ‘date’: ‘2014-05-31T10:32:11.556816-05:00’, ‘partner’:
{‘name’: u’BANCO DE LA
PRODUCCION S.A.’, ‘email’: False}, ‘to_process’:
[u’0101762748|3801110018902011’, u’0101956324|3801130002352012’, u’0101997401|3805060000772012’]}",
‘job_type’: ‘one_time’,
‘repeats’: 1, ‘is_active’: True, ‘start_date’:
‘2014-05-31T10:32:11.556816-05:00’}

On Sat, May 31, 2014 at 10:21 AM, Axel Mendoza Pupo aekroft@gmail.com
wrote:

By the way the job task is created in zato but not scheduled in
APScheduler to be run. Also if I go through the webadmin interface and try
to execute it manually, the job is not executed neither, here is the log:
2014-05-31 10:17:44,584 - WARNING - 11796:Dummy-2899 -
zato.server.scheduler:22 - Job [Batch queries for BANCO DE LA PRODUCCION
S.A. at 2014-05-31 10:05:27.187315-05:00] is not scheduled, could not
execute it

On Sat, May 31, 2014 at 10:11 AM, Axel Mendoza Pupo aekroft@gmail.com
wrote:

example of my payload request generated at today at 10:06 am to be
executed today at 10:09:48, this is without timezone, the one that produce
"ValueError: Not adding job since it would never be run"

{‘cluster_id’: 1, ‘name’: u’Batch queries for BANCO DE LA PRODUCCION S.A.
at 2014-05-31 10:05:27.187315-05:00’, ‘service’:
u’openerp-job-generic-web-service-3.job_generic_web_service’, ‘extra’:
"{‘ftp_dir’: u’BANCO DE LA PRODUCCION S.A.’, ‘file_url’: u’
http://192.168.16.41:8069?db=rdc_test2#login=produccionreferencia&model=ir.attachment&id=23’,
‘filename’: u’Resultado-Procesos-Masivos-23-BANCO DE LA PRODUCCION
S.A.-2014-05-31 10:09:48.648989-05:00.zip’, ‘service_id’: 17, ‘date’:
‘2014-05-31T10:09:48.648989-05:00’, ‘partner’: {‘name’: u’BANCO DE LA
PRODUCCION S.A.’, ‘email’: False}, ‘to_process’:
[u’0101762748|3801110018902011’, u’0101956324|3801130002352012’, u’0101997401|3805060000772012’, u’0102069465|3802060024572011’, u’0102224052|3802110007112011’, u’0102845450|3802110004722011’, u’0102856978|3801130004462012’, u’0103284246|3801110024292012’, u’0103927554|3804060006202012’, u’0105965602|3801130003382012’]}",
‘job_type’: ‘one_time’, ‘repeats’: 1, ‘is_active’: True, ‘start_date’:
‘2014-05-31 10:09:48’}

the APScheduler compare with 2014-05-31 15:09:48.166245

On Sat, May 31, 2014 at 9:33 AM, Dariusz Suchojad dsuch@zato.io wrote:

On 05/31/2014 04:31 PM, Axel Mendoza Pupo wrote:

First of all, this errors are shown when I use the webservice client
API,

When I tried to create a job using a datetime with timezone this is the
error log

Can you please attach a request you’re invoking the service with?

thanks,

–
Dariusz Suchojad

https://zato.io
ESB, SOA, REST, APIs and cloud integrations in Python

forum-admin · June 20, 2016, 4:17pm

I just test it and it’s work ok, sorry that I miss the UTC convention,
maybe the webadmin column need to be formated to show the localtime in the
scheduler list??

All is working ok without timezone, in UTC, thanks for the point.

Cheers

On Sat, May 31, 2014 at 10:40 AM, Dariusz Suchojad dsuch@zato.io wrote:

On 05/31/2014 05:32 PM, Axel Mendoza Pupo wrote:

here is another request payload example with start_date datetime with
timezone, the one that produce “TypeError: can’t compare offset-naive and
offset-aware datetimes”.

‘repeats’: 1, ‘is_active’: True, ‘start_date’:
‘2014-05-31T10:32:11.556816-05:00’}

Right, I’ve just updated the documentation and added the format
start_date should be in.

https://zato.io/docs/public-api/details/zato.scheduler.job.create.html

I’ve also added a note that start_date is in UTC. This was already
mentioned in the section on API’s conventions but it’s better to
emphasize it again.

https://zato.io/docs/public-api/intro.html#conventions