(Migrated) Schedules jobs execute twice

(This message has been automatically imported from the retired mailing list)

Hi,

When I schedule a job with the Scheduler it always runs twice. I guess it h=
as something to do with the fact that we a running a cluster with two Zato =
processes.

How can we ensure that scheduled jobs are only run once?

Regards,

Finn Gruwier Larsen

Forbrugerr?det Taenk er en uafhaengig medlemsorganisation, der arbejder for=
et Danmark, hvor alle forbrugere kan traeffe et trygt valg.
F? nyheder, informationer om test, tilbud og gode r?d 1-2 gange om ugen. Ti=
lmeld dig vores nyhedsbreve p? taenk.dk/nyhedsbrev

Hi guys, I noticed the same thing.
I created a simple interval-based job and logged the results. I noticed a
double log on the first interval only and then it seemed to work itself
out.

2014-06-10 08:19:00,004 - INFO - 28153:Thread-46 - apscheduler.scheduler:22

  • Job “run my service (trigger: interval[0:01:00], next run at: 2014-06-10
    12:20:00)” executed successfully
    2014-06-10 08:19:00,006 - INFO - 28152:Dummy-5 -
    test-service.get-client-details:22 - Caught notification from scheduler
    2014-06-10 08:19:00,008 - INFO - 28152:Dummy-6 -
    test-service.get-client-details:22 - Caught notification from scheduler

I thought I had done something wrong. I installed from the ubuntu repo via
apt-get. I’m running the latest Linux Mint.

regards,
Brian

On Tue, Jun 10, 2014 at 6:27 AM, Dariusz Suchojad dsuch@zato.io wrote:

On 06/10/2014 12:23 PM, Finn Gruwier Larsen wrote:

I thought this was a general issue, but I can understand from your mail
that it must
be very specific to our environment (which you know in detail already).
So I’ll send a request to you directly.

Thanks and nope - servers should certainly not send double scheduler
tasks regardless of how many servers or processes there are.


Dariusz Suchojad

https://zato.io
ESB, SOA, REST, APIs and cloud integrations in Python

Yes, in fact, when I read Finns message, I retried it from scratch - wrote
a 2 liner Service and scheduled a 1 min interval from some date/time in the
future.

On Tue, Jun 10, 2014 at 8:29 AM, Dariusz Suchojad dsuch@zato.io wrote:

On 06/10/2014 02:22 PM, Brian Martin wrote:

I thought I had done something wrong. I installed from the ubuntu repo
via
apt-get. I’m running the latest Linux Mint.

Thanks Brian - can you reproduce it easily? Does it happen for any job
the first time it’s executed?

thanks a lot,


Dariusz Suchojad

https://zato.io
ESB, SOA, REST, APIs and cloud integrations in Python

Here’s a bzipped log - I truncated the top half so it starts around the
time I hot-deployed. Looks like its getting scheduled a second time when
you create it in the web interface ( you’ll see it getting logged twice
that its creating the scheduler - that apscheduler thing) . But, you’ll
notice it gets executed twice every other interval. I set it to repeat
every minute in the web interface.

grep the log for: “test-service.get-client-details”

On Tue, Jun 10, 2014 at 8:55 AM, Dariusz Suchojad dsuch@zato.io wrote:

On 06/10/2014 02:37 PM, Brian Martin wrote:

Yes, in fact, when I read Finns message, I retried it from scratch -
wrote
a 2 liner Service and scheduled a 1 min interval from some date/time in
the
future.

Can you please send across a snippet from server.log showing, on INFO
level for now, what it looks like when a job is scheduled and invoked?

Please check the attachment which shows a job ‘1’ scheduled to invoke
service ‘zato.ping’.

It’s git master.

thanks,


Dariusz Suchojad

https://zato.io
ESB, SOA, REST, APIs and cloud integrations in Python

On 06/10/2014 10:21 AM, Finn Gruwier Larsen wrote:

When I schedule a job with the Scheduler it always runs twice. I guess it has something to do with the fact that we a running a cluster with two Zato processes.

Hi Finn, can you please check the following?

  • Which Zato version is it? If git, what commit?

  • How do you mean two processes? One server with two gunicorn workers?
    Two servers with one gunicorn worker each?

  • How do you mean they run twice? Each server (if there are two) runs
    the job?

  • Do they run in parallel or in succession?

  • Do internal jobs, such as
    ’zato.server.cluster-wide-singleton-keep-alive’, also run twice?

thanks,

Hi Dariusz,

I thought this was a general issue, but I can understand from your mail that it must be very specific to our environment (which you know in detail already). So I’ll send a request to you directly.

Cheers,

Finn

-----Oprindelig meddelelse-----
Fra: Dariusz Suchojad [mailto:dsuch@zato.io]
Sendt: 10. juni 2014 11:31
Til: Finn Gruwier Larsen; zato-discuss@lists.zato.io
Emne: Re: [Zato-discuss] Schedules jobs execute twice

On 06/10/2014 10:21 AM, Finn Gruwier Larsen wrote:

When I schedule a job with the Scheduler it always runs twice. I guess it has something to do with the fact that we a running a cluster with two Zato processes.

Hi Finn, can you please check the following?

  • Which Zato version is it? If git, what commit?

  • How do you mean two processes? One server with two gunicorn workers?
    Two servers with one gunicorn worker each?

  • How do you mean they run twice? Each server (if there are two) runs the job?

  • Do they run in parallel or in succession?

  • Do internal jobs, such as
    ’zato.server.cluster-wide-singleton-keep-alive’, also run twice?

thanks,


Dariusz Suchojad

https://zato.io
ESB, SOA, REST, APIs and cloud integrations in Python

On 06/10/2014 12:23 PM, Finn Gruwier Larsen wrote:

I thought this was a general issue, but I can understand from your mail that it must
be very specific to our environment (which you know in detail already).
So I’ll send a request to you directly.

Thanks and nope - servers should certainly not send double scheduler
tasks regardless of how many servers or processes there are.

I restarted it and it appears to stop the duplication but something else is
not working properly. The event is not firing in my service anymore ( or
its not logging it) . I am going to play with it some more. I’ll keep you
posted.

On Tue, Jun 10, 2014 at 9:34 AM, Dariusz Suchojad dsuch@zato.io wrote:

Thanks Brian - just to confirm it.

This is back to normal, i.e. no duplicates, when you stop the
environment and start it again?

I’m just trying to pinpoint the moment it begins, if it’s because they
are stored twice in the config store or because they’re created twice
from web-admin.

thanks a lot again,


Dariusz Suchojad

https://zato.io
ESB, SOA, REST, APIs and cloud integrations in Python

Hi everybody
I saw this behavior in the client tests and I notice that this is something
related to the number of servers in the cluster, maybe a broker message
related thing because when a job is get to be executed it is published in a
Redis channel that and I think that all the servers are pulling the task
message. I my environment I use only one server to execute scheduled jobs
that’s why I am not seeing it now but I create the test cluster using
quickstart command and my clients reported me that they are seeing double
mail notification when the task is executed

On Tue, Jun 10, 2014 at 9:58 PM, Brian Martin bmartin@qualbe.com wrote:

I restarted it and it appears to stop the duplication but something else
is not working properly. The event is not firing in my service anymore ( or
its not logging it) . I am going to play with it some more. I’ll keep you
posted.

On Tue, Jun 10, 2014 at 9:34 AM, Dariusz Suchojad dsuch@zato.io wrote:

Thanks Brian - just to confirm it.

This is back to normal, i.e. no duplicates, when you stop the
environment and start it again?

I’m just trying to pinpoint the moment it begins, if it’s because they
are stored twice in the config store or because they’re created twice
from web-admin.

thanks a lot again,


Dariusz Suchojad

https://zato.io
ESB, SOA, REST, APIs and cloud integrations in Python


Brian Martin
Software Developer
Qualbe Marketing Group
(682) 593-5985 - Office
(613) 240-7676 - Cell

That’s good news. Thanks for fixing so quickly. Maybe I can drop my Maven
book finally. :wink:
On Jun 11, 2014 5:44 AM, “Dariusz Suchojad” dsuch@zato.io wrote:

On 06/11/2014 04:33 AM, Axel Mendoza Pupo wrote:

I my environment I use only one server to execute scheduled jobs
that’s why I am not seeing it now but I create the test cluster using
quickstart command and my clients reported me that they are seeing double
mail notification when the task is executed

Hi folks,

this has been just fixed.

If you use git master - run a git pull.

If you are on 1.1 - assuming it’s installed in /opt/zato/1.1

$ cd /opt/zato/1.1
$ ./hotfixman.sh

This will pull a hotfix that deals with double scheduler tasks.

The underlying reason was:

  • There are separate schedulers for internal and user-defined jobs
  • When created from web-admin or API, internal schedulers would run user
    jobs and that should not be the case
  • However, this would happen only until servers were first time restarted

Given that during development or tests most servers are restarted from
time to time it took that long to spot the issue and fix it.

Thanks to everyone to helped out in resolving it!


Dariusz Suchojad

https://zato.io
ESB, SOA, REST, APIs and cloud integrations in Python