(Migrated) zato service

(This message has been automatically imported from the retired mailing list)

hi
I would like to create service that run permanently , have state, do processing on their own, send messages and write to the databases and optionally respond to requests . kind of like to traditional concept of a service like a unix service, database etc…
I realize that zato can do similar things with the scheduler and zato services etc… but I could not find the best practice for long running services that do work without receiving any requests…
I was thinking of running a job every hour that would just check if it’s already running and abort or start but I don’t know if there are issues with jobs that never exit, (timeout…)
my idea of a ‘service’ is closer to services in ginkgo for example, but zato is a much richer framework, so I would rather use zato if possible.

thank you

On 14/02/15 18:49, Christian_cadieux wrote:

I would like to create service that run permanently ,
have state, do processing on their own, send messages and write
to the databases and optionally respond to requests .

Hi Christian,

if I understand it correctly, the reason you’d like for the service to
be a long-running one is that it be able to keep its own state around?

If so, you could simply store the state in Redis so that after each
invocation it looked it up and updated as needed.

Actually, I’m not 100% if this is what you need because you also said
you’d like for the service to work without external input?

Can you please describe your use case step-by-step so we can together
find the best approach?

thanks,

On 15/02/15 16:43, Christian_cadieux wrote:

i have lots of example:

Hi Christian,

a thing to keep in mind is that Zato is based on gunicorn using gevent
which, ultimately on the lowest level, uses libev. In other words, it’s
a set of asynchronous processes running in parallel, each one capable of
accepting hundreds of requests a second.

  • the zato scheduler is one example, it’s always running , watching the time and generating messages.
    there is only one scheduler. I understand it could be done as a service that starts every second
    but that’s not very efficient and you don’t want 2 running at the same time.

I’m not sure why you’re saying it’s not efficient? At one point I was
starting 50 new services a second from the scheduler and I’m sure it’s
not the end of it. Have you observed anything less?

As for not running 2 services at the same time, you just need to run the
critical paths under a distributed lock:

https://zato.io/docs/progguide/dist-locks.html

  • I have a service that need to do long/hanging polls against other systems
    and generate appropriate messages . it cannot stop or else it will
    lose information. it needs to react quickly

Ok, in that case simply:

def handle(self):

while True:
… # Here goes the long running code

  • I have services that extract stats from other services and keep a small
    statistical window (like a moving average of the least 10 samples) I need
    to react quickly, much faster than one second . it usually will stay connected,
    calling every 100 ms is not efficient , and it needs to send messages when
    the moving average breaks a certain threshold.

I don’t understand what it will stay connected to? And how will the
service be informed of the new messages if you cannot invoke the system
you connect to because it’s not efficient? Also, why it’s not efficient,
does the system have problems it being called 10 times a second?

  • I have a service that listen for messages and send then on websockets to browsers.
    it has a persistent list of connected websockets that it pushes messages to.
    gui users can login / logout and add/remove from that list of websockets.
    this service has to be up, cannot go away or it may miss some new
    connections,need to listen async for messages at the same time and
    push the messages to a list of active websockets.

It sounds like a regular service though I’m not sure what it means to
’listen async for messages’? I understand it’s a simple service and your
requirement is to ensure redundancy.

In that case, simply start more than one server in the cluster and
that’s it, all servers always work in the active-active configuration so
the service will always available on all the servers and the
load-balancer will distribute the load in a fair manner between them.

-I have a log service that listen for snmp traps and convert
them to messages . it cannot go away or it will miss traps.

I’m not familiar with SNMP. Can you please explain what it means to
listen for traps?

  • most services that listen for messages and need to react in
    real time seems to be in that category: they cannot go away,
    they have to act immediately, they need to do other things while
    listening for async events.

I say ‘message’ loosely . the messages I generate are amqp or redis
but inbound they may be entries in logs, hanging polls, traps, repeated polls from other system.

anyway , you get the picture. services are real-time ,
have state (some of the state cannot be serialized like websockets) , they have to be up.

I’m still not clear what ‘being up’ means. How do you handle this
requirement in the current system - seeing that you know the domain very
well I suspect you already have one?

I believe the thing to keep it mind is that a Zato service is not a OS
daemon. The name may be the same but it’s best to find a better parallel

  • consider instead that a service is a Python function running in a
    distributed environment.

If you approach Zato from that angle you’ll ask yourself - how do I make
a service, once it’s called, listen for new events, issue connections
and at the same time keep its state around.

The answer is - in a ‘while True:’ loop.

Well, technically, you could use some yield or recursion tricks if it
/actually/ were a Python function but that as a side note.

So that’s the only thing you need to do, really, a ‘while True:’ loop.

Make the service run forever and make it start automatically when
servers start:

https://zato.io/docs/admin/guide/install-config/config-server.html#startup-services-first-worker

Also, make it store a flag in Redis each 30 seconds and a scheduler’s
job run each 90 seconds - if the flag is not fresh enough, start the
service again.

And like I said, a service is a function running in a greenlet - it’s OK
to start hundreds or thousands of services/functions/greenlets a second.
That it’s a greenlet is completely transparent to you as a programmer
but I mention it just so you’re sure it’s OK to start new ones.

I don’t have any example doing exactly what you need handy but here’s
the closest one from Zato sources:

https://github.com/zatosource/zato/blob/master/code/zato-server/src/zato/server/service/internal/notif/init.py#L67

  • NotifierService is a service implementing common functionality across
    services that serve notifications.

  • It’s run under the ‘while self.keep_running:’ and - here’s where it
    differs from what you need - sleeps for N seconds between doing useful
    things

  • Upon being awaken, it spawns a new greenlet that will do the things.
    Spawning a greenlet means a new lightweight thread of execution that
    will not block the main service.

  • The greenlet consults the service’s configuration and confirms that
    the service is still to be run, if not, the 'while self.keep_running’
    loop stops

  • If it’s still good to go, it runs the thing to be run under a
    distributed lock which makes sure only one instance of the service can
    be invoked at any time

  • The main service passes any additional info to the greenlet using
    self.environ:

https://zato.io/docs/progguide/service-dev.html#environ

This is not 100% what you need but this is something that is related - a
long-running service, and that’s how it was implemented. You can base
your own on it.

I saw the example of accessing openstack swift by polling every 5 seconds.
I would probably try to avoid this for efficiently and just have one service listening for openstack events. same problem.

Again, there’s nothing inefficient in polling Swift periodically. If a
system can publish notifications, Zato will listen for them, say ZeroMQ
or even regular HTTP calls are of that family. But Swift isn’t so it’s
OK to poll it.