Zato scheduler jobs stop working after Gunicorn worker timeout and restart (2.0.7 on RHEL 6.7)


#1

Hello,

I am using paramiko 1.18 in some zato services which periodically pool remote servers for files and process them accordingly. But it seems there are some situations where the library does not timeout appropriately and hangs indefinitely. This eventually leads to a gunicorn timeout and the worker restarts, which is not ideal, but OK.

The problem is sometimes after this happens, all the scheduler jobs do not run anymore. On the singleton.log, nothing is written as well. Only after I restart one of the workers, the scheduler resumes properly.

Questions:

  • I tried several ways of provoking natural timeouts of Paramiko, unsuccessfully. Since it was a pain to finally find an able SFTP library, I am not looking to replace it anytime soon. So I tried using the signal alarm class to induce a timeout for my entire service, which works sometimes, but not always. It seems the library is able to completely hang in a way I cannot influence from my code. Do you see any way zato could help me avoid this issue? Or any other way I could use to simply drop my execution entirely? All the code is made from the perspective of supporting some hard crashes without losing anything, so I just need to avoid the zato gunicorn timeout.

  • Independently from the above problem, is it expected when a gunicorn timeout happens for the singleton mechanism (and by consequence the scheduling) to stop working? Do you think upgrading to 2.0.8 could improve anything related to this?

I don’t want to go looking for another SFTP library again but maybe we will have to open another topic to solve it. The main problem being, in the case of Paramiko 2.x, having dependencies already bundled within zato. I tried using a VM connected to the internet to create an rpm with all the dependencies working, but some of these libraries when updated break zato, so I don’t know if there is a way for zato to keep using it’s own versions and paramiko to use another in the same environment… But don’t focus on this part for now. Let’s keep the discussion focused on the scheduling issues.

Thanks,
Ricardo


Zato services and worker timeouts
#2

Hi Ricardo
That situation seems very familiar to me. Could you describe more your setup and interaction with sftp using paramiko? In my case I remember that my sftp server implemented using paramiko too does not support properly gevent monkey patching causing the file transfer to start but hang up after a moment indefinitely without ending.


Setting up SFTP connection
#3

Basically your answer is that Paramiko is not gevent friendly, at least for the server protocol as the creator of paramiko answered to me in an issue. But you could use it without gevent monkey patching or try another scriptable sftp server like asyncssh for python 3
My sftp server implementation used with zato jobs for batch processing is here and it worked ok with zato 1.1.


#4

My setup is very simple. I have 3 physical servers, each with 2 workers. On the scheduler I have a service which pools remotely for xml files, open then to extract some information on the headers and creates an internal queue (saved on Redis). Another service pools this queue and periodically moves those files for processing by another server (unrelated to Zato). All those reads and moves are in the same remote SFTP server. In other words, I use only the SFTP client part of paramiko. It works well 99% of the time, but when it hangs, it’s a cascade of problems all over the environment and sometimes we need a human (or a dumb script) to bring the platform back, which is not professional.

I am trying to understand more about the “gevent monkey patching”, started from https://github.com/paramiko/paramiko/issues/389. If you know some information or page I could read to reach this faster, I would appreciate it.

Thanks for the help so far!
Ricardo


#5

I looked into paramiko code and no sign of the monkey patch. So should I add it? Do I do it straight on the __init__.py of paramiko (I have no problems doing this if it solves the issue)? Or is it the opposite? If so, how can I tell zato/gunicorn to not patch it?

I imagine I should not do the monkey patch on my zato service, since gunicorn and zato should already have done this, alright?

Sorry if my questions are noobish, I am still learning the ropes regarding python and all the base libraries used by zato.

[]'s


#6

Sorry, but if I remember properly my situation was related to the SFTP server part, as you could see here


The monkey.patch_all() was commented to get it working. Also, this is could be something related to big files and perhaps it’s part and part of the client(with paramiko) and server (any server/platform having some kind of problem with coroutines usage functions). I remember the copy/send logs of the files started to log ok and after a very fixed amount of time the copy/send hangs and the worker need to be restarted
I have solved the situation in my implementation of the SFTP server. You should try it to see if the problem goes away using solt_sftp.


#7

Hello @rtrind,

check out the parallel-ssh library which says requires gevent to work which would imply is gevent-friendly:

https://pypi.python.org/pypi/parallel-ssh/

I understand that you would like for SSH or SCP to simply work and be done with it but this is not one of protocols that Zato supports out of the box and apparently paramiko is somehow mixing things up. It’s quite difficult to say why it’s the case because this is not part of what Zato depends on by default.

I plan to add support for this protocol, in which case it will work out of the box and will be supported but this won’t be done for 3.0 unless there is a sponsor for the development effort. Feel free to contact me privately if you’d like to discuss it (dsuch@zato.io).

Regards.


#8

Hello,

Thanks for the suggestion, but parallel-ssh uses paramiko as a requisite so I always dismissed it as a “the problems will probably be the same, since the underlying library is still paramiko”. If this is not true, I believe the library will be easy to change and I will try change the code accordingly. Please advise.

Thanks!


#9

Let me also add why gevent-friendliness is a topic that one needs to take into account.

I will write it as a broad explanation of how Zato works in certain aspects because it may be useful to you and to future people is similar situations. I realize that some of it may be clear to you but I’m writing it for others as well.

Ideally, a Zato server would reply in 0 milliseconds at the same time being able to handle a billion requests a second on a single 50 kHz CPU.

This is not the case, with Zato or any other server for that matter, and there are several factors that are in play here. A given workload of a Zato server can be:

  • Network-bound
  • CPU-bound
  • Disk-bound
  • Memory-bound

The last two are least frequently seen so I will go through them first.

A disk-bound server is limited by how fast disks are, simply. In most environments, this is not a concern at all because Zato is not a file server responding with images for web or anything like that. When a Zato server boots up, it reads from ODB (disk) all of its configuration and populates all the internal structures or caches. Then in run-time, ODB is hardly ever used really so disk is almost never a factor. About the only thing is HTTP access log, it adds some overhead but if it is a concern then one needs to use SSD and if even this is not enough then one simply logs through UDP to syslog and that is it.

Then, a server may be memory bound. For instance, you need to process very large documents (e.g. monthly orders from a company). There is nothing we can do on Python end if RAM is not fast enough short of suggesting that users buy faster RAM but what we can do is to suggest that all the existing JSON and XML libraries be evaluated using one’s own actual JSON or XML documents - there are vast variations in RAM consumption across libraries, some will work better with flatter structures, for some nested documents are easier to parse.

In general, one can expect at least 5-10 times memory use than a given document is. For instance, if your JSON orders are 1 MB big then you can expect 5-10 MB of RAM are needed to parse and work with them. This is simply how currently available libraries work.

I actually have an idea of creating a new JSON Python library that would learn in run-time about how user documents look like and which would in turn let it better manage memory and CPU access patterns but this is just an idea for now.

One thing that can reduce RAM usage is disabling various internal services - for instance, if you don’t need AMQP or ZeroMQ, then their relevant services don’t need to be deployed = less RAM is needed. In 3.0 one will be also able to disable components through server.conf which will mean less RAM and less CPU used.

Which bring us to the topic of CPU-bound applications. In a similar vein, there are applications for whom the limiting factor is how fast a CPU is. If you are computing the π number then you doing exactly that - computing. You are not really reading anything from disk and you are not reading the number from RAM because you need to compute it in the first place :slight_smile: (I am skipping details of what actual hardware is needed to compute π to a great degree because I’m not familiar with it myself but I gave it as an example).

So it looks that you are bound by how fast a CPU is. Now, being a Zato server, most of the time what you are doing is actually accessing some database, be it SQL, Cassandra, Redis, Solr, Vault, ZeroMQ, anything really, in your case this is an SSH server. This means that it turns out that what you are really doing is waiting for these remote resources to reply over TCP. When one measures it, it can be found that the overhead of using TCP over using a CPU is quite large, this is nicely shown here:

This in other words means that a Zato server is network-bound. Its performance is limited by how fast a network is. This is a generic term and means not only cabling through which packets go but also the remote databases that Zato waits for, all of this is considered part of the network.

All of this leads us to the idea of writing server software around libraries and kernel primitives that take advantage of the fact that, for certain classes of software, such as Zato, these applications will spend most of their time waiting for the network. For historical reasons, they are called asynchronous or non-blocking servers in opposition to what up to around 2005 was the norm, which is synchronous or blocking servers. The latter still exist but by definition, they cannot achieve the same performance as asynchronous servers such as Zato, they are not built around the idea that network is always the slowest part. The blocking ones have their uses and are very nice but simply cannot achieve hundreds or thousands of requests a second without much more hardware.

This in turn explains why we are using gevent - it’s an asynchronous server on top of which Zato is built. It’s actually an implementation detail, programmers authoring Zato services don’t really need to know about it. But in your case, since you are not using what Zato comes with by default, it needs to be taken into account.

Now, what gevent does is two things. One is making certain changes to Python’s standard library, which is called gevent monkey-patching. These changes let libraries that are otherwise synchronous to become asynchronous without changes to these libraries. Part of monkey patching are changes to Python’s socket library and making it aware of so called greenlets.

Greenlets are like very, very small threads that give an outwardly impression of running in parallel. I’m not really familiar with Erlang nor Golang but my understanding is that these languages have similar notions. In Python, we have greenlets.

The important part is all greenlets are actually not run in parallel, it’s just an impression. For instance, if you have 100 greenlets and 1 of them takes up 5 seconds to do its job, no matter what, then the remaining ones will not run. Keep that in mind, that was an important detail.

Getting back to Zato, each request that you send to Zato is actually executed in its own greenlet. Such a greenlet runs within the gevent’s event loop and gevent switches between them when they, essentially, wait for the network. This here is where you can see how it all is related - when one greenlet waits for network (and remember we are network bound), then another greenlet runs, then the other waits for network so a third one runs and so on. In practice, you can get up to a couple of thousands of such greenlets a second on a single CPU which in other words means that Zato can handle up to a couple of thousand requests a second on a single CPU (and because it knows how to use multiple CPUs, you can get up to about 8,000-10,000 req/s with Zato 3.0 on a quad core computer).

This works really nicely. Me and my colleagues, mainly myself, spend time making sure all of the libraries are gevent-friendly, all of it is thoroughly tested and ultimately, after some coding, GUI, CLI and so on, one can declare that a given technology is fully supported and can be used.

This is open-source so naturally you are free to add libraries of your own, this is why there is buildout, pip and zato_extra_paths. But what you need to take into account is whether what you are adding can actually work under gevent and in what circumstances, because if it can’t then you can expect exactly the things that you are observing.

If a library does not know how to yield control to another greenlet it is part of, then this library, this greenlet really, will overtake the whole of the CPU for itself. This means that other greenlets don’t run at all. And we have a few internal greenlets that need to run - one of them is the scheduler’s greenlet. If paramiko doesn’t let the scheduler run then it won’t run, it has no way, it doesn’t get its share of CPU to run on.

As a side note - this is one of reasons why in Zato 3.0 the scheduler is a separate component started with zato /path/to/scheduler. There have been situations like yours before (but not with SSH) and the simplest approach was simply to move it over aside so that nothing blocks it and it cannot block anything else.

The overtaking of a CPU can happen in other situations too - for instance, when you parse a 0.5 GB JSON document this is a CPU-bound task and it will need at least several seconds during which no other greenlet will run (not to mention it will need a lot of RAM). This is why when you have a mixed transaction-processing and batch workload, it’s better to spread over two Zato clusters, one for each type of work.

About your question which Python library to use - I really don’t know. Our job is to spend at least several days full-time to research and investigate these subjects before a technology gets into core Zato. This is not a matter of simply finding the first one and starting to use it. Many times it will work just fine with gevent’s monkey patching but sometimes it won’t and it’s always time-consuming effort to find the root cause.

I truly understand your situation, I see that you don’t want to get through the whole effort again but I hope this message explains why you cannot just put everything aside and focus on why there is an issue with the scheduler - I know that you’d just like to spend time on your solution not on external libraries but it’s really not a simple question of what the culprit could be.


Issue with starting SQL Notification and Scheduler - Zato 2.0
#10

Thanks for the very clean explanation. Now it’s clear to me.

Since the library you sugestted seems to be gevent friendly, even if under paramiko, there is a better chance of yield properly when needed so zato can run the scheduler properly. I will test it soon and let others now if this is a better fit than pure paramiko.

Thanks again!