Zato 2.0.7 gunicorn error

Hello, guys,

I know this version is unsupported but I’m in the middle of a crisis right now and I need something quickly to deal with this, so any help is appreciated.

Since this weekend, Zato is randomly going down on 2 of my 3 machine clusters, with the following logs:

*** Regular lines from my services finishing here***
2019-07-01 19:20:04,048 - ^[[1;37mINFO^[[0m - 12200:Dummy-1 - gunicorn.main:22 - Handling signal: term
2019-07-01 19:20:04,200 - ^[[1;37mINFO^[[0m - 12235:Dummy-1 - gunicorn.main:22 - Worker exiting (pid: 12235)
2019-07-01 19:20:04,244 - ^[[1;31mERROR^[[0m - 12236:Dummy-391 - gunicorn.main:22 - Socket error processing request.
Traceback (most recent call last):
  File "/opt/zato/2.0.7/code/eggs/gunicorn-18.0-py2.7.egg/gunicorn/workers/async.py", line 45, in handle
    self.handle_request(listener, req, client, addr)
  File "/opt/zato/2.0.7/code/eggs/gunicorn-18.0-py2.7.egg/gunicorn/workers/ggevent.py", line 150, in handle_request
    super(GeventWorker, self).handle_request(*args)
  File "/opt/zato/2.0.7/code/eggs/gunicorn-18.0-py2.7.egg/gunicorn/workers/async.py", line 83, in handle_request
    listener.getsockname(), self.cfg)
  File "<string>", line 1, in getsockname
error: [Errno 9] Bad file descriptor
2019-07-01 19:20:05,067 - ^[[1;31mERROR^[[0m - 12236:Dummy-392 - gunicorn.main:22 - Socket error processing request.
Traceback (most recent call last):
  File "/opt/zato/2.0.7/code/eggs/gunicorn-18.0-py2.7.egg/gunicorn/workers/async.py", line 45, in handle
    self.handle_request(listener, req, client, addr)
  File "/opt/zato/2.0.7/code/eggs/gunicorn-18.0-py2.7.egg/gunicorn/workers/ggevent.py", line 150, in handle_request
    super(GeventWorker, self).handle_request(*args)
  File "/opt/zato/2.0.7/code/eggs/gunicorn-18.0-py2.7.egg/gunicorn/workers/async.py", line 83, in handle_request
    listener.getsockname(), self.cfg)
  File "<string>", line 1, in getsockname
error: [Errno 9] Bad file descriptor
2019-07-01 19:20:05,188 - ^[[1;37mINFO^[[0m - 12236:Dummy-1 - gunicorn.main:22 - Worker exiting (pid: 12236)
2019-07-01 19:20:05,793 - ^[[1;37mINFO^[[0m - 12236:Dummy-394 - zato.server.pickup.gevent_pickup:22 - Successfully stopped the notifier
2019-07-01 19:20:05,955 - ^[[1;37mINFO^[[0m - 12200:Dummy-1 - gunicorn.main:22 - Shutting down: Master
2019-07-01 19:22:46,274 - ^[[1;37mINFO^[[0m - 44422:Dummy-1 - gunicorn.main:22 - Starting gunicorn 18.0
2019-07-01 19:22:46,277 - ^[[1;31mERROR^[[0m - 44422:Dummy-1 - gunicorn.main:22 - Connection in use: ('0.0.0.0', 17010)
2019-07-01 19:22:46,278 - ^[[1;31mERROR^[[0m - 44422:Dummy-1 - gunicorn.main:22 - Retrying in 1 second.
2019-07-01 19:22:47,279 - ^[[1;31mERROR^[[0m - 44422:Dummy-1 - gunicorn.main:22 - Connection in use: ('0.0.0.0', 17010)
2019-07-01 19:22:47,279 - ^[[1;31mERROR^[[0m - 44422:Dummy-1 - gunicorn.main:22 - Retrying in 1 second.
2019-07-01 19:22:48,280 - ^[[1;31mERROR^[[0m - 44422:Dummy-1 - gunicorn.main:22 - Connection in use: ('0.0.0.0', 17010)
2019-07-01 19:22:48,280 - ^[[1;31mERROR^[[0m - 44422:Dummy-1 - gunicorn.main:22 - Retrying in 1 second.
2019-07-01 19:22:49,280 - ^[[1;31mERROR^[[0m - 44422:Dummy-1 - gunicorn.main:22 - Connection in use: ('0.0.0.0', 17010)
2019-07-01 19:22:49,281 - ^[[1;31mERROR^[[0m - 44422:Dummy-1 - gunicorn.main:22 - Retrying in 1 second.
2019-07-01 19:22:50,281 - ^[[1;31mERROR^[[0m - 44422:Dummy-1 - gunicorn.main:22 - Connection in use: ('0.0.0.0', 17010)
2019-07-01 19:22:50,281 - ^[[1;31mERROR^[[0m - 44422:Dummy-1 - gunicorn.main:22 - Retrying in 1 second.
2019-07-01 19:22:51,282 - ^[[1;31mERROR^[[0m - 44422:Dummy-1 - gunicorn.main:22 - Can't connect to ('0.0.0.0', 17010)

Thankfully, the third machine seems stable and the load balancers is handling the situation when it happens. We could not find any configuration differences between the servers.

We could not find the root cause of the issue. Either something is requesting Zato to exit (which we could not find any evidence) or Zato/gunicorn is deciding this by itself (which again we could not see anything outside of normal on the logs).

I cannot upgrade Zato for now to 3.1 version, since our OS is unsupported now (RHEL 6.7) and if this is a obscure gunicorn bug, version 2.0.8 also cannot be used (since it uses the same version). Maybe going to 3.0, but would not like to upgrade to an already unsupported version, the problem will be the same if it doesn’t work anyway.

Any ideas on how could I deal with this issue? For now, I am trying to find a way to restart the application as soon as possible, but we cannot stay like this for too long. Any help is appreciated.

[]'s

Hello @rtrind,

I am sorry but there is no free support for this version of Zato anymore. Only clients covered by commercial support receive it and we are in the process of migrating to 3.x with most of them at this point.

Regards.