(Migrated) Zato 2.0.5 Fail to reestablish connection to rabbitmq

(This message has been automatically imported from the retired mailing list)

if there is a network interruption or the rabbit server is restarted =
zato fails to establish the connection, to test this I did the =
following:

I=E2=80=99ve created an AMQP definition with a 2 second heartbeat =
interval:

max frame size: 131072
heartbeat interval: 2

I also have 2 AMQP channels both using the above definition:

2015-09-29 08:40:54,953 - INFO - zato.broker.thread_client - Starting =
broker client, host:[localhost], port:[6379], =
name:[amqp-consuming-connector-K05HQJZHNYAS28PHV2T7CEBVS860], =
topics:[[’/zato/connector/amqp/all’, =
’/zato/connector/amqp/consuming/all’]]
2015-09-29 08:40:55,123 - INFO - zato.broker.thread_client - Starting =
broker client, host:[localhost], port:[6379], =
name:[amqp-consuming-connector-K06YC8GJCFY4FW63CHY33QFKRE4M], =
topics:[[’/zato/connector/amqp/all’, =
’/zato/connector/amqp/consuming/all=E2=80=99]]

they are consuming from the queues just fine, and they are seen in =
rabbit.

kano ~ # rabbitmqctl list_connections client_properties state channels
Listing connections …
[{“platform”,“Python 2.7.6”},{“product”,“Pika Python Client =
Library”},{“version”,“0.9.12”},{“capabilities”,[{“consumer_cancel_notify”,=
true},{“publisher_confirms”,true},{“basic.nack”,true}]},{“information”,“Se=
e http://pika.github.com”}] running 1
[{“platform”,“Python 2.7.6”},{“product”,“Pika Python Client =
Library”},{“version”,“0.9.12”},{“capabilities”,[{“consumer_cancel_notify”,=
true},{“publisher_confirms”,true},{“basic.nack”,true}]},{“information”,“Se=
e http://pika.github.com”}] running 1

But after restarting rabbitmq:

kano ~ # service rabbitmq-server restart
[ ok ] Restarting message broker: rabbitmq-server.
kano ~ # rabbitmqctl list_connections client_properties state channels
Listing connections =E2=80=A6
kano ~ #

or if the network connection is lost:

kano ~ # rabbitmqctl list_connections client_properties state channels
Listing connections …
[{“platform”,“Python 2.7.6”},{“product”,“Pika Python Client =
Library”},{“version”,“0.9.12”},{“capabilities”,[{“consumer_cancel_notify”,=
true},{“publisher_confirms”,true},{“basic.nack”,true}]},{“information”,“Se=
e http://pika.github.com”}] running 1
[{“platform”,“Python 2.7.6”},{“product”,“Pika Python Client =
Library”},{“version”,“0.9.12”},{“capabilities”,[{“consumer_cancel_notify”,=
true},{“publisher_confirms”,true},{“basic.nack”,true}]},{“information”,“Se=
e http://pika.github.com”}] running 1
kano ~ # Write failed: Host is down
ivan@kotalkhan ~ $ ssh kano
ivan@kano ~ $ sudo rabbitmqctl list_connections client_properties state =
channels
Listing connections …
kano ~ #

Zato doesn=E2=80=99t establish the connection effectively missing all =
the messages, the only way to restore this is by either editing the AMQP =
definition, or restarting the servers.

There are no log messages in server.log when connection is lost.

In comparison this is a celery log using the same server, showing the =
expected behavior:

[2015-09-29 08:35:36,538: WARNING/MainProcess] consumer: Connection to =
broker lost. Trying to re-establish the connection…
Traceback (most recent call last):
File =
"/Users/ivan/VirtualEnv/seleniumapi/lib/python2.7/site-packages/celery/wor=
ker/consumer.py", line 278, in start
blueprint.start(self)
File =
"/Users/ivan/VirtualEnv/seleniumapi/lib/python2.7/site-packages/celery/boo=
tsteps.py", line 123, in start
step.start(parent)
File =
"/Users/ivan/VirtualEnv/seleniumapi/lib/python2.7/site-packages/celery/wor=
ker/consumer.py", line 821, in start
c.loop(*c.loop_args())
File =
"/Users/ivan/VirtualEnv/seleniumapi/lib/python2.7/site-packages/celery/wor=
ker/loops.py", line 70, in asynloop
next(loop)
File =
"/Users/ivan/VirtualEnv/seleniumapi/lib/python2.7/site-packages/kombu/asyn=
c/hub.py", line 324, in create_loop
cb(*cbargs)
File =
"/Users/ivan/VirtualEnv/seleniumapi/lib/python2.7/site-packages/kombu/tran=
sport/base.py", line 162, in on_readable
reader(loop)
File =
"/Users/ivan/VirtualEnv/seleniumapi/lib/python2.7/site-packages/kombu/tran=
sport/base.py", line 144, in _read
drain_events(timeout=3D0)
File =
"/Users/ivan/VirtualEnv/seleniumapi/lib/python2.7/site-packages/amqp/conne=
ction.py", line 323, in drain_events
return amqp_method(channel, args)
File =
"/Users/ivan/VirtualEnv/seleniumapi/lib/python2.7/site-packages/amqp/conne=
ction.py", line 526, in _close
self._x_close_ok()
File =
"/Users/ivan/VirtualEnv/seleniumapi/lib/python2.7/site-packages/amqp/conne=
ction.py", line 554, in _x_close_ok
self._send_method((10, 51))
File =
"/Users/ivan/VirtualEnv/seleniumapi/lib/python2.7/site-packages/amqp/abstr=
act_channel.py", line 56, in _send_method
self.channel_id, method_sig, args, content,
File =
"/Users/ivan/VirtualEnv/seleniumapi/lib/python2.7/site-packages/amqp/metho=
d_framing.py", line 221, in write_method
write_frame(1, channel, payload)
File =
"/Users/ivan/VirtualEnv/seleniumapi/lib/python2.7/site-packages/amqp/trans=
port.py", line 182, in write_frame
frame_type, channel, size, payload, 0xce,
File =
"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/so=
cket.py", line 228, in meth
return getattr(self._sock,name)(*args)
error: [Errno 32] Broken pipe
[2015-09-29 08:35:36,967: ERROR/MainProcess] consumer: Cannot connect to =
amqp://ivan:**@kano.fiv:5672/dev: [Errno 61] Connection refused.
Trying again in 2.00 seconds…

[2015-09-29 08:35:38,984: ERROR/MainProcess] consumer: Cannot connect to =
amqp://ivan:**@kano.fiv:5672/dev: [Errno 61] Connection refused.
Trying again in 4.00 seconds…

[2015-09-29 08:35:43,012: ERROR/MainProcess] consumer: Cannot connect to =
amqp://ivan:**@kano.fiv:5672/dev: [Errno 61] Connection refused.
Trying again in 6.00 seconds…

[2015-09-29 08:35:49,454: ERROR/MainProcess] consumer: Cannot connect to =
amqp://ivan:**@kano.fiv:5672/dev: [Errno 61] Connection refused.
Trying again in 8.00 seconds…

[2015-09-29 08:35:57,500: INFO/MainProcess] Connected to =
amqp://ivan:**@kano.fiv:5672/dev=