Endless loop obtaining suds clients

Community support
#1

Hi,

I am making an outgoing SOAP connection to “https://test.triple-ip.com/onecall3/iiip3webservice/iiip3service.asmx?WSDL”.
Serialization type is SUDS.

Zato gets in an endless loop. The log shows:

0/2 Suds SOAP clients obtained to https://test.triple-ip.com/onecall3/iiip3webservice/iiip3service.asmx?WSDL
It never get past 0/2. I only have this behaviour on a Redhat Enterprise server. On an Ubuntu server it works OK.

Has anyone seen this behaviour and possibly knows what can be wrong?

Looking at the code it seems that is has something to do with the gevent.Queue.

There is a while loop with while not self.queue.full() in zato.server.connection.queue that never quits.

Regards, Jan

#2

Hello @jjmurre,

I will look into it - is that WSDL address under your control? Can you make sure it doesn’t go away?

I wonder if this is not specific to this host in terms of TCP connections between RHEL and that SOAP server.

What is the Zato version you are on?

#3

Hi @dsuch,

Tx. that you want to have a look. The wsdl is not under our control, but is stable. Zato version is 2.0.7

Regards,

Jan

#4

I have just successfully obtained SOAP clients from the WSDL under a Zato server on Ubuntu 14.04 so this is indeed something specific to this RHEL environment.

The part of code that establishes SOAP clients was not, however, wrapped in a try/except block and, because it runs in its own greenlet, any exceptions while the clients were being created, would be lost.

I have just pushed a commit to deal with it - all exceptions will now be logged in server.log:

https://github.com/zatosource/zato/commit/8e2f9c6a7a97ee4d17542f2f1f57d8f461174c8b.diff

Can you apply it in your environment? This is basically wrapping the whole of SudsSOAPWrapper.add_client in a try/except block and logging exceptions should any occur.

I’m sure this will shed some light on why it happens on RHEL only.

#5

Hi @dsuch,

Tx. for the patch, I applied it and very soon found out what the problem was.

It seems that Redhat, in their wisdom, backported some ssl stuff. This causes problems with gevent.
Because gevent patches SSLSocket, and the backport does change the callspec. for SSLSocket.
Gevent take this change into account, but only for python > 2.7.9
The problem is that Rehat backported this ssl change to python 2.7.5

This is a know problem, see: https://bugzilla.redhat.com/show_bug.cgi?id=1384491

Zato 2.0.7 uses gevent 1.0.2. I tried to use 1.1.2 and 1.2.1, but that does not solve the problem.

So I directly patched the gevent 1.0.2 code in the gevent egg for now.

class SSLSocket(socket):

    def __init__(self, sock, keyfile=None, certfile=None,
                 server_side=False, cert_reqs=CERT_NONE,
                 ssl_version=PROTOCOL_SSLv23, ca_certs=None,
                 do_handshake_on_connect=True,
                 suppress_ragged_eofs=True,
                 ciphers=None,
                 server_hostname=None,    # <- added
                 _context=None):                # <- added

Regards, Jan

#6

Ah, it’s nice you’ve got it working.

I saw a similar thing some time ago while porting an environment from Zato 1.1 to the latest version (three years of uninterrupted uptime!) from Ubuntu 12.04 to 16.04 - the same situation with server_hostname.