(Migrated) HTTPS problem in 2.0.5

(This message has been automatically imported from the retired mailing list)

After upgrading a system to 2.0.5, we are occasionally seeing the
following exceptions when making outbound HTTPS/SOAP calls, which we
weren’t seeing before.

exception: Error: []

traceback:
<< snip application >>
resp = zato.outgoing.soap.get(chan).conn.send(zato.cid, body)
File
"/opt/zato/2.0.5/zato-server/src/zato/server/connection/http_soap/outgoing.py",
line 323, in post
return self.http_request(‘POST’, cid, data, params, *args, **kwargs)
File
"/opt/zato/2.0.5/zato-server/src/zato/server/connection/http_soap/outgoing.py",
line 299, in http_request
response = self.invoke_http(cid, method, address, data, headers,
{}, params=qs_params, *args, **kwargs)
File
"/opt/zato/2.0.5/zato-server/src/zato/server/connection/http_soap/outgoing.py",
line 79, in invoke_http
cert=cert, verify=verify, timeout=self.config[‘timeout’], *args,
**kwargs)
File
"/opt/zato/2.0.5/eggs/requests-2.6.0-py2.7.egg/requests/sessions.py",
line 464, in request
resp = self.send(prep, **send_kwargs)
File
"/opt/zato/2.0.5/eggs/requests-2.6.0-py2.7.egg/requests/sessions.py",
line 579, in send
r = adapter.send(request, **kwargs)
File
"/opt/zato/2.0.5/eggs/requests-2.6.0-py2.7.egg/requests/adapters.py",
line 370, in send
timeout=timeout
File
"/opt/zato/2.0.5/eggs/requests-2.6.0-py2.7.egg/requests/packages/urllib3/connectionpool.py",
line 527, in urlopen
conn = self._get_conn(timeout=pool_timeout)
File
"/opt/zato/2.0.5/eggs/requests-2.6.0-py2.7.egg/requests/packages/urllib3/connectionpool.py",
line 239, in _get_conn
conn.close()
File “/usr/lib/python2.7/httplib.py”, line 780, in close
self.sock.close() # close it manually… there may be other refs
File
"/opt/zato/2.0.5/eggs/requests-2.6.0-py2.7.egg/requests/packages/urllib3/contrib/pyopenssl.py",
line 230, in close
return self.connection.shutdown()
File “/opt/zato/2.0.5/eggs/pyOpenSSL-0.14-py2.7.egg/OpenSSL/SSL.py”,
line 1157, in shutdown
_raise_current_error()
File
"/opt/zato/2.0.5/eggs/pyOpenSSL-0.14-py2.7.egg/OpenSSL/_util.py", line
22, in exception_from_error_queue
raise exceptionType(errors)

The service we are POSTing to has a long startup time of 30-40 seconds.
To compensate for that we set the timeout in the channel to 60 seconds,
and with zato 2.0.3 that seemed sufficient. The pool size is 10,
although these are only occasional POSTs and are unlikely to be more
than 1 concurrently.

However, I don’t think timeouts are the problem here. When I look at
zato server logs I see the following in every case:

2015-06-19 13:08:32,281 - ESC[1;37mINFOESC[0m - 20242:Dummy-1685 -
srv-cfflow.poll-single:22 - Starting PollSingle u’216’
2015-06-19 13:08:32,330 - ESC[1;37mINFOESC[0m - 20242:Dummy-1685 -
zato.server.connection.http_soap.outgoing:22 -
CID:[K067GBFQTMRZAYK9Q08GXMPWCE1V], address:[<>],
qs_params:[{}], auth:[None], kwargs:[{}]
2015-06-19 13:08:32,340 - ESC[1;37mINFOESC[0m - 20242:Dummy-1685 -
srv-cfflow.poll-single:22 - Poll 216 Completed

The first and last messages there are logged when the service starts and
finishes, and the middle message shows that the outgoing connection is
being made. Therefore it appears that the connection attempt aborts
immediately, without a timeout.

(The exceptions are being trapped within the service and written to a
mysql database, so the service itself completes successfully)

Therefore it does seem like there is a problem in the new implementation
of outbound HTTPS, but I have not been able to replicate it within zato
POSTing to itself - I only see it with the live service.

Any ideas from the above backtrace what’s going on? The exception
message from openssl of “Error: []” is rather unhelpful.

This is ubuntu 14.04 and python 2.7.6

Thanks,

Brian.

On 19/06/15 16:03, Brian Candler wrote:

Therefore it does seem like there is a problem in the new implementation
of outbound HTTPS, but I have not been able to replicate it within zato
POSTing to itself - I only see it with the live service.

Any ideas from the above backtrace what’s going on? The exception
message from openssl of “Error: []” is rather unhelpful.

The new implementation you mentioned does only what PyOpenSSL insists
on, namely, on providing payload as a string instead of unicode. That
was the only change made in 2.0.5 on that front.

Otherwise the code wouldn’t have been touched and it’s not something we
can easily back off from.

Is this consistent and reproducible 100% of time?