Too many open files

#1

Hi,

We are receiving the following error when there is load and long-running processes on zato in the production servers.

2019-05-13 18:15:08,627 - e[1;31mERRORe[0m - 31715:DummyThread-7752189 - service.enqueue:126 - Exception occurred while processing the enqueue service ---- [Errno 24] Too many open files
Traceback (most recent call last):
File “/opt/zato/perch/server1/work/hot-deploy/current/service.py”, line 30, in handle

File “/opt/zato/3.0/code/local/lib/python2.7/site-packages/openerplib/main.py”, line 313, in proxy
args, kw)
File “/opt/zato/3.0/code/local/lib/python2.7/site-packages/openerplib/main.py”, line 180, in proxy
result = self.connector.send(self.service_name, method, *args)
File “/opt/zato/3.0/code/local/lib/python2.7/site-packages/openerplib/main.py”, line 83, in send
return getattr(service, method)(*args)
File “/usr/lib/python2.7/xmlrpclib.py”, line 1243, in call
return self.__send(self.__name, args)
File “/usr/lib/python2.7/xmlrpclib.py”, line 1602, in __request
verbose=self.__verbose
File “/usr/lib/python2.7/xmlrpclib.py”, line 1283, in request
return self.single_request(host, handler, request_body, verbose)
File “/usr/lib/python2.7/xmlrpclib.py”, line 1311, in single_request
self.send_content(h, request_body)
File “/usr/lib/python2.7/xmlrpclib.py”, line 1459, in send_content
connection.endheaders(request_body)
File “/usr/lib/python2.7/httplib.py”, line 1038, in endheaders
self._send_output(message_body)
File “/usr/lib/python2.7/httplib.py”, line 882, in _send_output
self.send(msg)
File “/usr/lib/python2.7/httplib.py”, line 844, in send
self.connect()
File “/usr/lib/python2.7/httplib.py”, line 821, in connect
self.timeout, self.source_address)
File “/opt/zato/3.0/code/local/lib/python2.7/site-packages/gevent/socket.py”, line 86, in create_connection
raise err
error: [Errno 24] Too many open files

When we monitored the open files created by zato some of the files are not been closed. Not sure why these are not getting closed even after the requests are processed.

Before invoking 200 HTTP requests, the open file count for zato server was around 300 but after the invoking of requests it spiked to 500 but never reached back to 300 but it ended around 350. Because of this the open files are getting piled up and receiving the above error.

Are there any issues with some of the internal processing of closing the open files after the requests are processed?

Thanks,
Sai Bhargav

0 Likes

#2

None that I am aware of, it is the first time I hear about it.

The open files that you observe keep growing - can you say what kind of files these are? Do you have any more details?

0 Likes