(Migrated) 504 Gateway Time-out occasionally

(This message has been automatically imported from the retired mailing list)

Hi,

I’ve got a problem with 504 Gateway Time-out response, but not all the
time. Some requests after 15 seconds get a “504 Gateway Time-out” response:


504 Gateway Time-out

The server didn't respond in time. ---

I have changed configuration on
http://xxx/zato/load-balancer/manage/cluster/1/
I set the line from timeouts to:
Timeouts (ms) connect: 100000, client: 100000, server: 100000

In my zato app folder
(instance_of_zato/load-balancer/config/repo/zato.config) changes are
visible:


defaults
log global
option httpclose

stats uri /zato-lb-stats # ZATO defaults:stats uri

timeout connect 100000 # ZATO defaults:timeout connect
timeout client 100000 # ZATO defaults:timeout client
timeout server 100000 # ZATO defaults:timeout server

I have got 2 servers added to load balancer, so when I send request to
specified connector, one of these servers executes connector code. But as I
said before - from time to time the response is “504 Gateway Time-out”.
It’s strange for me, because sometimes I receive proper response (HTTP 200)
with good response XML. Both servers responds 504 error and 200 OK.

I have added time.sleep(15) to the handle() method and still some responses
have HTTP code 200, some have HTTP code 504. After that, of course, all
responses are sending from the server after more than 15s from request.

Let me paste a sample from Newman log testing tool:


200 17784ms http://xxx/my-connector [POST] http://xxx/my-connector
? Status code is 200
? has proper xml
? has executionResult
=C3=97 executionResult is OK
=C3=97 Zapis zaj=C4=99ty przez innego u=C5=BCytkownika.
504 15007ms http://xxx/my-connector [POST] http://xxx/my-connector
=C3=97 Status code is 200
=C3=97 has proper xml
200 18034ms http://xxx/my-connector [POST] http://xxx/my-connector
? Status code is 200
? has proper xml
? has executionResult
? executionResult is OK
200 18500ms http://xxx/my-connector [POST] http://xxx/my-connector
? Status code is 200
? has proper xml
? has executionResult
? executionResult is OK
504 15022ms http://xxx/my-connector [POST] http://xxx/my-connector
=C3=97 Status code is 200
=C3=97 has proper xml

Have someone has similar problem to me?

Pozdrawiam / Best regards

Szymon Kalinowski

On 25/02/16 15:41, Szymon Kalinowski wrote:

I’ve got a problem with 504 Gateway Time-out response, but not all the
time. Some requests after 15 seconds get a “504 Gateway Time-out” response:


504 Gateway Time-out

The server didn't respond in time.

Hi,

does it ever happen if you connect to either server directly, i.e.
bypassing the load balancer?

On 25/02/16 18:58, Szymon Kalinowski wrote:

I did’t try to send message to the server directly, because I did’n need
to and I don’t know how to do this. From a documentation
(https://zato.io/docs/security/attack-surface.html#servers ) we could
read that it could be done by a socket. Is it mentioned somewhere how
exactly to send a message to the server directly?

But when the client gets a 504 response from the load-balancer, on the
server execution of the code is still in progress.

Ok, this is all understood.

But actually, please first describe your environment.

I understand that there is a LB + 2 servers. Do they all run in the same
system?

The test HTTP client, where does it run? Is it your local box or the
same system LB and serves are in?

thanks,

Big thanks for fast reply!

But actually, please first describe your environment.

I understand that there is a LB + 2 servers. Do they all run in the same
system?

I have one Linux server, where Zato was installed. All servers are on one
machine.

Name Server host/FQDN Server state Server state update LB state LB address
server1 xxx.com.pl/xxx.com.pl running 24-02-2016 11:40:58 UP 127.0.0.1:17010
server2 xxx.com.pl/xxx.com.pl running 24-02-2016 11:41:02 UP 127.0.0.1:17013
server3 xxx.com.pl/xxx.com.pl running (unknown) (unknown) (unknown)
server4 xxx.com.pl/xxx.com.pl running (unknown) (unknown) (unknown)

The test HTTP client, where does it run? Is it your local box or the
same system LB and serves are in?

Tests was made from my PC, which is together with the server in one
network. Operations below made from my PC:

C:>tracert 172.16.167.22
Tracing route to xxx.com.pl [172.16.167.22]
over a maximum of 30 hops:
1 7 ms 2 ms 1 ms xxx.com.pl [172.16.167.22]
Trace complete.
C:>curl http://esb.dsr.com.pl:11223/pist-input-connector
Method GET is not allowed here

So communication with Zato should work.

Best Regards
Szymon Kalinowski

On 26/02/16 10:42, Szymon Kalinowski wrote:

I have one Linux server, where Zato was installed. All servers are on
one machine.

NameServer host/FQDNServer stateServer state updateLB stateLB address
server1xxx.com.pl/xxx.com.pl
http://xxx.com.pl/xxx.com.plrunning24-02-2016
11:40:58UP127.0.0.1:17010 http://127.0.0.1:17010
server2xxx.com.pl/xxx.com.pl
http://xxx.com.pl/xxx.com.plrunning24-02-2016
11:41:02UP127.0.0.1:17013 http://127.0.0.1:17013
server3xxx.com.pl/xxx.com.pl
http://xxx.com.pl/xxx.com.plrunning(unknown)(unknown)(unknown)
server4xxx.com.pl/xxx.com.pl
http://xxx.com.pl/xxx.com.plrunning(unknown)(unknown)(unknown)

The test HTTP client, where does it run? Is it your local box or the
same system LB and serves are in?

Tests was made from my PC, which is together with the server in one
network. Operations below made from my PC:

Right, OK.

Please repeat the same test now from your local PC against each of the
servers directly, without the load-balancer.

Next, do the same test but from the OS Zato runs on, i.e. not from your
PC. This will need to be done in three steps as well:

  • Test with the load-balancer
  • Test with server1
  • Test with server2

That will cover all of the connection paths to shed some light on where
the culprit may be.

Please repeat the same test now from your local PC against each of the
servers directly, without the load-balancer.

Could you give me a hint how to send a request directly to the server,
without the load-balancer?

I could only make tests on the server to
http://localhost:11223/pist-input-connector , so to the load-balancer.


[root@xxx szk]# curl http://localhost:11223/pist-input-connector -d @a.xml

504 Gateway Time-out

The server didn't respond in time. [root@xxx szk]# curl http://localhost:11223/pist-input-connector -d @a.xml expected response ---

Regards,
Szymon Kalinowski

On 26/02/16 13:52, Szymon Kalinowski wrote:

Could you give me a hint how to send a request directly to the server,
without the load-balancer?

What is the output of this command run in the server?

$ sudo netstat -tulpn

On 26/02/16 15:32, Szymon Kalinowski wrote:

What is the output of this command run in the server?

$ sudo netstat -tulpn

[root@xxx szk]# sudo netstat -tulpn
Active Internet connections (only servers)
tcp 0 0 127.0.0.1:17021 http://127.0.0.1:17021
0.0.0.0:* LISTEN 35227/gunicorn
tcp 0 0 127.0.0.1:35101 http://127.0.0.1:35101
0.0.0.0:* LISTEN 7493/python
tcp 0 0 0.0.0.0:445 http://0.0.0.0:445
0.0.0.0:* LISTEN 4399/smbd
tcp 0 0 127.0.0.1:17022 http://127.0.0.1:17022
0.0.0.0:* LISTEN 35257/gunicorn
tcp 0 0 127.0.0.1:35102 http://127.0.0.1:35102
0.0.0.0:* LISTEN 7494/python
tcp 0 0 0.0.0.0:8383 http://0.0.0.0:8383
0.0.0.0:* LISTEN 36001/python
tcp 0 0 127.0.0.1:17023 http://127.0.0.1:17023
0.0.0.0:* LISTEN 35282/gunicorn
tcp 0 0 0.0.0.0:12223 http://0.0.0.0:12223
0.0.0.0:* LISTEN 34681/haproxy
tcp 0 0 127.0.0.1:20351 http://127.0.0.1:20351
0.0.0.0:* LISTEN 7917/python
tcp 0 0 127.0.0.1:35201 http://127.0.0.1:35201
0.0.0.0:* LISTEN 7041/python
tcp 0 0 0.0.0.0:5665 http://0.0.0.0:5665
0.0.0.0:* LISTEN 4282/icinga2
tcp 0 0 127.0.0.1:35202 http://127.0.0.1:35202
0.0.0.0:* LISTEN 7042/python
tcp 0 0 127.0.0.1:35301 http://127.0.0.1:35301
0.0.0.0:* LISTEN 8231/python
tcp 0 0 127.0.0.1:17030 http://127.0.0.1:17030
0.0.0.0:* LISTEN 35899/gunicorn
tcp 0 0 127.0.0.1:35302 http://127.0.0.1:35302
0.0.0.0:* LISTEN 8232/python
tcp 0 0 127.0.0.1:17031 http://127.0.0.1:17031
0.0.0.0:* LISTEN 35923/gunicorn
tcp 0 0 0.0.0.0:13223 http://0.0.0.0:13223
0.0.0.0:* LISTEN 7932/haproxy
tcp 0 0 127.0.0.1:35111 http://127.0.0.1:35111
0.0.0.0:* LISTEN 7495/python
tcp 0 0 127.0.0.1:199 http://127.0.0.1:199
0.0.0.0:* LISTEN 2533/snmpd
tcp 0 0 127.0.0.1:17032 http://127.0.0.1:17032
0.0.0.0:* LISTEN 35958/gunicorn
tcp 0 0 127.0.0.1:17033 http://127.0.0.1:17033
0.0.0.0:* LISTEN 35982/gunicorn
tcp 0 0 127.0.0.1:35211 http://127.0.0.1:35211
0.0.0.0:* LISTEN 7043/python
tcp 0 0 0.0.0.0:139 http://0.0.0.0:139
0.0.0.0:* LISTEN 4399/smbd
tcp 0 0 0.0.0.0:6379 http://0.0.0.0:6379
0.0.0.0:* LISTEN 3235/redis-server *
tcp 0 0 127.0.0.1:35311 http://127.0.0.1:35311
0.0.0.0:* LISTEN 8233/python
tcp 0 0 0.0.0.0:6479 http://0.0.0.0:6479
0.0.0.0:* LISTEN 3243/redis-server *
tcp 0 0 0.0.0.0:111 http://0.0.0.0:111
0.0.0.0:* LISTEN 1604/rpcbind
tcp 0 0 0.0.0.0:80 http://0.0.0.0:80
0.0.0.0:* LISTEN 3532/httpd
tcp 0 0 127.0.0.1:17010 http://127.0.0.1:17010
0.0.0.0:* LISTEN 35761/gunicorn
tcp 0 0 127.0.0.1:17011 http://127.0.0.1:17011
0.0.0.0:* LISTEN 35777/gunicorn
tcp 0 0 0.0.0.0:6579 http://0.0.0.0:6579
0.0.0.0:* LISTEN 3252/redis-server *
tcp 0 0 127.0.0.1:17012 http://127.0.0.1:17012
0.0.0.0:* LISTEN 35811/gunicorn
tcp 0 0 127.0.0.1:17013 http://127.0.0.1:17013
0.0.0.0:* LISTEN 35832/gunicorn
tcp 0 0 0.0.0.0:22 http://0.0.0.0:22
0.0.0.0:* LISTEN 3217/sshd
tcp 0 0 0.0.0.0:11223 http://0.0.0.0:11223
0.0.0.0:* LISTEN 40566/haproxy
tcp 0 0 127.0.0.1:20151 http://127.0.0.1:20151
0.0.0.0:* LISTEN 40327/python
tcp 0 0 0.0.0.0:8183 http://0.0.0.0:8183
0.0.0.0:* LISTEN 35859/python
tcp 0 0 0.0.0.0:6679 http://0.0.0.0:6679
0.0.0.0:* LISTEN 3260/redis-server *
tcp 0 0 127.0.0.1:631 http://127.0.0.1:631
0.0.0.0:* LISTEN 2180/cupsd
tcp 0 0 0.0.0.0:35287 http://0.0.0.0:35287
0.0.0.0:* LISTEN 1802/rpc.statd
tcp 0 0 127.0.0.1:5432 http://127.0.0.1:5432
0.0.0.0:* LISTEN 3315/postmaster
tcp 0 0 127.0.0.1:25 http://127.0.0.1:25
0.0.0.0:* LISTEN 3406/master
tcp 0 0 0.0.0.0:8283 http://0.0.0.0:8283
0.0.0.0:* LISTEN 35303/python
tcp 0 0 127.0.0.1:20251 http://127.0.0.1:20251
0.0.0.0:* LISTEN 34669/python
tcp 0 0 0.0.0.0:6779 http://0.0.0.0:6779
0.0.0.0:* LISTEN 3268/redis-server *
tcp 0 0 127.0.0.1:17020 http://127.0.0.1:17020
0.0.0.0:* LISTEN 35201/gunicorn
tcp 0 0 :::5989 :::*
LISTEN 4516/cimserver
tcp 0 0 :::6379 :::*
LISTEN 3235/redis-server *
tcp 0 0 :::6479 :::*
LISTEN 3243/redis-server *
tcp 0 0 :::6579 :::*
LISTEN 3252/redis-server *
tcp 0 0 :::6679 :::*
LISTEN 3260/redis-server *
tcp 0 0 :::6779 :::*
LISTEN 3268/redis-server *
udp 0 0 0.0.0.0:37132 http://0.0.0.0:37132
0.0.0.0:* 7932/haproxy
udp 0 0 0.0.0.0:913 http://0.0.0.0:913
0.0.0.0:* 1604/rpcbind
udp 0 0 0.0.0.0:161 http://0.0.0.0:161
0.0.0.0:* 2533/snmpd
udp 0 0 0.0.0.0:58916 http://0.0.0.0:58916
0.0.0.0:* 40566/haproxy
udp 0 0 0.0.0.0:33079 http://0.0.0.0:33079
0.0.0.0:* 39772/haproxy
udp 0 0 127.0.0.1:708 http://127.0.0.1:708
0.0.0.0:* 1802/rpc.statd
udp 0 0 0.0.0.0:50258 http://0.0.0.0:50258
0.0.0.0:* 34681/haproxy
udp 0 0 0.0.0.0:54889 http://0.0.0.0:54889
0.0.0.0:* 32941/haproxy
udp 0 0 0.0.0.0:53610 http://0.0.0.0:53610
0.0.0.0:* 1802/rpc.statd
udp 0 0 0.0.0.0:111 http://0.0.0.0:111
0.0.0.0:* 1604/rpcbind
udp 0 0 0.0.0.0:631 http://0.0.0.0:631
0.0.0.0:* 2180/cupsd
udp 0 0 172.16.167.22:123 http://172.16.167.22:123
0.0.0.0:* 3228/ntpd
udp 0 0 127.0.0.1:123 http://127.0.0.1:123
0.0.0.0:* 3228/ntpd
udp 0 0 0.0.0.0:123 http://0.0.0.0:123
0.0.0.0:* 3228/ntpd
udp 0 0 :::123 :::*
3228/ntpd

Ok - I can see 12 Zato server processes running in this system. What
Zato version is it and how many CPUs do you have allocated to that OS?

Ok - I can see 12 Zato server processes running in this system. What
Zato version is it and how many CPUs do you have allocated to that OS?

I hope following info will be helpful:

[root@xxx szk]# less /proc/meminfo
MemTotal: 8054728 kB
MemFree: 175012 kB

[root@xxx szk]# cat /proc/cpuinfo | grep "model name"
model name : Intel® Xeon® CPU E5-2650 0 @ 2.00GHz
model name : Intel® Xeon® CPU E5-2650 0 @ 2.00GHz
model name : Intel® Xeon® CPU E5-2650 0 @ 2.00GHz
model name : Intel® Xeon® CPU E5-2650 0 @ 2.00GHz

[zato@xxx szk]$ zato --version
Zato 2.0.7.rev-dfee180c

So, there are 4 cores.

Detailed info about single core:

processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 45
model name : Intel® Xeon® CPU E5-2650 0 @ 2.00GHz
stepping : 7
microcode : 1808
cpu MHz : 2000.000
cache size : 20480 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm
constant_tsc arch_perfmon pebs bts xtopology tsc_reliable nonstop_tsc
aperfmperf unfair_spinlock pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2
x2apic popcnt aes xsave avx hypervisor lahf_lm ida arat epb pln pts dts
bogomips : 4000.00
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

Best regards,
Szymon Kalinowski

On Fri, Feb 26, 2016 at 3:36 PM, Dariusz Suchojad dsuch@zato.io wrote:

On 26/02/16 15:32, Szymon Kalinowski wrote:

What is the output of this command run in the server?

$ sudo netstat -tulpn

[root@xxx szk]# sudo netstat -tulpn
Active Internet connections (only servers)
tcp 0 0 127.0.0.1:17021 http://127.0.0.1:17021
0.0.0.0:* LISTEN 35227/gunicorn
tcp 0 0 127.0.0.1:35101 http://127.0.0.1:35101
0.0.0.0:* LISTEN 7493/python
tcp 0 0 0.0.0.0:445 http://0.0.0.0:445
0.0.0.0:* LISTEN 4399/smbd
tcp 0 0 127.0.0.1:17022 http://127.0.0.1:17022
0.0.0.0:* LISTEN 35257/gunicorn
tcp 0 0 127.0.0.1:35102 http://127.0.0.1:35102
0.0.0.0:* LISTEN 7494/python
tcp 0 0 0.0.0.0:8383 http://0.0.0.0:8383
0.0.0.0:* LISTEN 36001/python
tcp 0 0 127.0.0.1:17023 http://127.0.0.1:17023
0.0.0.0:* LISTEN 35282/gunicorn
tcp 0 0 0.0.0.0:12223 http://0.0.0.0:12223
0.0.0.0:* LISTEN 34681/haproxy
tcp 0 0 127.0.0.1:20351 http://127.0.0.1:20351
0.0.0.0:* LISTEN 7917/python
tcp 0 0 127.0.0.1:35201 http://127.0.0.1:35201
0.0.0.0:* LISTEN 7041/python
tcp 0 0 0.0.0.0:5665 http://0.0.0.0:5665
0.0.0.0:* LISTEN 4282/icinga2
tcp 0 0 127.0.0.1:35202 http://127.0.0.1:35202
0.0.0.0:* LISTEN 7042/python
tcp 0 0 127.0.0.1:35301 http://127.0.0.1:35301
0.0.0.0:* LISTEN 8231/python
tcp 0 0 127.0.0.1:17030 http://127.0.0.1:17030
0.0.0.0:* LISTEN 35899/gunicorn
tcp 0 0 127.0.0.1:35302 http://127.0.0.1:35302
0.0.0.0:* LISTEN 8232/python
tcp 0 0 127.0.0.1:17031 http://127.0.0.1:17031
0.0.0.0:* LISTEN 35923/gunicorn
tcp 0 0 0.0.0.0:13223 http://0.0.0.0:13223
0.0.0.0:* LISTEN 7932/haproxy
tcp 0 0 127.0.0.1:35111 http://127.0.0.1:35111
0.0.0.0:* LISTEN 7495/python
tcp 0 0 127.0.0.1:199 http://127.0.0.1:199
0.0.0.0:* LISTEN 2533/snmpd
tcp 0 0 127.0.0.1:17032 http://127.0.0.1:17032
0.0.0.0:* LISTEN 35958/gunicorn
tcp 0 0 127.0.0.1:17033 http://127.0.0.1:17033
0.0.0.0:* LISTEN 35982/gunicorn
tcp 0 0 127.0.0.1:35211 http://127.0.0.1:35211
0.0.0.0:* LISTEN 7043/python
tcp 0 0 0.0.0.0:139 http://0.0.0.0:139
0.0.0.0:* LISTEN 4399/smbd
tcp 0 0 0.0.0.0:6379 http://0.0.0.0:6379
0.0.0.0:* LISTEN 3235/redis-server *
tcp 0 0 127.0.0.1:35311 http://127.0.0.1:35311
0.0.0.0:* LISTEN 8233/python
tcp 0 0 0.0.0.0:6479 http://0.0.0.0:6479
0.0.0.0:* LISTEN 3243/redis-server *
tcp 0 0 0.0.0.0:111 http://0.0.0.0:111
0.0.0.0:* LISTEN 1604/rpcbind
tcp 0 0 0.0.0.0:80 http://0.0.0.0:80
0.0.0.0:* LISTEN 3532/httpd
tcp 0 0 127.0.0.1:17010 http://127.0.0.1:17010
0.0.0.0:* LISTEN 35761/gunicorn
tcp 0 0 127.0.0.1:17011 http://127.0.0.1:17011
0.0.0.0:* LISTEN 35777/gunicorn
tcp 0 0 0.0.0.0:6579 http://0.0.0.0:6579
0.0.0.0:* LISTEN 3252/redis-server *
tcp 0 0 127.0.0.1:17012 http://127.0.0.1:17012
0.0.0.0:* LISTEN 35811/gunicorn
tcp 0 0 127.0.0.1:17013 http://127.0.0.1:17013
0.0.0.0:* LISTEN 35832/gunicorn
tcp 0 0 0.0.0.0:22 http://0.0.0.0:22
0.0.0.0:* LISTEN 3217/sshd
tcp 0 0 0.0.0.0:11223 http://0.0.0.0:11223
0.0.0.0:* LISTEN 40566/haproxy
tcp 0 0 127.0.0.1:20151 http://127.0.0.1:20151
0.0.0.0:* LISTEN 40327/python
tcp 0 0 0.0.0.0:8183 http://0.0.0.0:8183
0.0.0.0:* LISTEN 35859/python
tcp 0 0 0.0.0.0:6679 http://0.0.0.0:6679
0.0.0.0:* LISTEN 3260/redis-server *
tcp 0 0 127.0.0.1:631 http://127.0.0.1:631
0.0.0.0:* LISTEN 2180/cupsd
tcp 0 0 0.0.0.0:35287 http://0.0.0.0:35287
0.0.0.0:* LISTEN 1802/rpc.statd
tcp 0 0 127.0.0.1:5432 http://127.0.0.1:5432
0.0.0.0:* LISTEN 3315/postmaster
tcp 0 0 127.0.0.1:25 http://127.0.0.1:25
0.0.0.0:* LISTEN 3406/master
tcp 0 0 0.0.0.0:8283 http://0.0.0.0:8283
0.0.0.0:* LISTEN 35303/python
tcp 0 0 127.0.0.1:20251 http://127.0.0.1:20251
0.0.0.0:* LISTEN 34669/python
tcp 0 0 0.0.0.0:6779 http://0.0.0.0:6779
0.0.0.0:* LISTEN 3268/redis-server *
tcp 0 0 127.0.0.1:17020 http://127.0.0.1:17020
0.0.0.0:* LISTEN 35201/gunicorn
tcp 0 0 :::5989 :::*
LISTEN 4516/cimserver
tcp 0 0 :::6379 :::*
LISTEN 3235/redis-server *
tcp 0 0 :::6479 :::*
LISTEN 3243/redis-server *
tcp 0 0 :::6579 :::*
LISTEN 3252/redis-server *
tcp 0 0 :::6679 :::*
LISTEN 3260/redis-server *
tcp 0 0 :::6779 :::*
LISTEN 3268/redis-server *
udp 0 0 0.0.0.0:37132 http://0.0.0.0:37132
0.0.0.0:* 7932/haproxy
udp 0 0 0.0.0.0:913 http://0.0.0.0:913
0.0.0.0:* 1604/rpcbind
udp 0 0 0.0.0.0:161 http://0.0.0.0:161
0.0.0.0:* 2533/snmpd
udp 0 0 0.0.0.0:58916 http://0.0.0.0:58916
0.0.0.0:* 40566/haproxy
udp 0 0 0.0.0.0:33079 http://0.0.0.0:33079
0.0.0.0:* 39772/haproxy
udp 0 0 127.0.0.1:708 http://127.0.0.1:708
0.0.0.0:* 1802/rpc.statd
udp 0 0 0.0.0.0:50258 http://0.0.0.0:50258
0.0.0.0:* 34681/haproxy
udp 0 0 0.0.0.0:54889 http://0.0.0.0:54889
0.0.0.0:* 32941/haproxy
udp 0 0 0.0.0.0:53610 http://0.0.0.0:53610
0.0.0.0:* 1802/rpc.statd
udp 0 0 0.0.0.0:111 http://0.0.0.0:111
0.0.0.0:* 1604/rpcbind
udp 0 0 0.0.0.0:631 http://0.0.0.0:631
0.0.0.0:* 2180/cupsd
udp 0 0 172.16.167.22:123 http://172.16.167.22:123
0.0.0.0:* 3228/ntpd
udp 0 0 127.0.0.1:123 http://127.0.0.1:123
0.0.0.0:* 3228/ntpd
udp 0 0 0.0.0.0:123 http://0.0.0.0:123
0.0.0.0:* 3228/ntpd
udp 0 0 :::123 :::*
3228/ntpd

Ok - I can see 12 Zato server processes running in this system. What
Zato version is it and how many CPUs do you have allocated to that OS?


Dariusz Suchojad

https://zato.io
ESB, SOA, REST, APIs and Cloud Integrations in Python

On 26/02/16 16:09, Szymon Kalinowski wrote:

[root@xxx szk]# cat /proc/cpuinfo | grep "model name"
model name : Intel® Xeon® CPU E5-2650 0 @ 2.00GHz
model name : Intel® Xeon® CPU E5-2650 0 @ 2.00GHz
model name : Intel® Xeon® CPU E5-2650 0 @ 2.00GHz
model name : Intel® Xeon® CPU E5-2650 0 @ 2.00GHz

There are 4 physical cores with 12 server processes.

I understand that this is likely a testbed environment but in an actual
one, if performance is a factor, you need to always have 1 server
process = 1 CPU and that server process should have its CPU affinity set
from command line so that its main thread does not migrate across CPUs.

But that was an aside only.

The actual question is if you are 100% sure that you are testing the
correct load-balancer?

There are 3 load-balancers in this system - are you positive your
changes in timeouts were applied to the one you are testing things with?