Hi @dsuch
Below is the output from info
None of this looks alarming to me…
What is concerning though is that a single server is using A LOT more RAM than another. If I understand what you are saying correctly, the servers will evenly pull messages from the tasks queue and there shouldn’t be this much bias of resources.
Server 1
+--------------------------------+--------------------------------------------------------------------------------------------+
| Key | Value |
+================================+============================================================================================+
| component_details | {"created_user_host": "zato@ip-10-137-170-186", "version": "Zato 2.0.8.rev-050c6697", |
| | "component": "SERVER", "created_ts": "2018-01-31T08:57:09.957902"} |
+--------------------------------+--------------------------------------------------------------------------------------------+
| component_full_path | /opt/zato/env/server1 |
+--------------------------------+--------------------------------------------------------------------------------------------+
| component_host | ip-10-137-170-186/ip-10-137-170-186.ec2.internal |
+--------------------------------+--------------------------------------------------------------------------------------------+
| component_running | True |
+--------------------------------+--------------------------------------------------------------------------------------------+
| current_time | 2018-04-16T12:35:24.941080 |
+--------------------------------+--------------------------------------------------------------------------------------------+
| current_time_utc | 2018-04-16T12:35:24.941100 |
+--------------------------------+--------------------------------------------------------------------------------------------+
| master_proc_connections | {'LISTEN': [{u'to': None, u'from': u'127.0.0.1:17010', u'formatted': u'127.0.0.1:17010'}]} |
+--------------------------------+--------------------------------------------------------------------------------------------+
| master_proc_create_time | 2018-04-16T08:11:36.930000 |
+--------------------------------+--------------------------------------------------------------------------------------------+
| master_proc_create_time_utc | 2018-04-16T08:11:36.930000+00:00 |
+--------------------------------+--------------------------------------------------------------------------------------------+
| master_proc_name | gunicorn: |
+--------------------------------+--------------------------------------------------------------------------------------------+
| master_proc_pid | 1610 |
+--------------------------------+--------------------------------------------------------------------------------------------+
| master_proc_username | zato |
+--------------------------------+--------------------------------------------------------------------------------------------+
| master_proc_workers_no | 2 |
+--------------------------------+--------------------------------------------------------------------------------------------+
| master_proc_workers_pids | [1619, 1620] |
+--------------------------------+--------------------------------------------------------------------------------------------+
| worker_1619_connections | {'ESTABLISHED': [{u'to': u'127.0.0.1:6379', u'from': u'127.0.0.1:56644', u'formatted': |
| | u'127.0.0.1:56644 -> 127.0.0.1:6379'}, {u'to': u'127.0.0.1:6379', u'from': |
| | u'127.0.0.1:56900', u'formatted': u'127.0.0.1:56900 -> 127.0.0.1:6379'}, |
| | {u'to': u'127.0.0.1:6379', u'from': u'127.0.0.1:56970', u'formatted': u'127.0.0.1:56970 |
| | -> 127.0.0.1:6379'}, {u'to': u'127.0.0.1:6379', u'from': u'127.0.0.1:56948', |
| | u'formatted': u'127.0.0.1:56948 -> 127.0.0.1:6379'}, {u'to': |
| | u'127.0.0.1:6379', u'from': u'127.0.0.1:56902', u'formatted': u'127.0.0.1:56902 -> |
| | 127.0.0.1:6379'}, {u'to': u'127.0.0.1:6379', u'from': u'127.0.0.1:56904', u'formatted': |
| | u'127.0.0.1:56904 -> 127.0.0.1:6379'}, {u'to': u'127.0.0.1:6379', u'from': |
| | u'127.0.0.1:35182', u'formatted': u'127.0.0.1:35182 -> 127.0.0.1:6379'}, |
| | {u'to': u'127.0.0.1:6379', u'from': u'127.0.0.1:33862', u'formatted': u'127.0.0.1:33862 |
| | -> 127.0.0.1:6379'}], 'LISTEN': [{u'to': None, u'from': u'127.0.0.1:17010', |
| | u'formatted': u'127.0.0.1:17010'}]} |
+--------------------------------+--------------------------------------------------------------------------------------------+
| worker_1619_create_time | 2018-04-16T08:11:39.670000 |
+--------------------------------+--------------------------------------------------------------------------------------------+
| worker_1619_create_time_utc | 2018-04-16T08:11:39.670000+00:00 |
+--------------------------------+--------------------------------------------------------------------------------------------+
| worker_1620_connections | {'ESTABLISHED': [{u'to': u'127.0.0.1:6379', u'from': u'127.0.0.1:57892', u'formatted': |
| | u'127.0.0.1:57892 -> 127.0.0.1:6379'}, {u'to': u'127.0.0.1:6379', u'from': |
| | u'127.0.0.1:50898', u'formatted': u'127.0.0.1:50898 -> 127.0.0.1:6379'}, |
| | {u'to': u'127.0.0.1:6379', u'from': u'127.0.0.1:56646', u'formatted': u'127.0.0.1:56646 |
| | -> 127.0.0.1:6379'}, {u'to': u'127.0.0.1:6379', u'from': u'127.0.0.1:35186', |
| | u'formatted': u'127.0.0.1:35186 -> 127.0.0.1:6379'}, {u'to': |
| | u'127.0.0.1:6379', u'from': u'127.0.0.1:56932', u'formatted': u'127.0.0.1:56932 -> |
| | 127.0.0.1:6379'}, {u'to': u'127.0.0.1:6379', u'from': u'127.0.0.1:57346', u'formatted': |
| | u'127.0.0.1:57346 -> 127.0.0.1:6379'}, {u'to': u'137.117.185.69:443', |
| | u'from': u'10.148.3.19:41718', u'formatted': u'10.148.3.19:41718 -> |
| | 137.117.185.69:443'}, {u'to': u'127.0.0.1:6379', u'from': u'127.0.0.1:35184', |
| | u'formatted': u'127.0.0.1:35184 -> 127.0.0.1:6379'}, {u'to': |
| | u'52.201.30.197:443', u'from': u'10.148.3.19:46870', u'formatted': u'10.148.3.19:46870 |
| | -> 52.201.30.197:443'}], 'CLOSE_WAIT': [{u'to': u'52.206.189.228:443', u'from': |
| | u'10.148.3.19:50712', u'formatted': u'10.148.3.19:50712 -> 52.206.189.228:443'}, |
| | {u'to': u'52.201.30.197:443', u'from': u'10.148.3.19:43262', u'formatted': |
| | u'10.148.3.19:43262 -> 52.201.30.197:443'}], 'LISTEN': [{u'to': None, u'from': |
| | u'127.0.0.1:17010', u'formatted': u'127.0.0.1:17010'}]} |
+--------------------------------+--------------------------------------------------------------------------------------------+
| worker_1620_create_time | 2018-04-16T08:11:39.690000 |
+--------------------------------+--------------------------------------------------------------------------------------------+
| worker_1620_create_time_utc | 2018-04-16T08:11:39.690000+00:00 |
+--------------------------------+--------------------------------------------------------------------------------------------+
Server 2
+--------------------------------+--------------------------------------------------------------------------------------------+
| Key | Value |
+================================+============================================================================================+
| component_details | {"created_user_host": "zato@ip-10-137-170-186", "version": "Zato 2.0.8.rev-050c6697", |
| | "component": "SERVER", "created_ts": "2018-01-31T08:57:09.972700"} |
+--------------------------------+--------------------------------------------------------------------------------------------+
| component_full_path | /opt/zato/env/server2 |
+--------------------------------+--------------------------------------------------------------------------------------------+
| component_host | ip-10-137-170-186/ip-10-137-170-186.ec2.internal |
+--------------------------------+--------------------------------------------------------------------------------------------+
| component_running | True |
+--------------------------------+--------------------------------------------------------------------------------------------+
| current_time | 2018-04-16T12:37:47.884327 |
+--------------------------------+--------------------------------------------------------------------------------------------+
| current_time_utc | 2018-04-16T12:37:47.884347 |
+--------------------------------+--------------------------------------------------------------------------------------------+
| master_proc_connections | {'LISTEN': [{u'to': None, u'from': u'127.0.0.1:17011', u'formatted': u'127.0.0.1:17011'}]} |
+--------------------------------+--------------------------------------------------------------------------------------------+
| master_proc_create_time | 2018-04-16T08:11:40 |
+--------------------------------+--------------------------------------------------------------------------------------------+
| master_proc_create_time_utc | 2018-04-16T08:11:40+00:00 |
+--------------------------------+--------------------------------------------------------------------------------------------+
| master_proc_name | gunicorn: |
+--------------------------------+--------------------------------------------------------------------------------------------+
| master_proc_pid | 1621 |
+--------------------------------+--------------------------------------------------------------------------------------------+
| master_proc_username | zato |
+--------------------------------+--------------------------------------------------------------------------------------------+
| master_proc_workers_no | 2 |
+--------------------------------+--------------------------------------------------------------------------------------------+
| master_proc_workers_pids | [1631, 1632] |
+--------------------------------+--------------------------------------------------------------------------------------------+
| worker_1631_connections | {'ESTABLISHED': [{u'to': u'52.201.30.197:443', u'from': u'10.148.3.19:47208', |
| | u'formatted': u'10.148.3.19:47208 -> 52.201.30.197:443'}, {u'to': |
| | u'127.0.0.1:6379', u'from': u'127.0.0.1:35188', u'formatted': u'127.0.0.1:35188 -> |
| | 127.0.0.1:6379'}, {u'to': u'137.117.185.69:443', u'from': u'10.148.3.19:41350', |
| | u'formatted': u'10.148.3.19:41350 -> 137.117.185.69:443'}, {u'to': |
| | u'127.0.0.1:6379', u'from': u'127.0.0.1:56858', u'formatted': u'127.0.0.1:56858 -> |
| | 127.0.0.1:6379'}, {u'to': u'127.0.0.1:6379', u'from': u'127.0.0.1:58748', u'formatted': |
| | u'127.0.0.1:58748 -> 127.0.0.1:6379'}, {u'to': u'127.0.0.1:6379', u'from': |
| | u'127.0.0.1:56656', u'formatted': u'127.0.0.1:56656 -> 127.0.0.1:6379'}, |
| | {u'to': u'127.0.0.1:6379', u'from': u'127.0.0.1:56856', u'formatted': u'127.0.0.1:56856 |
| | -> 127.0.0.1:6379'}, {u'to': u'127.0.0.1:6379', u'from': u'127.0.0.1:56854', |
| | u'formatted': u'127.0.0.1:56854 -> 127.0.0.1:6379'}, {u'to': |
| | u'127.0.0.1:6379', u'from': u'127.0.0.1:56852', u'formatted': u'127.0.0.1:56852 -> |
| | 127.0.0.1:6379'}, {u'to': u'52.206.189.228:443', u'from': u'10.148.3.19:51694', |
| | u'formatted': u'10.148.3.19:51694 -> 52.206.189.228:443'}], 'CLOSE_WAIT': [{u'to': |
| | u'52.206.189.228:443', u'from': u'10.148.3.19:51116', u'formatted': u'10.148.3.19:51116 |
| | -> 52.206.189.228:443'}, {u'to': u'52.206.189.228:443', u'from': u'10.148.3.19:41608', |
| | u'formatted': u'10.148.3.19:41608 -> 52.206.189.228:443'}, {u'to': |
| | u'158.85.167.221:80', u'from': u'10.148.3.19:60504', u'formatted': u'10.148.3.19:60504 |
| | -> 158.85.167.221:80'}, {u'to': u'52.201.30.197:443', u'from': u'10.148.3.19:46500', |
| | u'formatted': u'10.148.3.19:46500 -> 52.201.30.197:443'}, {u'to': |
| | u'52.206.189.228:443', u'from': u'10.148.3.19:49324', u'formatted': u'10.148.3.19:49324 |
| | -> 52.206.189.228:443'}, {u'to': u'52.206.189.228:443', u'from': u'10.148.3.19:50024', |
| | u'formatted': u'10.148.3.19:50024 -> 52.206.189.228:443'}], 'LISTEN': [{u'to': |
| | None, u'from': u'127.0.0.1:17011', u'formatted': u'127.0.0.1:17011'}]} |
+--------------------------------+--------------------------------------------------------------------------------------------+
| worker_1631_create_time | 2018-04-16T08:11:42.120000 |
+--------------------------------+--------------------------------------------------------------------------------------------+
| worker_1631_create_time_utc | 2018-04-16T08:11:42.120000+00:00 |
+--------------------------------+--------------------------------------------------------------------------------------------+
| worker_1632_connections | {'ESTABLISHED': [{u'to': u'127.0.0.1:6379', u'from': u'127.0.0.1:56972', u'formatted': |
| | u'127.0.0.1:56972 -> 127.0.0.1:6379'}, {u'to': u'127.0.0.1:6379', u'from': |
| | u'127.0.0.1:56934', u'formatted': u'127.0.0.1:56934 -> 127.0.0.1:6379'}, |
| | {u'to': u'127.0.0.1:6379', u'from': u'127.0.0.1:57884', u'formatted': u'127.0.0.1:57884 |
| | -> 127.0.0.1:6379'}, {u'to': u'137.117.185.69:443', u'from': u'10.148.3.19:42042', |
| | u'formatted': u'10.148.3.19:42042 -> 137.117.185.69:443'}, {u'to': |
| | u'127.0.0.1:6379', u'from': u'127.0.0.1:59350', u'formatted': u'127.0.0.1:59350 -> |
| | 127.0.0.1:6379'}, {u'to': u'52.201.30.197:443', u'from': u'10.148.3.19:47194', |
| | u'formatted': u'10.148.3.19:47194 -> 52.201.30.197:443'}, {u'to': |
| | u'127.0.0.1:6379', u'from': u'127.0.0.1:56658', u'formatted': u'127.0.0.1:56658 -> |
| | 127.0.0.1:6379'}], 'CLOSE_WAIT': [{u'to': u'52.201.30.197:443', u'from': |
| | u'10.148.3.19:45148', u'formatted': u'10.148.3.19:45148 -> 52.201.30.197:443'}, |
| | {u'to': u'52.206.189.228:443', u'from': u'10.148.3.19:48794', u'formatted': |
| | u'10.148.3.19:48794 -> 52.206.189.228:443'}], 'LISTEN': [{u'to': None, u'from': |
| | u'127.0.0.1:17011', u'formatted': u'127.0.0.1:17011'}]} |
+--------------------------------+--------------------------------------------------------------------------------------------+
| worker_1632_create_time | 2018-04-16T08:11:42.200000 |
+--------------------------------+--------------------------------------------------------------------------------------------+
| worker_1632_create_time_utc | 2018-04-16T08:11:42.200000+00:00 |
+--------------------------------+--------------------------------------------------------------------------------------------+