Errors/warnings while installing/running Zato into Docker

jsabater · October 9, 2018, 10:21am

Hi everyone!

So I deleted my previous container (to give the problems described in this thread a fresh approach) and, while following the instructions to install Zato 3.0 into Docker (latest version) from scratch, I noticed these errors/warnings:

Step 8/37 : RUN git pull
 ---> Running in 9a16b6e6b3b9
From https://github.com/zatosource/zato
   f5dbf26..acf62f0  support/3.0 -> origin/support/3.0
   0901b2f..e4f4db5  dsuch-f-gh755-pub-sub-sql-dsuch2 -> origin/dsuch-f-gh755-pub-sub-sql-dsuch2
   9fe9b34..eadc46a  dsuch-f-gh847-pre-3.0-cleanup -> origin/dsuch-f-gh847-pre-3.0-cleanup
 * [new branch]      dsuch-f-gh870-wsx-outconns -> origin/dsuch-f-gh870-wsx-outconns
 * [new branch]      dsuch-singleton1 -> origin/dsuch-singleton1
 * [new branch]      dsuch-tmp1 -> origin/dsuch-tmp1
   82ecf46..cb84372  main       -> origin/main
 * [new branch]      pre-support/3.0 -> origin/pre-support/3.0
 * [new branch]      revert-865-rtrind-f-gh862-create-scheduler -> origin/revert-865-rtrind-f-gh862-create-scheduler
   28b4288..b157ae5  u/nus      -> origin/u/nus
Updating f5dbf26..acf62f0
Fast-forward
 code/_install-alpine.sh                            |  10 +-
 code/_install-rhel.sh                              |   6 +-
[..]
 .../src/zato/admin/web/views/stats.py              |   2 +-
 code/zato-web-admin/test/__init__.py               |   2 +-
 code/zato-web-admin/test/zato/__init__.py          |   2 +-
 code/zato-web-admin/test/zato/admin/__init__.py    |   2 +-
 .../zato-web-admin/test/zato/admin/web/__init__.py |   2 +-
 .../test/zato/admin/web/views/__init__.py          |   2 +-
 .../test/zato/admin/web/views/test_stats.py        |   2 +-
 233 files changed, 4220 insertions(+), 1068 deletions(-)
 create mode 100644 code/zato-common/src/zato/common/odb/query/generic.py
 create mode 100644 code/zato-common/src/zato/common/wsx_client.py
 create mode 100644 code/zato-server/src/zato/server/base/worker/generic.py
 delete mode 100644 code/zato-server/src/zato/server/connection/web_socket/outgoing.py
 create mode 100644 code/zato-server/src/zato/server/generic/__init__.py
 create mode 100644 code/zato-server/src/zato/server/generic/api/__init__.py
 create mode 100644 code/zato-server/src/zato/server/generic/api/outconn_wsx.py
 create mode 100644 code/zato-server/src/zato/server/generic/connection.py
 create mode 100644 code/zato-server/src/zato/server/service/internal/generic/__init__.py
 create mode 100644 code/zato-server/src/zato/server/service/internal/generic/connection.py
 create mode 100644 code/zato-web-admin/src/zato/admin/static/js/outgoing/wsx.js
 create mode 100644 code/zato-web-admin/src/zato/admin/templates/zato/outgoing/wsx.html
 create mode 100644 code/zato-web-admin/src/zato/admin/web/forms/outgoing/wsx.py
 create mode 100644 code/zato-web-admin/src/zato/admin/web/views/outgoing/wsx.py
Removing intermediate container 9a16b6e6b3b9
 ---> 4ff516bbc664
Step 9/37 : RUN ./bin/pip install -e ./zato-cy
 ---> Running in 5a3ccda4151c
Obtaining file:///opt/zato/3.0/code/zato-cy
watchdog 0.8.3 has requirement argh>=0.24.1, but you'll have argh 0.23.2 which is incompatible.
sphinx 1.7.5 has requirement Pygments>=2.0, but you'll have pygments 1.5 which is incompatible.
zato-apitest 1.11 has requirement base32-crockford==0.2.0, but you'll have base32-crockford 0.3.0 which is incompatible.
zato-apitest 1.11 has requirement cassandra-driver==2.1.1, but you'll have cassandra-driver 3.0.0 which is incompatible.
zato-apitest 1.11 has requirement click==4.0, but you'll have click 1.1 which is incompatible.
zato-apitest 1.11 has requirement configobj==5.0.5, but you'll have configobj 5.0.6 which is incompatible.
zato-apitest 1.11 has requirement lxml==3.3.5, but you'll have lxml 3.7.3 which is incompatible.
zato-apitest 1.11 has requirement parse==1.6.4, but you'll have parse 1.6.6 which is incompatible.
zato-apitest 1.11 has requirement psycopg2==2.5.3, but you'll have psycopg2 2.7.4 which is incompatible.
zato-apitest 1.11 has requirement python-dateutil==2.2, but you'll have python-dateutil 2.6.0 which is incompatible.
zato-apitest 1.11 has requirement requests==2.3.0, but you'll have requests 2.19.0 which is incompatible.
zato-apitest 1.11 has requirement six==1.6.1, but you'll have six 1.10.0 which is incompatible.
zato-apitest 1.11 has requirement sqlalchemy==0.9.7, but you'll have sqlalchemy 1.2.8 which is incompatible.
Installing collected packages: zato-cy
  Found existing installation: zato-cy 3.0.0+rev.f5dbf26b
    Can't uninstall 'zato-cy'. No files were found to uninstall.
  Running setup.py develop for zato-cy
Successfully installed zato-cy
You are using pip version 10.0.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Some of them seem innocuous enough, like psycopg2 or sqlalchemy requested versions not being the latest, but some others… well, I am not sure. Should we be worried?

The installation did seem to finish okay, though. Then, when running the container, I noticed these initial error messages related to postgresql:

2018-10-09 10:03:29,127 CRIT Set uid to user 0
2018-10-09 10:03:29,132 INFO supervisord started with pid 8
2018-10-09 10:03:30,134 INFO spawned: 'postgresql' with pid 11
2018-10-09 10:03:30,136 INFO spawned: 'zato-web-admin' with pid 12
2018-10-09 10:03:30,138 INFO spawned: 'redis' with pid 13
2018-10-09 10:03:30,140 INFO spawned: 'zato-server1' with pid 14
2018-10-09 10:03:30,144 INFO spawned: 'zato-server2' with pid 16
2018-10-09 10:03:30,145 INFO spawned: 'ssh' with pid 17
2018-10-09 10:03:30,147 INFO spawned: 'zato-scheduler' with pid 18
2018-10-09 10:03:30,148 INFO spawned: 'zato-load-balancer' with pid 20
2018-10-09 10:03:30,150 DEBG fd 6 closed, stopped monitoring <POutputDispatcher at 139729000157912 for <Subprocess at 139729000524976 with name postgresql in state STARTING> (stdout)>
2018-10-09 10:03:30,151 DEBG 'postgresql' stderr output:
postgresql: unrecognized service

2018-10-09 10:03:30,151 DEBG fd 8 closed, stopped monitoring <POutputDispatcher at 139729000157552 for <Subprocess at 139729000524976 with name postgresql in state STARTING> (stderr)>
2018-10-09 10:03:30,151 INFO exited: postgresql (exit status 1; not expected)
2018-10-09 10:03:30,151 DEBG received SIGCLD indicating a child quit
2018-10-09 10:03:30,154 DEBG 'redis' stdout output:
13:C 09 Oct 10:03:30.152 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf

Grepping the installation inside the container I found these:

supervisord.conf:[program:postgresql]
supervisord.conf:command = service postgresql start
supervisord.log:2018-10-09 10:03:30,134 INFO spawned: 'postgresql' with pid 11
supervisord.log:2018-10-09 10:03:30,150 DEBG fd 6 closed, stopped monitoring <POutputDispatcher at 139729000157912 for <Subprocess at 139729000524976 with name postgresql in state STARTING> (stdout)>
supervisord.log:2018-10-09 10:03:30,151 DEBG 'postgresql' stderr output:
supervisord.log:postgresql: unrecognized service
supervisord.log:2018-10-09 10:03:30,151 DEBG fd 8 closed, stopped monitoring <POutputDispatcher at 139729000157552 for <Subprocess at 139729000524976 with name postgresql in state STARTING> (stderr)>
supervisord.log:2018-10-09 10:03:30,151 INFO exited: postgresql (exit status 1; not expected)
supervisord.log:2018-10-09 10:03:31,157 INFO spawned: 'postgresql' with pid 45
supervisord.log:2018-10-09 10:03:31,189 DEBG fd 6 closed, stopped monitoring <POutputDispatcher at 139729000157768 for <Subprocess at 139729000524976 with name postgresql in state STARTING> (stdout)>
supervisord.log:2018-10-09 10:03:31,190 DEBG 'postgresql' stderr output:
supervisord.log:postgresql: unrecognized service
supervisord.log:2018-10-09 10:03:31,190 DEBG fd 10 closed, stopped monitoring <POutputDispatcher at 139729000157552 for <Subprocess at 139729000524976 with name postgresql in state STARTING> (stderr)>
supervisord.log:2018-10-09 10:03:31,190 INFO exited: postgresql (exit status 1; not expected)
supervisord.log:2018-10-09 10:03:33,193 INFO spawned: 'postgresql' with pid 105
supervisord.log:2018-10-09 10:03:33,227 DEBG fd 6 closed, stopped monitoring <POutputDispatcher at 139729000158056 for <Subprocess at 139729000524976 with name postgresql in state STARTING> (stdout)>
supervisord.log:2018-10-09 10:03:33,227 DEBG 'postgresql' stderr output:
supervisord.log:postgresql: unrecognized service
supervisord.log:2018-10-09 10:03:33,227 DEBG fd 10 closed, stopped monitoring <POutputDispatcher at 139729000157552 for <Subprocess at 139729000524976 with name postgresql in state STARTING> (stderr)>
supervisord.log:2018-10-09 10:03:33,228 INFO exited: postgresql (exit status 1; not expected)
supervisord.log:2018-10-09 10:03:36,470 INFO spawned: 'postgresql' with pid 120
supervisord.log:2018-10-09 10:03:36,479 DEBG 'postgresql' stderr output:
supervisord.log:postgresql: unrecognized service
supervisord.log:2018-10-09 10:03:36,479 DEBG fd 10 closed, stopped monitoring <POutputDispatcher at 139729000157552 for <Subprocess at 139729000524976 with name postgresql in state STARTING> (stderr)>
supervisord.log:2018-10-09 10:03:36,479 DEBG fd 6 closed, stopped monitoring <POutputDispatcher at 139729000157768 for <Subprocess at 139729000524976 with name postgresql in state STARTING> (stdout)>
supervisord.log:2018-10-09 10:03:36,479 INFO exited: postgresql (exit status 1; not expected)
supervisord.log:2018-10-09 10:03:37,480 INFO gave up: postgresql entered FATAL state, too many start retries too quickly

The container started fine enough, it seems, and I am working with it right now, so far with no problems. Just thought these postgresql messages may require some touches on the configuration files

jsabater · October 9, 2018, 1:26pm

Also I noticed that, at the end of the installation, the following files have wrong owner and group (root, should be zato):

/opt/zato/3.0/.git/FETCH_HEAD
/opt/zato/3.0/.git/ORIG_HEAD
/opt/zato/3.0/.git/index

dsuch · October 9, 2018, 1:44pm

zato-apitest uses older dependencies and in fact it would be better to decouple it from the main installation so as not to have such spurious warnings.

As for PostgreSQL, the automatically created Zato cluster from the Dockerfile here uses SQLite rather than PostgreSQL so these warnings are harmless in the sense that your cluster will work, yet, you are right that it warrants a closer investigation because eventually I would like for the default Docker-based cluster to use PostgreSQL. But for now, this will not interfere with your installation.

jsabater · January 14, 2019, 8:44am

So after a while starting and stopping my Docker container, here you are some more information which I hope helps debug and improve the integration of Zato with Docker.

First of all, there’s a few differences between the Dockerfile. Here’s the list of exposed ports I use:
EXPOSE 6379 8183 17010 17011 11223
Here is the list of packages installed on Ubuntu:

# Install helper programs used during Zato installation
RUN apt-get update && apt-get install -y \
    apt-transport-https \
    apt-utils \
    curl \
    dialog \
    git \
    htop \
    libcurl4-openssl-dev \
    mc \
    net-tools \
    python-software-properties \
    redis-server \
    software-properties-common \
    sudo \
    supervisor \
    vim

Now, down to the bone, I am still getting a started Docker container that sometimes has both servers running and sometimes one of the two is down. So I’ve been collecting log files, which I’ve uploaded to Pastebin:

They all should have in common that both server1 or server2 are spawned but either of them is not running by the end of the log file (i.e. by the time I docker exec -u 0 -it musing_jennings bash and check the list of running processes).

I hope it helps.

dsuch · January 14, 2019, 9:53am

Thank you @jsabater,

like I said earlier, I have never seen it before, under Docker or otherwise, but I will investigate it in some time when I get around to the closer integration for Docker under orchestration tools.

In the meantime, can you tell me what your OS, kernel and Docker versions are? Also, is that a stock release or a derivative / customized version, e.g. offered by one of cloud providers?

jsabater · January 14, 2019, 11:10am

It’s my developer setup on my laptop. Ubuntu 18.04 with the latest Docker version:

KDE Neon (Ubuntu 18.04 with latest KDE on top).
Docker 18.09.1 from PPA at download.docker.com
Kernel 4.15.0-43

Inside the container there is, as per the installation instructions on Zato’s website:

Ubuntu 16.04.5 and Zato 3.x.
Kernel 4.15.0-43

The way I got here was:

Install Docker from the PPA in their website.
Install Zato inside Docker as per the instructions on the website.

A few, random notes:

My application is located at /opt/genesisng inside the container. Owner and group is Zato.
Application inside /opt/genesisng, as zato user, has been installed in development mode using pip: pip install -e .
Services are hot-deployed using docker cp genesisng/services/service.py <container-id>:/opt/zato/env/qs-1/server1/pickup/incoming/services/
~/env/qs-1/serverX/config/repo/server.conf includes user configuration:

    [user_config]
    genesisng=/opt/genesisng/genesisng/config.ini

Inside /opt/zato/3.0/code/zato_extra_paths there is a soft link genesisng pointing at /opt/genesisng

dsuch · January 29, 2019, 4:59am

Hello @jsabater,

I have updated the Dockerfile for a quickstart installation. It uses Ubuntu 18.04 instead of 16.04 and SQLite instead of PostgreSQL.

Can you try it out now and let me know if it works correctly for you? I have never been able to reproduce your exact situation but I think the above should help.

Thank you.

jsabater · April 9, 2019, 10:59am

I have tried the new Dockerfile and it works fine during installation, but when you try to start the Docker container a second time it attemps to create the zato database on PostgreSQL again and it fails:

$ docker start --interactive ecstatic_rosalind 
Reading the password for zato user from the file
Reading the password for web admin from the file
Initializing Postgresql database
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".

Data page checksums are disabled.

initdb: directory "/var/lib/postgresql/data" exists but is not empty
If you want to create a new database system, either remove or empty
the directory "/var/lib/postgresql/data" or run initdb
with an argument other than "/var/lib/postgresql/data".
pg_ctl: another server might be running; trying to start server anyway
waiting for server to start....2019-04-09 10:53:18.398 UTC [31] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2019-04-09 10:53:18.406 UTC [31] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2019-04-09 10:53:18.430 UTC [32] LOG:  database system was interrupted; last known up at 2019-04-08 11:42:24 UTC
2019-04-09 10:53:18.649 UTC [32] LOG:  database system was not properly shut down; automatic recovery in progress
2019-04-09 10:53:18.655 UTC [32] LOG:  redo starts at 0/25AFC28
2019-04-09 10:53:18.655 UTC [32] LOG:  invalid record length at 0/25AFD08: wanted 24, got 0
2019-04-09 10:53:18.655 UTC [32] LOG:  redo done at 0/25AFCD0
2019-04-09 10:53:18.714 UTC [31] LOG:  database system is ready to accept connections
 done
server started
2019-04-09 10:53:18.984 UTC [47] ERROR:  database "zato" already exists
2019-04-09 10:53:18.984 UTC [47] STATEMENT:  CREATE DATABASE "zato" ;
ERROR:  database "zato" already exists

PostgreSQL does not support IF NOT EXISTS on CREATE DATABASE, unfortunately.

Also may I suggest setting the database encoding to UTF8?

anielkis · April 9, 2019, 5:06pm

Hello, @jsabater, thank you for your report.

The correction was pushed, can you try it now and let me know if it works correctly for you?

Thank you.

jsabater · April 10, 2019, 8:29am

Hi, @anielkis!

During the first run (the creation of the container), a huge amount of duplicate constraint errors were shown on the console. Here you are an excerpt and some context to help you debug it:

[..]

2019-04-10 08:17:57,883 DEBG 'zato-server2' stdout output:
2019-04-10 08:17:57,883 - INFO - 216:MainThread - zato.common.repo:41 - Location [/opt/zato/env/qs-1/server2/config/repo] is not a Bazaar branch. Will turn it into one.

2019-04-10 08:17:57,883 DEBG 'zato-server1' stdout output:
2019-04-10 08:17:57,883 - INFO - 219:MainThread - zato.common.repo:41 - Location [/opt/zato/env/qs-1/server1/config/repo] is not a Bazaar branch. Will turn it into one.

2019-04-10 08:17:58,226 DEBG 'zato-server1' stdout output:
2019-04-10 08:17:58,225 - INFO - 219:MainThread - gunicorn.main:176 - Starting gunicorn 18.0

2019-04-10 08:17:58,228 DEBG 'zato-server2' stdout output:
2019-04-10 08:17:58,228 - INFO - 216:MainThread - gunicorn.main:176 - Starting gunicorn 18.0

2019-04-10 08:17:58,232 DEBG 'zato-server1' stdout output:
2019-04-10 08:17:58,231 - INFO - 219:MainThread - gunicorn.main:176 - Listening at: http://0.0.0.0:17010 (219)

2019-04-10 08:17:58,232 DEBG 'zato-server1' stdout output:
2019-04-10 08:17:58,232 - INFO - 219:MainThread - gunicorn.main:176 - Using worker: gevent

2019-04-10 08:17:58,233 DEBG 'zato-server2' stdout output:
2019-04-10 08:17:58,232 - INFO - 216:MainThread - gunicorn.main:176 - Listening at: http://0.0.0.0:17011 (216)

2019-04-10 08:17:58,233 DEBG 'zato-server2' stdout output:
2019-04-10 08:17:58,233 - INFO - 216:MainThread - gunicorn.main:176 - Using worker: gevent

2019-04-10 08:17:58,240 DEBG 'zato-server1' stdout output:
2019-04-10 08:17:58,239 - INFO - 278:MainThread - gunicorn.main:176 - Booting worker with pid: 278

2019-04-10 08:17:58,241 DEBG 'zato-server2' stdout output:
2019-04-10 08:17:58,240 - INFO - 279:MainThread - gunicorn.main:176 - Booting worker with pid: 279

2019-04-10 08:17:58,720 DEBG 'zato-server1' stdout output:
2019-04-10 08:17:58,719 - INFO - 278:MainThread - zato.server.base.parallel:449 - Preferred address of `server1@quickstart-758738` (pid: 278) is `http://172.17.0.1:17010`

2019-04-10 08:17:58,729 DEBG 'zato-server2' stdout output:
2019-04-10 08:17:58,729 - INFO - 279:MainThread - zato.server.base.parallel:449 - Preferred address of `server2@quickstart-758738` (pid: 279) is `http://172.17.0.1:17011`

2019-04-10 08:17:59,901 DEBG 'zato-server2' stdout output:
2019-04-10 08:17:59,900 - INFO - 279:MainThread - zato.server.service.store:283 - Deploying and caching internal services (server2)

2019-04-10 08:17:59,919 DEBG 'postgresql' stderr output:
2019-04-10 08:17:59.919 UTC [281] ERROR:  duplicate key value violates unique constraint "service_name_cluster_id_key"
2019-04-10 08:17:59.919 UTC [281] DETAIL:  Key (name, cluster_id)=(zato.security.rbac.role-permission.create, 1) already exists.
2019-04-10 08:17:59.919 UTC [281] STATEMENT:  INSERT INTO service (id, name, is_active, impl_name, is_internal, wsdl, wsdl_name, slow_threshold, opaque1, cluster_id) VALUES (nextval('service_id_seq'), $1, $2, $3, $4, $5, $6, $7, $8, $9) RETURNING service.id

2019-04-10 08:17:59,949 DEBG 'postgresql' stderr output:
2019-04-10 08:17:59.949 UTC [281] ERROR:  duplicate key value violates unique constraint "service_name_cluster_id_key"
2019-04-10 08:17:59.949 UTC [281] DETAIL:  Key (name, cluster_id)=(zato.security.rbac.role-permission.delete, 1) already exists.
2019-04-10 08:17:59.949 UTC [281] STATEMENT:  INSERT INTO service (id, name, is_active, impl_name, is_internal, wsdl, wsdl_name, slow_threshold, opaque1, cluster_id) VALUES (nextval('service_id_seq'), $1, $2, $3, $4, $5, $6, $7, $8, $9) RETURNING service.id

2019-04-10 08:17:59,980 DEBG 'postgresql' stderr output:
2019-04-10 08:17:59.980 UTC [281] ERROR:  duplicate key value violates unique constraint "service_name_cluster_id_key"
2019-04-10 08:17:59.980 UTC [281] DETAIL:  Key (name, cluster_id)=(zato.security.rbac.role-permission.get-list, 1) already exists.
2019-04-10 08:17:59.980 UTC [281] STATEMENT:  INSERT INTO service (id, name, is_active, impl_name, is_internal, wsdl, wsdl_name, slow_threshold, opaque1, cluster_id) VALUES (nextval('service_id_seq'), $1, $2, $3, $4, $5, $6, $7, $8, $9) RETURNING service.id

2019-04-10 08:17:59,997 DEBG 'postgresql' stderr output:
2019-04-10 08:17:59.997 UTC [281] ERROR:  duplicate key value violates unique constraint "service_name_cluster_id_key"
2019-04-10 08:17:59.997 UTC [281] DETAIL:  Key (name, cluster_id)=(zato.security.basic-auth.change-password, 1) already exists.
2019-04-10 08:17:59.997 UTC [281] STATEMENT:  INSERT INTO service (id, name, is_active, impl_name, is_internal, wsdl, wsdl_name, slow_threshold, opaque1, cluster_id) VALUES (nextval('service_id_seq'), $1, $2, $3, $4, $5, $6, $7, $8, $9) RETURNING service.id

[..]

2019-04-10 08:18:27,175 DEBG 'postgresql' stderr output:
2019-04-10 08:18:27.175 UTC [283] ERROR:  duplicate key value violates unique constraint "deployed_service_pkey"
2019-04-10 08:18:27.175 UTC [283] DETAIL:  Key (server_id, service_id)=(1, 343) already exists.
2019-04-10 08:18:27.175 UTC [283] STATEMENT:  INSERT INTO deployed_service (deployment_time, details, source, source_path, source_hash, source_hash_method, opaque1, server_id, service_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)

2019-04-10 08:18:27,227 DEBG 'zato-server2' stdout output:
2019-04-10 08:18:27,227 - INFO - 279:MainThread - zato.server.service.store:303 - Deployed and cached 558 internal services (server2)

2019-04-10 08:18:27,228 DEBG 'zato-server2' stdout output:
2019-04-10 08:18:27,227 - INFO - 279:MainThread - zato.server.base.parallel:263 - Deploying user-defined services (server2)

2019-04-10 08:18:27,228 DEBG 'zato-server2' stdout output:
2019-04-10 08:18:27,228 - INFO - 279:MainThread - zato.server.base.parallel:273 - Deployed 0 user-defined services  (server2)

2019-04-10 08:18:27,232 DEBG 'postgresql' stderr output:
2019-04-10 08:18:27.232 UTC [283] ERROR:  duplicate key value violates unique constraint "service_name_cluster_id_key"
2019-04-10 08:18:27.232 UTC [283] DETAIL:  Key (name, cluster_id)=(zato.ping, 1) already exists.
2019-04-10 08:18:27.232 UTC [283] STATEMENT:  INSERT INTO service (id, name, is_active, impl_name, is_internal, wsdl, wsdl_name, slow_threshold, opaque1, cluster_id) VALUES (nextval('service_id_seq'), $1, $2, $3, $4, $5, $6, $7, $8, $9) RETURNING service.id

[..]

2019-04-10 08:18:27,594 DEBG 'postgresql' stderr output:
2019-04-10 08:18:27.594 UTC [283] ERROR:  duplicate key value violates unique constraint "service_name_cluster_id_key"
2019-04-10 08:18:27.594 UTC [283] DETAIL:  Key (name, cluster_id)=(helpers.web-sockets-gateway, 1) already exists.
2019-04-10 08:18:27.594 UTC [283] STATEMENT:  INSERT INTO service (id, name, is_active, impl_name, is_internal, wsdl, wsdl_name, slow_threshold, opaque1, cluster_id) VALUES (nextval('service_id_seq'), $1, $2, $3, $4, $5, $6, $7, $8, $9) RETURNING service.id

2019-04-10 08:18:27,611 DEBG 'postgresql' stderr output:
2019-04-10 08:18:27.611 UTC [283] ERROR:  duplicate key value violates unique constraint "service_name_cluster_id_key"
2019-04-10 08:18:27.611 UTC [283] DETAIL:  Key (name, cluster_id)=(helpers.web-sockets-pub-sub-gateway, 1) already exists.
2019-04-10 08:18:27.611 UTC [283] STATEMENT:  INSERT INTO service (id, name, is_active, impl_name, is_internal, wsdl, wsdl_name, slow_threshold, opaque1, cluster_id) VALUES (nextval('service_id_seq'), $1, $2, $3, $4, $5, $6, $7, $8, $9) RETURNING service.id

2019-04-10 08:18:27,670 DEBG 'zato-server1' stdout output:
2019-04-10 08:18:27,670 - INFO - 278:MainThread - zato.server.service.store:303 - Deployed and cached 558 internal services (server1)

2019-04-10 08:18:27,671 DEBG 'zato-server1' stdout output:
2019-04-10 08:18:27,671 - INFO - 278:MainThread - zato.server.base.parallel:263 - Deploying user-defined services (server1)

2019-04-10 08:18:27,671 DEBG 'zato-server1' stdout output:
2019-04-10 08:18:27,671 - INFO - 278:MainThread - zato.server.base.parallel:273 - Deployed 0 user-defined services  (server1)

[..]

I could, however, start the container after the first run:

docker start --interactive agitated_antonelli

But the errors happened again, although the amount was not so huge:

[..]

2019-04-10 08:24:44,926 - INFO - 165:MainThread - gunicorn.main:176 - Booting worker with pid: 165

2019-04-10 08:24:45,015 DEBG fd 21 closed, stopped monitoring <POutputDispatcher at 140390672786568 for <Subprocess at 140390673089024 with name zato-server1 in state RUNNING> (stdout)>
2019-04-10 08:24:45,015 DEBG fd 25 closed, stopped monitoring <POutputDispatcher at 140390672786928 for <Subprocess at 140390673089024 with name zato-server1 in state RUNNING> (stderr)>
2019-04-10 08:24:45,016 INFO exited: zato-server1 (exit status 18; not expected)
2019-04-10 08:24:45,016 DEBG received SIGCLD indicating a child quit
2019-04-10 08:24:45,382 DEBG 'zato-server2' stdout output:
2019-04-10 08:24:45,382 - INFO - 165:MainThread - zato.server.base.parallel:449 - Preferred address of `server2@quickstart-758738` (pid: 165) is `http://172.17.0.1:17011`

2019-04-10 08:24:45,383 INFO spawned: 'zato-server1' with pid 168
2019-04-10 08:24:46,385 INFO success: zato-server1 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-04-10 08:24:47,647 DEBG 'zato-server2' stdout output:
2019-04-10 08:24:47,646 - INFO - 165:MainThread - zato.server.service.store:316 - Deploying 558 cached internal services (server2)

2019-04-10 08:24:47,991 DEBG 'postgresql' stderr output:
2019-04-10 08:24:47.991 UTC [186] ERROR:  duplicate key value violates unique constraint "deployed_service_pkey"
2019-04-10 08:24:47.991 UTC [186] DETAIL:  Key (server_id, service_id)=(2, 343) already exists.
2019-04-10 08:24:47.991 UTC [186] STATEMENT:  INSERT INTO deployed_service (deployment_time, details, source, source_path, source_hash, source_hash_method, opaque1, server_id, service_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)

[..]

2019-04-10 08:25:08,625 DEBG 'postgresql' stderr output:
2019-04-10 08:25:08.624 UTC [205] ERROR:  duplicate key value violates unique constraint "deployed_service_pkey"
2019-04-10 08:25:08.624 UTC [205] DETAIL:  Key (server_id, service_id)=(1, 343) already exists.
2019-04-10 08:25:08.624 UTC [205] STATEMENT:  INSERT INTO deployed_service (deployment_time, details, source, source_path, source_hash, source_hash_method, opaque1, server_id, service_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)

2019-04-10 08:25:08,886 DEBG 'postgresql' stderr output:
2019-04-10 08:25:08.886 UTC [203] ERROR:  duplicate key value violates unique constraint "deployed_service_pkey"
2019-04-10 08:25:08.886 UTC [203] DETAIL:  Key (server_id, service_id)=(1, 281) already exists.
2019-04-10 08:25:08.886 UTC [203] STATEMENT:  INSERT INTO deployed_service (deployment_time, details, source, source_path, source_hash, source_hash_method, opaque1, server_id, service_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)

2019-04-10 08:25:08,927 DEBG 'postgresql' stderr output:
2019-04-10 08:25:08.926 UTC [205] ERROR:  duplicate key value violates unique constraint "service_name_cluster_id_key"
2019-04-10 08:25:08.926 UTC [205] DETAIL:  Key (name, cluster_id)=(helpers.raw-request-logger, 1) already exists.
2019-04-10 08:25:08.926 UTC [205] STATEMENT:  INSERT INTO service (id, name, is_active, impl_name, is_internal, wsdl, wsdl_name, slow_threshold, opaque1, cluster_id) VALUES (nextval('service_id_seq'), $1, $2, $3, $4, $5, $6, $7, $8, $9) RETURNING service.id

2019-04-10 08:25:08,937 DEBG 'postgresql' stderr output:
2019-04-10 08:25:08.937 UTC [203] ERROR:  duplicate key value violates unique constraint "deployed_service_pkey"
2019-04-10 08:25:08.937 UTC [203] DETAIL:  Key (server_id, service_id)=(1, 1429) already exists.
2019-04-10 08:25:08.937 UTC [203] STATEMENT:  INSERT INTO deployed_service (deployment_time, details, source, source_path, source_hash, source_hash_method, opaque1, server_id, service_id) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)

2019-04-10 08:25:08,963 DEBG 'zato-server1' stdout output:
2019-04-10 08:25:08,963 - INFO - 201:MainThread - zato.server.service.store:321 - Deployed 558 cached internal services (server1)

[..]

anielkis · April 11, 2019, 9:02am

Thank you, @jsabater, these exceptions are harmless (part of the synchronization process) but we made a change to silence them out
Can you try it now?
Thanks

jsabater · April 12, 2019, 8:43am

Yup, all the constraint errors are gone. Cheers!