Tutorial to set up a test environment for Zato 3.1 and Python 3

#1

Hello everyone!

When testing the current state of the upcoming Zato version 3.1 under Python 3, I wrote down the steps I followed to have the testing environment inside a Docker container under Ubuntu 18.04 and I thought it would be good to share it with you all.

PostgreSQL (for Zato’s SQL Operational Database) and Redis (for Zato’s Key/Value Database) are kept outside of the Docker container for my convenience. Just use the instructions provided inside the container to have it all together.

Please let me know if you find any errors and I’ll fix them.

Redis

Install from the distribution sources:

sudo apt install --yes redis-server

The default configuration binds Redis to the localhost IP address, which is just what we need. Nonetheless, to test access to it the following command can be issued:

$ redis-cli -h 127.0.0.1 -p 6379 ping
PONG

In case of having to reinstall Zato it will be good to delete all keys from all existing databases, which can be done through the following command:

$ redis-cli -h 127.0.0.1 -p 6379 flushall
OK

PostgreSQL

Create the file /etc/apt/sources.list.d/pgdg.list and add the following line to it:

deb http://apt.postgresql.org/pub/repos/apt/ bionic-pgdg main

Import the repository signing key, and update the package lists:

wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
sudo apt-get update
sudo apt-cache policy postgresql-11

Install PostgreSQL:

sudo apt-get install postgresql-11 postgresql-client-11 postgresql-server-dev-11

Edit /etc/postgresql/11/main/pg_hba.conf to add access rules:

# IPv4 local connections:
host    zato            zato            127.0.0.1/32            trust
host    all             all             127.0.0.1/32            md5

And reload the server:

sudo service postgresql reload

Create the user and the database. As postgres user:

createuser --no-createdb --no-createrole --no-superuser zato
createdb --encoding=UTF8 --owner=zato --template=template0 zato

In case of having to reinstall Zato it will be necessary to recreate the zato database, which can be done through the following commands:

dropdb zato
createdb --encoding=UTF8 --owner=zato --template=template0 zato

Docker

Install Docker, as per instructions at Digital Ocean:

sudo apt install apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable" 
sudo apt update
sudo apt-cache policy docker-ce
sudo apt install docker-ce
sudo systemctl status docker

Add your username to the docker group so there’s no need to use sudo to launch the container:

sudo usermod -aG docker ${USER}

Apply the changes:

su - ${USER}

Ubuntu image inside Docker

Install the ubuntu docker image, which currently uses Ubuntu 18.04:

docker pull ubuntu

Run the container. Given that it is for for development purposes, we can just run the container using the network host mode instead of publishing ports:

docker run --interactive --tty --network="host" --name zato ubuntu

You are now root inside the container.

root@myhostname:~#

From outside of the container, use the docker command to find out the id and the name of the container:

$ docker ps --all
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                  PORTS               NAMES
d949fc88e3c0        ubuntu              "/bin/bash"         8 days ago          Up 2 days                                   zato
fe6629ea90cd        zato-3.0            "/entrypoint.sh"    9 days ago          Exited (0) 9 days ago                       vigilant_bassi

Whenever you exit the container (using the exit command), you can start it again with the following command:

docker start --interactive zato

And you can stop the container from outside with the following command:

docker stop zato

You can get a bash terminal, as root user, by executing the following command:

docker exec --user root --interactive --tty zato bash

Zato inside Docker using the Ubuntu image

Install dependencies in the form of Debian packages.

Install Python 3 and PIP:

apt install --yes python3 python3-dev python3-pip

Optionally, install helpers in the form of Debian packages:

apt install --yes apt-utils ccze dialog htop mc net-tools sudo tzdata vim

Install Git version control system:

apt-get install --yes git

Create the zato user and set a password:

groupadd zato useradd --comment "Zato Enterprise Service Bus" --home /opt/zato --create-home --shell /bin/bash --gid zato zato adduser zato sudo passwd zato

Become zato and clone the repository:

su - zato git clone https://github.com/zatosource/zato

Compile and install:

cd zato
./code/install.sh -p python3

If testing the connection from inside the container is necessary, try this:

$ sudo apt-get install postgresql-client $ psql --host=127.0.0.1 --username=zato --dbname=zato --tuples-only --no-align --command="SELECT 1"
1

Zato cluster

Create a cluster using the quickstart option:

$ mkdir --parents /opt/zato/env/qs-1
$ /opt/zato/zato/code/bin/zato quickstart create --odb_host 127.0.0.1 --odb_port 5432 --odb_user zato --odb_db_name zato --odb_password '' --kvdb_password '' /opt/zato/env/qs-1/ postgresql 127.0.0.1 6379
[1/9] Certificate authority created
[2/9] ODB schema created
[3/9] ODB initial data created
[4/9] server1 created
[5/9] server2 created
[6/9] Load-balancer created
Superuser created successfully.
[7/9] Web admin created
[8/9] Scheduler created
[9/9] Management scripts created
Quickstart cluster quickstart-726934 created
Web admin user:[admin], password:[9bR3Ith8w4LePItJ-8eulU9VgNnH4L8s]
Start the cluster by issuing the /opt/zato/env/qs-1/zato-qs-start.sh command
Visit https://zato.io/support for more information and support options

Edit the user’s profile file ~/.profile to add the binaries to the path:

# set PATH so it includes user's private bin if it exists
if [ -d "$HOME/zato/code/bin" ] ; then
    PATH="$HOME/zato/code/bin:$PATH" 
fi

Re-log as the zato user or reload the configuration:

source ~/.profile

Change the given password for the admin user, that will be required when accessing the web admin panel, by using the following command:

$ zato update password --password admin ~/env/qs-1/web-admin/ admin
Changing password for user 'admin'
OK

Start the cluster to test that the installation was correct:

$ /opt/zato/env/qs-1/zato-qs-start.sh
Starting Zato cluster quickstart-726934
Checking configuration
[1/8] Redis connection OK
[2/8] SQL ODB connection OK
[3/8] Checking TCP ports availability
[4/8] Load-balancer started
[5/8] server1 started
[6/8] server2 started
[7/8] Scheduler started
[8/8] Web admin started
Zato cluster quickstart-726934 started
Visit https://zato.io/support for more information and support options

$ ps --user zato x
  PID TTY      STAT   TIME COMMAND
   44 pts/0    S      0:00 -su
  131 pts/0    S      0:02 /opt/zato/zato/code/bin/python -m zato.agent.load_balancer.main /opt/zato/env/qs-1/load-balancer/config/repo fg=FalseZATO_ZATO_ZATOsync_internal=FalseZATO_ZATO_ZATOsecret_key=
  173 ?        Ss     0:00 haproxy -D -f /opt/zato/env/qs-1/load-balancer/config/repo/zato.config -p /opt/zato/env/qs-1/load-balancer/pidfile
  191 pts/0    Sl     0:02 gunicorn: master [gunicorn]
  232 pts/0    Sl     0:02 gunicorn: master [gunicorn]
  275 pts/0    Sl     0:02 /opt/zato/zato/code/bin/python -m zato.scheduler.main fg=FalseZATO_ZATO_ZATOsync_internal=FalseZATO_ZATO_ZATOsecret_key=
  307 pts/0    S      0:02 /opt/zato/zato/code/bin/python -m zato.admin.main fg=FalseZATO_ZATO_ZATOsync_internal=FalseZATO_ZATO_ZATOsecret_key=
  353 pts/0    R+     0:00 ps -u zato x

From here onwards the tutorial gets into installing the prototype application I am developing to test Zato. I leave it here as I find it potentially useful both as a reference and to discuss ways to use Zato. This application does not yet use Python Wheels to be installed (soon it will, hopefully):

Application

The next step is to install the application package inside the Docker container and let Zato have access to it.

A fresh installation can be done by using the Mercurial VCS or by copying all the necessary files and directories. Inside the Docker container, as root :

apt install mercurial
cd /opt
hg clone https://bitbucket.org/jsabater/genesisng

Change ownership of the repository to the user zato :

chown --recursive zato.zato /opt/genesisng

Install the package in development mode:

su - zato cd /opt/genesisng pip3 install --editable .

Make the package available to Zato as an extra path to the library:

ln --symbolic /opt/genesisng /opt/zato/zato/code/zato_extra_paths/genesisng

Edit the configuration files of both servers (located at /opt/zato/env/qs-1/server1/config/repo/server.conf and /opt/zato/env/qs-1/server2/config/repo/server.conf) to add the application local configuration file:

[user_config]
# All paths are either absolute or relative to the directory server.conf is in
genesisng=/opt/genesisng/genesisng/config.ini

Restart the server for the changes to take effect. As zato user:

/opt/zato/env/qs-1/zato-qs-restart.sh

From the working copy of the repository outside of the Docker container ( ~/Projects/genesisng ), hot-deploy the services:

cd ~/Projects/genesisng
for f in genesisng/services/*.py; do docker cp $f zato:/opt/zato/env/qs-1/server1/pickup/incoming/services/; done

Load the cluster configuration from the backup found at the config.yml file:

zato enmasse /opt/zato/env/qs-1/server1 --input /opt/genesisng/config.yml --import --replace-odb-objects

Configuration was originally created through the web admin panel and exported into that file by executing the following command:

zato enmasse /opt/zato/env/qs-1/server1 --export-odb --dump-format yaml

And copied to the repository outside of the Docker container by using the following command:

docker cp zato:/opt/zato/env/qs-1/config.yml ~/Projects/genesisng/
0 Likes

#2

Thank You - good stuff.

0 Likes