Python 3 support preview

#1

Hello,

I am happy to let you know that Python 3 support is available in Zato as a developer preview. Follow the instructions below to install it in your system.

  • Clone the repository:

    $ git clone https://github.com/zatosource/zato
    $ cd zato
    
  • Build Zato from source, providing the name of the Python binary to use

    $ ./code/install.sh -p python3.4
    
  • In the line above, python3.4 needs to point to an actual command, either via $PATH or as an absolute path, such as /usr/bin/python3.6

  • If you do not specify any Python binary, python2.7 will be used to preserve backward compatibility

  • Confirm the installation:

    $ ./code/bin/zato --version
    Zato 3.0.0+rev.a1b8e667-py3.4.3
    $
    
    
  • You can freely mix Python 2.7 and Python 3 in your environments - for instance, existing Python 2.7 environments can be used with Python 3 or the other way around

  • The point above means that it will be possible for current Python 2.7-based environments to start to use Python 3 as soon as Python 3 support is fully merged to supported branches - there will be no need to create new environments

  • If you spot anything unexpected make sure you git pull and, if it still persists, report it either here or through GitHub

Regards.

0 Likes

#2

Hello,

please note that Python 3 support is now merged to the main branch of development, to be released as Zato 3.1 in June 2019.

However, Python 3-related functionality will not be backported to branch support/3.0 - this is the moment when branches main and support/3.0 diverge.

0 Likes

#3

Nice work! Thank you!

0 Likes

#4

I’ll be testing this when I have the chance to set up a new development environment under Docker and Ubuntu 18.04. Thanks for the work. Looking forward to it!

0 Likes

#5

Hi, @dsuch!

I am setting an environment to test Python 3 support on Zato 3.1 (or, more specifically, the current status of the main branch of the repository) using the instructions to install from sources and to then use create quickstart to get started. These are the steps I have been following so far (please feel free to instruct me on how to do it better):

$ docker pull ubuntu
$ docker run --interactive --tty --network="host" ubuntu

$ apt install -y apt-transport-https apt-utils curl dialog git libcurl4-openssl-dev \
  python3-software-properties redis-server software-properties-common sudo supervisor \
  build-essential curl git haproxy libbz2-dev libev-dev libev4 libevent-dev \
  libffi-dev libkeyutils-dev libldap2-dev libmemcached-dev libpq-dev \
  libsasl2-dev libssl-dev libxml2-dev libxslt1-dev libyaml-dev openssl \
  python3 python3-dev python-pip swig uuid-dev uuid-runtime wget zlib1g-dev lsb-release
$ apt install -y htop mc net-tools vim

$ groupadd zato
$ useradd --comment "Zato Enterprise Service Bus" --home /opt/zato --create-home --shell /bin/bash --gid zato zato
$ adduser zato sudo
$ passwd zato

$ su - zato
$ git clone https://github.com/zatosource/zato
$ which python3
/usr/bin/python3
$ cd zato
$ ./code/install.sh -p python3

And this is how far I’ve reached:

$ mkdir --parents /opt/zato/env/qs-1
$ code/bin/zato quickstart create ~/env/qs-1 sqlite localhost 6379 --verbose

Key/value database password (will not echo): 
Kvdb_password again (will not echo): 
[1/9] Certificate authority created
[2/9] ODB schema created
[3/9] ODB initial data created
[4/9] server1 created
[5/9] server2 created
[6/9] Load-balancer created
File "code/bin/zato", line 11, in <module>
  File "/opt/zato/zato/code/zato-cli/src/zato/cli/zato_command.py", line 392, in main
    return run_command(get_parser().parse_args())
  File "/opt/zato/zato/code/zato-cli/src/zato/cli/__init__.py", line 374, in run_command
    command_class[args.command](args).run(args)
  Traceback (most recent call last):
  File "/opt/zato/zato/code/zato-cli/src/zato/cli/__init__.py", line 650, in run
    return_code = self.execute(args)
  File "/opt/zato/zato/code/zato-cli/src/zato/cli/quickstart.py", line 410, in execute
    create_web_admin_args, False, web_admin_password, True)
  File "/opt/zato/zato/code/zato-cli/src/zato/cli/create_web_admin.py", line 154, in execute
    update_globals(config, self.target_dir)
  File "/opt/zato/zato/code/zato-web-admin/src/zato/admin/zato_settings.py", line 38, in update_globals
    zato_secret_key = resolve_secret_key(zato_secret_key)
  File "/opt/zato/zato/code/zato-common/src/zato/common/crypto.py", line 379, in resolve_secret_key
    if not secret_key.startswith(_url_prefix):
TypeError: startswith first arg must be bytes or a tuple of bytes, not str

Incidentally, following the current instructions of Docker installation gives me no problem whatsoever, but that won’t get me to test Python 3 support.

0 Likes

#6

I cannot reproduce it, I use Zato 3.1.0pre1+rev.0a12fe04-py3.6.7 on Ubuntu 18.04. Can you pull latest commits from main, try again and tell me what your Zato and Ubuntu versions are?

As for the installation process, you do not need to create users or groups - you can install Zato as any user that has sudo powers simply by invoking install.sh.

Regards.

0 Likes

#7

Hi, @dsuch!

I am on it as of now, taking notes of each and every step I take. I am on Ubuntu 18.04 under Docker and the latest version from Git. I am trying to figure out the steps done by the Dockerfile as per the installation instructions and adapt them to the installation instructions from sources so that I have both (so to speak).

Regarding the user zato, I did that because I was mimicking the end result from the Dockerfile. Nevertheless, given that, when running or starting the Ubuntu Docker container one has but a root user, I’d say it’s a must.

May my troubles come from not adapting the default installations of HAProxy and Reddis packages?

I hope to be able to finish the job tomorrow and be able to provide more feedback. I’d like the spend a few days of the Easter holidays (from next Thursday until Monday where I live) to test the adaptation to Python 3 :slight_smile:

0 Likes

#8

Hi Jaume,

many thanks for getting involved - I have had a closer look at the past few commits and it turns out that one of them was related to this exception so I am quite confident it will no longer happen now.

0 Likes

#9

Hey there!

The previous error is gone but there are new ones. This is the command I executed as zato user:

/opt/zato/zato/code/bin/zato quickstart create --odb_host 127.0.0.1 --odb_port 5432 --odb_user zato --odb_db_name zato --odb_password '' --kvdb_password '' /opt/zato/env/qs-1/ postgresql localhost 6379

PostgreSQL is outside of the Docker container running Ubuntu 18.04 and the ODB database is zato, accessible via the username zato with no password. I checked access was right using psql inside the Docker container.

[1/9] Certificate authority created
[2/9] ODB schema already exists
[3/9] ODB initial data created
Could not create the server, e:`Traceback (most recent call last):
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 508, in do_execute
    cursor.execute(statement, parameters)
psycopg2.DataError: value too long for type character varying(32)


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/zato/zato/code/zato-cli/src/zato/cli/create_server.py", line 778, in execute
    session.commit()
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 943, in commit
    self.transaction.commit()
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 467, in commit
    self._prepare_impl()
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 447, in _prepare_impl
    self.session.flush()
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2254, in flush
    self._flush(objects)
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2380, in _flush
    transaction.rollback(_capture_exception=True)
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 187, in reraise
    raise value
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2344, in _flush
    flush_context.execute()
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 391, in execute
    rec.execute(self)
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 556, in execute
    uow
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 181, in save_obj
    mapper, table, insert)
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 866, in _emit_insert_statements
    execute(statement, params)
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 948, in execute
    return meth(self, multiparams, params)
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
    context)
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
    exc_info
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 186, in reraise
    raise value.with_traceback(tb)
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 508, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.DataError: (psycopg2.DataError) value too long for type character varying(32)
 [SQL: "INSERT INTO server (id, name, host, bind_host, bind_port, preferred_address, crypto_use_tls, last_join_status, last_join_mod_date, last_join_mod_by, up_status, up_mod_date, token, opaque1, cluster_id) VALUES (nextval('server_id_seq'), %(name)s, %(host)s, %(bind_host)s, %(bind_port)s, %(preferred_address)s, %(crypto_use_tls)s, %(last_join_status)s, %(last_join_mod_date)s, %(last_join_mod_by)s, %(up_status)s, %(up_mod_date)s, %(token)s, %(opaque1)s, %(cluster_id)s) RETURNING server.id"] [parameters: {'name': 'server1', 'host': None, 'bind_host': None, 'bind_port': None, 'preferred_address': None, 'crypto_use_tls': None, 'last_join_status': 'accepted', 'last_join_mod_date': datetime.datetime(2019, 4, 16, 14, 2, 44, 867894), 'last_join_mod_by': 'zato@ranma', 'up_status': None, 'up_mod_date': None, 'token': b'601ca7b42111451ea4f2ed74d78ac03d', 'opaque1': 'null', 'cluster_id': 2}] (Background on this error at: http://sqlalche.me/e/9h9h)
`
[4/9] server1 created
Could not create the server, e:`Traceback (most recent call last):
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 508, in do_execute
    cursor.execute(statement, parameters)
psycopg2.DataError: value too long for type character varying(32)


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/zato/zato/code/zato-cli/src/zato/cli/create_server.py", line 778, in execute
    session.commit()
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 943, in commit
    self.transaction.commit()
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 467, in commit
    self._prepare_impl()
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 447, in _prepare_impl
    self.session.flush()
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2254, in flush
    self._flush(objects)
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2380, in _flush
    transaction.rollback(_capture_exception=True)
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 187, in reraise
    raise value
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2344, in _flush
    flush_context.execute()
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 391, in execute
    rec.execute(self)
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 556, in execute
    uow
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 181, in save_obj
    mapper, table, insert)
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 866, in _emit_insert_statements
    execute(statement, params)
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 948, in execute
    return meth(self, multiparams, params)
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
    context)
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
    exc_info
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 186, in reraise
    raise value.with_traceback(tb)
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/opt/zato/zato/code/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 508, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.DataError: (psycopg2.DataError) value too long for type character varying(32)
 [SQL: "INSERT INTO server (id, name, host, bind_host, bind_port, preferred_address, crypto_use_tls, last_join_status, last_join_mod_date, last_join_mod_by, up_status, up_mod_date, token, opaque1, cluster_id) VALUES (nextval('server_id_seq'), %(name)s, %(host)s, %(bind_host)s, %(bind_port)s, %(preferred_address)s, %(crypto_use_tls)s, %(last_join_status)s, %(last_join_mod_date)s, %(last_join_mod_by)s, %(up_status)s, %(up_mod_date)s, %(token)s, %(opaque1)s, %(cluster_id)s) RETURNING server.id"] [parameters: {'name': 'server2', 'host': None, 'bind_host': None, 'bind_port': None, 'preferred_address': None, 'crypto_use_tls': None, 'last_join_status': 'accepted', 'last_join_mod_date': datetime.datetime(2019, 4, 16, 14, 2, 44, 912457), 'last_join_mod_by': 'zato@ranma', 'up_status': None, 'up_mod_date': None, 'token': b'2a2f6f15ce424fc3a1cc86d1c2428c48', 'opaque1': 'null', 'cluster_id': 2}] (Background on this error at: http://sqlalche.me/e/9h9h)
`
[5/9] server2 created
[6/9] Load-balancer created
[7/9] Web admin created
[8/9] Scheduler created
[9/9] Management scripts created
Quickstart cluster quickstart-730370 created
User [admin] already exists in the ODB
Start the cluster by issuing the /opt/zato/env/qs-1/zato-qs-start.sh command
Visit https://zato.io/support for more information and support options

Incidentally, I also found these warnings during the installation process (./code/install.sh -p python3):

hvac 0.7.2 has requirement requests>=2.21.0, but you'll have requests 2.19.0 which is incompatible.
jsonschema 3.0.1 has requirement six>=1.11.0, but you'll have six 1.10.0 which is incompatible.
warlock 1.1.0 has requirement jsonschema<3,>=0.7, but you'll have jsonschema 3.0.1 which is incompatible.
pyasn1-modules 0.2.4 has requirement pyasn1<0.5.0,>=0.4.1, but you'll have pyasn1 0.1.8 which is incompatible.
python-ldap 3.2.0 has requirement pyasn1>=0.3.7, but you'll have pyasn1 0.1.8 which is incompatible.

Thanks.

0 Likes

#10

Hi,

the last warnings about python packages, I think that if you lauch the install doing ‘sudo -H’ avoid that problems.

Regards.

0 Likes

#11

Thanks - can you try it now?

0 Likes

#12

Working fine now!

$ /opt/zato/zato/code/bin/zato quickstart create --odb_host 127.0.0.1 --odb_port 5432 --odb_user zato --odb_db_name zato --odb_password '' --kvdb_password '' /opt/zato/env/qs-1/ postgresql localhost 6379
[1/9] Certificate authority created
[2/9] ODB schema created
[3/9] ODB initial data created
[4/9] server1 created
[5/9] server2 created
[6/9] Load-balancer created
Superuser created successfully.
[7/9] Web admin created
[8/9] Scheduler created
[9/9] Management scripts created
Quickstart cluster quickstart-726934 created
Web admin user:[admin], password:[9bR3Ith8w4LePItJ-8eulU9VgNnH4L8s]
Start the cluster by issuing the /opt/zato/env/qs-1/zato-qs-start.sh command
Visit https://zato.io/support for more information and support options

Cheers!

0 Likes

#13

Next in line, there seems to be a problem with some libraries and gevent:

$ /opt/zato/env/qs-1/zato-qs-start.sh
Starting Zato cluster quickstart-726934
Checking configuration
[1/8] Redis connection OK
[2/8] SQL ODB connection OK
[3/8] Checking TCP ports availability
[4/8] Load-balancer started
Stderr received from program `/opt/zato/zato/code/bin/py -m zato.server.main /opt/zato/env/qs-1/server1 fg=FalseZATO_ZATO_ZATOsync_internal=FalseZATO_ZATO_ZATOsecret_key= 1> /dev/null 2> /tmp/tmprq0n_lur-zato-start-server.txt` e:`/opt/zato/zato/code/zato-server/src/zato/server/main.py:26: MonkeyPatchWarning: Monkey-patching ssl after ssl has already been imported may lead to errors, including RecursionError on Python 3.6. It may also silently lead to incorrect behaviour on Python 3.7. Please monkey-patch earlier. See https://github.com/gevent/gevent/issues/1016. Modules that had direct imports (NOT patched): ['urllib3.util (/opt/zato/zato/code/lib/python3.6/site-packages/urllib3/util/__init__.py)', 'urllib3.contrib.pyopenssl (/opt/zato/zato/code/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py)']. 
  patch_all()
`, kw:`{'async': True}`
0 Likes

#14

Troubleshooting and debugging of Zato 3.1 using Python 3.x continues on this issue on Github.

0 Likes