Zato InvalidToken

Community support
1

Hi

I started a quickstart cluster this morning, that has been running for about a week, (no code changes in that time) and it is now starting to fail with

2023-06-01 02:07:20,789 - INFO - 3469:MainThread - zato.server.base.parallel:0 - First worker of server1 is 3469
2023-06-01 02:07:20,795 - INFO - 3469:Dummy-13 - zato.common.util.api:0 - Getting configuration from /opt/zato/env/qs-1/server1/config/repo/server.conf
2023-06-01 02:07:20,801 - INFO - 3469:Dummy-13 - zato.common.util.api:0 - Getting configuration from /opt/zato/env/qs-1/server1/config/repo/sql.conf
2023-06-01 02:07:20,805 - ERROR - 3469:MainThread - zato:0 - Exception in worker process → (InvalidToken(), '')
Traceback (most recent call last):
File “/opt/zato/3.2.0/code/zato-server/src/zato/server/ext/zunicorn/arbiter.py”, line 638, in spawn_worker
self.cfg.post_fork(self, worker)
File “/opt/zato/3.2.0/code/zato-server/src/zato/server/base/parallel/init.py”, line 1635, in post_fork
ParallelServer.start_server(server, arbiter.zato_deployment_key)
File “/opt/zato/3.2.0/code/zato-server/src/zato/server/base/parallel/init.py”, line 1055, in start_server
self.init_ipc()
File “/opt/zato/3.2.0/code/zato-server/src/zato/server/base/parallel/init.py”, line 1100, in init_ipc
spawn_greenlet(self.ipc_api.start_server,
File “/opt/zato/3.2.0/code/zato-common/src/zato/common/util/api.py”, line 1709, in spawn_greenlet
raise_(exc_type, None, traceback)
File “/opt/zato/3.2.0/code/zato-common/src/zato/common/ext/future/utils/init.py”, line 464, in raise_
raise exc.with_traceback(tb)
File “src/gevent/greenlet.py”, line 908, in gevent.gevent_cgreenlet.Greenlet.run
File “/opt/zato/3.2.0/code/zato-common/src/zato/common/ipc/api.py”, line 69, in start_server
IPCServer.start(
File “/opt/zato/3.2.0/code/zato-common/src/zato/common/aux_server/base.py”, line 215, in start
config = class.config_class.from_repo_location(
File “/opt/zato/3.2.0/code/zato-common/src/zato/common/aux_server/base.py”, line 115, in from_repo_location
config.main.odb.password = config.crypto_manager.decrypt(config.main.odb.password)
File “/opt/zato/3.2.0/code/zato-common/src/zato/common/crypto/api.py”, line 230, in decrypt
return self.secret_key.decrypt(encrypted).decode(‘utf8’)
File “/opt/zato/3.2.0/code/lib/python3.8/site-packages/cryptography/fernet.py”, line 85, in decrypt
timestamp, data = Fernet._get_unverified_token_data(token)
File “/opt/zato/3.2.0/code/lib/python3.8/site-packages/cryptography/fernet.py”, line 118, in _get_unverified_token_data
raise InvalidToken
Exception: (InvalidToken(), ‘’)

If a run the get passwords script before it shuts down it is able to retrieve all of the passwords (for mysql)

2

A little bit more information>

TASK [Start the quickstart environment (server) (02)] **************************
changed: [localhost]

TASK [Start the quickstart environment (dashboard) (02)] ***********************
changed: [localhost]

TASK [Start the quickstart environment (load balancer) (02)] *******************
changed: [localhost]

TASK [Start the quickstart environment (scheduler) (02)] ***********************
changed: [localhost]

TASK [Waiting for the environment to start (02)] *******************************
Restarting Server
timh@LT-J4DHKS3:~/projects/zato$ fatal: [localhost]: FAILED! => {“changed”: true, “cmd”: “~/current/bin/zato wait --path /opt/zato/env/qs-1/server1 --timeout 120 --silent\n”, “delta”: “0:02:01.222995”, “end”: “2023-06-01 03:11:51.574415”, “msg”: “non-zero return code”, “rc”: 28, “start”: “2023-06-01 03:09:50.351420”, “stderr”: “”, “stderr_lines”: [], “stdout”: “”, “stdout_lines”: []}

3

Hi Tim,

this is corrected - please use the latest Docker image:

https://zato.io/en/docs/admin/guide/install/docker.html

4

Hi

Gave it a try, but the current docker image is now failing to start much earlier whilst trying to update apt packages

fatal: [localhost]: FAILED! => {“changed”: false, “msg”: “Failed to update apt cache: E:Release file for http://security.ubuntu.com/ubuntu/dists/focal-security/InRelease is not valid yet (invalid for another 12h 43min 28s). Updates for this repository will not be applied., E:Release file for http://archive.ubuntu.com/ubuntu/dists/focal-updates/InRelease is not valid yet (invalid for another 13h 46min 49s). Updates for this repository will not be applied., E:Release file for http://archive.ubuntu.com/ubuntu/dists/focal-backports/InRelease is not valid yet (invalid for another 12h 46min 42s). Updates for this repository will not be applied.”}

5

Arghh, ignore that my bad. Running under wsl2 and the clock was wrong ;-( (machine had slept)

6

However a separate issue had cropped up (at the same time as the original) and still persists

Scheduled jobs invoked manually and when they fire fail, with the following error

raise ConnectionError(e, request=request)

requests.exceptions.ConnectionError: HTTPConnectionPool(host=‘127.0.0.1’, port=31530): Max retries exceeded with url: / (Caused by NewConnectionError(‘<urllib3.connection.HTTPConnection object at 0x7fc4e6dbaeb0>: Failed to establish a new connection: [Errno 111] Connection refused’))

2023-06-07 03:46:29,786 - WARNING - 4084:Dummy-42 - zato.broker.client:0 - Invocation error; server → scheduler → HTTPConnectionPool(host=‘127.0.0.1’, port=31530): Max retries exceeded with url: / (Caused by NewConnectionError(‘<urllib3.connection.HTTPConnection object at 0x7fc4e6dbaeb0>: Failed to establish a new connection: [Errno 111] Connection refused’)) (1:‘100003’)

7

Just adding a little more detail.
I can confirm that nothing is listening on port 31530
and going back through the logs I can’t see it even attempting to start

8

Restarted quickstart instance (pull always) today and now the scheduler is working again ;-(