Clean domain names for loadbalancer and webadmin

Hi all,

I recently discovered Zato and I am really intrigued. It looks really impressive…

Tried to install on a personal server I have, with the requirements of having clean urls for the load balancer and the webadmin,https://zato.mydomain.com and https://zatoadmin.mydomain.com, respectively. My understanding being that a frontent application will be using the https://zato.mydomain.com domain for all backend requests and the https://zatoadmin.mydomain.com to be used for administrative purposes (or it could be my-fancy-app.mydomain.com and admin.mydomain.com, you get the point)

My server is already running Docker Swarm and already has a Traefik instance that runs on it’s own stack and uses a traefik-public docker network.

This allows me to start other different stacks (Zulip Chat, for example) that employs multiple internal services, but still use the external Traefik as a general purpose reverse proxy for it, or any other docker swarm stacks. This strategy has already worked succesfully for many swarm stacks, so far, so I know this methodology works!

Now in regards to Zato, I tried to make some changes to the suggested docker/swarm/docker-compose.yml.template to accomplish what I need.

My final edited docker-compose.yml:

version: '3.5'

services:
  postgres:
    image: 'postgres:alpine'
    deploy:
      restart_policy:
        condition: any
        delay: 5s
        max_attempts: 3
        window: 120s
    environment:
      - POSTGRES_PASSWORD=MVIVnzw3gfRKWuxBJkWbzpvxSEHAUiNOwtvcQHnQrgA=
      - POSTGRES_USER=zato
  redis:
    image: redis
    deploy:
      restart_policy:
        condition: any
        delay: 5s
        max_attempts: 3
        window: 120s
      resources:
        limits:
          memory: 300M
        reservations:
          memory: 20M
  zato_scheduler:
    image: registry.gitlab.com/zatosource/docker-registry/cloud:3.1
    deploy:
      restart_policy:
        condition: any
        delay: 5s
        max_attempts: 3
        window: 120s
    environment:
      - VERBOSE=y
      - LB_HOSTNAME=zato.mydomain.com
      - LB_PORT=443
      - LB_AGENT_PORT=20151
      - CLUSTER_NAME=zato
      - SECRET_KEY=7EY6Bh3kVUVotJFJKpG9sAw0h0tuhNFiDHazPUvWF88=
      - JWT_SECRET_KEY=9zrtuhHs9jxfQsmmhZO5vhJqhb+NXFU5Vrg2qFZoO70=
      - ZATO_POSITION=scheduler
      - ZATO_WEB_ADMIN_PASSWORD=ppaML9HqBAQjDbEEbGWkwmXJBEYeu71hGQ12gSpHIVQ=
      - ZATO_ADMIN_INVOKE_PASSWORD=OaWYrjbNrqLjfOn9jS3kkdMXWKz10e+wiIfvFqoDjBI=
      - ZATO_IDE_PUBLISHER_PASSWORD=C6GtZ2a2XIiw0A6eqUn78VnGiC5MJl8Z3DHDL5dMuf4=
      - REDIS_HOSTNAME=redis
      - ODB_TYPE=postgresql
      - ODB_HOSTNAME=postgres
      - ODB_PORT=5432
      - ODB_NAME=zato
      - ODB_USERNAME=zato
      - ODB_PASSWORD=MVIVnzw3gfRKWuxBJkWbzpvxSEHAUiNOwtvcQHnQrgA=
  zato_bootstrap:
    image: registry.gitlab.com/zatosource/docker-registry/cloud:3.1
    environment:
      - VERBOSE=y
      - LB_HOSTNAME=zato.mydomain.com
      - LB_PORT=443
      - LB_AGENT_PORT=20151
      - CLUSTER_NAME=zato
      - SECRET_KEY=7EY6Bh3kVUVotJFJKpG9sAw0h0tuhNFiDHazPUvWF88=
      - JWT_SECRET_KEY=9zrtuhHs9jxfQsmmhZO5vhJqhb+NXFU5Vrg2qFZoO70=
      - ZATO_POSITION=load-balancer
      - ZATO_WEB_ADMIN_PASSWORD=ppaML9HqBAQjDbEEbGWkwmXJBEYeu71hGQ12gSpHIVQ=
      - ZATO_ADMIN_INVOKE_PASSWORD=OaWYrjbNrqLjfOn9jS3kkdMXWKz10e+wiIfvFqoDjBI=
      - ZATO_IDE_PUBLISHER_PASSWORD=C6GtZ2a2XIiw0A6eqUn78VnGiC5MJl8Z3DHDL5dMuf4=
      - REDIS_HOSTNAME=redis
      - ODB_TYPE=postgresql
      - ODB_HOSTNAME=postgres
      - ODB_PORT=5432
      - ODB_NAME=zato
      - ODB_USERNAME=zato
      - ODB_PASSWORD=MVIVnzw3gfRKWuxBJkWbzpvxSEHAUiNOwtvcQHnQrgA=
  zato_server:
    image: registry.gitlab.com/zatosource/docker-registry/cloud:3.1
    
    deploy:
      replicas: 2
      update_config:
        parallelism: 1
        delay: 10s
      restart_policy:
        condition: any
        delay: 5s
        max_attempts: 3
        window: 120s
      labels:
        - traefik.enable=true
        - traefik.http.routers.zato_server.entrypoints=websecure
        - traefik.http.routers.zato_server.rule=Host(`zato.mydomain.com`)
        - traefik.http.routers.zato_server.tls.certresolver=letsencryptresolver
        - traefik.http.services.zato_server.loadbalancer.server.port=17010
        - traefik.http.services.zato_server.loadbalancer.healthcheck.path=/zato/ping
        - traefik.http.middlewares.testHeader.headers.customrequestheaders.X-Auth-Token=true
        - traefik.http.middlewares.testHeader.headers.customresponseheaders.X-Auth-Token=true
    environment:
      - VERBOSE=y
      - LB_HOSTNAME=zato.mydomain.com
      - LB_PORT=443
      - LB_AGENT_PORT=20151
      
      - CLUSTER_NAME=zato
      - SECRET_KEY=7EY6Bh3kVUVotJFJKpG9sAw0h0tuhNFiDHazPUvWF88=
      - JWT_SECRET_KEY=9zrtuhHs9jxfQsmmhZO5vhJqhb+NXFU5Vrg2qFZoO70=
      - ZATO_POSITION=server
      - SERVER_NAME=server1
      - ZATO_WEB_ADMIN_PASSWORD=ppaML9HqBAQjDbEEbGWkwmXJBEYeu71hGQ12gSpHIVQ=
      - ZATO_ADMIN_INVOKE_PASSWORD=OaWYrjbNrqLjfOn9jS3kkdMXWKz10e+wiIfvFqoDjBI=
      - ZATO_IDE_PUBLISHER_PASSWORD=C6GtZ2a2XIiw0A6eqUn78VnGiC5MJl8Z3DHDL5dMuf4=
      - REDIS_HOSTNAME=redis
      - ODB_TYPE=postgresql
      - ODB_HOSTNAME=postgres
      - ODB_PORT=5432
      - ODB_NAME=zato
      - ODB_USERNAME=zato
      - ODB_PASSWORD=MVIVnzw3gfRKWuxBJkWbzpvxSEHAUiNOwtvcQHnQrgA=
    ports:
      - 17010
    networks:
      - default
      - traefik-public
  zato_webadmin:
    deploy:
      labels:
        - traefik.enable=true
        - traefik.http.routers.zato_webadmin.entrypoints=websecure
        - traefik.http.routers.zato_webadmin.rule=Host(`zatoadmin.mydomain.com`)
        - traefik.http.routers.zato_webadmin.tls.certresolver=letsencryptresolver
        - traefik.http.services.zato_webadmin.loadbalancer.server.port=8183
    image: registry.gitlab.com/zatosource/docker-registry/cloud:3.1
    deploy:
      restart_policy:
        condition: any
        delay: 5s
        max_attempts: 3
        window: 120s
    environment:
      - VERBOSE=y
      - LB_HOSTNAME=zato.mydomain.com
      - LB_PORT=443
      - LB_AGENT_PORT=20151
      - CLUSTER_NAME=zato
      - SECRET_KEY=7EY6Bh3kVUVotJFJKpG9sAw0h0tuhNFiDHazPUvWF88=
      - JWT_SECRET_KEY=9zrtuhHs9jxfQsmmhZO5vhJqhb+NXFU5Vrg2qFZoO70=
      - ZATO_POSITION=webadmin
      - ZATO_WEB_ADMIN_PASSWORD=ppaML9HqBAQjDbEEbGWkwmXJBEYeu71hGQ12gSpHIVQ=
      - ZATO_ADMIN_INVOKE_PASSWORD=OaWYrjbNrqLjfOn9jS3kkdMXWKz10e+wiIfvFqoDjBI=
      - ZATO_IDE_PUBLISHER_PASSWORD=C6GtZ2a2XIiw0A6eqUn78VnGiC5MJl8Z3DHDL5dMuf4=
      - REDIS_HOSTNAME=redis
      - ODB_TYPE=postgresql
      - ODB_HOSTNAME=postgres
      - ODB_PORT=5432
      - ODB_NAME=zato
      - ODB_USERNAME=zato
      - ODB_PASSWORD=MVIVnzw3gfRKWuxBJkWbzpvxSEHAUiNOwtvcQHnQrgA=
    ports:
      - 8183:8183
    networks:
      - default
      - traefik-public
networks:
  default:
  traefik-public:
    external: true
    name: traefik-public

My environment file (although not that important) is:

ZATO_WEB_ADMIN_PASSWORD=ppaML9HqBAQjDbEEbGWkwmXJBEYeu71hGQ12gSpHIVQ=
IDE_PUBLISHER_PASSWORD=C6GtZ2a2XIiw0A6eqUn78VnGiC5MJl8Z3DHDL5dMuf4=
ZATO_ADMIN_INVOKE_PASSWORD=OaWYrjbNrqLjfOn9jS3kkdMXWKz10e+wiIfvFqoDjBI=
CLUSTER_NAME=zato
SECRET_KEY=7EY6Bh3kVUVotJFJKpG9sAw0h0tuhNFiDHazPUvWF88=
JWT_SECRET_KEY=9zrtuhHs9jxfQsmmhZO5vhJqhb+NXFU5Vrg2qFZoO70=
ODB_PASSWORD=MVIVnzw3gfRKWuxBJkWbzpvxSEHAUiNOwtvcQHnQrgA=
VERBOSE=y
TRAEFIK_NETWORK=traefik-public
LB_HOSTNAME=zato.mydomain.com
LB_PORT=443
WEBADMIN_HOSTNAME=zatoadmin.mydomain.com

Obviously I have edited the docker-compose.yml.template according to my needs (it generates the docker-compose.yml above)

My external traefik stack is:

version: "3.5"
services:
  proxy:
    image: traefik:v2.2
    command:
      - --api.debug=true
      - --api.dashboard=true
      - --providers.docker=true
      - --providers.docker.swarmMode=true
      - --providers.docker.exposedByDefault=false
      - --accessLog=true
      - --log.level=DEBUG
      - --providers.docker.network=traefik-public
      - --entrypoints.web.address=:80
      - --entrypoints.websecure.address=:443
      - --certificatesresolvers.letsencryptresolver.acme.httpchallenge=true
      - --certificatesresolvers.letsencryptresolver.acme.httpchallenge.entrypoint=web
      - --certificatesresolvers.letsencryptresolver.acme.email=myemail@gmail.com
      - --certificatesresolvers.letsencryptresolver.acme.storage=/letsencrypt/acme.json
    ports:
      - 80:80
      - 443:443
      - 8080:8080
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /data/disk2/docker_data/traefik/letsencrypt:/letsencrypt
    networks:
      - traefik-public
    logging:
      driver: json-file
    deploy:
      labels:
        traefik.http.routers.proxy.middlewares: test-auth
        traefik.http.routers.proxy.tls.certresolver: letsencryptresolver
        traefik.http.services.proxy.loadbalancer.server.port: "80"
        traefik.http.routers.http-catchall.middlewares: redirect-to-https@docker
        traefik.http.routers.http-catchall.rule: hostregexp(`{host:.+}`)
        traefik.http.middlewares.redirect-to-https.redirectscheme.scheme: https
        traefik.http.routers.proxy.entrypoints: websecure
        traefik.http.routers.http-catchall.entrypoints: web
        traefik.http.middlewares.test-auth.basicauth.users: myname:$$2y$$05$$Y7q6zrYorKySU34dfsH6hOAPVLqX6v.0KjBYBaBHTjrR1/BU9lb2m
        traefik.http.routers.proxy.service: api@internal
        traefik.enable: "true"
        traefik.http.routers.proxy.rule: Host(`traefik.mydomain.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))
      placement:
        constraints:
          - node.role == manager
networks:
  traefik-public:
    external: true
    # driver: overlay
    # attachable: true
    name: traefik-public

The installation “seems” to be working but I have a weird issue though.

I can access webadmin at https://zatoadmin.mydomain.com, I get a login window and I can login fine.

I get the front page at https://zatoadmin.mydomain.com/zato/. But when I visit
Services/List Services (or any other menu), the page refreshes, I see something populated but very quickly the page refreshes and I get a page with contents:

Traceback (most recent call last):
  File "/opt/zato/3.1.0/code/zato-client/src/zato/client/__init__.py", line 250, in init
    json = loads(self.inner.text)
  File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.6/json/decoder.py", line 342, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 1 column 5 (char 4)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/zato/3.1.0/code/zato-web-admin/src/zato/admin/web/views/__init__.py", line 397, in __call__
    response = self.invoke_admin_service()
  File "/opt/zato/3.1.0/code/zato-web-admin/src/zato/admin/web/views/__init__.py", line 347, in invoke_admin_service
    return func(service_name, request)
  File "/opt/zato/3.1.0/code/zato-web-admin/src/zato/admin/middleware.py", line 79, in invoke
    response = super(Client, self).invoke(*args, headers={'X-Zato-Forwarded-For': self.forwarded_for}, **kwargs)
  File "/opt/zato/3.1.0/code/zato-client/src/zato/client/__init__.py", line 492, in invoke
    return self._invoke(is_async=False, *args, **kwargs)
  File "/opt/zato/3.1.0/code/zato-client/src/zato/client/__init__.py", line 489, in _invoke
    ServiceInvokeResponse, is_async, headers, output_repeated)
  File "/opt/zato/3.1.0/code/zato-client/src/zato/client/__init__.py", line 421, in invoke
    return self.inner_invoke(request, response_class, is_async, headers)
  File "/opt/zato/3.1.0/code/zato-client/src/zato/client/__init__.py", line 406, in inner_invoke
    self.max_cid_repr, self.logger, output_repeated)
  File "/opt/zato/3.1.0/code/zato-client/src/zato/client/__init__.py", line 316, in __init__
    super(ServiceInvokeResponse, self).__init__(*args, **kwargs)
  File "/opt/zato/3.1.0/code/zato-client/src/zato/client/__init__.py", line 155, in __init__
    self.init()
  File "/opt/zato/3.1.0/code/zato-client/src/zato/client/__init__.py", line 254, in init
    raise ValueError(msg)
ValueError: inner.status_code `404`, JSON parsing error `404 page not found
`

I can get the https://zato.mydomain.com/zato/ping to reply with:

{"zato_ping_response": {"pong": "zato"}, "zato_env": {"result":
"ZATO_OK", "cid": "dc0ca225e90ce1638c269d0c", "details": ""}}

so I consider that the Load Balancer is correctly accessible.

I feel there is an implied hosting of the webadmin on a /zato subfolder that might be causing some issues but I have not found anything in the docs and I do not know how to further deal with it.

Does any of these help you see what the problem might be…?

I think some clear instructions on how to achieve some clean installation with well defined domain urls is really missing from the docs (which are excellent by the way), so maybe we can work something out?

Or am I going the long/wrong way here?

Any help is appreciated.

Thanks!

Hi @stratosgear,

yes, you are right that web-admin expects for URL paths to begin with /zato regardless of what domain it is under- this is just how it works.

In the traceback that you posted there is this line - #250 in __init__.py - can you please try to add a print state right before to check what self.inner.text is? Clearly, it is not a JSON document but let’s see what it actually is, this will give us more details as to what is going on.

Thanks.

Traceback (most recent call last):
  File "/opt/zato/3.1.0/code/zato-client/src/zato/client/__init__.py", line 250, in init
    json = loads(self.inner.text)