Editing Zato log format to have JSON

Hi everyone.

Zato have an inside logger (self.logger) and most of the time the format printing is the following:

2022-04-22 14:46:55,433 - INFO - 3075:Dummy-165 - api.user.get-details:0 - BOOOOOD

I would like to know if it is possible to change the log format in json. For example, if I do,

self.logger.info(request="my_request", something="string for something")

I would like to have something near:

{ "timestamp": 2022-04-22 14:46:55,433, "level": "INFO", "service": "api.user.get-details:0", "request": "my_request", "something": "string for something"  }

Hello @louistwiice,

Zato uses loggers based on Python’s own logging library so you can use any features that Python offers.

In this case, I think what you need is a custom formatter that emits JSON instead of regular text. Either your own or perhaps there is already one on PyPI.

The config file to edit is logging.conf, this is where you specify the details of how to configure logging.

Regards.

Hi,

Could you please, show me the location of logging.conf in Zato (PS: I am using zato on Docker)

Thank you

The easiest way to find files under Linux is to use the “locate” command, e.g. “locate logging.conf”.

Assuming that you have Docker Quickstart, the one that you need right now, for servers, is “/opt/zato/env/qs-1/server1/config/repo/logging.conf”.

1 Like

Thank you very much. I will try to edit logging.conf file a see how it is going. After editing, should we restart the container or it will take effect immediately?

This is something that hot-deployment does not cover yet and you need to restart the server yourself.

However, do not restart the whole container as that will spawn a clean one afterwards and all your modifications will be lost.

Instead, I suggest the following workflow:

  • Work in your own IDE on how to configure Python logging to emit JSON.
  • Once you know how to do it in isolation, enter the container, apply your edits to logging.conf and use “zato stop” and “zato start” to restart the server inside it.
  • When you have it all confirmed, you need to ensure that newly started containers will include your work. I think you use Docker Compose in your work so you can map a logging.conf file from host to the container using Compose and then the server inside the Docker will start using your configuration.

After you are done with everything, please post your solution here as this is something that other people may be interested in too.

Thank you.

Okay. I will try this. I let you know later.

Thank you again :+1:

Some more notes:

  • When I work on this kind of things, I always run the server in foreground, i.e. “zato start /path/to/server --fg” which means that I can stop it quickly with Ctrl-C.
  • Depending on a particular need, I often start such a server directly from VS Code, under debugger, which lets me easily observe what it is doing inside.

It is strange! I can not do zato start or zato stop inside the docker container. I have the following bash error

root@565cbe306348:/opt# zato stop
bash: zato: command not found

I don’t if I should be placed on a specific location

You need to be user zato, not root.

Hi @dsuch

I have found a package that could help to do that :python-json-logger . Here are steps to edit zato logs format (PS: I have tried this only for Zato Docker’s image)

1. Run the container

$ docker run --pull=always -it --rm -p 22022:22 -p 8183:8183 -p 11223:11223 -p 17010:17010 \
  --name zato-3.2-quickstart \
  ghcr.io/zatosource/zato-3.2-quickstart

2. Install python-json-logger

root@1232efab23# /opt/zato/current/bin/pip install python-json-logger

After that you need to edit logging.conf file.

root@1232efab23# cd /opt/zato/env/qs-1/server1/config/repo/
root@1232efab23:/opt/zato/env/qs-1/server1/config/repo/# vi logging.conf

Go to formatters section and edit the section like below:

loggers:
    .
    .
    .
formatters:
    audit_pii:
        format: '%(message)s'
        class: pythonjsonlogger.jsonlogger.JsonFormatter # <----- Line to add
    default:
        format: '%(asctime)s %(levelname)s %(process)d %(threadName)s %(name)s %(lineno)d %
(message)s' # <------- remove the '-'
        class: pythonjsonlogger.jsonlogger.JsonFormatter # <----- Line to add
    http_access_log:
        format: '%(remote_ip)s %(cid_resp_time)s "%(channel_name)s" [%(req_timestamp)s] "%(method)s %(path)s %(http_version)s" %(status_code)s %(response_size)s "-" "%(user_agent)s"'
        class: pythonjsonlogger.jsonlogger.JsonFormatter # <----- Line to add
    colour:
        format: '%(asctime)s - %(levelname)s - %(process)d:%(threadName)s - %(name)s:%(lineno)d - %(message)s'
        (): zato.common.util.api.ColorFormatter
        class: pythonjsonlogger.jsonlogger.JsonFormatter # <----- Line to add

3. Restart the server

root@1232efab23:/opt/zato/env/qs-1/server1/config/repo/# sudo su zato
zato@1232efab23:/opt/zato/env/qs-1/server1/config/repo/$ zato stop /opt/zato/env/qs-1/server1/
zato@1232efab23:/opt/zato/env/qs-1/server1/config/repo/$ zato start /opt/zato/env/qs-1/server1/

Logs will change and become

zato-3.2-quickstart  | {"asctime": "2022-04-25 10:08:52,021", "levelname": "\u001b[1;37mINFO\u001b[0m", "process": 3473, "threadName": "MainThread", "name": "zato.server.base.parallel", "lineno": 0, "message": "First worker of `server1` is 3473"}
zato-3.2-quickstart  | {"asctime": "2022-04-25 10:08:52,032", "levelname": "\u001b[1;37mINFO\u001b[0m", "process": 3473, "threadName": "MainThread", "name": "zato.server.connection.connector.subprocess_.ipc", "lineno": 0, "message": "Starting Zato events connector for server `server1` on port `34568`", "remote_ip": null}

The interested thing with this package is that you can custom you JSON log. For that I have created a package named wzLogFormatter

import re
import logging
from datetime import datetime
from pythonjsonlogger import jsonlogger


class CustomJsonFormatter(jsonlogger.JsonFormatter):

    def add_fields(self, log_record, record, message_dict):
        super(CustomJsonFormatter, self).add_fields(log_record, record, message_dict)
        log_record['event'] = log_record['message']  # We change the name so that it could match structlog format
        log_record.pop('message')

        if not log_record.get('asctime') or not not log_record.get('timestamp'):
            # this doesn't use record.created, so it is slightly off
            now = datetime.utcnow().strftime('%Y-%m-%dT%H:%M:%S.%fZ')
            log_record['timestamp'] = now
        else:
            # We replace asctime my timestamp so that it could match structlog format.
            log_record['timestamp'] = log_record['asctime']
            log_record.pop('asctime')

        reaesc = re.compile(r'\x1b[^m]*m')  # Zato logs comes with color formatter for system. We have to remove it
        if log_record.get('level'):
            level_without_color = reaesc.sub('', log_record['level'])
            log_record['level'] = level_without_color
        else:
            level_without_color = reaesc.sub('', record.levelname)
            log_record['level'] = level_without_color

    @property
    def getLogHandler(self, format=None):
        logHandler = logging.StreamHandler()

        if format:
            formatter = CustomJsonFormatter(format)
        else:
            formatter = CustomJsonFormatter('%(asctime)s %(levelname)s %(process)d %(threadName)s %(name)s %(lineno)d %(message)s')

        logHandler.setFormatter(formatter)

        return logHandler

After that, create the package. Install it. Replace in logging.conf class = pythonjsonlogger.jsonlogger.JsonFormatter # <----- Line to add by wzLogFormatter.<filename>.CustomJsonFormatter and it should work fine

Problem

  1. Removing ansi_code around log level

The trouble I am facing is that I haven’t yet found a way to write log level in a clean way in the JSON. It is always something like "levelname": "\u001b[1;37mINFO\u001b[0m" . I have tried to do some regex but in my custom package but it doesn’t work

  1. Loading logging.conf At the begining of Docker image start
    The thing I want is my logging.conf file taking effect at the start of the container. I have created a docker-compose file and i have tried to pass my logging.conf. But I received and error
version: "3"

services:
  zato:
    image: ghcr.io/zatosource/zato-3.2-quickstart:latest
    container_name: zato${SERVER}
    ports:
      - "8183:8183"
      - "8083:11223"
    environment:
      Zato_Dashboard_Password: "*********"
    volumes:
      - ./libs/requirements.txt:/opt/hot-deploy/python-reqs/requirements.txt
      - ./zato/:/opt/hot-deploy/services
      - ./libs:/opt/libs
      - ./libs/logging.conf:/opt/zato/env/qs-1/server1/config/repo/logging.conf:ro

When doing docker-compose up I have the following error during starting

zato  | TASK [Start HashiCorp Vault (dev) (02)] ****************************************
zato  | skipping: [localhost]
zato  | 
zato  | TASK [Make sure any previous environment is deleted (02)] **********************
zato  | fatal: [localhost]: FAILED! => {"changed": true, "cmd": "rm -rf /opt/zato/env/qs-1\n", "delta": "0:00:00.004168", "end": "2022-04-25 11:30:49.172662", "msg": "non-zero return code", "rc": 1, "start": "2022-04-25 11:30:49.168494", "stderr": "rm: cannot remove '/opt/zato/env/qs-1/server1/config/repo/logging.conf': Device or resource busy", "stderr_lines": ["rm: cannot remove '/opt/zato/env/qs-1/server1/config/repo/logging.conf': Device or resource busy"], "stdout": "", "stdout_lines": []}
zato  | 
zato  | PLAY RECAP *********************************************************************
zato  | localhost                  : ok=34   changed=23   unreachable=0    failed=1    skipped=19   rescued=0    ignored=0   
zato  | 
zato exited with code 2

So that is where I am. I don’t know if someone could help debug this.
Thank you very much, once again, for your help

Many thanks for the detailed write-up, @louistwiice - I am sure this will be useful to everyone interested in the subject.

I am not familiar with Docker Compose myself and I cannot help you with that particular question - but I wonder if the same happens when you use Docker bind mounts, as shown in the documentation?

As to the various characters, such as "\u001b[1;37mINFO\u001b[0m" - this looks to be added by a given logger’s formatter. In particular, the “stdout” formatter uses the “colour” formatter, thanks to which log levels such as INFO or WARN are shown in colors and in bold.

You may want to remove the “colour” formatter from “stdout” or you may perhaps want not to log to stdout at all. I am not sure what your preference is - the stdout logger is just for your convenience because the main log file is “server.log” anyway and this means that stdout can be disabled if you do not need it and if the “colour” formatter adds these unneeded characters.

Yes the same issue happens. I think it is because the file is already in use by the container. So when doing the bind mounts, it returns

"rm: cannot remove '/opt/zato/env/qs-1/server1/config/repo/logging.conf': Device or resource busy"

I think the docker-compose tries to delete and recreate the file. And it is impossible because, at this moment, the server is already up and is using the file. I am trying to find a moment to bind to file before zato server starts running.

I will let you know

About the characters, I have done what you have suggested. I have replace “colour” formatteur with default and now it works fine

Also I would like to know if it is possible to increase the number of server in Zato Docker (And How to do it).
If yes, I think i would have to do the same steps for all servers due to the logging.conf path location (/opt/zato/env/qs-1/serverX)

Hi

I have found a way to avoid using command zato stop and zato start. If you want logging.conf to take effect at beginning (since docker run command), here are the steps.

1. Next to logging.conf create a file zato-quickstart-02-edited.yaml

2. Create a Docker image

FROM ghcr.io/zatosource/zato-3.2-quickstart:latest
COPY <logging_path>/logging.conf /opt/tmp/logging.conf
COPY<path_to_edited_ansible_file>/zato-quickstart-02-edited.yaml /zato-ansible/zato-quickstart-02.yaml

2. Edit ansible file
It seems Docker zato use ansible to start and prepare environment. So, copy the content of the second ansible file zato ansible file number 2 and paste into into your file zato-quickstart-02-edited.yaml. I have added the following lines just after Enable Redis (02) task

- name:  Copy logging.conf file to server location
  become: true
  become_user: "zato"
  copy:
    src: <path_in_docker>/logging.conf
    dest: "{{ zato_env_path }}/server1/config/repo/logging.conf"

3. Create a docker-compose file

version: "3"

services:
  zato:
    build:
      context: .
    container_name: zato${SERVER}
    ports:
      - "8183:8183"
      - "11223:11223"
    environment:
      ZATO_WEB_ADMIN_PASSWORD: "*******"

4. Run docker compose

docker-compose up -d --build

Everything should work fine !!! :grinning: :grinning: