Storing and accessing security file from a service

Are there best practices on where to place a plain text file that I would like to access from a service? I have a service that makes an api call to Google Analytics Reporting API. The authentication process involves loading a json file that is downloaded from Google Analytics Reporting API. Are there best practices on where to store and how to retrieve such a file? The JSON file contains the following:

{
“type”: “”,
“project_id”: “”,
“private_key_id”: “”,
“private_key”: “”,
“client_email”: “”,
“client_id”: “”,
“auth_uri”: “”,
“token_uri”: “https://oauth2.googleapis.com/token”,
“auth_provider_x509_cert_url”: “https://www.googleapis.com/oauth2/v1/certs”,
“client_x509_cert_url”: “”
}

Hello,

is the file the same for each request? Is the flow, basically, load a file, fill out the details, send it to a remote endpoint? Is it possible for the structure of the file to change between requests?

That aside, the file looks like a JSON request - why does it have to be a file kept in the file system? Do you download it first and then make use of it in subsequent requests?

@dsuch thanks for the response.

The file is the same for each request and the flow is basically load the file from which credentials are generated from which a resource object that interacts with the google api is created. The structure of the file remains the same for all requests and has to be downloaded and saved on the server manually from which the service will load it to make the API requests.

If the data remains the same then perhaps instead of a JSON file, you could keep the information in an .ini file?

If you create a file such as google_credentials.ini and store it in /path/to/server/config/repo/user-config/google_credentials.ini then you will have access to it in runtime from your service through self.user_config.google_credentials.

That object will be dict-like - with standard methods such as .get, .keys, items etc. but it will also support attribute-based access, e.g. client_id = self.user_config.google_credentials.main.client_id.

Such configuration is read in when a server starts so it all is then served straight from RAM, without re-reading the file each time it is needed.

Naturally, under Docker, you can remap the user-conf directory and make it point to a mount point of your choice.

Let me know if this is what you need or if it actually needs to be a JSON file in which case I will suggest a different approach.

Thanks for the hint. I am however running into an issue starting the container
This is how I am starting the container:

docker run -it --env-file ./zato_env --rm -p 22 -p 6379:6379 -p 8183:8183 -p 17010:17010 -p 17011:17011 -p 11223:11223 -v ~/Development/zato/services/:/opt/hot-deploy/ -v ~/Development/zato/config/server1/:/opt/zato/env/qs-1/server1/config/repo/user-conf/ -v ~/Development/zato/config/server2/:/opt/zato/env/qs-1/server2/config/repo/user-conf/ --name zato --verbose registry.gitlab.com/zatosource/docker-registry/quickstart:3.1-py3

Below is the error I am getting;

2020-08-18 20:25:29.153 UTC [166] LOG: incomplete startup packet + /opt/zato/current/bin/zato quickstart create --odb_host localhost --odb_port 5432 --odb_user postgres --odb_db_name zato --odb_password f3e4b6e3-410f-4c56-8753-68ba6a723544 --kvdb_password '' /opt/zato/env/qs-1/ postgresql localhost 6379 Directory /opt/zato/env/qs-1 is not empty, please re-run the command in an empty directory 2020-08-18 20:25:35,020 - INFO - 169:MainThread - Create:624 - Directory /opt/zato/env/qs-1 is not empty, please re-run the command in an empty directory + /opt/zato/current/bin/zato from-config /opt/zato/update_password.config PermissionError: (13, 'Permission denied') (Hint: re-run with --verbose for full traceback) 2020-08-18 20:25:39,269 - ERROR - 188:MainThread - FromConfig:670 - PermissionError: (13, 'Permission denied') (Hint: re-run with --verbose for full traceback) + sed -i s/127.0.0.1:11223/0.0.0.0:11223/g /opt/zato/env/qs-1/load-balancer/config/repo/zato.config sed: can't read /opt/zato/env/qs-1/load-balancer/config/repo/zato.config: No such file or directory + sed -i s/gunicorn_workers=2/gunicorn_workers=1/g /opt/zato/env/qs-1/server1/config/repo/server.conf sed: can't read /opt/zato/env/qs-1/server1/config/repo/server.conf: No such file or directory + sed -i s/gunicorn_workers=2/gunicorn_workers=1/g /opt/zato/env/qs-1/server2/config/repo/server.conf sed: can't read /opt/zato/env/qs-1/server2/config/repo/server.conf: No such file or directory + set +x Stopping initialization of Postgresql 2020-08-18 20:25:39.501 UTC [144] LOG: received fast shutdown request waiting for server to shut down....2020-08-18 20:25:39.505 UTC [144] LOG: aborting any active transactions 2020-08-18 20:25:39.510 UTC [144] LOG: worker process: logical replication launcher (PID 151) exited with exit code 1 2020-08-18 20:25:39.513 UTC [146] LOG: shutting down 2020-08-18 20:25:39.536 UTC [144] LOG: database system is shut down done server stopped sed: can't read /opt/zato/env/qs-1/server1/config/repo/server.conf: No such file or directory

Please format your post correctly - it is impossible to read it.

It doesn’t seem to like me adding google_credentials.ini in

/opt/zato/env/qs-1/server1/config/repo/user-conf/google_credentials.ini

/opt/zato/env/qs-1/server2/config/repo/user-conf/google_credentials.ini


Running quickstart-bootstrap

  • /opt/zato/current/bin/zato quickstart create --odb_host localhost --odb_port 5432 --odb_user postgres --odb_db_name zato --odb_password 44d6aab7-dedb-46d8-9128-1524c1a11fb6 --kvdb_password ‘’ /opt/zato/env/qs-1/ postgresql localhost 6379

    Directory /opt/zato/env/qs-1 is not empty, please re-run the command in an empty directory

    2020-08-18 21:30:18,881 - INFO - 167:MainThread - Create:624 - Directory /opt/zato/env/qs-1 is not empty, please re-run the command in an empty directory
  • /opt/zato/current/bin/zato from-config /opt/zato/update_password.config

    PermissionError: (13, ‘Permission denied’) (Hint: re-run with --verbose for full traceback)

    2020-08-18 21:30:23,068 - ERROR - 186:MainThread - FromConfig:670 - PermissionError: (13, ‘Permission denied’) (Hint: re-run with --verbose for full traceback)
  • sed -i s/127.0.0.1:11223/0.0.0.0:11223/g /opt/zato/env/qs-1/load-balancer/config/repo/zato.config

    sed: can’t read /opt/zato/env/qs-1/load-balancer/config/repo/zato.config: No such file or directory
  • sed -i s/gunicorn_workers=2/gunicorn_workers=1/g /opt/zato/env/qs-1/server1/config/repo/server.conf

    sed: can’t read /opt/zato/env/qs-1/server1/config/repo/server.conf: No such file or directory
  • sed -i s/gunicorn_workers=2/gunicorn_workers=1/g /opt/zato/env/qs-1/server2/config/repo/server.conf

    sed: can’t read /opt/zato/env/qs-1/server2/config/repo/server.conf: No such file or directory
  • set +x

    Stopping initialization of Postgresql

    2020-08-18 21:30:23.313 UTC [142] LOG: received fast shutdown request

    waiting for server to shut down…2020-08-18 21:30:23.317 UTC [142] LOG: aborting any active transactions

    2020-08-18 21:30:23.324 UTC [142] LOG: worker process: logical replication launcher (PID 149) exited with exit code 1

    2020-08-18 21:30:23.327 UTC [144] LOG: shutting down

    2020-08-18 21:30:23.362 UTC [142] LOG: database system is shut down
    done

    server stopped

    sed: can’t read /opt/zato/env/qs-1/server1/config/repo/server.conf: No such file or directory

    Thanks

Hello @myke,

The Docker quickstart image look for the file /opt/zato/env/qs-1/zato-qs-restart.sh to check if the cluster is initialized or not, in you case it should not exists and it assumes that the cluster is not initialized and will run zato quickstart create to initialize it.

You are mounting server1/config/repo/user-conf and server2/config/repo/user-conf and the command zato quickstart create fails to run with this error:

2020-08-18 21:30:18,881 - INFO - 167:MainThread - Create:624 - Directory /opt/zato/env/qs-1 is not empty, please re-run the command in an empty directory

If you are trying to use other configuration files, you should mount /opt/zato/env/qs-1 instead of some of the subdirectories.

Currently the Docker quickstart image does not support including some custom files when you will initialize the cluster.

Please, can you try this?:

  1. Run the Docker quickstart image mounting an empty folder in /opt/zato/env/qs-1 with the parameter -v ~/Development/zato/config/:/opt/zato/env/qs-1/ (or any other empty directory).
  2. After the cluster is initialized you can stop the Docker container and copy the file google_credentials.ini to ~/Development/zato/config/server1/config/repo/user-conf/ and ~/Development/zato/config/server2/config/repo/user-conf/
  3. Start the cluster again with the same parameters of the 1st step.

Please, let me know if this worked for you. Thanks.

Thanks @anielkis for the hint. I was able to get the file into docker but the cluster never starts. However the clusters are unable to start. Without the mounted volume, it starts fine. See error below

2020-08-19 17:27:39,645 DEBG fd 27 closed, stopped monitoring <POutputDispatcher at 140510541003736 for (stdout)>
2020-08-19 17:27:39,645 DEBG fd 37 closed, stopped monitoring <POutputDispatcher at 140510541034560 for (stderr)>
2020-08-19 17:27:39,647 ERRO pool dependentstartup event buffer overflowed, discarding event 489
2020-08-19 17:27:39,647 INFO exited: zatoscheduler (exit status 16; not expected)
2020-08-19 17:27:39,651 DEBG received SIGCLD indicating a child quit
2020-08-19 17:27:39,849 DEBG 'zatowebadmin' stdout output: OperationalError: ('(psycopg2.OperationalError) could not connect to server: Connection refused\n\tIs the server running on host "localhost" (127.0.0.1) and accepting\n\tTCP/IP connections on port 5432?\ncould not connect to server: Cannot assign requested address\n\tIs the server running on host "localhost" (::1) and accepting\n\tTCP/IP connections on port 5432?\n',) (Hint: re-run with --verbose for full traceback)

2020-08-19 17:27:39,850 ERRO pool dependentstartup event buffer overflowed, discarding event 490

2020-08-19 17:27:39,879 INFO spawned: ‘zatoscheduler’ with pid 3571

2020-08-19 17:27:39,893 DEBG ‘zatowebadmin’ stderr output:

2020-08-19 17:27:39,848 - ERROR - 3500:MainThread - Start:670 - OperationalError: (’(psycopg2.OperationalError) could not connect to server: Connection refused\n\tIs the server running on host “localhost” (127.0.0.1) and accepting\n\tTCP/IP connections on port 5432?\ncould not connect to server: Cannot assign requested address\n\tIs the server running on host “localhost” (::1) and accepting\n\tTCP/IP connections on port 5432?\n’,) (Hint: re-run with --verbose for full traceback)

2020-08-19 17:27:40,524 DEBG ‘import’ stderr output:

2020/08/19 17:27:40 Waiting for tcp://localhost:17010: dial tcp 127.0.0.1:17010: connect: connection refused.

I am unable to access the admin page on localhost:8183

Thanks

Hello @myke,

It seems that PostgreSQL is not running

2020-08-19 17:27:39,849 DEBG 'zatowebadmin' stdout output: OperationalError: ('(psycopg2.OperationalError) could not connect to server: Connection refused\n\tIs the server running on host "localhost" (127.0.0.1) and accepting\n\tTCP/IP connections on port 5432?\ncould not connect to server: Cannot assign requested address\n\tIs the server running on host "localhost" (::1) and accepting\n\tTCP/IP connections on port 5432?\n',) (Hint: re-run with --verbose for full traceback)

The PostgreSQL server included in the Docker image starts anew every time and it depends of the ODB initialization process for the Zato cluster configuration. You have two options when you are reusing configuration files:

  1. Use an external PostgreSQL.
  2. Mount a folder into /var/lib/postgresql/data to keep the ODB information after the first initialization

Please, can you try one of these options? I hope you find it useful.

@anielkis thanks this helped greatly. I realized it also didn’t like the format of my INI file which is a JSON file that I need to load into my service. It isn’t clear where I can place this file so I can access it in the service.

Hello @myke,

I understand that you have PostgreSQL functioning properly now?

If so, then I am not sure which parts of the hints about the google_credentials.ini file did not work for you?

I.e. I am not clear what is not clear?

Hello @dsuch thanks for helping me out figure this. I essentially kept the contents of google_credentials.ini as JSON (see first message in this thread) which works when working out of Zato. However in Zato it fails. I have added to the following line to server.conf files for both server1 and server2 as shown below

[user_config]
user=./user.conf
data2=/opt/data/google_credentials.ini

Then in the service I call it this way:

self.logger.info('INI string %r ', self.user_config.data2)

When I call the service I get the following error logged

2020-08-20 18:03:46,850 DEBG ‘zatoserver1’ stderr output:

2020-08-20 18:03:46,850 - WARNING - 580:MainThread - zato.common.util.proc:113 - Stderr received from program /opt/zato/3.1.0/code/bin/py -m zato.server.main /opt/zato/env/qs-1/server1 fg=TrueZATO_ZATO_ZATOsync_internal=FalseZATO_ZATO_ZATOsecret_key=ZATO_ZATO_ZATOstderr_path=None 2> /tmp/tmp5i309kh5-zato-start-server.txt e:Traceback (most recent call last):</br> File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec)</br> File "/usr/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals)</br> File "/opt/zato/3.1.0/code/zato-server/src/zato/server/main.py", line 384, in <module></br> run(server_base_dir, options=parse_cmd_line_options(cmd_line_options))</br> File "/opt/zato/3.1.0/code/zato-server/src/zato/server/main.py", line 177, in run</br> server_config = get_config(repo_location, 'server.conf', crypto_manager=crypto_manager, secrets_conf=secrets_config)</br> File "/opt/zato/3.1.0/code/zato-common/src/zato/common/util/__init__.py", line 425, in get_config user_conf = ConfigObj(path)</br> File "/opt/zato/3.1.0/code/lib/python3.6/site-packages/configobj.py", line 1236, in __init__ self._load(infile, configspec)</br> File "/opt/zato/3.1.0/code/lib/python3.6/site-packages/configobj.py", line 1325, in _load raise error</br> configobj.ConfigObjError: Parsing failed with several errors.</br> First error at line 1. , kw:{'async': False}

Thanks

It will not work in this way - rather, the file truly needs to be an .ini one, e.g.:

[data]
private_key = "my_private_key.pem"
auth_uri = "https://auth.example.com"

Simply, convert all the JSON keys to .ini ones wrapped under a single section such as [data].

In the default installation, you will find a file called user.conf whose contents you can inspect for details and which you can first make use of in your service to get an understanding of how it works, e.g.

list_key = self.user_config.user.sample_section.list_key
self.logger.info('Result: %s', list_key)

Your use-case is an interesting one but we currently simply do not support JSON files on input in this way. Extending user config for the next Zato release to include JSON would not be difficult - please open a ticket on GitHub and this can be arranged.

Thanks.

By the way, if keeping the file as JSON is a necessity then it all can be done in another way, using a server startup service that loads it on when a server starts to store the contents in a server cache.

That involves a few lines of code while I just wanted to show you first something that is only a matter of configuration. But yes, it could be still a JSON file rather than .ini.

Thanks @dsuch for the help…I have created a service which I have added to zato_extra_paths and then I use that in the services. I am using a key in that I have added to user_config stanza to get the file reference which I use in the service added to the zato_extra_paths.

Thanks for the help