Adding MinIO support

Greetings everyone,

After being away for a long time, I finally have some Zato related work to do.
Firstly I gotta say thanks for Dsuch and others for their hard work with the Zato development!

Back to the topic

Lately, I have been looking for a more scalable / manageable solution for replacing FTP/SFTP servers used for storing various binary files. At some point, I found out about Minio [0], that by my understanding is quite compatible with AWS S3 services. As I remembered that Zato has support for AWS S3 connections, so I decided to try it out.

While the Minio connection was not supported directly, I was able to modify the code from s3.py module [1]
and make it successfully connect with locally hosted Minio instance and with the playground instance [2].

The edited s3.py module can be found from the GitHub gist:


[Edit]
Most of the changes are in add_client method of the S3Wrapper class (Lines 64-103)

About the changes related to s3.py module:

aws_minio flag
I wanted to keep the original behavior, so I created an aws_minio flag within the server configuration [3].
If this flag is not defined, the behavior should be the same as before, as the flag’s existence is checked.

[misc]

aws_minio = True

calling_format parameter
I do not completely understand this part, but based on GitHub issue comment [4], usage of boto 2
with Minio requires setting the calling_format argument as OrdinaryCallingFormat(). Without doing this, reading or writing objects and creating buckets is not working. As far I understand this is probably related to API’s paths.

Web-admin and the address field
When the aws_minio flag is enabled, the address field from the create/edit AWS S3 [5] is used for creating connection arguments (host, port and is_secured) from the URL (schema://host:port).
While I think that this is the cleanest way of doing this at the moment but it feels a bit hacky.

Final words

I was thinking about creating PR, but I’m not sure if these changes follow the official plans for the Zato.
And while I think that original implementation should work as before, I do not have access to real AWS S3 services for verifying this.

Any thoughts about these changes ?

Some refs:

[0] https://min.io/
[1] /opt/zato/current/zato-server/src/zato/server/connection/cloud/aws/s3.py
[2] https://play.minio.io:9000/minio/
[3] /example_path/server1/config/repo/server.conf
[4] https://github.com/minio/minio/issues/5422#issuecomment-358615603
[5] https://zato.io/docs/web-admin/cloud/aws/s3.html

Hi @liuz,

thanks for all the ideas and code - it surely makes sense to add support for this system.

I think it will be best to add it under its own outgoing connection type because, from what I can see, S3 is just an API for it and it is very well possible that in time MinIO may use other APIs too even if for now, to ease in migrations from other platforms, it uses S3.

In Zato, that new outgoing connection type will also use S3 but if we make it a separate entity, it will be easier to manage it in the longer term.

Can you just open a ticket in GitHub for it? I will reference your work and code accordingly.

Thanks again.

Hi @dsuch,

I agree with you, this should be separated to its own connection type.

While I believe that currently Minio is developed in a way that the AWS S3 API is supported, you never know what will happen in the future.

This would also allow using Minio package instead of using Boto 2/3 and possibly add some features that are not supported via Boto packages.

I will create a new ticket into the Github related to adding Minio support.

Thanks for the response