(Migrated) Adding FoundationDB support

(This message has been automatically imported from the retired mailing list)

I am looking to implement FoundationDB into Zato. It does support SQLAlchemy (https://foundationdb.com/layers/sql/documentation/GettingStarted/SQLAlchemy.html) but being a little new to the Zato source code, I am not quite sure where to start with implementing this.

My though was to use FoundationDB for the database and the kvdb to replace the need for Redis.

Any direction would be great, thanks!

Ryan J. Peterson

On 07/02/2014 08:51 PM, Ryan J. Peterson wrote:

I am looking to implement FoundationDB into Zato. It does support
SQLAlchemy (https://foundationdb.com/layers/sql/documentation/GettingStarted/SQLAlchemy.html)
but being a little new to the Zato source code, I am not quite sure where to start with implementing this.

My though was to use FoundationDB for the database and the kvdb to replace the need for Redis.

Hi Ryan,

thanks for suggesting it!

Adding a new DB which already has an SQLAlchemy wrapper will be the
easier of the two and can be certainly done before Zato 2.0 is cut.
Making Redis an optional part means more work and it’s a rather big
architectural change so it’s better to start with the former.

You’d need to work from the git version.

I’m not sure if you’ve already gone through the steps of installing from
source so here it is just in case:

  1. Make sure you are under Ubuntu, Debian or CentOS/RHEL
  2. Fork the repository on GH
  3. Git clone the repository
  4. Navigate to ./code
  5. Run ./install.sh
  6. Wait 10-30 minutes depending on your download link/CPU/whether you
    are under a VM

Now create a feature branch for your works. I have opened a ticket to
cover it https://github.com/zatosource/zato/issues/273

The convention for branch names is ‘username-X-gh###-description’, for
instance I am currently on ‘dsuch-f-gh272-show-user-ip-server-logs’ over
here, where

  • dsuch - my username on GH
  • f - it’s a feature
  • 272 - ticket number on GH
  • show-user-ip-server-logs - a short description from the ticket

In your situation it will be something like
‘rpeterson-f-273-add-foundationdb’.

Now the source code is built using buildout and consists of several
packages, each starting with zato-*

~/projects/zatosource-zato/code$ ls -1 | grep zato
zato-agent
zato-broker
zato-cli
zato-client
zato-common
zato_extra_paths
zato-server
zato-web-admin
~/projects/zatosource-zato/code$

What you are interested in are:

  • zato-cli - this is where the command line interface lives
  • zato-common - things reusable across the whole project
  • zato-server - servers
  • zato-web-admin - the web-admin interface

Basically, you’d need to find all the occurrences of strings say,
‘mysql’ and ‘postgres’, in these 4 places and consider:

  • zato-cli - whether any substitutions need to be performed on input
    provided by users. For instance, for convenience the CLI accepts a
    ‘mysql’ parameter but implementation-wise it gets changed to
    ‘mysql+pymysql’ because this is the SQLAlchemy driver to use.

  • zato-common - add FoundationDB to ping queries (i.e. SELECT 1+1) and
    the list of supported DBs

  • zato-server - does the FDB driver need any special process-wide
    initialization? Also, does it need name substitution or custom
    parameters passed to SQLAlchemy?

  • zato-web-admin - same as zato-server

Please remember that if you are running from source you don’t need to
run ./install.sh after each change in source code. Simply stop the
server and start it again and that will pick up the changes you’ve authored.

If you need any new libraries be added, the steps are:

  • Find their latest versions of PyPI
  • List each of them in that exact version in versions.cfg
  • Add each of them to buildout.cfg and buildout-darwin.cfg (I still keep
    faith we will have a working build on OS X one day)
  • Run ./bin/buldout

At times buildout may stop abruptly and you’ll find yourself with no
./bin/buildout around at all. In that situation do one of the following

  • ./install.sh (full build)
  • ./bin/python bootstrap.py -v 1.7.0 (takes 10 seconds)

Another good idea during development is to start components in foreground:

$ zato start ./web-admin --fg
$ zato start ./server1 --fg
$ zato start ./load-balancer (no --fg needed in this particular feature
you’d like to work on)

Please let post a message, either here or on GH, if you have any questions.

Thanks a lot :slight_smile:

On 07/02/2014 08:51 PM, Ryan J. Peterson wrote:

I am looking to implement FoundationDB into Zato. It does support
SQLAlchemy (https://foundationdb.com/layers/sql/documentation/GettingStarted/SQLAlchemy.html)
but being a little new to the Zato source code, I am not quite sure where to start with implementing this.

My though was to use FoundationDB for the database and the kvdb to replace the need for Redis.

Hi Ryan,

thanks for suggesting it!

Adding a new DB which already has an SQLAlchemy wrapper will be the
easier of the two and can be certainly done before Zato 2.0 is cut.
Making Redis an optional part means more work and it’s a rather big
architectural change so it’s better to start with the former.

You’d need to work from the git version.

I’m not sure if you’ve already gone through the steps of installing from
source so here it is just in case:

  1. Make sure you are under Ubuntu, Debian or CentOS/RHEL
  2. Fork the repository on GH
  3. Git clone the repository
  4. Navigate to ./code
  5. Run ./install.sh
  6. Wait 10-30 minutes depending on your download link/CPU/whether you
    are under a VM

Now create a feature branch for your works. I have opened a ticket to
cover it https://github.com/zatosource/zato/issues/273

The convention for branch names is ‘username-X-gh###-description’, for
instance I am currently on ‘dsuch-f-gh272-show-user-ip-server-logs’ over
here, where

  • dsuch - my username on GH
  • f - it’s a feature
  • 272 - ticket number on GH
  • show-user-ip-server-logs - a short description from the ticket

In your situation it will be something like
‘rpeterson-f-273-add-foundationdb’.

Now the source code is built using buildout and consists of several
packages, each starting with zato-*

~/projects/zatosource-zato/code$ ls -1 | grep zato
zato-agent
zato-broker
zato-cli
zato-client
zato-common
zato_extra_paths
zato-server
zato-web-admin
~/projects/zatosource-zato/code$

What you are interested in are:

  • zato-cli - this is where the command line interface lives
  • zato-common - things reusable across the whole project
  • zato-server - servers
  • zato-web-admin - the web-admin interface

Basically, you’d need to find all the occurrences of strings say,
‘mysql’ and ‘postgres’, in these 4 places and consider:

  • zato-cli - whether any substitutions need to be performed on input
    provided by users. For instance, for convenience the CLI accepts a
    ‘mysql’ parameter but implementation-wise it gets changed to
    ‘mysql+pymysql’ because this is the SQLAlchemy driver to use.

  • zato-common - add FoundationDB to ping queries (i.e. SELECT 1+1) and
    the list of supported DBs

  • zato-server - does the FDB driver need any special process-wide
    initialization? Also, does it need name substitution or custom
    parameters passed to SQLAlchemy?

  • zato-web-admin - same as zato-server

Please remember that if you are running from source you don’t need to
run ./install.sh after each change in source code. Simply stop the
server and start it again and that will pick up the changes you’ve authored.

If you need any new libraries be added, the steps are:

  • Find their latest versions of PyPI
  • List each of them in that exact version in versions.cfg
  • Add each of them to buildout.cfg and buildout-darwin.cfg (I still keep
    faith we will have a working build on OS X one day)
  • Run ./bin/buldout

At times buildout may stop abruptly and you’ll find yourself with no
./bin/buildout around at all. In that situation do one of the following

  • ./install.sh (full build)
  • ./bin/python bootstrap.py -v 1.7.0 (takes 10 seconds)

Another good idea during development is to start components in foreground:

$ zato start ./web-admin --fg
$ zato start ./server1 --fg
$ zato start ./load-balancer (no --fg needed in this particular feature
you’d like to work on)

Please let post a message, either here or on GH, if you have any questions.

Thanks a lot :slight_smile:

On 07/02/2014 09:47 PM, Dariusz Suchojad wrote:

Now create a feature branch for your works. I have opened a ticket to
cover it https://github.com/zatosource/zato/issues/273

Apologies - I forgot to add one thing.

Please make all the commits related to that ticket in the form of ‘GH
#273 …’

For instance ‘GH #273 - Added a dependency on FooBar’.

This lets one easily find all the commits for a ticket and the other way
around.

https://github.com/zatosource/zato/commits/master

thanks again,