(Migrated) Re: Adding FoundationDB support

(This message has been automatically imported from the retired mailing list)

Dariusz,

Thank you for all this info, good stuff. I think I was unclear about the =
Redis part, I don=E2=80=99t want to remove Redis, I just want to map its =
functionality to =46oundationDB as well.

Anyways, that can be for another time.

=E2=80=94=E2=80=94=E2=80=94

I am working on setting up a dev environment in Vagrant here. When I get =
to step =235 for install, I continue to get the following error:

/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distributi=
on option: ‘bugtrack=5Furl’
=C2=A0 warnings.warn(msg)
x86=5F64-linux-gnu-gcc: internal compiler error: Killed (program cc1)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-4.8/README.Bugs> for instructions.
error: Setup script exited with error: command ‘x86=5F64-linux-gnu-gcc’ f=
ailed with exit status 4
An error occurred when trying to install lxml 3.3.5. Look above this mess=
age for any errors that were output by easy=5Finstall.
While:
=C2=A0 Installing zato.
=C2=A0 Getting distribution for ‘lxml=3D=3D3.3.5’.
Error: Couldn’t install: lxml 3.3.5

I have tried patching this with:

apt-get -y apt-get install libxml2
apt-get -y apt-get install libxslt1.1
apt-get -y apt-get install libxml2-dev
apt-get -y apt-get install libxslt1-dev
apt-get -y apt-get install python-libxml2
apt-get -y apt-get install python-libxslt1
apt-get -y apt-get install python-dev
apt-get -y apt-get install python-setuptools
easy=5Finstall lxml

However, this doesn=E2=80=99t seem to consistently work. This is on Ubunt=
u 14.04 LTS x64 if that helps.

Not sure if anyone has run into this or not.

Thanks,

–=C2=A0
Ryan J. Peterson

On July 2, 2014 at 3:47:35 PM, Dariusz Suchojad (dsuch=40zato.io) wrote:
On 07/02/2014 08:51 PM, Ryan J. Peterson wrote:

I am looking to implement =46oundationDB into Zato. It does support
SQLAlchemy (https://foundationdb.com/layers/sql/documentation/GettingSt=
arted/SQLAlchemy.html)
but being a little new to the Zato source code, I am not quite sure whe=
re to start with implementing this.

My though was to use =46oundationDB for the database and the kvdb to re=
place the need for Redis.

Hi Ryan,

thanks for suggesting it=21

Adding a new DB which already has an SQLAlchemy wrapper will be the
easier of the two and can be certainly done before Zato 2.0 is cut.
Making Redis an optional part means more work and it’s a rather big
architectural change so it’s better to start with the former.

You’d need to work from the git version.

I’m not sure if you’ve already gone through the steps of installing from
source so here it is just in case:

  1. Make sure you are under Ubuntu, Debian or CentOS/RHEL
  2. =46ork the repository on GH
  3. Git clone the repository
  4. Navigate to ./code
  5. Run ./install.sh
  6. Wait 10-30 minutes depending on your download link/CPU/whether you
    are under a VM

Now create a feature branch for your works. I have opened a ticket to
cover it https://github.com/zatosource/zato/issues/273

The convention for branch names is ‘username-X-gh=23=23=23-description’, =
for
instance I am currently on ‘dsuch-f-gh272-show-user-ip-server-logs’ over
here, where

  • dsuch - my username on GH
  • f - it’s a feature
  • 272 - ticket number on GH
  • show-user-ip-server-logs - a short description from the ticket

In your situation it will be something like
‘rpeterson-f-273-add-foundationdb’.

Now the source code is built using buildout and consists of several
packages, each starting with zato-*

=7E/projects/zatosource-zato/code=24 ls -1 =7C grep zato
zato-agent
zato-broker
zato-cli
zato-client
zato-common
zato=5Fextra=5Fpaths
zato-server
zato-web-admin
=7E/projects/zatosource-zato/code=24

What you are interested in are:

  • zato-cli - this is where the command line interface lives
  • zato-common - things reusable across the whole project
  • zato-server - servers
  • zato-web-admin - the web-admin interface

Basically, you’d need to find all the occurrences of strings say,
‘mysql’ and ‘postgres’, in these 4 places and consider:

  • zato-cli - whether any substitutions need to be performed on input
    provided by users. =46or instance, for convenience the CLI accepts a
    ‘mysql’ parameter but implementation-wise it gets changed to
    ‘mysql+pymysql’ because this is the SQLAlchemy driver to use.

  • zato-common - add =46oundationDB to ping queries (i.e. SELECT 1+1) and
    the list of supported DBs

  • zato-server - does the =46DB driver need any special process-wide
    initialization=3F Also, does it need name substitution or custom
    parameters passed to SQLAlchemy=3F

  • zato-web-admin - same as zato-server

Please remember that if you are running from source you don’t need to
run ./install.sh after each change in source code. Simply stop the
server and start it again and that will pick up the changes you’ve author=
ed.

If you need any new libraries be added, the steps are:

  • =46ind their latest versions of PyPI
  • List each of them in that exact version in versions.cfg
  • Add each of them to buildout.cfg and buildout-darwin.cfg (I still keep
    faith we will have a working build on OS X one day)
  • Run ./bin/buldout

At times buildout may stop abruptly and you’ll find yourself with no
./bin/buildout around at all. In that situation do one of the following

  • ./install.sh (full build)
  • ./bin/python bootstrap.py -v 1.7.0 (takes 10 seconds)

Another good idea during development is to start components in foreground=
:

=24 zato start ./web-admin --fg
=24 zato start ./server1 --fg
=24 zato start ./load-balancer (no --fg needed in this particular feature=

you’d like to work on)

Please let post a message, either here or on GH, if you have any question=
s.

Thanks a lot :slight_smile:


Dariusz Suchojad

https://zato.io
ESB, SOA, REST, APIs and cloud integrations in Python

Okay, =C2=A0I will kill the VM and start from scratch to see if that fixe=
s things. Could be the Vagrant box I am using, will try with just raw ima=
ge of Ubuntu.

Thanks,
–=C2=A0
Ryan J. Peterson

On July 3, 2014 at 11:12:30 AM, Dariusz Suchojad (dsuch=40zato.io) wrote:=

On 07/03/2014 02:16 PM, Dariusz Suchojad wrote: =20

I’ll spin up an Ubuntu 14.04 LTS x64 VM and check it out too. =20

Everything went fine with no bumps, lxml including: =20

Getting distribution for ‘lxml=3D=3D3.3.5’. =20
Building lxml version 3.3.5. =20
Building without Cython. =20
Using build configuration of libxslt 1.1.28 =20
/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown =20
distribution option: ‘bugtrack=5Furl’ =20
warnings.warn(msg) =20
Got lxml 3.3.5. =20

=24 uname -a =20
Linux ubu140464 3.13.0-24-generic =2346-Ubuntu SMP Thu Apr 10 19:11:08 UT=
C =20
2014 x86=5F64 x86=5F64 x86=5F64 GNU/Linux =20
=24 =20

The only thing I did was ./install.sh from git master. =20

This is under VirtualBox with 1 CPU and 0.5 GB of RAM. =20

Upping the memory worked on my machine, something must have just been cra=
shing internally in the VM.

Thanks for checking on it,

–=C2=A0
Ryan J. Peterson
On July 3, 2014 at 11:14:14 AM, Ryan J. Peterson (rpeterson=40serenitysof=
t.com) wrote:

Okay, =C2=A0I will kill the VM and start from scratch to see if that fixe=
s things. Could be the Vagrant box I am using, will try with just raw ima=
ge of Ubuntu.

Thanks,
–=C2=A0
Ryan J. Peterson

On July 3, 2014 at 11:12:30 AM, Dariusz Suchojad (dsuch=40zato.io) wrote:=

On 07/03/2014 02:16 PM, Dariusz Suchojad wrote:

I’ll spin up an Ubuntu 14.04 LTS x64 VM and check it out too.

Everything went fine with no bumps, lxml including:

Getting distribution for ‘lxml=3D=3D3.3.5’.
Building lxml version 3.3.5.
Building without Cython.
Using build configuration of libxslt 1.1.28
/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown
distribution option: 'bugtrack=5Furl’
warnings.warn(msg)
Got lxml 3.3.5.

=24 uname -a
Linux ubu140464 3.13.0-24-generic =2346-Ubuntu SMP Thu Apr 10 19:11:08 UT=
C
2014 x86=5F64 x86=5F64 x86=5F64 GNU/Linux
=24

The only thing I did was ./install.sh from git master.

This is under VirtualBox with 1 CPU and 0.5 GB of RAM.


Dariusz Suchojad

https://zato.io
ESB, SOA, REST, APIs and cloud integrations in Python

On 07/03/2014 02:00 PM, Ryan J. Peterson wrote:

However, this doesn’t seem to consistently work. This is on Ubuntu 14.04 LTS x64 if that helps.

Not sure if anyone has run into this or not.

I’ve seen spurious build failures like this one that would go away the
next time it was tried and that’s one of the reasons I am not a great
fan of building from source if it can be avoided.

But I’m not sure if by not working consistently you mean it helps at
times but not always?

I’ll spin up an Ubuntu 14.04 LTS x64 VM and check it out too.

thanks,

On 07/03/2014 02:00 PM, Ryan J. Peterson wrote:

However, this doesn’t seem to consistently work. This is on Ubuntu 14.04 LTS x64 if that helps.

Not sure if anyone has run into this or not.

I’ve seen spurious build failures like this one that would go away the
next time it was tried and that’s one of the reasons I am not a great
fan of building from source if it can be avoided.

But I’m not sure if by not working consistently you mean it helps at
times but not always?

I’ll spin up an Ubuntu 14.04 LTS x64 VM and check it out too.

thanks,

On 07/03/2014 02:16 PM, Dariusz Suchojad wrote:

I’ll spin up an Ubuntu 14.04 LTS x64 VM and check it out too.

Everything went fine with no bumps, lxml including:

Getting distribution for ‘lxml==3.3.5’.
Building lxml version 3.3.5.
Building without Cython.
Using build configuration of libxslt 1.1.28
/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown
distribution option: 'bugtrack_url’
warnings.warn(msg)
Got lxml 3.3.5.

$ uname -a
Linux ubu140464 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC
2014 x86_64 x86_64 x86_64 GNU/Linux
$

The only thing I did was ./install.sh from git master.

This is under VirtualBox with 1 CPU and 0.5 GB of RAM.