(Migrated) Redis and Postgres depencencies

(This message has been automatically imported from the retired mailing list)

Hello Dariusz,

In a previous message you pointed at two GH issues (#206 and #207) which
have the objective of making Redis and Postgres optional dependencies.

Indeed, in an environment I am currently designing and developing I
would have used redis anyway but postgres is a big deal, especially
because I have high availability needs and this forces me to configure
and maintain a Postgres cluster only to make Zato run.

So, I have a couple of questions about this point:
Do you have any time estimation about issue 207?
And, can I take part of the work or help in some other way to speed it up?

And I have a suggestion/“feature request” which involves both issues:

In my case, the main database is MongoDB.

What about using it or any similar NoSQL DB as an (optional) alternative
to both Postgres and Redis?

Because MongoDB is neither a relational nor a key/value database but in
most situations it can replace both of them. So, I don’t know how
complex this would be inside Zato code but maybe it wouldn’t be too hard
to make replace the ODB and the KVDB all at once.

In any case, even if it’s too complex to make it replace a SQL DB, what
about adding MongoDB usage as an outgoing channel?

Regards,
Carles

On 05/02/2014 10:47 AM, Coeuz wrote:

Hello Carles,

Do you have any time estimation about issue 207?
And, can I take part of the work or help in some other way to speed it up?

The best one would be ‘for the next release’ which means several months
but it’s true that you could really help out simply by attempting to
create a server running with SQLite instead of Postgres.

This should really be a simple matter initially -
zato.common.init.py defines a couple of constant/templates where
Postgres is mentioned, such as SUPPORTED_DB_TYPES or engine_def. An
entry for SQLite should be added and at that point we’d find out if a
quickstart cluster starts up at all.

It’s okay if a library for SQLite is a wrapper around a C one. SQL ODB
is read up in its entirety when servers start and its contents is cached
locally - that’s why servers start several seconds or more instead of
0.2s - they cache everything so in runtime they don’t have to consult
SQL at all = they are faster.

The next place SQL ODB is used is when an object, say a channel, is
created. That information is stored in SQL ODB first before being
propagated and cached locally by all servers.

In other words, as far as SQL ODB is concerned, we don’t have to shun C
libraries even though servers are based on gevent.

What about using it or any similar NoSQL DB as an (optional) alternative
to both Postgres and Redis?

These are two questions:

  • As an alternative to Redis - this should be fairly easy and that’s why
    it’s called KVDB everywhere (key/value db) instead of Redis, so that
    other systems can be plugged in.

  • As an alternative to Postgres - this is probably a couple of months of
    work. First tests against all the API services would have to be added
    (more than 200 services in the next release) and then all the internal
    services - some 10-15k lines of code - would have to be rewritten to
    support a non-SQL backend. This is naturally doable but unless a sponsor
    or a dedicated contributor steps up, I can’t see its being done this year.

In any case, even if it’s too complex to make it replace a SQL DB, what
about adding MongoDB usage as an outgoing channel?

Sure, I’d be very happy to guide you but how to start it depends on
whether the MongDB library is:

  1. A dedicated library for gevent
  2. In pure Python and thread-safe
  3. In pure Python and not thread-safe
  4. A Python wrapper around a C library

We use greenlets so it becomes greenlet-safe for us but developers
usually only talk about thread safety instead of how a library
cooperates with gevent.

This ticket …

… added LDAP connections - of type 3) above - and I went with the user
through the major parts of adding such connections to the core. Please
have a look at it, this all still stands. Of the 4 types listed above,
which is the MongoDB library?

thanks,

On 05/02/2014 10:47 AM, Coeuz wrote:

Hello Carles,

Do you have any time estimation about issue 207?
And, can I take part of the work or help in some other way to speed it up?

The best one would be ‘for the next release’ which means several months
but it’s true that you could really help out simply by attempting to
create a server running with SQLite instead of Postgres.

This should really be a simple matter initially -
zato.common.init.py defines a couple of constant/templates where
Postgres is mentioned, such as SUPPORTED_DB_TYPES or engine_def. An
entry for SQLite should be added and at that point we’d find out if a
quickstart cluster starts up at all.

It’s okay if a library for SQLite is a wrapper around a C one. SQL ODB
is read up in its entirety when servers start and its contents is cached
locally - that’s why servers start several seconds or more instead of
0.2s - they cache everything so in runtime they don’t have to consult
SQL at all = they are faster.

The next place SQL ODB is used is when an object, say a channel, is
created. That information is stored in SQL ODB first before being
propagated and cached locally by all servers.

In other words, as far as SQL ODB is concerned, we don’t have to shun C
libraries even though servers are based on gevent.

What about using it or any similar NoSQL DB as an (optional) alternative
to both Postgres and Redis?

These are two questions:

  • As an alternative to Redis - this should be fairly easy and that’s why
    it’s called KVDB everywhere (key/value db) instead of Redis, so that
    other systems can be plugged in.

  • As an alternative to Postgres - this is probably a couple of months of
    work. First tests against all the API services would have to be added
    (more than 200 services in the next release) and then all the internal
    services - some 10-15k lines of code - would have to be rewritten to
    support a non-SQL backend. This is naturally doable but unless a sponsor
    or a dedicated contributor steps up, I can’t see its being done this year.

In any case, even if it’s too complex to make it replace a SQL DB, what
about adding MongoDB usage as an outgoing channel?

Sure, I’d be very happy to guide you but how to start it depends on
whether the MongDB library is:

  1. A dedicated library for gevent
  2. In pure Python and thread-safe
  3. In pure Python and not thread-safe
  4. A Python wrapper around a C library

We use greenlets so it becomes greenlet-safe for us but developers
usually only talk about thread safety instead of how a library
cooperates with gevent.

This ticket …

… added LDAP connections - of type 3) above - and I went with the user
through the major parts of adding such connections to the core. Please
have a look at it, this all still stands. Of the 4 types listed above,
which is the MongoDB library?

thanks,

On 05/08/2014 07:01 PM, Dariusz Suchojad wrote:

  • In your service you can invoke this method via ‘conn =
    self.app_context.your_method_name()’

Eh, sorry - please make it self.server.app_context.your_method_name()