(Migrated) Extending Zatos: Adding new Channel Types

(This message has been automatically imported from the retired mailing list)

Hello again :slight_smile:

I am interested in potentially using Kafka as our messaging backbone. I was
wondering what the level of effort would be for us to add a new channel typ=
e
to Zatos in a robust enough way that they would have a chance of being
accepted into the project. I haven=B9t read through all the source code for
existing channels yet, but from what I have seen I am guessing the workflow
might look l this:

  1. Create migrations to add appropriate configuration to the ODB
  2. Create the actual channel class and supporting classes
  3. Tests for #2
  4. Admin website forms / menu items
  5. Tests for #4
  6. Docs
    Are there any major missing pieces? Is there a better way?

Thank you,

Nathan=20

Again, thanks for the detailed answer, I think that gives us everything we
need to get started. As for UI tests, yes I suppose Selenium would do just
fine.

Thank you,

Nathan

On 12/31/14, 8:58 AM, “Dariusz Suchojad” dsuch@zato.io wrote:

On 31/12/14 00:30, Nathan Stults wrote:

  1. Create migrations to add appropriate configuration to the ODB
  2. Create the actual channel class and supporting classes
  3. Tests for #2
  4. Admin website forms / menu items
  5. Tests for #4
  6. Docs

Are there any major missing pieces? Is there a better way?

That’s the general approach though it would need to be broken down into
specifics. Let me offer an overview along with some broad remarks that
may be interesting for others as well.

  1. Yes, adding Alembic migrations would be needed

  2. This is the major part easily constituting 90% of the work

  3. Yes, both unit tests and API tests

  4. This would be quite easy - there’s a generic framework for adding new
    forms and auto-generating parts of the GUI. Menu items are added manually.

  5. There are no tests for the GUI yet, though more on that below, but
    this could be a good moment to add them.

  6. This would be easy - docs are in Sphinx and there’s a lot to base
    Kafka ones on.

  7. Alembic migrations
    =====================

This is only a matter of adding one or more migrations here
https://github.com/zatosource/zato/tree/master/code/alembic/versions -
nothing really fancy.

  1. Actual channel classes
    =========================

This is a few parts actually. On top of that, given that it would make
sense to use Kafka in both directions, both as channels and outgoing
generalconnections, the works would have to be split into Kafka
definitions and Kafka channels - akin to how AMQP connections work today.

2a. Admin API - CRUD services for Kafka connection definitions
2b. Admin API - CRUD services for Kafka channels
2c. Plugging definition services into the worker store
2d. Plugging channel services into the worker store
2e. Implementing classes establishing connections
2f. Implementing classes notifying services of new Kafka messages (the
actual channel part)

2e and 2f


The situation here is that in Zato the mantra is 'as few restarts as
possible'. Hence a lot of code goes out of its way to make everything
work when faced with questions:

- What to do if a starting server cannot establish a connection? How
long to keep retrying?

- What happens if a connection could not be established, we are paused
serverfor a few seconds and suddenly a user edits the connection
changing TCP destination? What if they change credentials to connect
with? What if they delete a connection and we are still trying to
establish it?

- What happens if the underlying Python client is not thread-safe at
all? (Zato uses greenlets, lightweight threads, but from the library's
point of view thread = greenlet).

- How to establish connections if the library is thread-safe, i.e. no
global variables but the connection objects are not?

- How to deal with connections established in C instead of Python given
that gevent cannot green such ones?

All of this questions deal with bullets 2e and 2f and these are the most
time-consuming parts but luckily there are a few base classes
implementation of which lets one deal on protocol-specific matters, most
of the time.

For instance ..

https://github.com/zatosource/zato/blob/master/code/zato-server/src/zato/s
erver/store.py

https://github.com/zatosource/zato/blob/master/code/zato-server/src/zato/s
erver/connection/queue.py

https://github.com/zatosource/zato/blob/master/code/zato-server/src/zato/s
erver/connection/__init__.py

.. this lets, say, the actual ElasticSearch and Solr classes be very thin:

https://github.com/zatosource/zato/blob/master/code/zato-server/src/zato/s
erver/connection/search/es.py

https://github.com/zatosource/zato/blob/master/code/zato-server/src/zato/s
erver/connection/search/solr.py

2a and 2b

Points 2a and 2b are fairly simple, there’s another framework for
auto-generating internal CRUD services …

https://github.com/zatosource/zato/blob/master/code/zato-server/src/zato/s
erver/service/meta.py

… and this means the implementation is very trivial:

https://github.com/zatosource/zato/blob/master/code/zato-server/src/zato/s
erver/service/internal/search/es.py

https://github.com/zatosource/zato/blob/master/code/zato-server/src/zato/s
erver/service/internal/search/solr.pyhttps://github.com/zatosource/zato/bl
ob/master/code/zato-server/test/zato/server/base/test_parallel.py

2c and 2d


Each server has a so called worker store which is a central place out of
which individual pieces of configuration branch off:

https://github.com/zatosource/zato/blob/master/code/zato-server/src/zato/s
erver/base/worker.py

This is also the object that receives configuration messages. For
instance, when you click 'Create connection' in web-admin, the road is:

- API service stores new connection in the database

- The same service published a message on a Redis-based internal broker
notifying other servers, including our own, that a new object should be
created

- Each of the worker stores picks up the message and actually create a
new connection

3. Tests for 2
==============

Two types of tests are needed:

- Unit tests for testing things in isolation, such as here:

https://github.com/zatosource/zato/blob/master/code/zato-server/test/zato/
server/base/test_parallel.py

- API tests for checking whole scenarios against live environments, such
as here:

https://github.com/zatosource/zato/blob/master/code/apitest/features/zato_
scheduler_job_interval_based.feature

API tests use zato-apitest:

https://github.com/zatosource/zato-apitest

Note that web-admin always uses API services for the management of
objects under its control hence creating API tests is an important part
of the web-admin testing as well - web-admin is just a Django-based
console.

4. Admin forms and menu
=======================

Menu items are added manually here:

https://github.com/zatosource/zato/blob/master/code/zato-web-admin/src/zat
o/admin/templates/zato/index.html

As for the forms, again, there's a sort of a JavaScript framework called
$.fn.zato.data_table:

https://github.com/zatosource/zato/blob/master/code/zato-web-admin/src/zat
o/admin/static/js/common.jshttps://github.com/zatosource/zato/blob/master/
code/zato-web-admin/src/zato/admin/static/js/common.js

This takes care of generating or validating the forms or object listings.

What you only need to do is:

- Prepare Django forms - super easy
- Prepare Django views - easy
- Prepare HTML - super easy
- Prepare JS declaring what fields to use and what happens on
Create/Edit/Delete events, basically, filling out place holders - easy

Here are some links:

https://github.com/zatosource/zato/blob/master/code/zato-web-admin/src/zat
o/admin/web/forms/search/es.py

https://github.com/zatosource/zato/blob/master/code/zato-web-admin/src/zat
o/admin/web/views/search/es.py

https://github.com/zatosource/zato/blob/master/code/zato-web-admin/src/zat
o/admin/templates/zato/search/es.html

https://github.com/zatosource/zato/blob/master/code/zato-web-admin/src/zat
o/admin/static/js/search/es.jsKanal

5. Tests for 4
==============

API tests take care of the majority of the work web-admin does but I
would certainly like for the project to have frontend tests as well -
what do you think would be the good tool here, Selenium?

6. Docs
=======

Simply copy/paste the general structure from, say, ZeroMQ and that's it.
What needs be added:

- Web admin forms
- API docs
- Usage examples

Summary
=======

That's the whole of it, 2c-2f are definitely the parts most of the time
can be spent on. I'd definitely tackle all the other parts before
starting out on them - luckily, all the parts can be implemented
independently.

On 31/12/14 00:30, Nathan Stults wrote:

  1. Create migrations to add appropriate configuration to the ODB
  2. Create the actual channel class and supporting classes
  3. Tests for #2
  4. Admin website forms / menu items
  5. Tests for #4
  6. Docs

Are there any major missing pieces? Is there a better way?

That’s the general approach though it would need to be broken down into
specifics. Let me offer an overview along with some broad remarks that
may be interesting for others as well.

  1. Yes, adding Alembic migrations would be needed

  2. This is the major part easily constituting 90% of the work

  3. Yes, both unit tests and API tests

  4. This would be quite easy - there’s a generic framework for adding new
    forms and auto-generating parts of the GUI. Menu items are added manually.

  5. There are no tests for the GUI yet, though more on that below, but
    this could be a good moment to add them.

  6. This would be easy - docs are in Sphinx and there’s a lot to base
    Kafka ones on.

  7. Alembic migrations
    =====================

This is only a matter of adding one or more migrations here
https://github.com/zatosource/zato/tree/master/code/alembic/versions -
nothing really fancy.

  1. Actual channel classes
    =========================

This is a few parts actually. On top of that, given that it would make
sense to use Kafka in both directions, both as channels and outgoing
generalconnections, the works would have to be split into Kafka
definitions and Kafka channels - akin to how AMQP connections work today.

2a. Admin API - CRUD services for Kafka connection definitions
2b. Admin API - CRUD services for Kafka channels
2c. Plugging definition services into the worker store
2d. Plugging channel services into the worker store
2e. Implementing classes establishing connections
2f. Implementing classes notifying services of new Kafka messages (the
actual channel part)

2e and 2f


The situation here is that in Zato the mantra is 'as few restarts as
possible'. Hence a lot of code goes out of its way to make everything
work when faced with questions:

- What to do if a starting server cannot establish a connection? How
long to keep retrying?

- What happens if a connection could not be established, we are paused
serverfor a few seconds and suddenly a user edits the connection
changing TCP destination? What if they change credentials to connect
with? What if they delete a connection and we are still trying to
establish it?

- What happens if the underlying Python client is not thread-safe at
all? (Zato uses greenlets, lightweight threads, but from the library's
point of view thread = greenlet).

- How to establish connections if the library is thread-safe, i.e. no
global variables but the connection objects are not?

- How to deal with connections established in C instead of Python given
that gevent cannot green such ones?

All of this questions deal with bullets 2e and 2f and these are the most
time-consuming parts but luckily there are a few base classes
implementation of which lets one deal on protocol-specific matters, most
of the time.

For instance ..

https://github.com/zatosource/zato/blob/master/code/zato-server/src/zato/server/store.py

https://github.com/zatosource/zato/blob/master/code/zato-server/src/zato/server/connection/queue.py

https://github.com/zatosource/zato/blob/master/code/zato-server/src/zato/server/connection/__init__.py

.. this lets, say, the actual ElasticSearch and Solr classes be very thin:

https://github.com/zatosource/zato/blob/master/code/zato-server/src/zato/server/connection/search/es.py

https://github.com/zatosource/zato/blob/master/code/zato-server/src/zato/server/connection/search/solr.py

2a and 2b

Points 2a and 2b are fairly simple, there’s another framework for
auto-generating internal CRUD services …

https://github.com/zatosource/zato/blob/master/code/zato-server/src/zato/server/service/meta.py

… and this means the implementation is very trivial:

https://github.com/zatosource/zato/blob/master/code/zato-server/src/zato/server/service/internal/search/es.py

https://github.com/zatosource/zato/blob/master/code/zato-server/src/zato/server/service/internal/search/solr.pyhttps://github.com/zatosource/zato/blob/master/code/zato-server/test/zato/server/base/test_parallel.py

2c and 2d


Each server has a so called worker store which is a central place out of
which individual pieces of configuration branch off:

https://github.com/zatosource/zato/blob/master/code/zato-server/src/zato/server/base/worker.py

This is also the object that receives configuration messages. For
instance, when you click 'Create connection' in web-admin, the road is:

- API service stores new connection in the database

- The same service published a message on a Redis-based internal broker
notifying other servers, including our own, that a new object should be
created

- Each of the worker stores picks up the message and actually create a
new connection

3. Tests for 2
==============

Two types of tests are needed:

- Unit tests for testing things in isolation, such as here:

https://github.com/zatosource/zato/blob/master/code/zato-server/test/zato/server/base/test_parallel.py

- API tests for checking whole scenarios against live environments, such
as here:

https://github.com/zatosource/zato/blob/master/code/apitest/features/zato_scheduler_job_interval_based.feature

API tests use zato-apitest:

https://github.com/zatosource/zato-apitest

Note that web-admin always uses API services for the management of
objects under its control hence creating API tests is an important part
of the web-admin testing as well - web-admin is just a Django-based console.

4. Admin forms and menu
=======================

Menu items are added manually here:

https://github.com/zatosource/zato/blob/master/code/zato-web-admin/src/zato/admin/templates/zato/index.html

As for the forms, again, there's a sort of a JavaScript framework called
$.fn.zato.data_table:

https://github.com/zatosource/zato/blob/master/code/zato-web-admin/src/zato/admin/static/js/common.jshttps://github.com/zatosource/zato/blob/master/code/zato-web-admin/src/zato/admin/static/js/common.js

This takes care of generating or validating the forms or object listings.

What you only need to do is:

- Prepare Django forms - super easy
- Prepare Django views - easy
- Prepare HTML - super easy
- Prepare JS declaring what fields to use and what happens on
Create/Edit/Delete events, basically, filling out place holders - easy

Here are some links:

https://github.com/zatosource/zato/blob/master/code/zato-web-admin/src/zato/admin/web/forms/search/es.py

https://github.com/zatosource/zato/blob/master/code/zato-web-admin/src/zato/admin/web/views/search/es.py

https://github.com/zatosource/zato/blob/master/code/zato-web-admin/src/zato/admin/templates/zato/search/es.html

https://github.com/zatosource/zato/blob/master/code/zato-web-admin/src/zato/admin/static/js/search/es.jsKanal

5. Tests for 4
==============

API tests take care of the majority of the work web-admin does but I
would certainly like for the project to have frontend tests as well -
what do you think would be the good tool here, Selenium?

6. Docs
=======

Simply copy/paste the general structure from, say, ZeroMQ and that's it.
What needs be added:

- Web admin forms
- API docs
- Usage examples

Summary
=======

That's the whole of it, 2c-2f are definitely the parts most of the time
can be spent on. I'd definitely tackle all the other parts before
starting out on them - luckily, all the parts can be implemented
independently.

On 31/12/14 00:30, Nathan Stults wrote:

  1. Create migrations to add appropriate configuration to the ODB
  2. Create the actual channel class and supporting classes
  3. Tests for #2
  4. Admin website forms / menu items
  5. Tests for #4
  6. Docs

Are there any major missing pieces? Is there a better way?

That’s the general approach though it would need to be broken down into
specifics. Let me offer an overview along with some broad remarks that
may be interesting for others as well.

  1. Yes, adding Alembic migrations would be needed

  2. This is the major part easily constituting 90% of the work

  3. Yes, both unit tests and API tests

  4. This would be quite easy - there’s a generic framework for adding new
    forms and auto-generating parts of the GUI. Menu items are added manually.

  5. There are no tests for the GUI yet, though more on that below, but
    this could be a good moment to add them.

  6. This would be easy - docs are in Sphinx and there’s a lot to base
    Kafka ones on.

  7. Alembic migrations
    =====================

This is only a matter of adding one or more migrations here
https://github.com/zatosource/zato/tree/master/code/alembic/versions -
nothing really fancy.

  1. Actual channel classes
    =========================

This is a few parts actually. On top of that, given that it would make
sense to use Kafka in both directions, both as channels and outgoing
generalconnections, the works would have to be split into Kafka
definitions and Kafka channels - akin to how AMQP connections work today.

2a. Admin API - CRUD services for Kafka connection definitions
2b. Admin API - CRUD services for Kafka channels
2c. Plugging definition services into the worker store
2d. Plugging channel services into the worker store
2e. Implementing classes establishing connections
2f. Implementing classes notifying services of new Kafka messages (the
actual channel part)

2e and 2f


The situation here is that in Zato the mantra is 'as few restarts as
possible'. Hence a lot of code goes out of its way to make everything
work when faced with questions:

- What to do if a starting server cannot establish a connection? How
long to keep retrying?

- What happens if a connection could not be established, we are paused
serverfor a few seconds and suddenly a user edits the connection
changing TCP destination? What if they change credentials to connect
with? What if they delete a connection and we are still trying to
establish it?

- What happens if the underlying Python client is not thread-safe at
all? (Zato uses greenlets, lightweight threads, but from the library's
point of view thread = greenlet).

- How to establish connections if the library is thread-safe, i.e. no
global variables but the connection objects are not?

- How to deal with connections established in C instead of Python given
that gevent cannot green such ones?

All of this questions deal with bullets 2e and 2f and these are the most
time-consuming parts but luckily there are a few base classes
implementation of which lets one deal on protocol-specific matters, most
of the time.

For instance ..

https://github.com/zatosource/zato/blob/master/code/zato-server/src/zato/server/store.py

https://github.com/zatosource/zato/blob/master/code/zato-server/src/zato/server/connection/queue.py

https://github.com/zatosource/zato/blob/master/code/zato-server/src/zato/server/connection/__init__.py

.. this lets, say, the actual ElasticSearch and Solr classes be very thin:

https://github.com/zatosource/zato/blob/master/code/zato-server/src/zato/server/connection/search/es.py

https://github.com/zatosource/zato/blob/master/code/zato-server/src/zato/server/connection/search/solr.py

2a and 2b

Points 2a and 2b are fairly simple, there’s another framework for
auto-generating internal CRUD services …

https://github.com/zatosource/zato/blob/master/code/zato-server/src/zato/server/service/meta.py

… and this means the implementation is very trivial:

https://github.com/zatosource/zato/blob/master/code/zato-server/src/zato/server/service/internal/search/es.py

https://github.com/zatosource/zato/blob/master/code/zato-server/src/zato/server/service/internal/search/solr.pyhttps://github.com/zatosource/zato/blob/master/code/zato-server/test/zato/server/base/test_parallel.py

2c and 2d


Each server has a so called worker store which is a central place out of
which individual pieces of configuration branch off:

https://github.com/zatosource/zato/blob/master/code/zato-server/src/zato/server/base/worker.py

This is also the object that receives configuration messages. For
instance, when you click 'Create connection' in web-admin, the road is:

- API service stores new connection in the database

- The same service published a message on a Redis-based internal broker
notifying other servers, including our own, that a new object should be
created

- Each of the worker stores picks up the message and actually create a
new connection

3. Tests for 2
==============

Two types of tests are needed:

- Unit tests for testing things in isolation, such as here:

https://github.com/zatosource/zato/blob/master/code/zato-server/test/zato/server/base/test_parallel.py

- API tests for checking whole scenarios against live environments, such
as here:

https://github.com/zatosource/zato/blob/master/code/apitest/features/zato_scheduler_job_interval_based.feature

API tests use zato-apitest:

https://github.com/zatosource/zato-apitest

Note that web-admin always uses API services for the management of
objects under its control hence creating API tests is an important part
of the web-admin testing as well - web-admin is just a Django-based console.

4. Admin forms and menu
=======================

Menu items are added manually here:

https://github.com/zatosource/zato/blob/master/code/zato-web-admin/src/zato/admin/templates/zato/index.html

As for the forms, again, there's a sort of a JavaScript framework called
$.fn.zato.data_table:

https://github.com/zatosource/zato/blob/master/code/zato-web-admin/src/zato/admin/static/js/common.jshttps://github.com/zatosource/zato/blob/master/code/zato-web-admin/src/zato/admin/static/js/common.js

This takes care of generating or validating the forms or object listings.

What you only need to do is:

- Prepare Django forms - super easy
- Prepare Django views - easy
- Prepare HTML - super easy
- Prepare JS declaring what fields to use and what happens on
Create/Edit/Delete events, basically, filling out place holders - easy

Here are some links:

https://github.com/zatosource/zato/blob/master/code/zato-web-admin/src/zato/admin/web/forms/search/es.py

https://github.com/zatosource/zato/blob/master/code/zato-web-admin/src/zato/admin/web/views/search/es.py

https://github.com/zatosource/zato/blob/master/code/zato-web-admin/src/zato/admin/templates/zato/search/es.html

https://github.com/zatosource/zato/blob/master/code/zato-web-admin/src/zato/admin/static/js/search/es.jsKanal

5. Tests for 4
==============

API tests take care of the majority of the work web-admin does but I
would certainly like for the project to have frontend tests as well -
what do you think would be the good tool here, Selenium?

6. Docs
=======

Simply copy/paste the general structure from, say, ZeroMQ and that's it.
What needs be added:

- Web admin forms
- API docs
- Usage examples

Summary
=======

That's the whole of it, 2c-2f are definitely the parts most of the time
can be spent on. I'd definitely tackle all the other parts before
starting out on them - luckily, all the parts can be implemented
independently.