On 31/12/14 00:30, Nathan Stults wrote:
- Create migrations to add appropriate configuration to the ODB
- Create the actual channel class and supporting classes
- Tests for #2
- Admin website forms / menu items
- Tests for #4
Are there any major missing pieces? Is there a better way?
That’s the general approach though it would need to be broken down into
specifics. Let me offer an overview along with some broad remarks that
may be interesting for others as well.
Yes, adding Alembic migrations would be needed
This is the major part easily constituting 90% of the work
Yes, both unit tests and API tests
This would be quite easy - there’s a generic framework for adding new
forms and auto-generating parts of the GUI. Menu items are added manually.
There are no tests for the GUI yet, though more on that below, but
this could be a good moment to add them.
This would be easy - docs are in Sphinx and there’s a lot to base
Kafka ones on.
This is only a matter of adding one or more migrations here
nothing really fancy.
- Actual channel classes
This is a few parts actually. On top of that, given that it would make
sense to use Kafka in both directions, both as channels and outgoing
generalconnections, the works would have to be split into Kafka
definitions and Kafka channels - akin to how AMQP connections work today.
2a. Admin API - CRUD services for Kafka connection definitions
2b. Admin API - CRUD services for Kafka channels
2c. Plugging definition services into the worker store
2d. Plugging channel services into the worker store
2e. Implementing classes establishing connections
2f. Implementing classes notifying services of new Kafka messages (the
actual channel part)
2e and 2f
The situation here is that in Zato the mantra is 'as few restarts as
possible'. Hence a lot of code goes out of its way to make everything
work when faced with questions:
- What to do if a starting server cannot establish a connection? How
long to keep retrying?
- What happens if a connection could not be established, we are paused
serverfor a few seconds and suddenly a user edits the connection
changing TCP destination? What if they change credentials to connect
with? What if they delete a connection and we are still trying to
- What happens if the underlying Python client is not thread-safe at
all? (Zato uses greenlets, lightweight threads, but from the library's
point of view thread = greenlet).
- How to establish connections if the library is thread-safe, i.e. no
global variables but the connection objects are not?
- How to deal with connections established in C instead of Python given
that gevent cannot green such ones?
All of this questions deal with bullets 2e and 2f and these are the most
time-consuming parts but luckily there are a few base classes
implementation of which lets one deal on protocol-specific matters, most
of the time.
For instance ..
.. this lets, say, the actual ElasticSearch and Solr classes be very thin:
2a and 2b
Points 2a and 2b are fairly simple, there’s another framework for
auto-generating internal CRUD services …
… and this means the implementation is very trivial:
2c and 2d
Each server has a so called worker store which is a central place out of
which individual pieces of configuration branch off:
This is also the object that receives configuration messages. For
instance, when you click 'Create connection' in web-admin, the road is:
- API service stores new connection in the database
- The same service published a message on a Redis-based internal broker
notifying other servers, including our own, that a new object should be
- Each of the worker stores picks up the message and actually create a
3. Tests for 2
Two types of tests are needed:
- Unit tests for testing things in isolation, such as here:
- API tests for checking whole scenarios against live environments, such
API tests use zato-apitest:
Note that web-admin always uses API services for the management of
objects under its control hence creating API tests is an important part
of the web-admin testing as well - web-admin is just a Django-based console.
4. Admin forms and menu
Menu items are added manually here:
This takes care of generating or validating the forms or object listings.
What you only need to do is:
- Prepare Django forms - super easy
- Prepare Django views - easy
- Prepare HTML - super easy
- Prepare JS declaring what fields to use and what happens on
Create/Edit/Delete events, basically, filling out place holders - easy
Here are some links:
5. Tests for 4
API tests take care of the majority of the work web-admin does but I
would certainly like for the project to have frontend tests as well -
what do you think would be the good tool here, Selenium?
Simply copy/paste the general structure from, say, ZeroMQ and that's it.
What needs be added:
- Web admin forms
- API docs
- Usage examples
That's the whole of it, 2c-2f are definitely the parts most of the time
can be spent on. I'd definitely tackle all the other parts before
starting out on them - luckily, all the parts can be implemented