(Migrated) Connection packs (idea)

(This message has been automatically imported from the retired mailing list)

Hi there,

following up on a recent IRC discussion regarding adding to Zato
connectors to external systems using the latter’s own API, I’d like to
discuss an idea I’ve been brewing for a moment.

Basically, it is about creating JSON documents to describe a given
domain system’s API, have this API uploaded to Zato in order for Zato to
automatically generate gluing services responsible for mapping user
input to an external system’s format. Such services would then invoke
the system and convert the response back to the called.

User code could then invoke these services using business objects only
(in Python terms this would simply anything that is dict or dict-like),
without thinking about any particularities of the underlying transport
or data format. This is already possible in Zato to a great level but
let’s push this idea even more.

The idea of describing interfaces surely rings a bell, it’s almost like
WSDLs in other parts of the server world, but here I am deliberately
thinking of scaling it down to a way simpler approach, one that would
deal with APIs only, no actions, includes, port types, endpoints -> this
Zato already has and it is known in its lingo under different names
(channels, outgoing connections, data_format etc.)

This wouldn’t also do any validation itself (but it would be possible
still, as explained later).

No validation is I think the only core thing /really/ making it
something different than WSDLs (apart from not being heavy XML people
generally look for ways to avoid, of course). Basically, this idea is
about mapping business objects (dicts) to external format, not about
validating anything. This would allow one to work with dicts or Bunches
(a very useful dict subclass), as it is customary to do in Python.

This could also deal with both JSON and more REST-ish (parameters in
URL) external systems. (XML/SOAP requests would be only slightly less
easier).

For instance, have a look at this Twitter API call:

https://dev.twitter.com/docs/api/1.1/post/friendships/update

Example address to invoke is:

https://api.twitter.com/1/statuses/show.json?id=112652479837110273&include_entities=true

The idea now is to create a JSON file along the lines of:

{‘conn_pack’: ‘Twitter’:
{
‘name’: ‘friendships_update’,
‘api_form’: ‘REST’,
‘method’: ‘POST’
}
}

out of which some sort of a glue service would be generated so that
users in their services would do

from zato.server.service import Service

class MyService(Service):
def handle(self):

     api_name = 'friendships_update'
     params = {'user_id':1, 'device':True}

     resp = self.outpack.get('Twitter').invoke(api_name, params)

     # resp is now a dict or a dict-like object

Naturally, not everything will be so flat as Twitter so another example
of using an API with more nested structures is

{‘conn_pack’: ‘MySystem’:
{
‘name’: ‘update’,
‘api_form’: ‘JSON’
‘method’: ‘POST’
}
}

Bunch https://pypi.python.org/pypi/bunch

from bunch import Bunch

Zato

from zato.server.service import Service

class MyService(Service):
def handle(self):

     customer = Bunch()
     customer.id = 123
     customer.name = 'Alice Grain'
     customer.segment = 'AKZ'

     product1 = Bunch()
     product1.type_id = 1
     product1.created = '2008-12-25'

     product2 = Bunch()
     product2.type_id = 2
     product2.created = '2013-05-14'

     customer.products = [product1, product2]

     self.outpack.get('MySystem').invoke('update', customer)

Some common attributes could be moved a level higher - so if each or
almost each of the API calls is of the same type (JSON, REST or SOAP),
it won’t make sense to specify it all the time for each call.

I won’t show it here but using XML/SOAP would look like almost exactly
the same - the only difference would be that you’d need to use
lxml.objectify instead of dict-like things. This would be similar to
what is already possible with Zato like here
https://zato.io/docs/progguide/xml.html#accessing-request-elements

Another thing is hooks. Like in other parts of Zato
(https://zato.io/docs/progguide/service-hooks.html) people will surely
need things beyond we will come up here so there must be a way to
customize things.

Hence, there should be at least 2 hooks for users to optionally
implement somewhere that would let them control additionally how
requests and responses are treated:

  • before_request - gets user input before it’s passed along to the
    external system and does anything is needed, for instance, uses XSD to
    validate the input before it leaves Zato

  • after_response - gets response from an external system and again, does
    what is needed, this could be used to plug validation it manually

Hooks are also where Zato could with time use XSD, RelaxNG or something
else automatically (but let’s not do it now).

The whole thing should also take API versioning into account. I haven’t
thought about it yet but I’m just signalling a need.

There are 2 things I think worth explaining still.

Why am I insisting on no XSD for now? Because the bigger systems to
integrate the less useful it is.

XSD sounds and works quite OK first but when you deal with thousands of
business objects (classes) all grouped into multi-level hierarchies,
almost inevitably everything in an XSD will be optional.

This is simply how it looks like.

The only usefulness of an XSD (at least as far as validation goes) is
that it checks the order of elements provided. But frankly, as longs as
you use Python instead of some custom-built XML parser, it’s really of
no concern.

In lower level languages XSD can also be used to generate all sorts of
useful things that let you deal with business classes in your code
instead of dealing with XML. This is cool and very handy indeed. But
this is Python - we can use business objects out of the box just like
that, as in the examples above.

And I’m not saying there will never be XSD/RelaxNG/anything - I’m just
saying it’s not a priority. If anyone truly needs it -> there are hooks
for that.

The other thing is that I /really/ like it to be something that
generates Python classes because there must be a way for users to
override anything that is automatically-generated. This classes must
also be not overly smart. Not too much magic, no metaclasses probably.
It must not look like SWIG, this must be something that is very easy to
customize manually because we won’t think of everything and we won’t
cover every need possible, hence it is 100% sure that people will edit
it sometimes themselves.

I don’t know exactly how to achieve it all but I think 95% of stuff
needed to complete it is already in Zato, it just needs to be nicely
connected now to provide a new value in form of that feature.

With time we can go further. Let’s add a ‘zato connpack’ CLI command to
https://zato.io/docs/admin/cli/index.html, such as

  • zato connpack get zimbra
  • zato connpack upgrade twitter
  • zato connpack install salesforce 3.1
  • zato connpack search fabecook -> (no ‘fabecook’ found, did you mean
    ’facebook’?)

etc. etc.

We can next create a central repository of user-provided connpacks to
let everyone collaborate on useful packs they themselves need. I’m
entirely sure very quickly people would be uploading packs to systems we
didn’t even know they existed. An ecosystem of APIs - that would be
something to look at.

Using Salesforce as an example, the only thing that is needed from Zato
now is a means to describe its Salesforce API within, say, 1 week of work.

If it’s made possible to express APIs of Salesforce’s size, people will
be able to add connpacks for smaller systems in one evening. Which will
be awesome :slight_smile:

What do you think?