(Migrated) Modelling of order provisioning

(This message has been automatically imported from the retired mailing list)

I’m having a look at Zato and finding it very interesting. I’d like some
advice on the best approach to take.

In the documentation it talks about a service as being “Interesting,
Re-usable, Atomic”. My question is, how might you best model a
provisioning order engine in terms of Zato service(s)?

Specifically I’m thinking about line ordering and provisioning,
implementing a service which implements ALA ND1651
http://www.niccstandards.org.uk/publications/ala.cfm

This is a work flow for provisioning a customer line such as DSL. An
order is a long lived process, i.e. it could take several weeks to
complete. The process is:

  • the buyer submits an order
  • multiple notifications are sent back to the buyer at different stages,
    giving order progress
  • the buyer may submit updates to the order at various points (e.g. to
    confirm an appointment, or confirm that an extra charge is acceptable)
  • the buyer may cancel the order at any point
  • at the end, a final notification is sent that the order is complete

During the execution of this service, call outs are made to other
back-end systems to perform the work (e.g. check availability, schedule
appointments etc)

The salient points are:

  • This is not a simple request-response type of service
  • It is “atomic”, in the sense that by the end, the whole order has
    either been successfully completed or cancelled/rejected
  • However, multiple steps and message exchanges with the requestor are
    required to reach that point

So, in Zato, can the whole order flow be considered a “service” - one
which generates multiple request/response messages during its lifetime?

Or should “request order”, “update order”, “cancel order”, “notify order
status” etc all be considered separate “services” in Zato? In that case,
it seems to me there would be a need for a separate workflow engine for
processing the order, since Zato would consider each individual message
finished once it had delivered it.

Additional information: in this standard all the messages are XML, and
there are multiple ways of transporting them. The simplest one is SOAP
over HTTP request-response exchanges (ND1651 section 13.2), e.g.

–> orderRequest
<

On 11/02/15 12:19, Brian Candler wrote:

Or should “request order”, “update order”, “cancel order”, “notify order
status” etc all be considered separate “services” in Zato? In that case,
it seems to me there would be a need for a separate workflow engine for
processing the order, since Zato would consider each individual message
finished once it had delivered it.

Hi Brian,

yes, that’s how I’d approach it. Have Zato focus on individual calls,
each implemented as a separate service and all of them contributing to
the higher-level processes.

Question is, how granular the services would be and that would depend on
how re-usable there were.

I’m not familiar with ALA ND1651 and it looks it goes straight into
implementation details without presenting a semi-formal overview of the
process - in particular, I’d really like to understand what sort parties
are being integrated here so I may be off below.

For instance, it would look that Cease should rather be a process rather
than a service.

That process has a start, which means someone initiates a request to
Zato (to a Zato service) that knows who, which systems, to notify that
such and such ALA service needs to cease. Some of the systems will be
simply notified and some will need to produce responses.

Zato doesn’t come with a workflow engine but I expect it will grow one
within 2-3 releases.

Right now it sounds you’d definitely want to make use of asynchronous
messaging patterns and publish/subscribe:

https://zato.io/docs/progguide/patterns/fan-out-fan-in.html
https://zato.io/docs/progguide/patterns/parallel-exec.html
https://zato.io/docs/progguide/patterns/invoke-retry.html
https://zato.io/docs/pubsub/index.html

All of them work on different levels of your implementation

Let’s say we continue with Cease.

  • An acceptCease service is invoked initially
  • That service fans out to 3 other services
  • One of them may publish a message using pub/sub
  • The other two may need to synchronously invoke some other 2 backend
    systems
  • Once all of them complete, a final callback service gets invoked which
    confirms to the initial that the Cease process ended successfully

At all points of the workflow you can, and in fact you should, store the
current state of the process in Redis so should in a catastrophic
situation your whole environment go down, you would be still able to
re-start the process.

Users also employ event-based approach in such situations. Essentially,
you keep a list of events and event handlers - where an event is an
interesting occurrence, such as the Cease business process starting and
https://zato.io/docs/progguide/patterns/async-invoke.htmlhandlers are
services registered for them. The list can be stored in Redis or cached
locally. Now each time an event occurs the list is consulted and
services registered are invoked. After they complete they invoke
services further in the chain so they are all loosely-coupled and the
only thing that binds them all together are the events they react to.

Here async callbacks will come in handy:

https://zato.io/docs/progguide/patterns/async-invoke.html

I’d definitely start with turning the concepts from ALA ND1651 into a
couple of diagrams to visualize how many actors there, what their role
is, and what the interactions between them are. It doesn’t need to be
BPMN right away - a set of simple diagrams like here would do:

https://zato.io/blog/posts/linux-journal-article-agile-esb-soa-rest-and-cloud-integrations-in-python.html

However, I’d use UML activity diagrams to specify actual steps once
actors and major state transitions are understood.

Zato services would follow from activity diagrams in that situations.

Let’s say we end up with a diagram like the one here - activities are
the parts that are closest to the service layer and decisions are simple
Python ifs:

http://www.learnsad.com/UMLACT.html

That’s how I’d translate it into Zato services:

  • Customer sends a request -> Zato service called create_request

  • System confirms the receipt -> create_request produces correct responses

  • Confirm the order - use pub/sub to notify the initial caller that the
    order is being processed

  • Dispatch the order - use pub/sub to notify some backend system that an
    order needs to be processed

As we can see, it turns out that this diagram doesn’t represent the
whole process - for instance, how is the original caller notified of the
progress? Is it at all? How does it know the processing is done?

All such questions need to be answered by the higher level design though
in your situation it will be quite easy because you already have a
formal spec.

However, Order Management can be surprising from time to time so I
simply wouldn’t jump straight into coding without an analysis phase the
product of which should be design documents.

Whether they are XML, SOAP or JSON is just an implementation detail -
generally, in Zato you work with pure-Python objects, mostly with dicts
or dict-like structures so you typically don’t deal with serialization,
headers and such unless you need to.

Happy to discuss it further if you can suggest a chapter from the spec
you’d like to go through in detail.

On 11/02/15 22:03, Brian Candler wrote:

Let me just clarify:

ext input --> Zato service X --> ext service Z [with callback Y]
<–response
Zato service Y

Ok, this is how you’d approach it in Zato:

That is, there would be three Zato services:

  • A channel one simply accepting the external input and possibly
    performing basic validation to make sure that are no gross validation
    issues, errors etc.

  • The channel stores in Redis cache everything that is important for
    that request by its CID and invokes the adapter telling the invoke_async
    call what the callback is. The adapter is also given the CID on input.

  • The adapter invokes an external service synchronously parameters and
    produces response. The response contains the original CID.

  • Zato’s async mechanisms understand that the adapter was actually
    invoked in async and find out that there is a callback for that
    invocation - the callback is executed

  • Callback knows that it’s a callback for some earlier service and given
    that it receives a CID, it looks up in Redis everything stashed away for it.

This is working code here on GitHub, you just need to fill in the blanks
with actual requests and responses:

I included a few other tips like using self.request.bunchified() which
are not strictly necessary but make work even more convenient.

Another point is that in bigger environments there will also be an
additional Business/Core (naming varies) layer between channels and
adapters so that the decision which backend to invoke and how to
construct composite services is not baked into channels directly.

Another point of clarification: does the asynchronous-call-with-
callback model only work for HTTP-style request-response exchanges?

The key point to always remember is that in Zato a service is just a
Python class with the ‘handle’ method and some other attributes. It’s
completely insulated from underlying protocols and can be exposed on
multiple channels completely independently - the same one can accepting
messages from HTTP, pub/sub, async, ZeroMQ, OpenStack Swift, SQL,
WebSphere MQ - simply, from all the channels and notification types.

In fact, by default a service receives input, performs some operations,
produces output and that’s it. If you must, you can check its
self.channel to understand which route the request came in but there is
a strict separation of integration logic from transports and security.

You’re absolutely right, for a reliable workflow system there needs
to be some sort of checkpointing at each completed stage

The feature is called guaranteed delivery but it didn’t make it for
release 2.0.

There was a prototype in Redis that still needs to be properly
implemented using an SQL backend rather than Redis.

https://github.com/zatosource/zato/blob/master/code/zato-common/src/zato/common/delivery.py

The other thing I really want is a good test framework

This is where zato-apitest should be used, check it out here:

You are right the docs don’t mention it and it’s a clear omission that
will be fixed.

On 02/11/2015 12:19 PM, Brian Candler wrote:

I’m having a look at Zato and finding it very interesting. I’d like some
advice on the best approach to take.

In the documentation it talks about a service as being “Interesting,
Re-usable, Atomic”. My question is, how might you best model a
provisioning order engine in terms of Zato service(s)?

(I’ve read most of the discussion, but I am answering here.)

To be honest I have one short answer:

Do not use Zato (nor any other ESB) for this.

Zato is a very good companion for a solution that can handle what you
are looking for. I am basically using it for parts (for now) of such
order/delivery/provisioning system for DSL/Fiber boxes.

But as an ESB, not Order and workflow engine.

What you need is something that can do the O & W which then talks to
Zato for al the communitcation between it and the (usually way too
many…) other systems involved in a value chain like that. Tnen you can
do a clean ND1651 workflow with it’s messages and Zato can be used to
transpose/translate between it and whatever the different systems use.

That will keep the complexity of the O & W system down as you wouldn’t
have to consider handling REST, SOAP, screen scraping (yes, it’s still
being done even in workflows like this in some cases) and so on, but let
Zato take care of it.

There is another approach aswell, using Zato as the B2B front end for
this and use some generic order and workflow engine for that part (DO
NOT USE A CRM SYSTEM FOR THIS!). Then you will support all messages
without having to hack the workflow system to understand it. Just store
the data, handle triggers based on it and send messages.

The last approach is a part of what I am doing and Zato will be a front
end for an external partner. Then they can just specify how they want to
talk with us (ND16xx here) and I can keep doing it my way on my side.
(Which is “REST”, tested by zato-apitest).

Thomas.

On 16/02/15 08:31, Thomas Lundquist wrote:

To me this defies the whole concept of SOA. At least my view on it. As I
see it the approach and meaning is to stay away from big and complex
monolithic systems and spread the functionality out to application
that’s simpler to handle (and replace for that matter.)

Then I could keep the Zato-ESB running as exactly that and if I
wanted, make a separate Zato-Workflow instance for that part. Without
having to wade through ESB specific stuff in the GUI or code.

Yes, indeed - anything that will be added will always have a good user
interface and naturally, will still be easy to use and manage.

We’re just considering automating things that are needed and implemented
by users anyway in a way that is reusable in multiple contexts and usage
scenarios.

Hi Thomas,

El 16/02/15 a les 08:31, Thomas Lundquist ha escrit:

Then I could keep the Zato-ESB running as exactly that and if I wanted,
make a separate Zato-Workflow instance for that part. Without having to
wade through ESB specific stuff in the GUI or code.

Why keeping it separated in terms of “projects” if you can have a single
one which is able to handle both jobs (together or separated)?

In my case, for example, I run a production environment with two
different zato platforms (several clusters each) with two clearly
different roles.

On one side, I run zato only as an ESB, with several integrations with
other systems and nearly no “intelligence” in it. It has a DB designed
to be as fast as possible, caches several things in order to speed up
communications and, in general, it is build to a robust platform able to
survive the failure of any of the other systems.

And, on the other side, I run zato purely as a backend server designed
in terms of Services and Workflows, running data mining and other
business processes on a scheduled and even driven basis, and which
communicates to the world only through the other one.

And, honestly, I can say that Zato performed extremely well in both
scenarios.

So, at the end, the role which the framework has (in this case zato) is
the one that the developers want to give it.
Then, why not working to have a platform which can handle all the
scenarios, and which can, if necessary, cover multiple roles at once?

Regards,
Carles

On 02/16/2015 11:27 AM, Coeuz wrote:

Hi Thomas,

El 16/02/15 a les 08:31, Thomas Lundquist ha escrit:

Then I could keep the Zato-ESB running as exactly that and if I wanted,
make a separate Zato-Workflow instance for that part. Without having to
wade through ESB specific stuff in the GUI or code.

Why keeping it separated in terms of “projects” if you can have a single
one which is able to handle both jobs (together or separated)?

You are answering this yourself…

In my case, for example, I run a production environment with two
different zato platforms (several clusters each) with two clearly
different roles.

…there.

And, honestly, I can say that Zato performed extremely well in both
scenarios.

Which is basically running Zato as an application framework with two
separate applications “on top” (or inside.)

I love reuseage, so this is prefectly OK with me.

Then, why not working to have a platform which can handle all the
scenarios, and which can, if necessary, cover multiple roles at once?

Because SOA and KISS.

The bigger something gets, the more complex it will end up being. And
the more you want to do out of the box the more complex it gets.

When I mentioned “DO NOT USE A CRM FOR THIS!” in the original answer
it’s because of too much experience with exactly that. The common
reasoning behind those choices (using a CRM) is “It’s basically there
already! Just add a little foo.”.

And from there it gets messy, always.

But it doesen’t mean the framework used to build the CRM cannot be used.

I’d be happy to see Zato branch out to Zato-Core, Zato-ESB,
Zato-Workflow but hope there will not be any Zato-ESB-Workflow-Appserver
all-in-one.

Thomas.