Is Zato suitable for my design?

Hello!

I’ve been looking at a bunch of ESB solutions to try to untangle our current processes. We’re mainly a MS SQL Server/SSIS/C# shop, but have a few Python systems in use which means we should get along quite well with a Python-based ESB.

I’ve read through the docs and set up a test Zato installation at home. I’m very impressed with its capabilities, but am wondering if it would be suitable for our situation.

We currently have rather large and unwieldy ETL processes built in SSIS. All of the orchestration is also handled via SSIS, as well as the ‘utility’ tasks like zipping and unzipping flat files, FTP uploads and downloads, etc.

There’s also a lot of polling. SSIS packages hammering SQL servers saying “what should I do next?”. All quite ugly :slight_smile:

So, here’s what my ideal world would look like:

  • Strip all the non-ETL stuff out of our SSIS packages
  • Use Zato for the orchestration, with a service whose job it is to execute things in the right order
  • ‘Utility’ services (uploader/downloaders, zipper/unzipper, etc)
  • ‘Execute’ services (capable of launching the stripped-down SSIS packages).

Now, here’s the thing that’s tripping me up. We would only want certain services to execute on certain machines. So, our ‘utility’ services might all run on one or two physical machines.

Also, the ‘Execute’ services would need to be tied to a single physical machine.

The flow would be something like:

  • Orchestrator service starts job 1234 for customer ABC
  • Orchestrator calls Utility services (on any ‘utility’ physical machine) do what they need to do
  • Orchestrator service checks metadata to see which physical machine customer ABC lives on
  • Orchestrator service launches ‘Execute’ service on that physical machine.

So, to boil it down:

  • Assume we have 40 physical machines (well, ok, they’re VMs :slight_smile: )
  • We want ‘utility’ services to be available on 5 of those, with work distributed between them.
  • The other 35 may each run several ‘Execute’ services concurrently, but we need to be able to specify which machine’s Zato servers receive the request to do work.

Any and all advice greatly appreciated!

Thanks,
Chris

Hi there,

yes, this can be done as long as the 5 specific VMs/systems can be somehow identified.

There is a feature originally implemented for use with a client who use a very complex networking configuration and a Zato cluster sitting in a mesh of DMZs and VPN connections.

Basically, the same thing was needed as yours - a way to have requests handled on specific servers only, e.g. ones belonging to a given VPN.

When you open your server.conf file you will note a stanza called invoke_target_patterns_allowed. The feature it deals with is so called ‘targets’. Each server is in fact a target of an invocation of a service, by default - it is an anonymous one which means that it will accept all requests.

But what you can do in your situation is that you can tell specific servers that they are to accept only specific targets, for instance:

[invoke_target_patterns_allowed]
order=true_false
utility=True
*=False

Which means - accept target ‘utility’, reject any other. Targets are simply labels, arbitrary strings.

There is a similar feature configured under ‘invoke_patterns_allowed’ - but here you specify which services can be invoked by their name.

Note that requests using these features are handled in background, asynchronously, and filtering out of messages that cannot be handled happens on receiver’s side (i.e. when you send a message to a target, all servers in the cluster receive it and 1 of the targets accepts it while the rest rejects it). This is how, for instance, ZeroMQ up until 3.x used to work.

All of these is documented here:

https://zato.io/docs/admin/guide/install-config/config-server.html#admin-guide-config-server-invoke-target-patterns-allowed

https://zato.io/docs/admin/guide/install-config/config-server.html#admin-guide-config-server-invoke-patterns-allowed

https://zato.io/docs/admin/guide/config-patterns.html

On the other hand, in current GitHub version we have means to synchronously invoke specific servers by their name or even more, to invoke specific processes within each server. But in Zato 2.0, this is not available yet.

These are rather uncommonly used features needed in specific situations so please let me know if you need any assistance.

Regards.

Thanks very much, dsuch!

That looks ideal. Being in the server config is great, since this is something ‘fixed’. We wouldn’t ever want a SQL server doing utility tasks or vice-versa.

As for requests being filtered by the servers themselves, that sounds fine. Our architecture suggests that our concurrent message volume will be pretty low.

I think I’ll set up 3.0 on a second test VM. I dare say that by the time we look to put something into production 3.0 may well be ready.

Just need to find some time to do a proof-of-concept. I’m really looking forward to untangling the horrendous ‘SSIS does everything!’ hole we’re in!

Cheers,
Chris