I’ve been looking at a bunch of ESB solutions to try to untangle our current processes. We’re mainly a MS SQL Server/SSIS/C# shop, but have a few Python systems in use which means we should get along quite well with a Python-based ESB.
I’ve read through the docs and set up a test Zato installation at home. I’m very impressed with its capabilities, but am wondering if it would be suitable for our situation.
We currently have rather large and unwieldy ETL processes built in SSIS. All of the orchestration is also handled via SSIS, as well as the ‘utility’ tasks like zipping and unzipping flat files, FTP uploads and downloads, etc.
There’s also a lot of polling. SSIS packages hammering SQL servers saying “what should I do next?”. All quite ugly
So, here’s what my ideal world would look like:
- Strip all the non-ETL stuff out of our SSIS packages
- Use Zato for the orchestration, with a service whose job it is to execute things in the right order
- ‘Utility’ services (uploader/downloaders, zipper/unzipper, etc)
- ‘Execute’ services (capable of launching the stripped-down SSIS packages).
Now, here’s the thing that’s tripping me up. We would only want certain services to execute on certain machines. So, our ‘utility’ services might all run on one or two physical machines.
Also, the ‘Execute’ services would need to be tied to a single physical machine.
The flow would be something like:
- Orchestrator service starts job 1234 for customer ABC
- Orchestrator calls Utility services (on any ‘utility’ physical machine) do what they need to do
- Orchestrator service checks metadata to see which physical machine customer ABC lives on
- Orchestrator service launches ‘Execute’ service on that physical machine.
So, to boil it down:
- Assume we have 40 physical machines (well, ok, they’re VMs )
- We want ‘utility’ services to be available on 5 of those, with work distributed between them.
- The other 35 may each run several ‘Execute’ services concurrently, but we need to be able to specify which machine’s Zato servers receive the request to do work.
Any and all advice greatly appreciated!