(Migrated) Network addresses

Community support
#1

(This message has been automatically imported from the retired mailing list)

Hello,

I’d like to ask you for feedback regarding a certain feature.

Background

We use Redis right now for server-server communication. For instance,
when you upload a service it is eventually routed through Redis to other
servers using Redis publish/subscribe.

But now that Redis will not be a core dependency a mechanism is needed
to deliver such information between servers directly, i.e. through
synchronous TCP calls (REST).

I prefer it over things like Zyre [1] (which is fine in itself) because
this will allow us to achieve two additional desirable goals:

  • When a service is hot-deployed there will be immediate feedback in
    web-admin if it succeeded or not so things like syntax errors will be
    caught immediately

  • We need to have something like below anyway:

response = self.servers[‘ServerName’].invoke(‘service’, request)

This is needed because in some scenarios it is important to invoke a
service on a selected server and none other - for instance, network
segmentation may dictate that only one server belongs to a VPN with an
external system Zato integrates with.

At the same time it is convenient to be able to simply specify in
server.conf …

[main]
gunicorn_bind=0.0.0.0:17010

… and be done with it.

Question

This however means that other servers, possibly in different operating
systems, do not know what a given server bound to, what its addresses are.

Hence there will be an idea of a preferred IP introduced for other
servers to call each other over.

(This is its default form in server.conf)

[bind_options]
preferred_server_to_server_ip=10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16,
eth0
bind_if_preferred_not_found=True
preferred_protocol=ipv4
allow_localhost_only=False

Now each server when it’s booting will store in ODB information what its
preferred IP address is so that other servers can always find it.

Thus everything should stay as convenient as today - one will be able to
use 0.0.0.0 and at the same time servers will find a way to announce
what their preferred IP is.

I just want to confirm if you think there are some use-cases not covered
by the logic below. The ones listed here will be supported just fine:

  • Run Zato in internal networks with fixed private IPs only

  • Run Zato in networks with unknown private IPs yet within a well-known
    known network range

  • Run Zato in networks with unknown IPs, private or public, but using
    well known network interfaces

  • Run Zato with public IPs only

  • If enabled, run Zato on localhost only

Implementation

The logic will be as follows:

  • Get all interfaces in the OS

  • If any is listed in preferred_server_to_server_ip, read its first IP
    address and use as the preferred one

  • If no network interface is explicitly listed in [bind_options] but
    network ranges or concrete addresses are, get IPs that 0.0.0.0 resolved
    to and use the first that matches what is in preferred_server_to_server_ip

  • If 0.0.0.0 does not resolve to anything in
    preferred_server_to_server_ip, arbitrarily pick first non-loopback
    interface and its first IP

  • If 0.0.0.0 turns out to point to localhost only then do not proceed
    unless allow_localhost_only is True because it may be the case that
    several servers in several systems all bind to their respective loopback
    interfaces so there is really no communication here possible

Conclusion

Do you think this will cover your environments? I realize that there are
many ways to configure networking so it is important that everything
available today can continue to work in 3.0.

[1] https://github.com/zeromq/zyre

regards,

#2

Hi Dariusz

I like the features that it brings like instant hotdeploy feedback for
errors and others benefits that you listed but I like the idea of having a
messaging integrator or orchestrator for some scenarios like the hotdeploy
of the services itself because without it, Zato Servers will end up calling
between each other and having to deal with all the stuffs that you listed
related to the Servers IP address that could lead to errors with things
like AWS EC2 Elastic IP or Round Robin DNS even if I couldn’t confirm that
it will be an issue. I just pointing that it will be more easily to
develop, maintain and configure if servers connect to a messaging protocol
to be notified of some events even when they could communicate directly
with each other for example to control where the service is executed and
not need to deploy a separate cluster just to control that.
Maybe integrating an event-driven orchestration solution like saltstack
https://saltstack.com/ that have a master and minions and it’s based on
ZeroMQ or ZeroMQ itself to collect the results from the executions in
scenarios that involve the deploy of services.
Also it will be great to have something like failover recovery for services
not deployed in servers that where down during the hotdeploy or new servers
that newly join to the cluster

Just my two cents.
Cheers

On Wed, Jun 15, 2016 at 2:21 PM, Dariusz Suchojad dsuch@zato.io wrote:

Hello,

I’d like to ask you for feedback regarding a certain feature.

Background

We use Redis right now for server-server communication. For instance,
when you upload a service it is eventually routed through Redis to other
servers using Redis publish/subscribe.

But now that Redis will not be a core dependency a mechanism is needed
to deliver such information between servers directly, i.e. through
synchronous TCP calls (REST).

I prefer it over things like Zyre [1] (which is fine in itself) because
this will allow us to achieve two additional desirable goals:

  • When a service is hot-deployed there will be immediate feedback in
    web-admin if it succeeded or not so things like syntax errors will be
    caught immediately

  • We need to have something like below anyway:

response = self.servers[‘ServerName’].invoke(‘service’, request)

This is needed because in some scenarios it is important to invoke a
service on a selected server and none other - for instance, network
segmentation may dictate that only one server belongs to a VPN with an
external system Zato integrates with.

At the same time it is convenient to be able to simply specify in
server.conf …

[main]
gunicorn_bind=0.0.0.0:17010

… and be done with it.

Question

This however means that other servers, possibly in different operating
systems, do not know what a given server bound to, what its addresses are.

Hence there will be an idea of a preferred IP introduced for other
servers to call each other over.

(This is its default form in server.conf)

[bind_options]
preferred_server_to_server_ip=10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16,
eth0
bind_if_preferred_not_found=True
preferred_protocol=ipv4
allow_localhost_only=False

Now each server when it’s booting will store in ODB information what its
preferred IP address is so that other servers can always find it.

Thus everything should stay as convenient as today - one will be able to
use 0.0.0.0 and at the same time servers will find a way to announce
what their preferred IP is.

I just want to confirm if you think there are some use-cases not covered
by the logic below. The ones listed here will be supported just fine:

  • Run Zato in internal networks with fixed private IPs only

  • Run Zato in networks with unknown private IPs yet within a well-known
    known network range

  • Run Zato in networks with unknown IPs, private or public, but using
    well known network interfaces

  • Run Zato with public IPs only

  • If enabled, run Zato on localhost only

Implementation

The logic will be as follows:

  • Get all interfaces in the OS

  • If any is listed in preferred_server_to_server_ip, read its first IP
    address and use as the preferred one

  • If no network interface is explicitly listed in [bind_options] but
    network ranges or concrete addresses are, get IPs that 0.0.0.0 resolved
    to and use the first that matches what is in preferred_server_to_server_ip

  • If 0.0.0.0 does not resolve to anything in
    preferred_server_to_server_ip, arbitrarily pick first non-loopback
    interface and its first IP

  • If 0.0.0.0 turns out to point to localhost only then do not proceed
    unless allow_localhost_only is True because it may be the case that
    several servers in several systems all bind to their respective loopback
    interfaces so there is really no communication here possible

Conclusion

Do you think this will cover your environments? I realize that there are
many ways to configure networking so it is important that everything
available today can continue to work in 3.0.

[1] https://github.com/zeromq/zyre

regards,

–
Dariusz Suchojad

https://zato.io
ESB, SOA, REST, APIs and Cloud Integrations in Python

#3

On 16/06/16 00:31, Axel Mendoza Pupo wrote:

Hi Axel,

[…]

I just pointing that it will be more easily to
develop, maintain and configure if servers connect to a messaging protocol
to be notified of some events even when they could communicate directly
with each other for example to control where the service is executed and
not need to deploy a separate cluster just to control that.
Maybe integrating an event-driven orchestration solution like saltstack
https://saltstack.com/ that have a master and minions and it’s based on
ZeroMQ or ZeroMQ itself to collect the results from the executions in
scenarios that involve the deploy of services.

Right - while I can say that there definitely won’t be a strong
dependency on SaltStack or anything external to Zato, I understand your
point though I’m not clear what you mean by issues with Elastic IP
addresses?

Surely they do not change themselves while systems are running and even
if they do because someone explicitly assigned a new IP to an interface,
it is understood that servers (Zato or not) using it in a given instance
need to be restarted?

This whole effort is meant to remove the dependency on Redis so I would
not like to replace it immediately with something else though at the
same time integration with configuration management or service discovery
software such as SaltStack or Consul as such absolutely makes sense.

This kind of worries of what sort of IPs change when is precisely
something I do not want to think about so now that I think of it, given
that we already have REST publish/subscribe …

https://zato.io/docs/pubsub/index.html

… then after all, why not take advantage of it and have servers use it
internally for deployment.

Synchronous calls between servers would be needed anyway but for
deployment our own pub/sub could be employed. In this fashion, servers
would only need connections to SQL ODB which is something that they need
anyway and that won’t change.

I’m pretty sure this would lead to a new feature of user-defined
services subscribing to a pub/sub topic.

Also it will be great to have something like failover recovery for services
not deployed in servers that where down during the hotdeploy or new servers
that newly join to the cluster

Ok, I’m not sure what 2.0.x version you are on but we have had it since
2.0.6 and this ticket:

When a server starts up it always asks other servers already running in
a given cluster for their list of services and if anything is missing,
it is deployed in the starting one.

regards,

#4

On 16/06/16 03:24, Axel Mendoza Pupo wrote:

If I look Zato servers closely they don’t even need an SQL ODB, with just a
good sync queue message protocol it’s enough because to be faster it have
caches of pretty much everything the servers need to build the response and
when a change it’s made Zato propagates the change in the cluster, or not?
I could sounds crazy but a master/broker is enough to maintain the state of
the cluster

This is all true but the thing is that an SQL database is the only
component that everyone is happy to accept as a core requirement.

Creating one’s own queue manager is nothing unheard of and could be done
but then we would be introducing a completely new piece that would still
clustering and many other things a proper database needs. That is an
interesting exercise but in the context of replacing Redis we would add
something entirely else that would still need internal HA and
communication so this is the same sort of questions moved elsewhere,
from servers to the queue manager.

The thing really is that at the end of the day an SQL database is the
only piece that everyone has and understands how to operate. Everything
else is optional.

#5

On 16/06/16 15:29, Samuel Rose wrote:

So, just to make sure I understand, the new outcome would be to use
https://zato.io/docs/pubsub/index.html but replacing redis with SQL
database as the store of information, yes?

Then, each server will PubSub, with data stored via ODB for IP
address, services deployed, etc

Right now this would be only for hot deployment and reconfiguration - in
2.0 we already use pub/sub with Redis to achieve it so only the backend
of the feature would change in 3.0 and it still would be hidden away
from users or at least not exposed conspicuously, just like today it isn’t.

Essentially, outwardly everything would look the same and only
internally an SQL database would be used for internal pub/sub instead of
Redis.

#6

On 16/06/16 16:16, Samuel Rose wrote:

it’s nice to have an option to use Redis easily, but I can see that it
is also very nice to not make Redis a requirement of the system.
Reducing complexity is a really good reason to make this change.

I understand it would only be internal. I think this change would be
very beneficial while at least from perspective would be low-ish
impact on existing zato systems

To be precise, what now uses Redis pub/sub would look as follows:

  • If there is one server, nothing is needed

  • If 2+ servers, but all in the same operating system, use ZeroMQ’s IPC
    sockets without SQL

  • If 2+ servers and least 2 systems then use SQL-backed pub/sub

That all would be internal details.

What possibly will need adjustment in 3.0 are preferred IP addresses but
then again, I think we have it covered with the bind options from my
original email so amendments by users should be really rare since by
default we will be able to automatically pick either an internal or
external IP to call servers by.

In addition to self.servers[‘ServerName’].invoke(‘service’, request) the
ability to invoke servers directly will be needed because in 3.0 the
scheduler will be its own component, i.e. with its ‘zato start
/path/to/scheduler’ command.

This is because I would like for the scheduler to invoke servers
directly, synchronously, so that any exceptions can be caught immediately.

#7

Axel - what are some specific scenarios regarding IP addresses that you
were thinking of?

Seeing as this server-server connectivity will be needed, if only so
that schedulers can invoke servers, I would like to make sure that
everything is covered.

thanks,

#8

Dariusz, I think that the combination of your needs and my ideas and concerns has a name. A consensus based system or algorithm like raft
https://raft.github.io/
http://thesecretlivesofdata.com/raft/

Scheduler fires same service twice
#10

Yes, consensus algorithms are cool and it looks we will need to employ one anyway since the deeper I look at into it the more I see we will need a custom broker in place of Redis but unlike Redis this broker will definitely not store any data, it all will be in SQL, and the broker - likely based on WebSockets - will be a mere dispatcher of requests across servers.

#11

I think that this could be an starting point to accomplish the task


It has the tools for build up a raft system based on ZMQ and be able to embed it on another applications

#13

Right @aekroft - this would be something along those lines but I would prefer WebSockets instead of ZeroMQ here because we need WebSockets anyway and this is a good opportunity to introduce the protocol in the platform.

#14

given
that we already have REST publish/subscribe …

https://zato.io/docs/pubsub/index.html1

… then after all, why not take advantage of it and have servers use it
internally for deployment.

Synchronous calls between servers would be needed anyway but for
deployment our own pub/sub could be employed. In this fashion, servers
would only need connections to SQL ODB which is something that they need
anyway and that won’t change.

I’m pretty sure this would lead to a new feature of user-defined
services subscribing to a pub/sub topic.

I am late to the discussion, but the other nice thing about the pub/sub approach is that user defined service could be made to pub/sub and update optional external service discovery mechanisms (like Consul) via their API. So, as new services are rolled out, via one mechanism zato can be updated internally, and update external systems too.