Why they used two services in link 1
Those links shows you how to call a Zato service from a Flask
application - which I think is the opposite of what you want to do.
They have exposed two Zato services: one which allows you to invoke any
Zato service over HTTP (hence potentially a big security hole unless you
know what you’re doing), and one which allows you to invoke only a
single Zato service (in which case you need a separate channel mapping
and URL for each Zato service you want to expose)
For link 2, My environment is already ready my application is
running on 127.0.0.1:5000/hello http://127.0.0.1:5000/hello
How can I expose these app over http on zato platform.
You have already exposed your services as HTTP. Why do you want to
expose them again as HTTP from Zato?
This might make sense if you want Zato to do authentication or logging
of your service invocations, or use Zato’s load balancer as a TLS
(https) front-end.
Option 2: You could migrate away from Flask to Zato entirely; that would
mean repackaging your application as a Zato service. i.e. instead of
@get '/hello’
def hello():
… do stuff
it would become
class Hello(Service):
def handle(self):
… do stuff
If you want the flexibility to do both, then you could package up ‘do
stuff’ into a separate python module (library), and you can call it from
either place. But beware that (a) zato can’t hot-deploy libraries, only
services; and (b) your code will be able to use either the flask
request/response helpers or the zato request/response helpers, but not both.
You also need to consider how you want to go about testing your
application. Zato doesn’t provide any test framework other than actually
running a server and invoking it over HTTP; whereas a Flask application
is just plain WSGI and can be tested using the Werkzeug test Client. http://flask.pocoo.org/docs/0.10/testing/
This is because customer.get1 returns response as string whereas
customer.get2 returns it as a Python dictionary - please go through this
article step by step and everything will be explained:
For link 2, My environment is already ready my application is running
on 127.0.0.1:5000/hello http://127.0.0.1:5000/hello
How can I expose these app over http on zato platform.
Likewise, I suggest simply spending a few moments with the whole of the
tutorial step by step - it covers your scenario + a few things more:
You also need to consider how you want to go about testing your
application. Zato doesn’t provide any test framework other than actually
running a server and invoking it over HTTP; whereas a Flask application
is just plain WSGI and can be tested using the Werkzeug test Client. http://flask.pocoo.org/docs/0.10/testing/
Hi Brian, Nilesh,
the test framework to use is zato-apitest which works end-to-end through
the full stack and domain systems as opposed to unittest-like frameworks.
I really apologize for asking simple questions but still not
understood how ESB comes into picture for my use case.
You might not need one at all. It depends on your use case.
My REST services uses ORM model. so the package contains other
information like database configuration and models.
You mean the python code you have written, which runs under Flask,
contains things like database configuration and models?
Yes indeed it will. And you can just leave it there.
I wanted to keep things simple and implement zato service, schedule it
and execute it periodically.
My question are as follows:
Do I need to implement separate services according to zato in
order to schedule them periodically? if yes how can I handle the other
things like db connections, models.
If I understand what you’re trying to do, you want to trigger your
Flask-running REST service periodically from Zato.
In that case:
You make a Zato service which invokes your REST service over HTTP
You use the Zato scheduler to invoke the Zato service periodically
These is no need to handle “db connections, models” because all of those
are handled within your Flask application. Zato doesn’t need to know
about any of them. All it is doing is making an outbound HTTP call to http://127.0.0.1:5000/whatever.
However, it’s up to you to make sure your Flask application is up and
running. If it isn’t, then when zato tries to connect to http://127.0.0.1:5000/ it will fail, obviously.
If you want to trigger multiple Flask services at different times or
with different parameters, then yes, you can make a different Zato
service for each variation; or you can create a single Zato service and
use the “extra data” setting in the scheduler to pass different (text)
parameters to it.
I have created two servers and configured these on load-balancer.
they are running on 127.0.01.:17010 http://127.0.01.:17010 and
127.0.0.1:17011 http://127.0.0.1:17011. My rest services are running
on http://x.x.x.x:5000/. In order to achieve ESB do I need to
configure rest service server ip address to servers?
I don’t really know what you mean here.
In the out-of-the-box configuration, Zato runs haproxy on the front end
listening on one port (default 11223), and two instances of gunicorn
running on two other ports (default 17010 and 17011).
haproxy is a standard Linux load-balancer, and gunicorn is a standard
python WSGI web server. Neither of them is really part of zato as such,
they are separate projects.
However zato runs inside gunicorn and listens for HTTP requests, in the
same way as your Flask app runs. (*)
So what is it you want? Is it:
clients connect to zato on port 11223
clients make requests to zato services using zato-style XML or JSON
behind the scenes, the zato service makes requests of your Flask
service on port 5000
then what you need to do is to write a bunch of little Zato services,
each of which makes outgoing http requests to the appropriate endpoints
on your x.x.x.x:5000 REST service, passing along the appropriate request
parameters, consuming the response, and (if necessary) reformatting the
response to return to the original client.
This involves writing and deploying a small bit of Python code for each
Zato service.
Does that make sense?
Does that make zato look rather like a proxy? Yes, in essence that is
what is is. You can write zato services which themselves directly open
database connections and implement business logic (that’s what I meant
by “converting your Flask app into a Zato service”); but if there is an
existing REST service running somewhere, then normally zato would just
make requests of that service over HTTP.
In doing this, of course now you have two completely separate web
servers, both of which have to be managed and scaled: zato, and your
Flask app. If your Flask app becomes very busy then you may have to
scale it horizontally, by adding more worker processes and possibly a
load-balancer in front of it. zato won’t do that for you (unless you
write your app directly as services within zato, in which case it runs
within the zato gunicorn workers and behind zato’s load balancer)
HTH,
Brian.
(*) Aside: seeing as it’s port 5000, I think you are probably using the
’development mode’ web server, but you could run your Flask app using
gunicorn if you wanted:
Hi Team,
Thank you Brian and Dariusz and I really appreciate your help.
Brian, I got your first option. I have created the outgoing http connection
and checked the service.
I really apologize for asking simple questions but still not understood how
ESB comes into picture for my use case.
My REST services uses ORM model. so the package contains other information
like database configuration and models.
I wanted to keep things simple and implement zato service, schedule it and
execute it periodically.
My question are as follows:
Do I need to implement separate services according to zato in order to
schedule them periodically? if yes how can I handle the other things like
db connections, models.
I have created two servers and configured these on load-balancer. they
are running on 127.0.01.:17010 and 127.0.0.1:17011. My rest services are
running on http://x.x.x.x:5000/. In order to achieve ESB do I need to
configure rest service server ip address to servers?
Thank you in Advance!
On Fri, Jan 8, 2016 at 8:18 PM, Dariusz Suchojad dsuch@zato.io wrote:
On 08/01/16 15:07, Brian Candler wrote:
You also need to consider how you want to go about testing your
application. Zato doesn’t provide any test framework other than actually
running a server and invoking it over HTTP; whereas a Flask application
is just plain WSGI and can be tested using the Werkzeug test Client. http://flask.pocoo.org/docs/0.10/testing/
Hi Brian, Nilesh,
the test framework to use is zato-apitest which works end-to-end through
the full stack and domain systems as opposed to unittest-like frameworks.
1. You make a Zato service which invokes your REST service over HTTP
How can I create service that invokes my REST service over HTTP?
I have gone through your documents but haven’t got any link for same.
Suppose my service is running as http://x.x.x.x:5000/helloworld. How
can I invoke this from zato service. For invoking other services
httplib2 is used, can we use same here or there is other way in zato?
You certainly can use httplib2 directly if you wish, but zato also
provides a wrapper for the ‘requests’ library to do this.
The example there does almost all you need, except that code the
handling of the HTTP response is broken: it just assigns it to a local
variable, not the zato response.
Now, I looked very hard, and I could not find anywhere in the zato
documentation which says the object type which is returned by
"conn.post". But by experimentation, it seems to be the native response
object from the ‘requests’ library. http://docs.python-requests.org/en/latest/user/quickstart/#response-content
But you’ll need to work out the exact details for yourself. Also, this
is the point at which if you want to massage the response into some
different format, you could do it here (let’s say convert XML to JSON or
vice versa)
(i.e. create the channel which is called “SetBillingInfo” in the example
code)
However you can ignore the zato channels entirely and just use a library
like httplib2 as you say; in that case, your service will have the
parameter http://x.x.x.x:5000/ hard-coded into it.
My service is not executed at scheduled time(detail snapshots are
attached). How can I handle this in handle?
(1) I don’t think a service can possibly be called “TestService”. If you
uploaded the service in file testservice.py containing class TestService
then it should be called something like testservice.TestService
This may be why the scheduler isn’t working (but then I didn’t think you
could create a scheduler entry which pointed to a non-existent service?)
What does the page which lists all services on your cluster show?
(2) Just to be clear, an “outgoing connection” is not a “service”
A “service” is real python code you write and upload.
In some ways, an outgoing connection is “service-like”: it is a thing
that can be called. Maybe Zato could have made outgoing connections be
the same as services, but it didn’t. Rather, an outgoing connection is
something that a service can look up, load and use.
(3) Your service looks up and loads the outgoing connection:
conn = self.outgoing.plain_http[...].conn
but it doesn’t actually use it. You need to do conn.post(…) or
conn.get(…) to make a HTTP call over that connection.
“conn” is just an object from the “requests” library, so you use it in
the same way as that.
Does zato exposes api’s to schedule job pragmatically?
You mean programatically?
Yes I think so, but it’s all hidden in the code. That is: I believe the
zato front-end in Django uses zato internal APIs to make its changes.
But as they’re not documented, you’ll need to work out from the code
what those API endpoints are.
(The internal services have names starting “zato.” but are hidden in the UI)
If you have special needs for dynamically changing the scheduling of a
job, not using the zato GUI, then maybe this is something you should
write yourself using one of the python scheduler libraries.
My services are exposed to user over HTTP, one of service takes
querystring as input from user. Now if I create a service and deploy
this service, Which IP address should I expose to user. I mean which
service is exposed to user, a zato service or flask service?
The end-user will contact the Zato IP address, and the Zato service will
contact the backend IP address.
This is assuming that you actually want to use Zato.
Having end-users contact Flask directly is also OK. Where Zato adds
value is when you have many different backend services on different URLs
used by many different clients; then you can point all the clients at
Zato, instead of having a spaghetti of many different clients all
pointing at different backends. Then if you want to change one
particular service, you can change it at one point rather than having to
go find and reconfigure all the clients which use that service. That’s
the idea behind an ESB anyway.
For some arguments why you might not want to use an ESB, see:
Also, if you are not going to do any data transformation, but just pass
(say) raw JSON requests and responses straight through, it’s harder to
see the value in an ESB.
I used inbuilt logger here to debug but not much powerful as pdb.
Can you suggest any other tool to debug the services?
Debugging Flask applications is outside of the scope of this list.
Debugging Zato services is tricky, as there’s no way provided to invoke
a service in a context other than the running zato server.
It would be nice if Zato had some user-callable scaffolding which will
set up the environment for a zato service and let you invoke it via the
zato stack.
What I do is keep make the actual service.py files as simple as
possible, and push as much logic as possible outside, in separate
libraries with their own test suites. The service itself is basically
just a wrapper which calls the library. If something fails, in the
service code, at least you get a backtrace in the zato server logs.
But what I’m doing is running fully-fledged services inside Zato.
This, in retrospect, was probably a mistake; I should have written and
deployed a standalone application like you are doing, under something
like Flask, and only put Zato in front if it was needed.
I started this using Zato in the expectation that it would free me from
doing things like SOAP encoding/decoding, but actually it’s not flexible
enough for my needs (*) so I’ve had to bypass that layer anyway.
I really apologize for asking simple questions but still not understood
how ESB comes into picture for my use case.
You might not need one at all. It depends on your use case.
My REST services uses ORM model. so the package contains other information
like database configuration and models.
You mean the python code you have written, which runs under Flask,
contains things like database configuration and models?
Yes indeed it will. And you can just leave it there.
I wanted to keep things simple and implement zato service, schedule it and
execute it periodically.
My question are as follows:
Do I need to implement separate services according to zato in order to
schedule them periodically? if yes how can I handle the other things like
db connections, models.
If I understand what you’re trying to do, you want to trigger your
Flask-running REST service periodically from Zato.
In that case:
You make a Zato service which invokes your REST service over HTTP
How can I create service that invokes my REST service over HTTP?
I have gone through your documents but haven’t got any link for same.
Suppose my service is running as http://x.x.x.x:5000/helloworld. How can I
invoke this from zato service. For invoking other services httplib2 is
used, can we use same here or there is other way in zato?
You use the Zato scheduler to invoke the Zato service periodically
Pretty clear, once I have service, I can schedule it periodically using
zato scheduler.
These is no need to handle “db connections, models” because all of those
are handled within your Flask application. Zato doesn’t need to know about
any of them. All it is doing is making an outbound HTTP call to http://127.0.0.1:5000/whatever.
However, it’s up to you to make sure your Flask application is up and
running. If it isn’t, then when zato tries to connect to http://127.0.0.1:5000/ it will fail, obviously.
If you want to trigger multiple Flask services at different times or with
different parameters, then yes, you can make a different Zato service for
each variation; or you can create a single Zato service and use the “extra
data” setting in the scheduler to pass different (text) parameters to it.
I have created two servers and configured these on load-balancer. they
are running on 127.0.01.:17010 and 127.0.0.1:17011. My rest services are
running on http://x.x.x.x:5000/. In order to achieve ESB do I need to
configure rest service server ip address to servers?
I don’t really know what you mean here.
In the out-of-the-box configuration, Zato runs haproxy on the front end
listening on one port (default 11223), and two instances of gunicorn
running on two other ports (default 17010 and 17011).
haproxy is a standard Linux load-balancer, and gunicorn is a standard
python WSGI web server. Neither of them is really part of zato as such,
they are separate projects.
However zato runs inside gunicorn and listens for HTTP requests, in the
same way as your Flask app runs. (*)
So what is it you want? Is it:
clients connect to zato on port 11223
clients make requests to zato services using zato-style XML or JSON
behind the scenes, the zato service makes requests of your Flask service
on port 5000
then what you need to do is to write a bunch of little Zato services, each
of which makes outgoing http requests to the appropriate endpoints on your
x.x.x.x:5000 REST service, passing along the appropriate request
parameters, consuming the response, and (if necessary) reformatting the
response to return to the original client.
This involves writing and deploying a small bit of Python code for each
Zato service.
Does that make sense?
Does that make zato look rather like a proxy? Yes, in essence that is what
is is. You can write zato services which themselves directly open
database connections and implement business logic (that’s what I meant by
"converting your Flask app into a Zato service"); but if there is an
existing REST service running somewhere, then normally zato would just make
requests of that service over HTTP.
In doing this, of course now you have two completely separate web servers,
both of which have to be managed and scaled: zato, and your Flask app. If
your Flask app becomes very busy then you may have to scale it
horizontally, by adding more worker processes and possibly a load-balancer
in front of it. zato won’t do that for you (unless you write your app
directly as services within zato, in which case it runs within the zato
gunicorn workers and behind zato’s load balancer)
HTH,
Brian.
(*) Aside: seeing as it’s port 5000, I think you are probably using the
’development mode’ web server, but you could run your Flask app using
gunicorn if you wanted:
Hi Brian,
Thanks Brain and sorry for poor english.
I have followed the above steps and created a TestService and deployed it
on zato platform. Also created outgoing http connection and scheduled the
service.
My questions are as follows:
My service is not executed at scheduled time(detail snapshots are
attached). How can I handle this in handle?
Does zato exposes api’s to schedule job pragmatically?
My services are exposed to user over HTTP, one of service takes
querystring as input from user. Now if I create a service and deploy this
service, Which IP address should I expose to user. I mean which service is
exposed to user, a zato service or flask service?
I used inbuilt logger here to debug but not much powerful as pdb. Can
you suggest any other tool to debug the services?
You make a Zato service which invokes your REST service over HTTP
How can I create service that invokes my REST service over HTTP?
I have gone through your documents but haven’t got any link for same.
Suppose my service is running as http://x.x.x.x:5000/helloworld http://x.x.x.x:5000/helloworld. How can I invoke this from zato service.
For invoking other services httplib2 is used, can we use same here or there
is other way in zato?
You certainly can use httplib2 directly if you wish, but zato also
provides a wrapper for the ‘requests’ library to do this.
The example there does almost all you need, except that code the handling
of the HTTP response is broken: it just assigns it to a local variable, not
the zato response.
Now, I looked very hard, and I could not find anywhere in the zato
documentation which says the object type which is returned by “conn.post”.
But by experimentation, it seems to be the native response object from the
’requests’ library. http://docs.python-requests.org/en/latest/user/quickstart/#response-content
But you’ll need to work out the exact details for yourself. Also, this is
the point at which if you want to massage the response into some different
format, you could do it here (let’s say convert XML to JSON or vice versa)
(i.e. create the channel which is called “SetBillingInfo” in the example
code)
However you can ignore the zato channels entirely and just use a library
like httplib2 as you say; in that case, your service will have the
parameter http://x.x.x.x:5000/ hard-coded into it.
Nilesh - please send the full request, I really mean the whole of it
without describing in words what it does. Just please issue a curl call,
in verbose mode, and send it all here as text, not as a screenshot of text.
Ok, thanks for the curl output - let’s please agree that you won’t send
screenshots of HTTP transactions from now on, only full verbose output
from curl. Thanks.
As for the line above - it says:
Find a plain HTTP outgoing connection called ‘featureurl’
Assign this connection to an object called ‘featureurl’ too
From this line …
INFO - 8989:Dummy-1330 - getattrs.get-attrs:22 - featureurl=None
… it follows that no such connection exists in your environment hence
None is logged, which is a correct and expected behaviour.
Please let me know if you have completed the tutorial yet. If not,
please do it - you are jumping the gun and trying to achieve too much
without learning the basics.
The tutorial will cover a lot of information that you need, please don’t
attempt to implement your own services until you will have thoroughly
understood what the tutorial offers:
Also created respective HTTP plain channels and Outgoing plain HTTP
according to Zato documentation.
When I ping my Plain HTTP outgoing connection. it gives 200 status.
I tried to access same using zato’s ip address. It calls my service. here I
wanted to get the parameters which I have passed as querystring. But I
found that request payload is empty.
The output in server log is:
2016-01-13 19:10:47,108 - INFO - 8990:Dummy-317 - getattrs.get-attrs:22 -
Request:
2016-01-13 19:10:47,109 - INFO - 8990:Dummy-317 - getattrs.get-attrs:22 -
featureurl=None
My service is not executed at scheduled time(detail snapshots are
attached). How can I handle this in handle?
(1) I don’t think a service can possibly be called “TestService”. If you
uploaded the service in file testservice.py containing class TestService
then it should be called something like testservice.TestService
This may be why the scheduler isn’t working (but then I didn’t think you
could create a scheduler entry which pointed to a non-existent service?)
What does the page which lists all services on your cluster show?
(2) Just to be clear, an “outgoing connection” is not a “service”
A “service” is real python code you write and upload.
In some ways, an outgoing connection is “service-like”: it is a thing that
can be called. Maybe Zato could have made outgoing connections be the same
as services, but it didn’t. Rather, an outgoing connection is something
that a service can look up, load and use.
(3) Your service looks up and loads the outgoing connection:
conn = self.outgoing.plain_http[...].conn
but it doesn’t actually use it. You need to do conn.post(…) or
conn.get(…) to make a HTTP call over that connection.
“conn” is just an object from the “requests” library, so you use it in the
same way as that.
Does zato exposes api’s to schedule job pragmatically?
You mean programatically?
Yes I think so, but it’s all hidden in the code. That is: I believe the
zato front-end in Django uses zato internal APIs to make its changes. But
as they’re not documented, you’ll need to work out from the code what those
API endpoints are.
(The internal services have names starting “zato.” but are hidden in the
UI)
If you have special needs for dynamically changing the scheduling of a
job, not using the zato GUI, then maybe this is something you should write
yourself using one of the python scheduler libraries.
My services are exposed to user over HTTP, one of service takes
querystring as input from user. Now if I create a service and deploy this
service, Which IP address should I expose to user. I mean which service is
exposed to user, a zato service or flask service?
The end-user will contact the Zato IP address, and the Zato service will
contact the backend IP address.
This is assuming that you actually want to use Zato.
Having end-users contact Flask directly is also OK. Where Zato adds value
is when you have many different backend services on different URLs used by
many different clients; then you can point all the clients at Zato, instead
of having a spaghetti of many different clients all pointing at different
backends. Then if you want to change one particular service, you can change
it at one point rather than having to go find and reconfigure all the
clients which use that service. That’s the idea behind an ESB anyway.
For some arguments why you might not want to use an ESB, see:
Also, if you are not going to do any data transformation, but just pass
(say) raw JSON requests and responses straight through, it’s harder to see
the value in an ESB.
I used inbuilt logger here to debug but not much powerful as pdb. Can
you suggest any other tool to debug the services?
Debugging Flask applications is outside of the scope of this list.
Debugging Zato services is tricky, as there’s no way provided to invoke a
service in a context other than the running zato server.
It would be nice if Zato had some user-callable scaffolding which will set
up the environment for a zato service and let you invoke it via the zato
stack.
What I do is keep make the actual service.py files as simple as possible,
and push as much logic as possible outside, in separate libraries with
their own test suites. The service itself is basically just a wrapper which
calls the library. If something fails, in the service code, at least you
get a backtrace in the zato server logs.
But what I’m doing is running fully-fledged services inside Zato. This,
in retrospect, was probably a mistake; I should have written and deployed a
standalone application like you are doing, under something like Flask, and
only put Zato in front if it was needed.
I started this using Zato in the expectation that it would free me from
doing things like SOAP encoding/decoding, but actually it’s not flexible
enough for my needs (*) so I’ve had to bypass that layer anyway.
It calls my service. I am catching the querystring which I passed in
request in my service.
Here is my service implementation:
class GetAttrs(Service):
def handle(self): self.logger.info(‘Request: {}’.format(self.request.payload))
featureurl = self.outgoing.plain_http.get(‘featureurl’) #featureurl = self.request.payload[‘featureurl’] self.logger.info(“featureurl={}”.format(featureurl))
conn = self.outgoing.plain_http[‘GetAttrs’].conn
the output in log file is as follows:
2016-01-13 19:56:36,102 - INFO - 8989:Dummy-1330 - getattrs.get-attrs:22 -
Request:
2016-01-13 19:56:36,102 - INFO - 8989:Dummy-1330 - getattrs.get-attrs:22 -
featureurl=None
Server Zato is not blacklisted
< Server: Zato
< Date: Wed, 13 Jan 2016 14:48:44 GMT
< Connection: close
< Transfer-Encoding: chunked
< Content-Type: application/json
< X-Zato-CID: K07B6H2540SEYBN237HPAX4GTZM6
<
Closing connection 0
Output in log file is
2016-01-13 20:18:44,841 - INFO - 8989:Dummy-1524 - getattrs.get-attrs:22 -
Request:
2016-01-13 20:18:44,842 - INFO - 8989:Dummy-1524 - getattrs.get-attrs:22 -
featureurl=None
On Wed, Jan 13, 2016 at 8:14 PM, Dariusz Suchojad dsuch@zato.io wrote:
On 13/01/16 15:42, Nilesh Sutar wrote:
I am issuing as follows:
curl -H “Accept:application/json”
Nilesh - please send the full request, I really mean the whole of it
without describing in words what it does. Just please issue a curl call,
in verbose mode, and send it all here as text, not as a screenshot of text.
thanks,
–
Dariusz Suchojad
https://zato.io
ESB, SOA, REST, APIs and Cloud Integrations in Python
