(Migrated) Development Environment

(This message has been automatically imported from the retired mailing list)

Happy New Year,

Are there any best practices around developing services for Zato in a test
driven way? What kind of project structure are you guys using for your Zato
services projects? Ideally our service code would live in a git repo with
branches for dev, staging and test and could be deployed to the various
environments from there and tested using some kind of CI, but I=B9m not quite
sure where to begin (unfortunately we=B9re pretty new to python, so this may
be self evident to all but us :slight_smile: )

Thank you,

Nathan Stults

That¹s great info, thank you.

Thank you,

Nathan Stults

On 1/5/15, 10:42 AM, “Samuel Rose” samuel.rose@gmail.com wrote:

Nathan, our project is using Vagrant/Ansible/VirtualBox for
development, and Jenkins for build/test.

So far, this set up has worked well.

Ansible is good for differentiation between dev/test/staging

On Mon, Jan 5, 2015 at 12:21 PM, Nathan Stults
nathan@sunfishtechnology.com wrote:

Happy New Year,

Are there any best practices around developing services for Zato in a
test
driven way? What kind of project structure are you guys using for your
Zato
services projects? Ideally our service code would live in a git repo
with
branches for dev, staging and test and could be deployed to the various
environments from there and tested using some kind of CI, but I¹m not
quite
sure where to begin (unfortunately we¹re pretty new to python, so this
may
be self evident to all but us :slight_smile: )

Thank you,

Nathan Stults

Thank you, I will explore that approach.

Nathan Stults

On 1/5/15, 10:17 AM, “Dariusz Suchojad” dsuch@zato.io wrote:

On 05/01/15 18:21, Nathan Stults wrote:

Are there any best practices around developing services for Zato in a
test driven way? What kind of project structure are you guys using for
your Zato services projects? Ideally our service code would live in a
git repo with branches for dev, staging and test and could be deployed
to the various environments from there and tested using some kind of CI,
but I=B9m not quite sure where to begin (unfortunately we=B9re pretty new to
python, so this may be self evident to all but us :slight_smile: )

I’m not sure about others but I always test integration services using
zato-apitest:

https://github.com/zatosource/zato-apitest

A Zato project is simply a directory of Python modules:

common.py
crm_billing_adapter.py
crm_account_adapter.py
crm_customer_adapter.py

(etc.)

Each of the modules contains one or more services.

On test and production, the services are deployed using Jenkins which
simply copies (cp) modules to a hot-deploy directory.

During the development, if I cannot work on a local server - for
instance if there is no VPN and a firewall to remote databases cannot be
opened due to internal policies - I deploy services remotely from
web-admin:

https://zato.io/docs/admin/guide/installing-services.html#hot-deploying-fr
om-the-web-admin

But if I can work on localhost and a client’s repo layout allows it, I
simply clone/check out code directly into a server’s pickup-dir. That
way, after setting server.conf’s misc.delete_after_pick_up to True, I
have deploy-on-save in IDE.

So my setup is pretty straightforward - people tend to use
Jenkins/Chef/Ansible to automate things or invoke API services for
deployment
https://zato.io/docs/public-api/details/zato.service.upload-package.html
in advanced cases but I haven’t actually done a lot on that front myself.

One thing with zato-apitest testing is that it can read variables to use
in test cases from environment (as in export MY_VAR=3D1). This is totally
cool and lets you, say, test whole environments in complete isolation
under Docker, before promoting code to the next environment.

–=20
Dariusz Suchojad

https://zato.io
ESB, SOA, REST, APIs and cloud integrations in Python

Nathan, our project is using Vagrant/Ansible/VirtualBox for
development, and Jenkins for build/test.

So far, this set up has worked well.

Ansible is good for differentiation between dev/test/staging

On Mon, Jan 5, 2015 at 12:21 PM, Nathan Stults
nathan@sunfishtechnology.com wrote:

Happy New Year,

Are there any best practices around developing services for Zato in a test
driven way? What kind of project structure are you guys using for your Zato
services projects? Ideally our service code would live in a git repo with
branches for dev, staging and test and could be deployed to the various
environments from there and tested using some kind of CI, but I’m not quite
sure where to begin (unfortunately we’re pretty new to python, so this may
be self evident to all but us :slight_smile: )

Thank you,

Nathan Stults

On 05/01/15 18:21, Nathan Stults wrote:

Are there any best practices around developing services for Zato in a
test driven way? What kind of project structure are you guys using for
your Zato services projects? Ideally our service code would live in a
git repo with branches for dev, staging and test and could be deployed
to the various environments from there and tested using some kind of CI,
but I’m not quite sure where to begin (unfortunately we’re pretty new to
python, so this may be self evident to all but us :slight_smile: )

I’m not sure about others but I always test integration services using
zato-apitest:

A Zato project is simply a directory of Python modules:

common.py
crm_billing_adapter.py
crm_account_adapter.py
crm_customer_adapter.py

(etc.)

Each of the modules contains one or more services.

On test and production, the services are deployed using Jenkins which
simply copies (cp) modules to a hot-deploy directory.

During the development, if I cannot work on a local server - for
instance if there is no VPN and a firewall to remote databases cannot be
opened due to internal policies - I deploy services remotely from
web-admin:

https://zato.io/docs/admin/guide/installing-services.html#hot-deploying-from-the-web-admin

But if I can work on localhost and a client’s repo layout allows it, I
simply clone/check out code directly into a server’s pickup-dir. That
way, after setting server.conf’s misc.delete_after_pick_up to True, I
have deploy-on-save in IDE.

So my setup is pretty straightforward - people tend to use
Jenkins/Chef/Ansible to automate things or invoke API services for
deployment
https://zato.io/docs/public-api/details/zato.service.upload-package.html
in advanced cases but I haven’t actually done a lot on that front myself.

One thing with zato-apitest testing is that it can read variables to use
in test cases from environment (as in export MY_VAR=1). This is totally
cool and lets you, say, test whole environments in complete isolation
under Docker, before promoting code to the next environment.

On 05/01/15 18:21, Nathan Stults wrote:

Are there any best practices around developing services for Zato in a
test driven way? What kind of project structure are you guys using for
your Zato services projects? Ideally our service code would live in a
git repo with branches for dev, staging and test and could be deployed
to the various environments from there and tested using some kind of CI,
but I’m not quite sure where to begin (unfortunately we’re pretty new to
python, so this may be self evident to all but us :slight_smile: )

I’m not sure about others but I always test integration services using
zato-apitest:

A Zato project is simply a directory of Python modules:

common.py
crm_billing_adapter.py
crm_account_adapter.py
crm_customer_adapter.py

(etc.)

Each of the modules contains one or more services.

On test and production, the services are deployed using Jenkins which
simply copies (cp) modules to a hot-deploy directory.

During the development, if I cannot work on a local server - for
instance if there is no VPN and a firewall to remote databases cannot be
opened due to internal policies - I deploy services remotely from
web-admin:

https://zato.io/docs/admin/guide/installing-services.html#hot-deploying-from-the-web-admin

But if I can work on localhost and a client’s repo layout allows it, I
simply clone/check out code directly into a server’s pickup-dir. That
way, after setting server.conf’s misc.delete_after_pick_up to True, I
have deploy-on-save in IDE.

So my setup is pretty straightforward - people tend to use
Jenkins/Chef/Ansible to automate things or invoke API services for
deployment
https://zato.io/docs/public-api/details/zato.service.upload-package.html
in advanced cases but I haven’t actually done a lot on that front myself.

One thing with zato-apitest testing is that it can read variables to use
in test cases from environment (as in export MY_VAR=1). This is totally
cool and lets you, say, test whole environments in complete isolation
under Docker, before promoting code to the next environment.

On 05/01/15 19:17, Dariusz Suchojad wrote:

One thing with zato-apitest testing is that it can read variables to use
in test cases from environment (as in export MY_VAR=1). This is totally
cool and lets you, say, test whole environments in complete isolation
under Docker, before promoting code to the next environment.

To be more clear - I find it cool because you can have a set of test
data common in 90% between dev/test/UAT/staging/other environments,
expressed in tests either directly or in INI config, but the remaining
10% - like credentials separate for each environment - can be read from
environment variables.

I came across that document last night and had the same idea, thanks. That
approach may also simplify developing Zato services via Vagrant. I saw
this line in the 2.0 changelog (we=B9ll be using 2.0 from the start)

  • Ability to hot-deploy code directly from a repository checkout (i.e.
    from GitHub)

Does this refer to the configuration scheme you just described, or
something else?

Thank you,

Nathan Stults

On 1/6/15, 7:14 AM, “Dariusz Suchojad” dsuch@zato.io wrote:

Nathan - here’s another idea.

You can set a selected server’s hot_deploy.pickup_dir to a directory
your services will be in.

https://zato.io/docs/admin/guide/install-config/config-server.html#hot-dep
loy-pickup-dir

And now, if that directory is under git (hg etc.) then on each pull, any
new/updated Python modules will be automatically hot-deployed into all
servers in a cluster.

That could automate installation - by deploying directly from a repo
clone.

–=20
Dariusz Suchojad

https://zato.io
ESB, SOA, REST, APIs and cloud integrations in Python

I had a follow up question:

It looked to me that Zato 1.1 has 4 documented methods to install or
hot deploy services:

https://zato.io/docs/admin/guide/installing-services.html

I could not see anything in the CLI with regard to deploy or install
services https://zato.io/docs/admin/cli/index.html

Lastly, this looked like a viable method:

https://zato.io/docs/public-api/details/zato.service.upload-package.html

So, I am exploring now the best way for our team to automate the hot
deployment of services after running ansible build, and configurations
that get zato up and running.

Advice and suggestions welcome/appreciated!

On Tue, Jan 6, 2015 at 10:14 AM, Dariusz Suchojad dsuch@zato.io wrote:

Nathan - here’s another idea.

You can set a selected server’s hot_deploy.pickup_dir to a directory
your services will be in.

https://zato.io/docs/admin/guide/install-config/config-server.html#hot-deploy-pickup-dir

And now, if that directory is under git (hg etc.) then on each pull, any
new/updated Python modules will be automatically hot-deployed into all
servers in a cluster.

That could automate installation - by deploying directly from a repo clone.


Dariusz Suchojad

https://zato.io
ESB, SOA, REST, APIs and cloud integrations in Python

Nathan, yes, theoretically even in Zato 1.1 when mediating deplyment
through Ansible, one could use ansible to pull in and place services
with GIT, then API or modification of service-sources.txt and restart
(all of which can be done by ansible)

(this is different than “Ability to hot-deploy code directly from a
repository checkout (i.e.
from GitHub)” as it is not direct from repo handled by Zato)

On Tue, Jan 6, 2015 at 11:46 AM, Nathan Stults
nathan@sunfishtechnology.com wrote:

I came across that document last night and had the same idea, thanks. Tha=
t
approach may also simplify developing Zato services via Vagrant. I saw
this line in the 2.0 changelog (we=C2=B9ll be using 2.0 from the start)

  • Ability to hot-deploy code directly from a repository checkout (i.e.
    from GitHub)

Does this refer to the configuration scheme you just described, or
something else?

Thank you,

Nathan Stults

On 1/6/15, 7:14 AM, “Dariusz Suchojad” dsuch@zato.io wrote:

Nathan - here’s another idea.

You can set a selected server’s hot_deploy.pickup_dir to a directory
your services will be in.

https://zato.io/docs/admin/guide/install-config/config-server.html#hot-de=
p

loy-pickup-dir

And now, if that directory is under git (hg etc.) then on each pull, any
new/updated Python modules will be automatically hot-deployed into all
servers in a cluster.

That could automate installation - by deploying directly from a repo
clone.


Dariusz Suchojad

https://zato.io
ESB, SOA, REST, APIs and cloud integrations in Python

On Tue, Jan 6, 2015 at 11:58 AM, Dariusz Suchojad dsuch@zato.io wrote:

On 06/01/15 17:37, Samuel Rose wrote:

So, I am exploring now the best way for our team to automate the hot
deployment of services after running ansible build, and configurations
that get zato up and running.

Advice and suggestions welcome/appreciated!

Could it be as simple as:

  • Check out code repo
  • $ cp /path/to/services/* /path/to/server/pickup-dir

Or would it not work in your situation?

I will have to test this out. It may very well be enough!

In production, is it generally a better practice to use
service-sources.txt to install services ? (I am driving toward an
"immutable infrastructure", meaning I could build and replace an
existing Zato install from scratch on-demand.)

I’ll test out automated command line vs service-sources.txt and report
back here.


Dariusz Suchojad

https://zato.io
ESB, SOA, REST, APIs and cloud integrations in Python

On Tue, Jan 6, 2015 at 12:22 PM, Dariusz Suchojad dsuch@zato.io wrote:

On 06/01/15 18:14, Samuel Rose wrote:

In production, is it generally a better practice to use
service-sources.txt to install services ? (I am driving toward an
"immutable infrastructure", meaning I could build and replace an
existing Zato install from scratch on-demand.)

This really depends on personal preferences.

I’m not familiar with Ansible but under Docker users tend to recreate
everything from scratch placing services under service-sources.txt
because that’s the methodology - restarts are not to be shunned. Hence
such users don’t hot-deploy at all.

Thanks Dariusz, this makes sense (and helps clarify the difference in
need of hot deply vs service-sources.txt).

Another school of thought is never to restart or at least very rarely.
Here users will only or predominately hot-deploy.

And there can be a mix of both putting common stuff in
service-sources.txt and services that can be often changed are hot-deployed.

I reckon there isn’t any objective absolute way to say which is better -
both can make sense to me depending on circumstances.

Again, makes sense.

But if you’re already leaning towards ‘immutable infrastructure’ then
why not give the former a go?


Dariusz Suchojad

https://zato.io
ESB, SOA, REST, APIs and cloud integrations in Python

On 06/01/15 17:46, Nathan Stults wrote:

  • Ability to hot-deploy code directly from a repository checkout (i.e.
    from GitHub)

Does this refer to the configuration scheme you just described, or
something else?

Yes, that’s the same thing.