(Migrated) pip install zato-common woes

(This message has been automatically imported from the retired mailing list)

I noticed several problems when trying to set up a virtual environment
with just zato-common and zato-client. Where can I report these as
issues? And where I do I find the source from which the zato-common and
zato-client packages on PyPI where built so I can make PRs?

When trying to install zato-common via pip, it fails, because bzr=3D=3D2.=
2.3
is not available on PyPI anymore.

This means you have to use the following command:

pip install --allow-external bzr --allow-unverified bzr zato-common
  • Several modules in zato.common, e.g. crypto, delivery, util and
    model, also rely on about half a dozen third-party packages, which
    are not declared as dependencies in setup.py.

    The ones I have found so far:

    gevent
    memory_profiler
    packaging
    paste
    psycopg2
    pyCrypto
    retools
    rsa
    SQLAlchemy
    textable
    WebHelpers

  • urllib3 is a dependency but not used at all, AFAICS

  • The outdated distutils2 package is declared as a dependency but optiona=
    l

  • zato.common.util also imports from zato.agent

  • zato.common.delivery imports from zato.redis_paginator

Regards,

–=20
Christopher Arndt christopher.arndt@pi-lar.net

On 16/03/15 17:15, Christopher Arndt wrote:

I noticed several problems when trying to set up a virtual environment
with just zato-common and zato-client. Where can I report these as
issues? And where I do I find the source from which the zato-common and
zato-client packages on PyPI where built so I can make PRs?

Hi Christopher,

I’ll have a look at it, thanks for raising it.

Reporting such things on ML or GH is fine, thanks. zato-client is part
of the zatosource/zato repository …

… and has not been updated lately because there has been no changes to it.

However, it may perhaps make sense to keep its version in sync with the
main Zato package, even if nothing changes, meaning there will be
zato-client 2.0.3 soon now after it’s cleaned up?

On 16/03/15 17:15, Christopher Arndt wrote:

I noticed several problems when trying to set up a virtual environment
with just zato-common and zato-client.

Christoph and everyone,

could you please help me out with the following?

I have cleaned up the dependencies and imports in zato-common …

https://github.com/zatosource/zato/blob/dsuch-t-gh397-zato-client-common/code/zato-common/setup.py

… but am now struggling with a cryptic traceback stemming from I’m not
sure what.

ValueError: A 0.7-series setuptools cannot be installed with

distribute. Found one at
/home/dsuch/tmp/gh397-zato-client-common/build/ConcurrentLogHandler/setuptools-0.7.7-py2.7.egg

Apparently, ConcurrentLogHandler uses setuptools in a way which is not
compatible with distribute 0.6.34 …

https://bazaar.launchpad.net/~lowell-alleman/python-concurrent-log-handler/master/view/head:/setup.py#L226

… but I’m not even sure how distribute made it into the equation
because there is no direct dependency on it.

I don’t even know if the version of setuptools ConcurrentLogHandler uses
is the one we are to use these days or an older one from before various
splits of the packaging projects.

Full traceback from an attempt to install zato-common from a local
directory is in the ticket on GH:

It’s just that being a zc.buildout user I simply went through all the
easy_install/distribute/packaging/setuptools/setuptools2/pre-0.6/pre-0.7
incompatibilities pretty much unscathed…

thanks,

Am 16.03.2015 um 21:31 schrieb Christopher Arndt:

$ virtualenv  --version

Sorry, the output is missing here.

$ virtualenv --version
12.0.7

Chris

Am 16.03.2015 um 17:15 schrieb Christopher Arndt:

  • Several modules in zato.common, e.g. crypto, delivery, util and
    model, also rely on about half a dozen third-party packages, which
    are not declared as dependencies in setup.py.
    =20
    The ones I have found so far:

Found two more (imported in zato/common/init.py):

  • boto
  • candv

Also zato/common/init.py has some funky code in it to find it’s
version number (line 39ff), which doesn’t work outside the zato source
tree. That code and the same stuff in zato-client needs to be seriously
refactored, IMHO.

Chris

On 17/03/15 10:07, Christopher Arndt wrote:

IMHO, such code should go into a script for preparing a release and/or a
post-commit script (like you already did with the revision) and should
write a static version number to the files where it is needed. I think
it’s ok to read the version number dynamically when you run the code
from your git working dir, but in a deployed package it can cause
problems (e.g. with zipped packages and other custom importers).

Yes, that’s a good suggestion, thanks Chris.

On 17/03/15 11:10, Christopher Arndt wrote:

Maybe it would be good idea to document these in a requirements-dev.txt
file in zato-common.

Chris - requirements-dev.txt, I cannot find any information on how it
differs from requirements.txt? Can you explain it a bit?

thanks a lot,

Am 17.03.2015 um 16:26 schrieb Dariusz Suchojad:

If no versions are pinned the dependencies pulled are in whatever
versions happen to be latest on PyPI, aren’t they?

Yes (unless you specify a maximum version).

But on the other hand, if people merrily start to use ‘pip install
zato-client’ without any versions pinned, how can I be serious in
providing any support if I cannot guarantee what works and what doesn’t=
?

If you don’t pin the versions in setup.py there’s really not much you
can do about it. On the other hand, people will get mad at you if, for
example, “pip install zato-client” downgrades existing packages in their
virtualenv, which they set up for their application, which just happens
to use zato-common/zato-client among other packages.

I would make a distinction between zato-common/zato-client on one hand,
and zato on the other. The former are packages and thus may be used in
contexts other than the zato server and thus you shouldn’t control the
deployment configuration. Zato, OTH, is an application and part of its
installation should be to specify exact requirements.

You could tell people, in order to get proper support, they should test
with a fresh virtualenv and install zato-common/zato-client with “pip
install -e requirements.txt” in it, where the requirements.txt would be
provided in the source dist.

BTW, “virtualenv .” is not recommended, because existing “bin” or "lib"
directories in the current dir may interfere with the virtualenv.
Specify a directory name or use virtualenvwrapper.

Chris

On 16/03/15 17:15, Christopher Arndt wrote:

I noticed several problems when trying to set up a virtual environment
with just zato-common and zato-client.

Hi Chris,

I have just pushed zato-client and zato-common 2.0.3.3 to PyPI.

The releases deal with everything you reported:

  • requirements.txt with exact versions is bundled through MANIFEST.in

  • setup.py’s install_requires uses dependencies but not versions from
    the requirements file

  • All dependencies have been brought up to date with the main project’s
    buildout.cfg and versions.cfg

There is no requirements-dev.txt though, everything went into
requirements.txt.

Installs cleanly under virtualenv and works correctly against Zato servers.

Can you please clarify the releases address all your needs?

thanks,

On 15-03-19 02:53 AM, Dariusz Suchojad wrote:

As for not pinning zato-common’s version zato-client uses - I’d like to
reduce the number of places that need be updated with a new version
string after each release.

Have you looked at bumpversion? https://pypi.python.org/pypi/bumpversion

Quite handy to un-crazy all that stuff.

On 19/03/15 13:40, Christopher Arndt wrote:

Yes, but the parse_requirements function in its setup.py doesn’t strip
off the version numbers like it does in zato-common’s setup.py.

Thank you again, Chris. It’s 2.0.3.5 then.