(Migrated) get unicode objects from outgoing MySQL connection

(This message has been automatically imported from the retired mailing list)

Hello,
I’m trying to configure an outgoing MySQL connection to give me unicode
objects when SELECTing text-like columns.

SQLAlchemy doc says there is the |convert_unicode|option for engine
creation:

http://docs.sqlalchemy.org/en/rel_1_0/core/engines.html?highlight=convert_unicode

I tried to put “convert_unicode=True” in the extra field of the outgoing
MySQL connection dialog in the web admin, but it does not work.

Any ideas?

On 02/11/15 10:53, Andrea Peter wrote:

I’m trying to configure an outgoing MySQL connection to give me unicode
objects when SELECTing text-like columns.

SQLAlchemy doc says there is the |convert_unicode|option for engine
creation:

http://docs.sqlalchemy.org/en/rel_1_0/core/engines.html?highlight=convert_unicode

I tried to put “convert_unicode=True” in the extra field of the outgoing
MySQL connection dialog in the web admin, but it does not work.

Hi Andrea,

please send in:

  • Zato version, MySQL version

  • A series of SQL CREATE and INSERT statements to recreate your environment

  • A minimal Zato service exhibiting the behaviour

  • A snippet from logs/admin.log showing the exact JSON request a server
    received when you clicked OK on the SQL connection edit form

thanks,

Hello, is there any news about this?

On 09/11/2015 07:52, Andrea Peter wrote:

Hello, is there any news about this?
Yes: on 3rd November, Dariusz asked you to send in the details of your
zato version and mysql version, how to replicate the issue in a
standalone way, and some specific logs.

https://mailman-mail5.webfaction.com/pipermail/zato-discuss/2015-November/001428.html

On 09/11/15 11:04, Brian Candler wrote:

On 09/11/2015 07:52, Andrea Peter wrote:

Hello, is there any news about this?
Yes: on 3rd November, Dariusz asked you to send in the details of your
zato version and mysql version, how to replicate the issue in a
standalone way, and some specific logs.

https://mailman-mail5.webfaction.com/pipermail/zato-discuss/2015-November/001428.html

Yes, I replied him in private, I’ll send the email to the list

On 09/11/15 11:34, Andrea Peter wrote:

Yes, I replied him in private, I’ll send the email to the list

Hi Andrea,

yes, please do. We are all very busy here in projects with customers and
have not been able to look into it and reproduce the situation yet.

thanks,

On 02/11/15 10:53, Andrea Peter wrote:

Hello,
I’m trying to configure an outgoing MySQL connection to give me
unicode objects when SELECTing text-like columns.

I tried to put “convert_unicode=True” in the extra field of the
outgoing MySQL connection dialog in the web admin, but it does not work.

Any ideas?

Got the solution for this,

the problems are the following:

  • (I think) the kwarg convert_unicode only works when using
    SQLAlchemy’s expression language or the ORM, not when (as in my case)
    using raw SQL
  • pymysql (which is the engine used by Zato for MySQL connection) uses
    latin1 by default and sets use_unicode=False when no charset is
    specified explicitly

SQLAlchemy detects arguments to pass to the connector in the url it
receives:
’{engine}://{username}:{password}@{host}:{port}/{db_name}?key=value’

key, value pairs are passed to the connector (not sure how they are
separated, i guess with a ‘&’?)

As a workaround we can inject those key-value pairs when specifying the
name of the database (in web admin or through public API) like:
dbname?charset=utf8

Wouldn’t it be useful to have an extra field for the options for the
connector when creating an outgoing SQL connection?

Thanks,
Andrea.