zato enmasse throws ParsingException


#1

Hi,

When I run the following command sudo -i -u zato zato enmasse /opt/zato/perch/server1/ --input /var/www/zato/perch/db_scripts/new_base/import.json --import --replace-odb-objects
The following error is thrown

Could not fetch objects of type zato_generic_connection: Traceback (most recent call last):
File “/opt/zato/3.0/code/zato-server/src/zato/server/connection/http_soap/channel.py”, line 268, in dispatch
payload, worker_store, self.simple_io_config, post_data, path_info, soap_action)
File “/opt/zato/3.0/code/zato-server/src/zato/server/connection/http_soap/channel.py”, line 495, in handle
params_priority=channel_item.params_pri)
File “/opt/zato/3.0/code/zato-server/src/zato/server/service/init.py”, line 474, in update_handle
self._invoke(service, channel)
File “/opt/zato/3.0/code/zato-server/src/zato/server/service/init.py”, line 396, in invoke
service.handle()
File “/opt/zato/3.0/code/zato-server/src/zato/server/service/internal/service.py”, line 311, in handle
response = func(id
, payload, channel, data_format, transport, serialize=True)
File “/opt/zato/3.0/code/zato-server/src/zato/server/service/init.py”, line 599, in invoke
return self.invoke_by_impl_name(self.server.service_store.name_to_impl_name[name], *args, **kwargs)
File “/opt/zato/3.0/code/zato-server/src/zato/server/service/init.py”, line 581, in invoke_by_impl_name
return self.update_handle(*invoke_args, **kwargs)
File “/opt/zato/3.0/code/zato-server/src/zato/server/service/init.py”, line 441, in update_handle
wmq_ctx=kwargs.get(‘wmq_ctx’))
File “/opt/zato/3.0/code/zato-server/src/zato/server/service/init.py”, line 942, in update
service._init(channel_type in _wsgi_channels)
File “/opt/zato/3.0/code/zato-server/src/zato/server/service/internal/init.py”, line 71, in init
super(AdminService, self).init(is_http)
File “/opt/zato/3.0/code/zato-server/src/zato/server/service/init.py”, line 351, in init
self.server.encrypt)
File “/opt/zato/3.0/code/zato-server/src/zato/server/service/reqresp/init.py”, line 141, in init
self.init_flat_sio(cid, sio, data_format, transport, wsgi_environ, required_list)
File “/opt/zato/3.0/code/zato-server/src/zato/server/service/reqresp/init.py”, line 184, in init_flat_sio
required_list, use_channel_params_only, path_prefix, default_value, use_text))
File “/opt/zato/3.0/code/zato-server/src/zato/server/service/reqresp/init.py”, line 220, in get_params
raise ParsingException(msg)
ParsingException: <ParsingException at 0x7f1c0df00820 cid:Caught an exception, param:type
, params_to_visit:(‘cluster_id’, 'type
’), has_simple_io_config:True, e:Traceback (most recent call last):
File “/opt/zato/3.0/code/zato-server/src/zato/server/service/reqresp/init.py”, line 213, in get_params
True, self.encrypt_func, self.encrypt_secrets, self.params_priority)
File “/opt/zato/3.0/code/zato-server/src/zato/server/service/reqresp/sio.py”, line 483, in convert_param
raise ParsingException(cid, msg)
ParsingException: <ParsingException at 0x7f1c0df2f500 cid:None, msg:Required input element:type
not found, value:ZATO_NONE, data_format:json, payload:{‘cluster_id’: 1}, channel_params:{}>, msg:None>

On using the verbose to the above command I could see that the error is invoked while creating zato_generic_connection. The import.json file which we are using doesn’t have any entry for zato_generic_connection, not sure why this is been invoked.

What could be the possible issue here?

Thanks,
Sai Bhargav


#2

I have something comparable, also on running enmasse:

    2018-09-19 15:42:28,464 - WARNING - 137:DummyThread-10 - zato.server.service:583 - Could not invoke `zato.generic.connection.get-list`, e:`Traceback (most recent call last):
  File "/opt/zato/3.0/code/zato-server/src/zato/server/service/__init__.py", line 581, in invoke_by_impl_name
    return self.update_handle(*invoke_args, **kwargs)
  File "/opt/zato/3.0/code/zato-server/src/zato/server/service/__init__.py", line 441, in update_handle
    wmq_ctx=kwargs.get('wmq_ctx'))
  File "/opt/zato/3.0/code/zato-server/src/zato/server/service/__init__.py", line 942, in update
    service._init(channel_type in _wsgi_channels)
  File "/opt/zato/3.0/code/zato-server/src/zato/server/service/internal/__init__.py", line 71, in _init
    super(AdminService, self)._init(is_http)
  File "/opt/zato/3.0/code/zato-server/src/zato/server/service/__init__.py", line 351, in _init
    self.server.encrypt)
  File "/opt/zato/3.0/code/zato-server/src/zato/server/service/reqresp/__init__.py", line 141, in init
    self.init_flat_sio(cid, sio, data_format, transport, wsgi_environ, required_list)
  File "/opt/zato/3.0/code/zato-server/src/zato/server/service/reqresp/__init__.py", line 184, in init_flat_sio
    required_list, use_channel_params_only, path_prefix, default_value, use_text))
  File "/opt/zato/3.0/code/zato-server/src/zato/server/service/reqresp/__init__.py", line 220, in get_params
    raise ParsingException(msg)
ParsingException: <ParsingException at 0x7f9f2de3daf0 cid:`Caught an exception, param:`type_`, params_to_visit:`('cluster_id', 'type_')`, has_simple_io_config:`True`, e:`Traceback (most recent call last):
  File "/opt/zato/3.0/code/zato-server/src/zato/server/service/reqresp/__init__.py", line 213, in get_params
    True, self.encrypt_func, self.encrypt_secrets, self.params_priority)
  File "/opt/zato/3.0/code/zato-server/src/zato/server/service/reqresp/sio.py", line 483, in convert_param
    raise ParsingException(cid, msg)
ParsingException: <ParsingException at 0x7f9f3d2578c0 cid:`None`, msg:`Required input element:`type_` not found, value:`ZATO_NONE`, data_format:`json`, payload:`{'cluster_id': 1}`, channel_params:`{}``>
``, msg:`None`>

After that, my channels are loading, so enmasse continues. But, the behaviour of the channels is not correct (problems with the routes).

Regards, Jan


#3

One of my older Zato 3 instances, does not have this problem, that instance is based on:

commit 0e74cdfe940a9548a01efbe1e68e6170df109539
Author: Dariusz Suchojad <dsuch-github@m.zato.io>
Date:   Wed Jul 18 08:37:44 2018 +0200

#4

@sai_bhargav @jjmurre Can you please update your codebase and try again with enmasse? I pushed a changset that deals with generic connections and the type_ element - note that it does not cover REST path precedence.

As for why generic connections get involved into it - part of what enmasse does is checking what objects already exist in the ODB, that is, it indiscriminately pulls everything and only then actual proceeds to import data from input files. Thus, the error was in fetching generic connections rather than importing them.


#5

@dsuch I can confirm that this error is fixed now. I pulled the ‘support/3.0’ branch (containing commit 5e3651b3eeb989744696ba886f8caea397dbb2b9).


#6

Hi dsuch,

I have pulled the latest code base and ran zato start /opt/zato/perch/server1/ --sync-internal and after that, I ran the enmasse and received the following error
Invoking zato.channel.amqp.get-list for channel_amqp
Invoking zato.channel.web-socket.get-list for web_socket
Invoking zato.pubsub.endpoint.get-list for pubsub_endpoint
Invoking zato.channel.jms-wmq.get-list for channel_jms_wmq
Invoking zato.channel.zmq.get-list for channel_zmq
Type def_sec has no ‘get-list’ service
Invoking zato.http-soap.get-list for http_soap
Invoking zato.scheduler.job.get-list for scheduler
Invoking zato.notif.sql.get-list for notif_sql
Invoking zato.outgoing.amqp.get-list for outconn_amqp
Invoking zato.outgoing.jms-wmq.get-list for outconn_jms_wmq
Invoking zato.query.cassandra.get-list for query_cassandra
Invoking zato.cloud.aws.s3.get-list for cloud_aws_s3
Invoking zato.security.ntlm.get-list for ntlm
Invoking zato.message.json-pointer.get-list for json_pointer
Invoking zato.scheduler.job.get-list for scheduler
Invoking zato.security.apikey.get-list for apikey
Invoking zato.security.basic-auth.get-list for basic_auth
Invoking zato.outgoing.sap.get-list for outconn_sap
Invoking zato.definition.amqp.get-list for def_amqp
Invoking zato.email.smtp.get-list for email_smtp
Invoking zato.security.aws.get-list for aws
Invoking zato.security.vault.connection.get-list for vault_connection
Invoking zato.security.rbac.permission.get-list for rbac_permission
Invoking zato.pubsub.topic.get-list for pubsub_topic
Invoking zato.email.imap.get-list for email_imap
Invoking zato.security.oauth.get-list for oauth
Invoking zato.security.tls.channel.get-list for tls_channel_sec
Invoking zato.cache.builtin.get-list for zato_cache_builtin
Invoking zato.definition.cassandra.get-list for def_cassandra
Invoking zato.cloud.openstack.swift.get-list for def_cloud_openstack_swift
Invoking zato.outgoing.stomp.get-list for outconn_stomp
Invoking zato.outgoing.odoo.get-list for outconn_odoo
Invoking zato.generic.connection.get-list for zato_generic_connection
Invoking zato.outgoing.sql.get-list for outconn_sql
Invoking zato.outgoing.zmq.get-list for outconn_zmq
Invoking zato.security.tls.key-cert.get-list for tls_key_cert
Invoking zato.security.rbac.role.get-list for rbac_role
Invoking zato.security.wss.get-list for wss
Invoking zato.definition.jms-wmq.get-list for def_jms_wmq
Invoking zato.search.es.get-list for search_es
Invoking zato.search.solr.get-list for search_solr
Invoking zato.sms.twilio.get-list for zato_sms_twilio
Invoking zato.message.xpath.get-list for xpath
Invoking zato.cache.memcached.get-list for zato_cache_memcached
Invoking zato.notif.cloud.openstack.swift.get-list for notif_cloud_openstack_swift
Invoking zato.outgoing.ftp.get-list for outconn_ftp
Invoking zato.message.namespace.get-list for def_namespace
Invoking zato.security.rbac.role-permission.get-list for rbac_role_permission
Invoking zato.channel.stomp.get-list for stomp
Invoking zato.channel.web-socket.get-list for web_socket
Invoking zato.security.jwt.get-list for jwt
Invoking zato.security.xpath.get-list for xpath_sec
Invoking zato.security.tls.ca-cert.get-list for tls_ca_cert
ODB objects read
ODB objects merged in
Traceback (most recent call last):
File “/opt/zato/current/bin/zato”, line 11, in
load_entry_point(‘zato-cli’, ‘console_scripts’, ‘zato’)()
File “/opt/zato/3.0/code/zato-cli/src/zato/cli/zato_command.py”, line 379, in main
return run_command(get_parser().parse_args())
File “/opt/zato/3.0/code/zato-cli/src/zato/cli/init.py”, line 356, in run_command
command_classargs.command.run(args)
File “/opt/zato/3.0/code/zato-cli/src/zato/cli/init.py”, line 632, in run
return_code = self.execute(args)
File “/opt/zato/3.0/code/zato-cli/src/zato/cli/init.py”, line 909, in execute
return self._get_dispatch()json_data[‘component’]
File “/opt/zato/3.0/code/zato-cli/src/zato/cli/enmasse.py”, line 1227, in _on_server
self.write_output()
File “/opt/zato/3.0/code/zato-cli/src/zato/cli/enmasse.py”, line 1269, in write_output
items.sort(key=lambda item: item[‘id’])
File “/opt/zato/3.0/code/zato-cli/src/zato/cli/enmasse.py”, line 1269, in
items.sort(key=lambda item: item[‘id’])
KeyError: u’id’

The git log shows the following in my machine
zato@ubuntu-xenial:~/current$ git log -1
commit f0d20c694f2d33331ab6bb696519d173c5a2e31e
Author: Dariusz Suchojad dsuch-github@m.zato.io
Date: Wed Sep 26 16:02:27 2018 +0200
GH #885 - Made generic connections work with enmasse.

Thanks,
Sai Bhargav


#7

@sai_bhargav - Please upload the input file that you are using with enmasse.


#8

Hi dsuch,

I have updated my source and restarted the zato and ran the enmasse and it worked without any issues. I was not able to reproduce it again. Will let you know if it reproduces again. Thanks for the support.

Regards,
Sai Bhargav