(Migrated) server restart fails

(This message has been automatically imported from the retired mailing list)

Hi,

One of 3 node zato cluster’ servers fails to restart with error below.
I modified a service and uploaded it via web admin. I can see that
service source code modified but it did not responded as I expected.
So I tried to restart zato server. But it failed. It became up state
and right after clean-down.

What should I do?

Thanks…

ps: I replaced some private service names with xxxx in error message lines

ParallelServer.start_server(worker.app.zato_wsgi_app,
arbiter.zato_deployment_key)
File “/opt/zato/2.0.7/zato-server/src/zato/server/base/parallel.py”,
line 815, in start_server
is_singleton = parallel_server._after_init_accepted(server,
zato_deployment_key, locally_deployed)
File “/opt/zato/2.0.7/zato-server/src/zato/server/base/parallel.py”,
line 685, in _after_init_accepted
self.odb.on_deployment_finished()
File “/opt/zato/2.0.7/zato-server/src/zato/server/odb.py”, line 62,
in on_deployment_finished
self._session.commit()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py”,
line 788, in commit
self.transaction.commit()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py”,
line 388, in commit
t[1].commit()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py”,
line 1445, in commit
self._do_commit()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py”,
line 1476, in _do_commit
self.connection._commit_impl()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py”,
line 642, in _commit_impl
self.connection._reset_agent is self.__transaction:
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py”,
line 283, in connection
return self._revalidate_connection()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py”,
line 359, in _revalidate_connection
raise exc.ResourceClosedError(“This Connection is closed”)
ResourceClosedError: This Connection is closed
warnings.warn(msg, UserWarning)
warnings.warn(msg, UserWarning)
warnings.warn(msg, UserWarning)
File “/opt/zato/2.0.7/zato-server/src/zato/server/odb.py”, line 233,
in add_service
self._session.commit()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py”,
line 788, in commit
self.transaction.commit()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py”,
line 388, in commit
t[1].commit()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py”,
line 1445, in commit
self._do_commit()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py”,
line 1476, in _do_commit
self.connection._commit_impl()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py”,
line 642, in _commit_impl
self.connection._reset_agent is self.__transaction:
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py”,
line 283, in connection
return self._revalidate_connection()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py”,
line 359, in _revalidate_connection
raise exc.ResourceClosedError(“This Connection is closed”)
ResourceClosedError: This Connection is closed
]
Traceback (most recent call last):
File “/opt/zato/2.0.7/eggs/gunicorn-18.0-py2.7.egg/gunicorn/arbiter.py”,
line 494, in spawn_worker
self.cfg.post_fork(self, worker)
File “/opt/zato/2.0.7/zato-server/src/zato/server/base/parallel.py”,
line 864, in post_fork
ParallelServer.start_server(worker.app.zato_wsgi_app,
arbiter.zato_deployment_key)
File “/opt/zato/2.0.7/zato-server/src/zato/server/base/parallel.py”,
line 815, in start_server
is_singleton = parallel_server._after_init_accepted(server,
zato_deployment_key, locally_deployed)
File “/opt/zato/2.0.7/zato-server/src/zato/server/base/parallel.py”,
line 685, in _after_init_accepted
self.odb.on_deployment_finished()
File “/opt/zato/2.0.7/zato-server/src/zato/server/odb.py”, line 62,
in on_deployment_finished
self._session.commit()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py”,
line 788, in commit
self.transaction.commit()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py”,
line 396, in commit
self.close()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py”,
line 466, in close
self.session.begin()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py”,
line 719, in begin
self, nested=nested)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py”,
line 181, in init
self._take_snapshot()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py”,
line 258, in _take_snapshot
self.session.flush()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py”,
line 1985, in flush
self._flush(objects)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py”,
line 2103, in _flush
transaction.rollback(_capture_exception=True)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/util/langhelpers.py”,
line 60, in exit
compat.reraise(exc_type, exc_value, exc_tb)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py”,
line 2067, in _flush
flush_context.execute()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/unitofwork.py”,
line 372, in execute
rec.execute(self)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/unitofwork.py”,
line 526, in execute
uow
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/persistence.py”,
line 65, in save_obj
mapper, table, insert)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/persistence.py”,
line 602, in _emit_insert_statements
execute(statement, params)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py”,
line 841, in execute
return meth(self, multiparams, params)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/sql/elements.py”,
line 322, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py”,
line 938, in _execute_clauseelement
compiled_sql, distilled_params
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py”,
line 1070, in _execute_context
context)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py”,
line 1271, in _handle_dbapi_exception
exc_info
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/util/compat.py”,
line 199, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py”,
line 1063, in _execute_context
context)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/engine/default.py”,
line 442, in do_execute
cursor.execute(statement, parameters)
File “/opt/zato/2.0.7/eggs/pg8000-1.10.2-py2.7.egg/pg8000/core.py”,
line 910, in execute
self._c.execute(self, operation, args)
File “/opt/zato/2.0.7/eggs/pg8000-1.10.2-py2.7.egg/pg8000/core.py”,
line 2040, in execute
self.handle_messages(cursor)
File “/opt/zato/2.0.7/eggs/pg8000-1.10.2-py2.7.egg/pg8000/core.py”,
line 2117, in handle_messages
raise self.error
ProgrammingError: (ProgrammingError) (u’ERROR’, u’23505’, u’duplicate
key value violates unique constraint “service_name_cluster_id_key”’,
u’Key (name, cluster_id)=(nvi.xxxxx-xxxxx, 1) already exists.’,
u’public’, u’service’, u’service_name_cluster_id_key’, u’nbtinsert.c’,
u’406’, u’_bt_check_unique’, u’’, u’’) u"INSERT INTO service (id,
name, is_active, impl_name, is_internal, wsdl, wsdl_name,
slow_threshold, cluster_id) VALUES (nextval(‘service_id_seq’), %s, %s,
%s, %s, %s, %s, %s, %s) RETURNING service.id" (u’nvi.xxxxx-xxxxx’,
True, u’nvi.XxxxXxxx’, False, None, None, 99999, 1)
Traceback (most recent call last):
File “/opt/zato/2.0.7/eggs/gunicorn-18.0-py2.7.egg/gunicorn/arbiter.py”,
line 494, in spawn_worker
self.cfg.post_fork(self, worker)
File “/opt/zato/2.0.7/zato-server/src/zato/server/base/parallel.py”,
line 864, in post_fork
ParallelServer.start_server(worker.app.zato_wsgi_app,
arbiter.zato_deployment_key)
File “/opt/zato/2.0.7/zato-server/src/zato/server/base/parallel.py”,
line 815, in start_server
is_singleton = parallel_server._after_init_accepted(server,
zato_deployment_key, locally_deployed)
File “/opt/zato/2.0.7/zato-server/src/zato/server/base/parallel.py”,
line 685, in _after_init_accepted
self.odb.on_deployment_finished()
File “/opt/zato/2.0.7/zato-server/src/zato/server/odb.py”, line 62,
in on_deployment_finished
self._session.commit()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py”,
line 788, in commit
self.transaction.commit()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py”,
line 396, in commit
self.close()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py”,
line 466, in close
self.session.begin()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py”,
line 719, in begin
self, nested=nested)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py”,
line 181, in init
self._take_snapshot()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py”,
line 258, in _take_snapshot
self.session.flush()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py”,
line 1985, in flush
self._flush(objects)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py”,
line 2103, in _flush
transaction.rollback(_capture_exception=True)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/util/langhelpers.py”,
line 60, in exit
compat.reraise(exc_type, exc_value, exc_tb)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py”,
line 2067, in _flush
flush_context.execute()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/unitofwork.py”,
line 372, in execute
rec.execute(self)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/unitofwork.py”,
line 526, in execute
uow
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/persistence.py”,
line 65, in save_obj
mapper, table, insert)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/persistence.py”,
line 602, in _emit_insert_statements
execute(statement, params)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py”,
line 841, in execute
return meth(self, multiparams, params)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/sql/elements.py”,
line 322, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py”,
line 938, in _execute_clauseelement
compiled_sql, distilled_params
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py”,
line 1070, in _execute_context
context)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py”,
line 1271, in _handle_dbapi_exception
exc_info
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/util/compat.py”,
line 199, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py”,
line 1063, in _execute_context
context)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/engine/default.py”,
line 442, in do_execute
cursor.execute(statement, parameters)
File “/opt/zato/2.0.7/eggs/pg8000-1.10.2-py2.7.egg/pg8000/core.py”,
line 910, in execute
self._c.execute(self, operation, args)
File “/opt/zato/2.0.7/eggs/pg8000-1.10.2-py2.7.egg/pg8000/core.py”,
line 2040, in execute
self.handle_messages(cursor)
File “/opt/zato/2.0.7/eggs/pg8000-1.10.2-py2.7.egg/pg8000/core.py”,
line 2117, in handle_messages
raise self.error
ProgrammingError: (ProgrammingError) (u’ERROR’, u’23505’, u’duplicate
key value violates unique constraint “service_name_cluster_id_key”’,
u’Key (name, cluster_id)=(nvi.xxxxx-xxxxx, 1) already exists.’,
u’public’, u’service’, u’service_name_cluster_id_key’, u’nbtinsert.c’,
u’406’, u’_bt_check_unique’, u’’, u’’) u"INSERT INTO service (id,
name, is_active, impl_name, is_internal, wsdl, wsdl_name,
slow_threshold, cluster_id) VALUES (nextval(‘service_id_seq’), %s, %s,
%s, %s, %s, %s, %s, %s) RETURNING service.id" (u’nvi.xxxxx-xxxxx’,
True, u’nvi.XxxxXxxx’, False, None, None, 99999, 1)
warnings.warn(msg, UserWarning)
Traceback (most recent call last):
File “/opt/zato/2.0.7/eggs/gunicorn-18.0-py2.7.egg/gunicorn/arbiter.py”,
line 494, in spawn_worker
self.cfg.post_fork(self, worker)
File “/opt/zato/2.0.7/zato-server/src/zato/server/base/parallel.py”,
line 864, in post_fork
ParallelServer.start_server(worker.app.zato_wsgi_app,
arbiter.zato_deployment_key)
File “/opt/zato/2.0.7/zato-server/src/zato/server/base/parallel.py”,
line 815, in start_server
is_singleton = parallel_server._after_init_accepted(server,
zato_deployment_key, locally_deployed)
File “/opt/zato/2.0.7/zato-server/src/zato/server/base/parallel.py”,
line 682, in _after_init_accepted
self.deploy_missing_services(locally_deployed)
File “/opt/zato/2.0.7/zato-server/src/zato/server/base/parallel.py”,
line 229, in deploy_missing_services
msg.package_id = hot_deploy(self, file_name, full_path, notify=False)
File “/opt/zato/2.0.7/zato-common/src/zato/common/util.py”, line
498, in hot_deploy
now, di, file_name, open(path, ‘rb’).read(), parallel_server.id)
File “/opt/zato/2.0.7/zato-server/src/zato/server/odb.py”, line 322,
in hot_deploy
filter(Cluster.id == self.server.cluster_id).
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py”,
line 2398, in one
ret = list(self)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py”,
line 2440, in iter
self.session._autoflush()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py”,
line 1264, in _autoflush
self.flush()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py”,
line 1985, in flush
self._flush(objects)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py”,
line 2103, in _flush
transaction.rollback(_capture_exception=True)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/util/langhelpers.py”,
line 60, in exit
compat.reraise(exc_type, exc_value, exc_tb)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py”,
line 2067, in _flush
flush_context.execute()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/unitofwork.py”,
line 372, in execute
rec.execute(self)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/unitofwork.py”,
line 526, in execute
uow
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/persistence.py”,
line 46, in save_obj
uowtransaction)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/persistence.py”,
line 141, in _organize_states_for_save
states):
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/persistence.py”,
line 849, in _connections_for_states
base_mapper)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py”,
line 232, in connection
return self._connection_for_bind(bind, execution_options)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py”,
line 315, in _connection_for_bind
conn = self._parent._connection_for_bind(bind, execution_options)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py”,
line 326, in _connection_for_bind
conn = bind.contextual_connect()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py”,
line 1910, in contextual_connect
self.pool.connect(),
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/pool.py”,
line 338, in connect
return _ConnectionFairy._checkout(self)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/pool.py”,
line 645, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/pool.py”,
line 440, in checkout
rec = pool._do_get()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/pool.py”,
line 960, in _do_get
(self.size(), self.overflow(), self._timeout))
TimeoutError: QueuePool limit of size 1 overflow 10 reached,
connection timed out, timeout 30
Traceback (most recent call last):
File “/opt/zato/2.0.7/eggs/gunicorn-18.0-py2.7.egg/gunicorn/arbiter.py”,
line 494, in spawn_worker
self.cfg.post_fork(self, worker)
File “/opt/zato/2.0.7/zato-server/src/zato/server/base/parallel.py”,
line 864, in post_fork
ParallelServer.start_server(worker.app.zato_wsgi_app,
arbiter.zato_deployment_key)
File “/opt/zato/2.0.7/zato-server/src/zato/server/base/parallel.py”,
line 815, in start_server
is_singleton = parallel_server._after_init_accepted(server,
zato_deployment_key, locally_deployed)
File “/opt/zato/2.0.7/zato-server/src/zato/server/base/parallel.py”,
line 682, in _after_init_accepted
self.deploy_missing_services(locally_deployed)
File “/opt/zato/2.0.7/zato-server/src/zato/server/base/parallel.py”,
line 229, in deploy_missing_services
msg.package_id = hot_deploy(self, file_name, full_path, notify=False)
File “/opt/zato/2.0.7/zato-common/src/zato/common/util.py”, line
498, in hot_deploy
now, di, file_name, open(path, ‘rb’).read(), parallel_server.id)
File “/opt/zato/2.0.7/zato-server/src/zato/server/odb.py”, line 322,
in hot_deploy
filter(Cluster.id == self.server.cluster_id).
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py”,
line 2398, in one
ret = list(self)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py”,
line 2440, in iter
self.session._autoflush()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py”,
line 1264, in _autoflush
self.flush()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py”,
line 1985, in flush
self._flush(objects)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py”,
line 2103, in _flush
transaction.rollback(_capture_exception=True)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/util/langhelpers.py”,
line 60, in exit
compat.reraise(exc_type, exc_value, exc_tb)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py”,
line 2067, in _flush
flush_context.execute()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/unitofwork.py”,
line 372, in execute
rec.execute(self)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/unitofwork.py”,
line 526, in execute
uow
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/persistence.py”,
line 46, in save_obj
uowtransaction)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/persistence.py”,
line 141, in _organize_states_for_save
states):
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/persistence.py”,
line 849, in _connections_for_states
base_mapper)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py”,
line 232, in connection
return self._connection_for_bind(bind, execution_options)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py”,
line 315, in _connection_for_bind
conn = self._parent._connection_for_bind(bind, execution_options)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py”,
line 326, in _connection_for_bind
conn = bind.contextual_connect()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py”,
line 1910, in contextual_connect
self.pool.connect(),
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/pool.py”,
line 338, in connect
return _ConnectionFairy._checkout(self)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/pool.py”,
line 645, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/pool.py”,
line 440, in checkout
rec = pool._do_get()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/pool.py”,
line 960, in _do_get
(self.size(), self.overflow(), self._timeout))
TimeoutError: QueuePool limit of size 1 overflow 10 reached,
connection timed out, timeout 30
File “/opt/zato/2.0.7/eggs/springpython-1.3.0rc1-py2.7.egg/springpython/context/init.py”,
line 106, in shutdown_hook
destroy_method()
File “/opt/zato/2.0.7/zato-server/src/zato/server/base/parallel.py”,
line 889, in destroy
self.broker_client.publish(msg, msg_type=msg_type)
AttributeError: ‘NoneType’ object has no attribute ‘publish’
'
Traceback (most recent call last):
File “/opt/zato/2.0.7/eggs/gunicorn-18.0-py2.7.egg/gunicorn/arbiter.py”,
line 494, in spawn_worker
self.cfg.post_fork(self, worker)
File “/opt/zato/2.0.7/zato-server/src/zato/server/base/parallel.py”,
line 864, in post_fork
ParallelServer.start_server(worker.app.zato_wsgi_app,
arbiter.zato_deployment_key)
File “/opt/zato/2.0.7/zato-server/src/zato/server/base/parallel.py”,
line 815, in start_server
is_singleton = parallel_server._after_init_accepted(server,
zato_deployment_key, locally_deployed)
File “/opt/zato/2.0.7/zato-server/src/zato/server/base/parallel.py”,
line 685, in _after_init_accepted
self.odb.on_deployment_finished()
File “/opt/zato/2.0.7/zato-server/src/zato/server/odb.py”, line 62,
in on_deployment_finished
self._session.commit()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py”,
line 788, in commit
self.transaction.commit()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py”,
line 388, in commit
t[1].commit()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py”,
line 1445, in commit
self._do_commit()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py”,
line 1476, in _do_commit
self.connection._commit_impl()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py”,
line 642, in _commit_impl
self.connection._reset_agent is self.__transaction:
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py”,
line 283, in connection
return self._revalidate_connection()
File “/opt/zato/2.0.7/eggs/SQLAlchemy-0.9.9-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py”,
line 359, in _revalidate_connection
raise exc.ResourceClosedError(“This Connection is closed”)
ResourceClosedError: This Connection is closed
Traceback (most recent call last):

On 02/12/15 09:49, Ali Rıza KELEŞ wrote:

One of 3 node zato cluster’ servers fails to restart with error below.
I modified a service and uploaded it via web admin. I can see that
service source code modified but it did not responded as I expected.
So I tried to restart zato server. But it failed. It became up state
and right after clean-down.

Hi Ali,

please describe what led to the whole situation. I understand what you
are trying to do now but what were the actions that led to the initial
error? How can it be reproduced?

thanks,

On 2 December 2015 at 12:03, Dariusz Suchojad dsuch@zato.io wrote:

On 02/12/15 09:49, Ali R=C4=B1za KELE=C5=9E wrote:

One of 3 node zato cluster’ servers fails to restart with error below.
I modified a service and uploaded it via web admin. I can see that
service source code modified but it did not responded as I expected.
So I tried to restart zato server. But it failed. It became up state
and right after clean-down.

Hi Ali,

please describe what led to the whole situation. I understand what you
are trying to do now but what were the actions that led to the initial
error? How can it be reproduced?

Hi Darius,

I deployed 2 different service. If it helps you can see here the source cod=
es:

First one is invoked by second one, while the first just writes some
keys into redis.

I got an error like “too many keys to unpack” here:
https://github.com/zetaops/ulakbus/blob/master/ulakbus/services/common/nvi_=
sts.py#L120

So, I add some logging lines just before this line to see types of
values, I expect them to be strings, not list or dict, etc…

I modified file and deployed it again via web admin interface. I
invoke it via curl.

curl localhost:17010/nvi-sts-token

But I did not see the type logs. So I restart zato components again:

$ sudo service zato restart

And result is as expected:

Load-balancer /opt/zato/ulakbus/load-balancer shutting down
Load-balancer’s agent /opt/zato/ulakbus/load-balancer shutting down
Server /opt/zato/ulakbus/node-01 shutting down
Web admin /opt/zato/ulakbus/web-admin shutting down
Starting ulakbus.load-balancer from /opt/zato/ulakbus/load-balancer…
OK
Starting ulakbus.node01 from /opt/zato/ulakbus/node-01…
OK
Starting ulakbus.web-admin from /opt/zato/ulakbus/web-admin…
OK

And then first attempt, produced same error and I did not see the logs too:

$ curl localhost:17010/nvi-sts-token

{“zato_env”: {“details”: “Traceback (most recent call last):\n File
”/opt/zato/2.0.7/zato-server/src/zato/server/connection/http_soap=
/channel.py",
line 198, in dispatch\n payload, worker_store,
self.simple_io_config, post_data)\n File
"/opt/zato/2.0.7/zato-server/src/zato/server/connection/http_soap=
/channel.py",
line 350, in handle\n params_priority=3Dchannel_item.params_pri)\n
File “/opt/zato/2.0.7/zato-server/src/zato/server/service/_init=
_.py”,
line 401, in update_handle\n self._invoke(service, channel)\n File
"/opt/zato/2.0.7/zato-server/src/zato/server/service/init.py=
",
line 344, in _invoke\n service.handle()\n File
"/opt/zato/ulakbus/node-01/work/hot-deploy/current/nvi_sts.py",
line 119, in handle\n for k, v in result:\nValueError: too many
values to unpack\n", “result”: “ZATO_ERROR”, “cid”:
“K05RN3K541327GTZESH30QJ123DY”}}

But the second connection refused

$ curl -X POST localhost:17010/nvi-sts-token
curl: (7) Failed to connect to localhost port 17010: Connection refused

The server had gone. At this point I saw the errors which i sent in
previous message. Then I tried to restart zato components, it can not
find process in zato servers pidfile. I removed pidfile and restart
again and again. It is same. It became up and after some time crashes
with same errors.

I don’t know if these infos helps.

Thanks.

–=20

Ali R=C4=B1za Kele=C5=9F

Hi Ali,

can you stop each server and delete everything in each of the server’s
directory holding hot-deployed code?

/path/to/server/work/hot-deploy/current

Then delete PID files, if any, and start servers individually back
without hot-deploying anything?

thanks,