Using sentry logging

#1

Has anyone experience in using Sentry logging with Zato.
The current raven version that is included in Zato 3 is quite old. Can it be upgraded without problems?

Another problem is that the default logging on uncaught exceptions from Zato cannot be interpreted properly by Sentry, so there is no error context available in the sentry logs.

Any tips/tricks about Zato - Sentry integration are welcome.

Regards, Jan

0 Likes

#2

Hello @jjmurre,

yes, feel free to update it using pip and post feedback about which version you are using - I will be happy to upgrade it to a newer version for Zato 3.1.

Regards.

0 Likes

#3

@jjmurre
Hey Jan-
We use Sentry heavily with Zato. It’s awesome - but we are not using the zato built-in sentry support since the new client is not backward compatible with the raven client.

That being said we are leaking memory heavily with Sentry. I’ve been working with their dev’s and have implemented some stuff - but it’s still leaking.

@dsuch
Hey-
We are basically doing sentry init in the entry point of each service.

Which is likely why we are leaking memory… and/or could be thread related. The sentry dev’s have implied they use threads heavily.

Not sure if there is a better way to implement this so it’s available in the global python space?

Thx/Chad

0 Likes

#4

Hi @chadical,

Could it be related with this issue?

Zato is gevent based, so maybe using that type of Sentry ‘transport’ could help.

Regards, Jan

0 Likes

#5

We have actually implemented some transport related stuff already. We use sentry for exception reporting AND additional message capturing as well. We have one project for exceptions, and another separate project for ‘data issues’. Using transports we can send an event to a specific project ( or multiple projects). For data issues, we use “with sentry_sdk.push_scope() as scope:” and set_extra on the message, then the sentry_send_event can determine which project it goes to.

Also, I JUST implement the sentry_cleanup this morning… monitoring memory usage now.

Feel free to use the code below as a starting point - it works well.

### general.py

import sentry_sdk
from sentry_sdk.integrations.logging import ignore_logger

def sentry_send_event(event):
    if 'extra' in event and 'project' in event['extra'] and 'data-issues' in event['extra']['project'].lower():
        sio_data_issues.capture_event(event)
    else:
        sio_zato_esb_production.capture_event(event)

def sentry_init(self):
    global sio_zato_esb_production
    global sio_data_issues
    global sio_init_owner

    if sentry_sdk.Hub.current.client is None:
        sio_init_owner = self.name
        self.logger.info('** sentry_init ** owner: ' + self.name)
        # zato-esb-production
        sio_zato_esb_production = sentry_sdk.client.Client("https://asdfasdfasdf@sentry.io/12341234")

        # data-integration-issues
        sio_data_issues = sentry_sdk.client.Client("https://hjgkghjkghjk@sentry.io/9987978")

        sentry_sdk.init(max_breadcrumbs=20, transport=sentry_send_event)
        sentry_sdk.integrations.logging.ignore_logger('zato.server.connection.http_soap.outgoing')

def sentry_cleanup(self):
    if sio_init_owner == self.name and sentry_sdk.Hub.current.client is not None:
        self.logger.info('** sentry_cleanup ** owner:' + sio_init_owner + ' vs cleanup caller:' + self.name)
        sio_zato_esb_production.close()
        sio_data_issues.close()

def sentry_capture_data_integration_issue(issue_title, issue_level='error', issue_description='', issue_resolution='', issue_source_name='', issue_source_url='', issue_destination_name='', issue_destination_url='', issue_payload_1={}, issue_payload_2={}, issue_exception_msg='{}'):
    # Logs sentry messages to the data-integration-issues sentry Project 
    with sentry_sdk.push_scope() as scope:
        scope.level = issue_level
        scope.set_extra( 'project', 'data-issues' ) # this tells sentry to use the data-issues client
        scope.set_extra( 'Issue Description', issue_description)
        scope.set_extra( 'Issue Resolution', issue_resolution)
        scope.set_extra( 'Issue Source', issue_source_name)
        scope.set_extra( 'Issue Source', issue_source_url)
        scope.set_extra( 'Issue Destination', issue_destination_name)
        scope.set_extra( 'Issue Destination', issue_destination_url)
        scope.set_extra( 'Issue Payload 1', issue_payload_1)
        scope.set_extra( 'Issue Payload 2', issue_payload_2)
        scope.set_extra( 'Issue Exception Message', loads(issue_exception_msg))
        scope.set_tag( 'Issue Source', issue_source_name)
        scope.set_tag( 'Issue Destination', issue_destination_name)
        scope.set_tag( 'Issue Source URL', issue_source_url)
        scope.set_tag( 'Issue Destination URL', issue_destination_url)
        sentry_sdk.capture_message(issue_title)
0 Likes

#6

Hello @chadical and @jjmurre ,

I will update Raven to the latest version.

As for obtaining a Sentry client in your own code, I would use self.server.user_ctx and self.server.user_ctx_lock.

The former is a process-wide dictionary-like object that can be used for sharing information across services. It is up to users to decide what it should contain, Zato will never use it for anything internally.

Note that there is one user_ctx per one process; thus, if you have one server with four sub-processes, each one will have its own user_ctx.

self.server.user_ctx_lock is a lock object - you can use it, for instance, to serialize write access to the function that creates a Sentry client.

Here is how it can look like

class MyService(Service):

    def _create_sentry_client(self):
        client = create_client()
        self.server.user_ctx['my.sentry.client'] = client

    def before_handle(self):

        # Obtain a Python process-wide lock
        with self.server.user_ctx_lock:

            # Does the client exist already?
            if not self.server.user_ctx.get('my.sentry.client'):
                self._create_sentry_client()

    def handle(self):

        # Here it is guaranteed that the client exists,
        # either because we have just created it in before_handle
        # or because it was already available.
        client = self.server.user_ctx['my.sentry.client']
0 Likes

#7

@dsuch - thanks for the additional optimization. I will implement that this week.

Also, the raven client is deprecated. Use the new sentry-sdk package…
You will find some dependency issues…

0 Likes