Invoking other services that expect parameters in QUERY_STRING

jsabater · 2018-10-16 20:57:55 UTC

Hi everyone!

In my test project I have some schema classes: guests, rooms and bookings. I have a number of services for each schema class in separate modules to get, delete, update, create and list records, among others.

Now I am building a more complicated service in which I want the list of bookings from a guest and the rooms related to those bookings. The channel would be something like:

/genesisng/guests/{id}/bookings

And the desired output would be something like:

{
    # Guest attributes
    "id": "1",
    // More attributes
    "bookings": [
        {
            # Booking attributes
            "id": "1",
            // More attributes
        },
        {
            # Booking attributes
            "id": "2",
            // More attributes
        },
    ],
    "rooms": [
        {
             # Room attributes
            "id": "1",
            // More attributes
        },
        {
             # Room attributes
            "id": "2",
            // More attributes
        }
    ]
}

To do this, my plan would be:

Invoke guest.get service.
Invoke booking.list service. Loop the results to get the list of room ids.
Invoke room.get multiple times or invoke room.list.

At the moment all my module.list services use query string parameters to handle filtering, fields projection, pagination, sorting and searching. For example:

curl -v -g "http://127.0.0.1:11223/genesisng/guests/list?page=4&size=40&sort_by=name&order_by=desc&fields=id&fields=name&filters=id|gt|50"

Invoking the guest.get module is working fine:

        id_ = self.request.input.id
        result = {}
        # Get guest data
        input_data = {'id': id_}
        guest = self.invoke('guest.get', input_data, as_bunch=True)
        if guest:
            result = guest

But invoking booking.list is not because I use this bit of code to parse the query string:

# Check for parameters in the query string
qs = parse_qs(self.wsgi_environ['QUERY_STRING'])
if qs:
    # Handle pagination
    # Handle sorting
    # Handle filtering
    # Handle fields projection
    # Handle search

The call I make is as follows:

# Get the list of bookings from the guest
result.bookings = []
input_data = {'filters': 'id_guest|eq|%s' % id_}
bookings = self.invoke('booking.list', input_data, as_bunch=True)
if bookings:
   rooms = [] # Used to later retrieve the list of rooms to be added to result
   for b in bookings:
        result.bookings.append(b)
        rooms.append(b.id_room)

And the error I get is:

2018-10-16 20:03:11,636 DEBG 'zato-server2' stdout output:
2018-10-16 20:03:11,636 - INFO - 116:DummyThread-55 - guest.bookings:429 - Invoking guest.get...

2018-10-16 20:03:11,643 DEBG 'zato-server2' stdout output:
2018-10-16 20:03:11,643 - INFO - 116:DummyThread-55 - guest.bookings:431 - Response is: response:
    address1: 221 B Baker St
    address2: Woolsthrope Manor
    birthdate: '1643-01-04'
    country: GB
    email: inewton@genesis.com
    gender: '1'
    home_phone: '+44.2079460702'
    id: 1
    locality: Lincolnshire
    mobile_phone: '+44.2079460701'
    name: Isaac
    passport: 12345678A
    postcode: 07180
    province: Midlands
    surname: Newton


2018-10-16 20:03:11,643 DEBG 'zato-server2' stdout output:
2018-10-16 20:03:11,643 - INFO - 116:DummyThread-55 - guest.bookings:438 - Invoking booking.list...

2018-10-16 20:03:11,644 DEBG 'zato-server2' stdout output:
2018-10-16 20:03:11,644 - WARNING - 116:DummyThread-55 - zato.server.service:501 - Traceback (most recent call last):
  File "/opt/zato/3.0/code/zato-server/src/zato/server/service/__init__.py", line 476, in update_handle
    self._invoke(service, channel)
  File "/opt/zato/3.0/code/zato-server/src/zato/server/service/__init__.py", line 398, in _invoke
    service.handle()
  File "/opt/zato/env/qs-1/server2/work/hot-deploy/current/booking.py", line 324, in handle
    qs = parse_qs(self.wsgi_environ['QUERY_STRING'])
KeyError: u'QUERY_STRING'


2018-10-16 20:03:11,644 DEBG 'zato-server2' stdout output:
2018-10-16 20:03:11,644 - WARNING - 116:DummyThread-55 - zato.server.service:585 - Could not invoke `booking.list`, e:`Traceback (most recent call last):
  File "/opt/zato/3.0/code/zato-server/src/zato/server/service/__init__.py", line 583, in invoke_by_impl_name
    return self.update_handle(*invoke_args, **kwargs)
  File "/opt/zato/3.0/code/zato-server/src/zato/server/service/__init__.py", line 476, in update_handle
    self._invoke(service, channel)
  File "/opt/zato/3.0/code/zato-server/src/zato/server/service/__init__.py", line 398, in _invoke
    service.handle()
  File "/opt/zato/env/qs-1/server2/work/hot-deploy/current/booking.py", line 324, in handle
    qs = parse_qs(self.wsgi_environ['QUERY_STRING'])
KeyError: u'QUERY_STRING'
`

2018-10-16 20:03:11,645 DEBG 'zato-server2' stdout output:
2018-10-16 20:03:11,644 - WARNING - 116:DummyThread-55 - zato.server.service:501 - Traceback (most recent call last):
  File "/opt/zato/3.0/code/zato-server/src/zato/server/service/__init__.py", line 476, in update_handle
    self._invoke(service, channel)
  File "/opt/zato/3.0/code/zato-server/src/zato/server/service/__init__.py", line 398, in _invoke
    service.handle()
  File "/opt/zato/env/qs-1/server2/work/hot-deploy/current/guest.py", line 439, in handle
    bookings = self.invoke('booking.list', input_data, as_bunch=True)
  File "/opt/zato/3.0/code/zato-server/src/zato/server/service/__init__.py", line 601, in invoke
    return self.invoke_by_impl_name(self.server.service_store.name_to_impl_name[name], *args, **kwargs)
  File "/opt/zato/3.0/code/zato-server/src/zato/server/service/__init__.py", line 583, in invoke_by_impl_name
    return self.update_handle(*invoke_args, **kwargs)
  File "/opt/zato/3.0/code/zato-server/src/zato/server/service/__init__.py", line 476, in update_handle
    self._invoke(service, channel)
  File "/opt/zato/3.0/code/zato-server/src/zato/server/service/__init__.py", line 398, in _invoke
    service.handle()
  File "/opt/zato/env/qs-1/server2/work/hot-deploy/current/booking.py", line 324, in handle
    qs = parse_qs(self.wsgi_environ['QUERY_STRING'])
KeyError: u'QUERY_STRING'


2018-10-16 20:03:11,645 DEBG 'zato-server2' stdout output:
2018-10-16 20:03:11,645 - ERROR - 116:DummyThread-55 - zato.server.connection.http_soap.channel:324 - Caught an exception, cid:`2a58008b147d56da39c57ce7`, status_code:`500`, _format_exc:`Traceback (most recent call last):
  File "/opt/zato/3.0/code/zato-server/src/zato/server/connection/http_soap/channel.py", line 268, in dispatch
    payload, worker_store, self.simple_io_config, post_data, path_info, soap_action)
  File "/opt/zato/3.0/code/zato-server/src/zato/server/connection/http_soap/channel.py", line 502, in handle
    params_priority=channel_item.params_pri)
  File "/opt/zato/3.0/code/zato-server/src/zato/server/service/__init__.py", line 476, in update_handle
    self._invoke(service, channel)
  File "/opt/zato/3.0/code/zato-server/src/zato/server/service/__init__.py", line 398, in _invoke
    service.handle()
  File "/opt/zato/env/qs-1/server2/work/hot-deploy/current/guest.py", line 439, in handle
    bookings = self.invoke('booking.list', input_data, as_bunch=True)
  File "/opt/zato/3.0/code/zato-server/src/zato/server/service/__init__.py", line 601, in invoke
    return self.invoke_by_impl_name(self.server.service_store.name_to_impl_name[name], *args, **kwargs)
  File "/opt/zato/3.0/code/zato-server/src/zato/server/service/__init__.py", line 583, in invoke_by_impl_name
    return self.update_handle(*invoke_args, **kwargs)
  File "/opt/zato/3.0/code/zato-server/src/zato/server/service/__init__.py", line 476, in update_handle
    self._invoke(service, channel)
  File "/opt/zato/3.0/code/zato-server/src/zato/server/service/__init__.py", line 398, in _invoke
    service.handle()
  File "/opt/zato/env/qs-1/server2/work/hot-deploy/current/booking.py", line 324, in handle
    qs = parse_qs(self.wsgi_environ['QUERY_STRING'])
KeyError: u'QUERY_STRING'
`

It is clear that, since the service is being invoked, self.wsgi_environ[‘QUERY_STRING’] does not exist. So, how should I proceed?

Check whether self.wsgi_environ[‘QUERY_STRING’] exists before attempting to extract parameter values from it.
Otherwise, see if the parameters expected in the QUERY_STRING are available at self.request.input?

Incidentally, I’ve been trying to invoke the room.list service with multiple filters with no luck. If we were using curl, what I want to do would look like:

curl -v -g "http://127.0.0.1:11223/genesisng/rooms/list?filters=id|eq|1&filters=id|eq|2"

If I were to invoke it, how would I have to do it? Something like the following?

input_data = {'filters': ['id|eq|1', 'id|eq|2']}
rooms = self.invoke('room.list', input_data, as_bunch=True)

Then loop self.request.input.filters, same as I would do with qs['filters']?

Thanks in advance.

dsuch · 2018-10-17 11:41:25 UTC

Is there a particular reason why you are parsing it manually?

You are using SimpleIO, so self.request.input.filters will be populated correctly from either query string parameters or the dictionary passed to self.invoke, depending on how you invoke your List service.

Can you not use it?

jsabater · 2018-10-17 13:27:35 UTC

My fault. That’s what happens when you post when it’s late and you are tired. I had my code working with the query string, knowing to always expect parameters with lists of values and I had forgotten about it.

Now I’ve transformed the code to always use self.request.input whether it comes from a call through the channel or through an invokation of a service from another service and I’ve realised that self.request.input.filters is a string or a list depending on whether the parameter filters is passed more than once on the request.

So:

curl -v -g "http://127.0.0.1:11223/genesisng/bookings/list?filters=id_guest|eq|1&filters=guests|gt|0"; echo ""

Produces a value of id_guest|eq|1 (unicode) at self.request.input.filters, whereas:

curl -v -g "http://127.0.0.1:11223/genesisng/bookings/list?filters=id_guest|eq|1&filters=guests|gt|0"; echo ""

Produces a value of [u’id_guest|eq|1’, u’guests|gt|0’] (list) at self.request.input.filters.

I take it that this behaviour is a feature, not a bug, so I take it that I always have to test whether self.request.input.filters (or any other variable in the query string or the input data when invoking from another service) is a list or not, and act accordingly.

Correct?

dsuch · 2018-10-17 14:34:17 UTC

Personally, I prefer for query string parameters to be always extracted out of lists if there is only one, even if in theory each QS parameter can be provided multiple times. This is why this is the behaviour that you are observing.

I recognize your use-case though, hence I have just pushed a change that will auto-wrap single elements in lists, unless they are already list-like, if you change your definition to:

from zato.server.service import List

class SimpleIO:
  input_optional = List('filters'), ...

This needs to be explicit because otherwise Zato would not know if a given QS parameter can ever be a list or not, most are not.

For completeness, let me also add that:

You do not need to deal with self.wsgi_environ if you do not want to, you can also access all input HTTP parameters through self.request.http.params - this will combine all SIO and HTTP-based parameters, which includes QS
If you ever actually need to provide a WSGI environment to another service, you can do it by passing it as a keyword argument with self.invoke(name, request, wsgi_environ={'key':'value'}). It needs to be a dictionary.

jsabater · 2018-10-21 15:30:50 UTC

Hi, @dsuch. I just finished refactoring the listing services, which now make use of self.request.input only, plus the List type. Everything is working fine. Thanks for your help and for your time!

dsuch · 2018-10-21 15:50:14 UTC

This is good to hear - thank you for all the thought-provoking ideas and comments that you make!