(Migrated) Pubsub issue

(This message has been automatically imported from the retired mailing list)

Hello -

I’m running into an issue with pub/sub in a fairly vanilla setup that
follows the tutorial.

I’ve configured the below topic / publisher / basic http auth

Then, when I try to access it:

$ curl -XPOST
http://192.168.2.199:11223/zato/pubsub/topic//vendor/product/create/ -d
“here” -u vendornewproducts:password
[K07NDMXG51H5T75SZ36JH72QSHYE] Unknown
URL:[/zato/pubsub/topic//vendor/product/create/] or SOAP action:[]

I can successfully publish a message through a Zato service or directly
through the admin interface, so I think the publisher is setup correctly.

Does anyone have a suggestion where my stupid self went wrong? Otherwise,
I’m really pumped about using Zato.

Thanks,
David

Sure thing - thanks for having a look.

KEYS zato:pubsub:*:

“[‘zato:pubsub:hash:last-seen-producer’, ‘zato:pubsub:hash:msg-metadata’,
‘zato:pubsub:hash:last-pub-time’]”

David Schenz
davidschenz@gmail.com
513-429-8884

On Sat, Feb 6, 2016 at 1:16 PM, Dariusz Suchojad dsuch@zato.io wrote:

On 06/02/16 18:41, David Schenz wrote:

$ curl -XPOST
http://192.168.2.199:11223/zato/pubsub/topic//vendor/product/create/ -d
"here" -u vendornewproducts:password
[K07NDMXG51H5T75SZ36JH72QSHYE] Unknown
URL:[/zato/pubsub/topic//vendor/product/create/] or SOAP action:[]

I can successfully publish a message through a Zato service or directly
through the admin interface, so I think the publisher is setup correctly.

Thanks David, I will be looking into it now.

In the meantime, can you please post the result of issuing this Redis
command?

KEYS zato:pubsub:*

https://zato.io/docs/web-admin/kvdb/remote-commands.html

thanks,


Dariusz Suchojad

https://zato.io
ESB, SOA, REST, APIs and Cloud Integrations in Python

Sure:

HGETALL ‘zato:pubsub:hash:msg-metadata’

“(None)”

David Schenz
davidschenz@gmail.com
513-429-8884

On Sat, Feb 6, 2016 at 1:20 PM, Dariusz Suchojad dsuch@zato.io wrote:

On 06/02/16 19:19, David Schenz wrote:

Sure thing - thanks for having a look.

KEYS zato:pubsub:*:

“[‘zato:pubsub:hash:last-seen-producer’,
‘zato:pubsub:hash:msg-metadata’, ‘zato:pubsub:hash:last-pub-time’]”

Great, could you also add this one?

HGETALL ‘zato:pubsub:hash:msg-metadata’


Dariusz Suchojad

https://zato.io
ESB, SOA, REST, APIs and Cloud Integrations in Python

Thanks, that makes sense. I switched to using periods but I also hit the
same problem:

$ curl -XPOST
http://192.168.2.199:11223/zato/pubsub/topic/vendor.product.create -d
"here" -u vendornewproducts:password
[K041M0C2XMS9V35564FBR8ET78Z4] Unknown
URL:[/zato/pubsub/topic/vendor.product.create] or SOAP action:[]

I tried the same thing with CamelCase, and hit the same error.

Settings for the topic are here: http://imgur.com/aGBRnl2

Thank you very much for having a look.

Thanks,
David

David Schenz
davidschenz@gmail.com
513-429-8884

On Sat, Feb 6, 2016 at 1:45 PM, Dariusz Suchojad dsuch@zato.io wrote:

On 06/02/16 19:23, David Schenz wrote:

HGETALL ‘zato:pubsub:hash:msg-metadata’

“(None)”

Mhm, that is fine - messages sent from web-admin or services already
expired, right.

OK, I know what this is and here is the full story.

Our URL matcher is here:

https://github.com/zatosource/zato/blob/master/code/zato-server/src/zato/server/connection/http_soap/url_data.py#L45

It is very fast and that is due to the fact that matching is done using
the ‘regex’ library https://pypi.python.org/pypi/regex.

Now, a very popular assumption is the ability to have clean URLs in HTTP
requests, such as:

/order/{order_id}/customer/{cust_id}/

Thanks to SimpleIO https://zato.io/docs/progguide/examples/sio.html this
gets translated to:

class MyService(Service):
class SimpleIO:
input_required = (order_id, cust_id)

def handle(self):
self.logger.info(‘Order ID %s’, self.request.input.order_id)
self.logger.info(‘Cust ID %s’, self.request.input.cust_id)

On top of that, pretty much everyone expects for the ‘/’ character in
URLs to separate segments of data, such as order_id and cust_id above.

In other words, slash is a special characters in URLs and this special
meaning was attached to it because of the expectations regarding clean
URLs.

However, this feature was implemented after documentation for pub/sub
was written so the latter still uses ‘/’ as though they were permitted
where in fact they are not allowed anymore.

So my recommendation is to rename your topic to 'vendor.product.create’
which allows for the same kind of namespace separation and documentation
for pub/sub will be amended.

As for why it would work in web-admin - there is no URL pattern matching
there, web-admin uses semi-internal APIs so that is why it never
surfaced in your tests.

thanks a lot,


Dariusz Suchojad

https://zato.io
ESB, SOA, REST, APIs and Cloud Integrations in Python

Ahhh! Perfect, that did the trick. Thank you very much for your help!

David Schenz
davidschenz@gmail.com
513-429-8884

On Sat, Feb 6, 2016 at 3:04 PM, Dariusz Suchojad dsuch@zato.io wrote:

On 06/02/16 20:53, David Schenz wrote:

Thanks, that makes sense. I switched to using periods but I also hit the
same problem:

$ curl -XPOST
http://192.168.2.199:11223/zato/pubsub/topic/vendor.product.create -d
"here" -u vendornewproducts:password
[K041M0C2XMS9V35564FBR8ET78Z4] Unknown
URL:[/zato/pubsub/topic/vendor.product.create] or SOAP action:[]

Hi David,

please note that the pattern is still /zato/pubsub/topic/{name}/ i.e.
the trailing / is required precisely because it signifies that {name}
segment ends there.

Can you try it out?

On 06/02/16 18:41, David Schenz wrote:

$ curl -XPOST
http://192.168.2.199:11223/zato/pubsub/topic//vendor/product/create/ -d
"here" -u vendornewproducts:password
[K07NDMXG51H5T75SZ36JH72QSHYE] Unknown
URL:[/zato/pubsub/topic//vendor/product/create/] or SOAP action:[]

I can successfully publish a message through a Zato service or directly
through the admin interface, so I think the publisher is setup correctly.

Thanks David, I will be looking into it now.

In the meantime, can you please post the result of issuing this Redis
command?

KEYS zato:pubsub:*

https://zato.io/docs/web-admin/kvdb/remote-commands.html

thanks,

On 06/02/16 19:19, David Schenz wrote:

Sure thing - thanks for having a look.

KEYS zato:pubsub:*:

“[‘zato:pubsub:hash:last-seen-producer’,
‘zato:pubsub:hash:msg-metadata’, ‘zato:pubsub:hash:last-pub-time’]”

Great, could you also add this one?

HGETALL ‘zato:pubsub:hash:msg-metadata’

On 06/02/16 19:23, David Schenz wrote:

HGETALL ‘zato:pubsub:hash:msg-metadata’

“(None)”

Mhm, that is fine - messages sent from web-admin or services already
expired, right.

OK, I know what this is and here is the full story.

Our URL matcher is here:

https://github.com/zatosource/zato/blob/master/code/zato-server/src/zato/server/connection/http_soap/url_data.py#L45

It is very fast and that is due to the fact that matching is done using
the ‘regex’ library https://pypi.python.org/pypi/regex.

Now, a very popular assumption is the ability to have clean URLs in HTTP
requests, such as:

/order/{order_id}/customer/{cust_id}/

Thanks to SimpleIO https://zato.io/docs/progguide/examples/sio.html this
gets translated to:

class MyService(Service):
class SimpleIO:
input_required = (order_id, cust_id)

def handle(self):
self.logger.info(‘Order ID %s’, self.request.input.order_id)
self.logger.info(‘Cust ID %s’, self.request.input.cust_id)

On top of that, pretty much everyone expects for the ‘/’ character in
URLs to separate segments of data, such as order_id and cust_id above.

In other words, slash is a special characters in URLs and this special
meaning was attached to it because of the expectations regarding clean URLs.

However, this feature was implemented after documentation for pub/sub
was written so the latter still uses ‘/’ as though they were permitted
where in fact they are not allowed anymore.

So my recommendation is to rename your topic to 'vendor.product.create’
which allows for the same kind of namespace separation and documentation
for pub/sub will be amended.

As for why it would work in web-admin - there is no URL pattern matching
there, web-admin uses semi-internal APIs so that is why it never
surfaced in your tests.

thanks a lot,

On 06/02/16 20:53, David Schenz wrote:

Thanks, that makes sense. I switched to using periods but I also hit the
same problem:

$ curl -XPOST
http://192.168.2.199:11223/zato/pubsub/topic/vendor.product.create -d
"here" -u vendornewproducts:password
[K041M0C2XMS9V35564FBR8ET78Z4] Unknown
URL:[/zato/pubsub/topic/vendor.product.create] or SOAP action:[]

Hi David,

please note that the pattern is still /zato/pubsub/topic/{name}/ i.e.
the trailing / is required precisely because it signifies that {name}
segment ends there.

Can you try it out?