Pubsub issue with zato 3.0 with topics beginnig with "/"


#1

Hello,

I currently migrate some services from Zato 2.0 to zato 3.0 on Pubsub
and I am facing an issue on Zato 3.0.

Context :

  • Services publishe messages to a topic with pubsub.publish() method
  • A REST service is used to collect each days messages on topics
    Quit simple.

Issue in Zato 3.0 is about topic syntax :

  • I can publish from a service with pubsub.publish method() on topic “/test/topic1” but not “Test.Topic1”
  • I can call a REST service (by example with curl) on topic “Test.Topic1” but not “/test/topic1”

I made some tests on different configurations and with different versions of Zato 3.0 (including latest with a git pull).

Is there any way to activate pubsub.publish method on topic “Test.topics1”
or better to get working REST services with topic that contain a leading “/” ?

Thank you for your anwser,
Best regards,


#2

Hello @bip68,

can you please clarify what you mean by the inability to use such topic names?

For both cases, please provide the exact code that you use to publish messages along with extracts from server.log showing exceptions or any other information that seems related to what prevents you from the publication.

Thank you.


#3

Topics definition :

First tests from services (for the demo, done on Publish defaut on topic) :

  • publish on /test/topics1 works fine when publisher is zato.pubsub.internal.endpoint
  • publish on Test/Topics1 does not work because zato.pubsub.internal.endpoint has sub=/,
    but cannot be modified to sub=
    or add sub=Test.Topics1

Second tests with curl :

< HTTP/1.1 200 OK
< Server: Zato
< Content-Type: application/json
< X-Zato-CID: 1a48851818534d82d479eb89
<

  • Connection #0 to host localhost left intact
    []
  • upload completely sent off: 53 out of 53 bytes

< HTTP/1.1 404 Not Found
< X-Zato-CID: 383ca402a26aa95e5b03c512
<

  • Connection #0 to host localhost left intact
    CID:383ca402a26aa95e5b03c512 Unknown URL:/zato/pubsub/topic//test/topic1 or SOAP action:``

#4

Thank you @bip68 - I am setting it all up locally and I have a quick question.

One of the topics is called Test.Topic1 but endpoint ‘test’ has permissions to publish to Test.topic1 instead (lowercase topic). Was that intentional and should it not be pub=Test.Topic1 in permissions?


#5

Interresting, It was not intentionnal, just a mistake when I build this new test.
Test was working fine with curl -v -XPOST http://test:test@localhost:17011/zato/pubsub/topic/Test.Topic1 . So I seems that pub= instruction is not case sensitive, isn’t it ?

For http://test:test@localhost:17011/zato/pubsub/topic//test/topic1,
it seems that zato does not recognize URL as /zato/pubsub/topic/{topic}
but as a standard channel URL … obviously not found (404)

Does my zato(s) could lack some definitions to identify this pattern ?


#6

I am trying to reproduce the situation and I will get back to you as soon as I know more details.

I have another side question, though, can you tell me what OS and web browser you are using? This is not strictly related to the situation at hand, I just do not recognize what the browser is.


#7

Hi @bip68,

I think there were two matters here.

One was that you had a typo in the topic’s name in pub permissions and that prevented you from publications in web-admin.

The other one was that REST endpoints for pub/sub did not have slash character matching explicitly enabled which prevented you from using topic names starting with or containing slashes. I have just pushed a fix for it and you can install it.

I believe that you are using a fresh Zato 3.0 environment so after installing the updates the easiest way to apply the fix will be to create a new cluster.

If you need anything else, just let me know.


#8

Hi dsuch,

Thank you for your quick patch.
I have setup a fresh Zato 3.0 + update (mysql Quickstart + git pull + install)
on unbuntu 16.04 (my browser is latest Firefox).
(As Quickstart execute create cluster, I suppose that is OK to create cluster)

I have created :

  • a topic : /test/topic1
  • an endpoint : test - REST - Subscriber - Auth/test - sub=/test/*
  • Subscription can be created :
    image

But curl still complain that url is not found :

zato@zatotest:~/env$ curl -v http://test:test@localhost:17010/zato/pubsub/topic//test/topic1 -d ‘{“sub_key”:“zpsk.rest.None.7920622026e389dcc6181941”}’; echo

  • Trying ::1…
  • TCP_NODELAY set
  • connect to ::1 port 17010 failed: Connection refused
  • Trying 127.0.0.1…
  • TCP_NODELAY set
  • Connected to localhost (127.0.0.1) port 17010 (#0)
  • Server auth using Basic with user ‘test’

POST /zato/pubsub/topic//test/topic1 HTTP/1.1
Host: localhost:17010
Authorization: Basic dGVzdDp0ZXN0
User-Agent: curl/7.58.0
Accept: /
Content-Length: 67
Content-Type: application/x-www-form-urlencoded

  • upload completely sent off: 67 out of 67 bytes
    < HTTP/1.1 404 Not Found
    < Server: Zato
    < Date: Fri, 07 Dec 2018 19:03:54 GMT
    < Connection: keep-alive
    < Transfer-Encoding: chunked
    < X-Zato-CID: f0806ca59286e8d6ce621955
    <
  • Connection #0 to host localhost left intact
    CID:f0806ca59286e8d6ce621955 Unknown URL:/zato/pubsub/topic//test/topic1 or SOAP action:``

I also try with demo/sample without success :
curl -v http://zato.pubsub.demo:azerty@localhost:17010/zato/pubsub/topic//zato/demo/sample -d ‘{“sub_key”:“zpsk.rest.zeci.d25ab836a3dbfeb8e5d6ba59”}’; echo
=> 404 ; Unknown URL:/zato/pubsub/topic//zato/demo/sample

After a lokk at the DB, It seems ok for pubsub infomation in pubsub_topic, pubsub_sub.

perhaps, can I activate some logs in debug mode to identify issue ?

Thank you for your help,


#9

Hi @bip68,

if this is a new cluster then your configuration is correct.

Can you please post the result of this SQL query against your MySQL server?

SELECT name, url_path, opaque1 from http_soap WHERE name LIKE '%pubsub.topic.topic_name%'

Also, can you post the output from this command?

$ cd /opt/zato/current
$ git log -1

#10

MariaDB [fimetest]> SELECT name, url_path, opaque1 from http_soap WHERE name LIKE ‘%pubsub.topic.topic_name%’;
±-----------------------------±--------------------------------±--------------------------+
| name | url_path | opaque1 |
±-----------------------------±--------------------------------±--------------------------+
| zato.pubsub.topic.topic_name | /zato/pubsub/topic/{topic_name} | “{“match_slash”: true}” |
±-----------------------------±--------------------------------±--------------------------+
1 row in set (0.01 sec)

zato@zatotest:~/current$ git log -1
commit 7a193ecde7564425a31ee669f0d3c1e6aa2239d5 (HEAD -> support/3.0, origin/support/3.0)
Merge: 952bca43 5e0ff32d
Author: Dariusz Suchojad dsuch-github@m.zato.io
Date: Thu Dec 6 19:27:40 2018 +0100

Merge branch 'pre-support/3.0' into support/3.0

#11

Thanks again for the details. Can you pull the latest updates and try again? There is no need to recreate the environment, just stop and stop the servers per the instructions:

https://zato.io/docs/admin/guide/install/update.html


#12

Hi dsuch,

Thank you very much for this fix.
Zato is really a wonderful piece of code and support is perfect :slight_smile:
I hope migrate soon all my services build last year from V2.0 to V3.0.

I have a few more questions if I may.
I plan to have 2 zatos to manage 2 zones (with 2 backups for the 2 zones), and 2 others for dev and Qa
- Should I share the same ca for all zato (for failover backup, it is obvious, but for the others ?)
- For QA, I add some consumers to topics to push production data to QA.
Do you have other working strategies ?

by the way, I have found an other small issue when you try to see the message from subscription screen :
image

I make the following quick hack that is working for me if this can help.
zato@zatotest:~/3.0/code/zato-server/src/zato/server/service/internal/pubsub$ diff endpoint.py.old endpoint.py
269c269
< if item.last_interaction_time:

        if 'last_iteraction_time' in item and item.last_interaction_time:

thank you again for your support,


#13

Thank you @bip68.

As for the CA, this is more of an organizational question, i.e. I would recommend to set it up according to your organization’s policies because as far as the architecture goes, there is no difference for Zato as long as there is some CA that produces / signs the relevant keys and certificates.

Regards.


#14

Ok, I understand
Thank you very much.

Regards,