Prefetch count configuration on AMQP channel


#1

Hello,

We are currently exploring on the AMQP Queues on Zato 3 for our application which uses Odoo for the backend. Right now with Zato 2.0.7, the queue gets emptied pretty fast pushing all the items into the consumer memory, so that we are forced to do this process in batches (so that the application won’t crash with the huge amount of data) and we are also losing data in case of any server restarts. We figured out that RabbitMQ has the facility to set a prefetch count in order to limit the data being pushed into the consumer memory and thus prevent the queue from getting emptied all at once. We are looking into it using Zato 3 right now.

After some exploration, I have succeeded in setting the prefetch count on the channel from the Zato dequeue service end (as I didn’t find any UI configuration option neither for the channel nor for the queue), as below.

self.request.amqp.msg.channel.basic_qos(prefetch_size=0, prefetch_count=1, a_global=False)

as you can see I am invoking the functionality on the kombu channel object.
But this would be set for every queue item processed, which doesn’t make sense. So I tweaked the Zato code to set the prefetch count at the consumer level, in this file.

/zato/code/zato-server/src/zato/server/connection/ampq_.py

def _get_consumer(self, _no_ack=no_ack, _gevent_sleep=sleep): 
""" Creates a new connection and consumer to an AMQP broker.  
""" 
     <some code>
     consumer.qos(prefetch_size=0, prefetch_count=20, apply_global=False) 
    consumer.consume() 

This works fine but tweaking the source code obviously is not the way to go, and it also doesn’t have any flexibility to configure differently depending on the queues/consumers. Is there any way to set the prefetch count for a channel or can this be made as a configuration on the channel level from UI (channel definition), so that all the consumers fetching data from a particular queue will adhere to that prefetch value?

Thanks,
Umesh


#2

Hi,

Do we have prefetch configurable in Zato3 for synchronous AMQP channel?

Thanks,
Prasantarmaye


#3

Hello,

there is no such functionality in 3.0.

Regards.


#4

Hi,

Would it be possible to give us the pull request so that we can make the changes for the prefetch count?

Thanks,
Prasantarmaye


#5

Hello @prasantarmaye,

using GitHub, you don’t need a prior approval to make a fork of our repository, work on the feature in your fork and then send a pull request to our repository back.

But it would be a good idea to discuss it first what you are planning to do exactly, this would let me point you into direction of similar functionality which could in turn let you implement the new one in a smooth way.