EzDevInfo.com

Beanstalkc Timeout Question

I am using beanstalkc in Python for a queuing process for a program which has to parse a list of URLs. Hence I am using timeout in beanstalk for avoiding huge time consumption by any URL. But even after using it my process doesn't time out in the limit and is taking a lot of time for parsing few URLs. I am using the following code:

for seed in seedlist:
    print 'Put data: %s' % seed
    bean.put(seed,ttr =5)
while True: 
    job = bean.reserve() 
    spider.spider(job.body)
    print 'Got data: %s' % job.body

Reload beanstalkd configuration without restaring

Is there a way to reload the beanstalkd config without actually restarting beanstalkd?

When I use sudo service beanstalkd restart or sudo service beanstalkd force-reload it clears all the queues.

(Yes, in the future I will run it in persistent mode)

Search in beanstalk queue

I have a use case which requires placing two different types of jobs in a beanstalk queue, say type a and type b. I put type a job whenever a new one arrives, but for type b I want that there should be a single job of type b in a queue at a time (there should never be two or more jobs of type b in queue). When I go for inserting type b job in queue, I first want to check if there is any type b job already in queue? If it is, then delay that job and don't insert a new one. If there is no type b job in queue, insert a new one. So is it possible to search a job in beanstalk queue?

How to use beanstalkc in Python to queue URLs and perform jobs

I have a function named spider which takes seed as an argument. seed is the name of the URL I send to the spider function. Now my question is how do I use beanstalkc in Python to queue the URLs and perform the jobs.

Getting jobs from beanstalkd - timed out exception

Thanks for reading.

I am using Python 2.7, beanstalkd server with beanstalkc as the client library.

It takes about 500 to 1500 ms to process each job, depending on the size of the job.

I have a cron job that will keep adding jobs to the beanstalkd queue and a "worker" that will run in an infinite loop getting jobs and processing them.

eg:

def get_job(self):
    while True:
        job = self.beanstalk.reserve(timeout=0)
        if job is None:
            timeout = 10 #seconds
            continue
        else:
            timeout = 0 #seconds
            self.process_job(job)

This results in "timed out" exception.

Is this the best practice to pull a job from the queue?

Could someone please help me out here?

Non-Blocking WebSocketHandler while receiving jobs from a queue

Setup:

• Tornado HTTP/WebSocket Server. WebSocketHandler reacts on messages from the client (e.g. put them in the job-queue)
• A beanstalk job-queue which sends jobs to the different components
• Some other components communicating over beanstalk, but those are unrelated to my problem.

Problem:

• WebSocketHandler should react on jobs, but if he is listening on beanstalk, its blocking. A job could be e.g. 'send data xy to client xyz'

How can this be solved nicely? My first approach was running a jobqueue-listener in a separate thread which contained a list of the pickled WebSocketHandler. All should be stored in a redis-db. Since WebsocketHandler can't be pickled (and this approach seems to be very ugly) I'm searching for another solution.

Any ideas?

When I use the 'reserve' operation in beanstalked, it is blocked for a long time

I need to use beanstalked in my project.So I download the 1.1 version. I use the beanstalkc for python client. A simple Producer-Consumer model.

Producer:

       import beanstalkc
       beanstalk = beanstalkc.Connection('localhost')
       beanstalk.use('foo')
       beanstalk.put('hello')
       beanstalk.put('world')

Consumer:

      import beanstalkc
      beanstalk.use('foo')
      beanstalk = beanstalkc.Connection('localhost')
      job = beanstalk.reserve()

But the reserve operation is blocked for a long time.Has anybody encounter this problem?

Connection refused in Django_beanstalkd

I'm very stuck with this. I have an app in Django that uses beanstalkd to establish a connection with vlcserver. Vlcserver captures video from an rtsp stream of an IP camera, and after that, that video is transcoded to a h264 format. The problem comes when I try to establish a connection with the beanstalkd server.

beanstalkc.Connection('127.0.0.1', 11300)

When this instruction is executed, it appears an exception: [errno 111] Connection refused. The port is open in the firewall. I don't know what I can do to fix this. Help please.

Thank you in advance.

Check if all reserved jobs have been completed using beanstalkc

I want one of my scripts to know if all of the remaining queued jobs in beanstalkd have been completed.

I'm currently planning to iterate through every job id and run peek() to see if anything is returned.

Is there a more elegant way to do this?

Change beanstalkd default TTR

I run beanstalkd as a service using the standard /etc/default/beanstalkd.

Sometimes my code throws a NOT_FOUND error when I try to delete a job, because it got released due to exceeding the TTR. I would like to increase the TTR for all jobs that get inserted into the tubes.

Is there a way to set a default TTR for beanstalkd jobs? My guess is that I can change this somewhere in /etc/default/beanstalkd, but I couldn't find this in the beanstalkd docs.