EzDevInfo.com

Do priorities work across queues/tubes in Beanstalkd?

I am a little confused as to whether priorities work at the job or queue/tube level. The reason I ask is that I am using a beanstalkd integration module in Drupal. This module enables one to define queues/tubes and assign a priority value to each queue/tube created.

What I am trying to work out is the following. Let's say I have two queues/tubes in Beanstalkd (queue A and queue B). If items assigned to Queue A have a higher priority than items in Queue B, does that mean items in Queue B will only be processed when Queue A is empty?

How to properly use an object in a PHP service

The specific scenario is this: a server running beanstalkd, using the PHP libary Pheanstalk. I have a PHP worker running as a service, so it theoretically never stops running.

The initial code for the worker is just a function call made to do the work. I wanted to shift that over to OOP. This would instantiate an object for every job that was processed. I imagine that without proper clean up, this would create a memory leak.

What would be the proper way to create/manage/destroy objects for use in this kind of scenario without causing memory leaks or excess system usage?

Class 'Pheanstalk_Pheanstalk' not found in Laravel 4 with Beanstalk

I'm trying to get beanstalk up and running and I'm getting the exception when trying to run Queue::push():

Class 'Pheanstalk_Pheanstalk' not found

I've run composer require pda/pheanstalk, and even regenerated my auto-load file an extra time.

It's happening in ./vendor/laravel/framework/src/Illuminate/Queue/Connectors/BeanstalkdConnector.php

Any ideas?

How to load Pheanstalk in PHP?

I'm trying to get Pheanstalk working for PHP but I can't load it.

I downloaded the source code from https://github.com/pda/pheanstalk, I moved src/Pheanstalk to my project directory, and then did the following in test.php:

use Pheanstalk\Pheanstalk;
$pheanstalk = new Pheanstalk('127.0.0.1');

But this gives me the following error:

Fatal error: Class 'Pheanstalk\Pheanstalk' not found in test.php on line 2

How do I just use Pheanstalk from the cloned git repository?

Beanstalkd: How to auto-bury a job that hits TTR?

In Beanstalkd, when a job hits TTR it will kill the worker handling the job and release the job back into the pool. In our system, this should be a super-rare event, and in that case I would like to just bury the job instead.

How would one go about doing this?

(If it matters, we are using PHP + Pheanstalk)

Beanstalk Getting Socket error 110: Connection timed out

I am using Beanstalk server with Pheanstalk Connection in php application.

Previously it was fine, but suddenly it is getting error

Critical: Fatal. Socket error 110: Connection timed out

Any help would be appreciated.

Thanks

Laravel 4.1 mail queueing with beanstalkd driver - "Insufficient data for unserializing"

I'm working on my first Laravel app, and hit an odd problem when trying to queue up email sending using Mail::queue to send emails.

I was originally using the sync driver and everything worked fine, however having now switched over to the beanstalkd driver it's stopped sending my emails.

My config is as follows:

• Ubuntu 13.10
• Laravel 4.1
• PHP 5.5.3
• Beanstalkd 1.8 installed via apt-get
• "pda/pheanstalk": "2.1.*" in composer.json

What I've noticed is that every time I run

$ php artisan queue:work

I'm getting:

  [ErrorException]                                                  
  Insufficient data for unserializing - 2570 required, 168 present  


queue:work [--queue[="..."]] [--delay[="..."]] [--force] [--memory[="..."]] [--sleep[="..."]] [--tries[="..."]] [connection]

I've tried selecting the redis driver and that wasn't giving me the error, so clearly there is something wrong with how pheanstalk is running on my machine.

I've tried changing the pheanstalk version in composer.json, but I'm still getting the same problem no matter which version I seem to use.

Problem is, this issue doesn't seem particularly widely documented, and the error message doesn't really give much away at all...

Any suggesions?

Beanstalk + Pheanstalk Delay Existing Job

I'm intermediate in Beanstalk + Pheanstalk. I'm developing my core part (background process) using Beanstalk. It's excellent tool that's no doubt but I'm stuck in one situation. I'm using beanstalk using Yii framework.

Let's say, I have two jobs A (Tube - A-Jobs) and B (Tube - B-Jobs) and both are under different tube and workers. B is dependent on A. It may possible that B will fetch first by worker before A and will start process on it. In that case, I need B to delay for some seconds (120 secs) untill A will execute.

As per my knowledge, there isn't any direct way to make job delay. For this you need to delete job B and put back to same queue with delay time. Correct me if I'm wrong here.

Now situation is that, I'm not able to get actual tube name in that case, because pheanstalk only providing Job Id and Job Data. I'm pasting you my code below,

$pheanstalkA = Yii::app()->beanstalk->getClient($client);
$pheanstalkA->watch('A-tube');

$pheanstalkB = Yii::app()->beanstalk->getClient($client);
$pheanstalkB->watch('B-tube');

Now, if I got any job using reserve function,

$jobB = $pheanstalkB->reserve();

It will give you output as follow, and we can see that beanstalk is not providing tube name from which it fetched this job.

Pheanstalk_Job Object
(
    [_id:Pheanstalk_Job:private] => 2
    [_data:Pheanstalk_Job:private] => Job Data ....
)

As we know that, single worker can watch in N tube(s), so if I want to delay this job and want to put under same tube. How can we find job's tube name?

Thanks...

Beanstalkd (via pheanstalk) allowing duplicate, simultaneous reserves?

Trying to wrap Pheanstalk in my PHP job base class. I'm testing the reserve and reserve with delay functionality and I've found that I can reserve a job from a second instances of my base class without the first instance releasing the job or the TTR timing out. This is unexpected since I was thinking this is exactly the thing job queues are supposed to prevent. Here are the beanstalkd commands for the first put and the first reserve along with time stamps. I also do a stats-job request at the end:

01:40:15: Sending command: use QueuedCoreEvent
01:40:15: Got response: USING QueuedCoreEvent

01:40:15: Sending command: put 1024 0 300 233
a:4:{s:9:"eventName";s:21:"ReQueueJob_eawu7xr9bi";s:6:"params";a:2:{s:12:"InstanceName";s:21:"ReQueueJob_eawu7xr9bi";s:17:"aValueToIncrement";i:123456;}s:9:"behaviors";a:1:{i:0;s:22:"BehMCoreEventTestDummy";}s:12:"failureCount";i:0;}
01:40:15: Got response: INSERTED 10

01:40:15: Sending command: watch QueuedCoreEvent
01:40:15: Got response: WATCHING 2

01:40:15: Sending command: ignore default
01:40:15: Got response: WATCHING 1

01:40:15: Sending command: reserve-with-timeout 0
01:40:15: Got response: RESERVED 10 233
01:40:15: Data: a:4:{s:9:"eventName";s:21:"ReQueueJob_eawu7xr9bi";s:6:"params";a:2:{s:12:"InstanceName";s:21:"ReQueueJob_eawu7xr9bi";s:17:"aValueToIncrement";i:123456;}s:9:"behaviors";a:1:{i:0;s:22:"BehMCoreEventTestDummy";}s:12:"failureCount";i:0;}

01:40:15: Sending command: stats-job 10
01:40:15: Got response: OK 162
01:40:15: Data: ---
id: 10
tube: QueuedCoreEvent
state: reserved
pri: 1024
age: 0
delay: 0
ttr: 300
time-left: 299
file: 0
reserves: 1
timeouts: 0
releases: 0
buries: 0
kicks: 0

So far, so good. Now I do another reserve from a second instance of my base class followed by another stats-job request. Notice the time stamps are within the same second, nowhere near the 300 second TTR I've set. Also notice in this second stats-job printout that there are 2 reserves of this job with 0 timeouts and 0 releases.

01:40:15: Sending command: watch QueuedCoreEvent
01:40:15: Got response: WATCHING 2

01:40:15: Sending command: ignore default
01:40:15: Got response: WATCHING 1

01:40:15: Sending command: reserve-with-timeout 0
01:40:15: Got response: RESERVED 10 233
01:40:15: Data: a:4:{s:9:"eventName";s:21:"ReQueueJob_eawu7xr9bi";s:6:"params";a:2:{s:12:"InstanceName";s:21:"ReQueueJob_eawu7xr9bi";s:17:"aValueToIncrement";i:123456;}s:9:"behaviors";a:1:{i:0;s:22:"BehMCoreEventTestDummy";}s:12:"failureCount";i:0;}

01:40:15: Sending command: stats-job 10
01:40:15: Got response: OK 162
01:40:15: Data: ---
id: 10
tube: QueuedCoreEvent
state: reserved
pri: 1024
age: 0
delay: 0
ttr: 300
time-left: 299
file: 0
reserves: 2
timeouts: 0
releases: 0
buries: 0
kicks: 0

Anyone have any ideas on what I might be doing wrong? Is there something I have to do to tell the queue I want jobs to only be accessed by one worker at a time? I'm doing an "unset" on the pheanstalk instance as soon as I get the job off the queue which I believe terminates the session with beanstalkd. Could this cause beanstalkd to decide the worker has died and automatically release the job without a timeout? I'm uncertain of how much beanstalkd relies on session state to determine worker state. I was assuming that I could open and close sessions with impunity and that job id was the only thing that beanstalkd cared about to tie job operations together, but that may have been foolish on my part... This is my first foray into job queues.

Thanks!

Beanstalkd / Pheanstalk - Cant get it working at all! (WAMP/PHP/LARAVEL)

So im trying to get Pheanstalk queue working in Laravel 4, which is built in (i had to composer instal the beanstalkd lib).

Now thats done... im trying to send jobs to my workers. Everytime i do i get this error

[2013-04-25 08:55:03] log.ERROR: exception 'Symfony\Component\Debug\Exception\FatalErrorException' with message 'Error: Maximum execution time of 30 seconds exceeded in C:\wamp\www\adam\L4\vendor\pda\pheanstalk\classes\Pheanstalk\Socket\StreamFunctions.php line 55' in C:\wamp\www\adam\L4\vendor\pda\pheanstalk\classes\Pheanstalk\Socket\StreamFunctions.php:55
Stack trace:
#0 C:\wamp\www\adam\L4\bootstrap\compiled.php(5006): Symfony\Component\Debug\ErrorHandler->handleFatal()
#1 [internal function]: Illuminate\Exception\ExceptionServiceProvider->Illuminate\Exception\{closure}()
#2 {main} [] []

I also get this error in my log:

exception 'ErrorException' with message 'Catchable Fatal Error: Argument 1 passed to Illuminate\Queue\Jobs\Job::resolveAndFire() must be of the type array, null given

Which seems to be looped as soon as i run 'php artisan queue:listen'.

I have downloaded the console for beanstalkd and configure it to listen to server localhost:11300.

Which gives me an error: unhandled response.

I have changed my httpd.conf and added Listen 11300 to it and restarted.

Any ideas where im going wrong? I cant see much docs for Beanstalkd and the Laravel docs dont help either.

Thanks for any ideas!

laravel mail queueing - Insufficient data for unserializing

I am using

Ubuntu
laravel 4.2
beanstalked

when i try to

php artisan queue:work

it returns

 [ErrorException]                                                  
  Insufficient data for unserializing - 1403 required, 218 present  

mail function (confide package)

Mail::queueOn(
                    Config::get('confide::email_queue'),
                    Config::get('confide::email_account_confirmation'),
                    compact('user'),
                    function ($message) use ($user) {
                        $message
                            ->to($user->email, $user->username)
                            ->subject(Lang::get('confide::confide.email.account_confirmation.subject'));
                    }
                );

Pheanstalk in Laravel, Everything ends up as an unhandled response?

Ive installed the Pheanstalk bundle for Laravel (https://github.com/mikelbring/Pheanstalk) I am now following the basic example code to try and pass a job to the queue and retrieve it back again.

This is my code:

Route::get('queue', function()
{

Pheanstalk::useTube('testtube')->put("job payload goes here\n");

$pheanstalk = Pheanstalk::connection();

$job = $pheanstalk
  ->watch('testtube')
  ->ignore('default')
  ->reserve();

echo $job->getData();

$pheanstalk->delete($job);
});

Yet when i run it, i get an 'Unhandled Response' error, like nothing has been returned.

Unhandled Exception

Message:

Unhandled response: 
Location:

C:\wamp\www\adam\phire\bundles\pheanstalk\pheanstalk\classes\Pheanstalk\Command\PutCommand.php on line 108
Stack Trace:

 #0 C:\wamp\www\adam\phire\bundles\pheanstalk\pheanstalk\classes\Pheanstalk\Connection.php(121): Pheanstalk_Command_PutCommand->parseResponse('', NULL)
#1 C:\wamp\www\adam\phire\bundles\pheanstalk\pheanstalk\classes\Pheanstalk.php(511): Pheanstalk_Connection->dispatchCommand(Object(Pheanstalk_Command_PutCommand))
#2 C:\wamp\www\adam\phire\bundles\pheanstalk\pheanstalk\classes\Pheanstalk.php(273): Pheanstalk\Pheanstalk->_dispatch(Object(Pheanstalk_Command_PutCommand))
#3 C:\wamp\www\adam\phire\application\routes.php(45): Pheanstalk\Pheanstalk->put('job payload goe...')
#4 [internal function]: Laravel\Bundle::{closure}()
#5 C:\wamp\www\adam\phire\laravel\routing\route.php(163): call_user_func_array(Object(Closure), Array)
#6 C:\wamp\www\adam\phire\laravel\routing\route.php(124): Laravel\Routing\Route->response()
#7 C:\wamp\www\adam\phire\laravel\laravel.php(167): Laravel\Routing\Route->call()
#8 C:\wamp\www\adam\phire\public\index.php(34): require('C:\wamp\www\ada...')
#9 {main}

Is anyone familiar with Pheanstalk and can provide some assistance?

Thanks Adam.

JOB_TOO_BIG Pheanstalk - what can be done?

On Laravel 4.2 & Laravel Forge

I Made a mistake and accidentally pushed some code on to the production sever, but there was a bug and it pushed a job to the queue without deleting it once done. Now I can't push anything in the queue anymore, I get:

Pheanstalk_Exception JOB_TOO_BIG: job data exceeds server-enforced limit

What can I do?

List all the jobs in Beanstalkd

We have beanstalkd running in our server. All our application records are stored in beanstalk before they are written into the mysql tables. Now few items got stuck inside this beanstalk queue. They are not being written into the database. So I would like to fetch all the items which are stuck in this Queue so that we can analyse further to see if they are corrupt entries. I could not find any way to list all the items from beanstalkd. Is there any way I can do it?

Pheanstalk touch job to ask more time

How do I make my worker ask for more time to work on the job ?

I am using the pheanstalk library. Work is a parsing of a file which depending on the file may take more than 60 secs, the default ttr.

while( $job = $pheanstalk->watch($tubeName)->reserve() ){ 
        // get the job body
        $parserExec = unserialize( $job->getData() );

        // execute parser - will take more than 1 min most of the time
        exec( $parserExec['command'] ); 

        // rest of the stuff - delete job etc
}

I am new to queue stuff and I have been going through the examples I found online. If I am doing this wrong, please do tell. I tried to get the jobstats and write a conditional based on the time-left value but I could not get it to work. When I use the jobStats function in pheanstalk the result is a private object array. What I want is to have the script wait till the exec is complete however long that may take.