Order of task processing

[tsmgeek]

I started a post on gearmand about order of tasks being reverse to which they were added.
I see no note on the PHP docs that its LIFO instead of FIFO.

gearman/gearmand#319

[oleg-st]

Do you have code to reproduce the issue?

[tsmgeek]

Literally just the standard code from examples, add lots of tasks and sequentially increase the unique id.

This is my test code, WN_GearmanClient is just a wrapper around GearmanClient so I do not need to do addServer etc.

• Client

<?php
$gmclient = new WN_GearmanClient();

# set a function to be called when the work is complete
$gmclient->setCompleteCallback("complete");

# add a task to perform the "reverse" function on the string "Hello World!"
$gmclient->addTask("reverse", "Hello World!", null, "1");
$gmclient->addTask("reverse", "!dlroW olleH", null, "2");
$gmclient->addTask("reverse", "Hello World!", null, "3");
$gmclient->addTask("reverse", "!dlroW olleH", null, "4");
$gmclient->addTask("reverse", "Hello World!", null, "5");

# run the tasks
$gmclient->runTasks();

function complete($task)
{
  print "COMPLETE: UniqID-" . $task->unique() . ", Data- " . $task->data() . "\n";
}

• Client Output

COMPLETE: UniqID-5, Data- 1: !dlroW olleH
COMPLETE: UniqID-4, Data- 2: Hello World!
COMPLETE: UniqID-3, Data- 3: !dlroW olleH
COMPLETE: UniqID-2, Data- 4: Hello World!
COMPLETE: UniqID-1, Data- 5: !dlroW olleH

• Worker

<?php
$worker = new WN_GearmanWorker();

# define a variable to hold application data
$context = new stdClass();
$context->cnt=0;

# add the "reverse" function
$worker->addFunction("reverse", "reverse_cb", $context);

# start the worker
while ($worker->work());

function reverse_cb($job, $ctx)
{
  $ctx->cnt++;
  print "$ctx->cnt: " .$job->unique() ." - ". strrev($job->workload())."\n";
  return "$ctx->cnt: " . strrev($job->workload());
}

• Worker Output

1630176236.240423 INFO (6): Gearman addFunction - reverse
1: 5 - !dlroW olleH
2: 4 - Hello World!
3: 3 - !dlroW olleH
4: 2 - Hello World!
5: 1 - !dlroW olleH

[oleg-st]

I think addTask / runTasks are needed to run multiple tasks at the same time (to be handled by multiple workers). The FIFO order will be for batches of tasks, not for tasks in a single batch.
However, the php extension does nothing special here, it only calls gearman_client_add_task / gearman_client_run_tasks in the appropriate order.

[tsmgeek]

I've been told that tasks is just a concept of client library, the server itself just looks at them as individual jobs.

So what this is pointing too is an error in libgearman and maybe how it pushes the jobs to gearman itself.
But it's not even close to being in order, I understand that it is dependent on how quick workers respond, but I can post 1000 with a single worker and it will be returned in exact reverse.

[dmeziere]

I have the same problem than @tsmgeek , and would add even worse : Imagine a queue with 1000 jobs waiting, and say 100 runing. If i declare another 1000 jobs with addTasks(), these new jobs will be executed before the ones already in the queue. So the ones in the queue may be indefinitely pushed back by incoming jobs.