Is there a way to run Solid Queue alongside Falcon?

[izaguirrejoe]

Hello,

Loving Falcon so far. I want to use Solid Queue as my Active Job backend. Is there a way to run Falcon alongside SolidQueue, similar to how you can run Puma with Solid Queue using the official Puma plugin? I prefer not to have a separate jobs server if possible. Thanks in advance.

[ioquatix]

To be honest, this is not something I've explored much. I don't think it's a bad idea but I just haven't really had the time or a reason to go deep on it.

Most things these days are compatible with Async and the fiber scheduler, so there is no reason why it shouldn't work, unless the design of Solid Queue is deliberately going out of it's way to be obtuse (e.g. per-thread state or something). So, why not try it and see how it goes?

For reference, my general advice is to take a look at https://github.com/socketry/async-job-adapter-active_job and see how it works. Does SolidQueue use ActiveRecord internally? Maybe you can give it a try and report back the issues you run into and we can work through them.

[ioquatix]

It's very poorly documented and tested but I believe it's all working: https://github.com/socketry/async-cron

[trevorturk]

Oooo TIL!

[izaguirrejoe]

Sorry for late response, ya'll. I've had a little time to work on this, but I think I have something working. I'm using Kamal to deploy with the default Rails Dockerfile and docker-entrypoint file. I've just added the following lines to my docker-entrypoint:

./bin/jobs &
JOBS_PID=$!

trap "echo 'Stopping background jobs...'; kill $JOBS_PID" SIGTERM SIGINT

Playing around with it a little, it seems to work just fine. I'm not an expert on processes, child processes, though, so I don't know the downsides to doing it like this.

In researching, the Puma plugin just forks the parent process and runs SolidQueue in a child process:

launcher.events.on_booted do
      @solid_queue_pid = fork do
        Thread.new { monitor_puma }
        SolidQueue::Supervisor.start
      end
    end

I found this neat async supervisor repo, as well as the async-service gem. Could these be used to implement this better? Forgive my ignorance on processes, supervisors, etc 🙇🏽‍♂️

[ioquatix]

Yes you can run the solid queue as part of your falcon.rb config, you'd need to define a default environment and service class. You can see a simple ish example here which runs ffmpeg as a child process:

https://github.com/ioquatix/chick.nz/tree/main/lib/camcast/service

And the config itself:

https://github.com/ioquatix/chick.nz/blob/526f7fb04176561f9a8a58fd7c54d89a8df7bd05/falcon.rb#L25-L27