mercredi 12 août 2015

Daemons-Rails: scaling up to multiple workers

So, I've been given a code base which uses daemons, daemons-rails and delayed jobs to trigger a number of *_ctl files in /lib/daemons/ but here's the problem:

If two people do an action which starts the daemons doing some heavy lifting then whichever one clicks second will have to wait for the first to complete. Not good. We need to start multiple daemons on each queue.

Ideally what I want to do is read a config file like this:

default:
  queues: default ordering 
  num_workers: 10
hours:
  queues: slow_admin_tasks
  num_workers: 2
minutes:
  queues: minute
  num_workers: 2

This would mean that 10 daemon processes are started to listen to the default and ordering queues, 2 for slow_admin tasks etc.

How would I go about defining multiple daemons like this, it looks like it might be in one of these places:

  • /lib/daemons/*_ctl
  • /lib/daemons/*.rb
  • /lib/daemons/daemons

I thought it might be a change to the daemons-rails rake tasks, but they just hit the daemons file.

Has anyone looked in to scaling daemons-rails in this way? Where can I get more information?

Aucun commentaire:

Enregistrer un commentaire