SolidQueue Batches: evolution and architecture

When Rosa first released SolidQueue, she did a fantastic talk on the internals of how it works and what trade-offs she made when building it. You should give it a watch. It’s where I first learned how the SolidQueue approach works, and why it performs as well as it does.

I wrote the batch implementation for SolidQueue (with feedback from loads of folks!), so as a complement to Rosa’s original talk, here’s a bit about how the internals of batches work.

As a quick refresher of what batches look like:

# Batch entrypoint:
SolidQueue::Batch.enqueue(
  # `on_finish` is one of three callback types, fired when the entire batch of jobs finishes:
  on_finish: SummarizeResearchJob.new(
    research_results
  )
) do
  # Every job enqueued inside of `Batch.enqueue` joins the batch
  research_topics.each do |topic|
    ResearchTopicJob.perform_later(
      research_results, topic
    )
  end
end

# Any regular ActiveJob can join a batch
def ResearchTopicJob < ApplicationJob
  def perform(research_results, topic)
    response = RubyLLM.chat.ask(topic)
    research_results.add_result(response)
  end
end

# Any regular ActiveJob can also be a callback
class SummarizeResearchJob < ApplicationJob
  def perform(research_results)
    puts "#{batch.total_jobs} completed, starting summarization..."
    summarize_research(research_results)
  end
end

If you aren’t familiar with SolidQueue batches, take a look at my guide on using batches in SolidQueue or the batch docs themselves.

Before getting into how batches evolved, let’s discuss a bit of how SolidQueue runs jobs internally.

Job Executions

One of the core concepts of SolidQueue is the concept of an “execution”.

Most RDBMS-based job systems you use in the Ruby world store all of their job data in a single table. Generally this table is called something like <my_job_system>_jobs, and it stores all the information you need about the job: enqueued at, started at, finished at, priority, queue, payload, etc.

Having one table is an understandable approach, but it comes with scaling and complexity issues. That job row becomes very “hot”, ie, every step of the process is constantly accessing, locking and manipulating that central row.

SolidQueue was built with a Job table as well, but it didn’t stop there. The system also breaks up a job into different modes of execution, each backed by its own table/model. Five of them are mutually exclusive states:

• ReadyExecution: job is runnable now, and ready for a worker to pick up
• ClaimedExecution: a worker has claimed the job and is running it
• BlockedExecution: a concurrency control is blocking this job from being ready because another job already has the concurrency semaphore
• FailedExecution: job is permanently failed unless manually retried
• ScheduledExecution: job is scheduled to be run at a later time

The system is built so none of those five states exist at the same time. When a job is claimed, the ReadyExecution is deleted and ClaimedExecution is created. If it fails, ClaimedExecution is deleted and FailedExecution is created.

In addition, executions can also model other properties of a job:

• RecurringExection: represents that the job is running on behalf of a recurring schedule. This allows SolidQueue to block duplicate recurrences from running. This type of execution is informative to batches later on in the post

Why this matters, and makes SolidQueue uniquely well designed, is it means queries operate on these isolated tables instead of constantly reading/updating/deleting job rows. This spreads out the load and allows Job to avoid being a hot spot.

If you want a TL;DR on the final batch architecture, skip ahead to Attempt #3. But if you have any interest in how a single feature can evolve and stop and start over multiple years, read on.

Approaching an architecture for batches

I opened a draft batch PR in February 2024, not knowing any of that execution architecture, just that I thought batches would be a good feature, and one I’d used extensively in Sidekiq Pro.

What did I get myself into? That simple act led me through multiple deep code revisions, learning the internals of SolidQueue, making multiple great Ruby friends, and collaborating the most deeply I’d ever done on an open source project. It was a process that evolved over three phases and 2.5 years.

Attempt #1: Update the Dispatcher

I opened the PR with my first attempt to get the discussion going, knowing at the time that Rosa had already said batches were not a feature they were yet open to accepting.

Thankfully Rosa never actually reviewed Attempt #1, because it meant she never had to read my initial approach 😅.

To get batches started, I searched around for code that was already running on a schedule and checking for work to be done. I found a class called Dispatcher, which ran on an interval and handled scheduled jobs. So I started by hijacking that flow. After dispatching the next batch of scheduled jobs, it now also checked for finished batches. Janky, but I figured it could end up in its own class once I proved the idea:

def dispatch_next_batch
  with_polling_volume do 
    ScheduledExecution.dispatch_next_batch(
      batch_size
    )
    SolidQueue::Batch.dispatch_finished_batches
  end
end

But how could I make dispatch_finished_batches aware of which batches to process?

Much like other libraries and their central jobs tables, the earliest version of batches had a single solid_queue_batches table, with two columns for monitoring progress: changed_at and last_changed_at:

create_table "solid_queue_batches" do |t|
  t.datetime "changed_at"
  t.datetime "last_changed_at"
  # It of course has other characteristics as well, for things
  #  like batch callbacks, starting/final status of the batch
  #  and metadata
  t.string "description"
  t.text "on_finish"
  t.text "on_success"
  t.text "on_failure"
  t.text "metadata"
  t.datetime "enqueued_at"
  t.datetime "finished_at"
  t.datetime "failed_at"
  t.datetime "created_at", null: false
  t.datetime "updated_at", null: false
end

Whenever a job finished and updated it’s finished_at, or a FailedExecution was created (which happens when a job runs out of retries and completely fails), it would touch both columns on the related batch:

if job.batch_id.present?
  job.job_batch.touch(
    :changed_at, :last_changed_at
  )
end

dispatch_finished_jobs would get a FOR UPDATE SKIP LOCKED lock on the batches that were incomplete, attempting to finish each one. incomplete meant changed_at was not null, or it had been over an hour since the last_changed_at¹:

scope :incomplete, -> {
  where(finished_at: nil).where("changed_at IS NOT NULL OR last_changed_at < ?", 1.hour.ago)
}

def dispatch_finished_batches
  incomplete.order(:id).pluck(:id).each do |id|
    transaction do
      where(id:).non_blocking_lock.each(&:finish)
    end
  end
end

It was a bit clunky, but not overly expensive. The incomplete lookup was pretty fast on a properly indexed column. But it did mean a lot of potential wasted work, constantly checking for batch changes.

Much worse, however, was what happened once it detected a change. It would then iterate over every job related to the batch, checking the status of each one:

def finish
  return if finished?
  reset_changed_at
  jobs.find_each do |next_job|
    return unless next_job.finished? || next_job.failed?
  end

  attrs = {}

  if job_class.present?
    job_klass = job_class.constantize
    active_job = job_klass.perform_later(self)
    attrs[:job] = Job.find_by(active_job_id: active_job.job_id)
  end

  update!({ finished_at: Time.zone.now }.merge(attrs))
end

Wildly inefficient. This could easily result in a worst case complexity of O(n^2) by the time the batch completed². Once upon a time I fixed an O(n^2) in a popular JavaScript library, and here I was introducing it into Ruby 😔. This code also meant that you could not delete a job if it was related to a batch. If you did, it was impossible to figure out which callbacks to enqueue 🔥.

But, it did mostly work and it described the API. That original API stuck around with only minor changes vs what merged in SolidQueue 1.7.

But yikes, those internals.

Promise-driven development

The first few months after opening the PR I played around with it and tried to clean things up. Many folks commented on the PR, hoping batches would land. There was definitely demand for it!

Despite that, the PR sat around for awhile. From initial PR open, It took about 18 months before it got a kickstart, all thanks to RailsConf 2025.

At that final RailsConf, I got to meet Rosa in person. She’s a great person, you should say hi to her if you see her at a conference!

We talked a bit about SolidQueue, and that batches should really end up in it. We sat next to each other at a lightning talk segment later that day, where Jeremy Smith was doing a lightning talk called Programming in the Low Memory Environment of Your Brain:

While watching his talk, I noticed a small line about batches in SolidQueue, which mentioned me directly by name:

He had very unintentionally put me on blast 😅 I doubt anyone else noticed, but it shined painfully in my eyes. Afterwards I introduced myself to Jeremy and we nerded out about job systems for quite a while.

If you watched Rosa’s talk, she makes an amazing point about what she jokingly refers to as “Promises Driven Development”:

I call this methodology “Promises Driven Development”… if you are struggling with motivation… I recommend your boss go to a conference and promised the delivery of your projects in front of hundreds of people. I guarantee that you will recover your motivation right away.

It wasn’t DHH in front of hundreds of people promising SolidQueue batches, but it was enough to get me sufficiently motivated to push batches to the finish line!

Attempt #2: Counters, callbacks and jobs

Soon after RailsConf, I got batches updated and running again. When I looked at open PRs, I immediately noticed a PR from Mikael Henriksson:

While my PR had been sitting around gathering comments and dust, Mikael built his own variation of batches. For example:

MyJob.perform_batch_later([
  { user_id: 1, action: "update" },
  { user_id: 2, action: "update" },
  { user_id: 3, action: "update" },
  on_success: { 
    job: ImportSuccessJob, 
    args: { email: "admin@example.com" }
  },
  on_failure: { 
    job: ImportFailureJob, 
    args: { email: "admin@example.com" }
  },
  on_complete: { 
    job: ImportCompleteJob
  },
  metadata: {
    source: "api", 
    imported_by: current_user.id
  }
)

The API was different, but conceptually similar. In the final batch API that equates to:

SolidQueue::Batch.enqueue(
  on_success: ImportSuccessJob.new(
    email: "admin@example.com"
  ),
  on_failure: ImportFailureJob.new(
    email: "admin@example.com"
  ),
  on_finish: ImportCompleteJob,
  source: "api",
  imported_by: current_user.id
) do
  MyJob.perform_later(user_id: 1, action: "update")
  MyJob.perform_later(user_id: 2, action: "update")
  MyJob.perform_later(user_id: 3, action: "update")
end

I had done more work integrating batches deeper into SolidQueue as an API - regular job enqueues worked, bulk enqueue worked, it used normal ActiveJob instances - but Mikael’s implementation avoided the constant batch queries, and the O(n^2) job checks. Instead of that, it used counters to track the progress of a batch. Instead of constantly asking if a batch had finished - whenever a job finished, it would update a progress counter and ask the batch about completion directly.

There were four counters:

1. total_jobs: total job count
2. pending_jobs: jobs left to finish
3. completed_jobs: jobs that finished successfully
4. failed_jobs: jobs that exhausted all retries and failed

Which resulted in a simple modification to the schema:

create_table "solid_queue_batches", do |t|
  #...
  t.integer "total_jobs", default: 0, null: false
  t.integer "completed_jobs", default: 0, null: false
  t.integer "failed_jobs", default: 0, null: false
  t.integer "pending_jobs", default: 0, null: false
end

We discussed a bit over GitHub and DMs the best way to combine our work. He closed his PR, and I adapted his approach into mine. Now, whenever a job was updated, or a FailedExecution was created, we’d trigger a job to check batch progress:

# Job::Batchable
after_update :update_batch_progress, if: :batch_id?

def update_batch_progress
  return unless saved_change_to_finished_at? && finished_at.present?
  return unless batch_id.present?

  BatchUpdateJob.perform_later(self)
  #...
end


# FailedExecution::Batchable
after_create :update_batch_progress, if: -> { job.batch_id? }

def update_batch_progress
  BatchUpdateJob.perform_later(job)
  #...
end

If a SolidQueue::Job was updated and finished_at was added, it would trigger BatchUpdateJob. It enqueued the same job whenever a FailedExecution was created.

The BatchUpdateJob simply checked if the job was finished:

class BatchUpdateJob < ActiveJob::Base
  def perform(job)
    #...
    job.batch.job_finished!(job)
  end
end

job_finished! was the key to properly tracking whether all jobs were complete:

def job_finished!(job)
  return if finished?

  transaction do
    if job.failed_execution.present?
      self.class.where(id: id).update_all(
        "failed_jobs = failed_jobs + 1, pending_jobs = pending_jobs - 1"
      )
    else
      self.class.where(id: id).update_all(
        "completed_jobs = completed_jobs + 1, pending_jobs = pending_jobs - 1"
      )
    end

    reload
    check_completion!
  end
end

As each job finished or failed, the pending count would decrease and the completed or failed counts would increase. Once pending_jobs was zero, check_completion! would complete the batch and fire callbacks.

Pretty simple - ActiveRecord callbacks enqueued a progress job, and Batch counters incremented and decremented. Unfortunately, you may see issues already with the potential performance of this approach.

A performance bottleneck

After adapting it in, I started doing some local load testing, comparing job performance on its own, and as part of batches.

To my dismay, batches added significant overhead. It was multiple times the cost of running a normal job, and it got worse the more jobs you attempted to run in a batch. It caused 3-5x overhead or worse as the size increased.

The performance overhead came down to two things:

1. Excessive jobs
2. “Hot” row contention

Excessive jobs

Excessive jobs seemed pretty obvious the moment I started hitting it. For every job you ran as part of a batch, we triggered yet another job to check if the batch was done:

• 10k batch jobs meant 20k jobs
• 100k meant 200k jobs
• 500k meant 1 million jobs

I needed to improve it, but it was likely as simple as cutting the job out entirely. The problem that surprised me was how much of a bottleneck the single row update was.

“Hot” rows and slotted counters

Even after removing the BatchUpdateJob, I couldn’t get the performance under control. The Batch row now became the bottleneck.

if job.failed_execution.present?
  self.class.where(id: id).update_all(
    "failed_jobs = failed_jobs + 1, pending_jobs = pending_jobs - 1"
  )
else
  self.class.where(id: id).update_all(
    "completed_jobs = completed_jobs + 1, pending_jobs = pending_jobs - 1"
  )
end

Every time a batch job finished or failed, it needed to update the same Batch row.

I started searching around for something that could improve hot row counters. The PlanetScale blog came to the rescue with an article on slotted counters³.

The basic idea is that instead of updating the single row, the updates are spread across N number of rows for the same logical counter. I created a counters table to represent this:

create_table "solid_queue_batches_counters", do |t|
  #...
  t.integer "batch_id", null: false
  t.integer "slot", default: 0, null: false
  t.integer "total_jobs", default: 0, null: false
  t.integer "completed_jobs", default: 0, null: false
  t.integer "failed_jobs", default: 0, null: false
  t.integer "pending_jobs", default: 0, null: false
end

Depending on the number of slots configured, it took a modulus of the job id and upserted into that row. It could then SUM the relevant records when you need the count, or periodically aggregate:

mattr_accessor :slots
self.slots = 32

def slot_for(job_id)
  job_id.to_i % slots
end

def update_on_completion(job, status)
  return unless job.batch_id.present?
  upsert_increments([{
    batch_id: job.batch_id,
    slot: slot_for(job.id),
    total: 0,
    pending: -1,
    completed: (status == "failed" ? 0 : 1),
    failed: (status == "failed" ? 1 : 0)
  }])
end

def completed_jobs(batch)
  where(batch_id: batch.id).sum(:completed_jobs)
end

# def total_jobs, failed_jobs, etc

Clearly, I really wanted the counter approach to work. Slotted counters helped with the hot row problem, but it introduced issues of its own. The SUM still caused overhead, and now there were timing and transaction visibility issues to deal with.

It also just seemed… off, and too complicated to introduce into SolidQueue.

Attempt #3: BatchExecution, duh

It was around this point that I actually, finally, watched Rosa’s talk about the internals of SolidQueue. I’d had it on a list for a while⁴. Why didn’t I watch it sooner?!

Having watched it, I strongly felt that the “execution” model was relevant to batches, but how? And anyway, could it even help me with my hot row problem?

It took some experimentation and false starts, but I finally landed on the general approach, created a new BatchExecution model, and narrowed in on the architecture that landed in SolidQueue 1.7. The final architecture landed on four rules:

1. BatchExecution represents a job’s active membership in a batch

Every job running in a Batch in SolidQueue has a batch_id, but BatchExecution represents active participation in the batch.

create_table "solid_queue_batch_executions" do |t|
  t.bigint "job_id", null: false
  t.bigint "batch_id", null: false
  t.datetime "created_at", null: false
end

2. Every job in a batch starts off with a BatchExecution

When enqueueing a job as part of a batch (either in perform_later or perform_all_later), always create a BatchExecution record alongside it.

This:

SolidQueue::Batch.enqueue do
  MyJob.perform_later
end

Causes a BatchExecution to get created:

after_create :create_batch_execution, if: :batched?

def create_batch_execution
  BatchExecution.create_all_from_jobs([ self ])
end

This:

SolidQueue::Batch.enqueue do
  ActiveJob.perform_all_later([
    MyJob.new("1"),
    MyJob.new("2"),
    MyJob.new("3")
  ])
end

Does the same:

def prepare_all_for_execution(jobs)
  batch_all(jobs) # Batch it up!

  due, not_yet_due = jobs.partition(&:due?)
  dispatch_all(due) + schedule_all(not_yet_due)
end

BatchExecution is a type of Execution (class BatchExecution < Execution), so it comes with the same create_all_from_jobs bulk-efficient method as all other Execution models:

def batch_all(jobs)
  BatchExecution.create_all_from_jobs(jobs) if Batch.migrated?
end

3. BatchExecution represents a viable job

While a job is still viable (ie, it has not finished or failed), it continues to have an associated BatchExecution row.

4. Counters are a snapshot of final state

BatchExecutions are the means to track batch completion. That meant pending_jobs was no longer needed, and total_jobs became a cache counter for total BatchExecutions. total_jobs is incremented as new jobs are added, but until a batch finishes, the other counters are live queries. Once a batch finishes, they get updated to the final counts.

Tracking a batch now revolves around BatchExecutions. They are created when a job is created, and they are destroyed when a job finishes or fails. When they are destroyed, we check if a batch is finished:

module SolidQueue
  class BatchExecution < Execution
    after_commit :finish_batch, on: :destroy
    #...
    def finish_batch
      #...
      batch.finish
    end
  end
end

And in that check, we can use batch_executions.exists? to efficiently check if we’re ready to complete the batch. batch_executions.exists? short-circuits the moment it finds a related BatchExecution row, so this is cheap, even on very large batches:

def finish
  #...
  return if batch_executions.exists?
  #...
end

Finally, we atomically compare-and-set inside of a transaction. If we’re racing another transaction to complete a batch, only one of us will have an updated > 0:

def finish
  #...
  transaction do
    updated = Batch.where(id: id).unfinished.enqueued.without_executions.update_all(finished_at: Time.current)
    finalize if updated > 0
  end
end

I now had an architecture that felt strong and performed well. What was left?

Production readiness

There were a variety of other things that evolved over the course of developing batches internals, and that needed to be added to make it ready for actual production environments.

Real production use

Early on, Rosa made a fair point:

Without production use, it’s hard to gauge how effective a batch API would be. This was a critical step for me.

By running it in production, it allowed me to find some API ergonomics tweaks, and some real edge cases. Before it got into main, it was run against hundreds of millions of jobs and batches in production environments.

Running that many batches allowed 0.01% kind of edge cases to surface, which is why the sweep_stalled method was introduced⁵, to deal with batches getting stuck and not starting:

module SolidQueue
  class Batch
    # Repairs batches that the regular completion detection can't finish on
    # its own: jobs removed via bulk discards, processes that crashed after
    # enqueueing jobs but before starting their batch, or completions whose
    # callback enqueueing failed and rolled back.
    module Sweepable
      extend ActiveSupport::Concern

      class_methods do
        def sweep_stalled(stalled_for: 5.minutes)
          sweep_stale_executions
          finish_stalled_batches
          start_stalled_batches(stalled_for:)
        end

The need to monitor for stuck batches largely stems from two things. First:

1. after_commit

This is something that impacts anyone using after_commit, and it gets under emphasized in the community. Batches wait until all surrounding transactions commit to start itself by using after_all_transactions_commit. Doing something after_commit means you have no transactional safety. Which also means you have to find ways to make sure if a server crashes, or a database connection fails, or an error occurs in your code during after_commit, that you have a way to mitigate it.

With enqueue_after_transaction_commit becoming the default in Rails 8.2, ideally we’d come up with a pattern in Rails for people who have mission critical jobs which might silently be lost in the case of a server crash (like some kind of Transactional Outbox⁶)

2. Separate databases

Separate databases are the default behavior of SolidQueue, and the encouraged approach. This is great for database health, but means you’re in a bit of an eternal after_commit, because you have no choice but to be in a separate transaction from your main app database. Batches have mitigations for it, but your own code likely doesn’t. It’s up to you to keep an eye on your mission critical jobs.

Batches overhead

As of the 1.7 release, a job running in a batch incurs about a 20% performance penalty. I tested batches with hundreds of thousands of no-op jobs, and this was pretty consistent.

That sounds bad! In reality, It’s about 0.5 to 1ms overhead per job. Unless your jobs are also no-ops, you will not notice. Still, I’d like any overhead to be minimal and I have a stack of promising, small changes that may get it down to around 8% overhead. I think that is pretty reasonable.

enqueue_after_transaction_commit

As a notable aside, enqueue_after_transaction_commit is a handy Rails feature that is brutal to support in batches. GoodJob and Sidekiq Pro batches don’t support it at all. But as a Rails default option, I felt SolidQueue needed to.

What this means for you is that you have less to think about when combining features of SolidQueue/ActiveJob in natural ways:

ActiveRecord::Base.transaction do
  SolidQueue::Batch.enqueue do
    ImmediateJob.perform_later
    DeferredJob.perform_later
  end
end
# It just works...

It seems so simple, doesn’t it? In concept, it is! But in implementation, that DeferredJob gets hijacked out of the normal flow and completely bypasses the adapter job system until the after_commit. As far as the job system knows, the job never existed until the transaction was over, well after Batch.enqueue has finished running!

There are a couple sharp edges still remaining, but I’ve also got work in progress to fix that too.

If you don’t know what enqueue_after_transaction_commit is, you should read about it. Then re-read my earlier recommendations.

You got here, you did it!

I think that about sums up the highlights of batches evolution and architecture. I hope it makes some people feel better about anything they’ve had sitting around, or work that’s taken them a long time to complete. There were many more conversations, DMs, Github comments, and experiments than what I mentioned here! Thanks to everyone who discussed, poked, prodded and encouraged the completion of batches! 🤙