JP Camara

When good threads go bad

Tue, 30 Dec 2025 11:20:16 -0500

👋🏼 This is part of series on concurrency, parallelism and asynchronous programming in Ruby. It’s a deep dive, so it’s divided into several parts:

Your Ruby programs are always multi-threaded: Part 1

Your Ruby programs are always multi-threaded: Part 2

Consistent, request-local state

Ruby methods are colorless

The Thread API

Bitmasks, Ruby Threads and Interrupts, oh my!

When good threads go bad

Thread and its MaNy friends

Fibers

Processes, Ractors and alternative runtimes

Scaling concurrency with streaming

Abstracted, concurrent Ruby

Closing thoughts, kicking the tires and tangents

How I dive into CRuby concurrency

You’re reading “When good threads go bad”. I’ll update the links as each part is released, and include these links in each post.

Threads out after curfew 🧵

It’s late, and you start getting alerts that requests to your web server are failing. You try to load a page and it hangs endlessly. The server isn’t responding to anything, and requests are continuing to queue up.

It’s 10PM. Do you know where your ~~children~~ threads are?

📺 iykyk

Not knowing what else to do, you trigger a server restart. Even doing that, things remain unresponsive for another 30 seconds. Finally you see the server stop, and start up again. As if by magic, everything is running fine again.

You’re running Puma with threads. It seemed like every thread was unresponsive. What happened to those threads?!

Stuck threads

The reality of the situation is probably mundane.

Are you looking up data in a query? Did you remember to index your columns?
Do you have N+1 queries?¹
Did you remember to set statement and lock timeouts before running a schema migration?
Are you sure it isn’t 1, 2 or 3?

If it isn’t those things, there are other ways your threads can go rogue. Let’s look at some ways a thread can get stuck:

Deadlocks
Livelocks
Long-running CPU
Long-running IO
Native extensions

Deadlocks

The conventional example of a deadlock is two threads attempting to acquire a mutexheld by the other thread.

They can never make progress, so they’re dead in the water:

mutex_1 = Mutex.new
mutex_2 = Mutex.new
thread_1 = Thread.new do
  mutex_1.synchronize do
    sleep 1
    mutex_2.lock
  end
end
thread_2 = Thread.new do
  mutex_2.synchronize do
    sleep 1
    mutex_1.lock
  end
end
thread_1.join

In our example, thread_1 acquires mutex_1. Then thread_2 acquires mutex_2. Next, thread_1 attempts to acquire mutex_2 and blocks. Then thread_2 attempts to require mutex_1 and blocks. Neither can make progress, and are stuck in place.

It’s a traditional example, but Ruby is very good at detecting it! It detects the problem and raises an error, checking if any threads are capable of making progress:

'Thread#join': No live threads left. Deadlock? (fatal)
  3 threads, 3 sleeps current:0x0000000a35e90c00 main thread:0x0000000101262de0
  * #<Thread:0x00000001001fafa8 sleep_forever>
    rb_thread_t:0x0000000101262de0 native:0x0000000202cfc800 int:0
	   
  * #<Thread:0x0000000121727c08 (irb):101 sleep_forever>
  rb_thread_t:0x0000000a35e90e00 native:0x00000001700f7000 int:0 mutex:0x0000000a3638c000 cond:1
    depended by: tb_thread_id:0x0000000101262de0
	   
  * #<Thread:0x00000001217279d8 (irb):107 sleep_forever>
  rb_thread_t:0x0000000a35e90c00 native:0x0000000170203000 int:0

Ruby detects that thread_1 and thread_2 are sleeping, and the third thread is Thread.main, which sleeps waiting for thread_1 to finish.

Most long running programs are likely to have some other thread running, and Ruby only detects if all threads are stuck. We can get our deadlock example to work by adding an extra thread in a work loop:

mutex_1 = Mutex.new
mutex_2 = Mutex.new
thread_1 = Thread.new do
  mutex_1.synchronize do
    sleep 1
    mutex_2.lock
  end
end
thread_2 = Thread.new do
  mutex_2.synchronize do
    sleep 1
    mutex_1.lock
  end
end

thread_3 = Thread.new do
  loop do
    # process some work...
  end
end

thread_1.join

Now threads 1 and 2 will never progress, and Ruby lets the program continue running because thread_3 is still active.

Livelocks

While a deadlock means a thread has stopped processing, a livelock happens when a thread keeps running, but never makes any progress. Here’s another example using an alternative approach for acquiring a mutex:

mutex_1 = Mutex.new
mutex_2 = Mutex.new
thread_1 = Thread.new do
  mutex_1.synchronize do
    sleep 0.1
    while !mutex_2.try_lock; end
  end
end
thread_2 = Thread.new do
  mutex_2.synchronize do
    sleep 0.1
    while !mutex_1.try_lock; end
  end
end
thread_1.join

This is similar to our deadlock example, but this time we use #try_lock instead of #lock. Unlike #lock, which blocks until the mutex is available, #try_lock returns false if attempting the lock fails. We do a short sleep in each thread to give them time to acquire the initial locks, then iterate infinitely attempting #try_lock. The locks will never be acquired, and the loops will run forever. Burn, CPU, burn 🔥.

Personally I’ve rarely encountered deadlocks and livelocks in threaded code. But I’ve definitely encountered them in databases!

thread_1 = Thread.new do
  User.find(1).with_lock do
    sleep 0.1
    User.find(2).lock!
  end
end

thread_2 = Thread.new do
  User.find(2).with_lock do
    sleep 0.1
    User.find(1).lock!
  end
end

thread_1.join

On PostgreSQL, it will detect this deadlock and raise an error:

PG::TRDeadlockDetected: ERROR:  deadlock detected (ActiveRecord::Deadlocked)
DETAIL:  Process 581 waits for ShareLock on transaction 4003; blocked by process 582.
Process 582 waits for ShareLock on transaction 4002; blocked by process 581.
CONTEXT:  while locking tuple (0,1) in relation "users"

How do we solve deadlocks and livelocks? The answer is a consistent order for locking.

mutex_1 = Mutex.new
mutex_2 = Mutex.new
thread_1 = Thread.new do
  mutex_1.synchronize do
    sleep 1
    mutex_2.lock
  end
end
thread_2 = Thread.new do
  mutex_1.synchronize do
    sleep 1
    mutex_2.lock
  end
end

thread_3 = Thread.new do
  loop do
    # process some work...
  end
end

thread_1.join

This is the same example as before, but this time both threads attempt to acquire mutexes in identical order. As long as you acquire in a consistent order, you should never hit deadlocks or livelocks.

Long-running CPU

As far as I know, this isn’t actually possible in pure Ruby. What I mean by “pure” Ruby is a program that only runs Ruby code, and no C/Rust/Zig extensions. The CRuby runtime controls how pure Ruby code runs, and makes sure we can’t hog threads. Mostly.

In Bitmasks, Ruby Threads and Interrupts, oh my!, we dug into the TIMER_INTERRUPT_MASKand how it utilizes priority. It allows a thread to influence how large of a time slice it gets:

def calculate_priority(priority, limit)
  priority > 0 ? limit << priority : limit >> -priority
end
	
calculate_priority(0, 100)        # => 100ms
calculate_priority(2, 100)        # => 400ms
calculate_priority(-2, 100)       # => 25ms

At a default priority of 0, we get Ruby’s default time slice of 100ms. At -2, we get 25ms. At 2, we get 400ms. This means in theory, can we starve out other threads by increasing our priority?

calculate_priority(5, 100)       # => 3276800ms

3,276,800 milliseconds is 54 minutes. Can we really block things for 54 minutes!?

counts = Array.new(10, 0)
done = false

threads = 10.times.map do |i|
  Thread.new do
    j = 0
    loop do
      break if done
      j += 1
      counts[i] += 1 if j % 1_000_000 == 0
    end
  end.tap do |t|
    t.priority = 15 if i == 5
  end
end

sleep 10
done = true
counts.each_with_index { |c, i| puts "#{i}: #{c}" }

There’s a bunch going on here - but there are only a few details to focus on:

For the 6th thread (i == 5), we set the priority to 15. That’s the priority that in theory gives us a time slice of 54 minutes.
We set a count every million iterations in each thread.
We sleep for 10 seconds, then set a killswitch on each thread by setting done to false.
We print how many times each thread was able to increment their slice of the array.

Here’s the output:

As we can see - the priority made a difference! The thread at index 5 runs around 7 times as much as every other thread. But, it does not fully hog the thread like we might have thought. This is only 10 seconds, well under 54 minutes, and the other threads still get prioritized by the scheduler.

This shows that we can influence the scheduler, but we can’t completely hog the runtime from pure Ruby code.

In more practical Ruby code, the Sidekiq gem gives you the ability to set priority on the threads it creates:

Sidekiq.configure_server do |cfg|
  cfg.thread_priority = 15
end

Interestingly, by default, Sidekiq sets its threads to a priority of -1, which is less than the 0 that Ruby uses by default. It describes the rationale:

Ruby’s default thread priority is 0, which uses 100ms time slices. This can lead to some surprising thread starvation; if using a lot of CPU-heavy concurrency, it may take several seconds before a Thread gets on the CPU.

Negative priorities lower the timeslice by half, so -1 = 50ms, -2 = 25ms, etc. With more frequent timeslices, we reduce the risk of unintentional timeouts and starvation.

Fascinating to see a real-world use-case of priority like this! Sidekiq has run trillions of jobs across hundreds of thousands of apps, and they made the decision to switch the priority.

Since Ruby 3.4, you can achieve the same thing globally by using RUBY_THREAD_TIMESLICE. You can set RUBY_THREAD_TIMESLICE=50 and keep the priority the same, but now the time slice is 50ms.

Long-running IO

This is the likeliest scenario that will saturate your threads: long-running IO. In Ruby methods are colorless, we discussed how threads are great at handing off work when blocked on IO:

As soon as you do any IO operation, it just parks that thread/fiber and resumes any other one that isn’t blocked on IO.

However, this only works as long as:

You have other threads available to pickup work
You’ve tuned your thread counts based on your workload. Amdal’s Law helps you decide how many threads you should run, based on how much IO is running in your code

Earlier we mentioned slow queries as a possible IO blocker. But let’s say you have a web server running with 5 threads, and you allow users to download files²:

class DownloadsController < ApplicationController
  def index
    send_file Rails.root.join("public/large.txt"),
      filename: "large.txt",
      type: "application/octet-stream"
  end
end

We have a simple Rails controller action, which sends a file to the client making the request. It’s pretty straightforward! The large.txt file I tested with locally is about 130mb. I’ll run it in a basic Puma setup, using 5 threads:

bundle exec puma -t 5:5
	
Puma starting in single mode...
 * Puma version: 7.1.0 ("Neon Witch")
 * Ruby version: ruby 4.0.0 (2025-12-25 revision 553f1675f3) +PRISM [arm64-darwin23]
 *  Min threads: 5
 *  Max threads: 5

I’m running a Puma server with 5 threads. Let’s try some benchmarks against it. We’ll use Apache Bench (ab) to simulate traffic to Puma. -n means the number of total requests, and -c is how many concurrent requests to make. Let’s start with 1 request:

ab -n 1 -c 1 http://0.0.0.0:9292/downloads
	
Concurrency Level:      1
Time taken for tests:   0.070 seconds
Complete requests:      1

Cool - 0.07 seconds. How about 3?

ab -n 3 -c 3 http://0.0.0.0:9292/downloads
	
Concurrency Level:      3
Time taken for tests:   0.074 seconds
Complete requests:      3

Not much change from a single request! How about 5? This would match the maximum number of simultaneous requests our server can currently support, using 5 threads:

ab -n 5 -c 5 http://0.0.0.0:9292/downloads
	
Concurrency Level:      5
Time taken for tests:   0.134 seconds
Complete requests:      5

Things slow down a little bit once we max out our threads. But still reasonable. How about 50?

ab -n 50 -c 50 http://0.0.0.0:9292/downloads
	
Concurrency Level:      50
Time taken for tests:   0.820 seconds
Complete requests:      50

So far, so good. But we’re benefiting from a lot here: the file isn’t particularly large, and there is no latency. The server is responding quickly, and the client is consuming the response quickly. Let’s switch things up a bit - how well does it handle a client downloading slowly?

We’ll use curl to simulate a slow client. We can use limit-rate to simulate a client downloading only 1000k per second. curl ... --limit-rate 1000k & means we’ll download results at a rate of 1000k per second, running in the background (&). This means it will take our curl call 2 or 3 minutes to download a 130mb file. At the same time, we’ll run another apache bench to see how things perform:

curl http://0.0.0.0:9292/downloads -o /tmp/large.txt \
  --limit-rate 1000k &
	
ab -n 1 -c 1 http://0.0.0.0:9292/downloads
	
Concurrency Level:      1
Time taken for tests:   0.056 seconds
Complete requests:      1

Puma responds quickly. In this scenario, curl is occupying one thread, and ab occupies another. Let’s try running 4 curl commands:

for i in {1..4}; do
  curl http://0.0.0.0:9292/downloads -o /tmp/large_$i.txt \
    --limit-rate 1000k &
done
	
ab -n 1 -c 1 http://0.0.0.0:9292/downloads
	
Concurrency Level:      1
Time taken for tests:   0.059 seconds
Complete requests:      1

Puma still responds fine. curl is now occupying 4 threads, and ab uses the remaining 1 thread. Let’s add one more request using curl. We also increase the default timeout of ab (which is 30) to 200, for no particular reason…

for i in {1..5}; do
  curl http://0.0.0.0:9292/downloads -o /tmp/large_$i.txt \
    --limit-rate 1000k &
done
	
ab -n 1 -c 1 -s 200 http://0.0.0.0:9292/downloads
	
Concurrency Level:      1
Time taken for tests:   124.477 seconds
Complete requests:      1

Yikes. That did not go well. The moment we had 5 slow running requests from our curl calls, we saturated all available threads. Our 6th request using ab sat around waiting, finally finishing 2 minutes later!

This is a critical consideration - long-running web work is a throughput killer. Ideally, keep all work as fast as possible and offload long-running work to jobs/other services. Most commonly for a download you’d create a presigned url for a service like S3 and redirect to that URL.

If you need to run long-running IO, you need to allocate many more threads³.

Native extensions

In our Long-running CPU example, we couldn’t get pure Ruby code to completely hog the runtime. We can prioritize a thread higher than other threads, but work still continues to be distributed.

Once you start running native extensions, the Ruby runtime has more limited influence. It’s up to the extension to properly interface with Ruby and yield control back. Here’s a simple example that will block all other threads, using the standard openssl gem, using a function written in C, pbkdf2_hmac:

require "openssl"

Thread.new do
  loop do
    puts "tick #{Time.now}"
  end
end

t = Thread.new do
  loop do
    OpenSSL::KDF.pbkdf2_hmac("passphrase", salt: "salt", iterations: 12_800_000, length: 32, hash: "sha256")
  end
end

t.join

We have two threads running - one infinitely printing the time in a loop, and one infinitely calling pbkdf2_hmac in a loop. Here I give pbkdf2_hmac a ludicrous number of iterations to force the function to run, in C, for a long period of time:

tick 2025-12-29 15:20:30 -0500
tick 2025-12-29 15:20:45 -0500
tick 2025-12-29 15:20:59 -0500

The results show that despite each thread having the same priority, the thread with long-running C extension code hogs all of the runtime. The printing thread is able to print a timestamp roughly every 15 seconds.

There’s nothing the extension code is doing wrong per se, but because it runs purely in C, without yielding back to Ruby until its done, Ruby can’t do anything to keep work distribution fair.

In most cases, well-developed/mature native extensions won’t hit this issue. There are many popular gems that are, or include, native extensions. But if you do hit an expensive path in a C extension, be aware the Ruby runtime will not be able to control it. If you know you are interfacing with a slow piece of code in a native extension, keep it off the hot path, same as our long-running IO example.

What happened to our Puma server, anyways?

Remember our production panic scenario from earlier?

Not knowing what else to do, you trigger a server restart. Even doing that, things remain unresponsive for another 30 seconds. Finally you see the server stop, and start up again. As if by magic, everything is running fine again

You triggered a server restart, and still had to wait 30 seconds? Why wasn’t Puma able to stop sooner? Let’s reuse our download example from earlier, and explain some default Puma behaviors!

First, let’s start Puma, and see how quickly we can stop it:

bundle exec puma -t 5:5

Puma starting in single mode...
* Puma version: 7.1.0 ("Neon Witch")
* Ruby version: ruby 4.0.0 (2025-12-25 revision 553f1675f3) +PRISM [arm64-darwin23]
*  Min threads: 5
*  Max threads: 5

### hit ctrl + c to issue a SIGINT signal

Gracefully stopping, waiting for requests to finish
=== puma shutdown: 2025-12-29 23:45:21 -0500 ===
- Goodbye!

We instantly see two things:

Gracefully stopping, waiting for requests to finish
Goodbye!

Puma tells us it is shutting down “gracefully”. With no activity, it is able to instantly stop.

Now let’s use our DownloadController again:

class DownloadsController < ApplicationController
  def index
    send_file Rails.root.join("public/large.txt"),
      filename: "large.txt",
      type: "application/octet-stream"
  end
end

We start Puma again, then occupy each thread with a request:

bundle exec puma -t 5:5

for i in {1..5}; do
  curl http://0.0.0.0:9292/downloads -o /tmp/large_$i.txt \
    --limit-rate 1000k &
done

Now let’s trying issuing INT using ctrl+c again:

Puma starting in single mode...
...
Use Ctrl-C to stop

^C <-- ctrl + c

... ~120 second pass, all downloads finish ...

- Gracefully stopping, waiting for requests to finish
=== puma shutdown: 2025-12-30 00:04:23 -0500 ===
- Goodbye!

Ok, so we issued our interruption. But… it waited for every request to completely finish! We had to wait around 120 seconds before our server shutdown. That’s even worse than earlier!

This isn’t a blog post about Puma specifically, but I’ll discuss a few factors:

Running puma -t 5:5 puts you in Puma “single” mode. This is a lightweight way of running puma, at the cost of less control. By default, single-mode Puma does not kill any requests. Like it tells us, it simply is “waiting for requests to finish”
Alternatively, you can run puma in “cluster” mode. This is done by adding any worker count at all using the -w flag. Even a -w 1 will move you into cluster mode. Cluster mode has a parent worker which monitors and controls child workers. This costs more memory, but it means the parent worker can kill child workers more reliably

Let’s try this one more time, starting in cluster mode by setting -w 1:

bundle exec puma -t 5:5 -w 1

[39363] Puma starting in cluster mode...
[39363] *  Min threads: 5
[39363] *  Max threads: 5
[39363] *   Master PID: 39363
[39363] *      Workers: 1
[39363] Use Ctrl-C to stop
^C <-- ctrl + c
[45110] - Gracefully shutting down workers...

... ~30 second pass, downloads are cut early ...

[45110] === puma shutdown: 2025-12-30 00:24:21 -0500 ===
[45110] - Goodbye!

This time we replicate our earlier behavior - 30 seconds pass, and the server is shutdown. Now that we’re running in cluster mode, by default Puma uses a configuration called worker_shutdown_timeout, which defaults to 30 seconds. If you have a configuration file, you can set it yourself to something longer or shorter:

worker_shutdown_timeout 25 # instead of 30

As well, by default Puma never kills threads. In a moment we’re going to be talking about ways to kill a thread. Puma plays it extremely safe, and offers no ability to kill individual threads. And even when shutting down, it defaults to a thread shutdown policy of :forever, which means the only way the threads are killed is when the server is entirely shutdown, which shuts down the worker the threads live in.

You can change this. In the same configuration file you’d set worker_shutdown_timeout you can set force_shutdown_after:

force_shutdown_after 1 # an integer, :forever, or :immediate

Still - this doesn’t do much. It still only impacts full server shutdown. But with this setting, the internal Puma thread pool will raise on all threads, and then eventually run kill on them.

What all this means is that by default in Puma:

The primary way to fix misbehaving threads is to restart your server
This will take up to 30 seconds running in cluster mode, unless you change the worker_shutdown_timeout
Puma can’t kill long-running or stuck threads without a full server shutdown. Technically there is a control server you can setup, which can manage individual workers. But it cannot kill individual threads/requests

We’ll talk in-depth later about the available option for killing long-running threads in Puma.

📝 if you want to dig deeper into how Puma works, I highly recommend Dissecting Puma: Anatomy of a Ruby Web Server. The source of Puma is also pretty readable

Don't guess, measure

All of these thread issues are possibilities, but they mean nothing without empirical data. Measure, then decide your course of option.

Thread Shutdown

Ok, enough about how threads get stuck. Once they’re stuck - is there a way to stop them?

`raise` and `kill`

You want to kill… me? 🥺

⚠️ TL;DR You shouldn’t use these methods unless you really know what you’re doing. Instead, interrupt your thread safely. Incidentally, you should also avoid the timeout module.

if you’re writing a generic threaded framework you may need it - for custom one-off threads you can probably manage without it

Sometimes a thread is running and you need to shut it down. There’s two primary methods for achieving that: raise and kill.

raise will raise an error inside of the target thread. If the thread hasn’t started yet, in most cases it is killed before running anything:

t = Thread.new do
  puts "never runs"
end
t.raise("knock it off")
sleep 0.01
puts thread_status(t)
#<ThreadStatus status="failed w/ error: knock it off", error=#<RuntimeError: knock it off>

📝 thread_status is the helper we defined in the “Thread API” section on status

The error isn’t raised instantly - only at the point the thread is scheduled next. We sleep 0.1 to give thread t an opportunity to start. The thread scheduler starts it, and it immediately raises our “knock it off” error, effectively running right before puts "never runs".

If the thread gets a chance to start, the error will be raised on whatever line happened to be running last:

t = Thread.new do
  sleep 5
end
t.join(1)
# We're only one second into the threads sleep at this point, so knock it off is raised from the `sleep 5` line
t.raise("knock it off, sleepyhead!")
sleep 0.1
puts thread_status(t)
#<ThreadStatus status="failed w/ error: knock it off, sleepyhead!", error=#<RuntimeError: knock it off, sleepyhead!>

Because it raises whatever error is provided (a RuntimeError if just a string is provided), we can actually rescue the error, ignore it, and retry 😱:

class KnockItOffError < StandardError; end

t = Thread.new do
  sleep 5
  puts "✌️"
rescue KnockItOffError => e
  puts "Nice try #{e}"
  retry
end

t.join(1)
t.raise(KnockItOffError.new("👊"))
t.join

puts thread_status(t)
# Nice try 👊 
# ✌️
#<ThreadStatus status="finished" error=nil>

With raise and kill, issues start to creep in when errors are thrown in an ensure. Here we use a ConditionVariable (we dug into those in The Thread API) to guarantee we raise from the ensure block:

mutex = Mutex.new
condition = ConditionVariable.new
t = Thread.new do
  sleep 5
  puts "✌️"
ensure
  mutex.synchronize do
    # Signal the condition.wait in the main thread
    condition.signal
    # ...perform some cleanup...
    condition.wait(mutex)
    puts "I'll never fire 😔"
  end
end
	
mutex.synchronize do
  # This will wait until our Thread ensure runs and signals us
  condition.wait(mutex)
end
	
t.raise(KnockItOffError.new("👊"))
	
mutex.synchronize do
  # We've enqueued our error, now signal so condition#wait fires in the ensure block
  condition.signal
end
sleep 0.1
	
puts thread_status(t)
#<ThreadStatus status="failed w/ error: 👊", error=#<RuntimeError: 👊>

We don’t see “I’ll never fire 😔”. What happens to our cleanup? Shouldn’t ensure, erm, umm, ensure that things finish…

Moving on from raise, kill stops the thread from running anymore instructions, no matter what it’s doing. raise can be rescue’d, kill can’t.

t = Thread.new do
  sleep 5
  puts "✌️"
# Exception is the root of the Error class hierarchy. If you can't rescue it with this, you can't rescue it 
rescue Exception
  puts "#kill cannot be stopped..."
ensure
  puts "This will still run"
end
	
t.join(1)
t.kill
sleep 0.1
	
puts thread_status(t)
# A #kill'd thread gives no indication it was terminated 😔
#<ThreadStatus status="finished", error=nil>

📝 Technically, kill can be ignored, we’ll explain that when discussing handle_interrupt.

⚠️ Don’t rescue Exception, it’s a bad idea and you could accidentally rescue things like an OutOfMemoryError 😬

Because kill doesn’t raise an error, you actually can’t even tell that the thread was killed. We just get the normal false status, represented in our example by “finished”.

Like raise, kill can also disrupt your ensure methods:

mutex = Mutex.new
condition = ConditionVariable.new
t = Thread.new do
  sleep 5
  puts "✌️"
ensure
  mutex.synchronize do
    condition.signal
    # perform some cleanup
    condition.wait(mutex)
    puts "I'll never fire 😔"
  end
end
	
mutex.synchronize do
  condition.wait(mutex)
end
	
t.kill
	
mutex.synchronize do
  condition.signal
end
sleep 0.1
	
puts thread_status(t)
# ✌️
#<ThreadStatus status="finished" error=nil>

There are a few aliases for kill to be aware of as well:

t = Thread.new do
  Thread.exit # internally gets the current thread, and calls `kill`
end
	
t2 = Thread.new { sleep }
Thread.kill(t) # class method `kill`
t2.exit        # alias for `kill`
t2.terminate   # alias for `kill`

Case closed. Feel free to use raise and kill on your threads. No harm no foul… oh what’s this here?

Ruby’s Thread#raise, Thread#kill, timeout.rb, and net/protocol.rb libraries are broken

Why Ruby’s Timeout is dangerous (and Thread.raise is terrifying)

The Oldest Bug In Ruby - Why Rack::Timeout Might Hose your Server

Timeout: Ruby’s Most Dangerous API

Oh…

Strangely, the thread docs say nothing about the dangers of these methods⁴. These articles are from 2008, 2015 and 2017. Surely no one uses it anymore, considering all that?

Nah.

In fairness to threaded gems that use these methods, they are using the official way you shutdown a thread. And they’re usually taking as many precautions as possible, prior to calling them. For the most part, gems use them as a shutdown mechanism, and give plenty of room for the thread to finish normally first.

The basic problem is this: raise and kill force your code to die at any point, with no guarantee of properly cleaning up.

You might ask: “Couldn’t ctrl+c do the same thing?”. Yes, an OS signal could kill your process or program before an ensure runs, but then all related state is also removed - it can cause other issues, but at least your program cannot limp along in a corrupted state.

ensure
  # but, but Ruby - you _promised_ me this would run 😭
  @value = false
end

So are they pure evil? An occasional necessity? Somewhere in between? I’ll leave that discussion to the code philosophers… in the practical realm, follow these rules:

📝 A small slice of this next section may look familiar. I included a bit of it in The Thread API. This goes much more in-depth

Don’t use Thread.kill, kill or raise unless you really, really know what you’re doing. Same applies for the kill aliases exit and terminate
If you need to stop a thread, you want it to tear down safely. Make it safely interruptible by adding in a condition that can allow your thread to finish early
Perform resource cleanups in an ensure
Don’t use the timeout module
If you use rack-timeout, you really should use term_on_timeout
When you use threads, even implicitly (aka, via configuration in Puma, Sidekiq, Falcon and SolidQueue), you can end up in weird states from them shutting down. It should be rare, but misbehaving threads (very long transactions, runaway CPU) are the most likely to experience this. Pay attention to long running queries or runaway CPU usage and treat it as an important bug
If you have something very critical that must properly cleanup 100% of the time, you need handle_interrupt

(1) Don't kill threads

I’m watching you. Step away from that method, slowly, and no threads have to get hurt.

(2) Interrupt your thread safely

Instead of killing your thread, set it up to be interruptible. Most mature, threaded frameworks operate this way.

still_kickin = Concurrent::AtomicBoolean.new(true)
Thread.new do
  while still_kickin.true?
    # more work!
  end
end

still_kickin.make_false

(3) Ensure cleanup

Whenever you need something to run before a method finishes, you should always use an ensure block. ensure is kind of like a method lifeguard - even if something goes wrong, it’s there for you. It’s the place code goes to ensure it’s run before the method finishes (even when an error is raised).

def read_some_data
  read_it
  close_it # bad
end

def read_some_data
  read_it
ensure
  close_it # good
end

We know ensure is not a silver bullet. Thread#raise and Thread#kill do not respect it. But you’re the most likely to clean things up using an ensure.

(4) Don’t use `timeout`

If you see this in code, be concerned:

require "timeout"

Timeout.timeout(1) do
  # 😱 
end

For some reason, the timeout gem itself doesn’t warn about any issues. But Mike Perham summarizes it best:

There’s nothing that exactly matches what timeout offers: a blanket way of timing out any operation after the specified time limit. But most gems and Ruby features offer a way to be interrupted - there is a repository called The Ultimate Guide to Ruby Timeouts which details everything you need to know. It shows you how to set timeouts safely for basically every blocking operation you could care about timing out. For instance, how to properly handle timeouts using the redis gem:

Redis.new(
  connect_timeout: 1, 
  timeout: 1,
  #...
)

The one piece mentioned in that repository you should leave alone: Net::HTTP open_timeout. Behind the scenes it uses the timeout module 🙅‍♂️. Leave the 60 second default, it should almost never impact you, and you’re probably worse off lowering it.

Primarily people use the timeout gem to manage IO timeouts. In the unlikely case you want to timeout CPU-bound code, it’s up to you to implement it in your processing.

(5) If you use `rack-timeout`, you really should use `term_on_timeout`

rack-timeout works similarly to the timeout module. And I already told you not to use that. So what gives? It will call raise on your threads - isn’t that bad?

The short answer is yes, it’s still bad.

But, rack-timeout is the only real option you have for timing out a web request in Puma. It’s meant as a last resort. From their docs:

rack-timeout is not a solution to the problem of long-running requests, it’s a debug and remediation tool. App developers should track rack-timeout’s data and address recurring instances of particular timeouts, for example by refactoring code so it runs faster or offsetting lengthy work to happen asynchronously.

On top of that, you should have your own lower level timeouts set so that they would fire before rack-timeout.

You’ll want to set all relevant timeouts to something lower than Rack::Timeout’s service_timeout. Generally you want them to be at least 1s lower, so as to account for time spent elsewhere during the request’s lifetime while still giving libraries a chance to time out before Rack::Timeout.

The core issue of any thread raise/kill based solution is corrupted state. When using rack-timeout, you should be using term_on_timeout, ideally set to 1.

term_on_timeout will send a SIGTERM to the worker the thread is running in, which for most servers indicates a need for a graceful shutdown of that process - any potential corrupted state is isolated to that process and will be cleaned up once the process is shutdown.

term_on_timeout only works properly if you’ve got multiple processes serving your requests. And if you get lots and lots of timeouts, it could potentially cause performance problems. See the docs for proper configuration!

There is an alternative idea floating around out there of a way to achieve a “Safer Timeout”, at least in Rails apps:https://web.meetcleo.com/blog/safer-timeouts. Maybe I’ll detail it more in the future, but in the meantime, if you’re in a Rails app I would give it a read.

(6) Monitor your thread cost

Having threads that do not stop easily is a bug. If you’re seeing rack timeout errors, or jobs that can’t be shut down, track it and prioritize fixing it. Treat it like a bug and allocate time to improve it.

(7) Get a `handle_interrupt` on things

Thread.handle_interrupt is One Weird Trick Thread#kill Calls Don’t Want You To Know™. If we’re gonna discuss it, might as well go deep…

`Thread.handle_interrupt`

A thread can be externally “interrupted” by a few things:

Thread#kill
Thread#raise
Your program being exited
A signal, like Ctrl+C

handle_interrupt gives you the ability to control how your program reacts to 1-3. And it means you can define blocks of code which will guarantee their ensure blocks run.

handle_interrupt is a low-level interface and it’s also the one you are least likely to ever need. You’ll see it used in things like threaded web and job servers where low-level control and better cleanup guarantees are helpful. You’ll find examples of it in Sidekiq, the Async fiber scheduler, Homebrew, the parallel gem and more.

When you need the strongest guarantees possible about cleaning up your code in response to “interruption”, handle_interrupt is what you need.

Let’s look at a simple example:

class KnockItOffError < StandardError; end

t = Thread.new do
  sleep 2
  puts "done!"
end
t.join(1)
t.raise(KnockItOffError.new("👊"))
sleep 0.1

puts thread_status(t)
#<ThreadStatus status="failed w/ error: 👊" error=#<KnockItOffError: 👊>>

Run that code 👆 and you’ll never see “done!” print. This is the same type of code we saw in the raise and kill section. What can handle_interrupt do for us?

t = Thread.new do
  Thread.handle_interrupt(KnockItOffError => :never) do
    sleep 2
    puts "done!"
  end
end

t.join(1)
t.raise(KnockItOffError.new("👊"))
sleep 0.1

puts thread_status(t)
# done!
#<ThreadStatus status="failed w/ error: 👊" error=#<KnockItOffError: 👊>>

Now we see “done!” printed! To be clear, the error will still be raised eventually. It can only impact the section it encloses, so the error will be raised right after.

What’s with the interface - what does KnockItOffError => :never mean? Let’s break it down:

handle_interrupt takes a hash. Each key is an exception class object, and each value is a symbol representing how to respond to the exception
KnockItOffError in our example represents the class that will be handled. Descendants of the class are included, so it could be KnockItOffError or any of its descendants.
There are three different symbols allowed as values:
- :never indicates that the exception will “never” interrupt any of the code inside of the block.
- :on_blocking indicates that the exception can only be raised during a “blocking” operation. This includes things like IO read/write, sleep, and waiting on mutexes. Anything that releases the GVL.
- :immediate indicates that the exception should be handled “immediately”. This is effectively the default behavior, so you would generally use this to re-apply an exception ignored at a higher level.

Based on that knowledge, let’s demonstrate a more complex example:

t = Thread.new do
  Thread.handle_interrupt(KnockItOffError => :never) do
    # KnockItOffError can "never" run here
    Thread.handle_interrupt(KnockItOffError => :immediate) do
      # KnockItOffError runs "immediate"ly here
      sleep 2
      puts "done!"
    end
  # KnockItOffError can "never" run here
  ensure
    puts "Can't touch this!"
  end
end
	
t.join(1)
t.raise(KnockItOffError.new("👊"))
sleep 0.1
	
puts thread_status(t)
# Can't touch this!
#<ThreadStatus status="failed w/ error: 👊" error=#<KnockItOffError: 👊>>

In this example, “done!” is never printed because it is in the :immediate block. But we successfully print out “Can’t touch this!” message in our ensure, because we’re within the :never block for KnockItOffError. ensure is now… ensured.

———

We’ve used :never and :immediate, what about :on_blocking?

i = 0
t = Thread.new do
  Thread.handle_interrupt(
    KnockItOffError => :on_blocking
  ) do
    1_000_000.times do
      i += 1
    end
    puts "👋"
  end
end
t.join(0.01)
t.raise(KnockItOffError.new("👊"))
sleep 1
puts "i: #{i}"
puts thread_status(t)
# i: 1000000
#<ThreadStatus status="failed w/ error: 👊" error=#<KnockItOffError: 👊>>

Our increments work fine, as indicated by i being one million. But our puts is a “blocking” call so it gets the boot.

Should we have used a thread safe counter? Let’s try it again using Concurrent::AtomicFixnum from concurrent-ruby, and two threads. We should see i as two million afterwards:

require "concurrent"
	
i = Concurrent::AtomicFixnum.new(0)
	
def incrementing_thread(i)
  Thread.new do
    Thread.handle_interrupt(
      KnockItOffError => :on_blocking
    ) do
      1_000_000.times do
        i.increment
      end
      puts "👋"
    end
  end
end
	
t = incrementing_thread(i)
t2 = incrementing_thread(i)
	
t.join(0.01)
t2.join(0.01)
t.raise(KnockItOffError.new("👊"))
t2.raise(KnockItOffError.new("👊👊"))
sleep 1
puts "i: #{i.value}"
puts thread_status(t)
puts thread_status(t2)
# i: 1237719
#<ThreadStatus status="failed w/ error: 👊" error=#<KnockItOffError: 👊>>
#<ThreadStatus status="failed w/ error: 👊👊" error=#<KnockItOffError: 👊👊>>

Wait, why? i is 1237719? Why is i not two million? What blocked?!

📝 You’ll definitely see a different number. Sometimes you’ll see 1000000, sometimes you’ll see a higher number, but you’ll pretty much never see 2000000

As it turns out, Concurrent::AtomicFixnum uses a Mutex by default. If a Mutex waits to acquire a lock it is considered a blocking operation! That means it qualifies for :on_blocking and the error gets raised.

As a specific fix for AtomicFixnum, if you install the concurrent-ruby-ext gem then you get native extensions which are lock-free, no longer use a Mutex, and properly run our code.

Once we install concurrent-ruby-ext, we properly get 2000000!:

# You ran `gem install concurrent-ruby-ext`
# It automatically loads if present
# ...
puts "i: #{i.value}"
puts thread_status(t)
puts thread_status(t2)
# i: 2000000
#<ThreadStatus status="failed w/ error: 👊" error=#<KnockItOffError: 👊>>
#<ThreadStatus status="failed w/ error: 👊👊" error=#<KnockItOffError: 👊👊>>

But we also know that a Mutex or any other locking/waiting behavior can cause our :on_blocking interrupt to fire. So :on_blocking can have surprising behavior if some other internal of the code were to change later.

———

If the thread hasn’t started yet, handle_interrupt won’t help you. The error will be raised immediately in the thread, before handle_interrupt can be called:

t = Thread.new do
  Thread.handle_interrupt(
    KnockItOffError => :never
  ) do
    puts "welcome!"
    puts "later 👋"
  end
end
t.raise(KnockItOffError.new("👊"))
sleep 1
puts thread_status(t)
#<ThreadStatus status="failed w/ error: 👊" error=#<KnockItOffError: 👊>>

———

What happens after handle_interrupt? Once the error is allowed to raise, code directly after it won’t run:

t = Thread.new do
  Thread.handle_interrupt(
    KnockItOffError => :never
  ) do
    puts "can't stop won't stop"
    Thread.handle_interrupt(
      KnockItOffError => :immediate
    ) do
      sleep 2
      puts "this won't run"
    end
    puts "this won't run either"
  end
end

t.join(1)
t.raise(KnockItOffError.new("👊"))
sleep 0.1
puts thread_status(t)
# can't stop won't stop
#<ThreadStatus status="failed w/ error: 👊" error=#<KnockItOffError: 👊>>

But code after the inner handle_interrupt could run, it just depends on if the previous block raises. In this example, all of the code runs successfully because we don’t raise an error during the inner block:

t = Thread.new do
  Thread.handle_interrupt(
    KnockItOffError => :never
  ) do
    puts "can't stop won't stop"
    Thread.handle_interrupt(
      KnockItOffError => :immediate
    ) do
      sleep 2
      puts "this won't run"
    end
    sleep 2
    puts "this won't run either"
  end
end

t.join(3)
t.raise(KnockItOffError.new("👊"))
t.join # FIXME raises?
puts thread_status(t)
# can't stop won't stop
# this won't run
# this won't run either
#<ThreadStatus status="failed w/ error: 👊" error=#<KnockItOffError: 👊>>

But you’re better off guaranteeing code after the block runs. Use an ensure to make sure even if the inner block raises an error your code still runs:

t = Thread.new do
  Thread.handle_interrupt(
    KnockItOffError => :never
  ) do
    puts "can't stop won't stop"
    Thread.handle_interrupt(
      KnockItOffError => :immediate
    ) do
      sleep 2
      puts "this won't run"
    end
  ensure
    puts "this will consistently run"
  end
end
	
t.join(1)
t.raise(KnockItOffError.new("👊"))
sleep 0.1
puts thread_status(t)
# can't stop won't stop
# this will consistently run
#<ThreadStatus status="failed w/ error: 👊" error=#<KnockItOffError: 👊>>

———

Can we even stop the unstoppable Thread#kill? Yep! From the Thread.handle_interrupt docs:

For handling all interrupts, use Object and not Exception as the ExceptionClass, as kill/terminate interrupts are not handled by Exception.

So we can handle it - but we have to use Object, which looks a bit odd but works well:

t = Thread.new do
  Thread.handle_interrupt(Object => :never) do
    sleep 2
    puts "done!"
  end
end

t.join(1)
t.kill
t.join
puts thread_status(t)
# done!
#<ThreadStatus status="finished" error=nil>

The reason for this odd syntax is that the kill/terminate interrupts are internally handled not as Exception instances, but as integers. That means this would also work:

t = Thread.new do
  Thread.handle_interrupt(
    Integer => :never
  ) do
    sleep 2
    puts "done!"
  end
end

t.join(1)
t.kill
t.join
puts thread_status(t)
# done!
#<ThreadStatus status="finished" error=nil>

Still, you’re better off using Object to avoid the implementation detail.

———

Can we stop the timeout gem from raising at a bad time using handle_interrupt? The Thread API docs used to specifically use timeout as a use-case for handle_interrupt, but there’s a non-determinism bug around thread reuse within the timeout gem.

So once again, don’t use the timeout gem.

I removed the example from the docs because it’s too broken, so on Ruby 3.4+, the docs no longer mention handle_interrupt with the timeout gem.

———

We’ve looked at many handle_interrupt examples - what do real gems use it for?

In the async gem it uses handle_interrupt to ignore SignalException while it shuts down its child tasks:

# Stop all children, including transient children, ignoring any signals.
def stop
  Thread.handle_interrupt(::SignalException => :never) do
    @children&.each do |child|
      child.stop
    end
  end
end

In sidekiq, when it has gracefully attempted a shutdown and is forcing threads to finish, it raises a special error. That error extends Interrupt, which means most rescue blocks will not capture it because it is a child of Exception rather than StandardError:

module Sidekiq
  class Shutdown < Interrupt; end
end
# later...
t.raise(Sidekiq::Shutdown)

To avoid Sidekiq::Shutdown breaking everything (including its own internal code), Sidekiq also uses handle_interrupt to ignore the error in a small piece of shutdown code:

IGNORE_SHUTDOWN_INTERRUPTS = {Sidekiq::Shutdown => :never}
ALLOW_SHUTDOWN_INTERRUPTS = {Sidekiq::Shutdown => :immediate}

def process(uow)
  jobstr = uow.job
  queue = uow.queue_name
     
  #... process logic
  ack = false
  Thread.handle_interrupt(
    IGNORE_SHUTDOWN_INTERRUPTS
  ) do
    Thread.handle_interrupt(
      ALLOW_SHUTDOWN_INTERRUPTS
    ) do
      dispatch(...) do |inst|
        config.server_middleware... do
          execute_job(...)
        end
      end
      ack = true
    rescue Sidekiq::Shutdown
      # Had to force kill this job because it didn't finish
      # within the timeout.  Don't acknowledge the work since
      # we didn't properly finish it.
  ensure
    if ack
      uow.acknowledge
    end
  end
end

If this section hasn’t been enough for you, Ben Sheldon gives some additional interesting examples In his article Appropriately using Ruby’s thread handle_interrupt.

The way you `ensure` success

I’m pretty confident, started from scratch, Ruby would not be implemented with raise and kill again. I don’t know _which _ model they would choose - but something like a Java interrupt would be a good start. And minimally, making all ensure blocks uninterruptable, as well as all finalizers. I didn’t even get into finalizers - they’re a less common, but also important area that you really don’t want to interrupt.

Ruby is one of the only programming languages that lets you kill a thread from outside of the thread. It’s powerful, but mostly, it’s dangerous. It’s one of the sharpest tools available to you, and it should be used sparingly, or ideally not at all.

In threaded code, the best offense is a good defense:

Consider how to safely manage your threads. If your thread is going to do a lot of work, or expensive work, make sure you have an escape hatch (like a boolean check for interrupts)
Treat performance issues like a bug. If you have threads that are hogging resources and timing things out, you need to fix them. killing them is not a long-term answer
Ultimately, there is no way to guarantee the safety of your code 100% of the time. The OS could kill your program. Your server could get unplugged. You’re always going to have edge cases that you can’t foresee. But control what you can, and be aware of what can go wrong.

Now go forth, armed with the knowledge on what to do when good threads go bad.

Unless you’re in SQLite, where apparently N+1 queries are a virtue 😲 https://www.sqlite.org/np1queryprob.html ↩︎
In general, you shouldn’t do this directly ↩︎
And processes.

Or use Falcon. See me later in the series when we talk about Fibers 😏 ↩︎
There’s a small mention about how to handle Timeout errors, but it doesn’t explain much or warn at all ↩︎

Bitmasks, Ruby Threads and Interrupts, oh my!

Tue, 21 Oct 2025 21:12:00 -0500

👋🏼 This is part of series on concurrency, parallelism and asynchronous programming in Ruby. It’s a deep dive, so it’s divided into several parts:

Your Ruby programs are always multi-threaded: Part 1

Your Ruby programs are always multi-threaded: Part 2

Consistent, request-local state

Ruby methods are colorless

The Thread API

Bitmasks, Ruby Threads and Interrupts, oh my!

When good threads go bad

Thread and its MaNy friends

Fibers

Processes, Ractors and alternative runtimes

Scaling concurrency with streaming

Abstracted, concurrent Ruby

Closing thoughts, kicking the tires and tangents

How I dive into CRuby concurrency

You’re reading “Bitmasks, Threads and Interrupts, oh my!". I’ll update the links as each part is released, and include these links in each post.

Interrupting Threads

Interrupting Threads 🧵

The Ruby thread scheduler is rude. There, I said it.

It’s always telling your threads how to run, when to run, how long they can run - it stops them whenever it wants and then tells other threads they have to start working. It’s bossy as hell. On top of that, it’s not even polite about it. It feels free to interrupt your threads - it decides when, and your threads have to just play along and listen.

But threads put up with it. They even benefit from it, if you can believe that. The thread scheduler may be abrupt, but it’s looking out for the runtime. If that’s the case - there must be a good reason… Why do threads put up with these scheduler shenanigans?

Managing threads 👷🏼‍♂️👷🏻‍♀️

An important purpose of a thread scheduler is efficiency and fairness. It manages what threads are running, for how long, and what context each thread gets loaded with.

In normal operation this comes down to a few things:

Time sharing: every thread, under normal circumstances, gets roughly 100ms of CPU runtime¹. Since only one thread can run Ruby code at a time², this keeps a single thread from dominating the program³
Blocking operations: certain operations will “block” a thread. Most forms of IO, and sleep, for instance. When the thread blocks, the thread scheduler allows other threads to run
Priority: we can suggest a priority for our threads, and the thread scheduler will take it into consideration when choosing what to run next and for how long
Passing control: we can suggest actions to the thread scheduler, like passing control or stopping the current thread so other threads can take over
Locking: we can synchronize access to resources, and the thread scheduler chooses the order of access to the resource

How does it handle all this?

An important interruption

The scheduler isn’t a single object - it is the behavior produced by specific VM checks and functions which opt-in to thread scheduling behavior. In this post, we’ll focus on time sharing, priority, and interrupting threads - which are managed through a concept deeply woven into CRuby - a set of functions the VM calls at specific checkpoints.

In the CRuby runtime, this concept revolves around “interrupts”⁴. It contains a set of possible events that could “interrupt” the flow of a Ruby program in general, and different threads in particular. There are several interrupt events possible:

Timer interrupt
Trap interrupt
Pending interrupt
Terminate interrupt
VM barrier interrupt
Postponed job interrupt

In the CRuby internals, these are represented by an integer mask. If we took the C code and represented it in Ruby, it would look like this (each mask is a hex value):

TIMER_INTERRUPT_MASK         = 0x01
PENDING_INTERRUPT_MASK       = 0x02
POSTPONED_JOB_INTERRUPT_MASK = 0x04
TRAP_INTERRUPT_MASK          = 0x08
TERMINATE_INTERRUPT_MASK     = 0x10
VM_BARRIER_INTERRUPT_MASK    = 0x20

Why integer masks?

What is an integer mask, and why would CRuby represent internal states this way?

📝 “mask” is a conventional name for a pattern to isolate specific bits. A “mask” sounds like something that would cover up something else (ie, a mask covering your face). In a sense, these serve a similar purpose - the mask is put on or taken off, clearing or setting bits and testing them efficiently.

Integer masks provide a compact and efficient way to represent multiple program states within a single number. Each state is stored as a single bit - representing a power of 2 - so in concept a 64-bit integer can represent up to 64 independent on/off states. Here’s a visualization of the first 8 bits in a number (a byte):

0 0 0 0 0 0 0 0
| | | | | | | |
| | | | | | | +- 1        (2^0)     0x01
| | | | | | +--- 2        (2^1)     0x02
| | | | | +----- 4        (2^2)     0x04
| | | | +------- 8        (2^3)     0x08
| | | +--------- 16       (2^4)     0x10
| | +----------- 32       (2^5)     0x20
| +------------- 64       (2^6)     0x40
+--------------- 128      (2^7)     0x80
      Byte       Decimal  Power     Hex

This is an adaptation of a visual from a blog post about bitwise operations: https://www.hendrik-erz.de/post/bitwise-flags-are-beautiful-and-heres-why

Being able to represent all this information in a compact way is convenient, but checking whether it matches a particular mask is also very fast. CPUs are well optimized for this sort of thing.

There are several operators for performing these checks called “bitwise” operators. Here’s a table of the operators, and the impact they have on bits:

		AND	OR	XOR	NOT
A	B	A & B	A \| B	A ^ B	~A
0	0	0	0	0	1
0	1	0	1	1	1
1	0	0	1	1	0
1	1	1	1	0	0

We can demonstrate using the Ruby binary syntax⁵:

Bitwise AND:

  0b00000101 &
  0b00000110
# 0b00000100

Bitwise OR:

  0b00000101
  0b00000110
# 0b00000111

Bitwise XOR:

  0b00000101 ^
  0b00000110
# 0b00000011

Bitwise NOT:

  (~0b11111000) & 0xFF
# 0b00000111

The & 0xFF forces us into 8-bits to demonstrate the NOT correctly

These aren’t relevant to masks specifically, but for completeness, you can also shift bits left or right to change values:

  0b00000101 << 1 # Left shift
# 0b00001010

  0b00000101 >> 1 # Right shift
# 0b00000010

Checking for interrupts

The reason these efficient mask checks matter, is because these interrupts are checked a lot.

Here’s a program that simply iterates for a little while, incrementing a counter⁶:

i = 0
while i < 500_000
  i += 1
end

This will check interrupts five hundred thousand times⁷, one check for each iteration of the loop. That’s a lot. And if your programming language is going to do something a lot, it needs to be efficient. The overhead of checking for interrupts should be undetectable in your Ruby program. As discussed earlier, bit mask checks are one of the most efficient checks you can make.

But why does this innocuous program need to check for interrupts so often? It’s part of the thread scheduler opt-in! The Ruby virtual machine is filled with checkpoints where it is safe for Ruby internals to check for interruptions in the program. One of those checkpoints is an if statement (did you think I’d say while loop?!).

Let’s disassemble this into Ruby bytecode:

puts RubyVMInstructionSequence.compile(
  DATA.read
).disassemble
	
__END__
i = 0
while i < 500_000
  i += 1
end

Which gives us the following:

0000 putobject_INT2FIX_0_
0001 setlocal_WC_0           i@0    # | i = 0
0003 jump                    16     # | jump to while i < ... 
...
0009 getlocal_WC_0           i@0    # | i += 1
0011 putobject_INT2FIX_1_           # |
0012 opt_plus                       # |
0014 setlocal_WC_0           i@0.   # |____________
0016 getlocal_WC_0           i@0    # | i < 500_000
0018 putobject               500000 # |
0020 opt_lt                         # |
0022 branchif                9      # | jump to instruction 9,
                                    # | which is i += 1
                                    # |____________
0024 putnil
0025 leave

For the moment you can trust me that branchif is the critical section here. Let’s see how branchif is defined:

DEFINE_INSN
branchif
(OFFSET dst)
(VALUE val)
()
{
    if (RTEST(val)) {
        RUBY_VM_CHECK_INTS(ec);
        JUMP(dst);
    }
}

❗️Woah! What the heck is that weird syntax? Is that Ruby? Is that C?

It’s neither! This is a special, CRuby internal specific DSL that is similar to C. In CRuby, there is a file called insns.def which defines every instruction the Ruby Virtual Machine (YARV) can run.

DEFINE_INSN tells us we are defining an instruction

branchif is the instruction name

OFFSET dst is the argument - 9 in our case, which would take us to 0009 getlocal_WC_0

VALUE val is the last value pushed on the stack - the result of i < 500_000

() that last empty set of parens is the optional return value - we don’t have one - we jump if val is true, or we fall through

Interesting! A couple things stick out to me here when theRTEST(val) (our while condition) is true:

We’re running RUBY_VM_CHECK_INTS anytime we call an if statement. RUBY_VM_CHECK_INTS is a key function for checking the interrupt queue. It’s embedded within VM instructions themselves!
We JUMP to a destination⁸

Fun fact: one of the places RUBY_VM_CHECK_INTS is called is from the once bytecode instruction. An unexpected callback to my article The o in Ruby regex stands for “oh the humanity!”

Jumping to instructions

In the typical case of a branchif, it would jump to the appropriate part of an if statement:

if is_it_true?
  # if it is true, jump here!
else
  # if it isn't, jump here!
end

What caught my eye is that internally an if statement basically acts like goto! Sorry Dijkstra⁹.

And because branchif can jump anywhere you tell it, that also means it can jump to previous code as well. In the case of a while loop, branchif truly takes on its goto roots. Instead of jumping to a future piece of code, it reruns the content of the while loop by jumping back to earlier instructions!

Pervasive interrupts

Want to double the number of checks from our example? Let’s add a method:

def add(a, b)
  a + b
end
	
i = 0
j = 0
while i < 500_000
  i += 1
  j = add(i, j)
end

Now CRuby checks the interrupt queue one million times¹⁰! That’s because of the opt_send_without_block instruction, which is one of the instructions for Ruby method calls:

DEFINE_INSN
opt_send_without_block
(CALL_DATA cd)
(...)
(VALUE val)
{
    // ...
    val = vm_sendish(ec, GET_CFP(), cd, bh, mexp_search_method);
    // Before returning from exec, int check!
    JIT_EXEC(ec, val);
    // ...
}

👆More of that fancy CRuby DSL

We know that interrupts are woven into vm instructions themselves in insns.def, but it’s not alone. Interrupts are checked throughout CRuby. For example, in:

IO
Threads
Processes
The Regex engine
BigNumber

And you’ll find the checks in various forms: RUBY_VM_CHECK_INTS_BLOCKING, RUBY_VM_CHECK_INTS, rb_thread_check_ints, vm_check_ints_blocking, vm_check_ints, etc.

We know what gets called to check for interrupts - but how do these “ints” get set by CRuby?

Interrupt masks

CRuby has macros for setting each of the interrupt flags:

#define RUBY_VM_SET_TIMER_INTERRUPT(ec)
  ATOMIC_OR((ec)->interrupt_flag, TIMER_INTERRUPT_MASK)
#define RUBY_VM_SET_INTERRUPT(ec)
  ATOMIC_OR((ec)->interrupt_flag, PENDING_INTERRUPT_MASK)
#define RUBY_VM_SET_POSTPONED_JOB_INTERRUPT(ec)
  ATOMIC_OR((ec)->interrupt_flag, POSTPONED_JOB_INTERRUPT_MASK)
#define RUBY_VM_SET_TRAP_INTERRUPT(ec)
  ATOMIC_OR((ec)->interrupt_flag, TRAP_INTERRUPT_MASK)
#define RUBY_VM_SET_TERMINATE_INTERRUPT(ec)
  ATOMIC_OR((ec)->interrupt_flag, TERMINATE_INTERRUPT_MASK)
#define RUBY_VM_SET_VM_BARRIER_INTERRUPT(ec)
  ATOMIC_OR((ec)->interrupt_flag, VM_BARRIER_INTERRUPT_MASK)

📝 ec in these examples refers to the “execution context”, which contains per-thread information about the running Ruby program

This ATOMIC_OR macro is an abstraction on top of bitwise operations that stays roughly as efficient, but makes sure the operations run atomically. Multiple operating system threads can run this at the same time - ATOMIC_OR helps to avoid read-modify-write issues.

Those abstractions obscure the actual bitwise operation - we learned a bit about Ruby bitwise operations earlier, let’s show these macros in Ruby form for clarity:

class ExecutionContext
  def initialize
    @interrupt_flag = 0
  end

  def ruby_vm_set_timer_interrupt
    @interrupt_flag |= TIMER_INTERRUPT_MASK
    self
  end
	
  def ruby_vm_set_interrupt
    @interrupt_flag |= PENDING_INTERRUPT_MASK
    self
  end
	
  def ruby_vm_set_postponed_job_interrupt
    @interrupt_flag |= POSTPONED_JOB_INTERRUPT_MASK
    self
  end
	
  def ruby_vm_set_trap_interrupt
    @interrupt_flag |= TRAP_INTERRUPT_MASK
    self
  end
	
  def ruby_vm_set_terminate_interrupt
    @interrupt_flag |= TERMINATE_INTERRUPT_MASK
    self
  end
	
  def ruby_vm_set_vm_barrier_interrupt
    @interrupt_flag |= VM_BARRIER_INTERRUPT_MASK
    self
  end
end

Let’s also add some binary conversion methods so we can conveniently print our results:

def to_b(number)
  number.to_s(2).rjust(8, '0')
end
	
class ExecutionContext
  # ...
  def interrupt_to_b
    to_b(@interrupt_flag)
  end
end

📝 Integer#to_s can be handed a base, which converts to the specified base before returning as a string. In our case, we are converting it to base 2 to show it as binary. We then rjust to pad the left side with 0’s. So for instance, this returns to_b(2) as 00000010.

For reference, here are the CRuby interrupt masks and which bits they set in our interrupt_flag:

0 0 0 0 0 0 0 0 = 0x0 = interrupt_flag
    | | | | | |
    | | | | | +- TIMER_INTERRUPT_MASK
    | | | | +--- PENDING_INTERRUPT_MASK
    | | | +----- POSTPONED_JOB_INTERRUPT_MASK
    | | +------- TRAP_INTERRUPT_MASK
    | +--------- TERMINATE_INTERRUPT_MASK
    +----------- VM_BARRIER_INTERRUPT_MASK

Knowing that, and equipped with our ExecutionContext class, let’s set some flags!

def new_ec
  ExecutionContext.new
end
	
new_ec.ruby_vm_set_timer_interrupt.interrupt_to_b
# => "00000001"
new_ec.ruby_vm_set_interrupt.interrupt_to_b
# => "00000010"
new_ec.ruby_vm_set_postponed_job_interrupt.interrupt_to_b
# => "00000100"
new_ec.ruby_vm_set_trap_interrupt.interrupt_to_b
# => "00001000"
new_ec.ruby_vm_set_terminate_interrupt.interrupt_to_b
# => "00010000"
new_ec.ruby_vm_set_vm_barrier_interrupt.interrupt_to_b
# => "00100000"
	
new_ec.ruby_vm_set_timer_interrupt
  .ruby_vm_set_interrupt
  .ruby_vm_set_postponed_job_interrupt
  .ruby_vm_set_trap_interrupt
  .ruby_vm_set_terminate_interrupt
  .ruby_vm_set_vm_barrier_interrupt
  .interrupt_to_b
# => "00111111"
	
new_ec.ruby_vm_set_timer_interrupt
  .ruby_vm_set_interrupt
  .ruby_vm_set_trap_interrupt
  .ruby_vm_set_vm_barrier_interrupt
  .interrupt_to_b
# => "00101011"

It’s nice we understand how to set them, but they don’t do anything on their own. They must be interpreted by one of the opt-in functions. We’ve been beating around the bush long enough. We’re opting-in, great. What do these opt-in functions actually do?

It's masks all the way down

Let’s start with RUBY_VM_CHECK_INTS. This is a macro that gets replaced with a function call to rb_vm_check_ints. Inside of rb_vm_check_ints, it calls RUBY_VM_INTERRUPTED_ANY, and if that is true it calls rb_threadptr_execute_interrupts:

#define RUBY_VM_CHECK_INTS(ec) rb_vm_check_ints(ec)
static inline void
rb_vm_check_ints(rb_execution_context_t *ec)
{
  if (UNLIKELY(RUBY_VM_INTERRUPTED_ANY(ec))) {
    rb_threadptr_execute_interrupts(rb_ec_thread_ptr(ec), 0);
  }
}

We want to get to rb_threadptr_execute_interrupts, but what does RUBY_VM_INTERRUPTED_ANY do?

static inline bool
RUBY_VM_INTERRUPTED_ANY(rb_execution_context_t *ec)
{
	// ...
    return ATOMIC_LOAD_RELAXED(ec->interrupt_flag) & ~(ec)->interrupt_mask;
}

Seems simple. Let’s translate the code into our Ruby ExecutionContext class:

class ExecutionContext
  # ...
  def ruby_vm_interrupted_any?
    # (flag & ~mask) != 0
    (@interrupt_flag & ~@interrupt_mask) != 0
  end
end

Then set a flag and try it:

new_ec.ruby_vm_set_interrupt.ruby_vm_interrupted_any?
# `ruby_vm_interrupted_any?': undefined method `~' for nil
# (@interrupt_flag & ~@interrupt_mask) != 0
#                    ^

Oops! I didn’t define @interrupt_mask. What is that exactly!? Looks like it’s defined alongside the interrupt_flag on the execution context.

struct rb_execution_context_struct {
  // ...
  rb_atomic_t interrupt_flag;
  rb_atomic_t interrupt_mask;
  // ...
}

👺 It’s a mask! It’s a flag! It’s a… confusing mental model…

We have interrupt_flag, we have the various *_INTERRUPT_MASK constants, and now interrupt_mask. Getting a little lost? I was.

I think it’s helpful to think of interrupt_flag as interrupt_pending, and interrupt_mask as interrupt_blocked. interrupt_flag contains operations waiting to run, and interrupt_mask contains operations that are currently blocked from running.

What is that & ~ business? Remember that & is Bitwise AND, and will only return 1 if both bits are 1. ~ is Bitwise NOT, and will change 1s to 0s, and 0s to 1s. As an example, using the TRAP_INTERRUPT_MASK:

 0 0 0 0 0 0 0 0 = 0x0 = interrupt_flag
 0 0 0 0 0 0 0 0 = 0x0 = interrupt_mask
         |
         +------- TRAP_INTERRUPT_MASK

 0 0 0 0 1 0 0 0 &     # interrupt_flag
~0 0 0 0 1 0 0 0       # interrupt_mask

 0 0 0 0 1 0 0 0 &     # interrupt_flag
 1 1 1 1 0 1 1 1       # interrupt_mask

 0 0 0 0 0 0 0 0       # TRAP_INTERRUPT_MASK is blocked!

It’s only used in a few places - but it seems to serve roles on critical paths, like preventing recursive calls within Signal#trap handlers:

static int
signal_exec(VALUE cmd, int sig)
{
    rb_execution_context_t *ec = GET_EC();
    volatile rb_atomic_t old_interrupt_mask = ec->interrupt_mask;
    // ...
    ec->interrupt_mask |= TRAP_INTERRUPT_MASK;
    // run signal handlers like Signal#trap
    ec->interrupt_mask = old_interrupt_mask;
    // ...
}

Because the interrupt_mask matches the interrupt_flag, RUBY_VM_INTERRUPTED_ANY won’t allow us to recursively trigger a signal handler. If we were to remove the interrupt_mask check, this code would call itself recursively forever and stack overflow:

pid = fork do
  Signal.trap("TERM") do
    Process.kill("TERM", Process.pid)
  end
end
Process.kill("TERM", pid)
Process.waitall
	
# Process.kill': stack level too deep (SystemStackError)

But with the interrupt block code, it just runs forever, endlessly queueing up another trap. It’s kind of hard to find a compelling example of this mask - it mostly seems like very defensive programming!

If you’ve ever written a signal handler in rails and tried using a Rails.logger, you’ve hit the interrupt_mask.

trap("TERM") do
  Rails.logger.info("TRAP fired!")
end
Process.kill("TERM", Process.pid)
# log writing failed. can't be called from trap context

This is because Mutex#lock raises an error if it is used inside an interrupt trap. If the interrupt mask has TRAP_INTERRUPT_MASK set, it means we’re running in a trap and blocking anymore trap interrupts from firing:

static VALUE
do_mutex_lock(VALUE self, int interruptible_p)
{
  rb_execution_context_t *ec = GET_EC();
  rb_thread_t *th = ec->thread_ptr;

  if (th->ec->interrupt_mask & TRAP_INTERRUPT_MASK) {
    rb_raise(rb_eThreadError, "can't be called from trap context");
  }
  // ...
}

Internally, Rails.logger is a Logger instance from the logger gem. That Logger writes to logs using a LogDevice. It uses the MonitorMixin, which gives it a built-in Monitor instance to synchronize with:

class Logger
  class LogDevice
    include MonitorMixin
	
    def write(message)
      handle_write_errors("writing") do
        synchronize do # We can't lock a mutex in a signal!
          # ...
        end
      end
    end
  end
end

📝 You can learn more about Monitors and synchronize in my post on The Thread API

For completeness, let’s add @interrupt_mask to our ExecutionContext class:

class ExecutionContext
  def initialize
    @interrupt_flag = 0
    @interrupt_mask = 0
  end
  # ...
	
  def with_interrupt_mask(mask)
    old_interrupt_mask = @interrupt_mask
    @interrupt_mask |= mask
  ensure
    @interrupt_mask = old_interrupt_mask    
  end
	
  def ruby_vm_interrupted_any?
    # (flag & ~mask) != 0
    (@interrupt_flag & ~@interrupt_mask) != 0
  end
	
  def mask_to_b
    to_b(@interrupt_mask)
  end
end

Now #ruby_vm_interrupted_any? should work! And we can create sections where we block certain interrupts from being fired:

ec = ExecutionContext.new
ec.ruby_vm_set_trap_interrupt
ec.ruby_vm_interrupted_any? # => true
ec.with_interrupt_mask(TRAP_INTERRUPT_MASK) do
  ec.ruby_vm_interrupted_any? # => false
end

The `interrupt`ion we've all been waiting for

Ok, now we know how to check and block the flags, we know the general places they are checked, and we know why it’s valuable for those checks to be efficient. Let’s look at what this has all led up to. What actually happens when an interrupt is detected? We’ll break it down piece-by-piece, but here’s the full function to start. Understanding this function gives us insight into when Ruby decides to yield, raise exceptions, and deliver signals:

int
rb_threadptr_execute_interrupts(rb_thread_t *th, int blocking_timing)
{
  rb_atomic_t interrupt;
  int postponed_job_interrupt = 0;
  int ret = FALSE;
	
  if (th->ec->raised_flag) return ret;
	
  while ((interrupt = threadptr_get_interrupts(th)) != 0) {
    int sig;
    int timer_interrupt;
    int pending_interrupt;
    int trap_interrupt;
    int terminate_interrupt;
	
    timer_interrupt = interrupt & TIMER_INTERRUPT_MASK;
    pending_interrupt = interrupt & PENDING_INTERRUPT_MASK;
    postponed_job_interrupt = interrupt & POSTPONED_JOB_INTERRUPT_MASK;
    trap_interrupt = interrupt & TRAP_INTERRUPT_MASK;
    terminate_interrupt = interrupt & TERMINATE_INTERRUPT_MASK; // request from other ractors
	
    if (interrupt & VM_BARRIER_INTERRUPT_MASK) {
      RB_VM_LOCKING();
    }
	
    if (postponed_job_interrupt) {
      rb_postponed_job_flush(th->vm);
    }
	
    if (trap_interrupt) {
      /* signal handling */
      if (th == th->vm->ractor.main_thread) {
        enum rb_thread_status prev_status = th->status;

        th->status = THREAD_RUNNABLE;
        {
          while ((sig = rb_get_next_signal()) != 0) {
            ret |= rb_signal_exec(th, sig);
          }
        }
        th->status = prev_status;
      }
	
      if (!ccan_list_empty(&th->interrupt_exec_tasks)) {
        enum rb_thread_status prev_status = th->status;

        th->status = THREAD_RUNNABLE;
        {
          threadptr_interrupt_exec_exec(th);
        }
        th->status = prev_status;
      }
    }
	
    /* exception from another thread */
    if (pending_interrupt && threadptr_pending_interrupt_active_p(th)) {
      VALUE err = rb_threadptr_pending_interrupt_deque(th, blocking_timing ? INTERRUPT_ON_BLOCKING : INTERRUPT_NONE);
      ret = TRUE;
	
      if (UNDEF_P(err)) {
        /* no error */
      }
      else if (err == RUBY_FATAL_THREAD_KILLED        /* Thread#kill received */   ||
               err == RUBY_FATAL_THREAD_TERMINATED   /* Terminate thread */       ||
               err == INT2FIX(TAG_FATAL) /* Thread.exit etc. */         ) {
        terminate_interrupt = 1;
      }
      else {
        if (err == th->vm->special_exceptions[ruby_error_stream_closed]) {
          /* the only special exception to be queued across thread */
          err = ruby_vm_special_exception_copy(err);
        }
        /* set runnable if th was slept. */
        if (th->status == THREAD_STOPPED ||
            th->status == THREAD_STOPPED_FOREVER)
          th->status = THREAD_RUNNABLE;
        rb_exc_raise(err);
      }
    }
	
    if (terminate_interrupt) {
      rb_threadptr_to_kill(th);
    }
	
    if (timer_interrupt) {
      uint32_t limits_us = thread_default_quantum_ms * 1000;
	
      if (th->priority > 0)
        limits_us <<= th->priority;
      else
        limits_us >>= -th->priority;
	
      if (th->status == THREAD_RUNNABLE)
        th->running_time_us += 10 * 1000; // 10ms = 10_000us
	
      EXEC_EVENT_HOOK(th->ec, RUBY_INTERNAL_EVENT_SWITCH, th->ec->cfp->self,
                      0, 0, 0, Qundef);
	
      rb_thread_schedule_limits(limits_us);
    }
  }
  return ret;
}

There it is. Guess we’re done here! See you next time!

What a joker I am 🙄.

Let’s start walking through the function. Most of the function lives inside of a while loop. The while loop sets interrupt to the return value of threadptr_get_interrupts. That function gets the current interrupt_flag & ~interrupt_mask, clearing out everything in ec->interrupt_flag in the process (except what was hidden by ec->interrupt_mask). We continue to iterate as long as the interrupt flag doesn’t come back with 0:

int
rb_threadptr_execute_interrupts(rb_thread_t *th, int blocking_timing)
{
  rb_atomic_t interrupt;
  int postponed_job_interrupt = 0;
  int ret = FALSE;

  // ...

  while ((interrupt = threadptr_get_interrupts(th)) != 0) {
    // ...
  }
}

Why are we using while on a single int field, where we check for every mask at once? While we’re checking existing values we’ve pulled from the interrupt_flag, it’s possible new bit masks have been set. If another mask gets set while we’re processing the current interrupts, we keep looping until we return 0.

Next up we use a Bitwise AND to check which masks are currently set. If they’re set, the int will be a non-zero value (truthy), otherwise 0 (falsey). We’ll use those for the if statements later on:

int
rb_threadptr_execute_interrupts(rb_thread_t *th, int blocking_timing)
{
  // ...
	
  while ((interrupt = threadptr_get_interrupts(th)) != 0) {
    int sig;
    int timer_interrupt;
    int pending_interrupt;
    int trap_interrupt;
    int terminate_interrupt;
	
    timer_interrupt = interrupt & TIMER_INTERRUPT_MASK;
    pending_interrupt = interrupt & PENDING_INTERRUPT_MASK;
    postponed_job_interrupt = interrupt & POSTPONED_JOB_INTERRUPT_MASK;
    trap_interrupt = interrupt & TRAP_INTERRUPT_MASK;
    terminate_interrupt = interrupt & TERMINATE_INTERRUPT_MASK; // request from other ractors
    // ...
  }
}

`TIMER_INTERRUPT_MASK`

Now we start checking for work. Let’s start with time slice and priority handling with TIMER_INTERRUPT_MASK. In Your Ruby programs are always multi-threaded: Part 1 I discussed how context gets switched between threads in Ruby:

There are two common reasons context gets switched between threads in CRuby, which can result in operations only partially completing (ie, setting the proper result, then checking that result):

~100ms of Ruby processing have elapsed

A blocking operation has been invoked

The TIMER_INTERRUPT_MASK condition is where we check for that processing time¹¹. On Linux/Unix, CRuby maintains a timer thread which typically checks for work every 10ms. As part of that, it calls RUBY_VM_SET_TIMER_INTERRUPT, which sets the TIMER_INTERRUPT_MASK.

The timer interrupt is fairly straightforward:

Get the current “quantum” (the CRuby name for the amount of time each thread can run before being context switched)
Priority is used to increase or decrease the amount of time a thread can run above or below the default
It is assumed the thread ran 10ms before this code, so it adds 10ms to the running_time
An event hook is fired notifying interested plugins that a thread context switch is happening
Calls rb_thread_schedule_limits.

if (timer_interrupt) {
  uint32_t limits_us = thread_default_quantum_ms * 1000;
	
  if (th->priority > 0)
    limits_us <<= th->priority;
  else
    limits_us >>= -th->priority;
	
  if (th->status == THREAD_RUNNABLE)
    th->running_time_us += 10 * 1000; // 10ms = 10_000us
	
  EXEC_EVENT_HOOK(th->ec,
    RUBY_INTERNAL_EVENT_SWITCH, th->ec->cfp->self,
    0, 0, 0, Qundef);
	
  rb_thread_schedule_limits(limits_us);
}

Since Ruby 3.4, you can set your own thread_default_quantum_ms using the env variable RUBY_THREAD_TIMESLICE. This means the long-held CRuby constant of 100ms time slices is now adjustable, and folks have been adjusting it to handle different CPU saturated workloads.

rb_thread_schedule_limits checks if the thread is over its allotted running time, and yields if so:

static void
rb_thread_schedule_limits(uint32_t limits_us)
{
  rb_thread_t *th = GET_THREAD();

  if (th->running_time_us >= limits_us) {
    thread_sched_yield(TH_SCHED(th), th);
    rb_ractor_thread_switch(th->ractor, th, true);
  }
}

We’ve discussed bit manipulation quite a bit - feels negligent to not briefly discuss that right and left bit shift for priority 🤷🏻‍♂️.

if (th->priority > 0)
  limits_us <<= th->priority;
else
  limits_us >>= -th->priority;

If th->priority is greater than zero, we shift every bit left. If not, it negates the priority (so negative priorities turn positive) and shifts every bit right. We can demonstrate how this would work easily in Ruby, using milliseconds (ms) instead of microseconds (us) for simplicity:

def calculate_priority(priority, limit)
  priority > 0 ? limit << priority : limit >> -priority
end
	
calculate_priority(0, 100)        # => 100
calculate_priority(2, 100)        # => 400
calculate_priority(-2, 100)       # => 25
to_b(100)                         # => 01100100  = 100
to_b(calculate_priority(0, 100))  # => 01100100  = 100
to_b(calculate_priority(2, 100))  # => 110010000 = 400
to_b(calculate_priority(-2, 100)) # => 00011001  = 25
	
#  01100100    01100100
#    << 2        >> 2
# 110010000    00011001

That means that at the default quantum of 100ms, if you give a CRuby thread a priority of 2, it will be given 400ms of runtime before being forced to switch! And -2 means your thread will only run for 25ms at a time. When we shift right, we lose bits, which is why the value is lower.

`TRAP_INTERRUPT_MASK`

Now we’re onto signal handling using TRAP_INTERRUPT_MASK. The first part is what you might expect from a “trap” interrupt - signal handling. According to this code - you’ll only ever run trap handlers on the main thread. If there is a trap mask and we aren’t on the main thread, we ignore it:

/* signal handling */
if (th == th->vm->ractor.main_thread) {
  // ...
}

The thread in this Ruby example will always equal Thread#main:

trap("INT") do
  puts "hello from #{Thread.current}: #{Thread.current == Thread.main}"
  # => hello from #<Thread:0x000000010445b2a8 run>: true
end

Next we iterate through each available signal. If multiple signals have not been processed, we process them all here. rb_signal_exec internally calls signal_exec, which we looked at earlier:

while ((sig = rb_get_next_signal()) != 0) {
  ret |= rb_signal_exec(th, sig);
}

Prior to Ruby 3.4, that was the exclusive purpose of TRAP_INTERRUPT_MASK. Ruby 3.4+ also uses it to alert other threads that there is work for them to execute. You put work into the threads interrupt_exec_tasks list, and call threadptr_interrupt_exec_exec on each thread:

if (!ccan_list_empty(&th->interrupt_exec_tasks)) {
  // ...
  threadptr_interrupt_exec_exec(th);
  // ...
}

threadptr_interrupt_exec_exec runs the requested task (a function), either in a new thread, or inline:

if (task->flags & rb_interrupt_exec_flag_new_thread) {
  rb_thread_create(task->func, task->data);
}
else {
  (*task->func)(task->data);
}

Seems generally handy, but was introduced for a specific purpose: supporting require and autoload inside of Ractors:

# Ruby < 3.3
Ractor.new { pp "hey there!" } # autoloads `pp`
# => `require': can not access non-shareable objects in constant Kernel::RUBYGEMS_ACTIVATION_MONITOR by non-main ractor. (Ractor::IsolationError)
Ractor.new {
  require "json"
  puts JSON.parse('"hey there!"')
}
# => `require': can not access non-shareable objects in constant Kernel::RUBYGEMS_ACTIVATION_MONITOR by non-main ractor. (Ractor::IsolationError)

# Ruby >= 3.4
Ractor.new { pp "hey there!" }
# => "hey there!"
Ractor.new {
  require "json"
  puts JSON.parse('"hey there!"')
}
# => hey there!

Requiring a gem requires accessing non-shareable objects - Ractors cannot access any state that is non-shareable. The only Ractor with access to these non-shareable objects is the main Ractor, Ractor.main. To get around this, non-main Ractors add a task to the interrupt_exec_tasks list on the main Ractor thread, and set TRAP_INTERRUPT_MASK:

rb_ractor_t *main_r = GET_VM()->ractor.main_ractor;
// for `require` calls
rb_ractor_interrupt_exec(main_r, ractor_require_func...)
// for autoloading, like when calling `pp`
rb_ractor_interrupt_exec(main_r, ractor_autoload_load_func...)

// ...
// ractor_require_func/ractor_autoload_load_func are
//   referenced in task->node
ccan_list_add_tail(&th->interrupt_exec_tasks, &task->node);

`PENDING_INTERRUPT_MASK`

Now we’ve got the heavy-hitter of thread interrupts - PENDING_INTERRUPT_MASK. It’s not clear from the name, but this mask gets set by Thread#raise and Thread#kill. It doesn’t get much more interruptive than arbitrarily raising an error within, or killing a thread.

The reason it’s called PENDING_INTERRUPT_MASK is because it indicates there are errors waiting to be evaluated in the threads pending_interrupt_queue. Every thread has a pending_interrupt_queue, and it manages the interrupts that have been enqueued by calls like Thread#raise and Thread#kill. Sometimes those interrupts are actual error instances (Thread#raise), and sometimes they are integer flags (Thread#kill).

We start off by checking if there are any active pending interrupts in the queue. If there are, we dequeue the first available interrupt. The blocking_timing relates to the #handle_interrupt method, and we’ll dig into those next time in “When good threads go bad”. For now, just know it gives you the ability to defer the thread being interrupted:

/* exception from another thread */
if (pending_interrupt && threadptr_pending_interrupt_active_p(th)) {
  VALUE err = rb_threadptr_pending_interrupt_deque(th, blocking_timing ? INTERRUPT_ON_BLOCKING : INTERRUPT_NONE);
  // ...
}

We check if the dequeued interrupt is one of the flags set by Thread#kill/Thread#terminate/Thread#exit, all representing that the thread should be killed immediately. We set the terminate_interrupt, which later in the function triggers rb_threadptr_to_kill. This kills the thread and cannot be rescued:

if (/* Thread#kill received */
    err == RUBY_FATAL_THREAD_KILLED ||
    /* Terminate thread */
    err == RUBY_FATAL_THREAD_TERMINATED ||
    /* Thread.exit etc. */
    err == INT2FIX(TAG_FATAL)) {
  terminate_interrupt = 1;
}

// ...
// outside of the pending interrupt if statement
if (terminate_interrupt) {
  rb_threadptr_to_kill(th);
}

If the error isn’t one of the Thread#kill flags, it must be an actual Ruby exception. We make sure the thread is in a running state. Then we force it to raise an error at whatever point in the code it goes to execute next. This raises whatever error we set with Thread#raise:

/* set runnable if th was slept. */
if (th->status == THREAD_STOPPED ||
    th->status == THREAD_STOPPED_FOREVER)
  th->status = THREAD_RUNNABLE;
rb_exc_raise(err);

Personally, I was surprised to find that these interrupts are stored in a queue! Can we try to prove it in our Ruby code? Let’s try:

CatchyError = Class.new(StandardError)

class ErrorCatcher
  def self.===(exception)
    exception.message =~ /1|2|3/
  end
end

t = Thread.new do
  sleep
rescue ErrorCatcher
  redo
rescue CatchyError
  raise
end

sleep 0.1
t.raise(CatchyError.new('1'))
t.raise(CatchyError.new('2'))
t.raise(CatchyError.new('3'))
t.raise(CatchyError.new('4'))
t.join
# => #<Thread:0x0000000120961420 (irb):84 run> terminated with exception (report_on_exception is true):
#   in 'Kernel#sleep': 4 (CatchyError)

In the above code:

We setup a dynamic error matcher so we can raise the same error, but catch it differently depending on the message¹²
We rescue and redo if we get a CatchyError with 1, 2, or 3 as the message
Even though we t.raise four times, only the fourth CatchyError is raised. If you change the regex to match /1|2/, it will fail on the third error instead
It really is running through the queue of errors!

`TERMINATE_INTERRUPT_MASK`

TERMINATE_INTERRUPT_MASK is pretty niche. You’ll remember this code from the Thread#kill code earlier triggered by the pending_interrupt_queue:

if (terminate_interrupt) {
  rb_threadptr_to_kill(th);
}

There are two ways to trigger that code:

Using Thread#kill, as we already learned
When a Ruby process is shutting down. As part of that shutdown, all Ractors are terminated, which set TERMINATE_INTERRUPT_MASK on each of their threads

`POSTPONED_JOB_INTERRUPT_MASK`

Still in the niche-zone, we’ve got POSTPONED_JOB_INTERRUPT_MASK.

This mask is used when work needs to be performed, but can’t safely run in its current context. By making it an interrupt mask, the work can be inserted into a safe point for execution in the CRuby runtime:

if (postponed_job_interrupt) {
  rb_postponed_job_flush(th->vm);
}

The rb_postponed_job_flush function iterates through work in the postponed_job_queue, calling each function in the queue.

In CRuby, I can only find references to it in the Tracepoint source code. In concept, it seems very similar to the interrupt_exec_tasks used for Ractor#require. I’m sure there is a CRuby committer who could explain this further - I’d be curious to understand it better!

`VM_BARRIER_INTERRUPT_MASK`

Not to be outdone by TERMINATE_INTERRUPT_MASK and POSTPONED_JOB_INTERRUPT_MASK, we’ve got king-niche: VM_BARRIER_INTERRUPT_MASK. When set, it runs RB_VM_LOCKING() on the thread:

if (interrupt & VM_BARRIER_INTERRUPT_MASK) {
  RB_VM_LOCKING();
}

It’s niche, but seems to play an important role in giving the entire VM exclusive access to an operation. This appears to have been introduced with Ractors in Ruby 3. That makes sense - Ractors are the first truly parallel unit of execution in Ruby.

Certain operations, like YJIT compiling bytecode, require exclusive access to the VM when running. For instance, when rb_yjit_compile_iseq is called, the first thing it does is call rb_vm_barrier:

void
rb_yjit_compile_iseq(const rb_iseq_t *iseq, rb_execution_context_t *ec, bool jit_exception)
{
    RB_VM_LOCKING() {
        rb_vm_barrier();

rb_vm_barrier sets the VM_BARRIER_INTERRUPT_MASK on all running threads across Ractors, then waits for each to stop at the barrier:

// interrupts all running threads
rb_thread_t *ith;
ccan_list_for_each(&vm->ractor.sched.running_threads, ith, sched.node.running_threads) {
  if (ith->ractor != cr) {
    RUBY_VM_SET_VM_BARRIER_INTERRUPT(ith->ec);
  }
}

// wait for other ractors
while (!ractor_sched_barrier_completed_p(vm)) {
  ractor_sched_set_unlocked(vm, cr);
  rb_native_cond_wait(&vm->ractor.sched.barrier_complete_cond, &vm->ractor.sched.lock);
  ractor_sched_set_locked(vm, cr);
}

———

😮‍💨 We dug deep in this one. Bitmasks, CRuby internals, Thread management - what could be next? With all this knowledge, we’re primed and ready to dig into what to do when a thread goes rogue. See you next time in “When good threads go bad” 👋🏼

With Aaron Patterson’s PR to have a configurable quantum, this can be configured now. But whatever it’s set to is still static during a programs execution, and still defaults to 100ms ↩︎
https://jpcamara.com/2024/06/04/your-ruby-programs.html#heisenbugs ↩︎
It can still happen, but it’s less likely. We’ll discuss ways it can happen later in the series ↩︎
JRuby does as well! https://github.com/jruby/jruby/blob/master/core/src/main/java/org/jruby/RubyThread.java#L822 ↩︎
The different number syntaxes in Ruby (binary, octal, decimal, hex) are really just sugar on the Integer class. So when you run these code snippets you’ll actually see a decimal integer rather than the binary I show in the comment ↩︎
If you don’t want to use a while loop, there’s a great alternative here: https://bsky.app/profile/jpcamara.com/post/3m2ntcgzhe22c ↩︎
1,001,102, to be exact

See https://jpcamara.com/2024/11/28/counting-c-method.html for how I got that number ↩︎
If we simplified the loop to be while(true); end, we’d actually exclusively use the jump instruction: https://redgetan.cc/understanding-timeouts-in-cruby/#6-working-examples ↩︎
But interesting way to think about an if statement! ↩︎
Fun fact - if you run this example with YJIT it goes back down to five hundred thousand. YJIT seems to inline the method call and that bypasses the ints check ↩︎
This particular check is technically linux/unix specific. On Windows, thread_win32.c is used and it maintains its own timer thread and priority controls specific to Windows. ↩︎
Thanks to the Honeybadger blog for the tip on dynamic exception matchers!

https://www.honeybadger.io/blog/level-up-ruby-rescue-with-dynamic-exception-matchers/ ↩︎

The /o in Ruby regex stands for “oh the humanity!”

Sat, 02 Aug 2025 00:16:06 -0500

Your code using the /o modifier

Source: wikipedia

Hi there! Do you like Regex? Do you like performance? Do you like creating confounding bugs for yourself rooted in the mechanics of the Ruby VM itself?

If you said yes to all of the above, have I got a feature for you!

But first, let’s start with a story.

The cliffs of insanity

I was recently reviewing some code, and part of the functionality was about matching. A class took an array of strings, and you could call a method to see if an input matched part of any of the strings. Stripped down, it was effectively the following code:

class Matcher
  def initialize(matchables)
    @matchables = matchables
  end

  def matches_any?(input)
    @matchables.any? { |m| m.match?(/#{input}/io) }
  end
end

I know there are some of you reading this code and thinking “does this really need a regex?”, “couldn’t it just use include? and some downcasing?”, “does this even need to exist?”, etc, etc. I see you, I hear you, I’d probably think the same, and I promise you the specifics of this method aren’t that important.

Functionally, the code looked ok to me. I knew what /i was (a regex in case-insensitive mode), but I didn’t recognize /o. It didn’t seem critically important to lookup yet. Tests were not exhaustive but were green, and so I went to run the code in a console:

Matcher.new(["Ruby!"]).matches_any?("ruby")
=> true
Matcher.new(["Ruby!"]).matches_any?("something else")
=> true
Matcher.new(["Ruby!"]).matches_any?("javascript")
=> true

Well, that seemed broken. Assuming it was a bug, I looked at the code to see what was wrong. But nothing stuck out. The code seemed ok, and simple:

@matchables.any? |m| m.match?(/#{input/io) }

So I deconstructed the code and ran it directly, outside the class:

["Ruby!"].any? { |m| m.match?(/ruby/io) }
=> true
["Ruby!"].any? { |m| m.match?(/something else/io) }
=> false

Even weirder. Run directly, it ran as expected. What if I started a new console, and ran the original class in a different order?

Matcher.new(["Ruby!"]).matches_any?("something else")
=> false
Matcher.new(["Ruby!"]).matches_any?("ruby")
=> false
Matcher.new(["something else"]).matches_any?("Ruby!")
=> true

Well, that was interesting. It seems like whatever value I sent to matches_any? was cached for every run after that point, even for brand new objects.

I looked through the class again. There was no weird memoization. No class-level variables. No thread locals. I was instantiating the class each time I ran matches_any?. I was out of ideas for the predictable things that cause unpredictable results. What else was there?

/o the humanity!

With nothing else to investigate, I finally looked up the docs for what the /o regex modifier does. /o is referred to as “Interpolation mode”, which sounded pretty harmless. The Ruby docs have a succinct section on the expected behavior: https://ruby-doc.org/3.4.1/Regexp.html#class-Regexp-label-Interpolation+Mode

Modifier o means that the first time a literal regexp with interpolations is encountered, the generated Regexp object is saved and used for all future evaluations of that literal regexp

Just reading that, I still wasn’t sure I’d expect what I was seeing. It almost sounded like internally Ruby would cache each different interpolation that comes through. As if it would maybe reuse an internal regex if the same string value was interpolated. They provide a code example that makes it a little clearer:

def letters; sleep 5; /[A-Z][a-z]/; end
words = %w[abc def xyz]
start = Time.now
words.each {|word| word.match(/\A[#{letters}]+\z/) }
Time.now - start # => 15.0174892

start = Time.now
words.each {|word| word.match(/\A[#{letters}]+\z/o) }
Time.now - start # => 5.0010866

The letters method in the first example is called three times. One time for each iteration, as expected, taking 15 seconds total.

In the second example, the code iterates three times, but the letters method is called once, taking 5 seconds total.

Knowing that, I went back to the original code with new eyes:

class Matcher
  def initialize(matchables)
    @matchables = matchables
  end
	
  def matches_any?(input)
    @matchables.any? { |m|
      m.match?(
        /#{input}/io # This is run once, _ever_
      )
    }
  end
end

What that meant for me is that the regular expression inside of matches_any? was interpolating the first value it ever receives. Past that point the regex never interpolated another value.

As it turned out, the developer had read about it as a potential regex optimization. It seemed harmless enough, so they added it. Now that I knew why I was hitting the issue, I removed the /o and everything started working properly.

But it still was not clear to me how this was possible. What in the world is Ruby doing internally? Let’s figure it out together.

Inside the VM

Sometimes the only way to understand a behavior in Ruby is to drop a bit lower. Let’s disassemble the code into Ruby VM byte code, to see if it gives us any clues. I’ll use the DATA feature to be able to put it into a script directly (you can find more about that syntax here).

puts RubyVM::InstructionSequence.compile(DATA.read).disassemble
	
__END__
class Matcher
  def initialize(matchables)
    @matchables = matchables
  end
	
  def matches_any?(input)
    @matchables.any? { |m| m.match?(/#{input}/io) }
  end
end

There are a bunch of other instructions you’ll see if you run that code, but we’ll focus on the instructions specific to the block in the matches_any? method:

# { |m| m.match?(/#{input/io) }

== disasm: #<ISeq:block in matches_any?@<compiled>:7 (7,21)-(7,51)>
local table (size: 1, argc: 1 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1])
[ 1] m@0<AmbiguousArg>
0000 getlocal_WC_0                          m@0                       (   7)[LiBc]
0002 once                                   block (2 levels) in matches_any?, <is:0>
0005 opt_send_without_block                 <calldata!mid:match?, argc:1, ARGS_SIMPLE>
0007 leave                                  [Br]

It describes:

Getting local variable m
Running the… once instruction?
Calling m with opt_send_without_block with a method id (mid) of match?
Then leaveing.

All of these instructions are very common in the Ruby VM. All except one - the once instruction. I’ve never heard of that!

What is once? It can’t exist almost solely for the purposes of this extremely strange regex modifier, can it? Surely this modifier is not built into the very bones of Ruby?

`once` upon a time

If you are ever curious to know how Ruby VM instructions get interpreted in CRuby, there is a central file to all of it called insns.def. It contains all of the available YARV (Yet Another Ruby VM) instructions in a C-esque format which is compiled into an actual C file as part of building the language.

In normal program execution, without JIT optimizations applied (like YJIT, ZJIT, MJIT), you can trace how each instruction is executed by reading insns.def. Let’s look at the once definition to see what kind of dark magic is being invoked.

/* run iseq only once */
DEFINE_INSN
once
(ISEQ iseq, ISE ise)
()
(VALUE val)
{
    val = vm_once_dispatch(ec, iseq, ise);
}

In essence it’s very simple - it takes the instruction and runs it using vm_once_dispatch. iseq is the instruction sequence, and ise is an “Inline Storage Entry”, which is a place to cache an instruction result. What does vm_once_dispatch do?

static VALUE
vm_once_dispatch(rb_execution_context_t *ec, ISEQ iseq, ISE is)
{
    rb_thread_t *th = rb_ec_thread_ptr(ec);
    rb_thread_t *const RUNNING_THREAD_ONCE_DONE = (rb_thread_t *)(0x1);
	
  again:
    if (is->once.running_thread == RUNNING_THREAD_ONCE_DONE) {
        return is->once.value;
    }
    else if (is->once.running_thread == NULL) {
        VALUE val;
        is->once.running_thread = th;
        val = rb_ensure(vm_once_exec, (VALUE)iseq, vm_once_clear, (VALUE)is);
        // ... skipped ...
        is->once.running_thread = RUNNING_THREAD_ONCE_DONE; /* success */
        return val;
    }
    else if (is->once.running_thread == th) {
        /* recursive once */
        return vm_once_exec((VALUE)iseq);
    }
    else {
        /* waiting for finish */
        RUBY_VM_CHECK_INTS(ec);
        rb_thread_schedule();
        goto again;
    }
}

It’s a little daunting at first look, but broken down it’s just some simple if statements:

is->once.running_thread == RUNNING_THREAD_ONCE_DONE
If the instruction cache is set to RUNNING_THREAD_ONCE_DONE (a flag to identify being done vs pointing at a thread), it returns its cached value. Forever.
is->once.running_thread == NULL
If there is no running thread, congratulations! Your thread is the winner! You get to decide what value gets cached! Every other thread trying to run this instruction will wait until you have produced a value (by calling vm_once_exec), then use whatever value you produced. Forever.
is->once.running_thread == th
If there is a running thread set, and it’s the current thread, we’re in a recursive once (terrifying…)
Otherwise you are a different thread, and you have to wait. You’ll keep checking for a value immediately, or get rescheduled and check it a little later (once some other threads have been given a slice of time)

We’ve broken down the logic. Now let’s break down the implications.

If you use /o in your regex, the content of the regex will be evaluated once, ever. Even if it’s inside of an instance method. Even if it’s inside of a loop with a thousand iterations. It will never be evaluated again. That’s the content you’ve got for your regex. This is why the code I was reviewing was so confusing - even though it lived in an object, and was only created local to that method - it implicitly created a constant, immutable value.

Now we can understand how the original code worked, by evaluating it relative to how the CRuby internals work. The first call we have no value set - past that point we will always use the value literally cached inside of the VM itself:

# We haven't run the code yet, so in our starting state:
#   is->once.running_thread == NULL
#   m.match?(/#{input}/io)
> Matcher.new(["Ruby!"]).matches_any?("something else")
# Now we've run the code once, we're done!
#   is->once.running_thread == RUNNING_THREAD_ONCE_DONE
#   m.match?(/something else/io)

> Matcher.new(["Ruby!"]).matches_any?("ruby")
#   is->once.running_thread == RUNNING_THREAD_ONCE_DONE
#   m.match?(/something else/io)

> Matcher.new(["something else"]).matches_any?("Ruby!")
#   is->once.running_thread == RUNNING_THREAD_ONCE_DONE
#   m.match?(/something else/io)

If the code has not been run yet, you’re in the starting state of is->once.running_thread == NULL. This means two things. One is clear - the first code to run this gets to determine the value set. Two is a little less clear - it is non-deterministic what value will win!

If you’ve read any of my series on Ruby concurrency, you’ll know I love a good thread non-determinism example. Here we create a method containing an “interpolation mode” regex. Then we create five threads, and call the method from each thread. To introduce some context switching, we sleep for random amounts (this could also be caused by IO, or long-running code, alternatively):

def once_and_for_all(input)
  /#{input}/o
end

5.times.map do |i|
  Thread.new { sleep(rand); p once_and_for_all(i) }
end.map(&:join)

# Run it once, it prints /3/ 5 times
# Run it again, it prints /1/ 5 times
# Run it again...

In my first run, I see the regex /3/ printed 5 times. Each run gave me a different result. Run this code several times and you will likely see a different value printed each time. It may repeat at times, but there will be no consistency.

Quite the behavior! This is pretty close to being a Heisenbug. Non-determinism at its finest.

The code has been run already, but now it’s being run again before being set as RUNNING_THREAD_ONCE_DONE: is->once.running_thread == th. This can only happen within the same thread, and it’s there to handle recursion. It’s hard to imagine using the /o at all, let alone recursively. If you were to do that I’d only have one question for you: “Who hurt you?”

But let’s do the damn thing. Here’s a recursive case of the /o modifier. Heaven help us.

def recursive_regex(n)
  puts "Evaluating regex for n=#{n}"
  if n > 0
    /#{recursive_regex(n - 1)}/o # This calls itself recursively
  else
    "base"
  end
end

recursive_regex(5)
# Evaluating regex for n=5
# Evaluating regex for n=4
# Evaluating regex for n=3
# Evaluating regex for n=2
# Evaluating regex for n=1
# Evaluating regex for n=0

recursive_regex(5)
# Evaluating regex for n=5

# Call it with whatever value you want, it's never running the method recursively again.
recursive_regex(500)
# Evaluating regex for n=500

I promise it hurt me more to write that than it did for you to read it.

This example does bring up an interesting point though! Our examples so far have been simple values interpolated into the regex. In this case, we’re calling a method and it is still only being evaluated once. The operation you run in the interpolation is irrelevant - whatever is interpolated will never be run again with the /o modifier.

If none of the other conditions are met, it means the interpolation is being evaluated by a different thread. All we can do here is wait to see what the value is.

This is the same concept as the threading example, but this is about the threads that lose the race. The most interesting part here is that it means this interpolation is guaranteed to be thread safe in CRuby (at least in the sense that other threads are locked out of it, not that you get a deterministic result). That will actually matter a little later, if you can believe it.

Why does it exist?

I didn’t find a clear origin of this modifier. But it’s been around for 20+ years. It’s likely been around almost as long as Ruby itself. Matz, you scoundrel, you.

Shockingly, every other forum or blog post I found in relation to the /o modifier exclusively spoke about the performance benefits of it. I didn’t find any warnings at all. But mostly I found it used in a scripting context. In a single, simple script run, maybe you’re safe. But the downside seems way worse than any potential upside. If you really need a speed boost - just cache the regex yourself?

interpolated_once = /#{ARGV.first}/
interpolated_once.match?("a string")
interpolated_once.match?("a string 2")
interpolated_once.match?("a string 3")
# Just as performant as /o?

The earliest reference I could find to the existence of a /o modifier in regex was in a 2003 post on a Perl forum. Based on this, I’m guessing Ruby borrowed it from Perl. Even in 2003, the prevailing wisdom was already clear:

The best heuristic is: Never use /o

I agree with that 2003 Perl person. You don’t need it. You should not use it. Step away from the modifier, and no one has to get hurt.

I’ll take your `once`, and raise you…

We’re all having fun here, right? Should we take this confounding modifier, and muddy things a bit more?

There is actually a way to force the /o modifier to re-evaluate.

hushed whispers!

someone in the back of the room faints!

I stumbled upon it because I was testing this code in a Rails console. I found that every time I called reload!, I was able to test my method from scratch again. Why would that be?

It’s because when you reload!, all of your code is re-evaluated by Ruby. And when it gets re-evaluated, you get new bytecode, and a new cache! Take this for instance:

def one_time(input)
  /#{input}/o
end

p one_time("hi there!")

def one_time(input)
  /#{input}/o
end

p one_time("how are you?")
# /hi there!/
# /how are you?/

Even though we used the /o modifier, we were able to re-evaluate it. That’s because we overwrote our one_time implementation, which gave us a new once cache.

This matters because it means you can’t even truly guarantee your /o regex will only be run once. If a piece of monkey patch code gets loaded and overwrites your method, you will get even more non-deterministic behavior.

The inmates are running the asylum

In a moment of pure serendipity, the day after I started writing this post I happened to read Jared Norman’s Code Reloading for Rack Apps. The article teaches you how to build the same kind of code reloading Rails has, but in a standalone Rack application. You should give it a read.

In it, he creates a class called Once.

This class let’s you create an object that encapsulates a bit of code and only ever lets it run once, even if it’s called across multiple threads.

-Jared Norman

class Once
  def initialize(&block)
    @block = block
    @mutex = Mutex.new
  end

  def call
    @mutex&.synchronize do
      return unless @mutex

      # Ignore the \, it's for a markdown issue...
      \@block.call

      @mutex = nil
    end
  end
end
	
o = Once.new do
  puts "I should only happen once"
end
	
100.times.map { Thread.new { o.call } }.each(&:join)
# => I should only happen once

Any thread attempting to run this code will block on the @mutex.synchronize. It runs @block.call, then at the end of the synchronize block, the @mutex is set to nil. This means that after the first thread, every other thread either exits early because @mutex is nil, or never runs the synchronize block at all because of the safe-nav. Makes sense!

As if I was destined to read this by some kind of trickster regex god, what did I see at the end of the article? A reimplementation of his Once class using the /o modifier!

A veritable devil on Jared’s shoulder, John Hawthorn gave him the idea. It would make sense that John, a Ruby internals wizard, would suggest such a thing. Then Jared, author of Advent of Criminally Bad Ruby Code, would actually decide to include it.

Here is their ingenious abomination.

class Once
  def initialize(&block)
    @block = block
  end

  def call
    /#{@block.call}/o # THE HORROR
  end
end

o = Once.new do
  puts "I should only happen once"
end

100.times.map { Thread.new { o.call } }.each(&:join)
# => I should only happen once

Using /o, the code examples still behave correctly and only run once despite the many threads attempting to run it.

It does, however, have a fatal flaw in comparison to the original code. Can you guess it?

Let’s demonstrate by creating multiple instances:

o1 = Once.new { puts "I should only happen once 1" }
o2 = Once.new { puts "I should only happen once 2" }
o3 = Once.new { puts "I should only happen once 3" }

100.times.map { Thread.new { o1.call } }.each(&:join)
100.times.map { Thread.new { o2.call } }.each(&:join)
100.times.map { Thread.new { o3.call } }.each(&:join)

In the original, mutex based implementation, we’d see the following:

# => I should only happen once 1
# => I should only happen once 2
# => I should only happen once 3

In the /o based implementation, we see this:

# => I should only happen once 1

Because of the instruction-level caching, we can only have one result from Once#call, ever. A class that literally can only be run reliably a single time. “Once”, indeed.

It’s probably more efficient than the mutex approach and maybe in some bizarro world where you truly needed a piece of code to be thread safe and lazy evaluated and as efficient up front as possible and only ever run once… maybe…

No… I cannot condone it - don’t do it!

Side note: Jared also has an excellent podcast called Dead Code, with loads of episodes with fantastic guests. I went on to speak about concurrency - give it a listen!

This is the `END`

Earlier, I said:

What is once? It can’t exist almost solely for the purposes of this extremely strange regex modifier, can it? Surely this modifier is not built into the very bones of Ruby?

And the truth is, /o is not the only reason the once instruction exists. It took a little digging, but I found another piece of Ruby code that uses once - the END language sytax. I was surprised to find that END existed! I had never heard of it or used it before.

Not to be confused with the end keyword that closes out blocks, END defines a block that is run at the end of the Ruby program. For instance:

END { puts "do this last please" }
puts "1"
puts "2"
	
# => 1
# => 2
# => do this last please

Note that if you run this code in irb, the END block will not run until you exit. irb is just a Ruby program, and it doesn’t END until irb is exited.

If you disassemble this code, we’ll see a familiar instruction:

puts RubyVM::InstructionSequence.compile(DATA.read).disassemble
	
__END__
END { puts "do this last please" }
puts "1"
puts "2"
	
# == disasm: #<ISeq:<compiled>@<compiled>:1 (1,0)-(4,8)>
# 0000 once          block in <compiled>, <is:0>(   2)

So the END block, but not the end of a block, uses the once instruction, an instruction that is used twice in Ruby. An appropriately confusing close to an article about a fascinating feature I think you should never use - the /o modifier.

A silly optimization: adding opt_respond_to to the Ruby VM, part 6

Mon, 06 Jan 2025 23:34:49 -0500

In part 5, we finally got our new instruction defined and outputting as part of our bytecode. if you didn’t run it yourself, you just had to trust me that it really did run.

But, I just dropped most of the implementation code in without explaining it. Let’s start off by walking through the basic version, then start planning for the true optimization.

The progress so far

Here’s our sample Ruby program:

puts "Did you know you can write to $stdout?" if $stdout.respond_to?(:write)

First, we’ll disassemble the code using make run, and run it using our C changes (you can pull the work in progress here):

RUNOPT0=--dump=insns make run

This gives us a new set of instructions. Most of it is the same as Ruby master, but opt_send_without_block is changed to opt_respond_to. The calldata containing respond_to? is still there, and I think it’ll stay even once we finish the whole implementation:

# == disasm: #<ISeq:<main>./test.rb:1 (1,0)-(1,76)>
0000 getglobal                :$stdout                  (   1)[Li]
0002 putobject                :write
# our new instruction!
0004 opt_respond_to           <calldata!mid:respond_to?, argc:1, ARGS_SIMPLE>
0006 branchunless             14
0008 putself
0009 putchilledstring         "Did you know you can write to $stdout?"
0011 opt_send_without_block   <calldata!mid:puts, argc:1, FCALL|ARGS_SIMPLE>
0013 leave
0014 putnil
0015 leave

Our current implementation is mostly just a pass through to the normal respond_to? method, with some debug information printed. Running it without the dump=insns option, this is the output we get:

> make run

symbol:File
Did you know you can write to $stdout?

File is the type of the receiver, $stdout, and symbol is the type of the method argument, :write.

📝 In previous posts, we used make runruby and make lldb-ruby/make gdb-ruby. Based on feedback from Ruby maintainers in the know (like byroot), it seems like make run and make lldb/make gdb are the better options in 99% of cases. These commands use “miniruby”, which is all the Ruby syntax without loading stdlib and gems, so it should run faster. If you do need the stdlib and standard gems, you’ll want to continue using make runruby and friends

Breaking down the changes

The last post was running pretty long, so I dumped all the code at the end without explanation. Let’s break each section down, starting with our insns.def change to the virtual machine DSL:

//insns.def
DEFINE_INSN
opt_respond_to
(CALL_DATA cd)
(VALUE recv, VALUE mid)
(VALUE val)
{
    val = vm_opt_respond_to(recv, mid);
    CALL_SIMPLE_METHOD();
}

We have some context for how a virtual machine instruction is defined from the previous post, so let’s break this down:

opt_respond_to is the name of the instruction
(CALL_DATA cd) is the one “operand”, the call data of the method. I don’t think we’ll need this for our optimized version, but I think if we use a fallback it would still be required
(VALUE recv, VALUE mid) are the values this instruction is expecting to be popped off the stack so they can be used in the call. In our sample program instructions this should correspond to getglobal :$stdout and putobject :write. $stdout is recv, or the “receiver”. :write is mid, or the “method id”
(VALUE val) is the return value. Whatever gets set to val gets pushed onto the stack at the end of the instruction. The next instruction in our example is branchunless, which pops our val off the stack and tests it
Next is the body of the instruction:
- val = vm_opt_respond_to(recv, mid); here I followed the convention of other instructions which need some custom logic - they put their code inside of a vm_ prefixed function named after their instruction, and define it in vm_insnhelper.c. My function takes the receiver and the method id, and we’ll dive into that in a bit
- I think CALL_SIMPLE_METHOD(); will use the calldata to call the original method. Normally you would check the return value of the vm_ function to determine whether you want to pass through to the original implementation. In my case, my function is just printing some debug information so I let it always call the original

We’ve dug into most of the pattern matching logic in compile.c in previous posts, so I’ll skip that part and focus on the instruction override:

// compile.c
const struct rb_callinfo *ci = (struct rb_callinfo *)OPERAND_AT(iobj, 0);
//...
iobj->insn_id = BIN(opt_respond_to);
iobj->operand_size = 1;
iobj->operands = compile_data_calloc2(
  iseq, 
  iobj->operand_size, 
  sizeof(VALUE)
);
iobj->operands[0] = (VALUE)ci;

Once it’s found an instruction that matches a send to respond_to?, we override the current information. First we set insn_id to BIN(opt_respond_to), which we know expands to the enum value YARVINSN_opt_respond_to.

The rest seems… redundant? It already had ci at the first operand position, it was already an operand_size of 1. It’s possible I don’t need to recompile this, but I’ll need some guidance around that. It’s probably not harmful, but possibly unnecessary.

Last we’ve got our vm_opt_respond_to function:

// vm_insnhelper.c
static VALUE
vm_opt_respond_to(VALUE recv, VALUE mid)
{
  if (SYMBOL_P(mid)) {
    printf("symbol:");
  } else if (STRING_P(mid)) {
    printf("string:");
  }
  printf("%s\n", rb_builtin_type_name(TYPE(recv)));
  return Qundef;
}

It’s purely a debug function right now. It prints “symbol:” if mid is a symbol (SYMBOL_P and STRING_P are each “predicate” functions, hence the _P), “string:” if we have a string. Then it prints the type of the receiver and a new line. This is how we end up with symbol:File when we run our program:

puts "Did you know you can write to $stdout?" if $stdout.respond_to?(:write)
# symbol:File
# Did you know you can write to $stdout?

What’s next?

I’m missing some things at the moment:

Tests
Logic for handling the private/protected param
Actual optimization code 😅

1. Tests

There should already be tests for respond_to?, so I’ll start running those and rely on them for the moment.

As might be expected for an entire language, there are tons of tests. There is also RubySpec, which is the standard spec suite for every Ruby language implementation. It’s automatically included in the repository as well.

I’ll rely on those specs for now:

> make test-spec SPECOPTS="../spec/ruby/core/kernel/respond_to_spec.rb"

ruby 3.5.0dev (2025-01-04T14:32:13Z opt-respond-to 5688434f63) +PRISM [arm64-darwin24]
[\ | ==================100%================== | 00:00:00]      0F      0E 

Finished in 0.007758 seconds

1 file, 13 examples, 24 expectations, 0 failures, 0 errors, 0 tagged

As expected, it still works so far since my version is basically a pass-through. We’ll see if we need more specs later on or if the base set is enough.

2. Logic for handling the private/protected param

respond_to? takes a second parameter - include_all - which determines whether to include private and protected methods.

I’ve never seen someone use this second parameter, but I’m sure it’s out there somewhere 🤷‍♂️. Piotr Szotkowski recently told me he’s a fan of the flip-flop operator - so the world is full of surprises 😉! Part of me wants to ignore it for optimizing and just pass through in that case, but that’s a total cop out.

I think there is some VM magic I need to utilize to handle an optional argument, applying special attributes for dynamic stack pointer adjustment. For instance, opt_send_without_block is defined like this:

DEFINE_INSN
opt_send_without_block
(CALL_DATA cd)
(...)
(VALUE val)
// attr bool handles_sp = true;
// attr rb_snum_t sp_inc = sp_inc_of_sendish(cd->ci);
// attr rb_snum_t comptime_sp_inc = sp_inc_of_sendish(ci);
{
  //...
}

It doesn’t specify the pop values, but instead uses the syntax (...) similar to argument forwarding in Ruby. It then specifies some stack pointer (“sp”) counts (those comments are actual code!), which I think allows it to handle a dynamic number of values to pop off the stack.

This seems complex for my case, where I have one required and one optional argument. I’ll defer this one for the moment.

3. Actual optimization code

I actually don’t know if this is optimizable in a meaningful way. I’d be lying if I said I didn’t care if there’s an optimization win here - that’s the most satisfying/impactful outcome.

This entire series is inspired by Optimizing Ruby’s JSON, Part 2, and one of the goals of that work was to reduce setup costs. Here’s some of the JSON.dump method in its original form:

def dump(obj, anIO = nil, limit = nil, kwargs = nil)
  #...
  if anIO.respond_to?(:to_io)
    anIO = anIO.to_io
  elsif limit.nil? && !anIO.respond_to?(:write)
    anIO, limit = nil, anIO
  end
  #...
end

The majority of the time, anIO is nil, so it won’t have a to_io or write method. That means in a micro-benchmark running millions of times the call to respond_to? is pure overhead. The solution in the post was to avoid the call when nil, but how fast can we make it if we did a silly, nil-specific optimization?

Setting up a performance baseline

Let’s setup a benchmark to see what our current performance is, as a baseline. In CRuby there are built-in benchmarking scripts we can use. We’ll define a new benchmark for respond_to?:

# benchmark/object_respond_to.yml
prelude: |
  class Base; def foo; end end
  class OneTwentyEight < Base
    128.times { include(Module.new) }
  end
  obj = OneTwentyEight.new  
benchmark:
  respond_to_false: obj.respond_to?(:bar)
  respond_to_true: obj.respond_to?(:foo)
  respond_to_nil_false: nil.respond_to?(:bar)
loop_count: 1_000_000

This YAML first sets up a prelude, which is Ruby code to setup our benchmark:

It defines a Base class with a foo method
Creates a child class called OneTwentyEight, which extends the Base class
Includes Module.new 128 times, to create alot of ancestors to search for methods
Instantiates OneTwentyEight to call from the benchmark

The benchmark keys specify what operations to run. respond_to_false checks respond_to? for a method that doesn’t exist, and respond_to_true checks for a method that does exist. respond_to_nil_false is unrelated to the prelude, but let’s me test how fast looking for a method on nil is.

The loop_count is how many iterations the code will run. I believe it runs several times, and then calculates how many times per second it should be able to run. Aaron Patterson created this benchmark in a PR that never merged, so thanks to him for that!

We can run the benchmark using make benchmark ITEM='respond_to'. I get the following output on a clean master branch:

# Iteration per second (i/s)
|                      |compare-ruby|built-ruby|
|:---------------------|-----------:|---------:|
|respond_to_nil_false  |     29.029M|   28.259M|
|                      |       1.03x|         -|
|respond_to_false      |     29.177M|   29.121M|
|                      |       1.00x|         -|
|respond_to_true       |     33.503M|   32.481M|
|                      |       1.03x|         -|

compare-ruby is the version of Ruby the project was built with (yes, building Ruby requires Ruby 🫨). For me, that’s Ruby 3.4. built-ruby is my local, built version. The differences in performance are pretty negligable - probably differences in compile flags used to build Rubies. The performance of each stays pretty close, and can flip-flip a bit between iterations.

You can run alot of respond_to?s in a second! The found method cases are the fastest, and the miss cases are consistently slower.

A first silly optimization

Now that we have a baseline, let’s try two optimizations to see what our upper-limit might be:

A nil specific check that always returns false
A nil specific check that has a hard-coded set of possible methods

First, we’ll change opt_respond_to into a common pattern. Many instructions will call a method, and if the method returns Qundef, they’ll revert to a base-case path. In our case right now, that’s CALL_SIMPLE_METHOD(). I assume Qundef exists to specify “undefined” behavior, to differentiate from Qnil which could be a valid return value:

// insns.def
DEFINE_INSN
opt_respond_to
(CALL_DATA cd)
(VALUE recv, VALUE mid)
(VALUE val)
{
  val = vm_opt_respond_to(recv, mid);
  if (UNDEF_P(val)) {
    CALL_SIMPLE_METHOD();
  }
}

And here is our silliest optimization. If recv is nil, always return false. Otherwise, return Qundef:

// vm_insnhelper.c
static VALUE
vm_opt_respond_to(VALUE recv, VALUE mid)
{
  if (NIL_P(recv)) {
    return Qfalse;
  }

  return Qundef;
}

Let’s rerun our benchmark, and see what we get:

> make benchmark ITEM='respond_to'

# Iteration per second (i/s)
|                      |compare-ruby|built-ruby|
|:---------------------|-----------:|---------:|
|respond_to_false      |     29.121M|   27.795M|
|                      |       1.05x|         -|
|respond_to_true       |     32.241M|   31.544M|
|                      |       1.02x|         -|
|respond_to_nil_false  |     26.872M|   57.894M|
|                      |           -|     2.15x|

Oh, not bad! Around a 2x improvement. But ya know, it’s totally incorrect. We can add a spec to the respond_to_spec to check. It fails, as expected:

it "returns true for checking for `==` on nil" do
  nil.respond_to?(:==).should == true
end

# make test-spec SPECOPTS="../spec/ruby/core/kernel/respond_to_spec.rb"
# 1)
# Kernel#respond_to? returns true for checking for `==` on nil FAILED
# Expected false == true
# to be truthy but was false
# [/ | ==================100%================== | 00:00:00]      1F      0E 
# Finished in 0.017146 seconds
# 1 file, 14 examples, 25 expectations, 1 failure, 0 errors, 0 tagged

A second, slightly less silly optimization

What if I added some overhead, but not a ton of overhead. First, I got every method available to me from an irb session:

nil.methods
# "rationalize", "&", "===", "inspect", "=~", "to_a",...

Then I took that and put it into an array of chars in C. The first time we call our vm_opt_respond_to function, it populates a rb_id_table with each of the available method names using rb_id_table_insert. rb_id_table is an internal CRuby hashtable structure which revolves around IDs, which I believe typically correspond to method names.

If the recv is nil, we use method_id_table to check if one of our hard-coded method names is being checked by respond_to?, using rb_id_table_lookup. If it returns true, we return Qtrue, otherwise Qfalse.

static struct rb_id_table *method_id_table = NULL;

static VALUE
vm_opt_respond_to(VALUE recv, VALUE mid)
{
  if (method_id_table == NULL) {
    const char *method_names[] = {
      "rationalize", "&", "===", "inspect", "=~", "to_a", "to_s", "to_i", "to_f", "to_r",
      "to_c", "nil?", "pretty_print_cycle", "|", "to_h", "^", "to_json", "to_yaml",
      "pretty_print", "pretty_print_instance_variables", "pretty_print_inspect", "singleton_class",
      "dup", "itself", "methods", "singleton_methods", "protected_methods", "private_methods",
      "public_methods", "instance_variables", "instance_variable_get", "instance_variable_set",
      "instance_variable_defined?", "remove_instance_variable", "instance_of?", "kind_of?",
      "is_a?", "display", "frozen?", "class", "then", "yield_self", "tap", "TypeName",
      "public_send", "extend", "clone", "<=>", "pretty_inspect", "!~", "method", "eql?",
      "respond_to?", "public_method", "singleton_method", "define_singleton_method", "hash",
      "freeze", "object_id", "Namespace", "send", "to_enum", "enum_for", "equal?", "!",
      "__send__", "==", "!=", "__id__", "instance_eval", "instance_exec"
    };

    size_t method_names_size = sizeof(method_names) / sizeof(method_names[0]);
    method_id_table = rb_id_table_create(method_names_size);

    for (size_t i = 0; i < method_names_size; i++) {
      ID id = rb_intern(method_names[i]);
      rb_id_table_insert(method_id_table, id, Qtrue);
    }
  }
  if (NIL_P(recv)) {
    ID id = rb_check_id(&mid);
    if (!id) return Qfalse;

    VALUE val;
    if (rb_id_table_lookup(method_id_table, id, &val)) {
      return Qtrue;
    } else {
      return Qfalse;
    }
  }

  return Qundef;
}

How fast is this version, now that we’re doing some actual work?

# Iteration per second (i/s)
|                      |compare-ruby|built-ruby|
|:---------------------|-----------:|---------:|
|respond_to_false      |     29.668M|   28.738M|
|                      |       1.03x|         -|
|respond_to_true       |     33.320M|   29.829M|
|                      |       1.12x|         -|
|respond_to_nil_false  |     28.610M|   53.084M|
|                      |           -|     1.86x|

Still pretty fast! It even passes our spec now:

[| | ==================100%================== | 00:00:00]      0F      0E 
Finished in 0.008847 seconds
1 file, 14 examples, 25 expectations, 0 failures, 0 errors, 0 tagged

Back to reality

Ok - we described our base code. We walked through next steps. We ran some specs and got a feel for some benchmarks. It seems like our upper limit on performance may be about 2x how fast it currently runs - and it’s probably unattainable. But it’s nice to know the potential ceiling on performance from where things currently are.

Next time we’ll dig into some previous optimization improvements to respond_to? in older PRs, how respond_to? works currently, and hopefully make our first real optimization improvement. See you next time!

PS - You can find the code changes made in the branch here.

Defining an instruction: adding opt_respond_to to the Ruby VM, part 5

Tue, 31 Dec 2024 22:43:30 -0500

In Peephole optimizations: adding opt_respond_to to the Ruby VM, part 4, we dug deep. We found the connection between prism compilation and the specialization we need for our new bytecode, called “peephole optimization”. We learned how to debug and step through C code in the Ruby runtime, and we added some logic for matching the “pattern” of the instruction we want to change.

Now that we know where the specialization needs to go and how to match what needs to be specialized - what do we actually replace it with? How do we get the virtual machine to recognize opt_respond_to?

Pattern matching bytecode instructions

opt_ary_freeze has been a great learning tool - let’s see what it teaches us about adding a new instruction name.

Here’s a refresher on how iseq_peephole_optimize matches on newarray, and then replaces it with opt_ary_freeze:

if (
  IS_INSN_ID(iobj, newarray)
) {
  LINK_ELEMENT *next = iobj->link.next;
  if (
    IS_INSN(next) &&
    IS_INSN_ID(next, send)
  ) {
    //... more if statements
    iobj->insn_id = BIN(opt_ary_freeze);

it first checks if the instruction id of iobj is newarray
it grabs the next element and checks if its an instruction, then checks if the instruction is send (it also checks if the method id is “freeze”, not shown above)
if those checks match, it replaces the instruction id with opt_ary_freeze

That’s pretty reasonable to follow. But how do IS_INSN and IS_INSN_ID work? What is BIN? What type is opt_ary_freeze - where is it defined? How do we add new instructions ourselves?

Macros and enums

BIN, IS_INSN and IS_INSN_ID are all C macros that revolve around interacting with virtual machine instructions.

Macros in C get embedded directly into your code in a preprocessing step before being compiled, so you can write things that look pretty odd compared to a typical C-like syntax. Here’s the definition for BIN:

#define BIN(n) YARVINSN_##n

📝 BIN probably stands for “Binary INstruction”

That ## is kind of like string interpolation, but the result is a static part of your actual code. This means that anywhere BIN is called, it’s kind of like saying YARVINSN_#{n} in Ruby. So this code:

iobj->insn_id = BIN(opt_ary_freeze);

Gets transformed into this, right before the program is compiled:

iobj->insn_id = YARVINSN_opt_ary_freeze;

Here’s the definition for IS_INSN_ID:

#define IS_INSN_ID(iobj, insn) (INSN_OF(iobj) == BIN(insn))

Based on our understanding of macros and BIN, so far, it gets transformed into:

#define IS_INSN_ID(iobj, insn) (INSN_OF(iobj) == YARVINSN_##insn)

Here’s the definition for INSN_OF, it just casts insn to an INSN type, and accesses its instruction id:

#define INSN_OF(insn) \
  (((INSN*)(insn))->insn_id)

That means the expanded version of IS_INSN_ID is:

#define IS_INSN_ID(iobj, insn) \
  (((INSN*)(insn))->insn_id == YARVINSN_##insn)

Here’s the definition for IS_INSN:

#define IS_INSN(link) ((link)->type == ISEQ_ELEMENT_INSN)

If we combined all of it together, and manually inline it, here’s what our original pattern matching code looks like:

if (
  (((INSN*)(iobj))->insn_id == YARVINSN_newarray)
) {
  LINK_ELEMENT *next = iobj->link.next;
  if (
    (next)->type == ISEQ_ELEMENT_INSN &&
    (((INSN*)(next))->insn_id == YARVINSN_send)
  ) {
    //... more if statements
    iobj->insn_id = YARVINSN_opt_ary_freeze;

I’m glad CRuby adds those macros… this expanded code is a lot less readable.

Why did I expand it and make that original code less clear? I wanted to think through what the instructions really look like at runtime, and why. The reason I can infer, is that all of these macros let you focus on a syntax that looks just like our VM instructions, while making sure there are no name collisions behind the scenes.

Ok, I still haven’t shown where the instruction comes from. Here’s the file you can actually find an enum containing the entire list of vm instructions:

// insns.inc
enum ruby_vminsn_type {
  BIN(nop),
  BIN(getlocal),
  //...
  BIN(opt_ary_freeze),
  //...
}

// or, expanded by the preprocessor!
enum ruby_vminsn_type {
  YARVINSN_nop,
  YARVINSN_getlocal,
  //...
  YARVINSN_opt_ary_freeze,
  //...
}

insns.inc gets included anywhere we need instruction checks, like in compile.c. These enum values are globally available anywhere this file is included. Thanks to BIN prepending all of their names with YARVINSN_, we can use them in a convenient syntax without having any collisions.

So if I search the CRuby repo for insns.inc, where can I find it? Hmmm, I can’t 🤔. insns.inc is a generated file! I can only see it locally, after compiling the entire project. Where does that file get generated from?

A virtual machine DSL

While insns.inc tells us the name of each instruction available, the file it is generated from defines every instruction available in the Ruby virtual machine, and how it should respond to that instruction. It’s called insns.def. The file looks a lot like C, but it’s actually a kind of DSL.

It lets you define a simplified set of information for the instruction. That simplified format is then compiled into a more comprehensive, C compatible version.

It’s compilers all the way down… 😵‍💫

The top of the file defines the format. I don’t fully understand it, but let’s walk through it:

/*
DEFINE_INSN
instruction_name
(type operand, type operand, ..)
(pop_values, ..)
(return values ..)
// attr type name contents..
{
  .. // insn body
}
*/

An instructions consists of:

A name ✅
Operands - like our “call data”. I’m not sure what other types of operands are typically used for, but I know opt_ary_freezs puts the frozen array here as well
Values to pop off the virtual machine stack so we can operate on them. This should be values that we’ve seen pushed onto the stack in previous instructions
A return value
(I don’t fully understand the value of attr type but it seems to influence what code gets generated by the instruction definition)
A C-compatible body

That’s a lot, baked into a small interface. Let’s look at a very simple example. Here’s one of the simplest instructions available, putnil:

DEFINE_INSN
putnil
()
()
(VALUE val)
{
    val = Qnil;
}

Looks… pointless? Theputnil instruction takes no arguments, and has a return value of val. The only thing the code block does is set val equal to Qnil, which is a special value in CRuby representing Ruby’s nil. What does that accomplish?

📝 VALUE is a special type in CRuby that points at a Ruby object, usually located on the heap. When you see VALUE, this often means we’re looking at a value you’d use in a Ruby program.

This file is compiled into regular C code, and the context of this simple instruction becomes clearer:

// vm.inc
/* insn putnil()()(val) */
INSN_ENTRY(putnil)
{
  //...
  VALUE val;
  //...
#   define NAME_OF_CURRENT_INSN putnil
#   line 331 "../insns.def"
{
  val = Qnil;
}
  //...
  INC_SP(INSN_ATTR(sp_inc));
  TOPN(0) = val;
  //...
}

The return value VALUE val is declared
It’s set to val = Qnil, the instruction we saw in insns.def
INC_SP is called, which I believe “increments” the “stack pointer”, giving us extra space on the stack to push onto?
TOPN(0) = val sets val to the top of the stack

I think I’ll dig more into that next time. But let’s get back to the task at hand - it’s time to try and get our respond_to? call replaced with opt_respond_to!

Adding to the DSL

It took me a bit of banging my head against a wall, but here is the working instruction and specialization, in a basic form:

// insns.def
DEFINE_INSN
opt_respond_to
(CALL_DATA cd)
(VALUE recv, VALUE mid)
(VALUE val)
{
    val = vm_opt_respond_to(recv, mid);
    CALL_SIMPLE_METHOD();
}

// compile.c
if (IS_INSN_ID(iobj, send)) {
  const struct rb_callinfo *ci = (struct rb_callinfo *)OPERAND_AT(iobj, 0);
  const rb_iseq_t *blockiseq = (rb_iseq_t *)OPERAND_AT(iobj, 1);

  if (vm_ci_simple(ci) && vm_ci_argc(ci) == 1 && blockiseq == NULL && vm_ci_mid(ci) == idRespond_to) {
      iobj->insn_id = BIN(opt_respond_to);
      iobj->operand_size = 1;
      iobj->operands = compile_data_calloc2(iseq, iobj->operand_size, sizeof(VALUE));
      iobj->operands[0] = (VALUE)ci;
  }
}

If we dump the instructions, we finally see our new instruction opt_respond_to. It’s not really doing anything yet, but it’s there!

puts "Did you know you can write to $stdout?" if $stdout.respond_to?(:write)

# > RUNOPT0=--dump=insns make run
# == disasm: #<ISeq:<main>./test.rb:1 (1,0)-(1,76)>
# 0000 getglobal                :$stdout                  (   1)[Li]
# 0002 putobject                :write
# 0004 opt_respond_to           <calldata!mid:respond_to?, argc:1, ARGS_SIMPLE>
# 0006 branchunless             14
# 0008 putself
# 0009 putchilledstring         "Did you know you can write to $stdout?"
# 0011 opt_send_without_block   <calldata!mid:puts, argc:1, FCALL|ARGS_SIMPLE>
# 0013 leave
# 0014 putnil
# 0015 leave

Sorry to just dump the code here, and split. We’ll dig into it more, and expand on it next time! There is lots more to do, and to explain, but i’m excited about this milestone! See you next time! 👋🏼

PS - since something is working now, i’ve pushed up my basic code so far, here.

Peephole optimizations: adding `opt_respond_to` to the Ruby VM, part 4

Fri, 27 Dec 2024 23:36:19 -0500

In The Ruby Syntax Holy Grail: adding opt_respond_to to the Ruby VM, part 3, I found what I referred to as the “Holy Grail” of Ruby syntax. I’m way overstating it, but it’s a readable, sequential way of viewing how a large portion of the Ruby syntax is compiled. Here’s a snippet of it as a reminder:

// prism_compile.c
static void
pm_compile_node(rb_iseq_t *iseq, const pm_node_t *node, LINK_ANCHOR *const ret, bool popped, pm_scope_node_t *scope_node)
{
    const pm_parser_t *parser = scope_node->parser;
    //...
    switch (PM_NODE_TYPE(node)) {
      //...
      case PM_ARRAY_NODE: {
        // [foo, bar, baz]
        // ^^^^^^^^^^^^^^^
        const pm_array_node_t *cast = (const pm_array_node_t *) node;
        pm_compile_array_node(iseq, (const pm_node_t *) cast, &cast->elements, &location, ret, popped, scope_node);
        return;
      }
      //...
      case PM_MODULE_NODE: {
        // module Foo; end
        //...
      }
      //...
}

The file that code lives in, prism_compile.c, is enormous. pm_compile_node itself is 1800+ lines, and the overall file is 11 thousand lines. It’s daunting to say the least, but there are some obvious directions I can ignore - i’m trying to optimize a method call to respond_to?, so I can sidestep a majority of the Ruby syntax.

Still, where I do go, specifically?

Sage wisdom

Helpfully, I got two identical sets of direction based on part 3. One from Kevin Newton, creator of Prism:

https://x.com/kddnewton/status/1872280281409105925?s=46

And one from byroot, who inspired this whole series:

https://bsky.app/profile/byroot.bsky.social/post/3le6xypzykc2x

I don’t want to jump to conclusions, but I think I need to look at the peephole optimizer 😆.

And exactly what is a “peephole optimizer”? Kevin described the process as “specialization comes after compilation”. From Wikipedia:

Peephole optimization is an optimization technique performed on a small set of compiler-generated instructions, known as a peephole or window, that involves replacing the instructions with a logically equivalent set that has better performance. https://en.wikipedia.org/wiki/Peephole_optimization

This seems to fit my goal pretty well. I want to replace the current opt_send_without_block instruction with a specialized opt_respond_to instruction, optimized for respond_to? method calls.

Finding the optimizer

So where are peephole optimizations happening in CRuby today? In Étienne’s PR, he added optimization code to a function called… iseq_peephole_optimize. A little on the nose, don’t you think? Kevin’s comment also mentioned iseq_peephole_optimize - seems like the winner.

I want to make the link between iseq_peephole_optimize and where we left off at pm_compile_node. Let’s dig into some code!

Disassembling an existing optimization

I’m going to use Étienne’s frozen array optimization to get to the optimizer and see how it relates. If you want to follow along, start with the setup instructions from part 3.

His optimization only applies to array and hash literals being frozen. So we’ll write a teensy Ruby program to demonstrate, and put it in test.rb at the root of our CRuby project:

# test.rb
pp [].freeze

The best way to run test.rb here is to use make. It will not only run the file, but also make sure things like C files get recompiled as necessary when you make changes. Let’s run our file, but dump the instructions it would generate for the Ruby VM:

RUNOPT0=--dump=insns make runruby

RUNOPT0 lets us add an option to the ruby call, so it’s effectively ruby --dump=insns test.rb. Here’s the instructions we see - we can confirm that we are getting the optimized opt_ary_freeze instruction from Étienne PR:

== disasm: #<ISeq:<main>./test.rb:3 (3,0)-(3,12)>
0000 putself                      (   3)[Li]
0001 opt_ary_freeze               [], <calldata!mid:freeze, argc:0, ARGS_SIMPLE>
0004 opt_send_without_block       <calldata!mid:pp, argc:1, FCALL|ARGS_SIMPLE>
0006 leave

You never know what code is truly doing until you run it. So far, I’ve just been reading and navigating the CRuby source. iseq_peephole_optimize lives in compile.c - let’s set a breakpoint and take a look 🕵🏼‍♂️.

Using the debugger

We can debug C code in CRuby almost as easily as we can use a debugger/binding.pry.

For MacOS, you can use lldb, and for Docker/Linux, you can use gdb. I’m going to do everything in lldb to start, but I’ll show some equivalent commands for gdb after.

Let’s start by looking at the peephole optimization code for [].freeze, inside of iseq_peephole_optimize. I’ll add comments above each line to explain what I think it’s doing:

// compile.c
static int
iseq_peephole_optimize(rb_iseq_t *iseq, LINK_ELEMENT *list, const int do_tailcallopt)
{
         // ...
         // if the instruction is a `newarray` of zero length
3469:    if (IS_INSN_ID(iobj, newarray) && iobj->operands[0] == INT2FIX(0)) {
             // grab the next element after the current instruction
3470:        LINK_ELEMENT *next = iobj->link.next;
             // if `next` is an instruction, and the instruction is `send`
3471:        if (IS_INSN(next) && (IS_INSN_ID(next, send))) {
3472:            const struct rb_callinfo *ci = (struct rb_callinfo *)OPERAND_AT(next, 0);
3473:            const rb_iseq_t *blockiseq = (rb_iseq_t *)OPERAND_AT(next, 1);
3474:
                 // if the callinfo is "simple", with zero arguments,
                 // and there isn't a block provided(?), and the method id (mid) is `freeze`
                 // which is represented by `idFreeze`
3475:            if (vm_ci_simple(ci) && vm_ci_argc(ci) == 0 && blockiseq == NULL && vm_ci_mid(ci) == idFreeze) {
                     // change the instruction to `opt_ary_freeze`
3476:                iobj->insn_id = BIN(opt_ary_freeze);
                     // remove the `send` instruction, we don't need it anymore
3481:                ELEM_REMOVE(next);

Now i’ll use lldb to see where this code runs in relation to our prism compilation. In CRuby, to debug you run make lldb-ruby instead of make runruby. You’ll see some setup code run, and then you’ll be left at a prompt, prefixed by (lldb):

> make lldb-ruby
lldb  -o 'command script import -r ../misc/lldb_cruby.py' ruby --  ../test.rb
(lldb) target create "ruby"
Current executable set to '/Users/johncamara/Projects/ruby/build/ruby' (arm64).
(lldb) settings set -- target.run-args  "../test.rb"
(lldb) command script import -r ../misc/lldb_cruby.py
lldb scripts for ruby has been installed.
(lldb)

At this point, we haven’t actually run anything. We can now set our breakpoint, then run the program. I’ll add a breakpoint right after all if statements have succeeded:

(lldb) break set --file compile.c --line 3476
Breakpoint 1: where = ruby`iseq_peephole_optimize + 2276 at compile.c:3476:17

With our breakpoint set, we call run to run the program:

(lldb) run

You’ll see something like the following. It ran the program until it hit our breakpoint, right after identifying a frozen array literal:

(lldb) run
Process 50923 launched: '/ruby/build/ruby' (arm64)
Process 50923 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
    frame #0: ruby`iseq_peephole_optimize(...) at compile.c:3476:17
   3473             const rb_iseq_t *blockiseq = (rb_iseq_t *)OPERAND_AT(next, 1);
   3474
   3475             if (vm_ci_simple(ci) && vm_ci_argc(ci) == 0 && blockiseq == NULL && vm_ci_mid(ci) == idFreeze) {
-> 3476                 iobj->insn_id = BIN(opt_ary_freeze);
   3477                 iobj->operand_size = 2;
   3478                 iobj->operands = compile_data_calloc2(iseq, iobj->operand_size, sizeof(VALUE));
   3479                 iobj->operands[0] = rb_cArray_empty_frozen;

I want to see where we are in relation to all our prism compilation code. We can use bt to get the backtrace:

(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
  * frame #0: ruby`iseq_peephole_optimize(...) at compile.c:3476:29
    frame #1: ruby`iseq_optimize(...) at compile.c:4352:17
    frame #2: ruby`iseq_setup_insn(...) at compile.c:1619:5
    frame #3: ruby`pm_iseq_compile_node(...) at prism_compile.c:10139:5
    frame #4: ruby`pm_iseq_new_with_opt_try(...) at iseq.c:1029:5
    frame #5: ruby`rb_protect(...) at eval.c:1033:18
    frame #6: ruby`pm_iseq_new_with_opt(...) at iseq.c:1082:5
    frame #7: ruby`pm_new_child_iseq(...) at prism_compile.c:1271:27
    frame #8: ruby`pm_compile_node(...) at prism_compile.c:9458:40
    frame #9: ruby`pm_compile_node(...) at prism_compile.c:9911:17
    frame #10: ruby`pm_compile_scope_node(...) at prism_compile.c:6598:13
    frame #11: ruby`pm_compile_node(...) at prism_compile.c:9784:9
    frame #12: ruby`pm_iseq_compile_node(...) at prism_compile.c:10122:9
    frame #13: ruby`pm_iseq_new_with_opt_try(...) at iseq.c:1029:5
    frame #14: ruby`rb_protect(...) at eval.c:1033:18
    frame #15: ruby`pm_iseq_new_with_opt(...) at iseq.c:1082:5
    frame #16: ruby`pm_iseq_new_top(...) at iseq.c:906:12
    frame #17: ruby`load_iseq_eval(...) at load.c:756:24
    frame #18: ruby`require_internal(...) at load.c:1296:21
    frame #19: ruby`rb_require_string_internal(...) at load.c:1402:22
    frame #20: ruby`rb_require_string(...) at load.c:1388:12
    frame #21: ruby`rb_f_require(...) at load.c:1029:12
    frame #22: ruby`ractor_safe_call_cfunc_1(...) at vm_insnhelper.c:3624:12
    frame #23: ruby`vm_call_cfunc_with_frame_(...) at vm_insnhelper.c:3801:11
    frame #24: ruby`vm_call_cfunc_with_frame(...) at vm_insnhelper.c:3847:12
    frame #25: ruby`vm_call_cfunc_other(...) at vm_insnhelper.c:3873:16
    frame #26: ruby`vm_call_cfunc(...) at vm_insnhelper.c:3955:12
    frame #27: ruby`vm_call_method_each_type(...) at vm_insnhelper.c:4779:16
    frame #28: ruby`vm_call_method(...) at vm_insnhelper.c:4916:20
    frame #29: ruby`vm_call_general(...) at vm_insnhelper.c:4949:12
    frame #30: ruby`vm_sendish(...) at vm_insnhelper.c:5968:15
    frame #31: ruby`vm_exec_core(...) at insns.def:898:11
    frame #32: ruby`rb_vm_exec(...) at vm.c:2595:22
    frame #33: ruby`rb_iseq_eval(...) at vm.c:2850:11
    frame #34: ruby`rb_load_with_builtin_functions(...) at builtin.c:54:5
    frame #35: ruby`Init_builtin_features at builtin.c:74:5
    frame #36: ruby`ruby_init_prelude at ruby.c:1750:5
    frame #37: ruby`ruby_opt_init(...) at ruby.c:1811:5
    frame #38: ruby`prism_script(...) at ruby.c:2215:13
    frame #39: ruby`process_options(...) at ruby.c:2538:9
    frame #40: ruby`ruby_process_options(...) at ruby.c:3169:12
    frame #41: ruby`ruby_options(...) at eval.c:117:16
    frame #42: ruby`rb_main(...) at main.c:43:26
    frame #43: ruby`main(...) at main.c:68:12

Whoa. That thing is huge! This is not the backtrace I was expecting! Seems like I missed a codepath in my earlier explorations. I got it right, up until prism_script:

main
which calls rb_main
which calls ruby_options, then ruby_process_options, then process_options
which calls prism_script
The next instruction I expected was pm_iseq_new_main, but instead we head into ruby_opt_init
which calls Init_builtin_features

This path seems to go through some gem preloading logic, which is why we see the rb_require calls:

void
Init_builtin_features(void)
{
    rb_load_with_builtin_functions("gem_prelude", NULL);
}

By default CRuby loads gem_prelude, which lives in ruby/gem_prelude.rb. Here’s that file, shortened for brevity:

require 'rubygems'
require 'error_highlight'
require 'did_you_mean'
require 'syntax_suggest/core_ext'

Compiling on-the-fly

There’s something i’ve learned here that seems obvious in hindsight, but I hadn’t considered. Ruby will only compile what is actually loaded, and only at the point it gets loaded. If I never load a particular piece of code, it never gets compiled. Or if I defer loading it until later, it does not get compiled until later.

We can actually demonstrate this by deferring a require:

sleep 10

require "net/http"

If we run this this using make lldb-ruby, we can see the delayed compilation in action:

(lldb) break set --file ruby.c --line 2616
(lldb) run
// hits our prism compile code
(lldb) next
(lldb) break set --file compile.c --line 3476
(lldb) continue
// waits 10 seconds, then compiles the contents of "net/http"

Getting to our test.rb file

I’d rather see just my code in test.rb get compiled, so I’m going to set a breakpoint directly on pm_iseq_new_main, which for me is in ruby.c on line 2616:

(lldb) break set --file ruby.c --line 2616
(lldb) run
Process 32534 launched: '/ruby/build/ruby' (arm64)
Process 32534 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
    frame #0: ruby`process_options(...) at ruby.c:2616:38
   2613         if (!result.ast) {
   2614             pm_parse_result_t *pm = &result.prism;
   2615             int error_state;
-> 2616             iseq = pm_iseq_new_main(&pm->node, opt->script_name, path, parent, optimize, &error_state);
   2617
   2618             pm_parse_result_free(pm);
   2619

Now when we run the backtrace I am seeing what I expected, because we’ve skipped the gem_prelude compilation. This is the exact flow I walked through in part 2:

(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
  * frame #0: ruby`process_options(...) at ruby.c:2616:38
    frame #1: ruby`ruby_process_options(...) at ruby.c:3169:12
    frame #2: ruby`ruby_options(...) at eval.c:117:16
    frame #3: ruby`rb_main(...) at main.c:43:26
    frame #4: ruby`main(...) at main.c:68:12

From here, we can set our iseq_peephole_optimize breakpoint and see only our specific code get compiled. Since we’re already in the running program, we call continue to keep executing:

(lldb) break set --file compile.c --line 3476
Breakpoint 2: where = ruby`iseq_peephole_optimize + 2276 at compile.c:3476:17
(lldb) continue
Process 55336 resuming
Process 55336 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 2.1
    frame #0: ruby`iseq_peephole_optimize() at compile.c:3476:17
   3473             const rb_iseq_t *blockiseq = (rb_iseq_t *)OPERAND_AT(next, 1);
   3474
   3475             if (vm_ci_simple(ci) && vm_ci_argc(ci) == 0 && blockiseq == NULL && vm_ci_mid(ci) == idFreeze) {
-> 3476                 iobj->insn_id = BIN(opt_ary_freeze);
   3477                 iobj->operand_size = 2;
   3478                 iobj->operands = compile_data_calloc2(iseq, iobj->operand_size, sizeof(VALUE));
   3479                 iobj->operands[0] = rb_cArray_empty_frozen;

If we call bt from here to get the backtrace, we finally see the connection between prism_compile.c and compile.c. pm_iseq_compile_node calls iseq_setup_insn, which runs the optimization logic. In the previous post, I saw iseq_setup_insn, but I didn’t know what it meant or what it did. Now we know. This is what Kevin Newton referred to earlier: specialization comes after compilation. Prism compiles the node in the standard way, then the peephole optimization layer - the specialization - is applied after:

(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 2.1
  * frame #0: ruby`iseq_peephole_optimize(...) at compile.c:3476:17
    frame #1: ruby`iseq_optimize(...) at compile.c:4352:17
    frame #2: ruby`iseq_setup_insn(...) at compile.c:1619:5
    frame #3: ruby`pm_iseq_compile_node(...) at prism_compile.c:10139:5
    frame #4: ruby`pm_iseq_new_with_opt_try(...) at iseq.c:1029:5
    frame #5: ruby`rb_protect(...) at eval.c:1033:18
    frame #6: ruby`pm_iseq_new_with_opt(...) at iseq.c:1082:5
    frame #7: ruby`pm_iseq_new_main(...) at iseq.c:930:12
    frame #8: ruby`process_options(...) at ruby.c:2616:20
    frame #9: ruby`ruby_process_options(...) at ruby.c:3169:12
    frame #10: ruby`ruby_options(...) at eval.c:117:16
    frame #11: ruby`rb_main(...) at main.c:43:26
    frame #12: ruby`main(...) at main.c:68:12

From here, we can inspect and see the current instruction using expr:

(lldb) expr *(iobj)
(INSN) $4 = {
  link = {
    type = ISEQ_ELEMENT_INSN
    next = 0x000000011f6568d0
    prev = 0x000000011f656850
  }
  insn_id = YARVINSN_newarray
  operand_size = 1
  sc_state = 0
  operands = 0x000000011f640118
  insn_info = (line_no = 1, node_id = 3, events = 0)
}

We see that iobj contains a link to a subsequent instruction, as well as an insn_id and some other metadata. The instruction is currently YARVINSN_newarray. If we run next, that should run iobj->insn_id = BIN(opt_ary_freeze);, and our instruction should change:

(lldb) next
(lldb) expr *(iobj)
(INSN) $5 = {
  //...
  insn_id = YARVINSN_opt_ary_freeze
  //...
}

It does! The instruction was changed from newarray to opt_ary_freeze! The optimization is at least partially complete (i’m not sure if more is involved, yet).

Making one small step towards `opt_respond_to`

This is already the longest and densest post in the series. But i’d love to make some actual progress towards a new instruction. Let’s pattern match on respond_to? in the peephole optimizer.

Here is our sample program:

puts "Did you know you can write to $stdout?" if $stdout.respond_to?(:write)

Run with RUNOPT0=--dump=insns make runruby, we get the following instructions:

== disasm: #<ISeq:<main>./test.rb:1 (1,0)-(1,76)>
0000 getglobal                              :$stdout                  (   1)[Li]
0002 putobject                              :write
0004 opt_send_without_block                 <calldata!mid:respond_to?, argc:1, ARGS_SIMPLE>
0006 branchunless                           14
0008 putself
0009 putchilledstring                       "Did you know you can write to $stdout?"
0011 opt_send_without_block                 <calldata!mid:puts, argc:1, FCALL|ARGS_SIMPLE>
0013 leave
0014 putnil
0015 leave

I want to match on this line:

0004 opt_send_without_block       <calldata!mid:respond_to?, argc:1, ARGS_SIMPLE>

Here’s my attempt. I’m going to copy what the newarray freeze optimization is doing, and just try changing a few things to match my example. Right underneath the code we’ve been debugging for newarray, i’m adding this:

// If the instruction is `send_without_block`, ie `0004 opt_send_without_block`
if (IS_INSN_ID(iobj, send_without_block)) {
    // Pull the same info the `newarray` optimization does
    const struct rb_callinfo *ci = (struct rb_callinfo *)OPERAND_AT(iobj, 0);
    const rb_iseq_t *blockiseq = (rb_iseq_t *)OPERAND_AT(iobj, 1);

    // <calldata!mid:respond_to?, argc:1, ARGS_SIMPLE>
    // 1. We have ARGS_SIMPLE, which is probably what `vm_ci_simple(ci)` checks for
    // 2. We have argc:1, which should match `vm_ci_argc(ci) == 1`
    // 3. We send without a block, hence blockiseq == NULL
    // 4. The method id (mid) for `vm_ci_mid(ci)` matches `idRespond_to`. I searched around for names
    //    that seemed similar to idFreeze, but replacing `idFreeze` with `idRespond` and found `idRespond_to`
    if (vm_ci_simple(ci) && vm_ci_argc(ci) == 1 && blockiseq == NULL && vm_ci_mid(ci) == idRespond_to) {
        int i = 0;
    }
}

Now i’ll follow the same debugging as before, but i’ll add a breakpoint in compile.c where I added my new code. Specifically, I’m setting a breakpoint at the int i = 0; so I am inside the if statement:

(lldb) break set --file ruby.c --line 2616
Breakpoint 1: where = ruby`process_options + 4068 at ruby.c:2616:38
(lldb) run
(lldb) break set --file compile.c --line 3491
Breakpoint 2: where = ruby`iseq_peephole_optimize + 2536 at compile.c:3491:17
(lldb) continue
Process 61925 resuming
Process 61925 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 2.1
    frame #0: ruby`iseq_peephole_optimize(...) at compile.c:3491:17
   3488         const rb_iseq_t *blockiseq = (rb_iseq_t *)OPERAND_AT(iobj, 1);
   3489
   3490         if (vm_ci_simple(ci) && vm_ci_argc(ci) == 1 && blockiseq == NULL && vm_ci_mid(ci) == idRespond_to) {
-> 3491             int i = 0;
   3492         }
   3493     }
   3494

I think it worked! It pattern matched on the characteristics of the respond_to? call, and hit the breakpoint set on int i = 0;. It’s a tiny step, but it’s a first step in the direction of adding the optimization.

Using `gdb`

For anyone wanting to do the same work using gdb, it’s pretty similar. Let’s start off by creating a breakpoints.gdb file in the root of your project. This will set you up with your initial breakpoint, similar to how we ran lldb, and set the breakpoint before calling run:

break ruby.c:2616

When you run make gdb-ruby, you can use the same backtrace command, bt:

> make gdb-ruby
Thread 1 "ruby" hit Breakpoint 4, process_options (...) at ../ruby.c:2616
2616	            iseq = pm_iseq_new_main(&pm->node, opt->script_name, path, parent, optimize, &error_state);
(gdb) bt
#0  process_options (...) at ../ruby.c:2616
#1  in ruby_process_options (...) at ../ruby.c:3169
#2  in ruby_options (...) at ../eval.c:117
#3  in rb_main (...) at ../main.c:43
#4  in main (...) at ../main.c:68
(gdb)

From here, you can set your next breakpoint so that you can see the compilation solely for the newarray instruction from our test.rb program:

(gdb) break compile.c:3476
Breakpoint 5 at 0xaaaabaa22f14: file ../compile.c, line 3476
(gdb) continue
Continuing.

Thread 1 "ruby" hit Breakpoint 5, iseq_peephole_optimize (...) at ../compile.c:3476
3476	                iobj->insn_id = BIN(opt_ary_freeze);

Similar to the lldb command expr, we can inspect the contents of locals using p or print in gdb:

(gdb) p *(iobj)
$2 = {link = {type = ISEQ_ELEMENT_INSN, next = 0xaaaace797ef0, prev = 0xaaaace797e70}, insn_id = YARVINSN_newarray,
  operand_size = 1, sc_state = 0, operands = 0xaaaace796ac8, insn_info = {line_no = 1, node_id = 3, events = 0}}

Finishing up

Ok, this went pretty long. Good on you for sticking in there with me! We’ve found the optimizer, and we’ve pattern matched our way to a respond_to? call. Next, we need to add the new instruction definition and try to actually replace the send with our new instruction. See you next time! 👋🏼

The Ruby Syntax Holy Grail: adding `opt_respond_to` to the Ruby VM, part 3

Wed, 25 Dec 2024 23:37:06 -0500

In Finding the compiler: adding opt_respond_to to the Ruby VM, part 2, I found the entrypoint into the compiler! It takes the root of our abstract syntax tree - pm->node - and produces a rb_iseq_t. rb_iseq_t is an “InstructionSequence”, which represents our virtual machine bytecode. Here’s the code where we left off:

// ruby.c
static VALUE
process_options(int argc, char **argv, ruby_cmdline_options_t *opt)
{
    //...
    pm_parse_result_t *pm = &result.prism;
    int error_state;
    iseq = pm_iseq_new_main(&pm->node, opt->script_name, path, parent, optimize, &error_state);

Nothing about this code screams “I’m the compiler!”. But I am taking an educated guess, since:

The compiler should produce instruction sequences, or iseqs
We know this is the function that returns our program’s iseq when main.c is called
This is the only line that produces an iseq in this function

With all that lining up, I’m confident this is the place we need to investigate further. Now let’s find what needs to change to add our new bytecode. Stepping into pm_iseq_new_main, there are a few layers I need to wade through to get to something that seems promising.

Getting your own environment setup

Before I dig in further, let’s take a quick step back. In case you want to join in at home, here are some simple(ish) steps for doing that.

Check out my guides on how to setup your development environment to hack on CRuby. I have a docker guide and a MacOS guide. The one thing I didn’t add to them was cloning the repo. So you’ll need to git clone the Ruby repository.
Building CRuby can take a few minutes the first time you run it. After you’ve built everything, you can easily test your local setup using a file named test.rb. Create a test.rb file in the root of your CRuby folder.
You can run make runruby, and it will run whatever is inside your test.rb file. You can even use debug tools to debug the C code you’re running and inspecting - we’ll talk more about those later.

Back to the investigation

First we’ve got the function pm_iseq_new_main, which seems to set us up as the <main> rb_iseq_t.

// iseq.c
rb_iseq_t *
pm_iseq_new_main(pm_scope_node_t *node, VALUE path, VALUE realpath, const rb_iseq_t *parent, int opt, int *error_state)
{
    iseq_new_setup_coverage(path, (int) (node->parser->newline_list.size - 1));

    return pm_iseq_new_with_opt(node, rb_fstring_lit("<main>"),
                                path, realpath, 0,
                                parent, 0, ISEQ_TYPE_MAIN, opt ? &COMPILE_OPTION_DEFAULT : &COMPILE_OPTION_FALSE, error_state);
}

This looked immediately familiar to me. It sticks out because i’ve seen that <main> before. Let’s run a simple Ruby program:

begin
  raise
rescue => e
  puts e.backtrace
end

All our program does it raise an error, rescue the error, then puts the backtrace. What does that backtrace look like?

../test.rb:2:in '<main>'

Oh yea! We are executing our code at the top-level of the program. And that top level is referred to as <main>. I think that’s being named by our pm_iseq_new_with_opt(node, rb_fstring_lit("<main>")... call - neat!

iseq_new_setup_coverage just sets up some optional coverage information, so let’s move to pm_iseq_new_with_opt:

rb_iseq_t *
pm_iseq_new_with_opt(pm_scope_node_t *node, VALUE name, VALUE path, VALUE realpath,
                     int first_lineno, const rb_iseq_t *parent, int isolated_depth,
                     enum rb_iseq_type type, const rb_compile_option_t *option, int *error_state)
{
    rb_iseq_t *iseq = iseq_alloc();
    ISEQ_BODY(iseq)->prism = true;
    //...
    struct pm_iseq_new_with_opt_data data = {
        .iseq = iseq,
        .node = node
    };
    rb_protect(pm_iseq_new_with_opt_try, (VALUE)&data, error_state);

    if (*error_state) return NULL;

    return iseq_translate(iseq);
}

This code allocates (iseq_alloc) an rb_iseq_t struct and sets it as being part of prism. I believe rb_protect is to allow handling of errors that might be raised while running a particular function? Looking at the git blame I see Peter Zhu added it to catch errors, so confirmed ✅. Not alot is happening here otherwise, so let’s jump into pm_iseq_new_with_opt_try:

VALUE
pm_iseq_new_with_opt_try(VALUE d)
{
    struct pm_iseq_new_with_opt_data *data = (struct pm_iseq_new_with_opt_data *)d;

    // This can compile child iseqs, which can raise syntax errors
    pm_iseq_compile_node(data->iseq, data->node);

    // This raises an exception if there is a syntax error
    finish_iseq_build(data->iseq);

    return Qundef;
}

This is the most promising piece of code so far. It’s the first thing that kindly tells me in explicit terms: “I am going to compile something”. Presumably pm_iseq_compile_node compiles data->node into the data->iseq. It’s in a new file called prism_compile.c. Let’s check it out!

// prism_compile.c
VALUE
pm_iseq_compile_node(rb_iseq_t *iseq, pm_scope_node_t *node)
{
    //...
    if (pm_iseq_pre_execution_p(iseq)) {
        //...
        pm_compile_node(iseq, (const pm_node_t *) node, body, false, node);
        //...
    }
    else {
        //...
        pm_compile_node(iseq, (const pm_node_t *) node, ret, false, node);
    }

    CHECK(iseq_setup_insn(iseq, ret));
    return iseq_setup(iseq, ret);
}

😮‍💨. There are many layers to this compilation. Primarily, this function seems to do two things: “compile” the node, then “setup” the iseq. I don’t know why the iseq “setup” is required yet. Let’s start with pm_compile_node and I’ll come back to the rest:

static void
pm_compile_node(rb_iseq_t *iseq, const pm_node_t *node, LINK_ANCHOR *const ret, bool popped, pm_scope_node_t *scope_node)
{
    const pm_parser_t *parser = scope_node->parser;
    //...
    switch (PM_NODE_TYPE(node)) {
      case PM_ALIAS_GLOBAL_VARIABLE_NODE:
        // alias $foo $bar
        // ^^^^^^^^^^^^^^^
        pm_compile_alias_global_variable_node(iseq, (const pm_alias_global_variable_node_t *) node, &location, ret, popped, scope_node);
        return;
      //...
      case PM_ARRAY_NODE: {
        // [foo, bar, baz]
        // ^^^^^^^^^^^^^^^
        const pm_array_node_t *cast = (const pm_array_node_t *) node;
        pm_compile_array_node(iseq, (const pm_node_t *) cast, &cast->elements, &location, ret, popped, scope_node);
        return;
      }
      //...
      case PM_FLIP_FLOP_NODE: {
        // if foo .. bar; end
        //    ^^^^^^^^^^
        const pm_flip_flop_node_t *cast = (const pm_flip_flop_node_t *) node;
        //...
        pm_compile_flip_flop(cast, else_label, then_label, iseq, location.line, ret, popped, scope_node);
        //...
      }
      //...
      case PM_IT_LOCAL_VARIABLE_READ_NODE: {
        // -> { it }
        //      ^^
        if (!popped) {
            PUSH_GETLOCAL(ret, location, scope_node->local_table_for_iseq_size, 0);
        }

        return;
      }
      //...
      case PM_MODULE_NODE: {
        // module Foo; end
        //...
      }
      //...
}

pm_compile_node puts the “fun” in “function”. It’s really cool! This 1800+ line monster seems to cover a huge swath of Ruby syntax. Maybe all of it? The prism_compile.c file is 11 thousand lines long, as each case of this switch statement branches off into more granular node compilations, like pm_compile_array_node and pm_compile_flip_flop.

With that in mind, it is also an utterly daunting file to consider for the opt_respond_to instruction. Do I edit this file? Where would I even start? I need to swap out a method call to respond_to? - there is code that seems to handle method calls:

case PM_CALL_NODE:
    // foo
    // ^^^
    //
    // foo.bar
    // ^^^^^^^
    //
    // foo.bar() {}
    // ^^^^^^^^^^^^
    pm_compile_call_node(iseq, (const pm_call_node_t *) node, ret, popped, scope_node);
    return;

Maybe that’s it?

I think I need to use a cheat code here to give me some direction. In the previous post, I mentioned Étienne Barrié’s PR to add optimized instructions for frozen literal Hash and Array. I’ve been mostly ignoring it so far, but I think it’s time I use that for a bit of direction on where to go from here.

I think we’re close! So far, I’ve navigated the code manually. In the next post, we’re going to actually run and debug some code, and dig a bit into Étienne’s work. See you then! 👋🏼

Finding the compiler: adding `opt_respond_to` to the Ruby VM, part 2

Mon, 23 Dec 2024 00:41:51 -0500

In Adding opt_respond_to to the Ruby VM: part 1, inspired by recent JSON gem optimizations, I setup my goal: I want to add a new bytecode instruction to the Ruby VM which optimizes respond_to? calls. I took this Ruby code:

if $stdout.respond_to?(:write)
  puts "Did you know you can write to $stdout?"
end

And identified what bytecode instructions matter most (I think):

== disasm: #<ISeq:<compiled>@<compiled>:1 (1,0)-(3,3)>
# ...
0002 putobject                 :write
0004 opt_send_without_block    <calldata!mid:respond_to?, argc:1, ARGS_SIMPLE>

This seems pretty low-level! But it’s still very high-level in terms of what I need to actually do. I know what instructions matter, but how can I change them?

A little help from Git

Thankfully, @byroot gave me some helpful direction in the form of a recent PR. Étienne Barrié recently merged a PR to add optimized instructions for frozen literal Hash and Array. It adds special handling for frozen, empty arrays and hashes so that when you use them in your code, calls like [].freeze will not result in any additional object allocations. Cool! A neat enhancement, and kind of perfect for me to analyze.

I think the use-case is simpler than what i’d need for opt_respond_to, but looking at that PR I can see an important part of adding a new bytecode instruction is in compile.c. I’ll eventually need to add some new logic there, but what steps does CRuby take to get to compile.c?

Starting from main

Knowing a bit about the CRuby source, and how C programs start, I know there is a main function that kicks everything off. In CRuby, it’s helpfully in main.c, at the root of the project:

int
main(int argc, char **argv)
{
    //...
    return rb_main(argc, argv);
}

static int
rb_main(int argc, char **argv)
{
    RUBY_INIT_STACK;
    ruby_init();
    return ruby_run_node(ruby_options(argc, argv));
}

I’m going to guess at not needing RUBY_INIT_STACK or ruby_init for adding a new instruction. I took a peek at it and it all seems related to setting up the runtime, and creating data structures needed for the Ruby virtual machine. Past that, there are only two function calls: ruby_options and ruby_run_node. ruby_options sounds like it would just get the options needed for the program. Maybe we need to go into ruby_run_node?

int
ruby_run_node(void *n)
{
    rb_execution_context_t *ec = GET_EC();
    int status;
    if (!ruby_executable_node(n, &status)) {
        rb_ec_cleanup(ec, (NIL_P(ec->errinfo) ? TAG_NONE : TAG_RAISE));
        return status;
    }
    return rb_ec_cleanup(ec, rb_ec_exec_node(ec, n));
}

Maybe? It looks like if the “node” n isn’t executable, it fails. I’ll concentrate instead on the success path on the last line. rb_ec_exec_node will run first, followed by rb_ec_cleanup.

📝 All this ec_* stuff seems to stand for execution_context, which presumably is the state of the runtime at any given point?

📝 I’m not Mr. C programmer - so I didn’t know what void *n meant. Looking it up, this seems to be a way of specifying a generic pointer type that can point to any data type

Let’s start by checking rb_ec_exec_node:

static int
rb_ec_exec_node(rb_execution_context_t *ec, void *n)
{
    volatile int state;
    rb_iseq_t *iseq = (rb_iseq_t *)n;
    if (!n) return 0;

    EC_PUSH_TAG(ec);
    if ((state = EC_EXEC_TAG()) == TAG_NONE) {
        rb_iseq_eval_main(iseq);
    }
    EC_POP_TAG();
    return state;
}

Hmmmm. This is the first place where I don’t like what i’m seeing. My primary concern is that void *n is getting cast to rb_iseq_t. The class we used to compile our Ruby sample in part 1 - RubyVM::InstructionSequence - is defined in a C file called iseq.c. So in CRuby, iseq stands for “InstructionSequence”. If we already have an iseq, I think it means our code has already been compiled and we’ve gone too far.

Stepping back to `ruby_options`

ruby_run_node doesn’t do much aside from calling rb_ec_exec_node. So if ruby_run_node and rb_ec_exec_node are not the right functions… that only leaves ruby_options. Not what I would expect, but let’s check:

void *
ruby_options(int argc, char **argv)
{
    rb_execution_context_t *ec = GET_EC();
    enum ruby_tag_type state;
    void *volatile iseq = 0;

    EC_PUSH_TAG(ec);
    if ((state = EC_EXEC_TAG()) == TAG_NONE) {
        iseq = ruby_process_options(argc, argv);
    }
    else {
        rb_ec_clear_current_thread_trace_func(ec);
        int exitcode = error_handle(ec, ec->errinfo, state);
        ec->errinfo = Qnil; /* just been handled */
        iseq = (void *)INT2FIX(exitcode);
    }
    EC_POP_TAG();
    return iseq;
}

There’s a lot going on in here, but I’m drawn to the iseq = ruby_process_options(argc, argv) line. Let’s dig into ruby_process_options. This is a big one:

void *
ruby_process_options(int argc, char **argv)
{
    ruby_cmdline_options_t opt;
    VALUE iseq;
    const char *script_name = (argc > 0 && argv[0]) ? argv[0] : ruby_engine;

    if (!origarg.argv || origarg.argc <= 0) {
        origarg.argc = argc;
        origarg.argv = argv;
    }
    set_progname(external_str_new_cstr(script_name));  /* for the time being */
    rb_argv0 = rb_str_new4(rb_progname);
    rb_vm_register_global_object(rb_argv0);

#ifndef HAVE_SETPROCTITLE
    ruby_init_setproctitle(argc, argv);
#endif

    iseq = process_options(argc, argv, cmdline_options_init(&opt));

    //...

    return (void*)(struct RData*)iseq;
}

Most of this function seems to be VM setup. But I think we’re getting closer with iseq = process_options(...).

Checking process_options… whoa, this is a ~350 line function! It’s a bit much to all paste in here, but scanning the code, I think we’re on the right track. There are all sorts of option initializations here:

static VALUE
process_options(int argc, char **argv, ruby_cmdline_options_t *opt)
{
    //...
    if (FEATURE_SET_P(opt->features, yjit)) {
        bool rb_yjit_option_disable(void);
        opt->yjit = !rb_yjit_option_disable(); // set opt->yjit for Init_ruby_description() and calling rb_yjit_init()
    }
    //...
    ruby_mn_threads_params();
    Init_ruby_description(opt);
    //...
    ruby_gc_set_params();
    ruby_init_loadpath();
    //...
}

Among many other things, it sets up options for yjit, mn threads, the program description, garbage collection params, and the loadpath. That’s just scratching the surface of this function. Then around 240 lines into the function, I see a very promising if statement:

static VALUE
process_options(int argc, char **argv, ruby_cmdline_options_t *opt)
{
    //...
    struct {
        rb_ast_t *ast;
        pm_parse_result_t prism;
    } result = {0};
    // ... ~240 lines of option handling
    if (!rb_ruby_prism_p()) {
        ast_value = process_script(opt);
        if (!(result.ast = rb_ruby_ast_data_get(ast_value))) return Qfalse;
    }
    else {
        prism_script(opt, &result.prism);
    }

The beginning of the function sets up a struct that contains either a rb_ast_t, or a pm_parse_result_t. Prism is the new default Ruby parser as of Ruby 3.4, so we’re getting close. rb_ast_t must be the format for the prior CRuby parser.

From a naming perspective, I would never have guessed that ruby_options is the place that parses our Ruby code. In principle I guess this is all preamble to actually running the program, so it kind of relates.

I won’t dig into prism_script, since it would create our Abstract Syntax Tree (AST), which I expect later will be used by the compiler:

typedef struct {
    //...
    /** The resulting scope node that will hold the generated AST. */
    pm_scope_node_t node;
    //...
} pm_parse_result_t;

Ok, here we go! I think we’ve got it with this next section! The pm_scope_node_t node (which should be set by prism_script) is used to create our rb_iseq_t *iseq inside of pm_iseq_new_main!

// ~320 lines into the function
pm_parse_result_t *pm = &result.prism;
int error_state;
iseq = pm_iseq_new_main(&pm->node, opt->script_name, path, parent, optimize, &error_state);

We now have the entrypoint into creating our InstructionSequence (our iseq, or rb_iseq_t). I wanted to start digging into the actual compiler, but I think I’ll stop here for today.

Now that we know the entrypoint into the compiler, we can start figuring out what code might need to change to add a new bytecode instruction. Next up, i’m hoping we can find the appropriate area that needs that change. See you then! 👋🏼

Adding `opt_respond_to` to the Ruby VM: part 1

Sun, 22 Dec 2024 01:22:24 -0500

@byroot has been posting a series on optimizations he and others have made to the json gem, and it’s been 🔥🔥🔥. Enjoyable and informative, I highly recommend reading what he’s posted so far.

In his second post, he mentions the possibility of improving performance by adding an additional method cache. This would involve compiling respond_to? calls in a special way:

It actually wouldn’t be too hard to add such a cache, we’d need to modify the Ruby compiler to compile respond_to? calls into a specialized opt_respond_to instruction that does have two caches instead of one. The first cache would be used to look up respond_to? on the object to make sure it wasn’t redefined, and the second one to look up the method we’re interested in. Or perhaps even 3 caches, as you also need to check if the object has a respond_to_missing? method defined in some cases.

That’s an idea I remember discussing in the past with some fellow committers, but I can’t quite remember if there was a reason we didn’t do it yet.

Inspired by his comment, I’m going to add a new bytecode instruction - opt_respond_to - to the Ruby VM, for fun. I don’t know how to add a new bytecode instruction (yet). I don’t know if one would get accepted by the Ruby team. I don’t know if adding it will actually provide a meaningful enhancement to performance. But let’s give it a try, shall we?

Understanding the requirements

I know I want to add a new Ruby Virtual Machine bytecode called opt_respond_to. What does code using respond_to? look like today, after being compiled? Here’s some code to evaluate:

if $stdout.respond_to?(:write)
  puts "Did you know you can write to $stdout?"
end

We can compile it using RubyVM::InstructionSequence. We compile the code, then we disassemble it to see the actual Ruby bytecode:

puts RubyVM::InstructionSequence.compile(DATA.read).disassemble

__END__
if $stdout.respond_to?(:write)
  puts "Did you know you can write to $stdout?"
end

📝 The __END__ format is just a convenient way of supplying some text to your program. Here we put all our Ruby code we want to compile after __END__ and it will be available to our program as an IO object called DATA. Thanks to Drew Bragg for the tip!

Compiling using InstructionSequence gives us:

== disasm: #<ISeq:<compiled>@<compiled>:1 (1,0)-(3,3)>
0000 getglobal                    :$stdout                  (   1)[Li]
0002 putobject                    :write
0004 opt_send_without_block       <calldata!mid:respond_to?, argc:1, ARGS_SIMPLE>
0006 branchunless                 14
0008 putself                                                (   2)[Li]
0009 putstring                    "Did you know you can write to $stdout?"
0011 opt_send_without_block       <calldata!mid:puts, argc:1, FCALL|ARGS_SIMPLE>
0013 leave
0014 putnil
0015 leave

I’m going to guess at the parts I think matter most - putobject, opt_send_without_block and branchunless.

putobject pushes the symbol :write onto the vm stack
opt_send_without_block is given <calldata!mid:respond_to?, argc:1, ARGS_SIMPLE>. I’m guessing calldata is a format for specifying metadata about what is being called. We’ve got the method name, respond_to?, how many args are being used, argc:1, and that the arguments are “simple”, ARGS_SIMPLE. mid stands for… “method id”?
branchunless wouldn’t be specifically related to creating a new instruction, but I just think it’s informative for how the respond_to? result is used. I believe it means “if the last result is false, jump to instruction 14”. 14 in this case is the putnil near the bottom of the bytecode. Each instruction seems to be prefixed with a hex value that identifies the location of the instruction. putobject is located at 0002, opt_send_without_block is located at 0004, branchunless is located at 0006 and putnil is located at 0014

I think the only thing I will be changing is taking calls to respond_to?, and making that a opt_respond_to bytecode instead of opt_send_without_block. We’ll see!

There are actually a few variations for respond_to? that I wasn’t aware of. The interface takes a symbol or a string as the first parameter, and then a boolean for whether to include private and protected methods:

$stdout.respond_to?("write")
$stdout.respond_to?("write", true)
$stdout.respond_to?(:write, true)

Let’s see the bytecode for these variations:

== disasm: #<ISeq:<compiled>@<compiled>:1 (1,0)-(3,33)>
0000 getglobal                    :$stdout                  (   1)[Li]
0002 putstring                    "write"
0004 opt_send_without_block       <calldata!mid:respond_to?, argc:1, ARGS_SIMPLE>
0006 pop
0007 getglobal                    :$stdout                  (   2)[Li]
0009 putstring                    "write"
0011 putobject                    true
0013 opt_send_without_block       <calldata!mid:respond_to?, argc:2, ARGS_SIMPLE>
0015 pop
0016 getglobal                    :$stdout                  (   3)[Li]
0018 putobject                    :write
0020 putobject                    true
0022 opt_send_without_block       <calldata!mid:respond_to?, argc:2, ARGS_SIMPLE>
0024 leave

It doesn’t make a huge difference. The string version is identical, except we see "write" instead of :write in the putobject call. The boolean versions add an additional putobject which pushes the boolean onto the stack. Then the opt_send_without_block call has argc:2 instead of argc:1. Good to understand, but functionally the same.

I’m going to keep these explorations shorter, and break them into parts, so i’m going to stop here. So far we’ve:

Identified that I want to create an opt_respond_to instruction for the Ruby VM
Compiled a simple respond_to? example, and examined what the current bytecode looks like
Identified what I think are the relevant instructions that need to be converted to opt_respond_to

Next up, i’m going to walk through the path CRuby takes from starting the program, to compiling our code, starting with main.c. Here we go!

20 days of ruby gems: part 1

Mon, 16 Dec 2024 20:12:46 -0500

Over on BlueSky, Gregory Brown suggested a #20daygemchallenge. Post gems you’ve either used time and time again, or have inspired you in some way, in no particular order. Mohit Sindhwani suggested writing about them at the end, which sounded like a great idea!

I’m breaking it into two parts. Here’s my breakdown of my first 10 gems posted:

First is HTTParty: https://github.com/jnunemaker/httparty

HTTPart is the OG http gem. It’s widely used and dead simple. There are a million http options out there, but HTTParty still remains a simple, common option. It sits on top of Net::HTTP, so it has the same timeout concern as any Net::HTTP usage. But many Ruby http gems use Net::HTTP, so that’s not a particular knock against it!

Second is Async: https://github.com/socketry/async

Async is the de facto FiberScheduler for Ruby. That’s what allows Fibers to parallelize blocking operations in Ruby 3+. It’s a great gem and a great ecosystem of tools as well, particularly revolving around all things IO and web protocols.

I talk about it in more detail in my In-Depth Ruby Concurrency talk from RubyConf and I’ll have future articles about it as well.

Third is Pitchfork: https://github.com/Shopify/pitchfork

Pitchfork is an evolution of the Unicorn web server. Its innovation is a forking technique called “Reforking”, where processes are forked multiple times from existing “warm” processes, getting to a point of maximally optimized Copy-on-Write performance.

Like async, I talk about it in more detail in my In-Depth Ruby Concurrency talk from RubyConf and I’ll have future articles about it as well.

Fourth is Ractor TVar: https://github.com/ko1/ractor-tvar

Ractor TVar is an implementation of software transactional memory in Ruby. I learned about it from the Mooro gem, which is a later gem pick. It’s a fascinating and largely unknown library that koichi seemed to have released alongside Ractors and not updated since. I’m very curious to read the source more to better understand how it works and maybe even give it a try in some real code. It only documents examples supporting Ractors but claims to also support threads.

Fifth is Strong Migrations: https://github.com/ankane/strong_migrations

I prefer my database migrations to be stress-free and zero-downtime. The strong migrations gem helps me sleep at night. It detects unsafe operations and blocks them from being run, offering safe alternatives.

There are a few other gems that help keep migrations safe, but I prefer the explicit style of strong migrations. Most other options are a bit “magical”, and will try to rewrite things for you.

Sixth is ZipKit: https://github.com/julik/zip_kit

I’ve mostly used ZipTricks in the past, and ZipKit is the successor to that gem. Being able to stream writes to a zip file is amazing for scaling and ZipKit makes it dead simple.

Using it you can, for instance, stream a file to S3, zipping it on the fly as it’s being uploaded! Streaming is the only way you can reasonably manage operations on very large files, so having this option is critical.

Seventh is Falcon: https://github.com/socketry/falcon

Falcon is a web server based on the FiberScheduler (provided by the async gem). It’s a very scalable server, particularly for IO bound operations. There’s a great talk focusing on it from RailsConf 2023 called Look ma, no jobs.

I also did some benchmarking of its web socket performance compared to a node.js implementation and it is very close in performance!

Here’s the code for it:

https://gist.github.com/jpcamara/8a1a09c9c67347c4e32384b9ce806b70

Like async and pitchfork, I talk about it in more detail in my In-Depth Ruby Concurrency talk from RubyConf and I’ll have future articles about it as well.

Eighth is OJ: https://github.com/ohler55/oj

OJ has historically been the fastest JSON parser in the Ruby world. Usually you can just drop it into a project as a JSON replacement and see things immediately speed up, especially if you are doing any heavy JSON processing.

The JSON gem was recently taken over by the Ruby GitHub organization and byroot has been making some big performance improvements to it - maybe we’ll see parity at some point but OJ is still a great choice.

Ninth is io-event: https://github.com/socketry/io-event

The async gem is the public interface, but io-event is what powers the scheduling at the OS level. It provides all of the integrations with each operating systems kernel event queue: io_uring and epoll for Linux, and kqueue for MacOS. IOCP support for windows is still in progress, so it falls back to a basic Ruby select there. If you don’t know why any of that is useful, it’s because it’s an important part of keeping the “Reactor” pattern of asynchronous IO efficient.

Like async, pitchfork and falcon (😅), I talk about the reactor pattern in more detail in my In-Depth Ruby Concurrency talk from RubyConf and I’ll have future articles about it as well. I obviously like concurrency 🙂.

Tenth is Glimmer: https://github.com/AndyObtiva/glimmer

I wasn’t familiar with Glimmer but I learned about it at RubyConf. It’s a DSL for building UIs with pure Ruby and has bindings for desktop app ui layers as well as the web. It’s a really cool concept and I look forward to learning more about it by watching How to build basic desktop applications in Ruby. I’ve been working on a cross platform app using React Native and Tauri - maybe I’ll port some of it to Glimmer as an experiment.

After I finish 11 through 20, I’ll post about them as well. Give these gems a try! 👋

My RubyConf talk is now on YouTube!

Sat, 14 Dec 2024 22:40:45 -0500

I was honored to present a talk on Ruby concurrency at RubyConf 2024. It represents a high-level distillation of much of my writing and research over the past year. The conference itself was great, and presenting was such a fun experience.

Here is the talk, titled “In-Depth Ruby Concurrency: Navigating the Ruby Concurrency Landscape”:

If you want the slides, I’ve published them as a short YouTube video:

I know it’s a bit odd to share my slides as a video, but there are so many animations it was the easiest option to preserve the original flow.

You can also find information about my talk and every other RubyConf 2024 talk at RubyVideo.

Speeding up Ruby by rewriting C… in Ruby

Wed, 04 Dec 2024 00:29:25 -0500

There is a recent language comparison repo which has been getting shared a lot. In it, CRuby was the third slowest option, only beating out R and Python.

The repo author, @BenjDicken, created a fun visualization of each language’s performance. Here’s one of the visualizations, which shows Ruby as the third slowest language benchmarked:

The code for this visualization is from https://benjdd.com/languages/, with permission from @BenjDicken

The repository describes itself as:

A repo for collaboratively building small benchmarks to compare languages.

It contains two different benchmarks:

“Loops”, which “Emphasizes loop, conditional, and basic math performance”
“Fibonacci”, which “Emphasizes function call overhead and recursion.”

The loop example iterates 1 billion times, utilizing a nested loop:

u = ARGV[0].to_i       
r = rand(10_000)                          
a = Array.new(10_000, 0)                 
	
(0...10_000).each do |i|                     
  (0...100_000).each do |j|               
    a[i] += j % u                     
  end
  a[i] += r                      
end
	
puts a[r]

The Fibonacci example is a basic “naive” Fibonacci implementation¹:

def fibonacci(n)
  return 0 if n == 0
  return 1 if n == 1
  fibonacci(n - 1) + fibonacci(n - 2)
end

u = ARGV[0].to_i
r = 0

(1...u).each do |i|
  r += fibonacci(i)
end

puts r

Run on @BenjDicken’s M3 MacBook Pro, Ruby 3.3.6 takes 28 seconds to run the loop iteration example, and 12 seconds to run the Fibonacci example. For comparison, node.js takes a little over a second for both examples - it’s not a great showing for Ruby.

	Fibonacci	Loops
Ruby	12.17s	28.80s
node.js	1.11s	1.03s

From this point on, I’ll use benchmarks relative to my own computer. Running the same benchmark on my M2 MacBook Air, I get 33.43 seconds for the loops and 16.33 seconds for fibonacci - even worse 🥺. Node runs a little over 1 second for fibonacci and 2 seconds for the loop example.

	Fibonacci	Loops
Ruby	16.33s	33.43s
node.js	1.36s	2.07s

Who cares?

In most ways, these types of benchmarks are meaningless. Python was the slowest language in the benchmark, and yet at the same time it’s the most used language on Github as of October 2024. Ruby runs some of the largest web apps in the world. I ran a benchmark recently of websocket performance between the Ruby Falcon web server and node.js, and the Ruby results were close to the node.js results. Are you doing a billion loop iterations or using web sockets?

A programming language should be reasonably efficient - after that the usefulness of the language, the type of tasks you work on, and language productivity outweigh the speed at which you can run a billion iterations of a loop, or complete an intentionally inefficient implementation of a Fibonacci method.

That said:

The programming world loves microbenchmarks 🤷‍♂️
Having a fast benchmark may not be valuable in practice but it has meaning for people’s interest in a language. Some would claim it means you’ll have an easier time scaling performance, but that’s arguable²
It’s disappointing if your language of choice doesn’t perform well. It’s nice to be able to say “I use and enjoy this language, and it runs fast in all benchmarks!”

In the case of this Ruby benchmark, I had a feeling that YJIT wasn’t being applied in the Ruby code, so I checked the repo. Lo and behold, the command was as follows:

ruby ./code.rb 40

We know my results from earlier (33 seconds and 16 seconds). What do we get with YJIT applied?

ruby --yjit ./code.rb 40

	Fibonacci	Loops
Ruby	2.06s	25.57s

Nice! With YJIT, Fibonacci gets a massive boost - going from 16.88 seconds down to 2.06 seconds. It’s close to the speed of node.js at that point!

YJIT makes a more modest difference for the looping example - going from 33.43 seconds down to 25.57 seconds. Why is that?

A team effort

I wasn’t alone in trying out these code samples with YJIT. On twitter, @bsilva96 had asked the same questions:

https://x.com/bsilva96/status/1861136096689606708

@k0kubun came through with insights into why things were slow and ways of improving the performance:

https://x.com/k0kubun/status/1861149512640979260

Let’s unpack his response. There are three parts to it:

Range#each is still written in C as of Ruby 3.4
Integer#times was converted from C to Ruby in Ruby 3.3
Array#each was converted from C to Ruby in Ruby 3.4

1. `Range#each` is still written in C, which YJIT can’t optimize

Looking back at our Ruby code:

(0...10_000).each do |i|                     
  (0...100_000).each do |j|               
    a[i] += j % u                     
  end
  a[i] += r                      
end

It’s written as a range, and range has its own each implementation, which is apparently written in C. The CRuby codebase is pretty easy to navigate - let’s find that implementation 🕵️‍♂️.

Most core classes in Ruby have top-level C files named after them - in this case we’ve got range.c at the root of the project. CRuby has a pretty readable interface for exposing C functions as classes and methods - there is an Init function, usually at the bottom of the file. Inside that Init our classes, modules and methods are exposed from C to Ruby. Here are the relevant pieces of Init_Range:

void
Init_Range(void)
{
  //...
  rb_cRange = rb_struct_define_without_accessor(
    "Range", rb_cObject, range_alloc,
    "begin", "end", "excl", NULL);

  rb_include_module(rb_cRange, rb_mEnumerable);
  // ...
  rb_define_method(rb_cRange, "each", range_each, 0);

First, we define our Range class using rb_struct_define.... We name it “Range”, with a super class of Object (rb_cObject), and some initialization parameters (“begin”, “end” and whether to exclude the last value, ie the .. vs ... range syntax).

Second, we include Enumerable using rb_include_module. That gives us all the great Ruby enumeration methods like map, select, include? and a bajillion others. All you have to do is provide an each implementation and it handles the rest.

Third, we define our “each” method. It’s implemented by the range_each function in C, and takes zero explicit arguments (blocks are not considered in this count).

range_each is hefty. It’s almost 100 lines long, and specializes into several versions of itself. I’ll highlight a few, collapsed all together:

static VALUE
range_each(VALUE range)
{
  //...
  range_each_fixnum_endless(beg);
  range_each_fixnum_loop(beg, end, range);
  range_each_bignum_endless(beg);
  rb_str_upto_endless_each(beg, sym_each_i, 0);
  // and even more...

These C functions handle all the variations of ranges you might use in your own code:

(0...).each
(0...100).each
("a"..."z").each
# and on...

Why does it matter that Range#each is written in C? It means YJIT can’t inspect it - optimizations stop at the function call and resume when the function call returns. C functions are fast, but YJIT can take things further by creating specializations for hot paths of code. There is a great article from Aaron Patterson called Ruby Outperforms C where you can learn more about some of those specialized optimizations.

2. Optimizing our loop: `Integer#times` was converted from C to Ruby in Ruby 3.3

The hot path (where most of our CPU time is spent) is Range#each, which is a C function. YJIT can’t optimize C functions - they’re a black box. So what can we do?

We converted Integer#times to Ruby in 3.3

Interesting! In Ruby 3.3, Integer#times was converted from a C function to a Ruby method! Here’s the 3.3+ version - its pretty simple:

def times
  #... a little C interop code
  i = 0
  while i < self
    yield i
    i = i.succ
  end
  self
end

Very simple. It’s just a basic while loop. Most importantly, it’s all Ruby code, which means YJIT should be able to introspect and optimize it!

An aside on `Integer#succ`

The slightly odd part of that code is i.succ. I’d never heard of Integer#succ, which apparently gives you the “successor” to an integer.

I’ve never seen this show, and yet it’s the first thing I thought of when I learned about this method. Thanks, advertising.

There was a PR to improve the performance of Integer#succ in early 2024, which helped me understand why anyone would ever use it:

We use Integer#succ when we rewrite loop methods in Ruby (e.g. Integer#times and Array#each) because opt_succ (i = i.succ) is faster to dispatch on the interpreter than putobject 1; opt_plus (i += 1).

https://github.com/ruby/ruby/pull/9519

Integer#succ is like a virtual machine cheat code. It takes a common operation (adding 1 to an integer) and turns it from two virtual machine operations into one. We can call disasm on the times method to see that in action:

puts RubyVM::InstructionSequence.disasm(1.method(:times))

The Integer#times method gets broken down into a lot of Ruby VM bytecode, but we only care about a few lines:

...
0025 getlocal_WC_0   i@0
0027 opt_succ        <calldata!mid:succ, ARGS_SIMPLE>[CcCr]
0029 setlocal_WC_0   i@0
...

getlocal_WC_0 gets our i variable from the current scope. That’s the i in i.succ
opt_succ performs the succ call in our i.succ. It will either call the actual Integer#succ method, or an optimized C function for small numbers
In Ruby 3.4 with YJIT enabled, small numbers get optimized even further into machine code (just a note, not shown in the VM machine code)
setlocal_WC_0 sets the result of opt_succ to our local variable i

If we change from i = i.succ to i += 1, we now have two VM operations take the place of opt_succ:

...
0025 getlocal_WC_0        i@0
0027 putobject_INT2FIX_1_
0028 opt_plus             <calldata!mid:+, argc:1, ARGS_SIMPLE>
0029 setlocal_WC_0        i@0
...

Everything is essentially the same as before, except now we have two steps to go through instead of one:

putobject_INT2FIX_1_ pushes the integer 1 onto the virtual machine stack
opt_plus is the + in our += 1, and calls either the Ruby + method or an optimized C function for small numbers
There is probably a YJIT optimization for opt_plus as well

If there is nothing else to learn from this code, it’s this: the kinds of optimizations you do at the VM and JIT level are deep. When writing general Ruby programs we typically don’t and shouldn’t consider the impact of one versus two machine code instructions. But at the JIT level, on the scale of millions and billions of operations, it matters!

Back to `Integer#times`

Let’s try running our benchmark code again, using times! Instead of iterating over ranges, we simply iterate for 10_000 and 100_000 times:

u = ARGV[0].to_i        
r = rand(10_000)        
a = Array.new(10_000, 0)
	
10_000.times do |i|
  100_000.times do |j|
    a[i] += j % u
  end
  a[i] += r
end
	
puts a[r]

	Loops
Range#each	25.57s
Integer#times	13.66s

Nice! YJIT makes a much larger impact using Integer#times. That trims things down significantly, taking it down to 13.66 seconds on my machine. On @k0kobun’s machine it actually goes down to 9 seconds (and 8 seconds on Ruby 3.4).

It’s probably Ruby 3.5’s job to make it faster than 8s though.

We might look forward to even faster performance in Ruby 3.5. We’ll see!

3. `Array#each` was converted from C to Ruby in Ruby 3.4

CRuby continues to see C code rewritten in Ruby, and in Ruby 3.4 Array#each was one of those changes. Here is an example of the first attempt at implementing it:

def each
  unless block_given?
    return to_enum(:each) { self.length }
  end
  i = 0
  while i < self.length
    yield self[i]
    i = i.succ
  end
  self
end

Super simple and readable! And YJIT optimizable!

Unfortunately, due to something related to CRuby internals, it contained race conditions. A later implementation landed in Ruby 3.4.

def each
  Primitive.attr! :inline_block, :c_trace

  unless defined?(yield)
    return Primitive.cexpr! 'SIZED_ENUMERATOR(self, 0, 0, ary_enum_length)'
  end
  _i = 0
  value = nil
  while Primitive.cexpr!(%q{ ary_fetch_next(self, LOCAL_PTR(_i), LOCAL_PTR(value)) })
    yield value
  end
  self
end

Unlike the first implementation, and unlike Integer#times, things are a bit more cryptic this time. This is definitely not pure Ruby code that anyone could be expected to write. Somehow, the Primitive module seems to allow evaluating C code from Ruby, and in doing so avoids the race conditions present in the pure Ruby solution³.

By fetching indexes and values using C code, I think it results in a more atomic operation. I have no idea why the Primitive.cexpr! is used to return the enumerator, or what value Primitive.attr! :inline_block provides. Please comment if you have insights there!

I was a little loose with my earlier Integer#times source code as well. That actually had a bit of this Primitive syntax as well. The core of the method is what we looked at, and it’s all Ruby, but the start of the method contains the same Primitive calls for :inline_block and returning the enumerator:

def times
  Primitive.attr! :inline_block
  unless defined?(yield)
    return Primitive.cexpr! 'SIZED_ENUMERATOR(self, 0, 0, int_dotimes_size)'
  end
  #...

Ok - it’s more cryptic than Integer#times was, but Array#each is mostly Ruby (on Ruby 3.4+). Let’s give it a try using arrays instead of ranges or times:

u = ARGV[0].to_i
r = rand(10_000)
a = Array.new(10_000, 0)
	
outer = (0...10_000).to_a.freeze
inner = (0...100_000).to_a.freeze
outer.each do |i|
  inner.each do |j|
    a[i] += j % u
  end
  a[i] += r
end
	
puts a[r]

Despite the embedded C code, YJIT still seems capable of making some hefty performance optimizations. It’s within the same range as Integer#times!

	Loops
Range#each	25.57s
Integer#times	13.66s
Array#each	13.96s

Microbenchmarking Ruby performance

I’ve forked the original language implementation repo, and created my own repository called “Ruby Microbench”. It takes all of the examples discussed, as well as several other forms of doing the iteration in Ruby: https://github.com/jpcamara/ruby_microbench

Here is the output of just running those using Ruby 3.4 with and without YJIT:

	fibonacci	array#each	range#each	times	for	while	loop do
Ruby 3.4 YJIT	2.19s	14.02s	26.61s	13.12s	14.91s	37.10s	13.95s
Ruby 3.4	16.49s	34.29s	33.88s	33.18s	36.32s	37.14s	50.65s

I have no idea why the while loop example I wrote seems to be so slow. I’d expected it to run much faster. Maybe there’s an issue with how I wrote it - feel free to open an issue or PR if you see something wrong with my implementation. The loop do (taken from @timtilberg’s example) runs around the same speed as Integer#times - although its performance is awful with YJIT turned off.

📝 The for in loop and array#each have very similar performance, and that’s because at the Ruby VM bytecode level they are almost identical. for in is mostly syntactic sugar that transforms into an #each call in the VM. Thanks to Daniel Colson for pointing this out, and you can read his for loops in Ruby article for some additional information and nuance around for in!

In addition to running Ruby 3.4, for fun I have it using rbenv to run:

Ruby 3.3
Ruby 3.3 YJIT
Ruby 3.2
Ruby 3.2 YJIT
TruffleRuby 24.1
Ruby Artichoke
MRuby

A few of the test runs are listed here:

	fibonacci	array#each	range#each	times	for	while	loop do
Ruby 3.4 YJIT	2.19s	14.02s	26.61s	13.12s	14.91s	37.10s	13.95s
Ruby 3.4	16.49s	34.29s	33.88s	33.18s	36.32s	37.14s	50.65s
TruffleRuby 24.1	0.92s	0.97s	0.92s	2.39s	2.06s	3.90s	0.77s
MRuby 3.3	28.83s	144.65s	126.40s	128.22s	133.58s	91.55s	144.93s
Artichoke	19.71s	236.10s	214.55s	214.51s	215.95s	174.70s	264.67s

Based on that, I’ve taken the original visualization and made a Ruby specific one here just for the fibonacci run:

Speeding up `range#each`

Can we, the non @k0kobun’s of the world, make range#each faster? If I monkey patch the Range class with a pure-ruby implementation, things do get much faster! Here’s my implementation:

class Range
  def each
    beginning = self.begin
    ending = self.end
    i = beginning
    loop do
      break if i == ending
      yield i
      i = i.succ
    end
  end
end

And here is the change in performance - 2 seconds slower than times - not bad!

	Time spent
Range#each in C	25.57s
Range#each in Ruby	16.64s

This is obviously over-simplified. I don’t handle all of the different cases of Range, and there may be nuances I am missing. Also, most of the Ruby rewritten methods I’ve seen invoke a Primitive class for certain operations. I’d love to learn more about when and why it’s needed.

But! It goes to show the power of moving things out of C and letting YJIT optimize our code. It can improve performance in ways that would be difficult or impossible to replicate in regular C code.

YJIT standard library

Last year Aaron Patterson wrote an article called Ruby Outperforms C, in which he rewrote a C extension in Ruby for some GraphQL parsing. The Ruby code outperformed C thanks to YJIT optimizations.

This got me thinking that it would be interesting to see a kind of “YJIT standard library” emerge, where core ruby functionality run in C could be swapped out for Ruby implementations for use by people using YJIT.

As it turns out, this is almost exactly what the core YJIT team has been doing. In many cases they’ve completely removed C code, but more recently they’ve created a with_yjit block. The code will only take effect if YJIT is enabled, and otherwise the C code will run. For example, this is howArray#each is implemented:

with_yjit do
  if Primitive.rb_builtin_basic_definition_p(:each)
    undef :each

    def each # :nodoc:
      # ... we examined this code earlier ...
    end
  end
end

As of Ruby 3.3, YJIT can be lazily initialized. Thankfully the with_yjit code handles this - the appropriate with_yjit versions of methods will be run once YJIT is enabled:

# Uses C-builtin
[1, 2, 3].each do |i|
  puts i
end

RubyVM::YJIT.enable

# Uses Ruby version, which can be YJIT optimized
[1, 2, 3].each do |i|
  puts i
end

This is because with_yjit is a YJIT “hook”, which is called the moment YJIT is enabled. After being called, it is removed from the runtime using undef :with_yjit.

Investigating YJIT optimizations

We’ve looked at Ruby code. We’ve looked at C code. We’ve looked at Ruby VM bytecode. Why not take it one step deeper and look at some machine code? And maybe some Rust code? Hey - where are you going! Don’t walk away while I’m talking to you!

If you haven’t walked away, or skipped to the next section, let’s take a look at a small sliver of YJIT while we’re here!

We can see the machine code YJIT generates 😱. It’s possible by building CRuby from source with YJIT debug flags. If you’re on a Mac you can see my MacOS setup for hacking on CRuby or my docker setup for hacking on CRuby for more elaborate instructions on building Ruby. But the simplified step is when you go to ./configure Ruby, you hand in an option of --enable-yjit=dev:

./configure --enable-yjit=dev
make install

Let’s use our Integer#times example from earlier as our example Ruby code:

u = ARGV[0].to_i
r = rand(10_000)
a = Array.new(10_000, 0)
	
10_000.times do |i|
  100_000.times do |j|
    a[i] += j % u
  end
  a[i] += r
end
	
puts a[r]

Because you’ve built Ruby with YJIT in dev mode, you can hand in the --yjit-dump-disasm flag when running your ruby program:

./ruby --yjit --yjit-dump-disasm test.rb 40

Using this, we can see the machine code created. We’ll just focus in on one tiny part - the machine code equivalent of the Ruby VM bytecode we read earlier. Here is the original VM bytecode for opt_succ, which is generated when you call i.succ, the Integer#succ method:

...
0027 opt_succ        <calldata!mid:succ, ARGS_SIMPLE>[CcCr]
...

And here is the machine code YJIT generates in this scenario, on my Mac M2 arm64 architecture:

# Block: times@<internal:numeric>:259 
# reg_mapping: [Some(Stack(0)), None, None, None, None]
# Insn: 0027 opt_succ (stack_size: 1)
# call to Integer#succ
# guard object is fixnum
0x1096808c4: tst x1, #1
0x1096808c8: b.eq #0x109683014
0x1096808cc: nop 
0x1096808d0: nop 
0x1096808d4: nop 
0x1096808d8: nop 
0x1096808dc: nop 
# Integer#succ
0x1096808e0: adds x11, x1, #2
0x1096808e4: b.vs #0x109683048
0x1096808e8: mov x1, x11

To be honest, I about 25% understand this, and 75% am combining my own logic and AI to learn it 🤫. Feel free to yell at me if I get it a little wrong, I’d love to learn more. But here’s how I break this down.

# Block: times@<internal:numeric>:259

👆🏼This roughly corresponds to the line i = i.succ in the Integer#times method in numeric.rb. I say roughly because in my current code I see that on line 258, but maybe it shows the end of the block it’s run in since YJIT compiles “blocks” of code:

256: while i < self
257:   yield i
258:   i = i.succ
259: end

# reg_mapping: [Some(Stack(0)), None, None, None, None]
# Insn: 0027 opt_succ (stack_size: 1)
# call to Integer#succ

👆🏼I have no idea what reg_mapping means - probably mapping how it uses a CPU register? Insn: 0027 opt_succ looks very familiar! That’s our VM bytecode! call to Integer#succ is just a helpful comment added. YJIT is capable of adding comments to the machine code. We still haven’t even left the safety of the comments 😅.

# guard object is fixnum

👆🏼This is interesting. I can find a corresponding bit of Rust code that maps directly to this. Let’s take a look at it:

fn jit_rb_int_succ(
  //...
  asm: &mut Assembler,
  //...
) -> bool {
  // Guard the receiver is fixnum
  let recv_type = asm.ctx.get_opnd_type(StackOpnd(0));
  let recv = asm.stack_pop(1);
  if recv_type != Type::Fixnum {
    asm_comment!(asm, "guard object is fixnum");
    asm.test(recv, Opnd::Imm(RUBY_FIXNUM_FLAG as i64));
         asm.jz(Target::side_exit(Counter::opt_succ_not_fixnum));
  }

  asm_comment!(asm, "Integer#succ");
  let out_val = asm.add(recv, Opnd::Imm(2)); // 2 is untagged Fixnum 1
  asm.jo(Target::side_exit(Counter::opt_succ_overflow));

  // Push the output onto the stack
  let dst = asm.stack_push(Type::Fixnum);
  asm.mov(dst, out_val);

  true
}

Oh nice! This is the actual YJIT Rust implementation of the opt_succ call. This is that optimization @k0kobun made to further improve opt_succ performance beyond the bytecode C function calls. We’re in the section that is checking if what we’re operating on is a Fixnum, which is a way small integers are stored internally in CRuby:

if recv_type != TypeFixnum 
  asm_comment!(asm, "guard object is fixnum");
  asm.test(recv, Opnd::Imm(RUBY_FIXNUM_FLAG as i64));
  asm.jz(Target::side_exit(Counter::opt_succ_not_fixnum));
}

That becomes this machine code:

# guard object is fixnum
0x1096808c4: tst x1, #1
0x1096808c8: b.eq #0x109683014

asm.test generates tst x1, #1, which according to an AI bot I asked is checking the least significant bit, which is a Fixnum “tag” that indicates this is a Fixnum. If it’s Fixnum, the result is 1 and b.eq is false. If it’s not a Fixnum, the result is 0 and b.eq is true and jumps away from this code.

0x1096808cc: nop 
0x1096808d0: nop 
0x1096808d4: nop 
0x1096808d8: nop 
0x1096808dc: nop

🤖 “NOPs for alignment/padding”. Thanks AI. I don’t know why it is needed, but at least I know what it probably is.

Finally, we actually add 1 to the number.

asm_comment!(asm, "Integer#succ");
let out_val = asm.add(recv, Opnd::Imm(2)); // 2 is untagged Fixnum 1
asm.jo(Target::side_exit(Counter::opt_succ_overflow));

// Push the output onto the stack
let dst = asm.stack_push(Type::Fixnum);
asm.mov(dst, out_val);

The Rust code generates our Integer#succ comment. Then, to add 1, because of the “Fixnum tag” data embedded within our integer, actually means we have to add 2 using adds x11, x1, #2 😵‍💫. If we overflow the space available, it exits to a different code path - b.vs is a branch on overflow. Otherwise, it stores the result with mov x1, x11!

# Integer#succ
0x1096808e0: adds x11, x1, #2
0x1096808e4: b.vs #0x109683048
0x1096808e8: mov x1, x11

😮‍💨. That was a lot. And it seems like alot of working is being done, but because it’s such low level machine code it’s presumably super fast. We examined a teensy tiny portion of what YJIT is capable of generating - JITs are complicated!

Thanks to @k0kobun for providing me with the commands and pointing me at the YJIT docs which contain tons of additional options as well.

The future of CRuby optimizations

The irony of language implementation is that you often work less in the language you’re implementing than you do in something lower-level - in Ruby’s case, that’s mostly C and some Rust.

With a layer like YJIT, it potentially opens up a future where more of the language becomes plain Ruby, and Ruby developer contribution is easier. Many languages have a smaller low level core, and the majority of the language is written in itself (like Java, for instance). Maybe that’s a future for CRuby, someday! Until then, keep the YJIT optimizations coming, YJIT team!

Naive in this case meaning that there are more efficient ways to calculate fibonacci numbers in a program ↩︎
MJIT, the precursor to YJIT, made Ruby much faster on certain benchmarks. But on large realistic Rails applications it actually made things slower ↩︎
When C code is running, it has to opt-in to releasing the GVL, so it’s more difficult for threads to corrupt or modify data mid-operation. The original Ruby version could yield the GVL at points that would invalidate the array. That’s my understanding of the situation anyways. ↩︎

My MacOS setup for hacking on CRuby

Mon, 02 Dec 2024 22:59:05 -0500

I recently posted my docker setup for hacking on CRuby, which showed how I test Linux features when working on CRuby. But most of the time, I just build CRuby directly on MacOS.

The Building Ruby guide from ruby-lang.org is the most up-to-date guide on doing this, but I like to spell it out exactly in order of how I do it day-to-day. So this is for me more than anything, but you may find it helpful!

# /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
xcode-select --install
brew update
brew install openssl@3
brew install autoconf
brew install gperf
brew install libffi
brew install libyaml
brew install zlib
brew install gmp
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

export CONFIGURE_ARGS=""
for ext in openssl readline libyaml zlib; do
  CONFIGURE_ARGS="${CONFIGURE_ARGS} --with-$ext-dir=$(brew --prefix $ext)"
done

./autogen.sh
mkdir build && cd build
mkdir ./.rubies

../configure --prefix="/path/to/ruby/build/.rubies/ruby-master" --disable-install-doc --config-cache --enable-debug-env optflags="-O0 -fno-omit-frame-pointer" CFLAGS="-DRUBY_DEBUG -O0" --with-opt-dir=$(brew --prefix gmp):$(brew --prefix jemalloc)

make install

That’s pretty much everything I do when setting things up!

If you want to run YJIT in “dev” mode, you add --enable-yjit=dev to the configure call:

../configure --prefix="/path/to/ruby/build/.rubies/ruby-master" --disable-install-doc --config-cache --enable-debug-env optflags="-O0 -fno-omit-frame-pointer" CFLAGS="-DRUBY_DEBUG -O0" --with-opt-dir=$(brew --prefix gmp):$(brew --prefix jemalloc) --enable-yjit=dev

From here, the simplest way to run some code is to place a test.rb file in the root of the project and run it using make runruby. To run it in a debug mode, you can run make lldb-ruby.

Counting C method calls in CRuby

Thu, 28 Nov 2024 06:30:00 -0500

There is a central macro in CRuby, RUBY_VM_CHECK_INTS, which is a very hot path for the Ruby runtime. It’s an important part of how threads are managed, and it’s called constantly. I was curious just how often it was called, and it turned out CRuby comes with some handy debugging functionality for just this scenario.

Inside of debug_counter.h, I changed #define USE_DEBUG_COUNTER 0 to #define USE_DEBUG_COUNTER 1 and added this line later in that file:

RB_DEBUG_COUNTER(rb_vm_check_ints)

Then inside vm_core.h I updated RUBY_VM_CHECK_INTS to add a debug increment:

#define RUBY_VM_CHECK_INTS(ec) rb_vm_check_ints(ec)
static inline void
rb_vm_check_ints(rb_execution_context_t *ec)
{
    RB_DEBUG_COUNTER_INC(rb_vm_check_ints); // increment!

After that I ran the following simple Ruby program:

10_000.times {}

And this was printed after it ran:

[RUBY_DEBUG_COUNTER]    rb_vm_check_ints    21,055

Iterating a loop ten thousand times results in twenty thousand calls to RUBY_VM_CHECK_INTS, exactly what I was looking to measure!

I’d like to know the proper configuration to compile without having to manually modify USE_DEBUG_COUNTER in the header file. Maybe someone can comment and let me know how? It has something to do with CFLAGS, I think.

Update 12/5/24 Thanks to Mohit Sindhwani for some advice on how to add the CFLAGS!

My docker setup for hacking on CRuby

Wed, 27 Nov 2024 06:43:15 -0500

I run on MacOS, but I often want to test Linux behaviors when working on the CRuby implementation.

Here’s the Dockerfile I use:

FROM ubuntu:24.04

# Preventing dialog prompts when installing packages
ENV DEBIAN_FRONTEND=noninteractive

# Update and install basic build dependencies and Rust
RUN apt-get update && apt-get install -y \
    git \
    curl \
    build-essential \
    autoconf \
    libreadline-dev \
    libssl-dev \
    libyaml-dev \
    libncurses5-dev \
    zlib1g-dev \
    libffi-dev \
    bison \
    libgdbm-dev \
    libgdbm-compat-dev \
    libreadline6-dev \
    libssl-dev \
    libgmp-dev \
    liburing-dev \
    && apt-get clean && rm -rf /var/lib/apt/lists/*

# Install Rust via rustup
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y

RUN apt-get update && apt-get install -y ruby
RUN apt-get update && apt-get install -y gdb

# Add Rust to the PATH so the cargo and rustc commands are available
ENV PATH="/root/.cargo/bin:${PATH}"

# Create a directory for the Ruby source code
WORKDIR /usr/src/ruby

# Copy Ruby source code from your local directory
COPY . .

# This will be the default command when you run the container.
CMD [ "/bin/bash" ]

To run the Dockerfile, you can use the following two commands:

docker build -t ruby-source-build-env .
docker run -it --mount type=bind,src=.,target=/usr/src/ruby ruby-source-build-env

Based on our Dockerfile, docker run will open up a bash shell for you. From there, I run the following commands to build CRuby:

./autogen.sh
mkdir build && cd build
mkdir ./.rubies
../configure --prefix="/usr/src/ruby/build/.rubies/ruby-master" --disable-install-doc --config-cache --enable-debug-env optflags="-O0 -fno-omit-frame-pointer"
make install

We now have CRuby operating under Ubuntu Linux! From here, the simplest way to run some code is to place a test.rb file in the root of the project and run it using make runruby.

Calculating the largest known prime in Ruby

Tue, 26 Nov 2024 21:53:43 -0500

Looking to impress your Ruby friends by calculating the largest known prime, 2 ** 136_279_841-1?

On Ruby 3.4.0-preview2 and earlier, 2 ** 136_279_841-1 logs a warning and returns Infinity 😔:

2 ** 136_279_841-1
# warning: in a**b, b may be too big
# => Infinity

Thanks to @mametter, Ruby 3.4 will handle this calculation just fine! See Do not round a**b to infinity.

Knowing this, you excitedly use your ruby manager of choice to pull down ruby master:

rvm install ruby-head

You run ruby -e "puts 2 ** 136_279_841-1", and your excitement is slowly eroded. An hour into calculating, you terminate the command in frustration 😫.

Is @mametter a liar?!

As it turns out, there is critically important library you need for accelerating “Bignum” calculations: GMP, the GNU Multiple Precision Arithmetic Library. It’s even specifically mentioned in the CRuby guide to building ruby.

Without it, you can kiss your largest prime calculating dreams goodbye 👋.

You reinstall ruby head, making sure gmp is available

brew install gmp
rvm reinstall ruby-head --with-gmp-dir=$(brew --prefix gmp)

With a bit of hope in your heart, you try again:

ruby -e "puts 2 ** 136_279_841-1"

Success! @mametter was telling the truth!

Within around 5 seconds, your terminal is filled with a beautiful output of 41,024,320 digits. Your Ruby friends cheer and carry you off on their shoulders.

This was all inspired by Matz’s keynote at RubyConf 2024 - where he mentioned that Ruby 3.4 can now calculate the largest known prime. For fun, I tried it on my mac and just let it keep running - 2 hours later, it was still running! I’d never heard of GMP, but now I know!

The Thread API : Concurrent, colorless Ruby

Mon, 26 Aug 2024 17:59:00 -0500

👋🏼 This is part of series on concurrency, parallelism and asynchronous programming in Ruby. It’s a deep dive, so it’s divided into 12 main parts:

Your Ruby programs are always multi-threaded: Part 1

Your Ruby programs are always multi-threaded: Part 2

Consistent, request-local state

Ruby methods are colorless

The Thread API

Bitmasks, Ruby Threads and Interrupts, oh my!

When good threads go bad

Thread and its MaNy friends

Fibers

Processes, Ractors and alternative runtimes

Scaling concurrency with streaming

Abstracted, concurrent Ruby

Closing thoughts, kicking the tires and tangents

How I dive into CRuby concurrency

You’re reading “The Thread API: Concurrent, colorless Ruby”. I’ll update the links as each part is released, and include these links in each post.

The Thread API

The Thread API 🧵

We’re going to break down threads into three parts:

The Thread API - All the tools available to you in the Ruby runtime to manage threads, and how they work
Interrupting Threads - How threads get stuck, and how to shut them down safely
Thread and its MaNy friends - Thread architecture and the GVL

This post covers the Thread API. We’ll go over every method available to you, why they matter, how to call them, and often how popular open source projects use them.

Don’t let the cute mascot fool you

Before we start digging into threads, I’d like to make a small disclaimer: writing safe, deterministic, bug-free threaded code is hard.

I think understanding threads is valuable knowledge. After all, whether you explicitly use threads or not, your Ruby programs are always multi-threaded. If you’re always in the context of a thread, it’s helpful to know how they work behind the scenes. Even if the most you ever do is set a thread count on a server, it still helps to know how they work - it better informs what changing those numbers can and can’t do for you.

Thread.new do
  puts "seems easy enough? 🤷🏻‍♂️"
end

Behind the simplicity of the thread interface lives a lot of complexity. An OS thread gives you the literal ability to perform tasks in parallel. And once things can run in parallel, our sequential thinking starts to fail us. It’s difficult to correctly read through code step by step if at any point that step by step code can swap out with another separate piece of code. No warning, no ability to determine exactly where your program will switch to next.

It’s the same when multiple people work on some tasks in parallel - you encounter communication breakdown. There can be contention over a shared resource. You can undo someone else’s work, or leave them with inconsistent information that causes them to finish their task, but incorrectly.

There’s a cognitive bias known as the Dunning Kruger Effect. Essentially someone with limited experience in something can overestimate their abilities, or the complexity of the task. Thread.new {} feels pretty simple - it’s easy to underestimate what goes into it. Diving into threads together helps us realize they wield a lot of power!

There’s a reason that most threaded code in gems like Rails, SolidQueue and GoodJob use the concurrent-ruby gem. Abstractions are your friend. We’ll dig into abstractions later in the series - for now we’ll learn about threads directly. But dont stop here! Learn the foundation, and when you need it yourself, learn the abstractions and tools that make it easier.

source: Eli and JP Camara, https://x.com/logicalcomic

This post will dig into all the options available in Ruby out of the box. You’ll learn each thread method and what they’re for, and we’ll discuss how to coordinate your threads once you create them. Let’s go!

A thread api primer (with examples!)

We’ll start off with some details on how you interact with threads. Let’s create a thread!

Thread.new do
  sleep 10
  puts "finished!"
end

Run that code¹ 👆🏼… hmmm, nothing happens. Nothing prints and the program exits silently. What happened?

Everything starts off running in the “main thread”, which is accessible by calling Thread.main. When a new thread is created it’s like a branch off of the main thread. Those branches exist independently and the main thread doesn’t wait for them to finish by default.

Waiting for a thread to complete

`join`

To wait on a specific thread, you join that thread and the current thread together. Here we have the main thread join with t:

t = Thread.new do
  sleep 10
  puts "finished!"
end
# We're at the top level in `Thread.main`, which "joins" with `t` until it finishes
t.join
# finished!

When the thread finishes, the thread object is returned from join.

———

`value`

You can also join on a thread and ask it for the last value returned using value:

t = Thread.new do
  sleep 10
  File.read(a_file_path)
end
puts t.value # contents of `a_file_path`

The simplest way to run some work concurrently is to start up several threads and then iterate over each one calling join or value. We’ll make 4 http requests here, and the overall time will take as long as the longest request:

# frozen_string_literal: true
	
require "net/http"
require "json"
	
def generate_uuid_thread
  url = "https://httpbin.org/uuid"
  Thread.new do
    response = Net::HTTP.get(URI(url))
    JSON.parse(response)["uuid"]
  end
end
	
uuids = 4.times.map do
  generate_uuid_thread
end.map(&:value)
puts uuids
# 6b76167d-9bac-45dc-8d0b-5b7af865b843
# a025a125-51a5-4773-b78d-ab93fba02eb3
# 98befb95-adf7-4c89-9fd0-d10f6b2a3d7a
# 95f486fd-5bc6-46fe-b65a-c52a87140dfb

Make sure you let each thread start first! If you join too early, you end up running them sequentially:

uuids = []
uuids << generate_uuid_thread.value
uuids << generate_uuid_thread.value
uuids << generate_uuid_thread.value
uuids << generate_uuid_thread.value
puts uuids
# 6b76167d-9bac-45dc-8d0b-5b7af865b843
# a025a125-51a5-4773-b78d-ab93fba02eb3
# 98befb95-adf7-4c89-9fd0-d10f6b2a3d7a
# 95f486fd-5bc6-46fe-b65a-c52a87140dfb

The end result looks identical, and the output is but the execution is totally different. When we create all the threads up front, certain operations can run in parallel:

Threads 1 through 4 get created and are “runnable”
The main thread blocks on the first iteration to call map(&:join)
As each thread blocks on the HTTP call, they run in parallel
The loop takes as long as the longest thread, so if it takes us 50ms for 3 requests, and the 4th is 100ms, we spend around 100ms total

Calling generate_uuid_thread.value one at a time, we’re just running the code sequentially:

Thread 1 gets created and run, returning its value
Thread 2 gets created and run, returning its value
Same for threads 3 and 4
We’re running sequentially, so the threads provided no value. It would take roughly 250ms. If anything, the threads likely added overhead.

⚠️ Never actually generate a uuid using a web service, that’s crazy 🤣

———

`join(timeout_in_seconds)`

You can also limit the amount of time you join. If you’re calling join, typically you just want to wait however long it takes. But using join(seconds) could be useful to periodically pop into some other work or alert that you’ve been running too long. join(seconds) will return the thread while it is still running, and return nil once it finishes.

t = Thread.new do
  sleep 10
end
t2 = Thread.new do
  sleep 20
end
	
while t2.alive?
  puts "wait a bit more... #{t2.join(5)}"
end
puts "done!"
# wait a bit more... 
# wait a bit more... 
# wait a bit more... 
# wait a bit more... #<Thread:0x0... main.rb:116 dead>
# done!
	
def fib(n)
  return n if n <= 1
  fib(n - 1) + fib(n - 2)
end
	
t2 = Thread.new do
  fib(40)
end
while t2.alive?
  puts "wait a bit more... #{t2.join(0.01)}"
end
puts "done! #{t2.value}" # get returned even after a join
# wait a bit more... 
# wait a bit more... 
# wait a bit more... 
# wait a bit more... 
# wait a bit more... 
# done! 9227465

You could use it to intermittently check for a thread finishing without wasting too many CPU cycles:

t = Thread.new do
  sleep 5
end
while t.join(1)
  puts "still waiting..."
end
puts "done!"
# still waiting...
# still waiting...
# still waiting...
# still waiting...
# done!

In the Puma web server, it uses join(seconds) to manage shutting down its thread pool. It iterates over each thread, adjusting the join timeout based on how much time has elapsed since starting the method:

join = ->(inner_timeout) do
  start = Process.clock_gettime(
    Process::CLOCK_MONOTONIC
  )
  threads.reject! do |t|
    elapsed = Process.clock_gettime(
      Process::CLOCK_MONOTONIC
    ) - start
    t.join inner_timeout - elapsed
  end
end
	
# Wait +timeout+ seconds for threads to finish.
join.call(timeout)

It uses join(timeout) to try and let each thread finish before forcing a shutdown with raise or kill. The reject! removes any threads that finish during that time - nil returned from join means the thread is still running, otherwise the thread object is returned (removing it from the array).

📝 A “thread pool” is a reusable, typically fixed size set of threads. Rather than create brand new threads every time they are needed, a thread pool saves on thread creation by reusing threads to perform operations. They are generally used to save on thread creation cost and limit the number of threads running at a time. concurrent-ruby comes with several types of thread pools.

Error Reporting

`report_on_exception` and `abort_on_exception`

What happens if something fails in your thread?

t = Thread.new do
  raise "oops!"
end
sleep # does the program sleep forever, or raise an error?

Running that 👆🏼, it just runs forever. Even though our thread has failed, it never impacts the program. We will see an error printed in our console, however:

terminated with exception (report_on_exception is true):
main.rb:2:in `block in <main>`: oops! (RuntimeError)

You can control whether it fails silently, but the default is to report the error. Keep that default - silent errors are not your friend. But if you really needed to, you can change it globally or with a per thread setting:

# No threads log their errors ☠️ 
Thread.report_on_exception = false
	
# This individual thread does not log
t.report_on_exception = true

What if you want to report it in the current thread? Similar to joining execution with a thread to wait for it to finish, you need to join with it to raise its error:

t = Thread.new do
  raise "oops!"
end
t.join # or .value, raises "oops!"

You can also force the thread to raise, even when running independently:

t = Thread.new do
  sleep 5
  raise "listen to me!"
end
t.abort_on_exception = true
sleep 10 # t blows up the program!

This can be set globally as well:

Thread.abort_on_exception = true
t = Thread.new do
  sleep 5
  raise "listen to me!"
end
sleep 10 # t blows up the program!

Rack timeout uses it in its Scheduler thread, applying it using Thread.current:

def run_loop!
  Thread.current.abort_on_exception = true

In general, you don’t see this much in a production setting. But it could be useful if you’re writing a one-off script and you want any thread that fails to kill the program.

Tracking threads

Can we find out what threads are running?

`Thread.list`

Thread.list will give you every thread that hasn’t already finished:

Thread.new { sleep 0.1 }
Thread.new { sleep 0.1 }
Thread.new { sleep 0.1 }
Thread.new {}
puts Thread.list
	
#<Thread:0x0... run>
#<Thread:0x0... main.rb:1 sleep>
#<Thread:0x0... main.rb:2 sleep>
#<Thread:0x0... main.rb:3 sleep>
#<Thread:0x0... main.rb:4 dead>

`name`

Ok… but how can you differentiate them? There’s a few ways you can achieve that:

File information
Setting a name
Inheritance

File information gets printed by default, and shows you the name of the file and what line the thread was started on:

t = Thread.new { sleep 0.1 }
t2 = Thread.new { sleep 0.1 }
t3 = Thread.new { sleep 0.1 }
t4 = Thread.new { sleep 0.1 }
puts Thread.list
	
#<Thread:0x0...main.rb:1 sleep>
#<Thread:0x0...main.rb:2 sleep>
#<Thread:0x0...main.rb:3 sleep>
#<Thread:0x0...main.rb:4 sleep>

The file info is useful, but often your threads are all started from the same place, so it doesn’t tell you much:

def start_thread
  Thread.new { sleep 0.1 }
end
t = start_thread
t2 = start_thread
t3 = start_thread
t4 = start_thread
puts Thread.list
	
#<Thread:0x0...main.rb:2 sleep>
#<Thread:0x0...main.rb:2 sleep>
#<Thread:0x0...main.rb:2 sleep>
#<Thread:0x0...main.rb:2 sleep>

In that case, you can set a name on each thread to differentiate them:

Thread.main.name = "main"
t.name = "first!"
t2.name = "second!"
t3.name = "third!"
t4.name = "fourth!"
puts Thread.list
	
#<Thread:0x0...@first!  main.rb:2 sleep>
#<Thread:0x0...@second! main.rb:2 sleep>
#<Thread:0x0...@third!  main.rb:2 sleep>
#<Thread:0x0...@fourth! main.rb:2 sleep>

The concurrent-ruby gem uses this to help differentiate threads created in its ThreadPoolExecutor:

# concurrent/executor/ruby_thread_pool_executor.rb
[@thread.name](http://thread.name) = [pool.name, 'worker', id].compact.join('-')
#...
	
require "concurrent"
	
pool = Concurrent::ThreadPoolExecutor.new(
  name: "🤖"
)
pool.post { sleep 0.1 }
pool.post { sleep 0.1 }
pool.post { sleep 0.1 }
pool.post { sleep 0.1 }
puts Thread.list
#<Thread:0x0...@🤖-worker-1...concurrent/executor/ruby_thread_pool_executor.rb:339 sleep_forever>
#<Thread:0x0...@🤖-worker-2...concurrent/executor/ruby_thread_pool_executor.rb:339 sleep_forever>
#<Thread:0x0...@🤖-worker-3...concurrent/executor/ruby_thread_pool_executor.rb:339 sleep_forever>
#<Thread:0x0...@🤖-worker-4...concurrent/executor/ruby_thread_pool_executor.rb:339 sleep_forever>

The Honeybadger gem runs a background thread when sending errors to their error reporting service. To differentiate their thread, they use inheritance. The information about each thread includes the class so it makes it easy to identify:

module Honeybadger
  class Worker
    class Thread < ::Thread; end
  end
end
	
Honeybadger::Worker::Thread.new { sleep 0.1 }
puts Thread.list
#<Thread:0x0... run>
#<Honeybadger::Worker::Thread:0x0... main.rb:119 sleep>

———

`status`, `alive?` and `stop?`

Can you keep track of the status of a running thread? Yep! Threads operate in one of 5 states - they’re not the most intuitive, but they’re what you’ve got:

“run” - the thread is running
“sleep” - the thread is “sleeping”. There is some blocking operation going on or the thread went to sleep or was put to sleep by the thread scheduler
“aborting” - the thread has failed but hasn’t finished running yet
nil - an error was raised and the thread is dead
false - the thread finished normally

Let’s demonstrate some statuses:

a = Thread.new { raise("bye bye") }
b = Thread.new { Thread.stop }
c = Thread.new {}
d = Thread.new { 
  IO.select(nil, nil, nil, 3)
}
d.join(1) # wait on d for 1 second
puts a.status.class          #=> NilClass
puts b.status                #=> "sleep"
puts c.status                #=> false
puts d.status                #=> "sleep"
puts Thread.current.status   #=> "run"

You’ll notice a few things about the above:

I didn’t show “aborting”. That’s because I’m not sure regular code could ever see that status. Catching a failed thread that hasn’t finished yet seems pretty hard to do. Internally the CRuby thread needs to be in a to_kill state. I would love it if someone knows a way to demonstrate it, though! It would perhaps require the assistance of a core CRuby wizard 🧙‍♀️.
“sleep” is not specific to the sleep method. In thread d we are making an IO.select call that takes three seconds. So the thread blocks while waiting, hence it is “sleep”ing.
Since you can’t run Ruby code in multiple threads in parallel, you pretty much would only ever see “run” on the current, active thread.

It would be nice if the information was a bit more readable. We can put together a little helper to make the output clearer. We’ll also add in one more internal status not directly exposed by status:

ThreadStatus = Data.define(
  :status, :error
)
	
def thread_status(thread)
  error = nil
  status = case thread.status
    when NilClass
      error = begin
        thread.join
      rescue => e
        e
      end
      "failed w/ error: #{error}"
    when FalseClass then "finished"
    when "run" then "running"
    when "sleep"
      parse_thread_sleep_status(thread)
  else thread.status
  end
  ThreadStatus.new(status:, error:)
end
	
def parse_thread_sleep_status(thread)
  status = thread.to_s
  status[status.index("sleep")..-2].sub(
    "sleep", "sleeping"
  )
end
	
# our previous thread code... then...
puts thread_status(a)
puts thread_status(b)
puts thread_status(c)
puts thread_status(d)
puts thread_status(Thread.current)
	
#<ThreadStatus status="failed w/ error: bye bye", error=#<RuntimeError: bye bye>
#<ThreadStatus status="sleeping_forever", error=nil>
#<ThreadStatus status="finished", error=nil>
#<ThreadStatus status="sleeping", error=nil>
#<ThreadStatus status="running", error=nil>

Using our new helper, we get a bit more readability and depth into our thread statuses.

Using join we are able to return more information about the failed thread - “failed” instead of nil, and the actual error it failed with
We return “finished” instead of false for a successful finish
sleep_forever let’s us differentiate actively blocked threads (like one doing IO.select) from a thread that is actually stopped, and won’t run again without intervention. We’ll talk more about Thread.stop in the next section

In addition to status, we can also use alive? and stop? to check on a threads status:

t = Thread.new {}
t2 = Thread.new { loop {} }
t3 = Thread.new { Thread.stop }
t.name = "quick"
t2.name = "slow"
t3.name = "on ice"
[t, t2, t3].each do |thread|
  # make sure it gets a chance to run
  thread.join(0.01)
  puts "#{thread.name}: " \
    "alive? #{thread.alive?}, " \
    "stopped? #{thread.stop?}"
end
# quick:  alive? false, stopped? true
# slow:   alive? true,  stopped? false
# on ice: alive? true,  stopped? true

Thread Scheduling

There are several methods for either taking direct action, or suggesting action to the thread scheduler. Before using them, keep in mind that you’re probably not smarter than the Ruby thread scheduler. It tries to do what makes the most sense for the runtime, and it’s been tuned extensively. But these tools exist, and they get used, so let’s discuss them a bit.

`Thread.pass`, `wakeup`, `Thread.stop`, `run`, and `priority`

pass and wakeup are kind of like nudges to the runtime. They request a particular action, but the scheduler does not have to honor them. Thread.pass tells the thread scheduler it can “pass” control to another thread:

t = Thread.new do
  Thread.pass
  puts "hi!"
end
t2 = Thread.new do
  puts "bye!"
end
t.join
t2.join
# Most of the time you will see:
# bye!
# hi!
# but it's not guaranteed

wakeup marks a thread as eligible for scheduling. It’s up to the thread scheduler whether that happens:

t = Thread.new do
  Thread.stop
  puts "hi!"
end
t.join(1)
t.wakeup
t.join
# hi!

Thread.stop and run are more direct commands to the thread scheduler:

t = Thread.new do
  now = Time.now
  sleep 10
  puts "Seconds slept: #{Time.now - now}"
end
t.join(1)
t.run
t.join
# Seconds slept: 1.000076481

Only one second has passed, but run caused the sleep to finish early.

t = Thread.new do
  Thread.stop
  puts "done!"
end
t.join(0.1)
puts thread_status(t)
puts "alive? #{t.alive?}"
puts "stopped? #{t.stop?}"
t.run
t.join
#<ThreadStatus status="sleeping_forever">
# alive? true
# stopped? true
# done!

priority gives a hint to the scheduler of which thread should be given more runtime. The thread docs have a good example of this:

count1 = count2 = 0
a = Thread.new do
      loop { count1 += 1 }
    end
a.priority = -1
	
b = Thread.new do
      loop { count2 += 1 }
    end
b.priority = -2
sleep 1
puts count1 #=> 21472634
puts count2 #=> 14256235

The threads run forever, but thread a gets higher priority so it adds to the counter more often.

For an open source example of run, you can check the rack-timeout scheduler:

def schedule(event)
  @mx_events.synchronize { 
    @events << event 
  }
  runner.run  # wakes up the runner thread so it can recalculate sleep length taking this new event into consideration
  return event
end

Thread.pass is a recommendation from Mike Perham if you have your jobs hogging CPU:

class ExpensiveJob
  include Sidekiq::Job
	
  def perform
    loop do
      # expensive stuff
      Thread.pass if occasional_condition
    end
  end
end

Thread Shutdown

`raise` and `kill`

You want to kill… me? 🥺

⚠️ TL;DR You shouldn’t use these methods unless you really know what you’re doing. Instead, interrupt your thread safely. Incidentally, you should also avoid the timeout module. We’ll dig deep into raise and kill in the next post on “Interrupting Threads”.

Interrupt your thread safely

Instead of killing your thread, set it up to be interruptible. Most mature, threaded frameworks operate this way.

still_kickin = Concurrent::AtomicBoolean.new(true)
Thread.new do
  while still_kickin.true?
    # more work!
  end
end
	
still_kickin.make_false

Don’t use timeout

If you see this in code, be concerned:

require "timeout"
	
Timeout.timeout(1) do
  # 😱 
end

For some reason, the timeout gem itself doesn’t warn about any issues. But Mike Perham summarizes it best:

There’s nothing that exactly matches what timeout offers: a blanket way of timing out any operation after the specified time limit. But instead of using the timeout gem, there is a repository called The Ultimate Guide to Ruby Timeouts. It shows you how to set timeouts safely for basically every blocking operation you could care about timing out. For instance, how to properly handle timeouts using the redis gem:

Redis.new(
  connect_timeout: 1, 
  timeout: 1,
  #...
)

———

`Thread.handle_interrupt`

A thread can be externally “interrupted” by a few things:

Thread#kill
Thread#raise
Your program being exited
A signal, like Ctrl+C

handle_interrupt gives you the ability to control how your program reacts to 1-3².

Because it primarily matters in the context of raise and kill, we’ll discuss it in the next post on “Interrupting Threads”.

———

`Process._fork`

Process#_fork isn’t a thread api, but it’s good be aware of for your threaded code.

What’s happens to a thread when a process forks?

t = Thread.new { sleep }
fork do
  puts "inside fork: #{thread_status(t)}"
end
puts "outside fork: #{thread_status(t)}"
# outside fork: #<ThreadStatus status="running">,  pid: 362
# inside fork:  #<ThreadStatus status="finished">, pid: 367

You can’t bring your threads with you when you fork. But you can recreate them, using _fork!

module OnFork
  def _fork
    pid = super
    if pid == 0
      # your code to restart threads
    end
    pid
  end
end
	
Process.singleton_class.prepend(OnFork)

It’s a little strange of a setup. You’re hooking into the inheritance chain for Process._fork, so you need to call super directly. No one calls _fork directly, super ultimately returns the result of fork itself. If the result is 0, we’re in the forked process, which means we can perform any kind of post-fork action. In the case of managing threads, that would involve recreating them.

The connection_pool gem uses this to run an after_fork method. It uses it to close out connections.

module ForkTracker
  def _fork
    pid = super
    if pid == 0
      ConnectionPool.after_fork
    end
    pid
  end
end
	
Process.singleton_class.prepend(
  ForkTracker
)

The redis-client gem uses _fork to track the pid in PIDCache, so it can determine whether it needs to close the inherited socket (threads are not inherited when forking, but file descriptors are).

def ensure_connected(retryable: true)
  close if !config.inherit_socket && @pid != PIDCache.pid

Coordinating Threads

Now that we know the different methods of interacting with a thread directly, how can we coordinate threads together safely?

📝 If you can avoid it, don’t coordinate at all! Immutable structures, or isolated work are your friends.

`Mutex`

Mutex is the core thread coordination primitive in Ruby. It stands for mutual exclusion, and it allows you to control single thread access to a particular resource. A thread or fiber acquires a lock on the mutex, and it is the only thing that can unlock that mutex.

📝 If you know database locks, a Mutex basically operates like an exclusive lock.

📝 You’re better off not sharing objects, it keeps things simpler. I’ll reference my own note from “Your Ruby programs are always multi-threaded: Part 2”:

My personal metric is that the right amount of mutexes in my code is zero. If I am using a mutex, I think hard to figure out a way to avoid it because it means I’m opening up myself and future devs to a lot of cognitive overhead: you need to think critically anytime you make a change relating to mutex code.

If you’re a library or framework author they may be unavoidable at some point to do interesting or useful things. In my own application code, I can pretty much always avoid them.

Earlier we used a class from concurrent-ruby called AtomicBoolean to implement an interruptible thread. What if we wanted to implement it ourselves?

class AtomicBoolean
  def initialize(default = false)
    @value = default
    @mutex = Mutex.new
  end
	
  def true?
    @mutex.synchronize { value == true }
  end
	
  def false?
    @mutex.synchronize { value == false }
  end
	
  def make_true
    @mutex.synchronize { @value = true } 
  end
	
  def make_false
    @mutex.synchronize { @value = false }
  end
end

To make sure our data stays consistent, we synchronize every access. That way we know we can’t corrupt anything and we have a consistent view of ``@value```:

still_kickin = AtomicBoolean.new(true)
Thread.new do
  while still_kickin.true?
    # more work!
  end
end
	
still_kickin.make_false

It doesn’t make sure we do things in any expected order, but it’s a foolproof way of having proper access and visibility.

📝 Truthfully, CRuby doesn’t need this kind of corruption guarantee. But true parallel Ruby runtimes like JRuby and TruffleRuby do.

synchronize is the probably the only method you’ll find yourself using on a Mutex. Look in different projects and that’s 95% of all Mutex usage. But there are more methods available you may see on occasion:

lock
unlock
try_lock
owned?
locked?
sleep

lock and unlock can be used to recreate what synchronize does:

def synchronize(mutex)
  mutex.lock
  yield
ensure
  mutex.unlock
end
	
synchronize(mutex) do
  # locked work
end

try_lock lets you attempt a lock without blocking. When you call lock or synchronize, your code will block until you are able to acquire a lock:

mutex = Mutex.new
t = Thread.new do
  mutex.lock
  loop {} # runs forever, never releasing
ensure
  mutex.unlock
end
t2 = Thread.new do
  mutex.synchronize do
    # do some work
  end
end
t.join(1)
t2.join # t never releases the lock, so t2 runs forever

But try_lock will just return a boolean if the lock worked:

mutex = Mutex.new
t = Thread.new { mutex.lock; loop {} }
t2 = Thread.new do
  if mutex.try_lock
    # do some work
  else
    raise "Couldn't acquire the lock!"
  end
ensure
  mutex.unlock if mutex.owned?
end
t.join(1)
t2.join # tries to acquire the lock, and raises an error because it can't

Notice the owned? call in the ensure? We don’t know if the lock was successfully acquired, so we only call unlock if the lock is owned?. Being owned? means the current thread successfully acquired the lock, and is the current “owner”. If you tried to call unlock on a thread that wasn’t the owner, an error would be raised.

locked? allows you to check if the lock is owned by some thread. Seems susceptible to some race conditions, but you could use it to determine if you need to perform an action. The aws-sdk-core uses it to determine whether to create a thread for an “async refresh”. If the thread has already started and is refreshing, locked? will be true and no thread will be created:

unless @mutex.locked?
  Thread.new do
    @mutex.synchronize do
      # refresh async
    end
  end
end

Last we have sleep(timeout = nil), which releases the lock for timeout seconds, or runs forever if given nil. This is exactly what the rack-timeout gem uses internally to create a Scheduler class which it uses to schedule request timeouts:

def initialize
  @mx_events = Mutex.new
  # ...
end
	
def run_loop!
  loop do # begin event reader loop
    @mx_events.synchronize {
      @events.reject!(&:cancelled?)
      sleep_for = [@events.map](http://events.map)(&:monotime).min
      @mx_events.sleep sleep_for

It acquires a lock using synchronize to safely operate on the @events array. It then finds the event with the shortest wait time, and sleeps the mutex for that period. That way other events can be added to the @events array using the appropriate lock, even while waiting. This supports the scheduler interface:

require "rack-timeout"
	
Scheduler = Rack::Timeout::Scheduler
Scheduler.run_in(5) { puts "I did a thing last!" }
Scheduler.run_in(3) { puts "whoop whoop, i'm second" }
Scheduler.run_in(1) { puts "yowza, i'm first" }
# yowza, i'm first
# whoop whoop, i'm second
# I did a thing last!

When you call run_in, a new event is appended to the @events array. run is called on the thread with the sleeping mutex , which causes @mx_events.sleep to wakeup and run_loop! to iterate again, checking for any events to fire and scheduling the shortest even duration to wait again using @mx_events.sleep.

def schedule(event)
  @mx_events.synchronize { @events << event }
  runner.run  # wakes up the runner thread so it can recalculate sleep length taking this new event into consideration
  return event
end

Similar to the principle of a database lock, the shorter you can keep the mutex lock the better for performance.

`ConditionVariable`

Similar to Mutex#sleep, a ConditionVariables purpose is to let you release a lock and sleep - you do that using the wait method. The difference is that it provides a direct communication mechanism to wake up: signal and broadcast. Let’s look at a small example - in it we won’t see the wait re-acquire the lock until we call signal:

mutex = Mutex.new
condition = ConditionVariable.new
t = Thread.new do
  mutex.synchronize do
    puts "hi!"
    condition.wait(mutex)
    puts "bye!"
  end
end
t.join(5)
puts "how are you?"
t.join(1)
puts "still waiting?"
condition.signal
t.join
# hi!
# how are you?
# still waiting?
# bye!

signal will only notify a single thread, whereas broadcast will notify all threads. Let’s create two threads and try it signal instead:

def waiter(mutex, condition)
  Thread.new do
    mutex.synchronize do
      puts "hi!"
      condition.wait(mutex)
      puts "bye!"
    end
  end
end
	
mutex = Mutex.new
condition = ConditionVariable.new
t = waiter(mutex, condition)
t2 = waiter(mutex, condition)
	
t.join(5)
puts "how are you?"
t2.join(1)
puts "still waiting?"
	
condition.signal
sleep 1
# hi!
# hi!
# how are you?
# still waiting?
# bye!

We never see the second thread say “bye!”, because only a single signal call has been made. A signal call attempts to wake up a single thread. If you try to join the second thread, Ruby will detect a deadlock condition because the wait will never finish:

condition.signal
t.join
t2.join
# hi!
# hi!
# how are you?
# still waiting?
# bye!
# main.rb:in `join': No live threads left. Deadlock? (fatal)
# 2 threads, 2 sleeps current:0x0000000001df10f0 main thread:0x0000000001df10f0
# * #<Thread:0x00007f9373bba9c8 sleep_forever>
#    rb_thread_t:0x0000000001df10f0 native:0x00007f938d474300 int:0   
# * #<Thread:0x00007f9371c124b8 main.rb:112 sleep_forever>
#    rb_thread_t:0x00000000026ac400 native:0x00007f9371ace6c0 int:0
#     depended by: tb_thread_id:0x0000000001df10f0

There are probably cases where signal makes sense - only one thread can acquire the lock so it’s cheaper to wake up a single thread than to wake up every thread. But broadcast covers more scenarios, and fixes our two thread example:

condition.broadcast
t.join
t2.join
# hi!
# hi!
# how are you?
# still waiting?
# bye!
# bye!

A ConditionVariable can only wait on a locked mutex:

mutex = Mutex.new
condition = ConditionVariable.new
t = Thread.new do
  condition.wait(mutex)
end
sleep 1
puts thread_status(t)
#<ThreadStatus status="failed w/ error: Attempt to lock a mutex which is unlocked", error=#<ThreadError:...>

And only for the thread that owns the mutex:

mutex = Mutex.new
condition = ConditionVariable.new
t = nil
mutex.synchronize do
  t = Thread.new do
    condition.wait(mutex)
  end
  sleep 1
end
puts thread_status(t)
#<ThreadStatus status="failed w/ error: Attempt to unlock a mutex which is locked by another thread/fiber", error=#<ThreadError:...>

We can use ConditionVariable#wait with a timeout in seconds, so we can also recreate the Mutex#sleep Scheduler code from rack-timeout (in a more basic form):

class Scheduler
  Schedule = Data.define(:block, :time)
	
  def initialize
    @mutex = Mutex.new
    @cond = ConditionVariable.new
    @schedules = []
    start
  end
	
  def start
    Thread.new do
      loop do
        @mutex.synchronize do
          schedule = @schedules.min_by { |s| s.time }
          if schedule
            now = Time.now
            sleep_duration = schedule.time - now
	
            if sleep_duration > 0
              @cond.wait(@mutex, sleep_duration)
            end
	
            if Time.now >= schedule.time
              schedule.block.call
              @schedules.delete(schedule)
            end
          else
            @cond.wait(@mutex)
          end
        end
      end
    end
  end
	
  def schedule(seconds, &block)
    @mutex.synchronize do
      target_time = Time.now + seconds
      @schedules << Schedule.new(block:, time: target_time)
      @cond.signal
    end
  end
end
	
s = Scheduler.new
puts Time.now
s.schedule(1) { puts "1! #{Time.now}" }
s.schedule(3) { puts "2! #{Time.now}" }
s.schedule(5) { puts "3! #{Time.now}" }
sleep
# 2024-08-26 21:58:23 +0000
# 1! 2024-08-26 21:58:24 +0000
# 3! 2024-08-26 21:58:26 +0000
# 5! 2024-08-26 21:58:28 +0000

In Your Ruby programs are always multi-threaded: part 2, we looked at an example of coordinating threads using a “CountdownLatch”.

class CountdownLatch
  def initialize(count)
    @count = count
    @mutex = Mutex.new
    @cond = ConditionVariable.new
  end
	
  def wait
    @mutex.synchronize do
      @cond.wait(@mutex) while @count > 0
    end
  end
	
  def count
    @mutex.synchronize { @count }
  end
	
  def count_down
    @mutex.synchronize do
      @count -= 1
      if @count == 0
        @cond.broadcast
      end
      @count
    end
  end
end

📝 Quick reminder that concurrent-ruby comes with a countdown latch so there’s no need to use this one ☝🏼. It’s just educational. It is very similar to the concurrent-ruby version though!

We’ve seen how wait and broadcast work. But why do we need that while @count > 0 check? Shouldn’t it only get woken up when @count == 0 and @cond.broadcast is called? Unfortunately, Mutex#sleep and ConditionVariable#wait can wake up randomly due to something called Spurious wakeups³.

Thread runtimes can decide for internal reasons to wake up a condition at random, so you should be ready to handle it - in our case we continually check the expected condition @count > 0 and continue to wait until it is false. This makes sure if we wake up due to a spurious wakeup we’ll immediately wait for the condition again.

`Monitor`

A Monitor is essentially the same as a Mutex, but it is also “re-entrant”. What does it mean to be re-entrant? Let’s go back to an example from Your Ruby programs are multi-threaded: part 2:

require "monitor"
	
class Result
  attr_accessor :value
end
	
class Fibonacci
  @fib_monitor = Monitor.new
	
  class << self
    def result=(value)
      @fib_monitor.synchronize { @result = value }
    end
	
    def result
      @fib_monitor.synchronize { @result }
    end
	
    def calculate(n)
      @fib_monitor.synchronize do
        self.result = Result.new
        result.value = fib(n)
      end
    end
	
    def fib(n)
      return n if n <= 1
	
      fib(n - 1) + fib(n - 2)
    end
  end
end

In it, when we call #calculate, internally it calls #result and #result=. calculate first acquires the lock using synchronize, then it calls result= which also tries to acquire the lock. It is re-entering the same lock. Let’s change @fib_monitor to a Mutex and see what happens:

class Fibonacci
  @fib_monitor = Mutex.new
end
	
Fibonacci.calculate(10)
# `synchronize': deadlock; recursive locking (ThreadError)

We immediately see an error raised: “deadlock; recursive locking”. By changing to a Monitor, everything works fine.

The redis-rb gem creates clients that are thread-safe. It uses a Monitor to do that, likely because it allows synchronized methods to call other synchronized methods without any recursive deadlocking errors:

class Redis
  def initialize(options = {})
    @monitor = Monitor.new
    # ...
    inherit_socket = @options.delete(:inherit_socket)
	
    @client = initialize_client(@options)
    @client.inherit_socket! if inherit_socket
  end
	
  def synchronize
    @monitor.synchronize { yield(@client) }
  end
	
  def send_command(command, &block)
    @monitor.synchronize do
      @client.call_v(command, &block)
    end
  rescue ::RedisClient::Error => error
    Client.translate_error!(error)
  end
	
  # lib/redis/commands/transactions.rb
  def multi
    synchronize do |client|
      client.multi do |raw_transaction|
        yield MultiConnection.new(raw_transaction)
      end
    end
  end

Monitor comes with some other conveniences around creating ConditionVariables related to it, which you can read more about in its documentation.

`Queue`

Queue is one of two thread-safe data structures that come out of the box with Ruby. It is a First-in, First-out queue which allows for safe communication between threads. It’s primarily used for implementing producer/consumer patterns between threads:

queue = Queue.new
producer = Thread.new do
  i = 0
  loop do
    queue << i
    i += 1
    sleep 1
  end
end
	
def create_consumer(name, queue)
  Thread.new do
    loop do
      item = queue.pop
      puts "#{name} got another item #{item} at #{Time.now}"
    end
  end
end
	
create_consumer("Consumer 1", queue)
create_consumer("Consumer 2", queue)
	
producer.join
# Consumer 1 got another item 0 at 2024-08-24 20:58:46 +0000
# Consumer 2 got another item 1 at 2024-08-24 20:58:47 +0000
# Consumer 1 got another item 2 at 2024-08-24 20:58:48 +0000
# Consumer 2 got another item 3 at 2024-08-24 20:58:49 +0000
# Consumer 1 got another item 4 at 2024-08-24 20:58:50 +0000
# Consumer 2 got another item 5 at 2024-08-24 20:58:51 +0000
# Consumer 1 got another item 6 at 2024-08-24 20:58:52 +0000
# Consumer 2 got another item 7 at 2024-08-24 20:58:53 +0000
# Consumer 1 got another item 8 at 2024-08-24 20:58:54 +0000
# Consumer 2 got another item 9 at 2024-08-24 20:58:55 +0000

The producer thread endlessly adds an item to the queue every 1 second, and two consumer threads pop items off the queue. If there is no item available, the thread sleeps until one becomes available.

If it seems like you could pretty easily implement this using a Mutex and a ConditionVariable, you’d be right! I’ll leave that for you to try as an example.

`SizedQueue`

SizedQueue is the other thread-safe data structure available in Ruby, and it’s just another flavor of the base Queue class. It allows you to create a fixed-size queue - when new items are added the queue blocks until space is available.

fixed_queue = SizedQueue.new(3)
fixed_queue << 1
fixed_queue << 2
fixed_queue << 3
fixed_queue << 4 # raises "No live threads left. Deadlock?"

This is a good use-case for throttling your own code to not overwhelm your application. The following code will block if more than 5 items exist in the queue, throttling the producers so consumers can keep up with it:

throttle = SizedQueue.new(5)
	
# Producers
Thread.new do
  i = 0
  loop do
    i += 1
    if i.even?
      throttle << "Producer 1: #{i}"
      Thread.pass
    end
  end
end
Thread.new do
  i = 0
  loop do
    i += 1
    if i.odd?
      throttle << "Producer 2: #{i}"
      Thread.pass
    end
  end
end
	
# Consumers
Thread.new do
  loop do
    puts "Thread 1: #{throttle.pop}"
    sleep 1
  end
end
Thread.new do
  loop do
    puts "Thread 2: #{throttle.pop}"
    sleep 2
  end
end
	
loop do
  puts "Threads waiting: #{throttle.num_waiting}"
  sleep 0.5
end
	
# Threads waiting: 0
# Thread 1: Producer 1: 2
# Thread 2: Producer 2: 1
# Threads waiting: 2
# Thread 1: Producer 1: 4
# Threads waiting: 2
# Threads waiting: 1
# Thread 2: Producer 2: 3
# Thread 1: Producer 1: 6
# Threads waiting: 1
# Threads waiting: 2
# Thread 1: Producer 2: 5
# Threads waiting: 1
# Threads waiting: 2
# Thread 2: Producer 1: 8
# Thread 1: Producer 2: 7

You’ll see threads waiting going between 1 and 2, as producers get blocked while consumers slowly consume data from the SizedQueue.

`ThreadGroup`

The Ruby docs describe ThreadGroup as “a means of keeping track of a number of threads as a group.” A thread is automatically a part of the “default” group:

t = Thread.new { loop {} }
t.group == ThreadGroup::Default is # true

And you can add threads to a new group:

t = Thread.new { loop {} }
t2 = Thread.new { sleep }
group = ThreadGroup.new
group.add(t)
group.add(t2)
puts group.list
#<Thread:0x0...main.rb:1 run>
#<Thread:0x0...main.rb:2 sleep>

I’m mentioning them here for completeness, but you almost never see them in real use. See their documentation to learn more.

——

That’s about it! Out of the box, Ruby comes with a pretty small set of Thread primitives. Most code you might see in real use will use either these, or options from the concurrent-ruby gem. We’ll dig more into concurrent Ruby in “Abstracted, concurrent Ruby” later on in the series.

Memory visibility

The last thing we’ll discuss before finishing up is a concept called “memory visibility”.

The simplest way to think of memory visibility is “what can each thread can see at any given time”. When threads are running on multiple CPUs, a common optimization is to localize things to the most performant memory caches located on the CPU itself. If that happens, it means two different threads can operate on a shared piece of data, and have completely different views of that data because they have localized, out of sync versions of it.

In addition, the CPU can actually reorder certain operations to optimize them.

How can you solve these problems? When you need to make sure each thread sees a consistent, accurate version of a shared piece of data, you utilize something called a memory barrier. How can you use a memory barrier from Ruby? A mutex! As long as you wrap each access to a particular shared resource, you’ll be guaranteed to see a consistent view of it:

class AtomicBoolean
  def initialize(default = false)
    @value = default
    @mutex = Mutex.new
  end
	
  def true?
    @mutex.synchronize { value == true }
  end
	
  def false?
    @mutex.synchronize { value == false }
  end
	
  def make_true
    @mutex.synchronize { @value = true } 
  end
	
  def make_false
    @mutex.synchronize { @value = false }
  end
end

Isolated to a single thread, or for immutable objects, memory visibility doesn’t really matter - it’s another reason attempting to share objects between threads can lead to issues, and avoiding it is better than trying to safely coordinate it.

📝 In CRuby, memory visibility is unlikely to ever be an issue for you. That’s because there is always a mutex involved when moving between threads: The GVL. We’ll talk more about the GVL in “Thread and its MaNy friends”. But to be on the cautious side, you’re better off synchronizing access consistently for reads/write to any data shared between threads.

———

We’ve now dug into the majority of the Thread API. The main piece we’ve only touched lightly, is around interrupting threads. It’s up next in “Interrupting Threads: Colorless, concurrent Ruby”. More soon! 👋🏼

There are a couple other interfacing for creating a thread, but you basically never see them in use.

a = 1

b = 2

Thread.new(a, b) { |a, b| puts a, b }

👆 this one has been described in some places as making copies to keep things thread safe - but that’s incorrect - it just passes the reference so it doesn’t provide much value ↩︎
You can handle signals in a couple ways that we’ll discuss later ↩︎
Spurious: not being what it purports to be; false or fake ↩︎

Ruby methods are colorless

Tue, 16 Jul 2024 18:25:00 -0500

👋🏼 This is part of series on concurrency, parallelism and asynchronous programming in Ruby. It’s a deep dive, so it’s divided into 12 main parts:

Your Ruby programs are always multi-threaded: Part 1

Your Ruby programs are always multi-threaded: Part 2

Consistent, request-local state

Ruby methods are colorless

The Thread API

Bitmasks, Ruby Threads and Interrupts, oh my! (Concurrent, colorless Ruby)

When good threads go bad

Thread and its MaNy friends

Fibers

Processes, Ractors and alternative runtimes

Scaling concurrency with streaming

Abstracted, concurrent Ruby

Closing thoughts, kicking the tires and tangents

How I dive into CRuby concurrency

You’re reading “Ruby methods are colorless”. I’ll update the links as each part is released, and include these links in each post.

📝 A quick note on Ruby runtimes

I’m approaching this series from the perspective of CRuby. This is the original and most popular/common Ruby runtime. I’m also focusing primarily on Ruby 3.2/3.3+. This matters because it informs how things like Threads and Fibers behave.

First, some context

Last year I read a tweet from @ThePrimagen about his experience with Go¹. In it he mentioned some strengths and weaknesses (cheekily), and then he mentioned a phrase I’d never heard before: “colorless functions”.

real talk, colorless functions [are] amazing and I think that most people don’t realize how great it really is

– @ThePrimeagen

Function colors

Curious to understand what a “colorless function” is, a quick search returned this article as the first result: What color is your function?²

In it, Bob Nystrom goes through a fictional language (spoiler: it’s Javascript/node.js) that breaks functions down into one of two “colors”:

Every function has a color

The way you call a function depends on its color

You can only call a red function from within a red function

Red functions are more painful to call

Some core library functions are red

It’s an allegory… red functions are asynchronous ones

– Nystrom

Red and blue functions equate to asynchronous and synchronous functions. And having to indicate the color of your function infects the rest of your code and the API decisions you make.

The road to hell is paved with callbacks

Nystrom’s article was written in 2015 while Javascript/node was still in its callback hell³ phase. Any operation that could block, like reading a file, making an HTTP call, or querying a database had to be done using a callback. Callbacks proliferated throughout your code - so asynchronous, red functions were plentiful and painful to interact with.

In JavaScript, everything happens in a single-ish threaded context called the “event loop”. You can’t block the loop - so you would set a callback to know when a blocking call completed:

readFile(path, (content, err) => {
  if (err) { throw ''; }
  saveFile(content, (file, err) => {
    // and on and on...
  });
});

After callbacks, he alludes to what would be the subsequent Promises phase of Javascript/node.js with his section “I promise the future is better”. An ergonomic improvement to callbacks but still cumbersome, and does not solve most red/blue pain points:

readFile(path)
  .then((content) => {
    saveFile(content)
      .then((file) => {})
      .catch(err) => {})
  })
  .catch((err) => console.error(err));

Finally he describes the async/await phase in “I’m awaiting a solution”, which leads us to modern Javascript:

try {
  const content = await readFile(path);
} catch (err) {
  console.error(err);
}
	
try {
  const file = await saveFile(content);
} catch (err) {
  console.error(err);
}

Invasive async/await

I’ll edit “I’m awaiting a solution” for modern JS, where he notes:

Synchronous functions return values, async ones return Promise<T> (Promise containing type T) wrappers around the value.

Sync functions are just called, async ones need an await.

If you call an async function you’ve got this wrapper object when you actually want the T. You can’t unwrap it unless you make your function async and await it.”

– Nystrom

#3 is the particularly infectious bit. Async code bubbles all the way to the top. If you want to use await, then you have to mark your function as async. Then if someone else calling your function wants to use await, they also have to mark themselves as async, on and on until the root of the call chain. If at any point you don’t then you have to use the async result (in JavaScript’s case a Promise<T>).

async function readFile(): Promise<string> {
  return await read();
}
	
async function iCallReadFile(): Promise<string> {
  return await readFile();
}
	
async function iCallICallReadFile(): Promise<string> {
  return await iCallReadFile();
}
	
function iGiveUp(callback) {
  iCallICallReadFile()
    .then(callback)
    .catch((err) => {})
}

Once your language is broken down into async and synchronous functions, you can’t escape it.

Even more onerous, if it isn’t built into your language core like JavaScript/node.js, adding it later means modifying your entire runtime, libraries and codebases to understand it. This was (still can be?) a painful transition for languages like Python and Rust.

// et tu, Rust?
async fn main() {
  let content = read_file().await;
}

What makes Ruby colorless?

So what does a “blue” (synchronous) method look like in Ruby?

file = File.read(path)

And what does that same “red” (asynchronous) call look like?

file = File.read(path)

Wait. Is that right? Surely there must be more nuance?

HTTP?

response = Net::HTTP.get(
  URI("https://example.com")
)

Shell calls!

output = `cat file.txt`

Process spawning??

pid = fork
  # good stuff
end
Process::Status.wait pid

Mutex locks?!?

mutex.synchronize {}

DNS resolution!??!

ip = Resolv.getaddress "ruby-lang.org"

Sleep!?!!!

sleep 5

😮‍💨😮‍💨😮‍💨

Well that’s nice! There is no difference between asynchronous and synchronous methods, so no color. We get asynchronous behavior for free. Blog post over!?

Well… not exactly. We still don’t know how we get that behavior.

In the section “What language isn’t colored?” Nystrom specifically mentions Ruby being one of a few colorless languages⁴:

…languages that don’t have this problem: [Java,] Go, Lua, and Ruby.

Any guess what they have in common?

Threads. Or, more precisely: multiple independent callstacks that can be switched between. It isn’t strictly necessary for them to be operating system threads. Goroutines in Go, coroutines in Lua, and fibers in Ruby are perfectly adequate.

– Nystrom

So Ruby is colorless, and to be colorless you need independent call stacks that you can switch between. And apparently Threads and Fibers can give you that.

What does that mean and what does that enable? And since Nystrom’s article was written in 2015 - is there anything new to Ruby since then?

A nested concurrency model 🪆

In Your Ruby programs are always multi-threaded: Part 2, we briefly discussed that Ruby has a layered concurrency model and that the best way to describe it is as a nesting doll - as you pull away the outer layers, you find new layers of concurrency inside.

Here’s my key for mapping icons to concurrency

💎 represents a Ruby process, which is essentially another instance of Ruby itself

🦖 represents a Ruby Ractor. Ractor stands for Ruby Actor, as in the Actor model. But I always think of the word “Raptor” instead, so I represent it with a Dinosaur 🤷🏻‍♂️

🧵 represents a Ruby Thread. It’s a thread icon, for obvious reasons

🌾 represents a Ruby Fiber. I think of wheat when I think of Fibers, hence 🌾

As a refresher, in your own Ruby code, you can easily inspect each layer directly:

puts "Process #{Process.pid}"
puts "  Ractor #{Ractor.current}"
puts "    Thread #{Thread.current}"
puts "      Fiber #{Fiber.current}"
	
# Process 123
#   Ractor #<Ractor:#1 running>
#     Thread #<Thread:0x0... run>
#       Fiber #<Fiber:0x0... (resumed)>

Your code is always operating within a specific instance of each layer. Mostly you can access those instances with a call to current.

Each layer has unique characteristics when it comes to running concurrent code. Some can operate in parallel, executing independent Ruby code at the exact same time. Others run your Ruby code concurrently, moving back and forth between them but never running simultaneously.

source: jpcamara.com

The outermost layer is a Process 💎. Every program on every common operating system (Linux, Windows, macOS) runs inside a process - it’s the top-level unit of execution in the OS. You can have as many processes as your OS will allow - but the number of CPU cores determine how many processes can run in parallel. Multiple Ruby processes can run in parallel but they live in isolated memory spaces - they have no simple way of communicating with each other.

pid = fork do
  #...
end
Process::Status.wait pid

Internally, a Ruby process is made up of Ractors 🦖. Ractors don’t have a direct parallel in the OS - they’re a unique Ruby structure which wrap native OS threads⁵. Multiple Ractors can run Ruby code in parallel, and they have strict sharing rules to keep them memory safe - so you can only communicate between them by passing messages. Those messages can copy data, or they can move objects completely between Ractors and change ownership. Ractors sound pretty nice, but are still experimental. Regardless, every Ruby process starts out with a single Ractor that your Ruby code runs inside of.

ractor = Ractor.new do
 #...
end
ractor.take

Inside of Ractors, you have Threads 🧵. Threads are created 1:1 with threads on your operating system⁶. These threads share the same memory space with each other, and every thread created within a process shares the same memory space as the process, and spends its lifetime there. Because threads share the same memory space they have to be carefully coordinated to safely manage state. Ruby threads cannot run CPU-bound Ruby code in parallel, but they can parallelize for blocking operations. Threads run preemptively, which means they can swap out of your code at any time, and move to other Ruby code. Every Ruby Ractor starts out with a single thread that your Ruby code runs inside of.

thread = Thread.new do
  #...
end
thread.join

Inside of threads, you have Fibers 🌾. Fibers are another structure with no direct OS equivalent (though they have parallels in other languages). Fibers are a tool intended for lightweight, cooperative concurrency. Like Ruby threads, Fibers share a memory space, but their coordination is more deterministic. Also like Ruby threads, Fibers cannot run CPU-bound Ruby code in parallel, but using a FiberScheduler can parallelize for blocking operations. Every Ruby thread starts out with a single fiber that your Ruby code runs inside of.

fiber = Fiber.new do
  #...
end
fiber.resume # transfer / Fiber.schedule

When a Ruby program starts, all four layers start with it. The Process 💎 starts, it creates a Ractor 🦖, which creates a Thread 🧵, which creates a Fiber 🌾. Ultimately, your Ruby code is in the context of all 4, but most specifically the fiber. Each layer is always active in a Ruby program whether you use them directly or not⁷. We can utilize each in unique ways as well.

That’s a lot! In subsequent posts you’ll learn why each layer exists and the values each provides. Every layer will get some discussion - but as it relates to asynchronous, colorless programming, threads and fibers are the most relevant.

Colorless threads 🧵 and fibers 🌾

source: jpcamara.com

Threads 🧵 are one of the oldest and most common units of concurrency in Ruby. They’re the OG for splitting up concurrent chunks of work. They’ve been in Ruby since Ruby 1.1⁸ and have enabled colorless calls for many years.

Fibers 🌾 aren’t too young themselves, but are about 10 years younger than their older sibling Threads. They were released in Ruby 1.9, but remained a bit of an esoteric option for a long time. They offered a form of concurrency, but for most use-cases it didn’t really help anyone - unlike Threads, they didn’t enable colorless programming at first.

But ever since Ruby 3, Fibers have been given superpowers in the form of the FiberScheduler. By using a FiberScheduler, Fibers can now provide seamless, colorless programming.

So both threads and fibers offer colorless, concurrent programming. We’ve got a definition for colorless, but what about concurrency?

What does it mean to be concurrent?

The most popular source for a definition of concurrency is from the creator of Go, Rob Pike, in his talk concurrency is not parallelism. Another member of the Go team summarized it:

When people hear the word concurrency they often think of parallelism, a related but quite distinct concept. In programming, concurrency is the composition of independently executing processes, while parallelism is the simultaneous execution of (possibly related) computations. Concurrency is about dealing with lots of things at once. Parallelism is about doing lots of things at once.

source: https://go.dev/blog/waza-talk

They describe it as a way of composing tasks. It allows you to spread out and coordinate work. How that work is ultimately executed is not relevant to whether the code is concurrent or not.

Look at the activity monitor for your OS right now. If you were to count the number of processes and threads, it would vastly exceed the number of available cores. And yet everything still chugs along smoothly. That’s concurrency in action.

There are over 200 threads associated with just these 5 apps. I don’t have 200 CPU cores, so only a handful of them could ever be active in parallel. But all of them could operate concurrently.

Having said all that - we do want our work to happen as parallel as possible in most cases. We want to maximize our resources and parallelize as much as we can.

In Rob Pike’s concurrency talk, he used the Go mascot, the gopher, to demonstrate how his task (incinerating obsolete language manuals 😅) was composed, and how it could be run in parallel, or with one gopher at a time, but it was still concurrent:

source: https://go.dev/talks/2012/waza.slide

Go’s gophers are kind of fun. Let’s use some concurrency mascots for Ruby to show our own example.

Process!

Ractor!

Thread!

Fiber!

CPU!

source: jpcamara.com

As a baseline, Processes and Ractors can run in parallel for as many cores as are available:

But Threads and Fibers, while concurrent, effectively only parallelize against one core at a time⁹:

What happens when we start scaling Processes and Ractors past the number of available cores?

Once we exceed the ability to parallelize, we are concurrent in the same way as Threads and Fibers! This model allows our units of concurrency adapt to the environment - when cores are available they can run in parallel, and when cores are not available they can swap between each other, transparently to the program itself. Every program eventually becomes concurrent as it scales, at least for CPU-bound processing.

But this leaves Threads and Fibers looking pretty limited. They allow you to breakup your work into independent, interleaving tasks, operating concurrently on Ruby code. But what good is that since they never operate in parallel… or do they?

Colorless calls

How does this all relate to colorless methods?

Let’s see with an example. In our example, we’re just traversing a few different websites we care about, and we want to retrieve them as efficiently as possible.

First we try it with threads. For demonstration purposes, we use the httpbin.com/delay endpoint to simulate delays in responses.

require "net/http"
	
def log_then_get(url)
  puts "Requesting #{url}..."
  get(url)
end
	
def get(uri)
  response = Net::HTTP.get(uri)
  puts caller(0).join("\n")
  response
end
	
def get_http_thread(url)
  Thread.new do
    log_then_get(URI(url))
  end
end
	
def get_http_via_threads
  threads = []
  threads << get_http_thread(
    "https://httpbin.org/delay/3?ex=1"
  )
  threads << get_http_thread(
    "https://httpbin.org/delay/3?ex=2"
  )
  threads << get_http_thread(
    "https://httpbin.org/delay/3?ex=3"
  )
  threads << get_http_thread(
    "https://httpbin.org/delay/3?ex=4"
  )
  threads.map(&:value)
end
	
now = Time.now
get_http_via_threads
puts "Thread runtime: #{Time.now - now}"

This code is doing a few things:

It’s split into multiple methods to create a callstack (effectively, a backtrace). This callstack allows us to demonstrate that we maintain the position and state of each method call, even when we switch between threads.
It creates four threads and appends them to an array. After those threads are initialized they go into a ready state, available to be run by the thread scheduler.
It calls valueon each thread. This blocks the program until each thread finishes and returns the last value returned from the thread. Once we block the main thread with value, the other threads have an immediate opportunity to start running. In this case we block until each thread finishes making an HTTP call.

# > bundle exec ruby main.rb
# 
# Requesting https://httpbin.org/delay/3?ex=1...
# Requesting https://httpbin.org/delay/3?ex=2...
# Requesting https://httpbin.org/delay/3?ex=3...
# Requesting https://httpbin.org/delay/3?ex=4...
# main.rb:12:in `get'
# main.rb:7:in `log_then_get'
# main.rb:18:in `block in get_http_thread'
# main.rb:12:in `get'
# main.rb:7:in `log_then_get'
# main.rb:18:in `block in get_http_thread'
# main.rb:12:in `get'
# main.rb:7:in `log_then_get'
# main.rb:18:in `block in get_http_thread'
# main.rb:12:in `get'
# main.rb:7:in `log_then_get'
# main.rb:18:in `block in get_http_thread'
# Thread runtime: 3.340238554

Second we try it with fibers:

require "net/http"
require "async"
	
# Same #log_then_get
# Same #get
	
def get_http_fiber(url, responses)
  Fiber.schedule do
    responses << log_then_get(URI(url))
  end
end
	
def get_http_via_fibers
  Fiber.set_scheduler(Async::Scheduler.new)
  responses = []
  responses << get_http_fiber(
    "https://httpbin.org/delay/3?ex=1", responses
  )
  responses << get_http_fiber(
    "https://httpbin.org/delay/3?ex=2", responses
  )
  responses << get_http_fiber(
    "https://httpbin.org/delay/3?ex=3", responses
  )
  responses << get_http_fiber(
    "https://httpbin.org/delay/3?ex=4", responses
  )
  responses
ensure
  Fiber.set_scheduler(nil)
end
	
now = Time.now
get_http_via_fibers
puts "Fiber runtime: #{Time.now - now}"

Same as before, this code is doing a few things:

We are split into multiple methods to create a callstack. Same as with threads, this callstack allows us to demonstrate that we maintain the position and state of each method call, even when we switch between fibers.
It schedules four Fibers. We append to an array inside the Fiber.schedule to get the result. Unlike threads, a scheduled fiber will start running its block immediately.
The fibers will coordinate between themselves and the current fiber. When we unset the FiberScheduler using set_scheduler(nil) in our ensure, the Async::Scheduler makes sure all fibers have finished running before returning.

This results in the following, similar backtrace:

# > bundle exec ruby main.rb
# 
# Requesting https://httpbin.org/delay/3?ex=1...
# Requesting https://httpbin.org/delay/3?ex=2...
# Requesting https://httpbin.org/delay/3?ex=3...
# Requesting https://httpbin.org/delay/3?ex=4...
# main.rb:12:in `get'
# main.rb:7:in `log_then_get'
# main.rb:24:in `block in get_http_fiber'
# .../async-2.6.3/lib/async/task.rb:160:in `block in run'
# .../async-2.6.3/lib/async/task.rb:330:in `block in schedule'
# main.rb:12:in `get'
# main.rb:7:in `log_then_get'
# main.rb:24:in `block in get_http_fiber'
# .../async-2.6.3/lib/async/task.rb:160:in `block in run'
# .../async-2.6.3/lib/async/task.rb:330:in `block in schedule'
# main.rb:12:in `get'
# main.rb:7:in `log_then_get'
# main.rb:24:in `block in get_http_fiber'
# .../async-2.6.3/lib/async/task.rb:160:in `block in run'
# .../async-2.6.3/lib/async/task.rb:330:in `block in schedule'
# main.rb:12:in `get'
# main.rb:7:in `log_then_get'
# main.rb:24:in `block in get_http_fiber'
# .../async-2.6.3/lib/async/task.rb:160:in `block in run'
# .../async-2.6.3/lib/async/task.rb:330:in `block in schedule'
# Fiber runtime: 3.291355669

If we visualize what’s happening, we see how things are getting coordinated. It’s nearly identical between the two:

Threads:

Fibers:

👆🏼This diagram does look pretty sequential - that’s odd huh?

And here is what happens once we are waiting for a response:

Threads:

Fibers:

👆🏼Oh, that seems better!

Running these HTTP methods is the same if I run it with no threads/fibers, 4 threads/fibers, or 400 threads/fibers. The code never changes, the execution context is all we need to start operating asynchronously. Something is coordinating for us behind-the-scenes, and we’re able to handle our HTTP calls in parallel 🙌🏼.

This lines up with our timing. In each example we ran 4 HTTP calls, all guaranteed to block for around 3 seconds. Despite that, each example finished in roughly 3 seconds, rather than the 12 seconds it would take if they had run sequentially.

# Fiber runtime: 3.291355669
# Thread runtime: 3.340238554

To paraphrase Bob Nystrom’s article one last time, we can apply his original points about Go almost perfectly to Ruby threads/fibers:

As soon as you do any IO operation, it just parks that thread/fiber and resumes any other one that isn’t blocked on IO.

If you look at the IO operations in the standard library, they seem synchronous. In other words, they just do work and then return a result when they are done. But it’s not that they’re synchronous in the sense that it would mean in JavaScript. Other Ruby code can run while one of these operations is pending. It’s that Ruby has eliminated the distinction between synchronous and asynchronous code.

As we saw in the initial examples of colorless method calls - it’s not purely IO that parks and resumes. It’s the majority of blocking operations. Here’s another example:

# Threads
threads = [
  Thread.new { sleep 5 },
  Thread.new { `ruby -v` },
  Thread.new { 
    noop = fork {}
    Process.wait noop
  }
]
	
results = threads.map(&:value)
	
# Fibers
Fiber.set_scheduler(Async::Scheduler.new)
fibers = [
  Async { sleep 5 },
  Async { `ruby -v` },
  Async { 
    noop = fork {}
    Process.wait noop
  }
]
	
results = fibers.map(&:wait)

We fork a ruby process, run a command line script to get the current ruby version using ruby -v and sleep - all in parallel.

A quick aside on Async

Manually setting the FiberScheduler and manually unsetting it is just to more clearly demonstrate the interface. But you would normally run the block format:

Async do
  fibers << get_http_fiber #...
end

You’d also use the Async helpers instead of Fiber.schedule, which have a more robust available API:

def get_http_fiber(url)
  Async do
    log_then_get(URI(url), Fiber.current)
  end
end

Digging deep

My toy examples aside, this is one of the things that makes threaded/fibered web servers like Puma/Falcon and threaded job servers like Sidekiq/GoodJob so powerful - parallelizing blocking operations. Every time one of your job or web server threads/fibers reads a file, queries a database, makes an HTTP call, or waits for a shared resource, Ruby parks the thread/fiber and another thread/fiber is able to take over. Depending on the type of workload you can have dozens/hundreds/thousands of threads/fibers running concurrently, and blocking in parallel. We’ll push some limits on that later in the series to see where that might break down, and how we can keep scaling past those limits!

Now that we understand what colorless Ruby programming means and how it works - we’re left with some questions.

Why would we need or want both Threads and Fibers?
Why do some languages choose to introduce color to their languages? Are there upsides to color?
What is the deal with those “VM locks”?
What does the “OS Kernel” have to do with parallel IO?
What’s a Reactor?
Is there a best concurrency option?
Are there abstractions you should be using?
What’s the point of concurrency without parallelism?
We also keep talking about parallelizing blocking operations - why don’t threads and fibers already allow you to parallelize everything?

To answer these questions (and more), we’re going to dig pretty far into the ruby runtime to better understand Ruby threads, fibers and other forms of concurrency in Ruby.

Let’s start with the OG, Threads, in “The Thread API: Concurrent, colorless Ruby”. More soon! 👋🏼

PS: A historical sidenote - Ruby had its own callback phase

It was 2009. The Black Eyed Peas were on top of the charts with “Boom Boom Pow”. Microsoft had just launched Windows 7. Avatar was taking off in the box office. And of course most important of all, node.js was released (😉). During a brief ensuing insanity, everyone thought it would devour the world of web development.

Like any new option, it ultimately took its place alongside other tools, but it had a big influence in the popularity of the reactor pattern.

The reactor pattern is an event loop that registers events with the operating system and efficiently multiplexes them. It was a way of solving the C10K problem - a solution for serving 10 thousand+ clients from a single server.

To get a great insight into the Ruby concurrency landscape at that time, and why the node.js model seemed so appealing, see this post from Yehuda Katz in 2010 - https://yehudakatz.com/2010/08/14/threads-in-ruby-enough-already/. In summary:

Early Rails was not thread-safe so you could only scale with Processes
Once Rails was “thread-safe”, it started with a giant mutex around the entire framework that meant only one thread could do anything at a time, including blocking operations
Even threaded servers at the time would often wrap things poorly and too broadly in mutexes
The most popular database for Ruby/Rails at the time, MySQL, had a Ruby driver which did not release the thread on blocking operations
Ruby 1.9 was the first Ruby to map to real operating system threads and loads of people still worked in Ruby 1.8
He doesn’t mention it specifically, but the gem landscape was also pretty scary in terms of thread safety because many people did not understand or worry about it prior to that point. Things are definitely better now, though I’d recommend reviewing my Your Ruby Programs are always multi-threaded posts, and specifically tips for auditing gems

It was a concurrency mess.

No wonder node.js seemed like a magical concurrency silver bullet. A single process handling hundreds to thousands of users vs dozens of heavyweight processes handling a paltry comparative amount.

In the Ruby world the reactor pattern was served by a tool called EventMachine. We’ll talk a bit deeper about EventMachine in the “Concurrent, Colorless Ruby” fiber post later, but it was solely callback based.

EventMachine.run do
  redis = EM::Hiredis.connect
  http = EM::HttpRequest.new(
    "http://google.com/"
  ).get query: { "keyname" => "value" }
	
  http.errback { EventMachine.stop }
  http.callback {
    parse(http.response).each |link| do
      req = EM::HttpRequest.new(link).get
      req.callback {
        body = req.response
        redis.set(link, body).callback {
          # and on and on
        }
      }
    end
  }
	
  EventMachine.stop
end

You can see how messy coding like this becomes, just like the original callback model we presented in JavaScript.

EventMachine was a powerful scaling option, and people still maintain EventMachine-based code today, but it was at odds with the normal flow of Ruby code. Ruby could have moved to red and blue style calls like some other languages to clean that up, but instead superseded those Reactor efforts with the FiberScheduler.

The programming language, not the game 🙂 ↩︎
Turns out prime did a reaction video on it as well https://youtu.be/MoKe4zvtNzA ↩︎
In case this seems like a confusing and bizarre title, it’s a play on “the road to hell is paved with good intention” https://en.wikipedia.org/wiki/The_road_to_hell_is_paved_with_good_intentions 🌈 ↩︎
I’m sure this is not exhaustive, the list is just illustrative and represents some popular languages. You don’t need to defend your language of choice 😉 ↩︎
They stand for Ruby Actors so it is a Ruby form of the actor model. It works similarly to JavaScript WebWorkers in that they are isolated and require strict rules for sharing to enforce thread safety ↩︎
For now, as we’ll discuss later ↩︎
As with most things, there’s even more nuance to the layering and we’ll touch on it a bit later ↩︎
But have evolved significantly since then. This also required community support in addition to language support ↩︎
It’s more nuanced than this, and this is more for illustrative puposes. Even though they don’t run Ruby code in parallel, threads and fibers may still be tied to separate CPUs. ↩︎

Consistent, request-local state

Mon, 01 Jul 2024 20:11:00 -0500

👋🏼 This is a series on concurrency, parallelism and asynchronous programming in Ruby. It’s a deep dive, so it’s divided into 12 main parts:

Your Ruby programs are always multi-threaded: Part 1

Your Ruby programs are always multi-threaded: Part 2

Consistent, request-local state

Ruby methods are colorless

The Thread API

Bitmasks, Ruby Threads and Interrupts, oh my! (Concurrent, colorless Ruby)

When good threads go bad

Thread and its MaNy friends

Fibers

Processes, Ractors and alternative runtimes

Scaling concurrency with streaming

Abstracted, concurrent Ruby

Closing thoughts, kicking the tires and tangents

How I dive into CRuby concurrency

You’re reading “Consistent, request-local state”. I’ll update the links as each part is released, and include these links in each post.

Rather than a full entry in the series, this is a brief-ish¹ concurrency mini - a follow up to Your Ruby programs are always multi-threaded: Part 2.

Recap

In Sharing state with fibers, we walked through a scenario where you had fiber-local state using Thread.current[], and it was not available inside nested Fibers.

# This is #<User id=123>
Thread.current[:app_user]
Async do # equivalent to Fiber.new
  # This is nil
  Thread.current[:app_user]
  params[:files].each do |file|
    Async { # Fiber.new
      # Still nil
      Thread.current[:app_user]
      # Upload to S3 fails...
    }
  end
end

We had fanned out user uploads to take advantage of parallelized IO. The problem was we tried to use fiber-local state and that gets reset in new Fibers.

To solve it, we tried true thread-local state with thread_variable_get and thread_variable_set.

# This is #<User id=123>
Thread.current.thread_variable_get(:app_user)
Async do
  # Still #<User id=123>
 Thread.current.thread_variable_get(:app_user)
  params[:files].each do |file|
    Async {
      # Still #<User id=123>!
      Thread.current.thread_variable_get(:app_user)
      # Upload to S3...
    }
  end
end

On a regular threaded server, that’s ok. It’s exactly how CurrentAttributes would function by default:

class AppContext < ActiveSupport::CurrentAttributes
  attribute :user, :locale

  def user=(user)
    super
    self.locale = user.locale
  end
end

# This is #<User id=123>
AppContext.user
Async do
  # Still #<User id=123>
  AppContext.user
  params[:files].each do |file|
    Async {
      # Still #<User id=123>!
      AppContext.user
      # Upload to S3...
    }
  end
end

That’s because CurrentAttributes works off of an isolation_level. It internally uses ActiveSupport::IsolatedExecutionContext, which is an abstraction on top of Thread/Fiber-local state. It has two possible isolation_levels:

:thread - This is the default, and works exactly like our code using thread_variable_get²
:fiber - This is the other available option, and works exactly like our code using Thread.current[]³

ActiveSupport::IsolatedExecutionContext.isolation_level = :thread # or :fiber

The reason this abstraction matters is because of the introduction of Fiber-based servers.

Careful with the Falcons

If you look into the async ecosystem, pretty quickly you’ll find a full-featured web server called falcon.

Falcon is a multi-process, multi-fiber rack-compatible HTTP server built on top of async, async-container and async-http. Each request is executed within a lightweight fiber and can block on up-stream requests without stalling the entire server process. Falcon supports HTTP/1 and HTTP/2 natively.

falcon is Fiber-based, meaning each request is run in a new Fiber, rather than a new Thread. Effectively, as it accepts each socket connection it hands it off to a new Fiber:

Async do # Fiber.new
  socket.accept do
    # Instead of Thread.new or a Thread pool
    Async {} # Fiber.new
  end
end

By default, falcon will set the IsolatedExecutionContext.isolation_level for you to use Fibers:

ActiveSupport::IsolatedExecutionContext.isolation_level = :fiber

Which means it localizes its state at the Fiber level. We’ve got our original problem again:

# This is #<User id=123>
AppContext.user
Async do
  # This is nil
  AppContext.user
  params[:files].each do |file|
    Async {
      # Still nil
      AppContext.user
      # Upload to S3 fails...
    }
  end
end

What would happen if you set the isolation_level back to :thread?

ActiveSupportIsolatedExecutionContext.isolation_level = :thread

class ImagesController
  before_action :set_user
	
  def create
    Async do
      # May be a user from a different fiber!
      # They all live in the same Thread, so they share
      #   thread-local state
      AppContext.app_user
      params[:files].each do |file|
        Async { S3Upload.new(file).call }
      end
    end
  end
end

Bad things. We start running into issues with shared global state again, because all of our fibers share the same Thread! We need to be Fiber-local when running on a Fiber-based server.

If Thread-locals are not safe on a Fiber-based server, how can we safely share this state?

Resetting context

One less than ideal option is to just reset the state in each nested Fiber:

# This is #<User id=123>
user = AppContext.user
Async do
  AppContext.user = user
  # This is #<User id=123>
  AppContext.user
  params[:files].each do |file|
    Async {
      AppContext.user = user
      # This is #<User id=123>
      AppContext.user
      # Upload to S3...
    }
  end
end

You don’t have to set it on every level - just the level that actually needs it. But you probably want to set it on every level to not surprise yourself later.

This works, but it’s manual and clunky.

Fiber storage

There’s a new option available as of Ruby 3.2.

It’s a new kind of “local”, called Fiber storage⁴, and it was designed to help with precisely this type of issue:

Fiber[:app_user] = user

# This is #<User id=123>
Fiber[:app_user]
Async do
  # This is #<User id=123>
  Fiber[:app_user]
  params[:files].each do |file|
    Async {
      # This is #<User id=123>
      Fiber[:app_user]
      # Upload to S3...
    }
  end
end

Fiber storage is a mechanism for inheriting state from a Fiber to any Ractors/Threads/Fibers started from that scope.

The best term for this type of storage is “request-local”. The definition of “request” i’m using here is very loose - it just means the context of some particular slice of execution. That might mean a web page request, a background job run, a cron job, etc.

So we could create a new AppContext using this approach:

class AppContext
  class << self
    def user
      Fiber[:app_user]
    end

    def user=(user)
      Fiber[:app_user] = user
      self.locale = user.locale
    end

    def locale
      Fiber[:app_locale]
    end

    def locale=(locale)
      Fiber[:app_locale] = locale
    end
  end
end

With this new AppContext, things are working again!

Async do
  # This is #<User id=123>
  AppContext.user
  params[:files].each do |file|
    Async {
      # This is #<User id=123>
      AppContext.user
      # Upload to S3...
    }
  end
end

Even more, this works for Threads too:

Thread.new do
  # This is #<User id=123>
  AppContext.user
  params[:files].each do |file|
    Thread.new {
      # This is #<User id=123>
      AppContext.user
      # Upload to S3...
    }
  end
end

And Ractors:

Ractor.new do
  # This is #<User id=123>
  AppContext.user
  Thread.new do
    # This is #<User id=123>
    AppContext.user
  end.join
end

📝 If it works for Ractors, Threads and Fibers, why call it “Fiber storage”?

Fibers are the innermost level of concurrency in Ruby (Process -> Ractor -> Thread -> Fiber). Since all code operates within a Fiber scope, it makes sense to have it be the storage layer for local state.

Every time you create a new Ractor/Thread, a new Fiber is created within it. Which means the thing actually inheriting the Fiber storage is the Fiber that exists within that new context, not the Ractor/Thread itself.

An ideal future state of libraries is that everything handling “request” type state would move to using Fiber storage (CurrentAttributes, RequestStore, and friends).

Caveats

Like with any new approach in an existing, robust ecosystem, there are some caveats to consider.

It requires framework buy-in

Every request needs to be wrapped in a new Fiber with fresh storage, so it doesn’t accidentally inherit cross-request storage:

# In a library like Puma/Sidekiq/Pitchfork
def new_request
  Fiber.new(storage: nil) do
    # fresh storage for this request
  end
end

Falcon already handles this for you. But no other framework does (yet). If you aren’t using Falcon, this may not be a viable option for you yet.

Each layer inherits a copy

Each layer copies the keys and values from the current Fiber storage scope into a new hash. You can’t override entries from the parent scope, so overrides only impact lower levels.

Fiber[:value] = 1
Fiber.new do
  Fiber[:value] = 2
end
Fiber[:value] # => 1

This is intentional, but just something to keep in mind.

Some things aren’t meant to be shared

As you fan out there are things you likely want to share, like the current user. But there are things you don’t want to, or can’t - like database connections or file handles.

That means existing solutions that try to share concepts (like CurrentAttributes and IsolatedExecutionContext) will need to split their approach. For a concept like CurrentAttributes you want inherited Fiber storage. For a database connection, you want things fiber-local so each Fiber has its own connection.

Takeaways

Use an abstraction, like CurrentAttributes. Over time, these abstractions should catch up with available options like Fiber storage and you’ll reap the benefits
If you’re using Falcon, Fiber storage is a great option for sharing state across child Fibers/Threads/Ractors
If you’re not using Falcon, there’s still some work for the community to catch up with this new option, so use it carefully or avoid it for now

Onward!

Ruby 3+ has introduced interesting new options for concurrency like the FiberScheduler/Fiber storage and thanks to many community efforts, things are starting to catch up. But there’s still work to be done and new abstractions needed to handle things seamlessly.

Thanks to Samuel Williams⁵ (@ioquatix, author of Async, Falcon, and the FiberScheduler) for suggesting the clarification and reviewing this post!

Ok, now we’ll be heading over to “Ruby methods are colorless”. More soon 👋🏼.

Some readers may be thinking “oh thank god, finally a brief one” 😆 ↩︎
It works the same but it’s not actually using that internally. It monkey-patches Thread and adds a new attr_accessor. If someone cares, I could talk about it more! ↩︎
Also not actually using that internally, same as with :thread. It monkey-patches Fiber and adds a new attr_accessor. If someone cares, I could talk about it more as well! ↩︎
There’s a bit more context in the original proposal https://bugs.ruby-lang.org/issues/19078 and in the docs for creating new Fibers https://docs.ruby-lang.org/en/master/Fiber.html#method-c-new ↩︎
Aka Mr Async, aka The Fiber Fella ↩︎

Your Ruby programs are always multi-threaded: Part 2

Wed, 26 Jun 2024 03:57:00 -0500

👋🏼 This is a series on concurrency, parallelism and asynchronous programming in Ruby. It’s a deep dive, so it’s divided into 12 main parts:

Your Ruby programs are always multi-threaded: Part 1

Your Ruby programs are always multi-threaded: Part 2

Consistent, request-local state

Ruby methods are colorless

The Thread API

Bitmasks, Ruby Threads and Interrupts, oh my! (Concurrent, colorless Ruby)

When good threads go bad

Thread and its MaNy friends

Fibers

Processes, Ractors and alternative runtimes

Scaling concurrency with streaming

Abstracted, concurrent Ruby

Closing thoughts, kicking the tires and tangents

How I dive into CRuby concurrency

You’re reading “Your Ruby programs are always multi-threaded: Part 2”. I’ll update the links as each part is released, and include these links in each post.

If you haven’t read “Your Ruby programs are always multi-threaded: Part 1”, you’re better off starting there. You’ll still get value of this post, but you’ll get the full context by reading both. All of the Table of Contents links under Part 1 link back to that previous article ↩️.

Now that you’re properly clearing out thread-local state between requests/jobs, you have a new feature to write. Up until this point you could only add an image from a url - users want to upload images from their computer.

It seems complex to take a file from the front end and somehow get it to a background job. So you’re going to do it in your controller - but you want to minimize the chance of timing out while uploading a bunch of files.

📝 Since this example app is built with rails, it should instead be using ActiveStorage. But let’s just play along at home 😌

You’ve heard of an interesting library called async that makes parallelizing IO¹ pretty easy and is more predictable than threads. It does it using this special Ruby 3+ FiberScheduler thing 🤔.

As a test you try uploading some files to S3 in the controller. You create an Async block, and inside of it you create as many Async blocks as there are blocking operations to parallelize:

class ImagesController
  before_action :set_user

  def create
    Async do
      params[:files].each do |file|
        Async { S3Upload.new(file).call }
      end
    end
  end
end

class S3Upload
  def initialize(file)
    [@file](https://micro.blog/file) = file
  end

  def call
    object.put(body: [@file](https://micro.blog/file))
  end

  private

  def object
    client = Aws::S3::Resource.new
    bucket = client.bucket(ENV["bucket"])
    bucket.object(object_path(@file))
  end

  def object_path
    [@file](https://micro.blog/file)
  end
end

source: jpcamara.com

You test this and they all upload in parallel! Nice! Your request takes as long as the longest upload, and no longer than that. But uploading random file names at the top level of your S3 bucket is not ideal so you prefix the files with the current user id:

# ...
Async do
  S3Upload.new(file, AppContext.user).call
end
# ...

class S3Upload
  def initialize(file, user)
    [@file](https://micro.blog/file) = file
    [@user](https://micro.blog/user) = user
  end

  # ...

  def object_path
    name = [@file](https://micro.blog/file)
    "#{@user.id}/#{name}"
  end
end

You test this and… it raises a NoMethodError on NilClass when trying to get user.id. Huh? You’re sure the user is defined so you print some information:

class ImagesController
  before_action :set_user

  def create
    puts "user[#{AppContext.user}]"
    # ...
  end
end

# ...
def object_path
  name = [@file](https://micro.blog/file)
  puts "upload user[#{AppContext.user}]"
  puts "upload [#{@user}]"
  "#{@user.id}/#{name}"
end

Which prints:

user[#<User id=123>]
upload user[]
upload [@user](https://micro.blog/user)[]
NoMethodError...

A wild Fiber appeared!

AppContext.user is nil when we’re inside that async block… I think it’s time to look at those Thread.current[] docs again…

Thread#[]

Returns the value of a fiber-local variable

Well that doesn’t seem right. Doesn’t everyone call them thread-locals? They’re not thread-local… they’re fiber-local?

What exactly does it mean for it to be fiber-local? And how can you fix it so can use your AppContext within Async?

Ruby concurrency has layers

The concurrency model in Ruby is multi-layered. The best way to describe it is as a nesting doll - as you pull away the outer layers, you find new layers of concurrency inside.

source: jpcamara.com

In your own Ruby code, you can easily inspect each layer directly:

puts "Process #{Process.pid}"
puts "  Ractor #{Ractor.current}"
puts "    Thread #{Thread.current}"
puts "      Fiber #{Fiber.current}"

# Process 123
#   Ractor #<Ractor:#1 running>
#     Thread #<Thread:0x0... run>
#       Fiber #<Fiber:0x0... (resumed)>

Your code is always operating in the context of a specific instance of each layer. Mostly you can access those instances with a call to current. Your code runs within a Fiber, which is inside a Thread, which is inside a Ractor, which is inside a Process. The next posts in this series will dig into every layer, but for now you’ve got a fiber problem².

Internally, every Async block is run in a new Fiber. That’s the innermost layer of Ruby concurrency. So that means if you have something that is stored as a fiber-local, every time you create a new fiber, you get an empty, fresh set of locals.

# Fiber.current #<Fiber>
puts "Main Fiber #{Fiber.current}"
puts Thread.current[:app_user]
Fiber.new do
  puts "Fiber.new #{Fiber.current}"
  puts Thread.current[:app_user]
end.resume
Async do
  puts "Async #{Fiber.current}"
  puts Thread.current[:app_user]
  Async do
    puts "  Async #{Fiber.current}"
    puts Thread.current[:app_user]
    Async do
      puts "    Async #{Fiber.current}"
      puts Thread.current[:app_user]
    end
  end
end

Running the above code you get:

Main Fiber #<Fiber:0x00007a77a0b5bbb0 (resumed)>
#<User id=123>
  Fiber.new #<Fiber:0x00007a77a01766b8 main.rb:940 (resumed)>
	
    Async #<Fiber:0x00007a77a0175858 task.rb:326 (resumed)>
	
      Async #<Fiber:0x00007a77a0174fc0 task.rb:326 (resumed)>
	
        Async #<Fiber:0x00007a77a0174cf0 task.rb:326 (resumed)>

That’s well and good to know, you think, but how can you fix it? You don’t care about being fiber-local, you’re trying to be thread-local so these fibers can use your AppContext🧵.

Actual thread-locals

Those same Thread.current docs point us in the right direction:

For thread-local variables, see thread_variable_get and thread_variable_set

Let’s update AppContext to use that:

class AppContext
  class << self
    def user
      Thread.current.thread_variable_get(:app_user)
    end

    def user=(user)
      Thread.current.thread_variable_set(:app_user, user)
      self.locale = user.locale
    end

    def locale
      Thread.current.thread_variable_get(:app_locale)
    end

    def locale=(locale)
      Thread.current.thread_variable_set(:app_locale, locale)
    end
  end
end

It’s a bit more verbose, but otherwise it feels the same. Run our upload code again and…

user[#<User id=123>]
upload user[#<User id=123>]
upload [@user](https://micro.blog/user)[#<User id=123>]

Success 🙌🏼. The data is now actually thread-local. It is still present inside of fibers because those fibers all belong to the current thread³.

There are actually two more core Ruby ways⁴ you could handle this situation 😵‍💫. But we’ve covered the two most common options, and it solves the problem at hand. We’ll dig into those other options later in the series.

This seems like it should be a solved problem, you think. Does everyone write this from scratch and encounter these same issues? Is there an easier way?

Yes, thankfully.

CurrentAttributes

If you’re in Rails, or you use ActiveSupport, you should be using CurrentAttributes.

CurrentAttributes provides the same basic behavior we built (plus more), and integrates seamlessly into both Rails controllers and jobs. It also has first class support from Sidekiq, even without Rails. Here’s the AppContext code, now using CurrentAttributes:

class AppContext < ActiveSupport::CurrentAttributes
  attribute :user, :locale

  def user=(user)
    super
    self.locale = user.locale
  end
end

At first glance, it’s already a much simpler and clearer setup. But the real benefit is now we don’t have to worry about anything else.

It hooks into the Rails executor lifecycle, so whenever a web request/job begins or completes, AppContext.reset is called
It internally uses IsolatedExecutionState, which is an abstraction around thread and fiber-local state. It is configurable to use thread or fiber-local state, and by default it uses thread-local state so it works out of the box for your example Rails app
Sidekiq offers first class support for it, so even if you don’t use Rails you can opt-in to resetting behavior, and Sidekiq will automatically load it at the start of your job based on the execution state present when the job was enqueued

Interestingly, CurrentAttributes doesn’t use the Thread.current[] or Thread.thread_variable_get approaches. We’ll talk about that more in a later post.

RequestStore

I would not recommend using this library. It only supports Thread.current, and it only automatically clears itself for Rails controllers - you’re still on the hook for managing job middleware/callbacks! It also will fail the same way as our example in an Async block. You can learn more about it from its GitHub repository.

Back in reality, part three…

If this example seemed oddly specific again, that’s because this issue is present in some change tracking gems.

In terms of monthly downloads, paper_trail is by far the most popular gem for change tracking. And internally it uses request_store which means:

It uses Thread.current[]
It does take care of Rails controller cleanup for you
it does not take care of job cleanup for you. If you set tracking information anywhere in your jobs, you are susceptible to leaking data between job runs 🙅🏻‍♂️.

If you’re going to use paper_trail, make sure you clear out your state in jobs. Since it uses request_store internally, you can clear out state using that.

# If using Sidekiq
class PaperTrailClearMiddleware
  include Sidekiq::ServerMiddleware

  def call(_worker, _job, _queue)
    yield
  ensure
    RequestStore.clear!
  end
end

Sidekiq.configure_server do |config|
  config.server_middleware do |chain|
    chain.add PaperTrailClearMiddleware
  end
end

# If using only ActiveJob
class ApplicationJob < ActiveJob::Base
  after_perform :clear_paper_trail_context

  def clear_paper_trail_context
    RequestStore.clear!
  end
end

There is a gem that does this for Sidekiq only for you, called request_store-sidekiq. It hasn’t been updated in several years, but it does what my middleware example does, automatically for you.

All change tracking gems use some type of thread/fiber state, so here’s the rundown on the remaining popular gems:

audited is the next most popular but it uses CurrentAttributes internally, so you’re good!
Logidze uses a block, so it’s basically impossible to leak data between threads. But, it uses Thread.current[] internally so it will break if you nest fibers (like Async blocks) inside of it
PublicActivity uses Thread.current[] but it’s unclear if it is cleaning it up appropriately. You’ll have to do you own due diligence there

4. Reusing objects

You do know that single-threaded servers can still have issues leaking state between requests - like our fiber-local issue. But can they hit real threading issues? Let’s ask GitHub:

Us: Hey GitHub!

GitHub: Ummm, hi?

Us: You run Unicorn as your web server don’t you?

Github: Who are you? How did you get in here??

Us: Can a single-threaded server like Unicorn encounter threading issues?

GitHub: …

GitHub: …yes 😔 https://github.blog/2021-03-18-how-we-found-and-fixed-a-rare-race-condition-in-our-session-handling/

TL;DR

Someone wrote threaded code to save on performance:

after_action :run_expensive_op_in_thread

It held a reference to a request environment object:

def run_expensive_op_in_thread
  Thread.new do
    env = request.env
    # some background logic
  end
end

That request object got reused between requests - it just cleared its data out instead of creating a new instance:

def after_request
  request.env.clear
end

In certain edge cases, sessions would get reused between users:

def run_expensive_op_in_thread
  Thread.new do
    if not_logged_in
      login(request.env)
    end
  end
end

So yes… it still involved explicit threading, but in the context of a single-threaded server. This is the crux of that “Ruby is always multi-threaded” thing. Threads pop up all over the place even when you think you’re not using them. If you defensively code assuming threads could eventually be involved, you reduce the risk of these types of issues popping up later.

source: https://github.blog/2021-03-18-how-we-found-and-fixed-a-rare-race-condition-in-our-session-handling/

The fix for the problem was to stop reusing environment objects between requests. Each new request created a new environment object, which means a thread could safely hold onto that environment information without anything leaking in from subsequent requests.

Create new instances. Unless you know they are incredibly expensive, it is not worth the risk of sharing them if it can be avoided.

5. Race conditions

You’ve been having a tough time with these bugs, yeesh. It’s as if some kind of external force is controlling you to intentionally introduce each of these issues for other people’s benefit. But that obviously can’t be true 🤫.

You’ve got a new requirement. And this one seems pretty… concurrent. You need to coordinate some of your jobs - you split your work into several pieces, run them in individual jobs, then perform one final piece once every job has finished.

How can you coordinate a set of jobs? Maybe there’s a concurrency concept you can try out!

You read about something called a “Countdown Latch”. You tell it what to countdown from, multiple threads can safely count down using it, and you can have another thread wait until it finishes:

latch = CountdownLatch.new(10)
10.times.map do |x|
  Thread.new do
    puts "hello #{x}"
    latch.count_down
  end
end

latch.wait
puts "world!"

This prints out

hello 0
hello 1
hello 2
hello 3
hello 4
hello 5
hello 6
hello 7
hello 8
hello 9
world!

The results look ordered but it’s just a coincidence of how it ran. The CountdownLatch only helps coordinate completing jobs, nothing internal to their ordering

Threading concepts

When you call latch.wait, the thread waits until the count_down is complete. What would implementing a CountdownLatch look like?

class CountdownLatch
  def initialize(count)
    @count = count
    @mutex = Mutex.new
    @cond = ConditionVariable.new
    puts "CountdownLatch initialized with a count of #{@count}"
  end

  def wait
    @mutex.synchronize do
      @cond.wait(@mutex) while @count > 0
    end
  end

  def count
    @mutex.synchronize { @count }
  end

  def count_down
    @mutex.synchronize do
      @count -= 1
      if @count == 0
        @cond.broadcast
      end
      @count
    end
  end
end

📝 The concurrent-ruby gem comes with its own CountdownLatch. I’ve written my own here for demonstration - but if you have a need for one, use theirs, not mine

A CountdownLatch is a special form of a computer science concept called a barrier. Barriers are used to help coordinate execution of multiple threads⁵ - a CountdownLatch is a one-time use form of a barrier where a thread can wait until the barrier counts down to zero.

To achieve that, you need a Mutex and a ConditionVariable. Mutexs and ConditionVariables are low-level tools available for coordinating access to resources in Ruby code. We’ll dig more into those in “Concurrent, colorless Ruby”.

For now, know that a Mutex can be used to safely lock a section of Ruby code. While you are in a synchronize block, any other thread using the same Mutex will wait until that block finishes. As far as the coordinating threads are concerned, anything that happens in that block happens atomically - individual pieces which would not normally be thread-safe are treated like a single, safe operation.

Ok. We’ve got our coordinator ✅. Now let’s try applying it to our jobs.

class TickerJob
  include Sidekiq::Job

  def perform
    # do some work
    latch = CountdownLatch.new(10)
    remaining = latch.count_down
    puts "Tick! [#{remaining}]"
  end
end

class CoordinatorJob
  include Sidekiq::Job

  def perform
    latch = CountdownLatch.new(10)
    10.times do
      TickerJob.perform_async
    end
    latch.wait
    puts "All jobs finished!"
  end
end
  end
end

Which gives us…!

CountdownLatch initialized with a count of 10
CountdownLatch initialized with a count of 10
Tick! [9]
CountdownLatch initialized with a count of 10
Tick! [9]
CountdownLatch initialized with a count of 10
Tick! [9]
CountdownLatch initialized with a count of 10
Tick! [9]
CountdownLatch initialized with a count of 10
Tick! [9]
CountdownLatch initialized with a count of 10
Tick! [9]
CountdownLatch initialized with a count of 10
Tick! [9]
CountdownLatch initialized with a count of 10
Tick! [9]
CountdownLatch initialized with a count of 10
Tick! [9]
CountdownLatch initialized with a count of 10
Tick! [9]

Uhhhh… hmmm… well that’s not working properly. It never finishes, and every count_down gives us 9. Oh right. You can’t create new countdowns everywhere you need them… you need to gulp… share them 😱.

You know the CountdownLatch class is thread safe, so that means multiple threads can use an instance of it together safely. You decide to use a global for now to try it out in your jobs… a class-level ivar 😅. Those are problematic, but the countdown code is thread safe so it should be ok, right? This is your time to shine!

class AppContext
  def self.latch
    @latch ||= CountdownLatch.new(10)
  end
end

Just to be safe you try it on some simple threads first…

10.times.map do
  Thread.new do
    latch = AppContext.latch
    # do some work
    puts "Count down! [#{latch.count_down}]"
  end
end

AppContext.latch.wait

CountdownLatch initialized with a count of 10
CountdownLatch initialized with a count of 10
Count down! [9]
CountdownLatch initialized with a count of 10
Count down! [9]
CountdownLatch initialized with a count of 10
Count down! [9]
CountdownLatch initialized with a count of 10
Count down! [9]
CountdownLatch initialized with a count of 10
Count down! [9]
CountdownLatch initialized with a count of 10
Count down! [9]
CountdownLatch initialized with a count of 10
Count down! [9]
CountdownLatch initialized with a count of 10
Count down! [9]
CountdownLatch initialized with a count of 10
Count down! [9]
CountdownLatch initialized with a count of 10
Count down! [9]

Well… So much for thread safety? What’s the deal?

You’re seeing CountdownLatch initialized with a count repeatedly, so it seems that the instance gets created over and over. Is the latch instance different every time?

10.times.map do |_x|
  Thread.new do
    latch = AppContext.latch
    # do some work
    puts "Count down! [#{latch.count_down}]"
    puts "latch id #{latch.object_id}"
  end
end

AppContext.latch.wait

🫠

...
latch id 2700
...
latch id 2720
...
latch id 2740
...
latch id 2760
...
latch id 2780
...
latch id 2800
...
latch id 2820
...
latch id 2840
...
latch id 2860
...
latch id 2880

You get a different latch every time.

Memoization / check-then-act

You’re hitting a race condition - multiple threads are racing to finish an operation. And this particular situation is so common it’s got a name: “check-then-act”.

In Ruby, the most common case of this is with memoization.

📝 If you don’t know - memoization is one of those funny looking terms, and sounds a bit like if the priest from princess bride tried to say “memorization”.

Mem-mwah-zatioooon

Memoization is just a means for lazy-loading code, or sharing an expensive resource. The first time you need it, you initialize it and store it to a variable you can reuse, like an instance variable.

Memoization is ok within a single thread, but if you’re sharing an object then memoization can lead to “check-then-act” conditions. The AppContext code is using memoization to share the CountdownLatch:

@latch ||= CountdownLatch.new(10)

It looks like a one-line call, but is actually multiple statements.

if !@latch
  @latch = CountdownLatch.new(10)
end

You “check” if the @latch is present, “then” you “act” on that information. If it’s present, you don’t do anything. If it isn’t, you create a new CountdownLatch and set it to @latch. Multiple threads cannot run this code in parallel, but they can swap between each other at inopportune times.

Thread 1 checks, and finds that @latch is nil. It instantiates CountdownLatch to set it to @latch. The initialize method in CountdownLatch has a puts statement. That’s blocking IO, so Thread 1 yields and…
Thread 2 checks, and finds that @latch is nil. As it instantiates CountdownLatch, it yields due to the blocking IO from puts…
Thread 3 checks, and finds that @latch is nil… seeing a pattern?

The puts statement exists very intentionally to force this condition. Without it, the race condition would happen much less frequently. But it could be a logger call, or it could open a connection to an external resource, or create a Tempfile⁶. Many things can easily causes the initialize to yield to other available threads.

If you want to use memoization safely between threads, you need a Mutex. Just like CountdownLatch uses a Mutex internally to count_down safely between threads, we need a Mutex to safely share our lazy-loaded instance:

class AppContext
  @latch_mutex = Mutex.new

  def self.latch
    return @latch if @latch
    @latch_mutex.synchronize do
      @latch ||= CountdownLatch.new(10)
    end
  end
end

💰 My personal metric is that the right amount of mutexes in my code is zero. If I am using a mutex, I think hard to figure out a way to avoid it because it means I’m opening up myself and future devs to a lot of cognitive overhead: you need to think critically anytime you make a change relating to mutex code.

If you’re a library or framework author they may be unavoidable at some point to do interesting or useful things. In my own application code, I can pretty much always avoid them.

To be perfectly honest, I’m not confident it’s ok to return the @latch early if it’s present. I’m pretty sure it is… but is there a memory visibility implication? Am I creating a new type of edge case or race condition? These are the questions you have to ask yourself when you introduce mutexes.

Ok, AppContex.latch memoization take three:

10.times.map do
  Thread.new do
    latch = AppContext.latch
    # do some work
    puts "Count down! [#{latch.count_down}]"
    puts "latch id #{latch.object_id}"
  end
end

AppContext.latch.wait
puts "All threads finished!"

Finally, you get a correct countdown!

CountdownLatch initialized with a count of 10
Count down! [9]
latch id 2700
Count down! [6]
latch id 2700
Count down! [3]
latch id 2700
Count down! [0]
latch id 2700
Count down! [7]
latch id 2700
Count down! [1]
latch id 2700
Count down! [4]
latch id 2700
Count down! [8]
latch id 2700
Count down! [5]
latch id 2700
Count down! [2]
latch id 2700
All threads finished!

Much better! We should be ready for our jobs now. But is this really the right direction? You’ve resolved the “check-then-act” issue, but you realize there are some pretty obvious limits to this approach:

Right now we can have one instance of the CountdownLatch to share, and it’s hard-coded to 10… What would be the best way to share an object, but also make it configurable?
What if you have more than one job server? Multiple job servers can pull jobs from the same queue 🤔. You can only synchronize across threads if they live in the same server process. Outside of that, your mutex’s ability to lock anything is just a wish in the breeze.

You need something to share, that’s independent of what’s in memory and can represent multiple coordinating sets of jobs.

read-modify-write

You’re going to branch a bit outside of threads for concurrency, and get distributed. You rewrite the CountdownLatch to:

Allow specifying an id to key off of, so we can support more than one CountdownLatch at a time
Store the countdown value in Redis, so it can be accessed and count down from anywhere⁷
Independently create CountdownLatch instances so you don’t need to share the instance itself. You are sharing the Redis database instead.

class DistributedCountdownLatch
  def initialize(id, count = nil)
    [@id](https://micro.blog/id) = id
    @redis = Redis.new
    @redis.set(key, count) if count
  end
	
  def wait
    while current > 0
      puts "Remaining in countdown: [#{current}]"
      sleep(1)
    end
  end
	
  def current
    @redis.get(key).to_i
  end
	
  def count_down
    new_current = current - 1
    @redis.set(key, new_current)
    new_current
  end
	
  def key
    "latch_#{@id}"
  end
end

Running it across some threads looks promising.

latch = DistributedCountdownLatch.new("hello", 10)
10.times.map do |x|
  Thread.new do
    puts "hello #{latch.count_down}"
  end
end

latch.wait
puts "world"

So far so good!

Remaining in countdown: [10]
hello 9
hello 8
hello 7
hello 6
hello 5
hello 4
hello 3
hello 2
hello 1
Remaining in countdown: [1]
hello 0
world

You run it several times and feel ready to use it in your Sidekiq jobs!

class TickerJob
  include Sidekiq::Job

  def perform
    # do some work
    latch = DistributedCountdownLatch.new("tick")
    remaining = latch.count_down
    puts "Tick! [#{remaining}]"
  end
end

class CoordinatorJob
  include Sidekiq::Job

  def perform
    latch = DistributedCountdownLatch.new("tick", 10)
    10.times do
      TickerJob.perform_async
    end
    latch.wait
    puts "All jobs finished!"
  end
end

Let’s see how it goes:

Tick! [9]
Tick! [9]
Tick! [8]
Tick! [7]
Tick! [7]
Tick! [6]
Tick! [6]
Remaining in countdown: [6]
Tick! [5]
Tick! [5]
Tick! [4]
Remaining in countdown: [4]
Remaining in countdown: [4]
Remaining in countdown: [4]
Remaining in countdown: [4]
Remaining...

Poorly. It went poorly 🥺.

If we walk through the code again:

Thread 1 reads the value from Redis. That’s blocking IO, so Thread 1 yields and…
Thread 2 reads the value from Redis. That’s blocking IO, so Thread 2 yields and…
Thread 3 reads the value from Redis. That’s blocking IO, so Thread 3 yields and…
Thread 1 gets the value back which is 10. Its subtracts 1 from it and writes 9 to Redis. That’s blocking IO, so Thread 1 yields and…
Thread 2 gets the value back which is 10. Its subtracts 1 from it and writes 9 to Redis. That’s blocking IO, so Thread 1 yields and…

🙅🏻‍♂️👈🏼👉🏼👇🏼👆🏼🙅🏻‍♂️

You’re experiencing another race condition, and once again it’s common enough to have a name: “read-modify-write”. You’re “read”ing in a value in its current state from Redis, “modify”ing it, then “write”ing it back to Redis. The problem is that for shared resources, each reader “read”s the value in the same state. We have to coordinate the way each client is pulling data from Redis.

Race conditions happen on shared resources - those shared resources don’t need to live on the same machine.

📝 Redis is a single-threaded database, meaning it can only ever execute one thing at a time, no matter how many clients are connected. But being single-threaded does not do anything to fix race conditions. It may only execute one operation at a time, but nothing about that guarantees you won’t execute those operations out of order.

What can you do?

We need something like a Mutex, but a Mutex that can work across independent servers. A Mutex is just a lock you acquire on a resource - can you get that with Redis?

You can simulate a lock using the Redis operation SET:

class DistributedCountdownLatch
  class WatchError < StandardError; end

  # ... same code as before

  def count_down
    lock_key = "lock_#{key}"
    loop do
      # `nx` stands for `not exists`
      # `ex` means the lock key will expire in 10 seconds
      if @redis.set(lock_key, 1, nx: true, ex: 10)
        new_count = current - 1
        # `multi` allows every operation to run atomically
        @redis.multi do |transaction|
          transaction.set(key, new_count)
          transaction.del(lock_key)
        end
        return new_count
      end
      sleep 0.1
    end
  end
end

First you set a key using the nx, or “not exists” flag. If a key already exists, the operation fails. In that case you sleep for 0.1 seconds, then try again.
Second you get the current value.
Third, you perform a Redis MULTI operation, which is a kind of Redis database transaction. We will SET the new count, and DELete our lock key, in one atomic call to Redis. It all works, or it all fails.

📝 This type of lock is called a “Pessimistic lock”. Pessimistic locking means you must obtain exclusive access to the lock before running an operation, and retry or raise an error if that is not possible. The alternative is “optimistic” locking, which means you attempt to run the operation, and before committing verify that no other operations have taken place. If they have you either have to retry, or raise an error.

📝 If you’re going to use Redis as a distributed lock, you should use the redlock-rb gem, which implements the Redis teams recommended algorithm for locking, the “Redlock” algorithm.

By creating a simple form of a distributed Mutex, this fixes our race condition! Now we will only attempt the update once we acquire the lock, and we’ll retry until we can acquire it:

Tick! [9]
Tick! [8]
Remaining in countdown [8]
Tick! [7]
Tick! [6]
Tick! [5]
Tick! [4]
Tick! [3]
Tick! [2]
Tick! [1]
Remaining in countdown [1]
Tick! [0]
All jobs finished!

Coordinating jobs

Even having fixed the “check-then-set” and “read-modify-write” issues, trying to coordinate your jobs using this approach is a lot of work to get right. I wouldn’t recommend it, even though the code is slightly more accurate now.

If you’re interested in more appropriate ways to coordinate jobs:

Some job servers offer a feature for running and coordinating jobs in batches, with optional success/finish/error callbacks.
- Sidekiq offers this in their paid pro tier
- GoodJob offers batch support
- I am working on batch support for SolidQueue - feel free to leave feedback!
If you still are interested in a CountdownLatch approach:
- Redis has a built-in INCR and DECR method, which handle these updates for you atomically
- ActiveRecord has an increment and decrement method you could use to decrement a value on a record atomically.

read-modify-write shows up in many ways

There’s a very good chance you have code susceptible to read-modify-write issues right now. Aside from reading a value and then modifying it, the most common situation I see is around enforcing uniqueness constraints:

You expect a record in a database to be unique, but you don’t have uniqueness constraints at the database-level.
You “read” the database and find the record doesn’t exist.
Another request comes in at the same time and also “read”s the database and finds the record doesn’t exist.
In this case the “modify” is creating the record. Each request now attempts to create the record.
With no database-level uniqueness constraint, the result is that each request thinks the record doesn’t exist, and you “write” two (or more) records, violating your uniqueness requirement.

Parting thoughts

Phew! That was a lot. We’ve gone deep into 5 different threading mistakes, ways to identify them, and ways to fix them. Let’s finish up with a few topics addressing how to look out for threading issues in your future endeavors, as well as answering some possible remaining questions!

Tips for auditing gems 💎

Here are some notes on what to look for when auditing a gem for Thread safety.

If the gem interface is a static method, check the source. It might be an indicator of class-level ivars or thread/fiber-local data
If the amount of code is small, take some time to understand it. Keep the “threading mistakes” we’ve discussed in mind and see if anything sticks out to you
If there is a lot of code, look through and make sure it has active maintenance and maintainers. You’re not likely to learn a larger code base so at least know it’s being actively maintained. There have been studies to show that the longer-lived a codebase is, the likelier it is developers and users will find and remove threading issues.
If there’s low activity, a lot of code, and you can’t take the time to understand it, you probably want to avoid it
Make sure main has been released into a gem version. The repository can sometimes be misleading - it may have a fix explaining an issue you’re hitting that hasn’t been released
If a gem is unmaintained or inactive, it doesn’t mean you can’t use it. But, effectively you own it. If you hit issues, you’ll be the one to fix them

I use Falcon - Fibers are safe from this right?

The short answer is “assume not”.

We’ll dig more into the nuances of fibers in “Concurrent, colorless Ruby: Fibers”, but you are always safer to run as if threads will eventually be involved. Falcon can run using Threads in addition to Fibers, for instance.

While Fibers are more deterministic, even with the FiberScheduler, they are still susceptible to some of the same issues as Threads.

Quick question: Can race conditions happen with Fibers? We’ll walk through that in “Concurrent, colorless Ruby: Fibers”. Or see the handy answer key below.

.seY: rewsnA

I use JRuby/TruffleRuby…

Good news! These examples will break really fast for you! Congrats!

We’ll talk more about this later in the series. But know for now that JRuby and TruffleRuby use Threads without any implicit locks (ie, the GVL). You will hit threading issues much faster in those environments than in CRuby.

Preforking / Reforking servers

Preforking and Reforking servers do share resources. They use fork, which initially shares the entire memory space between two processes using something called Copy-On-Write memory. That means things which are not safe to share, like file handles, are shared by default.

Thankfully many members of the Ruby community have worked to make that as safe as possible for most popular libraries. We’ll talk more on this topic later in the series, discussing areas where it is handled automatically and areas where it isn’t.

What about using mutexes for globals?

You may have seen some of the global examples and thought: “couldn’t you keep them global the way they are and keep them safe using a mutex?”

Good question!

That would mean using a lock to provide coordinated access to your global state. Like taking our Fibonacci example and using a lock here:

require "monitor"

class Result
  attr_accessor :value
end

class Fibonacci
  @fib_monitor = Monitor.new

  class << self
    def result=(value)
      @fib_monitor.synchronize { @result = value }
    end

    def result
      @fib_monitor.synchronize { @result }
    end

    def calculate(n)
      @fib_monitor.synchronize do
        self.result = Result.new
        result.value = fib(n)
      end
    end

    def fib(n)
      return n if n <= 1

      fib(n - 1) + fib(n - 2)
    end
  end
end

📝 We actually use a Monitor here instead of a Mutex because it supports reentrancy. That means we can re-enter the synchronize block from the same thread, which is necessary in our example. Otherwise, it is largely the same as a Mutex.

📝 You may have noticed I removed the attr_accessor and I both set and get using the Monitor. I did this out of caution for a concept of “visibility” of shared resources between Threads. We’ll discuss that more later.

Now we’ll run it again using run_forever:

answers = [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144]

run_forever do |iteration|
  n = iteration % answers.size
  Fibonacci.calculate(n)
  answer = answers[n]
  result = Fibonacci.result.value

  if result != answer
    raise "[#{result}] != [#{answer}]"
  end
rescue => e
  puts "Iteration[#{iteration}] #{e.message}"
end

It works. We don’t get any errors. The mutex allows the code to treat multiple operations as a single operation. Are there downsides?

If you’re going to use a Monitor/Mutex on a piece of a global state - keep your mutex scope small to reduce locking overhead
Don’t hog the GVL. Make sure you don’t wrap your GVL-releasing operations like blocking IO in a Mutex.
Don’t get your mutex scope wrong. If you get your scoping wrong, you may still run into issues like memory inconsistency between Threads.

We’ll dig into GVL hogs and surprising behaviors in “Colorless, concurrent Ruby”. For now the short answer is: you can, but avoid it, if possible. It may hurt your ability to parallelize your code.

It’s not doom and gloom, it’s detect and correct

When I highlight a bunch of pitfalls you might hit, it feels like I’m being a virtual storm cloud, just trying to ruin your sunny coding day. In my post PgBouncer is useful, important, and fraught with peril, I highlight the tricky parts to inform but I still use the tool happily every day!

It’s the same for Ruby and threading concerns. Almost every language has these same problems, or worse⁸. Threads are easy to get wrong, so you want to take some golden paths and stay simple when using them.

Your code is safer if you assume you’re always using threads. You are. And if by some oddity you really aren’t, there’s a strong chance you will be eventually.

Takeaways 🥡

Your Ruby code is threaded. Just assume it is and forget semantics.
The GVL can make threading bugs harder to produce - if you see suspicious code that seems to work, try again using run_forever
Think thread safety
If you write any explicitly threaded code, coordinate threads or share data - use existing vetted tools like job batches and concurrent-ruby
Other runtimes like Ruby/TruffleRuby will produce bugs much faster
As much as possible, don’t share resources between threads. Forget your lessons from childhood: sharing is bad.

Next up

source: jpcamara.com

This is the second step into this Ruby concurrency series. Next up we’ll discuss the concept of “colorless” programming, and how it enhances the Ruby experience. We’ll also take our first dive into how the layers of Ruby work, starting with Threads.

More soon 👋🏼!

And any blocking operation ↩︎
💩 ↩︎
There is a short section in the thread docs that describes this behavior as well https://docs.ruby-lang.org/en/3.3/Thread.html#class-Thread-label-Fiber-local+vs.+Thread-local ↩︎
Fiber#store and adding attr accessors to Thread/Fiber ↩︎
The concept can also be used for other units of concurrency, like fibers ↩︎
Or use a file mutex https://github.com/yegor256/futex ↩︎
Other databases would also work, but the Redis data model is the simplest to get going for a simple value. Sidekiq works off of Redis as well, so you already would have it in your stack ↩︎
So don’t let your Python or Java friends try to bully you about it 😉. JavaScript can suffer similar issues as well.

I’ve never supported a production Rust or Elixir application so I can’t speak from experience, but those are the only two languages I can think of that might fare a lot better in this regard (and similar languages like F#, Gleam, Clojure, etc).

Elixir because of its pure functional approach. Rust because it’s one of the first non-functional languages I’ve ever encountered that has strong consistency baked into its core. They eliminate a class of threading bugs by their nature, but can still experience issues unique to how they work. ↩︎

Your Ruby programs are always multi-threaded: Part 1

Tue, 04 Jun 2024 05:00:00 -0500

👋🏼 This is a series on concurrency, parallelism and asynchronous programming in Ruby. It’s a deep dive, so it’s divided into 12 main parts:

Your Ruby programs are always multi-threaded: Part 1

Your Ruby programs are always multi-threaded: Part 2

Consistent, request-local state

Ruby methods are colorless

The Thread API

Bitmasks, Ruby Threads and Interrupts, oh my!

When good threads go bad

Thread and its MaNy friends

Fibers

Processes, Ractors and alternative runtimes

Scaling concurrency with streaming

Abstracted, concurrent Ruby

Closing thoughts, kicking the tires and tangents

How I dive into CRuby concurrency

You’re reading “Your Ruby programs are always multi-threaded: Part 1”. I’ll update the links as each part is released, and include these links in each post.

It's all threaded

If you run a web server using Puma, Falcon¹, Webrick, or Agoo², you use threads.

If you run background jobs using Sidekiq, GoodJob, SolidQueue³, or Que, you use threads.

If you use ActiveRecord, or Rails, you use threads.

If you use the NewRelic, DataDog, Sentry, LaunchDarkly, or Honeybadger gems, you use threads.

If you use net/http (or any gem internally using it like HTTParty or RestClient), httpx, or the Timeout⁴ gem, you use threads 🧵.

And because of the above, even if you use a forked, single-threaded server like Pitchfork or Unicorn, or run Puma in worker mode with no additional threads, or run jobs with Resque, or just use a basic Ruby rake task, you are using threads!

The point isn’t to be exhaustive. There are plenty more gems which use threads internally - you’re probably using some I haven’t listed. I’m also intentionally avoiding nuance - there are varying levels of threaded-ness.

The point is to demonstrate that threads are everywhere in Ruby, even if you are not working with them explicitly. Always write your code to be as thread safe as possible. Act as if you’re in highly threaded code and you’ll live a much happier life.

Ok but how threaded is it really

The degree of thread safety concern you need to have does vary depending on how threaded your Ruby environment is.

Sidekiq, Puma, SolidQueue, or GoodJob: very threaded. If there are threading bugs in code you use they will come up, eventually.
Falcon: Can be configured to be threaded. If it is, threading bugs will come up eventually. By default it only uses Fibers, which can still be susceptible to some concurrency issues.
Pitchfork, Unicorn or Resque: not threaded, but you’re probably using threaded gems or writing threaded code. I’ll discuss later how these gems can still hit threading issues on process based servers, and other concurrency adjacent issues you might encounter.
A one-off Ruby script with no gems - you exist in a thread but if it’s just something like processing files sequentially, practically not threaded. You’re probably fine, but YMMV 😬

It’s not just about writing explicitly threaded code that is safe to run. If you’re using Thread.new, you know you’re working with threaded code, and you know you need to proceed carefully. If you don’t know that, be careful! Threading has many sharp edges - it’s easy to get wrong.

Threads introduce non-determinism into your code - what works on one execution may not work on the next, even with identical inputs. It can lull you into a false sense of security, because that failure state may not appear for a long time. Writing safe threaded code takes careful analysis and planning.

source: Eli and JP Camara, https://x.com/logicalcomic

Writing safe threaded code can be difficult, but writing safe non-threaded Ruby code in a threaded environment doesn’t have to be. There are some top mistakes I’ve seen (or personally made) to keep an eye out for in your own code, and code in gems you bring into your projects. Let’s take a look at them, and we’ll learn some core Ruby/CRuby principles along the way as well.

Threading mistakes

📝 For brevity, I’ll be referring to instance variables as “ivars”

If there’s a classic example of what not to do for thread safety, it’s maintaining and modifying class-level ivars⁵.

Not knowing that, you embark on implementing some file processing code. Finding yourself in need of code that splits a file into chunks by newline, you write the following:

class FileSplitter
  def self.split(file_path, max_lines)
    lines = []
    files = []
    File.open(file_path) do |file|
      file.each_line do |line|
        lines << line
        if lines.size > max_lines
          output = Tempfile.new
          lines.each do |line|
            output.write(line)
          end
          lines = []
          files << output
        end
      end

      # handle any remaining lines
      if lines.size > 0
        output = Tempfile.new
        lines.each do |line|
          output.write(line)
        end
        files << output
      end

      files
    end
  end
end

The split method takes a file path, a number of max lines per file, and returns an array of Tempfiles. Each Tempfile contains a chunk of the original file. In some cases you upload each chunk to a file service like S3, in other cases you hand them off to methods for further processing⁶.

The method is kind of hefty, and there’s some clear duplication so you decide to refactor.

source: https://theycantalk.com/post/710363232083361793/plans

You decide the logical split in the code is around writing to the temp files, a new method you’ll call write_lines. There are so many variables that it just seems easier to use ivars instead of passing everything around.

class FileSplitter
  def self.split(file_path, max_lines)
    @lines = []
    @files = []
    File.open(file_path) do |file|
      file.each_line do |line|
        @lines << line
        write_lines(max_lines)
      end
      # force remaining to write
      write_lines
      @files
    end
  end

  def self.write_lines(max = nil)
    return if @lines.empty?

    if max.nil? || @lines.size > max
      @output = Tempfile.new
      @lines.each do |line|
        @output.write(line)
      end
      @lines = []
      @files << @output
    end
  end
end

Since the method is at the class level, the ivars are class level as well. For any code calling these methods, they only exist once. Every thread shares the same class object.

📝 If this doesn’t look like class-level ivars, you may be used to the more explicit syntax using double at-signs. For example, @@lines. These are equivalent, but when you’re inside a class method, class-level ivars only need one at-sign. When inside a class method, the instance is the class itself.

~~Another format of this would be a class level attr_accessor. So if you see a class method with self.lines = [] or see code like FileSplitter.lines = [], these can also be class level ivars!~~

Some commenters have pointed at that the above is not accurate - until I update it more accurately, see the ruby lang docs for more information about class level instance variable differences

You are running this code in a background job using Sidekiq, running with multiple threads on CRuby 3.3. Most of the time the job runs fine, but you’ve received a few reports from people missing data, and some people even seeing data that wasn’t in their files! Uh oh, what is happening?!

The problem is that every thread is sharing the same data, and modifying it concurrently. If multiple jobs are running and calling FileSplitter.split, they are all trying to write to the same piece of memory. We can follow along with the code line by line to see where things break down:

### Thread 1
# FileSplitter.split(path, max) ->
@lines = []
@files = []
# ...
@output = Tempfile.new
@output.write(line) # `write` causes the thread to sleep so Thread 2 starts

### Thread 2
# Calling `split` resets the instance variables
# Now each thread is using the same variables!
# FileSplitter.split(path, max) ->
@lines = []
@files = []
# ...
# Next time it wakes up, Thread 1 will be appending to the same array as Thread 2 💀 
@lines << line
# ...
# Thread 1's Tempfile is overwritten and lost, along with anything written to it 👋🏼
@output = Tempfile.new
@output.write(line) # Now `write` causes Thread 2 to sleep

### Thread 1 wakes up and continues processing
# The next line from Thread 1 gets written to the output file from Thread 2
# We are now 100% off the rails...
# Pour one out for our mix and matched 
# user data ☠️☠️☠️
@lines.each do |line|
  @output.write(line)
end

Or the same flow, visualized:

source: jpcamara.com

The simplest advice about the above code?

Don’t use shared instance variables in classes, ever 🙅🏻‍♂️. Assume that a class level ivar is a code smell, until proven otherwise.

📝 There is a caveat to this, and its configuration and initialization class level ivars. It is common to set global configuration values and macros⁷ using class level ivars. The difference is that these are written once on load of your application, and then only ever read past that point. If they are not properly frozen you could still corrupt them by modifying them during program execution. Which is why things like Rails middleware freeze the middleware array to try and avoid this issue.

How can we make this code thread safe? Use regular Ruby instances. We don’t have to change much.

class FileSplitter
  def initialize(file_path, max_lines)
    @file_path = file_path
    @max_lines = max_lines
    @lines = []
  end

  def split
    @files = []

    File.open(@file_path) do |file|
      file.each_line do |line|
        @lines << line
        write_lines(max_lines)
      end
      # force remaining to write
      write_lines
      @files
    end
  end

  private

  def write_lines(max = nil)
    return if @lines.empty?

    if max.nil? || @lines.size > max
      # Also remove @output as an ivar,
      # we only use it right here so it
      # being an ivar wasn't necessary 
      output = Tempfile.new
      @lines.each do |line|
        output.write(line)
      end
      @lines = []
      @files << output
    end
  end
end

Now when you use this code, each thread creates its own FileSplitter:

### Thread 1
splitter = FileSplitter.new(path, max)
splitter.split

### Thread 2
splitter = FileSplitter.new(path, max)
splitter.split

Each thread now safely uses its own independent, isolated instance 😌.

Heisenbugs

source: https://thedailyomnivore.net/2012/11/30/heisenbug/

A heisenbug is a software bug that seems to disappear or alter its behavior when one attempts to study it.

The term is a pun on the name of Werner Heisenberg, the physicist who first asserted the observer effect of quantum mechanics, which states that the act of observing a system inevitably alters its state.

The effects of sharing class-level ivars can be deceptive. For instance, try running the following:

class Result
  attr_accessor :value
end

class Fibonacci
  class << self
    attr_accessor :result

    def calculate(n)
      self.result = Result.new
      self.result.value = fib(n)
    end

    def fib(n)
      return n if n <= 1
      fib(n - 1) + fib(n - 2)
    end
  end
end

answers = [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144]

answers.size.times.map do |n|
  Thread.new do
    Fibonacci.calculate(n)
    answer = answers[n]
    result = Fibonacci.result.value

    if result != answer
      raise "[#{result}] != [#{answer}]"
    end
  end
end.map(&:join)

Here we have a naive⁸, basic Fibonacci generator, and we verify each result against an array of precomputed results. We spawn as many threads as there are precomputed results to calculate against. We’re storing the result in a class-level ivar that is being shared across threads - that shouldn’t end well! If any results don’t match, we raise an error.

Run this example and… it doesn’t fail in CRuby 🤔. This might lull you into thinking it isn’t an issue.

Is CRuby somehow thread safe? No, we’ve already established with the FileSplitter example that it is not inherently thread safe.

Maybe that Global VM Lock (GVL) you’ve heard about is protecting you from this issue? Unintentionally, yes.

In CRuby, only one thread running Ruby code can run at a time. That means for our example you can’t get our class data to overwrite without what’s called a “context switch”: something that would cause the Ruby runtime to swap between our threads, even mid operation like in a method call or variable assignment.

There are two common reasons context gets switched between threads in CRuby, which can result in operations only partially completing (ie, setting the proper result, then checking that result):

~100ms of Ruby processing have elapsed
A blocking operation has been invoked

Neither of these cases are met by our example. You’re benefiting from low overhead and a lack of blocking operations - there isn’t enough going on to cause a context switch. Without (1) or (2), each thread runs and completes in entirety before the next thread runs.

📝 If you don’t know what the GVL is - or only know it from a high level - you’ll learn all about it in “Concurrent, colorless Ruby” later on

I titled this section “Heisenbugs” because you can get the bug to happen by altering things in what would appear to be innocuous ways. We’ll try two things:

For (1), we’ll try with more calculations:

answers = [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2584, 4181, 6765, 10946, 17711, 28657, 46368, 75025, 121393, 196418, 317811, 514229, 832040, 1346269, 2178309, 3524578, 5702887, 9227465, 14930352, 24157817]

answers.size.times.map do |n|
  Thread.new do
    Fibonacci.calculate(n)
    answer = answers[n]
    result = Fibonacci.result.value

    if result != answer
      raise "[#{result}] != [#{answer}]"
    end
  end
end.map(&:join)

💥! We get an error raised! So what happened?

[] != [832040] (RuntimeError)

In the recursive Fibonacci solution, the code gets exponentially slower the more numbers we try to generate. And we now know when pure Ruby code is being run, the CRuby thread scheduler only allows each thread to run in 100ms time slices. This is so each thread can share processing time.

📝 CPU intensive work doesn’t run more efficiently with threads in CRuby, but you will still encounter it on threaded servers.

We can see this time slicing in action with the gvl-tracing gem, which creates a timeline of thread context switching using the CRuby GVL Instrumentation API:

require "gvl-tracing"

GvlTracing.start("timeline.json") do
  answers.size.times.map do |n|
    Thread.new do
      Fibonacci.calculate(n)
      answer = answers[n]
      result = Fibonacci.result.value

      if result != answer
        raise "[#{result}] != [#{answer}]"
      end
    end
  end.map(&:join)
end

source: jpcamara.com

Our threads’ running slices form a sort of waterfall because of the GVL. They spend most of their time in the wants_gvl state, which means they’re ready for work but are waiting to acquire the GVL. When they run, they can only hold the GVL for around 100ms of Ruby processing before the CRuby scheduler passes control to another thread. You see each thread in a running state for small slices of time - the highlighted thread ran 97ms before having control pass back. As the Fibonacci numbers get larger, the threads need more 100ms slices to finish, which is why each thread has so many running blocks.

It’s also why we start to hit issues with our class level ivars. The scheduler can stop a thread mid operation, or in between variable assignments. It takes more iterations than shown, but we can visualize the issue like this:

source: jpcamara.com

Can we try a bit harder and get our original example to break as well? Even though it has low throughput?

Let’s create a helper to simulate some load.

require "concurrent-ruby"

def run_forever(pool_size: 10)
  pool = Concurrent::FixedThreadPool.new(
    pool_size
  )
  i = Concurrent::AtomicFixnum.new

  loop do
    pool.post do
      yield i.increment
    end
  end
end

In the run_forever method we utilize a gem called concurrent-ruby to endlessly run our code across multiple threads. It uses a pool of 10 threads, and each time a thread runs it maintains a thread-safe counter which it yields as the current iteration. We can now use it to run our original Fibonacci example:

answers = [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144]

run_forever do |iteration|
  n = iteration % answers.size
  Fibonacci.calculate(n)
  answer = answers[n]
  result = Fibonacci.result.value

  if result != answer
    raise "[#{result}] != [#{answer}]"
  end
rescue => e
  puts "Iteration[#{iteration}] #{e.message}"
end

💥. This time, we start seeing errors!

Iteration[36981] [13] != [34]
Iteration[154569] [89] != [144]
Iteration[173571] [89] != [21]
Iteration[197573] [] != [144]
Iteration[199483] [] != [89]
Iteration[201395] [] != [144]
Iteration[203330] [] != [55]
Iteration[207180] [5] != [144]
Iteration[209117] [5] != [144]
Iteration[211039] [5] != [55]
Iteration[214849] [5] != [89]
Iteration[234986] [8] != [89]
Iteration[221961] [1] != [144]
Iteration[223872] [1] != [144]
Iteration[225767] [1] != [34]
Iteration[244243] [0] != [144]
Iteration[218269] [1] != [144]
Iteration[236807] [8] != [144]
Iteration[240446] [55] != [89]
Iteration[231318] [1] != [34]
Iteration[246175] [0] != [13]

For (2), let’s try it again but call puts, a blocking operation:

answers.size.times.map do |n|
  Thread.new do
    Fibonacci.calculate(n)
    puts "Get ready to check for #{n}!"
    answer = answers[n]
    result = Fibonacci.result.value

    if result != answer
      raise "[#{result}] != [#{answer}]"
    end
  end
end.map(&:join)

puts gives us a pretty reliable failure:

[144] != [0] (RuntimeError)
Get ready to check for 1!

The same basic idea happens - we perform the calculation, but before checking the result we print the message Get ready to check for #{n}!. This is a blocking operation in CRuby⁹ - it causes our thread to yield and another thread to get scheduled. When the original thread regains control there’s a good chance the value has been swapped out and the comparison will fail. This is the main reason our original FileSplitter code failed so reliably - the Tempfile write method is a blocking operation and caused a context switch.

Ruby internals

Interesting side note, this was my original code for this section, which I initially tried on CRuby 3.2.

class Repeater
  class << self
    attr_accessor :result

    def repeat_by(content, n)
      self.result = [content] * n
    end
  end
end

100.times do
  100.times.map do |n|
    Thread.new do
      Repeater.repeat_by("hello", n)
      array_size = Repeater.result.size
      if array_size != n
        raise "[#{array_size}] should be [#{n}]?"
      end
    end
  end.map(&:join)
end

[88] should be [98]? (RuntimeError)

That code failed reliably on Mac and Linux on Ruby 3.2. But on Ruby 3.3, I couldn’t get it to fail no matter how many times I ran it. Even between minor Ruby versions the thread internals can change enough to invalidate an evaluation of unsafe code. In this case the code benefited from the change, but there are no guarantees the next version won’t fail even more frequently than 3.2¹⁰.

Using our run_forever helper, however, we can eventually get this to fail on CRuby 3.3:

run_forever do |iteration|
  n = rand(100)
  Repeater.repeat_by("hello", n)
  array_size = Repeater.result.size
  if array_size != n
    raise "[#{array_size}] should be [#{n}]?"
  end
rescue StandardError => e
  puts "Iteration[#{iteration}] #{e.message}"
end

Iteration[323650] [52] should be [35]?
Iteration[408269] [59] should be [18]?
Iteration[563087] [51] should be [16]?
Iteration[623992] [8] should be [91]?

It takes hundreds of thousands of attempts but it does ultimately fail. But why did running it so many times cause it to fail? There’s no blocking operation. It shouldn’t take 100ms to run. In “Concurrent, colorless Ruby” we’ll dig even further into other reasons your thread context switches.

Back in reality...

The community has a much better understanding of this type of issue now vs 10+ years ago. But it still happens and I’ve seen it recently in gems, even ones that are well maintained.

Watch out for this in your own code. Watch out for this in code reviews. Watch out for this in gems. If you see it and it seems to run ok, use run_forever and you’ll likely see it fail.

Assume shared class instance variables are a bad idea. Knowing that you try something new…

Copying state to instances

You’re maintaining the please_encrypt gem (a very polite encryption layer), and you have a class level macro that defines which attributes to encrypt:

class Member
  include PleaseEncrypt
  please_encrypt :home_address, :phone
end

Learning from your FileSplitter issue, you’ve removed all of your shared class level ivars in the gem. But you’ve just added a new class level ivar that you’re sure is safe.

You know not to share class-level instance variables, but you are now using some for default values. The macro defines class level ivars containing metadata about the encryption. It’s only defined once on class load, so you won’t be contending with threads overwriting each other¹¹.

When an instance uses the encryptables metadata, you dup the original value so nothing gets shared by accident:

# member.rb
class Member
  include PleaseEncrypt
  please_encrypt :home_address, :phone
end

# please_encrypt.rb
module PleaseEncrypt
  # ...
  def pleasant_options
    # `dup` to avoid sharing across threads
    @pleasant_options ||= self.class.encryptables.dup
  end

  def pleasant_attributes
    @pleasant_attributes ||= # ...
  end

  module ClassMethods
    attr_accessor :encryptables

    def please_encrypt(*fields)
      self.encryptables = {}
      fields.each do |field|
        self.encryptables[field] = {
          algorithm: "AES-256-GCM",
          key_method: :generate_key
        }
        
        define_method("#{field}=") do |value|
          result = encrypt(field, value)
          pleasant_attributes[field][:encrypted] = result
        end

        define_method(field) do
          decrypt(
            field,
            pleasant_attributes[field][:encrypted]
          )
        end
      end
    end
  end

  def encrypt(field, value)
    options = pleasant_options[field]
    cipher = OpenSSL::Cipher.new(
      options[:algorithm]
    )
    
    options[:key] = send(options[:key_method])
    cipher.encrypt
    # ...
  end

  def decrypt(field, value)
    options = pleasant_options[field]
    # ...
  end
  # ...
end

We’ll look at the full code later, but this highlights the major points. When please_encrypt is called it takes each symbol and makes them into readers and writers. When setting the field the value is encrypted. When getting the field the value is decrypted. Pleasant!

You release a new version of your gem and a few weeks after release you start seeing issues open up!

😥

😰

😱

You haven’t seen anything like this in your testing and can’t manage to reproduce it. What is going on?

A user has reported the only way they can reliably reproduce it. It requires running code using please_encrypt inside of sidekiq jobs for 3-5 minutes straight.

class EncryptionFailureJob
  include Sidekiq::Job

  def perform
    m = Member.new
    m.phone = "123-456-7890"
    m.phone
  end
end

# sidekiq -C 10
loop do
  EncryptionFailureJob.perform_async
end

The user can “fix” the error by forcing a retry in the code:

def perform
  m = Member.new
  m.phone = "123-456-7890"
  m.phone
rescue OpenSSL::Cipher::CipherError
  puts "retrying..."
  retry
end

You can use this code to reproduce the issue, and it always succeeds within one retry 👀.

Running this way, after a few minutes you usually see an OpenSSL::Cipher::CipherError. Yay…

But why does it take so long? Maybe the overhead of Sidekiq interacting with Redis and managing the job server is reducing load. You know how you can get some load - run_forever!

run_forever do |iteration|
  m = Member.new
  m.phone = "123-456-7890"
  m.phone
rescue => e
  puts "Iteration[#{iteration}] #{e.class}" 
end

Doing that, you’re able to raise an OpenSSL::Cipher::CipherError pretty quickly:

Iteration[3591] OpenSSL::Cipher::CipherError
Iteration[8093] OpenSSL::Cipher::CipherError
Iteration[12229] OpenSSL::Cipher::CipherError
Iteration[13368] OpenSSL::Cipher::CipherError
Iteration[18023] OpenSSL::Cipher::CipherError
Iteration[38276] OpenSSL::Cipher::CipherError
Iteration[63144] OpenSSL::Cipher::CipherError

You look through recent changes. Luckily, in your case the most recent change you made was your attribute dup, so you look deeper into that code. Is something wrong with the ||= maybe?

Here’s the full source code. Can you find the issue?

⚠️ This is working code so you can try it to replicate the issue. It will properly encrypt and decrypt. But it is not meant for real use! Use real, vetted libraries for anything encryption/security related 🔐

module PleaseEncrypt
  def self.included(base)
    base.extend ClassMethods
  end
  
  module ClassMethods
    attr_accessor :encryptables

    def please_encrypt(*fields)
      self.encryptables = {}
      fields.each do |field|
        self.encryptables[field] = {
          algorithm: "AES-256-GCM",
          key_method: :generate_key
        }
        
        define_method("#{field}=") do |value|
          result = encrypt(field, value)
          pleasant_attributes[field][:encrypted] = result
        end

        define_method(field) do
          decrypt(
            field,
            pleasant_attributes[field][:encrypted]
          )
        end
      end
    end
  end

  def encrypt(field, value)
    options = pleasant_options[field]
    cipher = OpenSSL::Cipher.new(
      options[:algorithm]
    )
    
    options[:key] = send(options[:key_method])
    cipher.encrypt
    options[:iv] = cipher.random_iv
    cipher.key = options[:key]
    encrypted = cipher.update(value) + cipher.final
    options[:auth_tag] = cipher.auth_tag
    Base64.encode64(encrypted)
  end

  def decrypt(field, value)
    options = pleasant_options[field]
    cipher = OpenSSL::Cipher.new(
      options[:algorithm]
    )
    
    data = Base64.decode64(value)
    cipher.decrypt
    cipher.iv = options[:iv]
    cipher.auth_tag = options[:auth_tag]
    cipher.key = options[:key]
    cipher.update(data) + cipher.final
  end

  def pleasant_options
    @pleasant_options ||= self.class.encryptables.dup
  end

  def pleasant_attributes
    @pleasant_attributes ||= hash_defaulter
  end

  def as_json
    pleasant_attributes.inject(hash_defaulter) do |hash, (k, v)|
      hash[k][:encrypted] = v[:encrypted]
      hash[k][:iv] = pleasant_options[k][:iv]
      hash[k][:auth_tag] = pleasant_options[k][:auth_tag]
      hash
    end
  end

  private

  def generate_key
    OpenSSL::Random.random_bytes(32)
  end

  def hash_defaulter
    Hash.new do |h, k|
      h[k] = {}
    end
  end
end

class Member
  include PleaseEncrypt
  please_encrypt :home_address, :phone
end

You print the options and attributes, which is when you see the following:

m = Member.new
m.phone = "123\n456\nok!"
m.home_address = "..."
puts m.pleasant_attributes
puts m.pleasant_options

{
  home_address: { ... },
  phone: { ... }
}

Oh no… you’re dup’ing the class-level data (good!), but because dup only performs a shallow copy, you are sharing nested objects (bad!).

members = [
  Member.new,
  Member.new,
  Member.new
]
puts 'options id'.ljust(12) +
     ' | home_address id'.ljust(18) +
     ' | phone id'.ljust(12)
puts '-----------------------------------------'

members.each do |member|
  opts = member.pleasant_options
  puts opts.object_id.to_s.ljust(15) +
       opts[:home_address].object_id.to_s.ljust(18) +
       opts[:phone].object_id.to_s.ljust(12)
end

This means that per class, per attribute, every thread is modifying the same hash 😩, even across what seemed like independent instances. When we print the object_id on each of the hashes, we see that the top-level hash is unique, but the nested hashes are all the same object.

options id   | home_address id | phone id 
-----------------------------------------
2000           1920              1960        
2020           1920              1960        
2040           1920              1960

But why does it take high load to break? You have your suspicions…

Threading issues in CRuby can be ticking time-bombs. The GVL blocks parallel execution of Ruby code, so it takes more effort to force the right timing for unsafe context switches. If you see code that seems like it’s not thread safe but “works” - assume it will eventually break.

The fix is frustratingly simple:

def pleasant_options
  @pleasant_options ||= begin
    duplicate = {}
    self.class.encryptables.each do |k, v|
      duplicate[k] = v.dup
    end
    duplicate
  end
end

Now you are duplicating the nested hashes as well. With this change, you no longer share any data between the class and the instances. Rerunning the run_forever example there are no more errors 🙌🏼.

This does leave you susceptible to deeply nested hashes being shared. If you’re using ActiveSupport or Rails, I’d recommend using deep_dup, which handles nested hashes. If not, you could write a recursive method to handle it, or check for a gem (there seem to be many).

Back in reality... part two

If this whole scenario seemed oddly specific, it’s because it is based on a real gem issue. This exact problem remained an open issue for years in the attr_encrypted¹² gem without anyone identifying the issue. All of the GitHub issue images are from real issues - running in Sidekiq and usually under load they saw intermittent encryption errors.

How could this have been avoided? The simplest answer will sound familiar: don’t share class-level state. Even though in this case the sharing was unintentional, the macro metadata could have instead been used separate rather than trying to mix it into the instance:

def encrypt(field, value)
  options = pleasant_options[field]
  cipher = OpenSSL::Cipher.new(
    options[:algorithm]
  )
    
  options[:key] = send(self.class.encryptables[field][:key_method])
  # ...
end

def decrypt(field, value)
  options = pleasant_options[field]
  cipher = OpenSSL::Cipher.new(
    self.class.encryptables[field][:algorithm]
  )
    
  # ...
end

def pleasant_options
  @pleasant_options ||= {}
end

This is easy to see in hindsight but is very easy to overlook in a larger code base.

attr_encrypted is not Rails specific, technically. But if you’re in the Rails world using ActiveRecord you should use the built-in encryption instead.

Cleaning up thread state

😮‍💨

Now you’re thinking “is there a way I can avoid all of these headaches?”. You’re so tired of these class level ivars and their surprising behaviors. You’ve heard about something called “thread-locals” that sound really promising. They stay silo’d to your thread and can even act “global”¹³ without any chance of sharing¹⁴ 😍.

As a test, you use them on the Fibonacci example from earlier and can’t reproduce any issues 😌. Within the thread its global, but each thread has its own local state, isolated from the others.

class Fibonacci
  class << self
    def result
      Thread.current[:fib_result]
    end

    def result=(value)
      Thread.current[:fib_result] = value
    end

    def calculate(n)
      self.result = Result.new
      self.result.value = fib(n)
    end

    def fib(n)
      return n if n <= 1
      fib(n - 1) + fib(n - 2)
    end
  end
end

source: jpcamara.com

No class level ivars ✅ , no duping ✅. Feeling confident that it’s a good approach when you need something global, you put this solution in your mental back pocket.

Now you’re writing a new application - It’s a Rails app for sharing articles (because we don’t have enough of those), with a User and Article model:

class User < ApplicationRecord
  has_many :articles
end

class Article < ApplicationRecord
end

It’s a typical CRUD web application, and a couple requirements come up:

Track the origin of a change. If an article is added, updated or deleted, a record is created to keep track of who did it and when.
Add some I18N. You’ve got Japanese and Polish users going wild for this highly original site.

There’s so many places you could need to access this information, you decide to store it in a thread-safe, global way using thread-locals 🙌🏼.

class AppContext
  class << self
    def user
      Thread.current[:app_user]
    end

    def user=(user)
	  Thread.current[:app_user] = user
      self.locale = user.locale
    end

    def locale
      Thread.current[:app_locale]
    end

    def locale=(locale)
      Thread.current[:app_locale] = locale
    end
  end
end

📝 Not to spoil the ending, but ultimately I will recommend you avoid directly using Thread.current[] in favor of other solutions, particularly since we’ll find later that it being “thread-local” is not really accurate.

Just in case you get so excited at the idea of Thread.current[] you start feverishly writing code using it and never finish reading part 2 🏃‍♂️

You use the thread-local data to pass the user into the Change we save.

# app/controllers/application_controller.rb
class ApplicationController
  def set_user
    AppContext.user = User.find(cookies.encrypted["user_id"])
  end
end

# app/models/articles_controller.rb
class ArticlesController < ApplicationController
  before_action :set_user

  def create
    Article.create!(article_params)
  end
end

# app/models/change.rb
class Change < ApplicationRecord
  belongs_to :trackable, polymorphic: true
  belongs_to :user
end

# app/models/article.rb
class Article
  include Trackable
end

# app/models/concerns/trackable.rb
module Trackable
  extend ActiveSupport::Concern

  included do
    before_save :track_changes
    before_destroy :track_destroy
    has_many :changes, as: :trackable
  end

  private

  def track_changes
    action = if new_record?
      "create"
    else
      "update"
    end

    track_change(action)
  end

  def track_destroy
    track_change("delete")
  end

  def track_change(action)
    Change.create(
      trackable: self,
      action:,
      changes: attributes,
      user: AppContext.user
    )
  end
end

This seems to work really well. Well enough that you decide to expose a change history to your users so they can see edits that have been made to an article, and by who.

class ArticlesController < ApplicationController
  # ...

  def show
    @article = Article.find(params[:id])
  end
end

<!-- app/views/articles/show.html.erb -->
	
<h1><%= t('articles.changes.title') %></h1>
	
<table>
  <thead>
    <tr>
      <th><%= t('articles.changes.headers.article_id') %></th>
      <th><%= t('articles.changes.headers.action') %></th>
      <th><%= t('articles.changes.headers.changes') %></th>
      <th><%= t('articles.changes.headers.user') %></th>
      <th><%= t('articles.changes.headers.changed_at') %></th>
    </tr>
  </thead>
  <tbody>
    <% @article.changes.each do |change| %>
      <tr>
        <td><%= change.trackable_id %></td>
        <td><%= t("articles.changes.actions.#{change.action}") %></td>
        <td>
          <ul>
            <% change.changes.each do |attribute, value| %>
              <li><%= t("articles.attributes.#{attribute}") %>: <%= value %></li>
            <% end %>
          </ul>
        </td>
        <td><%= change.user.email %></td>
        <td><%= l(change.created_at, format: :long) %></td>
      </tr>
    <% end %>
  </tbody>
</table>

Things are going smoothly, but what good is posting articles without being able to discuss them?

# app/models/article.rb
class Discussion
  include Trackable
end

# app/controllers/discussions_controller.rb
class DiscussionsController < ApplicationController
  def create
    Discussion.create!(discussion_params)
  end

  def show
    @discussion = Discussion.find(params[:id])
  end
end

# assume a near identical "changes" view...

You test this out and it works on your machine 😬. Guess it’s time to deploy 🤷🏻‍♂️.

Except… once you deploy these changes, you start receiving some weird reports from users. Users will sometimes see discussions render in a totally random language. Even worse, some changes are showing up associated with unrecognized users!

Co się dzieje?!

何が起こっているの？！

What is happening?!

You analyze your database and logs, and notice something strange:

The changes are only ever wrong for the Discussion model
The internationalizations are only ever wrong in the discussions views

What’s the difference between ArticlesController and DiscussionsController?

class ArticlesController < ApplicationController
  before_action :set_user

  def create
    Article.create!(article_params)
  end
end

class DiscussionsController < ApplicationController
  def create
    Discussion.create!(discussion_params)
  end
end

Oh simple, there it is, you’re not setting AppContext.user in the discussions controller. But if it’s not getting set then why is it a different user and locale? Why is it set to anything at all?

You check the docs for thread-locals only to find…

They exist for the lifetime of the thread
You know that everything in Ruby runs in a thread
On multi threaded servers like Puma, threads are reused between requests so they can leak data you don’t clear out
Even on single threaded servers like Unicorn, you still run within the Ruby “main” thread (Thread.main). So you can even leak data you don’t clear out in single-threaded servers 🫠

source: jpcamara.com

Well that was pretty bad. Whenever a user went to the DiscussionsController, it would use information from whichever user used that thread last. It might be the same user, but it could just as easily be someone else.

You make sure they get set on every controller now, but also get cleared out in every controller too.

class AppContext
  # ...
  def clear
    [:app_user, :app_locale].each do |key|
      Thread.current[key] = nil
    end
  end
end

class ApplicationController
  after_action :clear_app_context

  def clear_app_context
    AppContext.clear
  end
end

class DiscussionsController < ApplicationController
  before_action :set_user
  # ...
end

Now if someone forgets to add it to their controller, it won’t be set to any user at all - at the end of every request the thread-local is cleared out so it won’t be present¹⁵.

You deploy the change and almost everything is good.

Locales are good! No more Polish for English users, or Japanese for Polish users.
When someone posts to a discussion, it consistently shows the right user in the change history 😮‍💨
But, there’s still an intermittent change showing up as the wrong user. What other code is there?

The controllers are good. How are the background jobs?

class FirstTimePostingJob < ApplicationJob
  def perform(user)
    AppContext.user = user
    Congrats.new(user).call
  end
end

class ImageUploadJob < ApplicationJob
  def perform(article, file_url)
    # Uploads to a service 
    article.upload_from_url(file_url)
  end
end

You’re setting the user in one job, and picking it up in another 🤦🏻‍♂️. When the article is updated to add an image using the file_url, sometimes the change is null, and sometimes it’s a totally random user. Hardly ever is it the actual user that added the image 😩.

The same thing you did with controllers, you can’t forget to do in jobs. You need to clean them up the same way (after an action/job finishes), but the logic goes in a new place.

If you're using Sidekiq

If you’re using a mixture of Sidekiq and ActiveJob, or Sidekiq alone, a Sidekiq Middleware is your best option. It will work for both job types because it is called internally by the Sidekiq server when performing a job.

class ContextClearMiddleware
  include Sidekiq::ServerMiddleware

  def call(_worker, _job, _queue)
    yield
  ensure
    AppContext.clear
  end
end

Sidekiq.configure_server do |config|
  config.server_middleware do |chain|
    chain.add ContextClearMiddleware
  end
end

If you're using ActiveJob

If you exclusively use ActiveJob, then an after_perform in your base job class should work just as well, and will work even if you’re using SolidQueue, GoodJob, or any other job adapter that hooks into ActiveJob.

class ApplicationJob < ActiveJob::Base
  after_perform :clear_app_context

  def clear_app_context
    AppContext.clear
  end
end

Next Up

source: jpcamara.com

This is the first step into this Ruby concurrency series. In Part 2 we’ll dig into a few more topics: true thread locals + fibers, the layered Ruby concurrency model, reusing objects, some more complex threading coordination, and ideas for keeping your code and your dependencies thread safe. More soon 👋🏼!

It doesn’t have to use threads by default, but there’s a good chance you are configured for some ↩︎
None of these servers force you to use multiple threads, but will usually use them by default. Puma’s OOTB configuration in rails uses 3 threads, for instance. https://github.com/rails/rails/blob/main/railties/lib/rails/generators/rails/app/templates/config/puma.rb.tt

They use some threads internally as well even if you don’t serve requests using more than 1 thread, possibly with the exclusion of Falcon since its core is the FiberScheduler. ↩︎
You also don’t have to use these job queues with threads either. But they all use them by default, or at least have internal threaded elements. ↩︎
You shouldn’t though.

http://blog.headius.com/2008/02/ruby-threadraise-threadkill-timeoutrb.html

https://www.mikeperham.com/2015/05/08/timeout-rubys-most-dangerous-api/

But if you’re using net-http, you’re using the timeout gem and have no choice.

There was an attempt to remove timeout from net http that was reverted:https://github.com/ruby/ruby/commit/f88bff770578583a708093f4a0d8b1483a1d2039

It IS safe to use when used inside a fiber scheduler like the async gem, because it swaps out the thread for a fiber timeout ↩︎
It’s ok to do once, up front, or controlled by mutexes. We’ll discuss that more in a bit. ↩︎
Sometimes you bake them into a cake for your friends 🎂. You just really love tempfiles 🥹. ↩︎
By “macros” I mean things like column encryption in Rails using encrypts. Internally it uses a class level instance variable called encrypted_attributes to track columns you are encrypting on that model.

But it is only set once when the class is loaded. If the class is reloaded, it must be locked by a mutex to safely change that value - Rails has a “reloader” that handles this, for instance. ↩︎
Naive because this is a terribly inefficient way to calculate Fibonacci sequences, which is why it’s a great example for demonstrating how CPU intensive work causes CRuby context switching ↩︎
And most other languages ↩︎
Ruby 3.3 introduced the M:N thread scheduler, so there was a ton of refactoring of pthread internals. It may have been more impactful to threading than other versions - but every minor version is a year long large-scale effort that can easily have an impact on these private, internal behaviors ↩︎
This assumes you’re only loading classes once or safely handle reloads across threads (like Rails does) ↩︎
The reproduction steps are even a modified form of one of the issues https://github.com/attr-encrypted/attr_encrypted/issues/372 and here is the code that fixed the issue https://github.com/attr-encrypted/attr_encrypted/pull/320/files ↩︎
I’ll leave it to others to discuss the merits or pitfalls of using any globals at all, thread-local or otherwise 🤷🏻‍♂️ ↩︎
They’re totally susceptible to the copying state issue we just discussed but at least you’re aware of it now ↩︎
This could be achieved in a rack middleware as well, as opposed to an after_action ↩︎

A comprehensive guide to PgBouncer/Postgres compatibility

Thu, 25 May 2023 21:23:46 -0500

My previous post, PgBouncer is important, useful, and fraught with peril, was a deep dive into Postgres feature compatibility with different modes of PgBouncer.

I’m happy with how it came out and it was well received. I think it is the most comprehensive guide to Postgres/PgBouncer compatibility that exists. But it might be a daunting read for some (9k words 😅), and the name doesn’t clearly convey what it offers - I want it to be easily found when searching for PgBouncer/Postgres pooling questions, and be quickly digestible on the fly.

Below is a list of topics you may be looking for more information about. While PgBouncer is the most popular option, most of it applies to any of the Postgres pooling options available (Supavisor, PgCat, Odyssey, etc).

Looking to understand how different PgBouncer modes work? See session mode, statement mode, and transaction mode
Want to understand connection pooling a little better? See this section on pooling approaches
Looking at the PgBouncer compatibility table and wanting more detail on the implications of each feature? See the next section.

SQL feature map for pooling modes

Below is each compatibility table feature linked to the original post. If there isn’t a link, it means the feature works the same way with or without PgBouncer.

Feature	Session pooling	Transaction pooling
Startup parameters*	Yes	Yes
SET/RESET statement_timeout	Yes	Never
SET/RESET lock_timeout	Yes	Never
LISTEN/NOTIFY	Yes	Never
WITHOUT HOLD CURSOR	Yes	Yes
WITH HOLD CURSOR	Yes	Never
Protocol-level prepared plans	Yes	Yes, with some gotchas**
PREPARE / DEALLOCATE	Yes	Never
ON COMMIT DROP temp tables	Yes	Yes
PRESERVE/DELETE ROWS temp tables	Yes	Never
Cached plan reset	Yes	Yes
LOAD statement	Yes	Never
Session-level advisory locks	Yes	Never

* Startup parameters are: client_encoding, datestyle, timezone, and standard_conforming_strings. PgBouncer detects their changes and so it can guarantee they remain consistent for the client.

** PgBouncer 1.21 introduced protocol-level prepared plan support, though there are some current gotchas which may be resolved in Postgres 17. You can find out more in the original post.

I’m biased, but I think it’s a pretty good read in entirety as well. There’s more sections on other gotchas, community pooling suggestions, and a look at work improving Postgres internals to lighten the need for poolers.

Thanks for reading, and I hope it answers your questions!

PgBouncer is useful, important, and fraught with peril

Wed, 12 Apr 2023 21:40:33 -0500

Updated 2024-09-17 to reflect updated PgBouncer support for protocol-level prepared statements 🐘

To start, I want to say that I’m appreciative that PgBouncer exists and the work its open source maintainers put into it. I also love working with PostgreSQL, and I’m thankful for the incredible amount of work and improvements that go into it as well.

I also think community and industry enthusiasm around Postgres is at an all time high. There are more managed hosting options than ever (Crunchy Data, Render, Fly.io, and on and on), deep extensions like PostgresML, Citus and Timescale, serverless options like Neon, and real-time services like Supabase with Postgres at their center. Postgres is a robust, advanced and fast RDBMS capable of handling the needs of most every application.

I just find the current state of recommendations and guidance around scaling Postgres to be confounding. And it feels surprising for new Postgres users to discover that one of the most common scaling options relies on a solution like PgBouncer.

Over the years I’ve read dozens of articles around scaling and maintaining Postgres databases, and they always understate the impact of PgBouncer on your application. They casually mention unusable features without any exploration, or the numerous ways you can silently break expected query behavior. The advice is just to turn it on. I want it to be clear that as your application scales, PgBouncer is often necessary but isn’t free.

The following sections provide an overview of what connection pooling is in general, how connection pooling modes work in PgBouncer and similar tools, and then I dig into every Postgres feature that does not work in PgBouncer transaction mode and what the implications are. This is the PgBouncer article I wish existed the first time I used it - let’s get going 🐘!

What is connection pooling?

PgBouncer is a lightweight connection pooler for PostgreSQL. What does that mean exactly? What is connection pooling and why is it needed?

Opening a connection is expensive: a new Postgres client connection involves TCP setup, process creation and backend initialization – all of which are costly in terms of time and system resources. A connection pool keeps a set of connections available for reuse so we can avoid that overhead past initial connection.

There are three main levels of connection pooling¹:

Framework connection pooling. This is a common feature of many frameworks/libraries. Within a given process, you maintain a pool of active connections that are shared between code, generally running across threads. Whenever you handle some processing in a server request, a background process, a job, etc, you open a connection and keep that connection open. When that piece of work finishes and a new piece of work starts, you can reuse the connection without the expense of opening a new connection to the database every single time. These connections are usually local to a particular operating system process, so you gain no benefit outside of that process (and if you’re scaling Postgres, you probably have lots of processes)

One level higher, you can have client level connection pooling outside of your code. PgBouncer can handle this, and instead of independent unsharable process-isolated connections you proxy all of your connections through PgBouncer. But it runs on your server, so you still cannot share connections between servers (and again, when needing to do it you probably have lots of servers).

Server level connection pooling. Here we host PgBouncer independent of our servers and connect to a single central PgBouncer instance². This is the most robust form of connection pooling because independent of anything else in your code or server, you are guaranteed that any client connection is coming from the pool.

Why do I need a separate tool from Postgres?

That’s all great but… why do we need it?

There are two primary layers to this:

Maintaining connections is beneficial as a base feature. Less memory and io churn, less latency before running queries. Less pressure on the database constantly opening and closing connections.
Postgres connections get expensive very quickly. Surprisingly quickly.

Here are some general community guidelines around allowable Postgres connection counts based on a mixture of community experience and specific benchmarking:

In terms of what some managed services even offer: Supabase offers a max of 50 connections, Neon offers a max of 100 connections, and Render offers a max of 397 connections.
The general upper bound recommendation is a max of 500 active connections. Services like Heroku Postgres even enforce a hard limit of 500 connections³
Even at 500 connections, your server is going to be strained. This more recent (as of 2023) enterprisedb article analyzed connection performance and found that 300-400 active connections seems optimal. This article from Brandur is older (2018) but seems to reinforce this idea as well
There have been some more recent connection improvements in Postgres (as of version 14) handling idle connections more efficiently⁴, but active connections are still expensive⁵ and idle connections have not reached the scale of a dedicated pooler
The reality of 500 connections is it sounds extremely low but those connections can handle a ton of throughput. The problem is, as a metric of pure concurrency, real connections have a hard upper limit. So if you try to have five thousand clients connect simultaneously, you’re going to start getting loads of connection errors⁶.

To improve the cost of connection overhead, general connection pooling is helpful and a PgBouncer instance in its default session based mode can handle it. But to improve concurrency things have to get a bit… quirky.

There are two modes in PgBouncer which give clients access to more connections than Postgres actually has available. They rely on the idea that at any given time many of your connections are idle, so you can free up usage of idle connections to improve concurrency.

Can I just turn on PgBouncer and get scaling for free?

Kind of? But not really? It’s complicated.

Internally, PgBouncer will manage a pool of connections for you. The default pooling mode it starts with, session pooling, is conservative, and in most cases will not provide improved concurrency⁷.

I’m going to hand wave a bit past two of the modes and focus on the typical recommendation.

Session mode is the default and most conservative mode. This is a 1:1 - your local connection truly holds onto a full connection until you close it. This does little to help you scale connection concurrency, but it helps with latency and connection churn overhead.

Statement mode is the most aggressive mode and means your connection goes back into the pool after every statement. You lose the ability to use transactions 😰 - that seems wild and unusable for only the most specialized of use cases.

The mode that results in a more sane balance of improved concurrency and retained critical database features is transaction mode. Transaction mode means your connection stays consistent as long as you’re in a transaction. Once the transaction finishes, your code thinks it still has real connection but PgBouncer actually releases the connection back into the pool internally. This is session sharing, your session is going to be shared with other connections without being reset or closed.

Transaction mode is a powerful concept. Your code in general has lots of database downtime. Most code does not solely operate on the database - it takes CPU cycles, interacts with files, makes network calls, and calls other data stores. During that time, your connection sits idle and unused for what in computing and database terms is an eternity. By releasing that back into the pool outside of transactions you free up your idle connection for use by a client who actually needs it. This way your 500 available connections can services thousands of clients, instead of a 1:1 with the number of available connections.

-- connection is actually pulled from the pool inside PgBouncer
BEGIN;
INSERT INTO...;
UPDATE;
COMMIT;
-- connection goes back to the pool inside PgBouncer

The problem with transaction mode is that this tiny configuration change quietly changes not only your ability to scale, but also the way your connections behave. It breaks the expected command semantics between client and database server. And understanding whether you’ve gotten things right in transaction mode is very difficult.

Let’s say you’ve been operating with PgBouncer in session mode (or operating without a proxy at all), and you make the switch to transaction mode. Your perspective on how you can use Postgres needs to change - so now we’re onto the peril.

Perils

Many of the following items are documented shortcomings of PgBouncer in transaction mode. But:

They’re treated lightly
Their repercussions and downsides are not discussed
PgBouncer is often recommended without mentioning them
PgBouncer is often recommended at the same time as recommending incompatible transaction mode features like session level advisory locks and session level statement timeouts
The non-determinism introduced by using incompatible statements is not discussed (ie, I execute a statement in Process A and suddenly Process B errors out due to it)

Assume anytime I mention PgBouncer after this point I am referring to transaction mode. Here we go!

Detecting invalid statements 😑

PgBouncer happily accepts statements that are not supported in transaction mode. The problem is pushed onto the developer, which means they can and will get it wrong⁸.

This is by design. The sense I get is that PgBouncer was specifically architected to not analyze any statements and so it would be a big change for them to handle this⁹.

Amazon has a similar tool to PgBouncer called RDS Proxy, and it has a feature called “connection pinning”. If it detects a statement that is incompatible with transaction mode, it will automatically hold that connection for that client for the duration of their session.

This is both highly useful and simultaneously problematic. It means query behavior is consistent with your expectations (🙌🏼) but also that you can silently kill all concurrency benefits (😑). If enough queries are run that trigger connection pinning, all of a sudden you may throttle your throughput. But it does give you an escape hatch for safely running statements which are not transaction compatible without having to jump through any hoops.

I’d be fine with some logging I could monitor. As far as I can tell there is nothing like this in PgBouncer, and so all the burden lands on you to get it right. As one engineer, or a few engineers, all aware of potential issues, you can probably maintain that. But what about dozens of engineers? Or hundreds? Thousands? All with varying levels of experience with databases and poolers? There’s going to be mistakes.

Lock Timeouts (SET/RESET) 🔓

Unless you like app downtime, you should be using a lock_timeout when running DDL. It’s a critical aspect of zero downtime migrations.

The idea is to set it to a limit that would be acceptable for queries in your application to slowdown by - waiting to acquire a lock can cause related queries to queue up behind your DDL operation:

-- slow select
SELECT * FROM my_table;

-- DDL starts in separate process, blocked on acquiring the lock by the 
--    slow query
ALTER TABLE my_table...

-- Subsequent queries start queuing up...
SELECT * FROM my_table WHERE id = 123;
SELECT * FROM my_table WHERE id = 234;
--- ...

In that scenario, the slow query is running the show. Until it finishes, all the other queries to that table are stuck. That goes on long enough and users can’t use the system. A bit longer and your app starts timing out. A bit longer you’re running out of connections. Now you’re staring at a total app outage, about ready to kill all of your connections in a desperate attempt to salvage things, contemplating a career change to landscaping where you can at most impact one person at a time, right? That sounds nice, doesn’t it?

I’ve of course never experienced that. I’m just very creative 💀. But if you have experienced that, or you’d like to avoid experiencing that, use a lock_timeout:

SET lock_timeout TO '2s';

Now if your DDL cannot acquire a lock it will throw an error after 2 seconds. That should be an ok delay in running queries, and you can retry the operation later.

But wait! Are you connected to PgBouncer?! You may want to bring up that landscaping help-wanted listing again… 🌳

SET operations apply at the session level. This means that on a PgBouncer connection, there is no guarantee our lock_timeout will still be applied when we run our DDL:

-- Process 1
-- PgBouncer pulls connection 1
SET lock_timeout TO '2s';
-- connection 1 goes back to the pool

-- Meanwhile, in Process 2:
-- PgBouncer pulls connection 3
SELECT id FROM my_table, pg_sleep(30);

-- Back in Process 1:
-- PgBouncer pulls connection 2
-- This connection has no lock_timeout set, so it will hang 
--    until our pg_sleep query finishes 30 seconds later, and all
--    queries to my_table after it are stuck for those 30 seconds as well
ALTER TABLE my_table...

It’d be easy to argue “don’t have slow queries”. And that should be the goal! But we don’t call it “happy path uptime 🌼”, we call it “zero downtime”. It means even if things go wrong, you don’t go down. There can also be other operations that hold a lock on your table, so you simply can’t rely on successfully acquiring that lock¹⁰.

So what can we do? There are two options:

Bypass PgBouncer and go straight to the database
Use a transaction level lock_timeout

Bypassing PgBouncer

Your safest bet is to go with option (1). You should have some ability to directly connect to your database, so take advantage of it and don’t jump through hoops to run DDL safely.

The biggest obstacle you hit with (1) is transparency: PgBouncer really doesn’t want you to know whether you are connected to the real database or not. There’s no easy answer there, but by validating a setup where you consistently run your DDL process directly against Postgres then you’re set.

Use transaction level statements

There is a transaction local equivalent to the SET statement: SET LOCAL. Using our example from earlier:

-- Process 1
-- PgBouncer pulls connection 1
BEGIN;
SET LOCAL lock_timeout TO '2s';
-- connection 1 stays checked out

-- Meanwhile, in Process 2:
-- PgBouncer pulls connection 3
SELECT id FROM my_table, pg_sleep(30);

-- Back in Process 1:
-- Connection 1 is still checked out
ALTER TABLE my_table...
-- lock_timeout raises an error after 2 seconds waiting, and 
--    we avoid our downtime!

DDL in Postgres is transactional, so it’s valid to start our transaction, set our lock_timeout using SET LOCAL, then start our DDL operation. Our transaction local setting will stick with us until the transaction commits or rolls back, so we safely keep our timeout and rollback our DDL.

It’s not a terrible solution (1 is still better), except for two things:

Concurrent indexes
Tooling

Another zero downtime star is the concurrent index. When you create a new index on a table you run the chance of locking it up long enough to cause downtime. Here’s the answer to that problem:

CREATE INDEX CONCURRENTLY index_name ON my_table;

Concurrent indexes are created without an exclusive lock, so your normal operations keep going while it builds the index in the background. The problem is they can’t be run in a transaction, so SET LOCAL is not an option.

Because they don’t require an exclusive lock, setting a lock_timeout is less important. But if there is contention and you just can’t get that index to acquire it’s shared lock, do you really want it to run forever?

As for (2), popular tooling usually does not handle SET LOCAL for you. In the Rails/ActiveRecord world there are several libraries that will automatically apply zero downtime policies for you, but they all assume you have an exclusive connection and operate at the SET session level.

In PgBouncer, the road to downtime is paved with session level statements.

Just go with (1), keep your sanity, throw away the diary entries about living out your days breathing in the smell of fresh cut grass, and connect directly to Postgres to run DDL with SET lock_timeout calls.

Statement timeouts (SET/RESET) ⏳

Determined not to repeat your experiences from lock_timeout, you read about this thing called statement_timeout. This little magic wand makes it so you control how long a statement is allowed to run 🪄.

So here it is:

SET statement_timeout TO '2s';

Those greedy queries now don’t stand a chance. You can tame your long running queries and avoid blocking your DDL! You ignore my advice to always use lock_timeout, say “bye losers” to long running queries, and fire off that DDL again… oh god. Why are things slowing down. Now they’re timing out. The connections are filling up. What is happening?

Oh riiiight. You forgot. You’re using PgBouncer. SET is off the table. Should have set that lock_timeout 🔐…

If I had a nickel for every time someone mentioned SET statement_timeout and PgBouncer in the same article…¹¹ I know no one sharing this content is doing it maliciously, but be aware that these are misleading and incompatible features.

With lock_timeout, why does statement_timeout even matter?

Statement timeouts are helpful for long running queries so they cancel earlier. If a client disconnects, Postgres will periodically check for the connection and try to cancel the query when it goes away. But a query with runaway cpu usage will just keep running even if the client dies or disconnects. That means you lose that connection until the query finishes, which can take minutes (or hours)
The database default is 0, which means there is no limit. In some contexts this is not a problem, but particularly for web requests this is excessive

The first time I used statement_timeout was from a blog recommendation to limit statements for requests in web applications. In a web request, you usually have an upper limit on how long you allow them to run before they time out - this conserves resources, protects against runaway buggy code and helps with bad actors. It made sense that I’d set it to something conservative on all my web connections to deal with long running queries.

I deployed the code and for a little while things seemed to work well. Then I saw something odd. This started popping up:

canceling statement due to statement timeout

But in my… background jobs? My web requests were tuned to be fast, but the constraints around my background processes were a bit… looser. Can you guess what I had recently enabled? PgBouncer in transaction mode. My session level statement timeout was being swapped out from my web request, picked up by my job, and caused my job to timeout instead - web request safety was off the rails and longer running jobs were intermittently failing.

So is there any way we can use it? There’s a couple ways I know of, but nothing great when pooling.

Our old friend transaction

BEGIN;
SET LOCAL statement_timeout '5s';
SELECT ...
COMMIT;

Something about wrapping a SELECT in a transaction feels kind of strange, but it works. If you have targeted concerns, you can wrap particular queries in a transaction and use SET LOCAL to localize the statement_timeout.

This is absolutely not a viable solution for a whole request lifecycle. If I wanted to attempt my web request level timeouts again, no way am I wrapping every web request in one giant transaction. Postgres doesn’t have a concept of nested transactions so any code I have that may be operating transactionally is gonna be in for some confusing surprises¹². And most importantly, wrapping my whole request in a transaction means I’ve completely negated the benefit of proxy pooling - now my request lifecycles are basically 1:1 with my connection sessions.

Apply statement timeouts per user

I’ve never tried it, but I’ve seen it recommended to set statement timeouts per user when using PgBouncer. That seems to have a couple problems I can think of:

It’s not dynamically configurable.
It dilutes the pool of available connections per context

(1) is definitely inconvenient. If you have different contexts where you’d like to apply different timeout constraints, this would be way too cumbersome to maintain.

But (2) feels like a deal breaker. If I want to constrain my web requests to a conservative timeout, but give my background processes more wiggle room, my pool size of real connections is now split instead of sharing a pool of total available database connections. I also have to manage making sure each context uses the appropriate user, or things will go badly.

It’s technically an option, but seems trickier to maintain and monitor.

Transparency 👻

I don’t understand why my session features aren’t working. I always make sure to use plenty of Postgr…PgBouncer?!

It is very difficult to tell when you are or aren’t using PgBouncer, which is unfortunately by design. It considers itself a transparent proxy. In session mode, that’s pretty much true. But in transaction and statement mode you are working with bizarro Postgres. It all works the same except when it doesn’t.

So if you want a regular connection because you need a feature not available in transaction mode, being sure you did it right is extremely difficult.

I have had a hell of a time verifying that some servers are or aren’t running with PgBouncer. Server A is using pub sub, I don’t want it. Server B needs the throughput, I want it. How can I make sure someone never makes a mistake and attaches the server to the wrong place? Basically, I can’t.

When it comes to production code I like to be paranoid. On a large enough codebase, and team, and user base, unusual things are bound to happen, sometimes regularly. I try to write code and configure environments so the right way is easy and the wrong way is hard. PgBouncer does not make that easy.

On this particular point I’d love to say I have some kind of advice to act on, but it mostly takes testing and validating your setup. If someone out there has better ideas or tips, I am all ears.

Prepared Statements (PREPARE/DEALLOCATE, Protocol-level prepared plans) ✔️

📝 Update as of PgBouncer version 1.21 - protocol-level prepared statements are now supported! See this crunchy data article for more specifics. PREPARE style statements will still never be supported, and there are still some gotchas with protocol-level support you can learn about in that article.

Everything mentioned here is still relevant for versions < 1.21. As well, it still gives a thorough explanation of why “turning off” prepared statements while still utilizing libpq (as most libraries do) is a-ok ✌️

PgBouncer has a public relations problem when it comes to prepared statements. This is all the PgBouncer docs say about them:

Feature	Session pooling	Transaction pooling
`PREPARE` / `DEALLOCATE`	Yes	Never
Protocol-level prepared plans	Yes	No*

* It is possible to add support for that into PgBouncer

Kind of feels… alarming. No prepared statements in transaction mode?! Aren’t those… important? Even further when you go to use PgBouncer with Hibernate or ActiveRecord (and I’m sure others) you’ll see the recommendation to configure them to turn off prepared statements 😱. Does it surprise you a bit to hear that? Make you feel a little queasy maybe?

I had it drilled into me early in my career that prepared statements are a critical part of protecting against SQL injection. In the OWASP SQL Injection Prevention Cheatsheet the very first recommendation is:

Use of Prepared Statements (with Parameterized Queries)

So PgBouncer tells me I need to turn them off?

The first time I used PgBouncer in an application I spent a lot of time figuring out how turning off prepared statements was safe to do. It turns out that prepared statements in Postgres mean a few things, but come down to two main options:

Named prepared statements
Unnamed prepared statements

Named prepared statements are reusable, and are tied to the connection session.

Unnamed prepared statements are single use, and have no association to the connection session.

There are two ways to create a named prepared statement and one way to create an unnamed prepared statement:

PREPARE
Protocol-level Parse/Bind/Execute with a name specified
Protocol-level Parse/Bind/Execute with no name specified

PgBouncer says it doesn’t support prepared statements in either PREPARE or protocol-level format. What it actually doesn’t support are named prepared statements in any form. That’s because named prepared statements live in the session and in transaction mode you can switch sessions.

-- PgBouncer pulls connection 1
PREPARE bouncer_since (int, timestamp) AS
SELECT * 
FROM bouncers b
INNER JOIN guests g ON g.bouncer_id = b.id
WHERE b.id = $1 AND b.created > $2;
-- connection 1 goes back to the pool

-- PgBouncer pulls connection 2
EXECUTE bouncer_since(1, now() - INTERVAL '2 days');
-- 💣 ERROR: prepared statement "bouncer_since" does not exist 💣

But unnamed prepared statements are totally fine. In fact, I’d be shocked if the current client library you’re using to connect to Postgres does not already switch to them if “prepared statements” (again, so damn misleading) are “turned off”.

But wait. What the heck is an unnamed statement? PREPARE requires a name… how can I make a prepared statement without a name?

Protocol-level prepared plans

The alternative to the PREPARE statement is to directly communicate with Postgres at the protocol level.

I had to dig a bit to get a handle on this - I started from a common Ruby ORM called ActiveRecord, dug into the Ruby “pg” gem it uses, then went one layer deeper into libpq, which is part of Postgres itself.

If we use active record as an example, when prepared statements are “disabled”, the postgres adapter internally calls exec_no_cache in activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb:

def exec_no_cache(sql, name, binds...)
  #...
  conn.exec_params(sql, type_casted_binds)

That’s powered by the ruby “pg” gem, which when calling exec_params from ruby ultimately calls into the libpq function PQsendQueryParams:

// Ruby "pg" gem
// ext/pg_connection.c
static VALUE
pgconn_async_exec_params(int argc, VALUE *argv, VALUE self) {}

// internally calls...
static VALUE
pgconn_send_query_params(int argc, VALUE *argv, VALUE self) {}

// internally calls this from the libpq c postgres internals:
// src/interfaces/libpq/fe-exec.c
int PQsendQueryParams(PGconn *conn,
  const char *command,
  int nParams,
  const Oid *paramTypes,
  const char *const *paramValues,
  const int *paramLengths,
  const int *paramFormats,
  int resultFormat) {}

What does PQsendQueryParams do? It calls an internal method named PQsendQueryGuts. Notice the empty string and use unnamed statement comment 🤔.

return PQsendQueryGuts(conn,
    command,
    "", /* use unnamed statement */
    nParams,
    paramTypes,
    paramValues,
    paramLengths,
    paramFormats,
    resultFormat);

What does that function do (aside from making me laugh every time I read the name PQsendQueryGuts 😆)? Internally PQsendQueryGuts communicates with Postgres at the protocol level:

/* construct the Parse message */
if (pqPutMsgStart('P', conn) < 0 ||
  pqPuts(stmtName, conn) < 0 ||
  pqPuts(command, conn) < 0) {}

/* Construct the Bind message */
if (pqPutMsgStart('B', conn) < 0 ||
  pqPuts("", conn) < 0 ||
  pqPuts(stmtName, conn) < 0) {}

/* construct the Execute message */
if (pqPutMsgStart('E', conn) < 0 ||
  pqPuts("", conn) < 0 ||
  pqPutInt(0, 4, conn) < 0 ||
  pqPutMsgEnd(conn) < 0) {}

This is the Parse/Bind/Execute process I mentioned earlier.

The code sends a Parse message with the query and an optional name. In our case the name is empty
The code then Binds params to that query (if the query is parameterized)
It then Executes using the combination of the parsed query and the bound params

This is perfectly safe to do in transaction mode, and from a SQL safety perspective should behave identically to a named prepared statement.

Named protocol-level statements

For comparison, when ActiveRecord has prepared statements turned on, things look a bit different, but by the end we’re in the same place:

def exec_cache(sql, name, binds...)
  #...pseudo coded a bit but importantly
  #   it calls `prepare`
  if !cached
    stmt_key = conn.prepare(sql)
  # then it calls exec_prepared
  conn.exec_prepared(stmt_key, type_casted_binds)

It first has to call prepare with whatever sql we’re going to run. The caller is in charge of keeping track of whether the sql has been prepared before, otherwise Postgres will keep overwriting our previous sql and it might as well just execute an unnamed statement. Then it calls exec_prepared with only the stmt_key, which should match the name of a previously prepared query.

If we skip ahead to what gets called in libpq:

// conn.prepare(sql)
int
PQsendPrepare(PGconn *conn,
    const char *stmtName, 
    const char *query,
    int nParams, 
    const Oid *paramTypes) {
  //...
  if (pqPutMsgStart('P', conn) < 0 ||
      pqPuts(stmtName, conn) < 0 ||
      pqPuts(query, conn) < 0) {}
  //...
}

We see something similar to our earlier Parse/Bind/Execute, but now we’re only calling the Parse portion and this time we have a stmtName. We then trigger the prepared statement calling exec_prepared, which ultimately calls PQsendQueryPrepared:

// conn.exec_prepared(stmt_key, type_casted_binds)
int
PQsendQueryPrepared(PGconn *conn,
    const char *stmtName,
    int nParams,
    const char *const *paramValues,
    const int *paramLengths,
    const int *paramFormats,
    int resultFormat) {
  //...
  return PQsendQueryGuts(conn,
      NULL,     // no sql
      stmtName, // named
      nParams,
      NULL,
      paramValues,
      paramLengths,
      paramFormats,
      resultFormat);
  //...
}

Anything look familiar? That’s the same PQsendQueryGuts function we called for the unnamed statement! This time it doesn’t hand a command in because we already parsed our SQL in the earlier prepare call. We also have a stmtName defined, instead of handing in an empty string. This version goes on to skip the Parse, call the Bind with the stmtName, then call Execute - same flow as our unnamed version.

For SQL injection safety, both named and unnamed versions are equivalent: they separate query structure (Parse) from data values (Bind). Adding query bindings when not in a prepared statement simply makes an unnamed statement.

Nothing about these calls is specific to the libpq library, it’s just a rock solid implementation of them¹³ - any language could make the same protocol calls. If a library is utilizing this protocol, they are doing the same things when binding to an unnamed prepared statement as they are when binding to a named prepared statement¹⁴.

As long as your code uses parameterized queries, “turning off” prepared statements for PgBouncer is safe, even if it seems a bit unnerving. There is a PR to allow PgBouncer to track prepared statements, so maybe this won’t cause people like me as much heartburn in the future 🥲.

Pool throughput / Long running queries 🏃‍♂️

We’ve got two types of connections to Postgres: active and idle. Idle connections are the backbone of poolers - having idle connections means we’ve got capacity to swap around transactions for connected clients. What about active connections?

An active connection means that connection is actively tied up by the database. For that timespan, the connection cannot be swapped out to do anything else until its operation completes. We know that active connections get expensive quickly, and we also know that most managed services range somewhere from 50 to 500 allowed total, non-pooled connections.

Using a max PgBouncer connection pool of 10k and Render’s managed Postgres service with a max of 397 total connections means we’d have:

10000 / 397 = ~25 connections per active connection

Using Supabase’s 50 connections the spread is even higher:

10000 / 50 = ~200 connections per active connection

That means that for every long running operation, you are potentially down 200 connections worth of pooling.

These numbers are very back of the napkin and of course do not represent the true scaling capability and connection handling of a real pooler. But the point is this:

Active connections are very valuable to a pooler
Long running queries disproportionally impact concurrency

As an example, you’re using Render Postgres fronted by PgBouncer and you’ve got 10k available connections backed by the max of 397 Postgres connections. Let’s say a new feature is introduced for populating some graph data on your app’s landing page. It’s powered by a new query that looks great, has indexes, and seems well optimized. It’s even run against some load testing and representatively sized data as a QA check. It gets deployed to production and OOF, it’s taking 15 seconds per query 🐌. Users are logging in or navigating to the landing page all the time so within moments you’ve had thousands of hits to this query. Obviously this is going to get quickly rolled back, but what does it mean for your pool in the meantime?

It means you’re maxed out. Your pooler being there means at least you’re less likely to start erroring out right away, but transaction mode can’t fix a stuck query. For each of those 15 second chunks of time your concurrency basically went from 10k back down to 397.

This is not the general behavior you’ll see when using PgBouncer unless you’ve really got some intermittent trouble with runaway queries. But it does emphasize an important point to remember: these are not real Postgres connections. Your upper bound on long running, active queries is always constrained by your actual pool of real Postgres connections.

Guarding against slow queries

Log your slow queries using log_min_duration_statement. This option lets you set a threshold and if queries take long than that threshold Postgres will log the offending query. This won’t help the sudden mass slow query situation mentioned above, but it helps to keep an eye on overall app query health
Use streaming queries sparingly. In most client libraries you can set your query to run in “single row mode”. This means you retrieve your rows one at a time instead of getting one big result set at once. This is helpful for efficiency with very large result sets but is slower than a full result set query, and probably means you are running queries large enough to be slower in the first place
Use statement timeouts. This is tricky, especially when pooling, but see that section for ideas on how to approach it
Spread out reads across read replicas

Session Level Advisory Locks 🔐

Session level advisory locks work fine in PgBouncer.

Sorry 🙈.

If you’ve read the previous sections you’ve already picked up on the pattern: “session” anything means it probably doesn’t work in transaction mode. But what does that matter to you?

Advisory locks are a great option for creating simple, cross process, cross server application mutexes based on a provided integer key. Unlike traditional locks you use/encounter elsewhere in Postgres which are tied to tables or rows, advisory locks can be created independent of tables to control application level concerns. There are plenty of other tools you could use for this job outside of Postgres, but since Postgres is already part of your tech stack it’s a convenient and simple option.

Across languages a common use case for session level advisory locks is to hold a lock while database migrations (ie, DDL) are being run. For example:

-- 1234 is arbitrary, it can be any integer
SELECT pg_advisory_lock(1234);
SET lock_timeout TO '1s';
ALTER TABLE my_table...;
INSERT INTO migrations VALUES (1234567);
-- If we don't explicitly unlock here, the lock will be held until this 
--    connection is closed
SELECT pg_advisory_unlock(1234);

If another connection went to acquire the same lock, it would be blocked:

-- This will block indefinitely until the other connection is closed, 
--    or calls pg_advisory_unlock(1234)
SELECT pg_advisory_lock(1234);

This is largely an attempt to improve consistency of migration tracking, and help coordinate multi process deploys:

Continuous deployment with the potential to trigger multiple deployments in succession
Propagating code changes to multiple servers with deploy scripts automatically triggering migrations in each context

By waiting to acquire a lock at the Postgres level, each process waits for the first lock owner to finish before continuing, coordinating each process based on a shared lock key.

Once more, with feeling PgBouncer

Now for the obligatory example of trying the same thing when connected to PgBouncer 🫠:

-- Grab the lock on connection 1
SELECT pg_advisory_lock(1234);
-- Connection 1 goes back into pool
-- ...
-- Try to unlock on connection 2, which does not own the 1234 lock
SELECT pg_advisory_unlock(1234);
-- WARNING: you don't own a lock of type ExclusiveLock

We try to unlock, but because we’re on a different connection we can’t. The lock stays locked for as long as connection 1 stays alive, which means now no one else can acquire that lock unless that connection naturally closes at some point or is explicitly pg_cancel_backended 😓.

More session advisory lock use cases

Outside of migrations, advisory locks can serve other use cases:

Application mutexes on sensitive operations like ledger updates
Leader election for maintaining a single but constant daemon operation across servers
Exactly once run job controls for Postgres based job systems like GoodJob and Que

If these things sound interesting or useful, they are! But only if you connect directly to Postgres.

Transaction level locks

Advisory locks do have a transaction based companion:

-- Process 1
BEGIN;
SELECT pg_advisory_xact_lock(1234);

-- Process 2 
-- Blocks while process 1 is in the transaction
SELECT pg_advisory_lock(1234);

-- Back in Process 1
SET LOCAL lock_timeout TO '1s';
ALTER TABLE my_table...;
INSERT INTO migrations VALUES (1234567);
COMMIT; -- automatically unlocks on commit or rollback
-- Process 2 now can acquire the lock

-- If you need to manually unlock while still in the transaction 
-- SELECT pg_advisory_xact_unlock(1234);

You could use it as a replacement for certain scenarios, like the above migration operating transactionally. For custom purposes, it’s a good alternative!

Unfortunately most migration tooling, things like leader election, and request or job lifetime locks, all use or require a longer lived lock than a single transaction could reasonably provide.

Turn off advisory migration locks

If you need to run migrations against PgBouncer, in Rails you can turn them off with an advisory_locks flag in database.yml. Other migration tools likely have something similar. Do it at your own peril 🤷🏻‍♂️

Maintaining a separate direct connection to Postgres

If the lock is critical, but the operations past the lock fan out and acquire multiple connections, you could potentially have two pieces:

A direct connection to Postgres where you acquire a session level advisory lock
Your normal code level connection pooling using your PgBouncer connections so it can capitalize on the scaling opportunities provided there

There’s an obvious downside - you’re consuming an extra direct connection and potentially impacting throughput - but it’s an alternative available if needed.

Listen / Notify 📣

Postgres comes out of the box with a handy pub/sub feature called LISTEN/NOTIFY.

You simply call:

LISTEN channel_name;

And that connection will receive NOTIFY events:

NOTIFY channel_name, 'hi there!';

Like session level advisory locks, there are more robust pub/sub solutions out there. But the Postgres implementation works well, and you already have it available in your stack.

Looking at the example, you’ll notice that the LISTEN call is just a single statement, and it activates the listener for the current session. What have we said so many times already? Sessions bad. Transactions good… kind of.

kind of?

Similar to prepared statements, the docs are misleading when it comes to LISTEN/NOTIFY.

PgBouncer officially lists LISTEN/NOTIFY as an unsupported feature in transaction mode, which is not precisely true. LISTEN does not work in transaction mode, but NOTIFY does.

NOTIFY is a single statement, and doesn’t rely on any session semantics. It’s also transactional¹⁵:

BEGIN;
NOTIFY channel_name, 'hi!';
ROLLBACK; -- no notification is sent

Both NOTIFY formats (inside and outside a transaction) work fine with transaction mode pooling. If you want to use pub/sub, you just need to make sure your LISTENer is connected directly to Postgres. Since it can be hard to tell if you’re connected to Postgres or PgBouncer this is somewhat tricky, unfortunately.

I’ve built implementations LISTENing on a non-PgBouncer connection and NOTIFYing on PgBouncer that work fine. There’s not much writing on this, but I have found this approach to work well.

The single thread 🪡

In contrast to the multi process monster that is Postgres, PgBouncer runs on a paltry single process with a single thread.

This means that no matter how capable a server is, PgBouncer is only going to utilize a max of one CPU core so once you’ve maxed out on that core you can’t scale that single instance anymore.

A popular option is to load balance PgBouncer instances. Otherwise, almost every alternative to PgBouncer (like Odyssey, PgCat and Supavisor) utilize multiple cores.

If you’re using a managed Postgres service (like Crunchy Data, Supabase, Neon or Heroku), your default option is going with PgBouncer as a connection pooler - so it will be up to those services to offer a load balanced option.

pg_dump 🚮

If you’re running pg_dump against PgBouncer, it’s probably by mistake.

As far as I can tell, pg_dump is broken when run against PgBouncer. See https://github.com/pgbouncer/pgbouncer/issues/452.

The answer here is to make sure you’re using a direct connection to Postgres for utility operations like pg_dump.

Other unavailable features 🫥

There are some remaining features which transaction mode is incompatible with as well¹⁶. I have less or no experience with these:

WITH HOLD CURSOR - A WITH HOLD continues to exist outside of a transaction, which seems like it could have handy use cases but I’ve never personally used it in my day to day.
PRESERVE/DELETE ROWS temp tables - temporary tables are a session level feature so will not work properly, and preserve/delete rows are modifiers on how those temporary tables behave on commit, and are unsupported
LOAD statement - this is for loading shared libraries into Postgres, so it makes sense this is not something you should be doing through a pooler. I haven’t actually tried, so I’m not sure if PgBouncer would stop you, but it requires super user privileges so it’s very unlikely that’s what your PgBouncer user has

PgBouncer documents a simple “SQL feature map for pooling modes” where you can see all the features mentioned in this post.

Linting 🧶

Aside from having identified potential issues - what can we do to avoid them in an automated way?

Surprisingly, not much exists. And by not much, I mean i’ve found nothing outside of advice.

It makes me feel a bit like I’m exaggerating the importance of these issues. Maybe I’m the oddball that has actually encountered many of them in real production usage and had to address them. I’ve had statement timeouts and lock timeouts misapplied. I’ve had to deal with rearranging connections because of code using a session advisory lock and LISTEN/NOTIFY, or drop libraries that use them. I’ve had to remember to turn off prepared statements in my ORM to avoid named prepared statement errors.

The implications can feel small, but they can be surprising and particularly around migrations can cause real serious downtime.

We lint everywhere. As engineers we try to automate away as many mistakes as possible with linting and specs. As development teams grow, the importance of automation becomes critical to scaling because otherwise someone somewhere is going to do the wrong thing and it won’t get caught.

Some ideas that would be great to see:

PgBouncer optional process that detects bad queries and logs them
RDS connection pinning behavior
Static analysis tools for app queries
Runtime extension to client libraries
Making sure your development flow runs PgBouncer locally to try and encounter this behavior before running on production

In the rails world there are several active gems devoted to keeping a codebase safe from issues that would cause downtime while migrating tables (ie, zero downtime). But across ecosystems I could not find anything related to protecting against PgBouncer issues.

As a step in this direction, I’ve published a (currently experimental) gem for use in Rails/ActiveRecord apps called pg_pool_safe_query. It will log warnings if SQL is run that is incompatible with PgBouncer and raise an error if advisory locks and prepared statements are not disabled.

Can we improve connections without a pooler?

A more recent development in Postgres 14 was improvements to snapshot scalability, which seem to have resulted in big improvements in efficiently maintaining more idle connections in Postgres.

It’s exciting to see effort being applied to increasing connection efficiency in Postgres itself. The author of that snapshot scalability improvement lines up with my own frustrations:

Ideally Postgres would better handle traffic spikes without requiring a pooler
Poolers cut out useful database features
Postgres itself would ideally move towards architecture changes across several key areas, eventually culminating in a larger move towards a lighter weight process/thread/async model which better aligns with the C10k problem out of the box

Most of the work in the industry seems to concentrate on building better poolers, rather than improving the internals of Postgres connection handling itself¹⁷. Outside of PgBouncer you’ve got RDS Proxy, Odyssey, PgCat, Supavisor, PgPool II and I’m sure others. All have their own benefits but suffer from the same transactional scaling limitations.

In fairness to the incredible work that goes into Postgres - every performance improvement they make in every new version is also a connection scalability improvement. If the queries, indexes, plans, and processes are making big performance gains with each version then less connections can do more.

PgBouncer alternatives

There are alternatives to PgBouncer, but the same transaction limitations apply to all of them: each has a transaction mode (or operate exclusively in transaction mode) that offers the best scaling. Once in transaction mode you can’t support most session level features anymore and you’re working off of the fact that database connections spend more time being idle than active.

They all have their own unique benefits in comparison, but have the same fundamental transaction limitations.

Am I finally done with this post?

I think I’ve said enough.

Postgres is great. PgBouncer is important. Know what can go wrong and account for it.

🐘 ✌🏼 🐘

This article from Brandur details some additional nuances of handling connections and pools, but these three are the higher level version of it ↩︎
Technically it doesn’t have to be a single instance, it could be a round Robin of multiple PgBouncers, but from a client perspective you connect to a single one

https://www.crunchydata.com/blog/postgres-at-scale-running-multiple-pgbouncers ↩︎
It’s even lower on their lower powered options. It goes from 20, to 120, to 400, then 500 once you’re around their $400/mo plans.

Supabase has no standard plans with more than 50 connections.

Render.com’s managed Postgres offering is based on memory available on each plan: 6 gigs or less is 97 connections, less than 10 gigs is 197 connections and over 10 gigs is 397 connections.

This isn’t totally unreasonable - managing more connections requires more cores and more memory in a process based model especially. But at their highest tiers these services don’t exceed 500 available connections.

More generalized services like Azure and Amazon RDS will let you go as high as you like, but that’ll go badly. ↩︎
Which is very exciting to see concentrated work on improving this aspect in Postgres internals! 🤘🏼 ↩︎
This more recent crunchy data article on making sure your Postgres app is production ready implies the old standard of 500 connections is no longer accurate so I’d be curious to know more specifics since most resources still emphasize these numbers https://www.crunchydata.com/blog/is-your-postgres-ready-for-production ↩︎
AFAIK this is largely true of any database and MySQL also has connection pooling solutions, but it does seem to be particularly necessary with postgres ↩︎
There are a couple caveats to this statement. Just having a dedicated low latency pool is an improvement so may slightly help concurrency. PgBouncer can also proxy multiple databases so you could increase read concurrency at least this way.

The queueing behavior of poolers can also be a benefit since you can wait for a connection to be available for longer, vs Postgres instantly rejecting the attempt: https://www.percona.com/blog/connection-queuing-in-pgbouncer-is-it-a-magical-remedy/ ↩︎
They do mention this in their docs:

> “Note that “transaction” pooling breaks client expectations of the server by design and can be used only if the application cooperates by not using non-working features.” ↩︎
https://github.com/pgbouncer/pgbouncer/issues/653

https://github.com/pgbouncer/pgbouncer/issues/249

It also just seems to be something they’re not interested in doing ↩︎
The idea is you fail and continue to retry. See this article for some framework agnostic approaches to retries: https://postgres.ai/blog/20210923-zero-downtime-postgres-schema-migrations-lock-timeout-and-retries ↩︎
I might have five nickels, but still, it happens. Also again I am grateful for anyone taking the time to write up content and share their expertise. ↩︎
That’s a topic for another time… ↩︎
In addition to the Ruby pg gem It’s used by the python psycopg lib, and node node-libpq package (and I’m sure many others). So it seems like most client libraries handle things safely enough at the protocol level to turn off prepared statements

If you are using Go with the pure Go lib/pq driver see this issue for how to properly handle unnamed statements. The rust sqlx library seems to have a similar issue. Seems that if a library does not use libpq they end up in a bit of pain when trying to work with PgBouncer ↩︎
Named prepared statements can boost performance for repetitious queries because they bypass the Parse call on subsequent runs. That’s their primary benefit in comparison to unnamed statements ↩︎
LISTEN can be called in a transaction as well, but all that means is the session level listen won’t be triggered until the transaction commits, and won’t start listening at all if a rollback is triggered ↩︎
I do find myself asking “what is the point of nice features if you can’t use them at scale because of transaction mode pooling”? Not being able to use certain features at scale should never preclude them from being built - but it’s a disappointing reality ↩︎
I’m a little afraid to have made this statement and the potential for someone to come back at me angry about this being an oversimplification 😅 ↩︎

Making Tanstack Table 1000x faster with a 1 line change

Tue, 07 Mar 2023 17:10:00 -0500

A few months back I was working on a Javascript frontend for a large dataset using Tanstack Table. The relevant constraints were:

Up to 50k rows of content
Grouped by up to 3 columns

Using react and virtualized rendering, showing 50k rows was performing well. But when the Tanstack Table grouping feature was enabled, I was seeing slowdowns on a few thousand rows, and huge slowdowns on 50k rows.

It might have gone unnoticed if it was 100ms slower, or even 500ms slower. But in the worst case renders would go from less than a second without grouping up to 30-40 seconds with it.

Tracking down the issue

Initially I tried using the chrome Javascript profiler, but it can be tough to use when performance is so slow. The profiler adds noticeable overhead to your code, and since the code took 30-40 seconds already, it was basically unusable¹.

Unable to use the profiler, I reached for an old, simple standby: console.time². This is a convenient way to see how long a section of code takes, logged to your console:

console.time('expensive code');
thisIsExpensive();
console.timeEnd('expensive code');
// console.time
//   expensive code: 1000ms

A side note about optimizing code: as programmers we are full of ideas about what needs to be optimized and are terrible at getting it right.

We make educated guesses at what is important to optimize and where the problem is, and until we measure we’re usually wrong. I try now to wrap everything possible in a benchmark to make sure I’m even in the right place, then start narrowing the benchmark inside of the code from there.

When tracking down a performance issue this would be a general outline:

console.time('everything');
elements.forEach(() => {
  console.time('methodCall');
  methodCall(() => {
    console.time('build');
	build();
	console.timeEnd('build');
  });
  console.timeEnd('methodCall');
});
console.timeEnd('everything');
// build      49ms
// methodCall 50ms
// build      51ms
// methodCall 52ms
// everything 102ms

Back to the table rendering

As usual, before measuring, all of my guesses at the potential performance problem were wrong.

My own code was fine, so this was a case where it was actually a library bug, which was a surprise. There was no code path I could find that was performing poorly - all of the time was spent in the library. When using Tanstack table in React all of the logic happens in a pretty centralized place - the useReactTable hook - so it was easy to see the time was all there

console.time('everything');
customCode();
console.time('useReactTable');
useReactTable(...);
console.timeEnd('useReactTable');
console.timeEnd('everything');
// useReactTable 31500 ms
// everything    31537 ms

One of the nicest things about developing Javascript with packages is that at any time I can open up the node_modules folder and play around with third party code. In this case I was able to modify the Tanstack Table source code directly to add some timing information.
Turning on grouping was when everything slowed down, so it made the most sense to start timing that code

This is an abbreviated version of the grouped row source code, with my first pass at timing what I thought were the likeliest culprits. Pay attention primarily to console.time statements - you don’t have to understand everything going on.

function getGroupedRowModel<TData extends RowData>() {
  console.time('everything');
  //...
	
  console.time('grouping filter')
  const existing = grouping.filter(columnId =>
    table.getColumn(columnId)
  )
  console.timeEnd('grouping filter')
	
  const groupUpRecursively = (
    rows: Row<TData>[],
    depth = 0,
    parentId?: string
  ) => {
    if (depth >= existing.length) {
      return rows.map(row => {
      row.depth = depth
      //...
      if (row.subRows) {
        console.time('subRows')
        row.subRows = groupUpRecursively(
          row.subRows, 
          depth + 1
        )
        console.timeEnd('subRows')
      }
      return row
    });
	
    const columnId: string = existingGrouping[depth]!
    const rowGroupsMap = groupBy(
      rows, 
      columnId
    )
	
    const aggregatedGroupedRows = Array.from(rowGroupsMap.entries()).map(([groupingValue, groupedRows], index) => {
      let id = `${columnId}:${groupingValue}`
      id = parentId ? `${parentId}>${id}` : id
      console.time(
        'aggregatedGroupedRows groupUpRecursively'
      )
      const subRows = groupUpRecursively(
        groupedRows, 
        depth + 1,
        id
      )
      console.timeEnd(
        'aggregatedGroupedRows groupUpRecursively'
      )
      //...
      }
    }
  }
  console.timeEnd('everything');
}

My hunch was that the groupUpRecursively function was to blame - it made logical sense that tens of thousands of recursive calls could cause a slowdown (spoiler: as usual, I was wrong 😑):

First pass was a bust - it logged thousands of the subRows timers - every iteration was fast and there were too many of them to be useful so I cut it.

console.time
 subRows: 0 ms
   at map (packages/table-core/src/utils/getGroupedRowModel.ts:48:25)
       at Array.map (<anonymous>)
       at Array.map (<anonymous>)
       at Array.map (<anonymous>)
       at Array.map (<anonymous>)

console.time
 subRows: 0 ms
   at map (packages/table-core/src/utils/getGroupedRowModel.ts:48:25)
       at Array.map (<anonymous>)
       at Array.map (<anonymous>)
       at Array.map (<anonymous>)
       at Array.map (<anonymous>)

Removing that, I started to get closer. I was accounting for most of the time but I had two problems: I wasn’t accounting for all of the time (everything was 33 seconds and groupUpRecursively was only 23 seconds) and the chunk of time I was logging was too large to usefully identify the problem code:

console.time
  grouping filter: 1 ms
    at fn (packages/table-core/src/utils/getGroupedRowModel.ts:22:17)

console.time
  aggregatedGroupedRows groupUpRecursively: 23248 ms
    at map (packages/table-core/src/utils/getGroupedRowModel.ts:71:23)
        at Array.map (<anonymous>)
        at Array.map (<anonymous>)
        at Array.map (<anonymous>)

console.time
  everything: 33509 ms

I realized I had missed a function call - a little unassuming function called groupBy - so I added a console.time block there next:

console.time('groupBy')
const rowGroupsMap = groupBy(rows, columnId)
console.timeEnd('groupBy')

Got it! Almost the entirety of the 31 seconds was concentrated into 3 calls to groupBy.

console.time
  grouping filter: 2 ms
    at fn (packages/table-core/src/utils/getGroupedRowModel.ts:22:17)

console.time
  groupBy: 10279 ms
    at groupUpRecursively (packages/table-core/src/utils/getGroupedRowModel.ts:59:19)

console.time
  groupBy: 10868 ms
    at groupUpRecursively (packages/table-core/src/utils/getGroupedRowModel.ts:59:19)
        at Array.map (<anonymous>)

console.time
  groupBy: 10244 ms
    at groupUpRecursively (packages/table-core/src/utils/getGroupedRowModel.ts:59:19)
        at Array.map (<anonymous>)
        at Array.map (<anonymous>)

console.time
  aggregatedGroupedRows groupUpRecursively: 21159 ms
    at map (packages/table-core/src/utils/getGroupedRowModel.ts:71:23)
        at Array.map (<anonymous>)
        at Array.map (<anonymous>)

console.time
  everything: 31537 ms

For each grouped column, it was calling groupBy and each call took roughly 10 seconds.

So what the heck was going on in the groupBy function that was causing such a massive slowdown. Can you pick it out?

function groupBy<TData extends RowData>(rows: Row<TData>[], columnId: string) {
  const groupMap = new Map<any, Row<TData>[]>()
	
  return rows.reduce((map, row) => {
    const resKey = `${row.getValue(columnId)}`
	const previous = map.get(resKey)
	if (!previous) {
	  map.set(resKey, [row])
	} else {
	  map.set(resKey, [...previous, row])
	}
	return map
  }, groupMap)
}

I started chopping up the function. I switched the Map to be an object literal in case that was causing some kind of memory overhead and tried changing to a for loop instead of reduce in case the amount of iterations plus closures was causing an issue³.

Those had no effect so next I started commenting out lines of the function. When I commented out this line, all of a sudden everything started finishing instantly:

// map.set(resKey, [...previous, row])

What was the purpose of that line and why was it so slow?

const resKey = `${row.getValue(columnId)}`
const previous = map.get(resKey)
if (!previous) {
  map.set(resKey, [row])
} else {
  map.set(resKey, [...previous, row])
}

On each iteration of the reduce call, the code:

Used the value of that column cell as a map key. Let’s say the value is the string “New York”
If there was no value associated with “New York”, it would set a value of the current row wrapped in an array
If there was already a value, it would use the Javascript spread operator to concatenate the current row onto the end of the previous array value
This means that on each iteration of reduce, the spread operator was creating a new, incrementally larger array, getting slower with each iteration⁴

// 1st iteration
[...previous, row] => [1,2]
// 2nd iteration
[...previous, row] => [1,2,3]
// 3rd iteration
[...previous, row] => [1,2,3,4]
// ...
// 50,000th iteration
[...previous, row] => [1,...49998,49999]

What does spread do behind the scenes?

In our case the spread operator takes an iterable object and loops over it to create new array. It’s probably more nuanced than this, but the spread operator is likely O(n). That means that as the size of the array grows, the longer it takes to spread the values into a new array.

I’m certain there are additional efficiencies provided in the language internals but the following code is essentially equivalent⁵:

const a = [1, 2, 3, 4, 5, 6, 7];
	
// spread
const b = [...a, 8];
	
// manual loop
let c = [];
	
for (let i = 0; i < a.length; i++) {
  c.push(a[i]);
}
c.push(8);

Which means that the original version of that groupBy code was equivalent to this:

// original code
map.set(resKey, [...previous, row]
	
// manual spread
for (let i = 0; i < rows.length; i++) {
  const tempPrevious = [];
  for (let j = 0; j < previous.length; j++) {
    tempPrevious.push(previous[j]);
  }
  tempPrevious.push(rows[i]);
  previous = tempPrevious;
  map.set(resKey, tempPrevious);
}

Seeing the manual spread, one thing sticks out immediately: we have a nested loop.

A nested loop is a good shorthand for identifying one of the slower forms of algorithmic complexity: a “Big O” complexity of O(n^2)⁶. This means that for an array size of 50,000, the number of iterations required would be 50,000^2, or 2.5 billion iterations 😱.

In our specific case the array started out empty and grew to a size of 50,000, so it’s wasn’t quite as bad. Technically it was still a complexity of O(n^2), but practically it was O(n^2/2)⁷. Either way it still meant:

1,249,975,000, or 1 billion 249 million iterations
49,999 discarded arrays allocated for 1,249,925,001 entries
Times three groupBy calls that means 3,749,925,000 iterations, 149,997 discarded arrays and 3,749,775,003 allocated entries

Pour one out for the garbage collector and Javascript runtime 🙅🏻‍♂️☠️🙅🏻‍♂️. Honestly, for how many iterations and how much garbage we accumulated, it was operating pretty well 🫠.

So the question was - what could be done to improve it?

Spread is an immutable pattern, so should the code be kept immutable?

Aside from providing a convenient syntax for combining iterables, the spread operator also allows us to keep code immutable. This can be beneficial because immutable patterns mean we avoid mutations, which means we don’t change data from underneath other code that is using it.

If this code was written to be immutable, there were some other options we could try to improve performance which would behave the same as the spread operator:

// Array.from()
//   ~10 seconds, just as slow 😒
const arr = Array.from(previous)
arr.push(row)
map.set(resKey, arr)
	
// slice()
//   ~10 seconds, just as slow 🙃
const arr = previous.slice()
arr.push(row)
map.set(resKey, arr)
	
// concat(row)
//   ~10 seconds, just as slow 🧐
map.set(resKey, previous.concat(row))
	
// hand written for loop
//   ~14 seconds, slower than spread! 😱 
let arr = []
for (let i = 0; i < previous.length; i++) {
  arr.push(previous[i])
}
arr.push(row)
map.set(resKey, arr)
	
// concat([row])
//   ~4 seconds 🏃‍♂️ 
map.set(resKey, previous.concat([row]))
	
// using the `immer` node package
//   ~100ms 🚀 
return produce(groupMap, (draft) => {
  return rows.reduce((map, row) => {
    const resKey = `fixedKey`
    let previous = map.get(resKey)
    if (!previous) {
      map.set(resKey, [row])
    } else {
      previous.push(row)
    }
    return map
  }, draft)
})

Array.from, slice, and concat all had the same performance characteristics as the spread operator. There were a few surprises however:

A manual for loop was the slowest option, coming in at around 14 seconds. The native code versions of these methods are clearly much more optimized than the manual loop we can write
concat, when handed an array as the argument instead of a plain object, was consistently faster than spread on any v8 platform (node/chrome). It took half the time! Either there was a hard to find mistake in my code or the runtime had a special optimization for this use case. Still too slow however
Using immer performed significantly faster than all other options. immer provides regular Javascript interfaces on immutable data using proxies and structural sharing to make efficient changes to data structures without changing the original source object. However in this case the main benefit of the immer approach is that we were not copying any arrays - just pushing directly to them because they were created inline in our function

But did we even need immutability at all? Was the immer approach good enough? Could we get even faster?

Using push directly

For performance, you can’t beat mutations.

Mutations modify data in place, without copying and creating duplicated data. Looking over our groupBy function, there wasn’t any benefit to using immutable operations. Everything was created in the function so we could safely mutate it before we returned it. I don’t think the original code was written with immutability in mind - using the spread syntax was just convenient.

When using array push, we are effectively running with an algorithmic complexity of O(1). That means as our array grows, the cost of inserting new elements does not. It’s constant time: it’s as fast for 1 item as it is for 50,000.

Here’s the final version of the fix, now 1000x faster for my use case:

function groupBy<TData extends RowData>(rows: Row<TData>[], columnId: string) {
  const groupMap = new Map<any, Row<TData>[]>()
	
  return rows.reduce((map, row) => {
    const resKey = `${row.getValue(columnId)}`
    const previous = map.get(resKey)
    if (!previous) {
      map.set(resKey, [row])
    } else {
      previous.push(row)
    }
    return map
  }, groupMap)
}

Can you spot the difference? It’s this one simple change:

previous.push(row)

previous.push(row) mutates the existing array and brings our time down from 10 seconds per groupBy down to around 10ms, 1000x faster than the original code 🚀.

The takeaway here is that while spread is an expressive and convenient language feature, it’s important to see it for what it is at its core: a for loop. If you see a spread inside of a loop, you’ll want to rewrite it if there is any chance it will operate on a large set of values.

Source code

You can see the full PR here https://github.com/TanStack/table/pull/4495. Because it was such a small fix, the majority of the PR is just a spec to guard against a future performance regression.

And here’s the source code for each attempt, runnable on the Replit platform. If you run it there the timings are much slower because it is running a lower powered CPU than my local machine, but as a relative measure the results are consistent with the numbers in this post.

I also find it can be hard to get a great sense for a profile when profiling react code since certain internal react code paths are hit so often the results are tough to decipher ↩︎
It’s non standard, but in the future I need to remember to reach for console.profile as well.

That could have been helpful here for profiling smaller blocks of code rather than deciphering the whole React render stack.

Considering how slow the code was it still may not have been usable anyways.

You can learn more about it here: https://developer.mozilla.org/en-US/docs/Web/API/console/profile ↩︎
What I’ve found is that Javascript engines have gotten so optimized that a forEach/map/reduce call can be as efficient as a hand written for loop. They’re almost never the performance problem in Javascript, least of all in a large set of iterations where the JIT can aggressively optimize. ↩︎
I should note that the performance was at its worst for poorly distributed groupings. By that I mean if you had a grouped column with only one value then the grouping is concentrated into one key with all 50,000 values. If there were lots of unique values in the column the arrays were smaller and built more quickly ↩︎
The v8 internals are, unsurprisingly, very complicated. Logic for spread seems to exist in a variety of places because it’s such a core feature so I wasn’t able to track down a specific implementation ↩︎
If “Big O” complexity doesn’t mean anything to you, there’s lots of great resources online for it and it’s a great concept to be familiar with. The basic idea is it helps you identify how costly a particular piece of code will be to run.

For a deep dive into it, https://frontendmasters.com/courses/algorithms/ is a great free resource ↩︎
In terms of algorithmic complexity, O(n^2) and O(n^2/2) are considered equivalent, since you generally drop the constant (in this case, 1/2).

But from a practical perspective, if you’re going to have a slow algorithm you’d still rather have it be half as slow as it’s full potential slowness 😅 ↩︎

Kicking the social media habit with “one sec”

Thu, 09 Feb 2023 17:30:54 -0500

Twitter. Oh how I could stroll that infinite corridor of information.

Ever since having kids my personal time has been squeezed into a tiny ball. Still, I’d finish putting the kids to bed, open my phone, and where would I end up? On Twitter for 45 minutes. An hour. Hour 30. I’d sit there feeding whatever meager morsels of free time I had left to that tiny blue bird, it gobbling them up like some kind of attention soaked black oil sunflower seeds¹.

Twitter was my go to. I could certainly get lost on Instagram, or TikTok. But something about those platforms felt more wasteful. I wasn’t doing anything “worthwhile” there. Being on Twitter I could create warped justifications that I was actually learning things from people. Many times I was. But it was just as easy to get lost in pointless wanderings, sucked into controversy I didn’t care about 5 minutes earlier. And even when I learned something, was it best use of my time?

During one of these info dump excursions, I saw a tweet about an app called “one sec”. It got me curious - taking a deep breathe, delaying distracting apps… adding friction?

… isn’t that basically Screen Time?

It sounded interesting, but was it really any different from screen time limits?

I have had screen time limits on social media since that feature first released on iOS. At worst it is a minor annoyance that I hardly notice myself dismissing. At best it occasionally saves me after already wandering aimlessly for ages.

My rock solid screen time workflow:

You’ve reached your limit!
tap “Remind me in 15 minutes”
You’ve reached your limit!
tap for 15 more
You’ve reached your limit!
Ugh, I just want to finish reading one thing - I can probably do it in 1 minute?
You’ve reached your limit!
…give up and tap “Ignore limit for today”²

So what about “one sec” was going to make that any different?

The difference I found with “one sec” is specifically related to what they advertise: added friction. In full they describe it as “added friction makes distracting apps less appealing.”

Screen time does not make apps less appealing. Screen time is almost completely frictionless. How much friction does quickly tapping a screen cause? I’ve extended my screen time so many times over the years I probably could tap each option by muscle memory.

“One sec” is all friction. Here’s an example of what it’s Iike when you try to open an app connected to “one sec”:

That’s actually sped up a bit - the default is a 6-10 second animation. See how you are forced to sit and wait? That waiting is the key. It’s the friction they describe. I literally cannot get into that app any faster, and then I still have to decide whether I move forward once I finish waiting.

You sit there for 10 seconds - it’s intentionally aggravating, and it forces you to question how committed you are to opening the app. Does it really matter to me?

Here’s my new workflow with “one sec”:

Open app
“One sec” pops open
Take a deep breath. Wait for the screen to reveal my options
Phew, still waiting
This is only 10 seconds? Do I really want to open the app this badly?
Ok, moment of truth. Close or continue on?
Here I usually just drop out, but sometimes I “Continue to ‘app’”. Now I still have to make one more decision. What’s the reason I’m using it?
Not worth it, I’m out 👋🏼

For the rare times I actually stick around, I love the intention prompt. If the initial friction of waiting isn’t enough, here is one more layer of confronting my habit. You have a base set of intentions, and I’ve created a custom prompt for the one thing I ever do anymore on Instagram.

I don’t remember if this was on by default, but if it isn’t you should absolutely enable it through “Intention Tracking”

Is there anything more to it than friction?

Why would friction be so much more effective than screen time limits? Why would it be more powerful than just willing yourself to leave social media?

This excerpt from James Clear’s “Atomic Habits” sums it up really well:

Your environment is more important than your willpower

It is important to remember that the environment drives our good behaviors as well as our bad ones. People who seem to stick to good habits with ease are often benefitting from an environment that makes those behaviors easier.

Atomic Habits - Chapter 12

“One sec” changes your environment. Your typical phone environment is every app instantly available with a tap. With “one sec”, you are forced to evaluate that tap instead of mindlessly hopping around.

“One sec” has also been analyzed in two different academic studies about its impact on social media usage:

Social media usage cut in half through “one sec” intervention³

Friction causes behavioral change

By taking out the instant gratification, the urge for unintentional social media usage fades away over time: user’s brain now connects opening Instagram with something unpleasant, having to wait 10 seconds during the breathing intervention.

The takeaway: instant gratification (tap in and space out) is replaced with an unpleasant experience (waiting and being forced to acknowledge your own motivators). You are actually re-wiring your brain through this change. I can attest to that - the thought of opening apps and waiting through the “one sec” interstitial is pretty unpleasant, and stops me before I even consider opening them.

How do I use it?

For such a powerful app, the setup is a bit clunky.

As far as I can tell this isn’t “one sec”s fault - it’s likely not possible to block one app with another app just by having it installed. If it was, you could completely block a persons phone by convincing them to download your app.

To enable “one sec” you need the one sec app and the Shortcuts app. You create a personal automation that triggers “one sec” any time you go to open a specific application.

As of writing this I’m up to 8 automations connecting to “one sec”:

The biggest benefit you’ll get out of it is by using the “pro” version, which currently costs me about $16/yr. It’s absolutely worth it, but you can also use the free version to try out the behavior against a single app and see how you like it.

“One sec” displays the number of times it’s prevented you from opening each app, and estimates time saved. It’s interesting to analyze preventions, though it’s saved me way more time than it takes credit for.

The one sec site has great documentation, and the app offers videos on setup, so you can read and watch those once you download it.

What I’ve been doing instead

The clearest change for me is that I’m even writing this. It’s been years since I’ve written anything, and I’ve invested some of my time into a new writing workflow.

As well I’ve been learning a new programming language, reading books, exercising, and just generally formulating uses of my time that align with my values.

Social media, even when not all consuming, is a time suck. You probably have alternatives that would make you happier if only you left these platforms⁴.

Oh and I’ve been sleeping better. I have less options to distract me if I wake up at night - it’s a beautiful thing.

Throwing the baby out with the bath water⁵

Social media may be a huge time suck, but it’s not 100% useless. I think most people find some slices of learning, inspiration and connection on it somewhere.

For me, Twitter was often a valuable learning resource. It was a useful tool for keeping up with technology trends that matter to me personally and professionally. Unfortunately, I can’t disentangle the good (learning) from the bad (aimless wandering). Social media is designed to suck us in and keep us there as long as possible - I can’t fight that programming, I can only avoid it.

As a way to keep up with folks I follow without using Twitter directly, I’ve been using Mailbrew. It allows me to setup a digest based on my timeline and I can also choose specific Twitter handles to keep up with. It doesn’t recreate the experience of spontaneous discovery, but that means it also doesn’t trigger the accompanying variable ratio schedule⁶ with it.

Going even deeper with “one sec”

In writing about “one sec”, I’ve dug into the app quite a bit. I’ve found that “one sec” is deeply configurable and filled with additional features. Blocking apps was my main draw but it has a variety of other features including:

Blocking websites with a safari extension
Hooking into focus modes to completely block apps during those times
Starting a “block session” in the app which blocks every app and website you have configured
Triggering “don’t get lost” notifications if you’ve gone through to an app and have been there for awhile

“One sec” can do even more and I’ve found myself expanding not only what it enables but also what apps I apply it to. I now have periods where I apply “one sec” to email, slack and web browsers. I have my sleep focus setup to completely block them so I can’t wake up in the middle of the night and just hop over to bad habits.

It’s starting to feel like a distraction reducing superpower.

Next steps

Eventually, having subjected all apps on my phone to triggering “one sec”, I will finally setup “one sec” to trigger “one sec” whenever it opens. This will likely initiate an infinite loop that causes my phone to melt completely⁷.

In this highly probable scenario, maybe I’ll just switch back to a flip phone?

More likely I’ll buy a smartphone again… but I know the first app I install will be “one sec”.

I suppose I’m being presumptuous in assuming the Twitter bird likes black oil sunflower seeds. The birds in my yard go wild for the stuff, so seems like a strong likelihood ↩︎
I do wonder if screen limits are even meant to_actually_ stop you, or just check off a box on device usability for Apple. All screen time tracking usually does is make you think “wow.. I use this phone too much”. There’s no behavior change. Information in isolation does not change behavior.

This is a bit like mandates to make it clearer to consumers what the nutrition facts are for different foods so people can make “informed decisions”. This helps some people do that, but for most people it’s meaningless information that they completely ignore (or notice, but eat/drink it anyways and just feel bad about it later).

It’s highly anecdotal but I’ve never met anyone who found screen limits to be helpful. ↩︎
This is from the “one sec” blog, which is a good read in general.

Honestly some of their blog post illustrations are fantastic - much nicer than I expected such a utilitarian app to have. The referenced post in particular looks great, so here it is again https://one-sec.app/max-planck-study/ ↩︎
If not, then get your social media on! Who am I to judge if it makes you genuinely happy? ↩︎
The sentiment is right, but figures of speech are so weird. Also I like to think no one has ever actually done this 😰 ↩︎
https://rationalwiki.org/wiki/Variable_ratio

I think this tracks. Essentially using Twitter for me is like using a slot machine. I get hits of value, but on a variable schedule. Because I don’t know when the next hit will occur, but I know it probably will, I keep going indefinitely. ↩︎
I am genuinely curious whether it’s possible and if apple or the “one sec” developer have handled it but I’m too scared to try it 😅 I know my phone won’t melt, but could I crash it? ↩︎

JP Camara

When good threads go bad

Threads out after curfew 🧵

Stuck threads

Livelocks

Long-running CPU

Long-running IO

Native extensions

What happened to our Puma server, anyways?

Don't guess, measure

Thread Shutdown

raise and kill

(1) Don't kill threads

(2) Interrupt your thread safely

(3) Ensure cleanup

(4) Don’t use timeout

(5) If you use rack-timeout, you really should use term_on_timeout

(6) Monitor your thread cost

(7) Get a handle_interrupt on things

The way you ensure success

Bitmasks, Ruby Threads and Interrupts, oh my!

Interrupting Threads 🧵

Managing threads 👷🏼‍♂️👷🏻‍♀️

An important interruption

Why integer masks?

Checking for interrupts

Jumping to instructions

Pervasive interrupts

Interrupt masks

It's masks all the way down

The `interrupt`ion we've all been waiting for

`TIMER_INTERRUPT_MASK`

`TRAP_INTERRUPT_MASK`

`PENDING_INTERRUPT_MASK`

`TERMINATE_INTERRUPT_MASK`

`POSTPONED_JOB_INTERRUPT_MASK`

`VM_BARRIER_INTERRUPT_MASK`

The /o in Ruby regex stands for “oh the humanity!”

The cliffs of insanity

/o the humanity!

Inside the VM

once upon a time

Why does it exist?

I’ll take your once, and raise you…

The inmates are running the asylum

This is the END

A silly optimization: adding opt_respond_to to the Ruby VM, part 6

The progress so far

Breaking down the changes

What’s next?

1. Tests

2. Logic for handling the private/protected param

3. Actual optimization code

Setting up a performance baseline

A first silly optimization

A second, slightly less silly optimization

Back to reality

Defining an instruction: adding opt_respond_to to the Ruby VM, part 5

Pattern matching bytecode instructions

Macros and enums

A virtual machine DSL

Adding to the DSL

Peephole optimizations: adding `opt_respond_to` to the Ruby VM, part 4

Sage wisdom

Finding the optimizer

Disassembling an existing optimization

Using the debugger

Compiling on-the-fly

Getting to our test.rb file

Making one small step towards opt_respond_to

Using gdb

Finishing up

The Ruby Syntax Holy Grail: adding `opt_respond_to` to the Ruby VM, part 3

Getting your own environment setup

Back to the investigation

Finding the compiler: adding `opt_respond_to` to the Ruby VM, part 2

A little help from Git

Starting from main

Stepping back to ruby_options

Adding `opt_respond_to` to the Ruby VM: part 1

`raise` and `kill`

(4) Don’t use `timeout`

(5) If you use `rack-timeout`, you really should use `term_on_timeout`

(7) Get a `handle_interrupt` on things

The way you `ensure` success

`once` upon a time

I’ll take your `once`, and raise you…

This is the `END`

Making one small step towards `opt_respond_to`

Using `gdb`

Stepping back to `ruby_options`

1. `Range#each` is still written in C, which YJIT can’t optimize

2. Optimizing our loop: `Integer#times` was converted from C to Ruby in Ruby 3.3

An aside on `Integer#succ`

Back to `Integer#times`

3. `Array#each` was converted from C to Ruby in Ruby 3.4

Speeding up `range#each`

`report_on_exception` and `abort_on_exception`

`status`, `alive?` and `stop?`

`Thread.pass`, `wakeup`, `Thread.stop`, `run`, and `priority`

`raise` and `kill`