In Finding the compiler: adding opt_respond_to to the Ruby VM, part 2, I found the entrypoint into the compiler! It takes the root of our abstract syntax tree - pm->node - and produces a rb_iseq_t. rb_iseq_t is an “InstructionSequence”, which represents our virtual machine bytecode. Here’s the code where we left off:

// ruby.c
static VALUE
process_options(int argc, char **argv, ruby_cmdline_options_t *opt)
{
    //...
    pm_parse_result_t *pm = &result.prism;
    int error_state;
    iseq = pm_iseq_new_main(&pm->node, opt->script_name, path, parent, optimize, &error_state);

Nothing about this code screams “I’m the compiler!”. But I am taking an educated guess, since:

  • The compiler should produce instruction sequences, or iseqs
  • We know this is the function that returns our program’s iseq when main.c is called
  • This is the only line that produces an iseq in this function

With all that lining up, I’m confident this is the place we need to investigate further. Now let’s find what needs to change to add our new bytecode. Stepping into pm_iseq_new_main, there are a few layers I need to wade through to get to something that seems promising.

Getting your own environment setup

Before I dig in further, let’s take a quick step back. In case you want to join in at home, here are some simple(ish) steps for doing that.

  1. Check out my guides on how to setup your development environment to hack on CRuby. I have a docker guide and a MacOS guide. The one thing I didn’t add to them was cloning the repo. So you’ll need to git clone the Ruby repository.
  2. Building CRuby can take a few minutes the first time you run it. After you’ve built everything, you can easily test your local setup using a file named test.rb. Create a test.rb file in the root of your CRuby folder.
  3. You can run make runruby, and it will run whatever is inside your test.rb file. You can even use debug tools to debug the C code you’re running and inspecting - we’ll talk more about those later.

Back to the investigation

First we’ve got the function pm_iseq_new_main, which seems to set us up as the <main> rb_iseq_t.

// iseq.c
rb_iseq_t *
pm_iseq_new_main(pm_scope_node_t *node, VALUE path, VALUE realpath, const rb_iseq_t *parent, int opt, int *error_state)
{
    iseq_new_setup_coverage(path, (int) (node->parser->newline_list.size - 1));

    return pm_iseq_new_with_opt(node, rb_fstring_lit("<main>"),
                                path, realpath, 0,
                                parent, 0, ISEQ_TYPE_MAIN, opt ? &COMPILE_OPTION_DEFAULT : &COMPILE_OPTION_FALSE, error_state);
}

This looked immediately familiar to me. It sticks out because i’ve seen that <main> before. Let’s run a simple Ruby program:

begin
  raise
rescue => e
  puts e.backtrace
end

All our program does it raise an error, rescue the error, then puts the backtrace. What does that backtrace look like?

../test.rb:2:in '<main>'

Oh yea! We are executing our code at the top-level of the program. And that top level is referred to as <main>. I think that’s being named by our pm_iseq_new_with_opt(node, rb_fstring_lit("<main>")... call - neat!

iseq_new_setup_coverage just sets up some optional coverage information, so let’s move to pm_iseq_new_with_opt:

rb_iseq_t *
pm_iseq_new_with_opt(pm_scope_node_t *node, VALUE name, VALUE path, VALUE realpath,
                     int first_lineno, const rb_iseq_t *parent, int isolated_depth,
                     enum rb_iseq_type type, const rb_compile_option_t *option, int *error_state)
{
    rb_iseq_t *iseq = iseq_alloc();
    ISEQ_BODY(iseq)->prism = true;
    //...
    struct pm_iseq_new_with_opt_data data = {
        .iseq = iseq,
        .node = node
    };
    rb_protect(pm_iseq_new_with_opt_try, (VALUE)&data, error_state);

    if (*error_state) return NULL;

    return iseq_translate(iseq);
}

This code allocates (iseq_alloc) an rb_iseq_t struct and sets it as being part of prism. I believe rb_protect is to allow handling of errors that might be raised while running a particular function? Looking at the git blame I see Peter Zhu added it to catch errors, so confirmed ✅. Not alot is happening here otherwise, so let’s jump into pm_iseq_new_with_opt_try:

VALUE
pm_iseq_new_with_opt_try(VALUE d)
{
    struct pm_iseq_new_with_opt_data *data = (struct pm_iseq_new_with_opt_data *)d;

    // This can compile child iseqs, which can raise syntax errors
    pm_iseq_compile_node(data->iseq, data->node);

    // This raises an exception if there is a syntax error
    finish_iseq_build(data->iseq);

    return Qundef;
}

This is the most promising piece of code so far. It’s the first thing that kindly tells me in explicit terms: “I am going to compile something”. Presumably pm_iseq_compile_node compiles data->node into the data->iseq. It’s in a new file called prism_compile.c. Let’s check it out!

// prism_compile.c
VALUE
pm_iseq_compile_node(rb_iseq_t *iseq, pm_scope_node_t *node)
{
    //...
    if (pm_iseq_pre_execution_p(iseq)) {
        //...
        pm_compile_node(iseq, (const pm_node_t *) node, body, false, node);
        //...
    }
    else {
        //...
        pm_compile_node(iseq, (const pm_node_t *) node, ret, false, node);
    }

    CHECK(iseq_setup_insn(iseq, ret));
    return iseq_setup(iseq, ret);
}

😮‍💨. There are many layers to this compilation. Primarily, this function seems to do two things: “compile” the node, then “setup” the iseq. I don’t know why the iseq “setup” is required yet. Let’s start with pm_compile_node and I’ll come back to the rest:

static void
pm_compile_node(rb_iseq_t *iseq, const pm_node_t *node, LINK_ANCHOR *const ret, bool popped, pm_scope_node_t *scope_node)
{
    const pm_parser_t *parser = scope_node->parser;
    //...
    switch (PM_NODE_TYPE(node)) {
      case PM_ALIAS_GLOBAL_VARIABLE_NODE:
        // alias $foo $bar
        // ^^^^^^^^^^^^^^^
        pm_compile_alias_global_variable_node(iseq, (const pm_alias_global_variable_node_t *) node, &location, ret, popped, scope_node);
        return;
      //...
      case PM_ARRAY_NODE: {
        // [foo, bar, baz]
        // ^^^^^^^^^^^^^^^
        const pm_array_node_t *cast = (const pm_array_node_t *) node;
        pm_compile_array_node(iseq, (const pm_node_t *) cast, &cast->elements, &location, ret, popped, scope_node);
        return;
      }
      //...
      case PM_FLIP_FLOP_NODE: {
        // if foo .. bar; end
        //    ^^^^^^^^^^
        const pm_flip_flop_node_t *cast = (const pm_flip_flop_node_t *) node;
        //...
        pm_compile_flip_flop(cast, else_label, then_label, iseq, location.line, ret, popped, scope_node);
        //...
      }
      //...
      case PM_IT_LOCAL_VARIABLE_READ_NODE: {
        // -> { it }
        //      ^^
        if (!popped) {
            PUSH_GETLOCAL(ret, location, scope_node->local_table_for_iseq_size, 0);
        }

        return;
      }
      //...
      case PM_MODULE_NODE: {
        // module Foo; end
        //...
      }
      //...
}

pm_compile_node puts the “fun” in “function”. It’s really cool! This 1800+ line monster seems to cover a huge swath of Ruby syntax. Maybe all of it? The prism_compile.c file is 11 thousand lines long, as each case of this switch statement branches off into more granular node compilations, like pm_compile_array_node and pm_compile_flip_flop.

With that in mind, it is also an utterly daunting file to consider for the opt_respond_to instruction. Do I edit this file? Where would I even start? I need to swap out a method call to respond_to? - there is code that seems to handle method calls:

case PM_CALL_NODE:
    // foo
    // ^^^
    //
    // foo.bar
    // ^^^^^^^
    //
    // foo.bar() {}
    // ^^^^^^^^^^^^
    pm_compile_call_node(iseq, (const pm_call_node_t *) node, ret, popped, scope_node);
    return;

Maybe that’s it?

I think I need to use a cheat code here to give me some direction. In the previous post, I mentioned Étienne Barrié’s PR to add optimized instructions for frozen literal Hash and Array. I’ve been mostly ignoring it so far, but I think it’s time I use that for a bit of direction on where to go from here.

I think we’re close! So far, I’ve navigated the code manually. In the next post, we’re going to actually run and debug some code, and dig a bit into Étienne’s work. See you then! 👋🏼