From Scratch Code RSS Feed

Build Your First REST API in Rust - LIU Brooklyn Workshop

Mon, 06 Apr 2026 00:00:00 GMT

A few weeks ago, I dusted off my “Build Your First REST API in Rust” workshop and took it to another borough. Hello, LIU Brooklyn!

My workshops begin with a show of hands to see who has worked with Rust and REST APIs before. About 20 students attended this time, and the topics were new to nearly all of them! I’m hopeful they left feeling Rust was even a tiny bit more accessible.

I’m grateful to my friend and colleague Samir Iabbassen for hosting, and Professor Christopher League for seeding the discussion with some nerdy programming language topics.

After the primary REST API material, I also demoed Ozark, an example of Rust compiled to WebAssembly for use in the browser. No server required!

If you want to try it yourself, here are the materials:

See you at the next one!

Build Your First REST API in Rust - CCNY Workshop

Mon, 23 Mar 2026 00:00:00 GMT

Earlier this month, I ran my first in-person workshop, “Build Your First REST API in Rust,” for the Association for Computing Machinery (ACM) at the City College of New York (CCNY). That’s a lot of acronyms, and it went well!

It was a great group, very engaged, and I enjoyed chatting with several students afterwards. About 30 people attended, and we built a CRUD API together from scratch using Axum.

My goal was to make Rust feel more approachable and to introduce a practical topic that’s often missing from a school curriculum: building REST APIs.

One thing I’m thinking about for next time is easing into the command-line setup a bit more, but overall I was really happy with how it went.

A few photos from the session:

If you want to try it yourself, here are the materials:

I’m planning to run more workshops like this in the future.

See you there, maybe!

Write your first library

Mon, 17 Nov 2025 00:00:00 GMT

At my first real software internship, the highlight of the summer was a 24-hour hackathon. My team and I sat in a conference room for way too many hours, hacking together a prototype. I barely remember the end goal. A chat something?

What I recall most was the feeling of trying one JavaScript library after another, failing to get basic tabs to work. I felt helpless!

Clearly I needed more JS experience and should’ve asked for help, but there’s a different point I want to emphasize: the shift from being a library user to being a library writer.

What is a library?

You’ll see different terms depending on the person, language, and ecosystem: library, package, crate, SDK. These all represent code which is already written in your language which you can pull in freely to your project. I prefer the term library because it’s timeless, and would make Andrew Carnegie proud (I hope!).

The size can vary anywhere from a small date/time utility to a massive framework like React. Each of these is a library and was written by someone with a keyboard like you and me.

The code itself typically lives on GitHub, but could be anywhere, and a built/packaged version of that code lives in an online package index. For JavaScript, this is NPM, in Python it’s PyPI, in Rust it’s crates.io.

It’d be easy to get lost in the licensing details or best practices, but others have covered this better and more deeply than I will.

I’m going to make the case for why you should write your own, and how it will transform your relationship with your code.

You’ll feel more confident in your stack.

The vague uncertainty and powerlessness I felt during the hackathon? I had no idea what I was doing.

Each JavaScript library I found on the internet, each example I stumbled through, all I could think was “I hope this one works.” I was at the whim of what was published, potentially incomplete, years prior.

After you’ve written even one library, your thinking shifts to: “I understand the general shape of what’s happening here.”

When you see an import, you’ll know roughly what that maps back to on the file system, and how they exposed that set of symbols (functions, classes, etc) publicly.

You don’t need to be able to regurgitate the pseudocode behind the scenes. Just knowing you can navigate their repo turns what used to be a black box into just another piece of code in your system, this time written by someone else.

Once the black box feeling dissolves, something interesting happens: you start forming opinions. You start noticing which libraries feel intuitive and which feel clunky. That’s discernment, and it naturally leads you to the next question: what makes a good interface in the first place?

Your discernment around other libraries will increase.

The conventional wisdom for picking a library is to check their GitHub stars (a popularity contest for nerds), version number (be wary of anything below 1.0!), and how recent its commits are. This is all valid, but I’d encourage you to go one step further: look at their examples and see if they match your mental model of the problem you are trying to solve. This forces you to think through what you need the library for in the first place. Sometimes you'll think “uhhhh this looks more complicated than I expected,” other times "ah perfect, they make this super easy."

A library is ultimately a contract: when you ask for X, it will do Y or give Z. A good one will do this with minimal friction.

When I pull up the docs for a new library, my first thought is about their interface. What functions and classes they expose, how they fit together. And to be honest, I often look for a bit of awe, a sense of “how did they do this?!” (I wrote about this in Rust’s Axum.)

I’m not saying every library you use needs to be fancy, elegant, or clever. But there’s a sense of “does this match what I was expecting it to do?” that’s always worth asking yourself.

The moment you can recognize a clean interface in someone else’s library, you’re ready to design your own.

You'll learn to design interfaces.

You may recall when you wrote your first function and called it. Code reuse! Writing a library is exactly the same, just with harder interfaces and more pomp and circumstance.

Each time you think “this function could really use another parameter,” that’s interface work. Each time you realize a function printed an error rather than throwing or returning an error to the caller, that’s interface work.

Designing a usable interface is as much art as science. A good one will hide most of the details and expose a small surface area of knobs for the user at the right time. This will feel difficult and awkward at first, but with time you’ll get the hang of it.

Once you’ve designed your first library interface, you’ll start to notice places within your projects where a harder interface is more useful. The voice I often hear is “hmm this file shouldn’t need to know about this,” and it’s a sign my interface could be stronger.

At this point, the difference between “code I wrote and am now reusing” and “code someone else wrote and I’m now reusing” starts to go away.

It’s all just code, available for your next idea.

It’s empowering.

This is the real reason. Everything else about saving time and not reinventing the wheel is valid and probably what you should tell your manager.

There’s a deeper truth here. Early on in your career, you will use libraries to build applications. You may even have users, and hopefully your code is solving a problem for them.

When you write a library, your users are other engineers like yourself. If your goal is to build an app that changes the world, this isn’t the fastest path. But if you’re driven to feel like a real engineer, someone fully in control of their tools and their craft, this is the way.

Some people may say writing a library will take 10 pounds off and make people laugh harder at your jokes. I can’t speak to that, but it will give you something equally elusive: agency. The agency I was severely missing sitting in the conference room praying one of the libraries I found would finally work. No more fingers crossed that the next one will magically work. You get to be the person who writes the thing others depend on.

This post is about libraries, but this is also why I started From Scratch Code. This skill set we share, which is sometimes valued by the job market and sometimes not, is something no one can take away from you. Why not use it to add to the canon of code in the world?

Writing a library is the best first step to own your craft deeply. If you want to get started and aren’t sure how, I’d love to help.

Memphis goes online: adding socket support

Mon, 03 Nov 2025 00:00:00 GMT

Before we begin, let’s agree that “online” in this context means accepting bytes from localhost. Deal? Deal.

You may recall that sometime last century I had the brilliant idea to boot a Flask server from inside Memphis. Trudging toward that goal exposed me to a ton of types and functionality I didn’t yet support. I even ran an interpreter inside an interpreter! But it was a slog. Literally line 1 imports something from the stdlib which imports something from the stdlib which imports something else from the stdlib. I never got to line 2. At a certain point, I realized there was no light at the end of the tunnel.

Skip to a few weeks ago, it hit me I could make my whole life easier by writing my own HTTP library in Python, wire up the necessary system resources from the Rust side, and start running some curl commands.

This post documents Memphis’ journey from being air-gapped to responding in kind:

> curl localhost:8080
Hello from Enid (powered by Memphis)

We’ll approach this like this:

What we want a net module for Memphis to look like
How I implemented this
Introducing Enid, a micro HTTP framework to wrap this up (and allow us to pretend we got Flask working all along)

A `net` module for Memphis

The surface area to respond to an HTTP request is surprisingly small. (Maybe that’s why I keep reimplementing them? Let’s put a pin in that for later.)

Essentially, we need to:

Listen on a port
Accept new connections on that port

That’s it!

Here’s a minimal interface to do this.

from memphis.net import listen

sock = listen(("127.0.0.1", 8080))
print("Listening...")
conn, _ = sock.accept()
data = conn.recv(1024)
print("Sending...")
conn.send(b"HTTP/1.1 200 OK\r\n\r\nHello from Memphis!")
print("Closing...")
conn.close()

If you want to skip ahead, here’s a short video showing this in action.

This code should block on the sock.accept(), just like any socket-based server, until it receives a connection on localhost:8080.

In other words, this is the simplest possible Memphis web server, one that says hello over TCP. And we format the bytes we send back as HTTP so we can test this with curl.

You’ll notice we’re importing memphis.net.listen rather than socket like you would in CPython. That’s what we have to build. From scratch!

To do so, we’ll need:

a memphis.net module, which contains
a listen function, which, when called, returns
a Socket object, which contains an accept method which, when called, returns
a Connection object, which contains recv, send, and close methods.

That’s a lot to wire up! But we can split this into two parts:

the wiring up of that laundry list
connecting the Socket and Connection objects to use Rust utilities to actually do things

This is the paradox of implementing a programming language: you spend 90% of your time on the piping, and then just call your host language to do the real work.

Here our tools are Rust’s TcpListener and TcpStream. Let’s take a look at how we can connect those to our Python code.

Implementing the `net` module

1. Building the module

Our first step is to create a memphis.net module, which will contain the symbols listen, Socket, and Connection.


fn builtins() -> Vec<Box<dyn CloneableCallable>> {
    vec![Box::new(NetListenBuiltin)]
}

fn init(type_registry: &TypeRegistry) -> Module {
    let mut net_mod = Module::new(Source::default());
    for builtin in builtins() {
        net_mod.insert(&builtin.name(), TreewalkValue::BuiltinFunction(builtin));
    }

    register_native_class::<Socket>(&mut net_mod, "Socket", type_registry);
    register_native_class::<Connection>(&mut net_mod, "Connection", type_registry);

    net_mod
}

pub fn import(module_store: &mut ModuleStore, type_registry: &TypeRegistry) {
    let net_mod = init(type_registry);

    let memphis_mod = module_store.get_or_create_module(&ImportPath::from("memphis"));
    memphis_mod.borrow_mut().insert(
        "net",
        TreewalkValue::Module(Container::new(net_mod.clone())),
    );

    module_store.store_module(&ImportPath::from("memphis.net"), Container::new(net_mod));
}

We store our new module in the ModuleStore, which will allow us to find our module when a user’s code imports it.

Next, let’s take a look at the two native structs that back the classes we just registered.

2. Defining our native structs

First, here is our native Socket, which is a thin Rust wrapper.

use std::{
    io,
    net::{SocketAddr, TcpListener, TcpStream},
};

pub struct Socket {
    listener: TcpListener,
}

impl Socket {
    pub fn new(host: String, port: usize) -> io::Result<Self> {
        Ok(Self {
            listener: TcpListener::bind(format!("{}:{}", host, port))?,
        })
    }

    pub fn accept(&self) -> io::Result<(TcpStream, SocketAddr)> {
        self.listener.accept()
    }
}

And our native Connection, another thin Rust wrapper.

use std::{
    io::{self, Read, Write},
    net::{Shutdown, TcpStream},
};

pub struct Connection {
    stream: TcpStream,
}

impl Connection {
    pub fn new(stream: TcpStream) -> Self {
        Self { stream }
    }

    pub fn recv(&mut self, bufsize: usize) -> io::Result<Vec<u8>> {
        let mut buffer = vec![0; bufsize];
        let n = self.stream.read(&mut buffer)?;
        Ok(buffer[..n].to_vec())
    }

    pub fn send(&mut self, data: &[u8]) -> io::Result<()> {
        self.stream.write_all(data)
    }

    pub fn close(&mut self) -> io::Result<()> {
        self.stream.shutdown(Shutdown::Both)
    }
}

It’s worth mentioning that these are both synchronous. I could integrate these with tokio down the road, but that felt like a can of worms best left for future me.

These structs encapsulate the socket behaviors we need, but we still need to connect those up to their Python bindings. You’ll notice these two structs don’t reference our actual interpreter logic: no Module, no knowledge they will be used to evaluate Python code. That’s on purpose!

To make them useful, we’ll bridge our native structs to our Python type system.

3. Wiring them into the type system

In the snippet below, we register our builtin methods to Socket and Connection. impl_method_provider is a macro to reduce the boilerplate to assign our methods to a type.

impl_method_provider!(Socket, [AcceptBuiltin]);
impl_method_provider!(Connection, [ConnRecv, ConnSend, ConnClose,]);

fn register_native_class<T: MethodProvider>(
    mod_: &mut Module,
    name: &str,
    type_registry: &TypeRegistry,
) {
    let object_class = type_registry.get_type_class(&Type::Object);
    let type_class = type_registry.get_type_class(&Type::Type);

    let mut class = Class::new_direct(name, Some(type_class.clone()), vec![object_class.clone()]);

    for builtin in T::get_methods() {
        class.set_on_class(&builtin.name(), TreewalkValue::BuiltinMethod(builtin));
    }

    mod_.insert(name, TreewalkValue::Class(Container::new(class)));
}

In register_native_class, we create a Python Class for each and add these two new classes to our new memphis.net module, each with the specified builtin methods.

With this in place, we can finally begin filling out our builtin methods, starting with NetListenBuiltin.

4. Adding builtin functions

In Memphis, all builtin functions or methods are just structs which implement the Callable trait. This trait requires the struct provide its name and implement a call method. The struct itself doesn’t need any fields because those will all be passed in as runtime parameters.

When a builtin is invoked, it’s provided a reference to the interpreter &TreewalkInterpreter and any arguments Args. The former is needed to access any other runtime resources or raise errors, the latter because that’s how functions work.

pub struct NetListenBuiltin;

impl Callable for NetListenBuiltin {
    fn call(&self, interpreter: &TreewalkInterpreter, args: Args) -> TreewalkResult<TreewalkValue> {
        check_args(&args, |len| len == 1, interpreter)?;

        let host_port = args.get_arg(0).as_tuple().raise(interpreter)?;
        let host = host_port.first().as_str().raise(interpreter)?;
        let port = host_port.second().as_int().raise(interpreter)?;

        let socket = Socket::new(host, port as usize)
            .map_err(|e| interpreter.runtime_error_with(format!("Failed to bind Socket: {}", e)))?;

        let socket_class = interpreter
            .state
            .read_class(&ImportPath::from("memphis.net.Socket"))
            .ok_or_else(|| interpreter.runtime_error_with("Socket class not found"))?;

        let obj = Object::with_payload(socket_class.clone(), socket);

        Ok(TreewalkValue::Object(Container::new(obj)))
    }

    fn name(&self) -> String {
        "listen".into()
    }
}

In this builtin, we:

evaluate and type check the positional arguments
create a new Socket
look up the memphis.net.Socket class
create an Object of this class and provide it a Socket

For this last step to work, we must add native object support to our Python Object. Let’s take a look at that now.

5. Storing native payloads on objects

We start by adding a new field to our Object, an optional Box<dyn Any>. I considered using an enum here, something like Option<NativeResource>, but I was stubborn and went with dynamic dispatch. I’ll explain why shortly.

pub struct Object {
    class: Container<Class>,
    scope: Scope,
    native_payload: Option<Box<dyn Any>>, // this is new!!!!!
}

We are now up to the sock.accept() call.

struct AcceptBuiltin;

impl Callable for AcceptBuiltin {
    fn call(&self, interpreter: &TreewalkInterpreter, args: Args) -> TreewalkResult<TreewalkValue> {
        let self_val = args.get_self().raise(interpreter)?;
        let socket = self_val.as_native_object::<Socket>().raise(interpreter)?;

        let (stream, addr) = socket.accept().map_err(|e| {
            interpreter.runtime_error_with(format!("Socket.accept() failed: {}", e))
        })?;

        let conn = Connection::new(stream);

        let conn_class = interpreter
            .state
            .read_class(&ImportPath::from("memphis.net.Connection"))
            .ok_or_else(|| interpreter.runtime_error_with("Connection class not found"))?;

        let conn_obj = Object::with_payload(conn_class.clone(), conn);

        Ok(TreewalkValue::Tuple(Tuple::new(vec![
            TreewalkValue::Object(Container::new(conn_obj)),
            TreewalkValue::Str(Str::new(&addr.to_string())),
        ])))
    }

    fn name(&self) -> String {
        "accept".into()
    }
}

The key line here is as_native_object::<Socket>(), which reads our new Object.native_payload field and attempts to downcast it to a Socket object.

If I had gone with an enum, I could have used an as_socket() pattern or similar, but I just didn’t want to. There are other potential benefits to using Box<dyn Any>, like dynamic registration of native types, but that feels a ways off.

I won’t show the recv, send, and close methods on Connection, but they work very similarly to accept.

With that, our memphis.net module is wired up and implemented. Thanks for sticking with me! I know those code snippets were a slog.

To recap, we:

created a new memphis.net module
populated the module with a listen builtin function and Socket and Connection classes
added native object support to our Object, with the ability to downcast to retrieve the native objects
created a native Socket and Connection using TcpListener and TcpStream in Rust
created builtins for Socket.accept, Connection.recv, Connection.send, and Connection.close

Before we stop, let’s make our HTTP server a bit more fun.

Introducing Enid

With the net module hooked up and working, I wanted to make my HTTP calls look more like Flask, with its decorator usage and general clean feel. I chose to call this Enid, because that’s another small town in the middle of the US.

Our new API looks like this.

from enid import App, Response

app = App()

@app.get("/")
def home(req):
    return Response.text("Hello from Enid (powered by Memphis)")

if __name__ == "__main__":
    app.run()

Any calls to the root / should return a 200 with a text response, everything else should return a 404. Here’s an example curl session.

> curl localhost:8080
Hello from Enid (powered by Memphis)
> curl localhost:8080/
Hello from Enid (powered by Memphis)
> curl localhost:8080/another_endpoint
404 Not Found
> curl localhost:8080/anything_else
404 Not Found
> curl localhost:8080
Hello from Enid (powered by Memphis)

Wrapping up

Implementing socket support also revealed several Python details I’d missed. I didn’t support __name__, I had a ROUGH kwargs bug, plus I needed to implement str.encode, str.join, str.split, and bytes.decode.

If you want to try it on GitHub, you can run an Enid app on either Python or Memphis. With ChatGPT’s assistance, I wrote a shim so that it will try to import from Memphis, then fallback to Python’s stdlib, like so:

try:
    from memphis.net import listen
except ImportError:
    # Fallback to the pure-Python shim
    from .shim_net import listen

This whole project took about two weeks, which is massively shorter than if I’d kept on the road to Flask nirvana. The wiring to reach the inside of the listen function took about a day, getting the rest of the stdlib and pipes working about a week, and the last week was integrating it into the type system.

To show why the type system work was necessary, this snippet shows two things I wasn’t able to do even when I could technically respond to a curl command.

from memphis import net

type(net.Connection) # <class 'type'>
"send" in dir(net.Connection) # True

This reflection was only possible once we registered the Connection class into memphis.net and assigned it methods. This, plus adding the native payload to a Python object, was just as rewarding as seeing the curl work because it meant Memphis is growing up. And while I can’t say this was easy, I also don’t feel like the code is all going to fall apart tomorrow. So we proceed!

There’s a ton here I could do next. Maybe I’ll support route pattern matching (/endpoint/{id}) and middleware in Enid, or get this all working on my bytecode VM. I could even try to wrap my head around what an async version of this would look like. The order I tackle these will probably depend on how I feel when I wake up tomorrow.

With that, Memphis is officially online! Just don’t count on it to respond to your Slack messages.

Python’s operator chaining is mildly interesting

Mon, 20 Oct 2025 00:00:00 GMT

Nary a week ago, I typed a harmless expression into my Memphis REPL: 4 == 4 == 4.

My REPL’s response? False.

I’m glad it can speak its mind, but the gall!

Apparently, you can’t just treat a chain of comparison operators like a tree of binary expressions. According to my BlueSky timestamps, it took me 13 minutes to figure this out. And I’m here today so you don’t make the same misunderstanding I did.

Fixing the parser

Expressions in Python are parsed in a left-associative way, meaning something like 2 + 3 + 4 will be parsed as (2 + 3) + 4. This works fine for numerical operations like addition, and we can rely on operator precedence and parentheses to control evaluation order.

If you apply this same approach to comparison operators, like I was doing, you get this: (4 == 4) == 4. Which reduces to True == 4. Which is False. Where I thought my REPL was giving me sass, it was just doing math.

How can we fix this? We need to move from my tree-based approach to a flat structure, something that records all the operators and operands in order. I ended up introducing a new Expr variant to represent these chains.

enum Expr {
    ...
    BinaryOperation {
        left: Box<Expr>,
        op: BinOp,
        right: Box<Expr>,
    },
    ComparisonChain {
        left: Box<Expr>,
        ops: Vec<(CompareOp, Expr)>,
    },
    ....
}

You may be wondering why we need to store the CompareOp in between each right operand. My first thought was that we’d only need to store it once! As I looked more into the Python rules (AKA asked ChatGPT for test cases), I learned that Python allows you to write heterogeneous chains. Take these examples.

a = 1 < 2 == 2 < 3                    # True
b = 1 == 2 != 3                       # False
c = 2 in [1,2,3] in [[1,2,3],[4,5,6]] # True

Honestly, I’d be pretty annoyed if I saw any of these in code. But Python allows them, so we must fall in line.

Here are my new parser tests, along with a Rust macro to make it easier to build these expressions.

#[test]
fn operator_chaining() {
    let input = "a == b == c";
    let expected_ast = cmp_chain!(var!("a"), [(Equals, var!("b")), (Equals, var!("c")),]);

    assert_ast_eq!(input, expected_ast, Expr);

    let input = "a == b < c > d";
    let expected_ast = cmp_chain!(
        var!("a"),
        [
            (Equals, var!("b")),
            (LessThan, var!("c")),
            (GreaterThan, var!("d")),
        ]
    );

    assert_ast_eq!(input, expected_ast, Expr);
}

Now that the parser knows how to build these chains, we can begin to interpret them correctly.

Fixing the treewalk interpreter

The treewalk implementation is closest to my mental model, so I began there. The main trick is to reuse the right-hand side of each evaluation in the next comparison. Once we hit the first False result, we can short-circuit the entire operation. If we got to the end without seeing any False results, we are free to return True.

fn evaluate_comparison_chain(
    &self,
    left: &Expr,
    ops: &[(CompareOp, Expr)],
) -> TreewalkResult<TreewalkValue> {
    let mut left = self.evaluate_expr(left)?;

    for (op, right) in ops {
        let right = self.evaluate_expr(right)?;

        // ideally, we wouldn't need to clone `right` here, but this requires ownership for now.
        let result = self.evaluate_compare_op(left, op, right.clone())?;
        if !result.as_boolean() {
            return Ok(TreewalkValue::Bool(false));
        }

        left = right;
    }

    Ok(TreewalkValue::Bool(true))
}

The other fun part about the treewalk interpreter is that it’s further along, which means I support operator overloading. Each time a comparison chain evaluates a particular operation, under the hood it is calling a dunder method on an object.

fn evaluate_compare_op(
    &self,
    left: TreewalkValue,
    op: &CompareOp,
    right: TreewalkValue,
) -> TreewalkResult<TreewalkValue> {
    use CompareOp::*;

    match op {
        In => self.invoke_method(&left, &Dunder::Contains, args![right]),
        NotIn => {
            let contains = self.invoke_method(&left, &Dunder::Contains, args![right])?;
            Ok(contains.not())
        }
        Equals => self.invoke_method(&left, &Dunder::Eq, args![right]),
        NotEquals => self.invoke_method(&left, &Dunder::Ne, args![right]),
        LessThan => self.invoke_method(&left, &Dunder::Lt, args![right]),
        GreaterThan => self.invoke_method(&left, &Dunder::Gt, args![right]),
        LessThanOrEqual => self.invoke_method(&left, &Dunder::Le, args![right]),
        GreaterThanOrEqual => self.invoke_method(&left, &Dunder::Ge, args![right]),
        Is => Ok(TreewalkValue::Bool(left.is(&right))),
        IsNot => Ok(TreewalkValue::Bool(!left.is(&right))),
    }
}

This part technically already existed, but there’s something satisfying about watching a new feature click into an old mechanism. These comparisons used to live inside my binary operator evaluation, back when CompareOp didn’t exist and comparison chains and I didn’t yet understand each other.

With the treewalk interpreter working smoothly, it was time to bring the same behavior to the bytecode VM.

Fixing the bytecode VM

When I sat down to write this section, I read through my code to begin to explain it. In classic rubber-duck fashion, I realized I’d overcomplicated the whole thing. I now present the simplified pseudocode of how the bytecode looks for operator chaining.

evaluate left
for each (op, right):
    evaluate right
    compare op
    if false: goto end
    if not last iteration: pop true
    else: leave it
end:

Remember how we had to save the right-hand side for the next comparison? That’s what we’re still missing in this pseudocode. The line compare op will pop two operands off the stack and compare them, consuming both operands in the process. If we want the right operand to still be around afterwards, we’re gonna need to intervene.

We can do this by running a DupTop and a RotThree before we do the actual comparison. (CPython uses COPY and SWAP.) The former makes a copy of the right-hand operand, while the latter moves it to the third spot on the stack (hence the rotate three).

Here’s what this looks like in Rust.

fn compile_comparison_chain(
    &mut self,
    left: &Expr,
    ops: &[(CompareOp, Expr)],
) -> CompilerResult<()> {
    if ops.is_empty() {
        panic!("Comparison chain must have >= 1 op.");
    }

    self.compile_expr(left)?;

    let mut false_jumps = vec![];
    for (i, (op, right)) in ops.iter().enumerate() {
        let last_op = i == ops.len() - 1;

        self.compile_expr(right)?;

        // Preserve the right-hand side for the next comparison
        if !last_op {
            self.emit(Opcode::DupTop)?;
            self.emit(Opcode::RotThree)?;
        }

        self.emit(Opcode::from(op))?;

        // If any comparison evaluates to False, jump to end.
        // Otherwise, pop the True and continue the chain.
        // Unless it's the last operation, and we should leave the result on the stack.
        if !last_op {
            // At the end of the loop, we will patch this with a JumpIfFalse
            false_jumps.push(self.emit_placeholder()?);

            self.emit(Opcode::PopTop)?;
        }
    }

    // Patch all fail jumps to here
    for placeholder in false_jumps {
        let offset = self.forward_offset_to(placeholder)?;
        self.emit_at(placeholder, Opcode::JumpIfFalse(offset))?;
    }

    Ok(())
}

The updates to the VM itself were minimal, we just needed to support the new DupTop and RotThree.

    Opcode::DupTop => {
        let x = self.peek()?;
        self.push(x)?;
    }
    Opcode::RotThree => {
        // Before:
        // c <- TOS
        // b
        // a
        //
        // After:
        // b <- TOS
        // a
        // c
        let c = self.pop()?;
        let b = self.pop()?;
        let a = self.pop()?;
        self.push(c)?;
        self.push(a)?;
        self.push(b)?;
    }

My initial implementation only worked when all the operands were equal, like 2 == 2 == 2. That told me I had a bug in the RotThree implementation which I corrected (and left better notes behind for future-me).

Putting it all together

With both interpreter engines updated, it was time to add tests to crosscheck. Here’s a subset of the chaining tests which are run through both engines.

#[test]
fn operator_chaining() {
    let input = "2 == 2 == 2";
    assert_crosscheck_return!(input, MemphisValue::Boolean(true));

    let input = "2 == 2 == 22";
    assert_crosscheck_return!(input, MemphisValue::Boolean(false));

    let input = "2 == 2 == 2 < 3";
    assert_crosscheck_return!(input, MemphisValue::Boolean(true));

    let input = "1 < 2 < 3 < 4";
    assert_crosscheck_return!(input, MemphisValue::Boolean(true));

    let input = "1 < 2 < -3 < 4";
    assert_crosscheck_return!(input, MemphisValue::Boolean(false));

    let input = "1 < 2.0 < 3";
    assert_crosscheck_return!(input, MemphisValue::Boolean(true));

    let input = "2 in [1,2,3] in [[1,2,3],[4,5,6]]";
    assert_crosscheck_return!(input, MemphisValue::Boolean(true));

    let input = "2 in [1,2,3] in [[4,5,6]]";
    assert_crosscheck_return!(input, MemphisValue::Boolean(false));
}

👉 Curious what these examples compile to? Try it live using Ozark, the interactive bytecode compiler: fromscratchcode.com/ozark

Sometimes I get lazy and/or scared and don’t add the full test suite to crosscheck, but I was determined to do this feature right. (hence the blog post.)

Along the way, I realized I’d implemented 2 < 2.5 but not 2.5 < 3. The first calls Dunder::Lt on int, while the second calls it on float. It’s fun to catch and quickly fix oversights like this. In addition, 2 == 2.0 never returned True. This was because I’d assumed that any cross-type comparison should always return False.

The end

Operator chaining isn’t a flashy feature, yet implementing it reminded me what I love about continuing to develop Memphis: the more Python behaviors I properly model—across two engines—the purer my abstractions become. And what is expertise in software, if not fluency in abstractions? That’s what keeps me coming back to build this interpreter no one asked for.

What I’ve learned from 200+ hours helping developers grow

Mon, 14 Jul 2025 00:00:00 GMT

This isn’t a how-to guide. Just a few things I’ve noticed while mentoring people 1:1 over the last two years.

One-on-one mentorship wasn’t even part of my career plan. I viewed myself as a quiet builder and, hopefully, a good teammate.

This journey started as most things do in my life: an experiment because I was curious. I had an itch for more 1:1 conversation and a hope that I could help a person or two with Python, so I made a profile on Wyzant.

Two years and more than 200 hours later, it’s become one of the most human parts of my work. I’ve worked with college students, self-taught developers, and software professionals, focusing on helping them build confidence and clarity in topics like Python, Rust, React, and even digital logic and computer architecture.

Here’s what I’ve learned.

You don’t need to be extroverted

One of my early fears, both in mentorship and in starting my own business, was that I would fail if I wasn’t “on.” I wasn’t sales-y, and I worried I wouldn’t know how to manage a conversation that got away from me. I was even scared I wouldn’t know how to cut off a session at the end of the hour!

But none of that turned out to matter.

In fact, the structured sessions ended up being exactly what my brain needed. Already knowing someone wants to talk to me when I hop on a video call, even if it is loosely transactional, puts me at ease.

This let me focus on the real work: showing up respectfully, staying curious about what brought someone in, and posing the right question at the right time.

If they’re not asking questions, that might be a good sign

Sometimes I’ll catch myself thinking, Why aren’t they asking me anything? as I watch a mentee explore something on their own.

But I stop myself, because that’s actually a good sign. It means they’ve taken the wheel, and something about our environment still feels useful to them. My challenge then becomes to provide encouragement and nudge them with bigger questions.

A few months ago, one of my mentees was thinking aloud and said “I bet you’re gonna tell me to ask ChatGPT.” I smiled because it showed they had internalized when to use one of their tools. I was still there to help them interpret the next steps, but they were able to take the first step on their own.

Emotional safety looks different for everyone

Our first sessions are often knee-deep in technical detail, but as we zoom out, I try to get a sense of the environment around the work.

How do they like their manager? For the college students, do they have friends in class?

That stuff is often lurking beneath the surface. A lonely or unsupportive environment can be just as challenging as a gnarly stack trace. And I want to make space for both.

Not everyone engages deeply on those questions, but it doesn’t mean they won’t come back. I’ve had consistent mentees where we go deep into mental health issues and others where we never get past the weather. Learning to be okay with that has been part of the work too.

Code is often just the entry point, we stay for the connection

You hear this kind of thing about food or music, but I can’t say I hear many people say, “I code to connect with people.” As a craft, it can be the ultimate solitude. Even when folks talk about using code “for good,” it’s often from a distance. Which is fine!

But for me, it’s more personal. Nearly all of my close friends came from engineering school or work. It was an environment I felt confident in, and that confidence gave me a safe harbor from which to go meet people. Now I try to offer that same sense of support to people who might not have had it elsewhere.

People often reach out to get unstuck technically, which gets us into the space together. But I really enjoy the chance to hear what’s going on in their lives. What’s coming up after graduation, what’s been weighing on them.

I don’t get that deep with everyone I work with, but after three or so sessions we usually have some rapport. My goal is that they leave feeling more seen than when they came in, ideally with their code running.

That’s everything I’ve learned. Nothing else! Zip, zero, zilch.

Oh, and Vite replaced Create React App while I was asleep.

If you’re looking for technical mentorship that includes both the code and the human behind it, I’d love to work with you. Book a free intro call, and let’s get started.

This was cross-posted on From Scratch Press.

How global variables work in Python bytecode

Mon, 16 Jun 2025 00:00:00 GMT

Think globally, act locally. Who knew this oft-touted phrase was referring to Python bytecode?

Last time we acted on learning how local variables work, today let’s think about how global variables might work.

I came into my own bytecode journey assuming that a “global” was no different than a “local,” it was just at the outermost scope of a module. And boy, was I mistaken.

While the VM isn’t even aware of a local variable’s name, just its index, the VM performs dynamic name resolution to resolve each global variable. This is a key part of Python’s dynamism.

Once again, this post will discuss the mechanics of Memphis, my Python interpreter built in Rust. While it doesn’t match CPython 100%, the model is simple and maps closely to how most Python runtimes behave under the hood.

Bytecode Compilation

Consider the following Python program.

# A global variable which we'll later reference inside our function
y = 11

# A function which adds an unknown number (via global var) to the input
# parameter x, returning the result.
def add_useless(x):
    return x + y

👉 Curious what this compiles to? Try it live using Ozark, the interactive bytecode compiler: fromscratchcode.com/ozark

Here is the bytecode for our function, add_useless:

LOAD_FAST    0 (x)
LOAD_GLOBAL  0 (y)
ADD
RETURN_VALUE

Our CodeObject looks like the one below. Again, pardon my pseudo-Rust syntax! Or...

CodeObject {
  name: "add_useless",
  varnames: ["x"], // names of each local accessed by the function
  names: ["y"], // names of each global accessed by the function
}

We don’t see a constant of 11 here because this code object is specific to our add_useless function. The function will have to look up that value at runtime via the global store, since it isn’t embedded in the code object itself.

If you haven’t read the previous post, I’d encourage you to pause here and give it a read. It walks through the evaluation stack step-by-step and shows how the ADD operation gets its two operands off the stack.

VM Execution

What we’re interested in today is what, exactly, LOAD_GLOBAL is doing. First, let’s define a runtime concept I blew past earlier.

Global Store: a mapping from a variable name to an object reference. The object being referenced lives on the heap. This is roughly what you get when you call globals() in Python or CPython.

Let’s see what happens when the VM begins executing the bytecode. Like last time, we’ll ignore how the bytecode actually enters the function.

LOAD_FAST 0 means: read from slot 0 on the frame’s execution stack.
LOAD_GLOBAL 0 means: look up the identifier in names at index 0. It finds y, then goes to the global store to look for the key y, where it finds an object reference to the value of 11.
The remaining instructions, ADD and RETURN_VALUE, work just as they did in the previous post: the VM pops two operands off the stack, computes the result, and returns it.

You may have noticed: global variables don’t need to be defined when the function is defined, only when it is called. If y doesn’t exist at function call time, a NameError will be thrown. This differs from local variables, which must exist at the time they are first referenced. Otherwise, the bytecode compiler will assume you are referring to a global with that same name.

See how there is an extra level of indirection when resolving the global compared to the local? This is key to Python’s dynamism. Here’s an example to illustrate this.

y = 11

def add_useless(x):
    return x + y

print(add_useless(9)) # prints 20

y = 1

print(add_useless(9)) # prints 10

This example highlights the perils of impure functions, but otherwise might not seem that surprising at first.

Before we continue, it’s worth mentioning that the global store is specific to the module, while the heap is shared across modules. This means that multiple modules can have a global named y, which matches what we expect as users. Imagine having to know if every other module in your project, including those from third-party libraries, had previously used a global identifier? That would be a nightmare!

Global Store Mutations

To further illustrate how globals can be modified, let’s consider two more examples. These below move away from my Memphis implementation and focus on how dynamic global behavior shows up in CPython.

a = "First"
globals()['a'] = "Second"
print(a) # prints "Second"

This one accomplishes something similar to the previous example (reassigning a global), but by mutating the global store directly via globals(). We see here that a doesn’t point to a fixed memory location, but is resolved dynamically by name at runtime.

To take it one step further, consider the same idea applied across modules. We’ll also reassign a function rather than an integer.

# main.py
import other

def monkey_patch_foo():
    print("Got ya!")

other.foo = monkey_patch_foo
other.bar() # prints "Got ya!"

# other.py
def foo():
    print("I am foo.")

def bar():
    foo()

Our main module modifies the behavior of a function defined in another module, and that change affects the other module itself. Said another way: you can change the behavior of code from the outside, and the function itself doesn’t know.

This is referred to as monkey patching and is one of the most mind-bending things about Python. Monkey patching is often used to inject patch fixes or replace functionality at runtime. And it’s only possible because of Python’s dynamic name resolution for global variables.

To be transparent, Memphis doesn’t fully support this yet! I'm still working through object mutability and shared references, and how these vary across my treewalk implementation and bytecode VM. Monkey patching: WIP.

You might wonder why Python treats locals and globals so differently in bytecode.

The reason is to balance performance and flexibility. Functions are called frequently during the lifetime of a program, its locals are accessed often, so Python optimizes for speed using slot-based indexing. At the module level, the dynamic name resolution introduces a performance hit, but the language is more dynamic as a result.

The End

I didn’t fully appreciate this design until I implemented it myself. You really start to see how much of Python’s feel comes from how its variables are resolved in bytecode.

If you’re curious to explore more, I recommend trying out the dis module to inspect your own functions! It’ll be confusing at first, but you may begin to pick up one new tidbit each time.

Now that we’ve tackled locals and globals, we’re free to think about free variables next! I hope you’ll stay tuned.

Lastly, I’m continuing to experiment with open office hours this week. If you’re stuck in Python or Rust and want to talk through it live, here’s the booking link. The slots are pay-what-you-want, with zero pressure to tip. I’d love to meet you!

How local variables work in Python bytecode

Mon, 02 Jun 2025 00:00:00 GMT

Local variables are stored on the stack.

This line tormented me for years. I could spout it in an interview, but I didn’t really know what it meant. I could pull up the diagram showing the stack growing upwards and the heap growing downwards. And I definitely didn’t want to think about what happened if they ever crossed.

But to truly understand? I had to build it myself.

This post explains how local variables work by walking through their mechanics in Memphis, my Python interpreter built in Rust. While it doesn’t match CPython byte-for-byte, the model is simple and maps closely to how most Python runtimes behave under the hood.

If you’ve read my previous posts, you’ll recall I have two implementations: a treewalk interpreter and a bytecode VM. This post focuses on the bytecode VM because it had a steeper learning curve, blew my mind more often, and is closer to CPython.

Let’s begin with some definitions which will be used by our bytecode VM at runtime.

Bytecode VM concepts

CodeObject: An immutable structure produced by the bytecode compiler and consumed by the VM.

FunctionObject : A runtime representation of a CodeObject plus its captured environment. We’ll need this layer for when we introduce free variables, which underpin closures.

Frame: A runtime representation of a FunctionObject. It holds the program counter and the evaluation stack, including the space reserved for local variables.

CallStack: A stack of Frame objects, which tracks each execution context (i.e each new function call).

What the heap?!

Wait a minute. Are local variables, which so famously live on the stack, stored in the Frame or the CallStack? The answer is yes!

Each Frame contains its own evaluation stack, the bottom portion of which is reserved for local variables. This is what the term “stack-based virtual machine” refers to, and is in contrast to a register-based VM. Both are mechanisms for managing working memory during your program’s execution.

Meanwhile, the CallStack is one layer of abstraction higher. It holds a new Frame for each new function which is entered.

Before I go any farther, I want to address the elephant in the room. In addition to the incantation local variables are stored on the stack, I was pretty sure that objects in Python are stored on the heap. What gives?

Both things are true. Local variables are stored on the stack, but most variables in Python are references. The stack holds the pointer to the object (which could an int, string, list, custom object, etc.), while the heap does hold the actual data.

Let’s walk through a short function to see this in action.

The stack in action

Consider the following function. It takes in one parameter and defines one additional local variable. For the scope of this discussion, let’s assume our VM has already called our function, add_useless.

# A function which adds an unknown number (via local var) to the input
# parameter x, returning the result.
def add_useless(x):
    y = 11
    return x + y

👉 Curious what this compiles to? Try it live using Ozark, the interactive bytecode compiler: fromscratchcode.com/ozark

Remember how we said the bottom portion of the Frame's evaluation stack is reserved for local variables? We can visualize it like this.

┌────────────────┐
|   stack[1]     │
├────────────────┤
|   stack[0]     │
├────────────────┤
│   locals[1]    │
├────────────────┤
│   locals[0]    │
└────────────────┘

Before we set y = 11, the stack would hold two reserved spaces for our two local variables, x and y.

┌──────────────────┐
│ empty slot for y │
├──────────────────┤
│    value of x    │
└──────────────────┘

During the evaluation of y = 11, the stack briefly looks like this:

┌──────────────────┐
│         11       │
├──────────────────┤
│ empty slot for y │
├──────────────────┤
│     value of x   │
└──────────────────┘

After we set y = 11, the stack would be:

┌────────────┐
│    11 (y)  │
├────────────┤
│ value of x │
└────────────┘

During the evaluation of x + y, the stack briefly looks like this:

┌────────────────┐
|       11       │
├────────────────┤
|   value of x   │
├────────────────┤
│      11 (y)    │
├────────────────┤
│   value of x   │
└────────────────┘

After we add x + y, but before the return the stack would be:

┌────────────────┐
│ value of x + y │
├────────────────┤
│      11 (y)    │
├────────────────┤
│   value of x   │
└────────────────┘

Then we return the sum, discard the frame, and hope nothing blows up. (Can you tell I don’t want to explain that part right now?)

If you are screaming at your screen “why is only half of our stack the stack,” I get it! It confused me too that is one piece of contiguous memory, but we reserve the bottom n slots for n local variables.

So how do we know how many local variables to reserve? Because we already compiled the function’s bytecode before running it. Let’s take a look at how that works.

The bytecode in action

Before we look at the bytecode itself, let’s define a few more concepts. These are both fields on the CodeObject and are populated during the bytecode compilation.

constants : The list of constants taken directly from the user code. Think: any integer, float, bool, or string. Because CodeObjects themselves are immutable, they can also appear as constants. In Rust, I represent this with an enum to allow multiple types in the same list.

varnames: The list of identifiers for each local variable in a given CodeObject. In a previous post, I described how varnames (for locals) differs from names (for globals).

During compilation, each identifier assigned to (either as a function parameter or a local variable) is pushed into the varnames list of the current CodeObject. (We can ignore the global keyword for now.)

Whenever the compiler encounters a new function definition, it creates a fresh CodeObject and pushes it onto a code stack. This stack is used only during compilation to track lexical scope, and is not related to the runtime CallStack. Once a function body is fully compiled, the corresponding CodeObject is popped off and made available as a constant for its parent CodeObject.

Here is the bytecode for our add_useless function. The data in the parentheses annotates what each index refers to. I added this manually from my Memphis output, but you can get a similar output in CPython by running import dis; dis.dis(add_useless), which uses the dis module.

LOAD_CONST  0  (11)
STORE_FAST  1  (y)
LOAD_FAST   0  (x)
LOAD_FAST   1  (y)
ADD
RETURN_VALUE

We’re indexing into two separate lists here, both of which live inside a CodeObject. I write Rust, so naturally I visualized it like this:

CodeObject {
  constants: [11],
  varnames: ["x", "y"],
}

LOAD_CONST indexes into constants, while STORE_FAST and LOAD_FAST index into varnames.

The magic here is there’s no need for the runtime VM to know about the names of the local variables. varnames is used by the compiler to emit the right indices into the bytecode, but during execution, the VM just deals with stack slots.

Once a runtime Frame is initialized, the VM will execute each of these instructions and index into its own execution stack.

The End

If this felt a bit dense, I totally get it! It took me several tries to get this working and clean in my Memphis implementation.

I’m hoping to write similar posts covering global variables and free variables, so I hope you’ll stick around for those.

Lastly, I’m experimenting with open office hours this week. If you’re stuck in Python or Rust and want to talk through it live, here’s the booking link. The slots are pay-what-you-want, with zero pressure to tip. I’d love to meet you!

Verifying two interpreter engines with one test suite

Mon, 05 May 2025 00:00:00 GMT

Crosscheck, my cross-engine testing framework for Memphis, had been sitting atop my writing to-do list for a while.

Instead, I deleted it and replaced it. Memphis can now test itself across all engines.

What does this mean?

I can verify the treewalk interpreter and bytecode VM interpreter produce the same results (return value or symbol table entries) for a given snippet of Python.
This functionality is available in unit tests, rather than only in integration tests. In Rust, this means in src/ rather than tests/.

Let’s look at a couple of examples.

Testing expressions

Any interpreter worth its table salt should be able to evaluate 2 + 2.

[An expression returns a value, whereas a statement modifies control flow or the symbol table. I couldn’t have defined this clearly before I began this project, so I hope this helps.]

#[test]
fn binary_expression() {
    let input = "2 + 2";
    assert_crosscheck_return!(input, MemphisValue::Integer(4));
}

A few building blocks were necessary to get here.

First, I needed to represent a Python value independent of the runtime engine. This isn’t exactly necessary for integers, but come lists or objects, my architecture needed each engine to be able to manage interior mutability differently.

Second, I needed a simple initialization flow, which would hide the complexity of initializing two interpreter engines and running a snippet of code through both. I’d had a few builders over time, but they always did too much. I wanted to truly hide all this behind assert_crosscheck_return!. Speaking of which, let’s look inside.

macro_rules! assert_crosscheck_return {
    ($src:expr, $expected:expr) => {{
        let mut session =
            $crate::crosscheck::CrosscheckSession::new($crate::domain::Source::from_text($src));
        let (tw_val, vm_val) = session.eval();
        assert_eq!(
            tw_val, $expected,
            "Treewalk return value did not match expected"
        );
        assert_eq!(
            vm_val, $expected,
            "Bytecode VM return value did not match expected"
        );
    }};
}

Awesome! It calls something else. Sounds and reads like software.

It’s worth mentioning here why assert_crosscheck_return is a macro rather than a helper function. When a panic happens inside a macro, it reports the line number of your test file, not of the macro itself. This is due to how Rust resolves them before compile-time. Some of my macros call helper functions under the hood, but having the assertions at the macro-level makes debugging way simpler.

We create a CrosscheckSession, which creates two MemphisContext objects. Since this is test-only, I don’t mind loudly failing with expect inside of eval.

pub struct CrosscheckSession {
    treewalk: MemphisContext,
    vm: MemphisContext,
}

impl CrosscheckSession {
    pub fn new(source: Source) -> Self {
        let treewalk = MemphisContext::new(Engine::Treewalk, source.clone());
        let vm = MemphisContext::new(Engine::BytecodeVm, source);
        Self { treewalk, vm }
    }

    /// Run both engines; confirm they return the same value, then return the value. Useful
    /// for evaluating expressions or statements which only return a single value.
    pub fn eval(&mut self) -> (MemphisValue, MemphisValue) {
        let tw_val = self.treewalk.run().expect("Treewalk run failed.");
        let vm_val = self.vm.run().expect("VM run failed.");

        (tw_val, vm_val)
    }
}

Each MemphisContext manages the lexer/parser/interpreter lifetime for a single Engine.

pub struct MemphisContext {
    lexer: Lexer,
    interpreter: Box<dyn Interpreter>,
}

impl MemphisContext {
    pub fn new(engine: Engine, source: Source) -> Self {
        let lexer = Lexer::new(&source);
        let interpreter = init_interpreter(engine, source.clone());

        Self { lexer, interpreter }
    }

    pub fn run(&mut self) -> Result<MemphisValue, MemphisError> {
        let MemphisContext {
            lexer, interpreter, ..
        } = self;

        let mut parser = Parser::new(lexer);
        interpreter.run(&mut parser)
    }
}

Testing statements, classes, and more

Let’s look at a slightly more involved example, a test which defines a class with one attribute and a couple of methods.

#[test]
fn method_call() {
    let mut session = crosscheck_eval!(
        r#"
class Foo:
    def __init__(self, val):
        self.val = val

    def bar(self):
        return self.val

f = Foo(10)
b = f.bar()
"#
    );
    assert_crosscheck_eq!(session, "b", MemphisValue::Integer(10));
}

Our pair of new macros here allows us to start a crosscheck session. Each session will run the provided code snippet through each engine, then stay alive so we can query its symbol table.

macro_rules! crosscheck_eval {
    ($src:expr) => {{
        $crate::crosscheck::CrosscheckSession::new($crate::domain::Source::from_text($src))
            .run()
            .expect("Crosscheck session failed")
    }};
}

macro_rules! assert_crosscheck_eq {
    ($session:expr, $name:expr, $expected:expr) => {{
        let actual = $session.read($name).expect("Symbol not found");
        assert_eq!(actual, $expected);
    }};
}

Let’s look at the two new CrosscheckSession methods, run and read. You’ll notice I break my own rule about keeping assertions at the top level inside the read method. I’ll put a ticket in the backlog to fix that later.

impl CrosscheckSession {
    /// Run both engines; discard the return value and return the session. Useful for
    /// later reads from the symbol table with `CrosscheckSession::read`.
    pub fn run(mut self) -> Result<Self, MemphisError> {
        self.treewalk.run()?;
        self.vm.run()?;
        Ok(self)
    }

    /// Read a value from both engines; confirm they return the same value, then
    /// return the value.
    pub fn read(&mut self, name: &str) -> Option<MemphisValue> {
        let a = self.treewalk.read(name)?;
        let b = self.vm.read(name)?;

        assert_eq!(a, b, "Engines returned different values");
        Some(a)
    }
}

And that’s it! I have a few other macros and flows to validate an expected Python runtime error, but they follow the same ideas as what I’ve shown here.

Why did I delete the old `crosscheck`?

Because it sucked.

Okay not really, it did its job for nearly a year! But it required a lot of boilerplate code. It also expected each test file to manually define what engines to verify (using TreewalkAdapter and BytecodeVmAdapter).

Have a look for yourself:

fn run_binary_expression_test<T: InterpreterTest>(mut interpreter: T) {
    let input = "2 + 2";
    match interpreter.evaluate(input) {
        Err(e) => panic!("Interpreter error: {:?}", e),
        Ok(result) => {
            assert_eq!(result, MemphisValue::Integer(4));
        }
    }
}

#[test]
fn test_treewalk_binary_expression() {
    run_binary_expression_test(TreewalkAdapter::new());
}

#[test]
fn test_bytecode_vm_binary_expression() {
    run_binary_expression_test(BytecodeVmAdapter::new());
}

Early-2024-me was pleased with this, but it is heavy.

I wrote the original as an integration test (where tests live in tests/ rather than src/) because I didn’t know what I was doing. Rust produces a separate binary for integration tests, which was unnecessary for verifying cross-engine behavior.

It also meant I was using memphis as a library — interesting on its own (a Python interpreter you can embed in other Rust code! cool!), but not what I needed here.

The End

This work took roughly a year, though I took about 11 months off in the middle.

If the founder of the Single Responsibility Principle reads this, I hope they are proud of me. Their principle served as a useful North Star.

Along the way, I used MemphisContext to support both engines in the REPL. That’s also the reason the Lexer is kept alive but the Parser isn’t—to support streaming input. But that's a story for another day.

I hope you are well and your code returns the same value every time!

I'm embarrassed by how much code I cut from my test suite

Mon, 31 Mar 2025 00:00:00 GMT

My parser test suite was a house of cards I’ve never been prouder to see collapse. I found a pattern that worked, kept working, and then worked a little too well. Until rust-analyzer couldn't process my file anymore. Best practice says to refactor before your LSP craps out, but alas. Here's how I saved several thousand lines of test code in my Memphis parser.

Designing an expressive parser test suite

I’m going to break all rules of writing and tell this story Benjamin Button style.

We begin with this idyllic test case. Wouldn’t it be lovely to verify our AST in an expressive yet relaxed way? Yes, it is lovely.

#[test]
fn expression() {
    let input = "2 + 3 * (4 - 1)";
    let expected_ast = bin_op!(
        int!(2),
        Add,
        bin_op!(int!(3), Mul, bin_op!(int!(4), Sub, int!(1)))
    );

    assert_ast_eq!(input, expected_ast, Expr);

    let input = "2 // 3";
    let expected_ast = bin_op!(int!(2), IntegerDiv, int!(3));

    assert_ast_eq!(input, expected_ast, Expr);
}

This test wasn’t all sunflowers and rainbows. This 16-line fact-checker used to come in at a bloated 38 lines.

Have a peep yourself.

#[test]
fn expression() {
    let input = "2 + 3 * (4 - 1)";
    let context = init(input);

    let expected_ast = Expr::BinaryOperation {
        left: Box::new(Expr::Integer(2)),
        op: BinOp::Add,
        right: Box::new(Expr::BinaryOperation {
            left: Box::new(Expr::Integer(3)),
            op: BinOp::Mul,
            right: Box::new(Expr::BinaryOperation {
                left: Box::new(Expr::Integer(4)),
                op: BinOp::Sub,
                right: Box::new(Expr::Integer(1)),
            }),
        }),
    };

    match context.parse_oneshot::<Expr>() {
        Err(e) => panic!("Parser error: {:?}", e),
        Ok(ast) => assert_eq!(ast, expected_ast),
    }

    let input = "2 // 3";
    let context = init(input);

    let expected_ast = Expr::BinaryOperation {
        left: Box::new(Expr::Integer(2)),
        op: BinOp::IntegerDiv,
        right: Box::new(Expr::Integer(3)),
    };

    match context.parse_oneshot::<Expr>() {
        Err(e) => panic!("Parser error: {:?}", e),
        Ok(ast) => assert_eq!(ast, expected_ast),
    }
}

The tool I used to reduce my boilerplate was declarative macros, Rust’s way of generating Rust code at compile time.

By the end of my cleanup trance, I improved these areas:

expressing Python types (int, str, bool, list, set, tuple)
expressing operations (binary, unary, and logical ops)
wrapping the actual entrypoint to parse the input
wrapping error handling for the common case

The result? Shorter tests and clearer intentions.

Expressing Python types

I can now write int!(3) instead of Expr::Integer(3). Blah blah blah so what.

This one doesn’t save a whole lot, but let’s look at two more.

I can now write list![int!(1), int!(2), int!(3)] and set![int!(1), int!(2), int!(3)]. Glancing at the implementation for those macros, we see another key.

macro_rules! list {
    ($($expr:expr),* $(,)?) => {
        Expr::List(vec![
            $($expr),*
        ])
    };
}

macro_rules! set {
    ($($expr:expr),* $(,)?) => {
        Expr::Set(HashSet::from([
            $($expr),*
        ]))
    };
}

My parser tests no longer care that an Expr::List accepts a Vec, while a Expr::Set accepts a HashSet. One could argue I don’t need the HashSet at all because this is just the AST, not the evaluation stage of the interpreter. In that case, I could change the underlying representation by updating the macro.

Expressing operations

This one starts to get really fun. I can now write bin_op!(var!("a"), BitwiseAnd, var!("b")), which expands to the following.

macro_rules! bin_op {
    ($left:expr, $op:ident, $right:expr) => {
        Expr::BinaryOperation {
            left: Box::new($left),
            op: BinOp::$op,
            right: Box::new($right),
        }
    };
}

Not only does this macro hide the two Box initializations (necessary in a recursive enum), it also allows us to write BitwiseAnd rather than BinOp::BitwiseAnd, all without sacrificing any type checking.

Wrapping the parser entrypoint

Previously, I had to write this, which isn’t awful, but is also awful.

let context = init("2 + 3");

match context.parse_oneshot::<Expr>() {
   ...
}

We’re working with a MemphisContext object here, another bloated structure I use to orchestrate the whole evaluation flow. The problem is, this is an evolving interface. I learned the hard way that without a level of indirection in my tests, I’d have to tweak this pattern constantly. Across hundreds of tests, that is obnoxious.

The new approach uses a straight forward parse! macro.

let ast = parse!($input, Statement);
assert_stmt_eq!(ast, $expected);

Wrapping the happy path error handling

As Yogi Berra famously said, 90% of unit tests are one-half mental.

I applied this philosophy to design parse! to handle the happy path, the roughly 90% of my parser tests I expect to be able to parse their Python input successfully. Since these are tests and nothing matters, we can fail loudly on any unexpected exceptions and return the AST quietly otherwise.

macro_rules! parse {
    ($input:expr, $pattern:ident) => {
        match init($input).parse_oneshot::<$pattern>() {
            Err(e) => panic!("Parser error: {:?}", e),
            Ok(ast) => ast,
        }
    };
}

And for those 10% of tests where we expect a parse error? This will do just fine.

macro_rules! expect_error {
    ($input:expr, $pattern:ident) => {
        match init($input).parse_oneshot::<$pattern>() {
            Ok(_) => panic!("Expected a ParserError!"),
            Err(e) => e,
        }
    };
}

This is what it now looks like to confirm an improperly structured dict. While I love match statements, I love even more no longer having to write them.

let input = "{ 2, **second }";
let e = expect_error!(input, Expr);
assert_eq!(e, ParserError::SyntaxError);

The End

I’m embarrassed I didn’t do these steps sooner. Please learn from my mistakes and treat your tests the way you wish to be treated.

I left corporate and still do roadmaps + a Memphis update

Mon, 03 Mar 2025 00:00:00 GMT

Happy March!

This month I am putting a bow on my Q1 roadmap and continuing to unify the two Memphis execution engines.

I chose an update post because I want to “build in public.” What follows is what has been on my mind! I don’t write advice posts (Why you shouldn’t use Vec) or random technical overviews (Vectors vs. Slices in Rust: A Complete Guide) because I am unable to do either with a straight face. If I could, I’d probably still work in a large organization where those kind of ambitious but safe norms are politely celebrated. Someone else will write those pieces and I support them in the same way I support someone running a marathon; I’m not against it, but that doesn’t mean I want to.

It appears this “build in public” includes me breaking the fourth wall on my writing process! Can you tell I struggle in groups? If you’re still here, let’s continue.

Q1 Roadmap

I began Q1 with a list of 6 items I wanted to achieve for From Scratch. As someone who once flew to Chicago to write a list of personal goals on a public library whiteboard with a close friend, I’m no stranger to making goals. What felt new was doing them with that uniquely American fuel: a profit motive.

The Original 6 (the little-known prequel to The Magnificent 7) were:

Launch website testimonials (thanks Jakub!)
Improve SEO for fromscratchcode.com
Launch lead magnet
Run a small trial with paid ads
Write chapters 3-5 of my novella
File taxes and pay Q1 estimated taxes

I would later add two more:

Add CTA to and modernize my personal site
Create a runbook for follow-ups

I’m still working on Chapter 5 (Chapters 1-4 are on From Scratch Press) and the follow-up runbook, but everything else is DONE. Who knew you could accomplish things without Jira for Enterprise.

These records have unintentionally produced a fairly detailed account of the playbook I’m running to build my online business. Which I’m excited to share with you in case you too are interested in how to earn small amounts of money with only an internet connection.

I feel pride looking at this list because it makes my efforts feel less random. I can reassure myself Sure, I’m still growing, but here’s my strategy. I can (and have!) thrown this list into ChatGPT and asked what foundational pieces am I missing. And each time it says “Share your work on social media,” I close the tab and search for From Scratch on Google instead.

Multiple Execution Engines

Memphis has supported two execution engines for about a year now. But barely.

The treewalk interpreter is farthest along and, if you wanted to try to run real code, what you should use. I’ve been strengthening the bytecode VM’s foundation and it’s coming along but slowly. Because I have no deadlines, I’m being deliberate to define Memphis capabilities versus treewalk capabilities versus bytecode VM capabilities. Did I mention I have no deadlines?

The unification has proceeded in the following broad strokes.

Common Entrypoint

Initially, you could pick an engine like this.

# Treewalk is default FOR NOW
memphis example.py

# VM selected using an environment variable
MEMPHIS_ENGINE=vm memphis example.py

Which works fine! But after kicking off the runtime, there was no coming back together. Or reunification.

This remains the interface to select a non-default engine, but I’m gradually unifying more of the code behind the scenes. I’ve also floated the idea of using a Rust feature flag instead of an environment variable to produce a smaller binary.

Common Return Type

Because I respect those who came before me, I wanted to treat Python runtime errors as first-class. Meaning I wouldn’t trap them below deck after hitting an iceberg.

Instead of separate error types for treewalk and VM errors, I would combine them. This would also be a chance to turn this dumping ground I aspirationally called InterpreterError into a type which actually represented possible Python runtime errors.

// The treewalk version was first so it is known as "Interpreter" here
pub enum InterpreterError {
    Exception(DebugCallStack),
    TypeError(Option<String>, DebugCallStack),
    KeyError(String, DebugCallStack),
    ValueError(String, DebugCallStack),
    NameError(String, DebugCallStack),
    AttributeError(String, String, DebugCallStack),
    FunctionNotFound(String, DebugCallStack),
    MethodNotFound(String, DebugCallStack),
    ClassNotFound(String, DebugCallStack),
    ModuleNotFound(String, DebugCallStack),
    DivisionByZero(String, DebugCallStack),
    ExpectedVariable(DebugCallStack),
    ExpectedString(DebugCallStack),
    ExpectedInteger(DebugCallStack),
    ExpectedList(DebugCallStack),
    ExpectedTuple(DebugCallStack),
    ExpectedRange(DebugCallStack),
    ExpectedSet(DebugCallStack),
    ExpectedDict(DebugCallStack),
    ExpectedFloatingPoint(DebugCallStack),
    ExpectedBoolean(DebugCallStack),
    ExpectedObject(DebugCallStack),
    ExpectedClass(DebugCallStack),
    ExpectedFunction(DebugCallStack),
    ExpectedIterable(DebugCallStack),
    ExpectedCoroutine(DebugCallStack),
    WrongNumberOfArguments(usize, usize, DebugCallStack),
    StopIteration(DebugCallStack),
    AssertionError(DebugCallStack),
    MissingContextManagerProtocol(DebugCallStack),
    RuntimeError,
    EncounteredReturn(ExprResult),
    EncounteredRaise,
    EncounteredAwait,
    EncounteredSleep,
    EncounteredBreak,
    EncounteredContinue,
}

pub enum VmError {
    StackUnderflow,
    StackOverflow,
    NameError(String),
    RuntimeError,
}

I landed on this streamlined structure which could now be used on both engines. All my previous ExpectedString, ExpectedInteger, etc., variants are now just a TypeError. This mirrors CPython, where an optional String parameter provides more detail.

pub struct ExecutionError {
    pub debug_call_stack: DebugCallStack,
    pub execution_error_kind: ExecutionErrorKind,
}

pub enum ExecutionErrorKind {
    RuntimeError,
    ImportError(String),
    TypeError(Option<String>),
    KeyError(String),
    ValueError(String),
    NameError(String),
    AttributeError(String, String),
    DivisionByZero(String),
    StopIteration,
    AssertionError,
    MissingContextManagerProtocol,
}

With this unified type, I was able to test for an expected NameError in a crosscheck test. Which reminds me, I should really write more about crosscheck, my testing framework for both engines. I have a fun proc macro in the works there.

Now that both engines returned an ExecutionError, I needed to build an interface to push new stack frames to the DebugCallStack. These would be displayed to the user whenever a Python runtime error occurs in their user code.

Common Debug Stack Trace

I’m still actively working on unifying stack traces, but I’m excited because it is me saying “Memphis supports a debug stack trace, regardless of what execution engine you choose,” rather than just “both engines have a stack trace.” Do you see the difference? It’s a pLaTfOrM. Or maybe I mean fRaMeWoRk? It’s something.

I cleaned up my DebugStackTrace and DebugCallStack structs and moved them into a domain module to indicate they represent a Python stack trace, but should NOT be used as a source of runtime info for an engine.

One challenge is giving each engine access to the right-sized shared state object. This would be how each engine could register new stack frames; think something like state.push_stack_frame(function.to_stack_frame()) when entering a new function context. I’m attempting to balance platform capabilities (MemphisState) against freedom of implementation within each engine (TreewalkState and VmState).

The other challenge of implementing stack traces is it forces you to keep around your metadata for use at the right time. Metadata like file paths and line numbers aren’t necessary for actually running Python code, but they’re essential for debugging. When a statement is parsed and immediately evaluated (treewalk) or immediately compiled (bytecode VM, though this has a bug right now), we record the line number of the start of the current function and increment it each time we see a new line. This parser<>runtime communication is only necessary for stack traces (for me. so far.).

This work stream has been a crash course in applying the Single-Responsibility Principle a few decades after I first learned of it. It’s easy to understand the definition (”of course each function/struct/class only does one thing!”). But applying it? That’s harder (”we’ve got access to state here so I’ll put it there!”). The result is my code is GRADUALLY shifting from a medium number of medium-sized functions to a whole lot of tiny ones. I’ve always been an advocate for adding a second entrypoint, either through unit tests or a REPL, but Memphis has forced me to expand that thinking to an entirely different level.

The End

I’m putting the finishing touches on my Q2 roadmap, which should take my business to the max! Do people still say that?

I recently finished The Pathless Path by Paul Millerd and his message of finding the work each of us wants to do indefinitely resonated with me. With my Memphis engine work and my roadmap work, I believe I’m closer than ever to finding that. I’d also love to be a technical mentor to anyone reading this. Because money.

Hope you are well!

Build a software career with meaning: a playbook

Mon, 17 Feb 2025 00:00:00 GMT

Even with 20 fingers and toes, I barely have enough digits to count how many times during my 9-5 career I thought, “Wow, I feel frustrated/stuck/alone/hungry, maybe I should look for a new job.” A new job holds the allure of allowing you to let go of all your negative emotions (especially hunger) and start over.

There are plenty of reasons to switch jobs, particularly when advocating for fairer compensation or a less-toxic environment. But if you're like me and looking to build a meaningful software career, where you can use your brain to its fullest and maybe do a bit of good, job hopping is more like a salve on a wound than a complete fix.

With this in mind, I’m excited to announce my new email course: Build a Software Career with Meaning. While under development, I called this “How I went from ‘Hello World!’ to ‘How can I help?’ in just 19 years,” which is clearly a cheeky title but captures the work-in-progress feel of my career.

Did I mention it’s free? Over the course of 5 (business) days, you’ll receive a brief email with a piece of wisdom, along with an actionable thought experiment you can test on your career.

I also want to be transparent: in email marketing speak, this is called a “lead magnet.” I learned this term from reading (does YouTube still exist?), so I can only assume this is referring to the element from the periodic table with the symbol Pb (you know, the one pronounced ‘led’). And here I thought lead wasn't magnetic!

I'm offering the course for free because I want to share how I think about things. At the end of 5 (business) days, perhaps you’ll feel like you know a bit more about my values and want to work with me. If you don’t, that’s more than okay! You are welcome to stay on the list indefinitely, and the option to unsubscribe will always be at the bottom. There are no tricks here, just a marketing playbook I’ve been learning in between implementing my Markdown blog and nested functions in Python bytecode.

If you’ve ever felt stuck or disillusioned in your software career, I hope this course gives you a new perspective. If nothing else, it’ll be five (business) days of me popping into your email client with career advice just a tad more nuanced than “Quit your job!”

P.S. If you’re interested in reading the story about how my 9-5 career crashed-and-burned—and how I built something better from the wreckage—I have the complete account over on From Scratch Press. I’d love to hear if any of my experiences and catatonic thought loops (the kind where you forget to eat) mirror your own!

Building a Markdown blog with links optimized for Gatsby

Mon, 27 Jan 2025 00:00:00 GMT

While I was hoping this website would build itself, it ended up taking a good amount of fine-tuning! I wanted to share how I built this blog and a few of the challenges I encountered.

Background

My setup for this website is:

domain and DNS managed on GoDaddy
code lives in a private GitHub repository (should I make this public?)
static site written in React using Gatsby
site deployed using the free tier on Netlify

While Dev.to and Hashnode have been helpful to get me flowing on my technical writing, I really wanted a minimal blog over which I exercised full control. I wanted to be able style blockquotes the way I liked. I wanted the blog to reflect the From Scratch ethos that less is better.

The final straw: when I realized I was missing out on potential SEO improvements by not publishing my writing directly to fromscratchcode.com.

Thus this silly blog that I’m moderately proud of was born!

List of Requirements

My requirements for the new blog were:

Posts shall be written in Markdown and be portable, meaning require no changes to be thrown into other editors (Dev.to, Hashnode, etc). As a bonus, I currently write these in Notion because they copy out already in Markdown.
Posts shall support syntax highlighting for code blocks that actually look good. (This was a PAIN on my email provider. too bad this post isn’t about picking an email provider!)
Internal links within posts shall be optimized for navigation and SEO.

Gatsby is modern, flexible, and I’d barely used it before, which gave me the confidence I could pull this off. I wrote this post because these requirements got more difficult as I went and I hope this can be a resource for someone else. Here’s what I learned!

Supporting Markdown Posts

My motivation for writing in Markdown will not surprise you: I wanted to spend more time writing and less time formatting. To move a piece (including code!) between platforms and have it just work. Markdown checks all of those boxes.

Gatsby’s ecosystem is rich with markdown support. Here’s how I leveraged it!

I used the gatsby-transformer-remark plugin to read the Markdown files. Next, I used the createPages API in gatsby-node.js to register a new page with slug (this is a URL that for some reason is named after an insect).

Then, I used Gatsby’s createNodeField to attach additional metadata to the Markdown node. This ensures this metadata is available to components via GraphQL queries. I also used the reading-time library to calculate the reading time for each block of Markdown to give each post some fun deets and make this look like a real blog.

Here are these pieces assembled together in my gatsby-node.js file:

const BLOG_QUERY = `
    {
      allMarkdownRemark(
        filter: { fileAbsolutePath: { regex: "/blog/" } }
        sort: { frontmatter: { date: DESC } }
      ) {
        edges {
          node {
            frontmatter {
              slug
            }
          }
        }
      }
    }
  `

// This function runs a GraphQL query and creates pages at `${base_url}/${node.frontmatter.slug}`
// for each using the provided template.
// NOTE: each markdown file must provide its own slug in the frontmatter.
// TODO: could we provide a default in the case the frontmatter is not specified?
const createMarkdownPages = async (
  graphql,
  createPage,
  query,
  base_url,
  template
) => {
  const markdownFiles = await graphql(query)

  markdownFiles.data.allMarkdownRemark.edges.forEach(({ node }) => {
    const slug = `${base_url}/${node.frontmatter.slug}`
    createPage({
      path: slug,
      component: template,
      // The context is needed for the $slug param lookup in the query inside the
      // repsective template
      context: {
        slug,
      },
    })
  })
}

exports.createPages = async ({ graphql, actions }) => {
  const { createPage } = actions
  const blogPostTemplate = path.resolve(`src/templates/blogPostTemplate.js`)

  await createMarkdownPages(
    graphql,
    createPage,
    BLOG_QUERY,
    "/blog",
    blogPostTemplate
  )
}

exports.onCreateNode = ({ node, actions, getNode }) => {
  const { createNodeField } = actions
  if (node.internal.type === "MarkdownRemark") {
    let slug = node.frontmatter.slug

    // Determine the base path (e.g., 'policies' or 'blog')
    // This is the name from the gatsby-source-filesystem config in gatsby-config.js
    const sourceInstanceName = getNode(node.parent).sourceInstanceName

    // Add the slug field to the node, which will become queryable from GraphQL. This is safer
    // than relying on frontmatter within the component pages because we have applied all the
    // necessary transformations here before we write to the field.
    createNodeField({
      node,
      name: "slug",
      value: `/${sourceInstanceName}/${slug}`,
    })

    // Calculate the reading time of the markdown content and add it to the GraphQL
    const stats = readingTime(node.rawMarkdownBody)
    createNodeField({
      node,
      name: "readingTime",
      value: stats.text, // Example: "3 min read"
    })
  }
}

I trimmed for brevity (HA!), but I process my Terms of Use and Privacy Policy using the same procedure. While those would not need syntax highlighting, requirement #3 about optimized internal links would still apply.

Next, I used Gatsby’s GraphQL to query for the post content based on the slug of the rendered page.

This is my first pass at blogPostTemplate.js. I have stripped out a few pieces to focus on the GraphQL query.

import React from "react"
import { graphql } from "gatsby"

const BlogPost = ({ data }) => {
  const post = data.markdownRemark

  return (
    <Layout>
      <article>
        <h1>{post.frontmatter.title}</h1>
        <div dangerouslySetInnerHTML={{ __html: post.html }} />
      </article>
    </Layout>
  )
}
export const query = graphql`
  query ($slug: String!) {
    markdownRemark(fields: { slug: { eq: $slug } }) {
      frontmatter {
        title
        date(formatString: "MMM DD, YYYY")
      }
      fields {
        readingTime
      }
      html
    }
  }
`

export default BlogPost

Consider the following piece of frontmatter, the metadata at the top of a Markdown file:

---
title: "Introducing: From Scratch Code"
date: "2024-11-04"
slug: "introducing-from-scratch-code"
---

At this point, we have created and populated a page at /blog/introducing-from-scratch-code/. Awesome!

Enabling Syntax Highlighting

Syntax highlighting ended up taking two tries, both using PrismJS.

First Attempt: `gatsby-remark-prismjs`

My first approach used the gatsby-remark-prismjs plugin (yes, this is a plugin [prismjs] to a plugin [remark] to a framework [gatsby]) during the Gatsby build pipeline. Here was the addition to my gatsby-config.js file.

{
  resolve: `gatsby-transformer-remark`,
  options: {
    plugins: [
      {
        resolve: `gatsby-remark-prismjs`,
        options: {
          classPrefix: "language-", // Set a class prefix
          inlineCodeMarker: null, // Marker for inline code
          // Do not use prism to highlight inline code
          noInlineHighlight: true,
        },
      },
    ],
  }
}

This approach does these steps behind the scenes:

Read markdown
Convert it to HTML
Apply syntax highlighting via CSS classes
Let React render our HTML which we fetched by querying GraphQL

This worked out of the box!

To customize the theme, I added this to my gatsby-browser.js.

// Prism.js syntax highlighting
import "prismjs/themes/prism-tomorrow.css"

I had some trouble styling line numbers and the Copy-to-Clipboard button. There seemed to be some complexity around PrismJS plugins inside a static site processing pipeline such as Remark in Gatsby, so I punted on those.

This approached worked great until it didn’t, which brings me to requirement #3.

Optimizing Internal Links

When I say “internal link”, I mean a link on this website, such as /blog.

I had two motivations for optimizing these:

This is mostly me being particular, but in order to truly make my Markdown portable, I wanted to be able to write external links (such as https://fromscratchcode.com/mentorship) in Markdown and have it be treated as the internal link /mentorship.
This is the biggie: Gatsby applies a magic touch using its Link component, which consists of several things.
1. Under the hood, Gatsby prefetches any Link destination when the page loads.
2. When clicked, it uses @reach/router to navigate to it using ultra-smooth client-side navigation.
3. This is all while appearing to be an <a> component in the page source, which keeps these links *chef's kiss* perfect for SEO because they remain visible to crawlers such as Google’s.
4. In addition, our React state persists across clicks. Without this optimization, the state of the dark mode toggle would persist when using the top nav, but not when clicking a link discovered in a blog post. That is not ideal!

Second Attempt: Switching to Rehype

The challenge here is turning [mentorship](/mentorship) into <Link to="/mentorship">mentorship</Link> and have it still be rendered as React.

There is a strong ecosystem of JS libraries to help with this, but I needed to switch from using PrismJS in a Remark plugin to a Rehype plugin. That sentence would have been word salad to me a few ~~hours~~ weeks ago, so please bear with me. Gatsby’s Remark pipeline converts Markdown to HTML before React sees it, so internal links get set before React can modify them. By using Rehype, we get a hook where we can dynamically swap <a> for Link. Rehype also supports integrating PrismJS syntax highlighting. Here’s how I pulled it off!

The function below does this whole shebang in several steps:

Parse our raw markdown content into an AST (Abstract Syntax Tree).
Apply a plugin we would need to write to detect and convert external links to internal links.
Convert the markdown AST (used by Remark) into an HTML AST (used by Rehype).
Apply PrismJS syntax highlighting using a Rehype plugin.
Render the Rehype AST as React, converting any <a> elements into CustomLink components along the way. (CustomLink is a wrapper I created around Gatsby’s Link so that it works for internal or external links.)

// Render markdown to React
// This pipeline processes raw markdown and converts it into React components,
// applying syntax highlighting and internal link optimization.
const renderMarkdownToReact = markdown => {
  try {
    return unified()
      .use(remarkParse) // Parse markdown into an abstract syntax tree
      .use(optimizeInternalLinks) // Apply our plugin to detect internal links
      .use(remarkRehype) // Convert Markdown AST to Rehype AST
      .use(rehypePrism, { showLineNumbers: false }) // Add PrismJS syntax highlighting
      .use(rehypeReact, {
        jsx,
        jsxs,
        Fragment,
        components: {
          a: props => <CustomLink to={props.href} {...props} />,
        },
      })
      .processSync(markdown).result
  } catch (error) {
    console.error("Failed to render markdown:", error)
    return <div>Error rendering markdown</div>
  }
}

And our plugin to convert internal links:

const optimizeInternalLinks = () => tree => {
  visit(tree, "link", node => {
    const { url } = node
    if (url.startsWith("/") || url.startsWith(SITE_URL)) {
      const internalPath = url.replace(SITE_URL, "")
      node.url = internalPath
    }
  })
}

This function calls visit from unist-util-visit to allow us to walk the AST. Its interface matches what is expected by a Remark plugin.

PHEW. That is quite a pipeline. This is the type of problem I probably would have given up on pre-ChatGPT. With a tool that points me to exactly what libraries to use and gets me close on the initial syntax, I was able to take it the rest of the way.

Now you can click around this blog and, when I reference a past post, it should load nearly instantaneously and preserve your dark mode settings. We did it!

What’s next?

I’m pleased with how the blog has turned out! I’d like to eventually add support for tags, table of contents for each post, and perhaps a series so to link to all the Memphis posts. I could add comments using Disqus but, then again, would you invite a troll into your living room?

I’d love to hear from you! Please reach out if you find any bugs. I’m also curious what static site blog features have wow-ed you in the past, now that I am the proud owner of my own.

I’m gonna go write some Rust now.

Typed integers in Rust for safer Python bytecode compilation

Mon, 13 Jan 2025 00:00:00 GMT

Shortly after I shared my previous post, a helpful redditor pointed out that the typed integers I alluded to is known as the newtype pattern in Rust, a design that allows you to define types purely for compile-time safety and with no runtime hit.

And here I thought I invented it! Alas. 😂

I wanted to share a few details about what challenge I encountered and how I solved it.

The problem

Like I mentioned in the previous post, bytecode is a lower-level representation of your code created for efficient evaluation. One of the optimizations is that variable names are converted into indices during bytecode compilation, which supports faster lookup at runtime when the VM pumps through your bytecode.

At one point while implementing this myself, I was debugging a sequence that felt like this. (I’m using the wishy-washy “felt like” here because I have no idea what the actual bytecode looked like.)

LOAD_GLOBAL 0
LOAD_FAST 0
LOAD_CONST 0

Okay, so we’re loading the first global, the first local, and the first constant. Given the ongoing evolution of my VM design and implementation, the data structures I’m using to communicate the mappings of these indices to their symbol names and/or values between the compiler and VM have been in a steady flux.

I’d sit down to implement the execution of a given opcode in the VM, see that I was handed the value 0, and I’d think, “What do they want me to do with this?!” (“they” in the case being myself from 5 minutes prior hacking on the compiler stage.)

After interpreting a 0 wrong for the forth or fifth time, I decided THERE HAS TO BE A BETTER WAY. I asked myself, “Is there a way to get the compiler to tell me when I’m using a 0 incorrectly?” In addition to compile-time checking, with rust-analyzer configured in my Neovim, I should get a type error as soon as I made the mistake.

Could I add types to my integers?!

The CPython bytecode docs even hint at these distinctions, but incorporating them into my own implementation wasn’t immediately clear until I experienced the pain firsthand. For my three opcodes above, the respective documentation is:

LOAD_GLOBAL(namei)
Loads the global named co_names[namei>>1] onto the stack.

LOAD_FAST(var_num)
Pushes a reference to the local co_varnames[var_num] onto the stack.

LOAD_CONST(consti)
Pushes co_consts[consti] onto the stack.

co_names, co_varnames, and co_consts are three fields on a CPython code object, which is an immutable piece of compiled bytecode. In my interpreter, I’m using names and varnames to show my originality.

(Note to self: I treat constants separately at the moments, but I should probably unify it as I solidify my understanding of code objects.)

(Another note to self: how curious that they are shifting namei to the right during LOAD_GLOBAL. I hope to one day know why.)

The solution

I added typed integers to keep myself sane. They offered an additional benefit of providing me an opportunity to gain more experience with generics in Rust!

Here is a subset of my Opcode implementation.

pub enum Opcode {
    /// Push the value found at the specified index in the constant pool onto the stack.
    LoadConst(ConstantIndex),
    /// Read the local variable indicated by the specified index and push the value onto the stack.
    LoadFast(LocalIndex),
    /// Read the global variable indicated by the specified index and push the value onto the stack.
    LoadGlobal(NonlocalIndex),
}

My hope here is these new types, ConstantIndex, LocalIndex, and NonlocalIndex, would semantically illustrate the behavior of each opcode while providing type safety.

Generics entered the scene next as I was hoping to implement this with minimal code reuse.

Here is the next layer of the onion.

pub type ConstantIndex = Index<ConstantMarker>;
pub type LocalIndex = Index<LocalMarker>;
pub type NonlocalIndex = Index<NonlocalMarker>;

We’re beginning to see some code reuse–awesome!

What are these marker types? They are truly that: a type which is literally just a marker with no other data. These empty types are enforced at compile-time and do nothing at runtime.

pub struct ConstantMarker;
pub struct LocalMarker;
pub struct NonlocalMarker;

In Rust, PhantomData<T> from std::marker allows you to include a type in a struct purely for compile-time checks without affecting runtime performance—exactly the type safety for which we’re seeking! I’m using usize for the value because my use-case was indices, which should never be negative.

/// An unsigned integer wrapper which provides type safety. This is particularly useful when
/// dealing with indices used across the bytecode compiler and the VM as common integer values such
/// as 0, 1, etc, can be interpreted many different ways.
#[derive(Copy, Clone, PartialEq, Hash, Eq)]
pub struct Index<T> {
    value: usize,
    _marker: PhantomData<T>,
}

impl<T> Index<T> {
    pub fn new(value: usize) -> Self {
        Self {
            value,
            _marker: PhantomData,
        }
    }
}

impl<T> Deref for Index<T> {
    type Target = usize;

    fn deref(&self) -> &Self::Target {
        &self.value
    }
}

impl<T> Display for Index<T> {
    fn fmt(&self, f: &mut Formatter) -> Result<(), Error> {
        write!(f, "{}", self.value)
    }
}

The last piece of magic is impl<T> Deref for Index<T>, which allows us to treat instances of our new type as integers when dereferenced.

As a result, a piece of code like this would pass with flying colors.

#[test]
fn test_dereference() {
    let index: LocalIndex = Index::new(4);
    assert_eq!(*index, 4)
}

By this point, we are free to use these in the bytecode compiler! Most places in the code don’t require specifying the generic because it will be inferred by the function signature, like in this example.

fn get_or_set_local_index(&mut self, name: &str) -> LocalIndex {
    if let Some(index) = self.get_local_index(name) {
        index
    } else {
        let code = self.ensure_code_object_mut();
        let new_index = code.varnames.len();
        code.varnames.push(name.to_string());
        Index::new(new_index)
    }
}

We initialize a new index with Index::new(new_index), which the compiler knows to treat as an Index<LocalMarker>, which we have aliased to LocalIndex to allow us to be blissfully ignorant VM developers.

The end

This is a small but powerful example of how I’m falling head-over-heels for Rust’s type system. I love writing expressive code to implement core features from scratch in a way which manages complexity and makes people smile.

My mentor at my first big-boy job was a wizard at this and I credit him for showing me what is possible: spending more time thinking about what you’re trying to build rather than being bogged down by how you are trying to build it.

Have you used Rust’s type system in any fun and creative ways? Or done something similar in another language? I’d love to hear your thoughts in the comments. Be well!

How I added support for nested functions in Python bytecode

Mon, 30 Dec 2024 00:00:00 GMT

I wanted to share some pretty cool stuff I’ve been learning about Python bytecode with you, including how I added support for nested functions, but my guy at the printing press said I needed to keep it under 500 words.

It’s a holiday week, he shrugged. What do you expect me to do?

Excluding code snippets, I bargained.

Fine, he ceded.

Do you know why we use bytecode in the first place?

I just operate the printing press, I trust you though.

Fair enough. Let’s begin.

Why we use bytecode in the first place

Memphis, my Python interpreter written in Rust, has two execution engines. Neither can run all code but both can run some code.

My treewalk interpreter is what you would build if you didn’t know what you were doing. 🙋‍♂️ You tokenize the input Python code, generate an abstract syntax tree (AST), and then walk the tree and evaluate each node. Expressions return values and statements modify the symbol table, which is implemented as a series of scopes which respect Python scoping rules. Just remember the easy pneumonic LEGB: local, enclosing, global, builtin.

My bytecode VM is what you would build if you didn’t know what you were doing but wanted to act like you did. Also 🙋‍♂️. For this engine, the tokens and AST work the same, but rather than walking we take off sprinting. We compile the AST into an intermediate representation (IR) hereafter known as bytecode. We then create a stack-based virtual machine (VM), which conceptually acts like a CPU, executing bytecode instructions in sequence, but it’s implemented entirely in software.

(For a complete guide of both approaches without the ramblings, Crafting Interpreters is excellent.)

Why do we do this in the first place? Just remember the two Ps: portability and performance. Remember how in the early 2000s nobody would shut up about how Java bytecode was portable? All you need is a JVM and you can run a Java program compiled on any machine! Python chose not to go with this approach for both technical and marketing reasons, but in theory the same principles apply. (In practice, the compilation steps are different and I regret opening this can of worms.)

Performance is the biggie though. Rather than traversing an AST multiple times during the lifetime of a program, the compiled IR is a more efficient representation. We see improved performance from avoiding the overhead of repeatedly traversing an AST, and its flat structure often results in better branch prediction and cache locality at runtime.

(I don’t blame you for not thinking about caching if you don’t have a background in computer architecture—heck, I began my career in that industry and I think about caching far less than I think about how to avoid writing the same line of code twice. So just trust me on the performance piece. That’s my leadership style: blind trust.)

Hey buddy, that’s 500 words. We need to load up the frame and let ‘er rip.

Already?! You excluded code snippets?

There are no code snippets, my man.

Okay okay. Just 500 more. I promise.

Context matters for Python variables

I got kinda far before tabling my bytecode VM implementation about a year ago: I could define Python functions and classes and call those functions and instantiate those classes. I clamped down this behavior with some tests. But I knew my implementation was messy and that I’d need to revisit the fundamentals before adding more fun stuff. Now it’s Christmas week and I want to add fun stuff.

Consider this snippet for calling a function, keeping an eye on the TODO.

fn compile_function_call(
    &mut self,
    name: &str,
    args: &ParsedArguments)
) -> Result<Bytecode, CompileError> {
    let mut opcodes = vec![];

    // We push the args onto the stack in reverse call order so that we will pop
    // them off in call order.
    for arg in args.args.iter().rev() {
        opcodes.extend(self.compile_expr(arg)?);
    }

    let (_, index) = self.get_local_index(name);

    // TODO how does this know if it is a global or local index? this may not be the right
    // approach for calling a function
    opcodes.push(Opcode::Call(index));

    Ok(opcodes)
}

Are you done considering? We load the function arguments onto the stack and “call the function”. In bytecode, all names are converted into indices (because index access is faster during the VM runtime), but we don’t really have a way to know whether we are dealing with a local index or a global index here.

Now consider the improved version.

fn compile_function_call(
    &mut self,
    name: &str,
    args: &ParsedArguments)
) -> Result<Bytecode, CompileError> {
    let mut opcodes = vec![self.compile_load(name)];

    // We push the args onto the stack in reverse call order so that we will pop
    // them off in call order.
    for arg in args.args.iter().rev() {
        opcodes.extend(self.compile_expr(arg)?);
    }

    let argc = opcodes.len() - 1;
    opcodes.push(Opcode::Call(argc));

    Ok(opcodes)
}

Thank you for considering that code.

We now supported nested function calls! What changed?

The Call opcode now takes a number of positional arguments, rather than an index to the function. This instructs the VM how many arguments to pop off the stack before calling the function.
After popping the arguments off the stack, the function itself will be left on the stack and compile_load has already handled local versus global scope for us.

LOAD_GLOBAL versus LOAD_FAST

Let’s take a look at what compile_load is doing.

fn compile_load(&mut self, name: &str) -> Opcode {
    match self.ensure_context() {
        Context::Global => Opcode::LoadGlobal(self.get_or_set_nonlocal_index(name)),
        Context::Local => {
            // Check locals first
            if let Some(index) = self.get_local_index(name) {
                return Opcode::LoadFast(index);
            }

            // If not found locally, fall back to globals
            Opcode::LoadGlobal(self.get_or_set_nonlocal_index(name))
        }
    }
}

There are several key principles in action here:

We match based on the current context. Adhering to Python semantics, we can consider Context::Global to be at the top level of any module (not just your script’s entrypoint), and Context::Local is inside any block (i.e. function definition or class definition).
We now differentiate between a local index and a non-local index. (Because I was going crazy trying to decipher what the index 0 referred to in different places, I introduced typed-integers. LocalIndex and NonlocalIndex provide type-safety for otherwise untyped unsigned integers. I may write about this in the future!)
We can tell at bytecode-compilation time whether a local variable exists with a given name, and if it does not, at runtime we will search for a global variable. This speaks to the dynamism built into Python: as long as a variable is present in the global scope of that module by the time a function executes, its value can be resolved at runtime. However, this dynamic resolution comes with a performance hit. While local variable lookups are optimized to use stack indices, global lookups require searching the global namespace dictionary, which is slower. This dictionary is a mapping of names to objects, which themselves may live on the heap. Who knew that the saying “Think globally, act locally.” was actually referring to Python scopes?

What’s in a varname?

The last thing I’ll leave you with today is a peek into how these variables names are mapped. In the code snippet below, you’ll notice that local indices are found in code.varnames and nonlocal indices are found in code.names. Both live on a CodeObject, which contains the metadata for a block of Python bytecode, including its variable and name mappings.

fn get_or_set_local_index(&mut self, name: &str) -> LocalIndex {
    if let Some(index) = self.get_local_index(name) {
        index
    } else {
        let code = self.ensure_code_object_mut();
        let new_index = code.varnames.len();
        code.varnames.push(name.to_string());
        Index::new(new_index)
    }
}

fn get_local_index(&self, name: &str) -> Option<LocalIndex> {
    let code = self.ensure_code_object();
    find_index(&code.varnames, name).map(Index::new)
}

fn get_or_set_nonlocal_index(&mut self, name: &str) -> NonlocalIndex {
    let code = self.ensure_code_object_mut();
    if let Some(index) = find_index(&code.names, name) {
        Index::new(index)
    } else {
        let new_index = code.names.len();
        code.names.push(name.to_string());
        Index::new(new_index)
    }
}

The difference between varnames and names tormented me for weeks (CPython calls these co_varnames and co_names), but it’s actually fairly straightforward. varnames holds the variable names for all local variables in a given scope, and names does the same for all nonlocal.

Once we properly track this, everything else just works. At runtime, the VM sees a LOAD_GLOBAL or a LOAD_FAST and knows to look in the global namespace dictionary or the local stack, respectively.

Buddy! Mr Gutenberg is on the phone and says we can hold the presses no longer.

Okay! Fine! I get it! Let’s ship it. 🚢

What’s next for Memphis?

Shh! The printing press man doesn’t know I’m writing a conclusion, so I will be brief.

With variable scoping and function calls in a solid place, I’m gradually turning my attention to features like stack traces and async support. If you’ve enjoyed this dive into bytecode or have questions about building your own interpreter, I’d love to hear from you—drop a comment!

Improving memory efficiency in a working interpreter

Mon, 16 Dec 2024 00:00:00 GMT

Lifetimes are a fascinating feature of Rust and the human experience. This is a technical blog, so let’s focus on the former. I was admittedly a slow adopter for leveraging lifetimes to safely borrow data in Rust. In the treewalk implementation of Memphis, my Python interpreter written in Rust, I hardly leverage lifetimes (by cloning incessantly) and I repeatedly elude the borrow checker (by using interior mutability, also incessantly) whenever possible.

My fellow Rustaceans, I am here to today to tell you this ends now. Read my lips……no more shortcuts.

Okay okay, let’s be real. What is a shortcut versus what’s the right way is a matter of priorities and perspective. We’ve all made mistakes, and I’m here to take accountability for mine.

I began writing an interpreter six weeks after I first installed rustc because I have no chill. With that haranguing and posturing out of the way, let’s begin today’s lecture on how we can use lifetimes as our lifeline to improve my bloated interpreter codebase.

Identifying and avoiding cloned data

A Rust lifetime is a mechanism which provides a compile-time guarantee that any references do not outlive the objects to which they refer. They allow us to avoid the “dangling pointer” problem from C and C++.

This is assuming you leverage them at all! Cloning is a convenient workaround when you want to avoid the complexities associated with managing lifetimes, though the downside is increased memory usage and a slight delay related to each time data is copied.

Using lifetimes also forces you to think more idiomatically about owners and borrowing in Rust, which I was eager to do.

I chose my first candidate as the tokens from a Python input file. My original implementation, which relied heavily on ChatGPT guidance while I was sitting on Amtrak, used this flow:

we pass our Python text to a Builder
the Builder creates a Lexer, which tokenizes the input stream
the Builder then creates a Parser, which clones the token stream to hold its own copy
the Builder is used to create an Interpreter, which repeatedly asks the Parser for its next parsed statement and evaluates it until we reach the end of the token stream

The convenient aspect of cloning the token stream is that the Lexer was free to be dropped after step 3. By updating my architecture to have the Lexer own the tokens and the Parser just borrow them, the Lexer would now be required to stay alive much longer. Rust lifetimes would guarantee this for us: as long as the Parser existed holding a reference to a borrowed token, the compiler would guarantee that the Lexer which own those tokens still existed, ensuring a valid reference.

Like all code always, this ended up being a bigger change than I expected. Let’s see why!

The new parser

Before updating the Parser to borrow the tokens from the Lexer, it looked like this. The two fields of interest for today’s discussion are tokens and current_token. We have no idea how large the Vec<Token> is, but it is distinctly ours (i.e. we are not borrowing it).

pub struct Parser {
    state: Container<State>,
    tokens: Vec<Token>,
    current_token: Token,
    position: usize,
    line_number: usize,
    delimiter_depth: usize,
}

impl Parser {
    pub fn new(tokens: Vec<Token>, state: Container<State>) -> Self {
        let current_token = tokens.first().cloned().unwrap_or(Token::Eof);
        Parser {
            state,
            tokens,
            current_token,
            position: 0,
            line_number: 1,
            delimiter_depth: 0,
        }
    }
}

After borrowing the tokens from the Lexer, it looks fairly similar, but now we see a LIFETIME! By connecting tokens to the lifetime 'a, the Rust compiler will not allow the owner of the tokens (which is our Lexer) and the tokens themselves to be dropped while our Parser still references them. This feels safe and fancy!

static EOF: Token = Token::Eof;

/// A recursive-descent parser which attempts to encode the full Python grammar.
pub struct Parser<'a> {
    state: Container<State>,
    tokens: &'a [Token],
    current_token: &'a Token,
    position: usize,
    line_number: usize,
    delimiter_depth: usize,
}

impl<'a> Parser<'a> {
    pub fn new(tokens: &'a [Token], state: Container<State>) -> Self {
        let current_token = tokens.first().unwrap_or(&EOF);
        Parser {
            state,
            tokens,
            current_token,
            position: 0,
            line_number: 1,
            delimiter_depth: 0,
        }
    }
}

Another small difference you may notice is this line:

static EOF: Token = Token::Eof;

This is a small optimization that I began considering once my Parser was moving in the direction of “memory-efficient.” Rather than instantiating a new Token::Eof each time the Parser needs to check if it is at the end of the text stream, the new model allowed me to instantiate only a single token and reference &EOF repeatedly.

Again, this is a small optimization, but it speaks to the larger mindset of each piece of data existing only once in memory and every consumer just referencing it when needed, which Rust both encourages you to do and snugly holds your hand along the way.

Speaking of optimization, I really should have benchmarked the memory usage before and after. Since I did not, I have nothing more to say on the matter.

As I alluded to earlier, tying the lifetime of my Lexer and Parser together a large impact on my Builder pattern. Let’s see what that looks like!

The new Builder: MemphisContext

In the flow I described above, remember how I mentioned that the Lexer could be dropped as soon as the Parser created its own copy of the tokens? This had unintentionally influenced the design of my Builder, which was intended to be the component which supports orchestrating Lexer, Parser, and Interpreter interactions, whether you begin with a Python text stream or a path to a Python file.

As you can see below, there are a few other non-ideal aspects to this design:

needing to call a dangerous downcast method to get the Interpreter.
why did I think it was okay to return a Parser to every unit test just to then pass it right back into interpreter.run(&mut parser)?!

fn downcast<T: InterpreterEntrypoint + 'static>(input: T) -> Interpreter {
    let any_ref: &dyn Any = &input as &dyn Any;
    any_ref.downcast_ref::<Interpreter>().unwrap().clone()
}

fn init(text: &str) -> (Parser, Interpreter) {
    let (parser, interpreter) = Builder::new().text(text).build();

    (parser, downcast(interpreter))
}


#[test]
fn function_definition() {
     let input = r#"
def add(x, y):
    return x + y

a = add(2, 3)
"#;
    let (mut parser, mut interpreter) = init(input);

    match interpreter.run(&mut parser) {
        Err(e) => panic!("Interpreter error: {:?}", e),
        Ok(_) => {
            assert_eq!(
                interpreter.state.read("a"),
                Some(ExprResult::Integer(5.store()))
            );
        }
    }
}

Below is the new MemphisContext interface. This mechanism manages the Lexer lifetime internally (to keep our references alive long enough to keep our Parser happy!) and only exposes what is needed to run this test.

fn init(text: &str) -> MemphisContext {
    MemphisContext::from_text(text)
}

#[test]
fn function_definition() {
    let input = r#"
def add(x, y):
    return x + y

a = add(2, 3)
"#;
    let mut context = init(input);

    match context.run_and_return_interpreter() {
        Err(e) => panic!("Interpreter error: {:?}", e),
        Ok(interpreter) => {
            assert_eq!(
                interpreter.state.read("a"),
                Some(ExprResult::Integer(5.store()))
            );
        }
    }
}

context.run_and_return_interpreter() is still a bit clunky and speaks to another design problem I may tackle down the road: when you run the interpreter, do you want to return only the final return value or something which lets you access arbitrary values from the symbol table? This method opts for the latter approach. I actually think there’s a case to do both, and will keep tweaking my API to allow for this as we go.

Incidentally, this change improved my ability to evaluate an arbitrary piece of Python code. If you’ll recall from my WebAssembly saga, I had to rely on my crosscheck TreewalkAdapter to do that at the time. Now, our Wasm interface is much cleaner.

#[cfg(feature = "wasm")]
mod wasm {
    use console_error_panic_hook::set_once;
    use wasm_bindgen::prelude::wasm_bindgen;

    use super::init::MemphisContext;

    #[wasm_bindgen]
    pub fn evaluate(code: String) -> String {
        // Set the panic hook for better error messages in the browser console
        set_once();

        let mut context = MemphisContext::from_text(&code);
        let result = context
            .evaluate_oneshot()
            .expect("Failed to evaluate expression.");
        format!("{}", result)
    }
}

The interface context.evaluate_oneshot() returns the expression result rather than a full symbol table. I wonder if there’s a better way to ensure any of the “oneshot” methods can only operate on a context once, ensuring that no consumers use them in a stateful context. I’ll keep simmering on that!

Was this worth it?

Memphis is first-and-foremost a learning exercise, so this was absolutely worth it!

In addition to sharing the tokens between the Lexer and the Parser, I created an interface to evaluate Python code with significantly less boilerplate. While sharing data introduced additional complexity, these changes bring clear benefits: reduced memory usage, improved safety guarantees through stricter lifetime management, and a streamlined API that’s easier to maintain and extend.

I’m choosing to believe this was the right approach, mostly to maintain my self-esteem. Ultimately, I aim to write code that clearly reflects the principles of software and computer engineering. We can now open the Memphis source, point to the single owner of the tokens, and sleep soundly at night!

An interpreter inside an interpreter

Mon, 25 Nov 2024 00:00:00 GMT

A few months into development, I decided my north star for Memphis would be to run a Flask server entirely within my interpreter. I had no idea how much work this would entail, only that it sounded cool and would probably teach me a lot along the way. If I were making this goal today, I may pick FastAPI or nothing at all because that was silly of me.

Python stdlib

A big decision I encountered was how to deal with the Python standard lib. As you are likely familiar, the standard lib of a language is not technically part of the language definition or runtime. It is included with releases in order to make the language and runtime more useful. Imagine Python without threading or async support. You would still be able to evaluate expressions and instantiate classes, but most production-ready programs need some sort of concurrency support.

One option would be to rewrite the entire standard lib myself. I’m building an interpreter, aren’t I? I believe this is the approach taken by RustPython, which is an admirable path. I figured I had enough on my plate getting the runtime to work, was looking for any and all corners to cut, and decided against this.

The Python standard lib consists of two main parts: the parts implemented in Python and the parts implemented in C. Conveniently enough, I had my own Python interpreter. Could I just interpret the Python source file from the host machine to satisfy the former? Yes, I could. I’d need to support every syntax and feature they used, but after that, it would Just Work.

The C part is where it gets interesting. Way back yonder in 2023, I made a decision to embed a Python interpreter inside my Python interpreter without fully understanding what that meant. Now it was time to wrap my head around this and decide if I wanted to stay with this approach or chose another path.

The interop shop for Rust and Python is Pyo3. As the only game in town, Pyo3 uses the Foreign Function Interface (FFI) to allow your Rust code to make calls into the CPython binary. This works by agreeing on the Application Binary Interface (ABI), a concept I used during my career at AMD. Core software ftw!

Importing modules

My initial use-case was to run import sys and have it give me an object on which I could perform a member access operation. I’m getting into interpreter-speak here, but this is the type of REPL session I’m talking about.

Python 3.12.5 (main, Aug  6 2024, 19:08:49) [Clang 15.0.0 (clang-1500.3.9.4)]
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys
<module 'sys' (built-in)>
>>> type(sys.modules)
<class 'dict'>

Getting this functionality using Pyo3 was straightforward.

pub struct CPythonModule(PyObject);

impl CPythonModule {
    pub fn new(name: &str) -> Self {
        pyo3::prepare_freethreaded_python();
        let pymodule = Python::with_gil(|py|
            PyModule::import(py, name).expect("Failed to import module").into()
        );

        Self(pymodule)
    }
}

And we can use this to drive a similar REPL session in Memphis, assuming you remember the cocktail of features flags to get this to run.

memphis 0.1.0 REPL (Type 'exit()' to quit)
>>> import sys
>>> sys
<module 'sys' (built-in)>
>>> type(sys.modules)
<class 'dict' (built-in)>

If you’re asking yourself, couldn’t you just use this approach to import the entire standard lib (including the parts written in Python and C) and make your entire life, liberty, and the pursuit of happiness, easier, the answer is yes. That would be a valid approach! However, that would make my interpreter more of a shell around CPython than I would like. This is a learning exercise so I’m all for arbitrary decisions. For the purists out there who say loading any piece of CPython inside Memphis makes Memphis not a real interpreter, I would just say: please show me your interpreter.

I conducted a quick test with htop by running import sys inside a REPL session using both Memphis and CPython. On Memphis, because this load the CPython libraries into memory, it increased the RAM usage (Resident Set Size in htop) by about 5MB. For comparison, the Memphis REPL after loading the sys module uses about 9MB of RAM, while the Python REPL before and after loading the sys module uses about the same. I’m sure this isn’t an apples-to-apples comparison, but it at least told me that Memphis wasn’t gonna slowly choke my computer to death.

Converting objects and getting existential

The next complexity with this setup involves converting my Memphis object representation into CPython representations and vice versa. This is a work-in-progress and my primary directive was, initially, “do not fail” and, more recently, “show warnings when you do a lossy conversion.”

Here is my conversion from a PyObject, which is the object representation on the Pyo3 side, into an ExprResult, my Memphis representation.

pub mod utils {
    pub fn from_pyobject(py: Python, py_obj: &PyAny) -> ExprResult {
        if let Ok(value) = py_obj.extract::<i64>() {
            ExprResult::Integer(Container::new(value))
        } else if let Ok(value) = py_obj.extract::<f64>() {
            ExprResult::FloatingPoint(value)
        } else if let Ok(value) = py_obj.extract::<&str>() {
            ExprResult::String(Str::new(value.to_string()))
        } else if let Ok(py_tuple) = py_obj.extract::<&PyTuple>() {
            let elements = py_tuple
                .iter()
                .map(|item| from_pyobject(py, item))
                .collect();
            ExprResult::Tuple(Container::new(Tuple::new(elements)))
        } else if let Ok(py_module) = py_obj.extract::<&PyModule>() {
            let mut module = Module::default();

            // Get the module's __dict__ to iterate over all attributes
            for (key, value) in py_module.dict() {
                let key_str: String =
                  key.extract().expect("Key is not a string");
                let expr_value = from_pyobject(py, value);
                module.insert(&key_str, expr_value);
            }

            ExprResult::Module(Container::new(module))
        } else if let Ok(py_set) = py_obj.extract::<&PySet>() {
            let elements = py_set
                .iter()
                .map(|item| from_pyobject(py, item))
                .collect();
            ExprResult::Set(Container::new(Set::new(elements)))
        } else if let Ok(py_list) = py_obj.extract::<&PyList>() {
            let elements = py_list
                .iter()
                .map(|item| from_pyobject(py, item))
                .collect();
            ExprResult::List(Container::new(List::new(elements)))
        } else {
            // TODO think of a way to detect whether this is an object we can
            // convert or not
            // log(LogLevel::Warn, || {
            //     "Potentially ambiguous CPythonObject instance.".to_string()
            // });
            ExprResult::CPythonObject(CPythonObject::new(py_obj.into_py(py)))
        }
    }
}

And here is the reverse comparison. Note that for both of these we must pass in a Python object, which controls our access to the CPython GIL (global interpreter lock).

impl ToPyObject for ExprResult {
    fn to_object(&self, py: Python) -> PyObject {
        match self {
            ExprResult::None => py.None(),
            ExprResult::Boolean(b) => b.to_object(py),
            ExprResult::Integer(i) => i.borrow().to_object(py),
            ExprResult::String(s) => s.as_str().to_object(py),
            ExprResult::List(l) => {
                let list = PyList::empty(py);
                for item in l.clone().into_iter() {
                    list.append(item).expect("Failed to append to PyList");
                }
                list.to_object(py)
            }
            ExprResult::Function(_) => {
                // TODO our PyCFunction implementation is a no-op, we need to find a way to pass
                // the interpreter into here.
                let callback = |_args: &PyTuple, _kwargs: Option<&PyDict>| -> PyResult<bool> {
                    log(LogLevel::Warn, || {
                        "Potentially lossy PyCFunction invocation.".to_string()
                    });
                    Ok(true)
                };
                // TODO use real function name
                let py_cfunc = PyCFunction::new_closure(
                    py,
                    Some("memphis_func"),
                    None,
                    callback
                ).unwrap();
                py_cfunc.to_object(py)
            }
            ExprResult::Class(_) => {
                // TODO same here, our PyClass implementation does bring real fields
                Py::new(py, TestClass {}).unwrap().to_object(py)
            }
            ExprResult::Module(module) => {
                let py_module = PyModule::new(py, &module.borrow().name()).unwrap();

                // Flatten all key-value pairs from scope into the module
                for (key, value) in module.borrow().dict() {
                    py_module.add(key, value.to_object(py)).unwrap();
                }

                py_module.to_object(py)
            }
            ExprResult::CPythonModule(module) => module.borrow().0.to_object(py),
            ExprResult::CPythonObject(object) => object.0.to_object(py),
            _ =>
                unimplemented!(
                    "Attempting to convert {} to a PyObject, but {} conversion is not implemented!",
                    self,
                    self.get_type()
                ),
        }
    }
}

This is a rich area that I’d like to explore further. Here are some of the directions I’ve considered:

Convert each time an object crosses the FFI interface. (And yes, I realize that acronym expands to foreign function interface interface.) That’s roughly what I’m already doing, I would just need to own it and not feel like an imposter. This could be simple but inefficient.
Keep a registry so that each object exists at most once on each side. This would be more efficient than (1), but it’d require a stable value which you could use to lookup and link up these objects.
Aim for a single representation on the Rust side and use Pyo3 to proxy and lazily convert fields as needed. I believe this would still leverage the functionality of (1), but in a more efficient manner.
Make the memory layout of a Memphis object match that of a PyObject. Similar to how #[repr(C)] already works in Rust, this would be similar to the role an ABI plays for a function call. I’m not even sure if this one is possible given the difference in what each side needs to do its evaluation, but this intrigues me.

I’m getting ahead of myself because I can barely load a C module right now, but there’s truly no end to where my curiosity could take me in this area.

The End

I continue to poke at this when I hit a new conversion failure while plodding along towards getting Flask to boot. This exercise is a good reminder that all objects (or classes, modules, etc) are a set of attributes that exist in a known format in memory. If we understand that format well enough, we should be able to do incredible things, regardless of whether it is on the Memphis or CPython side.

This philosophy drives my work with From Scratch Code as well. If you are tired of being unable to get a library to work in your code, I encourage you to step back and ask: what the library is actually doing? Do you need it, or could a simpler solution work? I believe in cultivating this curiosity about software—and I’d be happy to help you incorporate this mindset into your toolbox.

Building for WebAssembly

Mon, 18 Nov 2024 00:00:00 GMT

I’m currently exploring two interesting topics for Memphis, my Python interpreter in Rust: building for WebAssembly and embedding CPython. With no major milestones to report this week, I thought I’d share some in-progress thoughts. For me, Memphis is been a project for expanding my conceptual understanding through practical experiments—hopefully, this post can do the same for you as we walk through some of the design decisions I'm exploring.

Python in the browser

Compiling Memphis to a WebAssembly target had been in the back of my mind for some time, and two Saturdays ago, I finally gave it a go. With a lukewarm cup of drip coffee on my coaster, I cracked my knuckles and began.

WebAssembly is a sandboxed execution environment inside modern web browsers which complements the traditional JavaScript environment. The Wasm environment is closer to native code and can be used for tasks which benefit from a more performant CPU context; think number crunching or silly busy loops. I was interested in it less from a performance perspective and more because it was possible at all. One of Rust’s selling points (out of literally bajillions) is it can target Wasm. How do, one might ask? This is possible because Rust uses LLVM as its compiler backend. The Rust compiler frontend produces LLVM Intermediate Representation (IR) code and LLVM can compile this to native code for dozens of targets.

That’s a pretty massive benefit and I was curious if it would Just Work for Memphis. I had given literally zero thought to running Python in the browser before, so this seemed like a perfect opportunity to test out the Wasm learning curve.

Setting Up Wasm-Pack and Building for WebAssembly

I fired up my AI assistant and asked for the launch sequence. It went beep boop beep boop. Below are the steps annotated with my learnings along the way.

# wasm-pack helps compile our Rust code to WebAssembly and bundle it with JavaScript bindings we
# can call from our HTML/JavaScript page.
cargo install wasm-pack

# wasm-pack also downloads the wasm32-unknown-unknown target via rustup for us.
# If for whatever reason it does not, you can use this: rustup target add wasm32-unknown-unknown
# We must specify a feature flag because our wasm_bindgen interface is behind the wasm feature flag.
wasm-pack build --target web --out-dir wasm_ui/pkg -- --features wasm

The build succeeded on my first try! However, because we haven’t marked any functions in our Rust binary as being available to call from WebAssembly, it doesn’t do much.

We can install the wasm-bindgen crate to do this, which I put behind a feature flag. I added this to my Cargo.toml.

[dependencies]
wasm-bindgen = { version = "0.2", optional = true }

[features]
wasm = ["wasm-bindgen"]

Here’s a small piece of code I added to my src/lib.rs file, behind the wasm feature flag. The greet function is decorated with #[wasm_bindgen] to make this symbol available in JavaScript.

#[cfg(feature = "wasm")]
mod wasm {
    use wasm_bindgen::prelude::wasm_bindgen;

    // Export a function to JavaScript
    #[wasm_bindgen]
    pub fn greet() -> String {
        "Hello from WebAssembly!".to_string()
    }
}

Creating a JavaScript Interface

I also asked my AI assistant for the smallest possible piece of JavaScript I could use to test my Wasm interface. When we call init(), the browser loads the .wasm file, performs a JIT compilation step to convert the portable WebAssembly binary into native code, and initializes memory for the WebAssembly runtime.

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <title>Wasm Test</title>
  </head>
  <body>
    <script type="module">
      import init, { greet } from "./pkg/memphis.js"

      async function run() {
        await init()
        console.log(greet())
      }

      run()
    </script>
  </body>
</html>

Like a miracle among miracles, it Just Worked. Granted, I wasn’t running any Python code in the browser, but interfacing with my binary was a HUGE step that younger-me-who-could-barely-install-java did not want to undervalue.

The next step was to give it a Python expression defined in JavaScript and have the Wasm binary crunch the numbers. As I mentioned in my REPL post, every entry point in a software project is an opportunity to improve my abstractions, and it would certainly be the case again here. As I thumbed through my Memphis repo, I realized Wow, I should really have a better interface to pass a string and evaluate it as Python. Like I said, I LOVE new entry points.

For the time being, I would use my crosscheck adapter. Crosscheck is my work-in-progress testing framework to validate the treewalk interpreter and bytecode VM produce the same behavior for a given Python input. It’s named after the thing flight attendants do.

Here is my updated Rust code.

#[cfg(feature = "wasm")]
mod wasm {
    use wasm_bindgen::prelude::wasm_bindgen;

    use crosscheck::{InterpreterTest, TreewalkAdapter};

    // Export a function to JavaScript
    #[wasm_bindgen]
    pub fn greet() -> String {
        "Hello from WebAssembly!".to_string()
    }

    #[wasm_bindgen]
    pub fn evaluate(code: String) -> String {
        let result = TreewalkAdapter.execute(&code);
        format!("{}", result)
    }
}

Here is my updated JavaScript code, which invokes the new Rust evaluate function.

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <title>Wasm Test</title>
  </head>
  <body>
    <script type="module">
      import init, { greet, evaluate } from "./pkg/memphis.js"

      async function run() {
        await init()
        console.log(greet())
        const expr = "[ 2 * i for i in range(5) if i % 2 == 0 ]"
        console.log(expr, "=", evaluate(expr))
      }

      run()
    </script>
  </body>
</html>

Debugging WebAssembly Errors

Now when I ran it I got……… a console error. It crashed with an unimplemented error.

I poked around a bit and it was not clear what was causing this. You can click into the source but for a Wasm build that is just a block of assembly without references to the original Rust functions.

I did some AI chatting/Googling and found two helpful approaches. One is console_log for use in Wasm builds, which displays log statements from your Rust code in your browser console. This helped some, but what I was really looking for was a stack trace. Enter console_error_panic_hook. It gave me the Rust stack trace immediately, which was CLUTCH. If you are doing your own Wasm build, stop reading this now and add this crate. I don’t even mind if you never finish reading this post. Ferris would want you to use this crate 🦀. Here’s how I added it to my Wasm interface.

#[cfg(feature = "wasm")]
mod wasm {
    use console_error_panic_hook::set_once;
    use wasm_bindgen::prelude::wasm_bindgen;

    #[wasm_bindgen]
    pub fn evaluate(code: String) -> String {
        // Set the panic hook for better error messages in the browser console
        set_once();

        let result = TreewalkAdapter.execute(&code);
        format!("{}", result)
    }
}

My stack trace pointed me to my culprit: I was using std::env to request some OS resources, which are not allowed in a Wasm runtime (that’s the sandboxed part). I put these calls behind a feature flag (they are related to how I hack-ily determine the location of the Python standard lib on the host machine) and fired up my build again. After a few small failures related to properly displaying my return types….

IT WORKED. Here’s what I now see in my browser console.

wasm_ui/:13: Hello from WebAssembly!
wasm_ui/:15: [ 2 * i for i in range(5) if i % 2 == 0 ] = [0, 4, 8]

tldr I can run Python in the browser. (To their credit, RustPython does this too. I haven’t looked deeply at their project but it seems comprehensive.) The Python list comprehension is defined in JavaScript in string form and the response list is evaluated by the Rust code compiled to Wasm and converted back into a string which can be displayed by JavaScript.

This setup only supports expressions at the moment. To evaluate statements (and later read back their results), I will need to keep state on the Rust side. I also dream of building a JavaScript REPL. That sounds like a problem for future-me (and a boring dream tbh).

The End

I’ve been talking long enough, so I’m going to hold off on discussing embedded Python until next Monday.

Apologies for the bait and switch. The content calendar waits for no one.

To be clear, by embedded Python, I mean embedding a CPython interpreter inside of Memphis, not running Python in an “embedded systems” environment. That would be hard for no reason. Unlike Memphis, which is hard for FUN.

Introducing: From Scratch Code

Mon, 04 Nov 2024 00:00:00 GMT

THE BIG CITY—From Scratch Enterprises LLC (ticker: FSE) announced its newest venture Monday, From Scratch Code (ticker: FSC). Members of the media gathered around the folding chair of its owlish founder, Jones Beach. Refreshments were not provided.

Whispers circulated among the media contingent that this was the same desk which produced the not-a-non-profit, From Scratch Press (ticker: FSP). The representative present could not confirm and barely glanced up from their phone.

“After becoming the market leader in telling autism stories no one asked for, we stepped back and asked ourselves what was next,” said Beach. “It became clear that we could go beyond the abcs and move into 1s and 0s.”

The event continued with a personal statement read aloud by Beach, which was a weird format, but seemed heartfelt.

I never set out to make the killer app. When I was building an early project—a website about stadiums—people asked me when they could expect an app, I looked at them feeling under-appreciated and directed them to my clumsy website. I’m not motivated by attempting to build the next big thing, but by creating something genuine and functional.

My skill set as a software engineer is typically valued through monetization rather than words of affirmation. I’m not asking for sympathy about this; I’m incredibly fortunate that people chose to pay me to write code for them for nearly a decade. But when this system stopped working for me, I looked around for what else I could do with my skills. I’m driven by curiosity and a genuine desire to support others, bringing humor and understanding to my work. These values give me far more satisfaction than fitting into the small box an employer needs each quarter, so I set out to build a business that embraces them, with as little BS as possible.

What mentally freed me to arrive at this point was letting go of the need to impress people who didn’t understand my tools, my craft, or my skill set. While that path works for some, it left me feeling unheard and used. Instead of building software to do something interesting, I chose to build software that is, itself, interesting—a kind of art for art’s sake and my personal rebellion against a system that seeks to control my time and monetize my output.

I landed on a service business because I crave 1:1 connection. My favorite moments in my 9-5 weren’t building products but nerding out with a colleague over an obscure programming language feature. Tutoring proved I could make non-zero dollars doing what I love most. Today, I’m expanding this into my own brand and platform, where creativity and emphasis on the individual can shine—free from high platform fees and other external constraints.

From Scratch Code is for people who already know how to write code and want to learn how to write even better code. Who want to build their own libraries in Rust and Python and understand how programming languages and computers work together under the hood. Who want to have a technical support system which takes not being serious very seriously. I’ll continue to work with students and beginner developers on Wyzant, but here, you’ll find a space where creativity and curiosity are the main drivers.

If any of this resonates with you, I encourage you to sign up for my email list. There, I’ll be telling silly stories about the Rust and Python code I’m building—like my current interpreter project—and the things we could learn together, either through mentorship or courses. I’ll continue to discuss the mental health and adult-diagnosed autism side of my story on From Scratch Press. I can’t think of a better way to fully present the two sides of myself to the modern economy than by maintaining parallel newsletters!

I’ll wrap up with this: my inability to form even a single sentence of what feels like BS played a large role in my decision to leave corporate, so everything I’m building with From Scratch Code is genuine and designed to help you thrive in your technical work. On the flip side, I’m a firm believer that humor and creative absurdism can make people smile and expand what’s considered possible. As such, I found myself with no patience for people who (or systems which encourage people to) say “I’m working with person A on project B” when everyone in the room knows person A doesn’t respond to emails and project B will be scrapped. Perhaps this impatience with non-reality is an autism thing. I’m drawn to the person who says “what if we built project C on the moon?!” Those are the people stretching boundaries and refusing to live in the small boxes corporate life often imposes.

I want to be that person for you, your career, and your technical work. Your unlicensed technical therapist. A supportive listener who doesn’t take insurance but can debug your code.

Feel free to share this message with anyone who might enjoy some offbeat creativity alongside their technical growth—I’d love to connect with them!

This is going to be fun. I hope you’ll join me!

The event concluded ten minutes after it began.

James Beach covers culture and satire in The Big City. He lives in The Big City alongside the rest of the literati. He is in no way related to Jones Beach and believes refreshments at press conferences pose an ethical dilemma.

This is cross-posted on From Scratch Press.

A REPL for fat-finger friendly typing

Mon, 21 Oct 2024 00:00:00 GMT

My Python interpreter, Memphis, has a REPL (read-eval-print loop)!

This is old news. As long as you made zero mistakes while interacting with the wise old owl 🦉, you could interpret to your heart’s content. Assuming you never wanted to evaluate the same statement twice, or if you did, didn’t mind retyping it. Also with zero mistakes.

I was perfectly content with this REPL. Thrilled even. I had written home about this REPL. But my bosses’s bosses’ bossi demanded we improve the REPL for the bottom line for the people. They called me into their office and nodded me into the chair across from their mahogany desk. “Some users are expecting the backspace key to work.” TO HELL WITH THE USERS! “The up arrow should bring up their last command.” ARROW KEY SUPPORT IS SO NINETIES! “Can you have this done by the end of Q5?” I QUIT!

So I went back to my desk and improved the REPL.

I improved it so much that all the keys worked. The backspace key, the up arrow, the down arrow, the left arrow, and last, but not least, the backspace arrow. An accountant could have a field day with the new REPL. tick tick tick tick tick tick tick tick tick. That’s the accountant typing numbers, not a bomb slowly diffusing.

I sent the REPL down to the lab and told my main machinist to put a rush job on this order. It’s the REPL I said and from the look in their eyes I could tell they understood. 750ms later the build was complete and we had arrow key support. I took the product back to the big wigs, begged for my job back, and asked them what they thought. They ran a few commands, printed some prints, and added some adds. They made a mistake and hit the backspace key. I rolled my eyes because seriously who makes mistakes but they seemed satisfied. They realized they didn’t want to run a long command they had already typed out and this is where my life went to hell in a handbasket. They. hit. Ctrl. C. Seriously, who does that?! You know that ends the current process, right? RIGHT???

“We need Ctrl-C support by the end of next year.” These people and their demands. I would add Ctrl-C support. But it absolutely would not be within the next two years.

So I went back to my desk and added Ctrl-C support.

What made this REPL worthy of people with fat-fingered tendencies?

Would I be a tool?

I have staked my entire professional and financial future on building things “from scratch,” so I faced a quandary on day 1 of this project. I chose to use crossterm for the key detection primarily because of the cross-platform support. Honestly though, crossterm was very, very good. The API is intuitive and I was especially pleased with KeyModifiers (which we needed to handle Ctrl-C, which I thought was unnecessary, see above).

Raw mode is a pain

We needed it so that the terminal wouldn't handle special keys for us. But damn, I didn't realize it would turn our screen into a malfunctioning typewriter. Anyway, I had to normalize all strings to add a carriage return before any newline characters. Which worked fine and I'm THRILLED about it.

/// When the terminal is in raw mode, we must emit a carriage return in addition to a newline,
/// because that does not happen automatically.
fn normalize<T: Display>(err: T) -> String {
    let formatted = format!("{}", err);
    if terminal::is_raw_mode_enabled().expect("Failed to query terminal raw mode") {
        formatted.replace("\n", "\n\r")
    } else {
        formatted.to_string()
    }
}

/// Print command which will normalize newlines + carriage returns before printing.
fn print_raw<T: Display>(val: T) {
    print!("{}", normalize(val));
    io::stdout().flush().expect("Failed to flush stdout");
}

Integration testing was fun

Under my old REPL (which I preferred, see above), I could test it integration-ally by just running the binary and passing in some Python code to stdin. That stopped working when using crossterm I think because of a contract dispute. I honestly can’t explain it fully, but event::read() would timeout and fail in the integration test provided with stdin input. So I mocked it.

pub trait TerminalIO {
    fn read_event(&mut self) -> Result<Event, io::Error>;
    fn write<T: Display>(&mut self, output: T) -> io::Result<()>;
    fn writeln<T: Display>(&mut self, output: T) -> io::Result<()>;
}

/// A mock for testing that doesn't use `crossterm`.
struct MockTerminalIO {
    /// Predefined events for testing
    events: Vec<Event>,

    /// Captured output for assertions
    output: Vec<String>,
}

impl TerminalIO for MockTerminalIO {
    fn read_event(&mut self) -> Result<Event, io::Error> {
        if self.events.is_empty() {
            Err(io::Error::new(io::ErrorKind::Other, "No more events"))
        } else {
            // remove from the front (semantically similar to VecDequeue::pop_front).
            Ok(self.events.remove(0))
        }
    }

    fn write<T: Display>(&mut self, output: T) -> io::Result<()> {
        self.output.push(format!("{}", output));
        Ok(())
    }

    fn writeln<T: Display>(&mut self, output: T) -> io::Result<()> {
        self.write(output)?;
        self.write("\n")?;
        Ok(())
    }
}

Which resulted in the whole thing becoming a unit test? Honestly I don’t know. At this point, I call it an integration test if I either a) call a binary inside another binary, or 2) launch a server / open a port / listen on a socket inside a test. If you have another definition you’d like to leave in the comments, please don’t because that sounds annoying TBH.

/// Run the complete flow, from input code string to return value string. If you need any Ctrl
/// modifiers, do not use this!
fn run_and_return(input: &str) -> String {
    let mut terminal = MockTerminalIO::from_str(input);
    Repl::new().run(&mut terminal);
    terminal.return_val()
}

fn string_to_events(input: &str) -> Vec<Event> {
    input
        .chars()
        .map(|c| {
            let key_code = match c {
                '\n' => KeyCode::Enter,
                _ => KeyCode::Char(c),
            };
            Event::Key(KeyEvent::new(key_code, KeyModifiers::NONE))
        })
        .collect()
}

We can now test these common scenarios with fairly little boilerplate.

#[test]
fn test_repl_name_error() {
    let return_val = run_and_return("e\n");
    assert!(return_val.contains("NameError: name 'e' is not defined"));
}

#[test]
fn test_repl_expr() {
    let third_from_last = run_and_return("12345\n");
    assert_eq!(third_from_last, "12345");
}

#[test]
fn test_repl_statement() {
    let return_val = run_and_return("a = 5.5\n");

    // empty string because a statement does not have a return value
    assert_eq!(return_val, "");
}

#[test]
fn test_repl_function() {
    let code = r#"
def foo():
    a = 10
    return 2 * a

foo()
"#;
    let return_val = run_and_return(code);
    assert_eq!(return_val, "20");
}

#[test]
fn test_repl_ctrl_c() {
    let mut events = string_to_events("123456789\n");
    let ctrl_c = Event::Key(KeyEvent::new(KeyCode::Char('c'), KeyModifiers::CONTROL));
    events.insert(4, ctrl_c);
    let mut terminal = MockTerminalIO::new(events);

    Repl::new().run(&mut terminal);
    assert_eq!(terminal.return_val(), "56789");
}

Code entrypoints get me out of bed in the morning

One of my motivations in adding a REPL at all was because I believe you make your code better when you add a second entrypoint. You are essentially becoming the second user for your library, which helps you get closer to understanding The One Perfect Abstraction we are all poking at our keyboards in search of. I mean this point earnestly.

“Zero Dependencies” 😉

The REPL is now behind a feature flag as a way to get back at management. I am keeping alive the ability to interpret Python code with the help of zero third-party crates, which means crossterm would either need to be an exception or I would introduce a feature flag. Now, if you compile without the REPL enabled and run “memphis”, it will politely tell you “wrong build, dumbass.”

Goodbye

The REPL is here. You can run it like this. If you want to buy it, that sounds like a scam. Be well & Talk soon.

Declarative macro magic from Axum in Rust

Mon, 07 Oct 2024 00:00:00 GMT

Take a quick glance at the code snippet below. Without thinking too hard, is get a function or a method? How about post?

Router::new()
    .route(“/one”, get(get_handler).post(post_handler))
    .route(“/two”, post(post_handler).get(get_handler))

What did you decide: is get a function or a method? It appears to be both! And the same with post!

If you are familiar with API development in Rust, you may recognize this syntax from Axum. This puzzling question piqued my curiosity and led me to build Cairo, where I re-implemented some key Axum concepts in a simpler way as a learning exercise. I’ve long been fascinated by the intermediate + advanced concepts library authors employ to make their interfaces feel frictionless. Rust sits in a brilliant space because of the way it melds high-level expressiveness with low-level control and performance. One of the ways it accomplishes this is using macros.

What is a macro in Rust?

Macros in Rust come in two key forms: declarative and procedural. This post will focus on declarative, but let’s briefly define both.

A declarative macro in Rust uses the function!(..) syntax, which you may be familiar with from print!(“Hello World!”) or panic!(“at the disco!”). It is a form of metaprogramming which allows developers to reduce boilerplate code. It does this by evaluating the declarative macro at compile time, which produces more Rust code which ultimately ends up in the binary of the program.

If you have wondered why print!(..) in Rust is a macro, it is due to its dynamic nature. All of these are valid Rust:

println!("Hello");
println!("Hello {}", "World");
println!("Hello {}{}", "World", "!");

In a ~~language with rules~~ strongly-typed language, a traditional function would typically not support a variable number of parameters like this (also called variable-arity). However, Rust’s declarative macros use a $(..),* syntax to support variable-arity by expanding to different code based on the number of parameters passed. The compiled code may end up calling different function implementations or generating specific code for each unique pattern of parameters.

A procedural macro in Rust uses the #[...] syntax, which you may have seen in #[tokio:main]. This macro is applied to a Rust function or struct, allowing the macro to modify the syntax tree of the annotated item at compile time. While it often helps reduce boilerplate code, it does so by transforming the function or struct as a whole rather than directly inserting statements into the function body. For example, #[tokio::main] wraps the entire main function in a tokio async runtime, enabling it to block on the execution of async code in main.

Code without macros can be repetitive and say the same thing in multiple ways

Axum is one of the most popular web frameworks in Rust. Its compatibility with the Tokio ecosystem and its powerful syntax, among other features, keep it near the front of the pack.

Skipping back to my original bafflement at whether get is a function or a method, the answer is both and it does this using a declarative macro.

If we look at our snippet of interest, get must be a function available in the outer scope and a method on whatever type is returned by invoking post. Similarly, post must be a function available in the outer scope and a method on whatever type is returned by invoking get. This symmetry pleases me. Let’s write some code that accomplishes exactly that and nothing more.

use std::collections::HashMap;

/// Define a trait with a single method `handle`.
/// This trait allows different types to implement handler logic, which is useful in API routing
/// systems.
trait Handler {
    fn handle(&self);
}

/// Define an enum representing HTTP methods.
/// Deriving `Hash`, `Eq`, and `PartialEq` allows this type to be used as a `HashMap` key.
#[derive(Hash, Eq, PartialEq, Debug)]
enum Method {
    Get,
    Post,
}

/// Start of a method-chaining function for the `GET` HTTP method.
fn get<H: Handler + 'static>(handler: H) -> InnerType {
    InnerType::new().on(Method::Get, handler)
}

/// Start of a method-chaining function for the `POST` HTTP method.
fn post<H: Handler + 'static>(handler: H) -> InnerType {
    InnerType::new().on(Method::Post, handler)
}

/// Define a struct that holds a collection of routes. We give it a silly name here to emphasize
/// this is internal to our Axum-like library. Specifically, each `Router` would hold several of
/// these struct instances, each mapped to a specific string-pattern representing a route. This is
/// outside the scope of this example, please see [Cairo](https://github.com/fromscratchcode/cairo)
/// if you are interested to learn more.
struct InnerType {
    /// `Box<dyn Handler>` enables dynamic dispatch, allowing for different handler types in the
    /// `HashMap`.
    routes: HashMap<Method, Box<dyn Handler>>,
}

impl InnerType {
    /// Initialize our empty instance.
    fn new() -> Self {
        Self {
            routes: HashMap::default(),
        }
    }

    /// Add a `Handler` for a specific HTTP method to the routes map.
    fn on<H: Handler + 'static>(mut self, method: Method, handler: H) -> Self {
        self.routes.insert(method, Box::new(handler));
        self
    }

    /// Method-chaining function for the `GET` HTTP method.
    fn get<H: Handler + 'static>(self, handler: H) -> Self {
        self.on(Method::Get, handler)
    }

    /// Method-chaining function for the `POST` HTTP method.
    fn post<H: Handler + 'static>(self, handler: H) -> Self {
        self.on(Method::Post, handler)
    }
}

At this point, we see some duplication, but the boilerplate isn’t overwhelmingly negative. However, what about when we add support for the remaining HTTP verbs: OPTIONS, HEAD, PUT, DELETE, PATCH? We’d find ourselves maintaining 7 functions and 7 methods. Given the choice between maintaining 14 blocks of code versus adding complexity to prove I know a language, I'll choose the latter every single time. Enter declarative macros.

Declarative Macros in Axum

The code below is a mix of Axum and Cairo. We introduce two macros: add_http_function and add_http_method.

/// A declarative macro which will define a function `$name` which accepts a `Handler` to be
/// registered to a particular HTTP `Method::$method`.
macro_rules! add_http_function {
    (
        $name:ident, $method:ident
    ) => {
        fn $name<H: Handler + 'static>(handler: H) -> InnerType {
            on(Method::$method, handler)
        }
    };
}

/// A declarative macro which will define a method `$name` which accepts `self` and a `Handler` to
/// be registered to a particular HTTP `Method::$method`.
macro_rules! add_http_method {
    (
        $name:ident, $method:ident
    ) => {
        fn $name<H: Handler + 'static>(self, handler: H) -> Self {
            self.on(Method::$method, handler)
        }
    };
}

You'll notice that these both accept two ident parameters, a $name and a $method. The $name becomes the function name, which will be get, post, etc., while the $method is concatenated to Method::, meaning we must give a valid Method enum variant. It's okay if this is confusing at first—metaprogramming is a different way of thinking about programming. Instead of asking "what should my code do?" we are now asking "what code should my code produce?"

You'll also notice that while they both define a "function" with the name $name, add_http_method accepts a parameter self. This means we must invoke our macro (by calling add_http_method!(get, Get)) in a context which is aware of a self. For us, this will be inside our impl InnerType block.

Putting it all together

Putting it all together, here is our new library with minimal boilerplate and support for method chaining for 7 HTTP methods.

use std::collections::HashMap;

/// Define a trait with a single method `handle`.
/// This trait allows different types to implement handler logic, which is useful in API routing
/// systems.
trait Handler {
    fn handle(&self);
}

/// Define an enum representing HTTP methods.
/// Deriving `Hash`, `Eq`, and `PartialEq` allows this type to be used as a `HashMap` key.
#[derive(Hash, Eq, PartialEq, Debug)]
enum Method {
    Get,
    Post,
    Options,
    Head,
    Put,
    Delete,
    Patch,
}

/// Define a function to be called by our `add_http_function` macro. For a given HTTP method, this
/// function will register a handler and return an type which supports method chaining.
fn on<H: Handler + 'static>(method: Method, handler: H) -> InnerType {
    InnerType::new().on(method, handler)
}

/// A declarative macro which will define a function `$name` which accepts a `Handler` to be
/// registered to a particular HTTP `Method::$method`.
macro_rules! add_http_function {
    (
        $name:ident, $method:ident
    ) => {
        fn $name<H: Handler + 'static>(handler: H) -> InnerType {
            on(Method::$method, handler)
        }
    };
}

// Invoke our declarative macro once for each HTTP `Method`.
add_http_function!(get, Get);
add_http_function!(post, Post);
add_http_function!(delete, Delete);
add_http_function!(head, Head);
add_http_function!(options, Options);
add_http_function!(patch, Patch);
add_http_function!(put, Put);

/// A declarative macro which will define a method `$name` which accepts `self` and a `Handler` to
/// be registered to a particular HTTP `Method::$method`.
macro_rules! add_http_method {
    (
        $name:ident, $method:ident
    ) => {
        fn $name<H: Handler + 'static>(self, handler: H) -> Self {
            self.on(Method::$method, handler)
        }
    };
}

/// Define a struct that holds a collection of routes. We give it a silly name here to emphasize
/// this is internal to our Axum-like library. Specifically, each `Router` would hold several of
/// these struct instances, each mapped to a specific string-pattern representing a route. This is
/// outside the scope of this example, please see [Cairo](https://github.com/fromscratchcode/cairo)
/// if you are interested to learn more.
struct InnerType {
    /// `Box<dyn Handler>` enables dynamic dispatch, allowing for different handler types in the
    /// `HashMap`.
    routes: HashMap<Method, Box<dyn Handler>>,
}

impl InnerType {
    /// Initialize our empty instance.
    fn new() -> Self {
        Self {
            routes: HashMap::default(),
        }
    }

    /// Add a `Handler` for a specific HTTP method to the routes map. This will be invoked from our
    /// `add_http_method` macro.
    fn on<H: Handler + 'static>(mut self, method: Method, handler: H) -> Self {
        self.routes.insert(method, Box::new(handler));
        self
    }

    // Invoke our declarative macro once for each HTTP `Method`.
    add_http_method!(get, Get);
    add_http_method!(post, Post);
    add_http_method!(delete, Delete);
    add_http_method!(head, Head);
    add_http_method!(options, Options);
    add_http_method!(patch, Patch);
    add_http_method!(put, Put);
}

The End

While we've only scratched the surface of the metapossibilities of Rust's declarative macros, I hope this post has inspired you to explore more of what Rust can do at compile time. Procedural macros are another wonderful beast which I would encourage you to dig into if you are interested in ASTs and language development.

If you are curious about more ways Axum and APIs work under-the-hood, I encourage you to check out my course HTTP Server in Rust. The final module is public as the repo Cairo, which shows the north star you will build towards throughout the course. In addition to the macro usage we described here, you’ll explore the details of HTTP and how to create a Router which accepts handlers with variable parameters and return types using advanced type erasure.

Now it's your turn! I'd love your perspective in the comments on these two questions:

How have you used declarative macros in your own Rust projects to reduce boilerplate code?
Are there any Rust crates you have used with an interfaces that makes you go "how did they do that?!"

From Scratch Code RSS Feed

Build Your First REST API in Rust - LIU Brooklyn Workshop

Build Your First REST API in Rust - CCNY Workshop

Write your first library

What is a library?

You’ll feel more confident in your stack.

Your discernment around other libraries will increase.

You'll learn to design interfaces.

It’s empowering.

Memphis goes online: adding socket support

A net module for Memphis

Implementing the net module

1. Building the module

2. Defining our native structs

3. Wiring them into the type system

4. Adding builtin functions

5. Storing native payloads on objects

Introducing Enid

Wrapping up

Python’s operator chaining is mildly interesting

Fixing the parser

Fixing the treewalk interpreter

Fixing the bytecode VM

Putting it all together

The end

What I’ve learned from 200+ hours helping developers grow

You don’t need to be extroverted

If they’re not asking questions, that might be a good sign

Emotional safety looks different for everyone

Code is often just the entry point, we stay for the connection

How global variables work in Python bytecode

Bytecode Compilation

VM Execution

Global Store Mutations

The End

How local variables work in Python bytecode

Bytecode VM concepts

What the heap?!

The stack in action

The bytecode in action

The End

Verifying two interpreter engines with one test suite

Testing expressions

Testing statements, classes, and more

Why did I delete the old crosscheck?

The End

I'm embarrassed by how much code I cut from my test suite

Designing an expressive parser test suite

Expressing Python types

Expressing operations

Wrapping the parser entrypoint

Wrapping the happy path error handling

The End

I left corporate and still do roadmaps + a Memphis update

Q1 Roadmap

Multiple Execution Engines

Common Entrypoint

Common Return Type

Common Debug Stack Trace

The End

Build a software career with meaning: a playbook

Building a Markdown blog with links optimized for Gatsby

Background

List of Requirements

Supporting Markdown Posts

Enabling Syntax Highlighting

First Attempt: gatsby-remark-prismjs

Optimizing Internal Links

Second Attempt: Switching to Rehype

What’s next?

Typed integers in Rust for safer Python bytecode compilation

The problem

The solution

The end

How I added support for nested functions in Python bytecode

Why we use bytecode in the first place

Context matters for Python variables

LOAD_GLOBAL versus LOAD_FAST

What’s in a varname?

What’s next for Memphis?

A `net` module for Memphis

Implementing the `net` module

Why did I delete the old `crosscheck`?

First Attempt: `gatsby-remark-prismjs`