Writing my Own Programming Language in Go

This project is licensed under MIT and open sourced on GitHub. Check it out here.

Most of my recent projects have revolved around the web, such as my portfolio site (welcome!) and my digital agency, Muon Design. As much as I love building cool websites, experiences, and interactions, I wanted to challenge myself with something completely different. I've always been fascinated by how magical programming languages are - seemingly turning snippets of pure text into something incredibly powerful and complex. Thus, I decided to set out on a programming adventure to create something of my very own!

During my initial research, I discovered that a lot of the available resources fell into one of two extremes: they were either too shallow to be significantly useful, or overwhelmingly deep and complex. Additionally, I really didn't want to build a compiler, as that's a whole journey on its own. Eventually, I found the Writing an Interpreter in Go by Thorsten Ball. As a fan of the Go programming language, I was immediately hooked on the concept and got my hands on it.

The book really helped to bridge the gaps between those two extremes, as the book itself has tons of useful insight, but is not overwhelmingly long, coming in at around only 200 pages in length.

Over the course of the next few days, I absolutely devoured the book, following the code step by step and making sure I understood everything I wrote. Along the way I noticed some areas that could be improved (such as UTF-8 support) and began forming a mental list of features I wanted in my own language. Once I finished the book, I was ready to begin designing and creating something new from scratch.

Core Features & Philosophy

The first thing I tackled was the language specification. I wanted something that was fun to write, but not something foreign or esoteric. As tempting as it was to create a programming language themed entirely around cats, I ultimately decided I wanted something a bit more professional to use for later projects down the line. For example, writing my own raytracer or game in the language. To support these design goals, I wanted something clean and approachable and slightly Go-inspired. Check out the snippet below:

1// Hey! Comments are supported.
2// Count to x or 5, whichever comes first
3fn countTo(x) {
4  a = 0 
5
6  while (a < x) {
7    a = a + 1 // += isn't supported (yet)
8    
9    // Control flow and comparisons
10    if (a > 5) {
11      break
12    } else {
13      // Log the result to the console
14      out(a)
15    }
16  }
17}
18
19out("Sen also supports UTF-8 😎")

As you can see from the snippet, the language is not revolutionary by any means, but it certainly gets the job done! Something that might jump out at you is the lack of semicolons. I ultimately decided not to include them in Sen, as after writing so much code in Go I really found it to be much cleaner, and also removes that slight friction of forgetting one at the end of the line. This was a surprisingly tricky thing to implement properly, as line endings differ between devices.

As for the functions, I was inspired by the conciseness of the Rust programming language, and as for variable assignments, the let keyword is optional. The control flow structure should be familiar, and I needed the language to support comparisons, console output, while loops, etc.

With my simple specification in hand, I got started writing the language.

Current Features

Before diving into the creation of Sen, I wanted to take a moment and list the features that it currently has!

• UTF-8 input
• Variable assignments
• Ints, strings, and booleans
• While & break statements
• Control flow via if and else statements
• Arrays
• Prefix & infix expressions
• Named and inline functions
• Closures!!!
• Block statements and scope
• Errors that state where the error occurred
• Comments
• Builtin functions including len, push, and out!

In the future, I would like to add the following features. If you're interested in programming languages and want to contribute, I encourage checking the project out on GitHub!

• Classes
• Modules
• A standard library
• File input and output
• Sen as an executable file
• For loops

Implementation

To build a programming language, it’s important to understand the pipeline that transforms raw text into something that can be executed. This involves three lexing, parsing, and evaluation. The lexer breaks down the raw code into individual tokens, parsing structures those tokens into a data structure, and finally the evaluator interprets the structure to produce an output!

Each stage plays a vital role in taking human-readable code and turning it into something the computer can understand and act on.

Lexer Implementation

First step of creating a programming language is to write a lexer! The lexer is super cool and is actually really fun to implement. Essentially, the lexer takes a chunk of text and transforms it into tokens. Tokens are bits of text that actually correspond to some sort of meaning. For example, the word fn is just two characters, but it has meaning behind it. In this case, that's the keyword that tells the programming language: "Hey! We need to define a function!" So, as we move through the text, we need to determine what each part of the text corresponds to the valid tokens for our programming language.

Parser Implementation

After the lexer turns the raw code into tokens, the next step is to give those tokens structure. A common data structure for this task is the abstract syntax tree, or AST. It allows us to easily traverse the structure of our code during the evaluation process.

For Sen, I used a Pratt parser, a parsing technique that makes order of operations easy breezy beautiful. I highly recommend checking out this blog post, as it goes into a lot more depth and detail into how it actually works. Essentially, the way it works is it breaks everything down into either an infix or prefix expression. An infix expression is something that goes between two values, such as +, -, ==, etc. These are common in mathematical operations such as 1 + 2 * 3. The other type is a prefix expression, such as (), [], and even function calls. The parser looks at each token, decides whether it's starting a new expression or continuing one, and builds up the structure of the code based on the rules and precedence you've defined. This approach makes it easy to handle complex expressions while keeping the parser logic simple and extensible.

Evaluator Implementation

Now that we have our AST, we can finally evaluate the code. The evaluator walks through the AST and executes the code that it represents. Each node in the tree triggers some sort of action, such as variable assignments, function calls, and control flow.

Under the hood, I used a simple tree-walking interpreter. It recursively visits each node in the AST and executes the corresponding logic. This approach makes it easy to add new features and experiment with language behavior.

One cool thing about this phase is that it’s where the language really comes alive. All the work done by the lexer and parser leads to this moment, where the code actually does something, whether that's printing to the screen, performing calculations, or calling a function.

Examples

This is a map function that applies a function to each element in an array. Eventually I want to write a standard library for Sen (in Sen code) and include useful functions such as this!

1fn map(array, func) {
2  length = len(array)
3  mapped = []
4  count = 0
5
6  while (count < length) {
7    mapped = push(mapped, func(array[count]))
8    count = count + 1
9  }
10
11  return mapped
12}
13
14
15fn square(x) {
16  x * x
17}
18
19arr = [1, 2, 3, 4, 5, 6]
20
21// "out" outputs values to the console
22out(map(arr, square))
23//>>> [1, 4, 9, 16, 25, 36]
24

Below you can see closures in action:

1fn makeMult(a, b) {
2  fn(c) {a * b * c}
3}
4
5// This will create a new function that will multiply our number by 10
6mult = makeMult(2, 5)
7
8// 2 * 5 * 5 = 50
9out(mult(5))
10
11//>>> 50

Naming the Language

This part was surprisingly tricky!! I had so many different names for the language, and every time I looked it up, it already existed. I had no idea there were so many programming languages out there. I had a few requirements for creating naming it. The name needed to be short, quick to type, and memorable. Eventually the word "Sen" came to mind, and after looking it up it appeared a language by that name didn't exist. As a bonus, I discovered it means lotus in Vietnamese, which is now the logo for the language!

Thanks for reading! This is my first blog post so if you have any feedback I'd love to hear it. If you'd like to contribute, fork, or view the code, it's licensed under MIT and open sourced on GitHub which you can checkout here.