Building a Functional Core in Elixir

Do Fun Things with Big, Loud Worker-Bees

Trinity College Library, Dublin

I admit it. I freely admit my guilt. I have neither excuses nor shame. I admit that I regularly buy programming books just so that I can curl up with them in a good chair and go through them from cover to cover, without ever opening a laptop. I thrive on the rabid consumption of new knowledge, no matter the subject.

That’s right, I’m that guy that reads a book multiple times. The first time I go through because I’m excited and inspired about some topic, and I don’t want to lose the momentum by putting the book down — without regard for the author’s advice to do just that. I get the “big picture” of the book from the first read. I love reading, I love books, but I loathe reading with a laptop or computer open next to me. It entirely ruins the ergonomics of reading for me.

It’s only on the second pass that I will actually stop and do the examples, and only then if the book’s material kept me inspired throughout my first traversal.

Yesterday I found myself with some spare time, idle hands, and an idea. For nearly two years now, my brain has been gnawing on the notion of an application that federates media libraries so I can “rent” a movie from my desktop tower to a RasPi or tablet or something portable to take with me on trips.

I’ve had a few false starts with this application before. I’ve run out of time, inspiration, or both. Yesterday I was frustrated again but remembered the chapter from Bruce Tate and James Edward Gray’s fantastic Designing Elixir Systems with OTP book on Building a Functional Core. I thought it might be fun to sit down and just start modeling a functional core based on my idea.

In the application crammed into my brain, there is a central “pool” of media libraries; an aggregate of discovered/registered libraries residing on disparate devices. Each library has a unique ID, a name, etc, and contains media items. Each media item has things like the title, description, file size, and maybe the run-time (so I can plan my mobile viewing accordingly).

There’s also an interesting aspect where I don’t want any of the media-holding devices to need a direct connection with each other. I want to be able to request (“rent”) a video from my tower, and it’ll appear magically on my tablet or other device eventually, and never require the two to have a direct connection. This is for both security (I don’t want to punch firewall holes in my house to allow me to access it from a hotel) and convenience. To manage this, I envision that there’s a staging area (which could be a cloud VM, or just some box I have isolated from my real home network), where the file contents are cached waiting for the requester/target to come online and fetch it. This might be me coming online from a hotel’s WiFi or some other passive activity that requires no interaction from me.

I’ve jumped into this project head first a number of times before. However, during each of those previous attempts, I’ve started in the weeds and wrote the streaming code, web sockets, file transfers, more networking stuff — all the distributed systems shiny bits that fire my neurons. After reading the aforementioned book, I was ready to try a new approach to getting started: building a functional core.

There’s a powerful concept in this book that dictates a hard separation between your functional core, your GenServers (actors), and the client API that other aspects of your application use to interact with your actors. It makes perfect, beautiful sense to me now but I’ve certainly never been disciplined enough in the past to do it, despite drawing similar hard lines on projects I’ve done in other languages.

So I started and created a module called FileBlock :

defmodule Librarian.Core.FileBlock do
    defstruct index: 0,
              hash: ""
def new(index, hash) do
        %__MODULE__{
            index: index,
            hash: hash
        }
    end
end

This is just a simple struct, but it helped get my brain thinking about the state I would need to maintain, and the types of mutations and translations I might need to make to data. Coming up with the types of data conversion activities led to creating a few more modules. As you’ll see, requests will use file blocks.

I created a MediaItem module, then MediaLibrary, then LibraryPool. These three interact with each other in the way you might expect — a pool holds multiple libraries, and each library holds multiple items. I wrote some functions to search through the pool to find a media item:

def find_items(pool, text) do
   pool
   |> all_items()
   |> items_matching_text(text)
end

I will be honest with you, before reading this book I would not have written my code like this. The authors remind us of the value of maintaining a single level of abstraction in the code. This means I don’t want to mix calls like Enum.filter with high-level functions like all_items etc. The code I pasted above is probably the 4th or 5th iteration and far cleaner than what I started with. You can see how, even if you’re not an Elixir developer, it’s pretty self-documenting.

Someone approaching this code with no previous context can easily see that we grab a list of all the items in a pool and then filter that list against a text input.

Take a look at the definition of all_items :

defp all_items(pool) do
   pool.libraries 
   |> Map.values 
   |> List.foldr([], fn x, acc -> acc ++ x.media_items end) 
end

This is pretty tight code (I’m not an Elixir expert, so I suspect this can be cleaned up further), but you can see how if I had placed that foldr expression into the same pipeline as the one in find_items, it would dramatically reduce the readability of the code and violate the single level of abstraction rule. Even as the code’s author, I would ultimately forget how it worked.

This is interesting, but querying the media pool isn’t all that complicated, so it might be hard to see the value-add of the functional core. I want to write servers and OTP and make network connections!

Let’s stay in the functional core and take a look at a more complicated aspect of my app. As I mentioned, I want to be able to use a staging area, and I need to keep track of which chunks of the file have been staged, which ones have been delivered, and which ones are still pending (not yet copied from source). In the future, I may also want to keep track of which blocks failed to transfer and are in need of re-transmission.

I’ve totally wrecked this implementation in the past, in multiple languages. It was a pretty ugly scene. This time around, I recalled the sample from the functional core chapter in the book that explained the functional programming concept of tokens and it inspired me. I decided to treat the request struct as the state of a request at a given point in time, and then move it through transformations like a token.

So I ended up with a pattern where my request has multiple fields, but the 3 key ones are: blocks_pending, blocks_staged, and blocks_delivered. Now, when a block arrives in the staging area, I want to move that block from one field to the other. Likewise, I’ll move it from staged to delivered when that event occurs. It’s worth pointing out that in the functional core, I don’t care how this transition happens. I’m also not keeping track of the actual bytes — that’s well off in “side-effect” land that can be handled by the OTP layer of my application. This also keeps my core eminently testable.

I started off with a module that looks like this:

defmodule Librarian.Core.Request do
    defstruct approved: false,
              destination_agent: "",
              blocks_pending: [],
              blocks_staged: [],
              blocks_delivered: [],
              media_item: nil
def new(media_item, agent) do
    %__MODULE__{
      blocks_pending: media_item.blocks,
      media_item: media_item,
      destination_agent: agent
    }
end
end

This constructor takes the blocks (again, block metadata, not the actual bytes) from the media item and sets them up as “pending”.

Now I want to write a (easily testable!) function that can move a block from the pending state to staged. This is where following the abstraction rules and other guidelines from the book really pays off.

Take a look at the functions to mark a block as staged or delivered:

def stage_block(request, block) do
    request
    |> move_block(block, :blocks_pending, :blocks_staged)
end
def deliver_block(request, block) do
    request 
    |> move_block(block, :blocks_staged, :blocks_delivered)
end

These functions are all at the same abstraction level, and it’s clear that the code is moving a block from one list to another, depending on the circumstances. Here’s the move_block function:

defp move_block(request, block, field_from, field_to) do
    request
    |> remove_block_from(block, field_from)
    |> add_block_to(block, field_to)
end

I iterated over this function a number of times as well. My first version again violated the single level of abstraction rule. Now, as you can see, even this function is easy to read, performs a single task, and is easily tested. I don’t drop down into the Elixir primitives until the remove_block_from or add_block_to functions:

defp remove_block_from(request, block, field) do
   new_list = Map.fetch!(request, field) |> List.delete(block)
   Map.put(request, field, new_list)
end
defp add_block_to(request, block, field) do
   new_list = Map.fetch!(request, field) ++ [block] 
   Map.put(request, field, new_list)
end

At this point, I suspect that there are a lot of readers who might be thinking that I’ve got “too many functions”. I might have actually agreed with that in the past, but doing things this way just feels right. I get the right vibe from this code, and that’s important to me. I can definitely imagine how much easier it will be to maintain and edit this code going forward, especially for those who weren’t around when the code was first created.

I can’t remember the last time I had fun while writing code. I’ve been productive, I’ve been impressed, and I’ve felt powerful. But fun? Not in a long time. Taking an idea in my head and starting to put form to it following the guidelines in Tate and Gray’s book has made this kind of thing fun again, and I am really looking forward to continuing on a journey that will likely be more rewarding than the end product.

While the mnemonic device from their book (do fun things with big, loud worker-bees) is to help you remember the right layers and abstractions, it also includes doing fun things.