Mutual TLS over gRPC with Elixir

In an ideal world, we would never have to write code that terminates SSL in our own applications, this would be left up to the platform on which we run, probably handled by reverse proxies or service meshes or the like.

However, we sometimes need to write applications or services in such a way that they need to start up and talk securely to other applications and we can’t count on infrastructure to be there to add security on our behalf. More importantly, it is extremely dangerous to assume that merely slapping an “S” on the end of http is going to make our communications secure and tamper-proof. We all should be able to remember a number of recent SSL failures and flaws.

The security I want to implement is called mutual TLS because the client needs to verify that the server is who they claim to be and the server needs to verify that the client is who they claim to be, and that no one has tampered with the messages sent between them. The encryption between the two endpoints, when using properly configured mutual TLS, should be pretty secure and tamper-proof.

If you’ve read many of my tweets or blog posts, you’ll know that I’m a huge fan of gRPC. So naturally when I wanted to see how well I could manage mutual TLS, I decided to build a streaming gRPC example. In this case, it’s a simple mock server that allows clients to report zombie sightings as well as stream a list of zombies nearby based on a location and radius.

The biggest key to success with mutual TLS is in figuring out how to deal with your certificates. Certificates are the bane of many people’s existence and they have on more than one occasion made me question all of my life decisions leading up to my interaction with certs.

For my mutual TLS sample, I tried to keep the certificate setup as simple as possible (I used certstrap to create all of the files):

• Create a Certificate Authority. For self-signed certs, I created one with a common name of the fictitious company managing the sample (Zombie Spotters Ltd).
• Create a cert request for the client and sign it using the new CA
• Create a cert request for the server and sign it using the new CA

To create the certs:

$ certstrap init --common-name "Zombie Spotters Ltd"
$ certstrap request-cert --common-name zombieclient
$ certstrap sign zombieclient --CA "Zombie Spotters Ltd"
$ certstrap request-cert --common-name zombieserver
$ certstrap sign zombieserver --CA "Zombie Spotters Ltd"

Now when I host my gRPC server, I will include the CA certificate and the server certificate. When I create my client, I’ll include the CA certificate and the client certificate. This strategy prevents anyone from talking to my server that doesn’t have a cert that was signed by my CA. For use cases where you’re running on an internal network or testing, self-signed is fine, but if you’re planning on trusting remote strangers in the cloud, you should use a real CA and real certs.

Now that I’ve got certificates, let’s get to building the gRPC server. To do this, I used this gRPC library. In typical gRPC fashion, there’s a language-specific plugin to the protoc binary that produces message data and RPC abstractions. In the case of Elixir, the wrappers were surprisingly small compared to what I’m used to with Go and Rust.

You can see all of the sample code in the umbrella application I created in my github repository.

Let’s take a look at the App module for the zombie server:

defmodule Zombieserver.App do
    use Application
    # cut out some constants for clarity
def start(_type, _args) do
   import Supervisor.Spec
   children = [
      supervisor(Zombieserver.Data, []),
      supervisor(GRPC.Server.Supervisor, [start_args()])
   ]
   opts = [strategy: :one_for_one, name: Zombieserver]
   Supervisor.start_link(children, opts)
end
defp start_args do
        cred = GRPC.Credential.new(ssl: [certfile: @cert_path, 
                keyfile: @key_path,
                cacertfile: @ca_cert_path,
                secure_renegotiate: true, 
                reuse_sessions: true,
                verify: :verify_peer,                                 
                fail_if_no_peer_cert: true]
            )
        IO.inspect cred
        {Zombieserver.Zombies.Server, 10000, cred: cred}
    end
end

The most important part here is the ssl parameter passed to the gRPC server. This is actually going to be passed directly to the underlying Erlang web server (Cowboy). Thanks to the patience and understanding of the folks on the Elixir slack, I finally figured out how to configure this using the Erlang docs as a reference.

Here’s a look at the implementation of the gRPC server (note that it’s not doing a bunch of work, it’s faked up to help me illustrate the TLS stuff):

defmodule Zombieserver.Zombies.Server do
    use GRPC.Server, service: Zombies.Zombies.Service 
    alias GRPC.Server
    alias Zombieserver.Data
   
    @spec report_sighting(Enumerable.t, GRPC.Server.Stream.t) :: Zombies.SightingSummary.t 
    def report_sighting(req_enum, _stream) do
        Zombies.SightingSummary.new(sighting_count: 1, radius: 0) 
    end
@spec zombies_nearby(Zombies.Target.t, GRPC.Server.Stream.t) :: any 
    def zombies_nearby(target, stream) do
        zombies = Data.fetch_zombies
        zombies 
          |> Enum.each(
            fn zombie -> Server.stream_send(stream, zombie) end)
    end
end

I was quite impressed with how simple and straightforward the code was in order to emit data to a gRPC stream.

Next let’s take a look at how the gRPC client is configured to allow for mutual TLS:

defmodule Zombieclient.App do
    use Application
@ca_cert_path   Path.expand("./Zombie_Spotters_Ltd.crt", :code.priv_dir(:zombieclient))
    @cert_path      Path.expand("./zombieclient.crt", :code.priv_dir(:zombieclient))
    @key_path       Path.expand("./zombieclient.key", :code.priv_dir(:zombieclient))
def start(_type, _args) do
        import Supervisor.Spec
   cred = 
     GRPC.Credential.new(ssl: 
      [cacertfile: @ca_cert_path, 
       certfile: @cert_path, 
       keyfile: @key_path,
       verify: :verify_peer,
       server_name_indication: 'zombieserver'])
    children = [
        supervisor(Zombieclient.Consumer, [cred]),            
    ]
          
    opts = [strategy: :one_for_one, name: Zombieclient]
    Supervisor.start_link(children, opts)
  end    
end

Here the configuration is much simpler — I’m not setting any of the SSL server options, I’m just including the certificates. Thanks again to the Elixir slack community for pointing out more security options here. We need to enable peer verification. Without it we’ll get encryption, but we won’t actively reject impostors. By setting server_name_indication and verifying peers, we’re saying that the common name of the server’s cert has to be zombieserver, and we can trust that name assertion because that server’s cert has to be signed by a CA we trust. If we set server_name_indication to some random value like “foo”, we will see a handshake failure and be unable to communicate.

It’s also worth pointing out that even though it’s called the “server name” indication, this isn’t tightly coupled to a host name. This check only cares about the common name of a cert in the chain. In cloud environments you often see this kind of security where you’ll have a cert with a name that indicates its role like master or agent, etc.

If you try and make the client’s SSL configuration the same as the server’s, you might get “handshake failures” like I did while trying to get this to work.

Finally, we can write the code that invokes the “gRPC Stub” to talk to the remote service. Note that once we’ve got a configured channel, none of the actual client code is explicitly aware of the SSL configuration or mutual TLS handshakes.

defmodule Zombieclient.Consumer do
    
    def start_link(cred) do
        IO.puts "started zombie sighting consumer"
        opts = [cred: cred]
        {:ok, channel} = GRPC.Stub.connect("localhost:10000", opts)
        get_zombies(channel)
        Agent.start_link(fn -> %{zombies: []} end, name: __MODULE__)
    end
    def get_zombies(channel) do
        IO.puts "getting zombies"
        center = 
          Zombies.Location.new(latitude: 400000000, 
                               longitude: -750000000)
        target = 
          Zombies.SearchTarget.new(center: center, radius: 12.5)
        stream = 
          channel 
            |> Zombies.Zombies.Stub.zombies_nearby(target)
        
        Enum.each stream, fn (zombie)->
            IO.inspect zombie
        end 
    end  
end

Some of this is just useless cruft that I threw in to make the sample work. For example, I don’t actually need the Agent there storing the list of zombies, but it illustrates how easy it would be to maintain a local/offline cache of the zombie sightings from inside this module.

My key takeaways from this learning exercise are as follows:

• The Elixir slack is a great community and the folks there are super helpful and welcoming, even to people covered in newbsauce like myself.
• gRPC streaming is incredibly useful. You really should explore this if you’re looking at gRPC at all.
• While Elixir often feels awkward to my brain that thinks in Go all day long and has occasional affairs with Rust, the effort required to go from concept to a fully functioning, mutual-TLS-secured gRPC microservice is extremely small and even smaller for people who work in Elixir regularly.

All in all, I thoroughly enjoyed this experiment. Of course, now I’m going to have to play with how to set up mutual TLS over gRPC in other languages to compare!