Using libsodium in Lambda, with Clojure

I store some data in S3, like a lot of you probably do. And one of my requirements is that the data is encrypted. You might need that too.

Sodium is a new, easy-to-use software library for encryption, decryption, signatures, password hashing and more.

Of the many ways to encrypt data, the one I chose is libsodium. It’s simple to use, can be used in virtually any platform, and provides good encryption.

My Clojure programs use caesium to interact with the underlying libsodium library. Encrypting is easy:

(defn encrypt
  "Encrypt a message with the given key. The message can be in any form that can be converted into a byte-array."
  [plaintext k]
  (-> plaintext
    bs/to-byte-array
    (msb/secretbox-nmr k)))

And so is decrypting:

(defn decrypt
  "Decrypt a message with the given key. The ciphertext can be in any form that can be converted into a byte-array."
  [ciphertext k]
  (-> ciphertext
    bs/to-byte-array
    (msb/open k)))

Oh! Do you want to know something weird? Apparently, something about Lambda’s environment doesn’t like emojis in your code:

{
    "errorMessage": "Error loading class app.handler: caesium/binding$✨",
    "errorType": "java.lang.NoClassDefFoundError"
}

So caesium needed a quick fix to not do that.

The key needs to be a byte array, but I need to pass environment variables in as a string, so I need a way to convert the key to/from a hex string.

(defn hex->bytes
  "Convert a hex string to a byte-array."
  [hex]
  (DatatypeConverter/parseHexBinary hex))
(defn bytes->hex
  "Convert a byte-array to a hex string."
  [bytes]
  (DatatypeConverter/printHexBinary bytes))

And all that works great … as long as libsodium is installed on the computer where your program is running. But when you’re running in Lambda, it isn’t. So we need to package a native dependency in our JAR, where we can use it.

The first thing we need to do is build libsodium for Lambda’s particular version of Linux. I can’t just package up the binary from my Mac and expect it to run in Lambda!

To build it, I created a new EC2 instance running the same AMI that Lambda uses: Amazon Linux AMI 2017.03.1.20170812 x86_64 HVM GP2. It doesn’t need to be a very big instance, since all we’re going to be doing with it is compiling.

sudo yum install -y gcc gcc-c++
wget https://download.libsodium.org/libsodium/releases/libsodium-1.0.15.tar.gz
tar xzf libsodium-1.0.15.tar.gz
cd libsodium-1.0.15
./configure
sudo make install

Now we have a libsodium binary installed on our server. Let’s make sure it works!

#include <sodium.h>
void main(void) {
    if (sodium_init() < 0) {
        // panic
        printf("libsodium failed\n");
        return 1;
    }
    printf("libsodium succeeded\n");
    return 0;
}

Let’s compile and run that dumb little program:

gcc test.c -lsodium -L/usr/local/lib
LD_LIBRARY_PATH=/usr/local/lib ./a.out

Great. Now, I grab /usr/local/lib/libsodium.so from the server using scp, and I need to put it in my JAR. Checking out the execution environment, I see that LD_LIBRARY_PATH is:

Contains /lib64, /usr/lib64, LAMBDA_TASK_ROOT, LAMBDA_TASK_ROOT/lib. Used to store helper libraries and function code.

And LAMBDA_TASK_ROOT:

Contains the path to your Lambda function code.

For a Clojure project, that means I need the binary in resources/lib:

$ tree resources/
resources/
├── lib
│   ├── libsodium.so -> libsodium.so.23.0.0
│   ├── libsodium.so.23 -> libsodium.so.23.0.0
│   └── libsodium.so.23.0.0

That’s a file at libsodium.so.23.0.0 (which we built in EC2), and a couple of symlinks pointing to it.

Now lein uberjar gets me a JAR that includes libsodium, and caesium will see it in the LD_LIBRARY_PATH just as if it were installed normally.

And because it encrypts/decrypts byte arrays, I can secure files that contains JSON just as easily as files that contain binary data like images. (I do both of those.) And because I’m using libsodium, I can transparently use this from any execution environment (whether my code is running in Lambda or on a more traditional server or locally), and from any platform (Clojure, or Go, or something else if I were some kind of monster who wrote in something other than one of those).

Pretty nice!