“En passant” Encryption in Go

Anyone storing critical data has to think about security. While you can rely on infrastructure and middleware to do some of the work (e.g encrypted filesystems and the like), application developers should not assume that such mechanisms are always in place (and should code as if they were not). To that end, many patterns have been developed to work with popular persistence frameworks — Ruby developers might define custom attribute accessors, while Java developers might follow a different set of patterns with e.g. JPA/Hibernate.

With the popularity of document-based/NoSQL databases, for which you cannot generally piggyback on relatively mature ORM layers, developers end up spending a lot of time converting data to and from JSON (or BSON, etc.). When I’m doing this kind of work in Go, I like to use a pattern that I call “en passant” encryption.

En passant is a particular kind of move in chess, wherein a pawn captures another pawn by moving through/past it — thus the “en passant.” I use the term here because it captures the sense of doing one thing (marshaling to JSON) with a positive side effect (encrypting critical fields) as a result. The main benefit is that it simplifies the code path and follows what is actually a very similar pattern to the other examples cited above: data is automatically encrypted when it is persisted, and it is decrypted when it is read from storage, and the encryption requires little or no additional action on the part of the developer, once the initial wiring is done.

Implementing this pattern is really quite simple. There are two mechanisms to employ — implementing custom implementations for json.Marshaler/Unmarshaler, and creating your own annotation tags. In this post I will be discussing the former approach (which I personally find the simpler). I may explore the other approach in a subsequent post.

Let’s assume that our data is contained in a struct type, and one of the fields in the struct must be encrypted before persisting. Our struct might look like this:

type StructWithEncryptedType struct {
  EncryptedField EncryptedString `json:"encrypted_field"`
  PlainField string `json:"plain_field"`
}

You’ll note that the type of EncryptedField is EncryptedString. This is just a type alias for string:

type EncryptedString string

All I need to do to ensure that, when writing this out as JSON, the field gets encrypted, is implement a custom implementation of the json.Marshaler interface, which has one method, MarshalJSON():

func (e EncryptedString) MarshalJSON() ([]byte, error) {
  plaintext := []byte(e)
  blockCipher, _ := aes.NewCipher(key)
 
  encryptor, _ := cipher.NewGCM(blockCipher)
  nonce := make([]byte, encryptor.NonceSize())
  rand.Read(nonce)
  encrypted := encryptor.Seal(nil, nonce, plaintext, nil)
  cipherbytes := append(nonce, encrypted...)
  
  ciphertext := base64.StdEncoding.EncodeToString(cipherbytes)
  return json.Marshal(ciphertext)
}

I’ve dispensed with any error-checking here, and key in this example is initialized in the init() method for the package (which is something I like to do as a rule). When you call json.Marshal(thing), go first checks if thing implements the MarshalJSON() function — if it does, then that is called, otherwise go uses reflection to get the type and does its best to marshal — in my case, since this is just a type alias to string, go would have simply returned the standard marshaling for a string. When I call the above code :

func main() {
 myStruct := StructWithEncryptedType{
   EncryptedField: "thisneedstobeencrypted",
   PlainField: "donotencryptthis",
 }
 
 b, _ := json.Marshal(myStruct)
 
 fmt.Printf("%s\n", string(b))
}

it returns the marshaled struct with the field encrypted (and base64-encoded) automatically:

{"encrypted_field":"AazY56GW6MZlZ+MSsAf+B3m5PIPk5Sk6KzeMdcEyJPXSVRqtMrpv5q0lpP/sXePY/wA=","plain_field":"donotencryptthis"}

(Note: since the nonce is random, you should not expect this exact output if you run this code via the go playground link at the bottom.)

Decryption is no more complex. In this case, you simply need to implement the json.Unmarshaler interface, which has a single method, UnmarshalJSON:

func (e *EncryptedString) UnmarshalJSON(b []byte) error {
  // Strip quotes because we aren't in the standard golang json
  // unmarshaler
  if len(b) > 0 && b[0] == 0x22 {
    b = b[1:]
  }
  if len(b) > 0 && b[len(b)-1] == 0x22 {
    b = b[:len(b)-1]
  }
  cipherbytes, _ := base64.StdEncoding.DecodeString(string(b))
  blockCipher, _ := aes.NewCipher(key)
  decryptor, _ := cipher.NewGCM(blockCipher)
 
  nonce := cipherbytes[:decryptor.NonceSize()]
  
  decrypted, _ := decryptor.Open(
    nil,
    nonce,
    cipherbytes[decryptor.NonceSize():],
    nil)
  *e = EncryptedString(string(decrypted))
  return nil
}

Again, I’ve dispensed with any error-checking here, but a couple of aspects are worth noting:

1. MarshalJSON() takes a value receiver, but UnmarshalJSON() takes a pointer receiver.
2. The default golang marshaling code will handle stripping quotes for you, but you’ll have to be sure to deal with that in a custom UnmarshalJSON().
3. Although the actual encryption here definitely falls into the “quick-n-dirty” category, I’ve nonetheless followed the pattern of storing the nonce with the encrypted value, since the nonce is randomized and generated anew for each message to be encrypted (what you should always do).
4. I’ve base64-encoded the encrypted value before storing it, and I decode it before decrypting. Since I’m storing this in a document database (or maybe a key-value store like etcd), I want to make sure that I’m working with printable strings.

So now that I’ve expanded main() to do both marshaling and unmarshaling:

func main() {
 myStruct := StructWithEncryptedType{
   EncryptedField: "thisneedstobeencrypted",
   PlainField: "donotencryptthis",
 }
 
 b, _ := json.Marshal(myStruct)
 
 fmt.Printf("%s\n", string(b))
 fmt.Println(strings.Repeat("=", 30))
 
 target := &StructWithEncryptedType{}
 
 json.Unmarshal(b, target)
 
 fmt.Printf("%v\n", target)
}

the output looks like this:

{"encrypted_field":"AazY56GW6MZlZ+MSsAf+B3m5PIPk5Sk6KzeMdcEyJPXSVRqtMrpv5q0lpP/sXePY/wA=","plain_field":"donotencryptthis"}
==============================
&{thisneedstobeencrypted donotencryptthis}

You can find the full sample program at the go playground, and I’ll put it up on GitHub as well. I find this a useful pattern, and it’s pretty easy to implement — once you’ve got the types defined with the interfaces implemented, all other developers need to do is be sure to use your custom types when building structs, and go will do most of the heavy lifting for you!