Running Go AWS Lambda functions locally

Saturday, January 27, 2018

AWS recently announced Go support for Lambda, giving developers more choice over how their functions are written.

In an attempt to kick the tires of the new runtime, I found myself rummaging around the open source library required when writing Lambda functions in Go, and was delighted to find a glimpse into what happens when your function is invoked. 

This post is a brief tour of what I’ve gathered, and describes a simple way of invoking your function in a local environment.

tl;dr: AWS Go Lambdas are invoked using net/rpc over TCP and make use of the Go standard library. You can “simulate” a lambda being invoked, which could be useful for integration tests or sanity checking, see below for an example

Background

A Go lambda needs two things to run

1. A handler to handle requests
2. A main function that calls lambda.Start(...) with your handler as an argument

package main

import (
    "strings"
    "github.com/aws/aws-lambda-go/lambda"
)
 
func ToUpperHandler(input string) (string, error) {
    return strings.ToUpper(input), nil
}
 
func main() {
    lambda.Start(ToUpperHandler)
}

lambda.Start(ToUpperHandler) is where the magic happens, it is a blocking call that will block until the process is killed or an exception is propagated that cannot be handled.

This gives us our first clue that AWS isn’t simply just running your go binary every time a function is invoked, it’s actively listening for requests and passing them to your handler. 

Doing it this way has a few performance benefits, it allows you to set up expensive, thread safe dependencies up front so they are warm if your function is called more than once¹.

Looking underneath

Taking a look at the code under the hood, we can see that it:

• Creates a TCP server listening on a port defined by the environment variable _LAMBDA_SERVER_PORT
• Uses the net/rpc package to handle Ping and Invoke requests from remote clients
• Uses the context package to store and manage state 
• Uses the encoding/json package to perform SerDe for objects you pass to and from your function

The use of net/rpc is really neat in a way, just plain old Go standard library code.

Remote calls

The two methods you can call via RPC are in function.go and they allow remote clients to:

• Perform Ping requests by sending a *messages.PingRequest object, and unsurprisingly this does exactly what it says on the tin. I’m assuming AWS use this to check liveliness of your function and whether it is still reachable wherever they are hosting it.
• Perform Invoke requests by sending a *messages.InvokeRequest object,
  • I’m wondering what the Deadline attribute is for. At one point in the execution path, they use whatever this is set to with the context.WithDeadline function, which leads me to believe this might be the timeout you have configured against your lambda.

These functions will be called by something in AWS that is managing the lifetime of an invocation.

Diagram

To help visualise this, I drew this crude, overly simplified diagram that describes the above interactions. Obviously the AWS Lambda service has a lot of components and infrastructure that we are not privy to, but I think conceptually it’s mostly right.

Testing locally

As the lambda is just listening on a port over TCP, it’s pretty simple to test the above behaviour locally.

By forcing the lambda to run on a known port²

_LAMBDA_SERVER_PORT=8001 go run lambda.go

And then writing a client to submit a InvokeRequest to it, you can successfully execute the function end-to-end, which might be useful for integration testing or whatever.

$ go run client.go 8001 "\"daniel\""
"DANIEL"

I’ve created a small library imaginatively called go-lambda-invoke that wraps up this logic, meaning you can just make the following call in your code

response, err := golambdainvoke.Run(8001, "daniel")

It probably has limited uses, in most cases just writing plain old unit tests for your logic should be sufficient, rather than testing the scaffolding AWS erects around it. However I could see it being useful if you want to test that you’ve built a valid linux binary and perform some pre-deploy sanity tests or something.

Conclusion

Speculating over what AWS are doing under the hood with Lambda is a pastime that has circled the internet ever since it launched, and this post doesn’t really reveal much to answer that question. Is it containers? Who knows. Probably.

But I think looking at how the Go programming model works, a plain old TCP server that allows RPC clients to connect to it, gives us at least a small glimpse into the interactions AWS are performing with your application.