Skip to main content

Zappa and LambCI

In the previous post, we talked about Python serverless architectures on Amazon Web Services with Zappa.

In addition to the previously-mentioned benefits of being able to concentrate directly on the code of apps we're building, instead of spending effort on running and maintaining servers, we get a few other new tricks. One good example of this is that we can allow our developers to deploy (Zappa calls this update) to shared dev and QA environments, directly, without having to involve anyone from ops (more on this in another post), nor even do we require a build/CI system to push out these types of builds.

That said, we do use a CI serversystem for this project, but it differs from our traditional setup. In the past, we used Jenkins, but found it a bit too heavy. Our current non-Lambda setup uses Buildbot to do full integration testing (it not only runs our apps' test suites, but it also spins up EC2 nodes, provisions them with Salt, and makes sure they pass the same health checks that our load balancers use to ensure the nodes should receive user requests).

On this new architecture, we still have a test suite, of course, but there are no nodes to spin up (Lambda handles this for us), no systems to provision (the "nodes" are containers that hold only our app, Amazon's defaults, and Zappa's bootstrap), and not even any load balancers to keep healthy (this is API Gateway's job).

In short, our tests and builds are simpler now, so we went looking for a simpler system. Plus, we didn't want to have to run one or more servers for CI if we're not even running any (permanent) servers for production.

So, we found LambCI. It's not a platform we would normally have chosen—we do quite a bit of JavaScript internally, but we don't currently run any other Node.js apps. It turns out that the platform doesn't really matter for this, though.

LambCI (as you might have guessed from the name) also runs on Lambda. It requires no permanent infrastructure, and it was actually a breeze to set up, thanks to its CloudFormation template. It ties into GitHub (via AWS SNS), and handles core duties like checking out the code, runing the suite only when configured to do so, and storing the build's output in S3. It's a little bit magical—the good kind of magic.

It's also very generic. It comes with some basic bootstrapping infrastructure, but otherwise relies primarily on configuration that you store in your Git repository. We store our build script there, too, so it's easy to maintain. Here's what our build script (do_ci_build) looks like (I've edited it a bit for this post):

#!/bin/bash

# more on this in a future post
export PYTHONDONTWRITEBYTECODE=1

# run our test suite with tox and capture its return value
pip install --user tox && tox
tox_ret=$?

# if tox fails, we're done
if [ $tox_ret -ne 0 ]; then
    echo "Tox didn't exit cleanly."
    exit $tox_ret
fi

echo "Tox exited cleanly."

set -x

# use LAMBCI_BRANCH unless LAMBCI_CHECKOUT_BRANCH is set
# this is because lambci considers a PR against master to be the PR branch
BRANCH=$LAMBCI_BRANCH
if [[ ! -z "$LAMBCI_CHECKOUT_BRANCH" ]]; then
    BRANCH=$LAMBCI_CHECKOUT_BRANCH
fi

# only do the `zappa update` for these branches
case $BRANCH in
    master)
        STAGE=dev
        ;;
    qa)
        STAGE=qa
        ;;
    staging)
        STAGE=staging
        ;;
    production)
        STAGE=production
        ;;
    *)
        echo "Not doing zappa update. (branch is $BRANCH)"
        exit $tox_ret
        ;;
esac

echo "Attempting zappa update. Stage: $STAGE"

# we remove these so they don't end up in the deployment zip
rm -r .tox/ .coverage

# virtualenv is needed for Zappa
pip install --user --upgrade virtualenv

# now build the venv
virtualenv /tmp/venv
. /tmp/venv/bin/activate

# set up our virtual environment from our requirements.txt
/tmp/venv/bin/pip install --upgrade -r requirements.txt --ignore-installed

# we use the IAM profile on this lambda container, but the default region is
# not part of that, so set it explicitly here:
export AWS_DEFAULT_REGION='us-east-1'

# do the zappa update; STAGE is set above and zappa is in the active virtualenv
zappa update $STAGE

# capture this value (and in this version we immediately return it)
zappa_ret=$?
exit $zappa_ret

This script, combined with our .lambci.json configuration file (also stored in the repository, as mentioned, and read by LambCI on checkout) is pretty much all we need:

{
    "cmd": "./do_ci_build",
    "branches": {
        "master": true,
        "qa": true,
        "staging": true,
        "production": true
    },
    "notifications": {
        "sns": {
            "topicArn": "arn:aws:sns:us-east-1:ACCOUNTNUMBER:TOPICNAME"
        }
    }
}

With this setup, our test suite runs automatically on the selected branches (and on pull request branches in GitHub), and if that's successful, it conditionally does a zappa update (which builds and deploys the code to existing stages).

Oh, and one of the best parts: we only pay for builds when they run. We're not paying hourly for a CI server to sit around doing nothing on the weekend, overnight, or when it's otherwise idle.

There are a few limitations (such as a time limit on lambda functions, which means that the test suite + build must run within that time limit), but frankly, those haven't been a problem yet.

If you need simple builds/CI, it might be exactly what you need.

Zappa

For the past few months, I've been focused primarily on a Python project that we're deploying without any servers.

Well, of course that's not really true, but we're deploying it without any permanent servers.

The idea of "serverless" architecture isn't brand new, anymore, but running serverless applications on the AWS infrastructure—which I've become very familiar with over the past few years—is still a pretty new concept.

AWS Lambda has been around for a few years, now. It's a platform that allows you to run arbitary code, in response to events, and pay only for gigabyte-seconds of RAM time. This means that someone else (Amazon) manages the servers, networking, storage, etc.

At some point, Lambda gained the ability to run Python code (instead of just JavaScript, C#, and Java). This piqued my interest, but we didn't have a whole lot of use for it in building web apps. Nevertheless, we used it to turn SNS notifications into IRC messages, so our #ops IRC channel would get inline notices that our Autoscalers were autoscaling.

In early 2015, I tweeted: "…too bad @AWSCloud Lambda can’t listen (and respond) to HTTP(S) events on Elastic Load Balancer…".

A while later, Amazon introduced API Gateway, which—amid other functionality that we don't use very much—had the ability to turn arbitrary HTTP(S) requests into AWS events. Things got interesting. You'll recall, from above, that Lambda functions can run in response to events.

Interesting in that we could respond to HTTP events, but it wasn't really possible to use regular tools and frameworks with API Gateway. We're used to building apps in Flask, not monolithic Python functions that do their own HTTP request parsing.

As time went on, these tools got a little more mature and gained more useful features. I kept thinking back to my tweet where we could just run code, not servers.

Then, in October—increasingly tired of the grind of otherwise-simple operations work—I went searching a bit harder for something to help with the monolithic lambda function problem, and I stumbled upon Zappa. It seemed to be exactly the kind of thing I was looking for. With a bit of boilerplate, hackery, and near-magic, it turns API Gateway request events into WSGI requests, and Flask (plus other Python tools) speaks WSGI. This looked great.

Little did I know that right around that same time, there were some new, barely-documented (at the time), changes to API Gateway that would help reduce the magic and hacky parts of the Zappa boilerplate.

I quickly built my first simple Zappa-based app (it was actually porting a 10-year-old PHP app), and deployed it: paste.website.

We're using this technology on a very large client project, too. It's exciting that we're going to be able to do it without having to worry about things like software upgrades, underutilized servers, and build nodes that cost us money while we're all sleeping.

I'm not going to let this turn into yet-another-Zappa-tutorial—there are plenty of those out there—but if you're interested in this kind of thing and hadn't heard of Zappa before now, well… now you have.

We (mostly Rich) even managed to get it working on the brand-new Python 3.6 target in Lambda.

DST pain

Tonight, in Montreal (and many other North American cities), we change from Standard Time to Daylight Time.

I know we programmers complain about date/time math relentlessly, but I thought it was worth sharing this real-life problem that someone asked me about on Reddit this weekend:

It sounds like this is a serious problem that has effected you on more than one occasion. Story?

The simplest complicated scenario is: let's say we have a call scheduled between our team on the east coast of North America and a colleague in the UK at 10AM Montreal time.

Normally Nottingham (UK, same as London time) is 5 hours ahead of Montreal. This is pretty easy. Our British colleague needs to join at 3PM.

However, tonight, we change from EST to EDT in Montreal (clocks move one hour ahead). But the UK will still be on GMT tomorrow. So, now, the daily 10AM call becomes a 2PM call for the Brits.

But this is only for the next 2 weeks, because BST starts on March 26th (BST is to GMT as EDT is to EST). Then, we go back to a 5 hour difference. So we can expect Europeans to show up an hour late for everything this week. Or maybe we're just an hour early on this side of the Atlantic.

To make this more difficult, we often have calls between not only Montreal and England, but also those two plus Korea and Brazil.

Korea doesn't employ Daylight Saving Time, so a standing 7AM call in Seoul (5PM in Montreal) becomes a 6PM call in Montreal.

And to even further complicate things, our partners in São Paulo switched FROM DST to standard time on Feb 17. Because they're in the southern hemisphere the clock change is the opposite direction of ours, on a different day.

So: yes. It has affected our team on many occasions. It's already very difficult to get that many international parties synced up. DST can make it nearly impossible.