1. Shortbread

    December 1st. Shortbread season begins today.

    (Here's something a little diffrent for long-time blog readers.)

    Four years ago, in December, I made dozens and dozens of batches of shortbread, slowly tweaking my recipe, batch after batch. By Christmas, that year, I had what I consider (everyone's tastes are different) the perfect ratio of flour/butter/sugar/salt, and exactly the right texture: dense, crisp, substantial, and short—not fluffy and light like whipped/cornstarch shortbread, and not crumbly like Walkers.

    I present for you, friends, that recipe.

    shortbread

    Ingredients

    • 70g granulated white sugar (don't use the fancy kinds)
    • 130g unsalted butter (slightly softened; it's colder than normal room temperature in my kitchen)
    • 200g all purpose white flour (I use unbleached)
    • 4g kosher salt

    Method

    In a stand mixer, with the paddle attachment, cream (mix until smooth) the sugar and butter. I like to throw the salt in here, too. You'll need to scrape down the sides with a spatula, as it mixes (turn the mixer off first, of course).

    When well-creamed, and with the mixer on low speed, add the flour in small batches; a little at a time. Mix until combined so it looks like cheese curds. It's not a huge deal if you over-mix, but the best cookies come when it's a little crumblier than a cohesive dough, in my experience.

    Turn out the near-dough onto a length of waxed paper, and roll into a log that's ~4cm in diameter, pressing the "curds" together, firmly. Wrapped in the wax paper, refrigerate for 30 minutes.

    Preheat your oven to 325°F, with a rack in the middle position.

    Slice the chilled log (with a sharp, non-searated knife) into ~1.5cm thick rounds, and place onto a baking sheet with a silicone mat or parchment paper. (If you refrigerate longer, you'll want to let it soften a little before slicing.)

    Bake until right before the tops/sides brown. In my oven, this takes 22 minutes. Remove from oven and allow to cool on the baking sheet.

    Eat and share!

    Don't try to add vanilla or top them with anything, unless you like inferior shortbread. (-; Avoid the temptation to eat them right away, because they're 100 times better when they've cooled. Pop a couple in the freezer for 5 mins if you're really impatient (and I am).

    Enjoy! Let me know if you make these, and how they turned out.

  2. API Gateway timeout workaround

    The last of the four (see previous posts) big API Gateway limitations is the 30 second integration timeout.

    This means that API Gateway will give up on trying to serve your request to the client after 30 seconds—even though Lambda has a 300 second limit.

    In my opinion, this is a reasonable limit. No one wants to be waiting around for an API for more than 30 seconds. And if something takes longer than 30s to render, it should probably be batched. "Render this for me, and I'll come back for it later. Thanks for the token. I'll use this to retrieve the results."

    In an ideal world, all HTTP requests should definitely be served within 30 seconds. But in practice, that's not always possible. Sometimes realtime requests need to go to a slow third party. Sometimes the client isn't advanced enough to use the batch/token method hinted at above.

    Indeed, 30s often falls below the client's own timeout. We're seeing a practical limitation where clients can often run for 90-600 seconds before timing out.

    Terrible user experience aside, I needed to find a way to serve long-running requests, and I really didn't want to violate our serverless architecure to do so.

    But this 30 second timeout in API gateway is a hard limit. It can't be increased via the regular AWS support request method. In fact, AWS says that it can't be increased at all—which might even be true. (-:

    As I mentioned in an earlier post, I did a lot of driving this summer. Lots of driving led to lots of thinking, and lots of thinking led to a partial solution to this problem.

    What if I could use API Gateway to handle the initial request, but buy an additional 30 seconds, somehow. Or better yet, what if I could buy up to an additional 270 seconds (5 minutes total).

    Simply put, an initial request can kick off an asynchronous job, and if it takes a long time, after nearly 30 seconds, we can return an HTTP 303 (See Other) redirect to another endpoint that checks the status of this job. If the result still isn't available after another (nearly) 30s, redirect again. Repeat until the Lambda function call is killed after the hard-limited 300s, but if we don't get to the hard timeout, and we find the job has finished, we can return that result instead of a 303.

    But I didn't really have a simple way to kick off an asynchronous job. Well, that's not quite true. I did have a way to do that: Zappa's asynchronous task execution. What I didn't have was a way to get the results from these jobs.

    So I wrote one, and Zappa's maintainer, Rich, graciously merged it. And this week, it was released. Here's a post I wrote about it over on the Zappa blog.

    The result:

    $ time curl -L 'https://REDACTED.apigwateway/dev/payload?delay=40'
    {
      "MESSAGE": "It took 40 seconds to generate this."
    }
    
    real    0m52.363s
    user    0m0.020s
    sys 0m0.025s
    

    Here's the code (that uses Flask and Zappa); you'll notice that it also uses a simple backoff algorithm:

    @app.route('/payload')
    def payload():
        delay = request.args.get('delay', 60)
        x = longrunner(delay)
        if request.args.get('noredirect', False):
            return 'Response will be here in ~{}s: <a href="{}">{}</a>'.format(
                delay, url_for('response', response_id=x.response_id), x.response_id)
        else:
            return redirect(url_for('response', response_id=x.response_id))
    
    
    @app.route('/async-response/<response_id>')
    def response(response_id):
        response = get_async_response(response_id)
        if response is None:
            abort(404)
    
        backoff = float(request.args.get('backoff', 1))
    
        if response['status'] == 'complete':
            return jsonify(response['response'])
    
        sleep(backoff)
        return "Not yet ready. Redirecting.", 303, {
            'Content-Type': 'text/plain; charset=utf-8',
            'Location': url_for(
                'response', response_id=response_id,
                backoff=min(backoff*1.5, 28)),
            'X-redirect-reason': "Not yet ready.",
        }
    
    
    @task(capture_response=True)
    def longrunner(delay):
        sleep(float(delay))
        return {'MESSAGE': "It took {} seconds to generate this.".format(delay)}
    

    That's it. Long-running tasks in API Gateway. Another tool for our serverless arsenal.

  3. API Gateway Limitations

    As I've mentioned a couple times in the past, I've been working with Lambda and API Gateway.

    We're using it to host/deploy a big app for a big client, as well as some of the ancillary tooling to support the app (such as testing/builds, scheduling, batch jobs, notifications, authentication services, etc.).

    For the most part, I love it. It's helped evaporate the most boring—and often most difficult—parts of deploying highly-available apps.

    But it's not all sunshine and rainbows. Once the necessary allowances are made for a new architecture (things like: if we have concurrency of 10,000, a runaway process's consequences are amplified, database connection pools are easily exhausted, there's no simple way to use static IP addresses), there are 4 main problems that I've encountered with serving an app on Lambda and API Gateway.

    The first two problems are essentially the same. Both headers and query string parameters are clobbered by API Gateway when it creates an API event object.

    Consider the following Lambda function (note: this does not use Zappa, but functions provisioned by Zappa have the same limitation):

    import json
    
    def respond(err, res=None):
        return {
            'statusCode': '400' if err else '200',
            'body': err.message if err else json.dumps(res),
            'headers': {
                'Content-Type': 'application/json',
            },
        }
    
    def lambda_handler(event, context):
        return respond(None, event.get('queryStringParameters'))
    

    Then if you call your (properly-configured) function via the API Gateway URL such as: https://lambdatest.example.com/test?foo=1&foo=2, you will only get the following queryStringParameters:

    {"foo": "2"}
    

    Similarly, a modified function that dumps the event's headers, when called with duplicate headers, such as with:

    curl 'https://lambdatest.example.com/test' -H'Foo: 1' -H'Foo: 2'
    

    …will result in the second header overwriting the first:

    {
        
        "headers":
            
            "Foo": "2",
            
        
    }
    

    The AWS folks have backed themselves into a bit of a corner, here. It's not trivial to change the way these events work, without affecting the thousands (millions?) of existing API Gateway deployments out there.

    If they could make a change like this, it might make sense to turn queryStringParameters into an array when there would previously have been a clobbering:

    {"foo": ["1", "2"]}
    

    This is a bit more dangerous for headers:

    {
        
        "headers":
            
            "Foo": [
                "1",
                "2"
            ],
            
        
    }
    

    This is not impossible, but it is a BC-breaking change.

    What AWS could do, without breaking BC, is (even optionally, based on the API's Gateway configuration/metadata) supply us with an additional field in the event object: rawQueryString. In our example above, it would be foo=1&foo=2, and it would be up to my app to parse this string into something more useful.

    Again, headers are a bit more difficult, but (optionally, as above), one solution might be to supply a rawHeaders field:

    {
        
        "rawHeaders": [
            "Foo: 1",
            "Foo: 2",
            
        ],
        
    }
    

    We've been lucky so far in that these first two quirks haven't been showstoppers for our apps. I was especially worried about a conflict with access-es, which is effectively a proxy server.

    The next two limitations (API Gateway, Lambda) are more difficult, but I've come up with some workarounds:

  4. Lambda payload size workaround

    Another of the AWS Lambda + API Gateway limitations is in the size of the response body we can return.

    AWS states that the full payload size for API Gateway is 10 MB, and the request body payload size is 6 MB in Lambda.

    In practice, I've found this number to be significantly lower than 6 MB, but perhaps I'm just calculating incorrectly.

    Using a Flask route like this:

    @app.route('/giant')
    def giant():
        payload = "x" * int(request.args.get('size', 1024 * 1024 * 6))
        return payload
    

    …and calling it with curl, I get the following cutoff:

    $ curl -s 'https://REDACTED/dev/giant?size=4718559' | wc -c
     4718559
    $ curl -s 'https://REDACTED/dev/giant?size=4718560'
    {"message": "Internal server error"}
    

    Checking the logs (with zappa tail), I see the non-obvious-unless-you've-come-across-this-before error message:

    body size is too long
    

    Let's just call this limit "4 MB" to be safe.

    So, why does this matter? Well, sometimes—like it or not—APIs need to return more than 4 MB of data. In my opinion, this should usually (but not always) be resolved by requesting smaller results. But sometimes we don't get control over this, or it's just not practical.

    Take Kibana, for example. In the past year, we started using Elasticsearch for logging certain types of structured data. We elected to use the AWS Elasticsearch Service to host this. AWS ES has an interesting authentication method: it requires signed requests, based on AWS IAM credentials. This is super useful for our Lambda-based app because we don't have to rely on DB connection pools, firewalls, VPCs, and much of the other pain that comes with using an RDBMS in a highly-distributed system. Our app can use its inherited IAM profile to sign requests to AWS ES quite easily, but we also wanted to give our developers and certain partners access to our structured logs.

    At first, we had our developers run a local copy of aws-es-kibana, which is a proxy server that uses the developer's own AWS credentials (we distribute user or role credentials to our devs) to sign requests. Running a local proxy is a bit of a pain, though—especially for 3rd parties.

    So, I wrote access-es (which is still in a very early "unstable" state, though we do use it "in production" (but not in user request flows)) to allow our users to access Kibana (and Elasticsearch). access-es runs on lambda and effectively works as a reverse HTTPS proxy that signs requests for session/cookie authenticated users, based on the IAM profile. This was a big win for our no-permanent-servers-managed-by-us architecture.

    But the very first day we used access-es to load some large logs in Kibana, it failed on us.

    It turns out that if you have large documents in Elasticsearch, Kibana loads very large blobs of JSON in order to render the discover stream (and possibly other streams). Larger than "4 MB", I noticed. Our (non-structured) logs filled with body size is too long messages, and I had to make some adjustments to the page size in the discover feed. This bought us some time, but we ran into the payload size limitation far too often, and at the most inopportune moments, such as when trying to rapidly diagnose a production issue.

    The "easy" solution to this problem is to concede that we probably can't use Lambda + API Gateway to serve this kind of app. Maybe we should fire up some EC2 instances, provision them with Salt, manage upgrades, updates, security alerts, autoscalers, load balancers… and all of those things that we know how to do so well, but were really hoping to leave behind with the new "serverless" (no permanent servers managed by us) architecture.

    This summer, I did a lot of driving, and during one of the longest of those driving sessions, I came up with an idea about how to handle this problem of using Lambda to serve documents that are larger than the Lambda maximum response size.

    "What if," I thought, "we could calculate the response, but never actually serve it with Lambda. That would fix it." Turns out it did. The solution—which will probably seem obvious once I state it—is to use Lambda to calculate the response body, store that response body in a bucket in S3 (where we don't have to manage any servers), use Lambda + API Gateway to redirect the client to the new resource on S3.

    Here's how I did it in access-es:

        req = method(
            target_url,
            auth=awsauth,
            params=request.query_string,
            data=request.data,
            headers=headers,
            stream=False
        )
    
        content = req.content
    
        if overflow_bucket is not None and len(content) > overflow_size:
    
            # the response would be bigger than overflow_size, so instead of trying to serve it,
            # we'll put the resulting body on S3, and redirect to a (temporary, signed) URL
            # this is especially useful because API Gateway has a body size limitation, and
            # Kibana serves *huge* blobs of JSON
    
            # UUID filename (same suffix as original request if possible)
            u = urlparse(target_url)
            if '.' in u.path:
                filename = str(uuid4()) + '.' + u.path.split('.')[-1]
            else:
                filename = str(uuid4())
    
            s3 = boto3.resource('s3')
            s3_client = boto3.client(
                's3', config=Config(signature_version='s3v4'))
    
            bucket = s3.Bucket(overflow_bucket)
    
            # actually put it in the bucket. beware that boto is really noisy
            # for this in debug log level
            obj = bucket.put_object(
                Key=filename,
                Body=content,
                ACL='authenticated-read',
                ContentType=req.headers['content-type']
            )
    
            # URL only works for 60 seconds
            url = s3_client.generate_presigned_url(
                'get_object',
                Params={'Bucket': overflow_bucket, 'Key': filename},
                ExpiresIn=60)
    
            # "see other"
            return redirect(url, 303)
    
        else:
            # otherwise, just serve it normally
            return Response(content, content_type=req.headers['content-type'])
    

    If the body size is larger than overflow_size, we store the result on S3, and the client receives a 303 see other with an appropriate Location header, completely bypassing the Lambda body size limitation, and saving the day for our "serverless" architecture.

    The resulting URL is signed by AWS to make it only valid for 60 seconds, and the resource isn't available without such a signature (unless otherwise authenticated with IAM + appropriate permissions). Additionally, we use S3's lifecycle management to automatically delete old objects.

    For clients that are modern browsers, though, you'll need to properly manage the CORS configuration on that S3 bucket.

    This approach fixed our Kibana problem, and now sits in our arsenal of tools for when we need to handle large responses in our other serverless Lambda + API Gateway apps.

  5. Zappa and LambCI

    In the previous post, we talked about Python serverless architectures on Amazon Web Services with Zappa.

    In addition to the previously-mentioned benefits of being able to concentrate directly on the code of apps we're building, instead of spending effort on running and maintaining servers, we get a few other new tricks. One good example of this is that we can allow our developers to deploy (Zappa calls this update) to shared dev and QA environments, directly, without having to involve anyone from ops (more on this in another post), nor even do we require a build/CI system to push out these types of builds.

    That said, we do use a CI serversystem for this project, but it differs from our traditional setup. In the past, we used Jenkins, but found it a bit too heavy. Our current non-Lambda setup uses Buildbot to do full integration testing (it not only runs our apps' test suites, but it also spins up EC2 nodes, provisions them with Salt, and makes sure they pass the same health checks that our load balancers use to ensure the nodes should receive user requests).

    On this new architecture, we still have a test suite, of course, but there are no nodes to spin up (Lambda handles this for us), no systems to provision (the "nodes" are containers that hold only our app, Amazon's defaults, and Zappa's bootstrap), and not even any load balancers to keep healthy (this is API Gateway's job).

    In short, our tests and builds are simpler now, so we went looking for a simpler system. Plus, we didn't want to have to run one or more servers for CI if we're not even running any (permanent) servers for production.

    So, we found LambCI. It's not a platform we would normally have chosen—we do quite a bit of JavaScript internally, but we don't currently run any other Node.js apps. It turns out that the platform doesn't really matter for this, though.

    LambCI (as you might have guessed from the name) also runs on Lambda. It requires no permanent infrastructure, and it was actually a breeze to set up, thanks to its CloudFormation template. It ties into GitHub (via AWS SNS), and handles core duties like checking out the code, runing the suite only when configured to do so, and storing the build's output in S3. It's a little bit magical—the good kind of magic.

    It's also very generic. It comes with some basic bootstrapping infrastructure, but otherwise relies primarily on configuration that you store in your Git repository. We store our build script there, too, so it's easy to maintain. Here's what our build script (do_ci_build) looks like (I've edited it a bit for this post):

    #!/bin/bash
    
    # more on this in a future post
    export PYTHONDONTWRITEBYTECODE=1
    
    # run our test suite with tox and capture its return value
    pip install --user tox && tox
    tox_ret=$?
    
    # if tox fails, we're done
    if [ $tox_ret -ne 0 ]; then
        echo "Tox didn't exit cleanly."
        exit $tox_ret
    fi
    
    echo "Tox exited cleanly."
    
    set -x
    
    # use LAMBCI_BRANCH unless LAMBCI_CHECKOUT_BRANCH is set
    # this is because lambci considers a PR against master to be the PR branch
    BRANCH=$LAMBCI_BRANCH
    if [[ ! -z "$LAMBCI_CHECKOUT_BRANCH" ]]; then
        BRANCH=$LAMBCI_CHECKOUT_BRANCH
    fi
    
    # only do the `zappa update` for these branches
    case $BRANCH in
        master)
            STAGE=dev
            ;;
        qa)
            STAGE=qa
            ;;
        staging)
            STAGE=staging
            ;;
        production)
            STAGE=production
            ;;
        *)
            echo "Not doing zappa update. (branch is $BRANCH)"
            exit $tox_ret
            ;;
    esac
    
    echo "Attempting zappa update. Stage: $STAGE"
    
    # we remove these so they don't end up in the deployment zip
    rm -r .tox/ .coverage
    
    # virtualenv is needed for Zappa
    pip install --user --upgrade virtualenv
    
    # now build the venv
    virtualenv /tmp/venv
    . /tmp/venv/bin/activate
    
    # set up our virtual environment from our requirements.txt
    /tmp/venv/bin/pip install --upgrade -r requirements.txt --ignore-installed
    
    # we use the IAM profile on this lambda container, but the default region is
    # not part of that, so set it explicitly here:
    export AWS_DEFAULT_REGION='us-east-1'
    
    # do the zappa update; STAGE is set above and zappa is in the active virtualenv
    zappa update $STAGE
    
    # capture this value (and in this version we immediately return it)
    zappa_ret=$?
    exit $zappa_ret
    

    This script, combined with our .lambci.json configuration file (also stored in the repository, as mentioned, and read by LambCI on checkout) is pretty much all we need:

    {
        "cmd": "./do_ci_build",
        "branches": {
            "master": true,
            "qa": true,
            "staging": true,
            "production": true
        },
        "notifications": {
            "sns": {
                "topicArn": "arn:aws:sns:us-east-1:ACCOUNTNUMBER:TOPICNAME"
            }
        }
    }
    

    With this setup, our test suite runs automatically on the selected branches (and on pull request branches in GitHub), and if that's successful, it conditionally does a zappa update (which builds and deploys the code to existing stages).

    Oh, and one of the best parts: we only pay for builds when they run. We're not paying hourly for a CI server to sit around doing nothing on the weekend, overnight, or when it's otherwise idle.

    There are a few limitations (such as a time limit on lambda functions, which means that the test suite + build must run within that time limit), but frankly, those haven't been a problem yet.

    If you need simple builds/CI, it might be exactly what you need.

  6. Zappa

    For the past few months, I've been focused primarily on a Python project that we're deploying without any servers.

    Well, of course that's not really true, but we're deploying it without any permanent servers.

    The idea of "serverless" architecture isn't brand new, anymore, but running serverless applications on the AWS infrastructure—which I've become very familiar with over the past few years—is still a pretty new concept.

    AWS Lambda has been around for a few years, now. It's a platform that allows you to run arbitary code, in response to events, and pay only for gigabyte-seconds of RAM time. This means that someone else (Amazon) manages the servers, networking, storage, etc.

    At some point, Lambda gained the ability to run Python code (instead of just JavaScript, C#, and Java). This piqued my interest, but we didn't have a whole lot of use for it in building web apps. Nevertheless, we used it to turn SNS notifications into IRC messages, so our #ops IRC channel would get inline notices that our Autoscalers were autoscaling.

    In early 2015, I tweeted: "…too bad @AWSCloud Lambda can’t listen (and respond) to HTTP(S) events on Elastic Load Balancer…".

    A while later, Amazon introduced API Gateway, which—amid other functionality that we don't use very much—had the ability to turn arbitrary HTTP(S) requests into AWS events. Things got interesting. You'll recall, from above, that Lambda functions can run in response to events.

    Interesting in that we could respond to HTTP events, but it wasn't really possible to use regular tools and frameworks with API Gateway. We're used to building apps in Flask, not monolithic Python functions that do their own HTTP request parsing.

    As time went on, these tools got a little more mature and gained more useful features. I kept thinking back to my tweet where we could just run code, not servers.

    Then, in October—increasingly tired of the grind of otherwise-simple operations work—I went searching a bit harder for something to help with the monolithic lambda function problem, and I stumbled upon Zappa. It seemed to be exactly the kind of thing I was looking for. With a bit of boilerplate, hackery, and near-magic, it turns API Gateway request events into WSGI requests, and Flask (plus other Python tools) speaks WSGI. This looked great.

    Little did I know that right around that same time, there were some new, barely-documented (at the time), changes to API Gateway that would help reduce the magic and hacky parts of the Zappa boilerplate.

    I quickly built my first simple Zappa-based app (it was actually porting a 10-year-old PHP app), and deployed it: paste.website.

    We're using this technology on a very large client project, too. It's exciting that we're going to be able to do it without having to worry about things like software upgrades, underutilized servers, and build nodes that cost us money while we're all sleeping.

    I'm not going to let this turn into yet-another-Zappa-tutorial—there are plenty of those out there—but if you're interested in this kind of thing and hadn't heard of Zappa before now, well… now you have.

    We (mostly Rich) even managed to get it working on the brand-new Python 3.6 target in Lambda.

  7. DST pain

    Tonight, in Montreal (and many other North American cities), we change from Standard Time to Daylight Time.

    I know we programmers complain about date/time math relentlessly, but I thought it was worth sharing this real-life problem that someone asked me about on Reddit this weekend:

    It sounds like this is a serious problem that has effected you on more than one occasion. Story?

    The simplest complicated scenario is: let's say we have a call scheduled between our team on the east coast of North America and a colleague in the UK at 10AM Montreal time.

    Normally Nottingham (UK, same as London time) is 5 hours ahead of Montreal. This is pretty easy. Our British colleague needs to join at 3PM.

    However, tonight, we change from EST to EDT in Montreal (clocks move one hour ahead). But the UK will still be on GMT tomorrow. So, now, the daily 10AM call becomes a 2PM call for the Brits.

    But this is only for the next 2 weeks, because BST starts on March 26th (BST is to GMT as EDT is to EST). Then, we go back to a 5 hour difference. So we can expect Europeans to show up an hour late for everything this week. Or maybe we're just an hour early on this side of the Atlantic.

    To make this more difficult, we often have calls between not only Montreal and England, but also those two plus Korea and Brazil.

    Korea doesn't employ Daylight Saving Time, so a standing 7AM call in Seoul (5PM in Montreal) becomes a 6PM call in Montreal.

    And to even further complicate things, our partners in São Paulo switched FROM DST to standard time on Feb 17. Because they're in the southern hemisphere the clock change is the opposite direction of ours, on a different day.

    So: yes. It has affected our team on many occasions. It's already very difficult to get that many international parties synced up. DST can make it nearly impossible.

  8. Vermont

    I get asked, from time to time, what things I would recommend when visiting Vermont. Here's my list. I'll update it as I learn about new gems.

  9. DNS for VMs

    Previously we talked about using Vagrant at Fictive Kin and how we typically have many Virtual Machines (VMs) on the go at once.

    Addressing each of these VMs with a real hostname was proving to be difficult. We couldn’t just use the IP addresses of the machines because they’re unreasonably hard to remember, and other problems like browser cookies don’t work properly.

    In the past, I’ve managed this by editing my local /etc/hosts file (or the Windows equivalent, whatever that’s called now). Turns out this wasn’t ideal. If my hosts don’t sync up with my colleagues’ hosts, stuff (usually cookies) can go wrong, for example. Plus, I strongly believe in setting up an environment that can be managed remotely (when possible) so less-technical members of our team don’t find themselves toiling under the burden of managing an obscurely-formatted text file deep within the parts of their operating systems that they — in all fairness — shouldn’t touch. Oh, and you also can’t do wildcards there.

    As I mentioned in a previous post, we have the great fortune of having all of our VM users conveniently on one operating system platform (Mac OS X), so this post will also focus there, but a similar strategy to this one could be used on Windows or Linux, without the shiny resolver bits — you’d just have to run all of your host’s DNS traffic through a VM-managed name resolver; and these other operating systems might have something similar to resolver that I simply haven’t been enlightened to, and surely someone will point out my error on Twitter or email (please).

    The short version (which I just hinted at) is that we run a DNS server on our gateway VM (all of our users have one of these), and we instruct the workstation’s operating system to resolve certain TLDs via this VM’s IP address.

    We set up the VM side of this with configuration management, in our Salt sates. Our specific implementation is a little too hacky to share (we have a custom Python script that loads hostname configuration from disk, running within systemd), but I’ve recently been tinkering with Dnsmasq, and we might roll that out in the non-distant future.

    Let’s say you want to manage the .sean TLD. Let’s additionally say that you have an app called saxophone (on a VM at 192.168.222.16) and another called trombone (on 192.168.222.17), and you’d like to address these via URLs like https://saxophone.sean/ and https://trombone.sean/, respectively. Let’s also say that you might want to make sure that http://www.trombone.sean/ redirects to https on trombone.sean (without the www). Finally, let’s say that the saxophone app has many subdomains like blog.saxophone.sean, admin.saxophone.sean, cdn.saxophone.sean, etc. As you can see, we’re now out of one-liner territory in /etc/hosts. (Well, maybe a couple long lines.)

    To configure the DNS-resolving VM (“gateway” for us), with Dnsmasq, the configuration lines would look something like this:

    address=/.saxophone.sean/192.168.222.16
    address=/.trombone.sean/192.168.222.17
    

    You can test with:

    $ dig +short @gateway.sean admin.saxophone.sean
    192.168.222.16
    $ dig +short @gateway.sean www.trombone.sean
    192.168.222.17
    $ dig +short @gateway.sean trombone.sean
    192.168.222.17
    

    Now we’ve got the VM side set up. How do we best instruct the OS to resolve the new (fake) sean TLD “properly”?

    Mac OS X has a mechanism called resolver that allows us to choose specific DNS servers for specific TLDs, which is very convenient.

    Again, the short version of this is that you’d add the following line to /etc/resolver/sean (assuming the gateway is on 192.168.222.2) on your workstation (not the VM):

    nameserver 192.168.222.2
    

    Once complete (and mDNSResponder has been reloaded), your computer will use the specified name server to resolve the .sean TLD.

    The longer version is that I don’t want to burden my VM users (especially those who get nervous touching anything in /etc — and with good reason), with this additional bit of configuration, so we manage this in our Vagrantfile, directly. Here’s an excerpt (we use something other than sean, but this has been altered to be consistent with our examples):

    # set up custom resolver
    if !File.exist? '/etc/resolver/sean'
      puts "Need to add the .sean resolver. We'll need sudo for this."
      puts "This should only happen once."
      print "\a"
      puts `sudo sh -c 'if [ ! -d /etc/resolver ]; then mkdir /etc/resolver; fi; echo "nameserver 192.168.222.2" > /etc/resolver/san; killall -HUP mDNSResponder;'`
    end
    

    Then, when the day comes that we want to add a new app — call it trumpet — we can do all of it through configuration management from the ops side. We create the new VM in Salt, and the next time the user’s gateway is highstated (that is: the configuration management is applied), the Vagrantfile is altered, and the DNS resolver configuration on the gateway VM is changed. Once the user has done vagrant up trumpet, they should be good to point their browsers at https://trumpet.sean/. We don’t (specifically Vagrant doesn’t) even need sudo on the workstation after the initial setup.

  10. SSH: jump servers, MFA, Salt, and advanced configuration

    Let’s take a short break from our discussion of Vagrant to talk about how we use SSH in production at Fictive Kin.

    Recently, I went on a working vacation to visit my family in New Brunswick (think: east of the eastern time zone in Canada). While there, I needed to log in to a few servers to check on a few processes. I’ve done this in past years, and am frequently away from my sort-of-static home IP address. Usually, this required wrangling of AWS EC2 Security Groups to temporarily allow access from my tethered connection (whose IP changes at least a few times a day), but not this time. This time things were different.

    Over the past year or so, we’ve been reworking most of our production architecture. We’ve moved everything into VPC, reworked tests, made pools work within auto scale groups, and generally made things better. And one of the better things we’ve done is set up SSH to work through a jump host.

    This is certainly not a new idea. I’ve used hosts like this for many years. Even the way we’ve set it up is far from groundbreaking, but I thought it was worth sharing, since I've had people ask me about it, and it’s much more secure.

    The short version is that we’ve set up a SSH “jump” host to allow global SSH access on a non-standard port, and that host — in turn — allows us to access our AWS servers, including QA and production if access has been granted. There is no direct SSH access to any servers except the jump host(s), and they are set up to require multi-factor authentication (“MFA”) with Google’s Authenticator PAM module.

    This is more secure because nearly none of our servers are listening on the public Internet for SSH connections, and our jump host(s) listens on a non-standard port. This helps prevent compromise from non-targetted attacks such as worms, script kiddies, IBR. Additionally the server is configured with a minimal set of services, contains no secrets, requires public keys (no passwords) to log in, has a limited set of accounts, harshly rate-limits failed connections, and has the aforementioned MFA module set up, which we require our jump host users to set up.

    In practice, this is pretty easy to set up and use, both from the server side and for our users.

    From a user’s standpoint, we provision the account, including their public key, through configuration management (we use Salt). They then need to SSH directly to the jump host one time to configure google-authenticator, which asks a few questions, generates a TOTP seed/key, and gives the user a QR code (or seed codes) that they can scan into their MFA app of choice. We have users on the Google Authenticator app (both Android and iOS), as well as 1Password (which we acknowledge is not actually MFA, but it’s still better than single-factor).

    Then, when they want to connect to a server in AWS, they connect via ssh — using their SSH private key — through the jump host (which asks for their current rotating TOTP/MFA code), and if successful allows them to proxy off to their desired server (which also requires their private key, but this is usually transparent to users).

    To illustrate, let’s say a user (sean) wants to connect to their app’s QA server (exampleappqa01.internal.example.net, which is in a VPC that has a CIDR of 10.77/16, or has IP addresses in the 10.77.* range). If they have their SSH configuration file set up properly, they can issue a command that looks like it’s connecting directly:

    ~$ ssh exampleappqa01.internal.example.net
    Authenticated with partial success.
    Verification code: XXXXXX
    
    sean@exampleappqa01:~$
    

    This magic is possible through SSH’s ProxyCommand configuration directive. Here’s a sample configuration for internal.example.net:

    # jump host ; used for connecting directly to the jump host
    Host jumphost01.public.example.net
      ForwardAgent yes
      Port 11122  # non-standard port
    
    # for hosts such as test.internal.example.net, through jumphost01
    Host *.internal.example.net
      ForwardAgent yes
      ProxyCommand nohup ssh -p 11122 %r@jumphost01.public.example.net nc -w1 %h %p
    
    # internal IP addresses for internal.example.net
    Host 10.77.*
      ForwardAgent yes
      ProxyCommand nohup ssh -p 11122 %r@jumphost01.public.example.net nc -w1 %h %p
    

    SSH transparently connects (via ssh) to the non-standard port (11122) on jumphost01.public.example.net and invokes nc (netcat — look it up if you’re unfamiliar, and you’re welcome! (-: ) to proxy the connection’s stream over to the actual host (%h) specified on the command line.

    Hope that all made sense. Please hit me up on Twitter (or email) if not.

    Here are a couple bonus scenes for reading this far. Our Salt state for installing Google Authenticator’s PAM module looks like this, on Debian:

    include:
        - apt  # for backports
        - sshd-mfa.openssh  # for an updated version of sshd
    
    libqrencode3:
        pkg.installed
    
    libpam-google-authenticator:
        pkg.installed:
            # from http://ftp.us.debian.org/debian/pool/main/g/google-authenticator/libpam-google-authenticator_20130529-2_amd64.deb
            - name: libpam-google-authenticator
            - require:
                - pkg: libqrencode3
    
    # see: http://delyan.me/securing-ssh-with-totp/
    # nullok means that users without a ~/.google_authenticator will be
    # allowed in without MFA; it's opt-in
    # additionally, the user needs to log in to run `google-authenticator`
    # before they'd have a configured MFA app/token anyway
    /etc/pam.d/sshd:
        file.replace:
            - pattern: '^@include common-auth$'
            - repl: |
                auth [success=done new_authtok_reqd=done default=die] pam_google_authenticator.so nullok
                @include common-auth # modified
            - require:
                - pkg: libpam-google-authenticator
            - watch_in:
                - service: openssh6.7
    
    /etc/ssh/sshd_config:
        file.replace:
            - pattern: 'ChallengeResponseAuthentication no'
            - repl: |
                ChallengeResponseAuthentication yes
                AuthenticationMethods publickey,keyboard-interactive:pam
            - append_if_not_found: True
            - watch_in:
                - service: openssh6.7
    

    Finally, on this topic: I’ve been playing with assh to help manage my ssh config file, and it’s been working out pretty well. I suggest you give it a look.