XSS Woes

A predominant PHP developer (whose name I didn't get permission to drop, so I won't, but many of you know who I mean) has been doing a bunch of research related to Cross Site Scripting (XSS), lately. It's really opened opened my eyes to how much I take user input for granted.

Don't get me wrong. I write by the "never trust users" mantra. The issue, in this case, is something abusable that completely slipped under my radar.

Most developers worth their paycheque, I'm sure, know the common rules of "never trust the user", such as "escape all user-supplied data on output," "always validate user input," and "don't rely on something not in your control to do so (ie. Javascript cannot be trusted)." "Don't output unescaped input" goes without saying, in most cases. Only a fool would "echo $_GET['param'];" (and we're all foolish sometimes, aren't we?).

The problem that was demonstrated to me exploited something I considered to be safe. The filename portion of request URI. Now I know just how wrong I was.

Consider this: you build a simple script; let's call it simple.php but that doesn't really matter. simple.php looks something like this:

<html>
 <body>
  <?php
  if (isset($_REQUEST['submitted']) && $_REQUEST['submitted'] == '1') {
    echo "Form submitted!";
  }
  ?>
  <form action="<?php echo $_SERVER['PHP_SELF']; ?>">
   <input type="hidden" name="submitted" value="1" />
   <input type="submit" value="Submit!" />
  </form>
 </body>
</html>

Alright. Let's put this script at: http://example.com/tests/simple.php. On a properly-configured web server, you would expect the script to always render to this, on request:

<html>
 <body>
  <form action="/tests/simple.php">
   <input type="hidden" name="submitted" value="1" />
   <input type="submit" value="Submit!" />
  </form>
 </body>
</html>

Right? No.

What I forgot about, as I suspect some of you have, too (or maybe I'm the only loser who didn't think of this (-; ), is that $_SERVER['PHP_SELF'] can be manipulated by the user.

How's that? If I put a script at /simple/test.php, $_SERVER['PHP_SELF'] should always be "/simple/test.php", right?

Wrong, again.

See, there's a feature of Apache (I think it's Apache, anyway) that you may have used for things like short URLs, or to optimize your query-string-heavy website to make it search-engine friendly. $_SERVER['PATH_INFO']-based URLs.

Quickly, this is when scripts are able to receive data in the GET string, but before the question mark that separates the file name from the parameters. In a URL like http://www.example.com/download.php/path/to/file, download.php would be

executed, and /path/to/file would (usually, depending on config) be available to the script via $_SERVER['PATH_INFO'].

The quirk is that $_SERVER['PHP_SELF'] contains this extra data, opening up the door to potential attack. Even something as simple the code above is vulnerable to such exploits.

Let's look at our simple.php script, again, but requested in a slightly different manner: http://example.com/tests/simple.php/extra_data_here

It would still "work"--the output, in this case, would be:

<html>
 <body>
  <form action="/tests/simple.php/extra_data_here">
   <input type="hidden" name="submitted" value="1" />
   <input type="submit" value="Submit!" />
  </form>
 </body>
</html>

I hope that the problem is now obvious. Consider: http://example.com/tests/simple.php/%22%3E%3Cscript%3Ealert('xss')%3C/script%3E%3Cfoo

The output suddenly becomes very alarming:

<html>
 <body>
  <form action="/tests/simple.php/"><script>alert('xss')</script><foo">
   <input type="hidden" name="submitted" value="1" />
   <input type="submit" value="Submit!" />
  </form>
 </body>
</html>

If you ignore the obviously-incorrect <foo"> tag, you'll see what's happening. The would-be attacker has successfully exploited a critical (if you consider XSS critical) flaw in your logic, and, by getting a user to click the link (even through a redirect script), he has executed the Javascript of his choice on your user's client (obviously, this requires the user to have Javascript enabled). My alert() example is non-malicious, but it's trivial to write similarly-invoked Javascript that changes the action of a form, or usurps cookies (and submits them in a hidden iframe, or through an image tag's URL, to a server that records this personal data).

The solution should also be obvious. Convert the user-supplied data to entities. The code becomes:

<html>
 <body>
  <?php
  if (isset($_REQUEST['submitted']) && $_REQUEST['submitted'] == '1') {
    echo "Form submitted!";
  }
  ?>
  <form action="<?php echo htmlentities($_SERVER['PHP_SELF']); ?>">
   <input type="hidden" name="submitted" value="1" />
   <input type="submit" value="Submit!" />
  </form>
 </body>
</html>

And an attack, as above, would be rendered:

<html>
 <body>
  <form action="/tests/simple.php/&amp;quot;&amp;gt;&amp;lt;script&amp;gt;alert('xss')&amp;lt;/script&amp;gt;&amp;lt;foo">
   <input type="hidden" name="submitted" value="1" />
   <input type="submit" value="Submit!" />
  </form>
 </body>
</html>

This still violates the assumption that the script name and path are the only data in $_SERVER['PHP_SELF'], but the payload has been neutralized.

Needless to say, I felt silly for not thinking of such a simple exploit, earlier. As the aforementioned PHP developer said, at the time (to paraphrase): if guys who consider themselves experts in PHP development don't notice these things, there's little hope for the unwashed masses who have just written their first 'echo "hello world!\n";'. He's working on a generic user-input filtering mechanism that can be applied globally to all user input. Hopefully we'll see it in PECL, soon. Don't forget about the other data in $_SERVER, either..

... ...

Upon experimenting with this exploit on my own server (and watching the raw data in my _SUPERGLOBALS, conveniently, via phpinfo()), I noticed something very interesting that reminded me that even though trusting this data was a stupid mistake on my part, I'm not the only one to do so. A fun (and by fun, I mean nauseating) little game to play: create a file called "info.php" (or whatever name you like). In it, place only "<php phpinfo(); ?>". Now request it like this: http://your-server/path/to/info.php/%22%3E%3Cimg%20src=http://www.perl.com/images/75-logo.jpg%3E%3Cblah

Nice huh? A little less nauseating: it's fixed in CVS.

Fun with the tokenizer...

I was reminded, this past week, of how cool the tokenizer is.

One of the guys who works in the same office as I do had what seemed to be a simple problem: he had a php file that contained ~50 functions, and wanted to summarize the API without parsing through the file, manually, and cutting out the function declarations.

We introduced him to in-line phpdoc blocks (he works (as a Jr.-level PHP developer) in the same office, but for a different company, so he doesn't have to follow our coding standards, but I digress..), but the 50-function library in question didn't have docblocks.

Sure, he could (and did) pull up a list function NAMES with get_defined_functions (I assume by using array_diff against a before-and-after capture), but this didn't give him the argument names, or even the number of arguments for a given function, so I broke out some old tokenizer code I'd written.

In case you aren't familiar with the tokenizer, the PHP manual defines it as:

“[an interface to let you write] your own PHP source analyzing or modification tools without having to deal with the language specification at the lexical level.”

The extension (which has been part of the PHP core distribution since 4.3.0) consists only of two functions: token_get_all and token_name, and a boatload of constants.

Enough babble, though, let's get to the meat. I pulled out this code I'd written for PEARClops (on EFNet #PEAR) that parses PHP source files and figures out what classes, functions/methods and associated parameters are included.

<?php

function get_protos($in)
{
  if (is_file(realpath($in)))
  {
    $in = file_get_contents($in);
  }
  $tokens = token_get_all($in);
  $funcs = array();
  $currClass = '';
  $classDepth = 0;

  for ($i=0; $i<count($tokens); $i++)
  {
    if (is_array($tokens[$i]) && $tokens[$i][0] == T_CLASS)
    {
      ++$i; // whitespace;
      $currClass = $tokens[++$i][1];
      while ($tokens[++$i] != '{') {}
      ++$i;
      $classDepth = 1;
      continue;
    }
    elseif (is_array($tokens[$i]) && $tokens[$i][0] == T_FUNCTION)
    {
      $nextByRef = FALSE;
      $thisFunc = array();
      
      while ($tokens[++$i] != ')')
      {
        if (is_array($tokens[$i]) && $tokens[$i][0] != T_WHITESPACE)
        {
          if (!$thisFunc)
          {
            $thisFunc = array(
              'name'  => $tokens[$i][1],
              'class' => $currClass,
            );
          }
          else
          {
            $thisFunc['params'][] = array(
              'byRef'   => $nextByRef,
              'name'    => $tokens[$i][1],
            );
            $nextByRef = FALSE;
          }
        }
        elseif ($tokens[$i] == '&')
        {
          $nextByRef = TRUE;
        }
        elseif ($tokens[$i] == '=')
        {
          while (!in_array($tokens[++$i], array(')',',')))
          {
            if ($tokens[$i][0] != T_WHITESPACE)
            {
              break;
            }
          }
          $thisFunc['params'][count($thisFunc['params']) - 1]['default'] = $tokens[$i][1];
        }
      }
      $funcs[] = $thisFunc;
    }
    elseif ($tokens[$i] == '{')
    {
      ++$classDepth;
    }
    elseif ($tokens[$i] == '}')
    {
      --$classDepth;
    }

    if ($classDepth == 0)
    {
      $currClass = '';
    }
  }

  return $funcs;
}

function parse_protos($funcs)
{  
  $protos = array();
  foreach ($funcs AS $funcData)
  {
    $proto = '';
    if ($funcData['class'])
    {
      $proto .= $funcData['class'];
      $proto .= '::';
    }
    $proto .= $funcData['name'];
    $proto .= '(';
    if ($funcData['params'])
    {
      $isFirst = TRUE;
      foreach ($funcData['params'] AS $param)
      {
        if ($isFirst)
        {
          $isFirst = FALSE;
        }
        else
        {
          $proto .= ', ';
        }

        if ($param['byRef'])
        {
          $proto .= '&';
        }
        $proto .= $param['name'];
      }
    }
    $proto .= ")";
    $protos[] = $proto;
  }
  return $protos;
}

echo "Functions in {$_SERVER['argv'][1]}:\n";
foreach (parse_protos(get_protos($_SERVER['argv'][1])) AS $proto)
{
  echo "  $proto\n";
}
?>

Save it as "parse_funcs.php" (or whatever you like) and call it like so: php parse_funcs.php /path/to/php_file

For instance:

sean@iconoclast:~/php/scripts$ php token_funcs_cli.php ~/php/cvs/Mail_Mime/mime.php
Functions in /home/sean/php/cvs/Mail_Mime/mime.php:
  Mail_mime::Mail_mime($crlf)
  Mail_mime::__wakeup()
  Mail_mime::setTXTBody($data, $isfile, $append)
  Mail_mime::setHTMLBody($data, $isfile)
  Mail_mime::addHTMLImage($file, $c_type, $name, $isfilename)
  Mail_mime::addAttachment($file, $c_type, $name, $isfilename, $encoding)
  Mail_mime::_file2str(&$file_name)
  Mail_mime::_addTextPart(&$obj, $text)
  Mail_mime::_addHtmlPart(&$obj)
  Mail_mime::_addMixedPart()
  Mail_mime::_addAlternativePart(&$obj)
  Mail_mime::_addRelatedPart(&$obj)
  Mail_mime::_addHtmlImagePart(&$obj, $value)
  Mail_mime::_addAttachmentPart(&$obj, $value)
  Mail_mime::get(&$build_params)
  Mail_mime::headers(&$xtra_headers)
  Mail_mime::txtHeaders($xtra_headers)
  Mail_mime::setSubject($subject)
  Mail_mime::setFrom($email)
  Mail_mime::addCc($email)
  Mail_mime::addBcc($email)
  Mail_mime::_encodeHeaders($input)
  Mail_mime::_setEOL($eol)

Not bad, huh?

There are some not-so-obvious bugs (inheritance, mostly), but for a relatively short script, it does a pretty good job.

This post doesn’t exactly fit the normal theme of my blog, but over the past few weeks, several people have asked me about this, so I thought it was worth jotting down a few notes.

In January 2021, after eyeballing the specs and possibilities for the past few months, I splurged and ordered the Anova Precision Oven. I’ve owned it for over a year, now, and I use it a lot. But I wish it was quite a bit better.

There were a few main features of the APO that had me interested.

First, we have a really nice Wolf stove that came with our house. The range hood is wonderful, and the burners are great. The oven is also good when we actually need it (and we do still need it, sometimes; see below), but it’s propane, so there are a few drawbacks: it takes a while to heat up because there’s a smart safety feature that’s basically a glow plug that won’t let gas flow until it’s built up enough heat to ignite the gas, preventing a situation where the oven has an ideal gas-air mix and is ready to explode. It’s also big. And it uses propane (which I love for the burners, but is unnecessary (mostly) for the oven, and not only is it relatively expensive to run (we have a good price on electricity in Quebec because of past investments in giant hydro-electric projects), it measurably reduces the air quality in the house if the hood fan isn’t running (and running the fan in the dead of winter or summer cools/heats the house in opposition to our preference).

The second feature that had me really interested in the APO is the steam. I’ve tried and mostly-failed many times to get my big oven (this gas one and my previous electric oven) to act like a steam oven. Despite trying the tricks like a pan of water to act as a hydration reservoir, and spraying the walls with a mist of water, it never really steamed like I’d hoped—especially when making baguette.

I’m happy to say that the APO meets both of these needs very well: it’s pretty quick to heat up—mostly because it’s smaller; I do think it’s under-powered (see below)—and the steam works great.

There are, however, a bunch of things wrong with the APO.

The first thing I noticed, after unpacking it and setting it up the first time, is that it doesn’t fit a half sheet pan. It almost fits. I’m sure there was a design or logistics restriction (like maybe these things fit significantly more on a pallet or container when shipping), but sheet pans come in standard sizes, and it’s a real bummer that I not only can’t use the pans (and silicone mats) I already owned, but finding the right sized pan for the APO is also difficult (I bought some quarter and eighth sheet pans, but they don’t fill up the space very well).

Speaking of the pan: the oven comes with one. That one, however, was unusable. It’s made in such a way that it warps when it gets hot. Not just a little bit—a LOT. So much that if there happens to be liquid on the pan, it will launch that liquid off of the pan and onto the walls of the oven when the pan deforms abruptly. Even solids are problematic on the stock pan. I noticed other people complaining online about this and that they had Anova Support send them a new pan. I tried this. Support was great, but the pan they sent is unusable in a different way: they “solved” the warping problem by adding rigidity to the flat bottom part of the pan by pressing ribs into it. This makes the pan impossible to use for baking anything flat like bread or cookies.

I had to contact Support again a few months later when the water tank (the oven uses this for steam, but also even when steam mode is 0%, to improve the temperature reading by feeding some of the water to the thermometer, in order to read the “wet bulb” temperature). The tank didn’t leak, but the clear plastic cracked in quite a large pattern, threatening to dump several litres of water all over my kitchen at any moment. Support sent me a new tank without asking many questions. Hopefully the new one holds up; it hasn’t cracked yet, after ~3 months.

Let’s talk about the steam for a moment: it’s great. I can get a wonderful texture on my breads by cranking it up, and it’s perfect for reheating foods that are prone to drying out, such as mac & cheese—it’s even ideal to run a small amount of steam for reheating pizza that might be a day or two too old. I rarely use our microwave oven for anything non-liquid (melting butter, reheating soups), and the APO is a great alternative way to reheat leftovers (slower than the microwave, sure, but it doesn’t turn foods into rubber, so it’s worth trading time for texture).

So it’s good for breads? Well, sort of. The steam is great for the crust, definitely. However, it has a couple problems. I mentioned above that it’s under-powered, and what I mean by that is two-fold: it has a maximum temperature of 250°C (482°F), and takes quite a long time to recover from the door opening—like, 10 minutes long. Both of these are detrimental to making an ideal bread. I’d normally bake bread at a much higher temperature—I do 550°F in the big oven, and pizza even hotter (especially in the outdoor pizza oven which easily gets up to >800°F). 482°F is—at least in my casual reasoning—pretty bad for “oven spring”. My baguettes look (and taste) great, but they’re always a bit too flat. The crust forms, but the steam bubbles don’t expand quite fast enough to get the loaf to inflate how I’d like. The recovery time certainly doesn’t help with this, either. I’ve managed to mitigate the slow-reheat problem by stacking a bunch of my cast iron pans in the oven to act as a sort of thermal ballast, and help the oven recover more quickly.

Also on the subject of bread: the oven is great for proofing/rising yeast doughs. Well, mostly great. It does a good job of holding the oven a bit warmer than my sometimes-cold-in-winter kitchen, and even without turning on the steam, it seems to avoid drying out the rising dough. I say “mostly” because one of the oven’s fans turns on whenever the oven is “on”, even at low temperatures. The oven has a pretty strong convection fan which is great, but this one seems to be the fan that cools the electronics. I realize this is necessary when running the oven itself, but it’s pretty annoying for the kitchen to have a fairly-loud fan running for 24-48+ hours while baguette dough is rising at near-ambient temperatures.

The oven has several “modes” where you can turn on different heating elements inside the oven. The main element is the “rear” one, which requires convection, but there’s a lower-power bottom element that’s best for proofing, and a top burner that works acceptably (it’s much less powerful than my big gas oven, for example) for broiling. One huge drawback to the default rear+convection mode, though, is that the oven blows a LOT of bubbling liquid all over the place when it’s operating. This means that it gets really dirty, really quickly (see the back wall in the photo with the warped pan, above). Much faster than my big oven (even when running the convection fan over there). This isn’t the end of the world, but it can be annoying.

The oven has controls on the door, as well as an app that works over WiFi (locally, and even when remote). I normally don’t want my appliances to be in the Internet (see Internet-Optional Things), but the door controls are pretty rough. The speed-up/slow-down algorithm they use when holding the buttons for temperature changes is painful. It always overshoots or goes way too slow. They’ve improved this slightly, with a firmware update, but it’s still rough.

The app is a tiny bit better, but it has all of the problems you might expect from a platform-agnostic mobile app that’s clearly built on a questionable web framework. The UI is rough. It always defaults to the wrong mode for me (I rarely use the sous-vide mode), and doesn’t seem to allow things like realtime temperature changes without adding a “stage” and then telling the oven to go to that stage. It’s also dangerous: you can tell the app to turn the oven on, without any sort of “did one of the kids leave something that’s going to catch fire inside the oven” interlock. I’d much prefer (even as optional configuration) a mode where I’m required to open and close the door within 90 seconds of turning the oven on, or it will turn off, or something like that.

Speaking of firmware… one night last summer, while I was sitting outside doing some work, my partner sent me a message “did you just do something to the oven? it keeps making the sound like it’s just turned on.” I checked the app and sure enough, it just did a firmware update. I told her “it’s probably just restarted after the firmware update.” When I went inside a little while later, I could hear it making the “ready” chime over and over. Every 10-15 seconds or so. I didn’t realize this is what she’d meant. I tried everything to get it to stop, but it was in a reboot loop. We had to unplug it to save our sanity. Again, I looked online to see if others were having this issue, and sure enough, there were thousands of complaints about how everyone’s ovens were doing this same thing. Some people were about to cook dinner, others had been rising bread for cooking that night, but we all had unusable ovens. They’d just reboot over and over, thanks to a botched (and automatic!) firmware update. Anova fixed this by the next morning, but it was a good reminder that software is terrible, and maybe our appliances shouldn’t be on the Internet. (I’ve since put it back online because of the aforementioned door controls and the convenience of the—even substandard—app. I wish we could just use the door better, though.)

So, should you buy it? Well, I don’t know. Truthfully, I’m happy we have this in our house. It’s definitely become our main oven, and it fits well in our kitchen (it’s kind of big, but we had a part of the counter top that turned out perfect for this). It needs its own circuit, really, and is still underpowered at 120V (~1800W). However, I very very often feel like I paid a lot of money to beta test a product for Anova (it was around the same price as I paid for my whole slide-in stove (oven + burners, “range”), at the previous house), and that’s a bummer.

If they announce a Version 2 that fixes the problems, I’d definitely suggest getting that, or even V1 if you need it sooner, and are willing to deal with the drawbacks—I just wish you didn’t have to.

In the previous post, we talked about Python serverless architectures on Amazon Web Services with Zappa.

In addition to the previously-mentioned benefits of being able to concentrate directly on the code of apps we’re building, instead of spending effort on running and maintaining servers, we get a few other new tricks. One good example of this is that we can allow our developers to deploy (Zappa calls this update) to shared dev and QA environments, directly, without having to involve anyone from ops (more on this in another post), nor even do we require a build/CI system to push out these types of builds.

That said, we do use a CI serversystem for this project, but it differs from our traditional setup. In the past, we used Jenkins, but found it a bit too heavy. Our current non-Lambda setup uses Buildbot to do full integration testing (it not only runs our apps’ test suites, but it also spins up EC2 nodes, provisions them with Salt, and makes sure they pass the same health checks that our load balancers use to ensure the nodes should receive user requests).

On this new architecture, we still have a test suite, of course, but there are no nodes to spin up (Lambda handles this for us), no systems to provision (the “nodes” are containers that hold only our app, Amazon’s defaults, and Zappa’s bootstrap), and not even any load balancers to keep healthy (this is API Gateway’s job).

In short, our tests and builds are simpler now, so we went looking for a simpler system. Plus, we didn’t want to have to run one or more servers for CI if we’re not even running any (permanent) servers for production.

So, we found LambCI. It’s not a platform we would normally have chosen—we do quite a bit of JavaScript internally, but we don’t currently run any other Node.js apps. It turns out that the platform doesn’t really matter for this, though.

LambCI (as you might have guessed from the name) also runs on Lambda. It requires no permanent infrastructure, and it was actually a breeze to set up, thanks to its CloudFormation template. It ties into GitHub (via AWS SNS), and handles core duties like checking out the code, runing the suite only when configured to do so, and storing the build’s output in S3. It’s a little bit magical—the good kind of magic.

It’s also very generic. It comes with some basic bootstrapping infrastructure, but otherwise relies primarily on configuration that you store in your Git repository. We store our build script there, too, so it’s easy to maintain. Here’s what our build script (do_ci_build) looks like (I’ve edited it a bit for this post):

#!/bin/bash

# more on this in a future post
export PYTHONDONTWRITEBYTECODE=1

# run our test suite with tox and capture its return value
pip install --user tox && tox
tox_ret=$?

# if tox fails, we're done
if [ $tox_ret -ne 0 ]; then
    echo "Tox didn't exit cleanly."
    exit $tox_ret
fi

echo "Tox exited cleanly."

set -x

# use LAMBCI_BRANCH unless LAMBCI_CHECKOUT_BRANCH is set
# this is because lambci considers a PR against master to be the PR branch
BRANCH=$LAMBCI_BRANCH
if [[ ! -z "$LAMBCI_CHECKOUT_BRANCH" ]]; then
    BRANCH=$LAMBCI_CHECKOUT_BRANCH
fi

# only do the `zappa update` for these branches
case $BRANCH in
    master)
        STAGE=dev
        ;;
    qa)
        STAGE=qa
        ;;
    staging)
        STAGE=staging
        ;;
    production)
        STAGE=production
        ;;
    *)
        echo "Not doing zappa update. (branch is $BRANCH)"
        exit $tox_ret
        ;;
esac

echo "Attempting zappa update. Stage: $STAGE"

# we remove these so they don't end up in the deployment zip
rm -r .tox/ .coverage

# virtualenv is needed for Zappa
pip install --user --upgrade virtualenv

# now build the venv
virtualenv /tmp/venv
. /tmp/venv/bin/activate

# set up our virtual environment from our requirements.txt
/tmp/venv/bin/pip install --upgrade -r requirements.txt --ignore-installed

# we use the IAM profile on this lambda container, but the default region is
# not part of that, so set it explicitly here:
export AWS_DEFAULT_REGION='us-east-1'

# do the zappa update; STAGE is set above and zappa is in the active virtualenv
zappa update $STAGE

# capture this value (and in this version we immediately return it)
zappa_ret=$?
exit $zappa_ret

This script, combined with our .lambci.json configuration file (also stored in the repository, as mentioned, and read by LambCI on checkout) is pretty much all we need:

{
    "cmd": "./do_ci_build",
    "branches": {
        "master": true,
        "qa": true,
        "staging": true,
        "production": true
    },
    "notifications": {
        "sns": {
            "topicArn": "arn:aws:sns:us-east-1:ACCOUNTNUMBER:TOPICNAME"
        }
    }
}

With this setup, our test suite runs automatically on the selected branches (and on pull request branches in GitHub), and if that’s successful, it conditionally does a zappa update (which builds and deploys the code to existing stages).

Oh, and one of the best parts: we only pay for builds when they run. We’re not paying hourly for a CI server to sit around doing nothing on the weekend, overnight, or when it’s otherwise idle.

There are a few limitations (such as a time limit on lambda functions, which means that the test suite + build must run within that time limit), but frankly, those haven’t been a problem yet.

If you need simple builds/CI, it might be exactly what you need.