HTTP 1.0 and the Connection header

I have a long backlog of things to write about. One of those things is Varnish (more on that in a future post). So, over these Christmas holidays, while intentionally taking a break from real work, I decided to finally do some of the research required before I can really write about how Varnish is going to make your web apps much faster.

To get some actual numbers, I broke out the Apache Benchmarking utility (ab), and decided to let it loose on my site (100 requests, 10 requests concurrently):

ab -n 100 -c 10 http://seancoates.com/codes

To my surprise, this didn't finish almost immediately. The command ran for what seemed like forever. Finally, I was presented with its output (excerpted for your reading pleasure):

Concurrency Level:      10
Time taken for tests:   152.476 seconds
Complete requests:      100
Failed requests:        0
Write errors:           0
Total transferred:      592500 bytes
HTML transferred:       566900 bytes
Requests per second:    0.66 [#/sec] (mean)
Time per request:       15247.644 [ms] (mean)
Time per request:       1524.764 [ms] (mean, across all concurrent requests)
Transfer rate:          3.79 [Kbytes/sec] received

Less than one request per second? That surely doesn't seem right. I chose /codes because the content does not depend on any sort of external service or expensive server-side processing (as described in an earlier post). Manually browsing to this same URL also feels much faster than one request per second. There's something fishy going on here.

I thought that there might be something off with my server configuration, so in order to rule out a concurrency issue, I decided to benchmark a single request:

ab -n 1 -c 1 http://seancoates.com/codes

I expected this page to load in less than 200ms. That seems reasonable for a light page that has no external dependencies, and doesn't even hit a database. Instead, I got this:

Concurrency Level:      1
Time taken for tests:   15.090 seconds
Complete requests:      1
Failed requests:        0
Write errors:           0
Total transferred:      5925 bytes
HTML transferred:       5669 bytes
Requests per second:    0.07 [#/sec] (mean)
Time per request:       15089.559 [ms] (mean)
Time per request:       15089.559 [ms] (mean, across all concurrent requests)
Transfer rate:          0.38 [Kbytes/sec] received

Over 15 seconds to render a single page‽ Clearly, this isn't what's actually happening on my site. I can confirm this with a browser, or very objectively with time and curl:

$ time curl -s http://seancoates.com/codes > /dev/null

real  0m0.122s
user  0m0.000s
sys   0m0.010s

The next step is to figure out what ab is actually doing that's taking an extra ~15 seconds. Let's crank up the verbosity (might as well go all the way to 11).

$ ab -v 11 -n 1 -c 1 http://seancoates.com/codes
(snip)
Benchmarking seancoates.com (be patient)...INFO: POST header == 
---
GET /codes HTTP/1.0
Host: seancoates.com
User-Agent: ApacheBench/2.3
Accept: */*


---
LOG: header received:
HTTP/1.1 200 OK
Date: Mon, 26 Dec 2011 16:27:32 GMT
Server: Apache/2.2.17 (Ubuntu) DAV/2 SVN/1.6.12 mod_fcgid/2.3.6 mod_ssl/2.2.17 OpenSSL/0.9.8o PHP/5.3.2
X-Powered-By: PHP/5.3.2
Vary: Accept-Encoding
Content-Length: 5669
Content-Type: text/html

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
(HTML snipped from here)

LOG: Response code = 200
..done

(snip)

This all looked just fine. The really strange thing is that the output stalled right after LOG: Response code = 200 and right before ..done. So, something was causing ab to stall after the request was answered (we got a 200, and it's a small number of bytes).

This is the part where I remembered that I've seen a similar behaviour before. I've lost countless hours of my life (and now one more) to this problem: some clients (such as PHP's streams) don't handle Keep-Alives in the way that one might expect.

HTTP is hard. Really hard. Way harder than you think. Actually, it's not that hard if you remember that what you think is probably wrong if you're not absolutely sure that you're right.

ab or httpd does the wrong thing. I'm not sure which one, and I'm not even 100% sure it's wrong (because the behaviour is not defined in the spec as far as I can tell), but since it's Apache Bench, and Apache httpd, we're talking about here, we'd think they could work together. We'd be wrong, though.

Here's what's happening: ab is sending a HTTP 1.0 request with no Connection header, and httpd is assuming that it wants to keep the connection open, despite this. So, httpd hangs on to the socket for an additional—you guessed it—15 seconds, after the request is answered.

There are two easy ways to solve this. First, we can tell ab to actually use keep-alives properly with the -k argument. This allows ab to drop the connection on the client side after the request is complete. It doesn't have to wait for the server to close the connection because it expects the server to keep the socket open, awaiting further requests on the same socket; in the previous scenario, the server behaved the same way, but the client waited for the server to close the connection.

A more reliable way to ensure that the server closes the connection (and to avoid strange keep-alive related benchmarking artifacts) is to explicitly tell the server to close the connection instead of assuming that it should be kept open. This can be easily accomplished by sending a Connection: close header along with the request:

$ ab -H "Connection: close" -n1 -c1 http://seancoates.com/codes
(snip)
Concurrency Level:      1
Time taken for tests:   0.118 seconds
Complete requests:      1
Failed requests:        0
Write errors:           0
Total transferred:      5944 bytes
HTML transferred:       5669 bytes
Requests per second:    8.48 [#/sec] (mean)
Time per request:       117.955 [ms] (mean)
Time per request:       117.955 [ms] (mean, across all concurrent requests)
Transfer rate:          49.21 [Kbytes/sec] received
(snip)

118ms? That's more like it! A longer, more aggressive (and concurrent) benchmark gives me a result of 88.25 requests per second. That's in the ballpark of what I was expecting for this hardware and URL.

The moral of the story: state the persistent connection behaviour explicitly whenever making HTTP requests.

Aficionadoʼs Curse / Pessimistic Optimism

As I've mentioned in previous posts, I like beer. I mean, I really like it. I've tasted many unique, special, rare, and extremely old beers. I even have Beer Judging credentials. I would go as far as to say that I'm a "beer aficionado." I find the idea of a cheap, poorly-made beer (especially when there are superior alternatives on hand) to be almost repulsive.

I know aficionados in other fields: wine, vodka, scotch, cheese, movies, music, specific genres of music, woodworking, home electronics, office design, and even one guy I might consider a dog food aficionado.

These people, like myself when it comes to beer, often suffer from what I call "Aficionado's Curse." This syndrome affects experts in many fields, and prevents them from truly enjoying the full gamut of their medium. They are able to truly appreciate the utmost quality, but are turned off by the bottom shelf. Others (non-aficionados) are perfectly happy consuming the most readily available items and occasionally treating themselves to something fancy from the middle.

Consider someone who is completely immersed in the world of hand-made Swiss timepieces. Now, consider that person wearing a Happy-Meal-derived digital watch with a made-in-China plastic band. Unless they're trying to perfect their ironic hipster look, the cheap watch likely wouldn't fly.

In high school, long before I considered that this thing might have a name, I had a media teacher who was a film aficionado. He once told our class "When you start making movies—no matter the length—your world will change. You will stop simply being entertained by what is on the screen before your eyes, and instead will wonder how they did that, or you might marvel at the complexity of a specific shot. You'll still enjoy film, but in a different way." This nicely sums up Aficionado's Curse.

Maybe ignorance is bliss.

In a phenomenon similar to Aficionado's Curse, I've noticed a trend that cloaks optimism in pessimism. Unfortunately, I am a victim of this, and I try to keep it reined in, but I often fail.

I have very high expectations for things… nearly everything, in fact. I think that high expectations are generally a good thing, as so many things are—shall we say—of a quality less than the highest.

I expect machines to work properly, traffic to flow, computers to perform at reasonable levels, food to taste good, service to be quick and friendly, events to respect their schedules and other similar things. More often than not, though, I am let down by this unmaintainable level of optimism. The bad part is that in my letdown, I often find myself complaining (or if I've managed to keep it under control, not complaining) about such things. Not because the thing I've just witnessed is completely broken, but more like because it's sub-optimal in some way. These complaints are (reasonably, I admit) perceived as pessimism. My optimism has precipitated as pessimism.

I think this happens to smart people quite a bit. I've worked with people who are extremely unpleasant, but also extremely kind and forgiving. This may have been due to the scenario described above.

I saw this on Fred Wilson's blog a while back, and I think it's relevant:

"sometimes we make money with brilliant people who are easy to get along with, most often we make money with brilliant people who are hard to get along with, but we rarely make money with normal people who are easy to get along with."

One of the greatest things I learned from Chris when working at OmniTI was something that he didn't intentionally teach me (at least not as part of my job): it's OK to be let down, but complaining about it doesn't often breed positive change. I've tried to apply this to my public persona in the past few years, and at risk of sounding like a complaint, I think we'd all do well to follow Chris's lead, and strive to be brilliant people who are also easy to get along with.

Webshell

Webshell is a console-based, JavaScripty web client utility that is great for consuming, debugging and interacting with APIs.

I use Firefox as my primary browser. The main reason I've been faithful to Mozilla is my set of add-ons. I use Firebug regularly, and I'm not sure what I'd do without JSONovich.

Last year, as I built Gimme Bar's internal API, I found myself using curl, extensively, and occasionally Poster, to test and debug my code.

These two tools have allowed me to interact with HTTP, but not in the most optimal way. Poster's UI is clunky and isn't scriptable (without diving into Firefox extension internals), and curl requires a lot of Unixy glue to process the results into something more usable than visual inspection.

I wanted something that would not only make requests, but would let me interact with the result of these requests.

When working with Evan to debug a problem one day, I mentioned my problem, and said "I really should build something that fixes this." Evan suggested that such a thing would be really useful to him, too, and that he'd be interested in working on it.

I'd planned on building my version of the tool in PHP. Evan is… not a PHP guy. He's a [whisper]Ruby[/whisper] guy.

If you've seen me speak at a conference, lately, you've probably seen this graphic:

Venn Diagram

It shows that we have diverse roles in Gimme Bar, but everyone who touches our code can speak JavaScript. (This is another, much longer post that I maybe should write, but in the meantime, see this past PHP Advent entry.)

Thus, Evan suggested that we write Webshell in JavaScript, with node.js as our "framework." Despite the aforementioned affinity for Ruby (cheap shots are fun! (-: ), Evan is a pretty smart guy. It turns out that this was not only convenient, but working with HTTP traffic (especially JSON results (of course)) is way better with JavaScript than it would have been with PHP.

So, Webshell was born. If you want to see exactly what it does, you should take a look at the readme, which outlines almost all of its functionality.

If you use curl, or any sort of other ad-hoc queries to inspect, consume, debug or otherwise touch HTTP, I hope you'll take a look at Webshell. It saves me several hours every week, and most of our Gimme Bar administration is done with it. Also, it's on GitHub so please fork and patch. I'd love to see pull requests.

Gimme Bar on MongoDB

I'm happy to report that Gimme Bar has been running very well on MongoDB since early February of this year. I previously posted on some of the reasons we decided to move off of CouchDB. If you haven't read that, please consider it a prerequisite for the consumption of this post.

Late last year, I knew that we had no choice but to get off of CouchDB. I was dreading the port. The dread was two-fold. I dreaded learning a new database software, its client interface, administration techniques, and general domain-knowledge, but I also dreaded taking time away from progress on Gimme Bar to do something that I knew would help us in the long term, but was hard to justify from a "product" standpoint.

I did a lot of reading on MongoDB, and I consulted with Andrei, who'd been using MongoDB with Mapalong since they launched. In the quiet void left by the holiday, on New Year's day this year, I seized the opportunity of absent co-workers, branched our git repository, put fingers-to-keyboard—which I suppose is the coding version of pen-to-paper—and started porting Gimme Bar to Mongo.

I expected the road to MongoDB to be long, twisty, and paved with uncertainty. Instead, what I found was remarkable—incredible, even.

Kristina Chodorow has done a near-perfect job of creating the wonderful tandem that makes up PHP's MongoDB extension and its most-excellent documentation. If it wasn't for Kristina (and her employer, 10gen for dedicating her time to this), the porting might have been as-expected: difficult and lengthy. Instead, the experience was pleasant and straightforward. We're not really used to this type of luxury in the PHP world. (-:

From the start, I knew that our choice of technologies carried a certain amount of risk. I'm kind of a risk-averse person, so I like to weigh the benefits (some of which I outlined in the aforementioned post), and mitigate this risk whenever possible. My mitigation technique involved making my models as dumb as possible about what happens in the code between the models and the database. I wasn't 100% successful in keeping things entirely separate, but the abstraction really paid off. I had to write a lot of code, still, but I didn't have to worry too much about how deep this code had to reach. Other than a few cases, I swapped my CouchDB client code out for an extremely thin wrapper/helper class and re-wrote my queries. The whole process took only around two weeks (of most of my time). Testing, syncing everyone, rebuilding production images and development virtual machine images, and deployment took at least as long.

That was the story part. Here's comes the opinion part (and remember, this is just my opinion; I could very well be wrong).

After using both, extensively (for a very specific application, admittedly), I firmly believe that MongoDB is a superior NoSQL datastore solution for PHP based, non-distributed (think Dropbox), non-mobile, web applications.

This opinion stems almost fully from Mongo's rich query API. In the current version of Gimme Bar, we have a single map/reduce job (for tags). Everything else has been replaced by a straightforward and familiar query. The map/reduce is actually practical, and things like sorting and counting are a breeze with Mongo's cursors. I did have to cheat in a few places that I don't expect to scale very well (I used $in when I should denormalize), but the beauty of this is that I can do these things now, where with Couch, my only option was to denormalize and map. Yes, I know this carries a scaling/sharding and performance penalty, but you know what? I don't care yet. ("Yet" is very important.).

MongoDB also provides a few other things to developers that were absent in CouchDB. For example, PHP speaks to Mongo through the extension and a native driver. CouchDB uses HTTP for transport. HTTP carries a lot of overhead when you need to do a lot of single-document requests (for example, when topping up a pagination set that's had records de-duplicated). My favourite difference, though, is in the atomic operations, such as findAndModify, which make a huge difference both logic- and performance-wise, at least for Gimme Bar.

Of course, there are two sides to every coin. There are CouchDB features that I miss. Namely: replication, change notification, CouchDB-Lucene (we're using ElasticSearch and manual indexing now), and Futon.

Do I think MongoDB is superior to CouchDB? It depends what you're using it for. If you need truly excellent eventual-consistency replication, CouchDB might be a better choice. If you want to have your JavaScript applications talk directly to the datastore, CouchDB is definitely the way to go. Do I have a problem with CouchDB, their developers or their community? Not at all. It's just not a good fit for the kind of app we're building.

The bottom line is that I'm extremely happy with our port to MongoDB, and I don't have any regrets about switching other than not doing it sooner.

Gimme Bar no longer on CouchDB

As mentioned in a previous post, we started building Gimme Bar a little over a year ago. We did a lot of things right, but we also did some things wrong.

Since—in those early days—I was the only developer, and since most of my professional development experience is in PHP, that choice was obvious. I also started building the API before the front-end. I chose a really simple routing model for the back-end, and got to work, sans framework. Our back-end code is still really lean (for the most part), and I'm still (mostly (-: ) proud of it.

When it came time to select a datastore, I chose something a bit more risky, with Cameron's blessing.

Having just spent the best part of a year and a half working with PostgreSQL at OmniTI, I felt it was time to try something new. We knew this carried risks, but the timing was good, and—quite frankly—I was simply bored of hacking on stored procedures in PL/pgSQL. We wanted something that could be expected to scale (eventually, when we need it), without deep in-house expertise, but also something that I'd find fun to work on. I love learning new things, so we thought we'd give one of the NoSQL solutions a whirl.

In those days (January 2010), the main NoSQL contenders for building a web application were—at least in our minds—CouchDB and MongoDB. Also in those days, MongoDB didn't quite feel like it was ready for us. I could, of course, be wrong, but I figured that since I didn't know either of these systems very well, the best way to find out was to just pick one and go with it. The thing that ultimately pushed us to the CouchDB camp was a mild professional relationship with some of the CouchDB folks. So, we built the first versions of Gimme Bar on top of Linux, Apache, PHP 5.3, Lithium (on the front-end), jQuery and CouchDB.

By summer 2010, we began work on adding social features (which have since been hidden) to Gimme Bar, and CouchDB started giving us trouble. This wasn't CouchDB's fault, really. It was more of an architectural problem. We were trying to solve a relational problem with a database that by-design knew nothing about how to handle relationships.

Now might be a good time to explain document-independence and map/reduce, but I fear that would take up more attention than you've kindly offered to this article, and it's going to be long even without a detailed tutorial. Here's the short version: CouchDB stores structured objects as (JSON) documents. These documents don't know anything about their peers. To "query" (for lack of a better term) Couch, you need to write a map function (in JavaScript or Erlang, by default) that is passed all documents in the database and emits keys and values to an index that matches your map's criteria. These keys can be (roughly) sorted, and to "query" your documents, you jump to a specific part of this sorted index and grab one or more documents in the sequence. From what I understand of map/reduce (and my only practical experience so far is with CouchDB), this is how other systems such as Hadoop work, too.

There is tremendous value to a system like this. Once the index is generated, it can be incrementally updated, and querying a huge dataset is fast and efficient. The reduce side of map/reduce (we had barely a handful of reduce functions) is also incredibly powerful for calculating aggregates of the map data, but it's also intentionally limited to small subsets of the mapped data. These intentional limits allow map/reduce functions to be highly parallelizable. To run a map on 100 servers, the dataset can be split into 100 pieces, and each server can process its individual chunk safely and in parallel.

This power and flexibility has an architectural cost. Over a decade of professional development with various relational databases taught me that in order to keep one's schema descriptive and robust, one must always (for small values of "always") keep data normalized until a performance problem forces denormalization. With a document-oriented datastore like CouchDB or MongoDB, denormalization is part of the design.

A while ago, I made an extremely stripped-down example of how something like user relationships are handled in Gimme Bar with CouchDB. This document is for the user named "aaron" (_id: c988a29740241c7d20fc7974be05ec54). Aaron is following bob (_id: c988a29740241c7d20fc7974be05f67d), chris (_id: c988a29740241c7d20fc7974be05ff71), and dale (_id: c988a29740241c7d20fc7974be060bb4). You can see the references to the "following" users in aaron's document. I also published example maps of how someone might go about querying this (small) set.

The specific problem that we ran into with CouchDB is that our "timeline" page showed the collected assets of users that the currently-logged-in user is following. So, aaron would see assets that belong to bob, chris and dale. This, in itself, isn't terribly difficult; we just needed to query once for each of aaron's follows. The problem was further complicated when a requirement was raised to not only see the above, but also to collapse duplicates into one displayed asset (if bob and chris collected the same asset, aaron would only see it once). Oh, and also, these assets needed to be sorted by their capture time. These requirements made the chain of documents extremely complicated to query. In a relational system, a few (admittedly expensive) joins would have taken care of it in short order.

I spent a lot of time fighting with CouchDB to solve this problem. I asked in the #couchdb channel on Freenode, posted to the mailing list and even resorted to StackOverflow (a couple times) before coming up with a "solution." I put the word "solution" in quotes there because what I was told to do only partially solved our problem.

The general concensus was that we should denormalize our follow/following + asset records in an extreme way (as you can see in the StackOverflow posts, above). I ended up creating an interim index of all of a user's followers/following links, plus an index of all of the media hashes (what we use to uniquely identify assets, even when captured by different users). Those documents got pretty big pretty quickly (even though we had less than 100 users at the time). Here's an example: Cameron's FollowersIndex document.

As you might guess, even a system designed to handle large documents like this (such as CouchDB) would have a hard time with the sheer size. Every time an asset was captured, it would get injected into the FollowersIndex documents, which caused a reindex… which used up a lot of RAM, and caused bottlenecks. Severe bottlenecks. Our 8GB of RAM was easily exhausted by our JavaScript map function. Think about that. 8GB… for <100 users. This was not going to survive. Turns out we were exhausting Erlang's memory allocator and actually crashing CouchDB. From userspace. I asked around, and the proposed solution to this problem-within-a-problem was to re-write the JavaScript map as Erlang to avoid the JSON conversion overhead. At this point, I was desperate. I had Evan (who is a valuable member of the team, and is a far superior computer scientist to me) translate the JS to Erlang. What he came up with made my head hurt, but it worked. And by "worked," I mean that it didn't crash CouchDB and send it into a recovery spiral (crash, respawn, reindex, crash, repeat)… but it did work. Enough to get us by for a few weeks, and that's what we did: get by. The index regeneration for the friends feed was so slow that I had to use delayed indexes and reindex in cron every minute. CouchDB was using most of our computing resources, and we knew we couldn't sustain any growth with this system.

At this point, we decided to cut our losses, and I went to investigate other options, including MySQL and MongoDB. My next blog post will be on why I think MongoDB is a superior solution for building web applications, despite CouchDB being better in certain areas.

PHP Community Conference

I was once told that "the only reason you're successful is that you were at the right place at the right time." Other than the word "only" in that declaration, the accuser was mostly right. The reason I'm [moderately] successful is that I was at the right place at the right time. The subtlety in the second statement is in the reason I was at the right place at the magical time.

I firmly believe that my technical skills are only part of my value, career-wise. Looking back on my career so far, I can definitely see opportunities that arose because of being at the right place. What wasn't considered in the flippant statement was why I was there, when I was.

To me, it's clear: I've taken measures to put myself in the right place, when it was beneficial to do so. I've been doing this for years, and it's paid off.

Want to know how I became the Editor-in-Chief of php|architect magazine, a Web Architect at OmniTI, and was put into contact with my co-founder for Gimme Bar? Sure, my abilities to build web stuff played into all of those roles, but the way I found myself in all of those positions was by asking. Yes, asking.

Was I in the right place at the right time when I noticed Marco commenting about having to edit the current issue of php|architect, and I chimed in "hey, I kind of actually like that sort of thing," half a decade ago? Definitely, but it's more complicated than "luck."

Similarly, when I approached Chris Shiflett about working with OmniTI, his immediate reaction was "Of course there's room on my team for you; we'll just need to work out the details." Am I that good, when it comes to coding, architecting large deployments, and managing a team? Definitely not—even less so back then.

The real question is why was I hanging out on IRC when Marco was venting, or how was it so easy for me to have Chris's ear? The answer is simple: I'd established myself as part of the PHP community, and had a standing with those guys, even without having ever worked with them, directly (I had written for php|architect before, but it wasn't under Marco's direct supervision).

I assume that many of you readers are already members of the community in some way. That could be as simple as participating on mailing lists or forums, helping reproduce bugs, or fixing grammatical errors in the manual. One of the best ways I've found to connect with the community, though, is in person.

Nearly everyone I know and have had a long-term relationship with, in the PHP community, I met at a conference. Sure, I'd often "known" someone from their online persona, but it's hard to really "know" someone until you've spent some face time with them, preferably with a beer or two between you.

This is one of the main reasons that I think that the PHP Community Conference in Nashville, in just about a month, is important, and why I think you should go. I have no personal stake in this (in fact, since it's run by the community, the only stake to be had is a potential loss by the organizers; there is no profit to be had), I just think it's going to be a great event, and a wonderful opportunity for attendees—and not just from a career perspective, but I expect everyone who attends will become more valuable to their current employers, too, based simply on knowledge gained and connections made. (There's a huge amount of value in being able to fire off a friendly email to the author of (e.g.) the memcached extension, when you get stuck, and to already be on a first-name basis.)

I'm also speaking, there, on Gimme Bar. It won't be a pitch. It will be more of a show-and-tell session on which technologies we use, how we've built what we have so far, what I think we've done right, and a frank discussion on the mistakes we've made (so far (-: ).

If you can, you should make it to the PHP Community Conference, and be in the right place at the right time, whether it's Nashville on April 21 and 22, or sometime in your future.

Ideas of March

Around two weeks ago, Chris wrote a blog post that I responded to, and I was reminded of some of the great conversations that helped build our community. Many of these took place on the blogs of the aughts.

Like Chris, I think we've lost a bit of that. I've seen what feels like hundreds of conversations fly by on Twitter, 140 characters at a time: incomplete thoughts crammed into a package that's simply too small for detailed and deep expression. Don't get me wrong—a stream like Twitter (or maybe not Twitter itself) is valuable for quick thoughts and light conversation, but we often need more than that.

Thus, like others, I am pledging to do more blogging this year than last, starting now.

I recently spoke at ConFoo, and I intend to turn my Fifty Things talk into a series of short blog posts. I've also been mulling over a post on how and why we ported Gimme Bar from CouchDB to MongoDB. Those will hopefully pave the way and form a habit and personal culture of blogging. Please feel free to hold me to this intent, and if you have a blog, I hope you'll join this effort of creating a blogging revival (and if you don't yet have a blog, check out Habari).

See you soon.

Gimme Bar: One Year Old

Exactly one year ago, today, Gimme Bar was born.

Gimme Bar has been the focus of my work for that entire time, and I haven't blogged about it (much).

Ironically (sort of), I've been far too busy working on Gimme Bar to do much writing about Gimme Bar, but I thought it fitting to take a couple minutes to write a few words about it today.

The elevator pitch for the project goes something like this: Gimme Bar is a personal utility to help you capture and collect interesting things you find in your day-to-day use of the Web.

I'm admittedly not very good at the pitch, but my colleague (and Gimme Bar's backer) Cameron is, and we released this demo video, yesterday:

We're in the middle of some huge changes on the technical side that I intend to blog about, and once those are released, I hope to add a lot more active users. If the video makes it sound like something you might be interested in, be sure to sign up for an invitation. We've got some really great stuff coming in the pipeline, too, for existing users.

I'll post more, soon, I hope.

Post-Advent 2010

As I write this on Christmas Eve, Chris is putting the finishing touches on PHP Advent 2010.

A brief search of my site indicates that I haven't actually blogged about PHP Advent since 2007, when I was lucky enough to write the first article. That first year, Chris put the advent articles up on his blog (and we do intend to copy them over to phpadvent.org, eventually). Sensing that Chris had entirely too much work to do, curating, and since we were working together by the time the season came around in 2008, I offered to help with editing and curation—I did, after all know the pain/joy of putting together a magazine.

Chris took me up on my offer, and he enlisted Jon and Jon to design and build a proper site. We commissioned authors a little too late, but they came through and PHP Advent 2008 was a success.

By the time 2009 came around, Chris was already deep into preparing to launch Analog, and I'd already announced (internally) that I was moving on to other things. As a result, 2009's Advent was hard. Really hard. We commissioned authors too late, didn't set solid deadlines (as much as we hate deadlines, this sort of date-sensitive project requires them), neglected to dedicate enough time to author herding and editing, and to top it all off, I was headed to Costa Rica for a much-needed vacation, leaving Chris holding the bag for the last five days of 2009's season. Things were so bad at one point, last year, that I took it upon myself to write an article just so that we didn't miss a day. Luckily, we made it through (and by we, I mean Chris, because by the time my flight to San Jose on Dec. 19th came around, I'd had quite enough of Advent for the year).

If we learned anything from PHP Advent 2009, it was sadly not from the great articles, but instead from our own failures. If we were going to do this again in 2010, we needed to get on it early, and we needed to attack with full-force. I set my calendar to start bugging me in August, but even though I was hassled by its weekly reminders, we found ourselves at the start of November, wrecked from just having organized a conference, and in the middle of two product launches. Despite feeling like we didn't want to have the trouble of Advent again in 2010, neither of us dared say it to the other…at least not in so many words.

Due only to the abilities and professionalism of our most excellent authors, PHP Advent 2010 was—at least in my opinion—the best year, yet. They wrote wonderful, substantial, punchy articles that informed our readers, and generated significantly more traffic than we've seen in previous years: over 70,000 views, from more than 25,000 unique visitors, so far, and data from past years tells us that these numbers drop slightly starting on the 25th as we cease to post new content, but remain strong into January, with constant, lower traffic and occasional blips throughout the year. The most popular article this year had more than 10,000 views!

As we post the last article of 2010, I'm encouraged but all of this, and—contrary to how I felt in 2009—am actually looking forward to making PHP Advent even better in 2011.

Thank you Chris, and thank you authors. Have a wonderful new year.

Remote pbcopy

I use the command line a lot. I'm sure many of you do, too.

I find myself often piping things between processes:

$ cat seancoates.com-access_log \
> | awk {'print $1'} \
> | sort \
> | uniq \
> | wc -l
627
$ # unique IPs

One particularly useful tool on my Mac is the pbcopy utility, which takes standard input and puts it on the pasteboard (this is known as the "clipboard" on some other systems). Its sister application, pbpaste is also useful (it outputs your pasteboard to standard output when your pasteboard contains data that can be represented in some sort of text form—if you have image data copied, for example, pbpaste yields no output).

$ cat seancoates.com-access_log \
> | awk {'print $1'} \
> | sort \
> | uniq \
> | pbcopy
$ # the list of unique IPs is now on my pasteboard

I find this particularly useful for getting information from the command line into a GUI application.

Wouldn't it be even more useful if we could pbcopy from a remote SSH session? Indeed it is useful. Here's how.

The first thing you need is a listener on your local machine. Luckily, Apple has provided us with launchd and its administration utility, launchctl. This is basically [x]inetd for your Mac (plus a bunch of other potentially great stuff that I simply don't understand). Put the following in ~/Library/LaunchAgents/pbcopy.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
     <key>Label</key>
     <string>localhost.pbcopy</string>
     <key>ProgramArguments</key>
     <array>
         <string>/usr/bin/pbcopy</string>
     </array>
     <key>inetdCompatibility</key>
     <dict>
          <key>Wait</key>
          <false/>
     </dict>
     <key>Sockets</key>
     <dict>
          <key>Listeners</key>
               <dict>
                    <key>SockServiceName</key>
                    <string>2224</string>
                    <key>SockNodeName</key>
                    <string>127.0.0.1</string>
               </dict>
     </dict>
</dict>
</plist>

…then, run: launchctl load ~/Library/LaunchAgents/pbcopy.plist

This sets up a listener on localhost (127.0.0.1) port 2224, and sends any data received on this socket to /usr/bin/pbcopy. You can try it with telnet:

$ telnet 127.0.0.1 2224
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
hello
^]
telnet> ^D
Connection closed.

…then try pasting. You should have hello (followed by a newline) on your pasteboard.

The next step is tying this into SSH. Add RemoteForward 2224 127.0.0.1:2224 to ~/.ssh/config. This will tell your SSH connections to automatically forward the remote machine's local port 2224 to your local machine, on the same port, over your encrypted SSH tunnel. It's essentially the same thing as adding -R2224:localhost:2224 to your SSH connection command.

Now you have a listener on your local machine, and a secure tunnel from remote servers to this listener. We need one more piece to tie everything together. Put the following in a file (preferably in your path) on the remote machine(s) where you'd like a pipe-friendly pasteboard:

cat | nc -q1 localhost 2224

…I like to put this in ~/bin/pbcopy or /usr/local/bin/pbcopy on servers where I have root. You'll also need to chmod +x this file to make it executable. You'll need the nc executable, which is often available in a package called netcat. This invocation of nc takes standard input and pushes it to localhost on port 2224.

Now you should have a useful pbcopy on your remote server(s). Be aware, though, that there is no additional security on this port connection. If someone on the remote machine can connect to localhost:2224, they can inject something into your pasteboard. This is usually safe, but you should definitely keep it in mind. Also, if you have multiple users using this technique on the same server, you'll probably want to change the port numbers for each user.

I use this technique all the time. Now you can too. Hope it's helpful.