User

You are logged in as Anonymous.

Want to log out?

My friend Paul has a cool service called Wonderproxy that lets you test and develop GeoIP-based apps without the normal headaches. If you need to simulate remote, international traffic, you should check it out.

Goodbye, OmniTI

Today is my last day at OmniTI.

From an email I just sent out to my soon-to-be-past colleagues:

“I sincerely wish you continued success as a company, and also as individuals who truly make up a significant portion of the best people in this industry. There are many things that OmniTI does very well, and I won't hesitate to refer business your way when the situation arises.

This past year and a half (or so) has been a bumpy road, but I'm absolutely sure I will look back on my time with OmniTI as a net-positive. Thank you all for supporting me and my team with our sometimes-(absurd | stupid | obvious | amateur | tough) questions and requests.”

The road ahead for OmniTI doesn't look nearly as bumpy, but after a very long period of thought, I finally decided to pursue other options around 6 weeks ago, and will now join the ranks of the funemployed.

Thanks for the opportunities, experience, insight, and tough problems, OmniTI.

2010 will be a great year. I'm already excited about some of the prospects that are in my future.

Bonus points if the title of this post seems familiar. (-:

Horrible Support

I fully acknowledge that this is a rant. If you're not into that sort of thing, scroll on by—nothing to see here. I do have a point at the end of the (long) narrative(s), if you do manage to read the whole thing, though.

Technical customer service sucks—at least for people who have the slightest clue about the technology they're calling about.

Videotron

Today, I spent all afternoon (and this evening, and this will carry into tomorrow, maybe Friday) without Internet service. After lunch, I was sitting at my dining room table with the laptop, where I have a clear view of the telephone/hydro/cable pole across the street. This pole services my house. I glanced up and noticed a Videotron (my cable provider) truck, and its driver up in the bucket, doing something to the wires.

I jokingly wrote this in the work IRC channel:

[14:08] sean: expecting internet to drop any second
[14:09] sean: there's a cable guy on the pole across the street

I was disconnected within seconds. I looked up again, and the truck was pulling away.

Thinking it might just be a one-off desynchronization, I reset my cable modem, and it didn't reconnect.

I hate calling technical support. It's always an absolute last resort. If I get to the point of desperation that I actually need to call tech support, you can be absolutely sure that I've fully exhausted every possible solution on my side. This was obviously related to the careless technician formerly of the pole across my street, and clearly not a problem on my side.

I waded through the phone menus, and got to speak to a support agent. After confirming my super-secret birth date with him to verify my identity, he asked me tell him what was wrong.

"One of your technicians was on the pole across the street from my house. My connection was working perfectly before he arrived. When he left, my modem wouldn't sync. He obviously broke something, could you please send him back?"

I was asked to reset my modem. I explained that I already tried this, and it still wouldn't sync. I was then asked to reboot my computer. I explained, calmly, that nothing has changed on my side, and the modem simply wouldn't connect. Nothing on my side of the modem mattered if the modem wouldn't connect. He offered to check the files to see if anyone else in my neighbourhood was complaining of outage, noticed that there was a technician in the area, and that he was fixing a problem that was reported earlier this morning.

I chuckled and told my support agent that he probably fixed my neighbour's problem, but in the process managed to seemlingly knock the pole side of my connection out. The agent told me that since there was a reported outage in the area already, he couldn't send a technician but there was someone working on it. I didn't believe him that someone was still working on it, since the truck pulled away. I was right.

It was obvious to me that the agent wouldn't understand Occam's razor, so I didn't bother.

Three hours later (5:30pm), I called my ISP again. I swam through the sea of menus, and spoke with a technician. I had to explain the whole situation again. He asked me to reset my modem, reboot my computer, and wanted to know if my router had changed configuration. I, again, calmly explained that my router configuration is moot if the modem won't sync. After checking what was presumably a connection diagnostic on his side, and once again verifying that my neighbours weren't having trouble, he had a eureka moment and informed me that there was a technician in my neighborhood repairing a problem earlier that day. I reminded him that I just told him the same thing (slightly less calmly—but not rudely—this time), and he admitted defeat and agreed to send a technician to my place. "He'll be there by 8pm."

I fully expected to have to call Videotron back at 8:05pm, and I was right. So, I tried resetting my modem one last time, then called support. The menus were getting easier. "Please choose from the 7 follo..." *keypad 2* I got a technician, confirmed my birthdate before he even asked, and started explaining that the technician didn't show up. He was confused. I explained the whole problem, and concluded by repeating that the technician didn't show up. "Oh, I see that a technician is supposed to visit you." Sighing, I said "Yes, that's why I'm calling you. He was supposed to be here by 8. It's after 8 now."

The agent said he'd have to call the technician's dispatcher to see what's up. Hold music. (Aside: the hold music gets interrupted by messages that explain how to fix your own problems by resetting your modem, but they warn not to do this if you're using Videotron's VOIP service (-: ) The agent comes back and starts grilling me about not being home when the technician got here, so I missed him. I was home the whole time. In fact, I spent the 2.5h checking the yard every time I heard a car drive by. The agent insists that the technician was here, and I wasn't. I insist that I was here, and the technician didn't want to come out after 8pm on a cold, wet autumn night, so is saying that I wasn't home. Stalemate.

The agent had an idea. Technicians are supposed to leave cards when they call and the customer neglected to be home to receive them. I checked both doors. No card. Nothing. This convinced the agent that the technician wasn't here, somehow. He offered to schedule a new appointment for tomorrow. I remind him that this isn't my fault in any way, and they should just come fix it now. That's not possible. I can schedule for one of three blocks, tomorrow. 7am-12pm, 12pm-5pm, or 5pm-8pm. None of these work for me. I have to take my daughter to school in the morning. I have tomorrow off, and I have errands that need to be run, so the 5-hour afternoon block is out, and I'll be out in the evening. Friday is the same situation. They require me to be home for FIVE hours just so they can fix a problem that I didn't cause.

He said it's impossible for them to fix the problem if I'm not going to be home. I know this isn't true, but I'm sure it's a policy on their side, so I didn't fight with the poor agent too much. "He was perfectly capable of breaking my service without coming into my house."

So now I'm in Internet limbo. I don't know when/if it will be fixed. I'm basically screwed until I can find a 5 hour window where I'll be home and when I don't need to be online. Normally, I'd just tell them to fix it or cancel my account, but these guys are the least-worst choice for broadband in Montreal. The only other option is DSL from Bell. (Not quite true: there are other options like 3G access from Rogers (another evil), Satellite (impossible latency), and resellers that use Bell's and Videotron's infrastructures; none of which are actually viable.)

Bell

Before getting Videotron at the house, I had DSL from Bell. I canceled them due to their incompetence.

One day, after a few months of good service, I started getting >50% packet loss. I checked everything on my side. It was fine. This was a problem with my DSL connection itself. So, I gave in and called tech support.

The usual annoying questions ensued. You'd think that if I said "I'm measuring 53% packet loss" it would automatically qualify me for escalation beyond the "is your computer on?" type of questions. Not so.

I rebooted. I bypassed the router. I installed their stupid PPPOE software (which was not necessary, but I obliged anyway). Magically, this didn't fix my packet loss problem. The agent acknowledged that they weren't getting a very good signal to my DSL modem. Then he asked me a stupid question. "How long is the telephone cable that connects your modem to the wall?" I replied with the truth "I don't know, offhand. I guess eight feet." Little did I know that this was a trick question. The correct answer to cable length queries was about to be revealed: six feet. "What?" "It needs to be six feet long." "Uh. No. It doesn't." "Yes, sir, with a longer cable, you will introduce noise, and you'll get packet loss." This was humourous but also frustrating. I asked the agent if the electrons magically changed into some sort of noise-proof signal upon entering the wall, as it was the same type of cable on both sides of the socket. He wasn't amused.

"Hold on a sec. *pause* OK. Now it's six feet." He was still unamused. "Sir, you can't just tell me it's six feet." "Oh, no. I wouldn't do that. It's six feet now." If you make up lies about things like this, it's fair for me to play your game.

He finally gave in and agreed to send a technician to fix the problem. The first appointment they had was four days later. Yes. Four days. I insisted that there must be an appointment before that. They disagreed. I pressed, anyway "Do you know that I can sign up for service with Videotron faster than you can get a technician out here to solve your DSL problem?" They held their ground, so I signed up with Videotron and canceled. Videotron has worked well up until today.

I hate having to manipulate tech support to solve a real problem, though. This reminds me of how I've had to deal with Dell in the past.

Dell

Around five years ago, I had a Dell laptop. After a few months of use, the power connector on the motherboard came loose, and it would only charge sporadically. We had purchased the super-mega-extended-warranty that Dell offers, so when I called tech support (obviously, this was not a problem I could solve on my own, or I would have), and convinced them that the hardware needed to be fixed, they sent a technician to my office the next morning (super warranty to the rescue).

The technician replaced the motherboard on my laptop. When he put everything back together, I gave it a quick test and was satisfied, so I thanked him and signed off on his work. Within an hour or so, I noticed that my computer was underperforming. Everything was slow.

To make my subjective observation into objective evidence, I found some online benchmarks for my laptop's model, and ran the same benchmarks locally. I was right: it was underperforming by around 50%.

I called tech support again, and explained the whole situation: a technician replaced my motherboard that day, and afterwards, my computer was performing much worse than before. Obviously something was wrong with the new motherboard. "Obviously" has a much different meaning to me than to Dell's technical support. I was forced to go through a procedure of rebooting multiple times, re-seating RAM, resetting my BIOS, explaining that I couldn't boot Windows into safe mode because I wasn't running Windows (this further confused the agent, and almost jeopardized my ability to actually get support). I was around 45 minutes into the call at this point, and I had no way of convincing the agent that the motherboard replacement was the obvious culpit. Her flow chart of how to solve my problem didn't include an actual solution to my problem, and every branch of her problem-solving scripts ended up in fruitless frustration.

Finally, she asked me to run the full Dell diagnostic tests. This came on a CD with the laptop, and I'd run it once before just to see what it did. It took several hours to run the full suite. She was ready to be through with me, rescued by an impossibly long procedure, but I wasn't ready to give up that easily. So, I dug out the disk, and asked her to "please hold." At this point, I was quite bored, and had to amuse myself, so I'd pick up the phone every ten minutes or so and ask her entertaining, yet covertly mean questions about her job. "Out of curiousity, is your performance judged by your average call duration?" "Will this 90 minute call negatively affect you?"

Around two hours in, I decided to give up. It was obvious that she didn't have a script that would allow her to turn my problems into a new visit from their technicians, no matter how many times I insisted that the motherboard was to blame. I had places to be, so I thanked her for her help and hung up without any sort of solution.

The next morning, I desperately called the technician directly. I had his number because Dell outsources on-site work to third parties, and I had to call him to schedule the first meeting. I explained the whole situation, from the slowness to my useless call with Dell tech support. He was sympathetic, but insisted that there was no way he could help without a work order. I understood, but asked him how he might suggest I actually solve this problem. "Well, if your computer has no power at all, then they'd have to replace the motherboard again." A lightbulb turned on. "I understand! Thank you!"

So, I called Dell tech support again, and played dumb. When asked to describe the problem, I said "my computer won't turn on." "It says in your file that your computer is running slowly..." "Yes, that was yesterday. Today, it just doesn't work." A few minor exercises involving removal and replacement (or so they thought) of the battery, they broke the bad news "I'm sorry sir, but we're going to have to replace your motherboard." I feigned sadness, and got a new work order number, and was told a technician would call.

The technician replaced my motherboard that afternoon, and everything returned to normal. I even had a working power connector after the ordeal.

Apple

The only time I can remember actually having good technical support from any company with more than 100 employees is from Apple.

This might read like a fanboy remark, but it's true. The few times I've had to visit the Apple Geniuses at their stores, they've actually listened to my problem, acknowledged that I've probably already tried the obvious solutions, and treated me like an actual person, and not just someone they want to get out of their queue as quickly as possible to improve their call time averages.

I've been genuinely impressed with them.

I wish more companies could be like Apple in this regard. It would have been trivial for Videotron and Dell to acknowledge that—in all likelihood—the problem was caused by obvious circumstances. It's truly nice to not be asked to "reboot, and call back" when talking with technicians who actually make an effort to understand and solve the problem.

In the meantime, I'll keep "borrowing" my neighbour's open wifi. Thanks, "default"!

Drupal Frustration Redux

Earlier this week, I wrote a piece on why I'm no longer willing to tolerate Drupal. It got a lot of traffic (for my blog), and a lot of comments.

Most of those comments were "classic apologetics" as one person put it, and upon reading them, I realized that I may not have done the best job of making my point (which boils down to: Drupal takes more effort to maintain than I'm willing to spend). If you missed it, I replied to most of the complaints.

Of all of the comments, though, one really stood out, and rang truer than anything else I've read on this topic:

“I think the problem isn't so much that Drupal didn't work with PHP 5.3 - it's that the Drupal upgrade process is so bad.

In fact there isn't much of an upgrade path for Drupal at all unless you stick only to core (and who does that?)

Drupal has too much "tribal knowledge" of it's own—too much is decided on IRC where it isn't available to more casual users.

More importantly Drupal has rejected great chunks of the tribal knowledge of programmers: OOP helps move projects towards self documentation - while design patterns speed understanding of a new system. In fact Drupal rejects so much that it has even implemented it's own bug tracking and project management tools built on Drupal—that have handcuffed Drupal to CVS for far too long.

Drupal is great for people who don't like programming, who can live with the user interface and don't care what the data model looks like.

It is possible to download Drupal and quickly build a rough prototype with little prior knowledge—or if you really invest in learning the Drupal way you can do pretty much whatever you want with it.

But if you're a programmer at heart and want to use/develop transferable skills maybe Drupal isn't the best tool.

...

There is some awareness of these problems within the Drupal community:

— Sean Burlington (reposted with permission)

Sean (the other Sean) did a much better job at conveying the point than I did. I think he really hit the nail on the head when he called it "Tribal Knowledge."

More of his thoughts on Drupal.

Why I Won't Recommend SilverStripe

In my search for a replacement for Drupal, I started playing with SilverStripe over the weekend.

The site looks pretty good, the demo—while minimal (which is totally fine by me)—seems to work decently. It made my "to test" list.

The bottom line:

A bunch of people replied asking "why?" This is my response to those people. My intent is not to bash SilverStripe, but to explain how it fell short of my wants and needs, quickly.

First stop: the code.

SilverStripe is built on a framework called Sapphire, which is developed by the same folks as SilverStripe (a bit like the CodeIgniter-Expression Engine relationship, I guess). A [very] quick look through the tutorials had me interested, and the actual functionality of SilverStripe seems to be something we could work with. I downloaded it, unpacked the archive, and dove straight into the code.

Again, maybe my expectations are too high.

Here's an excerpt from the root index.php.

<p><![CDATA[
// For linux
$_SERVER['SCRIPT_FILENAME'] = str_replace('/index.php','/sapphire/main.php', $_SERVER['SCRIPT_FILENAME']);
$_SERVER['SCRIPT_NAME'] = str_replace('/index.php','/sapphire/main.php', $_SERVER['SCRIPT_NAME']);
// And for windows
$_SERVER['SCRIPT_FILENAME'] = str_replace('\\index.php','\\sapphire\\main.php', $_SERVER['SCRIPT_FILENAME']);
$_SERVER['SCRIPT_NAME'] = str_replace('\\index.php','\\sapphire\\main.php', $_SERVER['SCRIPT_NAME']);
 
chdir('sapphire');
require_once('sapphire/main.php');
]]>

The useless str_replace calls aren't a huge problem, but they demonstrate sloppiness. It's easy enough to detect whether or not we're on Windows (hint: the PHP_OS constant) without redundantly replacing everything twice.

What really got my attention was the chdir call. They must have some reason to do this, but it's almost certainly a bad one. At the very least, it should have a comment indicating why they'd do something weird like this.

Next, sapphire/main.php (excerpt; the first two executable lines in the file):

<p><![CDATA[
$majorVersion = strtok(phpversion(),'.');
if($majorVersion < 5) {
...
]]>

This raised all kinds of warning flags to me. Maybe they're worried about PHP_VERSION constant being missing from some old versions of PHP (to their benefit, the manual doesn't say when this was introduced), and that's why they've decided to use the phpversion() function. There is, however, a function built in to PHP to compare versions: version_compare(). That's the safest way to do this. A more obscure, but faster way to handle this is to check if a function or class only defined in 5.x exists.

.htaccess also hints at something weird:

<p><![CDATA[
<Files *.ss>
Order deny,allow
Deny from all
Allow from 127.0.0.1
</Files>
]]>

"What's this *.ss stuff?" I wondered. It's SilverStripe's templating system. I didn't even bother to see how they're interpreted, but they're apparently not PHP-based, with syntax such as <% base_tag %> and $SilverStripeNavigator (the latter was not within tags of any sort).

I made it through barely a handful of files before deciding to just plug my nose and ignore the code, hoping that I would be so blown away by the functionality that I would forget what lies under the covers.

I tried the installer. It complained that I didn't have the mysql extension built. Fair enough, it's true. I didn't have the mysql extension. I did have pdo_mysql and mysqli but no, I didn't have the ancient extension installed. I expected better.

Once I rebuilt PHP with mysql-proper support, I went through the installation process, and when it finished, I was redirected to a page that said "The requested page couldn't be found." So, I gave up.

I could probably fix that problem, and none of the other problems on their own are insurmountable, but combined, they leave me with the impression that the code isn't what I'd like to commit to maintaining.

The search continues...

A Messy Breakup with Drupal (Least Worst, Part 1)

A little over two years ago, when I got involved with the MontreAlers (note: yes, site broken... read on), the homebrewing club in Montreal, I was quickly nominated to be the sucker who would help take their aging and mostly-neglected web site, and make it somehow useful.

At the time, the choice was simple: custom code or Drupal. Because custom code takes quite a bit of the most valuable commodity in life—time—and because I had (and continue to have) very limited spare time, I chose Drupal.

I hacked together a custom theme (based on one of the prepackaged templates), spent a few hours configuring modules, and spent probably an equal amount of time shaking my head at the procedural and poorly-designed (for small values of "design") plugin/extension API. I grabbed some pre-built modules, and cobbled together a reasonable community site. We didn't need much, and Drupal + a few plugins solved most of our problems.

The site was a hit. Membership in the club has climbed from a handful of members to what's often a troublingly large number of attendees to our monthly meetings. There's no direct correlation to the site, but I think all of the MontreAlers would agree that it has played a role in connecting with new likeminded people.

As with any project whose maintainer doesn't have sufficient available time, the site got dusty over the course of the first few weeks, then dirty over the next few months, and before we even realized, the site was in a state of partial disrepair. Databases were upgraded, hardware was swapped, DNS was migrated, and PHP versions were updated. Each of these tasks of regular maintenance were met with heightened amounts of finger crossing, cursing and furious patching. After upgrades or changes, I'd usually get inquiries about why certain functionality mysteriously stopped working, which often lead to more furious patching, cursing and finger crossing.

Over time, minor inconveniences can easily snowball into what seems like one large, insurmountable problem. The last straw for ongoing my relationship with Drupal was upon upgrading the PHP version on my server to 5.3.

Admittedly, 5.3 is fairly young, but I was ready to take on the challenge. I'm handy enough with PHP's changes that I felt confident that I'd be able to take on any problems that Drupal presented. The obvious first step, here, was to update Drupal to the most recent release. I don't remember the exact date of the upgrade, but I do remember my feelings clearly. It was the day I started actively searching for a Drupal replacement.

The upgrade was extremely painful. There was no clear path to upgrading from our old, stagnant Drupal install to the newest version. Updating the code was a bit painful, but possible. The Drupal upgrader/installer did take care of the database update, which was my biggest worry, and there was some documentation on how to do this on the Drupal site, but I had to dig pretty deep to find it (and I've since lost the links, so I apologize that I can't link to the references I used).

There were two main pain points: I had to manually patch Drupal to work on PHP 5.3, and some of the third-party Drupal modules the club now depended on were not possible to upgrade.

The patching for PHP 5.3 was not entirely unanticipated, and was reminiscent of making fixes in broken array_merge()s when jumping from 4.3 to 5.0. The real problem here is that Drupal still maintains its code in CVS, so I gave up on submitting my patches upstream shortly after diffing and trying (to no avail) to find their subversion or Git repository. I thought PHP was the last significant project to move from CVS to SVN, but I was obviously wrong.

I fully expect Drupal will (or maybe already has) catch up to the current release of PHP (aside: they had plenty of notice to get their code up to speed—PHP 5.3 was certainly long in the tooth by 5.3.0). The real problem with the latest upgrade was that some of our modules not only didn't work with the new version of Drupal, but there was often no available upgrade for them. Our event calendar went missing, user maps (home addresses for club meetings) disappeared, our photo gallery is completely AWOL. There may be suitable replacements (read: alternative functionality in other modules) for some of our wants and needs, but the fact remains that maintenance is painful, and "upgrades"—often for the purpose of plugging security holes—take much more time and effort than I have to give.

The underlying code in Drupal is old. It seems brittle. It doesn't do things I'd expect modern software to do such as have a good inheritance model (hint: it's [almost?] all prodedural code), have self-documenting or intuitive APIs, and it does tons of things that I simply don't care about (I'll probably blog on this last thing as a general poisonous open source trend in another episode). It's more pain than it's worth at this point, and I just need it out of my life.

So, Drupal, we're breaking up.

But I don't have anyone else to go (more on my short list in a near-future entry).

(Here's the rhetorical paragraph.) Is it too much to ask for PHP code to actually be well-written? It doesn't even need to be perfect. Habari is certainly far from perfect, and it has warts, but on a whole, it's good code. Is the barrier to entry for most projects too low? Or is there just a huge lack of experience in knowing how to do things right? Maybe all of the best people are too busy doing other things? Or maybe it's a community-management problem...? I don't know.

When it comes to software, and especially web software, I find myself always settling for "Least Worst." Habari is Least Worst. Trac is Least Worst. MediaWiki, PHPMyAdmin, Roundcube: all arguably best-in-class, but under the hood, they're often painful. I suspect my expectations are simply too high. Should I just lower them? I don't know that either.

 1 2 3 … 20 Next →


Clicky Web Analytics