Bug 1: Ajax post request

The first bug I had to fix was not really a javascript bug. The only buggy part about it, in terms of javascript, was that I was allowed to write buggy code with no alarms going off. The problematic code was sending a post request done through pure javascript – no jQuery magic or anything like it, just pure javascript. Because the request needed to be a fullblown multipart/form-data request, the headers were from constructed from scratch – this included creating the boundary string for the request. Unfortunately, the code I was using was creating a flawed boundary (in particular, it included a semicolon at the end of the boundary) but all browsers happily sent the request along. And everything worked smoothly, till we upgraded the PHP binary on our server: in version 5.3.9, PHP enforces the HTTP standard more strictly and so it simply drops the incoming POST data if the boundary string isn’t formatted properly. Which meant that all of a sudden, the ajax requests dropped like bricks.

How do you debug that? You can see that there’s no POST data coming through, but you also know the script sending the data worked until recently – and you know the script hasn’t changed. What you don’t know is that the server was upgraded – but you learn this after about 1 hour. However, you do know that Chrome sends the data through just fine – but IE and Firefox don’t. Where do you start looking for the problem? Nowhere do you get any errors (PHP doesn’t notify you that the data from the POST request was dropped – and the browsers don’t tell you that the request is faulty), so you have to guess at what the error might be.

What led me to the answer was that Google Chrome detected the problem and fixed it – although silently. Chrome rewrites the boundary identifier before sending the request through – if it detects a faulty boundary identifier. So when I tried googles boundary string, suddenly things worked. A bit of experimentation later and I had the solution – which also led me to the changelog for PHP 5.3.9, where this thing is mentioned.

Bug 2: cloning in IE9

The second bug that caused me massive headaches was cloning DOM elements in IE9. I do the cloning in order to send back html to the server, for converting to PDF. Turns out that it’s much, much, MUCH easier to deal with the html through the DOM (and especially through jQuery or other such libraries) than it is to deal with it through PHP. So what is done is that the main node is cloned, various bits/pieces removed and then the content is sent through a request as a text string. How could this go wrong, you ask? Well, turns out that IE9 decided to throw hissy fits and just randomly apply classes to elements. Well, not quite randomly: in the process of cloning, it came across one class-name it liked so much, that it decided to apply it to about 75% of all elements on the page (but not all the elements – that would have been too consistent).

What fascinates me about this is that all other browsers I have tried work just fine. Yes, that also means IE7 and IE8 (we don’t cater to IE6 on the project) – no trouble there. Microsoft actually worked hard to make sure that there were new bugs in IE9. Well, I suppose the world wouldn’t have been the same if one of their browsers actually worked as it should.

Of course, this is exaggerating things. Most likely, the fault is mine for cloning elements with IDs. Still, I think Microsoft got the golden rule of interoperability wrong (the one that goes something like: “be lax on your input but strict on your output”): instead of being lax about input, they decided to be strict about it … while they in general are rather lax about their output (another example of this sending a response to an ajax request – you had better not think to set the charset to utf8, as that will just result in errors you have no idea how decipher).

What makes me think that? Well, the fact that my hacky workaround works: grab the string representation of the node to clone, add an extra bit to all IDs, then create a new element from that and start working. And hey presto, problem solved. Do I feel dirty now? Yes. Is my loathing of IE and MS bigger? Yes. Was I fooled again by them, thinking that IE had in fact improved? Yes. Shame on me? Yes.

The twist

I have recently developed a small error-handling script in JS (after reading about a new service that lets you install that in an easy fashion on your site and thinking “I can do that”) and have put that up on the site. I had hoped to glean information from this but it turned out utterly useless for both bugs – because none of them were bugs in javascript or with my programming. They were bugs of the implementation of javascript in browsers – not of the language itself.

In fact, the error handler only served to confuse things, because it has picked up on a number of bugs, including errors in browser plugins and weird script handler issues in the Bing crawler. I don’t regret putting it into place but certainly needs a lot of analysing before it gives off any goodies.

In the end, only lucky guessing and some deductions provided answers. Of course, various tools could probably have helped, but I would have needed to know that these would be useful in tracking down the error – so I would still have had to guess at the problem. I have learned a bit more about browsers, and – most importantly – about errors. The worst possible error is the one disguised as success: either your typical silent error (such as dropping the POST vars without mentioning it) or your more atypical everything-is-fine error (where things fail but appearances are kept up).

I had a quick look at the scene and came up with the following candidates:

• Tonic
• Slim
• Recess
• Epiphany
• Frapi

There are others as well, but these looked like the best candidates for quickly setting up a small API.

Measuring

My needs for the framework are:

• small size
• get out of the way
• don’t force me to do more stuff than I need
• few requirements
• ability to get into it in 5-10 minutes

Tonic

This framework seems nicely done. However, after looking at the source and the setup for 5 minutes, I found more overhead than I want. Specifically, the process of dispatching from request to controller involved much more code than I think should be needed. On top of that, it seems to include Smarty, which is 100% overkill for my needs (and which I’m generally opposed to). And as a final hint that I wouldn’t like the framework, it included a module for doing basic http authentication. Not exactly something I want in my RESTful API.

Slim

Here there’s very little setup needed before you’re running a RESTful service. The framework really does all it can to get out of the way, so after looking for 2 minutes at the included example index file you’ll know how to get the basics going. The size of the framework is about 230K, which is not bad. Overall a very good contestant.

Recess

After installing this and setting it up, I browse to the test page only to find it needs a database to work. Instant dismissal: if my API doesn’t need a database I’m sure as hell not going to be bothered with creating one for the framework.

Epiphany

This is the second final contestant for the framework to use. Again, fairly simple setup and you’ll know very fast after installing it how it works, so you won’t waste a lot of time reading docs before you can begin to use it. I like it a bit less than Slim, though, as it seems to need more manual setup (more static calls than I would expect, before things are running). However, very small size (80K), nice set of libraries, a nice surprise in terms of flexibility, so this one is definitely not out of the race.

Frapi

This seems like a nice a fairly advanced framework, however, right out of the box it involves too much complexity to be of any use to me. First off, I need to setup two virtualhosts for it, one for the frontend and one for the backend. I don’t need that, as I don’t need a backend. Secondly, I couldn’t even get to the point of figuring out if Frapi needs a backend, as it wouldn’t run without APC which I don’t have running on my small netbook. So it got disqualified from the race right there. If it hadn’t, it would have been on account of size, as it packs in 31MB just in libraries.

Conclusions

I’ll probably go with Slim though I’ll try to do a minimal test-case of Slim and Epiphany (just with an auth module) to compare developing with both. I’m sure the other frameworks have positive sides as well, but they strike me as overkill for a RESTful API. I get why you would want to make things slightly more complex if the goal is developing web applications (as Tonic is advertised for, for instance) but including smarty? That’s simply unnecessary bloat to me.

One thing I skipped over here is obviously the existing bigger frameworks like CodeIgniter, CakePHP, Zend, etc. All these are way overkill for an API, if you ask me. They don’t get out of the way, instead they dictate most of your choices (not necessarily a bad thing – enforcing best practices is good in my book – just not needed here). Zend, for instance, has a good set of modules that could come in quite handy for a RESTful api (likely why Frapi includes a whole lot of Zend code), but Zend seems to be built upon the idea that Java development practices are good and can/should be used in PHP. Which is quite simply wrong, if you ask me.

So yeah, the proper tool for the job, which in this case amounts to Slim or Epiphany. Hereby recommended.

Fine, option #2: connect to Hotmail with an email client, say Thunderbird. Setup is easy, indeed it goes very fast, but what’s this? POP3? But I want IMAP, as I don’t want to bother with checking that messages are not deleted online and I also want to download everything in the various folders. Some more research and it’s revealed that while M$ has no problem bleeding money out of their crap search engine, they can’t be bothered to provide a proper interface to their webmail. How can these morons still be in business? How is it possible that they lure so many people into using their crap?

Grudgingly, we move on to option #3: Microsofts own Windows Live Mail. As the company is known for it’s loser-mentality (keep everything proprietary and make sure your own developers know “the secret APIs”) one might expect that this program (package, in fact: another loser-mentality streak – you need to download an entire package to get to their mail program, even though you have no use for the other crap) would work. However, that’s not the case: it only downloads whatever’s in the inbox. Microsoft does not believe in backup, apparently.

Conclusion: the only thing I actually managed to do to backup the emails was to print them to PDF. Yes, that’s how horribly bad Microsofts email service is. It really, truly boggles my mind why anyone would ever trust anything to do with email from this company (so please dump your POS exchange servers – there are better alternatives anyway and they’re free).

Well, not entirely true: I also managed to get so angry that I wrote to Microsoft that they could “fuck off and die”. That’s how bad my user experience was.

So to make sure I don’t forget these things after I’ve done them I’ve got a library of files containing notes on how to do things. On top of that, I’ve also got Trac running on my server. I installed it mainly to keep track of hosting info (such as server details, bugs and other things I needed to take care of, projects from clients, etc) but I discovered it would also be useful to migrate my library of notes onto Trac. Or rather, I haven’t migrated it, I’ve just copied it over to trac while keeping (and working on) the local copy. Hence the need to sync everything. Locally, I keep the notes in git, making for easy syncing across the network – but I don’t have anything like that setup for trac. Hence, a bit of late night coding and hey presto! git hooks. I still need to work a bit on the pre-commit hook but the post-commit hook is done. Just a very simple script to update trac after I’ve committed changes to my notes locally. It looks like this:

// script residing in the main repo folder

#!/usr/bin/php
<?php
define('TRAC_AUTH', 'username:password');
define('COOKIEFILE', tempnam('/tmp', 'cookies_'));

$self = array('web_hook.php', 'web_update.php');

if (empty($_SERVER['argv'][1])) {
    echo "Call with name of file" . PHP_EOL;
    exit(1);
}

if (in_array($_SERVER['argv'][1], $self)) {
    exit(0);
}

$url_base   = 'http://path/to/server/wiki/';
$url_suffix = '?action=edit';

$url_array = array(
    'server.txt'      => 'Server',
    'development.txt' => 'Development',
    'email.txt'       => 'Email',
    'git.txt'         => 'Git',
);

if (!isset($url_array[$_SERVER['argv'][1]])) {
    echo "No such file set: " . $_SERVER['argv'][1] . PHP_EOL;
    exit(1);
}

$url_part = $url_array[$_SERVER['argv'][1]];

$data = fetch_data($url_base . $url_part . $url_suffix);
if (!$data) {
    echo "Could not fetch data" . PHP_EOL;
    exit(1);
}

if (!put_data($data, $_SERVER['argv'][1], $url_base . $url_part)) {
    echo "Could not update trac" . PHP_EOL;
    exit(1);
}

/**
 * posts data to the proper
 * page, updating trac info
 *
 * @param string $data
 * @param string $filename
 * @param string $url
 *
 * @return string
 */
function put_data($data, $filename, $url) {
    $dom = new DOMDocument();
    $dom->loadHTML($data);

    $xpath = new DOMXPath($dom);

    $form_token  = $xpath->query('//form[@id="edit"]//input[@name="__FORM_TOKEN"]')->item(0)->getAttribute('value');
    $version     = $xpath->query('//form[@id="edit"]//input[@name="version"]')->item(0)->getAttribute('value');
    $action      = $xpath->query('//form[@id="edit"]//input[@name="action"]')->item(0)->getAttribute('value');
    $from_editor = $xpath->query('//form[@id="edit"]//input[@name="from_editor"]')->item(0)->getAttribute('value');

    $comment = "Saved from git hook";
    $text    = file_get_contents($filename);
    if (empty($text)) {
        echo "No text to save";
        exit(1);
    }

    $post_data = array(
        '__FORM_TOKEN'  => $form_token,
        'version'       => $version,
        'action'        => $action,
        'from_editor'   => $from_editor,
        'text'          => $text,
        'comment'       => $comment,
    );

    $curl_handle = curl_init();
    curl_setopt($curl_handle, CURLOPT_URL, $url);
    curl_setopt($curl_handle, CURLOPT_FOLLOWLOCATION, true);
    curl_setopt($curl_handle, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($curl_handle, CURLOPT_USERPWD, TRAC_AUTH);
    curl_setopt($curl_handle, CURLOPT_POST, true);
    curl_setopt($curl_handle, CURLOPT_POSTFIELDS, $post_data);
    curl_setopt($curl_handle, CURLOPT_COOKIEFILE, COOKIEFILE);

    return curl_exec($curl_handle);
}

/**
 * fetches the data from trac
 *
 * @param string $url
 *
 * @return string
 */
function fetch_data($url) {
    $curl_handle = curl_init();
    curl_setopt($curl_handle, CURLOPT_URL, $url);
    curl_setopt($curl_handle, CURLOPT_FOLLOWLOCATION, true);
    curl_setopt($curl_handle, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($curl_handle, CURLOPT_USERPWD, TRAC_AUTH);
    curl_setopt($curl_handle, CURLOPT_COOKIEJAR, COOKIEFILE);

    return curl_exec($curl_handle);
}

// this is the post-commit file in .git/hooks/
for LINE in `git log --oneline --name-only -1 HEAD | awk '{if ($2 == "") print $1}'`
do
    if ! ./web_update.php $LINE
    then
        echo "Failed to update trac with changes"
    fi
done

It should be fairly easy to figure out what’s going on. The post-commit hook checks all the lines from the log for the last commit and filters the output, passing it to the update script as needed. The update script fetches the edit page from trac for the given file, scrapes out the needed variables to post back to trac, then sends a post request to trac, thus updating the trac wiki.

Now, I’m sure there are better ways (most likely some sort of trac plugin or api) to update trac remotely … but the point is, this is just a bit of late night coding and done deal: I’m now updating my trac wiki whenever I commit my notes locally.

The interesting thing about nginx is that it’s so very lightweight. In fact, on my server, I run nginx with php as a proxy for web and mail to a number of VPS setups. The individual VPS setups that run webapps typically run Apache, and not one of them is remotely close to the same memory footprint as nginx + php. This is not much of an observation though, seeing as the Apache installs are not trimmed to be low-resource – however, I haven’t trimmed nginx either, it just is small-footprint resource-wise. Thus, for the average layperson sysadmin (a contradiction in terms, though not in practice) an nginx + PHP install can make rather good sense, especially if you’re on a low-powered VPS setup.

So I decided to look a bit more into nginx and PHP. As mentioned I already have it running on my server but that was a fairly fast setup without too much thought – just consulting the various tutorials I could find while trying to sidestep any obvious issues. Hence, more study was needed.

Installing nginx

On Debian/Ubuntu you can just apt-get nginx

sudo apt-get install nginx

That will, however, install a slightly older version of nginx. To get the newest stable, go to http://nginx.org/en/download.html and grab the latest.

sudo -s
cd /opt
wget http://nginx.org/download/nginx-1.0.5.tar.gz && tar -zxvvf nginx-1.0.5.tar.gz
rm nginx-1.0.5.tar.gz
cd nginx-1.0.5
./configure --http-log-path=/var/log/nginx/ --error-log-path=/var/log/nginx/ --with-http_ssl_module --with-mail --with-mail_ssl_module --with-imap --with-imap_ssl_module --with-pcre
make
make install

You’ll need a number of packages to be able to run this, but the configure command should complain and tell you which if you don’t know. Also note that the mail and imap configuration settings above are in place because I use nginx to proxy email access to the individual VPS machines I have setup – if you don’t need that, you don’t need those settings. After you’re through running the commands (apt-get or manual install), you should have nginx setup and ready to run (running actually, if you used apt-get).

After this, you need to get PHP installed, and here things get more tricky. There are a number of ways of going about things:

• compile PHP yourself from source (http://php.net/downloads.php) or install php5-cgi using apt-get. Then use your a homegrown script to manage PHP
• as above, but use spawn-fcgi to manage PHP (depending upon your OS version, you might need to install lighttpd as spawn-fcgi only became
• use PHP-FPM (manually compiled from http://php-fpm.org/ or installed via apt-get)

The configuration from this point on differs a bit, because you’ll either be setting up your own script, configuring spawn-fcgi or configuring PHP-FPM. However, the most relevant part for this post refers to nginx and PHP communicating – for info on the bits about setting things up, you can check out Linode Library for scripts and info. The nginx wiki also has good info. What it boils down to, in the end, is whether nginx will be communicating with PHP over TCP or through sockets. If you google you’ll find differing advices but the majority seems to point in the direction of sockets. The reasoning behind this is that sockets are faster than TCP – there’s much less overhead. You might also find some warnings that some people have had problems with sockets though – so it’s not an easy choice, and essentially you need to test things out yourself.

Which I then did – or rather, I settled for sockets, then decided to do some benchmark testing, and then figured out there was a need to test out both before you actually choose. The configuration I went with looks like:

# relevant nginx part
location / {
    root           html;
    fastcgi_pass   unix:/tmp/php5-fpm.sock;
    fastcgi_index  index.php;
    fastcgi_param  SCRIPT_FILENAME  /var/www/nginx-php/$fastcgi_script_name;
    include        fastcgi_params;
}

# relevant php-fpm part, from /etc/php5/fpm/pool.d/
listen = /tmp/php5-fpm.sock

Together, the two will make the server serve requests to php-fpm through sockets. Now, because I was playing with nginx and PHP-fastcgi I thought it would be interesting to see how it would handle a bunch of requests, so I fired up ab (Apache Benchmark) from apache and started flooding nginx. First I went with something like:

ab -kc 100 -n 1000 nginx-php:8000/

which worked fine (if you’re not familiar with the options for ab, the line above will fire 100 concurrent keepalive requests at the server at nginx-php, port 8000, until a 1000 requests have been sent. I had put a .php script with nothing but a single echo at the end of that url, just to check the throughput. Then, just for kicks, I upped the concurrency to 200 requests a second – and suddenly I start seeing errors. Checking the error log of nginx, I see the following:

connect() to unix:/tmp/php5-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream

I then added a -v 3 to the ab command line, to get some more detailed info about what nginx would output if that happened. Turns out you’ll get something like this:

WARNING: Response code not 2xx (502)
LOG: header received:
HTTP/1.1 502 Bad Gateway

Essentially, what happens is that nginx tries to open a connection to PHP-fastcgi, fails for one reason or another, then responds to the user with a 502 Bad Gateway.

Thinking that maybe it was a limitation based on the php setup I tried changing the fastcgi settings (how many PHP processes to keep around, max processes, etc) but no change – I kept coming up against the 502. The limit on my local setup seems to be around 150-160 concurrent requests – more than that and errors crop up. However, I then switched to a TCP based setup and the errors are gone. Testing a bit, I up the concurrency to 400 connections, still no errors. At triple the concurrency I start to see problems, but not before.

However, instead of calling this a win for TCP there’s the other aspect: speed. Running a test with 100 concurrent connections for a total of 50,000 requests, sockets gave a 15% speed increase compared to TCP. Looking at CPU consumption, on the other hand, showed a higher spike with sockets than TCP (reasonably explained by bigger throughput). Memory consumption didn’t spike much for either setup, though sockets used a tad more.

Conclusion

You need to figure out what your likely scenario is when setting up your server. That should more or less self-explanatory – however, it’s something you find out time and again as you play around with hardware and software.

While I have seen some possible solutions on how to ramp up the concurrency for nginx + PHP using sockets, the main thing to do is ask yourself: will I actually ever hit much more than 100 concurrent requests that I need to forward to PHP? If not, then you’re safe using sockets. If you expect to hit much more, you should probably look into how to optimize nginx with PHP – or consider other options that might scale easier.