I wish I had some cool screenshots for this one, but hopefully a text-based description will suffice.

So I fixed a bug at work, where certain AJAX calls were failing.  It turns out that our view templates were making some assumptions that they ought not to have been making.  I fixed that error, and our QA team pushed that code out to our test servers (along with other things– I wasn’t that important).  Well, the error, or at least its symptoms, showed up again.  Now, it only showed up sometimes, and only on the test server.  “It works on my machine,” not being much of a solution to a bug, I started investigating.

I looked into the error logs that were being generated, and saw what looked like the sort of error you get when you call a helper method which doesn’t exist.  As part of the bug fix, I added a helper method.  We checked manually the code that was running on the server, and it had the helper method and the full new version of the code.  At this point I began suspecting the work of some foul dark wizard– or at least a gremlin.  I mean, why else would a view template not be able to find a helper method that is there after the server has been restarted?

I’ll tell you why.  We have a cluster configuration, and Capistrano or something else, hadn’t shut down every process running the old code.  Since AJAX calls are full-blown server requests that just happen not to refresh the whole browser page, some of those calls were directed to server processes that hadn’t actually been restarted.  Rails doesn’t auto-refresh code in production mode (Good Thing), so those old processes quite naturally didn’t find the new code.  We killed the old processes and, presto!  The world is right again.

So, if you ever find yourself wondering why your view template can’t find your helper method, check to make sure that your server restart restarted ALL instances of your application. We’re still not entirely sure why all instances didn’t restart though.

So, Dark Wizards may still be involved.