At my previous job (link might stop working any day now) I was the lead web developer (and lead Windows developer, and lead PHP developer…). It was always exciting to be working on new features for our users. But I often got distracted by the triumvirate of managers leading the company. “Statistics!” they’d cry. “We need more statistics!” Occasionally, I’d grudgingly comply.

Getting the right view on your data can mean the difference between success and failure. Getting a good reporting system in place is one of those not-so-sexy infrastructure things that doesn’t get users all excited, but can provide crucial insight into how your systems work and how they get used — which in turn tells you what sexy new features are the most important.

Feedwhip has been generating a decent daily report for me for the past year, but as I’ve been continuing to tackle performance issues, it’s been clear I don’t have enough info. So, I started logging more detailed stats about how pages are generated.

Lo and behold: useful data was obtained! Did you know that Feedwhip handles an RSS request every six seconds? And that serving up RSS requests are by far the most db-intensive thing I do?

With those two facts in my back pocket, I realized it was time (probably long past time) to add support for etags and last-modified headers. These headers let me report back to an RSS reader that a page hasn’t changed — so I don’t need to regenerate the content.

I don’t cache the content I generate (mostly because it changes so often, although this could be another performance enhancer to explore), so every time there’s a request for an RSS feed I need to go searching among the 1.5 million RSS items in the database for ones that match the user’s filter requirements. Some RSS feeds took more than 100 seconds to generate — and most of that was spent rifling through the database. By avoiding the generation of these pages altogether, I’ll be taking a huge load off the database — which means everything else will respond that much more quickly.

So, it’s all just a matter of finding the right statistics to give you an informative view on your system. Now with that done, I’ll go back to adding some sexy new features.