(Geeky post alert. If you’re reading this on Facebook, the links and formatting are going to be all messed up.)
15 years ago, I wrote a blog post about a stereo cabinet glass door that spontaneously exploded. For some reason, this post attracted a lot of attention. If I had written it a few years later, one would say it “went viral.” It received tens of thousands of page views, and 330 comments.
At some point, I decided to export it to a static page, since every page load was causing my server – at the time, running on a Pentium in my home office across my DSL line – to slow down horribly. In the process, I managed to delete the page entirely (a long story within a long story) and I grabbed the page off of the Wayback Machine.
That page is HERE, by the way.
Each comment has a Gravatar logo next to it, which, due to the way curl (the tool I used to retrieve the static copy) works, has a name like avatar(230).php but is actually a jpeg file. That means that every time the page loads, it makes 330 calls to the php engine, which errors out because the file in question isn’t a php file, but is an actual on-disk jpeg file. Like this one, for example.
Then, several years ago, I switched from using mod_php to using php_fpm, which does the same thing, except more efficiently.
Finally, at some point, I added a mod_security ruleset that attempted to detect when people were DDoSing my site – the barrier it set was more than 30 requests in under a second.
These various things, all combined, resulted in a situation where whenever someone attempted to view that page, it would cause my server to crawl to a halt, and the visitor to be added to my mod_security deny list. This was not desired behavior.
Of course, this is all in retrospect. All I knew was that several times a day, I’d get failure notices from my server monitoring, and by the time I got there to see what was happening, the problem had cleared up. So, no big deal, right.
This has been going on for years.
Today, looking at error logs trying to figure out what was happening, I suddenly put all of the pieces together, and fixed the problem, in less time than it has taken me to write this blog post. The solution has a few parts.
First, we exclude anything in the /files/ directory from being processed by php:
# (old line) ProxyPassMatch ^/(.*\.php(/.*)?)$ fcgi://127.0.0.1:9000/var/www/vhosts/drbacchus/$1
ProxyPassMatch ^/(?!files)(.*\.php(/.*)?)$ fcgi://127.0.0.1:9000/var/www/vhosts/drbacchus/$1
That adds the (?!files) negative lookahead, which says “only do this if it DOESN’T match ‘files’
Next, we turn off the mod_security rule specifically for these requests:
<LocationMatch (Exploding)>
SecRuleEngine off
</LocationMatch>
Which says, don’t run the SecRuleEngine for requests that contain ‘Exploding’, which is in the URL of the static copy of the blog post.
Finally, I have to tell httpd that the .php files in the static copy are, in fact, jpeg files:
<Directory /var/www/vhosts/drbacchus/files>
AddType image/jpg .php
</Directory>
This has the added benefit that if anybody dropped a .php file in my files directory, it would be defanged, so to speak, and wouldn’t execute.