The robots outnumber the people

Watching my Apache log files for this web site, I was struck by something very interesting. I used to notice in my log files, the occasional request by a ‘robot’ or ‘spider’ – some automated program that downloads web content on behalf of their owner.

Today, I’m noticing the occasional request by a real browser. Almost all of the requests to my web site are by automated programs, which download my content either for publishing on some aggregator or other, or from something like Amphetadesk that serves the same purpose on the desktop.

So it seems that, finally, we have been relived of the drudgery of reading websites by programs that do it for us, even as Douglas Adams foresaw in “Long Dark Teatime of the Soul”, and in his talk at ApacheCon London 2000.