This morning I was glancing at my log files and noticed something quite odd:
193.29.77.* - - [11/Jul/2003:10:16:40 -0400] "GET /cgi-bin/MT/mt-tb.cgi?__mode=view&entry_id=3218 HTTP/1.0" 200 939 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
193.29.77.* - - [11/Jul/2003:10:16:51 -0400] "GET /2003/07/09/because_everything_is_better_with_bukkake.html HTTP/1.0" 200 6390 "-" "Mozilla/5.0 (X11; U; SunOS sun4u; en-US; rv:1.0rc3) Gecko/20020524"
193.29.77.* - - [11/Jul/2003:10:16:56 -0400] "GET /cgi-bin/MT/mt-comments.cgi?entry_id=3217 HTTP/1.0" 200 6006 "-" "Mozilla/4.0 (SunOS 5.8)"
193.29.77.* - - [11/Jul/2003:10:16:59 -0400] "GET /cgi-bin/MT/mt-tb.cgi?__mode=view&entry_id=3217 HTTP/1.0" 200 961 "-" "Mozilla/4.61 [en] (OS/2; U)"
193.29.77.* - - [11/Jul/2003:10:17:06 -0400] "GET /2003/07/09/qotd_07092003.html HTTP/1.0" 200 5166 "-" "Mozilla/5.0 (OS/2; U; Warp 4.5; en-US; rv:0.9.8) Gecko/20020204"
193.29.77.* - - [11/Jul/2003:10:17:09 -0400] "GET /cgi-bin/MT/mt-comments.cgi?entry_id=3216 HTTP/1.0" 200 4912 "-" "Mozilla/4.0 (Windows XP 5.1)"
193.29.77.* - - [11/Jul/2003:10:17:12 -0400] "GET /cgi-bin/MT/mt-tb.cgi?__mode=view&entry_id=3216 HTTP/1.0" 200 877 "-" "Mozilla/5.0 (Windows XP; U) Opera 6.01 [en]\""
Notice anything interesting? Take a look at the referrer string User-agent. The ip is the same, but the referrer changes with every query. What an ass. Thanks to the power of mod_rewrite that person will no longer be crawling my site (Though it's looking like I may just have to block his whole network).
Edit: I had to change the IP# because for some reason mod_rewrite of remote addr was catching the IP# in the log snipped above and screwing with the whole page. Hmm. Sounds like a bug?
Another Edit: Oh yeah, and the other thing they were doing was ignoring my robots.txt
