Attack of the bots

One out of every 5 hits to my site is from a bot. I spent a fair bit of time this weekend analysing my log file for last month (which runs to gigabytes, and I ended up learning a few things about file system optimisation, but more on that later). 80% of the hits were from regular browsers. 20% were from robots. Here's a sample of the user-agents: Mozilla/5.0 (compatible; Yahoo! Slurp; <a href="http://help.yahoo.com/help/us/ysearch/slurp)">http://help.yahoo.com/help/us/ysearch/slurp)</a> Mozilla/5.0 (compatible; Googlebot/2.1; +<a href="http://www.google.com/bot.html)">http://www.google.com/bot.html)</a> Mediapartners-Google DotBot/1.0.1 (<a href="http://www.dotnetdotcom.org/#info">http://www.dotnetdotcom.org/#info</a>, [email protected]) Mozilla/5.0 (Twiceler-0.9 <a href="http://www.cuill.com/twiceler/robot.html)">http://www.cuill.com/twiceler/robot.html)</a> msnbot/1.1 (+<a href="http://search.msn.com/msnbot.htm)">http://search.msn.com/msnbot.htm)</a> FeedBurner/1.0 (<a href="http://www.FeedBurner.com)">http://www.FeedBurner.com)</a> Mozilla/5.0 (compatible; attributor/1.13.2 +<a href="http://www.attributor.com)">http://www.attributor.com)</a> WebAlta Crawler/2.0 (<a href="http://www.webalta.net/ru/about_webmaster.html)">http://www.webalta.net/ru/about_webmaster.html)</a> (Windows; U; Windows NT 5.1; ru-RU) Yandex/1.01.001 (compatible; Win16; I) ... You get the idea. The bulk of these are search engines. Over two-thirds of the bot requests were from Yahoo Slurp. Now, this struck me as weird. If I take the top 3 search engines that are sending traffic my way, ...

Bollywood actors jigsaw quiz

I've jumbled up pictures of 10 Bollywood actors. You can move the jumbled blocks around, like a jigsaw. Can you guess the actors? Comments sathya 23 Aug 2008 12:53 am: are you sure jigsaw 5 is right ? I guess it is A Kh. I entered the wikipedia spelling and it still says it is wrong. Sorry for the partial spoiler. sathya 23 Aug 2008 12:54 am: are you sure jigsaw 5 is right ? I guess it is A Kh. I entered the wikipedia spelling and it still says it is wrong. Sorry for the partial spoiler. S Anand 23 Aug 2008 4:15 am: @Sathya: Jigsaw 5 is not Aamir Khan, Sathya. He is on the list, though. Saurabh 23 Aug 2008 5:20 am: Nice one, and was able to solve it as well within 5 mins :) kay 24 Aug 2008 8:46 am: Any idea why the clues don’t open up for me? I’d be interested in knowing. sathya 25 Aug 2008 2:30 am: I dint mean Aamir khan … but Ak Kh (son of an actor) … in case u r planning to post the answers, i’ll wait. S Anand 27 Aug 2008 7:13 am: @Sathya: I used his name without the “e” after the first name. That seemed to have more results on a Google search that with. But you’re right – Wikipedia spells his name with the “e”. Of course, all of this wouldn’t be a problem if I implemented a decent fuzzy search like my other quizzes. Will do that. sathya 28 Aug 2008 3:26 am: yeah. fuzzy search is a good idea. Another approach could be to use the “did u mean” like functionality that search engines offer. Peter Norvig’s approach could be used for this (which you had also linked in one of your posts). I would be curios to know which approach and how, you are going to implement the fuzzy search. Kalpesh 14 Sep 2008 6:05 am: Hi Anand, I dont know how I stubled upon your blo & since then has become a fan ;) Solved this one in 5 mins. The feed shows multiple posts for same title in bloglines. Is it because of you updating the same post? S Anand 15 Sep 2008 12:43 am: @Kalpesh: That’s right. Every time I make a minor tweak to the post, it’s shown again, I’m afraid. smrithy 1 Dec 2011 1:18 pm: hii, i found all actors except the 2nd ones.. can i get some clues for him???

Animated charts in Excel

Watch Hans Rosling's TED Talks on debunking third world myths and new insights on poverty and ask yourself: could I do this with my own data? Yes. Google has a gadget called MotionChart that lets you do this. Now, you could put this up on your web page, but that's not quite useful when presenting to a client. (It is shocking, but there are many practical problems getting an Internet connection at a client site. The room doesn't have a connection. The cable isn't long enough. You can't access the LAN. Their proxy requires authentication. The connection is too slow. Whatever.) ...

Resolving the Prisoners Dilemma

If you’re ever taken a course in Economics, and it discussed Game Theory, you may be familiar with The Prisoner’s Dilemma. Roughly, this is the problem. Assume you possess copious quantities of some item (money, for example), and wish to obtain some amount of another item (perhaps stamps, groceries, diamonds). You arrange a mutually agreeable trade with the only dealer of that item known to you. You are both satisfied with the amounts you will be giving and getting. For some reason, though, your trade must take place in secret. Each of you agrees to leave a bag at a designated place in the forest, and to pick up the other’s bag at the other’s designated place. Suppose it is clear to both of you that the two of you will never meet or have further dealings with each other again. ...

The Non-Designers Design Book

I’ve been thumbing through books on visual design for a while, and recently, picked up a copy of The Non-Designer’s Design Book by Robin Williams. If there’s one book that I’d suggest to a newbie on visual design, it’s this one. It’s rare among design books in that it offers 4 design principles that are easy to remember, easy to spot when violated, and easy to fix. Over 90% of the slides that I have reviewed violate at least one of these principles (often all), so I guess there’s a 90% chance this book will improve your design. ...

Caching pages on Apache

I don’t use any blogging software for my site. I just hand-wired it some years ago. When doing this, one of the biggest problems was caching. Consider each blog entry page. Each page has the same template, but different content. Both the template and content could be changed. So ideally, blog pages should be served dynamically. That is, every time someone requests the page, I should look up the content, look up the template, and put them together. ...

Bollywood actress jigsaw quiz

I've jumbled up pictures of 10 Bollywood actresses. You can move the jumbled blocks around, like a jigsaw. Can you guess the actresses? Comments sathya 14 Aug 2008 6:52 am: the score says 10/5 for all correct answers. pls check :-) ectp 14 Aug 2008 11:14 am: 10/10 :) Had problems with the spelling in q.8. Can’t complain, as the standard “Don’t worry about the spelling. Just spell it like it sounds, and the box will turn green.” was not there. :) S Anand 15 Aug 2008 12:47 am: Yeah, this quiz is a bit finicky on the names. Sorry about that! Malini 16 Aug 2008 2:36 am: me a fan of urs… crazy abt ur kannada song collection… Neela 20 Aug 2008 1:51 am: 10/10 :). Amazing quiz. Amazing blog. Great job! How do you find the time to do all this? warda 3 Nov 2008 12:11 pm: this quiz sux…all f d actreses were so easy 2 guess….wuff….i sure wasted my time mehru 8 Nov 2008 2:38 am: nice quiz…i dnt agree wid warda d actreses were nt dat easy 2 guess…gud job s,anand Senthilkumar.S 24 Mar 2009 9:19 am: got 9/10. interesting quiz… Nivethetha 11 Sep 2009 3:20 pm: Hi Anand, I got 9/10. Amazing quiz. Keep quizzing Juliet Rosalin Dash 21 May 2010 7:19 am: hey anand i have secured 10/10 keep going and publish this type of quizzes

How often to write

If you look at the number of entries I’ve written every month since 2005, there has been a clear decline. While I was averaging almost an entry a day in 2005 and 2006, that dropped to 2-3 entries a month since mid-2007. This doesn’t bother me. I’ve been lucky to never have lost sight of the purpose of this website. This website is meant for me. Not for you, the reader. For me, the author. ...

A new home page

I have a new home page design. (If you’re reading the RSS feed, check the home page.) One reason is that the old home page’s design sucked. Almost everyone told me that it was drab in black and white. Personally, I think the new home page sucks in terms of colours as well. There’s too many. I suck at picking colours. The only good thing about these colours is that I left it to the judgement of experts. These are the colours in Powerpoint 2007’s “Concourse” theme color, and I’ve just lifted them. ...

Launching applications

Opening programs from the Start - All Programs menu is painful. For many years, I relied on the quick launch bar. But it’s space constrained. There are only so many applications you can place there. I want space enough for frequently used documents as well. Recently, I decided that I need all the space on the screen. So my task bar is on auto hide, and that makes the quick launch bar a little tougher to use as well. And finally, I can’t use the quick launch bar with the keyboard. That’s important. ...