web-analytics on S Anand

The Bing effect

Fri, 12 Jun 2009 19:14:01 +0000

This graph is the number of referrals Microsoft’s new search engine, Bing, sent to my site over the last few days. Looks like the hype is dying out. Though Bing did leapfrog Yahoo briefly, that lasted just one day.

Short notes

Wed, 20 May 2009 05:15:32 +0000

I’m quite busy on a project right now, and don’t get time to write long articles. So for a while, I’m going to stick to short notes on interesting stuff.

Peter Bregman has a very interesting piece on Why You Should Encourage Weakness. It boils down to a choice: do you focus on on improving strengths or minimising weaknesses? Conventional performance evaluations focus on the latter. I very strongly support Bregman’s view on this. The weakness isn’t why you hired the person! Unless it’s killing the organisation, just leave them to focus on their strengths.
Google Analytics has a fairly interesting API that I hadn’t explored until recently. Picked up [Advanced Web Metrics with Google Analytics](http://www.s-anand.net/amazon-browser.html#advanced web metrics with google analytics) and learnt that you can track outbound clicks, page load times, Javascript events and error logs, almost anything at all using Google Analytics. You can also mirror the logging on your local server using pageTracker._setLocalRemoteServerMode()
The whole concept of a Sandbox environment seems to be picking up within Google. There’s a Checkout sandbox, an AJAX API playground, an AdWords sandbox, an AdSense API sandbox, the Mapstraction API sandbox, even an event called Developer Sandbox. (After saying Sandbox 6 times, I feel a bit like Hobbes.)

Mobile browsing

Wed, 03 Sep 2008 12:00:00 +0000

When I analysed my HTTP log last week, I had another motive: are there enough people accessing my site on a mobile device? Or is it too small at this stage for me to care about?

Well, have a look at the numbers.

Windows	98.4%
Mobile	0.6%
Linux	0.5%
OS X	0.5%

Yes, there are more people accessing my site through a mobile device than there are using Linux or OS X. That's shocking!

Now, I'm not saying that this is representative of the rest of the world or anything, but at least it tells me a couple of things.

Firstly, the whole mobile browsing thing is bigger than I thought it was. I started worrying about this a couple of months ago and got myself a HTC s620 phone and a BlackBerry (for free, through some innovative social engineering and smooth talking). It really does get pretty useful on the move... which is frankly anywhere outside of the home and the office, and sometimes even within. (It's handier to read recipes off the HTC than a laptop.) Google had caught on to the whole mobile browsing trend a very long time ago, and are rather well positioned to make use of it.

Secondly, it means that rather than worrying about my site working on Linux or OS X (i.e. worrying about what plugins to use), I should worry more about it working on mobile devices (i.e. small screen, no Javascript / CSS).

That's a fairly big shift in my thinking. Earlier, I had been all for shifting all the processing to client-side Javascript. Now it appears I need to design more towards plain HTML pages generated by Perl / PHP.

Comments

Dinakaran 3 Sep 2008 6:23 am: Nice to see that people visit your site your mobile. Mobile browsing is a very upcoming trend . It’s yet to catch up in India for the telecom infrastructure limitations.If there are better band width’s provided by the service providers , mobile browsing will move up to a new level.

A day when I’m able to play 30 minutes of non-interupted podcast through my mobile while travelling down to the office listening to the latest technology discussion is some thing which I dream to come as a reality in the near future.

Attack of the bots

Sun, 31 Aug 2008 12:00:00 +0000

One out of every 5 hits to my site is from a bot.

I spent a fair bit of time this weekend analysing my log file for last month (which runs to gigabytes, and I ended up learning a few things about file system optimisation, but more on that later). 80% of the hits were from regular browsers. 20% were from robots. Here's a sample of the user-agents:

Mozilla/5.0 (compatible; Yahoo! Slurp; <a href="http://help.yahoo.com/help/us/ysearch/slurp)">http://help.yahoo.com/help/us/ysearch/slurp)</a>
Mozilla/5.0 (compatible; Googlebot/2.1; +<a href="http://www.google.com/bot.html)">http://www.google.com/bot.html)</a>
Mediapartners-Google
DotBot/1.0.1 (<a href="http://www.dotnetdotcom.org/#info">http://www.dotnetdotcom.org/#info</a>, crawler@dotnetdotcom.org)
Mozilla/5.0 (Twiceler-0.9 <a href="http://www.cuill.com/twiceler/robot.html)">http://www.cuill.com/twiceler/robot.html)</a>
msnbot/1.1 (+<a href="http://search.msn.com/msnbot.htm)">http://search.msn.com/msnbot.htm)</a>
FeedBurner/1.0 (<a href="http://www.FeedBurner.com)">http://www.FeedBurner.com)</a>
Mozilla/5.0 (compatible; attributor/1.13.2 +<a href="http://www.attributor.com)">http://www.attributor.com)</a>
WebAlta Crawler/2.0 (<a href="http://www.webalta.net/ru/about_webmaster.html)">http://www.webalta.net/ru/about_webmaster.html)</a> (Windows; U; Windows NT 5.1; ru-RU)
Yandex/1.01.001 (compatible; Win16; I)
...

You get the idea. The bulk of these are search engines. Over two-thirds of the bot requests were from Yahoo Slurp. Now, this struck me as weird. If I take the top 3 search engines that are sending traffic my way,

	Referral %	Crawl %
Google	90%	24%
Yahoo	6%	66%
Microsoft	3%	0.3%
Others	1%	9%

The search engine that sends me the most traffic is being reasonably conservative, while Yahoo is just eating up the bandwidth on my site. Actually, this shouldn't bother me too much. It's not taking up too much bandwidth, or even CPU usage, given that all the bots put together make up only 20% of my traffic. But somehow... it's sub-optimal. Inelegant, even.

So I decided to take a closer look. Just how often are they crawling my site?

Yahoo	*Every 5 seconds*
Google	Every 13 seconds
DotBot	Every 9 minutes
Cuill	Every 9 minutes
Microsoft	Every 18 minutes
Feedburner	Every 18 minutes
Attributor	Every 23 minutes
Yandex	Every 27 minutes

Look at those numbers. Yahoo is hitting my site once every 5 seconds. No wonder there's a help page at Yahoo titled How can I reduce the number of requests you make on my web site? I followed their advice and set the crawl-delay to 60, so at least it slows down to once a minute.

Just that one little line change should (hopefully) reduce the load on my site by around 15%.

As for the other engines, I don't mind that much in terms of load.

Google, for all that it crawls every 13 seconds, has faithfully reported that it has only 11% of my site under its index, so I've no idea what they're doing, but I'm not complaining about the traffic that's coming my way.
DotBot. Today was the first I'd heard of them. Visited the site, and smiled. These guys can do all the crawling of my site that they like, and I hope something interesting comes out of their work.
Cuill, sends me 0.2% of my traffic, but it's a new search engine, I'm happy to give it time.
Microsoft's OK, sends me a tiny stream of traffic.
Feedburner is just pinging my RSS feed every 18 minutes.
Attributor and Yandex I'm hearing of for the first time, again. Not too much load on a system, so that's OK.

What's amazing is the sheer number of bots out there. Last month, I counted over 600 distinct user-agent strings just representing bots. So it's true. The Web is no longer just for humans. We do need a Semantic Web.

Comments

Dhar 31 Aug 2008 9:09 pm: Hmmm, curious as to:
1. Why the bots should crawl your site every 5 seconds or so?
2. How you can find out how much of your site has been indexed by Google.
Cheers, D.
S Anand 1 Sep 2008 12:11 am:
1. I think Yahoo’s crawler is aggressive in any case. My site doesn’t seem to be an exception: there are a lot of threads discussing this problem.
2. Google’s webmaster tools tells you how many URLs have been indexed from your sitemap.

Most bookmarked pages

Mon, 15 Jan 2007 12:00:00 +0000

These are the most bookmarked pages on my site:

My home page
Excel tips
Calvin & Hobbes quotes (I typed them all)
Indian torrents (I have a search engine for Indian torrents)
Tamil Transliterator (Lets you type Tamil in English)
Tamil songs quiz
Movie quote quiz
My best links
Top 10 lists

But this post is not about these links.

It’s about how I found this out.

Think about it… how could I know what pages have been bookmarked? The browser doesn’t send any information about bookmarks.

Some months ago, I moved away from Google Analytics mainly to have more control over tracking visitors. Among other things, I track referrers. When you click on any page and go to another one, the second page knows the first page you came from. That first page is the referrer.

So I know every page people clicked on to get to my site. Usually it’s Google. Sometimes it’s someone’s blog. Sometimes, it’s blank.

The blank referrers either indicate that the browser has blocked the referrer page, or that the person didn’t visit any page before mine. The former is rare (less than 1%). So, realistically, a blank referrer has either the person typed in the URL, or bookmarked it.

To make sure, I did a quick survey over the weekend. Those who came to my page without a referrer saw a survey form, asking where they came from. Almost all of them had either bookmarked or typed my page. The typists went directly to my home page. All the other links you see above are bookmarks.

So all I had to do was count the number of hits for each page with blank referrers. That’s the list above.

Absence of information can be a powerful indicator too.

Comments

Chitra 16 Jan 2007 7:11 am: Oho…how many people did the list comprise of?
S Anand 16 Jan 2007 12:56 pm: I had over 4,000 unique IP addresses, but I don’t believe that’s even representative of visitors, thanks to most ISPs using dynamic IPs. I get about 80 direct IP hits per day, so I guess the number of bookmarks is around twice that.
Rupesh 17 Jan 2007 9:41 am: Would you get to know if someone has subscribed to your site feed this way?
S Anand 17 Jan 2007 12:06 pm: Actually, I use Feedburner to track subscriptions to my site. But no, using this I can’t always tell if it’s a feed subscription. People coming through live bookmarks can’t be distinguished from normal bookmarks. But people using feed readers (Google Reader / Bloglines) can be identified by their referrer.
Shreyas 19 Jan 2007 5:46 am: Apologies for a completely off-topic comment but have you taken off the ‘Starred Items on Google Reader’ link? I kind of enjoyed reading the ones you had a couple of weeks back!

Demographics prediction from online behaviour

Mon, 26 Jun 2006 12:00:00 +0000

Microsoft adCenter Labs has a demographics prediction engine. Based on a person's search queries and web sites visited, it can predict their gender and age.

So I tried that on parts of the body, to see what men were interested in vs women.

topic	male	female
hair	25%	75%
eyes	33%	67%
cheek	33%	67%
hands	33%	67%
lips	36%	64%
ears	39%	61%
fingers	40%	60%
forehead	42%	58%
nose	43%	57%
neck	46%	54%
beard	55%	45%
moustache	58%	42%
leg	60%	40%
palm	61%	39%
toe	64%	36%

While I can understand men being more interested in beards and moustaches (perhaps even legs), why are they far more interested in toes than women?

Comments

Dhar 28 Jun 2006 2:10 am: Hey Anand, a quick question.
Are you using Google Spreadsheets? Yesterday, I realized that after the initial enthu, I have not visited the site. This makes me wonder if Google Spreadsheets in itself is a powerful enough solution to attract the masses.
Dhar 28 Jun 2006 2:12 am: As pointed out by someone on Slashdot, try using the URL http://www.google.com/ as the input and you will get “Female Oriented with following Confidence: Female 1.00, Male 0.00”
S Anand 28 Jun 2006 2:45 pm: I do use Google spreadsheets. Just see my next post.

Search queries to my site

Thu, 06 Apr 2006 12:00:00 +0000

On a related note, 60% of the search queries that lead to my site this year were Calvin and Hobbes quotes. “i can’t help but wonder what kind of desperate straits would drive a man to invent this thing.” topped the list (Calvin referring to a yo-yo), with i always catch these trick questions following closely.

People searching for Excel related stuff were next (20%): excel indirect(address(, row() excel offset address and the like.

A few were also looking for me by name or school (10%).

The last 10% ranged from the puzzling to the bizarre, including these gems.

michalengelo hidden skull. Probably looking for the alleged hidden skull in Last Judgement.
googlemail access between england and india. Why? Did he think there wouldn’t be any?
greenwich meridien time for india. This is usually the same in Greenwich and in India. Rest of the world too.
origin of monkey in fez. What?
address of sexy girl in ahmedabad. But why my site?
address of tool makers for converting html to xml in chennai. Do they have a license to convert?

Comments

Sai 11 Apr 2006 4:55 pm: Really funny! What kind of metics/tools do you have to know what searches lead to your site?
S Anand 11 Apr 2006 7:44 pm: I use Google Analytics.
ritzkini 14 Apr 2006 5:37 pm: address of sexy girl in ahmedabad !! :))
Krishna 15 Apr 2006 5:57 pm: Thats a real funny set of queries. But why your site? ;-)
The Calvin and Hobbes search Takedown | s-anand.net 21 May 2010 12:02 pm (pingback): […] also increased traffic to my site, which was a bit disconcerting. I didn’t want to attract attention. In 2007, I […]

No site has linked to my page

Fri, 31 May 2002 12:00:00 +0000

Almost no site has linked to my page. (I sort of like that a bit. It means the 30-odd hits a day that I get are from people who TYPE my URL.)

Wednesday is popular

Sun, 29 Jul 2001 12:00:00 +0000

Wednesday seems the most popular day for visiting my site. While I get 14 visitors a day on average, I seem to get 22 visitors on Wednesdays.

Weblog analyser

Thu, 01 Feb 2001 12:00:00 +0000

I’m using Analog for analysing the IIMB webserver stats. Anyone with enthu, find a better log analyzer and mail me its name.

Popular sites at IIMB

Mon, 10 Jul 2000 12:00:00 +0000

Whose sites are the most popular on this server? Last month, the top 10 sites were admissions, placement, Prof. Srinivasan’s, and Prof. Bandi’s. I ran http-analyze to get these results.