web-performance on S Anand

Audio data URI

Tue, 17 Jul 2012 11:41:29 +0000

Turns out that you can use data URIs in the <audio> tag.

Just upload an MP3 file to http://dataurl.net/#dataurlmaker and you’ll get a long string starting with data:audio/mp3;base64...

Insert this into your HTML:

<audio controls src=”data:audio/mp3;base64...”>

That’s it – the entire MP3 file is embedded into your HTML page without requiring additional downloads.

This takes a bit more bandwidth than the MP3, and won’t work on Internet Explorer. But for modern browsers, and small audio files, it reduces the overall load time – sort of like CSS sprites.

So, on my bus ride today, I built a little HTML5 musical keyboard that generates data URIs on the fly. Click to play.

Comments

Fabio Mazarotto 9 May 2013 8:07 pm: Can you confirm that it no longer works in Safari >= 6.0?

WP-SuperCache

Wed, 10 Feb 2010 17:27:09 +0000

That dip there in response time is thanks to WP-SuperCache. My average page load time has dropped from 1 second to 0.25 seconds.

Attack of the bots

Sun, 31 Aug 2008 12:00:00 +0000

One out of every 5 hits to my site is from a bot.

I spent a fair bit of time this weekend analysing my log file for last month (which runs to gigabytes, and I ended up learning a few things about file system optimisation, but more on that later). 80% of the hits were from regular browsers. 20% were from robots. Here's a sample of the user-agents:

Mozilla/5.0 (compatible; Yahoo! Slurp; <a href="http://help.yahoo.com/help/us/ysearch/slurp)">http://help.yahoo.com/help/us/ysearch/slurp)</a>
Mozilla/5.0 (compatible; Googlebot/2.1; +<a href="http://www.google.com/bot.html)">http://www.google.com/bot.html)</a>
Mediapartners-Google
DotBot/1.0.1 (<a href="http://www.dotnetdotcom.org/#info">http://www.dotnetdotcom.org/#info</a>, crawler@dotnetdotcom.org)
Mozilla/5.0 (Twiceler-0.9 <a href="http://www.cuill.com/twiceler/robot.html)">http://www.cuill.com/twiceler/robot.html)</a>
msnbot/1.1 (+<a href="http://search.msn.com/msnbot.htm)">http://search.msn.com/msnbot.htm)</a>
FeedBurner/1.0 (<a href="http://www.FeedBurner.com)">http://www.FeedBurner.com)</a>
Mozilla/5.0 (compatible; attributor/1.13.2 +<a href="http://www.attributor.com)">http://www.attributor.com)</a>
WebAlta Crawler/2.0 (<a href="http://www.webalta.net/ru/about_webmaster.html)">http://www.webalta.net/ru/about_webmaster.html)</a> (Windows; U; Windows NT 5.1; ru-RU)
Yandex/1.01.001 (compatible; Win16; I)
...

You get the idea. The bulk of these are search engines. Over two-thirds of the bot requests were from Yahoo Slurp. Now, this struck me as weird. If I take the top 3 search engines that are sending traffic my way,

	Referral %	Crawl %
Google	90%	24%
Yahoo	6%	66%
Microsoft	3%	0.3%
Others	1%	9%

The search engine that sends me the most traffic is being reasonably conservative, while Yahoo is just eating up the bandwidth on my site. Actually, this shouldn't bother me too much. It's not taking up too much bandwidth, or even CPU usage, given that all the bots put together make up only 20% of my traffic. But somehow... it's sub-optimal. Inelegant, even.

So I decided to take a closer look. Just how often are they crawling my site?

Yahoo	*Every 5 seconds*
Google	Every 13 seconds
DotBot	Every 9 minutes
Cuill	Every 9 minutes
Microsoft	Every 18 minutes
Feedburner	Every 18 minutes
Attributor	Every 23 minutes
Yandex	Every 27 minutes

Look at those numbers. Yahoo is hitting my site once every 5 seconds. No wonder there's a help page at Yahoo titled How can I reduce the number of requests you make on my web site? I followed their advice and set the crawl-delay to 60, so at least it slows down to once a minute.

Just that one little line change should (hopefully) reduce the load on my site by around 15%.

As for the other engines, I don't mind that much in terms of load.

Google, for all that it crawls every 13 seconds, has faithfully reported that it has only 11% of my site under its index, so I've no idea what they're doing, but I'm not complaining about the traffic that's coming my way.
DotBot. Today was the first I'd heard of them. Visited the site, and smiled. These guys can do all the crawling of my site that they like, and I hope something interesting comes out of their work.
Cuill, sends me 0.2% of my traffic, but it's a new search engine, I'm happy to give it time.
Microsoft's OK, sends me a tiny stream of traffic.
Feedburner is just pinging my RSS feed every 18 minutes.
Attributor and Yandex I'm hearing of for the first time, again. Not too much load on a system, so that's OK.

What's amazing is the sheer number of bots out there. Last month, I counted over 600 distinct user-agent strings just representing bots. So it's true. The Web is no longer just for humans. We do need a Semantic Web.

Comments

Dhar 31 Aug 2008 9:09 pm: Hmmm, curious as to:
1. Why the bots should crawl your site every 5 seconds or so?
2. How you can find out how much of your site has been indexed by Google.
Cheers, D.
S Anand 1 Sep 2008 12:11 am:
1. I think Yahoo’s crawler is aggressive in any case. My site doesn’t seem to be an exception: there are a lot of threads discussing this problem.
2. Google’s webmaster tools tells you how many URLs have been indexed from your sitemap.

HTTP download speeds

Wed, 25 Apr 2007 12:00:00 +0000

In some of the Web projects I’m working on, I have a choice of many small files vs few big files to download.

There are conflicting arguments. I’ve read that many small files are better, because you can choose to use only the required files, and they’ll be cached across the site. (These are typically CSS or Javascript files.) On the other hand, a single large file takes less time to download than the sum on many small files, because there’s less latency. (Latency is more important than bandwidth these days.)

I ran some tests, and the answer is rather easy. The graph below shows the average time taken to download a file of size 1KB - 100KB.

The X-axis is the size of the file. The Y-axis is the number of milliseconds taken to download the file, averaged over 10 attempts on my home computer. (I did the same thing at work and at a client site. The results are similar.)

The key thing is, it’s not linear. Larger files take less time. Specifically:

A file twice as big only takes 30% longer to load.
Breaking a file into two parts takes 54% longer to load.

These days, it looks like few big files are better.

To give you an example: my home page had the following components:

	Size (KB)	Load time (ms)
HTML	25	680
CSS	4	340
Javascript	6	400
Total	35	1420

The total time for these 3 files would be about 1.4 seconds. Instead, if I put them all on one file…

	Size (KB)	Load time (ms)
All-in-one	35	770

The combined file takes only 0.77 seconds – half the download time for split files. It’s a compelling argument to put all your CSS and Javascript (images, too, if possible) into a single page.

But what if people visit multiple pages on my site, and lose the benefit of caching? Not really. The benefit of caching is small. By having a single file, I have 770 - 680 = 90 ms additional time for each HTML to load. But I don’t have to load the CSS and Javascript individually, which takes 740 seconds. The breakeven is about 740 / 90 = 8 page visits.

So, on average, if people, visit more than 8 pages in my site, it’s worth breaking up the CSS and Javascript. But the average for my site is only 2 pages per person. (It’s a skewed distribution. Most, from Google, just visit one page. Few, from bookmarks, visit several pages. On average, it’s just over 2.) I’d argue that unless you’re using an external Javascript / CSS library (prototype, etc.) or people view many pages each visit, you’re better of having a single HTML+Javascript+CSS file.

Speed up your pages

Tue, 03 Jan 2006 12:00:00 +0000

Speed up your pages.

Windows Live Local

Thu, 08 Dec 2005 12:00:00 +0000

Windows Live Local tries to match Google Maps. But Google Maps is just too fast. As I mentioned earlier (Why Google Reader): the reason I like Google is largely speed.>

Why Google Reader

Thu, 20 Oct 2005 12:00:00 +0000

I switched to Google Reader as my blog reader (I was using Mozilla so far). The reason was simple: speed. Thanks to the Google site’s speed and keyboard navigation, I can read blog entries 10 times faster. Now there’s a unique proposition for Google that a lot of people are missing: that their site loads a whole lot faster than others. It makes a huge difference to the whole browsing experience.

Comments

Dhar 24 Oct 2005 9:38 am: Anand, check out Groovy.
Dhar 24 Oct 2005 9:38 am: Sorry, it should be Goowy
S Anand 24 Oct 2005 2:26 pm: Quite an app!

Firefox prefetches Google results

Fri, 01 Apr 2005 12:00:00 +0000

Firefox prefetches Google results.

Gmail performance at 1GB

Tue, 27 Jul 2004 12:00:00 +0000

Kevin Rose tests gmail performance at 1 GB.