I compare Indian IT service firms to prisons where professionals endure physical searches, strict clock-ins, and restricted internet access. I share my survey results on corporate web freedom and discuss the industry's pervasive lack of trust.

(I had intended to write this post sarcastically, a bit like my web freedom survey. But sarcasm’s confusing to read. So I’ll just be straight and mild.) If you’re a well-paid professional in an Indian IT services firm, your freedom is limited.\ (This holds if you’re a student, too.) You clock-in and clock-out. You’re searched on your way in and out. You need your boss’s permission to leave. You work on what you’ve been told to work on. You cannot be trusted to freely access the Internet. The last bit worries me the most. Perhaps because in all the other cases, there are humans I can put to shame or fight, face-to-face. Or because I am a Net addict. Don’t know why. Anyway, here’s the result of my survey (after de-duplicating and eliminating results where the company or geography was not clear). web-freedom-survey Some day, I will follow-this up with a post on “Surviving in Prison”, detailing out my experiences with the system, and beating it. Comments Kalidas 22 May 2010 3:06 am: Good to see your posts again ! Seems like a long hibernation. Great post but pls let me know the suriving tactics !! Not able to access yahoo/gmail from office is a real pain Saravanan 23 May 2010 1:12 am: Searching on the way out was the most demeaning thing I ever have to face while working at an Indian IT company. Other idiotic rule I had to endure was inability to change Wallpaper. I like nice wallpapers in my PCs and change them every few days. To make things worse, one day they put wallpaper with company's mission and vision statement across all PCs in the company. grrr.. There may be genuine case for blocking mail and docs site due to confidentiality and security purposes. But blocking social sites or blogs is meaning less. If employees wasting their time browsing instead of working, then they should be made responsible for their actions. Blocking sites won't work. If IT companies are worried about productivity, then they should start looking into tea breaks first :) Harman, You sound like Jackie chan, who told free society is not good for Chinese and they need to be controlled. You are from an elite group, so you don't need anyone to oversee or control you. Lower class people who work in Indian IT firms need some rules, because they can't be trusted. WTF.. If a company is going to treat their employees like kids, then expect them to behave like kids. Treat them as adults. If somebody abuses their freedom, then make responsible for their actions. Making stupid polices won't help. wanderlust 29 May 2010 11:53 pm: these strict rules apply mainly in the services sector, where there are people from different levels of enthu for job, different levels of education... basically more diversity than some r&d lab where everyone is mostly focussed, are in the job because they want this line of work, and hence are less inclined to be unproductive because of their internet addiction and/or are less likely to occupy a lot of bandwidth by downloading music and movies, and are more aware of the security risks of pornsites (which spread viruses, trojans, etc). basically these non-services firms take in mostly people for whom internet addiction is not a roadblock in the way of productivity. which is not completely true of the general population. as are a lot of other things. like for example, my previous employer was one of those where everyone's phone number was public for everyone else to see, and yet we did not face situations like those faced by my friends in infy where during the freshers training, girls had to contend with guys getting their number from the employee database and harassing them. Siva 21 May 2010 5:00 pm: Excellent!!!!! Raj 21 May 2010 10:55 pm: waiting for it eagerly....on how to survive in prison... Its so ridiculous sometimes that the bossess think that any free minute you get should be spent in thinking for the company. They should know that we are mature enough to put our mind to work when it is needed and take rest at teh same time. This attitude especially makes it sick for us to show loyalty to the company. Harman 22 May 2010 2:03 am: Sometimes people are treated as kids or prisoners because they are not ready for freedom. There are companies in India which don't have any of the above, and guess what, they pay far more and choose their employees far more carefully. Most of the "end-to-end solution providers" such as Infy, Wipro, TCS, Satyam, Accenture, CSC hire so many people and so many of them are freshers, that a blanket attitude of "respect the employee" is counter-productive. White-collar employees in India are evolving in their work-ethic and professionalism. In my old company, there were routine thefts of mice and keyboard and headphones. People were choking the internet bandwidth by downloading porn and mp3 songs. Many were unable to concentrate because of constant messaging on public IM networks. I am with you that this is a sorry state of affairs, but merely leaving it to the employees to improve themselves may not always work. In the longer run, it requires ethical training and better role models at work, which are sometimes sadly missing. Start treating your work with respect if you want to be treated with respect. It's that simple. I will be the first to say that white-collar IT jobs in India are mostly at the low end of software industry, and as such, it is hard to "love" one's work and see it as anything more than a road to money and privilege, but still, you were not forced into this work, it pays mighty well, and there are reasonable perks, so don't whine and try to "get back" at your company for not respecting you "enough" by giving you all you want. As for clocking time, that may be a requirement driven by daily billing concerns of "clients". And it would be ignorance to say that an average IT workers works any more than 3-4 hours a day (if even that), no matter how much time he is physically in office. IT jobs are privileged jobs in a poor country like India. These are sometimes the first private sector jobs for many families. Have some perspective. :-) Of course IITians and suchlike will feel insulted in such an atmosphere, not the least because of the quality of one's colleagues, but very few IITians have these companies as their first or second choice anyway. Kalpesh 23 May 2010 7:31 pm: 2 things. Lack of trust & Blanket rules to cover it (from companies point of view). Makes me think what do ceo/vp do, if they are not booked for meetings? Please note, I am not saying that a job of a ceo/vp is low tech. I am curious to understand what do they do, in absence of internet connectivity? work on excel/word? Thejesh GN 24 May 2010 5:48 am: "Some day, I will follow-this up with a post on “Surviving in Prison”, detailing out my experiences with the system, and beating it." I would like to contribute to that. Sasidhar 24 May 2010 12:06 pm: Wanna come work for Google? Thomas 24 Jun 2010 3:35 pm: The most recent attack on freedom. My machine should only have software approved by big momma... It's a strict no-no to about 5 of my inseparable productivity tools, 10 of my preferred tools and file viewers and 10 of the lovely bits of software which I can live without - but makes work and time at office dull and sadder by a wide margin! They have a tool which will monitor the installed software - and any exceptions need to be approved by the top honchos up there... Atleast a USB is not blocked - nor are admin rights. So it's time to go portable ! thank God for registry hacks which should help me beat the system for now ! Thomas 24 Jun 2010 3:52 pm: Also, anand, can u pls share the name of the IT firms which blocked google ?!!!! TJ 21 Jul 2010 11:49 am: What do you do as a manager if your employees are abusing the freedom? Have been struggling with this for months now, though I agree that policing is not a solution. In my experience there is a clear impact on productivity and quality of work of my employees due to time spent on and distraction due to frequent/continuous surfing. Rahul 8 Nov 2010 6:11 am: Yes its true that companies do block some websites in a hope to increase productivity. Harman correctly identified some examples where employees waste company's resources. In my view blocking /policing the internet is not the solution for increase the productivity... the root cause for this attitude needs to be resolved. The root cause is our, individual's sense of responsibility. Our education system, our society does not teach ourselves to be responsible... we learn to be selfish, we learn to do 'our' thing even if it infringes 'others' freedom. Hence the reflection of internet-policing in Indian IT firms... where someone higher up there need to tell the 'children' what to do and what not to. Of course they do not know what is good and what is not for them... someone need to tell them.

You are in prison


I found that Tamil Nadu students born in September score up to 10% higher than June-born peers. This "relative age effect" stems from school cutoff dates making older students more mature than their younger classmates.

Does it matter which month you’re born in? Based on the results of the 20 lakh students taking the Class XII exams at Tamil Nadu over the last 3 years (via Reportbee), it appears that the month you were born in can make a difference of as much as 120 marks out of 1,200 – or 10%! Most students who took the Class XII exams in 2011 were born between March 1991 and June 1992. The average marks of each student (out of 1200) is shown in the graph below. tn-2011 Students born in June 1991 scored the lowest – around 720/1200. This suddenly shoots up in July, then in August, and the students born in September score as much as 840/1200 on average. From there on, it’s downhill. This result is consistent across years. In 2009 and 2010, you see a similar pattern. tn-2009 tn-2010 Why could this be? Malcolm Gladwell’s book Outliers offers a clue. Outliers opens, for example, by examining why a hugely disproportionate number of professional hockey and soccer players are born in January, February and March. The answer turns out to be completely unrelated to numerology or astrology. It's simply that in Canada the eligibility cutoff for age-class hockey is January 1. A boy who turns ten on January 2, then, could be playing alongside someone who doesn't turn ten until the end of the year—and at that age, in preadolescence, a twelve-month gap in age represents an enormous difference in physical maturity. In Tamil Nadu, students must be 5 years old before entering Class 1. Schools open mid-June. So students born in June 1994 would barely make it in June 1999 – making them the youngest students in the class. July and August students would be missed – but since many schools implement this policy leniently, they sometimes make it in as well. September borns are often consistently the eldest students in a class. This pattern reflected in the marks. The eldest – the September 1993 borns – score the highest. The next eldest, the October 1993 borns, score a bit less. And so on. (There are older students who take the exam – the ones born before September 1993 – but many of these are failed students from the previous year, introducing a bias in the results.) Perhaps this initial advantage that the elder students have over their classmates continues through the years? Whatever the reason, it’s clear that if your child is born in September, he or she already has a 100 mark advantage! Comments abhijeet 16 May 2011 8:14 am: "Whatever the reason, it’s clear that if your child is born in September, he or she already has a 100 mark advantage!" But at the cost of being older and "losing" 8-9 months! Shankar V 16 May 2011 10:00 am: Nice...!! Thank God I was born in July! If not on the crest, I at least made it into the upward curve! ;) S Anand 16 May 2011 10:21 am: @abhijeet: That reminds me of the "Life is a race" dialogue in 3 Idiots :-) Well, not sure how bad being a bit older is. I personally don't mind, but maybe others do? Shailesh 16 May 2011 10:33 am: Oh...now I know why I was getting good marks.... :-) I am born in September...and my headmaster changed my birthdate (to Jul month) in school register to not waste my 8-9 months..... ravi atluri 16 May 2011 10:47 am: They made me study LKG "TWICE", because I was born in September and didn't have enough time to be 3! and I have been an anomaly to the september-born-high-performers trend. If the marks are related to the mugging capacity, which if related to age would explain the graphs only for the schooling during. Would like to see, how would these change in college and bachelors :D One more doubt, I have is, many parents have the dob's of after-aug-born's to sometime between may to june. If that's the case, the months on the graph have to be offset ;) S Anand 16 May 2011 12:35 pm: @Shailesh -- if you got into school early, then statistically, you had a DISadvantage. So guess you were MUCH brighter than your classmates! @Ravi -- true. No idea how to correct for the real date of birth, though... k 17 May 2011 2:53 am: this is similar to the paper on Israeli schools where an externally imposed rule created a disproportionate class size, which helps in identifying the effect of class size on performance. date of birth can help in identifying the effect on performance then. Vasant 17 May 2011 6:14 am: Ha! Now I know why I scored so less in the boards. It's when I was born! :-) Jokes aside, brilliant analysis. Sathya 17 May 2011 6:43 am: Good One ! If you still have access to data, would it be possible to plot the same graph against their "numerology numbers" ? ie., 28th born would be 2+8 = 10 => 1+0 = 1. I used to wonder that astrology ( perhaps numerology ) could provide certain insights and that it astro/numero predictions should be verified using statistics. ( Pls note that I do not suggest astro determines a person's fate. But perhaps it can predict events with some accuracy. How a person reacts to those events are not in the realm of astrology). I would love to do this if I had access to the data. In case you find it worthwhile, it would be interesting to see if numero number matters at all ! Swetha 19 May 2011 9:34 am: M a September born and a gold medalist . (\Grins\) Sumit Dhar 21 May 2011 3:36 am: And, I know this is extremely difficult, but was wondering if it would be possible to assess the impact of month in which you were born on your professional growth / salary growth. Or even more simply, if marks are correlated with professional success. We Indians tend to believe that good marks == success at work. Would love to know if data validates this belief. If only this kind of data were available... Maybe HR teams in organizations could perform such an assessment. Cheers, D. Vish 16 Aug 2011 9:59 am: http://news.bbc.co.uk/2/hi/8227268.stm Summer-borns tend to lose out at school because, in England and Wales, 1 September is the cut-off date for school entry, so they can be up to a year younger than their classmates. June baby Frank Lampard Research by the Institute for Fiscal Studies two years ago found only 53% of girls and 47% of boys born in August reached the expected educational level at age seven in state primary schools in England, compared with 80% and 70% of those born in September. Their sporting prowess is affected too. Older children tend to be bigger and more confident, receive more encouragement, and their advantage grows over time. For unto everyone who hath shall be given. Among the 25 most capped England football players, 11 were born between September and November, while only one, Frank Lampard, was born between June and August. In his case, having an England international for a father probably outweighed the disadvantage. So for educational and sporting opportunities, the cruellest and kindest months sit next to each other in the calendar like resentful neighbours.

Birthday matters


I built a codecasting tool using Popcorn.js and SyntaxHighlighter to sync audio with code walkthroughs. This lightweight alternative to video allows for copy-pasting and reduced my tutorial preparation time from four hours to thirty minutes.

The best way to explain code to a group of people is by walking through it. If they’re far away in space or time, then a video is the next best thing. You can recommend them to try out the best coding apps as well. The trouble with videos, though, is that they’re big. I can’t host them on my server – I’d need YouTube. Editing them is tough. You can’t copy & paste code from videos. And so on. One interesting alternative is to use presentations with audio. Slideshare, for instance, lets you share slides and sync it with audio. That almost works. But it’s still not good enough. I’d like code to be stored as code. What I really need is codecasting: a YouTube or Slideshare for code. The closest I’ve seen until day-before was etherpad or ttyrec – but neither support audio. Enter Popcorn. It’s a Javascript library from Mozilla that, among other things, can fire events when an audio/video element reaches a particular point. Watch a demo of how I used it for codecasting A look at the code will show you that I’m using two libraries: SyntaxHighlighter to highlight the code, and Popcorn. The meat of the code I’ve written is in this subtitle function. When called like this: ... it takes the #audio element, when it plays to 1 second, highlights lines 1,2,3; at 5 seconds, highlights lines 4,5,6; and so on. Another thing that helped was that my iPad has a much better mic than my laptop, and ClearRecord is a really simple way to create recordings with minimal noise. [Note to self: sampling at 16KHz and saving as a VBR MP3 (45-85kbps) seems the best trade-off.] With these tools, my time to prepare a tutorial went down from 4 hours to half an hour! Comments S Anand 17 Nov 2011 7:58 am: Want to try building one? :-) Tanu 17 Nov 2011 2:52 pm: Good one.. Want to experiment one.. vivek 20 Nov 2011 2:46 pm: Excellent! Will start using this right away! Thanks Anand. Veera 17 Nov 2011 6:30 am: Great idea for an web app/platform!

Codecasting


I used The Hacker’s Diet principles to build an interactive visualization of calorie density versus price. Using Tesco data, I found that custards and jellies are surprisingly better high-volume snacks than most cereals.

A couple of years ago, I managed to lose a fair bit of weight . At the start of 2010, I started putting it back on, and the trajectory continues. I’m at the stage where I seriously need to lose weight. I subscribe to The Hacker’s Diet principle – that you lose weight by eating less, not exercising. An hour of jogging is worth about one Cheese Whopper. Now, are you going to really spend an hour on the road every day just to burn off that extra burger? You don't exercise to lose weight (although it certainly helps). You exercise because you'll live longer and you'll feel better . I’m afraid I’ll live too long anyway, so I won't bother exercising just yet. It's down to eating less. Sadly, I like food. So to make my “diet” work, I need foods that add less calories per gram. Usually, when browsing stores, I check these manually. But being a geek , I figured there’s an easier way. Below is a graph of some foods (the kind I particularly need to avoid, but still end up eating). The ones on the top add a lot of calories (per 100g), and better to avoid. The ones at the right cost a lot more. Now, I’m no longer at the point where I need to worry about food expenses, but still, I can’t quite kick the habit, also you might want to check out this Rootine's comparison of B12 methylcobalamin and cyanocobalamin that will help you in your diet. Hover over the foods to see what they are, and click on them to visit the product. (If you’re using an RSS reader and this doesn’t work, read on my site .) .map { position: relative; width: 600px; height: 600px; } .map a { position: absolute; display: block; width: 6px; height: 6px; font-size: 6px; moz-box-shadow: 1px 1px 2px black; webkit-box-shadow: 1px 1px 2px black; box-shadow: 1px 1px 2px black; filter:progid:dximagetransform.microsoft.dropshadow(color='#222222',offx='1',offy='1'); } .map a:hover { z-index: 100; width:16px; height: 16px; margin-left: -5px; margin-top: -5px; border: 2px solid black; filter:progid:dximagetransform.microsoft.dropshadow(color='#222222',offx='1',offy='1'); } .desserts a { background-color: #a7c0de; border: 1px solid #4f81bd; } .icecream a { background-color: #c0504d; border: 1px solid #632523; } .snacks a { background-color: #9bbb59; border: 1px solid #4f6228; } .cake a { background-color: #8064a2; border: 1px solid #403152; } .cereals a { background-color: #fbcba3; border: 1px solid #f79646; } .map input { position: absolute; right: 0px; top: 0px; } a.hidden { display: none; } (function($){ var timeout = 0; var $foods = $('.map a'); var $search = $('.map input'); function search() { var keyword = $search.val(); keyword = new RegExp(keyword, "i"); $foods.each(function() { var $this = $(this); if ($this.attr('title').match(keyword)) { $this.removeClass('hidden'); } else { $this.addClass('hidden'); } }); } $search.keyup(function(e) { if (timeout) { clearTimeout(timeout); } timeout = setTimeout(search, 200); }); })(jQuery) (The data was picked from Tesco .) It’s interesting that cereals are in the middle of the calorie range. I always thought they’d be low calories per gram. Turns out that if I want to to have such foods, I’m better off with desserts or ice creams ( profiterole , lemon meringue or tiramisu ). In fact, even jams have less calories than cereals. But there are some desserts to avoid. Nuts are a disaster. So are chocolates . Gums , dates and honey are in the middle – about as good as cereals. Salsa dip seems surprisingly low. Custards seem to hit the sweet spot – cheap, and very low in calories. Same for jellies . So: custards and jelly. My daughter’s going to be happy. Comments Srinivas 2 Jun 2011 2:30 pm: well looking at the graph, the answer is simple. just dont go to Tesco! and seriously , the most important rule of thumb to eat right is, never eat anything that comes in a packet. Sathya 20 May 2011 8:11 am: Saar what about "arachi vitta kozhlambu", thovaiyal, medu bonda, Rasa vadai , usili ... and items form "Grand Sweets and Snacks" ? Where are these ? these are my staples ;-( jk ... Nice one ( some surprises as well ). Sumit Dhar 20 May 2011 4:34 pm: Anand, I know it sound crazy but increasingly research is showing that not all calories are equal. What that means is that, a calorie from carbs might result in greater fat / weight gain thank a calorie from proteins. (Reference: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC506782/) Additionally, processed sugar is something you should avoid. Sugar causes insulin spikes that results in greater fat storage. Sames goes for carbs. Even the desserts you mention, should ideally be avoided. (Reference: http://www.youtube.com/watch?v=dBnniua6-oM) On the other hand, I am consuming quite a bit of nuts on a daily basis: almonds, cashews, peanuts. They are a good source of proteins and vital fatty acids. If you are a veggie, cottage cheese is another good source of proteins (along with fat too). Milk is another source of proteins along with calcium. Why we get Fat by Gary Taubes is a good book to check out. While I am not a big believer in rapid fat loss or fast weight reduction, I have read good things about the Rapid Fat Loss Handbook. It reiterates ideas from Why we get Fat. In short, don't focus on calories. Focus on grams of carbs consumed per day. Cheers, Sumit D 26 May 2011 6:03 am: Anand, I would recommend you also consider one major strategy - nutrient timing, which I used to good effect over the last 6 months while consuming the same no. of calories. Also if you really like chocolate, you can try dark chocolate as an alternative - lower calories per gram. Even for vegetarians, protein shakes derived from whey and soy are a great alternative to other foods. I have employed each of these and can vouch for their effectiveness in managing weight Shankar V 27 May 2011 9:00 pm: I agree with you. Nothing like cutting on food to lose weight. Exercise requires perseverance and focus....I lack both. :) Cutting down on food also has issues.....you reduce intake for a few days, then pop comes a wedding or function where you get 'ilai potta sappadu' and all control is lost. You tell yourself, just this once and then very soon you are back to your original eating habits. Best option - must be born with a body that does not put on weight. Period!!! I envy those folks. :) Saurabh Mittal 15 Jan 2014 12:04 pm: Nice one Anand! I was at 82 kgs a few years back and managed to come down to 72 . This was a result of both exercise (have run two half marathons so far in 2012 and then again in 2013) and change of eating habits (more of proteins and less of carbs). There is a nice book as well by Tim Ferris (The Four Hour Body), but in short definitely look at incorporating more exercise in your routine..Also as Sumit pointed out above - processed sugars and salt are the two major culprits! We don't even realize how these are found in plenty in most of the foods! Keeping track of weight on a daily basis definitely is a good habit! Cheers Saurabh

Eating more for less


I compared data processing speeds using Python, cut, and awk across different hardware. I discovered that choosing the right combination of tools and server environments can improve performance by over 250x compared to standard local scripting.

I’ve been playing with big data lately. The good part is, it’s easy to get interesting results. The data is so unwieldy that even average value calculations provoke a “Amazing! I didn’t know that,” response (No exaggeration. I heard this from two separate $1bn businesses this month.) The bad part is that calculating even that simple average is slow. For example, take this 40MB file (380MB unzipped) and extract the first column. The simplest Python script to get the first column looks like this: That took a good 3 minutes to execute on my laptop. Since I’m used to UNIX data processing , I tried cut -f1 . Weirdly, that’s worse. 5 minutes. Paradoxically, awk '{print $1}' only takes 17 seconds. That's about 12 times faster. Clearly the tool makes a big difference. And we always knew UNIX was fast . But I also ran these on an Amazon EC2 server, and a Hostgator server. Here’re the results. python cut awk My Dell E5400 3:04 ( 1x ) 5:42 ( 0.5x ) 0:17 ( 11x ) EC2 standard 0:33 ( 6x ) 0:5.6 ( 33x ) 0:16 ( 11x ) Hostgator 0:19 ( 10x ) 0:2.5 ( 74x ) 0:0.7 ( 265x ) What took 3 minutes with Python my Dell E5400 took less than a second on Hostgator’s server with awk. Over 250 times faster. (Not 250%. 250 times ). And it’s not just hardware. A good tool (awk) made things 11x faster on my machine. Good hardware (hostgator) made the same program 10x faster. But choosing the right combination can make things go faster than 11 x 10 = 110 times. Much faster. There are a few of things I’m taking away from this. Good hardware can speed you up much as (or more than) choosing the right tool. Good hardware can be rented. From many places. Cheaply. Always test what’s fast. awk’s fastest on my machine and Hostgator, but not on EC2.

Faster data crunching


I demonstrate how to improve Excel table formatting through specific visualization tips. Use these basic methods to enhance clarity and professional appearance beyond the standard software defaults.

Formatting tables in Excel is a fairly common task, but there are a number of ways to improve on the way it's done most of the time. Here are a few tips. Fairly basic stuff, but hopefully useful. Comments Neela 18 Aug 2011 6:16 pm: Thanks a lot for the tips! I think there might be a small error in the video posted above, since the last part about conditional formatting is repeated twice. Very useful nonetheless! Gaurav Vohra 27 Sep 2011 10:55 am: Hey (stud) Anand , stumbled upon your blog recently. It is a great read. Lou Reed said "between thought and expression, lies a lifetime". I think you bridge that gap really well. You can add me to your list of avid followers now. :) I would especially recommend your blog to anyone who wants to get into the field of business analytics (all my students :) )

Formatting tables


I identified the "Cheerful Mandolin 07" Apple Loop from GarageBand in A.R. Rahman's song "Phir Se Ud Chala" from the movie Rockstar. I recognized the sample after using it myself in a project on my iPad.

A month ago, I was at the theatre watching Ra.One. The movie was terrible, yet enjoyable. But I’m going to talk about something else – a song I heard that caught my imagination. The song is Phir Se Ud Chala from Rockstar). Around 14 seconds into the video, you’ll hear a guitar start off at the background. That’s what caught my ear first – because I’d heard it before. Listen to this piece below: Mystic light I’d created this a couple of months ago with GarageBand on my iPad2. It just plays two Apple Loops one after another. photo The first one that you hear – Cheerful Mandolin 07 – is exactly the same background music that you hear in Phir Se Ud Chala. Guess A R Rahman uses GarageBand too! (The strange thing is, I found no mention of this anywhere on the internet, as of 2 Dec 2011. Thought I’d have a go and be the first… just in case someone searches for Apple Loops or GarageBand in Phir Se Ud Chala from Rockstar.) Comments Dinakaran 3 Dec 2011 4:21 pm: http://www.apple.com/logicstudio/in-action/arrahman/index2.html Viju 14 Dec 2011 4:27 pm: The second loop is from Shauk Hai (Guru). Is this an Apple loop too? Dinakaran 3 Dec 2011 4:21 pm: It could be that GarageBand came out as a product out of Logic Pro that ARR is using with similar sounds ? S Anand 14 Dec 2011 9:17 pm: @Viju -- you're right! http://www.youtube.com/watch?v=k1vAu0c96yY#t=152s

GarageBand in Phir Se Ud Chala


I reviewed the HTML 4 & 5 Complete Reference app for iOS. While I liked its task-based tag groupings and attribute details, the poor search functionality for HTML entities makes me prefer using the official PDF spec.

HTML-4-and-5-The-Complete-Reference HTML 4 & 5: The Complete Reference is an iPhone / iPad app that does exactly what it says: a reference for HTML 4 and 5. It has a list of all tags, clearly demarcated as HTML4, HTML5 or both. The application is fairly easy to scroll through to find the tag or attribute you want. Clicking on a tag, you get: a brief description of what it’s for what attributes are valid – the good part is you can see clearly which attributes are specific to the element, and which ones are common (like class, id, etc.). You can also see the possible values for the attribute, which helps. and an example of how the tag is used. The examples are quite simplistic, and there’s only one per tag, but it does have a rendered version of the code, which helps. You can also scroll through the list of attributes and see which tags they’re valid for. The part that quite interested me was the “characters” or HTML entities. Quite often, I’d want the pound (£) or right angle quotes (»), but wouldn’t know the character or entity reference. So far, I’ve been using this HTML entity reference to search for characters, where I can just type in the word (e.g. pound or quote) and it filters the list to show what I want. I was really hoping to see that on the app, but was disappointed. It lets you search, but it’s not search as you type. And the result points you to a section that contains the character – not directly to the character. (It’s a bit difficult to find the character in the longer sections). There’s also a section where you can see elements by “task” – e.g. Forms, Link-related, Document Setup, Interaction, etc. This is a pretty useful break-up if you’re looking for the right element for the job, or browsing for interesting new elements to discover in HTML5. (I found the and tags this way. You do have the option of just downloading the PDF version of the HTML5 spec and reading it in iBooks, of course. So while I find the book useful, without a search-as-you-type feature, I suspect it won’t do much for my speed of looking up things, so I’ll just stick to the spec for the moment. Disclosure: I’m writing this post as part of O’Reilly’s blogger review program. While I’m not getting paid to review the app, I did get it for free.

HTML 4 & 5: The complete Reference


I created an SVG map of India's 640 districts by tracing the 2011 census administrative PDF using Inkscape. This map allows for district-level data visualization, such as plotting historical temperature variations across the country.

I put together a district map of India in SVG this weekend. So what? You can now plot data available at a district level on a map, like the temperature in India over the last century (via IndiaWaterPortal). The rows are years (1901, 1911, … 2001) and the columns are months (Jan, Feb, … Dec). Red is hot, green is cold. temperature (Yeah, the west coast is a great place to live in, but I probably need to look into the rainfall.) districts.svg has has 640 districts (I’ve no idea what the 641st looks like) and is tagged with the State and District names as titles: How? I made it from the 2011 census map (0.4MB PDF). I opened it in Inkscape, removed the labels, added a layer for the districts, and used the paint bucket to fill each district’s area. I then saved the districts layer, cleaning it up a big. Then I labelled each district with a title. (Seemed like the easiest way to get this done.) Thanks to @planemad, @gkjohn, @arjunram for inputs. Play around. Feedback welcome. Comments Thejesh GN 22 Jul 2011 9:27 am: Thanks for the map. Very useful. Btw you can cname map your bitbucket account. Its free. S Anand 23 Jul 2011 9:39 am: @Thej -- AHA! Didn't know about the CNAMEing bitbucket. Thanks. Rishi 23 Jul 2011 9:15 am: Wonderful stuff Anand!!! I would say - upload this on Wikipedia - but then we would have several flamewars with our dear neighbours on the borders shown. Arulalan.T 29 Dec 2011 10:17 am: Thanks a lot for the India District Map in pdf and svg. From this post I got few ideas to generate shape (shp) files. I edited the India map in CDAT [1] shape files by generate the lat,lon points by hand itself [2]. I want to develop the India district shape file by automatic. Your project and code lights in my way. Thanks a lot. [1] http://www2-pcmdi.llnl.gov/cdat [2] http://tuxworld.wordpress.com/2010/02/13/how-to-edit-world-map-in-cdat-documentation/ Dibyo 11 Aug 2011 4:29 am: Awesome - I've struggled with this for a while now. Minh 12 Dec 2012 8:11 pm: Thanks a lot. Do you have the shapefiles for your svg data? I could not find the new district level (2011 census) anywhere? Can you share with me? Many thanks. mahir 19 Dec 2012 6:11 am: Dear Anand, Great work. I have made a tool to translate these location names using wikipedia's toolserver. people can translate these svg maps from http://toolserver.org/mahir/ I have successfully translated India political map from english to tamil for tamil wikipedia. http://ta.wikipedia.org/wiki/படிமம்:Ta\India\National\Highway\17.png Please try once. thanks Arijit Upadhyay 26 Dec 2013 3:37 am: Some district names have changed since you posted this. Do you have an update for this map. Or how to edit them. Linkscape destroys the XML tags. Prakash 18 Jan 2014 10:43 am: Dear Friend would it be possible to get shp and dbf file of indian state with undivided bihar, up and mp.

India district map


I compared JavaScript arrays and objects for storing large datasets. I found that while execution speeds are now similar, arrays are significantly more memory-efficient, using only 8MB compared to 57MB for objects with one million entries.

Summary : Arrays are a lot smaller than objects, but only slightly faster on newer browsers. I’m writing an in-memory Javascript app that handles several thousand rows. Each row could be stored either as an array [1,2,3] or an object {"x":1,"y":2,"z":3} . Having read up on the performance of arrays vs objects , I thought I’d do a few tests on storing numbers from 0 to 1 million. The results for Chrome are below. (Firefox 7 was similar.) Time Size (MB) Array: x[i] = i 2.44s 8 Object: x[i] = i 3.02s 57 Object: x["alongdummytestingstring"+i]=i 4.21s 238 The key lessons for me were: Browsers used to process arrays MUCH faster than objects. This gap has now shrunk. However, arrays are still better: not for their speed, but for their space efficiency. If you’re processing a million rows or less, don’t worry about memory. If you’re storing stuff as arrays, you can store 128 columns in 1GB of RAM (1024/8=128). Comments dineshsaravanank 11 Oct 2011 5:27 am: Also, I got completely different numbers than yours. Array x[i]=i, 43ms | Object x[i]=i, 43ms | Object x["akey"+i]=i, 2443ms. I tested in chrome, as you can see the keyed object is \significantly\ slower in my case... S Anand 11 Oct 2011 10:38 pm: @dinesh: When object keys are integers, browsers tend to optimise them in an array-like fashion. That explains the {1:1, 2:2} being faster than {"a1":1,"a2":2}. @kalpesh, the main point of this,actually, was that performance is increasingly less of a worry for most applications. Browsers are fast enough. So the choice of language is becoming increasingly less performance driven and more ease driven. dineshsaravanank 11 Oct 2011 5:18 am: After reading this I was curious about 1) How is the read performance, 2) How does the object with long key perform. So I ran some tests. And, 1. as expected reads are much faster than the writes and there is no significant difference between array and object when it came to read. 2. Longer keys take more time to execute, smaller keys are significantly faster than very long keys. But there are some surprises here. An object {"1":1,"2":2,"3":3} is much faster than {"a1":1, "a2":2, "a3":4}, while object {"a1":1,"a2":2} and object {"alittlelong1":1, "alittlelong2":2} are almost same. You can see the tests here http://jsfiddle.net/ETAb9/ Kalpesh 12 Oct 2011 4:56 pm: I agree. My question was to think of it, in terms of memory consumption. Your thoughts? Kalpesh 10 Oct 2011 10:50 am: Considering this, do you think OO based languages add overhead to space it occupies when run, in memory? Arun 13 Oct 2011 6:09 pm: Anand, This is because object is a fundamental data type in Javascript and an Array is implemented as a special kind of object. Hence the lookups (the Time column) would be implemented as a hash map lookup in both cases. However the storage could be an interesting case. As one of the early gotchas of Javascript teaches us: Person.Name is equivalent to Person['Name'] But in the case of Array's some magic is applied leading to optimisations in storage. Hence I suppose rather than storing unicode strings for "0", "1" etc., they are stored as integers. Javascript, like English, is berry phunny language :) Joe 29 Feb 2016 1:02 pm: I ran some tests and was surprised that array access is only slightly faster. Something strange has to be going on with that. My test compares numeric entries either way. The object implementation normally has the extra work of casting to string and hashing. I have a feeling it will be different with non-numeric versus numeric keys. Still, it hints at something strange going on.

Javascript arrays vs objects


I built a crowd-sourced platform to map Indian PIN codes after finding no reliable open databases. I used GeoNames and OpenStreetMap data to seed the project, allowing users to contribute coordinates under a Creative Commons license.

I haven’t found an open or reliable database providing the geo-location of Indian PIN codes. That’s a bother if you’re creating geographic mash-ups. The closest were commercial sources: a PIN code directory from the Postal Training Centre for Rs. 2,000, which probably just contains a list of PIN codes, and a PIN code map from MapMyIndia for Rs. 1,00,000, whose quality I’m not sure of. (I spoke to one of their sales representatives who mentioned that the data was gathered via companies such as Coca Cola, using their local distribution knowledge, perhaps GPSs.) Crowd-sourcing this might help. Here’s a site where you can map the location of any PIN code you know: pincode.datameet.org For example, if you knew the exact location of the PIN code 110083 (which happens to be Mongolpuri in New Delhi), just go to and move the marker to where it should be. I’ve initially populated the data from GeoNames. Arun has offered OpenStreetMap data. If you know of any sources that we could use, please let me know. And if you want to use the data, feel free. It’s CC licensed. You can check out the source on github too. Comments Arun Anantharaman 8 Mar 2011 3:14 am: Not sure if I am doing something wrong, but am unable to edit on the pincode.datameet.org page. Get an error if I click on the "go" button. Vanaja 7 Mar 2011 5:29 am: A searchable database by Bharatiya Dak!http://www.indiapost.gov.in/pin/pinsearch.aspx Prashanth 11 Mar 2011 7:00 am: Great initiative. Blogged about it over here - http://hellogeek.in/2011/03/crowd-sourced-pin-code-geomapping-project-lets-contribute/ Prateek 14 Oct 2012 1:57 am: great work! But for some reason I could not view the markers on map. Is this browser not supported: Chromium on Linux ? Rakesh Patil 27 May 2014 7:57 pm: Hi All, I was wondering if there is a database dumb which will give me list of all pin codes surrounding a particular pin code, do we have any such database? if not should we start crowd sourcing?. Please let me know if I am not clear with my question. Regard Rakesh Prabaker 10 Jun 2014 7:32 pm: Great! Check this out: http://www.gadm.org/country Any progress on the PIN code thing? I have a few questions for you. Maybe I will send them your way, once I figure out your email. vishal kumar 15 Mar 2016 12:14 pm: the website has been removed. i'm trying to find the coordinates of pincodes, i have the list of entire pincodes but where i found there coordinates

Mapping PIN codes


I built Markdress to instantly render Markdown files from any URL—like Dropbox public links—into formatted HTML. I wanted a simple way to view and share my documents across my iPad and iPhone without needing extra accounts.

This year, I’ve converted the bulk of my content into Markdown – a simple way of formatting text files in a way that can be rendered into HTML. Not out of choice, really. It was the only solution if I wanted to: Edit files on my iPad / iPhone (I’ve started doing that a lot more recently) Allow the contents to be viewable as HTML as well as text, and Allow non techies to edit the file As a bonus, it’s already the format Github and Bitbucket use for markup. If you toss Dropbox into the mix, there’s a powerful solution there. You can share files via Dropbox as Markdown, and publish them as web pages. There are already a number of solutions that let you do this. DropPages.com and Pancake.io let you share Dropbox files as web pages. Calepin.co lets you blog using Dropbox. My needs were a bit simpler, however. I sometimes publish Markdown files on Dropbox that I want to see in a formatted way – without having to create an account. Just to test things, or share temporarily. Enter Markdress.org. My project for this morning. Just add any URL after markdress.org to render it as Markdown. For example, to render the file at , visit . To test it out, create any text file in your Dropbox public folder, get the public link and append it to without the http:// prefix. Comments Gee 14 Jan 2012 12:38 am: Great news. And all the best in your new ventures. Meanwhile, I made a simple bookmarklet for you: https://gist.github.com/1609547 Thejesh GN 5 Dec 2011 6:47 am: Check this http://markdoc.org I have been using it locally for sometime. Its cool. S Anand 4 Dec 2011 12:18 pm: James, your suggestion on following URLs is spot-on! I'll put this in (one of these days, when I get back to this...) S Anand 6 Dec 2011 10:10 pm: You need to have an empty line before a list. For example: Test This is a list: \ one \ two \ three Michael Lajlev 5 Dec 2011 2:12 pm: I seem to struggle getting list items to work http://markdress.org/dl.dropbox.com/u/623576/test.md Any idea? Gee 12 Jan 2012 9:14 pm: Markdress is great. I have it running on my server. I would like to render files in a directory adjacent to, or inside the app folder, instead of entering a URL path each time. I would like to enter just filename.txt and it would default back to (effectively) /markdress/filename.txt I looked over index.php and tried a few things but I can't get relative files to render. Is this an enhancement you've considered? Thanks for the great tool. S Anand 13 Jan 2012 12:51 am: I personally use this sort of a variant, but havent gotten around to making the same changes on markdress. I will do that over the next few days, once I settle down. The Bootcamp Blog Begins | JFDI–Innov8 2012 Bootcamp 3 Dec 2011 8:16 pm (pingback): [...] kind of people who can whip up something like Markdress in a [...] James Manning 4 Dec 2011 9:08 am: Awesome! Nicely done! Not sure if it's by design or not, but FWIW the example given seems to show that relative url's aren't handled correctly - the bottom of this page: http://daringfireball.net/projects/markdown/index.text has this relative url: [tfmenu]: /graphics/markdown/mt\textformat\menu.png which gets faithfully rendered into: That fails, of course, with the attempted get operation returning: http://markdress.org/graphics/markdown/mt\textformat\menu.png Could not fetch http://graphics/markdown/mt\textformat\menu.png FWIW, inserting a base tag in the head seems to work fine, like: However, it would have to be the 'real' url in the base - IOW, this doesn't work: On a related note, IMHO it seems like markdress should handle redirects and 'pass through' as a redirect to the markdress version. IOW, looking at this example request from the post: http://markdress.org/goo.gl/zTG1q When markdress requested the page and got back a redirect, instead of following it, it should have instead passed back a redirect to: http://markdress.org/daringfireball.net/projects/markdown/index.text IMHO, that's a better user experience as it's more clear what's getting served up (and it would reflect in the title as well). If/when caching was added (assuming it's not there now), then you'd want to have the canonical url for targets anyway. :) That behavior isn't required to fix the relative-url issue, but since you'll have to know what requests redirect to (the call may be implicitly following it now, so you don't currently know?), then you'll have the necessary info to do this piece as well. :) Thanks again for marking markdress! It's definitely going to be incredibly useful! Jeff Byrnes 6 Dec 2012 3:14 pm: Markdress appears to have expired, any chance of its return? S Anand 7 Dec 2012 3:53 am: I didn't think anyone was using it -- so I just let the domain expire. Alternatives are at https://github.com/sanand0/markdress#alternatives. You could also host the code yourself.

Markdress


I use Excel's NORMINV function and scatterplots to visualize how moderating school marks works. I demonstrate a scaling formula that adjusts means and standard deviations to make internal and public exam scores comparable across different difficulty levels.

Sometimes, school marks are moderated. That is, the actual marks are adjusted to better reflect students' performances. For example, if an exam is very easy compared to another, you may want to scale down the marks on the easy exam to make it comparable. I was testing out the impact of moderation. In this video, I'll try and walk through the impact, visually, of using a simple scaling formula. BTW, this set of videos is intended for a very specific audience. You are not expected to understand this . Rough transcript First, let me show you how to generate marks randomly. Let's say we want marks with a mean of 50 and a standard deviation of 20. That means that two-thirds of the marks will be between 50 plus/minus 20. I use the NORMINV formula in Excel to generate the numbers. The formula =NORMINV(RAND(), Mean, SD) will generate a random mark that fits this distribution. Let's say we create 225 students' marks in this way. Now, I'll plot it as a scatterplot. We want the X-axis to range from 0 to 225. We want the Y-axis to range from 0 to 100. We can remove the title, axes and the gridlines. Now, we can shrink the graph and position it in a single column. It's a good idea to change the marker style to something smaller as well. Now, that's a quick visual representation of students' marks in one exam. Let's say our exam has a mean of 70 and a standard deviation of 10. The students have done fairly well here. If I want to compare the scores in this exam with another exam with a mean of 50 and standard deviation of 20, it's possible to scale that in a very simple way. We reduce the mean from the marks. We divide by the standard deviation. Then multiply by the new standard deviation. And add back the new mean. Let me plot this. I'll copy the original plot, position it, and change the data. Now, you can see that the mean has gone down a bit -- it's down from 70 to 50, and the spread has gone up as well -- from 10 to 20. Let's try and understand what this means. If the first column has the marks in a school internal exam, and the second in a public exam, we can scale the internal scores to be in line with the public exam scores for them to be comparable. The internal exam has a higher average, which means that it was easier, and a lower spread, which means that most of the students answered similarly. When scaling it to the public exam, students who performed well in the interal exam would continue to perform well after scaling. But students with an average performance would have their scores pulled down. This is because the internal exam is an easy one, and in order to make it comparable, we're stretching their marks to the same range. As a result, the good performers would continue getting a top score. But poor performers who've gotten a better score than they would have in a public exam lose out. Comments Vinu 13 Apr 2011 8:17 pm: Nice one... your analysis does assume that the performance of a group of students in an easy exam and the performance of the same group of students in a tough exam will both have similar distributions... I am sure you will recall how the loss of resolution in marks (marks were awarded in increments of no less than 1/2) and the clustering of results at the top end ( remember our 10th & 12th public results in '90 & '92 respy) led to very skewed distribution? In tough exams, I would expect that the skew would shift downward with a lot more positive Kurtosis... so besides the normalization of the exam scores, it would probably be necessary to perform some additional transformations to really compare 2 exams of different 'toughnesses'. All this, without even considering the 'performance factor' from the student introducing its own variations... After writing all this, I do realize that my points are perhaps academically motivated - but if all we want to do is to help teachers pick out the students who are more likely to need 'extra tuition assistance' to make a better showing in a tough public exam, then the basic normalization is sufficient. Of course, most good teachers would say not need statistics to tell them that - they know their students well enough to understand who needs help and who is on autopilot to ace the exams... (ring any bells ? ;) )

Moderating marks


I built a PHP library and .htaccess configuration to protect static files using OpenID authentication. It allows access only to specific Google email addresses, combining the simplicity of static sites with secure access control.

I moved from static HTML pages to web applications and back to static HTML files. There’s a lot to be said for the simplicity and portability of a bunch of files. Static site generators like Jekyll are increasingly popular; I’ve built a simple publisher that I use extensively. Web apps give you something else, though, that are still useful on a static site. Access control. I’ve been resorting to htpasswd to protect static files, and it’s far from optimal. I don’t want to know or manage users’ passwords. I don’t want them to remember a new ID. I just want to allow specific people to log in via their Google Accounts. (OpenID is too confusing, and most people use Google anyway.) The easiest option would be to use Google AppEngine. But their new pricing worries me. Hosting on EC2 is expensive in the long run. All my hosting is now out of a shared Hostgator server that offers Apache and PHP. So, obviously, I wrote a library protects static files on Apache/PHP using OpenID. Download the code Say you want to protect /home/www which is accessible at . Copy .htaccess and auth/ under /home/www. In .htaccess, change RewriteBase to / In auth/, copy config.sample.php into config.php, and change $AUTHPATH to add permitted email IDs to function allow() Now, when you visit , you’ll be taken to Google’s login page. Once you log in, if your email ID is allowed , you’ll be able to see the file. Feel free to try, or fork the code. Comments S Anand 19 Nov 2011 10:31 am: I could. But what I've learnt is that while the static content on this site has survived 14 years, moving from host to host, the same can't be said of ANY of the apps I've written to create the site. I've moved from shell scripts to Perl to Python to node.js... and something else will come up. I can't maintain this stuff. Fortunately, HTML will stay. So I'm making the content primary. I just have a bunch of static files, and that's the key. This is an OPTIONAL library that sits on the side. Without it, the worst that'll happen is that the content becomes public. But otherwise, all URLs will remain unbroken. That's a big win with this approach. That's also the reason I rejected AppEngine. For my volume of usage, pricing isn't a serious issue. It's simplicity. Manu 19 Nov 2011 5:59 am: If you are willing to host it outside, then heroku allows one free instance per project. S Anand 2 Dec 2011 10:09 pm: I too will probably stay on WordPress for a while. But recently, I've been creating quite a few microblogs for various topics, and find that the workflow of Markdown -> Dropbox -> PHP -> HTML far too compelling to ignore. It's worked quite well from an ease of publishing perspective. I'm still ironing out few edges. Will publish once I do. Thejesh GN 25 Nov 2011 11:56 am: I was planning to move to static using http://www.blogofile.com/ (python and variety of templating systems are supported). But I am still betting on wordpress for many things. One day I will probably move to static blog. Christophe-Marie Duquesne 22 Dec 2013 12:53 pm: Hi, Your post inspired me to write a lighttpd magnet script that protects content regardless of its nature (static or dynamic). I use it: for protecting my (static) photo gallery and only giving access to people I select for protecting my (dynamic) rss reader and only giving access to myself for protecting my (dynamic) online file manager and only giving access to my girlfriend and myself. It is completely pluggable, and it sets the REMOTE\USER server variable so that user-aware applications can use this mechanism to authentify you. https://lighttpd-external-auth.chmd.fr/

Protect static files on Apache with OpenID


I share random reflections on the freedom of blogging, the beauty of node.js documentation, and my typography preferences. I also discuss technical frustrations with git and a reminder for programmers to prioritize their physical health.

The whole point of a blog is to be able to write what I want, isn't it? Without the need to be coherent. Intelligent. Useful. I read somewhere, in a top 10 list of advice to programmers -- to avoid constipation. It's actually good advice. It gives you a headache for the rest of the day. That piece of code is really not worth it, trust me. But then, when you have a headache, there isn't anything quite as beautiful and relaxing as code. Or good documentation. Like node.js docs, for example. Just finished seeing Casino Royale. Usual average movie. 3,3,1. I hate that git doesn't work. I hate that my mobile broadband dongle doesn't work. I hate that I have too much to do. I love beautiful typography. line-height:1.5 and a nice Georgia to go with it. (Yeah, Ram had this one right: madness is not a result of being in GramEner. It's a pre-requisite.) Well, dinner's ready.

Random notes


I find smart recruits by searching GitHub for specific locations and skills like Node.js. I also look through comments on relevant industry blogs and use tracked email templates to measure my outreach success.

Find people. 1. Search on github by location and skill. 2. Anand's blog comments Reach out to people. 1. Have a standard set of template, and track the template's success.

Recruiting smart people in practice


I benchmarked Node.js against PHP, Tornado, and Nginx using a simple Hello World test. I found Node.js surprisingly faster than Nginx serving a static file, achieving 2,500 requests per second using the Apache Benchmark tool.

Yesterday, I wrote about node.js being fast. Here are some numbers. I ran Apache Benchmark on the simplest Hello World program possible, testing 10,000 requests with 100 concurrent connections ( ab -n 10000 -c 100 ). These are on my Dell E5400, with lots of application running, so take them with a pinch of salt. PHP5 on Apache 2.2.6 <?php echo “Hello world” ?> 1,550/sec Base case. But this isn’t too bad Tornado/Python See Tornadoweb example 1,900/sec Over 20% faster Static HTML on Apache 2.2.6 Hello world 2,250/sec Another 20% faster Static HTML on nginx 0.9.0 Hello world 2,400/sec 6% faster node.js 0.4.1 See nodejs.org example 2,500/sec Faster than a static file on nginx! I was definitely NOT expecting this result… but it looks like serving a static file with node.js could be faster than nginx. This might explain why Markup.io is exposing node.js directly , without an nginx or varnish proxy. Comments Olivier 12 Dec 2011 3:39 am: Nginx has to read the file from the disk each time. Nodejs has the content of the file in memory. So it's normal that nodejs is faster than Nginx. Re-test the same thing but setup Nginx so it caches files in RAM and you'll see the truth. Seamsky 4 Jun 2013 11:31 pm: Please, use "return 200 'Hello world';" for nginx instead of serving file from hard drive.

Server speed benchmarks


I updated my essential software list for a new laptop, moving toward browser-based tools, Cygwin, and Portable Apps. I’ve replaced heavy IDEs with Sublime Text 2 and added power utilities like AutoHotKey, ClipX, and R.

Time for a new laptop, and to replace software. Here’s my new list. A lot has changed in the last 5 years. Mainly, I use the browser, cygwin and Portable Apps a lot more. (The last is to escape jailers, not registry bloat.) Media Chrome [new]: For browsing and development. Fast, light, and stays out of the way. Firefox: I keep it just for printing. Chrome sucks at printing. Media Player Classic: Nothing against it, but I decided to stick to just one app, which is… VLC: Continues to be the best media player, IMHO. WinAmp: I just manage my playlists as M3U files, using Python programs. Audacity: Still the easiest way to record audio. Camstudio: The simplest free portable screen capture software I know. PicPick [new]: Lightweight, powerful screenshot grabber VirtualDub: Not the simplest, but still good for what I need: cropping and joining video. MediaCoder [new]: Good for video/audio conversions. Maybe I’ll install this later. Foxit Reader: The simples free portable PDF reader I know, better than… NitroPDF Reader [new]: … which is good for Printing PDFs – better than… Primo PDF: … which has trouble on rare occasions. Microsoft Reader: I have a lot of ebooks in .LIT. Kindle for PC [new]: I don’t own a Kindle, but I’ve bought a few ebooks. Paint.NET: Good enough for cropping and adjusting colours on images. Windows Live Writer [new]: The best way to write this blog WYSIWYG Inkscape [new]: I occasionally edit vector graphics. Google Earth. Google Maps is good enough. ImgBurn: I no longer use CDs/DVDs. Just flash drives and external hard disks. Picasa: I’ve stopped browsing pictures. No time. Sharing Dropbox [new]: Simplest way of sharing files. Skype: I use it more than my phone. Google Talk: For those friends who have chat enabled on Gmail. TeamViewer [new]: Pretty efficient screen sharing. Works better than Skype, I think. Google Calendar Sync: To keep Outlook in sync with Google Calendar. Utilities 7-Zip [new]: Covers all compressed formats, and has the best compression ratio. WinRAR: 7-Zip has it covered. AutoHotKey [new]: Shockingly powerful macro functionality. Shockingly underused. Clip [new]: Command line clipboard. dir | clip copies the directory to the clipboard. ClipX [new]: Stores multiple clipboard entries and history. Invaluable. DiskTT [new]: I’m paranoid about disk speed. I keep measuring it. WinDirStat [new]: Best way to find what’s taking up space on disk. ProcessExplorer [new]: Just in case Task Manager doesn’t show you everything. Google Desktop: Well, it’s dead. mDesktop [new]: A Virtual Desktop Manager (multiple screens) for Windows 7. PowerToys: doesn’t work on Windows 7, but I got X-Mouse working. Teracopy: I don’t worry too much about copying files any more. Maybe later. Junction Link Magic [new]: To map folders. But I now use Cygwin, and symlinks rock. uTorrent [new]: For bittorrent. ntlmaps [new]: proxies requiring a password to a proxy not requiring a password Putty [new]: SSH for Windows, but can also act as an SSH tunnel TrueCrypt [new]: To securely back up my bank details on the cloud. Development Cygwin: UNIX on Windows. I also install make, curl, lynx, wget, tidy, mercurial, git, openssh, rsync, optipng, pdftk, sqlite3, imagemagick and sgrep. Also cacert.pem. ActivePython: My primary programming language. I also install NumPy, SciPy, ipython, tornado, tabular, lxml and eyeD3. ActivePerl: … isn’t as readable. node.js): Soon becoming my favourite programming language. Fast, popular. XAMPP [new]: Fastest way of getting Apache + MySQL + PHP running nginx [new]: Faster than Apache, but no CGI, and I like CGI. Google AppEngine [new]: To maintain sites like 250. redis [new]: My new favourite database. In-memory, and fast. CouchDB [new]: Simple, persistent JSON store. (Or MongoDB. Both’re fine.) Fiddler2 [new]: To see where traffic is really coming from. IETester [new]: I’ve stopped developing for Internet Explorer 6 or 7. JDK: Sadly, some apps require a Java compiler. (I like the JRE. Can’t stand Java.) Notepad++ [new]: Excellent editor. I’ll probably go back to it some time, but… Sublime Text 2 [new]: … is just a little bit cooler. Crimson Editor: Didn’t have Unicode support a long time ago, so I switched. Subversion. I’ve switched to Mercurial and git. Data Visualisation R [new]. The God of all statistical packages. Install reshape and ggplot2. Gephi [new]: Does network visualisations quite well. GraphViz [new]: Does network visualisations not quite as well. Google Refine [new]: Helps clean up messy data. qhull [new]: For voronoi treemaps. Don’t ask. wkhtml2pdf [new]: To print web pages as PDF. What am I missing that you really like? Comments Scraping for a laptop | s-anand.net 14 Jan 2012 4:01 am (pingback): [...] for a laptop January 14th, 2012 How I do things S Anand I’ve returned my laptop, and it’s time to buy a new one. For the first time in my life, I’m buying a laptop for [...] shiva 10 Dec 2011 11:45 am: Wat abt notepad ++ 408wij 28 Sep 2011 11:56 pm: Evernote Zipeg (available on both Mac and Win, better UI than 7zip) Keepass Switcher (Expose-like feature on Win. A bit flaky and not supported.) Erunt (registry backup) I'm trialing Xplorer2. Arun 13 Oct 2011 6:11 pm: I tried to do something similar and then got lazy and simply installed VirtualBox ;) Rajasekar 4 Nov 2011 11:35 am: Good One. bhavesh 4 Jan 2012 5:24 pm: What about CClener and Unlocker? Naveen 28 Sep 2011 6:26 am: Finally :-) Ravi Atluri (@sonnes) 28 Sep 2011 7:14 am: One more thing, I wanted to know from you is - How do you organize your work, specifically code and how do you handle repository and workspaces (if you use an IDE) ? In case you don't use an IDE, what kind of coding environment? I got a new laptop a few months back and have been trying to figure out how to start organizing my work, repositories of things I work or experiment on and the IDE. Currently, I am trying to put in Java, Python and Scala into eclipse by using it's plugins, hoping eclipse would help me to organize the code and repos. Mohit 30 Sep 2011 8:37 am: At least some of this could be automated :) check out "ninite dot com" Shankar V 30 Sep 2011 3:06 pm: Excellent list. Thanks for sharing :) I agree about Eclipse - too heavy. But if you code in Java, I dont see a better alternative. Anyone remember NetBeans? Google Desktop - I find this amazingly useful and I am sad Google is retiring this. Wish there is a good alternative. I hate windows search. Venu 7 Oct 2011 7:22 am: Thanks for excellent compilation. Can you point to any free utility to optimize the performance of our computers TD 28 Sep 2011 8:32 am: Wordweb and Evernote TD 28 Sep 2011 8:38 am: Also, iTunes.... Not that I like it but for Ipod. S Anand 28 Sep 2011 3:13 pm: @Ravi: I don't use an IDE. Just Sublime Text + bash/Command Prompt + git/hg. Since I don't use Java, I can stay away from Eclipse (which I think is way too heavy for my liking). Karen 17 May 2012 5:23 am: Very nice list! I would add JV 16 Power Tools to the Utilities list. They have a free 30 day fully functional trial to try them out. It's not limited like most trial software. Vinu 11 Aug 2012 11:06 am: If you spend any time in Excel, ASAP Utilities is an absolute joy!... get it and I'm pretty sure, you'll swear by it in no time.. the Sparklines add-in by Fabrice Rimlinger is also excellent.. a bit raw, but powerful nevertheless. Vinayak 23 Jan 2013 7:40 am: Nice list. You should really try out: Xplorer2 (get the professional version if you can - it is $30 and worth every penny and more) from http://zabkat.com Wikidpad - for organizing information in an offline desktop wiki - from http://wikidpad.sourceforge.net/ Sublime Text - the best text editor - http://www.sublimetext.com/ Everything - the perfect replacement for Google Desktop - actually it is better since it does not hog memory - from http://www.voidtools.com/ Slickrun - the perfect program launcher - http://www.bayden.com/slickrun/ MindManager - for Mindmapping - from http://www.mindjet.com/ Irfanview - basic image viewer and editor - http://irfanview.com/ SumatraPDF - Single portable program for reading ebooks (ePub & Mobi), comics (CBR and CBZ) and PDFs - from http://blog.kowalczyk.info/software/sumatrapdf/free-pdf-reader.html Software I currently use | s-anand.net 9 May 2014 6:19 pm (pingback): […] few years, I review the software I use. Here are some of my earlier […] Perl, 1994-2011 - S Anand 28 Sep 2024 10:21 pm (pingback): […] 2011, I stopped installing Perl on my […]

Software for my new laptop 2


I used Wakoopa to track my annual software usage and ranked my top 100 tools by time spent. My results show a heavy reliance on Gmail, Notepad++, and Chrome for my daily communication and coding.

Time for the annual update on software I use. This time, I've got Wakoopa to help me with the relative usage as well. Here’s the top 100 software / web apps I’ve used recently, and how long I spent on them. 1. Gmail 186361 seconds 2. Notepad++ 130641 seconds 3. Google Chrome 79879 seconds 4. GitHub 43780 seconds 5. Windows Command Prompt 40967 seconds 6. Microsoft Excel 32578 seconds 7. Microsoft Word 27067 seconds 8. Microsoft PowerPoint 27059 seconds 9. Windows Explorer 20902 seconds 10. Google Docs 17989 seconds 11. Foxit Reader 17001 seconds 12. Microsoft Outlook 15855 seconds 13. Internet Explorer 15830 seconds 14. Google Search 15616 seconds 15. Skype 14423 seconds 16. Media Player Classic 14159 seconds 17. Google Groups 7061 seconds 18. Google Calendar 5531 seconds 19. Wesabe 2814 seconds 20. Google Analytics 2665 seconds 21. TeamViewer 1985 seconds 22. RGui 1875 seconds 23. LinkedIn 1528 seconds 24. YouTube 1400 seconds 25. Stack Overflow 1167 seconds 26. Acrobat Connect 964 seconds 27. Kongregate 914 seconds 28. HTML Help 871 seconds 29. PicPick 790 seconds 30. Zoundry Raven 684 seconds 31. Mockingbird 657 seconds 32. Twitter 655 seconds 33. iStockphoto 590 seconds 34. 7-Zip 584 seconds 35. Buzznet 552 seconds 36. Inkscape 516 seconds 37. Bitbucket 499 seconds 38. Microsoft Visio 496 seconds 39. Paint.NET 474 seconds 40. IrfanView 461 seconds 41. Tableau Public 436 seconds 42. µTorrent 435 seconds 43. HandBrake 422 seconds 44. Check Point Endpoint Security 411 seconds 45. Windows Task Manager 385 seconds 46. Microsoft Project 372 seconds 47. IETester 347 seconds 48. Google Maps 340 seconds 49. eBay 310 seconds 50. Spokn 270 seconds 51. Firefox 269 seconds 52. Google Calendar Sync 259 seconds 53. Windows Calculator 247 seconds 54. PayPal 246 seconds 55. JsonView 220 seconds 56. Windows Live Writer 184 seconds 57. Junction Link Magic 152 seconds 58. WinDirStat 142 seconds 59. Kindle 139 seconds 60. XAMPP 127 seconds 61. Wakoopa 105 seconds 62. Dropbox 100 seconds 63. Office Help Viewer 99 seconds 64. PrimoPDF 94 seconds 65. PuTTY 84 seconds 66. Python 80 seconds 67. Flavors.me 75 seconds 68. Google Sites 71 seconds 69. Process Explorer 70 seconds 70. Windows Volume Control 63 seconds 71. Wikipedia 58 seconds 72. Nitro PDF Reader 57 seconds 73. Management Console 47 seconds 74. PythonWin 45 seconds 75. Windows Based Script Host 45 seconds 76. WinDiff 45 seconds 77. VLC Media Player 39 seconds 78. ClipX 35 seconds 79. Windows Installer 35 seconds 80. The Internet Movie Database 32 seconds 81. ImageShack 31 seconds 82. WordPad 25 seconds 83. TeraCopy 22 seconds 84. Skype Portable 22 seconds 85. Picasa Web Albums 20 seconds 86. Syncplicity 17 seconds 87. Google Reader 16 seconds 88. Google Talk 15 seconds 89. VirtualDub 12 seconds 90. Adobe Manager 10 seconds 91. FreeCall 10 seconds 92. Notepad 8 seconds 93. Codebase 5 seconds 94. eTrust ITM 5 seconds 95. Google Checkout 5 seconds 96. GDI++ Tray Notifier 5 seconds 97. ImgBurn 2 seconds 98. Virtual Desktop Manager 2 seconds 99. Tesseract201 2 seconds 100. TortoiseHg 0 seconds Comments Somnath 1 Mar 2011 4:38 pm: More time on Gmail than browsers - how are you accessing Gmail then? S Anand 6 Mar 2011 8:51 pm: @Somnath, mostly breaking through proxies -- see http://goo.gl/6wyg0 and http://goo.gl/DNtui. @Thej, no idea I'm afraid, but before I used Wakoopa, I was using https://gist.github.com/857652 which worked just fine, except that it wasn't social and didn't have the pretty charts. You might want to tweak that for Linux. Thejesh GN 5 Mar 2011 5:29 pm: It doesnt run on Linux (only PC n MAC). Anything for me? Shankar V 28 Feb 2011 3:10 am: hi Anand how do you generate this list? Wakoopa is blocked at Infy. So could not check that one out. Also, surprised to note that you are a Chrome user against FF. I have used both and my preference is still FF. S Anand 28 Feb 2011 6:38 am: I work out of client sites -- so sites aren't blocked. Plus, it includes software from my home laptop. I shifted to Chrome a while ago, even for development, mostly because it's faster than FF. The only thing I miss is Firebug, really.

Software update


I watched The Social Network and was surprised by how much I enjoyed it. I appreciated the realistic Perl code and fast-paced dialogue, ultimately finding personal connections to Mark Zuckerberg's character despite my initial jealousy.

4:00pm. Just started watching The Social Network. I'm fairly sure I won't like the film, mostly because I'll be jealous of Mark. About 5 minutes into the movie. I find myself rewinding to catch the dialogues. They're very fast. Very, very fast. 10 minutes. I like the code. I stopped on a screen to start checking if it's real code. It's in Perl. Stopped myself before I started dry-running the code. 20 minutes. OK, some of this is a bit beyond me. But good movie. 1:20. Dropped a note to Ram asking (almost forcing) him to watch the film. A fair bit to learn from a stakeholding perspective. 1:45. Dammit. I see a bit of the anti-social me in him. Gripping. Overall, bloody good movie. Quite surprised. And no, definitely not feeling jealous.

The Social Network


I revised my student performance visualizations using Python and SVG to improve readability for teachers. These color-coded dashboards use grade distributions and trend graphs to identify bimodal distributions, top performers, and individual subject struggles at a glance.

This earlier visualisation was revised based feedback from teachers. It’s split into two parts: one focused on performance by subject, and another on performance of each student. Students’ performance by subject Visualisation by subject This is fairly simple. Under each subject, we have a list of students, sorted by marks and grouped by grade. The primary use of this is to identify top performers and bottom performers at a glance. It also gives an indication of the grade distribution. For example, here’s mathematics. Student scores in a subject Grades are colour-coded intuitively, like rainbow colours. Violet is high, Red is low. Colour coding of grades The little graphs on the left show the performance in individual exams, and can be used to identify trends. For example, from the graph to the left of Karen’s score: A single student's score … you can see that she’d have been an A1 student (the first two bars are coloured A1) but for the dip in the last exam (which is coloured A2). Finally, there’s a histogram showing the grades within the subject. Histogram of grades Incidentally, while the names are fictitious, the data is not. This graph shows a bimodal distribution and may indicate cheating. Students’ performance Visualisation by student This is useful when you want to take a closer look at a single student. On the left are the total scores across subjects. Visualisation of total scores Because of the colour coding, it’s easy to get a visual sense of a performance across subjects. For example, in the first row, Kristina is having some trouble with Mathematics. And on the last row, Elsie is doing quite well. To give a better sense of the performance, the next visualisation plots the relative performance of each student. Visualisation of relative performance From this, it’s easy to see that Kristina is the the bottom quarter of the class in English and Science, and isn’t doing to well in Mathematics either. Gretchen and Elsie, on the other hand, are consistently doing well. Patrick may need some help with Mathematics as well. (Incidentally, the colours have no meaning. They just make it overlaps less confusing.) Next to that is the break-up of each subject’s score. Visualisation of score break-up The first number in each subject is the total score. The colour indicates the grade. The graph next to it, as before, is the trend in marks across exams. The same scores are shown alongside as numbers inside circles. The colour of the circle is the grade for that exam. In some ways, this visualisation is less information-dense than the earlier visualisation. But this is intentional. Redundancy can help with speed of interpretation, and a reduced information density is also less intimidating to first-time readers. Comments Venkatesh 19 Jan 2011 5:56 pm: Anand, I was searching for BCG and landed on your blog, retraced and landed up in this page. I currently work for an AID India (www.aidindia.in) in TN. The above gui/code looks extremely useful for a particular initiative we have taken up recently. If you are okay with it, i would drop in a mail explaining the exact idea. If you are up for it, then lets take it from there... Thanks. Venkatesh Krishnamoorthy. V 4 Jan 2011 1:12 pm: Is this something that you plan to release in the open source? Also, just curious : how did you generate the chart?Protovis? S Anand 9 Jan 2011 10:12 am: Just SVG and Python, actually. The code is at http://github.com/sanand0/reportbee-dashboard V 9 Jan 2011 11:43 am: SVG is neat, but i have offlate fallen in love with protovis. Check it out.

Visualising student performance 2


I built a heatmap mapping IMDb ratings against vote counts to find movies I've missed. It helps me visualize popular high-rated outliers, track my viewing progress, and filter by genre to discover hidden gems outside the Top 250.

The IMDb Top 250, as a source of movies, dries out quickly. In my case, I’ve seen about 175/250. Not sure how much I want to see the rest. When chatting with Col Needham (who’s working his way through every movie with over 40,000 votes), I came up with this as a useful way of finding what movies to watch next. visualising-the-imdb-1 Each box is one or more movies. Darker boxes mean more movies. Those on the right have more votes. Those on top have a better rating. The ones I’ve seen are green, the rest are red. (I’ve seen more movies than that – just haven’t marked them green yet :-) I think people like to watch the movies on the top right – that popularity compensates (at least partly) for rating, and the number of votes is an indication of popularity. For example, my movie pattern tells me that I ought to see Cidade de Deus, Inglourious Basterds and Heat – which I knew from the IMDb Top 250, but also that I ought to cover Kick-Ass, The Hangover and Juno. visualising-the-imdb-2 It’s easy to pick movies in a specific genre as well. visualising-the-imdb-3 Clearly, there are many more Comedy movies in the list than any other type – though Romance and Action are doing fine too. And I seem the have a strong preference for the Fantasy genre, in stark contrast to Horror. (Incidentally, I’ve given up trying to see The Shining after three attempts. Stephen King’s scary enough. The novel) kept me awake checking under my bed for a week at night. Then there’s Stanley Kubrick’s style. A Clockwork Orange was disturbing enough, but Haley Joel Osment in the first part of A.I. was downright scary. Finally, there’s Jack Nicholson. Sorry, but I won’t risk that combination on a bright sunny day with the doors open.) You can track your list at . For those who want to play with the code, it’s at . Comments Umar Sharief 11 Dec 2011 7:40 am: Great Idea... would have been much more awesome if we were able to select multiple Genres .. Thejesh GN 7 Jul 2011 7:19 am: Love it. Recommended Movies | Brian Hurley's Blog 26 Dec 2012 10:34 pm (pingback): [...] is another way to find good movies to watch you might have missed, and the link for the visualization [...]

Visualising the IMDb


I migrated from Python and Tornado to Node.js to simplify corporate recruitment and leverage its asynchronous performance. Node.js offers superior execution speed over Rhino, stable event-driven architecture similar to Nginx, and the ability to share codebases.

I’ve moved from Python to Javascript on the server side – specifically, Tornado to Node.js. Three years ago, I moved from Perl to Python because I got free hosting at AppEngine. Python’s a cleaner language, but that was not enough to make me move. Free hosting was. Initially, my apps were on AppEngine, but that wouldn’t work for corporate apps, so I tried Django. IMHO, Django’s too bulky, has too much “magic”, and templates are restrictive. Then I tried Tornado: small; independent modules; easy to learn. I used it for almost 2 years. The unexpected bonus with Tornado was it’s event-based model: it wouldn’t wait for file or HTTP requests to be complete before serving the next request. I ended up getting a fair bit of performance from a single server. Trouble is, Python’s a rare skill. I tried selling Python in corporates a couple of times, and barring RBS (which used it before I came in, and made it really easy for me to build an IRR calculator), I’ve failed every time. Apart from general fear, uncertainty and doubt, getting people is tougher. Javascript’s a good choice. It has many of Python’s benefits. It’s easy to recruit people. Corporates aren’t terrified of it. Rhino was good enough a server. All it lacked was the “cool” factor, which node.js has now brought it. And besides, It’s fast. About 20 times faster than Rhino, by my crude benchmarks. It’s stable. (Well, at least, it feels stable. Rock solid stable. Sort of like nginx.) It’s asynchronous. So I don’t miss Tornado It has a pretty good set of libraries, thanks to everyone jumping on to it I can write code that works on the client and server – e.g. form validation Bye, Python. Comments Philmod 22 Mar 2011 12:09 pm: Which library do you use for both server and client form validation? S Anand 24 Mar 2011 3:28 pm: I don't have any at the moment, Phillippe -- I'm writing my own at the moment. Do you know of any good ones? Aravinda 12 Mar 2011 8:36 pm: I started experimenting with nodejs and CouchDb couple of months ago. It is really awesome. A few good points to note about Node Js http://ncannasse.fr/blog/is\nodejs\wrong Shankar V 17 Mar 2011 11:44 am: Oh Anand - Python I thought is such a wonderful language. That it is a rare skill should not be a reason to turn away from it. On the contrary, maybe it is good that the language is not getting dumbed down ;) Have you read this? - http://www.paulgraham.com/pypar.html Sathya 17 Mar 2011 2:23 pm: Javascript, as Doug Crockford said, is a misunderstood language. I was under the impression that enterprises are quite wary about server side javascript. I am quite surprised to learn that companies are more open to Javascript than Python (given that Google embraces Python). Also, in India, developers look at javascript more as a UI-bells-and-whistles stuff and it is quite difficult to hire a Quality serverside js developer. Anyway ... for rapid prototyping isn't Python good enough ? You are anyway used to the quirks of Django etc for 3 yrs. Am sure node.js would be having some baggage elsewhere. Or is it the case that change is the only constant ? Safe Hammad 15 Mar 2011 7:36 am: Very interesting post. I'd be keen to learn which stack of libraries and frameworks you end up using on top of node.js and your experiences with them. Also, what are your experiences so far of using node.js in a production setting and how do they compare with, for example, deploying Python code in Tornado? S Anand 4 Jun 2011 4:11 am: Yes, I did play with it. It's amazing. I'm waiting for a few more months, allowing it to get more popular, and once someone else on my team suggests it independently, we'll jump into it! Luis 3 Jun 2011 3:30 pm: If you like the benefits of JavaScript and node.js but you still miss python's syntax and productivity, you may want to play with Coffeescript. Check it out, it's addictive...

Why node.js


I discovered that HTML5 audio tags support data URIs for embedding base64-encoded MP3s directly in HTML. To test this, I built a musical keyboard that generates audio on the fly, reducing HTTP requests for small sound files.

Turns out that you can use data URIs in the tag. Just upload an MP3 file to and you’ll get a long string starting with data:audio/mp3;base64... Insert this into your HTML: That’s it – the entire MP3 file is embedded into your HTML page without requiring additional downloads. This takes a bit more bandwidth than the MP3, and won’t work on Internet Explorer. But for modern browsers, and small audio files, it reduces the overall load time – sort of like CSS sprites. So, on my bus ride today, I built a little HTML5 musical keyboard that generates data URIs on the fly. Click to play. keyboard Comments Fabio Mazarotto 9 May 2013 8:07 pm: Can you confirm that it no longer works in Safari >= 6.0?

Audio data URI


Inspired by Bret Victor, I wrote a lightweight vanilla JavaScript snippet that auto-reloads pages by polling the Last-Modified header every 300ms. It works without external dependencies like LiveReload and requires only a local web server.

After watching Bret Victor’s Inventing on Principle, I just had to figure out a way of getting live reloading to work. I know about LiveReload, of course, and everything I’ve heard about it is good. But their Windows version is in alpha, and I’m not about to experiment just yet. This little script does it for me instead: It checks the current page every 300 milliseconds and reloads it if the Last-Modified header is changed. I usually include it as a minified script: There are no dependencies on any library, like jQuery. However, it requires that the file be on a web server. (It’s easy to fix that, but since I always run a local webserver, I’ll let you solve that problem yourself.)

Auto reloading pages


I explore why RGB color spaces fail to match human perception and share the practical tools I use to pick colors, including Adobe Kuler, ColorBrewer, and the perceptually uniform HCL model for programmatic selection.

In reality, a colour is a combination of light waves with frequencies between 400-700THz, just like sound is a combination of sound waves with frequencies from 20-20000Hz. Just like mixing various pure notes produces a new sound, mixing various pure colours (like from a rainbow) produces new colours (like white, which isn’t on the rainbow.) Our eyes aren’t like our ears, though. They have 3 sensors that are triggered differently by different frequencies. The sensors roughly peak around red, green and blue. Roughly. It turns out that it’s possible to recreate most (not all) colours using a combination of just red, green and blue by mimicking these three sensors to the right level. That’s why TVs and monitors have red, blue and green cells, and we represent colours using hex triplets for RRGGBB – like #00ff00 (green). There are a number of problems with this from a computational perspective. Conceptually, we think of (R, G, B) as a 3-dimensional cube. That’d mean that 100% red is about as bright as 100% green or blue. Unfortunately, green is a lot brighter than red, which is a lot brighter than blue. Our 3 sensors are not equally sensitive. You’d also think that a colour that’s numerically mid-way between 2 colours should appear to be mid-way. Far from it. This means that if you’re picking colours using the RGB model, you’re using something very far from the intuitive human way of perceiving colours. Which is all very nice, but I’m usually in a rush. So what do I do? 1. I go to the Microsoft Office colour themes and use a colour picker to pick one. (I extracted them to make life easier.) These are generally good on the eye. 2. Failing that, I pick something from 3. Or I go to and pick a set of colours 4. If I absolutely have to do things programmatically, I use the HCL colour scheme. The good part is it’s perceptually uniform. The bad part is: not every interpolation is a valid colour. Comments isomorphismes 13 Mar 2013 5:20 pm: What colours can't be created by convex combinations of {R, G, B}? I mean within the visible spectrum.

Colour spaces


Using Reportbee’s Class 12 exam data, I analyzed the correlation between various subjects and Computer Science. I found that Physics and Chemistry are more highly correlated with CS scores than Mathematics or English.

A question from Dorai get me thinking: does being good at maths help in programming? I don’t have a personal view. But since Reportbee has data on the Class 12 examination results for the last three years, we thought we could do a bit of analysis. Here’s the correlation of the scores of various subjects with Computer Science. Correlation Subject 0.79 CHEMISTRY 0.79 PHYSICS 0.75 ENGLISH 0.75 MATHEMATICS 0.72 LANGUAGE 0.67 BIOLOGY 0.66 ECONOMICS 0.66 COMMERCE 0.65 ACCOUNTANCY 0.56 HISTORY 0.52 GEOGRAPHY It almost breaks neatly into four groups. Physics & Chemistry , both of which have a correlation of 0.79, and clearly are the most correlated with Computer Science Maths, English & Language , which have a correlation of 0.72 – 0.75 Biology, Economics, Commerce and Accountancy , which hover at around 0.66 History & Geography , which are 0.52 – 0.56 The results in 2010 are almost exactly the same. Correlation Subject 0.78 PHYSICS 0.78 CHEMISTRY 0.75 ENGLISH 0.75 MATHEMATICS 0.73 LANGUAGE 0.67 ACCOUNTANCY 0.65 ECONOMICS 0.65 COMMERCE 0.64 BIOLOGY 0.60 GEOGRAPHY 0.55 HISTORY I’m not sure what it is that leads to this kind of correlation. In fact, the full correlation between every pair of subjects (for 2011) is below: What inferences would you draw from this? And what do you think is the reason for this? Comments Arun Ravindran 12 Feb 2012 5:53 am: My inference is purely anecdotal but might be helpful in explaining this data. I had chosen Computer Science for my +2 in 1998. It was taught in C++ and mostly involved memorising operations on Data Structures such as Lists, Stacks and Queues. The C++ standard library had to be memorized including I/O, string and file functions. The exams were basically a test of memory rather than attacking a new problem space mathematically. Guess which are the other subjects which involve memorising a huge set of symbolic facts? -- Chemistry and to a certain extend, Physics. I believe the data is more revealing of our Computer Science pedagogical and evaluation methods than the subject itself. S Anand 12 Feb 2012 6:46 am: Good point. It's quite debatable whether marks in computer science are indicative of programming ability. Anamika 12 Feb 2012 10:21 am: Netruvarai neram poga villaiyae, unadhu arugae neram podhavillaiyae...! Sathya 16 Feb 2012 12:56 pm: In my experience, I've come across rock star programmers who have sound grasp of mathematics. But i dont have statistics to prove them. As we all know, programming (esp functional) is heavily influenced by mathematics. Coming to inferences, I would take the dataset with a pinch of salt. Is it diverse enough to be statistically significant ? I agree with most of the comments on relating "mugging up programs" to being good at Chemistry in particular. Sathyaraj 12 Feb 2012 7:44 pm: My 2 cents based on experience. People who take the computer science group generally have to always take one language and English subject apart from taking Physics, Chemistry, Maths. They would not be able to take classes in accountancy, economics, history, etc. Some of the students at the top of the class would have realized that in order to differentiate in terms of coming first in class/school, one would need to excel in language and english. It is given that one needs to get excellent scores in physics, chemistry and maths to get an overall good percentage. Also the fact, that scoring a big total would enable them to get admission in colleges in like BITS pilani which looks at overall score. Students who take the biology or accountancy group have no such pressure for them to excel in English and language subjects. Hope it makes sense. Also, if you can do a correlation between computer science students and the language that they take, I believe you will find that the majority of students would have chosen french rather than Tamil or English. Shankar V 13 Feb 2012 6:23 am: Anand Correlation is a misleading statistic if the dependent and independent variables are not "really" related that way. There could be compounding effects within the independent variables leading to wrong correlation coefficients. There are ways to detect and cleanse compounding - design of experiments (in statistics) deals in depth on this. It is quite possible that there could be some relationship in how the computer science and math exams are scheduled and the scores in these subjects. If they are too close to each other, the student may have had lesser time to prepare for the math exam. For example, if Physics and Math were scheduled one after the other, and Computer Science test is after the Math test, the student may have prepared better for Physics, not have had enough time for Math and then recovered to do well in Computer Science. This can be a pattern in the entire class as it is normal for kids to focus more on Physics (the dreaded subject!) v/s Math. The statistic may not reveal ability or natural alignment of the subjects. Evgeni 11 Dec 2012 3:37 am: I'd rearrange subjects such that they're more clustered together by correlation, like in a TreeMap view. That way it's easier to see the relationships. amanjot kaur 10 Sep 2015 12:28 pm: i need correlation of commerce with language

Correlating subjects


I'm sharing a call for donations to support the Kuppuswami Sastri Research Institute in Mylapore. The center houses rare palm leaf manuscripts on astronomy and science but is struggling without government funding. Please contribute via PayPal if possible.

The following article appeared in The Times of India earlier this month. The institute is struggling for funds. Please contribute, if you could, by calling +91 44 24985320 or via PayPal . Sanskrit centre struggles to stay alive The Kuppuswami Sastri Research Institute attached to the Sanskrit College in Mylapore is in doldrums because of lack of government patronage. The Institute, one of the three involved in Sanskrit research in the country, has been surviving on private donations. With not enough resources, the management is unable to pay the faculty the benefits of the sixth pay commission. Institute director V Kameswari said the Union government stopped its financial support in 1995, after which it has been solely dependent on donations. "The institute has a trove of rare palm leaf manuscripts and books not just about Sanskrit literature but also on architecture, fine arts, geography, history and astronomy in Sanskrit," says Kameswari. The two other such institutes are the R G Bandarkar Sanskrit Institute in Pune and the Ganganath Jha Sanskrit Institute in Allahabad. "We have requested a onetime grant from the Union planning commission and also annual assistance from the Rashtriya Sanskrit Sansthan, but are yet to get any support," says K S Balasubramanian, deputy director of the institute. The plan panel had given grants to the Mumbai Asiatic Society and Kolkata-based Asiatic Society. The institute was getting about 10 lakh till 1995 but due to a misunderstanding between the government-appointed members of the governing committee and the management, the aid was stopped. Today, there are 24 scholars at the institute, most of them women doing their PhDs. "Scholars from across the country and world visit the institute. We send out publications to many foreign universities and they in turn send their publications which are preserved here," says Kameswari. The institute was started as a private non-profit organisation in 1944 in memory of Kuppuswami Sastri, a renowned Sanskrit scholar. It has a library with books on astronomy, architecture, fine arts, mathematics, Vedas, Puranas, Upanishads and various branches of science. "A private entrepreneur made a donation with which we have air-conditioned the library. The palm-leaf manuscripts in the library are 600 to 1,000 years old. Many of them are in Grantha script. We also have books on Jainism that speak about solving mathematical equations and explain geographical concepts," says Kameswari, who is worried about keeping the ancient language alive. Comments Abhijeet 27 Apr 2012 1:42 pm: Recently there are talks about declaring Marathi language as a classical language. During the discussions on tv i got to know that govt of india provides Rs. 500 crore per year for a classical language. As far as i know, Sanskrit is already a classical language. Wonder why there are lack of funds! That said, anyways i am also eager to contribute.

Donations for Sanskrit College