I explain my workflow for downloading music from YouTube using Keepvid or youtube-dl, then converting the video to MP3 using VLC's media conversion tool or a handy command line script for automation.

Five years ago, I built a song search engine – mainly because I needed to listen to songs. Three years ago, I stopped updating it – mainly because I stopped listening to songs actively, and have been busy since. For those of you who have been using my site for music: my apologies. These days, I don’t really find the need to download music. YouTube has most of the songs I need. Bandwidth is pretty good too even when on the move. But when I do need to download music, this is my new workflow. 1. Find the song on YouTube. (Misspellings are still an issue, but you’ll usually find what you need) 2. Download the video. Keepvid is the simple option. youtube-dlis the geek’s option (for multiple downloads) 3. Use VLC – the swiss-army knife of media – to convert the video into an MP3. That last step requires a bit of explaining. It’s very simple once you know how, but it took me a few months to get it right. So here goes. Select the Convert / Save option in the Media menu. audio-conversion-1 Click on Add to open file you want to convert. You can pick a track from an disk as well if you want to rip an audio CD or a DVD. audio-conversion-2 Choose the file. audio-conversion-3 Click on Convert / Save. audio-conversion-4 Type the destination filename. Make sure you type the full file name, and not just the name of the folder. audio-conversion-5 Select the output format you want under Settings – Profile. You can tweak the bitrate with the settings button, but I usually don’t bother. audio-conversion-6 When you click on the Start button, the file will be converted or the CD will be ripped. You’ll see the position marker move fairly fast. audio-conversion-7 The only problem I have with this method is that I can’t seem to do batch conversions easily enough with the GUI. Does anyone have any other workflow they like? Update (31 Jul 2012): Aditya Sengupta suggests the following: (should've guessed VLC would have something up its sleeve) Comments 408wij 15 Mar 2012 1:48 pm: Coincidentally, I just ripped a bunch of MP3s from AVIs last night. I used a Windows program called Format Factory. It supports batch conversion. Some AVIs failed on first past, so I reset their status in the queue and reran. vB 16 Mar 2012 5:31 am: Hey Try using freemake. The best as per me. You paste youtube URL and select the format you want to save on your machine and it does the rest. btw - found your website pretty neat, do not remember what was i looking for when i discovered it couple of days back, but now have put it on my reader. Ramanujam 24 Mar 2012 1:18 pm: Alternatively, you can skip all the steps and use one of the numerous sites that will directly give you the mp3. Here is one that works pretty well. http://www.youtube-mp3.org/ Pravin 16 Mar 2012 5:50 pm: I've been using this nifty little freeware for a long time and i would suggest it to you if you are interested in an all-in-one video downloading and format conversion. The feature set is really impressive for a freeware. It has also got support for GPU acceleration which might prove use full for those who want to work while conversion is on. http://bit.ly/aayV0 Pravin 25 Mar 2012 12:48 pm: @Sanketh normally flv's have 32kbps audio bit rate if you could go for 480p mode you will get 128kbps(Audio CD Quality) audio bit rate Pankaj 10 Apr 2012 8:57 pm: Seems you take a lot pain in getting few MP3s. :P Try freemake software (freeware) or even simpler this one http://www.youtube-mp3.org/ Sanketh 20 Mar 2012 3:49 pm: Roughly what bitrate are the youtube flv encoded at? Ganesh Kondal 4 Apr 2012 4:54 pm: youtube-dl works like a charm. Thanks dude. Using it in Ubuntu 11 Chaitanya 20 Mar 2012 9:01 am: Hey Anand This blog post has been very useful. I just used Freemake and it's cool and easy. Also, converting few videos using VLC Player. Does it take as much time as the length of the video? Or conversion is slow because I am doing it on my pretty old spec'ed laptop. Sendhil 17 Mar 2012 6:18 pm: I heard of this one-step workflow from my colleague: http://www.singyoutube.com/ Deepak 17 Mar 2012 7:11 am: I have used Listen to Youtube (http://www.listentoyoutube.com/). Works very well, especially for my son and his nursery rhymes and we dont want to see the computer screen for a long time. Ganesh 29 Mar 2012 10:50 am: If listening to mp3 is what you want then there are several free mp3 search sites. I use www.mp3skull.com You get a lot of options with various bit rates, and can listen to the song before downloading it. Vishal Dalmia 24 May 2012 6:46 am: http://musicable.com/ is a simple site that does the download easily with no downloads or installation required. Vijay Gnanaraj 30 Sep 2012 12:29 am: The next best option wud be IDM, internetdownloadmanager, which can download any video (almost from any website) playing on ur screen Streaming audio to iOS via VLC | s-anand.net 13 Oct 2012 4:51 pm (pingback): [...] listen to it on your iPhone / iPad – converting your PC into a radio station. As with most things VLC related, it’s tough to figure out but obvious in [...]

Downloading songs from YouTube


I found a simple way to host GitHub Pages without needing a master branch on the remote. By mapping my local master to the remote gh-pages branch in .git/config, I can push web content directly to the live site.

Github offers Github Pages that let you host web pages on Github. You create these by adding a branch to git called gh-pages, and this is often in addition to the default branch master. I just needed the gh-pages branch. So thanks to YJL, here's the simplest way to do it. Create the repositoryon github. Create your local repository and git commitinto it. Type git push -u origin master:gh-pages In .git/config, under the [remote "origin"] section, add push = +refs/heads/master:refs/heads/gh-pages The magic is the last :gh-pages.

Github page-only repository


I discovered how to automatically extract dictionary keys used inside Python lambdas by inspecting the func_code.co_consts attribute. This method helped me manage over 100 metric formulas for plotting without manually maintaining dependency lists.

Lisp users would laugh, since they have macros), but Python supports some basic code inspection and modification. Consider the following pieces of code: What if you wanted another function that lists all the dictionary indices used in the function? That is, you wanted to extract cost and sales? This is a real-life problem I encountered this morning. I have 100 functions, each defining a metric. For example, 1. lambda v: v['X'] + v['Y'] 2. lambda v: v['X'] - v['Z'] 3. lambda v: (v['X'] + v['Y']) / v['Z'] 4. ... I had to plot the functions, as well as each of the corresponding elements ('X', 'Y' and 'Z') in the formula. Two options. One: along with each formula, maintain a list of the elements used. Two: figure it out automatically. Each function has a funccode attribute. So, when you take margin.funccode is a "code object". This has a bunch of interesting attributes, one of which is coconsts There -- I just pick the strings out of that list and we're done (for simple functions at least.) Check out and search for func -- you'll find a number of interesting things you can do with functions, such as 1. Finding and changing the default parameters 2. Accessing the global variables of the namespace where the function was defined (!) 3. Replacing the function code with new code Also search for co -- you'll find some even more interesting things you can do with the code: 1. Find all local variable names 2. Find all constants used in the code 3. Find the filename and line number where the code was compiled from Python also comes with a disassembly module dis. A look at its source is instructive. Comments Kamaal 4 Sep 2012 7:29 am: This is cool. But Python needs to get Multiline Lambdas. One of the things Higher order Perl programmers miss in Python are muliline lamdas. S Anand 4 Sep 2012 11:40 am: Yeah -- I miss that too. But I guess it'd go too strongly against the Python ethos of readability. I find it interesting that I miss multi-line lambdas ONLY when writing code. Never when reading it. So guess it isn't too bad a thing, given that I'm reading (my own, often) code 90% of the time, and writing it less than 10% :-)

Inspecting code in Python


I benchmarked Protocol Buffers against JSON and CSV in Python to compare serialization speed and file size. I found protobuf significantly slower than JSON, while gzipped JSON and CSV provided better compression and performance for structured data.

Google’s Protocol Buffers is a “language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler” XML is slow and large. There’s no doubting that. JSON’s my default alternative, though it’s a bit large. CSV’s ideal for tabular data, but ragged hierarchies are a bit difficult. I was trying to see if Protocol Buffers would be smaller and faster, at least when using Python. I took JSON as the base, and checked the write speed, read speed and file sizes. Here’s the comparison: image Protocol Buffers are 17 times slower to write and almost 6 times slower to read than JSON files. File sizes are smaller, but then, all it takes is a simple gzip operation to compress the JSON files even smaller. Reading json.gz files is just 2% slower than JSON files, and writing them is only 4 times slower. The code base is at On the whole, it appears that GZipped JSON files are smaller, faster, and just as simple as Protocol Buffers. What am I missing? Update: When you add GZipped CSV to the mix, it's twice as fast as GZipped JSON to read: clearly a huge win. It's only slightly slower to write, and but compresses a tiny bit more than JSON. Comments Kaushik 23 Aug 2012 7:40 am: 1. Typed data elements 2. Language bindings 3. Evolution via backward compatibility (as mentioned by Krishna in the comment above) Krishna 1 Aug 2012 7:50 am: Protocol Buffers offers the ability to version schema changes in a backwards compatible way. So, especially in analytics (client based - mobile / flash clients) that have custom analytics, you will have changes to the schema due to product changes / feature additions or removals and protocol buffers offers a painless way to deal with these changes. From: https://developers.google.com/protocol-buffers/docs/overview#whynotxml New fields could be easily introduced, and intermediate servers that didn't need to inspect the data could simply parse it and pass through the data without needing to know about all the fields. You would have to handle these in application code if you use JSON. Specifically, you don't want this: if (version == 3) { ... } else if (version > 4) { if (version == 5) { ... } ... } Zheng 1 Aug 2012 8:53 am: Even within Google, Protocol Buffer processing in pure python is known to be extremely slow, about 100 times slower than optimized C implementation. It is recommended to use swigged C implementation within Python. Not sure if the C implementation has been open sourced. paintball equipment guy 16 Dec 2012 6:34 pm: I've read on stackoverflow that in c#, the protobuf vs xml and json is much faster. very interesting this is not the case for py... Ray Luo 7 Aug 2013 8:24 am: @Krishna, those versioning things are \NOT\ the pitfalls of json. Nor even that google link suggests that. Json object is always flexible to contain new fields in new version, and old version servers can read only known fields. Dean 10 Jun 2013 7:20 am: The python implementation of protocol buffers is horribly slow. It basically uses meta classes and reflection, whereas all other language bindings use compile-time generated code -- which is orders of magnitude faster. So while this comparison is probably fair for Python, other languages will have quite different results (in particular, I'd expect protocol buffers to be faster than JSON in C/C++ and Java). Paul 19 Oct 2014 8:14 am: Protocol Buffers are able to contain strings with line endings etc. without breaking the container's syntax. You can also GZip protobuffer messages.

Is Protocol buffers worth it?


I discovered that Magnetix magnets can form a chain that doesn't break under its own weight. I explored the theoretical cost and materials required to build a chain spanning one light-year.

I wasn’t entirely sure, but now I’m somewhat convinced: Magnetix magnets can form an infinite chain that won’t break due by its own weight. photo 1 photo 2 (This is not true, however, if you introduce the steel bearing balls between them. That structure collapses pretty quickly if you pull it up like a chain.) So, this would be a really nice question for What If, IMHO. What if you made a 1 light-year chain of Magnetix? Well, to begin with, we’d need nearly 40 million trillion pieces. That’d cost at least 10 million trillion dollars based on the current prices at Amazon, and would be about 140,000 times the world’s GDP. I’m sure Randall could take this a lot further. Comments Kamaal 2 Oct 2012 10:46 am: Very interesting discovery and idea. Now what I would be interested in knowing is the implications of using it in further research on space elevators(http://en.wikipedia.org/wiki/Space\elevator). Space elevators would be awesome. Needless to say the material used to design space elevators needs to be strong and load sustaining. The biggest hurdle to that goal is the weight of the elevator rope itself. Shalini 11 Oct 2012 10:56 am: This blog gives very useful information about analytics, data science and such stuff. I am a big fan.

Magnetix


I put together a quiz featuring recent Tamil songs from 2012. Listen to the clips and guess the movie. I designed the interface to accept phonetic spellings, so the box turns green as soon as you're right.

After a long break, here's another quiz, featuring relatively recent Tamil songs. Can you guess which movie they are from? Don't worry about the spelling. Just spell it like it sounds, and the box will turn green. Comments Rosario 23 May 2012 2:46 am: Right answers can be given along with the result. Nice pass-time for youngsters. Thank You for your services. Rosario sriram 22 May 2012 4:50 pm: after a long time... :) but what a poor memory i have.... 8/20. :(:(:(:( S Anand 21 May 2012 3:49 pm: @vrraghy: Fixed song 10. Thanks! Vinoth B 22 May 2012 3:02 pm: Interesting..! But couldn't get many !! :-) Raghavan Rengachari (@vrraghy) 21 May 2012 2:44 pm: Just couldn't get a few :| and song 10 isn't working... thambidi 21 Jul 2012 9:17 pm: poor memory 07/20 priya 3 Jun 2012 1:26 pm: Nice..familiar song but could't find all movie name Jayapal Chandran 23 May 2012 8:59 pm: It isn't poor memory but it is the quality of the song which makes you listen to it again and again and eventually you will get well memorized in you. and i could get green only the songs which i knew very well... others i did not even try cause they never sounded good... or the song failed publicity in some way... kavilavu 16 Jun 2012 8:29 am: 11/20,.....we found it..nice job.. so interesting to play thanks kamlakrisnan 29 Oct 2012 3:51 pm: Hi i got 20/20, its very interesting to find the movie names. Niranjan 6 Dec 2012 3:00 pm: I got 20/20 at first try itself by identifying the song name immediately. But it was difficult to find movie names. I have listened to all these songs in music channels as well as in my car. But finding movie names was difficult for 3-4 songs. Vincent(Jv) 12 Oct 2012 4:29 am: i got 15/20 great work Thanks... radha 28 Sep 2012 8:17 am: hey..... i got 19/20, great work... thanks for your good work Sundar 21 Nov 2012 5:04 am: i think i'm the first to get all 20/20 songs, but very tough to find it. good job for the collections, many was out of memory as we hear new songs, afshin 23 Oct 2012 6:32 pm: wow!!!!!!!17/20,......... intersting!!!!!!!! Abi 31 Dec 2012 4:29 am: can't find 8,10 and 20 Shiva 22 Nov 2012 2:38 pm: HEy Intersting!!!!! got 16/20. tough work :) Akila 12 Dec 2012 9:55 am: 14/20... where are the answers please? anusha 14 Mar 2013 3:06 pm: wow!!all 20 right!! :D Priya Krishnamoorthy 3 Feb 2013 8:09 am: 18/20. every thing is easy but except 2(i.e,13 & 16) sribala arun 25 Dec 2013 3:21 pm: i cant get 16 and then 14 is 7am arivu but it is showing not correct anu 24 Dec 2013 1:51 pm: i got 19/20 its interesting!!! deepika 21 Feb 2014 2:11 pm: i got 18/20 nice game Anusha 14 Dec 2013 8:35 pm: Woohoo!!! All 20 correct! ;) Anusha 14 Dec 2013 8:36 pm: Woohoo!!! All 20 are right! ;) hbqdb 27 Apr 2013 5:51 am: can’t find 10,14&17 anu 24 Dec 2013 1:55 pm: i got 19/20 nice!!! kumar 8 Apr 2014 3:40 pm: 17/20 Divya kd 14 Mar 2015 7:59 am: Wow I got 20/20 rviji 6 Dec 2014 8:23 pm: wow!!!!20/20 sowndharya 30 Apr 2018 8:10 pm: wow .....got 15 out of 20 ATR 9 Jun 2016 8:53 am: Got 20/20 but 7aam arivu is taking up as answer Sab 18 Apr 2020 7:05 pm: bad memory sankar 30 Dec 2015 8:52 am: could you forward the link to download this and answers for our own party game. hema 9 Jul 2016 8:31 am: semma i got 20/20 Akshaya 28 Nov 2018 12:15 pm: it was interesting revo 29 Sep 2016 10:20 am: nice quiz........but where are the answers? ganga 17 Jun 2016 7:16 pm: i got ans for 17 and it's very interesting Siva 8 May 2018 7:24 pm: Good adithiya 23 Apr 2017 1:16 pm: wow...sema but i cant find songs Meena 10 Apr 2020 4:59 pm: I found 17.. I know another 2 song.. but I can't remember the lyrics.. the last one is very tough Priya 23 Sep 2019 7:06 pm: Wow I got it 18/20

Recent Tamil Songs Quiz


I prioritize restartability over parallelism in large-scale data processing. I explain how smart partitioning and UNIX commands like xargs allow me to handle massive datasets efficiently without ever needing complex frameworks like Hadoop.

When processing data at a large scale, there are two characteristics that make a huge difference to my life. Restartability. When something goes wrong, being able to continue from where it stopped. In my opinion, this is more important than parallelism. There’s nothing as depressing as having to start from scratch every time. Think of it as the ability to save a game as opposed to starting from Level 1 in every life. Parallelism. Being able to run multiple processes in parallel. Often, this is easy. You don’t need threads. Good old UNIX xargs can do a great job of it. Interestingly, I’ve never used Hadoop for any real-life problem. I’ve gotten by with UNIX commands and smart partitioning. The “smart partitioning” bit is important. For example, if you’re dealing with telecom data, you’d be calculating most of your metrics (e.g. did the number of calls grow or fall, are there more outgoing or incoming calls, etc.) are calculated on a single mobile number. So if you have multiple data sets, as long as all the data related to one mobile number are on the same system, you’re fine. If you have 100 machines, just split the data based on the last 2 digits of the mobile number. So data about 9012345678 would go to machine 78 (the last two digits). Given a mobile number for any type of data, you’d know exactly which machine would have that data. For all practical purposes, that gives you the basics of a distributed file system. (I’m not saying you don’t need Hadoop. Just that I haven’t needed it.) Comments kamaal 30 Aug 2012 4:04 pm: Anand, If you were to write down- What would be the must learn skills for a data scientist? I've sort been trying and working hard on this thing. Of course you must know programming, what else apart from a programmers ordinary skills set does a data scientist need. Karthik Joshi 3 Sep 2012 9:06 am: I am a reader of your blogs. I love your visualization ideas. This one blog about distribution of work is a great Idea. But I think, it would depend on the nature of data for effective utilization of all 100 machines. (Suppose we have few numbers which will not end with 09 or 88 ro so on) How about hashing? Even though the distributedness of hash algorithm is a debatable issue, do you think this would be beneficial ? Just a thought.

Restartable and Parallel


I built a 30-line Python scraper using ScraperWiki to extract detailed laptop specifications from Flipkart after finding that no Indian retail sites allowed filtering by hardware details like hard disk RPM.

I've returned my laptop, and it's time to buy a new one. For the first time in my life, I'm buying a laptop for myself. I have a fairly clear idea of what I want: a 500GB+ 7200 rpm hard disk with 4GB of RAM and an Intel Core i7. I thought that would make finding one of those powerful laptops for producing music since I record some stuff too out of hobby. Sheer naïveté. Not a single site let me filter by hard disk rpm in India. (To be fair, I haven't found any sites outside India that did that either.) After spending a good two hours hunting for the details and collating it, I did what I normally would: spend 30 minutes writing a scraper. The scraper runs through all laptops on Flipkart and pulls out all of their specs. Thanks to the diligence of the good folks at Flipkart, this information is readily available on each page. The HTML is structured quite neatly too, so it was just a 30-line program to scrape it all. Full credit to ScraperWiki as well — I could use it on a netbook without any developer tools installed. The scraper took 2 hours to run. Feel free to filter through the output (CSV) for your favourite laptop, or fork the code and pull any other data you like. Comments ravi atluri 14 Jan 2012 4:19 am: If weight is not a limitation...i would recommend XPS15 or XPS15z i prefer it because of the i7 and >8GB RAM. gives me flexibility to run linux on VMs and 8 cores and 8GB RAM gives a lot of VMs :P if you are looking for portability. mac book pro :D it best fits for coding too Subodh M 24 Jan 2012 1:05 pm: Anand, hv u explored Thinkpad T410i? I am using one with with an i5 processor with4GB RAM, 500GB HDD, @ 7200 rpm and am happy with it. S Anand 14 Jan 2012 10:29 am: Battery life is the other consideration, and may be more important than performance. I am edging towards the Sony Vaio S VPCSA35GG. Despite the terrible name, it seems to have a decent performance battery-life trade off. Indira 14 Jan 2012 5:40 pm: dell xps 14/15z comes with a good battery life. dell claims 7 hours, but i would think that would be with a slightly less than optimal settings of screen brightness, it would at least give you 4 hours, which is reasonable. Between sony and dell, my choice would be dell S Anand 14 Jan 2012 5:59 pm: @Indira: thanks! Have added Dell XPS to my list. Also considering an Thinkpad E520 (i5, 4GB, 750GB) that seems very reasonably priced by a local vendor. I'll try and avoid Sony. Yuvi 14 Jan 2012 6:47 pm: SSD! Somnath 14 Jan 2012 7:31 pm: If you are planning to buy from flipkart you may like to contact Ravi Vohra. He is a big shot at flipkart :D S Anand 16 Jan 2012 1:25 pm: @pavan: agreed. I already have an external SSD though, so I just need a USB 3.0 port to take advantage of it. Can live with a more cost effective laptop :-) pavan 16 Jan 2012 6:38 am: Anand, try to get a laptop with SSD, or upgrade later. Even the cheapest SSDs are 10 times faster than HDDs. Low noise and better battery life are nice to have side effects of SSD. S.C 8 Aug 2012 3:20 pm: I had gone to a HP authorized service center, since browsers other than IE wouldn't work on my 3+ yr. old HP Pavilion Multimedia laptop. The person in charge of this center said that my laptop had probably been infected with a virus and advised me to install Norton AV. He also offered to 'upgrade' the OS in it from the licensed version of Windows Vista to Windows 7. When I started working on the laptop at home, I found messages that the OS was an unlicensed one. A few days later, I found that the laptop was not booting in the first attempt. Even when it booted, I got 'blue screen' errors before booting and horizontal and vertical lines on the screen which sometimes intersected. On taking the laptop back to the service center, I was told that the NVIDIA graphics card on the laptop's motherboard had malfunctioned. I want to ask users through this forum, whether this could have happened because of the extra load imposed on the 3+ yr. old graphics card by Windows 7? Since the personnel at the center now claimed that they could repair the graphics card for Rs. 1500, I gave them the laptop. But, when I collected it back after a week, I found no change in the laptop. So, I took it to the main HP service center in my city (where laptops covered by warranty are repaired). The personnel at this center said that they would have to replace the mother-board, since they didn't do chip level repairing. So, in order to get a graphics card repaired, I would have to shell out Rs. 22K for a new mother-board. Though I accepted this, HP hit me on the head with their next condition, when they said that they could repair the laptop, only if I surrendered the existing mother-board (in the malfunctioning laptop) to them, free of cost. This was in spite of the fact, that they had not mentioned this when taking the laptop from me for repair. Since, I couldn't have taken the earlier HP service center personnel to court for cheating me out of Rs. 1500 (in the name of repairing the graphics card), I refused and had to come back from the service center without getting the laptop repaired, in spite of pleading with HP to repair the mother-board for a month. They not only refused repeatedly, saying that this was their policy world wide and they couldn't change it just for me, but added insult to injury, by charging me Rs. 300 for inspecting the laptop. The reason I am mentioning this incident here, is because I feel this is malafide on the part of HP. Since I pay for a laptop when I buy it, the motherboard in it becomes my property then and there. If it malfunctions, why should I have to surrender it, to get it replaced and that also free of cost? I can choose to do whatever I want with it, which in this case, will definitely be to take HP to court for deficiency of service and causing mental agony, which wouldn't have been possible if I had surrendered the defective mother-board. I have heard that DELL is also taking customers for a ride in a similar way. Have any of you faced this? S Anand 16 Jan 2012 1:27 pm: @Ravi, will reach out and take advice on how to get those discounts! Deepak 15 Jan 2012 4:17 am: Aww, that is awesome. Can you include the price as well? :) Sri 15 Jan 2012 9:26 am: Try to check the prices at Dell Web Site directly and speak to the sales guy as well, they may give you a better price. Also it is better to have a look at Croma where you get a chance to touch and feel the different products and you will get an idea of price. Thejesh GN 15 Jan 2012 1:53 pm: I use Hp Probook 6460b which actually matches all your requirements. Unfortunately Flipkart has i5 one :( S Anand 15 Jan 2012 3:08 pm: Thanks Sri, that's a great idea. Will check Croma tomorrow. S Anand 15 Jan 2012 3:28 pm: How are HP laptops, generally? I haven't heard anything one way or the other so far... ravi atluri 15 Jan 2012 6:20 pm: my dell xps15 gives me around 4.5 hours ..that's a 9 cell battery and is pretty heavy :P XPS is more of a portable desktop than a portable one ;) I prefer Dell, because of their better service in India. and if you are buying Dell...there are tricks to get discounts ;) dont order online directly :D How to Scrap Web Pages using ScraperWiki? | Python, ScraperWiki, HasGeek, Jobs | Sanspace Blog 28 Sep 2012 8:04 pm (pingback): [...] data from web pages in a programmable way. For example, Check out Anand’s post about how he scraped for a laptop on Flipkart. That’s how I came to know about ScraperWiki and recently I wrote some quick and [...]

Scraping for a laptop


I explore why data analysis fails when it lacks a narrative. Using my dry research on Indian meter readings and past jargon-filled reports, I argue that simplicity is essential for driving real-world understanding and action.

In a number of sessions I’ve been to, people ask analysts to make their results more interesting – to tell stories with them. I’m co-teaching a course, part of which involves telling stories with data. So this got me thinking: what is a story? How does one teach storytelling to, let’s say, an alien? Consider this mini-paper. You’re probably thinking: "I know why he’s put this example here. It must be a bad one. So, what a rotten paper it must be!" Well, not quite. It’s a good piece of analysis. I did it myself and there’s a fair bit of effort and care behind these short paragraphs. The trouble is, if I read it out to my daughter, she’d say "What?" and not understand a word. My wife’d say “So what?” and not care a bit. I might as well not have written it. It’s like that Zen thing: If a tree falls in a forest and no on hears it, does it make a sound? If you did a piece of analysis, and no one understands or cares about it, why did you do it in the first place? Why do you do it? That last question is important: why do we analyse? Sometimes, we do it for fun. The knowledge is beautiful. Knowing Tetris is NP-Complete is rewarding, even though my colleague sarcastically remarked, "Thank God! I'm sooo relieved now that I know that Tetris is NP whatever." If that's the case with you, great. Write the analysis any which way you'll enjoy. Sometimes, we do it because we're forced to. In class. At work. Wherever. But that's another way of saying "I don't know why I'm doing it." In that case, I'd gently recommend watching 3 Idiots. Most often, we do it to share knowledge and drive actions. In that case, if no on understands it, or does anything with it, why do it? Keep it simple We prerajulisation of Farhanitate flagellated with ... Would your audience understand that? Or are you just scared that simple words indicate a simple mind? I was once afraid. 15 years ago, when writing a paper on IBM India's competitive advantage for the CXOs, I was worried about it being too simple. I didn't know anything about management. So I filled it with jargon. They politely nodded when I presented it, but I wasn't fooling anyone. If there's no content, jargon doesn't help. Unfortunately, it's become polite to accept jargon as a substitute for substance. Why were they not ripping me apart? Or at least, kindly asking me what on earth I wanted to say? My friend Manoj did that. In his nice, humble way, he asked, "But Anand, what does this mean?" When I explained it to him, I found I didn't have a clue. He was OK with that. He just wanted to make sure he hadn't missed something. (That's the technique I use these days. Ask people to explain things clearly. It's OK if they're just lost in jargon. I just want to make sure I haven't missed something.) Don't cloak your ignorance. No one will think less of you. In the long run, you'll learn more, and won't need the jargon. Part 2 of the article will talk about focusing on people and actions; storylining and the pyramid principle; and the structure of messages. Comments Santanu 30 Aug 2012 4:28 am: Starting to get it. Good thoughts Anand. Navneeth 26 Dec 2012 9:55 am: I had the opportunity to be an audience on one of your presentations on data visualization. Now I know why I suck at doing presentations... even though I had content. eagerly waiting for part-2.

Storytelling: Part 1


I show how to stream audio from a PC to iOS by turning VLC into a personal radio station. I configure the HTTP stream and MIME types so you can listen to music through Safari on an iPhone or iPad.

You can play a song on your PC and listen to it on your iPhone / iPad – converting your PC into a radio station. As with most things VLC related, it’s tough to figure out but obvious in retrospect. The first thing to do is set up the MIME type for the streaming. This is a bug that has been fixed, but might not have made it into your version of VLC. Go to Tools – Preferences. vlc-pref-1 Click on “All” to see all the settings. vlc-pref-2 Under Stream output – Access output – HTTP, set Mime to audio/x-mpeg. vlc-pref-3 At this point, you should restart VLC. As I mentioned earlier, you might not need to do this if you have new enough a version of VLC that auto-detects the content’s MIME type. Re-open VLC, and go to the Media – Stream menu. vlc-stream-1 Click Add and choose the file you want to stream. Then click on Stream. vlc-stream-2 Click Next. vlc-stream-3 Select HTTP and click Add. vlc-stream-4 Select Audio – MP3 and click on Stream. vlc-stream-5 At this point, the audio is being streamed at port 8080 of your machine. You can change the port and path in the menu above. (To find your local IP address, open the Command Prompt and type ipconfig.) Open Safari on your iPhone or iPad, and visit vlc-ipad-streaming I haven’t figured out the right codec and MIME type to do this for videos yet, but hopefully will figure it out soon. Comments Shibu 19 Sep 2014 10:12 am: You should try out any of the light weight DLNA servers....

Streaming audio to iOS via VLC


I am transitioning my blogging style from long articles back to shorter, more frequent posts. Managing startup responsibilities has made deep writing unsustainable, so I'll be prioritizing consistent, brief updates over original long-form content.

Until 2007, my blog was mostly just linking to stuff I found interesting on the Web. Since 2007, I’ve tried to write longer articles, mostly based on my own experiences. At the moment, that’s unsustainable. Right now, being in a startup, I doing more stuff than I ever have in the past. (That does not mean working more hours, by the way.) My posts, going forward, are likely to be smaller, less original, but hopefully more frequent. Comments Veera 27 Aug 2012 6:46 pm: Lot of doing implies lot of things to share. But again, time is a real constraint. Anyway, looking forward to your posts. :)

Style of blogging


I scraped PyPI to find the most downloaded scientific Python packages. The results highlight foundational tools like NumPy and SciPy alongside specialized libraries for machine learning, GIS, and seismology, providing a snapshot of the 2012 ecosystem.

I just scraped the scientific packages on pypi . Here are the top 50 by downloads. Name Description Size Downloads numpy NumPy: array processing for numbers, strings, records, and objects. 2000000 133076 scipy SciPy: Scientific Library for Python 7000000 33990 pygraphviz Python interface to Graphviz 99000 22828 geopy Python Geocoding Toolbox 32000 18617 googlemaps Easy geocoding, reverse geocoding, driving directions, and local search in Python via Google. 69000 15135 Rtree R-Tree spatial index for Python GIS 495000 14370 nltk Natural Language Toolkit 1000000 12844 Shapely Geometric objects, predicates, and operations 93000 12635 pyutilib.component.doc Documentation for the PyUtilib Component Architecture. 372000 10181 geojson Encoder/decoder for simple GIS features 12000 9407 GDAL GDAL: Geospatial Data Abstraction Library 410000 8957 scikits.audiolab A python module to make noise from numpy arrays 1000000 8856 pupynere NetCDF file reader and writer. 16000 8809 scikits.statsmodels Statistical computations and models for use with SciPy 3000000 8761 munkres munkres algorithm for the Assignment Problem 42000 8409 scikit-learn A set of python modules for machine learning and data mining 2000000 7735 networkx Python package for creating and manipulating graphs and networks 1009000 7652 pyephem Scientific-grade astronomy routines 927000 7644 PyBrain PyBrain is the swiss army knife for neural networking. 255000 7313 scikits.learn A set of python modules for machine learning and data mining 1000000 7088 obspy.seisan SEISAN read support for ObsPy. 3000000 6990 obspy.wav WAV(audio) read and write support for ObsPy. 241000 6985 obspy.seishub SeisHub database client for ObsPy. 237000 6941 obspy.sh Q and ASC (Seismic Handler) read and write support for ObsPy. 285000 6926 crcmod CRC Generator 128000 6714 obspy.fissures DHI/Fissures request client for ObsPy. 1000000 6339 stsci.distutils distutils/packaging-related utilities used by some of STScI's packages 25000 6215 pyopencl Python wrapper for OpenCL 1000000 6124 Kivy A software library for rapid development of hardware-accelerated multitouch applications. 11000000 5879 speech A clean interface to Windows speech recognition and text-to-speech capabilities. 17000 5809 patsy A Python package for describing statistical models and for building design matrices. 276000 5517 periodictable Extensible periodic table of the elements 775000 5498 pymorphy Morphological analyzer (POS tagger + inflection engine) for Russian and English (+perhaps German) languages. 70000 5174 imposm.parser Fast and easy OpenStreetMap XML/PBF parser. 31000 4940 hcluster A hierarchical clustering package for Scipy. 442000 4761 obspy.core ObsPy - a Python framework for seismological observatories. 487000 4608 Pyevolve A complete python genetic algorithm framework 99000 4509 scikits.ann Approximate Nearest Neighbor library wrapper for Numpy 82000 4368 obspy.imaging Plotting routines for ObsPy. 324000 4356 obspy.xseed Dataless SEED, RESP and XML-SEED read and write support for ObsPy. 2000000 4331 obspy.sac SAC read and write support for ObsPy. 306000 4319 obspy.arclink ArcLink/WebDC client for ObsPy. 247000 4164 obspy.iris IRIS Web service client for ObsPy. 261000 4153 Orange Machine learning and interactive data mining toolbox. 14000000 4099 obspy.neries NERIES Web service client for ObsPy. 239000 4066 pandas Powerful data structures for data analysis, time series,and statistics 2000000 4037 pycuda Python wrapper for Nvidia CUDA 1000000 4030 GeoAlchemy Using SQLAlchemy with Spatial Databases 159000 3881 pyfits Reads FITS images and tables into numpy arrays and manipulates FITS headers 748000 3746 HTSeq A framework to process and analyze data from high-throughput sequencing (HTS) assays 523000 3720 pyopencv PyOpenCV - A Python wrapper for OpenCV 2.x using Boost.Python and NumPy 354000 3660 thredds THREDDS catalog generator. 25000 3622 hachoir-subfile Find subfile in any binary stream 16000 3540 fluid Procedures to study geophysical fluids on Python. 210000 3520 pygeocoder Python interface for Google Geocoding API V3. Can be used to easily geocode, reverse geocode, validate and format addresses. 7000 3514 csc-pysparse A fast sparse matrix library for Python (Commonsense Computing version) 111000 3455 topex A very simple library to interpret and load TOPEX/JASON altimetry data 7000 3378 arrayterator Buffered iterator for big arrays. 7000 3320 python-igraph High performance graph data structures and algorithms 3000000 3260 csvkit A library of utilities for working with CSV, the king of tabular file formats. 29000 3236 PyVISA Python VISA bindings for GPIB, RS232, and USB instruments 237000 3201 Quadtree Quadtree spatial index for Python GIS 40000 3000 ProxyHTTPServer ProxyHTTPServer -- from the creator of PyWebRun 3000 2991 mpmath Python library for arbitrary-precision floating-point arithmetic 1000000 2901 bigfloat Arbitrary precision correctly-rounded floating point arithmetic, via MPFR. 126000 2879 SimPy Event discrete, process based simulation for Python. 5000000 2871 Delny Delaunay triangulation 18000 2790 pymc Markov Chain Monte Carlo sampling toolkit. 1000000 2727 PyBUFR Pure Python library to encode and decode BUFR. 10000 2676 collective.geo.bundle Plone Maps (collective.geo) 11000 2676 dap DAP (Data Access Protocol) client and server for Python. 125000 2598 rq RQ is a simple, lightweight, library for creating background jobs, and processing them. 29000 2590 pyinterval Interval arithmetic in Python 397000 2558 StarCluster StarCluster is a utility for creating and managing computing clusters hosted on Amazon's Elastic Compute Cloud (EC2). 2000000 2521 fisher Fast Fisher's Exact Test 43000 2503 mathdom MathDOM - Content MathML in Python 169000 2482 img2txt superseded by asciiporn, http://pypi.python.org/pypi/asciiporn 443000 2436 DendroPy A Python library for phylogenetics and phylogenetic computing: reading, writing, simulation, processing and manipulation of phylogenetic trees (phylogenies) and characters. 6000000 2349 geolocator geolocator library: locate places and calculate distances between them 26000 2342 MyProxyClient MyProxy Client 67000 2325 PyUblas Seamless Numpy-UBlas interoperability 51000 2252 oroboros Astrology software 1000000 2228 textmining Python Text Mining Utilities 1000000 2198 scikits.talkbox Talkbox, a set of python modules for speech/signal processing 147000 2188 asciitable Extensible ASCII table reader and writer 312000 2160 scikits.samplerate A python module for high quality audio resampling 368000 2151 tabular Tabular data container and associated convenience routines in Python 52000 2114 pywcs Python wrappers to WCSLIB 2000000 2081 DeliciousAPI Unofficial Python API for retrieving data from Delicious.com 19000 2038 hachoir-regex Manipulation of regular expressions (regex) 31000 2031 Kamaelia Kamaelia - Multimedia & Server Development Kit 2000000 2007 seawater Seawater Libray for Python 2000000 1985 descartes Use geometric objects as matplotlib paths and patches 3000 1983 vectorformats geographic data serialization/deserialization library 10000 1949 PyMT A framework for making accelerated multitouch UI 18000000 1945 times Times is a small, minimalistic, Python library for dealing with time conversions between universal time and arbitrary timezones. 4000 1929 CocoPy Python implementation of the famous CoCo/R LL(k) compiler generator. 302000 1913 django-shapes Upload and export shapefiles using GeoDjango. 9000 1901 sympy Computer algebra system (CAS) in Python 5000000 1842 pyfasta fast, memory-efficient, pythonic (and command-line) access to fasta sequence files 14000 1836 Comments Ron Z 25 Jun 2013 10:19 pm: Another good one is simpleCV: http://www.simplecv.org/ Ravindranath M 8 Apr 2013 7:13 am: Nice list Anand. Very useful. This is Ravi from Comviva, who attended your introduction.

The most popular scientific Python modules


I left my consulting career in London to move back to Bangalore and pursue entrepreneurship. I'm joining Gramener as Chief Data Scientist after years of balancing a day job with multiple side projects and startups.

I'm writing this post on a one-way flight from London back to India. I've moved on from Infosys Consulting, and am starting up on my own. I've wanted to do this for a long time. There's always more freedom in your own company than someone else's. There's often more money in it too, if you're lucky enough. But my upbringing is a bit too conservative to make that bold step. However, given that my father runs his own firm, I figured it was just a question of time for me to do the same. Two years ago, in Jan 2010, I picked up Rashmi Bansal's Stay Hungry Stay Foolish at an airport. That book killed the last bit of resistance I had. If the people in that book could succeed, I felt I could too. And if what they did (building small companies, not huge ones) could be called a success, I could be successful too. After the flight, it was clear in my mind. I would be an entrepreneur. I would create a small company that would probably fold. Then I'd do it again. And again, 10 times, because 1 in 10 companies survive. And finally, I'd be running a small business that'd be called successful by virtue of having survived. A modest, achievable ambition that I had the courage for. I usually make big decisions without analysis, by just sleeping over them. I slept over it and announced it to my family the next day. I'm not sure they believed me. Two months later, along with a friend, I built a dynamic digital image resizing product. We had our wives start a company in the UK, and tried selling it to retailers. There clearly was a demand. The problem was, we didn't know how to sell. After a year and having spent £500 with no sales, it was clear to us that venture #1 had failed. We eventually shut it down. In the middle of this, my ex- boss from IBM told me that he was looking to start a venture, focusing on mobile, rural BPO and energy management. This later on changed to data analytics and visualisation. They all sounded like fun, so I said I'll help out in my spare time. A few months later, a classmate told me he'd started a business digitising school report cards. That sounded like fun too, so I said I'd help out in my spare time. Now, if that sounds like I had a lot of spare time on my hands -- you're right, I did. And it's time to talk about the jobs in my life. My first 3 years at IBM were fun. I was coding, learning, and leading a bachelor's life with friends, money, and no responsibilities. My 4 years at BCG were strenuous with 80-hour weeks, but it was interesting and challenging. I was newly married, and between work and home responsibilities, I had no time for fun. I moved to Infosys Consulting in the UK with the specific aim of rectifying that (and for health reasons as well). In the last 7 years, the work has (except on occasion) been a bit boring, but very relaxing. On most days, I would spend 4 hours working, and 4 hours learning new stuff. The things I learnt only helped me be more efficient. So I ended up getting even more work done in less time. Many things came out of this. Firstly, I recovered my health. We had a daughter, and I spent more time with her. I started coding in earnest again. By 2007, I was writing code as part of my projects -- stuff that others whose job it was were unable to. By 2009, I had a few websites running, like an Indian music search engine, an IMDb Top 250 tracker, a few transliterators, and so on. So when I said I'd help out with these startups, it wasn't an empty promise. For the last 18 months, I've had a day job and three night jobs. I never did justice to any of them in my opinion, but I had more fun than ever in my life, I learnt more than ever in my life, and I produced more tangible output than ever in my life. Sometimes, quantity beats quality or reliability. Both these startups are doing well today. Gramener.com offers data visualisation and IT services. I will be joining them as Chief Data Scientist. Reportbee.com offers a hosted report card solution. I will continue helping them out. And I will continue working with a few NGOs. You'll see me a lot more active online now. I can publicly write about my work -- something I've been unable to do the last 11 years. I am relocating to Bangalore. From a professional front, it's an obvious choice. That's where the geeks are. In my last visit to India, I was at Bangalore, Chennai and Hyderabad. In the latter two, it's tough to meet geeks. And when you do, it's no easier to find the next. Bangalore has many more geeks, and they're fairly well networked. From a personal front, too, Bangalore works well. It's close enough to Chennai without actually being in Chennai. It's 10am on Thu 12th Jan. Our flight is descending into Delhi airport. It's the start of a new chapter in my life. Scary, but exciting. Wish me luck! Comments Thejesh GN 13 Jan 2012 9:19 am: Welcome back. AnanthRK 13 Jan 2012 8:51 am: Great to hear that you are taking the plunge full-time. All the very best!! Madhur Singhal 13 Jan 2012 12:47 pm: Anand, best wishes!! Well - I never had a doubt that you take only 4 hrs for 8 hrs of work anyway. I have been guilty of being lazy in calling but thankfully, I travel to Bangalore sometimes. I am at 9873901163 and if you tell me your #, I will call you. cheers!! Ashish 13 Jan 2012 12:31 pm: Hey Anand! Good to hear that, best of luck for your ventures......happy Independance!! Do let me know if i can be of any help Ashish Raghu 13 Jan 2012 12:42 pm: Enjoy always and lead from the front! All the best and do well! Kalpesh 13 Jan 2012 10:22 am: Anand, I am sure, you have all things needed to do better things in life. Hopefully, we will meet & work together in future. Best Wishes Kalpesh Vanaja 13 Jan 2012 11:28 am: All the best Anand! I have always wondered why you were stuck with Infosys when you were so much more. This looks like the way to go. Good Luck and God Bless! Pawan Bhargava 13 Jan 2012 9:55 am: Hi SAnand, good call. It does not matter how it works out (trading in handsight), but all that matters is enjoying what you do. I am sure it would be immensely satisfying. Good luck. Ravish 13 Jan 2012 10:00 am: People said "search" was too late for Google.. they also said "social networking" was too late for Facebook.. good that neither Larry Page nor Mark Zuckerberg listened to them.. All the best Anand.. Good Luck !! Its never too late for new ideas!! Gaurav 13 Jan 2012 1:29 pm: Way to go Anand! You will love being an entrepreneur - the freedom to try different things and the satisfaction you get when something works... 408wij 13 Jan 2012 2:17 pm: Good luck. Joe sriram 13 Jan 2012 2:49 pm: All the best Thala.. :) Somnath 13 Jan 2012 3:35 pm: Just goes to show the fan following you have. I am really late in checking your latest post. Welcome back to Bangalore and India. I see couple of batchmates have posted to. Wish I can join small organization / startup but am hesitant to take the risk. Then again what is my true skill set :D Somnath Harish Kashyap 13 Jan 2012 4:59 pm: Welcome to place where you belong! I am sure you will do well. You can reach me at 99010 32469. It will be good to have a cup of coffee with you! Suresh 13 Jan 2012 5:27 pm: Wish you success. "Real success comes not from being invited onto the yacht, but from being able to paddle one’s own canoe" - Hugh McLeod Sridhar 13 Jan 2012 5:34 pm: All the very best Anand! I am sure you'll have no problem adding tremendous value to whatever you produce and that will ensure your success. praphul 13 Jan 2012 6:54 pm: Congratulations! San 13 Jan 2012 9:12 pm: You've always been an inspiration and it seems like you always will. All the best! Sathish 13 Jan 2012 10:13 pm: Didn't actually get the technical part you are talking about... but looks like your are excited and I'm happy for you... ALL THE VERY BEST!! Madhu 14 Jan 2012 2:53 am: Good luck, many dream of doing it and few take the bold step. I am glad you did. Congratulations and wish you the best. Vasant 13 Jan 2012 4:25 am: Awesome! Welcome back. And best of luck ahead. Girish 13 Jan 2012 3:07 am: Congratulations for new change Anand :-) S-Anand.net is always at the back of my mind for the content it has. satish 13 Jan 2012 7:09 am: Wow. Good to know sir. Wish you all the very best Krishna 13 Jan 2012 6:43 am: Nice to know this Anand. I've been following your exciting works in our intranet blogs. For sure, this is going to be the best decision you ever made :) Wish you loads of luck! Cipher 13 Jan 2012 6:55 am: Congrats and the best for your future Anand. Veera 13 Jan 2012 4:53 am: Whoa! that explains the number the data visualization posts in your blog. :) Congratulations for starting up on your own! Me too in the same situation, going to work on my own start up from coming March onwards :) harish 13 Jan 2012 3:44 am: Hi Anand..... wish you the very best and am sure you will have no problems succeeding... Venkatramanan 13 Jan 2012 3:51 am: Anand Surprising to see you very late to entrepreneurship! I thought you were managing your business! All the very best! As they say in school 'you will definitely come with flying colours!' Regards Venkat chaitanya 13 Jan 2012 1:35 am: All The Best, Anand. It was always knowledgable and pleasure to follow you. Good luck. saurabh nitin 13 Jan 2012 4:33 am: best of luck brother! cheers SThomas 13 Jan 2012 3:20 am: Congratulations Anand, we look forward to hearing more Manish 13 Jan 2012 4:36 am: Anand, congrats I was also struggling with the same thoughts to take up a job or start on own Ayyappa 13 Jan 2012 4:40 am: Best of luck Anand! Btw -I am a daily user of your mixamail for twitter and never said thanks! Thanks a lot for that! Also, I hope you write your experiences in the startup life for "other conservatives who are afraid to jump" to learn how it feels to be in the wild outside. sadhasivam 24 Jan 2012 9:33 pm: Good Luck Anand. Being associated with you as read for a decades i am thrilled to go through post. We wish a great journey ahead. vivek 22 Jan 2012 5:49 pm: Best of luck Anand. Sankar 23 Jan 2012 2:05 am: All the best Anand! MohaN 23 Jan 2012 3:26 pm: As someone mentioned, I stumbled across on a pursuit...Life an Unwrapped GIFT......Tear it or Peal it or Break it...something underneath it... Karthick 13 Jan 2012 8:41 am: All the best. Naveen 14 Jan 2012 5:37 am: Anand- Congratulations , the day I had been waiting for long time. You will make a big impact where ever you are. Good Luck to you and Gramener Rajesh bhura 14 Jan 2012 2:10 pm: Finally! This is teriffic news. From our campus days I always thought that this was the way to go for u. Remember the digital signature idea which u had come up with as our e-commerce project? It's a big industry now in 10 yrs. I don't have any doubt whatever u do will be successful in it's own way. All the best. Your fan for always - rajesh Ashwin 14 Jan 2012 5:42 pm: Anand - I have been following your blog for several years now. I wish you all the best !! Nirav Kanakia 3 Feb 2012 6:09 pm: Good luck Shobha 9 Feb 2012 9:27 am: Hi Anand, You do not know me. But I had read your posts in 2007 and I was inspired by your diverse interests in almost every area. Suddenly I remembered you today and read your recent post and gotta know you are starting your own firm in India. All the best and kudos to you. Shobha- Karthik A 12 Feb 2012 3:30 am: All the best - have thought several times of making the leap but never got out of the comfort zone - Very happy you did. Maybe probability of this visualization thing becoming the next infy is 100% All the best Ravindra 3 Feb 2012 12:17 pm: Thaliva...All the best! Looking forward the IPO in Year 2015 or earlier! Cheers Ramana 16 Jan 2012 3:13 am: All the best Anand! Anand S 16 Jan 2012 1:08 am: Anand Good luck. Our paths are similar (I am anand @ anands.net). After 12 years in silicon valley, I returned to Bangalore an year ago to run my own venture (started this 4 years ago). Let's meet in Bangalore. Venkataragavan 25 Jan 2012 5:25 pm: Hey Anand, wish you good luck and great success ahead. I got interested into data visualization after reading your blogs and now I am actively learning it. Hope to move on someday. Cheers ravi atluri 13 Jan 2012 2:19 am: this is great. Congratulations. Data in India is going to get more exciting :D Ravi 13 Jan 2012 6:01 am: All the best Anand. I've loved your site and all the tools you've built around it. I've always wondered how you got so much time to do all that you do. Gramener seems to be a happening place now and I am sure you will make it a grand success. Amitesh Jasrotia 13 Jan 2012 1:24 am: Hey Anand, I was not really inspired by Rashmi Bansal's book but I am mighty inspired by the piece you have written. Nothing else is more self-liberating than becoming the master of your own destiny. I have been following your blogs/site for quite some time now. Something tells me you will be successful. Still, Best of luck for your future endeavors! Nithin 16 Jan 2012 11:50 am: All the Best Anand... Anuradha Shastry 16 Jan 2012 3:46 am: Lots of good luck!! Landed on this blog just an hour ago. best, another Infoscion Sumit Dhar 16 Jan 2012 6:32 am: Now that you have moved to Bangalore, we should exchange numbers and plan on catching up. Where, in Bangalore, are you based out of ? Cheers, D. D 2 Mar 2012 12:10 pm: Anand, Wish you tremendous amount of luck. Nikhil 31 Jan 2012 9:31 am: Might be of interest to you http://kurt.karmalab.org/2012/01/30/building-a-data-team/ Sreenivasan L 1 Feb 2012 11:11 am: All the best Anand. Wishing you a great success ahead in your new journey. Judy H 23 Feb 2012 7:22 am: I have no idea how I found your website today for the first time, but I've enjoyed reading it. I wish you all the best in your endeavors! Sri 19 Jan 2012 2:50 pm: Hi Anand I have followed your blog several years now. It's great to hear about the new venture. Very inspirational post. Good luck and all the best. Vaalga Valamudan Sri Sajiv 18 Jan 2012 4:20 pm: All the best Anand. pankaj 18 Jan 2012 6:39 pm: Good luck with your new venture. Keep us posted. :) Ranganathan 18 Jan 2012 7:53 pm: Hey Anand, I used to follow ur blogs for the past 2 years. I think i came to know about ur blog when i was searching about the tips in excel . Sometimes if I am bored i used to check ur site for the movie list..:-) .. I just want to say 'All the very best ' for ur new journey.. I am sure definitely u will succeed.....this will be an inspiration for me some day for starting a new company.... S Anand 16 Jan 2012 1:26 pm: @Sumit: definitely. Still haven't move into Bangalore just yet, but my contact number is on the home page. Please feel free to call any time, Shankar V 16 Jan 2012 2:04 pm: hey Anand All the very best and trust me.....I knew this was coming! :) Welcome back to India and welcome to Bangalore. Hope you continue to post interesting stuff. Please do stay in touch. I had your Infosys email id. Don't think I have your personal id. Please pass it on. ranjit bawa 16 Jan 2012 4:09 pm: hey Anand - great to hear!! good luck with the new chapter and look forward to seeing you soon. hope to follow suit, but till then, shall live vicariously... S Anand 18 Jan 2012 1:46 am: @Shankar, I keep my latest phone number at the top of my home page. Will be good to hear from you! Rajesh B 18 Jan 2012 7:00 am: Great to know you are starting off on your own. Good luck. I took my one way ticket to India 3 years ago. Still burning through my first venture. Inspiring post. Am planning to start 9 other on the side just to be on the correct side of the odds. Jokes apart, can think of few others more equipped to start off on something. If ever you are interested in doing something on the education space, drop me an email. Will try to make that as my night job. Am good with creating content, abysmal poor in trying to sell it. So, ideal for a night-job. Best wishes. Cheers, Bee Ravi Tiruvaipati 18 Jan 2012 1:13 pm: Hi Anand, I have been following your posts and your applications...It is good hear that you are planning to start your own. I wish you all the best!! Sriram S 15 Jan 2012 6:14 am: Welcome back and all the best...I stay in BTM Layout in Bangalore..lets catch up sometime Sudheer 15 Jan 2012 9:35 am: All the best Anand! Looking forward to hear more and more about you in the coming days from sources other than the blog Jitendra 15 Jan 2012 3:25 pm: Wish you luck Anand! I am sure with your attitude and energy you'd do well in whatever you try to do! Raja 24 Aug 2012 3:25 am: I try to visit you website since 2008 whenever I get a chance and look for what geeks are doing lately :-) and just noticed today that you moved to India starting out your own venture. Hard work pays off for sure. I am sure you will succeed. Good Luck! Michael 22 Oct 2012 7:16 pm: Good luck with this Anand. hope it is going well.. Michael mohan 11 Nov 2012 7:50 am: all the best mr anand keep going your website is so informative with high entertainment value , you will suceed god bless you mohan. you will definitely suceed in Amutha 25 Nov 2013 9:37 pm: It's very late if I say gud luck,:) and I hope you are going great with your plans. Ankur 15 Apr 2013 8:22 am: Hi Anand, I first heard of you courtesy Rahul Guha during my first year at BCG in Mumbai in 2006. He mentioned you as a Consultant he had a very high opinion of. I came across you next a couple of years later, as I was trying to index by IMDB 250 list - I used the app you'd created and recommended it to many friends too. Today, I reached your blog while browsing through some data on poverty in India (datastories.in). Just wanted to let you know. I wish you all the best in your venture(s). Ankur Father of the bride - S Anand 1 Nov 2023 8:25 pm (pingback): […] In 2012, I started Gramener with half a dozen friends. […] 10 years later - S Anand 17 Mar 2022 1:24 pm (pingback): […] 12 Jan 2012, on a flight back from London, I […] Moving to Singapore - S Anand 2 Aug 2022 6:37 pm (pingback): […] “I lost my job. Please don’t tell anyone. Let’s go back to India.” We left 2 weeks later to join […] A challenge of blog questions - S Anand 4 Mar 2025 4:44 pm (pingback): […] The next chapter of my life, which I wrote on a one-way flight back from the UK to India to start Gramener. […]

The next chapter of my life


I believe the traditional "three Rs" must evolve: reading remains vital, but typing and computing have replaced handwriting and mental arithmetic. I plan to help schools prioritize Excel and digital literacy over outdated manual skills.

Reading, wRiting and aRithmetic are the 3 'R's that are taught at school. I was thinking about their relevance today. Reading continues to be relevant. The volume of information available today is more than before. So you need to read faster AND smarter. (If there was one good thing that came out of my IIM coaching classes, it was the ability to read fast, and making it subconscious.) But I wouldn't say the same of writing. In the last 10 years, I have typed several hundred more pages than I've written. So have all my friends. Yesterday, I was at a bank with a relationship manager as he was taking notes in paper and pen. I do the same on occassion. I looked at his notes later. I could not understand a single word. "Don't worry, sir, I can read it. I'll type it out and mail you," he said. And he did. Writing seems to have become a device for personal memory, not communication. He's faster at writing than typing, perhaps. Or note taking is more convenient on paper. But for communication, he still prefers a typed format. So do I, and most other people. Perhaps writing will fade. Perhaps not. I don't know. But what I do know is that typing has become more important than writing. Yet, writing is taught more at school than typing. (A broader aspect of writing, though, is expressing oneself. That will remain important, of course.) The third R is aRithmetic. When I was 12, I could multiply four-digit numbers in my head reasonably well. I could recite 50 digits of Pi. I could do long division. Today, I can't. Nor can my friends. Nor have we needed to. A good feel for the numbers has helped, but not the actual mechanics of the calculations. We had an undergraduate course in statistics that taught us how to solve a linear regression problem. That skill went completely unused. I've never since used regression without a computer. We had a graduate course in statistics that taught us how to INTERPRET the results of a linear regression. That was worth it's weight in gold. This is not a critique of the three Rs. Rather, an attempt to re-interpret them. It's about comprehension, expression and computation. Two decades ago, it was reading, writing and arithmetic. Today, it's reading, typing and computing. Computers will grow more powerful. It may be worth planning for it. Teaching the ability to use them can go a long way. A tool like Excel for general purpose computing gives incredible power in the hands of people. It's worth training children for that. If I oversimplified, I'd say children must learn typing and Excel. Over the next few years, this is something I plan to work on. Making sure schools and parents do this. Any suggestions or leads you may have are welcome! Comments Arun 7 Feb 2012 12:05 pm: I find the 3 Rs to be redundant but still useful for cognition. I find any computer input device to be limiting because I cannot doodle or sketch if I need to while mapping an idea. Paper offers freedom to graphically detail a problem in ways that cannot be replicated easily or effortlessly on a computer (or a tablet for that matter). Somehow whiteboards the best way to communicate to a team than Powerpoint. Agree about aRithmetic. We almost never find ourselves without a computing device . But should we be completely reliant on them? What if you are marooned in an island? :D Today, I would simply add a fourth R - 'Recursion'. Implying the art of programming, the lack of which has become a serious handicap. Somnath 23 Jan 2012 7:04 pm: You don't need to teach typing I guess - kids these days have computer at home if not at private schools. My 1.5 year old has seen me type and has taken to imitate that. Over time they will learn typing out of necessity. Excel I agree you need to teach as it might not be seen as necessary - not Word or Powerpoint though as they will pick it up out of necessity due to school work. Having said that I still carry a simple ball-point pen and a paper whenever I get in a meeting or on my desk to take notes. No technology can get in that as I see it as obstructive. Ken Jones 22 Feb 2012 6:59 pm: Anand, I stumbled across your site 4 years ago and have always loved it. I agree that Excel is a wonderful tool to teach. It has the beauty of being as simple or and fantastically intricate and complicated tool you can make it. And these days you can link it to all the xml you want. I have always thought that logic - especially as it applies to programming - is very useful. However, I feel that there need to be more visual tools to assist in its education. Kids are able to follow nested loops and if/then stateemnts if they have an easy way to navigate the logic initially. I became very interested in the way Montessori teaches mathematics at an early age. But I have yet to see effective tools like that for logic education at a young age ( 4-10) Judy H 23 Feb 2012 6:55 am: I have noticed for a long time that my handwriting has gone to hell ever since I've had a computer at my desk. I had nice handwriting, and now it looks awful and feels cumbersome as well. I have heard that school districts have stopped teaching cursive writing to children. Imagine! Not that I disagree, how much really free hand writing does anyone do anymore? Maybe this will free up time to teach them things they really need, as some noted already: logic, analytical thinking, problem solving, and common sense. My background is in math and statistics. I spent many years working with people in other sciences who had a tough time working through a problem, or setting up an experiment, in a logical and thoughtful way -- planning and executing, learning from the result, and setting up the next iteration. These types of skills, along with communications skills (getting your point across as well as listening), are critical for success in life. I don't think Excel has entirely taken away the need for learning arithmetic, but I would like to see kids get an answer to a problem and know whether they have gotten the right answer (so some time of quick estimation method is needed). I emphasized it with my kids -- you can easily use a calculator, but how do you know you got the right answer? Does the answer you got make sense? To round out the basic public school education, I would include lots of other topics, but mostly to expose kids to what is out there -- literature, geography, philosophy, anthropology -- to make them more curious to explore on their own. Every thing is at your fingertips these days, I love it! Why are we still forcing kids to read books they hate? I would rather let them gravitate toward what thrills them -- Lord of the Rings instead of Huck Finn, or Tolstoy instead of Shakespeare, whatever turns them on. Rajesh Bhura 14 Mar 2012 9:35 am: Anand, what do you think about the utility of tools like abacus for teaching aRithmetic to children? Spandana who is 9 yrs old has been going to this UCMAS class for 3-4 yrs and i am still to find true value in this. i know i am deviating from the main discussion but wanted to know your and the views of others pls. http://www.ucmasindia.com/mental\arithmetic/aboutusnew2.html Karthik A 12 Feb 2012 3:37 am: I feel that logic (aRgument) helps increase flow for children and sometimes, like sleeping a problem over, helps in generating leaps of logic (intuition or conceptualization). We need to teach these in place of writing. Long back I remember my grandfather telling me that they used to have logic as a subject! Ravi 10 Apr 2012 10:40 pm: http://www.themodernword.com/eco/eco\future\of\book.html This is from Umberto Eco, one of the foremost and important intellectual of our times. This is his area "Semiotics" - the science of signs. Somehow I remembered this discussion and thought it would make sense to post this here. Ravi 20 Feb 2012 9:44 pm: The three R's are going to be relevant as long as civilization exists - their forms and formats will undergo constant shifts and changes. All these three R's are at some level only abstractions. If you think about it, reading was always a tool of internalization of concepts and communicating and creating within, writing was for externalization and communication without, and arithmetic was just a name given to the abstract name for the reliazation of the natural phenomena through a number line. By writing, if you meant translating your thoughts on a paper using a pen/pencil - yes that is receding; but the act of translating your thoughts onto something has not changed. In fact, if a medeival saint who perhaps wrote on "tala" leaves using a stylus magically appeared in 1950's, he would be flabbergasted by the meaning "writing" had acquired; same surprise that our grandfathers would express watching our kids type on computer screens and tablets. Arithmetic, however if narrowed down to doing math in the mind has definitely as an academic discipline has taken a violent beating at the hands of xcel, the ubiquitious calculator; but the concept of numbers has not. Neither will till the civilization exists. deepan 24 Jan 2012 8:12 am: Programming is still a subject that scares many. Misguided Focus on syntax rather than on logic and problem solving ,makes the learner diffident .Excel offers an easy way to dabble ,solve problems and experience sheer joy of working smart. Shankar V 24 Jan 2012 9:21 am: I agree with Somnath. Note taking is still paper and pen in my case. I tried using the evernotes, the onenotes and the sticky notes. I simply lose my way there. With a notebook, it is all chronologically sequenced and I know what I am looking for the moment I open the pages. But writing skills other than personal note taking is almost dead. And we still have schools teaching "cursive writing" skills to kids. :) Rajesh B 26 Jan 2012 6:04 am: Good points. A redefinition of the 3 R's is definitely called for. But, I would still add typing and excel as auxiliary skill sets, and spend lots of time on the basics. Excel, typing are more "skill sets" to have. At very young ages, it is very important to build the mind to be able to process stuff, rather than be able to "know" stuff. Kind of like the difference between academic education and vocational education. No offence to excel, but it clearly falls under "skills" category. As in, if we can make our kids more receptive and intelligent, they can figure out excel without much difficulty. Writing (in the communication aspect, not the putting pen to paper aspect), Reading and arithmetic all play a role in shaping the mind. Comprehension, articulation and analysis (not necessarily just computation) are important bits in piecing together a keen mind. I would look at all of schooling as the phase where the mind is made better. And not really worry about pure "skill sets" In computer terminology, till age 16 or 18, work on the RAM. The apps can come later. If your RAM is cranked up really well, loading apps is very easy. And the hard-disk generally is sound enough for the basics as well. Most of the students that I interact with (all in the 21-30 age group) have good apps, hajaar heavy hard-drive, but are running on a 128 MB RAM. I would want entire schooling to focus on improving processing ability and stamina. Everything else can come later. We can (and probably should) de-emphasize writing, and supplement it with typing. We are essentially working on articulation. As far as arithmetic goes, we should de-emphasize computation and increase the role of context and analysis. Include logical reasoning type questions in the pedagogy in order to combine analysis and comprehension skills. We should definitely expand the role of reading. But I am a little old-fashioned so would want to keep original frameworks. My biggest fear is that we introduce an app in the hope that it will improve processing power and somewhere down the line it becomes a task in itself and loses all value. This is what has happened to our ability of multiplying two 2-digit numbers mentally. Originally, it was a way to teach kids to have an intuition about size. Now, it is a cr\p vedic maths module with no basis. Pardon my language, Anand. I do not know if I have communicated clearly here. When you (Anand) were asked to do 47 \ 74, it might have been to tell you that it is somewhere close to 50 \ 70. And to help you to process and remember at the same time. Now, it is a template and there is no-one who discusses how this is close to 50 \ 70. Amlan 17 Feb 2012 3:12 pm: Products like the Livescribe smartpen will merge the worlds of writing and computing. Ravi 10 Apr 2012 10:44 pm: I hasten to add that this was a lecture in a 1994 symposium when Steve Jobs was still disillusioned about Apple and was making movies and not dreaming his "i" devices :-) yet the discussion is useful and the cross-references to historical and intellectual motifs are invaluable.

The three Rs


I share the easiest method for setting up a Windows XP virtual machine using Microsoft Virtual PC. It's ideal for testing IE6-8, and I found the memory footprint surprisingly small compared to tools like VirtualBox or VMWare.

Here’s the easiest way to set up a Windows XP virtual machine that I could find. (This is useful if you want to try out programs without installing it on your main machine; test your code on a new machine; or test your website on IE6 / IE7 / IE8.) 1. Go to the Virtual PC download site. (I tried VirtualBox and VMWare Player. Virtual PC is better if you’re running Windows on Windows.)\ image\ If you have Windows 7 Starter or Home, select "Don’t need XP Mode and want VPC only? Download Windows Virtual PC without Windows XP Mode."\ If you have Windows Vista or Windows 7, select “Looking for Virtual PC 2007?” 2. Download it. (You may have to jump through a few hoops like activation.) 3. Download Windows XP and run it to extract the files. (It’s a 400MB download.) 4. Open the “Windows XP.vmc” file – just double-clicking ought to work. At this point, you have a working Windows XP version. (The Administrator password is “Password1”.) 5. Under Tools – Settings – Networking – Adapter 1, select “Shared Networking (NAT)”\ image That’s pretty much it. You’ve got a Windows XP machine running inside your other Windows machine. Update (18 Sep 2012): I noticed something weird. The memory usage of VMWindow and vpc.exe is tiny! image Between the two processes, they take up less than 30MB of memory. This is despite the Windows XP Task Manager inside the virtual machine showing me 170MB of usage. I’ve no clue what’s happening, but am beginning to enjoy virtualisation. I’ll start up a few more machines, and perhaps install a database cluster across them. Comments Vivek T 16 Sep 2012 6:53 pm: I agree that Virtual PC is the easiest to use. However, Microsoft has stopped all development on it for about 2 years now. We use Virtualbox since it's open-source and provides access to virtually all USB devices connected to the host and also provides 2D & 3D graphics acceleration. Fuu 9 May 2013 11:42 pm: how do you download windows xp.vmc file?

Windows XP virtual machine


I expect to be trusted when invited to corporate offices, refusing invasive security searches or ID checks. If an organization's protocols treat guests like threats, I prefer meeting outside or at my own office.

We are often subject to body searches, baggage inspections, and identity verifications. At malls. At airports. At offices. These are to ensure that no one carries ammunition inside, or goods or secrets outside. In other words, to deter terrorists and thieves. It's nothing personal, of course. When someone does not know me, I can choose to accept that (or not; the choice is mine). When I'm invited somewhere, however, I assume that I am not deemed a security threat. Therefore, I expect that: My and my belongings will not be searched or scanned I need not leave behind my personal belongings I need not carry an identity card Please afford me this courtesy if you are inviting me. For some months now, I've visited many corporate offices. The reception is comprised of security guards, a metal detector and a register. I'm given a tag and an escort. I'm not fussy. I'm not worried about being greeted, for example. I'm quite happy to plug into a power socket and work on my laptop until logistics are sorted out. But when that happens at the security outpost with no sitting space, or outside the gate in the rain, it inconveniences me. A few weeks ago, I was at Singapore, and visited a client's office in slippers. One of them complemented my choice of footwear, and remarked that he had not yet risen high enough in the corporate ladder to afford this luxury. (There’s a series of stories behind my footwear that I’ll get to later.) That told me something. After a long time, I now can afford this luxury. Especially if someone knows me well enough to invite me to their office. I hope to point them to this blog post and request that security be arranged so that I can be afforded this small courtesy; be treated with trust rather than as a terrorist or a thief. (If their organisation's practice does not permit this, I'm happy to meet outside. Besides, our office is happy to extend warm hospitality.)

Courtesy


I created a VBA function that geocodes addresses directly within Excel using the Google Maps API. It uses XPath to extract latitude and longitude from XML responses, making it easy to map data without leaving your spreadsheet.

It’s easy to convert addresses into latitudes and longitudes into addresses in Excel. Here's the Github project with a downloadable Excel file. This is via Visual Basic code for a GoogleGeocode function that geocodes addresses. Comments Ryan 8 Jun 2015 9:28 pm: I find this isn't working and says, Compile Error; User defined type not defined xDoc As New MSXML2.DOMDocument what do I change to fix it? Thank you Richie Lionell 27 Jul 2016 6:40 am: Ryan, Inside the VBE, Go to Tools -> References, then Select Microsoft XML, v6.0 . If that doesn't work unselect that and select Microsoft XML, v3.0

Geocoding in Excel


I am migrating away from Google products following the shutdown of Google Reader. I've uninstalled Drive and Talk, and I'm seeking self-hosted or open-source alternatives like OpenStreetMap and Chromium to protect my workflows from future service cancellations.

Google Reader was where I spent most of my browsing time, but now, it’s shutting down. Time for alternatives, but not just for Reader: for all Google products. I’m not sure when one of these might go down, become paid, or become unusable. I just uninstalled Google Drive and Google Talk. but I don’t use it much (I use Skype), so no loss. I’ll leave Chrome for the while, but I’m hearing reports that Firefox is improving faster than Chrome is. Or there’s Chromium. I’m not worried much about search services (including image, video, scholar and books). When needed, I can switch. Scholar might be a bit sad to lose, but I don’t use it much. Google Translate, too, isn’t essential. Likewise for content. YouTube’s not a problem. There’re enough other video services. Trends are useful, but not critical. Maps might be, so I’ll try and switch to OpenStreetMap. I don’t use News or Picasa much. I don’t care much for social media anyway, so Blogger, Orkut and Plus can die any time. Google’s apps are the worrying ones. Mail and Calendar, in particular. I’ll probably migrate away from them last, but the attempt is on. I’ll be documenting the alternatives I find at https://gist.github.com/sanand0/5176161 (safely cloned locally). Looks like there’s no safe long-term alternative to being able to host your own apps. Pity. Comments Hari K T 20 Mar 2013 12:23 pm: You are right. Probably many of them will follow your path :-) . Dan Murray 16 Mar 2013 2:01 pm: It is annoying...isn't it. Ravikumar Venkateswar 18 Mar 2013 6:33 pm: Reader's the one that bugs me the most, since they basically out-competed all of the alternatives, even the paid ones. I prefer paid apps, personally, because I know where the money's coming from, and there's less chance that they'd make them unusable. I switched from Picasa to Smugmug a while ago, and I'm probably going to move Mail and Calendar to free Google Apps or maybe my employer's service. Why would you host your own apps instead of paying directly? Hosting options | s-anand.net 1 Jun 2013 6:49 am (pingback): [...] means. I can’t just take that code and move it to another server. Besides, I’m a bit wary of Google pulling the plug. Heroku? Same problem. I just want to take the code elsewhere and run [...] Tarun 21 Mar 2013 11:44 am: What about the appengine, did you explore any alternatives for it? I was quite impressed with script.google.com also, its in-line with your style of coding . Dev E 22 Mar 2013 3:22 am: How about Nokia's here.com for maps? Mathew 23 Mar 2013 12:38 am: Feedly if you plan to keep Chrome. Om Shankar 16 Jul 2013 11:35 am: Dude. I seriously hope you were saying that all out of anger and they are not your actual insights of other Google products. (Some were ridiculous - like Maps, Gmail, +plus, Chrome, etc.) Coz, if it were, you definitely need to come back to internet in today's age and have a check again on your updates. Looking forward to your conf. at Bangalore JS. :-)

Goodbye Google


I compared hosting costs across GAE, Azure, and AWS. I chose Amazon EC2 spot instances in Singapore, using shell scripts for fast recovery to cut my monthly hosting bill from $230 to just $29.

I've been trying out a number of options for hosting recently, and have settled on Amazon spot instances . Here were my options: Application hosting, like Google AppEngine . I used this a lot until 2 years ago. Then they changed their pricing, and I realised what “lock-in” means. I can’t just take that code and move it to another server. Besides, I’m a bit wary of Google pulling the plug . Heroku ? Same problem. I just want to take the code elsewhere and run it. Shared hosting, like Hostgator . This blog is run on Hostgator and I’m extremely happy with them. But the trouble is, with shared hosting, I don’t get to run long-running processes on any ports I like. Run you own servers. The problem here is quite simple: power cuts in India. Dedicated hosting, like Amazon EC2 , Azure , GCE , etc. This remains as pretty much the main hosting option I’m a price optimisation freak. So I ran the numbers for a year’s worth of usage. I was looking at the CPU cost of a large machine with 7-8GB RAM. Bandwidth and storage are negligible. The cost per hour worked out to: Amazon: $0.32 / hr in Singapore, $0.24 in Virginia Google: $0.29 / hr in Europe Microsoft: $0.32 / hr in US The price is not all that different, but I need low latency , so Singapore it what it’ll have to be. EC2 location Latency (ms) Singapore 139 Oregon, US 334 Japan 517 Ireland 618 Australia 620 California, US 677 Virginia, US 710 Now comes the choice of the right model. At $0.32 per hour, that’s $230 a month. Amazon offers some ways of getting this down. Instead of on-demand instances , I could go for reserved instances . For a year of usage, that’d get the price down to about $131 a month, nearly halving it. ($739 upfront for a heavy utilisation large reserved instance, with $0.095 24 365.25 for the year.) In this case, I know I’ll need the servers for a year. Probably more, but then, I might want to switch later. So this isn’t a bad move. But we can do better. Amazon also offers spot instances . Spot instances might get shut down any time – but in reality, so can on-demand instances. I need to plan for it anyway. I’m not going to host anything that’s so sensitive that if it’s down for a few hours, I’ll have a problem. But what’s attractive is the pricing. Typically, it’s $0.04 per hour, making it about $29 per month. Even if it shoots up to twice that, at $58, it’s less than a fourth of the on-demand price and less than half the reserved instance price. I’ve managed to script the entire setup up sequence as shell scripts, and it takes less than an hour to get a new server up and running the software I need. I need to work out a decent backup mechanism. Plus, I could use more reliable storage like like Amazon’s EBS to preserve the data. But on the whole, the pricing is far too attractive and makes the risks worthwhile. Comments Anand Chitipothu 2 Jun 2013 3:36 pm: Did you look at Linode? I've been using Linode for couple of years and pretty happy with it. The 8GB RAM model costs about $160 per month. Sadly, they don't have a data center in Singapore. You can test the download speeds and latency of their data centers from https://www.linode.com/speedtest/. jouko 7 Jul 2013 4:21 pm: I am using Inmotion hosting for my blog. May be i should also shift to amazon cloud. S Anand 3 Jun 2013 7:30 am: I did try Linode early on, and it's a fairly decent option. It's just that for my needs, the $29/month that I can get with Amazon is too attractive. Amit Chakradeo 3 Jun 2013 8:33 pm: If you don't mind occasional downtimes and occasional ummm complete loss of data :-), you can consider low end VPS boxes. (http://lowendbox.com/) You can find some decent deals if you check out the posts there... Sriram 15 Jan 2014 4:18 pm: have u tried openshift ?

Hosting options


I'm highlighting the Agastya Foundation's mobile science labs, where volunteers carry experiment kits on motorbikes to rural schools. The project is a finalist for the Google Impact Challenge, aiming to bring hands-on STEM education to remote areas.

My cousin’s working on an interesting project at the Agastya Foundation. A group of scientifically inclined volunteers go around on a bike to schools, taking with them a science lab kit, and show children in rural schools a variety of experiments. Google will award this and 3 other projects (out of 10) Rs 3 crores based on public votes. You can vote for and read more at https://impactchallenge.withgoogle.com/india2013#/agastya|vote Comments Motorbike science lab http://t.co/WWsD9YxGEX - Thej Live 25 Oct 2013 2:14 pm (pingback): [...] Motorbike science lab s-anand.net/blog/motorbike… [...]

Motorbike science lab


I navigated corporate resistance to open source by demoing prototypes directly to business leads and avoiding the 'open source' label with IT. I learned to deliver the full stack personally while adapting to rigid infrastructure constraints and security policies.

[This is a post that I’d published internally in InfyBlogs in Dec 2009. Time to share it.] Last month, my first application went live. I’ve been writing code for 20 years. Not one line of my code has been officially deployed in a corporate. (Loser...) It’s a happy feeling. Someone defined happiness as the intersection of pleasure and meaning. Writing code is pleasurable. Others using it is meaningful. But this post isn’t quite about that. It’s about the hoops I’ve had to jump through to make this happen. I’ve been living in a nightmare since March 2009. That was when I decided that I’d try and get corporates to use open source. March 2009 It began with a pitch to a VC firm. They were looking to build a content management system (CMS). Normally we’d pull together slides that say we’ll deliver the moon. This time, we put together demo based on WordPress’ CMS plugins. The meeting went fabulously well. We said, “Here’s a demo we’ve built for you. Do you like it?” The business lead (Stuart) was drooling and declared that that’s exactly what they wanted. The IT lead (another Stuart) was happy too, but warned the business users: “Just remember: this isn’t how we do development, so don’t get your hopes up that we can deliver stuff like this :-)” Time to make my point. I asked, “What’s your policy on open source software?” The business lead went quiet. “I don’t know,” he finally said. Fair enough. I turned to the IT lead. “Well, we don’t use it as a matter of policy... there are security concerns...” he said. “Which web server do you use?” ”Oh, OK. I see what you mean. We use Apache. So on a case to case basis, we have exceptions. But generally we have security concerns.“ ”Why? Do you believe open source software is more insecure than commercial software?“ He thought about it for a while. “Well... maybe. I don’t know.” We debated this a bit. Then we found the real issue: “It’s just that we don’t have control over the process. We don’t know enough about it to decide.” A couple of weeks later, I tried pitching to a newspaper. This time, it was our sales team that raised the same question. “But... isn’t open source insecure?” I didn’t even bother pitching any open source stuff to them. But I’d learnt my lessons: 1. Demo the application. Don’t talk about it. 2. Show it to the business first, and then tackle IT. Aside: June 2009 In June, I got another chance at a client where we were building their new website. The very first thing I did was ask to see the Javascript. Total mess, and filled with browser-incompatible DOM requests. So I went over to their web development team. “Look, why don’t you guys use a Javascript library? It’ll get you cross browser compatibility and compact maintainable code at the same time.” And, to their credit, they said, “Sure. Which library?” I showed them this and we agreed on jQuery. So, if nothing else, I’ve managed to get one open source library into a corporate. July 2009 I was also looking at payments on the website, and our client was looking to replace their chargeback application. Since I had a week off, I built a working PCI compliant prototype on Django. (I must clarify what I mean by PCI compliant. You see, any application that stores credit card information must pass through a stringent security clearance process. I bypassed the problem by not storing the card information. I’ve realised that I’ve been building PCI compliant applications all my life – and it’s a huge benefit to let people know that.) This time, I applied the lessons I’d learned, and demo-ed it to the business, who were thrilled. Time to tackle IT. I started with the architecture team. Matt on the architecture team was the most approachable. So I went over, demo-ed it, and said, “Matt, this took a week to put together. It’s based on some new technologies. Are you game to try these out?” He was. And quite enthused about it too. So we put together a proposal for the architecture review board, proposing a new technology stack: Django / Python and MySQL. As before, I showed the demo before I talked technology. I had prepared answers to all security related questions upfront (and practically memorised section 3 of the PCI guidelines.) The clincher, though, was the business case. To build it on Java, it would cost 1,000 person days. On Django, I’d mostly done it in 5. There was no way of justifying 1,000 person days for an application that could save, at best £100,000 a year. So they said “Go ahead, we’re fine if operations and infrastructure are fine.” It was time to find a Django developer in Infy. I hunted for a couple of weeks but none was available. (Only 2 people that I knew knew Django in the first place.) So that effort got canned, and we were back to the 1,000 person day solution. (Which got canned too, later.)\ But in the process, I’d learned my third lesson. 3. If you’re trying new technologies, plan on delivering it yourself. October 2009 Another application popped up that looked like a prime candidate for introducing open source. They were using an Excel application to fraud screen orders, and wanted to make a web app out of it. I followed the same route as before. Demo it. Show it to business first, then IT. Built it myself. I skipped Architecture, since they’d already approved the technology stack, and took it straight to Infrastructure. “This application uses Apache as the web server, MySQL as the database, and uses PHP and Javascript for the application logic. Could we get a Linux server to host it?” Our entire conversation lasted 30 seconds. He said, “No. We use Windows servers” (I was fine) “... and you’ll need to chance Apache to IIS” (fine again) “... and we don’t support PHP, so it’ll have to be Java or .NET” (I don’t know .NET or Java... but fine) “... and we don’t support MySQL, it’ll have to be SQL Server” (fine, I guess) “... and we don’t have DBAs available until January, so you’ll have to wait.” (definitely not good.) So back to the drawing board on the technology stack. I needed something in Java (I know very little Java, but nothing at all in .NET) and to avoid the DBA headache, it would have to bundle in a database. I first explored key-value stores like CouchDB, Redis, etc. None of them worked on Java. The only one I found that did was Persevere, and it was a JSON data store, which fit perfectly with my plans. By this time, I’d also learn my my fourth and most important lesson. 4. Don’t try to promote open source. Just deliver the application I said, “This is a custom-built application that runs on Java. Could we get a Windows server to host it?” The answer was “Yes”, and we had it live the next day. PS: December 2009 The application’s deployed and running. It has about 10,000 orders fraud screened by now.\ And the lessons are well learnt. So when some came over asking if there was any image resizing solution I knew off, I said: “Sure, who’s your business sponsor?” Then I went over and said, “Let me show you this open source application called ImageMagick. It handles aspect ratios correctly, and can crop too. Doesn’t this look professional?” Then I went over to IT and said, “It’s open source, so you can change it. It has Java bindings, so you can integrate it into your environment. It can handle 8 3000x2400 images a second on my puny laptop. It’s used by your competitors. And I can build it for you if you like.” I might just have my second open source entry into a corporate this year. Comments Satyakam Goswami 13 Jun 2013 4:24 am: Hmm interesting post , we sure should exchange our notes on these kind of experiences sometime which i guess no MBA college can teach . Shankar V 20 Jun 2013 1:58 pm: hi Anand This is interesting. I am currently working on a development of a product and we are debating which way to go. Should we go with the standard Oracle tech stack of Oracle 12g + Weblogic + Fusion Middlware + ADF & OBIEE (for BI) and build using JDeveloper. Or should we instead go for MySQL + Hibernate + Springs FW + jQuery & Jasper/Pentaho (for BI) and develop using Eclipse? While the former is standard software from an established vendor, the latter brings down the cost of operations. Plus, we are providing product support anyway. So will that not tilt the CIO decision in our favour? Or do you think they will be apprehensive despite all this just cos it is Open Source? Will be interested to know your thoughts on this. shankar v Yuvi 21 Jun 2013 10:20 pm: I'm still struggling to answer a larger question of 'why bother?'. You probably weren't in a position where you were scared of not getting a job if you quit. Why did you stay? Tarun 3 Jul 2013 6:52 pm: All I can say is congratulations and I am J ... Narendran 22 Jun 2013 6:37 am: Very well written and I totally agree. According to corporates, bringing in an open source software as well as open sourcing a homegrown software - both are insecure. I am yet to find a plausible explanation. Apart from the problem of reinventing a wheel, this greatly limits a software developer's versatility (unless he gets his personal time to play around with tech). Eventually, he ends up using the one big hammer to nail down all petty problems (irrespective of the type, complexity and scope of the problem). But ultimately, as yuvi says, why bother about all these when you have a good command over technologies. Get to a place that appreciates and supports you instead of fighting an endless war. Naveen 6 Sep 2013 7:52 am: Open source is viewed by lot of corporates as "free" , I realize that anything given free is not of any value ....Nice once Anand, I can echo with lot many questions I asked you during our initial days of Gramener....you would have written me off as one other guy with same questions :-) Ashish 5 Jun 2013 9:09 am: Hi Anand, I am no technie but I find your dabbling (as I see) and outcomes of various nice tools interesting and been following you for couple of years now - started when I found your search music by name application. So much for introduction. Recently, I was looking for list of Hindi and English movie names to build an automated application (in Excel) to randomly pick a movie. Basically, motivation was to play Dumb Charade when there are not enough members so that we don't lose out one person who is supposed to give movie name. I couldn't find any easy to download list though BollywoodHungama and IMDB have them in format that you can probably scrape and build whole thing in less than an hour. So would you help me? It could be interesting tool for you too! If not, I just need dump of all movie names in txt line by line for my purpose. To filter out list since it is going to be too big and for me to process in Excel, we may put IMDB rating > 4 restriction or some such.

Open source in corporates