data-science on S Anand

Fri, 12 Sep 2025 00:00:00 +0000

Slides for my DataHack Summit talk (controversially) titled RIP Data Scientists are at https://sanand0.github.io/talks/2025-08-21-rip-data-scientists/

Summary: as data scientists we explore, clean, model, explain, deploy, and anonymize datasets. I live-vibe-coded each step with DGCA data in 35 minutes using ChatGPT.

Of course, it’s the tasks that are dying, not the role. Data scientists will leverage AI, differentiate on other skills, and move on.

But the highlight was an audience comment: “I’m no data scientist. I’m a domain person. I’ll tell you all this: If you don’t follow these practices, you won’t have a job with me!”

They’re catching on! 🙂

Mon, 28 Feb 2022 05:10:55 +0000

I’m planning to publish a 3-hour self-paced #onlinecourse.

But I don’t know which topic would be more useful.

One topic is #datascience tools for non-programmers.

Another is a step-by-step guide to #datastorytelling for analysts.

What’s more useful for you?

Could you share with people, so I work on the more useful course? (Thanks 🙏)

Mon, 08 Apr 2019 10:16:58 +0000

If you’re a seasoned developer that enjoys working with data, have good front-end skills, and are challenged by impossible deadlines, please drop me a note. I’d love to work with you at Gramener Mumbai.

Thu, 05 Nov 2015 12:25:36 +0000

Apparently, I’m one of the top 10 data scientists in India. http://analyticsindiamag.com/top-10-data-scientists-in-india-2015/

A-Z of my browsing history

Fri, 09 May 2014 16:16:38 +0000

When you start typing in the address bar, Chrome suggests a link to visit, based on frecency. What do my recommendations look like?

A is for airtel.in/smartbyte-s/page.html – the page where you can check your bandwidth usage. I used to check it infrequently until I upgraded to a 125GB connection. Now I check it every few days and feel miserable that I’ve nowhere near used up my quota. This has coerced me to watch many Telugu movies, of which I don’t understand a word.
B is for blog.gramener.com – I blog there on data stories. The last month or so has been fairly active thanks to the elections.
C is for calendar.google.com – which has become primarily a shared calendar. It was always indispensible to manage my time. Now it helps my colleagues pick when to call me. Right now, my calendar has events booked about two months in advance.
D is for docs.google.com – for effectively one single purpose: shared spreadsheets. This is such a common and powerful use case, and I’m surprised it hasn’t become much easier to use.
E is for epaper.timesofindia.com – some of our content has been published by The Economic Times, and I keep doing ego-searches in the print edition. But close behind is eci.nic.in which I’ve been scraping a lot, and election-results.ibnlive.in.com which we created for CNN-IBN.
F is for flipkart.com – not facebook.com. I’m not often on Facebook.
G is for gramener.com. Naturally. (It’s not surprising that it’s not google.com: I search directly from the address bar.)
H is for handsontable.com – a library that I’ve been using a lot recently, followed by html5please.com that tells me which HTML5 features are ready for use.
I is for ibn.gramener.com – another property we created, but it only just beats irctc.co.in.
J is for join.me – a clean way to share your screen without the audience having to install anything (though you the sharer do have to install the software.)
K is for kraken.io – an amazingly efficient image compressor. As you might have guessed, I lead a strange life.
L is for learn.gramener.com – our Intranet. Sorry, you can’t access this one.
M is for mail.google.com. I’ll probably be moving away from gmail as a backend this weekend to Mail-in-a-box, though. Google’s pulling the plug on Google Reader has shaken my faith.
N is for news.ycombinator.com. When I’m bored and want to watch something while I have dinner, I don’t open YouTube. I open Hacker News.
O is for odc.datameet.org – the Open Data Camp. I’m quite into open data.
P is for pay.airtel.com, but if you ignore the number of bills I pay, it would be pandas.pydata.org, the home page of a remarkable data processing library.
Q is for quirksmode.org, PPK’s remarkable browser-compatibility guide
R is for reader.s-anand.net, my self-hosted RSS reader. It used to be reader.google.com, but Google let me down there.
S is for s-anand.net – this blog.
T is for twitter.com. Unlike Facebook, I don’t dislike Twitter so much.
U is for underscorejs.org. Clearly I need to get a life.
V is for visualizing.org. They have a number of interesting data visualisations.
W is for webpagetest.org – it helps measure the speed of web pages.
X is for xem.github.io. I’ve probably visited this page once, but it’s the only one in my recent history that starts with X
Y is for youtube.com. I lied. I spend an order of magnitude more time watching Telugu movies on YouTube than on Hacker News.
Z is for zoemob.com. Again, a page I visited only once, but there’s nothing else in Z at the moment.

Comments

Software I currently use | s-anand.net 9 May 2014 6:24 pm (pingback): […] course, some of my apps apps have moved online, and my earlier post on the A-Z of my browsing history covers that. But there are a few applications that I’ve hosted which I must talk about. […]
chandigarh 13 Oct 2015 7:27 pm: you can delete your web search history through link https://history.google.com/history

The most popular scientific Python modules

Tue, 11 Dec 2012 02:37:52 +0000

I just scraped the scientific packages on pypi. Here are the top 50 by downloads.

Name	Description	Size	Downloads
numpy	NumPy: array processing for numbers, strings, records, and objects.	2000000	133076
scipy	SciPy: Scientific Library for Python	7000000	33990
pygraphviz	Python interface to Graphviz	99000	22828
geopy	Python Geocoding Toolbox	32000	18617
googlemaps	Easy geocoding, reverse geocoding, driving directions, and local search in Python via Google.	69000	15135
Rtree	R-Tree spatial index for Python GIS	495000	14370
nltk	Natural Language Toolkit	1000000	12844
Shapely	Geometric objects, predicates, and operations	93000	12635
pyutilib.component.doc	Documentation for the PyUtilib Component Architecture.	372000	10181
geojson	Encoder/decoder for simple GIS features	12000	9407
GDAL	GDAL: Geospatial Data Abstraction Library	410000	8957
scikits.audiolab	A python module to make noise from numpy arrays	1000000	8856
pupynere	NetCDF file reader and writer.	16000	8809
scikits.statsmodels	Statistical computations and models for use with SciPy	3000000	8761
munkres	munkres algorithm for the Assignment Problem	42000	8409
scikit-learn	A set of python modules for machine learning and data mining	2000000	7735
networkx	Python package for creating and manipulating graphs and networks	1009000	7652
pyephem	Scientific-grade astronomy routines	927000	7644
PyBrain	PyBrain is the swiss army knife for neural networking.	255000	7313
scikits.learn	A set of python modules for machine learning and data mining	1000000	7088
obspy.seisan	SEISAN read support for ObsPy.	3000000	6990
obspy.wav	WAV(audio) read and write support for ObsPy.	241000	6985
obspy.seishub	SeisHub database client for ObsPy.	237000	6941
obspy.sh	Q and ASC (Seismic Handler) read and write support for ObsPy.	285000	6926
crcmod	CRC Generator	128000	6714
obspy.fissures	DHI/Fissures request client for ObsPy.	1000000	6339
stsci.distutils	distutils/packaging-related utilities used by some of STScI's packages	25000	6215
pyopencl	Python wrapper for OpenCL	1000000	6124
Kivy	A software library for rapid development of hardware-accelerated multitouch applications.	11000000	5879
speech	A clean interface to Windows speech recognition and text-to-speech capabilities.	17000	5809
patsy	A Python package for describing statistical models and for building design matrices.	276000	5517
periodictable	Extensible periodic table of the elements	775000	5498
pymorphy	Morphological analyzer (POS tagger + inflection engine) for Russian and English (+perhaps German) languages.	70000	5174
imposm.parser	Fast and easy OpenStreetMap XML/PBF parser.	31000	4940
hcluster	A hierarchical clustering package for Scipy.	442000	4761
obspy.core	ObsPy - a Python framework for seismological observatories.	487000	4608
Pyevolve	A complete python genetic algorithm framework	99000	4509
scikits.ann	Approximate Nearest Neighbor library wrapper for Numpy	82000	4368
obspy.imaging	Plotting routines for ObsPy.	324000	4356
obspy.xseed	Dataless SEED, RESP and XML-SEED read and write support for ObsPy.	2000000	4331
obspy.sac	SAC read and write support for ObsPy.	306000	4319
obspy.arclink	ArcLink/WebDC client for ObsPy.	247000	4164
obspy.iris	IRIS Web service client for ObsPy.	261000	4153
Orange	Machine learning and interactive data mining toolbox.	14000000	4099
obspy.neries	NERIES Web service client for ObsPy.	239000	4066
pandas	Powerful data structures for data analysis, time series,and statistics	2000000	4037
pycuda	Python wrapper for Nvidia CUDA	1000000	4030
GeoAlchemy	Using SQLAlchemy with Spatial Databases	159000	3881
pyfits	Reads FITS images and tables into numpy arrays and manipulates FITS headers	748000	3746
HTSeq	A framework to process and analyze data from high-throughput sequencing (HTS) assays	523000	3720
pyopencv	PyOpenCV - A Python wrapper for OpenCV 2.x using Boost.Python and NumPy	354000	3660
thredds	THREDDS catalog generator.	25000	3622
hachoir-subfile	Find subfile in any binary stream	16000	3540
fluid	Procedures to study geophysical fluids on Python.	210000	3520
pygeocoder	Python interface for Google Geocoding API V3. Can be used to easily geocode, reverse geocode, validate and format addresses.	7000	3514
csc-pysparse	A fast sparse matrix library for Python (Commonsense Computing version)	111000	3455
topex	A very simple library to interpret and load TOPEX/JASON altimetry data	7000	3378
arrayterator	Buffered iterator for big arrays.	7000	3320
python-igraph	High performance graph data structures and algorithms	3000000	3260
csvkit	A library of utilities for working with CSV, the king of tabular file formats.	29000	3236
PyVISA	Python VISA bindings for GPIB, RS232, and USB instruments	237000	3201
Quadtree	Quadtree spatial index for Python GIS	40000	3000
ProxyHTTPServer	ProxyHTTPServer -- from the creator of PyWebRun	3000	2991
mpmath	Python library for arbitrary-precision floating-point arithmetic	1000000	2901
bigfloat	Arbitrary precision correctly-rounded floating point arithmetic, via MPFR.	126000	2879
SimPy	Event discrete, process based simulation for Python.	5000000	2871
Delny	Delaunay triangulation	18000	2790
pymc	Markov Chain Monte Carlo sampling toolkit.	1000000	2727
PyBUFR	Pure Python library to encode and decode BUFR.	10000	2676
collective.geo.bundle	Plone Maps (collective.geo)	11000	2676
dap	DAP (Data Access Protocol) client and server for Python.	125000	2598
rq	RQ is a simple, lightweight, library for creating background jobs, and processing them.	29000	2590
pyinterval	Interval arithmetic in Python	397000	2558
StarCluster	StarCluster is a utility for creating and managing computing clusters hosted on Amazon's Elastic Compute Cloud (EC2).	2000000	2521
fisher	Fast Fisher's Exact Test	43000	2503
mathdom	MathDOM - Content MathML in Python	169000	2482
img2txt	superseded by asciiporn, http://pypi.python.org/pypi/asciiporn	443000	2436
DendroPy	A Python library for phylogenetics and phylogenetic computing: reading, writing, simulation, processing and manipulation of phylogenetic trees (phylogenies) and characters.	6000000	2349
geolocator	geolocator library: locate places and calculate distances between them	26000	2342
MyProxyClient	MyProxy Client	67000	2325
PyUblas	Seamless Numpy-UBlas interoperability	51000	2252
oroboros	Astrology software	1000000	2228
textmining	Python Text Mining Utilities	1000000	2198
scikits.talkbox	Talkbox, a set of python modules for speech/signal processing	147000	2188
asciitable	Extensible ASCII table reader and writer	312000	2160
scikits.samplerate	A python module for high quality audio resampling	368000	2151
tabular	Tabular data container and associated convenience routines in Python	52000	2114
pywcs	Python wrappers to WCSLIB	2000000	2081
DeliciousAPI	Unofficial Python API for retrieving data from Delicious.com	19000	2038
hachoir-regex	Manipulation of regular expressions (regex)	31000	2031
Kamaelia	Kamaelia - Multimedia & Server Development Kit	2000000	2007
seawater	Seawater Libray for Python	2000000	1985
descartes	Use geometric objects as matplotlib paths and patches	3000	1983
vectorformats	geographic data serialization/deserialization library	10000	1949
PyMT	A framework for making accelerated multitouch UI	18000000	1945
times	Times is a small, minimalistic, Python library for dealing with time conversions between universal time and arbitrary timezones.	4000	1929
CocoPy	Python implementation of the famous CoCo/R LL(k) compiler generator.	302000	1913
django-shapes	Upload and export shapefiles using GeoDjango.	9000	1901
sympy	Computer algebra system (CAS) in Python	5000000	1842
pyfasta	fast, memory-efficient, pythonic (and command-line) access to fasta sequence files	14000	1836

Comments

Ron Z 25 Jun 2013 10:19 pm: Another good one is simpleCV: http://www.simplecv.org/
Ravindranath M 8 Apr 2013 7:13 am: Nice list Anand. Very useful. – This is Ravi from Comviva, who attended your introduction.

The next chapter of my life

Fri, 13 Jan 2012 01:01:56 +0000

I’m writing this post on a one-way flight from London back to India. I’ve moved on from Infosys Consulting, and am starting up on my own.

I’ve wanted to do this for a long time. There’s always more freedom in your own company than someone else’s. There’s often more money in it too, if you’re lucky enough. But my upbringing is a bit too conservative to make that bold step. However, given that my father runs his own firm, I figured it was just a question of time for me to do the same.

Two years ago, in Jan 2010, I picked up Rashmi Bansal’s Stay Hungry Stay Foolish at an airport. That book killed the last bit of resistance I had. If the people in that book could succeed, I felt I could too. And if what they did (building small companies, not huge ones) could be called a success, I could be successful too.

After the flight, it was clear in my mind. I would be an entrepreneur. I would create a small company that would probably fold. Then I’d do it again. And again, 10 times, because 1 in 10 companies survive. And finally, I’d be running a small business that’d be called successful by virtue of having survived. A modest, achievable ambition that I had the courage for.

I usually make big decisions without analysis, by just sleeping over them. I slept over it and announced it to my family the next day. I’m not sure they believed me.

Two months later, along with a friend, I built a dynamic digital image resizing product. We had our wives start a company in the UK, and tried selling it to retailers. There clearly was a demand. The problem was, we didn’t know how to sell. After a year and having spent £500 with no sales, it was clear to us that venture #1 had failed. We eventually shut it down.

In the middle of this, my ex- boss from IBM told me that he was looking to start a venture, focusing on mobile, rural BPO and energy management. This later on changed to data analytics and visualisation. They all sounded like fun, so I said I’ll help out in my spare time.

A few months later, a classmate told me he’d started a business digitising school report cards. That sounded like fun too, so I said I’d help out in my spare time.

Now, if that sounds like I had a lot of spare time on my hands – you’re right, I did. And it’s time to talk about the jobs in my life. My first 3 years at IBM were fun. I was coding, learning, and leading a bachelor’s life with friends, money, and no responsibilities. My 4 years at BCG were strenuous with 80-hour weeks, but it was interesting and challenging. I was newly married, and between work and home responsibilities, I had no time for fun.

I moved to Infosys Consulting in the UK with the specific aim of rectifying that (and for health reasons as well). In the last 7 years, the work has (except on occasion) been a bit boring, but very relaxing. On most days, I would spend 4 hours working, and 4 hours learning new stuff. The things I learnt only helped me be more efficient. So I ended up getting even more work done in less time.

Many things came out of this. Firstly, I recovered my health. We had a daughter, and I spent more time with her. I started coding in earnest again. By 2007, I was writing code as part of my projects – stuff that others whose job it was were unable to. By 2009, I had a few websites running, like an Indian music search engine, an IMDb Top 250 tracker, a few transliterators, and so on.

So when I said I’d help out with these startups, it wasn’t an empty promise. For the last 18 months, I’ve had a day job and three night jobs. I never did justice to any of them in my opinion, but I had more fun than ever in my life, I learnt more than ever in my life, and I produced more tangible output than ever in my life. Sometimes, quantity beats quality or reliability.

Both these startups are doing well today. Gramener.com offers data visualisation and IT services. I will be joining them as Chief Data Scientist. Reportbee.com offers a hosted report card solution. I will continue helping them out. And I will continue working with a few NGOs.

You’ll see me a lot more active online now. I can publicly write about my work – something I’ve been unable to do the last 11 years.

I am relocating to Bangalore. From a professional front, it’s an obvious choice. That’s where the geeks are. In my last visit to India, I was at Bangalore, Chennai and Hyderabad. In the latter two, it’s tough to meet geeks. And when you do, it’s no easier to find the next. Bangalore has many more geeks, and they’re fairly well networked.

From a personal front, too, Bangalore works well. It’s close enough to Chennai without actually being in Chennai.

It’s 10am on Thu 12th Jan. Our flight is descending into Delhi airport. It’s the start of a new chapter in my life. Scary, but exciting. Wish me luck!

Comments

Thejesh GN 13 Jan 2012 9:19 am: Welcome back.
AnanthRK 13 Jan 2012 8:51 am: Great to hear that you are taking the plunge full-time. All the very best!!
Madhur Singhal 13 Jan 2012 12:47 pm: Anand, best wishes!! Well - I never had a doubt that you take only 4 hrs for 8 hrs of work anyway. I have been guilty of being lazy in calling but thankfully, I travel to Bangalore sometimes. I am at 9873901163 and if you tell me your #, I will call you. cheers!!
Ashish 13 Jan 2012 12:31 pm: Hey Anand! Good to hear that, best of luck for your ventures……happy Independance!! Do let me know if i can be of any help Ashish
Raghu 13 Jan 2012 12:42 pm: Enjoy always and lead from the front! All the best and do well!
Kalpesh 13 Jan 2012 10:22 am: Anand, I am sure, you have all things needed to do better things in life. Hopefully, we will meet & work together in future. Best Wishes Kalpesh
Vanaja 13 Jan 2012 11:28 am: All the best Anand! I have always wondered why you were stuck with Infosys when you were so much more. This looks like the way to go. Good Luck and God Bless!
Pawan Bhargava 13 Jan 2012 9:55 am: Hi SAnand, good call. It does not matter how it works out (trading in handsight), but all that matters is enjoying what you do. I am sure it would be immensely satisfying. Good luck.
Ravish 13 Jan 2012 10:00 am: People said “search” was too late for Google.. they also said “social networking” was too late for Facebook.. good that neither Larry Page nor Mark Zuckerberg listened to them.. All the best Anand.. Good Luck !! Its never too late for new ideas!!
Gaurav 13 Jan 2012 1:29 pm: Way to go Anand! You will love being an entrepreneur - the freedom to try different things and the satisfaction you get when something works…
408wij 13 Jan 2012 2:17 pm: Good luck. –Joe
sriram 13 Jan 2012 2:49 pm: All the best Thala.. :)
Somnath 13 Jan 2012 3:35 pm: Just goes to show the fan following you have. I am really late in checking your latest post. Welcome back to Bangalore and India. I see couple of batchmates have posted to. Wish I can join small organization / startup but am hesitant to take the risk. Then again what is my true skill set :D Somnath
Harish Kashyap 13 Jan 2012 4:59 pm: Welcome to place where you belong! I am sure you will do well. You can reach me at 99010 32469. It will be good to have a cup of coffee with you!
Suresh 13 Jan 2012 5:27 pm: Wish you success. “Real success comes not from being invited onto the yacht, but from being able to paddle one’s own canoe” - Hugh McLeod
Sridhar 13 Jan 2012 5:34 pm: All the very best Anand! I am sure you’ll have no problem adding tremendous value to whatever you produce and that will ensure your success.
praphul 13 Jan 2012 6:54 pm: Congratulations!
San 13 Jan 2012 9:12 pm: You’ve always been an inspiration and it seems like you always will. All the best!
Sathish 13 Jan 2012 10:13 pm: Didn’t actually get the technical part you are talking about… but looks like your are excited and I’m happy for you… ALL THE VERY BEST!!
Madhu 14 Jan 2012 2:53 am: Good luck, many dream of doing it and few take the bold step. I am glad you did. Congratulations and wish you the best.
Vasant 13 Jan 2012 4:25 am: Awesome! Welcome back. And best of luck ahead.
Girish 13 Jan 2012 3:07 am: Congratulations for new change Anand :-) S-Anand.net is always at the back of my mind for the content it has.
satish 13 Jan 2012 7:09 am: Wow. Good to know sir. Wish you all the very best
Krishna 13 Jan 2012 6:43 am: Nice to know this Anand. I’ve been following your exciting works in our intranet blogs. For sure, this is going to be the best decision you ever made :) Wish you loads of luck!
Cipher 13 Jan 2012 6:55 am: Congrats and the best for your future Anand.
Veera 13 Jan 2012 4:53 am: Whoa! that explains the number the data visualization posts in your blog. :) Congratulations for starting up on your own! Me too in the same situation, going to work on my own start up from coming March onwards :)
harish 13 Jan 2012 3:44 am: Hi Anand….. wish you the very best and am sure you will have no problems succeeding…
Venkatramanan 13 Jan 2012 3:51 am: Anand Surprising to see you very late to entrepreneurship! I thought you were managing your business! All the very best! As they say in school ‘you will definitely come with flying colours!’ Regards Venkat
chaitanya 13 Jan 2012 1:35 am: All The Best, Anand. It was always knowledgable and pleasure to follow you. Good luck.
saurabh nitin 13 Jan 2012 4:33 am: best of luck brother! cheers
S_Thomas 13 Jan 2012 3:20 am: Congratulations Anand, we look forward to hearing more
Manish 13 Jan 2012 4:36 am: Anand, congrats I was also struggling with the same thoughts to take up a job or start on own
Ayyappa 13 Jan 2012 4:40 am: Best of luck Anand! Btw -I am a daily user of your mixamail for twitter and never said thanks! Thanks a lot for that! Also, I hope you write your experiences in the startup life for “other conservatives who are afraid to jump” to learn how it feels to be in the wild outside.
sadhasivam 24 Jan 2012 9:33 pm: Good Luck Anand. Being associated with you as read for a decades i am thrilled to go through post. We wish a great journey ahead.
vivek 22 Jan 2012 5:49 pm: Best of luck Anand.
Sankar 23 Jan 2012 2:05 am: All the best Anand!
MohaN 23 Jan 2012 3:26 pm: As someone mentioned, I stumbled across on a pursuit…Life an Unwrapped GIFT……Tear it or Peal it or Break it…something underneath it…
Karthick 13 Jan 2012 8:41 am: All the best.
Naveen 14 Jan 2012 5:37 am: Anand- Congratulations , the day I had been waiting for long time. You will make a big impact where ever you are. Good Luck to you and Gramener
Rajesh bhura 14 Jan 2012 2:10 pm: Finally! This is teriffic news. From our campus days I always thought that this was the way to go for u. Remember the digital signature idea which u had come up with as our e-commerce project? It’s a big industry now in 10 yrs. I don’t have any doubt whatever u do will be successful in it’s own way. All the best. Your fan for always - rajesh
Ashwin 14 Jan 2012 5:42 pm: Anand - I have been following your blog for several years now. I wish you all the best !!
Nirav Kanakia 3 Feb 2012 6:09 pm: Good luck
Shobha 9 Feb 2012 9:27 am: Hi Anand, You do not know me. But I had read your posts in 2007 and I was inspired by your diverse interests in almost every area. Suddenly I remembered you today and read your recent post and gotta know you are starting your own firm in India. All the best and kudos to you. -Shobha-
Karthik A 12 Feb 2012 3:30 am: All the best - have thought several times of making the leap but never got out of the comfort zone - Very happy you did. Maybe probability of this visualization thing becoming the next infy is 100% All the best
Ravindra 3 Feb 2012 12:17 pm: Thaliva…All the best! Looking forward the IPO in Year 2015 or earlier! Cheers
Ramana 16 Jan 2012 3:13 am: All the best Anand!
Anand S 16 Jan 2012 1:08 am: Anand Good luck. Our paths are similar (I am anand @ anands.net). After 12 years in silicon valley, I returned to Bangalore an year ago to run my own venture (started this 4 years ago). Let’s meet in Bangalore.
Venkataragavan 25 Jan 2012 5:25 pm: Hey Anand, wish you good luck and great success ahead. I got interested into data visualization after reading your blogs and now I am actively learning it. Hope to move on someday. Cheers
ravi atluri 13 Jan 2012 2:19 am: this is great. Congratulations. Data in India is going to get more exciting :D
Ravi 13 Jan 2012 6:01 am: All the best Anand. I’ve loved your site and all the tools you’ve built around it. I’ve always wondered how you got so much time to do all that you do. Gramener seems to be a happening place now and I am sure you will make it a grand success.
Amitesh Jasrotia 13 Jan 2012 1:24 am: Hey Anand, I was not really inspired by Rashmi Bansal’s book but I am mighty inspired by the piece you have written. Nothing else is more self-liberating than becoming the master of your own destiny. I have been following your blogs/site for quite some time now. Something tells me you will be successful. Still, Best of luck for your future endeavors!
Nithin 16 Jan 2012 11:50 am: All the Best Anand…
Anuradha Shastry 16 Jan 2012 3:46 am: Lots of good luck!! Landed on this blog just an hour ago. best, another Infoscion
Sumit Dhar 16 Jan 2012 6:32 am: Now that you have moved to Bangalore, we should exchange numbers and plan on catching up. Where, in Bangalore, are you based out of ? Cheers, D.
D 2 Mar 2012 12:10 pm: Anand, Wish you tremendous amount of luck.
Nikhil 31 Jan 2012 9:31 am: Might be of interest to you http://kurt.karmalab.org/2012/01/30/building-a-data-team/
Sreenivasan L 1 Feb 2012 11:11 am: All the best Anand. Wishing you a great success ahead in your new journey.
Judy H 23 Feb 2012 7:22 am: I have no idea how I found your website today for the first time, but I’ve enjoyed reading it. I wish you all the best in your endeavors!
Sri 19 Jan 2012 2:50 pm: Hi Anand I have followed your blog several years now. It’s great to hear about the new venture. Very inspirational post. Good luck and all the best. Vaalga Valamudan Sri
Sajiv 18 Jan 2012 4:20 pm: All the best Anand.
pankaj 18 Jan 2012 6:39 pm: Good luck with your new venture. Keep us posted. :)
Ranganathan 18 Jan 2012 7:53 pm: Hey Anand, I used to follow ur blogs for the past 2 years. I think i came to know about ur blog when i was searching about the tips in excel . Sometimes if I am bored i used to check ur site for the movie list..:-) .. I just want to say ‘All the very best ’ for ur new journey.. I am sure definitely u will succeed…..this will be an inspiration for me some day for starting a new company….
S Anand 16 Jan 2012 1:26 pm: @Sumit: definitely. Still haven’t move into Bangalore just yet, but my contact number is on the home page. Please feel free to call any time,
Shankar V 16 Jan 2012 2:04 pm: hey Anand All the very best and trust me…..I knew this was coming! :) Welcome back to India and welcome to Bangalore. Hope you continue to post interesting stuff. Please do stay in touch. I had your Infosys email id. Don’t think I have your personal id. Please pass it on.
ranjit bawa 16 Jan 2012 4:09 pm: hey Anand - great to hear!! good luck with the new chapter and look forward to seeing you soon. hope to follow suit, but till then, shall live vicariously…
S Anand 18 Jan 2012 1:46 am: @Shankar, I keep my latest phone number at the top of my home page. Will be good to hear from you!
Rajesh B 18 Jan 2012 7:00 am: Great to know you are starting off on your own. Good luck. I took my one way ticket to India 3 years ago. Still burning through my first venture. Inspiring post. Am planning to start 9 other on the side just to be on the correct side of the odds. Jokes apart, can think of few others more equipped to start off on something. If ever you are interested in doing something on the education space, drop me an email. Will try to make that as my night job. Am good with creating content, abysmal poor in trying to sell it. So, ideal for a night-job. Best wishes. Cheers, Bee
Ravi Tiruvaipati 18 Jan 2012 1:13 pm: Hi Anand, I have been following your posts and your applications…It is good hear that you are planning to start your own. I wish you all the best!!
Sriram S 15 Jan 2012 6:14 am: Welcome back and all the best…I stay in BTM Layout in Bangalore..lets catch up sometime
Sudheer 15 Jan 2012 9:35 am: All the best Anand! Looking forward to hear more and more about you in the coming days from sources other than the blog
Jitendra 15 Jan 2012 3:25 pm: Wish you luck Anand! I am sure with your attitude and energy you’d do well in whatever you try to do!
Raja 24 Aug 2012 3:25 am: I try to visit you website since 2008 whenever I get a chance and look for what geeks are doing lately :-) and just noticed today that you moved to India starting out your own venture. Hard work pays off for sure. I am sure you will succeed. Good Luck!
Michael 22 Oct 2012 7:16 pm: Good luck with this Anand. hope it is going well.. Michael
mohan 11 Nov 2012 7:50 am: all the best mr anand keep going your website is so informative with high entertainment value , you will suceed god bless you mohan. you will definitely suceed in
Amutha 25 Nov 2013 9:37 pm: It’s very late if I say gud luck,:) and I hope you are going great with your plans.
Ankur 15 Apr 2013 8:22 am: Hi Anand, I first heard of you courtesy Rahul Guha during my first year at BCG in Mumbai in 2006. He mentioned you as a Consultant he had a very high opinion of. I came across you next a couple of years later, as I was trying to index by IMDB 250 list - I used the app you’d created and recommended it to many friends too. Today, I reached your blog while browsing through some data on poverty in India (datastories.in). Just wanted to let you know. I wish you all the best in your venture(s). Ankur
Father of the bride - S Anand 1 Nov 2023 8:25 pm (pingback): […] In 2012, I started Gramener with half a dozen friends. […]
10 years later - S Anand 17 Mar 2022 1:24 pm (pingback): […] 12 Jan 2012, on a flight back from London, I […]
Moving to Singapore - S Anand 2 Aug 2022 6:37 pm (pingback): […] “I lost my job. Please don’t tell anyone. Let’s go back to India.” We left 2 weeks later to join […]
A challenge of blog questions - S Anand 4 Mar 2025 4:44 pm (pingback): […] The next chapter of my life, which I wrote on a one-way flight back from the UK to India to start Gramener. […]

Bayes’ Theorem

Mon, 11 Oct 2010 14:53:01 +0000

I’ve tried understanding Bayes’ Theorem several times. I’ve always managed to get confused. Specifically, I’ve always wondered why it’s better than simply using the average estimate from the past. So here’s a little attempt to jog my memory the next time I forget.

Q: A coin shows 5 heads when tossed 10 times. What’s the probability of a heads?
A: It’s not 0.5. That’s the most likely estimate. The probability distribution is actually:

That’s because you don’t really know the probability with which the coin will throw a heads. It could be any number p. So lets say we have a probability distribution for it, f(p).

Initially, you don’t know what this probability distribution is. So assume they’re all the same – a flat function: f(p) = 1

Now, given this, let’s say a heads falls on the next toss. What’s the revised probability distribution? It’s:

f(p) ← f(p) * probability(heads | x) / probability(heads) = 1 * (x^1 * (1-x)^0) / 1 = x

Let’s say the next is again a heads. Now it’s

f(p) ← f(p) * probability(heads | x) / probability(heads) = x * (x^1 * (1-x)^0) / 1 = x^2

Now if it’s a tails, it becomes:

f(p) ← f(p) * prob(tails | x) / prob(tails) = x^2 * (x^0 * (1-x)^1) / 1 = x^2 * (1-x)

… and so on. (This happens to be a called a Beta distribution.)

Now, instead of this being the probability of heads, it could be the probability of a person having blood pressure, or a document being spam. As you get more data, the probability distribution of the probability keeps getting revised.

Comments

Ram 18 Jan 2012 1:59 pm: Hi Anand, f(p) ← f(p) * probability(heads | x) / probability(heads) = 1 * (x^1 * (1-x)^0) / 1 = x What I understand from the above notation is f(p) stands for probability distribution function (which is used recursively), x stands for the probability of getting a head (which is unknown) and 1-x stands for the complement of x. Can you please explain why the distribution function is defined as you have done? i.e. , f(p) ← f(p) * probability(heads | x) / probability(heads)

Managing the data deluge

Sat, 10 Feb 2007 12:00:00 +0000

Peter Norvig’s brilliant talk on Managing the Data Deluge. Among other things, he talks about how having lots of data is sometimes better than having a carefully designed algorithm.

Math will rock your world

Sat, 14 Jan 2006 12:00:00 +0000

It’s a good time to be a mathematician.