data-analysis on S Anand

My most memorable anniversary

Wed, 03 Jun 2026 09:27:14 +0800

At 9:30 pm, I checked my calendar for tomorrow’s appointments, alt-tabbed frantically into ChatGPT, and started typing:

Tomorrow is my 24th anniversary. It’s a bit late for me to buy anything (except maybe an online service) or prepare something.

This has become a habit – leaving things to the last minute and asking ChatGPT to save my day.

I did give it good context, though.

You remember the OCBC expenses treemap you created by analyzing my transactions? That will give you a good guessable idea of the kinds of things she spends on and hopefully, therefore, what she likes.

I was analyzing my bank transactions over the weekend, so that came in handy.

What can I do (either create for her, like I made a peanut jelly nutella sandwich birthday for Dhyeya or buy - if it’s an online service / gift voucher kind of thing) in the next hour or so?

ChatGPT began with gyaan - which I promptly ignored:

Your best move is not “buy a gift”. It is “prove that I have noticed your life, then attach a redeemable escape hatch…

… and moved on to analyzing her personality and recommending from different perspectives:

Practical person

No-decision day: “Tomorrow, you decide nothing. I handle transport, food, booking, payment, and exit plan.”

Service relief voucher: 24 hours where she forwards any household/admin nuisance to you.

Personal-care float: ClassPass/spa/salon voucher plus the promise: “I will book the slot and taxi, not just give you a coupon.”

A romance playwright

24 tiny vows: not grand vows; operational vows: “I will stop asking where the charger is.”

The ordinary-places pilgrimage: one route across 3-5 mundane but meaningful places.

Private audio guide: 8 short voice notes she unlocks at home / taxi / restaurant / temple.

… and so on.

For some of these, I’m absolutely the worst person to execute. ClassPass? Spa? Salon? I don’t even know the difference between them.

But 1.2 and 2.1 were interesting. So, after a few iterations with ChatGPT, I woke up the next morning with:

Today’s our 24th anniversary. For the next 24 hours, I’m going to knock off 24 things from your TODO list.

I rescheduled my meetings. I spent the entire day doing nothing but what was on her list. Water purifier, doctor appointments, toothbrush shopping, investments, drying clothes, duct-taping, … and we dined out.

At 10:00 pm: “I think it was my most memorable anniversary. Thank you.”

Thank you, ChatGPT. Thank you, habit of setting low expectations.

Can AI Replace Human Paper Reviewers?

Tue, 13 Jan 2026 09:55:45 +0530

Stanford ran a conference called Agents for Science. It’s a conference for AI-authored papers, peer reviewed by AI.

They ran three different AI systems on every paper submitted, alongside some human reviewers. The details of each of the 315 papers and review are available on OpenReview.

I asked Codex to scrape the data, ChatGPT to analyze it, and Claude to render it as slides.

The results are interesting! I think they’re also a reasonably good summary of the current state of using AI for peer review.

The three AI reviewers wildly disagree with each other.

Imagine hiring three movie critics to rate the same film. One gives it 2 stars, another gives it 6 stars, and the third gives it 4 stars. Same movie, completely different conclusions. That’s what’s happening with these AI reviewers-on almost half of all papers.
“Averaging” the three AIs doesn’t actually help

You might think: “Just average the three scores! That’ll balance out their biases.” But here’s the problem: the generous AI (AIRev2) uses much bigger numbers. When you average, its voice drowns out the others. It’s like having three judges, but one shouts and two whisper.
Every AI claims to be 100% confident - even when they’re wrong

Reviewers are asked “How confident are you in your assessment?” on a 1-5 scale. Every single AI review said “5 out of 5-totally confident.” All 751 of them. Even when two AIs looked at the same paper and reached opposite conclusions, both claimed maximum confidence.
AI and human reviewers see different things

On papers that got both AI and human reviews, we compared their scores. The AIs were almost always more generous than humans-by about 1 full point on average. And in some cases, AI said “excellent!” while the human said “this is broken.”
AI reviewers can catch obvious problems

AI reviewers successfully flagged papers with impossible claims-like citing AI models that don’t exist yet, or referencing datasets from the future. These are “fact check” problems that don’t require deep expertise, just attention to detail.
Use AI disagreement as a signal, not noise

When the “generous AI” loves a paper but the “skeptical AI” hates it, that’s not random noise-it’s useful information. It means the paper’s fate depends on standards (rigor vs. novelty), not just quality. These are exactly the papers humans should look at.

Tue, 09 Dec 2025 05:36:05 +0000

In my Mining Digital Exhaust workshop on Saturday,

One discovered that they cycle when life is unstable, not for fitness.
Another found that their buys are good but sells are bad trades.
I learnt that I watch YouTube most at office (12-4 pm), not at home.

How? A fairly straight-forward process:

Export your personal data. (Use Chrome Devtools Protocol to scrape.)
Upload to ChatGPT, Gemini, Claude, … and have them analyze with code.
Have them narrate in the style of your favorite author.

Models are super smart, but everyone has equal access to them. Your personal data is unique. Combine them to get something powerful.

Workshop summary: https://sanand0.github.io/talks/2025-12-06-mining-digital-exhaust/

Organized by Hasgeek at the Sahaj Software office in Bangalore.

Mon, 10 Nov 2025 00:00:00 +0000

I always wondered why old movies are rated so high on IMDb.

For example, 12 Angry Men (1954) with just ~900K votes ranks about as high as Inception (2010) with ~2M votes. Few people I know have seen 12 Angry Men. So where does this high rating come from?

My theories were:

Old movies really are that good.
IMDb’s algorithm is biased towards old movies.
People remember older movies fondly.

Actually, it’s none of these. It’s selection bias.

Few people watch a 1950s black & white drama: cinephiles, film students, etc. They love it and give it 9s and 10s.

Everyone watched Inception. The casual majority thinks it’s fine, not life changing. Maybe a 7 or 8.

This creates a paradox: obscurity protects ratings while popularity is its own punishment. Only “devotees” watch obscure movies - leading to better ratings than widely seen movies.

PS: This data analysis and story were authored by Claude Code. That includes the statistical significance validation.

Story: https://sanand0.github.io/datastories/imdb-democracy-penalty/index.html Prompt: http://github.com/sanand0/datastories/tree/main/imdb-democracy-penalty Code: https://github.com/sanand0/imdbscrape/pull/1

Sat, 17 May 2025 07:11:21 +0000

This talk is an experiment. I am going to talk (literally) to ChatGPT on stage and have it do every kind of data analysis and visual storytelling I have ever done.

Bangalore. 27 June.

Of course, this is an LLM era away. So no promises. We might be doing something completely different on stage.

Mon, 10 Mar 2025 03:35:42 +0000

I’m at an open Hyderabad meet-up, Thu 20 Mar 4 pm. “Analyzing data with AI agents”."

It’s a public event by Hasgeek. Venue: Castlight Health, Sattva Knowledge Park.

We know LLMs suck at number crunching but are good with code. I’ll share what we’ve learnt by getting it to write code to analyze data instead.

Less lecturing, more interactive Q&A and demos in a cozy group. Mostly for analysts, data scientists, and programmers. Not so much for LLM researchers or managers.

Old songs in my music library

Mon, 06 Jun 2022 16:02:05 +0000

My music library has around 1,000 songs (mostly Tamil and Hindi, with some Telugu and English film songs).

I spent this morning tagging them by year with mp3tag. (Manually. You don’t automate the pleasures of life.)

I thought my 1990s collection would be the largest. I was in college, listening to lots of music then. But surprisingly, my collection has grown post the 1990s.

I have 3 guesses why.

Recency bias. I re-built this collection recently. Maybe I forgot older songs?
Digitization bias. Maybe I listened to more songs as the cost of transmission/storage fell?
Worsening standards. Maybe I used to be choosier about music?

Though I’m not sure of the above, there’s another interesting anomaly.

There is a spike in the 1960s.

I don’t need to guess this one. I know why. Those are the songs my parents liked. I grew up hearing them.

The oldest song Tamil song is from Thiruneelakantar (1939). It’s from my father’s collection. I’ve heard it often enough to still enjoy it.

The oldest Hindi song is from Jaal (1952). He has a fondness for Dev Anand’s songs. So do I. This one is a beauty.

The oldest Tamil song my mother introduced me to is from Parasakthi (1952). She used to dance to this song when young.

The earliest Hindi song she introduced me to was from Jhanak Jhanak Payal Baaje (1955). It’s the song I grew up on, and it’s still among my favorites. What a melody!

My wife prefers newer songs. But I have low standards and few preferences. It makes my life rather happy.

So, in celebration of Make Music Day on 21 June, I’m treating myself to 2 weeks of my collection from the 1960s!

PS: My full collection is at https://gist.github.com/sanand0/877637165b17239aa27beac03749c9a6

How to find a Chinese actor to cast in Hollywood

Sun, 20 Feb 2022 07:08:21 +0000

Film actors mostly act within their own industry.

For example, Hollywood actors act outside Hollywood just 10% of the time. Chinese actors act with non-Chinese actors just 1% of the time.

So, if you’re a Hollywood producer trying to cast a Chinese actor, how would you find them?

One way is to list Chinese actors with the largest number of Hollywood co-stars. Let’s see who tops that list.

#5. Pei-Pei Cheng

You may know her as Jade Fox, the sly governess in Ang Lee’s Crouching Tiger, Hidden Dragon (2000), or Golden Swallow, the skilled swordsman sister in Come Drink With Me (1966), or even as the voice of the matchmaker who disgraces Mulan in Mulan (2020).

She mainly acts in Chinese films, co-starring nearly 180 times with actors like Hua Yueh, Lieh Lo, and Chung-Hsin Huang. But she’s also co-starred over 20 times with Hollywood actors like Jamie King (of Sin City), Peter Bowles (of The Bank Job), and Sandra Oh (of Grey’s Anatomy).

#4. Jet Li

You may know him as Han Sing, the martial artist and ex-cop in Romeo Must Die (2000), or Gabe Law, the former MultiVerse Authority agent in The One (2001), or Yin Yang, the unarmed member of The Expendables (2010).

He has co-starred over 100 times with Chinese actors like Jackie Chan, Simon Yam, and Sammo Kam-Bo Hung. But he’s also co-starred 30 times with Hollywood actors like Antonio Banderas, Morgan Freeman, and Sylvester Stallone.

#3. Joan Chen

She’s famous as Wanrong, the Chinese empress in The Last Emperor (1987), Josie Packard, the owner of the Twin Peaks mill in Twin Peaks (1989), or Dr Ilsa Hayden, assistant to the villain Rico Dredd in Judge Dredd (1995).

She’s co-starred over 80 times with Chinese actors like Tony Chiu-Wai Leung, Leon Lai, and Tony Ka Fai Leung. But she’s co-starred over 40 times with Hollywood actors like Michael Caine, Peter O’Toole, and Christopher Walken.

#2. Jackie Chan

The most famous Chinese martial arts actor in the world, and one of the highest-paid actors in the world, is famous as Detective Inspector Lee in Rush Hour (1998), Mr Han in The Karate Kid (2010), and the voice of Monkey in Kung Fu Panda (2008).

He has co-starred nearly 200 times with Chinese actors like Sammo Kam-Bo Hung, Maggie Cheung, and Kent Cheng. But he’s co-starred over 50 times with Hollywood actors like Arnold Schwarzenegger, Owen Wilson, and Chris Tucker.

#1. Michelle Yeoh

You may know her as Wai Lin, the Chinese spy and James Bond’s ally in Tomorrow Never Dies (1997), Yu Shu Lien, the warrior swordswoman in Crouching Tiger, Hidden Dragon (2000), or as Eleanor Young, the domineering mother-in-law in Crazy Rich Asians (2018).

She’s an actress at the borderline of the Chinese - Hollywood clusters. She’s acted ~60 times with Chinese actors like Maggie Cheung, Chow Yun-Fat and Jet Li. But she’s acted almost as many times with Hollywood actors like Sigourney Weaver, Zoe Saldana and Sam Worthington.

More actors

Here are half a dozen more Chinese actors that have acted with Hollywood actors often.

It’s interesting to see that 3 of the top 6 (Chow Yun-Fat, Pei-Pei Cheng, and Michelle Yeoh) had all acted in the blockbuster Crouching Tiger, Hidden Dragon (2000).

So, perhaps the simple message to our Hollywood producer is to “look no further than the cast of the first foreign-language film to break the $100mn mark in the USA.”

How isolated is Bollywood from world cinema?

Wed, 05 Jan 2022 10:05:45 +0000

These are the major group actors based on who they act with most.

Language. Not country. For example, the Spanish / Mexican group is across countries. But Indian actors divide into North Indian and South Indian. It’s language, not country.
Time period. Old American actors are a separate group from Hollywood. (Naturally. Brad Pitt was born after Humphrey Bogart died. They couldn’t have acted together.)
Genre. Hollywood Porn actors don’t act with mainstream Hollywood. Same with Japanese Porn, Hollywood TV, and Hollywood Horror actors.

How are these groups themselves connected? Do Chinese actors act with Hollywood often? How isolated is Bollywood from world cinema?

Hollywood is the core group

Take groups that act with other groups at least 5% of the time. Mainstream Hollywood acts with British and Hollywood TV/Horror actors. All other clusters are isolated.

Indian & Japanese clusters emerge

Let’s go more liberal. Take groups that act with other groups at least 2% of the time. Hollywood forms a big connected cluster. It includes most of Europe – British, German, French, Czech, Yugoslavian & Italian actors.

North & South Indian actors form the first non-Hollywood cross-language cluster.

The Japanese and Japanese porn actors form a cluster too. (Interestingly, it’s easy for a Japanese porn actor to act with mainstream Japanese actors. Hollywood porn actors find it far harder to act with Hollywood.)

Chinese & Korean cluster emerges

Chinese & South Korean actors form the first cross-country cross-language cluster.

Hollywood expands to act with Scandinavian, Spanish, Polish, Brazilian & Nigerian films.

Other film industries (Russian, Greek, Egyptian – even Hollywood Porn – are still isolated.)

World Cinema vs the rest

Among groups that act with other groups at least 0.5% of the time, we have:

Turkish & Iranian groups coming together
Indonesian actors acting with the Chinese
Hollywood expanding to cover Russian, Greek, Egyptian, and finally, Hollywood Porn. (It’s easier for Brazilian / Nigerian to act with Hollywood than to be a Hollywood Porn actor.)

At this point, there are 6 actor groups that act with each other at least 1 out of 200 times (0.5%).

World Cinema (Hollywood & friends)
Japanese (mainstream & porn)
Indian (North & South)
Chinese, South Korean & Indonesian
Turkish & Iranian
Filipino

One world of cinema

If we look at groups that act with other groups at least 0.5% of the time, we have a far more unified picture. Almost every actor group acts with another group at least 1 out of 400 times.

But even here, there’s an exception. Filipino actors – the most insular major actor group in the world.

So, how isolated is Bollywood from World Cinema? For its size, it’s one of the most isolated actor groups. (But not as much as Iranian/Turkish or Filipino.)

Comments

Dr Urmila 5 Jan 2022 7:36 pm: Hi Anand….i am not a technical person but I do use similar representations in my research as bibliometric, scientometrics etc…we use to understand the publication patterns… Any very interesting piece of information. I enjoyed reading
How to find a Chinese actor to cast in Hollywood - S Anand 20 Feb 2022 12:39 pm (pingback): […] For example, Hollywood actors act outside Hollywood just 10% of the time. Chinese actors act with non-Chinese actors just 1% of the time. […]

Tue, 21 Dec 2021 13:39:09 +0000

Can foreigners break into #Hollywood?

A break into films is hard. Particularly when you’re a foreigner. But is Hollywood more open or less open than other countries?

If we go by iconic actors, the numbers are not encouraging. Just 2% of Will Smith’s co-stars are non-Hollywood. But over 30% of Jackie Chan’s co-stars are non-Chinese.

But #clustering shows that Hollywood is actually among the most open to foreigners. #dataanalysis

Can foreigners enter Hollywood?

Tue, 21 Dec 2021 13:25:46 +0000

An aspiring Malaysian actor posted on Reddit:

I am a 18-year old biracial Malaysian kid who wants to be an actor in Hollywood. I’m taking a diploma for performing arts in a college called Sunway University in 8 days and I’m considering pulling out of it because why do something that I like when my dreams might never be fulfilled and the price for taking this diploma is seriously expensive. I am starting to doubt my chances of making it to Hollywood and I suffer from extreme anxiety. Is it possible for someone like me to enter Hollywood? What are my chances?

Breaking into Hollywood is hard. As a foreigner, it would be even harder. So I asked myself:

Do Hollywood actors act with foreigners?

Let’s take Will Smith. He frequently acts with Martin Lawrence, Tommy Lee Jones, Jaden Smith, Jon Voight, and 84 other actors.

His every co-star is a Hollywood actor, except the Spanish actor Jordi Mollà in Bad Boys II, and the Dutch actor Marwan Kenzari in Aladdin. Will Smith acts with just 2% of foreign co-stars.

On the other hand, Jackie Chan is more cosmopolitan. He acts with:

Chinese actors like Yuen Siu-Tin in Drunken Master
Hollywood actors like Chris Tucker in Rush Hour
Japanese actors like Kumiko Goto in City Hunter – which is based on a Japanese Manga of the same name.
South Korean actors like Su-cheon Bae in Huo shao shao lin men – a Korean/Mandarin film
Indian actors like Disha Patani in Kung Fu Yoga - an Indo-Chinese film
Spanish actors like Eva Cobo in Operation Condor - shot in Spain
Danish actors like Pilou Asbæk in the upcoming film Snafu

Of his 224 co-stars, 70 are non-Chinese. Jackie Chan acts with over 30% foreign co-stars.

Are Chinese films be more foreigner-friendly? Should our Malaysian friend try there instead?

Is Hollywood less open to foreigners than other countries?

I took all movie actors across the world and broke them into groups using a community structure. Actors within the group act mostly within themselves, and less with other groups.

The largest group is Hollywood, with ~80,000 actors (mostly American). They act with each other 90% of the time and act with other groups only 10% of the time.

In comparison, the Chinese group has ~20,000 actors. They act with each other 98% of the time. When they do act outside the group, it’s mostly with Hollywood (0.5%), Japanese (0.3%), South Korean (0.3%), and Indonesian (0.1%)

Clearly, Jackie Chan is more the exception than the norm.

But among the large groups, there are 2 groups that are even more insular than Chinese actors.

The ~8,200 Turkish actors act only with each other 99.1% of the time, occasionally venturing to act with Iranian actors (0.2%).

Even more insular are the ~7,000 Filipino actors who act with each other 99.3% of the time. They occasionally venture out to act in Hollywood 0.2% of the time.

There are no other sizeable groups of actors that’re as insulated.

Hollywood is actually among the most cosmopolitan groups, along with the West European films. So, to our budding Malaysian actor, I’d say:

It’s hard to get an acting break. As a foreigner, it’s 10 times harder in Hollywood. But you’re better off in Hollwood or Western Europe than in any other country, where it would be 50 to 100 times as hard!

Comments

How isolated is Bollywood from world cinema? - S Anand 5 Jan 2022 3:35 pm (pingback): […] These are the major group actors based on who they act with most. […]
I tested the best ways to mail people - S Anand 6 Jan 2022 11:32 am (pingback): […] write more about did 100% worse at #4 than #2 (4.4% down to 0%) […]
Vidyasekar 21 Dec 2021 7:15 pm: I don’t know whether it has a business value or not. But it gives a hammer shot on the head and looks at the Cinema industry and its behaviors.
My Year in 2021 - S Anand 2 Jan 2022 1:36 pm (pingback): […] On learning, I discovered network clusters. My PyCon talk on movie networks is the start of a fascinating exploration of actors that I’ll write more about. […]
Vinu 1 Jan 2022 1:08 am: Interesting data driven view of comparing the level of tribalism (if I can use that term ) within the various film industries around the world. I wonder, though, If casting decisions are directly made by the actors or the initial production and directorial leadership. some stars will command more of a say in whom they will co-Star with and with whom they won’t… but I wonder if that’s relevant to but a small fraction. Its good mental gymnastics, but at the core, your advice is good ;)
How to find a Chinese actor to cast in Hollywood - S Anand 20 Feb 2022 12:38 pm (pingback): […] example, Hollywood actors act outside Hollywood just 10% of the time. Chinese actors act with non-Chinese actors just 1% of the […]

2 inches will change my life

Wed, 30 Dec 2020 10:45:33 +0000

I walked ~11 million steps in the last 3 years, at ~10K steps daily.

Since 1 Jan 2018, I've steadily increased my walking average until Aug 2018. Then my legs started aching. So I cut it down until Jan 2019. In Feb, I resumed and was fairly steady until May 2020. To complement workouts like this, products that are aimed for men over 50 can be used.

In May, my wife refused to let me walk for more than an hour a day. It took me a few months to convince her and level up. I ended 2020 averaging a little over 10K steps for the year.

I'm becoming more regular. I walked 10K/day 15% more in 2020 than in 2018.

2018: I walked 10K steps almost half the time.
2019: it grew to a bit more, to 56%.
2020: I walked 10K steps a day almost two-thirds of the time.

But in May 2020, I went for 5 days without walking even 3K steps.

In 2018, I started being more and more regular until my leg started aching.
2019 was fairly consistent.
2020 is when I applied brakes again -- for very different reasons.

I've never gone for 5 days without walking even 3K/day before, since 2018. At most, it was 3 days at a stretch.

But when my wife refused to let me walk for more than an hour a day in May 2020, I went on strike! 😉

I walk ~77 min daily. This has increased over the years.

In 2020, this has gone up slightly to 84 min -- but it's still under an hour-and-half. I spend most of this time on calls or listening to audio books / podcasts.
Instead of spending it with my family.

Sometimes, I lose myself in calls and walk for almost 3 hrs and 20K steps.

Naveen is usually to blame. But this happens rarely. I walked 20K steps just 6 times over the last 3 years.

Though the longest walk here indicates over 3 hrs, I've never walked 3 hrs in a day.

On 21 Nov, my daughter borrowed my phone and went for her walk. So my phone shows our combined walks, not mine. Many of the other long walks are spread out during the day when I commute by walking in Singapore.

Date	hrs	km	#	Why?
21-Nov-20	3.46	15.5	1	My daughter took my phone. These are her + my walking stats.
15-Nov-19	2.98	11.5	2	Walked to meetings in Singapore.
17-Sep-19	2.96	10.7	3	Walked to meetings in Singapore.
11-Jul-20	2.89	13.9	4	Was talking to Pratap & Ganes.
15-Oct-18	2.83	9.5	5	Walked to meetings in Singapore.
03-Sep-20	2.82	13.0	6	Was talking to Naveen & my coach.

I want to walk faster. I walk at ~4.4 km/hr. My target is 5 km/hr.

Walking at over 5 km/hr speeds the heart up and improves metabolism. (Or so I've heard.)

I was steadily going towards 5 km/hr in my early days of walking. I slowed down starting Aug 2018, since my legs were aching. Then I picked up speed in end-2018.

I slowed down again in Nov 2019 -- and I don't remember why.

In Jun 2020, I started walking much faster -- mainly to complete 10K steps within the hour my wife gave me. That seems to have had a lasting impact. I walked faster overall in 2020.

I've managed fast walking 66 times in 2020, a bit more than before.

In Jun 2020, I walked at over 5 km / hr on 20 / 30 days -- a very consistent high speed. I've never gotten close to this any other month.
(Clearly, there are adverse effects of being able to convince my wife.)

The fastest I walked was in 2018, at 6.8 km/hr. It might have led to my leg aches.

My top 5 walking speeds were in 2018. In 2020, I've managed to walk faster than 6 km / hr just once.

Fastest days	km/hr
07-Jun-2018	6.80
05-Jan-2019	6.65
16-Mar-2018	6.34
08-Jun-2018	6.31
06-Feb-2018	6.19
05-Jun-2020	6.02

The normal stride/height ratio is 0.43. I'm 5'8". My stride is 2.4 ft. That's almost exactly 0.43 times my height. So all is well.

By increasing my stride by 2 inches, I can cover 10,000 steps in 8 min less time.

For every inch I lengthen my stride, I walk ~0.2km/hr faster.

I've walked with a stride as long as 32", which is 3" more than my 2020 average stride. By walking with a 2" longer stride, I can be 9.2% faster.

So in 2021, I plan to get healthier (and scolded less) with a 2" longer stride.

A longer stride means a faster walk. That's a good cardio exercise.
A faster walk also means that it takes less time. So I'll get beaten up less.
All it takes is stretching my legs 2" more. Might hurt a bit. I'll report on this when I know better.

	Now	New	Change	Benefit
Longer stride	29"	31"	2"	Builds character?
Faster walk (kmph)	4.5	5.0	0.5	Better cardio exercise
Time to 10K steps (min)	84	77	-8	Less scolding from wife

PostScript: This analysis was done in Excel. Download see the sheet below.

Walking Excel sheet (Download)

Comments

Somnath 30 Dec 2020 11:28 am: Great to see you back blogging on your personal site. Interesting analysis on your walking with cool insights (and learning Excel tricks) from download :-) Enjoy walking in 2021 to stay healthy and meet your resolution(s).
My Year in 2022 - S Anand 31 Dec 2024 4:31 pm (pingback): […] Walk 10,000 steps daily. […]

Ashish 11 Aug 2015 6:31 pm: Ask Airtel to start 99/- per month national roaming pack to make all incoming roaming calls free.
408wij 11 Aug 2015 9:29 pm: A quick google search turned up two android apps: (1) Call Log Location and (2) Log Call Location. Perhaps their logs can be exported and cross-referenced w/ your bill.
Bala 1 Sep 2015 12:47 pm: You spend 1 day (>24 hours) in talking on your mobile.
Indrajit Ghosh 18 Jan 2017 1:57 pm: You can ask for a better Data deal from Airtel.

Storytelling: Part 1

Wed, 29 Aug 2012 04:13:00 +0000

In a number of sessions I’ve been to, people ask analysts to make their results more interesting – to tell stories with them. I’m co-teaching a course, part of which involves telling stories with data. So this got me thinking: what is a story? How does one teach storytelling to, let’s say, an alien?

Consider this mini-paper.

ABSTRACT: Meter readings exhibit spikes at slab boundaries. We also
find significant evidence of improbably events at round numbers.

Electricity shortage is a serious problem in most Indian states. Part
of this problem is due to the inaccuracy of reporting procedures used
in monitoring meter readings. Our focus here is not to document or
experimentally determine the degree of inaccuracy. We have adopted a
data driven approach to this problem and attempt to model the extent
of inaccuracy using basic statistical analysis techniques such as
histograms and the comparison of means.

Our dataset comprises of the frequency analysis 12-month dataset
containing monthly meter readings of 1.8 million customers in the
State of Andhra Pradesh.

We find that a histogram of these readings shows unexpectedly high
values at the slab boundaries: 50 (+45.342%, t > 13.431), 100
(+55.134%, t > 16.384), 200 (+33.341%, t > 15.232), and 300
(+42.138%, t > 19.958).

We also detected spikes at round numbers: 10 (+15.341%, t > 5.315),
20 (+18.576%, t > 6.152), 30 (+11.341%, t > 4.319).

The statistical significance of every deviation listed above is over
99.9%. Further, every deviation has a positive mantissa. This leads us
to confidently declare the existence of a systematic bias in the meter
readings analysed.

You’re probably thinking: “I know why he’s put this example here. It must be a bad one. So, what a rotten paper it must be!”

Well, not quite. It’s a good piece of analysis. I did it myself and there’s a fair bit of effort and care behind these short paragraphs.

The trouble is, if I read it out to my daughter, she’d say “What?” and not understand a word. My wife’d say “So what?” and not care a bit. I might as well not have written it.

It’s like that Zen thing: If a tree falls in a forest and no on hears it, does it make a sound?

If you did a piece of analysis, and no one understands or cares about it, why did you do it in the first place?

Why do you do it?

That last question is important: why do we analyse?

Sometimes, we do it for fun. The knowledge is beautiful. Knowing Tetris is NP-Complete is rewarding, even though my colleague sarcastically remarked, “Thank God! I’m sooo relieved now that I know that Tetris is NP whatever.” If that’s the case with you, great. Write the analysis any which way you’ll enjoy.

Sometimes, we do it because we’re forced to. In class. At work. Wherever. But that’s another way of saying “I don’t know why I’m doing it.” In that case, I’d gently recommend watching 3 Idiots.

Most often, we do it to share knowledge and drive actions. In that case, if no on understands it, or does anything with it, why do it?

Keep it simple

We prerajulisation of Farhanitate flagellated with ...

Would your audience understand that? Or are you just scared that simple words indicate a simple mind?

I was once afraid. 15 years ago, when writing a paper on IBM India’s competitive advantage for the CXOs, I was worried about it being too simple. I didn’t know anything about management. So I filled it with jargon. They politely nodded when I presented it, but I wasn’t fooling anyone. If there’s no content, jargon doesn’t help.

Unfortunately, it’s become polite to accept jargon as a substitute for substance. Why were they not ripping me apart? Or at least, kindly asking me what on earth I wanted to say?

My friend Manoj did that. In his nice, humble way, he asked, “But Anand, what does this mean?” When I explained it to him, I found I didn’t have a clue. He was OK with that. He just wanted to make sure he hadn’t missed something.

(That’s the technique I use these days. Ask people to explain things clearly. It’s OK if they’re just lost in jargon. I just want to make sure I haven’t missed something.)

Don’t cloak your ignorance. No one will think less of you. In the long run, you’ll learn more, and won’t need the jargon.

Part 2 of the article will talk about focusing on people and actions; storylining and the pyramid principle; and the structure of messages.

Comments

Santanu 30 Aug 2012 4:28 am: Starting to get it. Good thoughts Anand.
Navneeth 26 Dec 2012 9:55 am: I had the opportunity to be an audience on one of your presentations on data visualization. Now I know why I suck at doing presentations… even though I had content. eagerly waiting for part-2.

Birthday matters

Sun, 15 May 2011 20:33:23 +0000

Does it matter which month you’re born in?

Based on the results of the 20 lakh students taking the Class XII exams at Tamil Nadu over the last 3 years (via Reportbee), it appears that the month you were born in can make a difference of as much as 120 marks out of 1,200 – or 10%!

Most students who took the Class XII exams in 2011 were born between March 1991 and June 1992. The average marks of each student (out of 1200) is shown in the graph below.

Students born in June 1991 scored the lowest – around 720/1200. This suddenly shoots up in July, then in August, and the students born in September score as much as 840/1200 on average. From there on, it’s downhill.

This result is consistent across years. In 2009 and 2010, you see a similar pattern.

Why could this be?

Malcolm Gladwell’s book Outliers offers a clue.

Outliers opens, for example, by examining why a hugely disproportionate number of professional hockey and soccer players are born in January, February and March.

The answer turns out to be completely unrelated to numerology or astrology.

It’s simply that in Canada the eligibility cutoff for age-class hockey is January 1. A boy who turns ten on January 2, then, could be playing alongside someone who doesn’t turn ten until the end of the year—and at that age, in preadolescence, a twelve-month gap in age represents an enormous difference in physical maturity.

In Tamil Nadu, students must be 5 years old before entering Class 1. Schools open mid-June. So students born in June 1994 would barely make it in June 1999 – making them the youngest students in the class. July and August students would be missed – but since many schools implement this policy leniently, they sometimes make it in as well. September borns are often consistently the eldest students in a class.

This pattern reflected in the marks. The eldest – the September 1993 borns – score the highest. The next eldest, the October 1993 borns, score a bit less. And so on. (There are older students who take the exam – the ones born before September 1993 – but many of these are failed students from the previous year, introducing a bias in the results.)

Perhaps this initial advantage that the elder students have over their classmates continues through the years? Whatever the reason, it’s clear that if your child is born in September, he or she already has a 100 mark advantage!

Comments

abhijeet 16 May 2011 8:14 am: “Whatever the reason, it’s clear that if your child is born in September, he or she already has a 100 mark advantage!” But at the cost of being older and “losing” 8-9 months!
Shankar V 16 May 2011 10:00 am: Nice…!! Thank God I was born in July! If not on the crest, I at least made it into the upward curve! ;)
S Anand 16 May 2011 10:21 am: @abhijeet: That reminds me of the “Life is a race” dialogue in 3 Idiots :-) Well, not sure how bad being a bit older is. I personally don’t mind, but maybe others do?
Shailesh 16 May 2011 10:33 am: Oh…now I know why I was getting good marks…. :-) I am born in September…and my headmaster changed my birthdate (to Jul month) in school register to not waste my 8-9 months…..
ravi atluri 16 May 2011 10:47 am: They made me study LKG “TWICE”, because I was born in September and didn’t have enough time to be 3! and I have been an anomaly to the september-born-high-performers trend. If the marks are related to the mugging capacity, which if related to age would explain the graphs only for the schooling during. Would like to see, how would these change in college and bachelors :D One more doubt, I have is, many parents have the dob’s of after-aug-born’s to sometime between may to june. If that’s the case, the months on the graph have to be offset ;)
S Anand 16 May 2011 12:35 pm: @Shailesh – if you got into school early, then statistically, you had a DISadvantage. So guess you were MUCH brighter than your classmates! @Ravi – true. No idea how to correct for the real date of birth, though…
k 17 May 2011 2:53 am: this is similar to the paper on Israeli schools where an externally imposed rule created a disproportionate class size, which helps in identifying the effect of class size on performance. date of birth can help in identifying the effect on performance then.
Vasant 17 May 2011 6:14 am: Ha! Now I know why I scored so less in the boards. It’s when I was born! :-) Jokes aside, brilliant analysis.
Sathya 17 May 2011 6:43 am: Good One ! If you still have access to data, would it be possible to plot the same graph against their “numerology numbers” ? ie., 28th born would be 2+8 = 10 => 1+0 = 1. I used to wonder that astrology ( perhaps numerology ) could provide certain insights and that it astro/numero predictions should be verified using statistics. ( Pls note that I do not suggest astro determines a person’s fate. But perhaps it can predict events with some accuracy. How a person reacts to those events are not in the realm of astrology). I would love to do this if I had access to the data. In case you find it worthwhile, it would be interesting to see if numero number matters at all !
Swetha 19 May 2011 9:34 am: M a September born and a gold medalist . (*Grins*)
Sumit Dhar 21 May 2011 3:36 am: And, I know this is extremely difficult, but was wondering if it would be possible to assess the impact of month in which you were born on your professional growth / salary growth. Or even more simply, if marks are correlated with professional success. We Indians tend to believe that good marks == success at work. Would love to know if data validates this belief. If only this kind of data were available… Maybe HR teams in organizations could perform such an assessment. Cheers, D.
Vish 16 Aug 2011 9:59 am: http://news.bbc.co.uk/2/hi/8227268.stm Summer-borns tend to lose out at school because, in England and Wales, 1 September is the cut-off date for school entry, so they can be up to a year younger than their classmates. June baby Frank Lampard Research by the Institute for Fiscal Studies two years ago found only 53% of girls and 47% of boys born in August reached the expected educational level at age seven in state primary schools in England, compared with 80% and 70% of those born in September. Their sporting prowess is affected too. Older children tend to be bigger and more confident, receive more encouragement, and their advantage grows over time. For unto everyone who hath shall be given. Among the 25 most capped England football players, 11 were born between September and November, while only one, Frank Lampard, was born between June and August. In his case, having an England international for a father probably outweighed the disadvantage. So for educational and sporting opportunities, the cruellest and kindest months sit next to each other in the calendar like resentful neighbours.

What does India search for?

Thu, 02 Dec 2010 14:33:19 +0000

Over the last couple of years, I’ve been tracking the top 5 hot searches in India on Google Trends (http://www.google.co.in/trends). Here are the results:

If you're interested in making visualisations out of it, please feel free. But there's one particular thing I'm trying out, which is to categorise these searches and see if there's a trend around that. I've added a "Tag" column.

Could you please help me tag the spreadsheet: https://spreadsheets.google.com/ccc?key=0Av599tR_jVYgdE5zTU5QWjcxVWVCaTBuY3d0NkUtc1E&hl=en_GB

It’s publicly editable, no special access required. If you could stick to the tags I already have (Business, Education, Entertainment, News, Politics, Sports, Technology), that would be great. If not, that’s fine as well.

And if you’ve made any visualisations or done any analysis using this data, please do drop a comment.

Comments

Rockey Nebhwani 2 Dec 2010 11:50 pm: I think simple conclusion may be Bollywood, Politics, Jobs, Stock :-)

Shortening sentences

Wed, 10 Nov 2010 08:16:37 +0000

When writing Mixamail, I wanted tweets automatically shortened to 140 characters – but in the most readable manner.

Some steps are obvious. Removing redundant spaces, for example. And URL shortening. I use bit.ly because it has an API. I’ll switch to Goo.gl, once theirs is out.

I tried a few more strategies:

Replace words with short forms. “u” for “you”, “&” for and, etc.
Remove articles – a, an, the
Remove optional punctuation – comma, semicolon, colon and quotes, in particular
Replace “one” with “1”, “to” or “too” with 2, etc. “Before” becomes “Be4”, for example
Remove spaces after punctuations. So “a, b” becomes “a,b” – the space after the comma is removed
Remove vowels in the middle. nglsh s lgbl wtht vwls.

How did they pan out? I tested out these on the English sentences on the Tanaka Corpus, which has about 150,000 sentences. (No, they’re not typical tweets, but hey…). By just doing these, independently, here is the percentage reduction in the size of text:

2.0%	Remove optional punctuations – comma, semicolon, colon and quotes
2.2%	Remove spaces after punctuations. So “a, b” becomes “a,b”
3.3%	Replace words with short forms. “u” for “you”, “&” for and, etc.
3.3%	Replace “one” with “1”, “to” or “too” with 2, etc.
6.7%	Remove articles – a, an, the
18.2%	Remove vowels in the middle

Touching punctuations doesn’t have much impact. There aren’t that many of them anyway. Word substitution helps, but not too much. I could’ve gone in for a wider base, but the key is the last one: removing vowels in the middle kills a whopping 18%! That’s tough to beat with any strategy. So I decided to just stop there.

The overall reduction, applying all of the above, is about 22%. So there’s a decent chance you can type in a 180-character tweet, and Mixamail.com will still tweet it intelligibly.

I had one such tweet a few days ago. I try and stay well within 140, but this one was just too long.

The Lesson: If you're writing an app (or building anything), find a use for yourself. There's no better motivation -- and it won't ever be a wasted effort.

That was 156 characters. It got shortened to:

Lesson If u're writing app (or building anything) find use 4 yourself. There's no better motivation -- & it won't ever be wasted ef4t.

Perfectly acceptable.

You may notice that Mixamail didn’t have to employ vowel shortening. It makes the most readable shortenings first, checks if it’s within 140, and tries the next only if required.

If anyone has a simple, readable way of shortening Tweets further, please let me know!

Comments

408wij 11 Nov 2010 5:17 am:
1. Are removing punctuation and removing spaces after punctuation mutually exclusive, or do you have a rule for determining which punctuation is optional?
2. You give the example of “before” going to “be4.” It could go to “b4” with the rule that the “be” and “de” prefixes reduce to “b” and “d.”
3. You have “you’re” going to “u’re.” You could expand “you’re” to “you are” and then reducing to “u r.”
4. For readability, I suggest preserving the first and last letters of words. (Compare with http://www.boingboing.net/2003/09/14/scrambled-words-are-.html) E.g., reduce “English” to “Englsh” instead of to “nglsh.” Finally, I the point of the 140-char limit is to limit the scope of a tweet to a simple thought. Shortening words is cheating. But, if you’re going to cheat, I suggest an unshortener for the receiver of tweets. 408wij
Arthi 10 Nov 2010 12:38 pm: I prefer tweetlonger.com. Tweets look much better when they don’t go through reductions.
V 26 Nov 2010 8:36 am: I am writing something similar. Will share once done. But my preliminary attempts brought down the characters atleast 10% lesser than this. For eg. You have got 133 chars , whereas my system results in 123. I shall implement some more NLP techniques and bring it down to an acceptable level without sacrificing the semantics
iwebtalk 11 Jan 2011 8:41 pm: I use utwitmore.com .. no need to worry about 140 characters

Bayes’ Theorem

Mon, 11 Oct 2010 14:53:01 +0000

I’ve tried understanding Bayes’ Theorem several times. I’ve always managed to get confused. Specifically, I’ve always wondered why it’s better than simply using the average estimate from the past. So here’s a little attempt to jog my memory the next time I forget.

Q: A coin shows 5 heads when tossed 10 times. What’s the probability of a heads?
A: It’s not 0.5. That’s the most likely estimate. The probability distribution is actually:

That’s because you don’t really know the probability with which the coin will throw a heads. It could be any number p. So lets say we have a probability distribution for it, f(p).

Initially, you don’t know what this probability distribution is. So assume they’re all the same – a flat function: f(p) = 1

Now, given this, let’s say a heads falls on the next toss. What’s the revised probability distribution? It’s:

f(p) ← f(p) * probability(heads | x) / probability(heads) = 1 * (x^1 * (1-x)^0) / 1 = x

Let’s say the next is again a heads. Now it’s

f(p) ← f(p) * probability(heads | x) / probability(heads) = x * (x^1 * (1-x)^0) / 1 = x^2

Now if it’s a tails, it becomes:

f(p) ← f(p) * prob(tails | x) / prob(tails) = x^2 * (x^0 * (1-x)^1) / 1 = x^2 * (1-x)

… and so on. (This happens to be a called a Beta distribution.)

Now, instead of this being the probability of heads, it could be the probability of a person having blood pressure, or a document being spam. As you get more data, the probability distribution of the probability keeps getting revised.

Comments

Ram 18 Jan 2012 1:59 pm: Hi Anand, f(p) ← f(p) * probability(heads | x) / probability(heads) = 1 * (x^1 * (1-x)^0) / 1 = x What I understand from the above notation is f(p) stands for probability distribution function (which is used recursively), x stands for the probability of getting a head (which is unknown) and 1-x stands for the complement of x. Can you please explain why the distribution function is defined as you have done? i.e. , f(p) ← f(p) * probability(heads | x) / probability(heads)

Enron email analysis

Mon, 09 Oct 2006 12:00:00 +0000

1.5 million internal e-mails of Enron were released after it collapsed, to help figure out why. The UC Berkeley Enron Email Analysis Project has some links analysing these emails. Check out the visual analysis.

Comments

Prakash Ayer 9 Oct 2006 3:03 pm: Hi Anand, Did you mean Enron instead of Amazon here? Take Care
S Anand 9 Oct 2006 3:15 pm: Gosh, yes – sorry, typo on my side. (Wonder why that happened… must’ve been sleep-typing :-)
Ravi 16 Oct 2006 6:51 pm: Along the same network visualization lines, http://www.touchgraph.com/TGGoogleBrowser.html and http://liveplasma.com/ are pretty interesting too. Or if you like Digg, http://labs.digg.com/swarm/ works too. (drop me a line some time Anand - kumar (pulli) venkateswar (at) gmail (pulli) com)
Irrexu 27 Oct 2006 5:06 am: Long time since you’ve posted any article on your site.. Anand, just a suggestion though.. I think you should start writing more about yourself and the latest in your life.. I am sure there are a lot of takers for that like me. Cheers!

More Google services

Thu, 11 May 2006 12:00:00 +0000

Google launches Google Co-op, which lets you search deep content (and share deep content), and Google Trends, which is like Google Zeitgeist for your searches.