data-analysis 28

At 9:30 pm, I checked my calendar for tomorrow’s appointments, alt-tabbed frantically into ChatGPT, and started typing: Tomorrow is my 24th anniversary. It’s a bit late for me to buy anything (except maybe an online service) or prepare something. This has become a habit – leaving things to the last minute and asking ChatGPT to save my day. I did give it good context, though. You remember the OCBC expenses treemap you created by analyzing my transactions? That will give you a good guessable idea of the kinds of things she spends on and hopefully, therefore, what she likes. ...

Stanford ran a conference called Agents for Science. It’s a conference for AI-authored papers, peer reviewed by AI. They ran three different AI systems on every paper submitted, alongside some human reviewers. The details of each of the 315 papers and review are available on OpenReview. I asked Codex to scrape the data, ChatGPT to analyze it, and Claude to render it as slides. The results are interesting! I think they’re also a reasonably good summary of the current state of using AI for peer review. ...

In my Mining Digital Exhaust workshop on Saturday, One discovered that they cycle when life is unstable, not for fitness. Another found that their buys are good but sells are bad trades. I learnt that I watch YouTube most at office (12-4 pm), not at home. How? A fairly straight-forward process: Export your personal data. (Use Chrome Devtools Protocol to scrape.) Upload to ChatGPT, Gemini, Claude, … and have them analyze with code. Have them narrate in the style of your favorite author. Models are super smart, but everyone has equal access to them. Your personal data is unique. Combine them to get something powerful. ...

I always wondered why old movies are rated so high on IMDb. For example, 12 Angry Men (1954) with just ~900K votes ranks about as high as Inception (2010) with ~2M votes. Few people I know have seen 12 Angry Men. So where does this high rating come from? My theories were: Old movies really are that good. IMDb’s algorithm is biased towards old movies. People remember older movies fondly. Actually, it’s none of these. It’s selection bias. ...

This talk is an experiment. I am going to talk (literally) to ChatGPT on stage and have it do every kind of data analysis and visual storytelling I have ever done. Bangalore. 27 June. Of course, this is an LLM era away. So no promises. We might be doing something completely different on stage. LinkedIn

I’m at an open Hyderabad meet-up, Thu 20 Mar 4 pm. “Analyzing data with AI agents”." It’s a public event by Hasgeek. Venue: Castlight Health, Sattva Knowledge Park. We know LLMs suck at number crunching but are good with code. I’ll share what we’ve learnt by getting it to write code to analyze data instead. Less lecturing, more interactive Q&A and demos in a cozy group. Mostly for analysts, data scientists, and programmers. Not so much for LLM researchers or managers. ...

My music library has around 1,000 songs (mostly Tamil and Hindi, with some Telugu and English film songs). I spent this morning tagging them by year with mp3tag. (Manually. You don’t automate the pleasures of life.) I thought my 1990s collection would be the largest. I was in college, listening to lots of music then. But surprisingly, my collection has grown post the 1990s. ...

Film actors mostly act within their own industry. For example, Hollywood actors act outside Hollywood just 10% of the time. Chinese actors act with non-Chinese actors just 1% of the time. So, if you’re a Hollywood producer trying to cast a Chinese actor, how would you find them? One way is to list Chinese actors with the largest number of Hollywood co-stars. Let’s see who tops that list. ...

These are the major group actors based on who they act with most. Language. Not country. For example, the Spanish / Mexican group is across countries. But Indian actors divide into North Indian and South Indian. It’s language, not country. Time period. Old American actors are a separate group from Hollywood. (Naturally. Brad Pitt was born after Humphrey Bogart died. They couldn’t have acted together.) Genre. Hollywood Porn actors don’t act with mainstream Hollywood. Same with Japanese Porn, Hollywood TV, and Hollywood Horror actors. How are these groups themselves connected? Do Chinese actors act with Hollywood often? How isolated is Bollywood from world cinema? ...

Can foreigners break into #Hollywood? A break into films is hard. Particularly when you’re a foreigner. But is Hollywood more open or less open than other countries? If we go by iconic actors, the numbers are not encouraging. Just 2% of Will Smith’s co-stars are non-Hollywood. But over 30% of Jackie Chan’s co-stars are non-Chinese. But #clustering shows that Hollywood is actually among the most open to foreigners. #dataanalysis Read more at https://www.s-anand.net/blog/can-foreigners-enter-hollywood/ ...

An aspiring Malaysian actor posted on Reddit: I am a 18-year old biracial Malaysian kid who wants to be an actor in Hollywood. I’m taking a diploma for performing arts in a college called Sunway University in 8 days and I’m considering pulling out of it because why do something that I like when my dreams might never be fulfilled and the price for taking this diploma is seriously expensive. I am starting to doubt my chances of making it to Hollywood and I suffer from extreme anxiety. Is it possible for someone like me to enter Hollywood? What are my chances? ...

I walked ~11 million steps in the last 3 years, at ~10K steps daily. Since 1 Jan 2018, I've steadily increased my walking average until Aug 2018. Then my legs started aching. So I cut it down until Jan 2019. In Feb, I resumed and was fairly steady until May 2020. To complement workouts like this, products that are aimed for men over 50 can be used. In May, my wife refused to let me walk for more than an hour a day. It took me a few months to convince her and level up. I ended 2020 averaging a little over 10K steps for the year. ...

My monthly postpaid mobile bills have been in the Rs 2,000 – Rs 3,000 range for some time now, and I spent a few hours dissecting them yesterday. Page 3 had the good stuff. It’s a little hard to figure out, but what the last 2 columns say is that most of my spend is offset by discounts. What’s not getting offset are outgoing roaming calls. Followed by calls to local landlines. For all practical purposes, that’s the only thing that counts in this bill. Everything else is close enough to zero. ...

In a number of sessions I’ve been to, people ask analysts to make their results more interesting – to tell stories with them. I’m co-teaching a course, part of which involves telling stories with data. So this got me thinking: what is a story? How does one teach storytelling to, let’s say, an alien? Consider this mini-paper. ABSTRACT: Meter readings exhibit spikes at slab boundaries. We also find significant evidence of improbably events at round numbers. Electricity shortage is a serious problem in most Indian states. Part of this problem is due to the inaccuracy of reporting procedures used in monitoring meter readings. Our focus here is not to document or experimentally determine the degree of inaccuracy. We have adopted a data driven approach to this problem and attempt to model the extent of inaccuracy using basic statistical analysis techniques such as histograms and the comparison of means. Our dataset comprises of the frequency analysis 12-month dataset containing monthly meter readings of 1.8 million customers in the State of Andhra Pradesh. We find that a histogram of these readings shows unexpectedly high values at the slab boundaries: 50 (+45.342%, t > 13.431), 100 (+55.134%, t > 16.384), 200 (+33.341%, t > 15.232), and 300 (+42.138%, t > 19.958). We also detected spikes at round numbers: 10 (+15.341%, t > 5.315), 20 (+18.576%, t > 6.152), 30 (+11.341%, t > 4.319). The statistical significance of every deviation listed above is over 99.9%. Further, every deviation has a positive mantissa. This leads us to confidently declare the existence of a systematic bias in the meter readings analysed. You’re probably thinking: “I know why he’s put this example here. It must be a bad one. So, what a rotten paper it must be!” ...

Does it matter which month you’re born in? Based on the results of the 20 lakh students taking the Class XII exams at Tamil Nadu over the last 3 years (via Reportbee), it appears that the month you were born in can make a difference of as much as 120 marks out of 1,200 – or 10%! Most students who took the Class XII exams in 2011 were born between March 1991 and June 1992. The average marks of each student (out of 1200) is shown in the graph below. ...

Over the last couple of years, I’ve been tracking the top 5 hot searches in India on Google Trends (http://www.google.co.in/trends). Here are the results: If you're interested in making visualisations out of it, please feel free. But there's one particular thing I'm trying out, which is to categorise these searches and see if there's a trend around that. I've added a "Tag" column. Could you please help me tag the spreadsheet: https://spreadsheets.google.com/ccc?key=0Av599tR_jVYgdE5zTU5QWjcxVWVCaTBuY3d0NkUtc1E&hl=en_GB It’s publicly editable, no special access required. If you could stick to the tags I already have (Business, Education, Entertainment, News, Politics, Sports, Technology), that would be great. If not, that’s fine as well. And if you’ve made any visualisations or done any analysis using this data, please do drop a comment. ...

When writing Mixamail, I wanted tweets automatically shortened to 140 characters – but in the most readable manner. Some steps are obvious. Removing redundant spaces, for example. And URL shortening. I use bit.ly because it has an API. I’ll switch to Goo.gl, once theirs is out. I tried a few more strategies: Replace words with short forms. “u” for “you”, “&” for and, etc. Remove articles – a, an, the Remove optional punctuation – comma, semicolon, colon and quotes, in particular Replace “one” with “1”, “to” or “too” with 2, etc. “Before” becomes “Be4”, for example Remove spaces after punctuations. So “a, b” becomes “a,b” – the space after the comma is removed Remove vowels in the middle. nglsh s lgbl wtht vwls. How did they pan out? I tested out these on the English sentences on the Tanaka Corpus, which has about 150,000 sentences. (No, they’re not typical tweets, but hey…). By just doing these, independently, here is the percentage reduction in the size of text: ...

I’ve tried understanding Bayes’ Theorem several times. I’ve always managed to get confused. Specifically, I’ve always wondered why it’s better than simply using the average estimate from the past. So here’s a little attempt to jog my memory the next time I forget. Q: A coin shows 5 heads when tossed 10 times. What’s the probability of a heads? A: It’s not 0.5. That’s the most likely estimate. The probability distribution is actually: ...

1.5 million internal e-mails of Enron were released after it collapsed, to help figure out why. The UC Berkeley Enron Email Analysis Project has some links analysing these emails. Check out the visual analysis. Comments Prakash Ayer 9 Oct 2006 3:03 pm: Hi Anand, Did you mean Enron instead of Amazon here? Take Care S Anand 9 Oct 2006 3:15 pm: Gosh, yes – sorry, typo on my side. (Wonder why that happened… must’ve been sleep-typing :-) Ravi 16 Oct 2006 6:51 pm: Along the same network visualization lines, http://www.touchgraph.com/TGGoogleBrowser.html and http://liveplasma.com/ are pretty interesting too. Or if you like Digg, http://labs.digg.com/swarm/ works too. (drop me a line some time Anand - kumar (pulli) venkateswar (at) gmail (pulli) com) Irrexu 27 Oct 2006 5:06 am: Long time since you’ve posted any article on your site.. Anand, just a suggestion though.. I think you should start writing more about yourself and the latest in your life.. I am sure there are a lot of takers for that like me. Cheers!

Google launches Google Co-op, which lets you search deep content (and share deep content), and Google Trends, which is like Google Zeitgeist for your searches.

Why is nanotechnology in the top Google queries from India? Comments ravi 15 Apr 2006 10:48 am: from all countries in the list indians are the only one with something technical in their queries. hurrah for that Prabhu 17 Apr 2006 6:03 am: Could be due to the fact that Sujatha is writing an article about it in a tamil weekly S Anand 17 Apr 2006 7:37 am: If so, Tamil Nadu would have to be a big chunk of India’s Google searches. Given the absense of other Tamil queries, I doubt this was the only factor… but it sure must have contributed! Gautam 26 Apr 2006 6:05 pm: funny how sania mirza ranks over aishwarya, though !

On the IMDb top 250, you normally see a correlation between the number of votes and the rating for a movie. Better rated movies are more watched. The outliers are interesting. The movies that are popular despite not having a high rating are: The Matrix The Sixth Sense Gladiator Star Wars 3: Revenge of the Sith Pirates of the Caribbean I can understand why The Sixth Sense, Pirates of the Caribbean and especially The Matrix are on this list – geeks would have watched these and voted on IMDb, though their voting need not have been high. But why are Gladiator and Sixth Sense on that list? ...

Americans have more leisure time than before. But why do Americans feel so harried? Weirdly, prosperity may be to blame in two ways. First, thanks to rising real incomes, an American’s time is worth more now. A walk in the park is more expensive than it used to be. (When people complain to him about being too busy, Mr Hamermesh tells them that their real problem is too much money.) ...

Google web authoring statistics. An analysis of over a billion pages to see how people use HTML markup.

Python vs Perl. Sums up my feelings perfectly: Python may be better for larger projects, but for my meddling, I’ll stick to Perl. It’s served me well for 10 years. Until 1999, I used Perl a fair bit, but no more than Java or C or anything else. My first “real-life” use of Perl was in 2000, when I was processing 600MB of IBES data. Access and SPSS couldn’t handle the load. Perl slurped all the data in a few seconds, though. A few years later, when processing bank data (3GB worth, this time), Perl again was the only saviour. In fact, between Excel and Perl (and CPAN), I think I have all the data analysis power I’ve ever needed. This blog, for instance, is written in an Excel spreadsheet, exported to XML, and converted into the blog format by Perl.

Are the free Mac Minis for real? Hardy tries to get one – and succeeds. But… is it worth it? Pretty good economic analysis.

The authors of Freakonomics visited Google, and were asked “What would you do with our data?” BTW, there is a regular Freakonomics column on the Times.

Music helps identify software bugs. I’m sure it can be extended to many other forms of ordered data. DNA sequences, time series, etc.