Making a short data movie requires the same essentials as filmmaking—theme, hypothesis, screenplay, visuals, timing, and post-production—even when the tools are PowerPoint and iMovie.

How to direct a data movie Ganes and I created a data movie on speed-cubing records as part of a Gramener hackathon. Here's a video of us talking about how we created it. Anand: We picked the Rubik's cube story for this hackathon. Tell me more about how this excited you. Ganes: Since my son started solving the Rubik's cube a few months back, I've been fascinated with these competitions. I still don't know how to solve it, but I like watching it. Anand: But he does? Ganes: Yeah, he does. So, in the competitions, I've seen kids solving the Rubik's cube in under 10 seconds. So that was the first source of amazement. I've seen kids doing it with one hand, blindfolded. I first couldn't believe it. Doing it with their legs. So that got me really interested. When we were talking about this, and I was sharing my amazement, we were talking about the hackathon and the conversations kind of merged. So that, I think, the curiosity around it led to picking this as the story. Anand: And what was the next step? Ganes: I have always seen the World Cube Association publishing these records. Their website is great. So I thought maybe we could scrape from that, and that’s when I start looking at the website and the competitions we can pick. and then I stumbled on the export feature where they have multiple formats neatly curated that you can take and directly start the analysis. Anand: Which was actually a big factor in deciding to go for this. Big data set. Very rich, interesting possibilities. Ganes: So we had had some five or six ideas. This immediately shot up to the top. So after we got the idea, you kind of took over. I think after I mentioned that all these formats were available, it got you excited. So what did you do after that? Anand: Then it became a question of what all interesting things we can find. It's almost an exploratory data analysis, but my approach to EDA (exploratory data analysis) is: let's formulate the hypotheses and then validate, and see if there Is an interesting story behind it. So it begins with, for instance, the speed at which records have been broken. Today, it's at 3½ seconds. We know that. But how fast did it fall? Or: what's the spread of solving-speed for somebody who solves it fast? Does the same person solve it really fast sometimes and really slow sometimes? Is there a movement in their average? You said, "Let's see how much longer it takes to solve bigger cubes." Nikhil was going to take the demographics of solvers and see how they're spread out. There are definitely a lot of Chinese solves in the spread. So, the thing was, let's look at possible ideas that could lead to an interesting answer, and then validate those. Ganes: It was almost like "What would we be interested in finding out" and not necessarily like looking at the column of data. Anand: Yes. And that I think is important, because, from the data, there may be some ideas. But after absorbing it, knowing what's interesting is what should drive the story. Ganes: Right. Yeah. So that was a good starting point. We listed all of these on the board. Then, what did you do next? Anand: Then it's about proving these. So, we know here are some possible interesting stories, and let us explore and validate whether these are, in fact, interesting, or can be turned into something interesting. So, when I looked at the speed at which records were broken, for instance, I thought that would be an interesting story. But it wasn't. It was just getting broken at a steadily successive pace. But something that I did not expect emerged, which is that Wusheng Du, who holds the world record, is not the person who was there in the records consistently. In fact, Felix Zemdegs has been the consistent winner for the last 10 years and is the only cubing champion who's won the WCA twice. So, that was something that emerged from doing the analysis. So, that has the ability, therefore, of both proving what we're looking to prove (or disproving), and also coming up with new stuff that we can choose to incorporate into the story. Ganes: Almost like starting with a business hypothesis, or what, in the enterprise world, the business wants to know, and then once you get into the data, the data is revealing a few interesting insights, and then you kind of marry both. Looks just like that. Anand: Exactly. Exactly. Ganes: So, we identified the insights. And then, the target here was to come up with a 2 minute video. So how did you plan from insights to the video. Anand: So, one of my cousins is a director, and she tried explaining to me the concept of a screenplay. I never really understood it, even though I've read a number of screenplays. So, in the last hackathon, when I was creating a (data) movie, that's when I realised: as I started writing what I want to shoot (because it requires a whole lot of planning), I was effectively writing a screenplay. The steps are, basically, you have to decide what are the frames or the sequences you want to shoot. So, one sequence was: we want to introduce this Rubik's cube win. Another sequence was: we want to show how quickly different types of cubes can be solved, etc. So, for each of these, what I do is: create a storyline that has the following structure. One: what is the message I want people to take away from that. Ganes: The headline from there. Anand: Exactly. And then, in order to do that, what are the words I would narrate on top of it? That literally forms the dialogue. The third thing is, what are the visuals that prove the dialogue. That I structure in the form of a video. The fourth thing is the transition -- from one video to another, or from one sequence to another, how do I flow. These are the 4 things that I captured. When I write down the full dialogue. I speak it out, put in a timer, and then say "OK, this took 10 seconds, this took 15 seconds, this took 14 seconds" and so on. Then comes the process of recording (the audio). Assembling the visuals, yes, but timing it and sequencing it based on the recording is pretty critical. So, actually, I wanted your voice - it's better. And initially, I wanted you to do the recording, but because you were busy in the Dell workshop, I had to do the recording to make sure that I get the timing. Then you re-recorded post that. That recording makes a huge difference. The audio quality on my iPhone is better than the laptop. I transfer it via Dropbox on to the system. Ganes: Were there some issues because you have some insights and you have a certain sequence, but it may not add up to 2 minutes. Or, there might be something which will just not flow. How do you correct those issues? Anand: I found that I consistently underestimate (the time). I thought that we only have material for 1½ minutes, but I knew at that point that invariably, because of this bloat, it will somehow add up to 2 minutes. Which is exactly what happened. It moved to 2 minutes 4 seconds. Ganes: Yes. Exactly. Yeah. Anand: So, once you've done it once or twice, that amount of correction is there. It's in fact a whole lot easier to control a video than something as crazy as a (software) program, for instance. The estimation error in programming is much higher than this. The good part is that post production or editing can take care of a lot of stuff. That 2-minute video can be cut to 1½ if required. Ganes: Yeah, it can be improved, but my biggest fear is: after recording, the post production is a nightmare. It takes hours and hours of effort. A five-minute video, to post, probably takes 2 hours. Anand: That is true. Ganes: How do you go about it? After having these audio clippings, videos and images, how do you stitch all together into a video? Anand: My workflow is on PowerPoint, mostly, and then on Windows Video Editor. And then you introduced iMovie into the mix. PowerPoint makes it fairly simple. I can put in an audio in the background. I can handle the animations. It's not a great tool at all, but it's a tool I'm very familiar with. So, my workflow is: one slide is one shot or one headline in the storyline. Then I record the video independently or download it from YouTube, put it in the background or wherever. Create all the visuals, create the animations around it, put it there. At this point, the raw material is in. Then I insert the audio and let it play the background for that particular slide. Then I time the animation to the audio. This is a slow process because PowerPoint doesn't have the right tools. So I play the audio till that point and then set the animation. Then I start from the beginning again, play the audio to the next point, and then set that animation. Which takes a long duration. But once that's sorted out, I play that full slide and it works out, I then go back and correct. The good part is that the audio is the time keeper. I pre-recorded the audio. So I know that the entire duration is only going to be 1.8 minutes (and then towards the end we added a few more vidoes that took it to 2 minutes). So the audio keeps you in control, and if you synchronize everything to the audio, then it becomes easier. Then I exported it into a video file from PowerPoint directly, and then did a little bit of post-processing, adding a background music and adding a few captions, mostly, on Windows Video Editor, and then gave it to you. Which was at around 9 o'clock or so. What did you do from 9 o'clock to 3 o'clock? Ganes: So, the first thing -- on the PowerPoint, I couldn't believe that you'd done all this on PowerPoint. Yes, you're taking the tool beyond the limit it was designed for. I've been working with iMovie for a year, and I find it very powerful. For someone who doesn't come from that background, it was very easy for me to pick up. I had the images and raw video footage for the different portions we were trying to introduce. I was able to split the audio that you recorded from the video, and then was able to record mine and add it. iMovie has these multiple streams you can insert and remove. I had one stream for my audio for my voice over. And there was this video which you had. On top of that, I could overlay the pictures and other videos that I had towards the end -- two videos playing side-by-side. So all of that was possible. and then I could also introduce background music at the very end. iMovie makes it very easy to move all of these things around. And even the synchronization issue which you told about, that's much easier to resolve in iMovie. So, all of this finally coming together, I think, at 3 o'clock… when I had all of this, at 3 o'clock I was hunting for the background music (laughs). I was playing all kinds of clips and finally I chose one. So that's how we got the final YouTube video. Anand: My lesson from this is: make sure you have a team member who has a Mac! Ganes: Right, yeah. So let's go back and look at our video and see what we can learn from it. Thank you!

How to direct a data movie


I use the markdown-customblocks library to extend Python Markdown for complex layouts. This allows me to create nested Bootstrap columns and custom tags, like audio players, directly within Markdown files without writing raw HTML code.

One problem I've had in Markdown is rendering a content in columns. On Bootstrap, the markup would look like this: How do we get that into Markdown without writing HTML? On Python, the attribute lists extension lets you add a class. For example: ... renders This is some content . But I can't do that to multiple paragraphs. Nor can I next content, i.e. add a .col inside the .row. Enter markdown-customblocks. It's a Python module that extends Python Markdown. This lets me write: This translates to: Better yet, we can create our own custom HTML block types. For example, this code: ... lets you convert this piece of Markdown: ... into this HTML: markdown-customblocks is easily the most useful Python module I discovered last quarter.

How to extend Markdown with custom blocks


A Minecraft hotfix that restores diamond availability matters deeply when you have spent ten hours mining and still badly need Fortune III.

I spent 10 hours this week mining for diamonds. I found just 8! I badly need Fortune III on my pickaxe. But today, my biggest worry is sorted out. Diamond is back in usual quantities in Minecraft 1.17! https://www.windowscentral.com/minecraft-bedrock-edition-1-17-2-hotfix LinkedIn

I found just 8 diamonds in minecraft


A short LinkedIn note highlights the top fiction and nonfiction picks from a year of reading and points readers to the full one-line review list.

I read 52 #books in 2021. The best #nonfiction was The Almanack of Naval Ravikant. Succinct & deep. I can spend a decade practicing every sentence. The best #fiction was Brandon Sanderson's Rhythm of War. A brilliantly rich magic system, and what a plot, what an ending! My one-line #reviews of the books are below. http://www.s-anand.net/blog/books-in-2021/ LinkedIn

I read 52 books in 2021


A client leader's delighted reaction to a TV-rating analytics platform becomes a moment of validation for all the hidden weekend work that built it.

I had a lovely interaction with Kaushik Das at Star TV. The goosebump moment for me was when he said, "... my boss called me the next day, and said that I've been playing with it since you showed it to me. And yeah, this is amazing! Why didn't we do this before? Thanks, Star TV Network & Gramener teams, for your night-outs, weekends, and relentless energy -- and building a TV Rating #analytics platform you can be proud of. LinkedIn

Interaction with kaushik das at star tv


This teaser introduces a playful movie-network story in which Govinda supposedly reaches Angelina Jolie through surprisingly short paths in actor-collaboration graphs.

Jolie No. 1 There are more Bollywood actors in Hollywood. Some are even turning down Hollywood roles. So we wondered: How easily can a Bollywood actor connect to a Hollywood actor? As part of the Oct 2019 Gramener data story hackathon, Anand, Kishore, and Niyas created a Jolie No 1 — a data video where Govinda announces (in our imagination) that he will act with Angelina Jolie in Jolie No 1, but declines to comment on who introduced them. Here's the video: https://youtu.be/lcwMsPxPIjc Read the full article at http://www.s-anand.net/blog/jolie-no-1/ Jolie No. 1 LinkedIn

Jolie no 1 linkedin


A hackathon team turned actor-network data into a playful mini-movie by asking how Govinda might plausibly connect to Angelina Jolie through the shortest chain of co-stars.

Jolie No. 1 There are more Bollywood actors in Hollywood. Some are even turning down Hollywood roles. So we wondered: How easily can a Bollywood actor connect to a Hollywood actor? As part of the Oct 2019 Gramener data story hackathon, Anand, Kishore, and Niyas created a Jolie No 1 — a data video where Govinda announces (in our imagination) that he will act with Angelina Jolie in Jolie No 1, but declines to comment on who introduced them. We picked a theme first The hackathon theme was "movies". We explored 5 themes: 1. Who acts most in cameo roles, and what's the impact on revenue? (Based on The Numbers) 2. Which actors acted often together? (Based on IMDb data) 3. Which movies become hits on TV? (Based on BARC TV data) 4. What is the social network of actors in individual movies ( ) 5. Correlation of TV series actors and their revenues We explored insights next We picked the first two themes because we liked them. 1. Cameo appearances Some observations were: Stan Lee starred in 45 cameo roles. No one even comes close. Some roles are: A school bus driver in Avengers: Infinity War (2018) A strip club DJ in Deadpool (2016) A hot-dog vendor in X-Men (1995) Jay Leno (25) and Larry King (21) follow, mostly starring as themselves Alfred Hitchcock (16) has famous cameo appearances in most of his films, such as: Man mailing letter in Suspicion (1941) Man winding the clock in Rear Window (1954) Man walking the docs in The Birds (1963) We didn't have inflation-adjusted box-office revenues, so we couldn't compare the revenues. 2. Which actors acted often together Some observations were: Top hero-heroine combo: Overall: Prem Nazir & Jayabharati Hollywood: Billy Dee & Mike Horner (pornstars) Tollywood: Krishna Ghattamaneni & Jaya Prada Bollywood: Jeetendra & Rekha Top male combo: Sivaji Ganesan & Nagesh (more recently, Senthil & Goundamani) Top female combination: Lalitha & Padmini Top pair of: Shah Rukh Khan: Rani Mukherji Amitabh Bachchan: Hema Malini Kamal Haasan: Sridevi Rajinikanth: Sridevi Sridevi: Krishna Ghattamaneni Chiranjeevi: Vijayshanti Dev Anand: Madhubala The observations focus on Bollywood and Hollywood (because of our familiarity) -- but there are number of insights on Japanese and French films too. We decided to go with this theme because it offered multiple storylines: Some actors pair up with each other, e.g. Gemini - Savithri Some actors have a big "following" e.g. Rajinikanth, Kamal Hassan, Jitendra have acted most with Sridevi Some actors form cliques -- working only with each other Often, comedians are the bridge between cliques It's interesting to see how actors from one clique can connect to another Creating the storyline When exploring of actors' connections, we found a clearly delineated network structure. Actor SNA The group of densely clustered actors is the Bollywood-Tollywood-Mollywood-Kollywood nexus. It appears disconnected from the Hollywood cluster. (We excluded anyone who hadn't acted together in at least 4 films.) The data was created using this Jupyter notebook. We realized that it's tough for someone in Bollywood to connect to Hollywood. Maybe that could be the plot? For example, what if Amitabh Bachchan wants to act with Metryl Streep? But this isn't an interesting story. So we asked: Who is the most desirable heroine in Hollywood? Our guess was Angelina Jolie Who would make funny co-actor? We toggled between Rajanikanth, Brahmanandam, and finally picked Govinda. The plot summary was: Govinda wants to act with Angelina Jolie. Who can connect them? The analysis is in this Jupyter notebook. Write the screenplay The morning of the hackathon was spent finalizing the screenplay and dialogues, written on Dropbox Paper. Create the video Anand and Niyas created the visuals on PowerPoint, collaborating on Dropbox. This is the first version of the presentation. It uses morph transitions extensively. PPT screenshot Niyas and Kishore recorded the audio in two parts on their phone, shared it with Anand via WhatsApp. We integrated these using the Windows 10 video editor. It's simple, but now powerful. For our use, simplicity was more important. The process took 6 hours (from 8 am to 2 pm). Writing the screenplay and dialogues: 1.5 hours Creating the presentation: 2 hours Recording the audio: 1 hour Integrating into the video: 1.5 hours At the last minute, we picked the title "Jolie No. 1" as a parody of Govinda's No. 1 film series). We published this on Google Drive, and then on YouTube.

Jolie No. 1


This talk transcript shows how reshaping and joining mismatched maps unlocks political, administrative, and business insights that would otherwise stay hidden in disconnected geographic layers.

Maps, Delimitation, and Gerrymandering I delivered a talk at PyCon India 2019. My slides are on Github. This is a transcript of that talk. What I'm going to be talking about is how you can get insights by joining two maps but before we go there, just some basic bookkeeping things. In case you're tweeting, these are the hashtags, you probably want to be using the #PyconIndia, my hashtag my IDs, #SANAND0, you don't need to worry about the slides, they are online. I've already posted on Twitter, the link to the slide deck, the slide deck that you're using but if you desperately do want to take notes, then one small suggestion. Research has shown that taking notes on pen and paper is much better than taking notes on laptops if you want to remember stuff or on mobile phones. So this was a discovery for me. In fact, it was my discovery of the year and I'm following it diligently. Do give it a shot if you want to take notes. Let's dive in. The story begins at the Karnataka elections in 2018. Say about one-eighth of the voters are Muslim, and both the Congress as well as the JD(S), were trying to get their support while on the other hand, BJP was taking potshots saying both of them are just trying to appease the community. The Hindu newspaper wanted to write a piece about how large a factor this is, and where all the Muslim vote is strong. You see, here we have a problem. The thing is that the proportion of the population by religion is available only at the district level or the village level, depending on where you get the data from and this is from the census. Unfortunately, elections are not conducted by the district, elections are conducted by constituency and these are two very different maps. So, I have data in one map, which shows me how many Muslims exists in a particular region and I want to see how many Muslims live in a different region on another map and even though they overlap, there really is no direct way of getting the data from one layer on to the other. So, we literally don't know how many Muslims live in a constituency. So, how do we solve this problem? Well, the logical way is you could take one district and a constituency or a set of constituencies, and let’s say the district has a population of 100, out of which we know that 13% are Muslims and we want to split it evenly across a bunch of constituencies. We could just overlay them. So one district could cover multiple constituencies, one constituency could cover multiple districts, and there is a many to many mapping between these there is sometimes full coverage, sometimes partial coverage. So this district, for instance, covers at least one constituency fully and maybe this takes up about 1/3 of the total area. So I can say approximately 1/3 of the district's population, which is that red area lives in this constituency. Or let's take another constituency that overlaps with this district. So, now only a portion of this constituency overlaps with this district. So in this area, which takes up maybe about 1/5, or about 20% of that district population, I can say that population lives here. In other words, we are simply making an assumption that within a district, which is the lowest level of data that you have, or if you have village data that's far more granular, the population is uniformly distributed. That's the basic assumption. Now, what we can do is fragment each of these districts and constituencies by overlaying them and creating an intersection out of those, and reassembling those and this is a process that I call reshaping the map. How much of this can we do in Python? There is a library called reshaper that we put together. The reshaper Library is something that's very work in progress by the way. You can find it github.com/gramener/reshaper. It does exactly what I'm about to show you right now. So let's give it a shot. I'm going to open up the IPython notebook. The library that we are going to be using for this primarily, the core library is Geopandas. For those of you who have been working with data, Pandas is pretty much the de-facto library to use for any kind of data processing. Geopandas is becoming that kind of a standard for any shapefile. So if you have any shapefile and you want to do any kind of geospatial processing, an easy way of doing it is Geopandas and an easy way of installing it is through Conda using Anaconda. Rather than trying to do a pip install by yourself. It's a little more efficient on most machines. So let's import Geopandas. Now I have a shapefile that has the Karnataka census data which will eventually appear. I'm going to load it once it appears on the screen. (Just taking a long time. Okay, there we are back again.) So GPD, which is the abbreviation for Geopandas, has a fromfile function that lets you load any shapefile. Now, the other question you'll have is where am I going to get these shapefiles from? We'll come to that in a bit. It's not as difficult as you might think. Let's say you've downloaded the shapefiles. This particularly is the Karnataka census shapefile and what does this look like? Geopandas has a plotting function, which lets you see what the map looks like. So if you look at these districts, this is a pretty large district, this is Bangalore. Let's take the area for these. Geopandas offers an attribute called .geometry, which has an attribute called .area, which gets you the overall area of each of these regions, and if you want to look at what that data frame looks like, each of these regions corresponds to one row. So the Bagalkot district is one row, Bangalore rural district is one row and so on. All of the data in the shapefile also comes in here, you have a column called geometry, which has the additional geometry details, this is a pretty large column, which you probably won't be going into the details of it. We've just now added one column called area, and this has the area of each of these regions and at the very least, you can figure out which are the larger regions, which are the smaller regions. Let's do the same for the constituencies data set. So here we have the constituencies that are more, these are parliamentary constituencies, by the way, not assembly constituencies. The difference being if you're electing someone for the parliament, it's or an MP, then it's a parliamentary constituency. If you're electing them for the assembly, which is an MLA, then it's the assembly constituency. Parliamentary constituencies are bigger. So you'll notice that out here, there are multiple parliamentary constituencies that sit in the same region that this district sits in, but it's not a perfect match. Again, let's take the area and see what this looks like. We have a bunch of these parliamentary constituencies like Gulbarga, Bijapur, etc, and their respective areas. Now, Geopandas has a function called sjoin, which lets you take two shapefiles and create all the intersections around those shapefiles the fragments that I just showed you out here. So yeah, creating all these fragments is what the sjoin function does. So, if we do that, then what it's now done is created a new data frame called merged and that has all these shapes. Let's validate that. So there are 30 districts and 28 constituencies, but when you overlay them, it turns out that there are 147 fragments, each of which represents an intersection of a district and a constituency. Now, given this, it should be possible to just take any metric, like the percentage of Muslim voters or the number of Muslims, the size of the Muslim population, from the district data into the data that you have on the constituencies. But it turns out that it's a little trickier than that. So, you have to do a little more calculation and that's what's available in the reshaper library, you can take a look at the code, what it does is moves the metrics from one layer to another in a way that is seamless. Once we have this, the result is an Excel sheet that kind of looks like this. It has all the attributes from both layers. So, it says, for instance, that this particular assembly constituency is broken up into three regions, each of which maps to different districts. So some of it overlaps with more and some of it with shift saagar some of it with data and in fact, these are across different states and what is the area of each of these, along with a variety of other metrics that you can calculate and the proportion of area that is overlapping. Once you have this kind of data set, what can we do with it? So let's revert back to our story. What actually happened to the Muslim vote? Well, this is the constituency-wise Muslim voter population in Karnataka. This was used by the Hindu to publish an article around where exactly the bulk of the voters are concentrated. So, there is a chunk here, there's a chunk here, there's a chunk here. Now, what was happening at this particular point was there was a fight for an alliance. The AIMIM, which is a Muslim party whose name is very long, and I can't even say it fully. But they had won a number of seats in Telangana, and were looking to also participate in the Karnataka elections. They plan to contest in 60 seats. Now, to make sure that they get the Muslim vote, both JD(S) and the Congress were vying for an alliance with the party and in April 2018, AMIM decided that they will not be directly contesting in the elections, but instead would be supporting JD(S). Now, we have the results of the elections by constituency, we know the voter population by constituency. Let us see what happened to JD(S). Turns out that where there were more Muslim populations, JD(S) actually got lower votes. So you can see the net result of this election and the alliance. Congress, on the other hand, had a mildly higher vote share and where there was a significantly larger voter population. It turns out that BJP was the one that gained the most. Now while I'm moderately okay, at Python, I'm terrible at electoral analysis. So I have no idea what this means. Okay, I'll let you figure it out. The elections in Maharashtra and Haryana are also coming up and it turns out that Congress is aligned with AIMIM and, well, let's just leave it at that. So, what can we do with this? What kinds of datasets exist and what is the potential of being able to join data sets across two spaces? That's something that I'm pretty keen on. It turns out that in India, there are broadly three kinds of geographic hierarchies. There is a political boundary hierarchy, a postal boundary hierarchy and an administrative boundary hierarchy. By political boundary, I mean, the state parliamentary constituency, assembly constituency, going all the way down to polling booth. This has all of the results of the elections and one of the important aspects of this is that policies get made to a good extent at this level because the MPs and the MLS are focused on their respective constituencies. The second is a postal code boundary. There is a zone within which there is a sorting district within which there is a post office and there is a PIN code, there are about 110,000 of these in total. The third is the administrative boundary hierarchy. So there is a state there is a district, there's also something called a division, but we'll leave that aside, then there could be a sub district block or village, if it's a rural area, or it could be municipality zone and ward if it's a township. Now, this apart, there is one other way we can create our own hierarchies. But before that, in case you're looking for shapefiles, for many of these, the easiest way to get the shapefile for India is to search for “Datameet maps”. Datameet is a group that it's a discussion forum and there is a lot of active discussion on various kinds of maps, pretty much any kind of map, there's a decent chance that you'll find it on Datameet and if it's not there on Datameet, ask the people, they might be able to post something, and if not, it probably just doesn't exist. But you can also create your own boundaries. If you have a single location, you can look at the area that is closest to this particular location than any other location. So for example, if this were a network of, let's say, schools, then what is that vision that is closest to a particular school than any other school. So if I take this particular point as a school, then this red region represents all of those points which are closer to this school than any other school. This particular process is called Voronoi tessellation and is something that comes out of box with QGIS, it's something that you can create with the command line prompt again using the reshape or library, but what that means is that now you can take literally any point and convert that into a region and the potential for that is quite high. So if I look at the kinds of data sets that you can create with location boundaries, right, so there's… take all the hospitals, take all the schools, take all the bank branches, take all the petrol pumps take all the locations where crimes have been reported, take any address or take all the telephone towers, take all the locations where there are stores of a particular brand. All of these are datasets for which you can get an address and an address can be geocoded into a point. If it can be geocoded into a point, you can convert that into a region and for each of these, you naturally have some kind of data for schools, you know how many teachers or how many students that are for telecom towers, you know, which is the organization that runs that tower, potentially the telecom organization will know how many calls are flowing through it, if it's healthcare data, you know, how many facilities that hospital has, how many patients, how many doctors, all of these are data sets that can be added to that particular cell in your respective region. But what this means, therefore, is that if we take any of these data sets that which you can create from location boundaries, or that often already exist by administrative boundaries -- and this is a pretty powerful set as well. Census gives us demographic data, asset ownership, who owns laptops, internet connections, TV, cars, fridges, social and religious data, economic indicators, well, income, household indicators, is the house made of a mud roof brick roof, do you have a toilet in the house not have a toilet in the house, practically every government scheme is tracked this way. So how many people have benefited from the National Rural Employment Guarantee act? Banking data is reported this way health data is reported this way. So effectively, anything that the government runs is reported by administrative boundaries. Anything that the corporate sector runs, by and large, is reported by locations. So between these two, there is enormous potential. But there's also the fact of how decision making happens. Ultimately, political boundaries are owned, in some sense by an MP or an MLA. And, of course, there is also the associated IAS equivalents, who usually run it by administrative boundaries. So if I wanted somebody on the political side to make decisions, then I could take any of this data and put it on to the constituency boundaries. If I wanted an administrative official to make a decision, then I could take any of this data and put it on to a district. If I wanted a manager or a principal of a school, or the CEO of a hospital to make a decision, I could take all of that data and put it onto their geographic boundary. For example, one of the things that the Hindu again did was found that the Congress is doing much better in the agrarian areas and they did that by taking the census data, which had the percentage of farmers and mapping that on to the voter constituency regions. If we took, for example, census demographic data and school data, we can answer a question, where should we open new schools so that students don't have to travel far or where there is a reasonably equal distribution of students across schools? If we took economic indicators how well the country is growing versus bank branch data? Then we can answer questions like are the bank branches distributed? Based on population? That is, does every person roughly have equal access to the bank or based on wealth? Does every rupee have roughly equal access to the bank or if it's in between? How close is it from one to the other? We could find out whether increasing the district's wealth leads to more theft. So that means people get richer. So does that mean that it does that lead to increase in crime? Or does it lead to less theft? Because the people are richer, and they don't need to steal therefore? And these are data sets that are available and can be joined. Similarly, with health data, does poor health lead to an increase in the number of pharmacies that are set up in that region because the pharmacies can sell more. Vice versa, if you actually set up more pharmacies? And does that have a positive impact on the people's health in that particular region? Now, the reason these questions are trivial to ask, but nearly impossible to solve today is because merging the data across different kinds of layers of maps is non-trivial. But both conceptually and technologically is quite an easy exercise. What can we do to solve problems like this? Well, me personally, I'd love to see more of these hidden insights come out but there are a few things that you can do, literally right now. First, if you have an idea, take a look at these data sets, any of the data sets that you know and raise an issue on this particular repository and I’d invite all of you to share this with people. It'll be great to see what kinds of ideas can be solved using these problems and I’d like to crowd source this to a number of people on the administrative side, on the NGO side, and on the corporate side, to create a repository that says here are things that we can do. If you want to try solving one of these and discovering your own insight, to build your own portfolio to share some useful knowledge. Then start by finding a map. Like I said, Datameet is a good place where you can find them up. You can find the reshaper library on https://github.com/gramener/reshaper. The links are again, on https://github.com/gramener/pycon2019. This is the one link that you need to remember and if you find something, do share it on Twitter. Please tag me @sanand0, I'd love to share it at least with the media and get some people to understand the power of geospatial joins. If you want to contribute to the library right now it's in a terrible state. Or if you want to learn more, I'm planning to organize a series of workshops on geospatial joins, do drop me an email. My email ID is s.anand@gramener.com and I'll mail you the workshops.\ If nothing else, if you just enjoyed the talk and you've learned something about it, then tweet about it. The tags are #PyconIndia2019, my ID, @sanand0. More than anything else, I'd love to see some insights come out by joining data. Happy mapping! Question: So my question is, basically, I'm belonging to northeast part of India. So I'm from Assam. So what happened in terms of the documentation for this geographic data and so those are always kept in a sort of, you know, register. We call registers or something, so how we use that image processing and all like, to enable those things into a more of a like a public space? Answer: Okay. There are broadly three ways in which you can get this kind of data out. The first is beg, borrow, steal. Somebody in the government may have this data. So for example, if you go to the Survey of India, they sell these shape files. Of course, I've been trying to buy one of these shape files for the last six years now and have failed and I’ve tried it through the Prime Minister's office and I still failed. But it's actually easier to just walk over to the Survey of India office and give them a USB stick, and they'll give it. So, depending on how you approach it, it may prove relatively straightforward. On the other hand, sometimes the maps don't exist. So for example, most interesting anecdote was the former head of the postal College of Mysore was trying to create a postal map, the region of all of the PIN codes. It turns out that nobody knows what the region is that a PIN code covers. So he created that he uploaded that ISRO’s Bhuvan, and then after about a year and a half realize that people have permission to upload into ISRO but download from Bhuvan. So after one and a half years of putting all the data, the data is locked, it's not even there. So today, what is the best source of getting PIN code data? It turns out that what people did was took various locations, geocoded them, they said, this location is at that this particular PIN code, this location is at that this particular PIN code, let's draw a region around it using the concept of Voronoi polygons, and publish it. So the second possibility is to create such maps. The third possibility that you talked about, which is can we use image processing to detect it? Some features can be detected that way. So, for example, if you want to detect urban regions are constructed regions, that's possible using satellite photography, if you want to locate water bodies, and whether they are growing or shrinking, so for example, in Chennai, the Chembarambakkam lake actually drying up, that's something that you can draw a boundary around using image processing and that’s a straightforward method. But the thing is, I don't think a single method will work for a wide variety of data sets, which is why we have many of these. But the biggest lesson that I've learned is that 90% of the things that we want, somebody else has usually wanted, and has managed to get their inputs. So, I find that the most efficient ways to ask and Datameet is a pretty good place to ask if somebody already has his data. Question: Anand, thank you very much for your talk. I've got a question regarding shape files. I had the requirement of using the map of India a few times and I suddenly realized that our external boundaries in a lot of places are in dispute and the kind of shape files that we get are not matching with what politically we want our file boundaries to be. So is there any official place from where we can get these shape files because the only shape file which are available are those distorted shape files, and I finally had to change the shape files myself to use it. I couldn't find any official place from where to get the shape file. Answer: So, the official place is the Survey of India, which claims to sell these maps like I said, for the last five, six years now I've been trying to buy these maps it's actually not possible. But there are people who have succeeded and stock is being recorded, right? Okay. Let's just say that if you go to Datameet maps, you will get unofficial but correct maps. Question: So, since you are in the field, shouldn't we have a system of getting correct official maps? Isn’t there a process being put in place or something? Answer: I tried talking to a couple of people at the Prime Minister's office and suggested this. They put me on the phone with the Inspector General of Surveys or some such high ranking official who said yes, absolutely, connected me to some person, who connected me to some person, who connected me to some person, who is exactly the same person I talked to in the first place. So, I don't know. I'm sure there is a process. I don't know it well enough.

Maps, Delimitation, and Gerrymandering


I compiled classic comedy dialogues from Nagesh’s films into an interactive quiz. Challenge your knowledge of Tamil cinema by guessing the movie titles based on his iconic lines—just type them as they sound to see if you're right.

Nagesh Comedy Dialogues Quiz Here are dialogues from Nagesh 's films. Can you guess which movie they're from? Don’t worry about the spelling. Just spell it like it sounds, and the box will turn green. Comments Hema Subramanian 2 Jan 2021 9:37 pm: Didn't get 5 6 and 7

Nagesh Comedy Dialogues Quiz


Gift-giving feels easy only when the gift is a book or food; everything else exposes how little we know about other people and how often “safe” gifts miss the mark.

Picking gifts is hard What do you pick for someone you don't know well enough? I generally pick books. I know books well enough to match them to people's personalities. Even if they're not a book reader. (The risk is that they might have already read the book.) As for the kids, toys like the tiny tower diy playhouses for sale would bring them so much joy. The other safe item is food. Chocolates, dry fruits, etc. Everyone likes them. (Even if they're dieting, dry fruits and dark chocolates are fine.) Beyond that, it's a hard problem. I went through some popular gift choices. There are a few good ideas there, but very few safe bets. For example: A Tortoiseshell Face Mask Chain. What's that? A stress relief ball or perhaps other stress-relieving products from sites like CBD UK. Hmm... not a bad idea, actually. A notebook. Except that I stopped writing in 2001. A USB rechargeable lighter. But who do I know that smokes, whom I want to encourage? A purse. I've never bought one in my life. My wife never likes the ones I point to. So... A hot water flask. Again, not a bad idea, actually. ... and the lists go on. But most gifts I receive for my talks are promotional. Plaques or certificates. Some people put these up in their cabin. I don't have a cabin. Branded photo frames. I don't have photos either. Company-branded USB car USB charger socket. I don't drive much. Their book or brochure. I've either already read it or never will. If they're wrapped in bubble wrap, I throw away the gift and pop the bubble wrap with my daughter. So I just added a section to my talks page. "If you want to send gifts, I only accept dry fruits."

Picking gifts is hard


This short note claims that PowerPoint can do almost anything and points to custom shape design as one of its most underrated superpowers.

I've a theory. PowerPoint can do anything. (Excel too). Shape operations are especially powerful. In this tutorial, I create an address card icon and export it as SVG. I don't need Sketch / Illustrator for this. They're great. But no, thanks. I'm pretty good with PowerPoint 😄 Have you seen PowerPoint do anything that made you go "Wow! This is cool!"? Do share. I'd love to hear about it. http://www.s-anand.net/blog/designing-complex-shapes-in-powerpoint/ LinkedIn

Powerpoint can do anything


Reverse-engineering Minecraft's websocket interface turned a private obsession into a public guide for building bots and tools in Bedrock without official documentation.

Unofficial Guide to Programming Minecraft with Websockets. My daughter got me hooked on Minecraft. (If you see me staring intently at you during a Zoom call, I'm probably fighting a mob.) As it turns out, Minecraft is pretty programmable. You can send it messages using websockets that can place blocks, create mobs, get information, and more. None of this is officially documented, though. So I spent a few weeks reverse-engineering this and put together a guide to programming Minecraft. https://youtu.be/bjanNXXwQbo Here's the full article with source code and examples: http://www.s-anand.net/blog/programming-minecraft-with-websockets/ May you live in exciting worlds! LinkedIn

Programming minecraft with websockets linkedin


This hands-on tutorial reverse-engineers Minecraft Bedrock's websocket hooks to show how chat, commands, queues, and bots can be controlled programmatically from JavaScript or Python.

Programming Minecraft with Websockets Minecraft lets you connect to a websocket server when you're in a game. The server can receive and send any commands. This lets you build a bot that you can ... (well, I don't know what it can do, let's explore.) Minecraft has commands you can type on a chat window. For example, type / to start a command and type setblock 1 0 0 grass changes the block 1 north of you into grass. ( means relative to you. Coordinates are specified as X, Y and Z.) Minecraft grass block Note: These instructions were tested on Minecraft Bedrock 1.16. I haven't tested them on the Java Edition. Connect to Minecraft You can send any command to Minecraft from a websocket server. Let's use JavaScript for this. First, run npm install ws uuid. (We need ws for websockets and uuid to generate unique IDs.) Then create this mineserver1.js: On Minecraft > Settings > General > Profile, turn off the "Require Encrypted Websockets" setting. Run node mineserver1.js. Then type /connect localhost:3000 in a Minecraft chat window. You'll see 2 things: 1. MineCraft says "Connection established to server: ws://localhost:3000" 2. Node prints "Connected" Now, our program is connected to Minecraft, and can send/receive messages. Minecraft chat connect Notes: The Python equivalent is in mineserver1.py. Run python mineserver1.py. If you get an Uncaught Error: Cannot find module 'ws', make sure you ran npm install ws uuid. If you get an "Encrypted Session Required" error, make sure you turned off the "Require Encrypted Websockets" setting mentioned above. To disconnect, run /connect off Subscribe to chat messages Now let's listen to the players' chat. A connected websocket server can send a "subscribe" message to Minecraft saying it wants to "listen" to specific actions. For example, you can subscribe to "PlayerMessage". Whenever a player sents a chat message, Minecraft will notify the websocket client. Here's how to do that. Add this code in the wss.on('connection', socket => { ... }) function. Now, every time a player types something in the chat window, the socket will receive it. Add this code below the above code: This code parses all the messages it receives and prints them. This code in is mineserver2.js. Run node mineserver2.js. Then type /connect localhost:3000 in a Minecraft chat window. Then type a message (e.g. "alpha") in the chat window. You'll see a message like this in the console. Notes: The Python equivalent is in mineserver2.py. Run python mineserver2.py. The official Minecraft docs say that the MCWSS protocol is outdated. But it seems to work. The full list of things we can subscribe to is undocumented, but @jocopa3 has reverse-engineered a list of messages we can subscribe to, and they're somewhat meaningful. This Go package has code that explores the protocol further. This chat has more details. There's also an outdated list of JSON messages from @jocopa3. Here's a sample program that places a block in Minecraft Build structures using chat Let's create a pyramid of size 10 around us when we type pyramid 10 in the chat window. The first step is to check if the player sent a chat message like pyramid 10 (or another number). Add this code below the above code: If the user types "pyramid 3" on the chat window, drawpyramid(3) is called. In drawpyramid(), let's send commands to build a pyramid. To send a command, we need to create a JSON with the command (e.g. setblock 1 0 0 grass). Add this code below the above code: Let's write drawpyramid() to create a pyramid using glowstone by adding this code below the above code: This code in is mineserver3.js. Run node mineserver3.js. Then type /connect localhost:3000 in a Minecraft chat window. Then type pyramid 3 in the chat window. You'll be surrounded by a glowstone pyramid. Minecraft glowstone pyramid Notes: The Python equivalent is in mineserver3.py. Run python mineserver3.py. The "requestId" needs to be a UUID -- at least for block commands. I tried unique "requestId" values like 1, 2, 3 etc. That didn't work. Understand Minecraft's responses For every command you send, Minecraft sends a response. It's "header" looks like this: If the command is successful, the response has body.statusCode == 0. For example: If the command failed, the response has a negative body.statusCode. For example: To print these, add this to socket.on('message', ...): This code in is mineserver4.js. Run node mineserver4.js. Then type /connect localhost:3000 in a Minecraft chat window. Then type pyramid 3 in the chat window. You'll be surrounded by a glowstone pyramid, and the console will show every command response. Notes on common error messages: The block couldn't be placed (-2147352576): The same block was already at that location. Syntax error: Unexpected "xxx": at "0 9 -1 >>xxx Minecraft incomplete pyramid That's because Minecraft only allows up to 100 messages in its queue. On the 101st message, you get a Too many commands have been requested, wait for one to be done error. So let's modify send() to add to a queue and send in batches. We'll create two queues: In wss.on('connection', ...), when Minecraft completes a command, we'll remove it from the awaitedQueue. If the command has an error, we'll report it. Once we've processed Minecraft's response, we'll send pending messages from sendQueue, upto 100 and add them to the awaitedQueue. Finally, in function send(), instead of socket.send(JSON.stringify(msg)), we use sendQueue.push(msg) to add the message to the queue. This code in is mineserver5.js. Run node mineserver5.js. Then type /connect localhost:3000 in a Minecraft chat window. Then type pyramid 6 in the chat window. You'll be surrounded by a large glowstone pyramid. The console will print messages like setblock 0 6 0 glowstone The block couldn't be placed` because we're trying to place duplicate blocks. Minecraft glowstone pyramid Comments Rodland 8 Mar 2021 12:45 am:Thanks for a great tutorial.However when I try to implement the chat listener (server2.js) i get an JSON error from MineCraft (in node.js console):"kryptert økt kreves" = "Needs secure connection" Ready. On MineCraft chat, type /connect localhost:3000 Connected { body: { statusCode: -2147418107, statusMessage: 'Kryptert økt kreves' }, header: { messagePurpose: 'error', requestId: '3522bb53-5256-4030-b827-4b411ae72a6c', version: 1 } } S Anand 8 Mar 2021 4:07 pm: Under Settings > General > Profile, please turn off the "Require Encrypted Websockets" setting. I've added this to the tutorial. Thanks for flagging this! S Anand 2 Jan 2022 9:48 am: That's right. Looks like you resolved it and changed localhost to minecraft-server.warpedwartwars.repl.co 👍 WarpedWartWars 21 Jun 2022 6:48 am: Got it working on localhost, using the Python mineserver3.py, but the message protocol seems to have changed, to this: {'body': {'message': 'pyramid 32', 'receiver': '', 'sender': 'WarpedWartWars', 'type': 'chat'}, 'header': {'eventName': 'PlayerMessage', 'messagePurpose': 'event', 'version': 16842752}} WarpedWartWars 15 Dec 2021 8:27 am: I tried this, as it seemed to be the best guide I could find. I made a Replit Repl: https://replit.com/@WarpedWartWars/minecraft-server?v=1 and I ran it, then tried "/connect localhost:3000" in Minecraft Bedrock 1.18.2, and I got "Could not connect to server: ws://localhost:3000". I imagine I have to change "localhost" to something else. someGuy 23 Feb 2022 8:11 am: This will be deleted in couple of days but some of the reasons people are getting issues connecting is because of UWP loopback restrictions source: https://stackoverflow.com/questions/34589522/cant-see-localhost-from-uwp-app to resolve this problem for windows: https://docs.microsoft.com/en-us/minecraft/creator/documents/scriptdevelopertools. run this in system administrator command prompt: CheckNetIsolation.exe LoopbackExempt -a -p=S-1-15-2-1958404141-86561845-1752920682-3514627264-368642714-62675701-733520436

Programming Minecraft with Websockets


This is an appreciative nod to a beautifully structured data-visualization course in R designed specifically for artists and designers.

Arvind Venkatadri put together a #datavisualization course in #R titled "R for Artists and Designers"." Beautifully crafted. Brilliantly structured. I loved it! https://r-for-artists.netlify.app/ LinkedIn

R for artists


Geospatial AI helped the World Mosquito Program reduce release-planning time from weeks to hours by estimating population density from satellite imagery and targeting modified mosquitoes more precisely.

At PyCon Indonesia, I spoke about a project we worked on with the World Mosquito Program. The World Mosquito Program (WMP) modifies mosquitoes with a bacteria -- Wolbachia. This reduces their ability to carry deadly viruses. (It makes me perversely happy that we're infecting mosquitoes now 😉.) Modifying mosquitoes is an expensive process. With a limited set of “good mosquitoes”, it is critical to find the best release points that will help them replicate rapidly. But planning the release points took weeks of manual effort. It involved ground personnel going through several iterations. So our team took high-resolution satellite images, figured out the building density, estimated population density based on that, and generated a release plan. This model is 70% more accurate and reduced the time from 3 weeks to 2 hours. More details at the Gramener website. The slides for the talk are below. https://www.slideshare.net/gramener/saving-lives-with-geospatial-ai-pycon-indonesia-2020

Releasing modified mosquitoes precisely


This is a quick public request for crowdsourced data on what people actually paid for COVID-19 vaccination.

Could you spare a minute & share your COVID-19 vaccination cost, please? LinkedIn

Share covid 19 vaccination cost


This talk announcement argues that spatial joins are one of geospatial analysis's most underrated superpowers because they enable correlations between datasets that do not share boundaries naturally.

The most under-rated #geospatial capability is spatial joins, IMHO. It lets you draw correlations from #geodata. During the 2019 assembly elections in India, it allowed me to join voter data (by constituency) with religious population (by district) and show that BJP's vote share increased with Muslim population, while it reduced the AIMIM-allied JD(S) vote share. That's just the start. We can explore: Where to open new schools? Where to locate bank branches? Does wealth lead to more or less theft? Does poor health lead to more or less pharmacies? ... and hundreds of other random questions. My day job doesn't let me explore these much. I'd love to work with someone, though. Would anyone like to explore such correlations? Which one(s)? I'd love to support you. Talk video: https://youtu.be/ql3Kz8ZjhRg Transcript: http://www.s-anand.net/blog/maps-delimitation-and-gerrymandering/ LinkedIn

Spatial joins talk


A developer war story about technical debt doubles as a hiring post for tech leads who understand how today's shortcuts become tomorrow's humility.

Is your code built to last? Mohammed Niyas P shares a developer war story. His shortcut worked. It delivered working code. But 5 years later, when it failed, he didn't expect he'd still be around to clean it up. https://gramener.com/developerwarstories/this-tech-debt-made-me-humble Found this tale on technical debt interesting? Gramener is #hiring #developers for a technical lead role. See https://gramener.com/job/?id=90520 Nikhil Kabbin -- your story's up next 😉 LinkedIn

Technical debt niyas blog post


This embedded quiz turns famous Vadivelu punchlines into a movie-guessing game that tolerates phonetic spelling and rewards memory rather than orthography.

Vadivelu Comedy Dialogues Here are some famous funny dialogues by Tamil comedian Vadivelu . Can you guess which movie they’re from? Don’t worry about the spelling. Just spell it like it sounds, and the box will turn green.

Vadivelu Comedy Dialogues


This webinar invites developers and designers to learn how open-source web components can help build data-driven infographics with minimal code.

I'm running a #webinar for #developers & #designers on Thu 17 Jun. It's on creating data-driven infographics -- without much code. This is based on new open-source #webcomponents Gramener is releasing - which makes it easy to map SVG attributes to data. It'll be a light, fun workshop where YOU will build something. You can register at https://info.gramener.com/creating-data-driven-infographics-with-web-components Note: Webinar recording it at https://youtu.be/Np50SvvX8UY LinkedIn

Webinar for developers designers 2021


Ten years after imagining a modest company that might barely survive, the author reflects on Gramener crossing one meaningful threshold after another and still feeling both scared and grateful.

10 years later On 12 Jan 2012, on a flight back from London, I wrote: ... it was clear in my mind. I would be an entrepreneur. I would create a small company that would probably fold. Then I’d do it again. And again, 10 times, because 1 in 10 companies survive. And finally, I’d be running a small business that’d be called successful by virtue of having survived. A modest, achievable ambition that I had the courage for. 10 years later, Gramener successively crossed 10 employees, 10 clients, 10 years, $10 mn and is on its way to 10 offices. We just opened a new office at Hyderabad. I have the same request as 10 years ago. It's scary but exciting. Wish me luck!

10 years later


A single autocorrect from “habits” to “hobbits” becomes a reminder that language errors can create delight rather than just embarrassment.

I built an internal app that nudges people to respond to calendar invites in Gramener. A colleague replied, "... this is really helpful for improving hobbits". Autocorrect is delightfully serendipitous. I promptly watched all three #films in the Peter Jackson Hobbit series 🙂 LinkedIn

Autocorrect is serendipitous


A Pune hair salon named after Calvin and Hobbes becomes a delightful excuse to connect barbers, theology, fandom, and comic nostalgia.

Calvin & Hobbes Salon This is the Calvin & Hobbes Hair Salon in Hinjewadi, Pune. Calvin & Hobbes fans might remember Calvin's hairdresser, Pete, who certainly "knows which side HIS bread is buttered on." Maybe I too will go get "the top of my head shaved, and the sides dyed pink and cut in horizontal stripes" 😉 The signboard has a Tirupathi Namam drawn over the names of two famous theologians -- both of whom had great hair. That's a lot of hidden connections! I was thrilled to see that it's not the only Calvin & Hobbes salon in India. Google Maps shows almost a dozen ! 😲

Calvin & Hobbes Salon


This is a recruiting note for a senior advanced analytics role, aimed at alumni networks and people with both data-science and business-facing maturity.

Dear Alumni 🙂, A strategy consulting firm is looking for a senior person in advanced analytics at an associate partner level. Ideally someone who has come from data scientist route but has developed into business minded. If you're interested or know anyone who is, please mail me at s.anand@gramener.com. LinkedIn

Dear alumni


A colleague's remark about using every part of himself at Gramener becomes a short defense of generalists in an age that overvalues narrow specialization.

When Sunil Kardam said Gramener is the one place he got to apply all his skills - management, engineering, people skills and analytics - it reminded me how cross disciplinary Gramener is. There's something to be said for generalists in the era of deep focus. LinkedIn

Gramener applies your skills


Actor-network clustering shows that Bollywood is one of the world's more isolated film ecosystems, though not as isolated as some smaller national clusters.

How isolated is Bollywood from world cinema? These are the major group actors based on who they act with most. Actors mostly act with other actors in the same... 1. Language. Not country. For example, the Spanish / Mexican group is across countries. But Indian actors divide into North Indian and South Indian. It's language, not country. 2. Time period. Old American actors are a separate group from Hollywood. (Naturally. Brad Pitt was born after Humphrey Bogart died. They couldn't have acted together.) 3. Genre. Hollywood Porn actors don't act with mainstream Hollywood. Same with Japanese Porn, Hollywood TV, and Hollywood Horror actors. How are these groups themselves connected? Do Chinese actors act with Hollywood often? How isolated is Bollywood from world cinema? Hollywood is the core group Take groups that act with other groups at least 5% of the time. Mainstream Hollywood acts with British and Hollywood TV/Horror actors. All other clusters are isolated. Indian & Japanese clusters emerge Let's go more liberal. Take groups that act with other groups at least 2% of the time. Hollywood forms a big connected cluster. It includes most of Europe -- British, German, French, Czech, Yugoslavian & Italian actors. North & South Indian actors form the first non-Hollywood cross-language cluster. The Japanese and Japanese porn actors form a cluster too. (Interestingly, it's easy for a Japanese porn actor to act with mainstream Japanese actors. Hollywood porn actors find it far harder to act with Hollywood.) Among groups thatact with other groups at least 1% of the time, we have: Chinese & Korean cluster emerges Chinese & South Korean actors form the first cross-country cross-language cluster. Hollywood expands to act with Scandinavian, Spanish, Polish, Brazilian & Nigerian films. Other film industries (Russian, Greek, Egyptian -- even Hollywood Porn -- are still isolated.) World Cinema vs the rest Among groups that act with other groups at least 0.5% of the time, we have: 1. Turkish & Iranian groups coming together 2. Indonesian actors acting with the Chinese 3. Hollywood expanding to cover Russian, Greek, Egyptian, and finally, Hollywood Porn. (It's easier for Brazilian / Nigerian to act with Hollywood than to be a Hollywood Porn actor.) At this point, there are 6 actor groups that act with each other at least 1 out of 200 times (0.5%). 1. World Cinema (Hollywood & friends) 2. Japanese (mainstream & porn) 3. Indian (North & South) 4. Chinese, South Korean & Indonesian 5. Turkish & Iranian 6. Filipino One world of cinema If we look at groups that act with other groups at least 0.5% of the time, we have a far more unified picture. Almost every actor group acts with another group at least 1 out of 400 times. But even here, there's an exception. Filipino actors -- the most insular major actor group in the world. So, how isolated is Bollywood from World Cinema? For its size, it's one of the most isolated actor groups. (But not as much as Iranian/Turkish or Filipino.) Comments Dr Urmila 5 Jan 2022 7:36 pm: Hi Anand....i am not a technical person but I do use similar representations in my research as bibliometric, scientometrics etc...we use to understand the publication patterns... Any very interesting piece of information. I enjoyed reading How to find a Chinese actor to cast in Hollywood - S Anand 20 Feb 2022 12:39 pm (pingback): […] For example, Hollywood actors act outside Hollywood just 10% of the time. Chinese actors act with non-Chinese actors just 1% of the time. […]


Analyzing actor co-star networks suggests that the easiest Chinese actors for Hollywood producers to discover are the ones already sitting closest to Hollywood through repeated cross-industry collaborations.

How to find a Chinese actor to cast in Hollywood Film actors mostly act within their own industry. For example, Hollywood actors act outside Hollywood just 10% of the time. Chinese actors act with non-Chinese actors just 1% of the time. So, if you're a Hollywood producer trying to cast a Chinese actor, how would you find them? One way is to list Chinese actors with the largest number of Hollywood co-stars. Let's see who tops that list. 5. Pei-Pei Cheng You may know her as Jade Fox, the sly governess in Ang Lee's Crouching Tiger, Hidden Dragon (2000), or Golden Swallow, the skilled swordsman sister in Come Drink With Me (1966), or even as the voice of the matchmaker who disgraces Mulan in Mulan (2020). She mainly acts in Chinese films, co-starring nearly 180 times with actors like Hua Yueh, Lieh Lo, and Chung-Hsin Huang. But she's also co-starred over 20 times with Hollywood actors like Jamie King (of Sin City), Peter Bowles (of The Bank Job), and Sandra Oh (of Grey's Anatomy). 4. Jet Li You may know him as Han Sing, the martial artist and ex-cop in Romeo Must Die (2000), or Gabe Law, the former MultiVerse Authority agent in The One (2001), or Yin Yang, the unarmed member of The Expendables (2010). He has co-starred over 100 times with Chinese actors like Jackie Chan, Simon Yam, and Sammo Kam-Bo Hung. But he's also co-starred 30 times with Hollywood actors like Antonio Banderas, Morgan Freeman, and Sylvester Stallone. 3. Joan Chen She's famous as Wanrong, the Chinese empress in The Last Emperor (1987), Josie Packard, the owner of the Twin Peaks mill in Twin Peaks (1989), or Dr Ilsa Hayden, assistant to the villain Rico Dredd in Judge Dredd (1995). She's co-starred over 80 times with Chinese actors like Tony Chiu-Wai Leung, Leon Lai, and Tony Ka Fai Leung. But she's co-starred over 40 times with Hollywood actors like Michael Caine, Peter O'Toole, and Christopher Walken. 2. Jackie Chan The most famous Chinese martial arts actor in the world, and one of the highest-paid actors in the world, is famous as Detective Inspector Lee in Rush Hour (1998), Mr Han in The Karate Kid (2010), and the voice of Monkey in Kung Fu Panda (2008). He has co-starred nearly 200 times with Chinese actors like Sammo Kam-Bo Hung, Maggie Cheung, and Kent Cheng. But he's co-starred over 50 times with Hollywood actors like Arnold Schwarzenegger, Owen Wilson, and Chris Tucker. 1. Michelle Yeoh You may know her as Wai Lin, the Chinese spy and James Bond's ally in Tomorrow Never Dies (1997), Yu Shu Lien, the warrior swordswoman in Crouching Tiger, Hidden Dragon (2000), or as Eleanor Young, the domineering mother-in-law in Crazy Rich Asians (2018). She's an actress at the borderline of the Chinese - Hollywood clusters. She's acted 60 times with Chinese actors like Maggie Cheung, Chow Yun-Fat and Jet Li. But she's acted almost as many times with Hollywood actors like Sigourney Weaver, Zoe Saldana and Sam Worthington. More actors Here are half a dozen more Chinese actors that have acted with Hollywood actors often. Chow Yun-Fat Donnie Yen Andy Lau Simon Yam Gong Li Josie Ho It's interesting to see that 3 of the top 6 (Chow Yun-Fat, Pei-Pei Cheng, and Michelle Yeoh) had all acted in the blockbuster Crouching Tiger, Hidden Dragon (2000). So, perhaps the simple message to our Hollywood producer is to "look no further than the cast of the first foreign-language film to break the $100mn mark in the USA."


A simple email experiment found that friends open far more than strangers, top links beat lower links, and provocative subject lines are harder to design well than expected.

I tested the best ways to mail people I emailed My Year in 2021 to 2,700 people. It had 3 experiments. Do friends open my mail more than strangers? I split the list into 2 groups: 1. My contacts: 1,000 people I knew (I've mailed them) 2. Strangers: 1,700 people I didn't know (I've never mailed them) My guess: strangers would open the mail 30% less often. Reality: They opened it 40% less. 50% of my contacts opened the mail, vs only 28% of strangers. Are the first and last links most clicked? I sent 2 versions of the email to my contacts. The order of links was different. Book reviews did 30% worse at #6 than #1 (62% down to 43%) PyCon talk did 67% worse at #4 than #1 (10.4% down to 3.4%) Landmark course did 7% worse at #4 than #2 (20.7% down to 19.2%) write more about did 100% worse at #4 than #2 (4.4% down to 0%) Tools for Data Science did 39% better at #2 than #6 (17% down to 10%) Pranayama did 62% better at #5 than #3 (3.4% up to 5.5%) My guess: the first and last links would be clicked 20% more often than those in the middle. Reality: The links higher up were clicked more often 5/6 times. Click rates drop but don't climb up at the end. Do provocative subject lines increase open rates? I sent 2 versions of the email to my contacts. The subject lines were different. 1. Bland: "My year in 2021" 2. Provocative: "Where I failed in 2021" My guess: the provocative title will have 2X the open rate of the bland one. Reality: The open rates were about the same (49% for provocative, 51% for bland). Either the second title was not provocative enough, or the bland was interesting enough. I need to re-run this experiment. Learnings I learned 3 things. 1. Strangers open my mails less than I thought. Make more friends 🙂 2. People scan emails top-down (not top-down, then bottom-up - like I do) and click on top links. Move the main link first. 3. I'm no good at creating high-contrast variations in content. Take help.

I tested the best ways to mail people


A quick observation on the age profile of the author's music library turns into a two-week celebration of songs inherited from his parents' era.

I was surprised to see a big chunk of my #music library is from the 1960s -- thanks to the songs my parents grew up with, and passed on to me. In celebration of Music Day on 21 Jun, I'm treating myself to 2 weeks of songs from the #1960s. I've linked to my collection. Hope you discover some gems too! LinkedIn

I was surprised to see a big chunk of my music library is from the 1960s thanks


Color-coding calendar events by reach doubled the author's weekly impact while freeing ten hours of personal time, though the exercise also revealed how crude reach is as a proxy for effectiveness.

Increasing calendar effectiveness by 2X I took a 2022 goal to be 10X more effective. In Jan, I managed 2X. Here's how. What is effectiveness? I don't know. I'm figuring it out. But to start off, I measured the number of people my actions directly impact. For example: Discussing my Tools in Data Science Course or writing a blog post impacts 500 people. Mailing all Gramener employees impacts 200 people. Shopping with my wife impacts 2 people -- her and me (in very different ways). Clearly, the impact is not equal. But it's a start. How to measure it? Since Dec 1, I categorized all my Outlook calendar entries into one of these categories: Red is "low reach". Green is "high reach". This is what 6-10 Dec 2021 looked like: I continued this for 8 weeks. Did effectiveness increase? In Week 1, I reached 30 people on average. This was the control week. In Weeks 2-3, the reach increased from 30 to 77. In Weeks 4-8, it settled at 64. So, yes, effectiveness increased. in Jan 2022, I reached twice as many people per week as when I started off. I didn't measure quality/impact. One-on-one coaching has more impact than a lecture. Reach is just a crude first approximation for effectiveness. How did this happen? What gets measured, improves. I'd categorize each entry on my calendar. This enabled 3 things: 1. I'd try to remove low-reach ( = 50 reach - green) items. This increased from 12 to 18 hours a week. So, I now have 10 more hours of "me time" every week, while I still reach 2X as many people. What next? I'm exploring better measures of effectiveness. I believe: Effectiveness is goal alignment. It's personal, and purely a function of your priorities. Effectiveness is multipled by assets. Actions that create assets improve effectiveness. Once I discover a robust measure, I will to re-categorize my calendar and re-run this experiment. If you use a measure of effectiveness of impact, please let me know -- I'd love to learn from that. Comments My Year in 2022 - S Anand 31 Dec 2022 8:42 am (pingback): […] 10X more effective. I improved my calendar effectiveness 2X in Jan. But I realized this is actually efficiency. Not effectiveness. Maybe effectiveness […]

Increasing calendar effectiveness by 2X


Eleven hours of Microsoft Speaker Coach data exposed the author's filler words, pacing habits, and “maybe”-driven hypotheses, turning speaking analytics into a self-improvement tool.

Microsoft ported its PowerPoint Speaker Coach to Teams. Since September, it's given me suggestions covering 11 hours in 77 calls (I speak 10 min/call.) Learning to speak better I say "uhh" a lot. That's intentional I use the filler word "uhh" in 70% of my calls. That did not surprise me. I do that intentionally. 1. On a poor network, they know I'm still connected 2. They know I'm going to say something 3. I sound less confident. That invites critique I can learn from But I also use filler words like "You know" and "I mean" in half the calls, and "like", "actually", and "basically" in a fifth. That's NOT intentional, and I'll be conscious. | Filler words | % of calls | \# / call | | ------------ | ---------- | --------- | | uhh | 70% | 3.6 | | You know | 48% | 2.4 | | I mean | 43% | 2 | | like | 22% | 1.4 | | actually | 19% | 1 | | basically | 18% | 1.2 | | anyway | 14% | 1.1 | | hmm | 16% | 1.1 | | umm | 9% | 1.4 | | ah | 4% | 1.3 | I say "maybe" a lot. That's surprising What did surprise me was "maybe". I use it every fourth call, but when I do, I say "maybe" ten times per call. That's a lot of maybe! Sometimes, I say maybe because I'm communicating uncertainty. Maybe we'll have 20-30% success rate... So and I had to switch 3 laptops or maybe 4. ... then she said, "OK, maybe it's some other Sam" Sometimes I'm proposing tentatively. ... one of the reasons why I'm nudging towards that is maybe a large reuse initiative is high return, We can even put this in as part of the project by maybe offering it to different teams... Maybe by having dedicated support... Maybe I'll drop off. Bye But sometimes, it's testable hypotheses. Uh, maybe I'm getting the names wrong, but I think it was Socrates... Maybe it's me, but yeah, I guess... You know, maybe it's because I don't store any of my stuff in... One of my year's goals is to run 50 experiments. I'd been doing well until April, and then fizzled out. Partly motivation. Partly a lack of testable hypotheses. And now, in October, I discovered that I literally speak out one testable hypothesis every call -- roughly every 10 minutes I speak! I'm amazed at how blind I've been, and how easy it can be to find experiments to test. I guess I need more of a scientific mindset. (Or just plain curiosity.) The next time I say, "maybe" (or see it in my transcript), I'll write it down as a hypothesis to test. Repetitive words cluster Another discovery was: I tend to pick a phrase and use it repeatedly in calls. For example, I said "let's say" twelve times in just one call of 15 minutes. I said "main" 20 times over 2 calls of 8 minutes each. I said "cool" 7 times in an 11-minute call. | Repetitive word | \# calls | \# / call | | --------------- | -------- | --------- | | lets say | 1 | 12 | | main | 2 | 10 | | also | 1 | 8 | | only | 2 | 7.5 | | correct | 7 | 7.4 | | in terms of | 1 | 7 | | alright | 3 | 6.3 | | that is | 3 | 6 | | cool | 2 | 5 | Clearly it's something to watch out for. But maybe repetition of words isn't a bad thing if it's not the same phrase repeated across calls? (There! I said "maybe". Let me find out!) Modulate the pace In a third of my calls, I need to speed up. In a third of my calls, I need to slow down. (On some calls, I need to do both!) Clearly, I need to vary my pace a lot more, consciously. It's not that I talk fast or slow. I do both. But I get stuck in one mode of speaking for too long. Takeaways I used to think I was a pretty good speaker. That's not a bad thought, but it can blind me to feedback and improvements. There's no end to learning how to speak. Speaker Coach is a great "in-your-face" feedback mechanism. I hope Microsoft adds more features to it. But what I'm going to do now is: 1. Every time I say "maybe", write down an experiment 2. Speed up and slow down more in calls 3. Watch for words I use repeatedly

Learning to speak better


A life story told through geography explains how family, marriage, jobs, fear, and ambition repeatedly pulled the author toward or away from Chennai until the move to Singapore.

Moving to Singapore My family and I relocated to Singapore today. Most of my major life decisions have involved the distance from Chennai. In 1992, I wanted to study physics at IIT Kanpur or Kharagpur. My father erased the choices from my admission form and calmly said, "Tick anything in Chennai." I ticked everything except Chemical Engineering. Prof Kalyanakrishnan saw my rank, said "You'll get Chemical Engineering", and ticked it for me. No one heard me say, "But I don't like Chemical Engineering." In 1996, I got job offers from Ramco Systems, Chennai and IBM, Bangalore. I chose IBM partly because my mother said, "Move out of Chennai, else you'll live in your father's shadow." In 1999, I got offers from IIM Ahmedabad and Bangalore. I picked Bangalore. "You're declining the best IIM?" my couseller asked. But it was far from Chennai. In 1999, I lost a scholarship and was insecure during the internship interviews. I accepted my first offer (from Lehman Brothers), though it was in Tokyo. "Well, you've already accepted. All the best," my father said that evening with concern. In 2000, I declined Lehman Brothers' pre-placement offer. Tokyo's too far from Chennai. (60 days of Subway)'s Veggie Delight didn't help.) In 2001, my matrimonial profile mentioned just 2 things: "He likes curd rice and plans to settle in Chennai." My wife, Shobana, ignored it. She had other plans. By 2005, she convinced me to move to the US or Europe. London's physically and spiritually closer to Chennai than New York or Seattle. So I joined Infosys Consulting in London. By 2011, I'd had enough. After 2 months of careful planning, I walked home and told Shobana, "I lost my job. Please don't tell anyone. Let's go back to India." We left 2 weeks later to join Gramener. In 2019, I attended Landmark's Forum and told Shobana I'd tricked her. (She didn't speak to me for a day.) We decided to move again. Not the US or Dubai. Singapore's physically and spiritually closer to Chennai. She spoke to my father, who was OK to move too. COVID slowed things down (thankfully), but in 2022, my daughter would start Class 11. That's a logical time to shift. So as of 1 Aug 2022, we're in Singapore as a family. At least for 2 years, until my daughter starts college. After that, let's see. Comments Aman Khan 3 Aug 2022 10:48 am: You should have moved near MSD, nobody is more close to chennai than him as per common perception :-) Arun 27 Jan 2023 8:28 pm: Thank you for sharing this well-written article. Short & crisp. I enjoyed reading it. My Year in 2022 - S Anand 8 Jan 2023 10:17 am (pingback): […] I moved to Singapore in August. (Cycling and ComicCon were mind-blowing discoveries.) […] Somya Jain 3 Oct 2022 2:10 pm: Love the way you write sir The OLX UPI con - S Anand 2 Oct 2022 10:12 pm (pingback): […] moving to Singapore, we listed our furniture and white goods for sale on […]

Moving to Singapore


The author's 2021 review balances misses in weight loss and confrontation with wins in teaching, learning, self-improvement, and habit consistency, then sets 2022 goals around experiments, talks, and effectiveness.

My Year in 2021 In 2021, I made 3 resolutions. 1. Lose 10 kgs. I lost 5 kg in 3 months. But gained it back by the year-end. 2. Fail big. I practiced confronting people – and failed. I still run from fights. Even when important. 3. Calendar integrity. I stuck to my calendar 90% of the time. But personal commitments slipped. On learning, I discovered network clusters. My PyCon talk on movie networks is the start of a fascinating exploration of actors that I’ll write more about. On training, I designed a Tools for Data Science Course for IITM’s Bachelor’s in Data Science. I’m now a “faculty” at my alma mater, and no longer scared of it. On self-improvement, I completed a Landmark course and continued Pranayama. Both helped my resolutions. I also continued 2 habits from last year. 1. Walk 10,000 steps daily. I averaged 10,200. 2. Read 50 books. I read 52. Here are my reviews. (Which did you like? What would you recommend?) In 2022, I plan to: 1. Run 50 experiments. I’ll learn by disproving my beliefs with measurable tests. 2. Speak at 10 global forums on data stories, and spread the beauty of data. 3. Be 10X more effective. I’ll measure the impact and stop low-impact work. I'm curious -- what's ONE thing you'd like to do in 2022? Comments My Year in 2022 - S Anand 31 Dec 2022 8:42 am (pingback): […] integrity. Stick to my calendar. Especially the time I block for myself to work. (This is a 2021 habit I’ve slipped […] Learning to speak better - S Anand 17 Oct 2022 9:40 am (pingback): […] of my year’s goals is to run 50 experiments. I’d been doing well until April, and then fizzled out. Partly motivation. Partly a lack of […] Pratyush 2 Jan 2022 7:08 pm: I'd like to learn native mobile development. I want to try out Flutter, Kotlin and Swift. I have been developing websites for the last 15 years. But those skills suddenly feel so outdated. The new hardware provides so many exciting opportunities. I'd like to explore them. Increasing calendar effectiveness by 2X - S Anand 25 Jan 2022 1:31 pm (pingback): […] took a 2022 goal to be 10X more effective. In Jan, I managed 2X. Here’s […]

My Year in 2021


This year-end reflection measures mixed progress on experiments, speaking, effectiveness, and relocation while setting 2023 goals around experiments, environment design, and calendar discipline.

My Year in 2022 In 2022, I made 3 resolutions: 1. Run 50 experiments. I ran 20 until April (here are some), but stopped (for no reason). I'll continue. 2. Speak at 10 global forums. I delivered 10+ PyCon talks. They were pre-recorded, allowing me to scale. But recording videos and no feedback are boring. I'll explore how to scale enjoyably. 3. Be 10X more effective. I improved my calendar effectiveness 2X in Jan. But I realized this is actually efficiency. Not effectiveness. Maybe effectiveness shouldn't be optimized, but discovered. I'll continue to ponder. Milestones in 2022: 1. I completed 10 years at Gramener in February. 2. I moved to Singapore in August. (Cycling and ComicCon were mind-blowing discoveries.) 2 habits I continued from last year: 1. Walk 10,000 steps daily. 2. Read 50 books. In 2023, I plan to: 1. Run 50 experiments. I’ll learn by disproving my beliefs with measurable tests. 2. Make 1 change a month in my environment. 90% of our thoughts are shaped by our environment. (Read The Extended Mind.) So I'll use the remaining 10% to shape my environment. 3. Calendar integrity. Stick to my calendar. Especially the time I block for myself to work. (This is a 2021 habit I've slipped on.) I'm curious — what's ONE thing you'd like to do in 2023? Comments My Year in 2023 - S Anand 31 Dec 2023 7:22 am (pingback): […] For 2023, I made 3 resolutions: […] Picking books to read - S Anand 11 Jan 2023 8:50 am (pingback): […] 2023, I’m reshaping my environment. Picking books I usually won’t pick. (Read The Unknown Unknown: Bookshops and the Delight of […]

My Year in 2022


Tamil and Telugu films often fake newspapers, screens, and official documents so sloppily that reading the background text becomes a second layer of comedy.

Newspapers in Tamil movies Captain America: The Winter Soldier shows a newspaper with the headline "Howard and Maria Stark Die in Car Accident". The movie crew created a newspaper with a realistic font as well as text. It reads: Howard Star, founder of Stark Industries was confirmed dead this morning after suffering massive internal injuries from a fatal car crash. His Wife, Maria Stark, was also in the car ... Tamil film crews use a less-expensive approach. In Game - Oru Melliya Kodu (2016), I noted that the detective agreement is titled "Partnership Agreement: Mr Akshay and Ms Maaya". But the contents talk about CorelDRAW and CorelPhoto Content is central to any project. Whether it's the perfect font, clipart, photo, a previous project, or a client mockup, CorelDRAW Graphics Suite X5 helps you quickly access content so that you can focus on your design. Pity this movie crew didn't use CorelDRAW to create "perfect content". In Kee (2019), the Baasha virus has an IP address of 181.589.214.739.023. In Kolaigaran (2019), this newspaper article starts well: Hyderabad: Contract killer nabbed by Hyderabad police. A four member team executed the job. Kolavenkat, Chari team lead by Prabhakaran IPS. ... but the font changes mid-way, and so does the text. It goes on to say... This change of guard could be very important to Trump, almost 18 months into office, in the context of the fact that his political window of opportunity to put an enduring stamp on US foreign policy may soon narrow rapidly -- unless he wins a second term in 2020." It makes perfect sense that Prabhakaran IPS would influence Trump's global policy. I'm just noting the typographical and grammatical errors. I'm not sure if these errors are due to a lack of budget or attention to detail, but it certainly makes my movie-watching experience more fun! In Cobra (2022)), though, I did see an improvement. The Wikipedia page on "Dmitri Yugoslav" is a smart edit of the Wikipedia page on Sergei Shoigu. Even the phonetic pronunciation of his name "Дмитрий Югослав" is correct. On the other hand, the Telugu movie Liger (2022) sported this YouTube screen. The upvotes rapidly increase from 1,380 to 1,430 -- but the number of views is stuck at "10 000 views". The Warrior (2022) has a fax message with an order of suspension from the "Comissioner of Police" for the "Superintend of police" where "under ur regime as the superintend of police at kurnool havemisused your power ..." In Sarkaru Vaari Paata, "there are many variations of passages of Lorem Ipsum available, but the majority have suffered alteration in some form, by injected humour, or randomised words which don't look even slightly believable. If you are going to use..." -- which GitHub Copilot auto-completed without my having to prompt it!

Newspapers in Tamil movies


Tagging a thousand songs by year revealed an unexpected 1960s spike, highlighting how family influence and digitization shape personal music archives across decades.

Old songs in my music library My music library has around 1,000 songs (mostly Tamil and Hindi, with some Telugu and English film songs). I spent this morning tagging them by year with mp3tag. (Manually. You don't automate the pleasures of life.) I thought my 1990s collection would be the largest. I was in college, listening to lots of music then. But surprisingly, my collection has grown post the 1990s. I have 3 guesses why. 1. Recency bias. I re-built this collection recently. Maybe I forgot older songs? 2. Digitization bias. Maybe I listened to more songs as the cost of transmission/storage fell? 3. Worsening standards. Maybe I used to be choosier about music? Though I'm not sure of the above, there's another interesting anomaly. There is a spike in the 1960s. I don't need to guess this one. I know why. Those are the songs my parents liked. I grew up hearing them. The oldest song Tamil song is from Thiruneelakantar (1939). It's from my father's collection. I've heard it often enough to still enjoy it. The oldest Hindi song is from Jaal (1952)). He has a fondness for Dev Anand's songs. So do I. This one is a beauty. The oldest Tamil song my mother introduced me to is from Parasakthi (1952)). She used to dance to this song when young. The earliest Hindi song she introduced me to was from Jhanak Jhanak Payal Baaje (1955). It's the song I grew up on, and it's still among my favorites. What a melody! My wife prefers newer songs. But I have low standards and few preferences. It makes my life rather happy. So, in celebration of Make Music Day on 21 June, I'm treating myself to 2 weeks of my collection from the 1960s! PS: My full collection is at

Old songs in my music library


The office snack box becomes a tiny allegory of unfairness when the Oreos are buried beneath the Marie Gold biscuits and nobody digs for them.

I can't phrase this better than Sunil Sharma who said "As if the struggle in life is not enough, they put the Oreos so beneath th Marie Golds :-(" So true. Shame on those who ate the Oreos on top and didn't dig into the bottom, Gramener team! Work hard next time 🙂 "PS: Has anyone observed something like the "Oreos at the bottom" phenomenon in your office? LinkedIn

Oreos at the bottom


A comic mythological clip becomes a reflection on flattery as a real-world influence tactic, especially when asking experts to help with inconvenient problems.

Anand Madhav described this video as "Me, convincing tech lead to solve a problem on the weekend" 🙂" For those who don't understand Hindi, here's a rough translation. I've come for help. There's no one smarter at solving tricky problems. You're a wizard. You're strong AND smart. That's why I need your help. I've used this strategy when reaching out. Praise does help. But has it backfired on anyone? Any examples that you can share? https://x.com/samrad77/status/1569655565101539332 LinkedIn

Ravana asks for advice


Three attempted OLX scams show how UPI fraudsters exploit buyer psychology with fake QR flows, tiny trust-building payments, and aggressive confidence—until experience turns the mark into the interviewer.

The OLX UPI con Before moving to Singapore, we listed our furniture and white goods for sale on OLX. Caller 1: Vinod buys a bed On 18th April, I got a call from +91 70868 17420. He said he was Vinod, and owned a furniture shop in Ahmedabad. He wanted to buy our bed for Rs 6,000. It was not too bad an offer, so we agreed. Vinod: "I will pay by phone, sir. Give me your WhatsApp number." Me: (Wow! He's using WhatsApp Pay? That's a first.) "Sure. It's 9741 552 552." Vinod: "I've sent a picture, sir. Please scan that." Me: "Uh... OK. I got this QR code on WhatsApp. I scanned it on PayTM. But it's asking me to transfer to 9713902265.imb@icici. Is that right?" Vinod: "Yes sir. Please type 6,000 and send. You will get the amount." He sounded so confident, I thought I had it wrong. Me: "But this will pay you Rs 6,000. How will I get Rs 6,000?" Vinod: "Sir, it is a business account. I sent you Rs 6,000. But you will get it only if you enter 6,000." Me: "Vinod, I don't think payments work that way..." Vinod: "Sir, I will show you. I will pay you 1 rupee. You type 1 and send. You will get back 2 rupees." I didn't mind losing a rupee, so I paid his account 1 rupee. Vinod: "Sir, now please check your account. You will have 2 rupees." And I did have 2 UPI credits of 1 rupee each from his account. | Date | Transaction Remarks | Withdrawal | Deposit | | ----------- | ----------------------------------------------------------------------------------------------- | ---------- | ------- | | 18-Apr-2022 | UPI/210885173447/NA/BHARATPE0990555/Yes Bank Ltd/PTMF7DD556B1DFF42538B3F24D52AFD0A0F/ | 1.00 | | | 18-Apr-2022 | UPI/210868024141/UPI Payment/9713902265.imb@/Axis Bank Ltd./ICId74ef953e7bd4588bbcf226880cd4864 | | 1.00 | | 18-Apr-2022 | UPI/210868024950/UPI Payment/9713902265.imb@/Axis Bank Ltd./ICIb8cafc942184402dafda89620319a9e9 | | 1.00 | Me: "Sure, Vinod. You paid me 2 rupees. But when I pay you Rs 6,000, there's no guarantee you'll pay me Rs 12,000. Why don't you come in person, collect the sofa, and pay me then?" Vinod: "What sort of man are you? I already sent you money from my account. I cannot get it back now. I will have a loss. You tell me -- how will you give me back my money?" Me: (Um... He really does sound convincing.) "Vinod, I'm really not comfortable with this. Let's not go ahead." Vinod: "You're an asshole. Get off the line!" Caller #2: Gaurav buys a recliner On 15th May, I got a call from Gaurav at +91 84865 47620. Gaurav: "Sir, you have a sofa set on OLX for Rs 60,000. Can you give it for a lower amount?" Me: "Well, I'm OK with Rs 55,000, but not less." Gaurav: "OK sir. I will pay by UPI. Can I make a 50% advance payment, then you can take off the listing from OLX?" Me: "OK, sure." Gaurav: "What is your UPI account?" Me: "9741552552@upi." Gaurav: "I have paid Rs 27,500. I have sent you a picture. Can you scan it and enter 27,500?" Me: "I got an image that requests a payment to bharatpe09905995207@yesbank.ltd. But that's for ME to transfer to YOU. I have not received your payment." Gaurav: "I have already paid the advance sir. If you type 27,500 and send, you will get it." Me: "Sorry, Gaurav, I'm not going to do that. Bye." (click) By now, I was a bit wiser. Caller #3: Priti buys a recliner On 15th May, I got a call from yet another person. I forget his name. It might have been Priti. He: "Sir, I want to buy your recliner. How much is it for?" Me: "How will you pay and how much, please?" He: "Sir, I will pay on UPI. I can pay up to 55,000, sir. I will transfer the money and send you the confirmation code, sir" I let out a huge laugh, and continued laughing for a good 5 seconds. He: "Sir, what happened, sir?" Me: "Oh, this is so hilarious. How much money do you make in a month with scams?" He: "Sir, what sir?" Me: "I'd love to interview you for my blog. Do you change SIM cards often? Do you use channels other than OLX? What do you do in case someone files a cyber-crime complaint?" He: (click) Cyber-crime reporting This con is well-documented on OLX. I decided to file an online cyber-complaint at https://cybercrime.gov.in/. The process was not too painful. This is what the form asked for. Category of complaint Online and Social Media Related Crime Online Financial Fraud Business Email Compromise/Email Takeover Debit/Credit Card Fraud/Sim Swap Fraud Demat/Depository Fraud E-Wallet Related Fraud Fraud Call/Vishing Internet Banking Related Fraud UPI Fraud Hacking / Damage to computer,computer system etc Online Cyber Trafficking Online Gambling / Betting Ransomware Cryptocurrency Crime Cyber Terrorism Any Other Cyber Crime Have you lost money? Yes/No Approximate date & time of Incident/receiving/viewing of content: 18 Apr 2022 18:10 Is there any delay in reporting? No Where did the incident occur? Email Facebook Instagram Snapchat Twitter WhatsApp Website Url Youtube LinkedIn Telegram Other Other Media: OLX Supporting Evidence (Upload Media/Image/Pdf): Uploaded Please provide any additional information about the incident: ... My thanks to Vinod, Gaurav, and the third friend. I filed my first police complaint 🙂 How the police responded is another story... Comments Sai 15 Mar 2023 9:37 am: Haha. Thanks for sharing this. I went through exactly the same thing when I was relocating to Sgp too quite recently. Pratyush 14 Oct 2022 8:04 pm: Hey, thanks a lot for sharing this. Please also share how the police responded. We too filed a police complained but never saw any action.

The OLX UPI con


This brief note marks Gramener's new Hyderabad office as one more milestone in a decade-long startup journey that once aimed only to survive.

We inaugurated a new office in #Hyderabad. 10 years ago, I wrote on my blog: "I would create a small company that would probably fold. Then I’d do it again. And again, 10 times, because 1 in 10 companies survive. And finally, I’d be running a small business that’d be called successful by virtue of having survived. A modest, achievable ambition that I had the courage for. Naveen Gattu, Mayank Kapur, Ganes Kesari, J. Ramachandran, Ravinder Ireddy, Vengatesh CR. -- we've come a long way 😃 Gramener successively crossed 10 employees, 10 clients, 10 years, $10 mn and is on its way to 10 offices. I have the same request as 10 years ago. It's scary but exciting. Please wish me luck! LinkedIn

We inaugurated a new office in hyderabad


The author crowdsources demand before building a self-paced course, contrasting tools-for-non-programmers with data-storytelling-for-analysts as two plausible directions.

I'm planning to publish a 3-hour self-paced #onlinecourse. But I don't know which topic would be more useful. One topic is #datascience tools for non-programmers. Another is a step-by-step guide to #datastorytelling for analysts. What's more useful for you? Could you share with people, so I work on the more useful course? (Thanks 🙏) LinkedIn

What online course to publish