I discovered thousands of automated SSH brute-force attempts on my Amazon EC2 instance, arriving in rhythmic batches from a Chinese IP. It prompted me to reflect on my own history with web scraping and the importance of monitoring server logs.

I’m not that difficult to scare, and this log message certainly didn’t help: That’s the message I saw – one thousand five hundred and seventy times yesterday in /var/log/auth.log on one of my Amazon EC2 instances. Someone, presumably from China, has been patiently trying out a variety of SSH keys to log into this system. These were grouped as batches. There were exactly 314 attempts at 8am yesterday, then 314 at 12noon, then 314 at 4pm, then 314 at 8pm, then 232 at 3am today. (All times are in UTC – that is, UK time without daylight saving). Every burst took 9 minutes to run through all 314 attempts. The worst part was, when I tried using SSH this morning, I wasn’t able to log in. (It turned out that I had made a configuration error, but this is the sort of thing that gets me quite worried.) Perhaps I shouldn’t be complaining. I’ve written enough scrapers to make most webmasters cringe at their logs. I remember a few years ago, when I was working on a project at Tesco, and was scraping bestsellers lists from most sites. (Here’s a blog post about it.) We were putting together a prototype to see how real-time competitive pricing could help. The scraper was a pretty mild one. It would visit a hundred links, roughly at the pace of one a second. No images were loaded, of course, just the HTML. One fine day, a few weeks after this had started, I got a call from Andy. “Hi Anand, are you running any scrapers on our books website?” “Yes, why?” “Oh! The site’s very slow. Could you shut it down immediately?” Turns out that not a single page on the site loaded, and it had almost crawled to a halt. Now, obviously, my little 100-page script could hardly cause damage, but it’s easy to understand their reactions. No unauthorised scraping! After a few days of trying to figure out what the problem was, they increased the memory and things went back to normal. Not a bad solution, actually – throw hardware at the problem, and if it vanishes, it’s probably the cheapest solution. But anyway, I’m sure it’s some nice chap who’s just curious to know what I’ve got on my servers. I’d be happy to share some of it. And even if it’s not so nice a chap, there’s little that I can do, is there? Update (1pm India, 3rd June): Actually, I now realise that this has been happening ever four hours since May 29th, as regular as a clockwork. Wish I knew enough UNIX programming to pull a prank... Comments Rajeev 23 Dec 2013 2:54 pm: The unwanted trial to login and break in very common on the blogs and the websites that provides a login account. I had many such problems initially and I block such IP's every time once I notice there is something going on which is not expected. I loved your blog. Thanks Rajeev Amit Chakradeo 3 Jun 2013 8:38 pm: That is not necessarily an actual break-in attempt. SSH prints that if the reverse name lookup on the ip address does not resolve back to the same IP address. Some pointers to address these things: 1. Run ssh on non-standard port. (this kills 90% of people trying dictionary passwords out for root). 2. Prevent root login for ssh and disable password authentication (use keys only) 3. Run denyhosts which denies known bad IP's and hosts trying to exploit your box. Gopal 3 Jun 2013 8:06 am: Now you know why US is getting angry with china! :-) Anyways as you know it might be to put in some malware for DDOS bot attacks? Thejesh GN 4 Jun 2013 10:23 am: I use DenyHosts (http://denyhosts.sourceforge.net/) on all my servers. It blacklists ip addresses trying to brute force. Its easy to setup and run. I can see that the script blacklists at least one IP everyday.

The scary Internet


I shared slides from my series of talks on network visualization and relational data. These materials, including my presentation at The Fifth Elephant, explore the technical and aesthetic challenges of mapping complex connections and visual structures.

Some slides from my talks on visualising networks. (These are part of a series of talks I’m giving at a number of forums; the one at The Fifth Elephant is open to public.) visualising-networks-2

Visualising networks


I explain my utilitarian approach to life, prioritizing functional value and productivity over status or luxury. By choosing public transport and basic tools, I focus on practical utility where I don't see the value in prestige or fashion.

A couple of years ago, my HTC Explorer’s screen died. I bought a Micromax A50. This triggered a series of reactions prompting this post. I have many defects. Like most men, I can’t tell colours apart – like the difference between pink and purple – and am constantly corrected by my six-year-old. I can’t hear two people at the same time – or even in-between each other. I can’t find things outside of my narrow field of vision. I can’t recognise faces, and need at least three one-on-one interactions before I place people. (If you ask me “Do you recognise me?” and I say “Yes, of course!”, I’m usually lying.) I can’t place voices on the phone. My memory is terrible – my wife’s learnt to make me write errands on my laptop. I cannot identify cars – in fact, I couldn’t drive until recently. I also lack a fashion sense, despite being a keen student of design. I can understand rules of thumb, like how large line heights should be, or why high saturation colours are jarring. I can even give passable judgement on the quality of clothing. The trouble is, I don’t see much value in it. I’m a utilitarian. This post is an apology from all utilitarians like me. We’re sorry – we just don’t see the the point of a Mont Blanc pen or a Cartier watch. Our Bic pens and digital watches work just as well. We’re not saying you shouldn’t buy them. It’s just that we don’t understand why you would. This is not an argument against expensive items. I bought the iPod and loved it. Same with the iPhone 4. I have two iPads. I’m fairly picky about the earphones I buy. The thing is, the reason I buy these is because there’s a value that matches the price. Where I don’t see the value, that’s just throwing money away. So that’s why I travel in buses or autos. I can work on my laptop while someone else drives. That’s why I walk or climb stairs. I get to lose weight without wasting time at a gym. That’s why I don’t wear a watch and don’t subscribe to newspapers or TV. For my non-utilitarian friends out there, this is from us utilitarians. Please forgive us. We don’t see the value. Comments What did I learn from Vinod, Anand & Pinal’s session at GIDS | Some place else 25 Apr 2014 4:02 pm (pingback): […] Anand on other hand is the calmest speaker I have ever seen. He brings out data driven stories which highlight issues, evoke emotions like surprise, laughter. Wish One day I can bring that finish to my delivery. […] Somnath Manna 13 Apr 2014 4:59 pm: +++1 VFM = Utilitarian and am happy being that. Instead of BIC its Reynolds pen, MicroMax vs Galaxy Y for me though I subscribe to newspaper and DTH (otherwise there will be revolt at home :D) Shiva 23 Jun 2015 10:08 pm: What is the need to apologise? Or am I missing the sarcasm? :)

A utilitarian’s apology


I share my most frequent websites from A to Z using Chrome’s address bar suggestions. My browsing reveals a mix of data tools like Pandas and Underscore.js, development resources, open data projects, and my penchant for Telugu movies.

When you start typing in the address bar, Chrome suggests a link to visit, based on frecency. What do my recommendations look like? A is for airtel.in/smartbyte-s/page.html – the page where you can check your bandwidth usage. I used to check it infrequently until I upgraded to a 125GB connection. Now I check it every few days and feel miserable that I’ve nowhere near used up my quota. This has coerced me to watch many Telugu movies, of which I don’t understand a word. B is for blog.gramener.com – I blog there on data stories. The last month or so has been fairly active thanks to the elections. C is for calendar.google.com – which has become primarily a shared calendar. It was always indispensible to manage my time. Now it helps my colleagues pick when to call me. Right now, my calendar has events booked about two months in advance. D is for docs.google.com – for effectively one single purpose: shared spreadsheets. This is such a common and powerful use case, and I’m surprised it hasn’t become much easier to use. E is for epaper.timesofindia.com – some of our content has been published by The Economic Times, and I keep doing ego-searches in the print edition. But close behind is eci.nic.in which I’ve been scraping a lot, and election-results.ibnlive.in.com which we created for CNN-IBN. F is for flipkart.com – not facebook.com. I’m not often on Facebook. G is for gramener.com. Naturally. (It’s not surprising that it’s not google.com: I search directly from the address bar.) H is for handsontable.com – a library that I’ve been using a lot recently, followed by html5please.com that tells me which HTML5 features are ready for use. I is for ibn.gramener.com – another property we created, but it only just beats irctc.co.in. J is for join.me – a clean way to share your screen without the audience having to install anything (though you the sharer do have to install the software.) K is for kraken.io – an amazingly efficient image compressor. As you might have guessed, I lead a strange life. L is for learn.gramener.com – our Intranet. Sorry, you can’t access this one. M is for mail.google.com. I’ll probably be moving away from gmail as a backend this weekend to Mail-in-a-box, though. Google’s pulling the plug on Google Reader has shaken my faith. N is for news.ycombinator.com. When I’m bored and want to watch something while I have dinner, I don’t open YouTube. I open Hacker News. O is for odc.datameet.org – the Open Data Camp. I’m quite into open data. P is for pay.airtel.com, but if you ignore the number of bills I pay, it would be pandas.pydata.org, the home page of a remarkable data processing library. Q is for quirksmode.org, PPK’s remarkable browser-compatibility guide R is for reader.s-anand.net, my self-hosted RSS reader. It used to be reader.google.com, but Google let me down there. S is for s-anand.net – this blog. T is for twitter.com. Unlike Facebook, I don’t dislike Twitter so much. U is for underscorejs.org. Clearly I need to get a life. V is for visualizing.org. They have a number of interesting data visualisations. W is for webpagetest.org – it helps measure the speed of web pages. X is for xem.github.io. I’ve probably visited this page once, but it’s the only one in my recent history that starts with X Y is for youtube.com. I lied. I spend an order of magnitude more time watching Telugu movies on YouTube than on Hacker News. Z is for zoemob.com. Again, a page I visited only once, but there’s nothing else in Z at the moment. Comments Software I currently use | s-anand.net 9 May 2014 6:24 pm (pingback): […] course, some of my apps apps have moved online, and my earlier post on the A-Z of my browsing history covers that. But there are a few applications that I’ve hosted which I must talk about. […] chandigarh 13 Oct 2015 7:27 pm: you can delete your web search history through link https://history.google.com/history

A-Z of my browsing history


I rode in an ambulance through Bangalore and noticed that while most drivers remarkably give way, delays usually happen because people are distracted by headphones, panic, or don't know how to yield safely without crashing.

I rode in an ambulance yesterday. I've been in a few of these in the last 3 weeks, but yesterday was the first time I had to give directions, so I was paying a bit more attention to the traffic. It's remarkable how well Bangalore traffic responds to ambulances. Almost every single person gave way. (Not that this is easy. Merely slowing towards the left isn't always effective if it ends up blocking the way. Many people were wise enough to give way at the appropriate place, and our flow was not impeded.) Not everyone gave way, though. The first was the driver of a small car. (I can't identify vehicle models. This sort of looked like a Maruti.) He was driving right in front of us for a while, right in the middle of the road, without giving way on either side. After some time, when the road widened, we passed him. I noticed that he was wearing earphones. Based on his body posture, my guess was that he was listening to music -- as opposed to speaking on the phone, for example. Clearly, he had not heard the siren nor the horn. The second was a young girl riding a moped. (Scooty, or something like that.) The road was narrow. There was no chance of overtaking her. She did hear the siren, though, and tried her best to rush ahead of the ambulance. After about half a minute, the road widened again, and she gave way. My guess is that she was under 18, out on a ride in a relatively small and safe road, and had no idea what to do when an ambulance scares her noisily from behind. The third was a bus driver, though the circumstances were different. The car ahead of us (a call taxi) gave way to the left. The bus was approaching us from the opposite side. The bus stopped right next to the car. Given the width of the road, the ambulance could not pass. The taxi driver tried to guide the bus driver, telling him that he should move a bit forward to let us pass, but the bus driver (again, a very young chap) seemed frozen. He didn't (or couldn't budge). The taxi driver in front of us started his car, moved 50 metres ahead, and let us pass. The fourth caused the longest delay. A couple riding a bike were ahead of us. The road turned ahead. They tried to give way, skidded, and they fell right in the middle of the road. The were shaken, but not hurt, thankfully. It took a couple of minutes for them to gather themselves and their vehicle (with assistance from a passerby) and give way. We checked after confirming that they were unhurt. I'm not sure what to make of this. In every case, the cause for delay ways ignorance. Either not hearing, or not knowing what to do, or not knowing how to do it well. But it's gladdening that the bulk of Bangalore is both knowledgeable and responsive to the needs of those in ambulances. But: please don't endanger yourselves while giving way. Comments Uday Madpathy 9 Oct 2014 1:46 pm: This is good.. Have seen the opposite in other cities like Hyderabad.. Manoj Gaddam 9 Nov 2014 3:59 am: I am glad that people in Bangalore do understand "what is emergency". But how to change the behaviour or habit of the driver in the Maruti car. Good post! Sathya 13 Jan 2015 7:29 am: Was linked this posted in the comments of my latest blog post http://sathyabh.at/2015/01/07/ambulance-and-bangalore-traffic/ I think now the situation has changed that people are willing to give way to Ambulance, but simply are clueless about how. Do they slow down, speed up, keep to the current side, move to the right, move to the left? Our Traffic Regulations mention Each driver shall on the approach of a fire service vehicle or an ambulance allow it free passage by drawing it to the side of the road so I guess it means that we're supposed to move over to the left side of the road. But with our current clogged roads, this is near impossible. So people try to improvise and give way however they can.

An ambulance ride


I've curated a selection of Calvin and Hobbes comic strips that illustrate why I feel so much sympathy for Miss Wormwood. These panels highlight the exhaustion and patience required to handle Calvin’s disruptive imagination and relentless classroom antics.

It's hard not to feel sorry for Miss Wormwood. Comments Suraj 13 Feb 2016 4:34 am: I pity his babysitter Rosalyn more!

Poor Miss Wormwood


I share my go-to Windows software toolkit, covering essentials like Sublime Text 3 and Console2 alongside specialized data science tools like Anaconda, RStudio, and Open Refine for analysis and visualization.

Every few years, I review the software I use. Here are some of my earlier lists. Right now, among browsers, Chrome is my primary browser. What’s interesting is that IE 11 has overtaken Firefox in terms of usage. That’s partly because we’re working with Microsoft a lot, but also because Firefox has a number of weird bugs like IE6 used to have, and is slowly lagging in the race. Next to browsers, I spend most of my time on the command prompt. I use Console2 for tabbed console windows. Given the number of command prompts I open, this is often necessary. I use bash in Cygwin as the default shell. Haven’t had the need for PowerShell. The only text editor I use is Sublime Text 3. This is the only text editor I’ve used for the last 3 years. The only plugin I use is PlainTasks which I use as my todo list. I write my blog posts in Windows Live Writer, which makes blogging offline quite painless. For image editing, I use PicPick to capture screens and basic editing. Since I haven’t upgraded to Windows 8, I don’t have the snipping tool. But PicPick also lets me pick colors from the screen, which is pretty useful when copying designs. For slightly more serious editing like changing colours, adding annotations, etc., I use Paint.NET. It’s close enough to Photoshop for most practical uses. On rare occasions, I’ve needed to power of GIMP – especially to remove background on images. But when even this fails me, it’s ImageMagick to the rescue, with inscrutable command line options that can morph Obama into Osama. If I want to edit icon files (to create favicons, for example), I use IcoFX. For vector graphics, I use InkScape, which has a steep learning curve but doesn’t seem to have a good free alternative. To edit shapefiles, I use QGIS, and Shape Viewer to view them. For music and movies, I’ve kept it simple: I use VLC. It lets me stream on to my iPad. I can also watch/stream movies as they are being downloaded via μTorrent – which is probably the coolest feature feature of any torrent client. I store all my music in one large folder, and keep .m3u playlists. These are rsynced periodically into my Android phone. For audio editing, Audacity remains my best bet. However, for video editing, my needs have changed. It’s usually screen-recordings I need to create, so I don’t use VirtualDub much. I’ve moved from CamStudio to Microsoft Expression Encoder Screen Capture (long name for a rather nice piece of software that works reasonably well.) To read books, I’ve started using Calibre, simply because it can read both ePub and .mobi formats. Since then, I’ve been using Kindle less. I continue to use my old copy of Microsoft Reader, even though the product is dead, because I have a lot of .lit files. (That’s one of the advantages of software over online services. Even if they pull the plug, you can use an old copy of the software. And it works!) To read PDF files, I use Foxit PDF Reader. On the extremely rare occasion that I need to print PDF files from a software that does not support PDF printing, I use CutePDF Writer. For file sharing, I use Dropbox for files. It’s simple, popular and just works. I tried BitTorrent Sync as a peer-to-peer alternative to Dropbox, but the interface has a long way to go before it’s usable. I do hope something emerges. For screen sharing, I use TeamViewer (which is fast) or join.me (which doesn’t require a client). Though I use Skype for calls, I don’t find its screen-sharing fast enough. I play around with data a lot. This is mostly done in Python, for which I use Continuum’s Anaconda builds – they have most of the useful packages built-in. When I need to scrape Javascript-based websites, I try CasperJS on top of PhantomJS. This is particularly handy for the several ASPX based Government websites. I also have node.js installed, but don’t really use it much. I use RStudio as my R IDE. I’m experimenting with Tabula to see if it’s practical to extract PDF tables with it, but my current preference is to use xpdf to convert PDF to text and then process it. For data cleansing, there’s only one tool that I know that’s effective: Open Refine. For network visualisations, I use Gephi, though NodeXL can do a small but useful subset of that within Excel. For compression, I use 7-zip. The 7z format provides the best compression across most file types that I’ve seen, but even if you want to use ZIP files, 7-zip creates smaller ZIP files. For image compression, I use kraken.io, which offers the best compression I’ve seen. On the desktop, TruePNG and jpegoptim do the trick. There are several small utilities I use. WinDirStat tells me how my hard disk space is used up, helping clean drives and Dropbox folders. ClipX lets you copy and keep multiple items in the clipboard. Restoration can undelete even permanently deleted files. Truecrypt keeps files encrypted. Putty lets you connect via SSH if you don’t have cygwin. But the mother of all tools is AutoHotkey, which I use for everything ranging from typing my signature to resizing windows to storing our conference bridge numbers. I’ve a number of web servers on my system. I use XAMPP for Apache, MySQL and PHP, but also have nginx handy. But the simplest, easiest and smallest web server is perhaps Mongoose. Just run it in any folder to start a web server. python -mSimpleHTTPServer does the same for developers. I also have Fiddler installed as a proxy – partly to monitor what URLs my applications access, and partly to simulate slow speed connections for the web apps I build. Apart from MySQL in XAMPP, I have a few databases installed: SQL Server, SQLite and SQLite Studio to read the sqlite3 files. Of course, some of my apps apps have moved online, and my earlier post on the A-Z of my browsing history covers that. But there are a few applications that I’ve hosted which I must talk about. WordPress, which this blog runs on, is the primary one on the list. I also use gitlab as an internal alternative to Github, slideshare.net to share slides, and etherpad.mozilla.org to chat / collaborate on code. But the application that I spend the most time on is selfoss – an RSS reader, my replacement for the late beloved Google Reader. Comments Pravin 10 May 2014 8:04 pm: Try Sumatra (http://en.wikipedia.org/wiki/Sumatra\PDF), it is a reader that supports PDF, EPUB, MOBI, i suspect its support for .lit files but you can give it a try as it is very lite on the system(It opens my Excel 2013 Bible e-book in less time compared to Foxit PDF Reader). Jason 7 Sep 2014 1:59 am: Hey what happened to mDesktop? ;) Nipun 10 Jul 2014 11:02 pm: Get a mac, this list will be much smaller then ;) Vang Lian 28 Aug 2014 8:52 am: I use notepad++ as editor. Now sublime text looks very interesting. Thanks for the recommendation Raghavendra 14 Apr 2016 5:39 pm: Hey Ananad, hope you remember me. I know only one data scientist so far n its you. Happy to read you blogs. Khair 30 Mar 2016 5:29 pm: Since I haven’t upgraded to Windows 8, I don’t have the snipping tool Snipping Tool is available on Windows 7 as well, I just assumed you are on Windows OS ! :( Vigneshraja 7 Jun 2017 7:34 am: Try Canopy for python rpv 8 Aug 2017 1:46 am: What is the ereader of your choice if not Kindle?

Software I currently use


I explored a curated list of generative AI companies currently disrupting traditional SaaS incumbents in the martech space, focusing on how emerging AI tools are reshaping established software markets and categories.

This week, I learned: List of Gen AI companies disrupting SaaS incumbents: LinkedIn

Things I Learned - 16 Nov 2014


I’m moving away from rapid dieting toward a slow, sustainable weight loss plan. My approach focuses on losing 1kg monthly through mindful eating, stocking healthy foods, and incorporating simple habits like taking the stairs instead of the elevator.

A few years ago, I ended up losting weight, mostly by dieting. That worked out rather well up to a point: I lost about 20kgs rapidly. But I ended up putting them back on almost as rapidly. What I learnt from this was that dieting made me more short-tempered. It also reduced my metabolic rate. My body would adjust to the hunger and enter a “starvation-mode”, using the limited food ridiculously efficiently. So I’d have to eat even less to continue losing weight. This time, I’m going to try it the slow way. Firstly, my targets are moderate. I plan to lose about 1kg every month. (It’ll take me a few years to achieve my target. That’s good – it postpones the time when I’ll say “Ah, I’m thin. I can eat now” and become fat again.) Secondly, I’m not going to keep myself hungry. I’m just going try to stop eating when I’m not hungry. This happens for two reasons. One’s because I usually watch a movie when I eat, and keep eating until the movie’s done. The other’s because when I’m between activities, I raid the kitchen. It’s not realistic to pretend that I can curb these tendencies. But it’s possible to be a bit more aware of them. Stocking the house with healthier foods helps. I also find that some fruits in particular keep my stomach full for longer. They’re low value for money energy-wise, but great for dieting and health. Thirdly, I’m going to start exercising, but in my own, slow, way. I’m not good at running or going to the gym, but weirdly, I rather like climbing stairs So the next step is to abandon lifts and only use the stairs. None of these is a big step. But I’m not in a hurry, and these are more like habits I’d like to get into for the rest of my life rather than short-term measures. Here’s to a lighter 2014! Comments Pravin 19 Jan 2014 11:46 am: Similar to what you have said --> http://new.ted.com/talks/sandra\aamodt\why\dieting\doesn\t\usually\work Kalpesh 19 Jan 2014 3:49 pm: Good to see a blog post. I hope you get time to post often, specifically on things you are working on. If you are coming to Mumbai, post it. It will be great to meet you in person. Suren Reddy 17 Mar 2014 5:39 am: Definetly Anand sir, I agree with you... rapidly loosing weight makes you short tempered. I have experienced it at personal level... Somnath 27 Jan 2014 1:43 pm: Good to see you write after such a long time :D Taking the stairs is right option - I religiously follow at my apartment except when I am taking my kids up/down or have heavy load (weekly groceries etc.). At office post lunch I go down 5 levels, 10 min brisk walk and then climb up 5 levels. Well that has ensured my weight does not change much. Cheers to "Lighter 2014"! Shirisha 3 Apr 2014 6:55 am: Great decision!!! Scientists say that when we feel hunger our stomach releases some acids which are helpful for digestion, If we eat without feeling hunger those acids will not get released and the eaten food would not be digested properly!!! All the best for your good health!! Venkat 8 Apr 2014 5:51 am: Moderate cycling might also help, may be a good bicycle that can help you to get around for normal chores would be a great way to achieve this goal. chandigarh 12 Oct 2015 7:47 am: Actually regular exercise or 1 hour morning and 1 hour evening can do much more fore you. internet is here to help you to know the effective exercise. taking less but healthy food in disciplined intervals is necessary.

Weight lines, again


I’ve shifted my output from writing to public speaking, trading the permanence of blogging for the real-time feedback of live talks. While I miss search discoverability, speaking allows me to shape data visualization insights dynamically based on audience reaction.

My blog’s been through a number of phases. Between 1996 – 1999, it was just a website with a few facts about my and some of my juvenile ramblings. Inspired by robotwisdom.com, I converted it into a blog – except that I didn’t know what blogging was and just called it “updating my site every day.” It was mostly a link blog. In 2006, around the time when I moved from Mumbai to London, I reduced my link-blogging and started writing longer articles talking about my experiences. This was a fairly productive phase, and I was churning a few dozen articles every year until 2012. In the UK, I didn’t know many people, and wasn’t comfortable going out of the way to interact. My blog was the primary means of sharing my thoughts and work. In 2012, when I moved back to India, that changed. I started speaking at various events. (Some of my talks are recorded.) I’ve been speaking at one or two events every month, which is roughly the volume of blogging I was doing since 2006. So, effectively, my output medium has changed. Instead of writing, I speak. Correspondingly, my blogging has come down. How does it feel? Well, on the one hand, there’s a lot more direct feedback when you’re speaking to an audience. You can interact with them, ask questions – all of which I can do on a blog as well, but this is real-time. When my audience laughs, I steer my talk more towards funny insights. When my audience claps, I steer it towards more impressive techniques. When my audience reacts like dead fish, I switch to Q&A. When my audience is lost in their own conversation, I terminate the talk early. Effectively, my content is often shaped in real time. And that can be (usually) an exhilarating experience. I used to worry that the talks didn’t have the permanence of blog posts. But like I said, many of them are recorded. I also worried that the audience response would not be permanent, like blog comments. But Twitter fills that void. For example, yesterday, I was speaking at the Great Indian Developer Summit. Here are the tweets going out as I was speaking. Kashinath Pai. P.: #GIDS visualising data by Anand » Naresha: Looking forward for a cool 'Visualizing Big Data' presentation from @sanand0 #gids » ARAVIND CHEKKALURE: #gids watg for a solution in visualizating bigdata..woww » Raj: Analysis of big data and visualization of big data is very different #gids » Sundarraj Kaushik: Now to visualize data with Anand at #gids » Raj: Anand's session on visualization of big data surly interesting talk of day. I attended previously #gids » UK Gupta: Another Excitin &Interestin Session "Leveraging #Cloud Services2Build & Integrate Analytics in Ur #IoT Solutions" by @Ragural #IntelDZ #gids » Siva Narayanan: Doordarshi party is the worst loser in Indian politics #gids » Siva Narayanan: Very cool viz about Indian Elections from gramener #gids » ARAVIND CHEKKALURE: Examples for visualiztion that anad took is rally imprazv #gids » Siva Narayanan: Margin of victory for winner isn't affected by number of candidates. Affects runner up. #gids » Goutham G: Enjoyed every bit of information on xls by vinod #GIDS » Sanaulla: Very interesting facts and presentation by s anand in visualizing big data #gids » THIYAGARAJAN.R: #gids Hi everyone, Anand session on Data visualization is interesting... happening on Main Hall » Sanaulla: Visualization helps grasp huge amount of data quite easily #gids » Vignesh Rajendran: Sanand might be called @NateSilver538 of Indian politics analysis #gids » 45hw1nk5: By far the beat talk so far, visualising data by S Anand from Gramener. #gids » Rahul Sharma: On the Everest of knowledge with S Anand.. Courtesy - 'Big Data' :D #gids » Sundarraj Kaushik: A very pertinent subject visualization of election statistics at #gids. » Siva Narayanan: UdayKumar has 1600 cases against him #gids » Rishi Raj Srivastav: Great Indian politics (data) visualization by Anand. #GIDS » Siva Narayanan: Singh has been most popular last name in Indian elections every time #gids » ARAVIND CHEKKALURE: Big data visualization is this much easy..like anand speks #gids » Sundarraj Kaushik: A very colourful presentation without actual mention of big data or visualization. Wonderful presentation at #gids. » Govind Kanshi: Gujarat, Maharashtra have longest names fighting in elections #gids powerful story as usual by @sanand0 » Venkat ramanan v: Data visualization at it's best. #gids » Kamlesh ®: RT @govindk: Gujarat, Maharashtra have longest names fighting in elections #gids powerful story as usual by @sanand0 » brntbeer: Talk about last names and regions of India. I’m definitely an outsider! #gids » Sundarraj Kaushik: Is the dropout of girls the cause of better results of girls. Anand at #gids » Sanaulla: Best session of the day: visualizing big data #gids » ujwala: Visualizing big data session is very very interesting. Nicely done. #gids » Siva Narayanan: Sun sign has a big impact on exam performance #gids » Venkat ramanan v: Intriguing session on visualization #gids http://t.co/sezeRP48BM » Siva Narayanan: Almost nobody is born in august in India! #gids People are fudging birth dates. » Raj: Thanks #saltmartch for invite such a good speaker. #gids » ujwala: RT @K2\181: Almost nobody is born in august in India! #gids People are fudging birth dates. » Vinod Srinivas: @greatindiandev #gids #Anand was at his best in his session on #Visualisation » Kiran Bhat: Lets get people to SEE data #Gramener #gids » Sonali Patnaik: #GIDS "lets get people to see data" good session @sanand » Amol Khanapurkar: Easily the best session at #gids for me by S Anand from http://t.co/1lVuBMpPlW » isha jain: Amazing facts and awesome session on data visualization by Anand... #gids » Kashinath Pai. P.: Absolutely mind blowing presentation by s anand #GIDS » Sundarraj Kaushik: Thanks to Anand S for a marvellous and pertinent presentation at #gids 2014 » Naresha: @sanand0 Those were amazing visualizations of data. One of the best sessions of #gids. » Amol Khanapurkar: Data visualization can provide insights that no amount of analytic processing can hope to provide. #gids » Vijay Singh: Session on big data visualisation was a joy ride #gids » Mrugen Deshmukh: @Gramener Most entertaining talk yet. by S. Anand #gids » Kashinath Pai. P.: Wonderful work @greatindiandev . inviting @sanand0 was absolutely amazing. #GIDS » Sachin: That was an really awesome session on big data visualisation.. Had fun... #gids » ARAVIND CHEKKALURE: Its reLy gd session by anand on visualizatg Bigdata..but never touch any tools and technologis. DisAptD #gids » Harpreet Singh: Great session by anand Add visualisation to data to make it information #gids » Raja Guru T: #gids thing of awesomeness visualization of large data. Lovely session by Anand. Way to go Saltmarch. Loving it. » Raja Guru T: RT @ujwala: Visualizing big data session is very very interesting. Nicely done. #gids » Prashanth: #gids data visualization session was amazing » Subhashish Dutta: At #gids today, awesome visualization of some big data in the Indian context by Anand of Gramener. » Japesh Thyagarajan: An impressive and fun session from Anand on Visualising Big Data, amazing illustration of Election and Education system , Hats off #GIDS » Raj: I must say visualization of big data best session of #gids » Apart from being able to preserve comments, I get to hear of this feedback a lot quicker than on a blog. What I miss, though, is discoverability. When I blog, search engines index the content for anyone to find. I still get relevant comments on 15-year old blog posts. That, I suspect, will not be the case even for recorded talks. But in any case, I’m afraid I will continue blogging less and speaking more over the course of the next few years. Please bear with me until then! Comments deepan 29 Apr 2014 12:37 pm: My hunch was that since you started your own company, you are engrossed with lot of work. Thank you for the update, do continue blogging. Shankar V 30 Apr 2014 2:15 pm: I do miss your blog posts though! :) I did check some of your youtube videos and and I liked the ones on data visualization quite a bit. I am working on a product that tries to grab relevant data from twitter feeds and use that as part of its forecasting functionality. I was able to gather some useful insights in those. Of course, my product does not need visualization techniques - like the ones you have used in your seminars. :) Bargava 15 May 2014 3:12 pm: Really miss your interesting articles. Your talks are brilliant and interesting. It would be good if you would post some link articles/ talks that you attended/ presentations you did/presentations you found useful somewhere (preferably in this blog). Vasudevan Panhavoor 19 Feb 2015 4:16 pm: Your talk today at Scope international was very interesting.opened lot of new windows of thought and insight.Thanks

Why I’m blogging less


I was named one of the top 10 data scientists in India for 2015 by Analytics India Magazine. This post reflects on the recognition and links to the list celebrating leaders in the local data science community.

Apparently, I'm one of the top 10 data scientists in India. http://analyticsindiamag.com/top-10-data-scientists-in-india-2015/ LinkedIn

Apparently i m one of the top 10 data scientists in india https lnkd in dbecsbu


I analyzed my Airtel postpaid bills to identify unused data quotas and expensive roaming charges. By optimizing my myPacks and switching to a cheaper myPlan, I reduced my monthly spending while increasing my roaming benefits.

My monthly postpaid mobile bills have been in the Rs 2,000 – Rs 3,000 range for some time now, and I spent a few hours dissecting them yesterday. Page 3 had the good stuff. It’s a little hard to figure out, but what the last 2 columns say is that most of my spend is offset by discounts. 2015-08-11 182653-Start What’s not getting offset are outgoing roaming calls. Followed by calls to local landlines. For all practical purposes, that’s the only thing that counts in this bill. Everything else is close enough to zero. It took me some time to figure out that Airtel postpaid has something called myPlan. Based on your plan, you get set of “myPacks” or discounts. That determines your final bill. And it turns out that I was barely using my quota in some areas – specifically data. I had 3GB of data available. I was typically using 200MB – 500MB. The last 2 columns on page 2 show the usage of myPacks. image Clearly, I can do with less data, less SMS, less local mobile, and perhaps less STD mobile. I might need more outgoing roaming, but that’s about it. This means I need fewer myPacks. So I was able to switch to the Rs 799 plan from the Rs 999 plan, while simultaneously increasing the number of free outgoing roaming calls I can make. There seems to be no myPack for incoming roaming, so I’m actually better off calling people if I’m travelling, rather than receiving calls! The rest of the bill is a treasure-trove of data, listing every call and every pulse of data connection. I only wish it also had the location of the calls, and were available as CSV files. Comments Ashish 11 Aug 2015 6:31 pm: Ask Airtel to start 99/- per month national roaming pack to make all incoming roaming calls free. 408wij 11 Aug 2015 9:29 pm: A quick google search turned up two android apps: (1) Call Log Location and (2) Log Call Location. Perhaps their logs can be exported and cross-referenced w/ your bill. Bala 1 Sep 2015 12:47 pm: You spend 1 day (>24 hours) in talking on your mobile. Indrajit Ghosh 18 Jan 2017 1:57 pm: You can ask for a better Data deal from Airtel.

Dissecting my Airtel bills


I discovered a perpetual motion machine for happiness: scheduling meetings makes me feel productive, while having them cancelled grants me the joy of reclaimed free time. It is my win-win strategy for management and workplace satisfaction.

In my current thrust towards greater management responsibilities, I have discovered a mechanism for generating happiness. I set up meetings on important topics. That makes me happy -- I'm driving something useful. Often, the meeting gets cancelled. That makes me happy -- I've more free time. It's the perfect perpetual motion machine. Comments Vasant 10 May 2016 11:58 pm: Ha ha! Love it. Madan 30 Mar 2019 9:00 pm: Often, the meeting gets cancelled. That makes me happy — I’ve more free time. What a positive thinking Sir Jee !! vikram 5 May 2016 6:04 pm: i am a non tamil .i have a piece of music and i want to know which song is it exactly.This was briefly played in the movie madras cafe starring john abraham.can u just help me wih that song . Kindly ping me to my mail i ll share that piece with you udayamoorthy v 23 May 2016 12:34 pm: That's Great Idea. Kind of Win Win Situation. Happy to see you in the blog after a long time. I used to view your older posts regularly . All are great. Thanks Regards Uday Chirag 25 Sep 2017 9:03 pm: Haha ! Thats clever :)

Happiness generator


I enjoyed watching Richie's impressive Excel work and expressed my anticipation for his next creative feat: animating a Michael Jackson dance sequence inside a spreadsheet. It's fascinating to see such unconventional uses of data software.

Always a pleasure watching Richie work his magic on Excel. Looking forward to Richie showing Michael Jackson dance in Excel 😀 LinkedIn

Always a pleasure watching richie work his magic on excel looking forward to ric


I am looking for seasoned developers with data and front-end expertise to join my team at Gramener Mumbai. If you enjoy building data-driven applications under tight deadlines, I would love to work with you.

If you're a seasoned developer that enjoys working with data, have good front-end skills, and are challenged by impossible deadlines, please drop me a note. I'd love to work with you at Gramener Mumbai. LinkedIn

If you re a seasoned developer that enjoys working with data have good front end


I saw Richie Lionell demonstrate how augmented reality integrates with comics and data storytelling at the Indian School of Business. The session was compelling enough to hook attendees who wandered in from nearby AI workshops.

Richie Lionell demonstrating how #augmentedreality weaves in with #comics and #datastorytelling at the Indian School of Business The anecdote I loved about this event was when an attendee from the nearby AI workshop got bored, wandered in here, and was hooked ☺ LinkedIn

Richie lionell demonstrating how augmentedreality weaves in with comics and data


I shared an impressive project by Ramya Mylavarapu that merges comics, augmented reality, and data visualization. It demonstrates how gamification can transform static narratives into immersive, interactive data-driven experiences for a modern audience.

This is the most interesting thing I've seen in a while. Combining #comics with #augmentedreality, #datavizualization and #gamification Nice work Ramya Mylavarapu! https://twitter.com/i/status/1201170899434520576 Right amount of Data, Comic and Augmented Reality. Addictive like a perfect coffee! With @Gramener @musingswithcode @richielionell @thoughtisdead at @AgamiIn pic.twitter.com/uGUbY2DGft — Ramya Mylavarapu (@MylavarapuRamya) December 1, 2019 LinkedIn

This is the most interesting thing i ve seen in a while combining comics with au


I analyzed 11 million steps from my last three years of walking to optimize my health and efficiency. By lengthening my stride by two inches, I aim to reach 5 km/hr and save eight minutes per 10k steps.

I walked 11 million steps in the last 3 years, at 10K steps daily. Since 1 Jan 2018, I've steadily increased my walking average until Aug 2018. Then my legs started aching. So I cut it down until Jan 2019. In Feb, I resumed and was fairly steady until May 2020. To complement workouts like this, products that are aimed for men over 50 can be used. In May, my wife refused to let me walk for more than an hour a day. It took me a few months to convince her and level up. I ended 2020 averaging a little over 10K steps for the year . I'm becoming more regular. I walked 10K/day 15% more in 2020 than in 2018. 2018: I walked 10K steps almost half the time. 2019: it grew to a bit more, to 56%. 2020: I walked 10K steps a day almost two-thirds of the time. But in May 2020, I went for 5 days without walking even 3K steps. In 2018, I started being more and more regular until my leg started aching. 2019 was fairly consistent. 2020 is when I applied brakes again -- for very different reasons. I've never gone for 5 days without walking even 3K/day before, since 2018. At most, it was 3 days at a stretch. But when my wife refused to let me walk for more than an hour a day in May 2020, I went on strike! 😉 I walk 77 min daily. This has increased over the years. In 2020, this has gone up slightly to 84 min -- but it's still under an hour-and-half. I spend most of this time on calls or listening to audio books / podcasts. Instead of spending it with my family. Sometimes, I lose myself in calls and walk for almost 3 hrs and 20K steps . Naveen is usually to blame. But this happens rarely. I walked 20K steps just 6 times over the last 3 years. Though the longest walk here indicates over 3 hrs, I've never walked 3 hrs in a day. On 21 Nov, my daughter borrowed my phone and went for her walk. So my phone shows our combined walks, not mine. Many of the other long walks are spread out during the day when I commute by walking in Singapore. Date hrs km # Why? 21-Nov-20 3.46 15.5 1 My daughter took my phone. These are her + my walking stats. 15-Nov-19 2.98 11.5 2 Walked to meetings in Singapore. 17-Sep-19 2.96 10.7 3 Walked to meetings in Singapore. 11-Jul-20 2.89 13.9 4 Was talking to Pratap & Ganes. 15-Oct-18 2.83 9.5 5 Walked to meetings in Singapore. 03-Sep-20 2.82 13.0 6 Was talking to Naveen & my coach. I want to walk faster. I walk at 4.4 km/hr. My target is 5 km/hr. Walking at over 5 km/hr speeds the heart up and improves metabolism. (Or so I've heard.) I was steadily going towards 5 km/hr in my early days of walking. I slowed down starting Aug 2018, since my legs were aching. Then I picked up speed in end-2018. I slowed down again in Nov 2019 -- and I don't remember why. In Jun 2020, I started walking much faster -- mainly to complete 10K steps within the hour my wife gave me. That seems to have had a lasting impact. I walked faster overall in 2020. I've managed fast walking 66 times in 2020, a bit more than before. In Jun 2020, I walked at over 5 km / hr on 20 / 30 days -- a very consistent high speed. I've never gotten close to this any other month. (Clearly, there are adverse effects of being able to convince my wife.) The fastest I walked was in 2018, at 6.8 km/hr. It might have led to my leg aches. My top 5 walking speeds were in 2018. In 2020, I've managed to walk faster than 6 km / hr just once. Fastest days km/hr 07-Jun-2018 6.80 05-Jan-2019 6.65 16-Mar-2018 6.34 08-Jun-2018 6.31 06-Feb-2018 6.19 05-Jun-2020 6.02 The normal stride/height ratio is 0.43 . I'm 5'8". My stride is 2.4 ft. That's almost exactly 0.43 times my height. So all is well. By increasing my stride by 2 inches, I can cover 10,000 steps in 8 min less time. For every inch I lengthen my stride, I walk 0.2km/hr faster. I've walked with a stride as long as 32", which is 3" more than my 2020 average stride. By walking with a 2" longer stride, I can be 9.2% faster. So in 2021, I plan to get healthier (and scolded less) with a 2" longer stride. A longer stride means a faster walk. That's a good cardio exercise. A faster walk also means that it takes less time. So I'll get beaten up less. All it takes is stretching my legs 2" more. Might hurt a bit. I'll report on this when I know better. Now New Change Benefit Longer stride 29" 31" 2" Builds character? Faster walk (kmph) 4.5 5.0 0.5 Better cardio exercise Time to 10K steps (min) 84 77 -8 Less scolding from wife PostScript : This analysis was done in Excel. Download see the sheet below. Walking Excel sheet (Download) Comments Somnath 30 Dec 2020 11:28 am: Great to see you back blogging on your personal site. Interesting analysis on your walking with cool insights (and learning Excel tricks) from download :-) Enjoy walking in 2021 to stay healthy and meet your resolution(s). My Year in 2022 - S Anand 31 Dec 2024 4:31 pm (pingback): […] Walk 10,000 steps daily. […] <!-- wp-comments-end -

2 inches will change my life


I'm looking for candidates for an internship focused on designing and creating original comic book characters. This opportunity is ideal for artists wanting to build a portfolio in character development and conceptual illustration for the comic industry.

Anyone interested in an internship to create comic characters? LinkedIn


I share my 2020 reading list of 45 books, ranking non-fiction favorites like The Lean Startup and exploring fantasy series by Brandon Sanderson and Brent Weeks. I also include comics and poetry to help meet my Goodreads challenge.

My Goodreads 2020 Reading Challenge target is 50 books. I'm at 45/50, with little hope of getting to 50. (I managed 25/24 in 2019.) The 10 non-fiction books I read (most useful first) are below. 1. The Lean Startup by Eric Reis.\ The principle of Build - Measure - Learn is useful everywhere in life too, not just in startups. 2. Never Split The Difference by Chriss Voss.\ Shares principle-driven strategies to convince people. 3. The 4 Disciplines of Execution by McChesney, Covey & Huling.\ Teaches how to build execution rigor in an organization. A bit long at the end, but the first section is excellent. 4. Sprint by Jake Knapp.\ A detailed step-by-step guide to running product development sprints that you can follow blindly. 5. How to Fail at Almost Everything and Still Win Big by Scott Adams.\ Dilbert's author shares his strategies for life. Very readable, intelligent, and slightly provocative, but always interesting. 6. The Five Dysfunctions of a Team by Patrick Lencioni.\ Written as a story (like The Goal). Talks about the 5 problems in teams and how to overcome them. 7. The Culture Code by Daniel Coyle.\ Explains the elements of strong cultures - belongingness, shared vulnerability, and shared purpose. 8. Data-Driven Storytelling by Nathalie Henry Riche et al.\ Shares the latest points of view on telling data stories. My team and I read these chapters as a group. 9. Leaders Eat Last by Simon Sinek.\ Inspiring when I read it, but I don't remember what it said. 10. Deep Work by Cal Newport.\ Shares tactics to focus. Practical and useful. I also started, by haven't finished these four: 1. Hacking Growth by Sean Ellis & Morgan Brown 2. The Laws of Human Nature by Robert Greene 3. How to Fail at Almost Everything and Still Win Big by Scott Adams 4. Stories at Work by Indranil Chakraborty I read these 25 works of fiction -- mostly by Brandon Sanderson (my current favorite author) and Brent Weeks. 1. Lightbringer (Books 1-5) by Brent Weeks.\ In a world where color is woven as magic, the most powerful man is caught in politics. This series had enough twists and turns to keep me hooked till the end. 2. Skyward (Books 1-2) by Brandon Sanderson.\ An outcast girl on an outcast planet becomes a fighter pilot with an alien spaceship. I love the way this is developing. 3. The Wheel of Time (Books 1-6) by Robert Jordan.\ I picked it up again mainly because Brandon Sanderson wrote the last 3 books. It was great up to book 4 but has started dragging. 4. Alcatraz Versus The Evil Librarians (Books 1-4) by Brandon Sanderson.\ The author lies to you. Literally. And tells you that he will, in almost every other paragraph. Great book for kids to laugh over. 5. Night Angel (Books 1-3) by Brent Weeks.\ An assassin in a story that spans centuries of the history of magic. 6. Legion (Books 1-3) by Brandon Sanderson.\ A detective who has multiple split-personalities in him -- that help him solve cases. 7. Snapshot by Brandon Sanderson.\ What if you could create a snapshot of the world, enter it, interact with it, and solve crimes? 8. The Art of Letting Go: Poetry for the Seekers by Sanhita Baruah.\ It's my first poetry book. (I hate poetry.) I took this up to see if I could survive it, and get a fresh perspective. I survived. ... and these 10 comic books/series. 1. Batman, Volume 1: The Court of Owls 2. Batman, Volume 2: The City of Owls 3. World War Hulk (1-5) 4. Superman: Red Son (1-3) 5. Flashpoint (1-5) 6. Batman - The Long Halloween (1-13) 7. Batman - The Killing Joke 8. Kingdom Come (Vol 1-4) 9. Spiderman: Ends of the Earth 10. Amazing Spiderman, Vol. 1 At the moment, I'm at 45 books, with little hope of completing 5 more this month unless I pick up comics. So that's exactly what I'm going to do 😉 Comments Kalpana Behara 5 Dec 2020 6:10 pm: I am so glad you are still writing. I stumbled upon your site some 15 odd years ago and read your Illayaraja articles. They were very technical for me to understand but I remember completely being mesmerized by the technicalities. I come back every few years to read those articles and listen to the songs you reference. I did that again today and realized you continue to blog! This is one of my favorite blogs on Internet. Thanks for sharing. Deepan Chithambaranadhan 6 Jun 2021 1:32 pm: Please also read the Thinking Fast and Slow by Daniel Kahneman ;if you havenot read. Very happy to See your updated website it was down for a few weeks some time ago. I recommend to my mentees your website for insights on interviews and your popular analysis on birth dates and 12th standard grades. I wish you could write more on Object oriented programming and Clean code. Samrat Sen 28 Dec 2020 6:58 am: I have read some of these - The Lean Startup, Stories At Work and quite a few of the Batman comics (Superman - The Red Son is now on Prime). Hadn't heard of Brandon Sanderson but looks promising. Currently reading Shoe Dog by Phil Knight. Inspiring! Books in 2023 - S Anand 4 Jan 2024 4:46 pm (pingback): […] read 52 books in 2023 (about the same as in 2022, 2021 and 2020.) Here’s what I read (best books […] S Anand 8 Dec 2020 3:52 pm: Thanks, Kalpana! I've been away for many years now, but am back regularly 😊 Books in 2024 - S Anand 31 Dec 2024 6:31 pm (pingback): […] read 51 new books in 2024 (about the same as in 2023, 2022, 2021, and 2020.) But slightly […]

Books in 2020


I found that PowerPoint is an accessible SVG authoring tool for web visuals. I draw shapes or Smart Art, save them as SVG, and get clean, editable code suitable for interactive simulators or animations.

With Office 365, PowerPoint supports SVG editing. This is really powerful. It means you can draw in PowerPoint and render it on the web -- including as interactive or animated visuals. For example, the SVG in this simulator was created just with PowerPoint. [](https://gramener.com/processmonitor/simulator) The process is simple. Draw anything. Select any shapes and right-click. Select Save As Picture... and choose SVG. For example, you can use PowerPoint to create Smart Art, export it as SVG, and embed it into a page. See this example on CodePen. See the Pen Smart Art SVG by S Anand ( @sanand0 ) on CodePen . The SVG is fairly well structured and easy to edit. The code generated for these 2 simple shapes: ... is quite straight-forward -- just two SVG shapes. I was worried about the lack of SVG authoring tools in Windows. (InkScape is not usable, and Adobe's tools are complex and expensive.) PowerPoint fits perfectly.

Create SVG with PowerPoint


I benchmarked lxml against Node.js's htmlparser2 and discovered Python is actually faster for one-off HTML parsing tasks. lxml processed a 700KB page in 8.6ms, outperforming Node's 14.5ms initial results, so I'm keeping Python in my toolkit.

Given the blazing speed of Node.js these days, I expected HTML parsing to be faster on Node than on Python. So I compared lxml with htmlparser2 -- the fastest libraries on Python and JS in parsing the reddit home page (700KB). lxml took 8.6 milliseconds htmlparser2 took 14.5 milliseconds Looks like lxml is much faster. I'm likely to stick around with Python for pure HTML parsing (without JavaScript) for a while longer. Note: If I run the htmlparser2 code 100 times instead of 10, it only takes 7ms per loop. The more the number of loops, the faster it parses. I guess Node.js optimizes repeated loops. But I'm only interested in the first iteration, since I'll be parsing files only once.

lxml is fast enough


I moved from managing large, general text files to hundreds of specific micro-notes. Syncing Markdown via Dropbox and using the Everything search tool allows me to find and edit my notes instantly on both desktop and mobile.

I maintain my (extensive) notes in text files. I've explored Evernote, Onenote, Google Keep, Apple Notes, and many other platforms. But text files work. I store them as Markdown and sync them on DropBox. They used to be relatively large files (50-100KB) each, on broad topics. For example: todo.txt was a consolidated list of things I had to do people.txt was a list of everything I knew about people (addresses, birthdays, etc) towrite.txt was a list of everything I wanted to write about notes.txt was where I tracked notes about any topics ... and more This led to a couple of problems. 1. Searching across files was hard. I wouldn't remember if I wrote ideas for my next talk in todo.txt or towrite.txt, or if my meeting minutes where in notes.txt todo.txt. I had to open each file and search. 2. Files were getting too big. Editing them on mobile was harder. Scanning them was harder. So I changed this system a few years ago into micro-notes. These files became a folder. For example, my notes/ folder looks like this: time-management.txt has my time management notes book-never-split-the-difference.txt has book notes on Never Split the Difference eat-food-sleep-exercise-live-healthy.txt has notes on fitness The folder has nearly 300 files. Here's a glimpse of the latest files. Similarly, my people/ folder has details of my discussions with various people I interact with -- friends, colleagues & clients. What made this change possible is Everything, a fast file search tool on Windows that lets me find files as I type. For example, if I'm looking for my notes on SlideSense, I just type "notes s" and it appears on the list. I usually sort the files by run count (how often I opened them). That makes it easy to re-open the most used files. It also makes it easier to edit these notes on mobile. I sync the folder on Dropbox, and use IAWriter to edit them while on walks. Dictation is pretty good, so I've been using that to take notes too. Comments Saurabh Mittal 3 Dec 2020 1:20 pm: Hey Anand, great to see you back! After a loonngggg hiatus! S Anand 4 Dec 2020 10:37 am: Thanks, Saurabh! Ananth 26 Dec 2020 2:32 am: Welcome back Anand. I have been following your "blog" (hope the term is still relevant!) for more than a decade. Your matter-of-fact writing is something that I enjoy and your experiences using various tools/approaches have always been interesting. Hope we can see more of your writing. Ramamurthy 27 Dec 2020 7:03 pm: Hi Anand, I stumbled into your page accidentally while looking for something about Ilayaraja.I am your chittappa,Lalitha chitti's husband.We attended your wedding too. Much water has flown under the bridge and I have lost contact with our relatives.Ramnath and Shekar are in touch.How is your Sriram uncle and your dad.I hope you remember me!

Micro-notes


I mine Ancient Debris in the Minecraft Nether by using exploding beds as a cost-effective alternative to TNT. This method quickly clears large areas to gather the materials needed for crafting powerful Netherite armor and weapons.

I've been active on Minecraft for the last 6 months, thanks to my daughter. She keeps watching game videos for hours. I thought I'd see what the big deal was, and made one myself. In this 5-minute clip, I'm mining for Ancient Debris in the Nether by placing beds -- which explode when used in the Nether. That's a quick way to clear large areas and is cheaper than TNT. Ancient Debris is used to make Netherite Scrap which makes Netherite ingots that can upgrade to Netherite weapons and armor -- the strongest things in Minecraft. Why do I care? Well, when my friend's son said "You're the only adult I know who plays Minecraft", I felt 20 years younger 😊. Comments Thejesh GN 27 Dec 2020 7:54 am: Mincraft Bicycle Dogs made me popular amongst the kids in our apartment.

Mining for Ancient Debris


I reviewed my 2020 progress on reading goals and daily walking while sharing my 2021 resolutions for weight loss, embracing failure, and calendar integrity. I also highlight the productivity impact of Landmark and Art of Living courses.

In 2020 I made 3 resolutions. 1. Read 50 books. I almost made it. Here are my reviews. 2. Walk 10,000 steps daily. I managed it, like the last two years. 3. Lose 2 kgs. I failed -- and instead, put on 6 kgs. On self-improvement, I completed a Landmark course and an Art of Living course. Both had a huge productivity impact. (Mail me for details.) On software, I starting playing Minecraft and moved from Gmail to Windows 10 Mail. More on this. On training, we built a data storytelling course and intend to train 100,000 people. (You can register for a free 90-min online workshop.) In 2021, I'm taking up 3 new goals. 1. Lose 10 kgs. I'll share my stats publicly. 2. Fail big. Take up initiatives I'm likely to fail on and learn. 3. Calendar integrity. Do what my calendar says, no matter what. Hope you have an amazing 2021! Comments Krishna 28 Dec 2020 5:03 am: Great to see you are back to posting. Glad I didn't prune your feed from my rss reader :) My Year in 2021 - S Anand 31 Dec 2024 4:31 pm (pingback): […] In 2021, I made 3 resolutions. […]

My year in 2020


I investigated why 10-year Indian Gilt Funds reported returns over 10% despite underlying bonds yielding much less. I found that constant maturity funds capture both coupon accruals and capital appreciation during falling interest rate cycles.

This month, I sold half my Indian equity mutual funds and was researching funds to invest in. I was looking for something safe & long term. As I was exploring 10-year Gilt Funds (mutual funds that invest in the Indian Government's 10-year bond), I noticed that they had a pretty high yield -- mostly over 10%. I took a closer look at ICICI Prudential's Constant Maturity Gilt Fund. (They had the lowest expense ratio.) The annualized returns over the last 5 years were 10.77%, and it's never fallen below 10% in the last 5 years. But the strange thing is that the underlying 10-year bond always yielded less than 10% in the last 10 years. So, how does a mutual fund that buys only one bond yield more than that bond's ever done in the last 10 years? (I went ahead and invested. But this is going to worry me to no end.) Comments S Anand 2 Jan 2021 3:13 am: Thanks, Ashwin -- that was helpful. I researched this specific fund, and it looks like they only invest in the 10-year GoI Gilt fund. But obviously, the time of investment would have varied. Given that interest rates have been generally declining that may explain part of the rise in this fund as well. Ashwin 27 Dec 2020 3:53 am: Hi Anand: I have been reading your posts for several years now. To help you understand the conundrum, A constant maturity fund mixes gilts of different longer-term maturities such that the average portfolio maturity stays at 10 years. This strategy has two key benefits. The first and obvious is the absence of credit risk. By investing purely in gilt securities, these funds are less risky than even high-credit-quality debt funds. For instance, an event such as the IL&FS catastrophe can hurt such high-quality debt funds, but will still leave gilt funds unharmed. The second benefit is that there are no risks of wrong duration calls. Gilt funds, unlike constant maturity funds, will change their portfolio maturities based on the stage of the interest rate cycle. That is, they may go for short-term gilts, or medium-term or even very long-term 20+ years of gilt papers. Dynamic bond funds similarly switch between duration and accrual-based on the direction of the interest rate cycle. This introduces uncertainty in how well they can capture market opportunities. A dynamic duration strategy needs to get its calls correctly in order to book profits made from bond price rallies. Missteps here can hurt returns. Since constant maturity funds stick to a specific duration, this risk significantly reduces. The strict maturity also ensures that the fund can book and realize the gains made on price appreciation. As an investor, therefore, you get both the accumulated coupon and capital appreciation. Constant maturity funds have the ability to beat other debt fund categories such as a corporate bond, medium duration, medium to long duration, and other gilt funds especially over a period where interest rates are overall trending lower. Consider the 5-year rolling return for the period between 2011 to date, of the Nifty 10-year Benchmark G-Sec. While most debt funds from categories such as a corporate bond, dynamic bond, short duration, and medium duration beat the benchmark return, just about half were able to do so more than 60% of the time. That is, the 10-year benchmark G-Sec index was able to beat funds with a good degree of consistency. Constant maturity funds do not have too long a history. However, their 5-year returns in the past six-year period have beaten funds across debt categories. It is noteworthy that rate cycles have not only become shorter in the past 6 years but have also turned trickier to forecast and manage. This makes constant maturity a strategy devoid of the uncertainties of all other duration categories. Do remember these funds do not stick only to 10-year papers, but mixes different long-term maturities of 8-13 years such that the overall average maturity remains around 10 years.

Mystery of the extra returns


I list my essential 2020 tools, highlighting VS Code for notes and task management, Everything for fast file searches, and WSL for native Linux. I also share hardware finds like my Casio keyboard and budget earbuds.

My most-used apps in 2020 were: 1. Everything. Locates files. Like Finder). Fast and brilliant. 2. Chrome. But Edge is pretty good, and I'm using it for secondary accounts. 3. Visual Studio Code. It's my note-taker, TODO list, outliner, and IDE. 4. Minecraft. I'm addicted. 5. PowerPoint. I use it to make & edit videos, not just slides. 6. Zoom. Thanks to the lockdown. Breakout rooms are great. 7. Mail. It uses under 50MB. Gmail takes 250MB. 8. VLC. It still plays all formats, but I'm looking for a replacement. 9. Seafile. Our private Dropbox. 10. AutoHotKey. The best macro tool, but hard to use. The new utilities I started using recently are: 1. WizTree. Fast visual disk space analyzer. 2. PowerToys. FancyZones and ColorPicker are my favorites. 3. WSL. Run Linux on Windows, natively (it's fast). 4. ngrok. Expose local web apps publicly. 5. TabMemFree. Closes unused Chrome tabs, saves memory. The gadgets I bought this year are: 1. Casio CT-X9000IN keyboard. Good touch response. I play every day. 2. HP Pavilion x360. My first touchscreen laptop. Yet to explore touch apps. 3. Raspberry Pi. But I'm yet to use it 😟 4. Teqneq S530 Bluetooth earbud. 2% of an airpod's cost. 5. JBL Endurance Run Headphones. More durable than Foxbit FX500 / JBL C100SI — but one ear always conks off in a few months. 6. PureIT Eco. Consumes a lot less water than regular RO purifiers. 7. Treetop Air Purifier. Fairly quiet, and shows the air quality.

Software & Gadgets, 2020


I've maintained a daily habit of walking 10,000 steps since 2018. I share my journey of consistency, how I integrated walking into my routine, and the long-term lessons I've learned from this simple physical activity.

Since 2018, I've been walking 10,000 steps a day. Here's my journey.

Walking 10,000 steps a day


I am honored that our work was included in Andy Kirk's curated data visualization list. His recognition is a significant motivator for my team as we continue to develop innovative visual journalism and information design projects.

Always a pleasure when our work lands on Andy Kirk's list 😊 -- thanks for being a great motivator, Andy! LinkedIn

Andy kirk best  of the visualization web 2020


This 2021 reading retrospective organizes fifty-two books by impact, recommending titles that most changed the author's thinking about wealth, habits, mortality, storytelling, and fantasy.

Books in 2021 On my Goodreads 2021 reading challenge, I read 52/50 books in 2021. I managed 47/50 in 2020 (see 2020 reviews) and 26/24 in 2019. Here's what I read (best books first). Mind-blowing 1. The Almanack of Naval Ravikant. It's the best non-fiction I've read in 5 years. It focuses Wealth and Happiness. It's short. I finished it in a day. But it's deep. I can spend a decade practicing just a single sentence. It's available at navalmanack.com as a free e-book and audio book. 2. Rhythm of War. The 4th book of the Stormlight Archives is an action-packed fantasy. A great gift for teenagers. In an extra-ordinary magic system, Brandon Sanderson builds up to the greatest climax I've read. What an ending! 3. Death Note #1-#12. Light Yagami gets hold of a “death note”. If he writes a name on it, they die. “L” is out to catch him. In a cat-and-mouse psychological thriller, Light and L work next to each other, share their plans, and still try to outwit the other. It’s like chess. The pieces are visible. But it's the strategy that counts. A brilliant comic series. Life-changing 1. Atomic Habits. A systematic, well-researched approach to creating (and stopping) habits that last. It's the best "Habits" book in the market right now.. 2. Being Mortal. A thoughtful, practical guide on dealing with old age. Must read for those with aging parents. It helps that Atul Gawande is a great storyteller and draws from his personal experiences. 3. Originals. Teaches you how to be more creative and take risks safely. If Creativity Inc inspired you, this book is a way to build Pixar's magic into your teams. An easy-to-read piece by Adam Grant, backed by solid research. 4. Combatting Cult Mind Control. The gold-standard in knowing when someone's in a cult, and how to escape the cult. Opened up a whole new world for me. 5. Rich Dad Poor Dad. Teaches you to make money work for you rather than you working for money. I was shocked when I realized that the middle class buys liabilities (a house to live in) while the rich buy assets (a house to rent out). 6. Think Again. Teaches you how to stop fooling yourself and avoid blindspots by checking your assumptions, enjoy learning from mistakes, and open up people's minds -- especially your own. Yet another easy-to-read piece by Adam Grant, backed by solid research. 7. Influence. A research-backed guide on the science of influencing people subconsciously. Reciprocity, consistency, social proof, authority, scarcity -- these are signals we react to unknowingly. 8. Dawnshard. Book #3.5 of the Stormlight Archives. A handicapped shipowner and her winged reptile pet travel to a mysterious island that no one returns from. With a typical Brandon Sanderson climax that moves this from "interesting" to "life changing". 9. From Data to Stories. The first & only comic data story book, with step-by-step cricket analysis explained by comic characters. This was written by Gramener's Story Labs team using Comicgen characters. Interesting 1. The First Law #1-#3. Joe Abercrombie. A wizard assembles 3 flawed heroes for a quest. 2. Shoe Dog. The Nike founder story. 3. Skin In The Game. On accountability & commitment. 4. That Will Never Work. The Netflix co-founder story. 5. Sycamore Row. John Grisham. An old man's will leaves everything to his servant. 6. A Time To Kill. John Grisham. An African American's on trial for murdering his daughter's rapists. 7. The Psychology of Money. How to make money work for you than the other way around. 8. Detective William Warwick #2-#4. Jeffrey Archer. More adventures from Dt. Warwick. 9. Zoom. A wordless book that zooms out on every successive page, and nothing is what it appears. Readable 1. The Goblin Emperor. Katherine Addison. A Goblin half-son inherits the throne and political intrigue. 2. A Time for Mercy. John Grisham. An African American kid's on trial for shooting a cop. 3. The Rithmatist. Brandon Sanderson. In a world where chalk drawings come to life, a student investigates murders. 4. Karna: The King of Anga. Kevin Missal. A fictionalized story of how Karna re-takes the kingdom of Anga against Jarasandha. 5. Asterix #34-#38. The latest adventures of Asterix & Obelix, the gauls. 6. Infinity Blade #1-#2. Brandon Sanderson. Deathless immortals battle each other, and discover their origins. 7. Old Man's War. John Scalzi. Senior citizens are given a physical boost and sent to fight aliens. 8. Measure What Matters. The definitive guide on how to use OKRs (Objectives & Key Results). 9. The Maze Runner #1-#3. Kids escape from a maze prison and discover why they were there. How I read books 1. Select. I add book recommendations on my GoodReads - to read list. Then I sort by rating and pick the first one I like to read. 2. Listen. I listen to non-fiction audiobooks during walks. 3. Read: I read fiction as ePUBs on my laptop or phone. 4. Stop: I stop reading books that are boring, with no guilt. I've better things to do. Comments Puja Pathak 31 Dec 2021 9:28 am: Greetings Anand sir .. I was a student at upGrad and have seen your lecture videos. I was impressed by the way you communicate and share your knowledge. I started following you on LinkedIn and twitter. Working at Gramener is one of my ambitions. Reading is my hobby too and currently I am reading the Atomic Habits by James Clear. When I saw your post and the books choices, especially the way you pick up a book, I so identified with it. So writing this message. Thanks for sharing. Sandipan 30 Dec 2021 1:51 pm: This is extremely inspiring Anand. I am putting this into my habit. Just finished reading “The Richest Man in Babylon”. Nisheeta 2 Jan 2022 9:40 am: Atomic Habits was my favorite too! And I suspect Sleep Your Way to Success (being published soon) will be my favorite this year :) Balki 7 Jan 2022 7:49 pm: You continue to be a legend with all the time in the world for useful things!! Mukul 13 Jan 2022 5:10 pm: I start my non-fiction journey because of this list. Starting it with The Alma.... and Atomic .... Thank you Anand for this list and sharing your reviews. My Year in 2023 - S Anand 1 Jan 2024 4:35 pm (pingback): […] 50 books a year, like in 2023, 2022, 2021, and […] Books in 2023 - S Anand 1 Jan 2024 4:32 pm (pingback): […] read 52 books in 2022 (about the same as in 2022, 2021 and 2020.) Here’s what I read (best books […] Seema Gopal 3 Jan 2022 11:32 am: Great to read your blog. Especially liked the part where you said you leave the boring books midway without guilt :) The book i loved best this year was Sapiens. Its so popular that you have probably read it, but it changed my perspective to life forever! krishna 3 Jan 2022 10:21 am: I was planning on reading 12 books in the year 2022, and there you are reading a book a week with your busy schedule. Post reading your takeaways on the books and given that it's possible. I have upped my target to 25 in an year. I tested the best ways to mail people - S Anand 6 Jan 2022 11:31 am (pingback): […] Book reviews did 30% worse at #6 than #1 (62% down to 43%) […] Sachidananda Aithal 2 Jan 2022 6:47 pm: Very impressive Anand...will pick up some books from your list.. My Year in 2021 - S Anand 2 Jan 2022 1:37 pm (pingback): […] 50 books. I read 52. Here are my reviews. (Which did you like? What would you […] Ayush Mandowara 5 Jan 2022 7:12 pm: great blog. thanks for sharing your insights related to the books along with sharing the list. Just a small suggestion, please change the link on LinkedIn from HTTP to HTTPS :) Sunil sharma 31 Dec 2021 6:45 pm: Picking some from this to my read list. Have a great reading challenge in 2022 :) Books in 2024 - S Anand 31 Dec 2024 6:28 pm (pingback): […] read 51 new books in 2024 (about the same as in 2023, 2022, 2021, and 2020.) But slightly […]

Books in 2021


This short teaser frames Hollywood not as easy for foreigners, but as one of the least closed major film ecosystems compared with other national industries.

Can foreigners break into #Hollywood? A break into films is hard. Particularly when you're a foreigner. But is Hollywood more open or less open than other countries? If we go by iconic actors, the numbers are not encouraging. Just 2% of Will Smith's co-stars are non-Hollywood. But over 30% of Jackie Chan's co-stars are non-Chinese. But #clustering shows that Hollywood is actually among the most open to foreigners. #dataanalysis Read more at https://www.s-anand.net/blog/can-foreigners-enter-hollywood/ LinkedIn

Can foreigners break into hollywood


Co-star network analysis suggests that while breaking into Hollywood is hard for foreigners, Hollywood is still more internationally connected than most major film industries.

Can foreigners enter Hollywood? An aspiring Malaysian actor posted on Reddit: I am a 18-year old biracial Malaysian kid who wants to be an actor in Hollywood. I'm taking a diploma for performing arts in a college called Sunway University in 8 days and I'm considering pulling out of it because why do something that I like when my dreams might never be fulfilled and the price for taking this diploma is seriously expensive. I am starting to doubt my chances of making it to Hollywood and I suffer from extreme anxiety. Is it possible for someone like me to enter Hollywood? What are my chances? Breaking into Hollywood is hard. As a foreigner, it would be even harder. So I asked myself: Do Hollywood actors act with foreigners? Let's take Will Smith. He frequently acts with Martin Lawrence, Tommy Lee Jones, Jaden Smith, Jon Voight, and 84 other actors. His every co-star is a Hollywood actor, except the Spanish actor Jordi Mollà in Bad Boys II, and the Dutch actor Marwan Kenzari in Aladdin. Will Smith acts with just 2% of foreign co-stars. On the other hand, Jackie Chan is more cosmopolitan. He acts with: Chinese actors like Yuen Siu-Tin in Drunken Master Hollywood actors like Chris Tucker in Rush Hour Japanese actors like Kumiko Goto in City Hunter -- which is based on a Japanese Manga of the same name. South Korean actors like Su-cheon Bae in Huo shao shao lin men -- a Korean/Mandarin film Indian actors like Disha Patani in Kung Fu Yoga - an Indo-Chinese film Spanish actors like Eva Cobo in Operation Condor - shot in Spain Danish actors like Pilou Asbæk in the upcoming film Snafu Of his 224 co-stars, 70 are non-Chinese. Jackie Chan acts with over 30% foreign co-stars. Are Chinese films be more foreigner-friendly? Should our Malaysian friend try there instead? Is Hollywood less open to foreigners than other countries? I took all movie actors across the world and broke them into groups using a community structure. Actors within the group act mostly within themselves, and less with other groups. The largest group is Hollywood, with 80,000 actors (mostly American). They act with each other 90% of the time and act with other groups only 10% of the time. In comparison, the Chinese group has 20,000 actors. They act with each other 98% of the time. When they do act outside the group, it’s mostly with Hollywood (0.5%), Japanese (0.3%), South Korean (0.3%), and Indonesian (0.1%) Clearly, Jackie Chan is more the exception than the norm. But among the large groups, there are 2 groups that are even more insular than Chinese actors. The 8,200 Turkish actors act only with each other 99.1% of the time, occasionally venturing to act with Iranian actors (0.2%). Even more insular are the 7,000 Filipino actors who act with each other 99.3% of the time. They occasionally venture out to act in Hollywood 0.2% of the time. There are no other sizeable groups of actors that're as insulated. Hollywood is actually among the most cosmopolitan groups, along with the West European films. So, to our budding Malaysian actor, I'd say: It's hard to get an acting break. As a foreigner, it's 10 times harder in Hollywood. But you're better off in Hollwood or Western Europe than in any other country, where it would be 50 to 100 times as hard! Comments How isolated is Bollywood from world cinema? - S Anand 5 Jan 2022 3:35 pm (pingback): […] These are the major group actors based on who they act with most. […] I tested the best ways to mail people - S Anand 6 Jan 2022 11:32 am (pingback): […] write more about did 100% worse at #4 than #2 (4.4% down to 0%) […] Vidyasekar 21 Dec 2021 7:15 pm: I don't know whether it has a business value or not. But it gives a hammer shot on the head and looks at the Cinema industry and its behaviors. My Year in 2021 - S Anand 2 Jan 2022 1:36 pm (pingback): […] On learning, I discovered network clusters. My PyCon talk on movie networks is the start of a fascinating exploration of actors that I’ll write more about. […] Vinu 1 Jan 2022 1:08 am: Interesting data driven view of comparing the level of tribalism (if I can use that term ) within the various film industries around the world. I wonder, though, If casting decisions are directly made by the actors or the initial production and directorial leadership. some stars will command more of a say in whom they will co-Star with and with whom they won’t… but I wonder if that’s relevant to but a small fraction. Its good mental gymnastics, but at the core, your advice is good ;) How to find a Chinese actor to cast in Hollywood - S Anand 20 Feb 2022 12:38 pm (pingback): […] example, Hollywood actors act outside Hollywood just 10% of the time. Chinese actors act with non-Chinese actors just 1% of the […]

Can foreigners enter Hollywood?


Comicgen's new character builder creates an astonishing range of avatars, but the post honestly admits the harder problem is discovering who actually needs this flexibility and why.

Comicgen now has a very versatile character -- Aavatar. Pick your gender. Hairstyle. Emotion. Attire. Pose. Colors. Create your own character. (It's open-source.) https://gramener.com/comicgen/v1/ The variations are staggering. It throws up surprising ones too, like crossdressers or bald women in sarees 🙂 What I'm struggling with is: Who needs this? What could they do with it? I'm not sure. If you have ideas or know someone who might, please let me know 🙏 comics #apis #characterdesign #datacomic #visualstorytelling LinkedIn

Comicgen aavatar


Comicgen unexpectedly became a flagship Gramener property again when its Power BI plugin earned an Editor's Pick, extending comics into mainstream BI workflows.

When we started Comicgen -- a JavaScript library to generate comics - https://gramener.com/comicgen/ -- I didn't expect it to become the #1 page on the Gramener website. Today, we had another unexpected surprise. The Comicgen Power BI plugin is a Power BI Editor's Pick: https://powerbi.microsoft.com/en-us/blog/power-bi-march-2021-feature-summary/ It's open-source: https://appsource.microsoft.com/en-US/product/power-bi-visuals/wa200001420. Try it out. datacomics #comics #dataanalysis LinkedIn

Comicgen power bi plugin


Contronyms are delightfully unstable words whose opposite meanings coexist in a single term, making them a compact source of linguistic wonder.

Contronyms are words that have two meanings that are the opposite of each other. Sanction, for example, may mean restricting something (e.g. sanction against imports) or approving something (e.g. sanctioning imports). Scan may mean to look at cursorily (e.g. scan a document) or look at in detail (e.g. scan an X-Ray) Fine may mean excellent (e.g. fine wine) or average (e.g. the wine's fine). I enjoyed this list of 75 contronyms.

Contronyms


Cyborg scraping mixes manual scrolling with tiny browser-console scripts to extract hard-to-scrape LinkedIn data quickly, cheaply, and imperfectly—but often fast enough to matter.

Cyborg scraping LinkedIn has a page that shows the people who most recently followed you. At first, it shows just 20 people. But as you scroll, it keeps fetching the rest. I'd love to get the full list on a spreadsheet. I'm curious about: 1. What kind of people follow me? 2. Which of them has the most followers? 3. Who are my earliest followers? But first, I need to scrape this list. Normally, I'd spend a day writing a program. But I tried a different approach yesterday. Aside: it's easy to get bored in online meetings. I have a surplus of partially distracted time. So rather than writing code to save me time, I'd rather create simple tasks to keep me occupied. Like scrolling. So here's my workflow to scrape the list of followers. Step 1: Keep scrolling all the way to the bottom until you get all followers. Step 2: Press F12, open the Developer Tools - Console, and paste this code. Step 3: The name, headline, followers and link are now in the clipboard as JSON. Visit and paste it in "Select your input" under "Enter Data". Step 4: Click on the "Download Result" button. The JSON is converted into a CSV you can load into a spreadsheet. I call this "Cyborg scraping". I do half the work (scrolling, copy-pasting, etc.) The code does half the work. It's manual. It's a bit slow. But it gets the job done quick and dirty. I'll share later what I learned about my followers. For now, I'm looking forward to meetings 😉 PS: A similar script to scrape LinkedIn invitations is below. You can only see 100 invitations per page, though. PS: A similar script to scrape LinkedIn people search results is below. Comments mvark 8 Aug 2021 10:39 pm: I think rather than create JSON, building a comma separated string and console.log-ging that string will show the whole list in the Dev Tools Console. Using the Save As option in the context menu, the list can be saved as a .CSV file directly without Step 3 & 4. Tools to publish annotated talks from videos - S Anand 20 Oct 2024 12:53 pm (pingback): […] together. As you scroll, newer/older comments are loaded. So I needed to use my favorite technique: Cyborg Scraping. During Q&A, I kept scrolling to the bottom and […]

Cyborg scraping


This short note points data scientists toward a rare NGO role that combines legal and political data with the chance to publish visible, portfolio-worthy insights.

Daksh India is an NGO that works with legal & political data -- such as every single legal case in India. They're looking for a senior #datascientist who can explore such data and publish visual insights. What I like about such roles is that it allows you to publish your work and build your portfolio. You could apply at https://dakshindia.org/work-with-us/ (via Surya Prakash B S) LinkedIn

Daksh india data scientist job posting


The Data Comicgen awards showcase how difficult—and impressive—it is to turn raw data into simple, comic-style narratives that still carry insight.

Over 30 people have created data comic stories for the #DataComicgen awards. Getting insights from data is hard. Telling stories from these is harder. Telling comic stories is the hardest. Yet, these two dozen stories simplify data into simple (and even interactive) comic narratives. Hats off, participants. Hats off! https://gramener.com/comicgenfriday/awards/gallery2021/ LinkedIn

Data comicgen awards


The Data Comicgen event is pitched as a fun, collaborative challenge that combines Google Sheets analysis with comics-based storytelling and public sharing.

I'm really looking forward to this Data Comicgen #event. 1. Get the #data on 5 Aug 2. Analyze it with #googlesheets 3. Use Comicgen for #comics #storytelling 4. Submit on 26 Aug It's a great opportunity to find fellow data storytellers and comic enthusiasts -- to see their work and share yours. And win awards. LinkedIn

Data comicgen event


By tracing and merging simple shapes, PowerPoint can recreate complex icons like an address card, making it a viable substitute for Illustrator for many practical design tasks.

Designing Complex Shapes in PowerPoint I use PowerPoint instead of Adobe Illustrator or Sketch. I'm familiar with it, and it does everything I need. One of the features I'm really excited by in PowerPoint is the ability to manipulate shapes. Let's say you have a rectangle and a circle. You can select both of these shapes and in the Shape Format > Merge Shapes dropdown, you can: merge them with a union combine them (like an XOR operation in Boolean algebra) fragment them, which breaks them up into pieces intersect them subtract them This is so powerful that you can create any kind of shape. Let's take an icon from Font Awesome at random -- say an address card -- and create it. Here's the video of the process. I'll explain it step-by-step below. First, let's take a screenshot of this and copy it into PowerPoint. Now let's draw over this. So we'll start with a rounded rectangular box with the same color as the address card. We can use the eyedropper to pick the right color. Remove the outline. Then match the edges as closely as you can. (Add a bit of transparency so you can see through it -- that helps match edges closely.) Move this card boundary to a new page. Now, on top of the original image we copy-pasted from Font Awesome, trace 3 rounded rectangles for the address lines. Trace a circle over the head. Fill them white. Remove the outline. It should look like these. Next, let's create the body. We'll create a rounded rectangle that matches the bottom half of the body, another that matches the top half of the body, and intersect them, like this: Then, draw a large circle around the head and subtract it from the body, like this: Finally, copy all these shapes over the card boundary on the next page. Select the card boundary first. Then select these copied shapes (3 address lines, head, and bust). Select Shape Format > Merge Shapes > Subtract. With that, we have a single shape that contains the entire address card. The white areas are transparent. You can download the Merge-Shapes.pptx file below with each of the steps. Merge-Shapes (Download) Like I said, I don't bother with Adobe Illustrator or Sketch. PowerPoint does it all for me 😊.

Designing Complex Shapes in PowerPoint


This job post seeks a Chief Sales Officer who can combine value delivery, relentless selling, data-storytelling instinct, and scale ambition for a growing data company.

We are looking for a Chief Sales Officer. (For Princeton, New Jersey.) Do you ensure clients get value from what you sell? Do you have the energy to go after partners and logos relentlessly? Do you feel the thrill of using data to tell stories? Are you game to take a $10m data company to $50m? We would like to talk to you. Please visit 👇 https://gramener.com/job/?id=90554 LinkedIn

Gramener job post chief sales officer 2021


The launch of Gramex IDE marks a low-code push to make advanced data visualization creation accessible to people who would otherwise need to write code.

For 10 years, I've enjoyed building data visualizations by writing code. Today, we launched the Gramex IDE at https://gramex.gramener.com -- and it aims to make me redundant. Everyone can do the same, but without code. I'll miss the good old days, but looking forward to a brave new world of low code. LinkedIn

Gramex ide launch


This music quiz asks fans to identify Tamil films from Harris Jayaraj's first interludes, focusing on melody recognition rather than lyrics.

Harris Jayaraj First Interludes Here are beautiful first interludes from Harris Jayaraj 's Tamil film songs. Can you guess which movie they’re from? Don’t worry about the spelling. Just spell it like it sounds, and the box will turn green.

Harris Jayaraj First Interludes


A brief tutorial announcement shows how a Harry Potter screen-time bar chart race was built in PowerPoint, proving again that the tool is stranger and more capable than people assume.

The #HarryPotter Screen Time video featured a bar chart race made in PowerPoint. Many of you asked how I made it. Here's a step-by-step tutorial. https://youtu.be/E1SeoUV8awI (BTW, there are just 9 people on LinkedIn who match "bar chart race" and #powerpoint. This is for you, friends!) Gramener converted this feature into a product, SlideSense.ai. But more on that later 🙂 The original video is at https://www.linkedin.com/feed/update/urn%3Ali%3Aactivity%3A6801138072068214784/ LinkedIn

Harry potter bar chart race powerpoint


The author's first lightboard-style video combines on-camera explanation, a PowerPoint bar-chart race, and OBS to make data storytelling feel more live and physical.

This is my first Lightboard video -- interacting with a data visualization on-camera. Hans Rosling did something like this with BBC 4. That's been my inspiration. In this video, I share the screen time of the Harry Potter supporting cast. The bar chart race was created in PowerPoint, and the lightboard effect was created with OBS. (I'll share a tutorial soon.) What do you think of the video? Any comments? Questions? LinkedIn

Harry potter lightboard