data-extraction on S Anand

Sat, 15 Nov 2025 00:00:00 +0000

When I realized Aishwarya Rai begins and ends with AI, I had to find out if there were more like her.

It took a coding agent (Claude Code in this case) 10 minutes to find the 10 celebrities who share that distinction, at least across the 24,086 names on Wikipedia:

Ai Nagai - Japanese playwright
Aiguo Dai - Chinese-American atmospheric scientist
Ai (poet) - American poet
Aisea Nawai - Fijian rugby player
Ai (singer) - Japanese-American singer
Aisha Chughtai - Pakistani actress
Aiyappan Pillai - Indian social reformer
Aizawa Seishisai - Japanese Confucian scholar
Ainmuire mac Sétnai - Irish high king
Aisha Yousef al-Mannai - Qatari artist

Glory be to these AI bookends!

PS: It’s pretty cool that two celebrities are known just as “Ai”!

Vibe-Scraping: Write outcomes, not scrapers

Tue, 30 Sep 2025 13:18:20 +0000

There hasn’t been a box-office explosion like Dangal in the history of Bollywood. CPI inflation-adjusted to 2024, it is the only film in the ₹3,000 Cr club.

3 Idiots (2009) is the first member of the ₹1,000 Cr club (2024-inflation-adjusted). The hot streak was 2013-2017: each year, a film crossed that bar: Dhoom 3, PK, Bajrangi Bhaijaan, Dangal, Secret Superstar.

Since then, we never saw such a release except in 2023 (Jawan, Pathan).

But this story isn’t about the box-office drought. It’s about vibe-scraping.

To scrape the 1k Cr club data, here’s what my process would be in:

2008-2015. Requests + BeautifulSoup in Python. Takes ~1 day.
2015-2024. Puppeteer. Still takes ~1 day.
2025-Today. AI writes code. ~2 hours/site. 4x faster.
Today-???. Coding agents scrape directly. ~30 min/site. 16 times faster.

I passed Codex CLI (roughly) this prompt:

Write scrape.py to scrape the highest-grossing films from Wikipedia’s list of Hindi films: 1994 to 2024.
Read pages as required. Save results as CSV.

Here’s what it did.

Read the Wikipedia lists starting 1994
Failed on missing BeautifulSoup dependency. I allowed install.
Discovered that tables below “grossing” or “box office” headings are relevant.
Noticed “Rank” became “No” in the column header since 2016 and adapted.
Fixed all errors and generated a clean CSV.

That’s… incredible!

The code was a by-product. The prompt and evals matter. When sites change, agents can fix the code. Or better agents will rewrite it.

I guess I’ll call this vibe-scraping.

I also asked Claude Code vibe-code a data story. Here are the links:

This afternoon, in front of a client, I spoke with Codex:

Write a scrape.py that searches Dutch fashion merchant websites and lists what delivery carriers they use.

~10 minutes later, we had a table. The client spotted one error that I couldn’t have. Expert review still matters. But what’s redundant is my 20-year scraping experience!

If agents can scrape on the fly, what new questions do we ask?

Dissecting my Airtel bills

Tue, 11 Aug 2015 13:13:33 +0000

My monthly postpaid mobile bills have been in the Rs 2,000 – Rs 3,000 range for some time now, and I spent a few hours dissecting them yesterday.

Page 3 had the good stuff. It’s a little hard to figure out, but what the last 2 columns say is that most of my spend is offset by discounts.

What’s not getting offset are outgoing roaming calls. Followed by calls to local landlines. For all practical purposes, that’s the only thing that counts in this bill. Everything else is close enough to zero.

It took me some time to figure out that Airtel postpaid has something called myPlan. Based on your plan, you get set of “myPacks” or discounts. That determines your final bill. And it turns out that I was barely using my quota in some areas – specifically data. I had 3GB of data available. I was typically using 200MB – 500MB. The last 2 columns on page 2 show the usage of myPacks.

Clearly, I can do with less data, less SMS, less local mobile, and perhaps less STD mobile. I might need more outgoing roaming, but that’s about it. This means I need fewer myPacks. So I was able to switch to the Rs 799 plan from the Rs 999 plan, while simultaneously increasing the number of free outgoing roaming calls I can make.

There seems to be no myPack for incoming roaming, so I’m actually better off calling people if I’m travelling, rather than receiving calls!

The rest of the bill is a treasure-trove of data, listing every call and every pulse of data connection. I only wish it also had the location of the calls, and were available as CSV files.

Comments

Ashish 11 Aug 2015 6:31 pm: Ask Airtel to start 99/- per month national roaming pack to make all incoming roaming calls free.
408wij 11 Aug 2015 9:29 pm: A quick google search turned up two android apps: (1) Call Log Location and (2) Log Call Location. Perhaps their logs can be exported and cross-referenced w/ your bill.
Bala 1 Sep 2015 12:47 pm: You spend 1 day (>24 hours) in talking on your mobile.
Indrajit Ghosh 18 Jan 2017 1:57 pm: You can ask for a better Data deal from Airtel.

Scraping for a laptop

Sat, 14 Jan 2012 04:00:51 +0000

I’ve returned my laptop, and it’s time to buy a new one. For the first time in my life, I’m buying a laptop for myself.

I have a fairly clear idea of what I want: a 500GB+ 7200 rpm hard disk with 4GB of RAM and an Intel Core i7. I thought that would make finding one of those powerful laptops for producing music since I record some stuff too out of hobby.

Sheer naïveté. Not a single site let me filter by hard disk rpm in India. (To be fair, I haven’t found any sites outside India that did that either.)

After spending a good two hours hunting for the details and collating it, I did what I normally would: spend 30 minutes writing a scraper. The scraper runs through all laptops on Flipkart and pulls out all of their specs. Thanks to the diligence of the good folks at Flipkart, this information is readily available on each page. The HTML is structured quite neatly too, so it was just a 30-line program to scrape it all. Full credit to ScraperWiki as well — I could use it on a netbook without any developer tools installed.

The scraper took 2 hours to run. Feel free to filter through the output (CSV) for your favourite laptop, or fork the code and pull any other data you like.

Comments

ravi atluri 14 Jan 2012 4:19 am: If weight is not a limitation…i would recommend XPS15 or XPS15z i prefer it because of the i7 and >8GB RAM. gives me flexibility to run linux on VMs and 8 cores and 8GB RAM gives a lot of VMs :P if you are looking for portability. mac book pro :D it best fits for coding too
Subodh M 24 Jan 2012 1:05 pm: Anand, hv u explored Thinkpad T410i? I am using one with with an i5 processor with4GB RAM, 500GB HDD, @ 7200 rpm and am happy with it.
S Anand 14 Jan 2012 10:29 am: Battery life is the other consideration, and may be more important than performance. I am edging towards the Sony Vaio S VPCSA35GG. Despite the terrible name, it seems to have a decent performance battery-life trade off.
Indira 14 Jan 2012 5:40 pm: dell xps 14/15z comes with a good battery life. dell claims 7 hours, but i would think that would be with a slightly less than optimal settings of screen brightness, it would at least give you 4 hours, which is reasonable. Between sony and dell, my choice would be dell
S Anand 14 Jan 2012 5:59 pm: @Indira: thanks! Have added Dell XPS to my list. Also considering an Thinkpad E520 (i5, 4GB, 750GB) that seems very reasonably priced by a local vendor. I’ll try and avoid Sony.
Yuvi 14 Jan 2012 6:47 pm: SSD!
Somnath 14 Jan 2012 7:31 pm: If you are planning to buy from flipkart you may like to contact Ravi Vohra. He is a big shot at flipkart :D
S Anand 16 Jan 2012 1:25 pm: @pavan: agreed. I already have an external SSD though, so I just need a USB 3.0 port to take advantage of it. Can live with a more cost effective laptop :-)
pavan 16 Jan 2012 6:38 am: Anand, try to get a laptop with SSD, or upgrade later. Even the cheapest SSDs are 10 times faster than HDDs. Low noise and better battery life are nice to have side effects of SSD.
S.C 8 Aug 2012 3:20 pm: I had gone to a HP authorized service center, since browsers other than IE wouldn’t work on my 3+ yr. old HP Pavilion Multimedia laptop. The person in charge of this center said that my laptop had probably been infected with a virus and advised me to install Norton AV. He also offered to ‘upgrade’ the OS in it from the licensed version of Windows Vista to Windows 7. When I started working on the laptop at home, I found messages that the OS was an unlicensed one. A few days later, I found that the laptop was not booting in the first attempt. Even when it booted, I got ‘blue screen’ errors before booting and horizontal and vertical lines on the screen which sometimes intersected. On taking the laptop back to the service center, I was told that the NVIDIA graphics card on the laptop’s motherboard had malfunctioned. I want to ask users through this forum, whether this could have happened because of the extra load imposed on the 3+ yr. old graphics card by Windows 7? Since the personnel at the center now claimed that they could repair the graphics card for Rs. 1500, I gave them the laptop. But, when I collected it back after a week, I found no change in the laptop. So, I took it to the main HP service center in my city (where laptops covered by warranty are repaired). The personnel at this center said that they would have to replace the mother-board, since they didn’t do chip level repairing. So, in order to get a graphics card repaired, I would have to shell out Rs. 22K for a new mother-board. Though I accepted this, HP hit me on the head with their next condition, when they said that they could repair the laptop, only if I surrendered the existing mother-board (in the malfunctioning laptop) to them, free of cost. This was in spite of the fact, that they had not mentioned this when taking the laptop from me for repair. Since, I couldn’t have taken the earlier HP service center personnel to court for cheating me out of Rs. 1500 (in the name of repairing the graphics card), I refused and had to come back from the service center without getting the laptop repaired, in spite of pleading with HP to repair the mother-board for a month. They not only refused repeatedly, saying that this was their policy world wide and they couldn’t change it just for me, but added insult to injury, by charging me Rs. 300 for inspecting the laptop. The reason I am mentioning this incident here, is because I feel this is malafide on the part of HP. Since I pay for a laptop when I buy it, the motherboard in it becomes my property then and there. If it malfunctions, why should I have to surrender it, to get it replaced and that also free of cost? I can choose to do whatever I want with it, which in this case, will definitely be to take HP to court for deficiency of service and causing mental agony, which wouldn’t have been possible if I had surrendered the defective mother-board. I have heard that DELL is also taking customers for a ride in a similar way. Have any of you faced this?
S Anand 16 Jan 2012 1:27 pm: @Ravi, will reach out and take advice on how to get those discounts!
Deepak 15 Jan 2012 4:17 am: Aww, that is awesome. Can you include the price as well? :)
Sri 15 Jan 2012 9:26 am: Try to check the prices at Dell Web Site directly and speak to the sales guy as well, they may give you a better price. Also it is better to have a look at Croma where you get a chance to touch and feel the different products and you will get an idea of price.
Thejesh GN 15 Jan 2012 1:53 pm: I use Hp Probook 6460b which actually matches all your requirements. Unfortunately Flipkart has i5 one :(
S Anand 15 Jan 2012 3:08 pm: Thanks Sri, that’s a great idea. Will check Croma tomorrow.
S Anand 15 Jan 2012 3:28 pm: How are HP laptops, generally? I haven’t heard anything one way or the other so far…
ravi atluri 15 Jan 2012 6:20 pm: my dell xps15 gives me around 4.5 hours ..that’s a 9 cell battery and is pretty heavy :P XPS is more of a portable desktop than a portable one ;) I prefer Dell, because of their better service in India. and if you are buying Dell…there are tricks to get discounts ;) dont order online directly :D
How to Scrap Web Pages using ScraperWiki? | Python, ScraperWiki, HasGeek, Jobs | Sanspace Blog 28 Sep 2012 8:04 pm (pingback): […] data from web pages in a programmable way. For example, Check out Anand’s post about how he scraped for a laptop on Flipkart. That’s how I came to know about ScraperWiki and recently I wrote some quick and […]

Scraping RSS feeds using XPath

Mon, 17 Dec 2007 12:00:00 +0000

If a site doesn't have an RSS feed, your simplest option is to use Page2Rss, which gives a feed of what's changed on a page.

My needs, sometimes, are a bit more specific. For example, I want to track new movies on the IMDb Top 250. They don't offer a feed. I don't want to track all the other junk on that page. Just the top 250.

There's a standard called XPath. It can be used to search in an HTML document in a pretty straightforward way. Here are some examples:

//a	Matches all <a> links
//p/b	Matches all <b> bold items in a <p> para. (the <b> must be immediately under the <p>)
//table//a	Matches all links inside a table (the links need not be immediately inside the table -- anywhere inside the table works)

You get the idea. It's like a folder structure. / matches the a tag that's immediately below. // matches a tag that's somewhere below. You can play around with XPath using the Firefox XPath Checker add-on. Try it -- it's much easier to try it than to read the documentation.

The following XPath matches the IMDb Top 250 exactly.

//tr//tr//tr//td[3]//a

(It's a link inside the 3rd column in a table row in a table row in a table row.)

Now, all I need is to get something that converts that to an RSS feed. I couldn't find anything on the Web, so I wrote my own XPath server. The URL:

www.s-anand.net/xpath?
url=http://www.imdb.com/chart/top&
xpath=//tr//tr//tr//td[3]//a

When I subscribe to this URL on Google Reader, I get to know whenever there's a new movie on the IMDb Top 250.

This gives only the names of the movies, though, and I'd like the links as well. The XPath server supports this. It accepts a root XPath, and a bunch of sub-XPaths. So you can say something like:

xpath=//tr//tr//tr title->./td[3]//a link->./td[3]//a/@href

This says three things:

//tr//tr//tr	Pick all rows in a row in a row
title->./td[3]//a	For each row, set the title to the link text in the 3rd column
link->./td[3]//a	... and the link to the link href in the 3rd column

That provides a more satisfactory RSS feed -- one that I've subscribed to, in fact. Another one that I track is a list of . link->./@href">new popular movies that make it to the mininova top seeded movies category.

You can whiff up more complex examples. Give it a shot. Start simple, with something that works, and move up to what you need. Use XPath Checker liberally. Let me know if you have any isses. Enjoy!

Comments

Mark 17 Dec 2007 12:00 pm: Have you ever thought about introducing authentication to the XPath server? I would like to parse certain fields of a page that is authenticated with cookies.
Rog 28 Oct 2008 1:07 am: Any chance you could share your xpath.php code? It seems the server is no longer available.
S Anand 28 Oct 2008 1:43 am: Sure Rog. I’ve mailed it to you
S Anand 7 Mar 2009 10:35 am: Post Yahoo’s introduction of Yahoo Query Language, you’re much better off using that instead of my XPath utility. I’ve covered it in this article on client side scraping.
Scraping your way to RSS Feeds! « Technosiastic! 8 Apr 2009 10:16 am (pingback): […] another gem I figured which actually lets you run XPath query for scraping into a web page for RSS. It can be […]
Bart P 3 Mar 2012 11:48 am: It would be great if you could share this code, I really like to use this server, but want to remove session ids from the links (so my reader doesn’t think all links are new every time). Is that possible? :)
Ben 11 Jun 2015 12:46 am: Is that possible to share your xpath.php code? yahoo pipes is going to be shut down :(

Hacking using keyboard whispers

Wed, 12 May 2004 12:00:00 +0000

It’s possible to make sense out of keyboard whispers.