There hasn’t been a box-office explosion like Dangal in the history of Bollywood. CPI inflation-adjusted to 2024, it is the only film in the ₹3,000 Cr club. 3 Idiots (2009) is the first member of the ₹1,000 Cr club (2024-inflation-adjusted). The hot streak was 2013-2017: each year, a film crossed that bar: Dhoom 3, PK, Bajrangi Bhaijaan, Dangal, Secret Superstar. Since then, we never saw such a release except in 2023 (Jawan, Pathan). ...
This week, I learned: DuckDB is 2-10 times faster than Pandas. ClickHouse is supposedly faster but doesn’t run on Windows. Claude 3 Haiku input costs is $0.25/MTok. That’s half the GPT-3.5 cost. If it’s of comparable quality, it’s worth switching. But Claude 3 Opus is comparable to GPT-4 and twice the cost, so not worth it. Tavily is a search API for LLMs Interesting model garden models There are sites you TRULY cannot scrape even in the browser because of the isTrusted read-only property of events that you can never set to true. Oracle Service Cloud checks for isTrusted in mouse actions.
This week, I learned: Gwern Branwen says LLMs nudge his “… making heavier use of the languages I don’t know well (Emacs Lisp & Python) since I increasingly trust that an LLM can help me maintain them.” Undetectable.ai checks for AI content. But it had false positives AND negatives in the 5 checks I ran. GPTZero got 2/2 right and seems better at detecting AI content. CoVA scrapes web pages via OCR When coding with LLMs, have SHORT, RELIABLE feedback loops. Ref
LinkedIn has a page that shows the people who most recently followed you. At first, it shows just 20 people. But as you scroll, it keeps fetching the rest. I’d love to get the full list on a spreadsheet. I’m curious about: What kind of people follow me? Which of them has the most followers? Who are my earliest followers? But first, I need to scrape this list. Normally, I’d spend a day writing a program. But I tried a different approach yesterday. ...
I’m not that difficult to scare, and this log message certainly didn’t help: ip223.hichina.com [223.4.183.127] failed - POSSIBLE BREAK-IN ATTEMPT! That’s the message I saw – one thousand five hundred and seventy times yesterday in /var/log/auth.log on one of my Amazon EC2 instances. Someone, presumably from China, has been patiently trying out a variety of SSH keys to log into this system. These were grouped as batches. There were exactly 314 attempts at 8am yesterday, then 314 at 12noon, then 314 at 4pm, then 314 at 8pm, then 232 at 3am today. (All times are in UTC – that is, UK time without daylight saving). Every burst took 9 minutes to run through all 314 attempts. The worst part was, when I tried using SSH this morning, I wasn’t able to log in. (It turned out that I had made a configuration error, but this is the sort of thing that gets me quite worried.) Perhaps I shouldn’t be complaining. I’ve written enough scrapers to make most webmasters cringe at their logs. I remember a few years ago, when I was working on a project at Tesco, and was scraping bestsellers lists from most sites. (Here’s a blog post about it.) We were putting together a prototype to see how real-time competitive pricing could help. The scraper was a pretty mild one. It would visit a hundred links, roughly at the pace of one a second. No images were loaded, of course, just the HTML. One fine day, a few weeks after this had started, I got a call from Andy. “Hi Anand, are you running any scrapers on our books website?” “Yes, why?” “Oh! The site’s very slow. Could you shut it down immediately?” Turns out that not a single page on the site loaded, and it had almost crawled to a halt. Now, obviously, my little 100-page script could hardly cause damage, but it’s easy to understand their reactions. No unauthorised scraping! After a few days of trying to figure out what the problem was, they increased the memory and things went back to normal. Not a bad solution, actually – throw hardware at the problem, and if it vanishes, it’s probably the cheapest solution. But anyway, I’m sure it’s some nice chap who’s just curious to know what I’ve got on my servers. I’d be happy to share some of it. And even if it’s not so nice a chap, there’s little that I can do, is there? Update (1pm India, 3rd June): Actually, I now realise that this has been happening ever four hours since May 29th, as regular as a clockwork. Wish I knew enough UNIX programming to pull a prank… ...
I’ve returned my laptop, and it’s time to buy a new one. For the first time in my life, I’m buying a laptop for myself. I have a fairly clear idea of what I want: a 500GB+ 7200 rpm hard disk with 4GB of RAM and an Intel Core i7. I thought that would make finding one of those powerful laptops for producing music since I record some stuff too out of hobby. Sheer naïveté. Not a single site let me filter by hard disk rpm in India. (To be fair, I haven’t found any sites outside India that did that either.) ...
I just put together this little demo that scrapes John Lewis’ site and creates a MediaRSS file out of it. CoolIris has got to be the best way to shop. Apart from being really pretty, it’s quite useful when you know what something looks like, but don’t quite know how to search for it. For example, I was trying to look for a headphone-microphone (you know, the ones that connect into an iPhone or a Blackberry). I didn’t have a clue what it’s called. (TRRS, if you’re interested. I found out later.) The only way I could get it was to browse the wall… ...
A cool thing about Google Spreadsheets is that you can scrape websites using external data functions like importHtml. It’s really easy to use. The formula: =importHtml("http://www.imdb.com/chart/top", "table", 1) imports the Internet Movie Database top 250 table on to Google Spreadsheets. Since you can publish these as RSS feeds, it ought to, in theory, be a great way of generating RSS feeds out of arbitrary content. There’s just one problem: it doesn’t auto update. There are claims that it does every hour. Maybe it does when the sheet is open. I don’t know. But it definitely does not when the sheet is closed. I wrote a simple script that logs the time at which the script was accessed, and prints the log every time it is accessed. ...
“Scraping” is extracting content from a website. It’s often used to build something on top of the existing content. For example, I’ve built a site that tracks movies on the IMDb 250 by scraping content. There are libraries that simplify scraping in most languages: Perl: WWW::Mechanize Python: BeautifulSoup Ruby: HPricot PHP: XPath (built-in) Javascript: jQuery on env.js on Rhino But all of these are on the server side. That is, the program scrapes from your machine. Can you write a web page where the viewer’s machine does the scraping? ...
I’m thinking of buying a digital keyboard with touch sensitive keys and MIDI support. (The one other thing that I thought off – a pitch bend – puts the keyboards out of my budget.) I’d like a good deal. (Who doesn’t?) But I don’t like to spend time searching for one. (Who does?) So here’s the plan. Firstly, I’ll restrict my search to Amazon.co.uk. For electronics items, I haven’t found anyone consistently cheaper. Tesco has some pretty low prices, but not the range. eBuyer is pretty good, but not often enough. Google Products is the only other one that gets me consistent lower prices, but I’ve had my credit card identity stolen once before while shopping online, so I’d rather not pick any random seller listed on Google. ...
I’d written earlier about Web lookup in Excel. I showed an example how you could create a movie wishlist that showed the links to the torrents from Mininova. You can do that even easier on Google Spreadsheets. It has 4 functions that let you import external data: =importData(“URL of CSV or TSV file”). Imports a comma-separated or tab-separated file. =importFeed(URL).vLets you import any Atom or RSS feed. =importHtml(URL, “list” | “table”, index). Imports a table or list from any web page. =importXML(“URL”,“query”). Imports anything from any web page using XPath. Firstly, you can see straight off why it’s easy to view RSS feeds in Google Spreadsheets. Just use the importFeed function straight away. So, for example, if I wanted to track all 8GB iPods on Google Base, I can import its feed in Google Spreadsheets. ...
If a site doesn't have an RSS feed, your simplest option is to use Page2Rss, which gives a feed of what's changed on a page. My needs, sometimes, are a bit more specific. For example, I want to track new movies on the IMDb Top 250. They don't offer a feed. I don't want to track all the other junk on that page. Just the top 250. There's a standard called XPath. It can be used to search in an HTML document in a pretty straightforward way. Here are some examples: ...
The technique of Web lookups in Excel I described yesterday is very versatile. I will be running through some of the practical uses it can be put to over the next few days TO generalise things beyond just getting the Amazon price, I created a user-defined function called XPATH. It takes two parameters: URL of the XML feed to read Search XPath list string (separated by spaces) This function can be used to extract information out of any XML file on the Web and get it out as a table. For example, if you wanted to watch the Top 10 movies on the IMDb Top 250, and were looking for torrents, an RSS feed is available from mininova. The URL http://www.mininova.org/rss/movie_name/4 gives you an RSS file matching all movies with “movie_name”. From this, we need to extract the and <item><link> elements. That’s represented by “//item title link” on my search string. ...
Click here to search for Hindi songs. This is an article on how I wrote the search engine. I find it a nuisance to have to go to Raaga, search for a song, not find it, then go to MusicIndiaOnline, not find it, then go to Musicplug.in, and so on until Google. So I got the list of songs from some of these sites, put it together in one place, and implemented a find-as-you-type. ...
I watch a lot of movies. Over the last year, I’ve watched over 250 movies (and read 50 books, but that’s another story). Other than making time to watch movies, my biggest problem is figuring out what to watch next. The IMDb top 250 is a good guideline, and I’m running my way down the list. Twofifty.org has been useful to track what I’ve seen as well. But I have interests outside of the IMDb Top 250, and I need a way of tracking these. ...
Hacking Google Print