rss-feeds on S Anand

ImportHtml doesn’t auto-refresh

Fri, 12 Mar 2010 09:54:26 +0000

A cool thing about Google Spreadsheets is that you can scrape websites using external data functions like importHtml. It’s really easy to use. The formula:

=importHtml("http://www.imdb.com/chart/top", "table", 1)

imports the Internet Movie Database top 250 table on to Google Spreadsheets.

Since you can publish these as RSS feeds, it ought to, in theory, be a great way of generating RSS feeds out of arbitrary content.

There’s just one problem: it doesn’t auto update.

There are claims that it does every hour. Maybe it does when the sheet is open. I don’t know. But it definitely does not when the sheet is closed. I wrote a simple script that logs the time at which the script was accessed, and prints the log every time it is accessed.

#!/usr/bin/env python

import datetime, os.path

print 'Content-Type: text/plain; charset=utf-8'
print ''

logfile = 'timenow.log'
try:    timelog = open(logfile).readlines()
except: timelog = []
timelog.append(str(datetime.datetime.now()) + '\n')
open(logfile, 'w').writelines(timelog)
print ''.join(timelog)

Then I importHtml’ed it into Google spreadsheets, and left it on for the night. Result: absolutely no hits when the document is closed.

Pity. Guess YQL is still the best option.

Comments

pitching manual 24 Jun 2010 5:29 pm: I’ve just started using google spreadsheets. Really impressed with how user friendly they are. I went ine expecting the worse and was very suprised.
Barry 7 Jun 2012 6:27 pm: I am running into the same thing. Can’t get importHtml() to refresh when google spreadsheet is closed. Was thinking I could hammer force a cell value to equal the IH() function via a script but that seems awfully inelegant.
alois 6 Sep 2012 4:20 pm: so what happens when you add a parameter to the url with a value coming from a field which is set to show the current time - minute or hour ?

Lazy bargain hunting

Fri, 04 Jan 2008 12:00:00 +0000

I’m thinking of buying a digital keyboard with touch sensitive keys and MIDI support. (The one other thing that I thought off – a pitch bend – puts the keyboards out of my budget.)

I’d like a good deal. (Who doesn’t?) But I don’t like to spend time searching for one. (Who does?)

So here’s the plan.

Firstly, I’ll restrict my search to Amazon.co.uk. For electronics items, I haven’t found anyone consistently cheaper. Tesco has some pretty low prices, but not the range. eBuyer is pretty good, but not often enough. Google Products is the only other one that gets me consistent lower prices, but I’ve had my credit card identity stolen once before while shopping online, so I’d rather not pick any random seller listed on Google.

Amazon has a secret discount. You can search for electronics items with 30% off or more. And then you can narrow it down to Sound & Vision > Musical Instruments > MIDI Keyboards. Further cap a 100 - 200 GBP restriction. That leaves us with one product:

While that matches my criteria, I’m in no hurry and can wait for more offers to come up. But I don’t want to keep checking this page every day. So, RSS to the rescue. You probably think I can’t get enough of RSS feeds. And you’d be right. The thing is, as an attention mechanism, it is incredibly powerful, and I never cease to be amazed that the things it lets me do.

Using my XPath checker and a bit of trial and error, I figured all product links link to “amazon.co.uk/dp/…” with a <span> inside. So this XPath gets all the links:

//a[contains(@href,'/dp/')][span]

And I made an RSS feed out of that using my XPath server and subscribed to it on Google Reader.

Combining a bunch of such searches, I have a shopping folder on Google Reader has all the items I’m searching for. Now that’s lazy bargain hunting.

Which is all very fine. But given that I’m buying a car in a hurry right now, and I’m not doing any bargain hunting, it’s a classic case of being penny-wise and pound-foolish. Sigh…

Comments

Oracle 4 Jan 2008 12:00 pm: Alternate way is to set-up a google alert for the specific item in the chosen sites. Do you see any disadvantages with that?
Rajesh 11 Nov 2008 7:58 am: Never had someone’s personal website kept me so drawn in. You are smart and very articulate (u probably know this already) and it shows in each intersting and very often informative article you write. you can even try writing columns in magazines, websites or papers. i’ve learnt a lot from your site already and haven’t finished reading them all. i also see you play the keyboard and maybe interested in midis. i’ve sequenced some midis. u can can find them at

http://rajesh.annamalaisamy.googlepages.com/rajeshmidis

if u’re interested. The best tamil song midis in my opinion are done by kishmu. His midi works can be found at

www.geocities.com/kishmu/

Eagerly anticipating your next article.

Regards

Rajesh

Web lookup using Google Spreadsheets

Mon, 31 Dec 2007 12:00:00 +0000

I’d written earlier about Web lookup in Excel. I showed an example how you could create a movie wishlist that showed the links to the torrents from Mininova.

You can do that even easier on Google Spreadsheets. It has 4 functions that let you import external data:

=importData(“URL of CSV or TSV file”).
Imports a comma-separated or tab-separated file.
=importFeed(URL).vLets you import any Atom or RSS feed.
=importHtml(URL, “list” | “table”, index).
Imports a table or list from any web page.
=importXML(“URL”,“query”).
Imports anything from any web page using XPath.

Firstly, you can see straight off why it’s easy to view RSS feeds in Google Spreadsheets. Just use the importFeed function straight away. So, for example, if I wanted to track all 8GB iPods on Google Base, I can import its feed in Google Spreadsheets.

This automatically creates a list of the latest 8GB iPods.

Incidentally, the “Price” column doesn’t appear automatically. It’s a part of the description. But it’s quite easy to get the price using the standard Excel functions. Let’s say the description is in cell C1. =MID(C1, FIND("Price", C1), 20) gets you the 20 characters starting from “Price”. Then you can sort and play around as usual.

The other powerful thing about Google Spreadsheets is the CONTINUE function. The importFeed function creates a multi-dimensional array. You can extract any cell from the array (for example, row 3, column 2 from cell C1) using CONTINUE(C1, 3, 2). So you can just pick up the title and description, or only alternate rows, or put all rows and columns in a single column – whatever.

The most versatile of the import functions is the importXML function. It lets you import any URL (including an RSS feed), filtering only the XPath you need. As I mentioned earlier, you can scrape any site using XPath.

For example, =importXML("http://www.imdb.com/chart/top", "//table//table//table//a") imports the top 250 movies from the IMDb Top 250. the second parameter says, get all links (a) inside a table inside a table inside a table. This populates a list with the entire Top 250.

Now, against each of these, we could get a feed of Mininova’s torrents. Mininova’s RSS URL is http://www.mininova.org/rss/search_string. So, in cell B1, I can get a torrent for the cell A1 (The Godfather) using the importFeed function. (Note: you need to replace spaces with a + symbol. These functions don’t like invalid URLs.).

Just copy this formula down to get a torrent against each of the IMDb Top 250 movies!

Check out the sheet I’ve created. (You need a Google account to see the sheet. If you don’t want have one, you can view the sheet.)

Now, that’s still not the best of it. You can extract this file as an RSS feed! Google lets you publish your sheets as HTML, PDF, Text, XLS, etc. and RSS and Atom are included as well. Here’s the RSS feed for my sheet above.

Think about it. We now have an application that sucks in data from a web page, does a web-based vlookup on another service, and returns the results as an RSS feed!

There are only two catches to this. The first is that Google has restricted us to 50 import functions per sheet. So you can’t really have the IMDb Top 250 populated here – only the top 49. The second is that the spreadsheet updates only when you open it again. So it’s not really a dynamically updating feed. You need to open the spreadsheet to refresh it.

But if you really wanted these things, there’s always Yahoo! Pipes.

Comments

Gregory Dillon 31 Dec 2007 12:00 pm: Hey, thanks. Your page was the most helpful of all. In the example with the movie names, how did you figure out that it was a table x3. or table within table within table.
S Anand 30 Mar 2010 9:04 pm: Looks like the IMDb top 250 page format has changed. =importXML(“http://www.imdb.com/chart/top", “//div[@id=““main””]//table//font/a”) seems to work now.
Sintl 2 Jan 2011 4:24 am: Hi Is there a possibility to get selective text from webpage by using import url function. =ImportHtml(URL; “list” | “table”; index) I want to pull following text Updated Sunday, December 19, 2010 from a html page Thnx
Siamak 26 Aug 2011 11:03 pm: I have an excel file with the name of about 300 businesses that I want to look up their phone numbers or websites. Any idea how I can automate this, rather than type each entry in Google? Thanks
akc 30 Jul 2011 11:07 am: Hey Very Helpful website it is……….Please help me I wan to import data from (http://dsebd.org/latest_share_price_scroll_l.php ) . to google spreadsheet . I wan to import share price . Now tell me which way i will use. I m failed

Scraping RSS feeds using XPath

Mon, 17 Dec 2007 12:00:00 +0000

If a site doesn't have an RSS feed, your simplest option is to use Page2Rss, which gives a feed of what's changed on a page.

My needs, sometimes, are a bit more specific. For example, I want to track new movies on the IMDb Top 250. They don't offer a feed. I don't want to track all the other junk on that page. Just the top 250.

There's a standard called XPath. It can be used to search in an HTML document in a pretty straightforward way. Here are some examples:

//a	Matches all <a> links
//p/b	Matches all <b> bold items in a <p> para. (the <b> must be immediately under the <p>)
//table//a	Matches all links inside a table (the links need not be immediately inside the table -- anywhere inside the table works)

You get the idea. It's like a folder structure. / matches the a tag that's immediately below. // matches a tag that's somewhere below. You can play around with XPath using the Firefox XPath Checker add-on. Try it -- it's much easier to try it than to read the documentation.

The following XPath matches the IMDb Top 250 exactly.

//tr//tr//tr//td[3]//a

(It's a link inside the 3rd column in a table row in a table row in a table row.)

Now, all I need is to get something that converts that to an RSS feed. I couldn't find anything on the Web, so I wrote my own XPath server. The URL:

www.s-anand.net/xpath?
url=http://www.imdb.com/chart/top&
xpath=//tr//tr//tr//td[3]//a

When I subscribe to this URL on Google Reader, I get to know whenever there's a new movie on the IMDb Top 250.

This gives only the names of the movies, though, and I'd like the links as well. The XPath server supports this. It accepts a root XPath, and a bunch of sub-XPaths. So you can say something like:

xpath=//tr//tr//tr title->./td[3]//a link->./td[3]//a/@href

This says three things:

//tr//tr//tr	Pick all rows in a row in a row
title->./td[3]//a	For each row, set the title to the link text in the 3rd column
link->./td[3]//a	... and the link to the link href in the 3rd column

That provides a more satisfactory RSS feed -- one that I've subscribed to, in fact. Another one that I track is a list of . link->./@href">new popular movies that make it to the mininova top seeded movies category.

You can whiff up more complex examples. Give it a shot. Start simple, with something that works, and move up to what you need. Use XPath Checker liberally. Let me know if you have any isses. Enjoy!

Comments

Mark 17 Dec 2007 12:00 pm: Have you ever thought about introducing authentication to the XPath server? I would like to parse certain fields of a page that is authenticated with cookies.
Rog 28 Oct 2008 1:07 am: Any chance you could share your xpath.php code? It seems the server is no longer available.
S Anand 28 Oct 2008 1:43 am: Sure Rog. I’ve mailed it to you
S Anand 7 Mar 2009 10:35 am: Post Yahoo’s introduction of Yahoo Query Language, you’re much better off using that instead of my XPath utility. I’ve covered it in this article on client side scraping.
Scraping your way to RSS Feeds! « Technosiastic! 8 Apr 2009 10:16 am (pingback): […] another gem I figured which actually lets you run XPath query for scraping into a web page for RSS. It can be […]
Bart P 3 Mar 2012 11:48 am: It would be great if you could share this code, I really like to use this server, but want to remove session ids from the links (so my reader doesn’t think all links are new every time). Is that possible? :)
Ben 11 Jun 2015 12:46 am: Is that possible to share your xpath.php code? yahoo pipes is going to be shut down :(

Advanced Google Reader

Sun, 16 Dec 2007 12:00:00 +0000

I’ve stopped visiting websites.

No, really. There’s only one website I visit these days. Google Reader.

Google Reader is a feed reader. If you want to just catch up on the new stuff on a site, you can add the site to Google Reader. Anything new that is published on the site appears in Google Reader. Right now, I’ve subscribed to over 50 feeds. There’s no way I can remember to visit 50 sites – so I’m actually able to read more and miss less.

In a sense, that’s the essence of using feeds to read stuff: remember less, read more and miss less. If I ever find an interesting site, I just add it to Google Reader and forget about it.

But it goes beyond that. You can subscribe to search results.

Videos, for examples. Subscribe to a search for google engedu on Google Video. Any new TechTalk at Google, you get it in your Reader. (The talks are fabulous by the way.) Or search for hindi movies.

Photos? Track new photos of the Indian cricket team on Flickr.

Movies? Get an update of the latest movie torrents from mininova or torrentspy. Or just get updates on American Gangster DVDRip.

Presentations? Track anything on Javascript or consulting from SlideShare.

Documents? Find new eBooks on Scribd.

Not all pages offer a feed. But Page2Rss offers a reasonable solution. It converts pretty much any page into a feed.

It’s gotten to the point where anything I know I want to read, I put it on Google Reader. The rest of the Internet is when I don’t know what I want to read.

Comments

Jayan 16 Dec 2007 12:00 pm: Same here. I ve been using it for over 2 months now. I subscribe to over 200 feeds. This reader is coool.. - (http://ekalayva.wordpress.com)
Simon32 16 Dec 2007 12:00 pm: I agree, RSS is great! Btw, you might also want to checkout Feedity - www.feedity.com for creating free custom RSS feeds.

top 500 blogs

Tue, 23 Aug 2005 12:00:00 +0000

Top 500 blogs. Each month, Feedster brings you a list of 500 of the most interesting and important blogs. Enjoy browsing to see what people are reading, to find feeds that will bring topics of interest to you… and to discover new voices in the Blogosphere.

Blog experiments

Tue, 08 Feb 2005 12:00:00 +0000

I’m trying a couple of experiments with my blog. One is a feed of sorts that shows recently updated blogs. I’m conducting a trial with a few blogs I read, on the right-side tab “Recently updated blogs”. The last 3 posts of the blogs updated within 3 days one-and-half days show up here (using their RSS feed).

The other is creating an online catalog of digital resources I’ve encountered. Right now, I’ve put in books (fiction) I’ve read soft copies of on the right side tab “My fiction collection”. I’m planning to add non-fiction, music and movies.

The good part about having digital books, of course, is being able to lend them without worrying about losing my copy.

Comments

Navneet 8 Feb 2005 12:00 pm: You have digital copies of ALL these books? Where did you get them from?
Ravi 8 Feb 2005 12:00 pm: not able to see the “my fiction collection” list
S Anand 8 Feb 2005 12:00 pm: Bugs in the commenting system. Will fix them and let you know – including how to fix the TOPFRAME problem.

RSS feed

Tue, 25 Jan 2005 12:00:00 +0000

My RSS feed is back.