yql on S Anand

ImportHtml doesn’t auto-refresh

Fri, 12 Mar 2010 09:54:26 +0000

A cool thing about Google Spreadsheets is that you can scrape websites using external data functions like importHtml. It’s really easy to use. The formula:

=importHtml("http://www.imdb.com/chart/top", "table", 1)

imports the Internet Movie Database top 250 table on to Google Spreadsheets.

Since you can publish these as RSS feeds, it ought to, in theory, be a great way of generating RSS feeds out of arbitrary content.

There’s just one problem: it doesn’t auto update.

There are claims that it does every hour. Maybe it does when the sheet is open. I don’t know. But it definitely does not when the sheet is closed. I wrote a simple script that logs the time at which the script was accessed, and prints the log every time it is accessed.

#!/usr/bin/env python

import datetime, os.path

print 'Content-Type: text/plain; charset=utf-8'
print ''

logfile = 'timenow.log'
try:    timelog = open(logfile).readlines()
except: timelog = []
timelog.append(str(datetime.datetime.now()) + '\n')
open(logfile, 'w').writelines(timelog)
print ''.join(timelog)

Then I importHtml’ed it into Google spreadsheets, and left it on for the night. Result: absolutely no hits when the document is closed.

Pity. Guess YQL is still the best option.

Comments

pitching manual 24 Jun 2010 5:29 pm: I’ve just started using google spreadsheets. Really impressed with how user friendly they are. I went ine expecting the worse and was very suprised.
Barry 7 Jun 2012 6:27 pm: I am running into the same thing. Can’t get importHtml() to refresh when google spreadsheet is closed. Was thinking I could hammer force a cell value to equal the IH() function via a script but that seems awfully inelegant.
alois 6 Sep 2012 4:20 pm: so what happens when you add a parameter to the url with a value coming from a field which is set to show the current time - minute or hour ?

Client side scraping for contacts

Fri, 06 Mar 2009 17:01:56 +0000

By curious coincidence, just a day after my post on client side scraping, I had a chance to demo this to a client. They were making a contacts database. Now, there are two big problems with managing contacts.

Getting complete information
Keeping it up to date

Now, people happy to fill out information about themselves in great detail. If you look at the public profiles on LinkedIn, you’ll find enough and more details about most people.

Normally, when getting contact details about someone, I search for their name on Google with a “site:linkedin.com” and look at that information.

Could this be automated?

I spent a couple of hours and came up with a primitive contacts scraper. Click on the link, type in a name, and you should get the LinkedIn profile for that person. (Caveat: It’s very primitive. It works only for specific URL public profiles. Try ‘Peter Peverelli’ as an example.)

It uses two technologies. Google AJAX Search API and YQL. The search() function searches for a phrase…

google.load("search", "1");
google.setOnLoadCallback(function() {
  gs = new google.search.WebSearch();
  $("#getinfo").show();
});

function search(phrase, fn) {
  gs.setSearchCompleteCallback(gs, function() {
    fn(this.results);
  });
  gs.execute(phrase);
}

… and the linkedin() function takes a LinkedIn URL and extracts the relevant information from it, using XPath.

function scrape(url, xpath, fn) {
  $.getJSON("http://query.yahooapis.com/v1/public/yql?callback=?", {
    q: "select * from html where(url=\""
      + url + "\" and xpath=\"" + xpath + "\")",
    format: "json",
  }, fn);
}

function linkedin(url, fn) {
  scrape(url, "//li[@class][h3]", fn);
}

So if you wanted to find Peter Peverelli, it searches on Google for “Peter Peverelli site:linkedin.com” and picks the first result.

From this result, it displays all the <LI> tags which have a class and a <H3> element inside them (that’s what the //li[@class][h3] XPath does).

The real value of this is in bulk usage. When there’s a big list of contacts, you don’t need to scan each of them for updates. They can be automatically updated — even if all you know is the person’s name, and perhaps where they worked at some point in time.

Comments

Thejesh GN » RSS Feed Aamir Khans blog using YQL and Pipes 9 Mar 2009 2:44 pm (pingback): […] Blog to create a feed. I had written custom scraping code to create the feed.Today after reading Anand’s blog I parsed the blog using YQL and created the feed using Pipes. Using YQL/PIPE much easier thank […]
Prakash 8 Mar 2009 1:50 pm: Have you tried xobni add-in for outlook (incase you use that to manage contacts)?
S Anand 8 Mar 2009 4:07 pm: I don’t use it myself, but have tried it — and interestingly enough, recommended it to this client.
Thejesh GN 9 Mar 2009 2:46 pm: After reading this post. I redid Aamir Khans blog parsing using YQL and Pipes. It was very easy as YQL was available as module inside Pipes.
Vasanth Asokan 18 Dec 2010 6:05 am: Hi Anand, The scraper demo does not seem to be working (I tried the value you suggested as well ‘Peter Peverelli’. Can you take a look?
S Anand 18 Dec 2010 10:42 am: True, Vasanth. Honestly, scraping was only ever useful as a temporary strut until service providers opened up the data. It required too much maintenance anyway. With LinkedIn’s API, this is outdated, and I’m planning to leave it as it is. But thanks for pointing this out!

Client side scraping

Wed, 04 Mar 2009 18:00:57 +0000

“Scraping” is extracting content from a website. It’s often used to build something on top of the existing content. For example, I’ve built a site that tracks movies on the IMDb 250 by scraping content.

There are libraries that simplify scraping in most languages:

Perl: WWW::Mechanize
Python: BeautifulSoup
Ruby: HPricot
PHP: XPath (built-in)
Javascript: jQuery on env.js on Rhino

But all of these are on the server side. That is, the program scrapes from your machine. Can you write a web page where the viewer’s machine does the scraping?

Let’s take an example. I want to display Amazon’s bestsellers that cost less than $10. I could write a program that scrapes the site and get that information. But since the list updates hourly, I’ll have to run it every hour.

That may not be so bad. But consider Twitter. I want to display the latest iPhone tweets from http://search.twitter.com/search.atom?q=iPhone, but the results change so fast that your server can’t keep up.

Nor do you want it to. Ideally, your scraper should just be Javascript on your web page. Any time someone visits, their machine does the scraping. The bandwidth is theirs, and you avoid the popularity tax.

This is quite easily done using Yahoo Query Language. YQL converts the web into a database. All web pages are in a table called html, which has 2 fields: url and xpath. You can get IBM’s home page using:

select * from html where url="http://www.ibm.com"

Try it at Yahoo’s developer console. The whole page is loaded into the query.results element. This can be retrieved using JSONP. Assuming you have jQuery, try the following on Firebug. You should see the contents of IBM’s site on your page.

$.getJSON(
  "http://query.yahooapis.com/v1/public/yql?callback=?",
  {
    q: "select * from html where url=\"http://www.ibm.com\"",
    format: "json",
  },
  function(data) {
    console.log(data.query.results);
  },
);

That’s it! Now, it’s pretty easy to scrape, especially with XPath. To get the links on IBM’s page, just change the query to

select * from html where url="http://www.ibm.com" and xpath="//a"

Or to get all external links from IBM’s site:

select * from html where url="http://www.ibm.com" and xpath="//a[not(contains(@href,'ibm.com'))][contains(@href,'http')]""

Now you can display this on your own site, using jQuery.

This leads to interesting possibilities, such as Map-Reduce in the browser. Here’s one example. Each movie on the IMDb (e.g. The Dark Knight) comes with a list of recommendations (like this). I want to build a repository of recommendations based on the IMDb Top 250. So here’s the algorithm. First, I’ll get the IMDb Top 250 using:

select * from html where url="http://www.imdb.com/chart/top" and xpath="//tr//tr//tr//td[3]//a"

Then I’ll get a random movie’s recommendations like this:

select * from html where url="http://www.imdb.com/title/tt0468569/recommendations" and xpath="//td/font//a[contains(@href,'/title/')]"

Then I’ll send off the results to my aggregator.

Check out the full code at http://250.s-anand.net/build-reco.js.

In fact, if you visited my IMDb Top 250 tracker, you already ran this code. You didn’t know it, but you just shared a bit of your bandwidth and computation power with me. (Thank you.)

And, if you think a little further, here another way of monetising content: by borrowing a bit of the user’s computation power to build complex tasks. There already are startups built around this concept.

Comments

Aravinda 4 Mar 2009 7:20 pm: Great!
Sriram 4 Mar 2009 10:51 pm: Thank u anand. Nice Information.
Rishi 5 Mar 2009 8:19 am: Awesome… This is the future
grep imdb part 2 ? « taeyoungchoon 26 Mar 2009 1:16 pm (pingback): […] http://www.s-anand.net/blog/client-side-scraping/ […]
Khair 26 May 2009 4:10 pm: Many thanks for your efforts in simplifying the ‘scraping’.
Venkat 16 Nov 2010 1:40 pm: Hey Anand, Thanks. This post is really useful. I have managed to scrape Amazon wishlist using this technique (http://www.venkatsworld.com/WIP/JSON_amazon.html) However YQL seems to be taking some time to respond. I am not sure if you experienced this.I did a similar exercise with Picasa JSON feed and a Jquery plug-in for image scrolling. It works well there too.
Zheka 17 Jun 2012 8:35 pm: Interesting article! There is another powerful scraping technology available that offers Javascript, jQuery, CSS, and XPath instead of XPath-only. It’s called Bobik (http://usebobik.com). The is a cool example of scraping restaurant menus using Bobik at http://news.ycombinator.com/item?id=4066478.
LAvesh 12 Mar 2013 11:19 pm: Hey Anand, Great Article. Just one query though in the client side implementation its the client IP that will be hitting the website (in your case www.ibm.com)? or it will use yahoo SQL server IP to hit.

Scraping RSS feeds using XPath

Mon, 17 Dec 2007 12:00:00 +0000

If a site doesn't have an RSS feed, your simplest option is to use Page2Rss, which gives a feed of what's changed on a page.

My needs, sometimes, are a bit more specific. For example, I want to track new movies on the IMDb Top 250. They don't offer a feed. I don't want to track all the other junk on that page. Just the top 250.

There's a standard called XPath. It can be used to search in an HTML document in a pretty straightforward way. Here are some examples:

//a	Matches all <a> links
//p/b	Matches all <b> bold items in a <p> para. (the <b> must be immediately under the <p>)
//table//a	Matches all links inside a table (the links need not be immediately inside the table -- anywhere inside the table works)

You get the idea. It's like a folder structure. / matches the a tag that's immediately below. // matches a tag that's somewhere below. You can play around with XPath using the Firefox XPath Checker add-on. Try it -- it's much easier to try it than to read the documentation.

The following XPath matches the IMDb Top 250 exactly.

//tr//tr//tr//td[3]//a

(It's a link inside the 3rd column in a table row in a table row in a table row.)

Now, all I need is to get something that converts that to an RSS feed. I couldn't find anything on the Web, so I wrote my own XPath server. The URL:

www.s-anand.net/xpath?
url=http://www.imdb.com/chart/top&
xpath=//tr//tr//tr//td[3]//a

When I subscribe to this URL on Google Reader, I get to know whenever there's a new movie on the IMDb Top 250.

This gives only the names of the movies, though, and I'd like the links as well. The XPath server supports this. It accepts a root XPath, and a bunch of sub-XPaths. So you can say something like:

xpath=//tr//tr//tr title->./td[3]//a link->./td[3]//a/@href

This says three things:

//tr//tr//tr	Pick all rows in a row in a row
title->./td[3]//a	For each row, set the title to the link text in the 3rd column
link->./td[3]//a	... and the link to the link href in the 3rd column

That provides a more satisfactory RSS feed -- one that I've subscribed to, in fact. Another one that I track is a list of . link->./@href">new popular movies that make it to the mininova top seeded movies category.

You can whiff up more complex examples. Give it a shot. Start simple, with something that works, and move up to what you need. Use XPath Checker liberally. Let me know if you have any isses. Enjoy!

Comments

Mark 17 Dec 2007 12:00 pm: Have you ever thought about introducing authentication to the XPath server? I would like to parse certain fields of a page that is authenticated with cookies.
Rog 28 Oct 2008 1:07 am: Any chance you could share your xpath.php code? It seems the server is no longer available.
S Anand 28 Oct 2008 1:43 am: Sure Rog. I’ve mailed it to you
S Anand 7 Mar 2009 10:35 am: Post Yahoo’s introduction of Yahoo Query Language, you’re much better off using that instead of my XPath utility. I’ve covered it in this article on client side scraping.
Scraping your way to RSS Feeds! « Technosiastic! 8 Apr 2009 10:16 am (pingback): […] another gem I figured which actually lets you run XPath query for scraping into a web page for RSS. It can be […]
Bart P 3 Mar 2012 11:48 am: It would be great if you could share this code, I really like to use this server, but want to remove session ids from the links (so my reader doesn’t think all links are new every time). Is that possible? :)
Ben 11 Jun 2015 12:46 am: Is that possible to share your xpath.php code? yahoo pipes is going to be shut down :(