xpath on S Anand

Geocoding in Excel

Wed, 08 May 2013 09:17:03 +0000

It’s easy to convert addresses into latitudes and longitudes into addresses in Excel. Here’s the Github project with a downloadable Excel file.

This is via Visual Basic code for a GoogleGeocode function that geocodes addresses.

Function GoogleGeocode(address As String) As String
    Dim xDoc As New MSXML2.DOMDocument
    xDoc.async = False
    xDoc.Load ("http://maps.googleapis.com/maps/api/geocode/" + _
        "xml?address=" + address + "&sensor=false")
    If xDoc.parseError.ErrorCode <> 0 Then
        GoogleGeocode = xDoc.parseError.reason
    Else
        xDoc.setProperty "SelectionLanguage", "XPath"
        lat = xDoc.SelectSingleNode("//lat").Text
        lng = xDoc.SelectSingleNode("//lng").Text
        GoogleGeocode = lat & "," & lng
    End If
End Function

Comments

Ryan 8 Jun 2015 9:28 pm: I find this isn’t working and says, Compile Error; User defined type not defined xDoc As New MSXML2.DOMDocument what do I change to fix it? Thank you
Richie Lionell 27 Jul 2016 6:40 am: Ryan, Inside the VBE, Go to Tools -> References, then Select Microsoft XML, v6.0 . If that doesn’t work unselect that and select Microsoft XML, v3.0

Client side scraping for contacts

Fri, 06 Mar 2009 17:01:56 +0000

By curious coincidence, just a day after my post on client side scraping, I had a chance to demo this to a client. They were making a contacts database. Now, there are two big problems with managing contacts.

Getting complete information
Keeping it up to date

Now, people happy to fill out information about themselves in great detail. If you look at the public profiles on LinkedIn, you’ll find enough and more details about most people.

Normally, when getting contact details about someone, I search for their name on Google with a “site:linkedin.com” and look at that information.

Could this be automated?

I spent a couple of hours and came up with a primitive contacts scraper. Click on the link, type in a name, and you should get the LinkedIn profile for that person. (Caveat: It’s very primitive. It works only for specific URL public profiles. Try ‘Peter Peverelli’ as an example.)

It uses two technologies. Google AJAX Search API and YQL. The search() function searches for a phrase…

google.load("search", "1");
google.setOnLoadCallback(function() {
  gs = new google.search.WebSearch();
  $("#getinfo").show();
});

function search(phrase, fn) {
  gs.setSearchCompleteCallback(gs, function() {
    fn(this.results);
  });
  gs.execute(phrase);
}

… and the linkedin() function takes a LinkedIn URL and extracts the relevant information from it, using XPath.

function scrape(url, xpath, fn) {
  $.getJSON("http://query.yahooapis.com/v1/public/yql?callback=?", {
    q: "select * from html where(url=\""
      + url + "\" and xpath=\"" + xpath + "\")",
    format: "json",
  }, fn);
}

function linkedin(url, fn) {
  scrape(url, "//li[@class][h3]", fn);
}

So if you wanted to find Peter Peverelli, it searches on Google for “Peter Peverelli site:linkedin.com” and picks the first result.

From this result, it displays all the <LI> tags which have a class and a <H3> element inside them (that’s what the //li[@class][h3] XPath does).

The real value of this is in bulk usage. When there’s a big list of contacts, you don’t need to scan each of them for updates. They can be automatically updated — even if all you know is the person’s name, and perhaps where they worked at some point in time.

Comments

Thejesh GN » RSS Feed Aamir Khans blog using YQL and Pipes 9 Mar 2009 2:44 pm (pingback): […] Blog to create a feed. I had written custom scraping code to create the feed.Today after reading Anand’s blog I parsed the blog using YQL and created the feed using Pipes. Using YQL/PIPE much easier thank […]
Prakash 8 Mar 2009 1:50 pm: Have you tried xobni add-in for outlook (incase you use that to manage contacts)?
S Anand 8 Mar 2009 4:07 pm: I don’t use it myself, but have tried it — and interestingly enough, recommended it to this client.
Thejesh GN 9 Mar 2009 2:46 pm: After reading this post. I redid Aamir Khans blog parsing using YQL and Pipes. It was very easy as YQL was available as module inside Pipes.
Vasanth Asokan 18 Dec 2010 6:05 am: Hi Anand, The scraper demo does not seem to be working (I tried the value you suggested as well ‘Peter Peverelli’. Can you take a look?
S Anand 18 Dec 2010 10:42 am: True, Vasanth. Honestly, scraping was only ever useful as a temporary strut until service providers opened up the data. It required too much maintenance anyway. With LinkedIn’s API, this is outdated, and I’m planning to leave it as it is. But thanks for pointing this out!

Client side scraping

Wed, 04 Mar 2009 18:00:57 +0000

“Scraping” is extracting content from a website. It’s often used to build something on top of the existing content. For example, I’ve built a site that tracks movies on the IMDb 250 by scraping content.

There are libraries that simplify scraping in most languages:

Perl: WWW::Mechanize
Python: BeautifulSoup
Ruby: HPricot
PHP: XPath (built-in)
Javascript: jQuery on env.js on Rhino

But all of these are on the server side. That is, the program scrapes from your machine. Can you write a web page where the viewer’s machine does the scraping?

Let’s take an example. I want to display Amazon’s bestsellers that cost less than $10. I could write a program that scrapes the site and get that information. But since the list updates hourly, I’ll have to run it every hour.

That may not be so bad. But consider Twitter. I want to display the latest iPhone tweets from http://search.twitter.com/search.atom?q=iPhone, but the results change so fast that your server can’t keep up.

Nor do you want it to. Ideally, your scraper should just be Javascript on your web page. Any time someone visits, their machine does the scraping. The bandwidth is theirs, and you avoid the popularity tax.

This is quite easily done using Yahoo Query Language. YQL converts the web into a database. All web pages are in a table called html, which has 2 fields: url and xpath. You can get IBM’s home page using:

select * from html where url="http://www.ibm.com"

Try it at Yahoo’s developer console. The whole page is loaded into the query.results element. This can be retrieved using JSONP. Assuming you have jQuery, try the following on Firebug. You should see the contents of IBM’s site on your page.

$.getJSON(
  "http://query.yahooapis.com/v1/public/yql?callback=?",
  {
    q: "select * from html where url=\"http://www.ibm.com\"",
    format: "json",
  },
  function(data) {
    console.log(data.query.results);
  },
);

That’s it! Now, it’s pretty easy to scrape, especially with XPath. To get the links on IBM’s page, just change the query to

select * from html where url="http://www.ibm.com" and xpath="//a"

Or to get all external links from IBM’s site:

select * from html where url="http://www.ibm.com" and xpath="//a[not(contains(@href,'ibm.com'))][contains(@href,'http')]""

Now you can display this on your own site, using jQuery.

This leads to interesting possibilities, such as Map-Reduce in the browser. Here’s one example. Each movie on the IMDb (e.g. The Dark Knight) comes with a list of recommendations (like this). I want to build a repository of recommendations based on the IMDb Top 250. So here’s the algorithm. First, I’ll get the IMDb Top 250 using:

select * from html where url="http://www.imdb.com/chart/top" and xpath="//tr//tr//tr//td[3]//a"

Then I’ll get a random movie’s recommendations like this:

select * from html where url="http://www.imdb.com/title/tt0468569/recommendations" and xpath="//td/font//a[contains(@href,'/title/')]"

Then I’ll send off the results to my aggregator.

Check out the full code at http://250.s-anand.net/build-reco.js.

In fact, if you visited my IMDb Top 250 tracker, you already ran this code. You didn’t know it, but you just shared a bit of your bandwidth and computation power with me. (Thank you.)

And, if you think a little further, here another way of monetising content: by borrowing a bit of the user’s computation power to build complex tasks. There already are startups built around this concept.

Comments

Aravinda 4 Mar 2009 7:20 pm: Great!
Sriram 4 Mar 2009 10:51 pm: Thank u anand. Nice Information.
Rishi 5 Mar 2009 8:19 am: Awesome… This is the future
grep imdb part 2 ? « taeyoungchoon 26 Mar 2009 1:16 pm (pingback): […] http://www.s-anand.net/blog/client-side-scraping/ […]
Khair 26 May 2009 4:10 pm: Many thanks for your efforts in simplifying the ‘scraping’.
Venkat 16 Nov 2010 1:40 pm: Hey Anand, Thanks. This post is really useful. I have managed to scrape Amazon wishlist using this technique (http://www.venkatsworld.com/WIP/JSON_amazon.html) However YQL seems to be taking some time to respond. I am not sure if you experienced this.I did a similar exercise with Picasa JSON feed and a Jquery plug-in for image scrolling. It works well there too.
Zheka 17 Jun 2012 8:35 pm: Interesting article! There is another powerful scraping technology available that offers Javascript, jQuery, CSS, and XPath instead of XPath-only. It’s called Bobik (http://usebobik.com). The is a cool example of scraping restaurant menus using Bobik at http://news.ycombinator.com/item?id=4066478.
LAvesh 12 Mar 2013 11:19 pm: Hey Anand, Great Article. Just one query though in the client side implementation its the client IP that will be hitting the website (in your case www.ibm.com)? or it will use yahoo SQL server IP to hit.

JPath - XPath for Javascript

Thu, 29 May 2008 12:00:00 +0000

XPath is a neat way of navigating deep XML structures. It's like using a directory structure. /table//td gets all the TDs somewhere below TABLE.

Usually, you don't need this sort of a thing for data structures, particularly in JavaScript. Something like table.td would already work. But sometimes, it does help to have something like XPath even for data structures, so I built a simple XPath-like processor for Javascript called JPath.

Here are some examples of how it would work:

jpath(context, "para")	returns context.para
jpath(context, "*")	returns all values of context (for both arrays and objects)
jpath(context, "para[0]")	returns context.para[0]
jpath(context, "para[last()]")	returns context.para[context.para.length]
jpath(context, "*/para")	returns context[all children].para
jpath(context, "/doc/chapter[5]/section[2]")	returns context.doc.chapter[5].section[2]
jpath(context, "chapter//para")	returns all para elements inside context.chapter
jpath(context, "//para")	returns all para elements inside context
jpath(context, "//olist/item")	returns all olist.item elements inside context
jpath(context, ".")	returns the context
jpath(context, ".//para")	same as //para
jpath(context, "//para/..")	returns the parent of all para elements inside context

Some caveats:

This is an implementation of the abbreviated syntax of XPath. You can't use axis::nodetest
No functions are supported other than last()
Only node name tests are allowed, no nodetype tests. So you can't do text() and node()
Indices are zero-based, not 1-based

There are a couple of reasons why this sort of thing is useful.

Extracting attributes deep down. Suppose you had an array of arrays, and you wanted the first element of each array.

You could do this the long way:
```
for (var list = [], i = 0; i < data.length; i++) {
 list.push(data[i][0]);
}
```
... or the short way:
```
$.map(data, function(v) {
 return v[1];
});
```
But the best would be something like:
```
jpath(data, "//1");
```

Ragged data structures. Take for example the results from Google's AJAX feed API.

{"responseData": {
 "feed": {
  "title": "Digg",
  "link": "http://digg.com/",
  "author": "",
  "description": "Digg",
  "type": "rss20",
  "entries": [
   {
    "title": "The Pirate Bay Moves Servers to Egypt Due to Copyright Laws",
    "link": "http://digg.com/tech_news/The_Pirate_Bay_Moves_Servers_to_Egypt_Due_to_Copyright_Laws",
    "author": "",
    "publishedDate": "Mon, 31 Mar 2008 23:13:33 -0700",
    "contentSnippet": "Due to the new copyright legislation that are going ...",
    "content": "Due to the new copyright legislation that are going to take...",
    "categories": [
    ]
   },
   {
    "title": "Millions Dead/Dying in Recent Mass-Rick-Rolling by YouTube.",
    "link": "http://digg.com/comedy/Millions_Dead_Dying_in_Recent_Mass_Rick_Rolling_by_YouTube",
    "author": "",
    "publishedDate": "Mon, 31 Mar 2008 22:53:30 -0700",
    "contentSnippet": "Click on any \u0022Featured Videos\u0022. When will the insanity stop?",
    "content": "Click on any \u0022Featured Videos\u0022. When will the insanity stop?",
    "categories": [
    ]
   },
   ...
  ]
 }
}
, "responseDetails": null, "responseStatus": 200}

If you wanted all the title entries, including the feed title, the choice is between:

var titles = [ result.feed.title ];
for (var i=0, l=result.feed.entries.length; i<l; i++) {
    titles.push(result.feed.entries[i].title;
}

... versus...

titles = jpath(result, "//title");

If, further, you wanted the list of all categories at one shot, you could use:

jpath(result, "//categories/*");

Comments

Will 7 Mar 2010 1:15 pm: See also: http://www.w3schools.com/xpath/tryit.asp?filename=try_xpath_select_pricenodes_text

Lazy bargain hunting

Fri, 04 Jan 2008 12:00:00 +0000

I’m thinking of buying a digital keyboard with touch sensitive keys and MIDI support. (The one other thing that I thought off – a pitch bend – puts the keyboards out of my budget.)

I’d like a good deal. (Who doesn’t?) But I don’t like to spend time searching for one. (Who does?)

So here’s the plan.

Firstly, I’ll restrict my search to Amazon.co.uk. For electronics items, I haven’t found anyone consistently cheaper. Tesco has some pretty low prices, but not the range. eBuyer is pretty good, but not often enough. Google Products is the only other one that gets me consistent lower prices, but I’ve had my credit card identity stolen once before while shopping online, so I’d rather not pick any random seller listed on Google.

Amazon has a secret discount. You can search for electronics items with 30% off or more. And then you can narrow it down to Sound & Vision > Musical Instruments > MIDI Keyboards. Further cap a 100 - 200 GBP restriction. That leaves us with one product:

While that matches my criteria, I’m in no hurry and can wait for more offers to come up. But I don’t want to keep checking this page every day. So, RSS to the rescue. You probably think I can’t get enough of RSS feeds. And you’d be right. The thing is, as an attention mechanism, it is incredibly powerful, and I never cease to be amazed that the things it lets me do.

Using my XPath checker and a bit of trial and error, I figured all product links link to “amazon.co.uk/dp/…” with a  inside. So this XPath gets all the links:

//a[contains(@href,'/dp/')][span]

And I made an RSS feed out of that using my XPath server and subscribed to it on Google Reader.

Combining a bunch of such searches, I have a shopping folder on Google Reader has all the items I’m searching for. Now that’s lazy bargain hunting.

Which is all very fine. But given that I’m buying a car in a hurry right now, and I’m not doing any bargain hunting, it’s a classic case of being penny-wise and pound-foolish. Sigh…

Comments

Oracle 4 Jan 2008 12:00 pm: Alternate way is to set-up a google alert for the specific item in the chosen sites. Do you see any disadvantages with that?
Rajesh 11 Nov 2008 7:58 am: Never had someone’s personal website kept me so drawn in. You are smart and very articulate (u probably know this already) and it shows in each intersting and very often informative article you write. you can even try writing columns in magazines, websites or papers. i’ve learnt a lot from your site already and haven’t finished reading them all. i also see you play the keyboard and maybe interested in midis. i’ve sequenced some midis. u can can find them at

http://rajesh.annamalaisamy.googlepages.com/rajeshmidis

if u’re interested. The best tamil song midis in my opinion are done by kishmu. His midi works can be found at

www.geocities.com/kishmu/

Eagerly anticipating your next article.

Regards

Rajesh

Web lookup using Google Spreadsheets

Mon, 31 Dec 2007 12:00:00 +0000

I’d written earlier about Web lookup in Excel. I showed an example how you could create a movie wishlist that showed the links to the torrents from Mininova.

You can do that even easier on Google Spreadsheets. It has 4 functions that let you import external data:

=importData(“URL of CSV or TSV file”).
Imports a comma-separated or tab-separated file.
=importFeed(URL).vLets you import any Atom or RSS feed.
=importHtml(URL, “list” | “table”, index).
Imports a table or list from any web page.
=importXML(“URL”,“query”).
Imports anything from any web page using XPath.

Firstly, you can see straight off why it’s easy to view RSS feeds in Google Spreadsheets. Just use the importFeed function straight away. So, for example, if I wanted to track all 8GB iPods on Google Base, I can import its feed in Google Spreadsheets.

This automatically creates a list of the latest 8GB iPods.

Incidentally, the “Price” column doesn’t appear automatically. It’s a part of the description. But it’s quite easy to get the price using the standard Excel functions. Let’s say the description is in cell C1. =MID(C1, FIND("Price", C1), 20) gets you the 20 characters starting from “Price”. Then you can sort and play around as usual.

The other powerful thing about Google Spreadsheets is the CONTINUE function. The importFeed function creates a multi-dimensional array. You can extract any cell from the array (for example, row 3, column 2 from cell C1) using CONTINUE(C1, 3, 2). So you can just pick up the title and description, or only alternate rows, or put all rows and columns in a single column – whatever.

The most versatile of the import functions is the importXML function. It lets you import any URL (including an RSS feed), filtering only the XPath you need. As I mentioned earlier, you can scrape any site using XPath.

For example, =importXML("http://www.imdb.com/chart/top", "//table//table//table//a") imports the top 250 movies from the IMDb Top 250. the second parameter says, get all links (a) inside a table inside a table inside a table. This populates a list with the entire Top 250.

Now, against each of these, we could get a feed of Mininova’s torrents. Mininova’s RSS URL is http://www.mininova.org/rss/search_string. So, in cell B1, I can get a torrent for the cell A1 (The Godfather) using the importFeed function. (Note: you need to replace spaces with a + symbol. These functions don’t like invalid URLs.).

Just copy this formula down to get a torrent against each of the IMDb Top 250 movies!

Check out the sheet I’ve created. (You need a Google account to see the sheet. If you don’t want have one, you can view the sheet.)

Now, that’s still not the best of it. You can extract this file as an RSS feed! Google lets you publish your sheets as HTML, PDF, Text, XLS, etc. and RSS and Atom are included as well. Here’s the RSS feed for my sheet above.

Think about it. We now have an application that sucks in data from a web page, does a web-based vlookup on another service, and returns the results as an RSS feed!

There are only two catches to this. The first is that Google has restricted us to 50 import functions per sheet. So you can’t really have the IMDb Top 250 populated here – only the top 49. The second is that the spreadsheet updates only when you open it again. So it’s not really a dynamically updating feed. You need to open the spreadsheet to refresh it.

But if you really wanted these things, there’s always Yahoo! Pipes.

Comments

Gregory Dillon 31 Dec 2007 12:00 pm: Hey, thanks. Your page was the most helpful of all. In the example with the movie names, how did you figure out that it was a table x3. or table within table within table.
S Anand 30 Mar 2010 9:04 pm: Looks like the IMDb top 250 page format has changed. =importXML(“http://www.imdb.com/chart/top", “//div[@id=““main””]//table//font/a”) seems to work now.
Sintl 2 Jan 2011 4:24 am: Hi Is there a possibility to get selective text from webpage by using import url function. =ImportHtml(URL; “list” | “table”; index) I want to pull following text Updated Sunday, December 19, 2010 from a html page Thnx
Siamak 26 Aug 2011 11:03 pm: I have an excel file with the name of about 300 businesses that I want to look up their phone numbers or websites. Any idea how I can automate this, rather than type each entry in Google? Thanks
akc 30 Jul 2011 11:07 am: Hey Very Helpful website it is……….Please help me I wan to import data from (http://dsebd.org/latest_share_price_scroll_l.php ) . to google spreadsheet . I wan to import share price . Now tell me which way i will use. I m failed

Scraping RSS feeds using XPath

Mon, 17 Dec 2007 12:00:00 +0000

If a site doesn't have an RSS feed, your simplest option is to use Page2Rss, which gives a feed of what's changed on a page.

My needs, sometimes, are a bit more specific. For example, I want to track new movies on the IMDb Top 250. They don't offer a feed. I don't want to track all the other junk on that page. Just the top 250.

There's a standard called XPath. It can be used to search in an HTML document in a pretty straightforward way. Here are some examples:

//a	Matches all <a> links
//p/b	Matches all <b> bold items in a <p> para. (the <b> must be immediately under the <p>)
//table//a	Matches all links inside a table (the links need not be immediately inside the table -- anywhere inside the table works)

You get the idea. It's like a folder structure. / matches the a tag that's immediately below. // matches a tag that's somewhere below. You can play around with XPath using the Firefox XPath Checker add-on. Try it -- it's much easier to try it than to read the documentation.

The following XPath matches the IMDb Top 250 exactly.

//tr//tr//tr//td[3]//a

(It's a link inside the 3rd column in a table row in a table row in a table row.)

Now, all I need is to get something that converts that to an RSS feed. I couldn't find anything on the Web, so I wrote my own XPath server. The URL:

www.s-anand.net/xpath?
url=http://www.imdb.com/chart/top&
xpath=//tr//tr//tr//td[3]//a

When I subscribe to this URL on Google Reader, I get to know whenever there's a new movie on the IMDb Top 250.

This gives only the names of the movies, though, and I'd like the links as well. The XPath server supports this. It accepts a root XPath, and a bunch of sub-XPaths. So you can say something like:

xpath=//tr//tr//tr title->./td[3]//a link->./td[3]//a/@href

This says three things:

//tr//tr//tr	Pick all rows in a row in a row
title->./td[3]//a	For each row, set the title to the link text in the 3rd column
link->./td[3]//a	... and the link to the link href in the 3rd column

That provides a more satisfactory RSS feed -- one that I've subscribed to, in fact. Another one that I track is a list of . link->./@href">new popular movies that make it to the mininova top seeded movies category.

You can whiff up more complex examples. Give it a shot. Start simple, with something that works, and move up to what you need. Use XPath Checker liberally. Let me know if you have any isses. Enjoy!

Comments

Mark 17 Dec 2007 12:00 pm: Have you ever thought about introducing authentication to the XPath server? I would like to parse certain fields of a page that is authenticated with cookies.
Rog 28 Oct 2008 1:07 am: Any chance you could share your xpath.php code? It seems the server is no longer available.
S Anand 28 Oct 2008 1:43 am: Sure Rog. I’ve mailed it to you
S Anand 7 Mar 2009 10:35 am: Post Yahoo’s introduction of Yahoo Query Language, you’re much better off using that instead of my XPath utility. I’ve covered it in this article on client side scraping.
Scraping your way to RSS Feeds! « Technosiastic! 8 Apr 2009 10:16 am (pingback): […] another gem I figured which actually lets you run XPath query for scraping into a web page for RSS. It can be […]
Bart P 3 Mar 2012 11:48 am: It would be great if you could share this code, I really like to use this server, but want to remove session ids from the links (so my reader doesn’t think all links are new every time). Is that possible? :)
Ben 11 Jun 2015 12:46 am: Is that possible to share your xpath.php code? yahoo pipes is going to be shut down :(

RSS feeds in Excel

Thu, 03 May 2007 12:00:00 +0000

The technique of Web lookups in Excel I described yesterday is very versatile. I will be running through some of the practical uses it can be put to over the next few days

TO generalise things beyond just getting the Amazon price, I created a user-defined function called XPATH. It takes two parameters:

URL of the XML feed to read
Search XPath list string (separated by spaces)

This function can be used to extract information out of any XML file on the Web and get it out as a table. For example, if you wanted to watch the Top 10 movies on the IMDb Top 250, and were looking for torrents, an RSS feed is available from mininova. The URL http://www.mininova.org/rss/movie_name/4 gives you an RSS file matching all movies with “movie_name”. From this, we need to extract the and <item><link> elements. That’s represented by “//item title link” on my search string. <a href="https://www.s-anand.net/blog/assets/flickr-mininova-rss-feed-in-excel_482789516_o-png.webp"></a> The formula becomes <code>XPath2( "http://www.mininova.org/rss/"&A2&"/4", "//item title link")</code>. The result is a 2-dimensional array returning individual items in rows, and the columns are title and link. Pulling it all together, you can get the sheet above. All of this could be done using a command-line program. Excel has one huge advantage though. It’s one of the most powerful user-interfaces. Increasingly, I’m beginning to rely on just two user interfaces for almost any task. One is the browser, and the other is Excel. With Excel, I could have a sheet that has my <a href="https://www.s-anand.net/blog/wishlist-for-movies/">movie wishlist</a> (which changes often), and add check to see if the torrent exists. Every time I add a bunch of movies to the wishlist, it’s just a matter of copying the formula down. No need to visit a torrent search engine and typing each movie in, one by one. Another example. Someone suggests 10 movies to watch. I’d rather watch the ones with a higher IMDb rating. Again, enter the Web lookup. Type in the movie names. Use a function like this to look up the rating on IMDb, and sort by the rating. The possibilities are endless. <hr> <h2 id="comments">Comments</h2>  <ul> <li>Isit? 3 May 2007 12:00 pm: Is it possible to have the feeds update cumulative so the old feeds dont get replaced in excel?</li> <li>Alex 3 May 2007 12:00 pm: Hi, I have an Excel spreadsheet with URLs of move titles on IMDB, and I would like to retrieve the movie rating. I’ve read your tutorials, and this one comes closest to what I need. However I still cannot figure out what needs to be done. It looks like you have this problem solved. If so, can you possibly share your solution with me? Thank you, Alex</li> <li>Kishan 3 May 2007 12:00 pm: I came here searching for the other way round. That is to publish an RSS feed of what I have in Xcel! Thanks Anyways</li> <li>venkat 15 Oct 2010 4:09 pm: Anand Can you provide the xpath function?, I need to read a rss feed into excel, and i want only some columns of data, i would appreciate if you could send the xpath function to my email id Thanks venkat</li> <li><a href="http://www.light-sensors.com">Light Sensors</a> 2 Dec 2010 7:08 pm: movie ratings really depend on how cool the movie is, i usually rate movies depending on the story line `-;</li> <li>rt 27 Jun 2013 5:41 pm: Hi, nice job ! can you share with me your xpath2 (i tried your other one without succes on amazon) ?Thanks, R</li> </ul>  </article> <article> <h1>Web lookup using Excel</h1> Thu, 03 May 2007 12:00:00 +0000 Take a look at the Excel screenshot below. <a href="https://www.s-anand.net/blog/assets/flickr-amazon-web-prices_481952444_o-png.webp"></a> Yes, that’s right. I have a <a href="https://www.s-anand.net/blog/user-defined-functions-in-excel/">user-defined function</a> called AMAZONPRICE. And it returns these cameras’ prices directly from Amazon.com. (Given the category and some keywords, it returns the price of the bestselling item on Amazon.com.) Here’s the code behind the function. <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-javascript" data-lang="javascript">Function AmazonPrice(index As String, keywords As String) As String Dim xDoc As MSXML2.DOMDocument30 Set xDoc = New MSXML2.DOMDocument30 xDoc.async = False If xDoc.Load("http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService" _ "&Version=2005-03-23&Operation=ItemSearch&SubscriptionId=03SDGQDFEB455W53SB82" _ "&AssociateTag=sanand-20&MinimumPrice=10000&ResponseGroup=OfferSummary,Small" _ "&Sort=salesrank&SearchIndex=" & index & "&Keywords=" & keywords) Then xDoc.setProperty "SelectionLanguage", "XPath" xDoc.setProperty "SelectionNamespaces", _ "xmlns:a=""http://webservices.amazon.com/AWSECommerceService/2005-03-23""" AmazonPrice = xDoc.selectSingleNode("/a:ItemSearchResponse//a:Amount").Text End If End Function </code></pre></div><hr> This is how it all started… Flickr has a <a href="http://www.flickr.com/cameras/">camera finder</a> that shows the most popular cameras in the Flickr community. <a href="https://www.s-anand.net/blog/assets/flickr-flickr-camera-finder_481952446_o-png.webp"></a> I <a href="https://www.s-anand.net/blog/how-i-buy-gadgets/">love comparing gadgets</a>, I’d been doing some research around these cameras, and the <a href="http://www.flickr.com/cameras/fujifilm/">Fuji series</a> (because I own a <a href="https://www.s-anand.net/blog/my-fuji-finepix-s5600/">Fuji Finepix S5600</a>). I’d normally make a spreadsheet that compares these cameras on various parameters, including price. Since I believe in <a href="https://www.s-anand.net/blog/excel-never-type-in-data/">never typing in data</a>, wondered if there was a way to get the prices in automatically… Two things made this possible. <ol> <li>Amazon offers <a href="http://www.amazon.com/gp/browse.html?node=3435361">web services</a> which let you get price (and almost any other) data for their products</li> <li>Visual Basic lets you use pretty much any ActiveX object as a control. Microsoft offers <a href="http://msdn2.microsoft.com/en-us/xml/bb291077.aspx">MSXML</a> which you can use to load (or download) any XML file, and parse it.</li> </ol> The first task is to get the XML feed for a product you want. Amazon lets you do that through by typing in a URL. The best way to construct the URL is through <a href="http://www.awszone.com/">AWSZone</a>. I picked the <a href="http://www.awszone.com/scratchpads/aws/ecs.us/ItemSearch.aws">US ItemSearch</a> method, which searches for a title or keyword within a category, and returns the matches. The feed for the Canon EOS Digital Rebel XT, based on this, would be at: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text">http://ecs.amazonaws.com/onca/xml? Service=AWSECommerceService& Version=2005-03-23& Operation=ItemSearch& SubscriptionId=0525E2PQ81DD7ZTWTK82& SearchIndex=Electronics& Keywords=Canon EOS Digital Rebel XT& Sort=salesrank& ResponseGroup=Offers,Small </code></pre></div>(You really need to replace the Subscription ID with your own.) If you <a href="http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService&Version=2005-03-23&Operation=ItemSearch&SubscriptionId=0525E2PQ81DD7ZTWTK82&SearchIndex=Electronics&Keywords=Canon%20EOS%20Digital%20Rebel%20XT&Sort=salesrank&ResponseGroup=Offers,Small">retrieved this URL</a>, you have an XML file containing the details of all Canon EOS Digital Rebel XTs, sorted by sales rank. To load this in Excel, you need to create a <a href="https://www.s-anand.net/blog/user-defined-functions-in-excel/">UDF</a> in Visual Basic. First, go to Tools - References and enable Microsoft XML, v3.0 or v4.0. Now, to load an XML document, do this: <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-vb" data-lang="vb">Dim xDoc As MSXML2.DOMDocument30 Set xDoc = New MSXML2.DOMDocument30 xDoc.async = False xDoc.Load(url) </code></pre></div>If the load succeeds, then you can extract the information fairly easily, using <a href="http://www.w3.org/TR/xpath">XPath</a>. <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-vb" data-lang="vb">xDoc.setProperty "SelectionLanguage", "XPath" xDoc.setProperty "SelectionNamespaces", _ "xmlns:a=""http://webservices.amazon.com/AWSECommerceService/2005-03-23""" </code></pre></div>The first line says we’ll be searching using XPath. The second line is a <a href="http://support.microsoft.com/kb/313372">workaround to support default namespaces</a>. (Don’t worry about it. I don’t quite get it either.) Finally, you get the price from the XML tree. In this case, it’s under ItemSearchResponse/Items/Item/OfferSummary/LowestNewPrice/Amount, and it’s in cents. <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-vb" data-lang="vb">AmazonPrice = xDoc.selectSingleNode("/a:ItemSearchResponse//a:Amount").Text </code></pre></div>That’s it! Now that this function is defined, just pass it the category and keywords, and you have the price of the first matching product. You can retrieve any other information about products as well – like sales rank, weight, ratings, reviews, etc. <hr> Here’s the <a href="https://files.s-anand.net/blog/a/AmazonPrices.xls">spreadsheet</a> for you to play around with. <hr> <ul> <li><a href="http://www.amazon.com/gp/product/0764540726">Excel 2003 Power Programming with VBA (Excel Power Programming With Vba</a></li> <li><a href="http://www.amazon.com/gp/product/0596003595">Writing Excel Macros with VBA, 2nd Edition</a></li> </ul> <hr> <h2 id="comments">Comments</h2>  <ul> <li>Saurabh 3 May 2007 7:38 am: There is a simpler way as well. Use Data->Import External Data->New Web Query. While this is not as powerful as the user defined function that you have created, but definitely a good starting point for lesser geeks :) I use this, for example, to have a dashboard kind of view for the status of the multiple projects that I handle at my work. We rely heavily on Microsoft Sharepoint at work, and using the Web Query option, I can easily get a snapshot of the status of various projects. Same can also fetch shareprices from various websites.</li> <li>S Anand 3 May 2007 8:16 am: Good point, Saurabh! Web Query does make getting stuff out of HTML very easy. My function operates in a different domain, though. It can access XML, which Web Query won’t, but can’t access HTML, which Web Query does. I’m planning a few more posts on how to extend this concept… that will explain what I mean.</li> <li>rick sherrin 3 May 2007 12:00 pm: Hi, I work at a public library, in the acquisitions department. The majority of the work is manually keying in order information (that is available electronically elsewhere). I read your article about using Excel to retrieve amazon information. This is something that would be very useful to our library and would make my job easier. I have a little excel knowledge, but I have no experience with visual basic or xml, etc. So although it looks straightforward, it is still out of my reach. I would like to be able to enter a column of ISBN numbers in one column, and retrieve title, author, publisher, price, reviews, synopsis, cover image, in adjacent columns. I would appreciate your help if you would give me an example of how I could easily do this. Thanks Rick</li> <li>dat 12 Aug 2008 8:10 am: useful article! Btw, how do I retrieve other info from amazon (published date, sales rank, customer review, etc)??? thanks</li> <li>S Anand 13 Aug 2008 2:15 am: @dat: Have a look at <a href="http://www.awszone.com/">http://www.awszone.com/</a> for examples on other parameters.</li> <li>Lydia 3 May 2010 12:05 am: this worked until august last year when amazon changed the way the query signs into their system…it now requires a signature with a timestamp in addition to the subscription ID</li> <li>Bobby Baker 10 Feb 2015 7:09 am: Have you updated the search functions for this or do you have a new code for this?</li> <li>Vlad 26 Aug 2015 10:00 pm: Hey Annand is it possible o get the racking status fro fedex for a column of tracking numbers?</li> </ul>  </article> </main></body></html>