Automating Internet Explorer with jQuery

Sun, 18 May 2008 12:00:00 +0000

Most of my screen-scraping so far has been through Perl (typically WWW::Mechanize). The big problem is that it doesn't support Javascript, which can often be an issue:

The content may be Javascript-based. For example, Amazon.com shows the bestseller book list only if you have Javascript enabled. So if you're scraping the Amazon main page for the books bestseller list, you won't get it from the static HTML.
The navigation may require Javascript. Instead of links or buttons in forms, you might have Javascript functions. Many pages use these, and not all of them degrade gracefully into HTML. (Try using Google Video without Javascript.)
The login page uses Javascript. It creates some crazy session ID, and you need Javascript to reproduce what it does.
You might be testing a Javascript-based web-page. This was my main problem: how do I automate testing my pages, given that I make a lot of mistakes?

There are many approaches to overcoming this. The easiest is to use Win32::IE::Mechanize, which uses Internet Explorer in the background to actually load the page and do the scraping. It's a bit slower than scraping just the HTML, but it'll get the job done.

Another is to use Rhino. John Resig has written env.js that mimics the browser environment, and on most simple pages, it handles the Javascript quite well.

I would rather have a hybrid of both approaches. I don't like the WWW::Mechanize interface. I've gotten used to jQuery's rather powerful selectors and chainability. So I'll tell you a way of using jQuery to screen-scrape offline using Python. (It doesn't have to be Python. Perl, Ruby, Javascript... any scripting language that can use COM on Windows will work.)

Let's take Google Video. Currently, it relies almost entirely on Javascript. The video marked in red below appears only if you have Javascript.

I'd like an automated way of checking what video is on top on Google Video every hour, and save the details. Clearly a task for automation, and clearly not one for pure HTML-scraping.

I know the video's details are stored in elements with the following IDs (thanks to XPath checker):

ID	What's there
hs_title_link	Link to the video
hs_duration_date	Duration and date
hs_ratings	Ratings. The stars indicate the rating and the span.Votes element inside it has the number of people who rated it.
hs_site	The site that hosts the video
hs_description	Short description

So I could do the following on Win32::IE::Mechanize.

use Win32::IE::Mechanize;
my $ie = Win32::IE::Mechanize->new( visible => 1 );
$ie->get("http://video.google.com/");
my @links = $ie->links
# ... then what?

I could go through each link to extract the hs_title_link, but there's no way to get the other stuff.

Instead, we could take advantage of a couple of facts:

Internet Explorer exposes a COM interface. That's what Win32::IE::Mechanize uses. You can use it in any scripting language (Perl, Ruby, Javascript, ...) on Windows to control IE.
You can load jQuery on to any page. Just add a <script> tag pointing to jQuery. Then, you can call jQuery from the scripting language!

Let's take this step by step. This Python program opens IE, loads Google Video and prints the text.

# Start Internet Explorer
import win32com.client
ie = win32com.client.Dispatch("InternetExplorer.Application")

# Display IE, so you'll know what's happening
ie.visible = 1

# Go to Google Video
ie.navigate("http://video.google.com/")

# Wait till the page is loaded
from time import sleep
while ie.Busy: sleep(0.2)

# Print the contents
# Watch out for Unicode
print ie.document.body.innertext.encode("utf-8")

The next step is to add jQuery to the Google Video page.

# Add the jQuery script to the browser
def addJQuery(browser,
    url="http://jqueryjs.googlecode.com/files/jquery-1.2.4.js"

    document = browser.document
    window = document.parentWindow
    head = document.getElementsByTagName("head")[0]
    script = document.createElement("script")
    script.type = "text/javascript"
    script.src = url
    head.appendChild(script)
    while not window.jQuery: sleep(0.1)
    return window.jQuery

jQuery = addJQuery(ie)

Now the variable jQuery contains the Javascript jQuery object. From here on, you can hardly tell if you're working in Javascript or Python. Below are the expressions (in Python!) to get the video's details.

# Video title: "McCain's YouTube Problem ..."
jQuery("#hs_title_link").text()

# Title link: '/videoplay?docid=1750591377151076231'
jQuery("#hs_title_link").attr("href")

# Duration and date: '3 min - May 18, 2008 - '
jQuery("#hs_duration_date").text()

# Rating: 5.0
jQuery("#hs_ratings img").length

# Number of ratings '(8,288 Ratings) '
jQuery("#hs_ratings span.Votes").text()

# Site: 'Watch this video on youtube.com'
jQuery("#hs_site").text()

# Video description
jQuery("#hs_description").text()

This wouldn't have worked out as neatly in Perl, simply because you'd need to use -> instead of . (dot). With Python (and with Ruby and Javascript on cscript), you can almost cut-and-paste jQuery code.

If you want to click on the top video link, use:

jQuery("#hs_title_link").get(0).click()

In addition, you can use the keyboard as well. If you want to type username TAB password, use this:

shell = win32com.client.Dispatch("WScript.Shell")
shell.sendkeys("username{TAB}password")

You can use any of the arrow keys, control keys, etc. Refer to the SendKeys Method on MSDN.

Comments

Franky 18 May 2008 12:00 pm: Holy crap. I am in way over my head here, but this is pretty sweet stuff if I’m reading it correctly. Thanks for the info.
Alessandro 5 Jul 2009 11:50 am: The idea is really good. Unfortunately it doesn’t seem to work under Perl. When a JS function/script is invoked (jQuery is a JS function) IE does not return a COM object and the control from Perl is lost. There are, of course, methods to overcome the obstacle but not as simple as $window->jQuery!
Anonymous 5 Mar 2010 11:25 pm: Wow, kick ass! Exactly what I was looking for. Used AutoIt on a host machine to instantiate the IE object and thanks to you was able to use jQuery to do my scraping. Combined with the IE Developer Toolbar, this has probably saved me weeks of time.
Manikandan 19 Nov 2009 3:30 pm: I am trying to do a screenscrape and the code executes jquery. Whenever I had to execute javascript i could execute it by ie.navigate(javascript:….) But when I tried the same thing with jquery it does not execute. The code behind is something like this I tried ie.navigate(jQuery1258644477483=“2”) It did not thing. Appreciate your help!!!
Powershell: Including jQuery in InternetExplorer.Application | DeveloperQuestion.com 30 Oct 2010 1:32 pm (pingback): […] script document I Followed this article, explaining how to accomplish in Python what I want to do in Powershell. So now I have this […]
Joe 27 Nov 2010 2:42 pm: Hello Mr. Anand, I am trying to do this in Ruby, but I am not sure how to start. Maybe I am lucky and you have you moved to ruby since May 2008? You start with import win32com.client, and I suppose that is to support an IE browser API. I have Firefox, Chrome, Safari and others available to me. The basic aim I guess is to inject JQuery into my page object. Mechanize provides me a page. I am that far. How do I it give it JQuery superpowers. Mechanize uses Nokogiri selectors under the hood. But I would love to standardize on JQuery because it’s useful in so many other contexts…in the way that regex is. Joe
Powershell: Using jQuery through InternetExplorer.Application 12 Feb 2012 2:48 am (pingback): […] followed this article, explaining how to spice up an Internet Explorer COM-Object with jQuery. While the author used […]

Netscape Opera Internet Explorer and Mozilla

Sat, 16 Oct 2004 12:00:00 +0000

A decade with Netscape. I’ve been wavering between IE, Opera and Mozilla over the last few months, and have come a full circle.

IE: Default browser. Works with BCG’s intranet.
Opera: Great offline browsing. Fast downloading. Tabbed browsing. Mouse gestures.
Mozilla: Powerful plugins.
IE: A9 toolbar. Fantastic way of searching.
IE again: Google desktop. Solves all offline browsing problems I had.

MSN hates Opera

Sun, 09 Feb 2003 12:00:00 +0000

MSN’s attempt to foil Opera. via andersja

101 things you can do in Mozilla

Fri, 15 Nov 2002 12:00:00 +0000

101 things you can do in Mozilla and not IE. But apart from 1. Tabbed browsing and 2. Popup blocking, I don’t quite use the other features. Mozilla (and Opera) still need some catching up to do. via New Architect

Move away from IE

Tue, 12 Mar 2002 12:00:00 +0000

AOL, and hence its 30 million subscribers, could move away from IE towards Mozilla. That’s big.

Block MSN access

Sun, 28 Oct 2001 12:00:00 +0000

Microsoft first tries to block MSN access to non-IE browsers, and then surrenders. I still can’t access their photos section through Opera.

IE6

Thu, 30 Aug 2001 12:00:00 +0000

IE6 is out with new features. My site has already had quite a few hits with people using IE6! BTW, it doesn’t have Java or Plugin support.

OggVorbis

Sat, 23 Jun 2001 12:00:00 +0000

Will OggVorbis beat MP3 Pro? Will Netscape 6.1 beat IE 5.5? Will OpenBSD 2.9 overtake Linux? Unlikely. But should be interesting to watch.

Opera 5

Fri, 08 Jun 2001 12:00:00 +0000

I shifted from Netscape Navigator to Internet Explorer because I could cut and paste tables from IE to Excel. But now that I’ve downloaded Opera 5.11, used its zoom function, it’s gesture keys, and so on, I’m moving over to Opera. Long live Opera! (Which, given Netscape’s fate, is perhaps no more than a fond hope.)

Browser stats

Thu, 10 Aug 2000 12:00:00 +0000

The latest browser stats say that 69% of the world uses, Internet Explorer, while only 27% use Netscape. BrowserWatch says 57%-26%. Netscape 6 might do something about it, given that Microsoft’s split.

internet-explorer on S Anand