jquery on S Anand

Eating more for less

Thu, 19 May 2011 18:12:23 +0000

A couple of years ago, I managed to lose a fair bit of weight. At the start of 2010, I started putting it back on, and the trajectory continues. I’m at the stage where I seriously need to lose weight.

I subscribe to The Hacker’s Diet principle – that you lose weight by eating less, not exercising.

An hour of jogging is worth about one Cheese Whopper. Now, are you going to really spend an hour on the road every day just to burn off that extra burger?
You don’t exercise to lose weight (although it certainly helps). You exercise because you’ll live longer and you’ll feel better.

I’m afraid I’ll live too long anyway, so I won’t bother exercising just yet. It’s down to eating less.

Sadly, I like food. So to make my “diet” work, I need foods that add less calories per gram. Usually, when browsing stores, I check these manually. But being a geek, I figured there’s an easier way.

Below is a graph of some foods (the kind I particularly need to avoid, but still end up eating). The ones on the top add a lot of calories (per 100g), and better to avoid. The ones at the right cost a lot more. Now, I’m no longer at the point where I need to worry about food expenses, but still, I can’t quite kick the habit, also you might want to check out this Rootine’s comparison of B12 methylcobalamin and cyanocobalamin that will help you in your diet.

Hover over the foods to see what they are, and click on them to visit the product. (If you’re using an RSS reader and this doesn’t work, read on my site.)

(The data was picked from Tesco.)

It’s interesting that cereals are in the middle of the calorie range. I always thought they’d be low calories per gram. Turns out that if I want to to have such foods, I’m better off with desserts or ice creams (profiterole, lemon meringue or tiramisu). In fact, even jams have less calories than cereals.

But there are some desserts to avoid. Nuts are a disaster. So are chocolates. Gums, dates and honey are in the middle – about as good as cereals. Salsa dip seems surprisingly low. Custards seem to hit the sweet spot – cheap, and very low in calories. Same for jellies.

So: custards and jelly. My daughter’s going to be happy.

Comments

Srinivas 2 Jun 2011 2:30 pm: well looking at the graph, the answer is simple. just dont go to Tesco! and seriously , the most important rule of thumb to eat right is, never eat anything that comes in a packet.
Sathya 20 May 2011 8:11 am: Saar what about “arachi vitta kozhlambu”, thovaiyal, medu bonda, Rasa vadai , usili … and items form “Grand Sweets and Snacks” ? Where are these ? these are my staples ;-( jk … Nice one ( some surprises as well ).
Sumit Dhar 20 May 2011 4:34 pm: Anand, I know it sound crazy but increasingly research is showing that not all calories are equal. What that means is that, a calorie from carbs might result in greater fat / weight gain thank a calorie from proteins. (Reference: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC506782/) Additionally, processed sugar is something you should avoid. Sugar causes insulin spikes that results in greater fat storage. Sames goes for carbs. Even the desserts you mention, should ideally be avoided. (Reference: http://www.youtube.com/watch?v=dBnniua6-oM) On the other hand, I am consuming quite a bit of nuts on a daily basis: almonds, cashews, peanuts. They are a good source of proteins and vital fatty acids. If you are a veggie, cottage cheese is another good source of proteins (along with fat too). Milk is another source of proteins along with calcium. Why we get Fat by Gary Taubes is a good book to check out. While I am not a big believer in rapid fat loss or fast weight reduction, I have read good things about the Rapid Fat Loss Handbook. It reiterates ideas from Why we get Fat. In short, don’t focus on calories. Focus on grams of carbs consumed per day. Cheers, Sumit
D 26 May 2011 6:03 am: Anand, I would recommend you also consider one major strategy - nutrient timing, which I used to good effect over the last 6 months while consuming the same no. of calories. Also if you really like chocolate, you can try dark chocolate as an alternative - lower calories per gram. Even for vegetarians, protein shakes derived from whey and soy are a great alternative to other foods. I have employed each of these and can vouch for their effectiveness in managing weight
Shankar V 27 May 2011 9:00 pm: I agree with you. Nothing like cutting on food to lose weight. Exercise requires perseverance and focus….I lack both. :) Cutting down on food also has issues…..you reduce intake for a few days, then pop comes a wedding or function where you get ‘ilai potta sappadu’ and all control is lost. You tell yourself, just this once and then very soon you are back to your original eating habits. Best option - must be born with a body that does not put on weight. Period!!! I envy those folks. :)
Saurabh Mittal 15 Jan 2014 12:04 pm: Nice one Anand! I was at 82 kgs a few years back and managed to come down to 72 . This was a result of both exercise (have run two half marathons so far in 2012 and then again in 2013) and change of eating habits (more of proteins and less of carbs). There is a nice book as well by Tim Ferris (The Four Hour Body), but in short definitely look at incorporating more exercise in your routine..Also as Sumit pointed out above - processed sugars and salt are the two major culprits! We don’t even realize how these are found in plenty in most of the foods! Keeping track of weight on a daily basis definitely is a good habit! Cheers Saurabh

Open source in corporates

Mon, 14 Dec 2009 21:59:30 +0000

Last month, my first application went live.

I’ve been writing code for 20 years. Not one line of my code has been officially deployed in a corporate. (Loser…)

It’s a happy feeling. Someone defined happiness as the intersection of pleasure and meaning. Writing code is pleasurable. Others using it is meaningful.

But this post isn’t quite about that. It’s about the hoops I’ve had to jump through to make this happen.

I’ve been living in a nightmare since March 2009. That was when I decided that I’d try and get corporates to use open source.

March 2009

It began with a pitch to a VC firm. They were looking to build a content management system (CMS). Normally we’d pull together slides that say we’ll deliver the moon. This time, we put together demo based on WordPress’ CMS plugins.

The meeting went fabulously well. We said, “Here’s a demo we’ve built for you. Do you like it?” The business lead (Stuart) was drooling and declared that that’s exactly what they wanted. The IT lead (another Stuart) was happy too, but warned the business users: “Just remember: this isn’t how we do development, so don’t get your hopes up that we can deliver stuff like this :-)”

Time to make my point. I asked, “What’s your policy on open source software?”

The business lead went quiet. “I don’t know,” he finally said. Fair enough.

I turned to the IT lead. “Well, we don’t use it as a matter of policy… there are security concerns…” he said.

“Which web server do you use?”

”Oh, OK. I see what you mean. We use Apache. So on a case to case basis, we have exceptions. But generally we have security concerns.“

”Why? Do you believe open source software is more insecure than commercial software?“

He thought about it for a while. “Well… maybe. I don’t know.” We debated this a bit. Then we found the real issue: “It’s just that we don’t have control over the process. We don’t know enough about it to decide.”

A couple of weeks later, I tried pitching to a newspaper company. This time, it was our sales team that raised the same question. “But… isn’t open source insecure?”

I didn’t even bother pitching any open source stuff to them. But I’d learnt my lessons:

Demo the application. Don’t talk about it.

Show it to the business first, and then tackle IT.

Aside: June 2009 In June, I got another chance. I was building the website for a large retailer. The very first thing I did was ask to see the Javascript. Total mess, and filled with browser-incompatible DOM requests. So I went over to their web development team.

“Look, why don’t you guys use a Javascript library? It’ll get you cross browser compatibility and compact maintainable code at the same time.”

And, to their credit, they said, “Sure. Which library?”

I showed them this comparison of jQuery (blue), dojo, scriptaculous and mootools…

https://trends.google.com/trends/explore?date=all&q=jquery,prototype,dojo,ext,yui

… and we agreed on jQuery. So, if nothing else, I’ve managed to get one open source library into a corporate.

July 2009

I was also looking at payments, and retailer was looking to replace their chargeback application. Since I had a week off, I built a working PCI compliant prototype on Django. This time, I applied the lessons I’d learned, and demo-ed it to the business, who were thrilled. Time to tackle IT.

I started with the architecture team. Matt on the architecture team was the most approachable. So I went over, demo-ed it, and said, “Matt, this took a week to put together. It’s based on some new technologies. Are you game to try these out?”

He was. And quite enthused about it too. So we put together a proposal for the architecture review board, proposing a new technology stack: Django / Python and MySQL. As before, I showed the demo before I talked technology. I had prepared answers to all security related questions upfront (and practically memorised section 3 of the PCI guidelines.) The clincher, though, was the business case. To build it on Java, it would cost ~1,000 person days. On Django, I’d mostly done it in 5. There was no way of justifying 1,000 person days for an application that could save, at best £100,000 a year.

So they said “Go ahead, we’re fine if operations and infrastructure are fine.”

It was time to find a Django developer in Infosys. I hunted for a couple of weeks but none was available. (Only 2 people knew Django in the first place.) So that effort got canned, and we were back to the 1,000 person day solution. (Which got canned too, later.)

But in the process, I’d learned my third lesson.

If you’re trying new technologies, plan on delivering it yourself.

October 2009

Another application popped up that looked like a prime candidate for introducing open source. They were using an Excel application to fraud screen orders, and wanted to make a web app out of it.

I followed the same route as before. Demo it. Show it to business first, then IT. Built it myself. I skipped Architecture, since they’d already approved the technology stack, and took it straight to Infrastructure.

“This application uses Apache as the web server, MySQL as the database, and uses PHP and Javascript for the application logic. Could we get a Linux server to host it?”

Our entire conversation lasted 30 seconds. He said, “No. We use Windows servers” (I was fine)

“… and you’ll need to chance Apache to IIS” (fine again)

“… and we don’t support PHP, so it’ll have to be Java or .NET” (I don’t know .NET or Java… but fine)

“… and we don’t support MySQL, it’ll have to be SQL Server” (fine, I guess)

“… and we don’t have DBAs available until January, so you’ll have to wait.” (definitely not good.)

So back to the drawing board on the technology stack. I needed something in Java (I know very little Java, but nothing at all in .NET) and to avoid the DBA headache, it would have to bundle in a database. I first explored key-value stores like CouchDB, Redis, etc. None of them worked on Java. The only one I found that did was Persevere, and it was a JSON data store, which fit perfectly with my plans.

By this time, I’d also learn my my fourth and most important lesson.

Don’t try to promote open source. Just deliver the application

I said, “This is a custom-built application that runs on Java. Could we get a Windows server to host it?”

The answer was “Yes”, and we had it the next day.

PS: December 2009

The application’s deployed and running. It has about 10,000 orders fraud screened by now.

And the lessons are well learnt. So when some came over asking if there was any image resizing solution I knew off, I said: “Sure, who’s your business sponsor?” Then I went over and said, “Let me show you this open source application called ImageMagick. It handles aspect ratios correctly, and can crop too. Doesn’t this look professional?” Then I went over to IT and said, “It’s open source, so you can change it. It has Java bindings, so you can integrate it into your environment. It can handle 8 3000x2400 images a second on my puny laptop. It’s used by your competitors. And I can build it for you if you like.”

I might just have my second open source entry into a corporate this year.

Comments

Kalpesh 15 Dec 2009 4:27 am: Great post. The feeling one gets, when the software is used by people and one hears a few praises, is priceless. I have enjoyed working with end-users & they have a lot to tell you, when they use the software. i.e. ask the viewers about the movie & not the critics ;) From your post, I think business guys are scared of the word “open source” without knowing some of the best software is open-source. I agree with your idea of “show them the software first, let them use it & be happy with it. They won’t worry if its open source”. Good to see a post from you after quite some time.
Thejesh GN 15 Dec 2009 6:08 am: I had to pitch only one customer this year and he agreed to use JQuery (the biggest plus was, it wasn’t gnu, its MIT). I think by default many customers use open source components with out actually realizing they are opensource. Like for example most of apache products, eclipse. But on the other side, I think the challenge is to convince IT Provider ( or the customer ) to go opensource. Not using but releasing some of their IP under OpenSource. How do we pitch this to them? Not undermining your efforts. I think the challenge is to convert this big users into contributors.
Ashwin Iyengar 20 Jan 2010 9:55 pm: Hey anand, Got to your webpage after a long time. Good to see that you’re still a true “end-to-end” consultant…we should have a chat about personal financial management solutions sometime later…-a
Anand.S 20 Dec 2009 10:13 pm: Nice blog… Interesting and gives good insight in details .
jyoti 21 Dec 2009 6:00 pm: nice blog,ur blog sounds great.keep wrtng.
Chitra 21 Dec 2009 10:30 pm: Interesting! And congrats :-)! I won’t pretend that I understood everything you penned (er…typed) here, but I do realize the inhibition of most users regarding open source technology due to ‘security’ issues. Perhaps you could list out the most quoted issues sometime……
Somnath 23 Dec 2009 4:04 am: Interesting - I thought your job is that of a management consultant or at least that’s what your employer sells your service as. But you end up doing all the technical stuff as well. Guess with all this you are ready to setup your own company now. Happy holidays! BR, Somnath
Khair 18 Dec 2009 3:58 pm: Thanks for your post, seems you have been busy with work/vacation. Frankly, quite a lot of IT Managers have no clue at all when you speak to them of OpenSource. Though ‘unknowingly’ they have been using either as Apache or PHP or Java. If you carefully look at how the IT Managers are recruited or groomed, you will see a culture of everything has to be ‘MICROSOFT’ ! Believe me for end-users/customers it makes no difference as long as the application delivers the need! Also one has to commend the aggressive marketing strategy of Microsoft. I am not against proprietery software but think corporates need to know and look for what’s best for the clients and end-users. Which in turn could reduce the money spent on Infrastructure/development/deployment. With money saved they can generate more employment opportunities.
Recruiting smart people | s-anand.net 14 Feb 2010 6:46 pm (pingback): […] at least the smart people) that hang out on Stack Overflow for a given topic. Last year, when I was looking for a Django developer, I scoured the Infosys internal blogs for similar networks. (Didn’t find many, but it sure […]
Harish Kashyap 5 Mar 2010 7:56 am: Hi Anand, I could relate to the sequence of events… Fantastic post… I picked the learning to be that it is hard to change the corporate…Sometime you may have to put in more effort and time to change the corporate rather than to change the solution itself! I.e. Do not view as a constraint as it would prevent the change!
Vinoth 21 May 2010 3:52 pm: Very insightful blog.. I work for Infosys too and your point about non-availability of django developers is interesting. I think this is one other prime reason why business do not prefer open source. There are simply not enough knowledgeable developers in latest open source frameworks and hence IT vendors cannot confidently pitch for open source.

Client side scraping

Wed, 04 Mar 2009 18:00:57 +0000

“Scraping” is extracting content from a website. It’s often used to build something on top of the existing content. For example, I’ve built a site that tracks movies on the IMDb 250 by scraping content.

There are libraries that simplify scraping in most languages:

Perl: WWW::Mechanize
Python: BeautifulSoup
Ruby: HPricot
PHP: XPath (built-in)
Javascript: jQuery on env.js on Rhino

But all of these are on the server side. That is, the program scrapes from your machine. Can you write a web page where the viewer’s machine does the scraping?

Let’s take an example. I want to display Amazon’s bestsellers that cost less than $10. I could write a program that scrapes the site and get that information. But since the list updates hourly, I’ll have to run it every hour.

That may not be so bad. But consider Twitter. I want to display the latest iPhone tweets from http://search.twitter.com/search.atom?q=iPhone, but the results change so fast that your server can’t keep up.

Nor do you want it to. Ideally, your scraper should just be Javascript on your web page. Any time someone visits, their machine does the scraping. The bandwidth is theirs, and you avoid the popularity tax.

This is quite easily done using Yahoo Query Language. YQL converts the web into a database. All web pages are in a table called html, which has 2 fields: url and xpath. You can get IBM’s home page using:

select * from html where url="http://www.ibm.com"

Try it at Yahoo’s developer console. The whole page is loaded into the query.results element. This can be retrieved using JSONP. Assuming you have jQuery, try the following on Firebug. You should see the contents of IBM’s site on your page.

$.getJSON(
  "http://query.yahooapis.com/v1/public/yql?callback=?",
  {
    q: "select * from html where url=\"http://www.ibm.com\"",
    format: "json",
  },
  function(data) {
    console.log(data.query.results);
  },
);

That’s it! Now, it’s pretty easy to scrape, especially with XPath. To get the links on IBM’s page, just change the query to

select * from html where url="http://www.ibm.com" and xpath="//a"

Or to get all external links from IBM’s site:

select * from html where url="http://www.ibm.com" and xpath="//a[not(contains(@href,'ibm.com'))][contains(@href,'http')]""

Now you can display this on your own site, using jQuery.

This leads to interesting possibilities, such as Map-Reduce in the browser. Here’s one example. Each movie on the IMDb (e.g. The Dark Knight) comes with a list of recommendations (like this). I want to build a repository of recommendations based on the IMDb Top 250. So here’s the algorithm. First, I’ll get the IMDb Top 250 using:

select * from html where url="http://www.imdb.com/chart/top" and xpath="//tr//tr//tr//td[3]//a"

Then I’ll get a random movie’s recommendations like this:

select * from html where url="http://www.imdb.com/title/tt0468569/recommendations" and xpath="//td/font//a[contains(@href,'/title/')]"

Then I’ll send off the results to my aggregator.

Check out the full code at http://250.s-anand.net/build-reco.js.

In fact, if you visited my IMDb Top 250 tracker, you already ran this code. You didn’t know it, but you just shared a bit of your bandwidth and computation power with me. (Thank you.)

And, if you think a little further, here another way of monetising content: by borrowing a bit of the user’s computation power to build complex tasks. There already are startups built around this concept.

Comments

Aravinda 4 Mar 2009 7:20 pm: Great!
Sriram 4 Mar 2009 10:51 pm: Thank u anand. Nice Information.
Rishi 5 Mar 2009 8:19 am: Awesome… This is the future
grep imdb part 2 ? « taeyoungchoon 26 Mar 2009 1:16 pm (pingback): […] http://www.s-anand.net/blog/client-side-scraping/ […]
Khair 26 May 2009 4:10 pm: Many thanks for your efforts in simplifying the ‘scraping’.
Venkat 16 Nov 2010 1:40 pm: Hey Anand, Thanks. This post is really useful. I have managed to scrape Amazon wishlist using this technique (http://www.venkatsworld.com/WIP/JSON_amazon.html) However YQL seems to be taking some time to respond. I am not sure if you experienced this.I did a similar exercise with Picasa JSON feed and a Jquery plug-in for image scrolling. It works well there too.
Zheka 17 Jun 2012 8:35 pm: Interesting article! There is another powerful scraping technology available that offers Javascript, jQuery, CSS, and XPath instead of XPath-only. It’s called Bobik (http://usebobik.com). The is a cool example of scraping restaurant menus using Bobik at http://news.ycombinator.com/item?id=4066478.
LAvesh 12 Mar 2013 11:19 pm: Hey Anand, Great Article. Just one query though in the client side implementation its the client IP that will be hitting the website (in your case www.ibm.com)? or it will use yahoo SQL server IP to hit.

Infyblogs dashboard

Mon, 26 Jan 2009 12:00:00 +0000

I just finished Stephen Few’s book on Information Dashboard Design. It talks about what’s wrong with the dashboards most Business Intelligence vendors (Business Objects, Oracle, Informatica, Cognos, Hyperion, etc.), and brings Tuftian principles of chart design to dashboards.

So I took a shot at designing a dashboard based on those principles, and made this dashboard for InfyBLOGS.

You can try for yourself. Go to http://www.s-anand.net/reco/
Note: This only works within the Infosys intranet.

Right click on the “Infyblog Dashboard” link and click “Add to Favourites…” (Non-IE users – drag and drop it to your links bar)
If you get a security alert, say “Yes” to continue
Return to InfyBLOGS, make sure you’re logged in (that’s important) and click on the “Infyblog Dashboard” bookmark
You’ll see a dashboard for your account, with comments and statistics

The rest of this article discusses design principles and the technology behind the implementation. (It’s long. Skim by reading just the bold headlines.)

Dashboards are minimalist

I’ll use the design of the dashboard to highlight some of the concepts in the book.

I designed the dashboard first on Powerpoint, keeping these principles in mind.

Fits in one screen. No scrolling. Otherwise, it isn’t a dashboard.
Shows only what I need to see. Ideally, from this dashboard, I should receive all the information I need to act on, and no more.
Minimises the data-ink ratio. Don’t draw a single pixel that’s not required.

The first was easy. I just constrained myself to one page of PowerPoint, though if you had a lot of toolbars, the viewing area of your browser may be less than mine.

The second is a matter of picking what you want to see. For me, these are the things I look for when I log into InfyBLOGS:

Any new comments?
Any new posts from my friends?
What’s new? What’s hot?

Then I dig deeper, occasionally, into popular entries, popular referrers, how fast the blogs are growing, etc. So I’ve put in what I think are the useful things.

The third is reflected in the way some of this information is shown. Let me explain.

Keep the charts bare

Consider the graphs on the right. They look like this.

Notice the wiggly line to the right. It’s a graph called sparkline, and was introduced by Edward Tufte. Sparklines are great to show trends in a compact way. Forget the axes. Forget the axes labels. Forget the gridlines. The text on the left (“visitors per day”) tells you what it is. The number (10475) is the current value. And the line is the trend. Clearly the number of visitors has exploded recently, from a relatively slow and flat start. The labels and axes aren’t going to tell you much more.

Boldly highlight what’s important

The most important thing here, the title, is on top. It’s pure black, in large font, positioned right on top, and has a line segregating it from the rest of the page.

The sections are highlighted by a bigger font, different colour, and a line, but the effect is a bit more muted.

The numbers on the right are prominent only by virtue of size and position. If anything, the colour de-emphasizes them. This is to make sure that they don’t overwhelm your attention. (They would if they were in red, for instance.)

The number 10475 is carefully set to occupy exactly two line spaces. That alignment is very important. The small lines are at a font size of 11px, and line spacing is 1.5. So at a font size of 2 x 11px x 1.5 = 33px, the size of the large number covers exactly two rows.

The labels, such as “visitors” or “sites” are in blue, bold. Nothing too out of the way, but visible enough that they stand out.

The “View more…” links just use colour to stand out. They’re pretty unimportant.

The bulk of the text is actually made of links, unlike traditional links, they’re not underlined and they’re not blue. It would just add to the noise if everything where underlined. But on mouseover, they turn blue and are underlined, clearly indicating that they’re links.

I’ve used four mechanisms to highlight relative importance: position, size, colour and lines. There are many more: font, styling, boxes, underlining, indentation, etc.

The purpose of good design is to draw attention to what’s important, to direct the flow of the eye in a way that builds a good narrative. Don’t be shy of using every tool at your disposal in doing that. While on the topic, the Non-Designer’s Design Book is a fantastic and readable book on design for engineers.

Always use grids to display

I can’t say this any better than Mark Boulton of subtraction.com, in his presentation Grids are Good. Grids are pervasive in every form of good design. It’s a fantastic starting point as well. Just read the slideshow and follow it blindly. You can’t go wrong.

This dashboard uses 12-grid. The page is divided into two vertically. The left half has mostly text and the right half has statistics and help. There are 3 blocks within each, and they’re all the same size (alignment is critical in good design). Two of the blocks on the left are subdivided into halves, while the bottom right “Links and help” section is subdivided into three. Well, it’s easier to show it than explain it:

Copy shamelessly

The design for this dashboard was copied in part from Wordpress 2.7’s new dashboard, part from the dashboards on Stephen Few’s book, and part from the winners of the 2008 Excel dashboard competition.

Most of these designs are minimalist. There’s no extra graphics, jazzy logos, etc. that detract from the informational content. This is an informational dashboard, not a painting.

Good design is everywhere. View this presentation on How to be a Creative Sponge to get a sense of the many sources where you can draw inspiration from. You’re much much better of copying good design than inventing bad ones.

You can hack the dashboard yourself

I’ve uploaded the source for the dashboard at http://infyblog-dashboard.googlecode.com/

Please feel free to browse through it, but don’t stop there! Go ahead and tweak it to suit what you think should be on an ideal dashboard. I’ll give access to the project to anyone who asks (leave a comment, mail me or call me at +44 7957 440 260).

Please hack at it. Besides, it’s great fun learning jQuery and working on a distributed open source project.

Now for some notes on how this thing works.

Bookmarklets bring Greasemonkey to any browser

This is implemented as a bookmarklet (a bookmark written in Javascript). It just loads the file http://www.s-anand.net/infyblog-dashboard.js which does all the grunt work. This is the code for the bookmarklet.

// Create a new a script element
var s = document.createElement("script");

// Set the source of the script element to my bookmarklet script
s.setAttribute("src", "http://www.s-anand.net/infyblog-dashboard.js");

// Add the script element to the head
// This does the the equivalent of:
// <head><script src="..."></script></head>
document.getElementsByTagName("head")[0]
  .appendChild(s);

// Return void. Not sure why, but this is required
void s;

This form of the bookmarklet is perhaps it’s most powerful use. It lets you inject any script into any site. If you want to change it’s appearance, content, anything, just write it as a bookmarklet. (Think of it as the IE and Safari equivalent of Firefox’s Greasemonkey.)

So if you wanted to load jQuery into any page:

Change the URL in line 5 above to the jQuery URL and add it as a bookmark. (Or use Karl’s jQuery bookmarklet. It’s better)
Go to any page, say the IMDB Top 250 for instance
Click on the bookmarklet. Now your page is jQueryified. (If you used Karl’s bookmarklet, it’ll tell you so.)
On the address bar, type “javascript:alert($(’table:eq(10) tr’).length)” and press enter.
You should see 252, the number of rows on main table of the IMDB Top 250

Notes:

There are other ways of adding Greasemonkey functionality to your browser
There are many bookmarklet directories where you can find bookmarklets
The dashboard is better written as a server-side app, but I don’t own any servers in our Intranet

jQuery is a powerful Javascript library

Firstly, you don’t want to program Javascript without a library. You just don’t. The state of browser incompatibilities and DOM usability is just to pathetic to contemplate. Stop learning DOM methods. Pick any library instead.

Going by popularity, you would do well to pick jQuery. Here’s a graph of searches on Google for the popular Javascript libraries, and jQuery leads the pack.

https://trends.google.com/trends/explore?date=all&q=jquery,prototype,dojo,ext,yui

I picked it in early 2007, and my reasons were that it:

Is small. At that time, it was 19KB and was the smallest of the libraries. (Today it’s 18KB and has more features.)
Makes your code compact. It lets you chain functions, and overloads the $() function to work with DOM elements, HTML strings, arrays, objects, anything.
Doesn’t pollute the namespace. Just 2 global variables: jQuery and $. And you can make it not use $ if you like.
Is very intuitive. You can learn it in an hour, and the documention is extremely helpful.
Is fully functional. Apart from DOM manipulation, it covers CSS, animations and AJAX, which is all I want from a library.

These days, I have some additional reasons:

It’s extensible. The plugin ecosystem is rich.
It’s hosted at Google, and is really fast to load. Most often, it’s already cached in your users’ browser.
John Resig is a genius. After env.js and processing.js, I trust him to advance jQuery better than other libraries.

So anyway, the first thing I do in the dashboard script is to load jQuery, using the same code outlined in the bookmarklet section above. The only addition is:

script.onload = script.onreadystatechange = function() {
  if (
    !this.readyState || this.readyState == "loaded"
    || this.readyState == "complete"
  ) {
    callback();
  }
};

This tells the browser to run the script “callback” once the script is loaded. My main dashboard function runs once jQuery is loaded.

InfyBLOGS has most statistics the dashboard needs

Thanks to posts from Utkarsh and Simon, I knew where to find most of the information for the dashboard:

Comments page for your latest comments
Profile page for your statistics: number of entries, interests, friends, etc.
Directory page for recently updated blogs
Communities page for recently updated communities
Usage page for Infyblog statistics and popular referrers
Meme page for recently popular links

The only things I couldn’t find were:

Friends posts. I’d have liked to show the titles of recent posts by friends. Yeah, I know: blogs/user_name/friends has it, but thanks to user styles, it’s nearly impossible to parse in a standard way across users. I’d really like an RSS feed for friends.
Interests posts. Would be cool to show recent posts by users who shared your your interest.
Communities posts. Again, I’d have liked to show the recent posts in communities, rather than just the names of communities.

Using IFRAMEs, we can load these statistics onto any page

Let’s say we want the latest 10 comments. These are available on the comments page. To load data from another page, we’d normally use XMLHTTPRequest, get the data, and parse it — perhaps using regular expressions.

$.get('/tools/recent_comments.bml').function(data) {
    // Do some regular expression parsing with data
});

But from a readability and maintainability perspective, regular expressions suck. A cleaner way is to use jQuery itself to parse the HTML.

$.get('/tools/recent_comments.bml').function(data) {
    var doc = $(data);
    // Now process the document
});

This works very well for simple pages, but sadly, for our statistics, this throws a stack overflow error (at least on Firefox; I didn’t have the guts to try it on IE.)

So on to a third approach. Why bother using Javascript to parse HTML, when we’re running the whole application in a browser, the most perfect HTML parser? Using IFRAMEs, we can load the whole page within the same page, let the browser do the HTML parsing.

$("<iframe src=\"/tools/recent_comments.bml\"></iframe>")
  .appendTo("body")
  .load(function() {
    var doc = $(this).contents();
    // Now process the document
  });

This way, you can read any page from Javascript within the same domain. Since both the bookmarklet and statistics are on the InfyBLOGS domain, we’re fine.

jQuery parses these statistics rather well

Now the doc variable contains the entire comments page, and we can start process it. For example, the comments are on the rows of the second table in the page. (The first table is the navigation on the left.). So

doc.find("table:eq(1) tr");

gets the rows in the second table (table:eq(0) is the first table). Now, we can go through each row, extract the user, when the comment was made, links to the entry and to the comment.

doc.find('table:eq(1) tr').slice(0,8).map(function() {
    var e = $(this),
        user = e.find('td:eq(0) b').text(),
        when = e.find('td:eq(0)').text().replace(user, ''),
        post = e.find('td:eq(1) a:contains(Entry Link)').attr('href'),
        cmnt = e.find('td:eq(1) a:contains(Comment Link)').attr('href'),
        // return HTML constructed using the above information
    });

Google Charts API displays dynamic charts

Another cool thing is that we can use Google Charts to display charts without having to create an image beforehand. For instance, the URL:

http://quickchart.io/chart?cht=lc&chs=300x200&chd=t:5,10,20,40,80

shows an image with a line chart through the values (5,10,20,40,80).

The dashboard uses sparklines to plot the trends in InfyBLOG statistics. These are supported by Google Charts. We can extract the data form the usage page and create a new image using Google Charts that contains the sparkline.

If you want to play around with Google Charts without mucking around with the URL structure, I have a relatively easy to use chart constructor.

Always use CSS frameworks

Libraries aren’t restricted to Javascript. CSS frameworks exist, and as with Javascript libraries, these are a no-brainer. It’s not worth hand-coding CSS from scratch. Just build on top of one of these.

The state of CSS frameworks isn’t as advanced as in Javascript. Yahoo has it’s UI grids, Blueprint’s pretty popular and I found Tripoli to be nice, but the one I’m using here is 960.gs. It lets you create a 12-grid and 16-grid on a 960-pixel wide frame, which is good enough for my purposes.

At nearly 2,500 words, this is the longest post I’ve ever written. It took a day to design and code the dashboard, but longer to write this post. Like Paul Buchheit says, it’s easier to communicate with code. I’ve also violated my principle of Less is More. My apologies. But to re-iterate:

Please hack it

Download the code and data.

Comments

sujays 26 Jan 2009 7:14 am: am an x-infy guy and my id(sujaychandiran_s) still works in ur “suggest friends” feature.. does this connect 2 infy DB?.. if yes, doesnt this pose a security threat ? am jus curious !

me cant hack, so cant confirm :)

Sujay
S Anand 26 Jan 2009 9:23 am: @Sujay: nah, it’s not a security risk. It’s just a static dump of data, and
is a bit out-dated in any case. If anything, it may be a privacy risk:
recruiters might try to mine for interests and pick people that way. But in
this economy, I guess no one’s worried about that.
gwatts 26 Jan 2009 6:52 pm: If you like jQuery, and you like sparklines then you can combine the two and avoid the browser roundtrips to fetch externally rendered images:
http://omnipotent.net/jquery.sparkline/
Thejesh GN 26 Jan 2009 10:33 pm: Very cool.
I guess we can add buttons for add/edit posts.
S Anand 31 Jan 2009 5:59 am: That’s a great plugin, Gareth! Look forward to using it. Thanks.

Automating Internet Explorer with jQuery

Sun, 18 May 2008 12:00:00 +0000

Most of my screen-scraping so far has been through Perl (typically WWW::Mechanize). The big problem is that it doesn't support Javascript, which can often be an issue:

The content may be Javascript-based. For example, Amazon.com shows the bestseller book list only if you have Javascript enabled. So if you're scraping the Amazon main page for the books bestseller list, you won't get it from the static HTML.
The navigation may require Javascript. Instead of links or buttons in forms, you might have Javascript functions. Many pages use these, and not all of them degrade gracefully into HTML. (Try using Google Video without Javascript.)
The login page uses Javascript. It creates some crazy session ID, and you need Javascript to reproduce what it does.
You might be testing a Javascript-based web-page. This was my main problem: how do I automate testing my pages, given that I make a lot of mistakes?

There are many approaches to overcoming this. The easiest is to use Win32::IE::Mechanize, which uses Internet Explorer in the background to actually load the page and do the scraping. It's a bit slower than scraping just the HTML, but it'll get the job done.

Another is to use Rhino. John Resig has written env.js that mimics the browser environment, and on most simple pages, it handles the Javascript quite well.

I would rather have a hybrid of both approaches. I don't like the WWW::Mechanize interface. I've gotten used to jQuery's rather powerful selectors and chainability. So I'll tell you a way of using jQuery to screen-scrape offline using Python. (It doesn't have to be Python. Perl, Ruby, Javascript... any scripting language that can use COM on Windows will work.)

Let's take Google Video. Currently, it relies almost entirely on Javascript. The video marked in red below appears only if you have Javascript.

I'd like an automated way of checking what video is on top on Google Video every hour, and save the details. Clearly a task for automation, and clearly not one for pure HTML-scraping.

I know the video's details are stored in elements with the following IDs (thanks to XPath checker):

ID	What's there
hs_title_link	Link to the video
hs_duration_date	Duration and date
hs_ratings	Ratings. The stars indicate the rating and the span.Votes element inside it has the number of people who rated it.
hs_site	The site that hosts the video
hs_description	Short description

So I could do the following on Win32::IE::Mechanize.

use Win32::IE::Mechanize;
my $ie = Win32::IE::Mechanize->new( visible => 1 );
$ie->get("http://video.google.com/");
my @links = $ie->links
# ... then what?

I could go through each link to extract the hs_title_link, but there's no way to get the other stuff.

Instead, we could take advantage of a couple of facts:

Internet Explorer exposes a COM interface. That's what Win32::IE::Mechanize uses. You can use it in any scripting language (Perl, Ruby, Javascript, ...) on Windows to control IE.
You can load jQuery on to any page. Just add a <script> tag pointing to jQuery. Then, you can call jQuery from the scripting language!

Let's take this step by step. This Python program opens IE, loads Google Video and prints the text.

# Start Internet Explorer
import win32com.client
ie = win32com.client.Dispatch("InternetExplorer.Application")

# Display IE, so you'll know what's happening
ie.visible = 1

# Go to Google Video
ie.navigate("http://video.google.com/")

# Wait till the page is loaded
from time import sleep
while ie.Busy: sleep(0.2)

# Print the contents
# Watch out for Unicode
print ie.document.body.innertext.encode("utf-8")

The next step is to add jQuery to the Google Video page.

# Add the jQuery script to the browser
def addJQuery(browser,
    url="http://jqueryjs.googlecode.com/files/jquery-1.2.4.js"

    document = browser.document
    window = document.parentWindow
    head = document.getElementsByTagName("head")[0]
    script = document.createElement("script")
    script.type = "text/javascript"
    script.src = url
    head.appendChild(script)
    while not window.jQuery: sleep(0.1)
    return window.jQuery

jQuery = addJQuery(ie)

Now the variable jQuery contains the Javascript jQuery object. From here on, you can hardly tell if you're working in Javascript or Python. Below are the expressions (in Python!) to get the video's details.

# Video title: "McCain's YouTube Problem ..."
jQuery("#hs_title_link").text()

# Title link: '/videoplay?docid=1750591377151076231'
jQuery("#hs_title_link").attr("href")

# Duration and date: '3 min - May 18, 2008 - '
jQuery("#hs_duration_date").text()

# Rating: 5.0
jQuery("#hs_ratings img").length

# Number of ratings '(8,288 Ratings) '
jQuery("#hs_ratings span.Votes").text()

# Site: 'Watch this video on youtube.com'
jQuery("#hs_site").text()

# Video description
jQuery("#hs_description").text()

This wouldn't have worked out as neatly in Perl, simply because you'd need to use -> instead of . (dot). With Python (and with Ruby and Javascript on cscript), you can almost cut-and-paste jQuery code.

If you want to click on the top video link, use:

jQuery("#hs_title_link").get(0).click()

In addition, you can use the keyboard as well. If you want to type username TAB password, use this:

shell = win32com.client.Dispatch("WScript.Shell")
shell.sendkeys("username{TAB}password")

You can use any of the arrow keys, control keys, etc. Refer to the SendKeys Method on MSDN.

Comments

Franky 18 May 2008 12:00 pm: Holy crap. I am in way over my head here, but this is pretty sweet stuff if I’m reading it correctly. Thanks for the info.
Alessandro 5 Jul 2009 11:50 am: The idea is really good. Unfortunately it doesn’t seem to work under Perl. When a JS function/script is invoked (jQuery is a JS function) IE does not return a COM object and the control from Perl is lost. There are, of course, methods to overcome the obstacle but not as simple as $window->jQuery!
Anonymous 5 Mar 2010 11:25 pm: Wow, kick ass! Exactly what I was looking for. Used AutoIt on a host machine to instantiate the IE object and thanks to you was able to use jQuery to do my scraping. Combined with the IE Developer Toolbar, this has probably saved me weeks of time.
Manikandan 19 Nov 2009 3:30 pm: I am trying to do a screenscrape and the code executes jquery. Whenever I had to execute javascript i could execute it by ie.navigate(javascript:….) But when I tried the same thing with jquery it does not execute. The code behind is something like this I tried ie.navigate(jQuery1258644477483=“2”) It did not thing. Appreciate your help!!!
Powershell: Including jQuery in InternetExplorer.Application | DeveloperQuestion.com 30 Oct 2010 1:32 pm (pingback): […] script document I Followed this article, explaining how to accomplish in Python what I want to do in Powershell. So now I have this […]
Joe 27 Nov 2010 2:42 pm: Hello Mr. Anand, I am trying to do this in Ruby, but I am not sure how to start. Maybe I am lucky and you have you moved to ruby since May 2008? You start with import win32com.client, and I suppose that is to support an IE browser API. I have Firefox, Chrome, Safari and others available to me. The basic aim I guess is to inject JQuery into my page object. Mechanize provides me a page. I am that far. How do I it give it JQuery superpowers. Mechanize uses Nokogiri selectors under the hood. But I would love to standardize on JQuery because it’s useful in so many other contexts…in the way that regex is. Joe
Powershell: Using jQuery through InternetExplorer.Application 12 Feb 2012 2:48 am (pingback): […] followed this article, explaining how to spice up an Internet Explorer COM-Object with jQuery. While the author used […]

Chaining functions in Javascript

Mon, 18 Feb 2008 12:00:00 +0000

One of the coolest features of jQuery is the ability to chain functions. The output of a function is the calling object. So instead of writing:

var a = $("<div></div>");
a.appendTo($("#id"));
a.hide();

… I can instead write:

$("<div></div>").appendTo($("#id")).hide();

A reasonable number of predefined Javascript functions can be used this way. I make extensive use of it with the String.replace function.

But where this feature is not available, you an create it in a fairly unobstrusive way. Just add this code to your script:

Function.prototype.chain = function() {
  var that = this;
  return function() {
    // New function runs the old function
    var retVal = that.apply(this, arguments);
    // Returns "this" if old function returned nothing
    if (typeof retVal == "undefined") return this;
    // else returns old value
    else return retVal;
  };
};
var chain = function(obj) {
  for (var fn in obj) {
    if (typeof obj[fn] == "function") {
      obj[fn] = obj[fn].chain();
    }
  }
  return obj;
};

Now, chain(object) returns the same object, with all its functions replaced with chainable versions.

What’s the use? Well, take the Google AJAX search API. Normally, to search for the top 8 “Harry Potter” PDFs on esnips.com, I’d have to do:

var searcher = new google.search.WebSearch();
searcher.setQueryAddition("filetype:PDF");
searcher.setResultSetSize(google.search.Search.LARGE_RESULTSET);
searcher.setSiteRestriction("esnips.com");
searcher.setSearchCompleteCallback(onSearch);
searcher.execute("Harry Potter");

Instead, I can now do this:

chain(new google.search.WebSearch())
  .setQueryAddition("filetype:PDF")
  .setResultSetSize(google.search.Search.LARGE_RESULTSET)
  .setSiteRestriction("esnips.com")
  .setSearchCompleteCallback(onSearch)
  .execute("Harry Potter");

(On the whole, it’s probably not worth the effort. Somehow, I just like code that looks like this.)

Comments

T 18 Feb 2008 12:00 pm: looks sweet!