google-spreadsheets on S Anand

ImportHtml doesn’t auto-refresh

Fri, 12 Mar 2010 09:54:26 +0000

A cool thing about Google Spreadsheets is that you can scrape websites using external data functions like importHtml. It’s really easy to use. The formula:

=importHtml("http://www.imdb.com/chart/top", "table", 1)

imports the Internet Movie Database top 250 table on to Google Spreadsheets.

Since you can publish these as RSS feeds, it ought to, in theory, be a great way of generating RSS feeds out of arbitrary content.

There’s just one problem: it doesn’t auto update.

There are claims that it does every hour. Maybe it does when the sheet is open. I don’t know. But it definitely does not when the sheet is closed. I wrote a simple script that logs the time at which the script was accessed, and prints the log every time it is accessed.

#!/usr/bin/env python

import datetime, os.path

print 'Content-Type: text/plain; charset=utf-8'
print ''

logfile = 'timenow.log'
try:    timelog = open(logfile).readlines()
except: timelog = []
timelog.append(str(datetime.datetime.now()) + '\n')
open(logfile, 'w').writelines(timelog)
print ''.join(timelog)

Then I importHtml’ed it into Google spreadsheets, and left it on for the night. Result: absolutely no hits when the document is closed.

Pity. Guess YQL is still the best option.

Comments

pitching manual 24 Jun 2010 5:29 pm: I’ve just started using google spreadsheets. Really impressed with how user friendly they are. I went ine expecting the worse and was very suprised.
Barry 7 Jun 2012 6:27 pm: I am running into the same thing. Can’t get importHtml() to refresh when google spreadsheet is closed. Was thinking I could hammer force a cell value to equal the IH() function via a script but that seems awfully inelegant.
alois 6 Sep 2012 4:20 pm: so what happens when you add a parameter to the url with a value coming from a field which is set to show the current time - minute or hour ?

Split text

Sat, 06 Sep 2008 12:00:00 +0000

This is a series on what Google Spreadsheets can do that Excel can’t.

SPLIT(string, delimiter) splits a string using a delimiter. So if you have “one,two,three,four” in cell A1, you could split that into 4 cells using =SPLIT(A1,",")

That’s similar to Data > Text to Columns, except that if the original data changed, Text to Columns does not revise the output. SPLIT can give you dynamic text-to-columns. This is pretty useful when processing text data, in three ways:

You retain the original data
You don’t need to re-apply Text to Columns. Extending the formula will work (and that’s quicker)
It’s dynamic. If the data changes, your split changes

Since SPLIT returns an array, you can do a bunch of useful things with it.

=COUNTA(SPLIT(A1," ")) gives you the number of words in a string

=SUM(SPLIT(A1,",")) sums up a comma separated list. “1,2,3,4” is added up to 10.

=ARRAYFORMULA(SUM(LEN(SPLIT(A1,",")))) sums up the word lengths. So “one,two,three,four” splits into 4 words of length 3,3,5,4 each, which adds up to 15.

The ability to join and split also lets you sort by multiple keys. For example, say you had income by country and product. You want to show it sorted by Country & Product. You also want to show it by Product & Country.

So first take the data sorted by Country and Product.

Now, in column E, create a key that’s sorted by Product and then by Income. Type

=SORT(ARRAYFORMULA(B2:B10&":"&A2:A10&":"&C2:C10))

… in cell E2. That will give you all the data in one cell, sorted by Product and then by Country. Now, just split the data, as shown here.

Note: You could have done the whole thing using one formula:

=ARRAYFORMULA(SPLIT(SORT(ARRAYFORMULA(B2:B10&":"&A2:A10&":"&C2:C10)),":"))

But for some reason, this doesn’t seem to show the first row properly. No idea why.

Can I do that in Excel?

Well, not really. You’re best off creating a user-defined function to duplicate the SPLIT function.

Comments

Prakash H Ayer 9 Sep 2008 8:25 am: Thanks Anand. This is very useful.
Prakash Ayer 9 Sep 2008 8:25 am: Thanks Anand. This is very useful.
Sridhar 17 Sep 2008 10:19 am: Cant “mid” with “find” and “len” replace “split”? Even though its a round about way i think it can still do it.
Sridhar 17 Sep 2008 11:11 am: @Sridhar: Yes, this can be done, within reason. Chandoo has a post about Splitting text in excel using formulas.
St 27 Sep 2008 6:54 am: It can be done in excel - we used excel to split up a sentence field into constituent words - but it is a bit roundabout - search for spaces or commas, count, split by count of space/comma

Dynamically eliminate duplicates

Thu, 04 Sep 2008 12:00:00 +0000

This is a series on what Google Spreadsheets can do that Excel can’t

To get a list of unique values from a list, use the UNIQUE function on Google Spreadsheets.

For example, if you have a list of browsers in column A, type =UNIQUE(A1:A17) at cell B1 to get a unique list of browsers. This is a dynamic list. If you change the list of browsers, the unique list gets updated automatically.

You can use UNIQUE to create a dynamic pivot table. Quite often, you end up creating a pivot table simply to summarise by one column. The main purpose the pivot table serves is in getting a list of unique values on that column. Plus, it’s a bit heavy on the UI. And every time the data changes, you need to refresh the pivot. But with the UNIQUE function, you can get a dynamic list of unique values, and you can use the COUNTIF and SUMIF function next to each value. Here is an example showing the frequency table of the browsers shown earlier. Column C does a COUNTIF of the unique values on the original list.

You can also use UNIQUE as the input to another formula:

=COUNT(UNIQUE(LIST)) counts the number of unique values

=COUNT(LIST)-COUNT(UNIQUE(LIST)) gives the number of duplicates

=INDEX(UNIQUE(LIST),3) gives you the third unique value

=LARGE(UNIQUE(LIST),3) gives you the third largest unique value

… and so on.

Can I do that in Excel?

You can, but not easily. There are two approaches, but each has its limitations.

A. Use Advanced Filters: easy but static

Create an advanced filter on column A (Alt-D-F-A)
Select Copy to another location
Click in the Copy to box, and then click the cell B1
Select Unique records only
Click OK

But the list of unique values that you get here is static. If you changed one of the values, the list of unique values does not change.

B. Use a complex formulae that are dynamic

First, blank out the duplicates by typing this formula:

=IF(COUNTIF(A$1:A1,A1)=1,A1,"")

adjacent to the first cell (into B1), and dragging it all the way down (to B17).

Now, create a named range (Alt-I-N-D) for these cells (B1:B17) called WithBlanks and another named range called NoBlanks for the cells one column to the right (C1:C17).

On the first cell of NoBlanks (C1), type this formula:

=IF(ROW()-ROW(NoBlanks)+1>ROWS(WithBlanks)-COUNTBLANK(WithBlanks),"",
    INDIRECT(ADDRESS(SMALL((IF(WithBlanks<>"",ROW(WithBlanks),ROW()+
    ROWS(WithBlanks))),ROW()-ROW(NoBlanks)+1),COLUMN(WithBlanks),4)))

Press Ctrl-Shift-Enter rather than Enter, because it’s an array formula. Now drag this all the way down (to C17).

The list in column C is dynamic. If you change a cell in column A, column C is updated. But the formula can only handle one column. Google Spreadsheets’ UNIQUE function works with any number of columns. If you had data in the range A1:D100 and wanted the unique rows, UNIQUE(A1:D100) gets that for you.

Note: I’m staying away from user defined functions. You could, of course, create a UNIQUE function in Excel using Visual Basic. In fact, you should!

Comments

Ken J 19 Sep 2008 1:10 pm: Great entry. I love your site. It is perfect for advanced users.

Let’s hope Google’s pressure will force some innovation at excel.
hk 21 May 2009 3:30 am: What about Data/Consolidate from menu?

Web lookup using Google Spreadsheets

Mon, 31 Dec 2007 12:00:00 +0000

I’d written earlier about Web lookup in Excel. I showed an example how you could create a movie wishlist that showed the links to the torrents from Mininova.

You can do that even easier on Google Spreadsheets. It has 4 functions that let you import external data:

=importData(“URL of CSV or TSV file”).
Imports a comma-separated or tab-separated file.
=importFeed(URL).vLets you import any Atom or RSS feed.
=importHtml(URL, “list” | “table”, index).
Imports a table or list from any web page.
=importXML(“URL”,“query”).
Imports anything from any web page using XPath.

Firstly, you can see straight off why it’s easy to view RSS feeds in Google Spreadsheets. Just use the importFeed function straight away. So, for example, if I wanted to track all 8GB iPods on Google Base, I can import its feed in Google Spreadsheets.

This automatically creates a list of the latest 8GB iPods.

Incidentally, the “Price” column doesn’t appear automatically. It’s a part of the description. But it’s quite easy to get the price using the standard Excel functions. Let’s say the description is in cell C1. =MID(C1, FIND("Price", C1), 20) gets you the 20 characters starting from “Price”. Then you can sort and play around as usual.

The other powerful thing about Google Spreadsheets is the CONTINUE function. The importFeed function creates a multi-dimensional array. You can extract any cell from the array (for example, row 3, column 2 from cell C1) using CONTINUE(C1, 3, 2). So you can just pick up the title and description, or only alternate rows, or put all rows and columns in a single column – whatever.

The most versatile of the import functions is the importXML function. It lets you import any URL (including an RSS feed), filtering only the XPath you need. As I mentioned earlier, you can scrape any site using XPath.

For example, =importXML("http://www.imdb.com/chart/top", "//table//table//table//a") imports the top 250 movies from the IMDb Top 250. the second parameter says, get all links (a) inside a table inside a table inside a table. This populates a list with the entire Top 250.

Now, against each of these, we could get a feed of Mininova’s torrents. Mininova’s RSS URL is http://www.mininova.org/rss/search_string. So, in cell B1, I can get a torrent for the cell A1 (The Godfather) using the importFeed function. (Note: you need to replace spaces with a + symbol. These functions don’t like invalid URLs.).

Just copy this formula down to get a torrent against each of the IMDb Top 250 movies!

Check out the sheet I’ve created. (You need a Google account to see the sheet. If you don’t want have one, you can view the sheet.)

Now, that’s still not the best of it. You can extract this file as an RSS feed! Google lets you publish your sheets as HTML, PDF, Text, XLS, etc. and RSS and Atom are included as well. Here’s the RSS feed for my sheet above.

Think about it. We now have an application that sucks in data from a web page, does a web-based vlookup on another service, and returns the results as an RSS feed!

There are only two catches to this. The first is that Google has restricted us to 50 import functions per sheet. So you can’t really have the IMDb Top 250 populated here – only the top 49. The second is that the spreadsheet updates only when you open it again. So it’s not really a dynamically updating feed. You need to open the spreadsheet to refresh it.

But if you really wanted these things, there’s always Yahoo! Pipes.

Comments

Gregory Dillon 31 Dec 2007 12:00 pm: Hey, thanks. Your page was the most helpful of all. In the example with the movie names, how did you figure out that it was a table x3. or table within table within table.
S Anand 30 Mar 2010 9:04 pm: Looks like the IMDb top 250 page format has changed. =importXML(“http://www.imdb.com/chart/top", “//div[@id=““main””]//table//font/a”) seems to work now.
Sintl 2 Jan 2011 4:24 am: Hi Is there a possibility to get selective text from webpage by using import url function. =ImportHtml(URL; “list” | “table”; index) I want to pull following text Updated Sunday, December 19, 2010 from a html page Thnx
Siamak 26 Aug 2011 11:03 pm: I have an excel file with the name of about 300 businesses that I want to look up their phone numbers or websites. Any idea how I can automate this, rather than type each entry in Google? Thanks
akc 30 Jul 2011 11:07 am: Hey Very Helpful website it is……….Please help me I wan to import data from (http://dsebd.org/latest_share_price_scroll_l.php ) . to google spreadsheet . I wan to import share price . Now tell me which way i will use. I m failed

GTalk is more than a messenger

Fri, 25 Aug 2006 12:00:00 +0000

Google Talk, Not Just An Instant Messenger. It’s used in Gmail, Google Desktop and Google Spreadsheets already. Google plans to use it in more of its services.

How I use Google Spreadsheets

Wed, 28 Jun 2006 12:00:00 +0000

I work across multiple computers (my office laptop, home laptop, client desktop) on a daily basis.

I used to transfer data across these by e-mailing them before I travelled. (I often forgot to do so.) Mostly, these are notes – like telephone numbers, things to buy, places to visit, etc.

Google Notebook solves the problem. But not entirely. I store a lot of my notes on spreadsheets, as lists. For example:

Gadgets to buy (and accompanying research)
Movies I’ve seen
Books to read (and which library they’re available from)
To do lists

That’s what I use Google Spreadsheets for – to share lists with myself, across computers.

Comments

Anand Srinivasan 29 Nov 2006 11:10 pm: I love google docs as the service is now called. I hope that Google will steadily add features to make this service comparable to Excel and Word. Being able to collaborate with others without having to mail these documents is the best feature in my opinion. Do you use Google Calendar ? I love it though i want google to introduce the feature of tasks and to-do lists which may not fit into a daily schedule.
S Anand 30 Nov 2006 12:25 pm: Yeah, collaboration is clearly a big help. In fact, I used it on a project in Nov 2006. I do use GCal as well, mostly for public calendars and reminders (on my mobile).
Nozikins 28 Jun 2006 12:00 pm: I use gmail as my notes, reminders. google spreadseets for work, and data tracking and gathering among peers.

Google Spreadsheets

Thu, 08 Jun 2006 12:00:00 +0000

I got an invite for Google Spreadsheets. Very impressive AJAX interface.

Comments

Dhar 9 Jun 2006 11:36 am: Tried Google Spreadsheets. Love the concept, but I seem to be missing the Grand Picture. How is Google going to integrate things like Calendar, Spreadsheets, Notebook, Browser Sync to give us an “Aha” moment? The glue that should hold these applications together seems to be missing. Quite a few of them are very average and scratch the itch for a small section of web population. Nothing earthshaking like Gmail or search. :(
S Anand 9 Jun 2006 2:31 pm: Funnily, the very apps you mentioned (Calendar, Notebook, Spreadsheets and Browser Sync) are the ones I use the most – as much as GMail and search! You’re right, there isn’t much integration, except perhaps between Calendar and GMail. Maybe it’s coming. But even as standalone apps, I’m surprised at how much I use them. I’ll write about how I’m using these sometime…
Madhu 12 Jun 2006 7:03 pm: Its more when than how i think

google-spreadsheets on S Anand

ImportHtml doesn’t auto-refresh

Comments

Split text

Comments

Dynamically eliminate duplicates

Comments

Web lookup using Google Spreadsheets

Comments

Most popular movies on IMDb

Comments

GTalk is more than a messenger

How I use Google Spreadsheets

Comments

Google Spreadsheets

Comments