user-defined-functions on S Anand

Web lookup using Excel

Thu, 03 May 2007 12:00:00 +0000

Take a look at the Excel screenshot below.

Yes, that’s right. I have a user-defined function called AMAZONPRICE. And it returns these cameras’ prices directly from Amazon.com. (Given the category and some keywords, it returns the price of the bestselling item on Amazon.com.)

Here’s the code behind the function.

Function AmazonPrice(index As String, keywords As String) As String
  Dim xDoc As MSXML2.DOMDocument30
  Set xDoc = New MSXML2.DOMDocument30
  xDoc.async = False
  If xDoc.Load("http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService" _
    "&Version=2005-03-23&Operation=ItemSearch&SubscriptionId=03SDGQDFEB455W53SB82" _
    "&AssociateTag=sanand-20&MinimumPrice=10000&ResponseGroup=OfferSummary,Small" _
    "&Sort=salesrank&SearchIndex=" & index & "&Keywords=" & keywords) Then
    xDoc.setProperty "SelectionLanguage", "XPath"
    xDoc.setProperty "SelectionNamespaces", _
      "xmlns:a=""http://webservices.amazon.com/AWSECommerceService/2005-03-23"""
    AmazonPrice = xDoc.selectSingleNode("/a:ItemSearchResponse//a:Amount").Text
  End If
End Function

This is how it all started…

Flickr has a camera finder that shows the most popular cameras in the Flickr community.

I love comparing gadgets, I’d been doing some research around these cameras, and the Fuji series (because I own a Fuji Finepix S5600). I’d normally make a spreadsheet that compares these cameras on various parameters, including price.

Since I believe in never typing in data, wondered if there was a way to get the prices in automatically…

Two things made this possible.

Amazon offers web services which let you get price (and almost any other) data for their products
Visual Basic lets you use pretty much any ActiveX object as a control. Microsoft offers MSXML which you can use to load (or download) any XML file, and parse it.

The first task is to get the XML feed for a product you want. Amazon lets you do that through by typing in a URL. The best way to construct the URL is through AWSZone. I picked the US ItemSearch method, which searches for a title or keyword within a category, and returns the matches. The feed for the Canon EOS Digital Rebel XT, based on this, would be at:

http://ecs.amazonaws.com/onca/xml?
    Service=AWSECommerceService&
    Version=2005-03-23&
    Operation=ItemSearch&
    SubscriptionId=0525E2PQ81DD7ZTWTK82&
    SearchIndex=Electronics&
    Keywords=Canon EOS Digital Rebel XT&
    Sort=salesrank&
    ResponseGroup=Offers,Small

(You really need to replace the Subscription ID with your own.)

If you retrieved this URL, you have an XML file containing the details of all Canon EOS Digital Rebel XTs, sorted by sales rank.

To load this in Excel, you need to create a UDF in Visual Basic. First, go to Tools - References and enable Microsoft XML, v3.0 or v4.0. Now, to load an XML document, do this:

Dim xDoc As MSXML2.DOMDocument30
Set xDoc = New MSXML2.DOMDocument30
xDoc.async = False
xDoc.Load(url)

If the load succeeds, then you can extract the information fairly easily, using XPath.

xDoc.setProperty "SelectionLanguage", "XPath"
xDoc.setProperty "SelectionNamespaces", _
        "xmlns:a=""http://webservices.amazon.com/AWSECommerceService/2005-03-23"""

The first line says we’ll be searching using XPath. The second line is a workaround to support default namespaces. (Don’t worry about it. I don’t quite get it either.)

Finally, you get the price from the XML tree. In this case, it’s under ItemSearchResponse/Items/Item/OfferSummary/LowestNewPrice/Amount, and it’s in cents.

AmazonPrice = xDoc.selectSingleNode("/a:ItemSearchResponse//a:Amount").Text

That’s it! Now that this function is defined, just pass it the category and keywords, and you have the price of the first matching product. You can retrieve any other information about products as well – like sales rank, weight, ratings, reviews, etc.

Here’s the spreadsheet for you to play around with.

Comments

Saurabh 3 May 2007 7:38 am: There is a simpler way as well. Use Data->Import External Data->New Web Query. While this is not as powerful as the user defined function that you have created, but definitely a good starting point for lesser geeks :) I use this, for example, to have a dashboard kind of view for the status of the multiple projects that I handle at my work. We rely heavily on Microsoft Sharepoint at work, and using the Web Query option, I can easily get a snapshot of the status of various projects. Same can also fetch shareprices from various websites.
S Anand 3 May 2007 8:16 am: Good point, Saurabh! Web Query does make getting stuff out of HTML very easy. My function operates in a different domain, though. It can access XML, which Web Query won’t, but can’t access HTML, which Web Query does. I’m planning a few more posts on how to extend this concept… that will explain what I mean.
rick sherrin 3 May 2007 12:00 pm: Hi, I work at a public library, in the acquisitions department. The majority of the work is manually keying in order information (that is available electronically elsewhere). I read your article about using Excel to retrieve amazon information. This is something that would be very useful to our library and would make my job easier. I have a little excel knowledge, but I have no experience with visual basic or xml, etc. So although it looks straightforward, it is still out of my reach. I would like to be able to enter a column of ISBN numbers in one column, and retrieve title, author, publisher, price, reviews, synopsis, cover image, in adjacent columns. I would appreciate your help if you would give me an example of how I could easily do this. Thanks Rick
dat 12 Aug 2008 8:10 am: useful article! Btw, how do I retrieve other info from amazon (published date, sales rank, customer review, etc)??? thanks
S Anand 13 Aug 2008 2:15 am: @dat: Have a look at http://www.awszone.com/ for examples on other parameters.
Lydia 3 May 2010 12:05 am: this worked until august last year when amazon changed the way the query signs into their system…it now requires a signature with a timestamp in addition to the subscription ID
Bobby Baker 10 Feb 2015 7:09 am: Have you updated the search functions for this or do you have a new code for this?
Vlad 26 Aug 2015 10:00 pm: Hey Annand is it possible o get the racking status fro fedex for a column of tracking numbers?

User defined array functions in Excel

Fri, 23 Feb 2007 12:00:00 +0000

Many languages have functions to process lists (array). These functions usually return a list, so you can pass that to another list function. This chaining of functions is really powerful.

UNIX provides this sort of chaining capability. If I had a cities (with some repetitions) and I wanted to find out how many started with the letter ‘A’, I’d just type:

cat cities | sort | uniq | grep "^A" | wc

cat: types the cities.
sort: sorts the cities alphabetically.
uniq: finds unique cities (works only if sorted - that's why we had to sort the list).
grep: filters the cities. Only allows cities beginning with A.
wc: word count

To do this on Excel, the only way is to

get the unique values. Data - Filter - Advanced Filter, and select “Unique records only”, “Copy the list to another location”, and select a location
get the first letter. =LEFT(cell,1) returns the first letter of the cell.
count the number of “A"s. =COUNT(range, “A”) counts the number of “A"s.

But ideally, I’d like a 1-line formula like

=LENGTH(UNIQUE(GREP("^A", range)))

Excel doesn’t provide these functions by default, but you can add them as user defined functions. Doing this lets you condense several cells into one. Instead of having to copy all your data into a set of unique values, and then adding a column for the first cell, the entire operation can be condensed into one formula.

I consider the following functions the a basic set for list processing.

LENGTH(list) counts the number of elements in a list
INDEX(list, n) returns the nth element of the list
GREP(string, list) returns elements of the list that have the string
UNIQUE(list) filters unique values
UNION(list, list) returns elements in at least one of the lists
INTERSECTION(list, list) returns elements in both lists
DIFFERENCE(list, list) returns the elements in the first list but not the second
REVERSE(list) reverses the order of the list
STRJOIN(separator, list) joins the elements of the list into a string, separated by a separator
STRSPLIT(separator, string) splits the string into a list, using a separator
MVLOOKUP(value, lookup, result) looks up value in “lookup”, and returns the corresponding MULTIPLE values from “result”

I created these UDFs. You can download the functions and play with them. Below are some tasks that you can do with them, that are difficult otherwise.

Get the file name from a path.

=INDEX(REVERSE(STRSPLIT(”", filename)), 1)

- Count the number of unique elements in a range.
  ```excel
=LENGTH(UNIQUE(range))

How many common elements are there in range 1 and range 2?

=LENGTH(INTERSECTION(range1, range2))

- How many words are there in a string?
  ```excel
=LENGTH(STRSPLIT(" ", string))

Get the smallest unique numbers in a range

=SMALL(UNIQUE(range), 5)

- Count the number of mismatches between two lists.
  ```excel
=COUNT(Range1)+COUNT(Range2) - COUNT(INTERSECTION(Range1,Range2))

Get a list of mismatches between two lists.

=STRJOIN(”,",UNION(DIFFERENCE(Range1,Range2), DIFFERENCE(Range2,Range1)))

- Count duplicate entries in a range.
  ```excel
=LENGTH(Range)-LENGTH(UNIQUE(Range))

VLOOKUP multiple values

=MVLOOKUP(A1,Lookup_Range,Return_Range)

- Count the unique matches in a VLOOKUP
  ```excel
=COUNT(UNIQUE(MVLOOKUP(A1,Lookup_Range,Return_Range)

This is a small sample. The power of list processing is phenomenal, especially when combined with array formulas. Download these macros and play with them!

Comments

Krishna 24 Feb 2007 4:10 am: Anand, thanks for an extremely useful stuff.In connection to our discussion on enhancing my excel skills, i think it would help me largely. Krishna
Sundar 26 Feb 2007 4:06 am: Very neat job Anand. Congrats for a super job. Waht is your e-mail address? Thanx in advance…………S.
Anand Srini 26 Feb 2007 9:40 pm: Hi I reproduce this extract from your article “Instead of having to copy all your data into a set of unique values, and then adding a column for the first cell, the entire operation can be condensed into one formula” can you explain why a column has to be added for the first cell ?
ashwin 27 Feb 2007 4:41 pm: Dear Anand, As usual; another useful post from you. Thanks, Ashwin
Kabir 5 Mar 2007 9:01 pm: you truly are a genius
Shiva 6 Mar 2007 2:03 pm: Your site is turning to a Dexters Laboratory
Ramly 19 Mar 2007 4:35 pm: Thank you very much this will help a lot
Mansi 11 Apr 2007 4:22 am: Hi, was just randomly searching for how to generate unique values on excel.. and came across your website. Must say, truly impressive work and this has helped me a lot! Thank you and keep up the good work! :)
James 23 Feb 2007 12:00 pm: Cheers, a lot of great stuff here, makes my Excel work much easier. My hat off to you!
Pete Watkins 23 Feb 2007 12:00 pm: Thanks ever so much for your work in developing these functions, as these have allowed me to avoid a huge amount of drudgery in processing data!

User-defined functions in Excel

Fri, 01 Sep 2006 12:00:00 +0000

Excel lets you create your own functions. If you wanted to create a function that returned the distance between two points (x1,y1) and (x2,y2), you can create a function DIST that takes these 4 parameters, and use it as shown below.

To create such a function,

press Alt-F11 to open the Visual Basic Editor
insert a new module (Alt-I-M)
type the following code:

Anything you declare as a “Function” in Excel’s Visual Basic automatically becomes visible in the Insert-Function dialog box under the category “User Defined” (see examples). The function is normally saved with the file. This is a good idea if you’re going to distribute the file. You can also save your functions in your personal.xls file and load it on startup.

There are 3 places where I suggest using UDFs.

You need to repeat a formula or use an additional cell to get the job done (e.g. replace Excel errors with empty strings)
You can’t get the information from a formula (e.g. a cell’s background colour)
It’s very cumbersome to get the information using formulas (e.g. regular expressions)

Let’s take the first one: replace Excel errors with empty strings. Normally, you’d store the results in a cell (say A2), and have another cell with the formula =IF(ISERROR(A2),"",A2). Instead, create this function NOERROR:

Now you can enclose any Excel function inside a NOERROR() and it’ll filter out the errors.

Notice that cell E2 would’ve had a #N/A error if you’d just used the VLOOKUP. This function also filters out #REF, #DIV/0!, #NAME? and all other errors.

BTW, you see column F displaying the formula in column E. I didn’t type it out. That’s another UDF.

I will talk about the other two places where you use UDFs tomorrow.

Comments

Greg 4 Apr 2007 3:15 pm: This is very helpful! I am having a problem however; I defined the functions in personal.xls (loaded upon startup) and they always work there, but they aren’t available when working with a different file (I get the #NAME error). I also then created variants of them in the specific file, but although they worked, once I made some changes in the file (added a column), they no longer worked even though I re-did the reference properly. Any ideas?