Business realities

In a number of sessions I’ve been to, people ask analysts to make their results more interesting – to tell stories with them. I’m co-teaching a course, part of which involves telling stories with data. So this got me thinking: what is a story? How does one teach storytelling to, let’s say, an alien?

Consider this mini-paper.

ABSTRACT: Meter readings exhibit spikes at slab boundaries. We also
find significant evidence of improbably events at round numbers.

Electricity shortage is a serious problem in most Indian states. Part
of this problem is due to the inaccuracy of reporting procedures used
in monitoring meter readings. Our focus here is not to document or
experimentally determine the degree of inaccuracy. We have adopted a
data driven approach to this problem and attempt to model the extent
of inaccuracy using basic statistical analysis techniques such as
histograms and the comparison of means.

Our dataset comprises of the frequency analysis 12-month dataset
containing monthly meter readings of 1.8 million customers in the
State of Andhra Pradesh.

We find that a histogram of these readings shows unexpectedly high
values at the slab boundaries: 50 (+45.342%, t > 13.431), 100
(+55.134%, t > 16.384), 200 (+33.341%, t > 15.232), and 300
(+42.138%, t > 19.958).

We also detected spikes at round numbers: 10 (+15.341%, t > 5.315),
20 (+18.576%, t > 6.152), 30 (+11.341%, t > 4.319).

The statistical significance of every deviation listed above is over
99.9%. Further, every deviation has a positive mantissa. This leads us
to confidently declare the existence of a systematic bias in the meter
readings analysed.

You’re probably thinking: “I know why he’s put this example here. It must be a bad one. So, what a rotten paper it must be!”

Well, not quite. It’s a good piece of analysis. I did it myself and there’s a fair bit of effort and care behind these short paragraphs.

The trouble is, if I read it out to my daughter, she’d say “What?” and not understand a word. My wife’d say “So what?” and not care a bit. I might as well not have written it.

It’s like that Zen thing: If a tree falls in a forest and no on hears it, does it make a sound?

If you did a piece of analysis, and no one understands or cares about it, why did you do it in the first place?

Why do you do it?

That last question is important: why do we analyse?

Sometimes, we do it for fun. The knowledge is beautiful. Knowing Tetris is NP-Complete is rewarding, even though my colleague sarcastically remarked, “Thank God! I’m sooo relieved now that I know that Tetris is NP whatever.” If that’s the case with you, great. Write the analysis any which way you’ll enjoy.

Sometimes, we do it because we’re forced to. In class. At work. Wherever. But that’s another way of saying “I don’t know why I’m doing it.” In that case, I’d gently recommend watching 3 Idiots.

Most often, we do it to share knowledge and drive actions. In that case, if no on understands it, or does anything with it, why do it?

Keep it simple

We prerajulisation of Farhanitate flagellated with ...

Would your audience understand that? Or are you just scared that simple words indicate a simple mind?

I was once afraid. 15 years ago, when writing a paper on IBM India’s competitive advantage for the CXOs, I was worried about it being too simple. I didn’t know anything about management. So I filled it with jargon. They politely nodded when I presented it, but I wasn’t fooling anyone. If there’s no content, jargon doesn’t help.

Unfortunately, it’s become polite to accept jargon as a substitute for substance. Why were they not ripping me apart? Or at least, kindly asking me what on earth I wanted to say?

My friend Manoj did that. In his nice, humble way, he asked, “But Anand, what does this mean?” When I explained it to him, I found I didn’t have a clue. He was OK with that. He just wanted to make sure he hadn’t missed something.

(That’s the technique I use these days. Ask people to explain things clearly. It’s OK if they’re just lost in jargon. I just want to make sure I haven’t missed something.)

Don’t cloak your ignorance. No one will think less of you. In the long run, you’ll learn more, and won’t need the jargon.

Part 2 of the article will talk about focusing on people and actions; storylining and the pyramid principle; and the structure of messages.

As promised, here are some tips from the trenches on surviving in prison. (For those who don’t follow my blog, prison is where your Internet access is restricted.)

There are two things you need to know better: software and people. I’ll try and cover the software in this post, and the more important topic in the next.

Portable apps

You’re often not in control of your laptops / PCs. You don’t have administrator access. You can’t install software. The solution is to install Portable Apps. Most popular applications have been converted into Portable Apps that you can install on to a USB stick. Just plug them into any machine and use them. I use Firefox and Skype quite extensively this way, but increasingly, I have a preference for Portable Apps for just about everything. It makes my bloated Start Menu a lot more manageable. Some of the other portable apps I have are: Audacity, Camstudio, GIMP, Inkscape and Notepad++.

Admin access

The other possibility is that you try and gain admin access. I did this once at a client site (a large bank). We didn’t have admin access. I wasn’t particularly thrilled. So I borrowed a floppy, installed an offline password recovery tool, rebooted, and got the admin password within a few minutes. This is with the full knowledge of the (somewhat worried) client. This is where the people part comes in, and I’ll talk about that later.

Proxies

But before you do any of these, you need to be able to download the files, most of which are executables. Those are probably blocked. Heck, the sites from which you can download these files are probably blocked in the first place.

Sometimes, internal proxies help. Proxies for different geographies may have different degrees of freedom. When I was at IBM, the Internet was accessible from most US proxies, just not from the Indian proxy. So it may just be a matter of finding the right internal proxy.

Or you can search for external public proxies. Sadly, many of these are blocked. Another option is for you to set up your own proxy. You can install mirrorrr on AppEngine for free, for example.

The most effective option, of course, is to use SSH tunnels. I’ve covered this is some detail earlier.

Google

Google has a wide range of tools that can help access blocked sites. If the site you’re accessing provides public RSS feeds, use Google Reader to access these. Public feeds for Twitter, for example, are available as RSS feeds.

Google’s cache is another way of getting the same information. Search for the URL, click on the “Cache” link to read the text even if it’s blocked.

To find more such help, Google for it!

Peopleware

… but all of this is, honestly, just a small part of it. The key, really, is to understand the people restricting your access. I’ll talk about this next.

Eight years ago, I started typing out each of the Calvin and Hobbes strips by hand. Four years ago, I set up a site that let people search for strips. Early this month, I was asked to take it down.

This is the story.

I can’t quite remember when I started reading Calvin & Hobbes. The earliest reference I can find in my blogs is in July 1999. I remember it didn’t take me long to become a fan. I’d read every strip on the newspaper; hunt them out at bookshops; and spend a fair bit of time searching for archives online.

At some point, I discovered a few archives of the complete Calvin & Hobbes images. These aren’t hard to find, and they’re still around in plenty. So that gave me a few more months of delight.

The trouble, though, was that I never could quite find a strip when I wanted to. A friend would refuse to accept something, and I’d want to pull out that strip where Calvin declares to reside in the state of “Denial”. Or if they said something fancy, I’d want to pull out the one where Hobbes says “I notice your oeuvre is monochromatic”. Or those strips where Calvin’s Dad explains how things work (“They build bigger and bigger trucks over the bridge until it breaks.”)

There were a few Calvin and Hobbes search engines around. None quite did what I wanted them to – which was to search the text, and show me the strip, with a nice scrollable interface.

So I set out to build one. I can’t remember when, exactly, but it was before Sep 11, 2002.

It took me many years. I’d spend several train rides and evenings typing this stuff out. My friends, employers and family were a bit puzzled, but just added it to my list of eccentricities and carried on. I was halfway there in 2005, pushed further in 2006, and with some help, I managed to finally complete it.

I was able to do a lot of cool stuff with this, like statistically improbable phrases and some amusing posts as well.

It also increased traffic to my site, which was a bit disconcerting. I didn’t want to attract attention. In 2007, I removed the page from Google’s indexes, which cut the number of hits a fair bit. Since then, the site was only visited by a few people that knew of it, and the occasional stumblers.

A month ago, I got reddit-ed and MetaFiltered.

It didn’t take me long to figure that a takedown notice would be on its way. It turned out to be quite a friendly mail, actually – scary only in parts. (A bit of a carrot-and-stick approach, perhaps.) Anyway, it took me all of 2 minutes to remove all of the pages and links.

Of course, the reason I went to all of this effort was because the original Calvin & Hobbes site does not have the search feature. I’ve reached out to United Media, offering my transcripts and code. Let’s see what happens.

Recently, I have ended up giving bits of advice to people recruiting at start-ups, and a few patterns have emerged that are worth sharing.

Before I go ahead, I should warn you that I have no qualifications whatsoever. (All consulting advice should come with this caveat, perhaps!) You might be better off reading Joel Spolsky’s Smart and Get Things Done (read). I haven’t read it myself, but from what little I see of it, the thoughts seem similar.

The key is to realise that smart people are probably 10 times as productive. OK, that may be wrong. It probably originated with Fred Brooks, and has been debated to death. But it seems fairly well accepted that the best people contribute more than they are better paid. (The best guy is probably paid twice the average, but is worth more than twice the average guy.)

This isn’t because they do more work. It’s because they solve harder problems. You can get two people to do two people’s work. You can’t solve a problem twice as hard even with twenty people.

For a startup, the problem is acute. You don’t have the luxury of being able to manage a large number of people.

Since smart people typically work for a lot less than they’re probably worth, it’s a bargain to hire smart people. You pay them twice as much, and they’ll solve problems twenty others couldn’t solve.

The problem boils down to finding smart people and getting them on board.

Finding smart people

You need to go after the smart people. They won’t come to you. Many reasons. You’re not big enough. There aren’t that many of them. They’re not in the market that much (no one lets go of them anyway).

So that just demolishes the traditional recruitment model straight away. You don’t advertise for people and filter their resumes. You find the people you want and go after them.

The good thing is, smart people cluster. They tend to know other smart people, meet up with other smart people, read the same things as other smart people, etc. That gives some useful starting points.

Matt Biddulph talks about Algorithmic recruitment with Github. The premise is that smart programmers are at the centre of the social networks in their respective areas. Just go after them. I advised a friend similarly: to look for the network (or at least the smart people) that hang out on Stack Overflow for a given topic. Last year, when I was looking for a Django developer, I scoured the Infosys internal blogs for similar networks. (Found only a few, but it sure introduced me to a lot of really smart people that I didn’t know existed!)

Conferences are another place to look for them. I tend to periodically check out Upcoming and Meetup to see who’s taking part in what, go over, meet them, and see what they do. I find it a great way of figuring out who’re the experts in a field. (I once met one of the guys who wrote TiddlyWiki, and it was immediately obvious that he was in a different league from the others that day at the Javascript Meetup.)

You can go a step further. Since smart people cluster, they form networks, and control of that network is power. So why not organise those conferences? A lot of these smart people just need a place to hang out and learn from each other. I know the Javascript Meetup was struggling to find a place to meet. Pubs don’t give you the quiet atmosphere needed to learn from each other, and it’s certainly impossible to have a talk there. The folks at Hackspace have done this really well, renting a place and equipment for people to tinker with electronics.

That’s what smart people want, mostly: a nice quiet place, good company, and perhaps pizza. Skills Matter does this beautifully. They organise free workshops every now and then. The list of people that attend these is invaluable.

Getting them on board

Once you’ve spotted a smart person, what do you offer them?

Remember – they’re probably 10 times as productive. Money is quite likely to be worth offering. If that works, great. But if you’re a startup, you probably don’t have the money. You probably could offer a stake in the firm. That might work too.

But, to quote Dan Pink: “One of the most robust findings of social science is that incentives dull the mind and hamper creativity. Yet, businesses ignore it.” Some people aren’t motivated by money. You might get better results if you didn’t pay money than if you did. (Read this story on motivation by Peter Bregman.)

Suppose you said, “I have this problem… I’ve no idea how to solve it. Would you be able to help me?” Most smart people would probably help you. For free. The feel good feeling is worth more than the transaction cost of extracting payment from you.

Or you might be championing a worthy cause – anywhere from world hunger, rural poverty or cure for cancer down to organising a scout camp. The thing about this is they are intrinsically attractive. You probably just need to open up and say “This is what I’m doing, can you help?”

The flip side of it is loss of control. Jonty told me about how Hackspace London was run: “it’s as loosely organised as possible without falling apart”. You don’t manage these people like traditional organisations. You manage them like a community of volunteers. Like parents at a school day function. Like family at a wedding. You don’t pay them. You don’t order them around either.

Part of that is the flexibility of being a startup. You can afford that loss of control. Yes, you don’t have the money. No, not everyone’s working for money. (The planet as a whole is fairly well off. Smart people particularly so.) But you might offer something interesting. Just as long as you’re willing to let go of some control in your mind…