scalability on S Anand

Restartable and Parallel

Thu, 30 Aug 2012 14:38:00 +0000

When processing data at a large scale, there are two characteristics that make a huge difference to my life.

Restartability. When something goes wrong, being able to continue from where it stopped. In my opinion, this is more important than parallelism. There’s nothing as depressing as having to start from scratch every time. Think of it as the ability to save a game as opposed to starting from Level 1 in every life.

Parallelism. Being able to run multiple processes in parallel. Often, this is easy. You don’t need threads. Good old UNIX xargs can do a great job of it. Interestingly, I’ve never used Hadoop for any real-life problem. I’ve gotten by with UNIX commands and smart partitioning.

The “smart partitioning” bit is important. For example, if you’re dealing with telecom data, you’d be calculating most of your metrics (e.g. did the number of calls grow or fall, are there more outgoing or incoming calls, etc.) are calculated on a single mobile number. So if you have multiple data sets, as long as all the data related to one mobile number are on the same system, you’re fine. If you have 100 machines, just split the data based on the last 2 digits of the mobile number. So data about 9012345678 would go to machine 78 (the last two digits). Given a mobile number for any type of data, you’d know exactly which machine would have that data. For all practical purposes, that gives you the basics of a distributed file system.

(I’m not saying you don’t need Hadoop. Just that I haven’t needed it.)

Comments

kamaal 30 Aug 2012 4:04 pm: Anand, If you were to write down- What would be the must learn skills for a data scientist? I’ve sort been trying and working hard on this thing. Of course you must know programming, what else apart from a programmers ordinary skills set does a data scientist need.
Karthik Joshi 3 Sep 2012 9:06 am: I am a reader of your blogs. I love your visualization ideas. This one blog about distribution of work is a great Idea. But I think, it would depend on the nature of data for effective utilization of all 100 machines. (Suppose we have few numbers which will not end with 09 or 88 ro so on) How about hashing? Even though the distributedness of hash algorithm is a debatable issue, do you think this would be beneficial ? Just a thought.

Channel economics

Sun, 22 Jan 2006 12:00:00 +0000

We were working with the financing subsidiary of a conglomerate. They had two divisions that gave loans for buying vehicles (mostly trucks, but also cars). One division used the direct channel. They had direct marketing agents (DMAs) who were paid a commission for getting the contract, and the division collected the monthly installments. The other used the dealer channel. The dealers would get the contract as well as collect the installments.

They wanted to cut costs, and asked us which channel had more flab. Since the company used IRR (internal rate of return), we defined the operating cost as the reduction in IRR. For example:

12% IRR paid by customer (through monthly installments)

9% IRR to subsidiary after reducing the cost of processing his loan

3% is therefore the operating cost.

After two months of analysis, we confirmed the subsidiary's own opinion: the dealer channel had lower operating cost. The direct channel's operating cost was 3.8% while the the dealer channel's was 2.7%.

So we said the direct channel is flabby.

But the direct channel guys didn't agree, and fought every inch of the decision.

"Do you think these figures are wrong?" I asked.

"Look, all we're saying is, we KNOW they pay out huge commissions to dealers. We KNOW they're overstaffed. They just CAN'T have a lower operating cost."

I was tasked with resolving the issue. After a month of breaking the cost every single way, something interesting emerged. If we measured the operating cost per contract in Rupees, both divisions had the same cost per contract: Rs 18,500. That is, the total cost incurred in getting the customer and servicing the loan over the lifetime of the loan was Rs 18,500 in both divisions.

It turned out that the size of the loan was different: the dealer channel was still lending mainly for trucks, while the direct channel had entered the high growth passenger car market. Cars cost less than trucks. So while the dealer channel was paying 18,500 and getting interest on a large truck, the direct channel was paying the same 18,500 for less interest on smaller cars.

This is a strategic decision. The subsidiary had chosen to enter the car business knowing it would be less profitable but have higher growth.

But the story had a twist.

The 18,500 of operating cost per contract broke down as follows:

	Dealer	Direct
Getting the loan	5,000	7,000
Servicing the loan	13,500	11,500

The dealers are paid a servicing cost as a percentage of the loan. Servicing in the dealer channel is a variable cost. The direct channel, however, employs its own people, and incurs a higher cost only when it hires more people. About half of the costs are fixed. If the business doubles, the number of people you need increases only by about 50%. Servicing in the direct channel is more a fixed cost.

This subsidiary was planning to double their business in two years. At that point, the dealer channel would still cost Rs 18,500 per contract, but the direct channel would have come down to around Rs 13,000. So, going forward, the direct channel is really cheaper!

We told them to try and reduce the dealer commission.

Postscript: The subsidiary still went ahead and cut costs aggressively in the direct channel. It's easier to fire your own people than to tell 500 dealers to reduce their commission, especially when you need them to also sell your trucks.

Comments

jayant 22 Jan 2006 11:39 am: Just out of curiosity, how did the company’s (guess Tata Motors) car sales performed?
S Anand 22 Jan 2006 5:39 pm: Pretty good car sales, from what I hear. But the truck sales really boomed (nothing to do with our work, though).
Madhu 24 Jan 2006 12:43 pm: Just created a blog. grandebelf.blogspot.com. Currently just a information provider
S Anand 24 Jan 2006 5:43 pm: What does ‘grandebelf’ mean?
Madhu 25 Jan 2006 3:22 pm: Grande = Big, Belf=Baby Elephant

Gmail performance at 1GB

Tue, 27 Jul 2004 12:00:00 +0000

Kevin Rose tests gmail performance at 1 GB.

Daypop is out of disk space

Wed, 18 Sep 2002 12:00:00 +0000

Daypop is out of disk space. Poor thing! Could we lend it some? I mean, could sites like Daypop work out a distributed architecture, using web services? Or using things like space drives? via Scripting News