data-processing on S Anand

Dissecting my Airtel bills

Tue, 11 Aug 2015 13:13:33 +0000

My monthly postpaid mobile bills have been in the Rs 2,000 – Rs 3,000 range for some time now, and I spent a few hours dissecting them yesterday.

Page 3 had the good stuff. It’s a little hard to figure out, but what the last 2 columns say is that most of my spend is offset by discounts.

What’s not getting offset are outgoing roaming calls. Followed by calls to local landlines. For all practical purposes, that’s the only thing that counts in this bill. Everything else is close enough to zero.

It took me some time to figure out that Airtel postpaid has something called myPlan. Based on your plan, you get set of “myPacks” or discounts. That determines your final bill. And it turns out that I was barely using my quota in some areas – specifically data. I had 3GB of data available. I was typically using 200MB – 500MB. The last 2 columns on page 2 show the usage of myPacks.

Clearly, I can do with less data, less SMS, less local mobile, and perhaps less STD mobile. I might need more outgoing roaming, but that’s about it. This means I need fewer myPacks. So I was able to switch to the Rs 799 plan from the Rs 999 plan, while simultaneously increasing the number of free outgoing roaming calls I can make.

There seems to be no myPack for incoming roaming, so I’m actually better off calling people if I’m travelling, rather than receiving calls!

The rest of the bill is a treasure-trove of data, listing every call and every pulse of data connection. I only wish it also had the location of the calls, and were available as CSV files.

Comments

Ashish 11 Aug 2015 6:31 pm: Ask Airtel to start 99/- per month national roaming pack to make all incoming roaming calls free.
408wij 11 Aug 2015 9:29 pm: A quick google search turned up two android apps: (1) Call Log Location and (2) Log Call Location. Perhaps their logs can be exported and cross-referenced w/ your bill.
Bala 1 Sep 2015 12:47 pm: You spend 1 day (>24 hours) in talking on your mobile.
Indrajit Ghosh 18 Jan 2017 1:57 pm: You can ask for a better Data deal from Airtel.

Restartable and Parallel

Thu, 30 Aug 2012 14:38:00 +0000

When processing data at a large scale, there are two characteristics that make a huge difference to my life.

Restartability. When something goes wrong, being able to continue from where it stopped. In my opinion, this is more important than parallelism. There’s nothing as depressing as having to start from scratch every time. Think of it as the ability to save a game as opposed to starting from Level 1 in every life.

Parallelism. Being able to run multiple processes in parallel. Often, this is easy. You don’t need threads. Good old UNIX xargs can do a great job of it. Interestingly, I’ve never used Hadoop for any real-life problem. I’ve gotten by with UNIX commands and smart partitioning.

The “smart partitioning” bit is important. For example, if you’re dealing with telecom data, you’d be calculating most of your metrics (e.g. did the number of calls grow or fall, are there more outgoing or incoming calls, etc.) are calculated on a single mobile number. So if you have multiple data sets, as long as all the data related to one mobile number are on the same system, you’re fine. If you have 100 machines, just split the data based on the last 2 digits of the mobile number. So data about 9012345678 would go to machine 78 (the last two digits). Given a mobile number for any type of data, you’d know exactly which machine would have that data. For all practical purposes, that gives you the basics of a distributed file system.

(I’m not saying you don’t need Hadoop. Just that I haven’t needed it.)

Comments

kamaal 30 Aug 2012 4:04 pm: Anand, If you were to write down- What would be the must learn skills for a data scientist? I’ve sort been trying and working hard on this thing. Of course you must know programming, what else apart from a programmers ordinary skills set does a data scientist need.
Karthik Joshi 3 Sep 2012 9:06 am: I am a reader of your blogs. I love your visualization ideas. This one blog about distribution of work is a great Idea. But I think, it would depend on the nature of data for effective utilization of all 100 machines. (Suppose we have few numbers which will not end with 09 or 88 ro so on) How about hashing? Even though the distributedness of hash algorithm is a debatable issue, do you think this would be beneficial ? Just a thought.

Faster data crunching

Fri, 23 Sep 2011 18:20:10 +0000

I’ve been playing with big data lately.

The good part is, it’s easy to get interesting results. The data is so unwieldy that even average value calculations provoke a “Amazing! I didn’t know that,” response (No exaggeration. I heard this from two separate ~ $1bn businesses this month.)

The bad part is that calculating even that simple average is slow.

For example, take this 40MB file (380MB unzipped) and extract the first column.

The simplest Python script to get the first column looks like this:

```python for row in csv.reader(fileinput.input(), delimiter='\t'): if len(row) > 0: print row[0] ```

That took a good 3 minutes to execute on my laptop.

Since I’m used to UNIX data processing, I tried cut -f1. Weirdly, that’s worse. 5 minutes. Paradoxically, awk '{print $1}' only takes 17 seconds. That's about 12 times faster. Clearly the tool makes a big difference. And we always knew UNIX was fast.

But I also ran these on an Amazon EC2 server, and a Hostgator server. Here’re the results.

	python	cut	awk
My Dell E5400	3:04 (1x)	5:42 (0.5x)	0:17 (11x)
EC2 standard	0:33 (6x)	0:5.6 (33x)	0:16 (11x)
Hostgator	0:19 (10x)	0:2.5 (74x)	0:0.7 (265x)

What took 3 minutes with Python my Dell E5400 took less than a second on Hostgator’s server with awk. Over 250 times faster. (Not 250%. 250 times).

And it’s not just hardware. A good tool (awk) made things 11x faster on my machine. Good hardware (hostgator) made the same program 10x faster. But choosing the right combination can make things go faster than 11 x 10 = 110 times. Much faster.

There are a few of things I’m taking away from this.

Good hardware can speed you up much as (or more than) choosing the right tool.
Good hardware can be rented. From many places. Cheaply.
Always test what’s fast. awk’s fastest on my machine and Hostgator, but not on EC2.

Python vs Perl

Wed, 11 Jan 2006 12:00:00 +0000

Python vs Perl. Sums up my feelings perfectly: Python may be better for larger projects, but for my meddling, I’ll stick to Perl. It’s served me well for 10 years.

Until 1999, I used Perl a fair bit, but no more than Java or C or anything else. My first “real-life” use of Perl was in 2000, when I was processing 600MB of IBES data. Access and SPSS couldn’t handle the load. Perl slurped all the data in a few seconds, though. A few years later, when processing bank data (3GB worth, this time), Perl again was the only saviour. In fact, between Excel and Perl (and CPAN), I think I have all the data analysis power I’ve ever needed. This blog, for instance, is written in an Excel spreadsheet, exported to XML, and converted into the blog format by Perl.

DNA computer

Tue, 19 Mar 2002 12:00:00 +0000

This DNA computer seems to have done some good work.