Coding

HTML5: Up and Running

HTML5: Up and Running is the book version of Mark Pilgrim’s comprehensive introduction to HTML5 at DiveIntoHTML5.org. Whether you buy the book or read it online, it’s the best introduction to the topic you’ll find.

Mark begins with the history of HTML5 (using email archaeology, as he calls it). You’d never guess that many of the problems we have with XHTML, MIME types, etc. have roots in discussions over 20 years ago. From then on, he moves into feature detection (which uses the Modernizr library), new tags, canvas, video, geo-location, storage, offline web apps, new form features and microdata. Each chapter can be read independently – so if you’re planning to use this as a reference, you may be better of reading the links kept up-to-date at DiveIntoHTML5.org. If you’re interesting in learning about the features, it’s a very readable book, terse, simple, and above all, delightfully intelligent.

Incidentally, if you’re starting off on a new HTML5 project, you’re probably best off using HTML5BoilerPlate.com. It’s very actively maintained, and contains some really nifty tricks you can learn like the protocol-relative URL.

Disclosure: I’m writing this post as part of O’Reilly’s blogger review program. While I’m not getting paid to review books, I sure am getting to read them for free.

Modular CSS frameworks

A fair number of the CSS frameworks I’ve seen – Blueprint, Tripoli, YUI, SenCSS – are monolithic. What I’d like is to be able to mix and match specific components of these.

For example, 960.gs has a simple grid system that I’d love to combine with the vertical rhythm that SenCSS offers. (Vertical rhythm ensures that sentences align vertically.) I’d love to have a CSS framework that just sets the fonts, for example, and touches nothing else. Or something that defines the colour schemes, and lets you change the theme like Microsoft Office does.

LessCSS

Less CSS has been invaluable in helping with this. It extends the CSS language without deviating significantly from it. Compared to SASS and CleverCSS, I’d say it has a better chance of getting incorporated as into, say, CSS4.

LessCSS offers variables. I can define a variable:

@foreground: #112233

and use it like this:

h1 { color: @foreground; }
a:hover { background-color: @foreground; }

When I change @foreground, it’s replaced everywhere.

LessCSS offers multiple inheritance.

.highlight { color: red; }
.button { border-radius: 10px; }
.action {
  .highlight;
  .button;
}

This assigns the properties of the highlight and the button classes to the action class. Any changes made to the parents automatically get inherited.

LessCSS has a Javascript pre-processor. So I can include it directly in the HTML, and add the pre-processor, which converts it into CSS.

<link rel="stylesheet/less" href="style.less">
<script src="less.js"></script>

I now use LessCSS as the basis of all new projects.

CSS libraries

My first attempt to consolidate modular CSS libraries is at bitbucket.org/sanand0/csslibs. As far as possible, I’ve tried to avoid creating new libraries, or even tweaking existing ones. Over time, I hope to completely eliminate any new code.

There are two types 2 types of libraries. Some just have variable definitions. Others actually define styles. For example, I’ve got three libraries that just define variables:

color_themes.less

Defines a standard set of color themes (based on the Office 2007 color themes)

font_stacks.less

Defines Web-safe font stacks (based on Sitepoint’s article)

backgrounds.less

Transparent background patterns (randomly useful images)

Including the above libraries will have no effect. You need to explicitly use them. For example:

@import "font_stacks.less";         // Does nothing
h1 { font-family: .font[@serif]; }  // Makes H1 a serif font

The following libraries define styles. Including them will define new classes or change the style of tags / classes.

reset.less

Resets default styles consistently across browsers. I chose YUI3 CSS Reset arbitrarily. I think HTML5boilerplate’s CSS reset may be a better choice, though.

grids.less

Defines classes for fixed and fluid grids. I choose YUI3 CSS Grids over 960.gs (which I’ve used for some years) because of its ability to offer fixed as well as fluid layouts, and the sheer brilliance of its minimality.

lineheight.less

Sets font sizes, ensuring that lines have a vertical rhythm. This is a stripped-down version of SenCSS, but over time, I’ll phase this out and use some standard framework someone comes up with.

Between these, I think the base infrastructure for most applications is in place. What’s required next are widgets. Specifically, I’d like:

  • Buttons. A really good, cross-browser, non-image-based button that offers rounded corners, gradients and borders.
  • Forms. Consistent form styling, without forcing me to use a specific form layout.
  • Icons. A standard icon library with replaceable CSS sprite-sets.

I’ll try keep the code updated as I find these. Do pass me any suggestions you may have.

Make backgrounds transparent

This is the simplest way that I’ve found to make the background colour of an image transparent.

  1. Download GIMP
  2. Open your image. I’ll pick this one:
  3. Optional: Select Image – Mode – RGB if it’s not RGB.
  4. Select Colors – Colors to Alpha…
  5. Click on the white button next to “From” and select the eye-dropper.
  6. Pick the green colour on the image, and click OK

The anti-aliasing is preserved as well.

Shopping with Cooliris

John Lewis jackets scrolling on CoolIris plugin

Zoom-in view of a jacket at John Lewis

I just put together this little demo that scrapes John Lewis’ site and creates a MediaRSS file out of it.

CoolIris has got to be the best way to shop. Apart from being really pretty, it’s quite useful when you know what something looks like, but don’t quite know how to search for it. For example, I was trying to look for a headphone-microphone (you know, the ones that connect into an iPhone or a Blackberry). I didn’t have a clue what it’s called. (TRRS, if you’re interested. I found out later.) The only way I could get it was to browse the wall…

Amazon search for ear microphones on CoolIris

For the curious, here’s the 50-line source code.

ImportHtml doesn’t auto-refresh

A cool thing about Google Spreadsheets is that you can scrape websites using external data functions like importHtml. It’s really easy to use. The formula:

=importHtml("http://www.imdb.com/chart/top", "table", 1)

imports the Internet Movie Database top 250 table on to Google Spreadsheets.

Since you can publish these as RSS feeds, it ought to, in theory, be a great way of generating RSS feeds out of arbitrary content.

There’s just one problem: it doesn’t auto update.

There are claims that it does every hour. Maybe it does when the sheet is open. I don’t know. But it definitely does not when the sheet is closed. I wrote a simple script that logs the time at which the script was accessed, and prints the log every time it is accessed.

#!/usr/bin/env python
 
import datetime, os.path
 
print 'Content-Type: text/plain; charset=utf-8'
print ''
 
logfile = 'timenow.log'
try:    timelog = open(logfile).readlines()
except: timelog = []
timelog.append(str(datetime.datetime.now()) + '\n')
open(logfile, 'w').writelines(timelog)
print ''.join(timelog)

Then I importHtml’ed it into Google spreadsheets, and left it on for the night. Result: absolutely no hits when the document is closed.

Pity. Guess YQL is still the best option.

Command line alarm

When I’m in front of my laptop, I usually forget the world around. Sadly, the world around has important things that need to get done on time. Like eating medicines, turning off the washing machine or the hob, etc.

The one thing I’ve been lacking on my machine was a simple alarm system. I’d like to set an alarm to remind me to do something in 5 minutes, for example. And it should be dead simple to set up.

After hunting around a fair for freeware to do this, I’ve finally settled on writing this tiny piece of Visual Basic code.

Set WshShell = CreateObject("WScript.Shell")
If WScript.Arguments.length &lt; 2 Then
  WScript.Echo "Usage: alarm <time-in-minutes> <message>"
Else
  WScript.Sleep WScript.Arguments.Item(0) * 60 * 1000
  msg = ""
  For i = 1 to WScript.Arguments.Count - 1
      msg = msg + WScript.Arguments.Item(i) + " "
  Next
  WshShell.Popup msg, -1, "Alarm", 64
End If

I’ve saved this as “alarm.vbs” somewhere in my path. When I need to set an alarm, I just type

alarm 5 Turn off the hob

This pops up a window in 5 minutes with the alarm:

An informational popup window saying Turn off the hob

This turned out to be a life-saver yesterday. I had to catch a flight at the Bangalore airport, and traffic is notoriously bad. To be on the safe side, I set up the following:

alarm 25 Catch the flight
alarm 30 You really need to go now
alarm 35 You've missed the flight

Turned out to be a wise thing. I ignored the first alarm. On the second, I said “OK, OK, just 1 minute…” and it really took the third alarm to get me going. Just barely made it to the flight.

SSH Tunneling through web filters

You can defeat most web filters by spending around 8 cents/hr 0 cents/hr on Amazon EC2. (It’s usually worth the money. It’s a fraction of the cost a phone call or a sandwich. And I usually end up wasting that money anyway on calling someone or eating my way out of the misery of corporate proxies.)

Most web filters and proxies block all ports except the HTTP port (80) and the HTTPS port (443). But it’s used to carry encrypted traffic, and, as Mark explains:

since all the traffic that passed through the tunnel is supposed to be SSL encrypted (so as to form an unhindered SSL session between the browser and the HTTPS server), there are little or no access controls possible on such a tunnel

That means web filters can’t really block HTTPS traffic. So we can redirect web traffic to a local HTTPS server, and set up a server outside the firewall that redirects them back to the regular servers.

Putty will be our local HTTPS server. Amazon EC2 gives us a server outside the firewall.

So here’s a 16-step recipe to bypass your web filter. (This is the simplest I could make it.)

In Steps 1-7, we’ll launch a server on Amazon EC2 with 2 tweaks. Step 1 enables Port 443, and step 6 re-configures SSH to run on Port 443 instead of on Port 22. (Remember: most proxies block all ports other than 80 and 443). Alestic’s article on how to Automate EC2 Instance Setup with user-data Scripts and this thread on running SSH on port 443 are invaluable.

In Steps 8-13, we’ll set up Putty as our local HTTPS server. Read how to set up Putty as a SOCKS server and how to use Putty with a HTTP proxy. All I did was to combine the two.

In steps 14-16, we’ll configure the browser to use the Putty as the SOCKS server.

Ingredients

  1. Amazon AWS account (sign up for free – you won’t be charged until you use it)
  2. Putty (which may be available on your Intranet, if you’re lucky)

Directions

  1. On the AWS EC2 Console, click on Security Groups and select the default security group. At the bottom, select HTTPS as the connection method, and save it.
  2. Click on Key Pairs, select Create Key Pair and type in some name. Click on the Create button and you’ll be asked to download a key file. Save it somewhere safe.
  3. Run PuttyGen (it comes with Putty), click Load and select the key file you just saved. Now click on Save private key and save it as privatekey.ppk.
  4. Back on the AWS EC2 Console, click on Launch Instance.
  5. Select Community AMIs and find ami-ccf615a5. It’s a Ubunty Jaunty 9.04 instance that’s been customised to run scripts passed as user-data. You may pick any other alestic instance. (The screenshot below picks a different instance. Ignore that.)
  6. Continue until you get to Advanced Instance Options. Here, copy and paste the following under User Data. Do not make a mistake here!
    #!/bin/bash
    mv /etc/ssh/sshd_config /etc/ssh/x
    sed "s/^#\?Port.*/Port 443/" /etc/ssh/x > /etc/ssh/sshd_config
    /etc/init.d/ssh restart

  7. Keep pressing Continue and Launch the instance. Once launched, click on “Instances” on the left, and keep refreshing the page until the status turns green (running). Now, copy the Public DNS of the instance.
  8. Run Putty. Type in root@<the-public-DNS-you-just-copied> as the host name, and 443 as the port
  9. Under Connection > Proxy, set HTTP as the proxy type. Type in the Proxy hostname and Port you normally use to access the Internet. Select Yes for Do DNS name lookup at proxy end. Type in your Windows login ID and password.
  10. Under Connection > SSH, select Enable Compression.
  11. Under Connection > SSH > Auth, click Browse and select the privatekey.ppk file you’d saved earlier.
  12. Under Connection > SSH > Tunnels, type 9090 as the Source port, Dynamic as the Destination, and click Add.
  13. Now click Open. You should get a terminal into your Amazon EC2 instance.
  14. Open your Browser, and set the SOCKS server to localhost:9090. For Internet Explorer, go to Tools – Options – Connections – LAN Settings, select Use a proxy …, click on Advanced, and type localhost:9090 as the Socks server. Leave all other fields blank.
  15. For Firefox, go to Tools – Options – Advanced – Network – Settings and select Manual proxy configuration. Set the Socks Host to localhost:9090 and leave all other fields blank.
  16. Also, go to URL about:config, and make sure that network.proxy.socks_remote_dns is set to true.

That’s it. You should now be able to check most blocked sites like Facebook and YouTube.

Those who favour the command line may want to automate Steps 1-7 by downloading Amazon’s EC2 API tools. EC2 API tools work from behind a proxy too. The commands you’ll need to use to setup are:

set EC2_HOME=your-ec2-home-directory
set EC2_CERT=your-ec2-certificate
set EC2_PRIVATE_KEY=your-ec2-private-key
ec2-add-keypair mykeypair
ec2-authorize default -p 443
set EC2_JVM_ARGS=-DproxySet=true -DproxyHost=yourproxy \
-DproxyPort=yourport -Dhttps.proxySet=true \
-Dhttps.proxyHost=yourproxy -Dhttps.proxyPort=yourport \
-Dhttp.proxyUser=yourusername -Dhttps.proxyUser=yourusername \
-Dhttp.proxyPass=yourpassword -Dhttps.proxyPass=yourpassword
ec2-run-instances ami-ccf615a5 --key mykeypair --user-data-file your-startup-file-containing-lines-in-step-6

You can go further and use any software (such as Skype) if you install FreeCap. More details are in this article on Secure Firefox and IM with Putty.

Linux users may want to check out ProxyTunnel and this article on Tunneling SSH over HTTP(S).

Update: Follow-ups on hacker news comments, twitter, delicious and digg.

Open source in corporates

Last month, my first application went live.

I’ve been writing code for 20 years. Not one line of my code has been officially deployed in a corporate. (Loser…)

It’s a happy feeling. Someone defined happiness as the intersection of pleasure and meaning. Writing code is pleasurable. Others using it is meaningful.

But this post isn’t quite about that. It’s about the hoops I’ve had to jump through to make this happen.

I’ve been living in a nightmare since March 2009. That was when I decided that I’d try and get corporates to use open source.

March 2009
It began with a pitch to a VC firm. They were looking to build a content management system (CMS). Normally we’d pull together slides that say we’ll deliver the moon. This time, we put together demo based on WordPress’ CMS plugins.

The meeting went fabulously well. We said, “Here’s a demo we’ve built for you. Do you like it?” The business lead (Stuart) was drooling and declared that that’s exactly what they wanted. The IT lead (another Stuart) was happy too, but warned the business users: “Just remember: this isn’t how we do development, so don’t get your hopes up that we can deliver stuff like this :-)”

Time to make my point. I asked, “What’s your policy on open source software?”

The business lead went quiet. “I don’t know,” he finally said. Fair enough.

I turned to the IT lead. “Well, we don’t use it as a matter of policy… there are security concerns…” he said.

“Which web server do you use?”

”Oh, OK. I see what you mean. We use Apache. So on a case to case basis, we have exceptions. But generally we have security concerns.“

”Why? Do you believe open source software is more insecure than commercial software?“

He thought about it for a while. “Well… maybe. I don’t know.” We debated this a bit. Then we found the real issue: “It’s just that we don’t have control over the process. We don’t know enough about it to decide.”

A couple of weeks later, I tried pitching to a newspaper company. This time, it was our sales team that raised the same question. “But… isn’t open source insecure?”

I didn’t even bother pitching any open source stuff to them. But I’d learnt my lessons:

1. Demo the application. Don’t talk about it.
2. Show it to the business first, and then tackle IT.

Aside: June 2009

In June, I got another chance. I was building the website for a large retailer. The very first thing I did was ask to see the Javascript. Total mess, and filled with browser-incompatible DOM requests. So I went over to their web development team.

“Look, why don’t you guys use a Javascript library? It’ll get you cross browser compatibility and compact maintainable code at the same time.”

And, to their credit, they said, “Sure. Which library?”

I showed them this comparison of jQuery (blue), dojo, scriptaculous and mootools…

… and we agreed on jQuery. So, if nothing else, I’ve managed to get one open source library into a corporate.

July 2009

I was also looking at payments, and retailer was looking to replace their chargeback application. Since I had a week off, I built a working PCI compliant prototype on Django. This time, I applied the lessons I’d learned, and demo-ed it to the business, who were thrilled. Time to tackle IT.

I started with the architecture team. Matt on the architecture team was the most approachable. So I went over, demo-ed it, and said, “Matt, this took a week to put together. It’s based on some new technologies. Are you game to try these out?”

He was. And quite enthused about it too. So we put together a proposal for the architecture review board, proposing a new technology stack: Django / Python and MySQL. As before, I showed the demo before I talked technology. I had prepared answers to all security related questions upfront (and practically memorised section 3 of the PCI guidelines.) The clincher, though, was the business case. To build it on Java, it would cost ~1,000 person days. On Django, I’d mostly done it in 5. There was no way of justifying 1,000 person days for an application that could save, at best £100,000 a year.

So they said “Go ahead, we’re fine if operations and infrastructure are fine.”

It was time to find a Django developer in Infosys. I hunted for a couple of weeks but none was available. (Only 2 people knew Django in the first place.) So that effort got canned, and we were back to the 1,000 person day solution. (Which got canned too, later.)

But in the process, I’d learned my third lesson.

3. If you’re trying new technologies, plan on delivering it yourself.

October 2009

Another application popped up that looked like a prime candidate for introducing open source. They were using an Excel application to fraud screen orders, and wanted to make a web app out of it.

I followed the same route as before. Demo it. Show it to business first, then IT. Built it myself. I skipped Architecture, since they’d already approved the technology stack, and took it straight to Infrastructure.

“This application uses Apache as the web server, MySQL as the database, and uses PHP and Javascript for the application logic. Could we get a Linux server to host it?”

Our entire conversation lasted 30 seconds. He said, “No. We use Windows servers” (I was fine)

“… and you’ll need to chance Apache to IIS” (fine again)

“… and we don’t support PHP, so it’ll have to be Java or .NET” (I don’t know .NET or Java… but fine)

“… and we don’t support MySQL, it’ll have to be SQL Server” (fine, I guess)

“… and we don’t have DBAs available until January, so you’ll have to wait.” (definitely not good.)

So back to the drawing board on the technology stack. I needed something in Java (I know very little Java, but nothing at all in .NET) and to avoid the DBA headache, it would have to bundle in a database. I first explored key-value stores like CouchDB, Redis, etc. None of them worked on Java. The only one I found that did was Persevere, and it was a JSON data store, which fit perfectly with my plans.

By this time, I’d also learn my my fourth and most important lesson.

4. Don’t try to promote open source. Just deliver the application

I said, “This is a custom-built application that runs on Java. Could we get a Windows server to host it?”

The answer was “Yes”, and we had it the next day.

PS: December 2009

The application’s deployed and running. It has about 10,000 orders fraud screened by now.

And the lessons are well learnt. So when some came over asking if there was any image resizing solution I knew off, I said: “Sure, who’s your business sponsor?” Then I went over and said, “Let me show you this open source application called ImageMagick. It handles aspect ratios correctly, and can crop too. Doesn’t this look professional?” Then I went over to IT and said, “It’s open source, so you can change it. It has Java bindings, so you can integrate it into your environment. It can handle 8 3000×2400 images a second on my puny laptop. It’s used by your competitors. And I can build it for you if you like.”

I might just have my second open source entry into a corporate this year.

Inline form validation

A List Apart’s article on Inline Validation is one of the most informative I’ve read in a while — and it’s backed by solid data.

Some useful lessons:

  1. Inline validation can reduce form completion time by 40%
  2. Use inline validations where the user doesn’t know if they’ll get it wrong (e.g. is a username available?). Don’t use them if user knows the answer (e.g. their name)
  3. Validate on blur, not on keypress (it’s distracting, and users can’t multitask)

Round buttons with Python Image Library

After much hunting, I finally settled on Hedger Wang’s simple round CSS links as the most acceptable cross-browser round button implementation. The minified CSS is about 2.5KB, and the syntax is very simple. To make an input button into a round button, just wrap it within a <span class="button">:

<span class="button"><input type="submit"></span>

… and it’s just as easy to convert a link into a rounded button:

<a class="button" href=”/”><span>Home</span></a>

It works by using a transparent PNG / GIF that looks like this:

The first button is the default button. The second appears on hover. The bottom two are for disabled buttons.

Can we easily create buttons in different colours?

That’s what this post is about: creating that image with round buttons and gradients.

When I tried creating these rounded buttons myself (and trying to do it in an automated was as usual), I saw 3 possible approaches:

  1. Create it using PowerPoint via Python and export as a PNG.
    So we make a curved box, put in the appropriate gradients and borders, and export it as a PNG. But the problem is I couldn’t figure out how to get transparent PNGs.
  2. Create it in GIMP using script-fu plugin.
    The problem is, I don’t know scheme or GIMP’s API. So I gave up on this as well.
  3. Create it using Python Image Library.
    This was inspired by Nadia’s PIL Tutorial: How to Create a Button Generator. Let me explain how this works.

The first step is to create a ‘button-mask.png’ like this one:

  1. Create a transparent 300 x 120 image in GIMP
  2. Selecting a box from (0,0) to (300,30)
  3. Shrink it by 2 pixels
  4. Convert it to a rounded rectangle with a radius of 80%
  5. Fill this in white
  6. Copy it to 60 pixels below

Now, we need code to create a gradient:

start, end = (192, 192, 224), (255, 255, 255)
grad = Image.new('RGBA', (300, 120))
draw = ImageDraw.Draw(grad)
for y in range(0, 30):
    draw.line(((0,y),(300,y)), fill=rgb(start, end, y/30.0))
    draw.line(((0,y+60),(300,y+60)), fill=rgb(start, end, 1.0-y/30.0))

Now that the gradient is created, convert that into a round button by loading button-mask.png’s alpha layer onto the gradient:

mask = Image.open('button-mask.png').convert('RGBA').split()[3]
border = Image.open('button-border.png').convert('RGBA')
grad.putalpha(mask)
grad.save('button.png')

There it is: a simple round button generator. You can see a sample of these buttons at my Dilbert search site.