htaccess on S Anand

Protect static files on Apache with OpenID

Fri, 18 Nov 2011 23:11:50 +0000

I moved from static HTML pages to web applications and back to static HTML files. There’s a lot to be said for the simplicity and portability of a bunch of files. Static site generators like Jekyll are increasingly popular; I’ve built a simple publisher that I use extensively.

Web apps give you something else, though, that are still useful on a static site. Access control. I’ve been resorting to htpasswd to protect static files, and it’s far from optimal. I don’t want to know or manage users’ passwords. I don’t want them to remember a new ID. I just want to allow specific people to log in via their Google Accounts. (OpenID is too confusing, and most people use Google anyway.)

The easiest option would be to use Google AppEngine. But their new pricing worries me. Hosting on EC2 is expensive in the long run. All my hosting is now out of a shared Hostgator server that offers Apache and PHP.

So, obviously, I wrote a library protects static files on Apache/PHP using OpenID.

Download the code

Say you want to protect /home/www which is accessible at http://example.com/.

Copy .htaccess and _auth/ under /home/www.
In .htaccess, change RewriteBase to /
In _auth/, copy config.sample.php into config.php, and
change $AUTH_PATH to http://example.com/
add permitted email IDs to function allow()

Now, when you visit http://example.com, you’ll be taken to Google’s login page. Once you log in, if your email ID is allowed , you’ll be able to see the file.

Feel free to try, or fork the code.

Comments

S Anand 19 Nov 2011 10:31 am: I could. But what I’ve learnt is that while the static content on this site has survived 14 years, moving from host to host, the same can’t be said of ANY of the apps I’ve written to create the site. I’ve moved from shell scripts to Perl to Python to node.js… and something else will come up. I can’t maintain this stuff. Fortunately, HTML will stay. So I’m making the content primary. I just have a bunch of static files, and that’s the key. This is an OPTIONAL library that sits on the side. Without it, the worst that’ll happen is that the content becomes public. But otherwise, all URLs will remain unbroken. That’s a big win with this approach. That’s also the reason I rejected AppEngine. For my volume of usage, pricing isn’t a serious issue. It’s simplicity.
Manu 19 Nov 2011 5:59 am: If you are willing to host it outside, then heroku allows one free instance per project.
S Anand 2 Dec 2011 10:09 pm: I too will probably stay on WordPress for a while. But recently, I’ve been creating quite a few microblogs for various topics, and find that the workflow of Markdown -> Dropbox -> PHP -> HTML far too compelling to ignore. It’s worked quite well from an ease of publishing perspective. I’m still ironing out few edges. Will publish once I do.
Thejesh GN 25 Nov 2011 11:56 am: I was planning to move to static using http://www.blogofile.com/ (python and variety of templating systems are supported). But I am still betting on wordpress for many things. One day I will probably move to static blog.
Christophe-Marie Duquesne 22 Dec 2013 12:53 pm: Hi, Your post inspired me to write a lighttpd magnet script that protects content regardless of its nature (static or dynamic). I use it:
- for protecting my (static) photo gallery and only giving access to people I select
- for protecting my (dynamic) rss reader and only giving access to myself
- for protecting my (dynamic) online file manager and only giving access to my girlfriend and myself. It is completely pluggable, and it sets the REMOTE_USER server variable so that user-aware applications can use this mechanism to authentify you. https://lighttpd-external-auth.chmd.fr/

Short URLs

Sat, 11 Apr 2009 18:59:45 +0000

With all the discussion around URL shorteners, Diggbar, blocking it, and the rev=canonical proposal, I decided to implement a URL shortening service on this blog with the least effort possible. This probably won’t impact you just yet, but when tools become more popular and sophisticated, it would hopefully eliminate the need for tinyurl, bit.ly, etc.

Since the blog runs on WordPress, every post has an ID. The short URL for any post will simply be http://www.s-anand.net/the_ID. For example, http://s-anand.net/17 is a link to post on Ubuntu on a Dell Latitude D420. At 21 characters, it’s roughly the same size as most URL shorteners could make it.

The code is easy: just one line added to index.php:

<link rev="canonical" href="http://s-anand.net/<?php the_ID(); ?>">

… and one line in my .htaccess:

RewriteRule ^([0-9]+)$ blog/?p=$1 [L,R=301,QSA]

Hopefully someone will come up with a WordPress plugin some time soon that does this. Until then, this should work for you.

Caching pages on Apache

Thu, 14 Aug 2008 12:00:00 +0000

I don’t use any blogging software for my site. I just hand-wired it some years ago. When doing this, one of the biggest problems was caching.

Consider each blog entry page. Each page has the same template, but different content. Both the template and content could be changed. So ideally, blog pages should be served dynamically. That is, every time someone requests the page, I should look up the content, look up the template, and put them together.

I did that, and within a few days outgrew my hosting service’s CPU usage limit. Running such a program for every page hit is too heavy on the CPU.

One way around this is to create the pages beforehand and serve it as regular HTML. But every time the template changes, you need to re-generate every single page. I had over 2,500 pages. That would kill the CPU usage if I changed the template often.

At that point, I did a piece of analysis. Do I really need to regenerate all 2000 blog entries? Wouldn’t the 80-20 rule apply? The Apache log confirmed that 20% of the URLs were accounting for 76% of the hits. So I’d be wasting my time regenerating all the pages every time I changed the template.

So based on this, I decided to dynamically cache the pages. When a page is requested for the first time, I create the page and save it in a cache. The next time, I’d just serve it from the cache. If the template changes, I just need to delete the cache. This way, I only generate pages that are requested, and they’re only generated once.

OK, so that’s the background. Now let me get to how I did it.

I wrote a Perl script, blog.pl, that would generate a page in the html folder whenever it is called. Next, I changed Apache’s .htaccess to run this program only if the page did not exist in the html folder.

# Redirect to cache first
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^([^/]*)\.html$       html/$1.html

# If not found, run program to create page
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^html/([^/]*)\.html$  blog.pl?t=$1

The first block redirects Apache to the cache. The second block checks if the file exists in the cache. If it doesn’t, the Apache redirects to the program. The program creates the page in the cache and displays it. Thereafter, Apache will just serve the file from the cache.

This Apache trick can be used in another way. I keep files organised in different folders to simplify my work. But to visitors of this site, that organisation is irrelevant. So I effectively merge these folders into one. For example, I have a folder called a in which I keep my static content. I also have this piece of code:

RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^([^/]+)$   a/$1

If any file is not found in the main folder, just check in the a/ folder. So I can access the file /a/hindholam.midi as </hindholam.midi> as well.

This can be extended to a series of folders: either as a cascade of caches, or to merge many folders into one.

Comments

Kannan Ekanath 17 Aug 2008 7:00 am: [http://random-thoughts-and-rambling.blogspot.com/]

Hi, I wanted to know a comparison/contrast how setting up blog etc in your own hosted space compares with things like http://sites.google.com. I have always postponed this buying of a domain in favour of things like sites.google.com. Obviously I understand that the advantages would be you can control the LAF, the content, the organisation etc. But have you got a comparison doc or something on what precisely one would get if he hosts things in his own domain instead of a google/yahoo powered wiki style page?
S Anand 17 Aug 2008 1:31 pm: @Kannan: Don’t have anything offhand, Kannan. But buying a domain would make sense anyway – you could point your domain to sites.google.com transparently. If you wanted to build your own blogging system for your site, though, I’d advise against it. I did it because I started in 1999 and there weren’t many blogging systems then. Today, it just doesn’t make sense.
Nirupam 18 Aug 2008 7:43 am: Hi Want to create a site for myself. Should I go for own hosted space or should I start using sites like Geosites etc. I want to deploy web-based applications in my site. Can you please advise?
S Anand 19 Aug 2008 11:24 am: @Nirupam: If you want to deploy your own applications, you should probably create your own site. Get a free hosting service to start with. I’ve used 110mb.com, freestarthost.com and awardspace.com, and found them to be OK.

Handling missing pages

Mon, 24 Dec 2007 12:00:00 +0000

If something goes wrong with my site, I like to know of it. My top three problems are:

The site is down

A page is missing

Javascript isn’t working

This article covers the second topic.

One thing I’m curious about is hits to non-existent pages (404s) on my site. I usually get 404s because:

I renamed the page
Someone typed a wrong URL
Someone followed a wrong link

Find the 404

The first problem is to know when someone gets a 404. I’ve seen sites that tell you to contact the administrator in case of a 404. That’s crazy. The administrator should automatically detect of 404s! Almost every web server provides this facility.

The real issue is attention. I receive 700 404s a day. That’s too much to manually inspect. And most of these are not for proper web pages, but for images (for example, almost all my 404s used to be for browsers requesting favicon.ico) or weird MS Office files.

I’m interested in a small subset of 404 errors. Those that hit a web page, not support files. And those requested by a human, not a search engine or a program.

A decent way of filtering these is to use Javascript in your 404 page. Javascript is typically executed only by browsers (i.e. humans, not search engines), and only in a web page (not images, etc.) So if you serve Javascript in your 404 page, and it gets executed, it’s likely to be a human requesting a web page.

I have a piece of Javascript in my custom 404 page that looks something like this:

<script>(new Image()).src = "/log.pl";</script>

Every time this code runs, it loads a new image. The source of the image is a Perl script, log.pl. Every time log.pl is accessed, it logs the URL from which it was called. I’m reasonably guaranteed that these are web pages a human tried to access.

The reduction in volume is tremendous. On a typical month, I get ~20,000 404 errors. With the Javascript logging, it’s down to around 200 a month, and most of them quite meaningful.

Point to the right page

Sometimes, the change happens because I changed the URLs. I keep fiddling with the site structure. Someone would have links to an old page that I’ve renamed. I may not even know that. Even if I did, they can’t be bothered with requests to change the link. So I’ve got to handle it.

The quickest way, I find, is to use Apache’s mod_rewrite. You can simply redirect the old URL to the new URL. For example, I used to have a link to /calvin.html which I now point to /calvin/. That becomes a simple line on my .htaccess file:

RewriteRule calvin.html  calvinandhobbes.html

I don’t do this for every site restructuring, though. I just restructure, wait for someone to request a wrong page, and when my 404 error log warns me, I create a line in the .htaccess. It keeps the redirections down to a minimum, and only for those links that are actually visited.

Be flexible with the URL structure

Sometimes people type in a wrong link. Often, these are unintentional. Here are some common misspellings for my Hindi songs search.

s-anand.net/hindi/
s-anand.net/Hindi
s-anand.net/hiundi

Occasionally, people are exploring the structure of my site:

s-anand.net/excel
s-anand.net/music
s-anand.net/hits

I need to decide what to do with both cases. For the former, sometimes my URL structure is too restrictive. I mean, why should someone have to remember to type /hindi instead of /Hindi or /hindi/? Who cares about case? Who cares about a trailing slash?

In such cases, I map all the variants to the right URL using mod_rewrite. For example, typing s-anand.net/HiNDi (with or without caps, with or without a slash at the end) will still take you to the right page.

As I keep discovering new mis-spellings, I take a call on whether to add it or not. The decision is usually based on volume. If two people make the same spelling mistake in a day, I almost certainly add the variant. Most of the time, it’s just typing errors like /hiundi which isn’t repeated oftener than once a month.

Provide search

To handle the exploratory URLs, and people following wrong links, I’ve turned my custom 404 page into a search engine.

For example, when someone types s-anand.net/excel, I know they’re searching for Excel. So I just do a Google Custom Search within my site for “excel” – that is, anything following the URL.

It’s a bit more complex than that, actually. I do a bit of tweaking to the URL, like convert punctuations (underscore, hyphen, full-stop, etc.) to spaces, remove common suffixes (.html, .htm) and ignore numbers. Quite often, it matches something on my site that they’re looking for. If not, ideally, I ought to try for various alternatives and subsets of the original search string to figure out a good match. But given that the number of mismatches is down to about one a day, I’m fairly comfortable right now.

What this means, incidentally, is that my site is, by default, a search engine for itself. To search for movie-related stuff on my site, just type s-anand.net/movie and you get a search of the word “movie” on my site. (Sort of like on a9.com, where searching for a9.com/keyword does a search on the keyword.)