javascript on S Anand

Mapscii Rendering

Wed, 26 Nov 2025 00:00:00 +0530

I asked Claude Sonnet 4.5 on GitHub Copilot:

How does mapscii render vector tiles as text blocks on the terminal? What are the libraries it uses and how does it use them? Explain as a detailed tutorial with examples so that I can learn how to rasterize vector shapes on the terminal.

Below’s the explanation. What’s noteworthy to me is:

How it uses Braille characters to get 8 pixels per character cell (2x4 grid).
How it fills polygons using (a) triangulation and (b) getting all edge points using Bresenham’s algorithm, then (c) filling horizontal spans between edge pairs.

I’ll help you understand how mapscii renders vector tiles as text blocks on the terminal. Let me search through the repository to find the relevant code and understand the architecture. Now let me get more specific files to understand the rendering pipeline better:# How MapSCII Renders Vector Tiles as Text Blocks on the Terminal

MapSCII is a fascinating project that transforms geographic vector tiles into ASCII/Braille art on your terminal. Let me break down the complete rendering pipeline with detailed examples so you can learn how to rasterize vector shapes on the terminal.

Architecture Overview

The rendering pipeline consists of several key components:

TileSource - Fetches and manages vector tiles
Tile - Parses vector tile data (Protobuf format)
Renderer - Orchestrates the rendering process
Canvas - Provides drawing primitives (lines, polygons)
BrailleBuffer - Converts pixels to Braille/ASCII characters

Key Libraries Used

{
  "@mapbox/vector-tile": "^1.3.1", // Parse vector tile format
  "pbf": "^3.2.1", // Protobuf decoding
  "earcut": "^2.2.2", // Polygon triangulation
  "rbush": "^3.0.1", // 2D spatial indexing
  "bresenham": "0.0.4", // Line rasterization
  "simplify-js": "^1.2.4", // Polyline simplification
  "x256": "0.0.2" // Color conversion to xterm-256
}

Step-by-Step Tutorial

Step 1: Understanding Braille Characters for Pixels

Braille characters are perfect for terminal graphics because each character represents a 2×4 pixel grid:

// BrailleBuffer uses Unicode Braille patterns (U+2800 - U+28FF)
// Each Braille character has 8 dots in a 2×4 grid:
//
// ⠁ ⠂  (left column: 0x1, 0x2, 0x4, 0x40)
// ⠄ ⠈  (right column: 0x8, 0x10, 0x20, 0x80)
// ⠐ ⠠
// ⠈ ⡀

const brailleMap = [
  [0x1, 0x8],
  [0x2, 0x10],
  [0x4, 0x20],
  [0x40, 0x80],
];

Each Braille character gives you 8 pixels in just one terminal cell, effectively quadrupling your resolution!

Step 2: Creating a Braille Buffer

class BrailleBuffer {
  constructor(width, height) {
    // Map of which bits correspond to which dot positions
    this.brailleMap = [
      [0x1, 0x8],
      [0x2, 0x10],
      [0x4, 0x20],
      [0x40, 0x80],
    ];

    // Store pixel state, colors
    const size = (width * height) / 8; // Each cell stores 8 pixels
    this.pixelBuffer = Buffer.alloc(size);
    this.foregroundBuffer = Buffer.alloc(size);
    this.backgroundBuffer = Buffer.alloc(size);
  }

  // Set a single pixel at (x, y)
  setPixel(x, y, color) {
    if (x >= 0 && x < this.width && y >= 0 && y < this.height) {
      // Project x,y to buffer index
      const idx = (x >> 1) + (this.width >> 1) * (y >> 2);

      // Get the specific dot mask for this pixel
      const mask = this.brailleMap[y & 3][x & 1];

      // Set the bit
      this.pixelBuffer[idx] |= mask;
      this.foregroundBuffer[idx] = color;
    }
  }

  // Generate terminal output
  frame() {
    const output = [];
    let currentColor = null;

    for (let y = 0; y < this.height / 4; y++) {
      for (let x = 0; x < this.width / 2; x++) {
        const idx = (y * this.width) / 2 + x;

        // Add ANSI color codes when color changes
        const colorCode = this._termColor(
          this.foregroundBuffer[idx],
          this.backgroundBuffer[idx],
        );
        if (currentColor !== colorCode) {
          output.push(currentColor = colorCode);
        }

        // Convert pixel bits to Braille character
        const brailleChar = String.fromCharCode(0x2800 + this.pixelBuffer[idx]);
        output.push(brailleChar);
      }
      output.push("\n");
    }

    return output.join("");
  }

  _termColor(foreground, background) {
    if (foreground && background) {
      return `\x1B[38;5;${foreground};48;5;${background}m`;
    } else if (foreground) {
      return `\x1B[49;38;5;${foreground}m`;
    }
    return "\x1B[39;49m"; // Reset
  }
}

Step 3: Drawing Lines with Bresenham’s Algorithm

const bresenham = require("bresenham");

class Canvas {
  constructor(width, height) {
    this.buffer = new BrailleBuffer(width, height);
  }

  // Draw a line from point A to point B
  line(from, to, color, width = 1) {
    if (width === 1) {
      // Simple bresenham for single-pixel width
      bresenham(from.x, from.y, to.x, to.y, (x, y) => {
        this.buffer.setPixel(x, y, color);
      });
    } else {
      // Thick line using Bresenham with width
      this._thickLine(from.x, from.y, to.x, to.y, width, color);
    }
  }

  // Draw a polyline (e.g., roads, rivers)
  polyline(points, color, width = 1) {
    for (let i = 1; i < points.length; i++) {
      this.line(points[i - 1], points[i], color, width);
    }
  }
}

Step 4: Drawing Filled Polygons with Earcut

For filled areas (buildings, water bodies), MapSCII uses polygon triangulation:

const earcut = require("earcut");

class Canvas {
  polygon(rings, color) {
    // Convert rings to flat array of vertices
    const vertices = [];
    const holes = [];

    for (const ring of rings) {
      if (vertices.length) {
        holes.push(vertices.length / 2); // Mark hole start
      }
      for (const point of ring) {
        vertices.push(point.x);
        vertices.push(point.y);
      }
    }

    // Triangulate the polygon
    const triangles = earcut(vertices, holes);

    // Draw each triangle
    for (let i = 0; i < triangles.length; i += 3) {
      const pa = [vertices[triangles[i] * 2], vertices[triangles[i] * 2 + 1]];
      const pb = [
        vertices[triangles[i + 1] * 2],
        vertices[triangles[i + 1] * 2 + 1],
      ];
      const pc = [
        vertices[triangles[i + 2] * 2],
        vertices[triangles[i + 2] * 2 + 1],
      ];

      this._filledTriangle(pa, pb, pc, color);
    }
  }

  _filledTriangle(pointA, pointB, pointC, color) {
    // Get edges using Bresenham
    const a = bresenham(pointB[0], pointB[1], pointC[0], pointC[1]);
    const b = bresenham(pointA[0], pointA[1], pointC[0], pointC[1]);
    const c = bresenham(pointA[0], pointA[1], pointB[0], pointB[1]);

    // Combine and sort all edge points by Y, then X
    const points = a
      .concat(b)
      .concat(c)
      .filter((p) => p.y >= 0 && p.y < this.height)
      .sort((a, b) => (a.y === b.y ? a.x - b.x : a.y - b.y));

    // Fill horizontal spans between edge pairs
    for (let i = 0; i < points.length - 1; i++) {
      const point = points[i];
      const next = points[i + 1];

      if (point.y === next.y) {
        // Draw horizontal line
        for (let x = point.x; x <= next.x; x++) {
          this.buffer.setPixel(x, point.y, color);
        }
      } else {
        this.buffer.setPixel(point.x, point.y, color);
      }
    }
  }
}

Step 5: Processing Vector Tiles

const VectorTile = require("@mapbox/vector-tile").VectorTile;
const Protobuf = require("pbf");
const RBush = require("rbush");
const zlib = require("zlib");

class Tile {
  async load(buffer) {
    // Decompress if gzipped
    if (this._isGzipped(buffer)) {
      buffer = await new Promise((resolve, reject) => {
        zlib.gunzip(buffer, (err, data) => {
          err ? reject(err) : resolve(data);
        });
      });
    }

    // Parse Protobuf
    this.tile = new VectorTile(new Protobuf(buffer));

    // Process each layer
    const layers = {};
    for (const name in this.tile.layers) {
      const layer = this.tile.layers[name];
      const nodes = [];

      for (let i = 0; i < layer.length; i++) {
        const feature = layer.feature(i);
        const geometries = feature.loadGeometry();

        // Get style for this feature
        const style = this.styler.getStyleFor(name, feature);
        if (!style) continue;

        // Convert geometries to nodes with boundaries
        for (const points of geometries) {
          nodes.push(
            this._addBoundaries({
              layer: name,
              style,
              points,
              color: this._getColor(style),
            }),
          );
        }
      }

      // Create spatial index for efficient querying
      const tree = new RBush(18);
      tree.load(nodes);

      layers[name] = {
        extent: layer.extent,
        tree,
      };
    }

    return layers;
  }
}

Step 6: The Complete Rendering Pipeline

class Renderer {
  async draw(center, zoom) {
    this.canvas.clear();

    // 1. Calculate visible tiles
    const tiles = this._visibleTiles(center, zoom);

    // 2.  Fetch all tiles in parallel
    await Promise.all(
      tiles.map(async (tile) => {
        tile.data = await this.tileSource.getTile(
          tile.xyz.z,
          tile.xyz.x,
          tile.xyz.y,
        );

        // 3. Query spatial index for visible features
        const scale = tile.data.layers[layerId].extent / tileSize;
        tile.layers[layerId].features = tile.data.layers[layerId].tree.search({
          minX: -tile.position.x * scale,
          minY: -tile.position.y * scale,
          maxX: (this.width - tile.position.x) * scale,
          maxY: (this.height - tile.position.y) * scale,
        });
      }),
    );

    // 4. Draw features in correct order (background to foreground)
    const drawOrder = ["water", "landuse", "building", "road", "poi_label"];
    for (const layerId of drawOrder) {
      for (const tile of tiles) {
        for (const feature of tile.layers[layerId].features) {
          this._drawFeature(tile, feature, scale);
        }
      }
    }

    // 5. Generate terminal output
    return this.canvas.frame();
  }

  _drawFeature(tile, feature, scale) {
    // Transform coordinates from tile space to screen space
    const points = feature.points.map((p) => ({
      x: Math.floor(tile.position.x + p.x / scale),
      y: Math.floor(tile.position.y + p.y / scale),
    }));

    // Draw based on geometry type
    switch (feature.style.type) {
      case "line":
        this.canvas.polyline(
          points,
          feature.color,
          feature.style.paint["line-width"],
        );
        break;
      case "fill":
        this.canvas.polygon([points], feature.color);
        break;
      case "symbol":
        this.canvas.text(
          feature.label,
          points[0].x,
          points[0].y,
          feature.color,
        );
        break;
    }
  }
}

Step 7: Coordinate Transformations

const utils = {
  // Convert lat/lon to tile coordinates at zoom level
  ll2tile(lon, lat, zoom) {
    const n = Math.pow(2, zoom);
    return {
      x: ((lon + 180) / 360) * n,
      y: ((1
        - Math.log(
            Math.tan((lat * Math.PI) / 180)
              + 1 / Math.cos((lat * Math.PI) / 180),
          ) / Math.PI) / 2) * n,
    };
  },

  // Convert tile coordinates back to lat/lon
  tile2ll(x, y, zoom) {
    const n = Math.pow(2, zoom);
    return {
      lon: (x / n) * 360 - 180,
      lat: (Math.atan(Math.sinh(Math.PI * (1 - (2 * y) / n))) * 180) / Math.PI,
    };
  },

  // Get tile size in pixels at fractional zoom
  tilesizeAtZoom(zoom) {
    return Math.pow(2, zoom - Math.floor(zoom)) * 256;
  },
};

Complete Example: Drawing a Simple Map

const Canvas = require("./Canvas");
const x256 = require("x256");

// Create a 160x64 pixel canvas (80x16 terminal cells in Braille mode)
const canvas = new Canvas(160, 64);

// Define some colors (xterm-256 color codes)
const BLUE = x256([0, 0, 255]);
const GREEN = x256([0, 200, 0]);
const GRAY = x256([128, 128, 128]);

// Draw a river (polyline)
const riverPoints = [
  { x: 20, y: 50 },
  { x: 40, y: 45 },
  { x: 60, y: 40 },
  { x: 80, y: 35 },
  { x: 100, y: 30 },
];
canvas.polyline(riverPoints, BLUE, 3);

// Draw a building (polygon)
const buildingRing = [
  { x: 70, y: 20 },
  { x: 90, y: 20 },
  { x: 90, y: 40 },
  { x: 70, y: 40 },
];
canvas.polygon([buildingRing], GRAY);

// Draw a road (polyline)
const roadPoints = [
  { x: 0, y: 25 },
  { x: 160, y: 25 },
];
canvas.polyline(roadPoints, GRAY, 2);

// Draw a label
canvas.text("Town Hall", 80, 30, GRAY);

// Output to terminal
process.stdout.write(canvas.frame());

Performance Optimizations

MapSCII uses several clever optimizations:

RBush Spatial Index: Only processes features visible in the viewport
Polyline Simplification: Uses simplify-js to reduce point counts
Tile Padding: Renders slightly beyond viewport to avoid edge artifacts
Draw Order Optimization: Different layer orders for different zoom levels
Color Caching: Avoids repeated color conversions

Key Takeaways

To rasterize vector shapes on the terminal:

Use Braille characters for 4x resolution boost (2×4 pixels per cell)
Implement a pixel buffer that maps coordinates to Braille dot positions
Use Bresenham’s algorithm for line rasterization
Use Earcut for polygon triangulation and scanline filling
Leverage xterm-256 colors with ANSI escape codes
Transform coordinates from geographic space to screen space
Use spatial indexing (RBush) for efficient viewport queries

The complete rendering pipeline is: Vector Tiles → Parse Protobuf → Query Spatial Index → Transform Coordinates → Rasterize to Pixels → Convert to Braille → Output ANSI Codes

You can explore more implementation details in the rastapasta/mapscii repository.

Patterns for Short Code

Wed, 26 Nov 2025 00:00:00 +0530

I had Claude Code create a PR to update my Unicoder tool. As part of that, I prompted it to repeatedly:

Shorten the code by exploring opportunities to rewrite more elegantly.

Finally, I asked it:

You applied several changes to refactor the code for elegance and brevity.

What were the principles you applied? List them all with examples. The objective is to teach me how to shorten and simplify code elegantly.

Its response was a good tutorial on refactoring to shorten and simplify code.

Excellent question! Here are the key refactoring principles I applied, with concrete examples from the code:

1. Data-Driven Design Over Repetitive Conditionals

Replace repetitive if/else chains with data structures.

Before (30 lines):

const toAscii = (char) => {
  const code = char.codePointAt(0);

  if (code >= 0x1d5d4 && code <= 0x1d5ed) return String.fromCharCode(code - UNICODE_OFFSETS.BOLD_UPPER);
  if (code >= 0x1d5ee && code <= 0x1d607) return String.fromCharCode(code - UNICODE_OFFSETS.BOLD_LOWER);
  if (code >= 0x1d608 && code <= 0x1d621) return String.fromCharCode(code - UNICODE_OFFSETS.ITALIC_UPPER);
  // ... 4 more similar checks
  return char;
};

const detectStyle = (char) => {
  const code = char.codePointAt(0);
  if ((code >= 0x1d5d4 && code <= 0x1d5ed) || (code >= 0x1d5ee && code <= 0x1d607)) return "bold";
  // ... 2 more similar checks
  return null;
};

After (15 lines):

const CHAR_RANGES = [
  { start: 0x1d5d4, end: 0x1d5ed, offset: UNICODE_OFFSETS.BOLD_UPPER, style: "bold" },
  { start: 0x1d5ee, end: 0x1d607, offset: UNICODE_OFFSETS.BOLD_LOWER, style: "bold" },
  // ... all ranges in one place
];

const toAscii = (char) => {
  const code = char.codePointAt(0);
  const range = CHAR_RANGES.find((r) => code >= r.start && code <= r.end);
  return range ? String.fromCharCode(code - range.offset) : char;
};

const detectStyle = (char) => {
  const code = char.codePointAt(0);
  const range = CHAR_RANGES.find((r) => code >= r.start && code <= r.end);
  return range?.style ?? null;
};

Principle: Define data once, process it generically. Both functions now share the same data.

2. Optional Chaining & Nullish Coalescing

Use modern JavaScript operators for cleaner null handling.

Before:

const formatSegment = (text, style) => {
  if (!style) return text;
  switch (style) {
    case "bold": return `**${text}**`;
    case "italic": return `*${text}*`;
    case "mono": return `\`${text}\``;
    default: return text;
  }
};

After:

const formatSegment = (text, style) => {
  if (!style) return text;
  const formatters = { bold: (t) => `**${t}**`, italic: (t) => `*${t}*`, mono: (t) => `\`${t}\`` };
  return formatters[style]?.(text) ?? text;
};

Principle: ?. safely accesses properties, ?? provides defaults. Object lookup replaces switch.

3. Pattern Arrays for Iterative Operations

Use data structures to eliminate repetitive code.

Before:

const mergeStyledSegments = (markdown) => {
  markdown = markdown.replace(/\*\*([^*]+)\*\* \*\*([^*]+)\*\*/g, "**$1 $2**");
  markdown = markdown.replace(/\*([^*]+)\* \*([^*]+)\*/g, "*$1 $2*");
  markdown = markdown.replace(/`([^`]+)` `([^`]+)`/g, "`$1 $2`");

  const original = markdown;
  markdown = markdown.replace(/\*\*([^*]+)\*\* \*\*([^*]+)\*\*/g, "**$1 $2**");
  markdown = markdown.replace(/\*([^*]+)\* \*([^*]+)\*/g, "*$1 $2*");
  markdown = markdown.replace(/`([^`]+)` `([^`]+)`/g, "`$1 $2`");

  return markdown !== original ? mergeStyledSegments(markdown) : markdown;
};

After:

const mergeStyledSegments = (markdown) => {
  const patterns = [
    [/\*\*([^*]+)\*\* \*\*([^*]+)\*\*/g, "**$1 $2**"],
    [/\*([^*]+)\* \*([^*]+)\*/g, "*$1 $2*"],
    [/`([^`]+)` `([^`]+)`/g, "`$1 $2`"],
  ];

  let prev;
  do {
    prev = markdown;
    patterns.forEach(([pattern, replacement]) => markdown = markdown.replace(pattern, replacement));
  } while (markdown !== prev);

  return markdown;
};

Principle: DRY—define patterns once, iterate over them. Easier to add new patterns.

4. Destructuring for Clarity

Extract only what you need from objects.

Before:

const copyToClipboard = (text, button) => {
  const originalHTML = button.innerHTML;
  const originalClasses = [...button.classList];

  button.textContent = "Copied!";
  button.classList.remove("btn-light");
  button.classList.add("btn-success");

  setTimeout(() => {
    button.innerHTML = originalHTML;
    button.className = "";
    originalClasses.forEach((cls) => button.classList.add(cls));
  }, 2000);
};

After:

const copyToClipboard = (text, button) => {
  const { innerHTML, className } = button;

  button.textContent = "Copied!";
  button.className = button.className.replace("btn-light", "btn-success");

  setTimeout(() => {
    button.innerHTML = innerHTML;
    button.className = className;
  }, 2000);
};

Principle: Destructure to extract values. Use className string instead of classList manipulation.

5. Arrow Functions with Implicit Returns

Eliminate unnecessary syntax for single-expression functions.

Before:

const handleCopy = (outputId, buttonId) => () => {
  copyToClipboard(document.getElementById(outputId).innerText, document.getElementById(buttonId));
};

After:

const handleCopy = (outputId, buttonId) => () =>
  copyToClipboard(document.getElementById(outputId).innerText, document.getElementById(buttonId));

Principle: Single-expression functions don’t need braces or return. One line when clear.

Merge functions that serve similar purposes.

Before:

const showError = (message) => {
  const errorContainer = document.getElementById("error-container");
  errorContainer.textContent = `Error: ${message}`;
  errorContainer.classList.remove("d-none");
};

const hideError = () => {
  document.getElementById("error-container").classList.add("d-none");
};

After:

const setError = (message = null) => {
  const el = document.getElementById("error-container");
  if (message) {
    el.textContent = `Error: ${message}`;
    el.classList.remove("d-none");
  } else {
    el.classList.add("d-none");
  }
};

Principle: One function with optional parameter beats two separate functions. setError() hides, setError(msg) shows.

7. Object Literal Shorthand

Use computed expressions in object literals.

Before:

const convertCodeBlock = (text) => {
  const converted = [...text].map((c) => (detectStyle(c) === "mono" ? toAscii(c) : c)).join("");
  return { markdown: "```\n" + converted + "\n```" };
};

After:

const convertCodeBlock = (text) => ({
  markdown: "```\n" + [...text].map((c) => (detectStyle(c) === "mono" ? toAscii(c) : c)).join("") + "\n```",
});

Principle: Eliminate intermediate variables when computation is clear. Use ({...}) for implicit object return.

8. Remove Dead Code

Delete unused functions ruthlessly.

Removed:

const fromUnicodeStyle = (text) => {
  return text.split("").map((char) => toAscii(char)).join("");
};

Principle: If it’s only exported for tests but unused in production, remove it. YAGNI (You Aren’t Gonna Need It).

9. Inline Single-Use Helpers

Don’t create helpers for operations done once or twice.

Before (separate helper):

const renderOutput = (outputId, content) => {
  const output = document.getElementById(outputId);
  output.replaceChildren();
  output.insertAdjacentHTML("beforeend", `<div class="m-0" style="white-space: pre-wrap; word-break: break-word;">${content}</div>`);
};

const updateMarkdownOutput = () => {
  hideError();
  renderOutput("unicode-output", convertMarkdownToUnicode(document.getElementById("markdown-input").value));
};

After (kept helper but simplified):

const renderOutput = (id, content) => {
  const el = document.getElementById(id);
  el.replaceChildren();
  el.insertAdjacentHTML("beforeend", `<div class="m-0" style="white-space: pre-wrap; word-break: break-word;">${content}</div>`);
};

Principle: Kept the helper (reused 2x), but shortened variable names (outputId→id, output→el).

10. Single-Line Simple Functions

When a function is trivial, make it one line.

Before:

const convertBulletsToLists = (markdown) => {
  return markdown.replace(/^• /gm, "- ");
};

After:

const convertBulletsToLists = (markdown) => markdown.replace(/^• /gm, "- ");

Principle: Trivial transforms don’t need braces or explicit returns.

11. Early Returns Over Nested Conditions

Fail fast to reduce indentation.

Example (already applied):

const copyToClipboard = (text, button) => {
  if (!text.trim()) return setError("Nothing to copy");
  // ... rest of logic
};

Principle: Guard clauses at the top reduce nesting and cognitive load.

12. Spread Operator for String Iteration

Use [...str] instead of str.split("") for Unicode safety.

Example:

// Handles surrogate pairs correctly
[...text].map((c) => detectStyle(c) === "mono" ? toAscii(c) : c)

Principle: [...] respects code points; split("") splits surrogate pairs incorrectly.

13. Concise Module Exports

Before:

module.exports = {
  convertMarkdownToUnicode,
  convertUnicodeToMarkdown,
  toBold,
  toItalic,
  toMonospace,
  toAscii,
  detectStyle,
  fromUnicodeStyle,
};

After:

module.exports = { convertMarkdownToUnicode, convertUnicodeToMarkdown, toBold, toItalic, toMonospace, toAscii, detectStyle };

Principle: One line when it fits. Remove unused exports.

Summary: Core Principles

Data over code. Use structures, not conditionals
DRY. Never repeat the same logic
Modern JS features. Use ?., ??, destructuring, spread
YAGNI. Remove unused code immediately
Single expression. Eliminate intermediate variables when clear
Implicit returns. Arrow functions with => not => { return }
Early returns. Guard clauses reduce nesting
Shorter names. When scope is small (output → el)
One concern. Each function does one thing well
Readability first. Terse is good; cryptic is bad

Result: 144 lines shorter, more maintainable, same functionality! 🎯

Things I Learned - 06 Jul 2025

Sun, 06 Jul 2025 00:00:00 +0000

This week, I learned:

When adding a coding benchmark for LLMs, here’s a question I’d like to add. #benchmark

How do I use Apache Arrow in the browser via cdn.jsdelivr.net to create a .parquet file and download it? Give me minimal working code I can paste in the browser console to test.
LinkedIn has an undocumented link that shows schedules posts at https://www.linkedin.com/share/management/ which redirects to https://www.linkedin.com/feed/?shareActive=true&view=management

Here’s a JS snippet you can paste in the DevTools console of an npm package version page (example) to get a Markdown list showing the versions and dates

copy(
  $$('table[aria-labelledby="version-history"] tbody tr')
    .map((tr) => {
      const a = tr.querySelector("a");
      const date = new Date(tr.querySelector("time").getAttribute("datetime")).toLocaleDateString("en-GB", {
        day: "numeric",
        month: "short",
        year: "numeric",
      });
      return `- [${a.textContent.trim()}](https://npmjs.com${a.getAttribute("href")}): ${date}.`;
    })
    .join("\n"),
);

DuckDB can read JSON APIs! Ref
⭐ When bringing in humans-in-the-loop, applications must make it easier to review and to edit the work.

Breaking mental coding barriers with LLMs

Mon, 02 Sep 2024 05:59:02 +0000

Today, I stepped a bit beyond my comfort zone. Usually, I prefer micro-managing LLMs when writing code. This time, I was macro-managing.

I needed to create a mock history of the status of a manuscript, e.g. it was submitted on this date. THEN it moved to this state on this date. THEN … etc. I have no idea what the states could be, though. So, I could send it to an LLM, and it would give a different set of states each time. Or I could write a program and lose out on variety.

Instead, I asked ChatGPT.

Me: When looking up the status of a manuscript in a database, what are typical fields it might return?

ChatGPT: When looking up the status of a manuscript in a database, typical fields it might return include:

Manuscript ID: A unique identifier for the manuscript.

Title: The title of the manuscript.

Author(s): Names of the author(s) who submitted the manuscript.

Submission Date: The date the manuscript was submitted.

Status: The current status of the manuscript (e.g., Under Review, Accepted, Rejected, In Revision).

…

That helps. Then I said:

Me: Based on this, at any point, what “Status” values could a manuscript be in? Draw a mermaid flow graph for this.

ChatGPT: Here’s a mermaid flow graph depicting possible “Status” values a manuscript could be in throughout its lifecycle:

This is cool – it gives me a starting point that’s better than I could have conceived of.

Finally, I said:

Me: Write JS code that will randomly pick from one of these states and create records containing random dates and states representing the history of the manuscript up to that point

ChatGPT: Here’s a JavaScript code snippet that randomly picks from the manuscript states and creates records containing random dates and states, representing the history of the manuscript up to that point:

function getRandomDate(start, end) {
  return new Date(
    start.getTime() + Math.random() * (end.getTime() - start.getTime()),
  );
}

function getRandomElement(arr) {
  return arr[Math.floor(Math.random() * arr.length)];
}

function generateManuscriptHistory() {
  const history = [];
  let currentState = "Submitted";
  let currentDate = getRandomDate(new Date(2020, 0, 1), new Date());

  history.push({ date: currentDate, state: currentState });

  while (
    currentState !== "Accepted" && currentState !== "Rejected"
    && currentState !== "Published"
  ) {
    let possibleNextStates;

    switch (currentState) {
      case "Submitted":
        possibleNextStates = ["Under Review"];
        break;
      case "Under Review":
        possibleNextStates = ["Review Completed"];
        break;
      case "Review Completed":
        possibleNextStates = [
          "Accepted",
          "Rejected",
          "Major Revision Required",
          "Minor Revision Required",
        ];
        break;
      case "Major Revision Required":
      case "Minor Revision Required":
        possibleNextStates = ["In Revision"];
        break;
      case "In Revision":
        possibleNextStates = ["Resubmitted"];
        break;
      case "Resubmitted":
        possibleNextStates = ["Under Review"];
        break;
      case "Accepted":
        possibleNextStates = ["In Production"];
        break;
      case "In Production":
        possibleNextStates = ["Published"];
        break;
      default:
        possibleNextStates = [];
        break;
    }

    if (possibleNextStates.length === 0) break;

    currentState = getRandomElement(possibleNextStates);
    currentDate = getRandomDate(currentDate, new Date());

    history.push({ date: currentDate, state: currentState });
  }

  return history;
}

// Example usage:
console.log(generateManuscriptHistory());

The above code implements a state-transition diagram – something I’ve never been good at doing. I have a subconscious bias, and I steer clear of even thinking about these. Now, I have one implemented (and it worked quite well!)

Yet another example that human imagination is the bottleneck to leveraging LLMs.

Cyborg scraping

Fri, 25 Jun 2021 04:03:23 +0000

LinkedIn has a page that shows the people who most recently followed you.

At first, it shows just 20 people. But as you scroll, it keeps fetching the rest. I’d love to get the full list on a spreadsheet. I’m curious about:

What kind of people follow me?
Which of them has the most followers?
Who are my earliest followers?

But first, I need to scrape this list. Normally, I’d spend a day writing a program. But I tried a different approach yesterday.

Aside: it’s easy to get bored in online meetings. I have a surplus of partially distracted time. So rather than writing code to save me time, I’d rather create simple tasks to keep me occupied. Like scrolling.

So here’s my workflow to scrape the list of followers.

Step 1: Keep scrolling all the way to the bottom until you get all followers.

Step 2: Press F12, open the Developer Tools - Console, and paste this code.

copy(
  $$(".follows-recommendation-card").map(v => {
    let name = v.querySelector(".follows-recommendation-card__name");
    let headline = v.querySelector(".follows-recommendation-card__headline");
    let subtext = v.querySelector(".follows-recommendation-card__subtext");
    let link = v.querySelector(".follows-recommendation-card__avatar-link");
    let followers = "", match;
    if (subtext) {
      if (match = subtext.innerText.match(/([\d\.K]+) follower/)) {
        followers = match[1];
      } else if (match = subtext.innerText.match(/([\d\.K]+) other/)) {
        followers = match[1];
      }
    }
    followers = followers.match(/K$/)
      ? parseFloat(followers) * 1000
      : parseFloat(followers);
    return {
      name: name ? name.innerText : "",
      headline: headline ? headline.innerText : "",
      followers: followers,
      link: link ? link.href : "",
    };
  }),
);

Step 3: The name, headline, followers and link are now in the clipboard as JSON. Visit https://www.convertcsv.com/json-to-csv.htm and paste it in “Select your input” under “Enter Data”.

Step 4: Click on the “Download Result” button. The JSON is converted into a CSV you can load into a spreadsheet.

I call this “Cyborg scraping”. I do half the work (scrolling, copy-pasting, etc.) The code does half the work. It’s manual. It’s a bit slow. But it gets the job done quick and dirty.

I’ll share later what I learned about my followers. For now, I’m looking forward to meetings 😉

PS: A similar script to scrape LinkedIn invitations is below. You can only see 100 invitations per page, though.

copy(
  $$(".invitation-card").map(v => ({
    name: (v.querySelector(".invitation-card__title") || {}).innerText || "",
    link: v.querySelector(".invitation-card__link").href,
    subtitle: (v.querySelector(".invitation-card__subtitle") || {}).innerText
      || "",
    common: (v.querySelector(".member-insights__count") || {}).innerText || "",
    message:
      (v.querySelector(".invitation-card__custom-message") || {}).innerText
      || "",
  })),
);

PS: A similar script to scrape LinkedIn people search results is below.

copy(
  $$(".entity-result").map(v => {
    const name = v.querySelector(
      ".entity-result__title-text [aria-hidden=\"true\"]",
    );
    const link = v.querySelector("a");
    const badge = v.querySelector(
      ".entity-result__badge [aria-hidden=\"true\"]",
    );
    const title = v.querySelector(".entity-result__primary-subtitle");
    const subtitle = v.querySelector(".entity-result__secondary-subtitle");
    const summary = v.querySelector(".entity-result__summary--2-lines");
    const insight = v.querySelector(".entity-result__simple-insight-text");
    return {
      name: name?.innerText || "",
      link: (link?.href || "").split("?")[0],
      badge: badge?.innerText || "",
      title: title?.innerText || "",
      subtitle: subtitle?.innerText || "",
      summary: summary?.innerText || "",
      insight: insight?.innerText || "",
    };
  }),
);

Comments

mvark 8 Aug 2021 10:39 pm: I think rather than create JSON, building a comma separated string and console.log-ging that string will show the whole list in the Dev Tools Console. Using the Save As option in the context menu, the list can be saved as a .CSV file directly without Step 3 & 4.
Tools to publish annotated talks from videos - S Anand 20 Oct 2024 12:53 pm (pingback): […] together. As you scroll, newer/older comments are loaded. So I needed to use my favorite technique: Cyborg Scraping. During Q&A, I kept scrolling to the bottom and […]

Programming Minecraft with Websockets

Wed, 20 Jan 2021 03:40:09 +0000

Minecraft lets you connect to a websocket server when you’re in a game. The server can receive and send any commands. This lets you build a bot that you can … (well, I don’t know what it can do, let’s explore.)

Minecraft has commands you can type on a chat window. For example, type / to start a command and type setblock ~1 ~0 ~0 grass changes the block 1 north of you into grass. (~ means relative to you. Coordinates are specified as X, Y and Z.)

Note: These instructions were tested on Minecraft Bedrock 1.16. I haven’t tested them on the Java Edition.

Connect to Minecraft

You can send any command to Minecraft from a websocket server. Let’s use JavaScript for this.

First, run npm install ws uuid. (We need ws for websockets and uuid to generate unique IDs.)

Then create this mineserver1.js:

const WebSocket = require("ws");
const uuid = require("uuid"); // For later use

// Create a new websocket server on port 3000
console.log("Ready. On MineCraft chat, type /connect localhost:3000");
const wss = new WebSocket.Server({ port: 3000 });

// On Minecraft, when you type "/connect localhost:3000" it creates a connection
wss.on("connection", socket => {
  console.log("Connected");
});

On Minecraft > Settings > General > Profile, turn off the “Require Encrypted Websockets” setting.

Run node mineserver1.js. Then type /connect localhost:3000 in a Minecraft chat window. You’ll see 2 things:

MineCraft says “Connection established to server: ws://localhost:3000”
Node prints “Connected”

Now, our program is connected to Minecraft, and can send/receive messages.

Notes:

The Python equivalent is in mineserver1.py. Run python mineserver1.py.
If you get an Uncaught Error: Cannot find module 'ws', make sure you ran npm install ws uuid.
If you get an “Encrypted Session Required” error, make sure you turned off the “Require Encrypted Websockets” setting mentioned above.
To disconnect, run /connect off

Now let’s listen to the players’ chat.

A connected websocket server can send a “subscribe” message to Minecraft saying it wants to “listen” to specific actions. For example, you can subscribe to “PlayerMessage”. Whenever a player sents a chat message, Minecraft will notify the websocket client.

Here’s how to do that. Add this code in the wss.on('connection', socket => { ... }) function.

// Tell Minecraft to send all chat messages. Required once after Minecraft starts
socket.send(JSON.stringify({
  "header": {
    "version": 1, // We're using the version 1 message protocol
    "requestId": uuid.v4(), // A unique ID for the request
    "messageType": "commandRequest", // This is a request ...
    "messagePurpose": "subscribe", // ... to subscribe to ...
  },
  "body": {
    "eventName": "PlayerMessage", // ... all player messages.
  },
}));

Now, every time a player types something in the chat window, the socket will receive it. Add this code below the above code:

// When MineCraft sends a message (e.g. on player chat), print it.
socket.on("message", packet => {
  const msg = JSON.parse(packet);
  console.log(msg);
});

This code parses all the messages it receives and prints them.

This code in is mineserver2.js. Run node mineserver2.js. Then type /connect localhost:3000 in a Minecraft chat window. Then type a message (e.g. “alpha”) in the chat window. You’ll see a message like this in the console.

{
  header: {
    messagePurpose: 'event',        // This is an event
    requestId: '00000000-0000-0000-0000-000000000000',
    version: 1                      // using version 1 message protocol
  },
  body: {
    eventName: 'PlayerMessage',
    measurements: null,
    properties: {
      AccountType: 1,
      ActiveSessionID: 'e0afde71-9a15-401b-ba38-82c64a94048d',
      AppSessionID: 'b2f5dddc-2a2d-4ec1-bf7b-578038967f9a',
      Biome: 1,                     // Plains Biome. https://minecraft.gamepedia.com/Biome
      Build: '1.16.201',            // That's my build
      BuildNum: '5131175',
      BuildPlat: 7,
      Cheevos: false,
      ClientId: 'fcaa9859-0921-348e-bc7c-1c91b72ccec1',
      CurrentNumDevices: 1,
      DeviceSessionId: 'b2f5dddc-2a2d-4ec1-bf7b-578038967f9a',
      Difficulty: 'NORMAL',         // I'm playing on normal difficulty
      Dim: 0,
      GlobalMultiplayerCorrelationId: '91967b8c-01c6-4708-8a31-f111ddaa8174',
      Message: 'alpha',             // This is the message I typed
      MessageType: 'chat',          // It's of type chat
      Mode: 1,
      NetworkType: 0,
      Plat: 'Win 10.0.19041.1',
      PlayerGameMode: 1,            // Creative. https://minecraft.gamepedia.com/Commands/gamemode
      Sender: 'Anand',              // That's me.
      Seq: 497,
      WorldFeature: 0,
      WorldSessionId: '8c9b4d3b-7118-4324-ba32-c357c709d682',
      editionType: 'win10',
      isTrial: 0,
      locale: 'en_IN',
      vrMode: false
    }
  }
}

Notes:

The Python equivalent is in mineserver2.py. Run python mineserver2.py.
The official Minecraft docs say that the MCWSS protocol is outdated. But it seems to work.
The full list of things we can subscribe to is undocumented, but @jocopa3 has reverse-engineered a list of messages we can subscribe to, and they’re somewhat meaningful.
This Go package has code that explores the protocol further.
This chat has more details. There’s also an outdated list of JSON messages from @jocopa3.
Here’s a sample program that places a block in Minecraft

Build structures using chat

Let’s create a pyramid of size 10 around us when we type pyramid 10 in the chat window.

The first step is to check if the player sent a chat message like pyramid 10 (or another number). Add this code below the above code:

// When MineCraft sends a message (e.g. on player chat), act on it.
socket.on("message", packet => {
  const msg = JSON.parse(packet);
  // If this is a chat window
  if (msg.body.eventName === "PlayerMessage") {
    // ... and it's like "pyramid 10" (or some number), draw a pyramid
    const match = msg.body.properties.Message.match(/^pyramid (\d+)/i);
    if (match) {
      draw_pyramid(+match[1]);
    }
  }
});

If the user types “pyramid 3” on the chat window, draw_pyramid(3) is called.

In draw_pyramid(), let’s send commands to build a pyramid. To send a command, we need to create a JSON with the command (e.g. setblock ~1 ~0 ~0 grass). Add this code below the above code:

function send(cmd) {
  const msg = {
    "header": {
      "version": 1,
      "requestId": uuid.v4(), // Send unique ID each time
      "messagePurpose": "commandRequest",
      "messageType": "commandRequest",
    },
    "body": {
      "version": 1, // TODO: Needed?
      "commandLine": cmd, // Define the command
      "origin": {
        "type": "player", // Message comes from player
      },
    },
  };
  socket.send(JSON.stringify(msg)); // Send the JSON string
}

Let’s write draw_pyramid() to create a pyramid using glowstone by adding this code below the above code:

// Draw a pyramid of size "size" around the player.
function draw_pyramid(size) {
  // y is the height of the pyramid. Start with y=0, and keep building up
  for (let y = 0; y < size + 1; y++) {
    // At the specified y, place blocks in a rectangle of size "side"
    let side = size - y;
    for (let x = -side; x < side + 1; x++) {
      send(`setblock ~${x} ~${y} ~${-side} glowstone`);
      send(`setblock ~${x} ~${y} ~${+side} glowstone`);
      send(`setblock ~${-side} ~${y} ~${x} glowstone`);
      send(`setblock ~${+side} ~${y} ~${x} glowstone`);
    }
  }
}

This code in is mineserver3.js.

Run node mineserver3.js.
Then type /connect localhost:3000 in a Minecraft chat window.
Then type pyramid 3 in the chat window.
You’ll be surrounded by a glowstone pyramid.

Notes:

The Python equivalent is in mineserver3.py. Run python mineserver3.py.
The “requestId” needs to be a UUID – at least for block commands. I tried unique “requestId” values like 1, 2, 3 etc. That didn’t work.

Understand Minecraft’s responses

For every command you send, Minecraft sends a response. It’s “header” looks like this:

{
  "header": {
    "version": 1,
    "messagePurpose": "commandResponse",                  // Response to your command
    "requestId": "97dee9a3-a716-4caa-aef9-ddbd642f2650"   // ... and your requestId
  }
}

If the command is successful, the response has body.statusCode == 0. For example:

{
  "body": {
    "statusCode": 0,                  // No error
    "statusMessage": "Block placed",  // It placed the block you wanted
    "position": { "x": 0, "y": 64, "z": 0 }   // ... at this location
  },
}

If the command failed, the response has a negative body.statusCode. For example:

{
  "body": {
    "statusCode": -2147352576,        // This is an error
    "statusMessage": "The block couldn't be placed"
  },
}

To print these, add this to socket.on('message', ...):

// If we get a command response, print it
if (msg.header.messagePurpose == "commandResponse") {
  console.log(msg);
}

This code in is mineserver4.js.

Run node mineserver4.js.
Then type /connect localhost:3000 in a Minecraft chat window.
Then type pyramid 3 in the chat window.
You’ll be surrounded by a glowstone pyramid, and the console will show every command response.

Notes on common error messages:

The block couldn't be placed (-2147352576): The same block was already at that location.
Syntax error: Unexpected "xxx": at "~0 ~9 ~-1 >>xxx<<" (-2147483648): You gave wrong arguments to the command.
Too many commands have been requested, wait for one to be done (-2147418109): Minecraft only allows 100 commands can be executed without waiting for their response.
More error messages here.

Wait for commands to be done

Typing “pyramid 3” works just fine. But try “pyramid 5” and your pyramid is incomplete.

That’s because Minecraft only allows up to 100 messages in its queue. On the 101st message, you get a Too many commands have been requested, wait for one to be done error.

{
  "header": {
    "version": 1,
    "messagePurpose": "error",
    "requestId": "a5051664-e9f4-4f9f-96b8-a56b5783117b"
  },
  "body": {
    "statusCode": -2147418109,
    "statusMessage": "Too many commands have been requested, wait for one to be done"
  }
}

So let’s modify send() to add to a queue and send in batches. We’ll create two queues:

const sendQueue = []; // Queue of commands to be sent
const awaitedQueue = {}; // Queue of responses awaited from Minecraft

In wss.on('connection', ...), when Minecraft completes a command, we’ll remove it from the awaitedQueue. If the command has an error, we’ll report it.

// If we get a command response
if (msg.header.messagePurpose == "commandResponse") {
  // ... and it's for an awaited command
  if (msg.header.requestId in awaitedQueue) {
    // Print errors 5(if any)
    if (msg.body.statusCode < 0) {
      console.log(
        awaitedQueue[msg.header.requestId].body.commandLine,
        msg.body.statusMessage,
      );
    }
    // ... and delete it from the awaited queue
    delete awaitedQueue[msg.header.requestId];
  }
}
// Now, we've cleared all completed commands from the awaitedQueue.

Once we’ve processed Minecraft’s response, we’ll send pending messages from sendQueue, upto 100 and add them to the awaitedQueue.

// We can send new commands from the sendQueue -- up to a maximum of 100.
let count = Math.min(100 - Object.keys(awaitedQueue).length, sendQueue.length);
for (let i = 0; i < count; i++) {
  // Each time, send the first command in sendQueue, and add it to the awaitedQueue
  let command = sendQueue.shift();
  socket.send(JSON.stringify(command));
  awaitedQueue[command.header.requestId] = command;
}
// Now we've sent as many commands as we can. Wait till the next PlayerMessage/commandResponse

Finally, in function send(), instead of socket.send(JSON.stringify(msg)), we use sendQueue.push(msg) to add the message to the queue.

This code in is mineserver5.js.

Run node mineserver5.js.
Then type /connect localhost:3000 in a Minecraft chat window.
Then type pyramid 6 in the chat window.
You’ll be surrounded by a large glowstone pyramid.
The console will print messages like setblock ~0 ~6 ~0 glowstone The block couldn't be placed because we’re trying to place duplicate blocks.

Comments

Rodland 8 Mar 2021 12:45 am:Thanks for a great tutorial.However when I try to implement the chat listener (server2.js) i get an JSON error from MineCraft (in node.js console):“kryptert økt kreves” = “Needs secure connection”
Ready. On MineCraft chat, type /connect localhost:3000 Connected { body: { statusCode: -2147418107, statusMessage: ‘Kryptert økt kreves’ }, header: { messagePurpose: ’error’, requestId: ‘3522bb53-5256-4030-b827-4b411ae72a6c’, version: 1 } }
S Anand 8 Mar 2021 4:07 pm: Under Settings > General > Profile, please turn off the “Require Encrypted Websockets” setting. I’ve added this to the tutorial. Thanks for flagging this!
S Anand 2 Jan 2022 9:48 am: That’s right. Looks like you resolved it and changed localhost to minecraft-server.warpedwartwars.repl.co 👍
WarpedWartWars 21 Jun 2022 6:48 am: Got it working on localhost, using the Python mineserver3.py, but the message protocol seems to have changed, to this: {‘body’: {‘message’: ‘pyramid 32’, ‘receiver’: ‘’, ‘sender’: ‘WarpedWartWars’, ’type’: ‘chat’}, ‘header’: {’eventName’: ‘PlayerMessage’, ‘messagePurpose’: ’event’, ‘version’: 16842752}}
WarpedWartWars 15 Dec 2021 8:27 am: I tried this, as it seemed to be the best guide I could find. I made a Replit Repl: https://replit.com/@WarpedWartWars/minecraft-server?v=1 and I ran it, then tried “/connect localhost:3000” in Minecraft Bedrock 1.18.2, and I got “Could not connect to server: ws://localhost:3000”. I imagine I have to change “localhost” to something else.
someGuy 23 Feb 2022 8:11 am: This will be deleted in couple of days but some of the reasons people are getting issues connecting is because of UWP loopback restrictions source: https://stackoverflow.com/questions/34589522/cant-see-localhost-from-uwp-app to resolve this problem for windows: https://docs.microsoft.com/en-us/minecraft/creator/documents/scriptdevelopertools. run this in system administrator command prompt: CheckNetIsolation.exe LoopbackExempt -a -p=S-1-15-2-1958404141-86561845-1752920682-3514627264-368642714-62675701-733520436

Auto reloading pages

Sun, 30 Sep 2012 10:33:15 +0000

After watching Bret Victor’s Inventing on Principle, I just had to figure out a way of getting live reloading to work. I know about LiveReload, of course, and everything I’ve heard about it is good. But their Windows version is in alpha, and I’m not about to experiment just yet.

This little script does it for me instead:

(function(interval, location) {
  var lastdate = "";
  function updateIfChanged() {
    var req = new XMLHttpRequest();
    req.open("HEAD", location.href, false);
    req.send(null);
    var date = req.getResponseHeader("Last-Modified");
    if (!lastdate) {
      lastdate = date;
    } else if (lastdate != date) {
      location.reload();
    }
  }
  setInterval(updateIfChanged, interval);
})(300, window.location);

It checks the current page every 300 milliseconds and reloads it if the Last-Modified header is changed. I usually include it as a minified script:

<script>(function(d,c){var b="";setInterval(function(){var a=new
XMLHttpRequest;a.open("HEAD",c.href,false);a.send(null);
a=a.getResponseHeader("Last-Modified");if(b)b!=a&&
c.reload();else b=a},d)})(300,window.location)</script>

There are no dependencies on any library, like jQuery. However, it requires that the file be on a web server. (It’s easy to fix that, but since I always run a local webserver, I’ll let you solve that problem yourself.)

Codecasting

Wed, 16 Nov 2011 20:05:52 +0000

The best way to explain code to a group of people is by walking through it. If they’re far away in space or time, then a video is the next best thing. You can recommend them to try out the best coding apps as well. The trouble with videos, though, is that they’re big. I can’t host them on my server – I’d need YouTube. Editing them is tough. You can’t copy & paste code from videos. And so on. One interesting alternative is to use presentations with audio. Slideshare, for instance, lets you share slides and sync it with audio. That almost works. But it’s still not good enough. I’d like code to be stored as code. What I really need is codecasting: a YouTube or Slideshare for code. The closest I’ve seen until day-before was etherpad or ttyrec – but neither support audio. Enter Popcorn. It’s a Javascript library from Mozilla that, among other things, can fire events when an audio/video element reaches a particular point.

Watch a demo of how I used it for codecasting

A look at the code will show you that I’m using two libraries: SyntaxHighlighter to highlight the code, and Popcorn. The meat of the code I’ve written is in this subtitle function.

function subtitle(media_node, pre_node, events) {
  var pop = Popcorn(media_node);
  for (var i=0, l=events.length; i<l; i++)="" {="" for="" (var="" j="0," line_selector="[]," line_no;="" line_no="events[i][1][j];" j++)="" line_selector.push(pre_node="" +="" '="" .number'="" line_no)="" }="" var="" start="events[i][0]" ,="" end="i<l-1" ?="" events[i+1][0]="" :="" events[i][0]+999;="" (function(start,="" end,="" selector)="" pop.code({start:="" start,="" end:end,="" onstart:="" function(o)="" $(selector).addclass('highlighted');="" },="" onend:="" $(selector).removeclass('highlighted');="" })="" })(start,="" line_selector.join(','));="" }<="" pre="">

When called like this:

subtitle("#audio", "pre", [
  [1, [1, 2, 3]],
  [5, [4, 5, 6]],
  [9, [7, 8]],
]);

… it takes the #audio element, when it plays to 1 second, highlights lines 1,2,3; at 5 seconds, highlights lines 4,5,6; and so on. Another thing that helped was that my iPad has a much better mic than my laptop, and ClearRecord is a really simple way to create recordings with minimal noise. [Note to self: sampling at 16KHz and saving as a VBR MP3 (45-85kbps) seems the best trade-off.] With these tools, my time to prepare a tutorial went down from 4 hours to half an hour!

</l;>

Comments

S Anand 17 Nov 2011 7:58 am: Want to try building one? :-)
Tanu 17 Nov 2011 2:52 pm: Good one.. Want to experiment one..
vivek 20 Nov 2011 2:46 pm: Excellent! Will start using this right away! Thanks Anand.
Veera 17 Nov 2011 6:30 am: Great idea for an web app/platform!

Javascript arrays vs objects

Thu, 06 Oct 2011 13:43:22 +0000

Summary: Arrays are a lot smaller than objects, but only slightly faster on newer browsers.

I’m writing an in-memory Javascript app that handles several thousand rows. Each row could be stored either as an array [1,2,3] or an object {"x":1,"y":2,"z":3}. Having read up on the performance of arrays vs objects, I thought I’d do a few tests on storing numbers from 0 to 1 million. The results for Chrome are below. (Firefox 7 was similar.)

	Time	Size (MB)
Array: `x[i] = i`	2.44s	8
Object: `x[i] = i`	3.02s	57
Object: `x["a_long_dummy_testing_string"+i]=i`	4.21s	238

The key lessons for me were:

Browsers used to process arrays MUCH faster than objects. This gap has now shrunk.
However, arrays are still better: not for their speed, but for their space efficiency.
If you’re processing a million rows or less, don’t worry about memory. If you’re storing stuff as arrays, you can store 128 columns in 1GB of RAM (1024/8=128).

Comments

dineshsaravanank 11 Oct 2011 5:27 am: Also, I got completely different numbers than yours. Array x[i]=i, 43ms | Object x[i]=i, 43ms | Object x[“akey”+i]=i, 2443ms. I tested in chrome, as you can see the keyed object is *significantly* slower in my case…
S Anand 11 Oct 2011 10:38 pm: @dinesh: When object keys are integers, browsers tend to optimise them in an array-like fashion. That explains the {1:1, 2:2} being faster than {“a1”:1,“a2”:2}. @kalpesh, the main point of this,actually, was that performance is increasingly less of a worry for most applications. Browsers are fast enough. So the choice of language is becoming increasingly less performance driven and more ease driven.
dineshsaravanank 11 Oct 2011 5:18 am: After reading this I was curious about 1) How is the read performance, 2) How does the object with long key perform. So I ran some tests. And,
1. as expected reads are much faster than the writes and there is no significant difference between array and object when it came to read.
2. Longer keys take more time to execute, smaller keys are significantly faster than very long keys. But there are some surprises here. An object {“1”:1,“2”:2,“3”:3} is much faster than {“a1”:1, “a2”:2, “a3”:4}, while object {“a1”:1,“a2”:2} and object {“alittlelong1”:1, “alittlelong2”:2} are almost same. You can see the tests here http://jsfiddle.net/ETAb9/
Kalpesh 12 Oct 2011 4:56 pm: I agree. My question was to think of it, in terms of memory consumption. Your thoughts?
Kalpesh 10 Oct 2011 10:50 am: Considering this, do you think OO based languages add overhead to space it occupies when run, in memory?
Arun 13 Oct 2011 6:09 pm: Anand, This is because object is a fundamental data type in Javascript and an Array is implemented as a special kind of object. Hence the lookups (the Time column) would be implemented as a hash map lookup in both cases. However the storage could be an interesting case. As one of the early gotchas of Javascript teaches us: Person.Name is equivalent to Person['Name'] But in the case of Array’s some magic is applied leading to optimisations in storage. Hence I suppose rather than storing unicode strings for “0”, “1” etc., they are stored as integers. Javascript, like English, is berry phunny language :)
Joe 29 Feb 2016 1:02 pm: I ran some tests and was surprised that array access is only slightly faster. Something strange has to be going on with that. My test compares numeric entries either way. The object implementation normally has the extra work of casting to string and hashing. I have a feeling it will be different with non-numeric versus numeric keys. Still, it hints at something strange going on.

Error logging with Google Analytics

Sat, 23 May 2009 19:29:01 +0000

A quick note: I blogged earlier about Javascript error logging, saying that you can wrap every function in your code (automatically) in a try{} catch{} block, and log the error message in the catch{} block.

I used to write the error message to a Perl script. But now I use Google’s event tracking.

var s = [];
for (var i in err) s.push(i + "=" + err[i]);
s = s.join(" ").substr(0, 500);
pageTracker._trackEvent("Error", function_name, s);

The good part is that it makes error monitoring a whole lot easier. Within a day of implementing this, I managed to get a couple of errors fixed that had been pending for months.

Comments

Thejesh GN 29 May 2009 7:40 am: That is a good tip. I can use the present infrastructure for error logging..

Random quotes generator

Mon, 23 Mar 2009 15:22:17 +0000

The Random Quotes Generator is a simple tool that creates quotes by mixing up words on a web page. The results are often funny, but sometimes surprisingly insightful.

Yes, this is the equivalent of a million monkeys typing Shakespeare, except that they’re using the works of Shakespeare as a starting point. And it doesn’t have to be Shakespeare. It could be you or your friends.

To try it out, visit this page, select the link and “Add to Favorites” or drag it into your browser’s bookmark toolbar.. Then go to any web page that has a lot of text, and click the link to generate random quotes.

Here’s an example of random text from TechCrunch.

The net will find monetization models of theater and sporting events before them. Indeed, there has to be some way to create websites that do other than Advertising. The expected drop in internet advertising will rapidly lose its value and its impact, for reasons that can easily be understood.

For the technically minded, Programming Pearls has a section on Generating Text that explains the concept. The bookmarklet uses an Order-2 word-level Markov chain. Translated into English, what that means is: I look at every pair of words in and find out what word is likely to follow that.

For example, in the Generating Text page, the pair of words “we can” are followed by the words “extend”, “also”, “get” and “write” with equal probability. We pick one randomly (say “also”) and write “we can also”. Then we look at the word pair “can also”, see what word follows that, pick one at random, and so on.

This is Order-2 because we pick pairs of words. And it’s word-level rather than letter-level because we use words instead of letters as the basic building blocks.

When you’re trying it out, make sure that the page is large enough. If not, you may find that the page’s content is reproduced verbatim.

The bookmarklet is built on top of the excellent Readability bookmarklet by Arc90, which helps identify the main content to be randomized.

Client side scraping for contacts

Fri, 06 Mar 2009 17:01:56 +0000

By curious coincidence, just a day after my post on client side scraping, I had a chance to demo this to a client. They were making a contacts database. Now, there are two big problems with managing contacts.

Getting complete information
Keeping it up to date

Now, people happy to fill out information about themselves in great detail. If you look at the public profiles on LinkedIn, you’ll find enough and more details about most people.

Normally, when getting contact details about someone, I search for their name on Google with a “site:linkedin.com” and look at that information.

Could this be automated?

I spent a couple of hours and came up with a primitive contacts scraper. Click on the link, type in a name, and you should get the LinkedIn profile for that person. (Caveat: It’s very primitive. It works only for specific URL public profiles. Try ‘Peter Peverelli’ as an example.)

It uses two technologies. Google AJAX Search API and YQL. The search() function searches for a phrase…

google.load("search", "1");
google.setOnLoadCallback(function() {
  gs = new google.search.WebSearch();
  $("#getinfo").show();
});

function search(phrase, fn) {
  gs.setSearchCompleteCallback(gs, function() {
    fn(this.results);
  });
  gs.execute(phrase);
}

… and the linkedin() function takes a LinkedIn URL and extracts the relevant information from it, using XPath.

function scrape(url, xpath, fn) {
  $.getJSON("http://query.yahooapis.com/v1/public/yql?callback=?", {
    q: "select * from html where(url=\""
      + url + "\" and xpath=\"" + xpath + "\")",
    format: "json",
  }, fn);
}

function linkedin(url, fn) {
  scrape(url, "//li[@class][h3]", fn);
}

So if you wanted to find Peter Peverelli, it searches on Google for “Peter Peverelli site:linkedin.com” and picks the first result.

From this result, it displays all the <LI> tags which have a class and a <H3> element inside them (that’s what the //li[@class][h3] XPath does).

The real value of this is in bulk usage. When there’s a big list of contacts, you don’t need to scan each of them for updates. They can be automatically updated — even if all you know is the person’s name, and perhaps where they worked at some point in time.

Comments

Thejesh GN » RSS Feed Aamir Khans blog using YQL and Pipes 9 Mar 2009 2:44 pm (pingback): […] Blog to create a feed. I had written custom scraping code to create the feed.Today after reading Anand’s blog I parsed the blog using YQL and created the feed using Pipes. Using YQL/PIPE much easier thank […]
Prakash 8 Mar 2009 1:50 pm: Have you tried xobni add-in for outlook (incase you use that to manage contacts)?
S Anand 8 Mar 2009 4:07 pm: I don’t use it myself, but have tried it — and interestingly enough, recommended it to this client.
Thejesh GN 9 Mar 2009 2:46 pm: After reading this post. I redid Aamir Khans blog parsing using YQL and Pipes. It was very easy as YQL was available as module inside Pipes.
Vasanth Asokan 18 Dec 2010 6:05 am: Hi Anand, The scraper demo does not seem to be working (I tried the value you suggested as well ‘Peter Peverelli’. Can you take a look?
S Anand 18 Dec 2010 10:42 am: True, Vasanth. Honestly, scraping was only ever useful as a temporary strut until service providers opened up the data. It required too much maintenance anyway. With LinkedIn’s API, this is outdated, and I’m planning to leave it as it is. But thanks for pointing this out!

Bound methods in Javascript

Wed, 12 Nov 2008 12:00:00 +0000

The popular way to create a class in Javascript is to define a function and add methods to its prototype. For example, let’s create a class Node that has a method hide().

var Node = function(id) {
  this.element = document.getElementById(id);
};
Node.prototype.hide = function() {
  this.style.display = "none";
};

If you had a header, say

, then this piece of code will hide the element.

var node = new Node("header");
node.hide();

If I wanted to hide the element a second later, I am tempted to use:

var node = new Node("header");
setTimeout(node.hide, 1000);

… except that it won’t work. setTimeout has no idea that the function node.hide has anything to do with the object node. It just runs the function. When node.hide() is called by setTimeout, the this object isn’t set to node, it’s set to window. node.hide() ends up trying to hide window, not node.

The standard way around this is:

var node = new Node("header");
setTimeout(function() {
  node.hide();
}, 1000);

I’ve been using this for a while, but it gets tiring. It’s so easy to forget to do this. Worse, it doesn’t work very well inside a loop:

for (var id in ["a", "b", "c"]) {
  var node = new Node(id);
  setTimeout(function() {
    node.hide();
  }, 1000);
}

This actually hides node “c” thrice, and doesn’t touch nodes “a” and “b”. You’ve got to remember to wrap every function that contains a function in a loop.

for (var id in ["a", "b", "c"]) {
  (function() {
    var node = new Node(id);
    setTimeout(function() {
      node.hide();
    }, 1000);
  })();
}

Now, compare that with this:

for (var id in ["a", "b", "c"]) {
  setTimeout((new Node(id)).hide, 1000);
}

Wouldn’t something this compact be nice?

To do this, the method node.hide must be bound to the object node. That is, node.hide must know that it belongs to node. And when we call another_node.hide, it must know that it belongs to another_node. This, incidentally, is the way most other languages behave. For example, on python, try the following:

>>> class Node:
...     def hide():
...             pass
...
>>> node = Node()
>>> node
<__main__.Node instance at 0x00BA32D8>
>>> node.hide
<bound method Node.hide of <__main__.Node instance at 0x00BA32D8>>

The method hide is bound to the object node.

To do this in Javascript, instead of adding the methods to the prototype, you need to do two things:

Add the methods to the object in the constructor
Don’t use this. Set another variable called that to this, and use that instead.

var Node = function(id) {
  var that = this;
  that.element = document.getElementById(id);
  that.hide = function() {
    that.element.style.display = "none";
  };
};

Now node.hide is bound to node. The following code will work.

var node = new Node("header");
setTimeout(node.hide, 1000);

I’ve taken to using this pattern almost exclusively these days, rather than prototype-based methods. It saves a lot of trouble, and I find it makes the code a lot compacter and easier to read.

Comments

Hari K T 3 Jan 2009 6:52 am: Good article . .. . .
Great work . I am trying to implement it tooo….
Mani 2 Dec 2011 8:25 pm: A small correction though var Node = function(id) { var that = this; that.element = document.getElementById(id); that.hide = function() { that.element.style.display = “none”; } };
S Anand 2 Dec 2011 10:12 pm: Thanks Mani – I’ve updated the code to reflect your correction.

JPath - XPath for Javascript

Thu, 29 May 2008 12:00:00 +0000

XPath is a neat way of navigating deep XML structures. It's like using a directory structure. /table//td gets all the TDs somewhere below TABLE.

Usually, you don't need this sort of a thing for data structures, particularly in JavaScript. Something like table.td would already work. But sometimes, it does help to have something like XPath even for data structures, so I built a simple XPath-like processor for Javascript called JPath.

Here are some examples of how it would work:

jpath(context, "para")	returns context.para
jpath(context, "*")	returns all values of context (for both arrays and objects)
jpath(context, "para[0]")	returns context.para[0]
jpath(context, "para[last()]")	returns context.para[context.para.length]
jpath(context, "*/para")	returns context[all children].para
jpath(context, "/doc/chapter[5]/section[2]")	returns context.doc.chapter[5].section[2]
jpath(context, "chapter//para")	returns all para elements inside context.chapter
jpath(context, "//para")	returns all para elements inside context
jpath(context, "//olist/item")	returns all olist.item elements inside context
jpath(context, ".")	returns the context
jpath(context, ".//para")	same as //para
jpath(context, "//para/..")	returns the parent of all para elements inside context

Some caveats:

This is an implementation of the abbreviated syntax of XPath. You can't use axis::nodetest
No functions are supported other than last()
Only node name tests are allowed, no nodetype tests. So you can't do text() and node()
Indices are zero-based, not 1-based

There are a couple of reasons why this sort of thing is useful.

Extracting attributes deep down. Suppose you had an array of arrays, and you wanted the first element of each array.

You could do this the long way:
```
for (var list = [], i = 0; i < data.length; i++) {
  list.push(data[i][0]);
}
```
... or the short way:
```
$.map(data, function(v) {
  return v[1];
});
```
But the best would be something like:
```
jpath(data, "//1");
```

Ragged data structures. Take for example the results from Google's AJAX feed API.

{"responseData": {
 "feed": {
  "title": "Digg",
  "link": "http://digg.com/",
  "author": "",
  "description": "Digg",
  "type": "rss20",
  "entries": [
   {
    "title": "The Pirate Bay Moves Servers to Egypt Due to Copyright Laws",
    "link": "http://digg.com/tech_news/The_Pirate_Bay_Moves_Servers_to_Egypt_Due_to_Copyright_Laws",
    "author": "",
    "publishedDate": "Mon, 31 Mar 2008 23:13:33 -0700",
    "contentSnippet": "Due to the new copyright legislation that are going ...",
    "content": "Due to the new copyright legislation that are going to take...",
    "categories": [
    ]
   },
   {
    "title": "Millions Dead/Dying in Recent Mass-Rick-Rolling by YouTube.",
    "link": "http://digg.com/comedy/Millions_Dead_Dying_in_Recent_Mass_Rick_Rolling_by_YouTube",
    "author": "",
    "publishedDate": "Mon, 31 Mar 2008 22:53:30 -0700",
    "contentSnippet": "Click on any \u0022Featured Videos\u0022. When will the insanity stop?",
    "content": "Click on any \u0022Featured Videos\u0022. When will the insanity stop?",
    "categories": [
    ]
   },
   ...
  ]
 }
}
, "responseDetails": null, "responseStatus": 200}

If you wanted all the title entries, including the feed title, the choice is between:

var titles = [ result.feed.title ];
for (var i=0, l=result.feed.entries.length; i<l; i++) {
    titles.push(result.feed.entries[i].title;
}

... versus...

titles = jpath(result, "//title");

If, further, you wanted the list of all categories at one shot, you could use:

jpath(result, "//categories/*");

Comments

Will 7 Mar 2010 1:15 pm: See also: http://www.w3schools.com/xpath/tryit.asp?filename=try_xpath_select_pricenodes_text

Chaining functions in Javascript

Mon, 18 Feb 2008 12:00:00 +0000

One of the coolest features of jQuery is the ability to chain functions. The output of a function is the calling object. So instead of writing:

var a = $("<div></div>");
a.appendTo($("#id"));
a.hide();

… I can instead write:

$("<div></div>").appendTo($("#id")).hide();

A reasonable number of predefined Javascript functions can be used this way. I make extensive use of it with the String.replace function.

But where this feature is not available, you an create it in a fairly unobstrusive way. Just add this code to your script:

Function.prototype.chain = function() {
  var that = this;
  return function() {
    // New function runs the old function
    var retVal = that.apply(this, arguments);
    // Returns "this" if old function returned nothing
    if (typeof retVal == "undefined") return this;
    // else returns old value
    else return retVal;
  };
};
var chain = function(obj) {
  for (var fn in obj) {
    if (typeof obj[fn] == "function") {
      obj[fn] = obj[fn].chain();
    }
  }
  return obj;
};

Now, chain(object) returns the same object, with all its functions replaced with chainable versions.

What’s the use? Well, take the Google AJAX search API. Normally, to search for the top 8 “Harry Potter” PDFs on esnips.com, I’d have to do:

var searcher = new google.search.WebSearch();
searcher.setQueryAddition("filetype:PDF");
searcher.setResultSetSize(google.search.Search.LARGE_RESULTSET);
searcher.setSiteRestriction("esnips.com");
searcher.setSearchCompleteCallback(onSearch);
searcher.execute("Harry Potter");

Instead, I can now do this:

chain(new google.search.WebSearch())
  .setQueryAddition("filetype:PDF")
  .setResultSetSize(google.search.Search.LARGE_RESULTSET)
  .setSiteRestriction("esnips.com")
  .setSearchCompleteCallback(onSearch)
  .execute("Harry Potter");

(On the whole, it’s probably not worth the effort. Somehow, I just like code that looks like this.)

Comments

T 18 Feb 2008 12:00 pm: looks sweet!

Javascript error logging

Tue, 25 Dec 2007 12:00:00 +0000

If something goes wrong with my site, I like to know of it. My top three problems are:

The site is down

A page is missing

Javascript isn’t working

This is the last of 3 articles on these topics.

I am a bad programmer I am not a professional developer. In fact, I’m not a developer at all. I’m a management consultant. (Usually, it’s myself I’m trying to convince.) Since no one pays me for what little code I write, no one shouts at me for getting it wrong. So I have a happy and sloppy coding style. I write what I feel like, and publish it. I don’t test it. Worse, sometimes, I don’t even run it once. I’ve sent little scripts off to people which wouldn’t even compile. I make changes to this site at midnight, upload it, and go off to sleep without checking if the change has crashed the site or not. But no one tells me so At work, that’s usually OK. On the few occasions where I’ve written Perl scripts or VB Macros that don’t work, people call me back within a few hours, very worried that THEY’d done something wrong. (Sometimes, I don’t contradict them.) It can be quite a stressful experience but good thing you can learn more here on how to cope up with it. On my site, I don’t always get that kind of feedback. People just click the back button and go elsewhere. Recently, I’ve been doing more Javascript work on my site than writing stuff. Usually, the code works for me. (I write it for myself in the first place.) But I end up optimising for Firefox rather than IE, and for the plugins I have, etc. When I try the same app a few months later on my media PC, it doesn’t work, and shockingly enough, no one’s bothered telling me about it all these months. They’d just click, nothing happens, they’d vanish. But their browsers can tell me The good part about writing code in Javascript is that I can catch exceptions. Any Javascript error can be trapped. So since the end of last year, I’ve started wrapping almost every Javascript function I write in a try {} catch() {} block. In the catch block, I send a log message reporting the error. The code looks something like this:

function log(e, msg) {
  for (var i in e) msg += i + "=" + e[i] + "\n";
  (new Image()).src = "log.pl?m=" + encodeURIComponent(msg);
}

function abc() {
  try {
    // ... function code
  } catch (e) {
    log(e, "abc");
  }
}

Any time there’s an error in function abc, the log function is called. It sends the function name ("abc") and the error details (the contents of the error event) to log.pl, which stores the error, along with details like the URL, browser, time and IP address. This way, I know exactly where what error occurs. This is a fantastic for a three reasons.

It tells me when I’ve goofed up. This is instantaneous feedback. I don’t have to wait for a human. If you run my program on your machine, and it fails, I get to know immediately. (Well, as soon as I read the error log, at least.)
It tells me where I’ve goofed up. The URL and the function name clearly indicate the point of failure.
It tells me why I’ve goofed up. Almost. Using the browser name and the error message, I can invariably pinpoint the reason for the error. Then it’s just a matter of taking the time to fix it.

I’d think this sort of error reporting should be the norm for any software. At least for a web app, given how easy it is to implement.

Comments

CodeKitten 25 Dec 2007 12:00 pm: You should bind an onerror-triggered method to call your log method aswell :)
Danko Stojanovic 10 May 2010 8:12 am: I feel your pain. I also moved away from programming to management. I have to live with customers saying “It doesn’t work” and programmers “It works just fine when I test it”. I so wish to know what is going on my client’s computer. Thanks for this article. It helped me decide to pursue this further
Allan Ebdrup 13 Nov 2011 7:23 am:The onerror event is a great way to start logging JavaScript errors, more people should use it. JavaScript errors can be just as bad as a webserver that is down, you really need to log those errors so you can do something about them.onerror, is quirky. It does not fire on iPad, iPhone, Safari and some other browseres. To catch those errors you have to insert try-catch’es arount your event handlers, xhr-requests and some other places. This will also let you get more detail about the errror (call stack in some browsers).But onerror is still much better than nothing.
We’ve solved a lot of these issues, with a solution that you install just like Google Analytics. Currently we are in a testing phase, private beta (free). To get an invite you can got to http://muscula.com and sign up.
Kiawaki 8 Jul 2011 5:45 am: A question from somebody who is even less of a pro web developer: how exactly do you use this code? Do you put it before or after the script or both? Thanks!
Offbeatmammal 23 Sep 2011 12:48 am: if you want a no-brain, minimal effort then have a look at jsErrLog … a simple utility that does this and logs it to a remote AppEngine service (all for free, in an open source project) - check out http://jsErrLog.appspot.com which links to the GitHub repository as well

Handling missing pages

Mon, 24 Dec 2007 12:00:00 +0000

If something goes wrong with my site, I like to know of it. My top three problems are:

The site is down

A page is missing

Javascript isn’t working

This article covers the second topic.

One thing I’m curious about is hits to non-existent pages (404s) on my site. I usually get 404s because:

I renamed the page
Someone typed a wrong URL
Someone followed a wrong link

Find the 404

The first problem is to know when someone gets a 404. I’ve seen sites that tell you to contact the administrator in case of a 404. That’s crazy. The administrator should automatically detect of 404s! Almost every web server provides this facility.

The real issue is attention. I receive 700 404s a day. That’s too much to manually inspect. And most of these are not for proper web pages, but for images (for example, almost all my 404s used to be for browsers requesting favicon.ico) or weird MS Office files.

I’m interested in a small subset of 404 errors. Those that hit a web page, not support files. And those requested by a human, not a search engine or a program.

A decent way of filtering these is to use Javascript in your 404 page. Javascript is typically executed only by browsers (i.e. humans, not search engines), and only in a web page (not images, etc.) So if you serve Javascript in your 404 page, and it gets executed, it’s likely to be a human requesting a web page.

I have a piece of Javascript in my custom 404 page that looks something like this:

<script>(new Image()).src = "/log.pl";</script>

Every time this code runs, it loads a new image. The source of the image is a Perl script, log.pl. Every time log.pl is accessed, it logs the URL from which it was called. I’m reasonably guaranteed that these are web pages a human tried to access.

The reduction in volume is tremendous. On a typical month, I get ~20,000 404 errors. With the Javascript logging, it’s down to around 200 a month, and most of them quite meaningful.

Point to the right page

Sometimes, the change happens because I changed the URLs. I keep fiddling with the site structure. Someone would have links to an old page that I’ve renamed. I may not even know that. Even if I did, they can’t be bothered with requests to change the link. So I’ve got to handle it.

The quickest way, I find, is to use Apache’s mod_rewrite. You can simply redirect the old URL to the new URL. For example, I used to have a link to /calvin.html which I now point to /calvin/. That becomes a simple line on my .htaccess file:

RewriteRule calvin.html  calvinandhobbes.html

I don’t do this for every site restructuring, though. I just restructure, wait for someone to request a wrong page, and when my 404 error log warns me, I create a line in the .htaccess. It keeps the redirections down to a minimum, and only for those links that are actually visited.

Be flexible with the URL structure

Sometimes people type in a wrong link. Often, these are unintentional. Here are some common misspellings for my Hindi songs search.

s-anand.net/hindi/
s-anand.net/Hindi
s-anand.net/hiundi

Occasionally, people are exploring the structure of my site:

s-anand.net/excel
s-anand.net/music
s-anand.net/hits

I need to decide what to do with both cases. For the former, sometimes my URL structure is too restrictive. I mean, why should someone have to remember to type /hindi instead of /Hindi or /hindi/? Who cares about case? Who cares about a trailing slash?

In such cases, I map all the variants to the right URL using mod_rewrite. For example, typing s-anand.net/HiNDi (with or without caps, with or without a slash at the end) will still take you to the right page.

As I keep discovering new mis-spellings, I take a call on whether to add it or not. The decision is usually based on volume. If two people make the same spelling mistake in a day, I almost certainly add the variant. Most of the time, it’s just typing errors like /hiundi which isn’t repeated oftener than once a month.

Provide search

To handle the exploratory URLs, and people following wrong links, I’ve turned my custom 404 page into a search engine.

For example, when someone types s-anand.net/excel, I know they’re searching for Excel. So I just do a Google Custom Search within my site for “excel” – that is, anything following the URL.

It’s a bit more complex than that, actually. I do a bit of tweaking to the URL, like convert punctuations (underscore, hyphen, full-stop, etc.) to spaces, remove common suffixes (.html, .htm) and ignore numbers. Quite often, it matches something on my site that they’re looking for. If not, ideally, I ought to try for various alternatives and subsets of the original search string to figure out a good match. But given that the number of mismatches is down to about one a day, I’m fairly comfortable right now.

What this means, incidentally, is that my site is, by default, a search engine for itself. To search for movie-related stuff on my site, just type s-anand.net/movie and you get a search of the word “movie” on my site. (Sort of like on a9.com, where searching for a9.com/keyword does a search on the keyword.)

Website load distribution using Javascript

Sat, 01 Sep 2007 12:00:00 +0000

My music search engine shows a list of songs as you type – sort of like Google’s autosuggest feature. I load my entire list of songs upfront for this to work. Though it’s compressed to load fast, each time you load the page, it downloads about 500KB worth of song titles.

My allotted bandwidth on my hosting service is 3GB per month. To ensure I don’t exceed it, I uploaded the songs list to an alternate free server: Freehostia. This keeps my load down. If I exceed Freehostia’s limit, my main site won’t be affected – just the songs. I also uploaded half of them to Google Pages, to be safe.

This all worked fine… until recently. Google Pages has a relatively low bandwidth restriction. (Not sure what, and they won’t reveal it, but my site is affected.) Freehostia is doing some maintenance, and their site goes down relatively often. So my song search goes down when any of these go down.

Now, these are rarely down simultaneously. Just one or the other. But whenever Freehostia is down, I can’t listen to one bunch of songs. When Google Pages is down, I can’t listen to another.

What I needed was a load distribution set-up. So I’ve made one in JavaScript.

Normally, I load the song list using an external javascript. I have a line that says:

<script src="http://sanand.freehostia.com/songs/..."></script>

… and the song’s loaded from Freehostia.

What I’d like to do is:

loadscripts(
  "list.hasLoaded()",
  [
    "http://sanand.freehostia.com/songs/...",
    "http://root.node.googlepages.com/...",
    "http://www.s-anand.net/songs/...",
  ],
);

If the function can’t load it from the first link, it loads it from the second, and so on, until list.hasLoaded() returns true.

Here’s how I’ve implemented the function:

function loadscripts(check, libs) {
  document.write(
    "<script src=\"" + libs.shift()
      + "\"><" + "/script>",
  );

  if (libs.length > 0) {
    document.write(
      "<script>if (!(" + check + ")) {"
        + "loadscripts(\"" + check + "\",[\"" + libs.join("\",\"") + "\"])"
        + "}</" + "script>",
    );
  }
}

The first document.write loads the first script in the list. The if condition checks if there’s more scripts to load. If yes, the second document.write writes a script that

checks whether the script is already loaded, and
if not, loads the rest of the scripts using the same function.

I’ve expanded the sites that have these free songs as well. So now, as long as my main site works, and at least one of the other sites work, the search engine will work.

PS: You can easily expand this to do random load distribution as well.

Comments

Saurabh 1 Sep 2007 12:00 pm: I think its time for you now to start your own company..:) Guess you have some great ideas and u just need to put them on paper and start executing!
RK 1 Sep 2007 12:00 pm: Have you looked at the DOJO toolkit. It does have the feature you are looking for namely including scripts and verifying if the included script loaded. You might have to customize it a bit to suit your needs as its an entire toolkit.
S Anand 1 Sep 2007 12:00 pm: Saurabh – you’’re right. Problem is, I’’m way to conservative!

RK – I know of Dojo, but didn’’t know of this functionality: thanks! You’’re right, I probably won’’t want the entire toolkit, it’’s quite bulky. But I’’m sure I could learn from their implementation.
jawahar 1 Sep 2007 12:00 pm: Hai anand, can u suggest or give an insight on how to keep track of files which we work on?? not program codes ? but reports we read ,write n edit.. so as to retrieve it faster .. since ur working in such an environment can u share ur exp on naming files and organising it. thanks, jawahar
kgnanasekaran 1 Sep 2007 12:00 pm: Dear Mr.Anand, I really enjoyd in this website lot. R u tamilan??? superb site.. plz carry on.. thanks a lot..?? if u like we will bcome frnds.. am always available in my gmail..kgnanasekaran@gmai.com keep n touch. with warm regards, shekar.
x_Vanity_! 1 Sep 2007 12:00 pm: I really enjoy your website. Thanks for all the information you share.
Ramesk K palleri 1 Sep 2007 12:00 pm: Kudos anand.. u have done a great job on yous music site. keep going..
RAVINDRA 1 Sep 2007 12:00 pm: I REALLY ENJOY YOUR WEBSITE. BUT I WANT LISTEN SONGS IN YOUR MEDIA PLAYER.
oracle 1 Sep 2007 12:00 pm: where are you oflate?missing your post
krishna rao jallipalli 1 Sep 2007 12:00 pm: DEAR MR. ANAND, INFACT YOUR WEBSITE… SONGS..SUPERB. YOU ARE DOING A GREAT JOB. THOUGH YOU IN AMERICA…INFACT.. THESE TYPE OF WEBSITES.. SO MANY.. ARE BEING MAINTAINED BY NRIs. REALLY GREAT. MY HEARTY WISHES. REALLY I AM ENJOYING. WITH BEST REGARDS J KRISHNA RAO GUNTUR A.P CELL:9949517103

A busy break from blogging

Fri, 24 Aug 2007 12:00:00 +0000

Between July 17th and August 22nd, I saw 57 movies and read 7 books. There were Saturdays when I watched four movies back-to-back. (I tried five. Couldn’t stay awake.) Amidst this, I also cooked, cleaned, shopped… and went to office. (Oh yes, I was working 10 hours a day.) And managed to build some interesting sites which I’ll release in a while.

But first, let me share the books with you.

Harry Potter and the Deathly Hallows

I wasn’t planning to buy it. I figured I’d just wait for the soft copy. On 21st July at 8am, I went shopping to the local Sainsbury’s to get groceries for my pre-movie cooking. I didn’t know that was the release date. And there it was. In a huge stack. 50% discount. Should I? Shouldn’t I? After finishing the rest of my shopping, and having deeply analysed the cost-benefit and ROI, I figured: if I didn’t buy it now, someone else might tell me the answers!

Was Snape evil? I couldn’t believe that. Not after Dumbledore’s implicit trust. Besides, I re-read Harry Potter and the Half-Blood Prince, and if Dumbledore was dismissing Harry’s explicit warnings about Snape, he had to know something more. Anyway, what was the significance of Dumbledore’s last words? “Severus… please…” Please what? Dumbledore begging for death seemed more likely than Dumbledore begging for life. I had to know.

Who dies? Voldemort, of course. But who else? It couldn’t be Harry, unless J K Rowling was looking to make herself one of the most hated novelists. Yet, it seems so… possible. Harry dying to take Voldemort out. Naah, can’t be. Not Ron or Hermione either. Same reason. One of the other Weasleys? Maybe. Plenty of them anyway. Hopefully Percy. Hargrid? Possible. Lupin? The last of Harry’s father’s friends?

And all the minor questions: What’s the significance of Harry’s eyes? What does Wormtail do to help Harry? What’s the significance of Voldemort having used Harry’s blood to resurrect himself? etc. etc.

So I bought it.

But didn’t start reading.

I knew that if I picked it up, I wouldn’t put it down. It was time to cook. And watch movies.

By 4:00pm, after three movies, I couldn’t stand it any more. So I picked it up. Read until 2:00am. Picked it up again on Sunday at 9:00am, and starved until 11:00am until I finished it.

Whew! What a book. Definitely the raciest of the lot. My earlier favourite in the series was The Prisoner of Azkaban, though The Half-Blood Prince came close. But this one beats them all. Resolves most of the mysteries till date, too. As Stephen King says in his review of Harry Potter, but by the time she penned the final line of Deathly Hallows, she had become one of the finer stylists in her native country.

How to be Good

And then there was Nick Hornby’s book. I’d seen a couple of his movies: About a Boy and High Fidelity. They were interesting, and I’d heard the books were good. Figured I’d pick one up.

And it was hilarious!

How to be Good is the funniest book I’ve read since Five Point Someone and The Inscrutable Americans. Most of my colleagues kept wondering what I was laughing out so loudly about.

This is the story. The author is a doctor and a good wife. “Gooder” than her husband, certainly, and that makes her feel good. Until he has suddently becomes GOOD. Truly good. Saint-like. And then she can’t stand him any more. The story is in first person, so you can see her thoughts almost verbatim. (See thoughts verbatim? Well, whatever the phrase is.)

The Runaway Jury

I had just seen the movie Runaway Jury, so I had to re-read it immediately. The movie was surprisingly good, though. Dustin Hoffman, Gene Hackman, John Cusack and Rachel Weisz. Many changes from the book, but didn’t detract from the experience.

Batman Year One, The Dark Knight Returns and Understanding Comics

I saw 300. WOW! Brilliant. One of the greatest visual experiences ever. Possibly better than V for Vendetta and certainly better than Sin City – both of which I thought had incredible visuals. The colours, the texture, the contrast, the surrealism – whew!

That’s when it hit me. Three of the best movies I’ve seen recently were based on graphic novels (comics). Two by Frank Miller. Maybe I should explore this a little more.

I got myself Batman Year One – and that’s when I realised where Batman Begins got its inspiration from. The graphics were pretty old style, but the story, incredible. Then I picked up The Dark Knight Returns. Now THAT is phenomenal graphics. And what a story! Christopher Nolan’s next movie is slated to be The Dark Knight. Really looking forward to that.

With all this, I ended up reading a bunch of new Superman comics as well (but they were lousy, so I won’t mention anything), and in the meanwhile, heard about Scott McCloud’s comic book on comics – Understanding Comics. Like I said, it’s a comic book, but non-fiction. It’s about the history and art of comics. Very nice reading, and quite insightful too. I think every visual designer should take a look at it.

Pro Javascript Techniques

I’d been reading up a lot of Javascript recently – learning mostly from Douglas Crockford, Peter Paul-Koch, Dean Edwards, and John Resig. So when I realised John had a book, I had to read it. Douglas Crockford recommends JavaScript: The Definitive Guide (5th Edition) as the least bad among many bad books. I read it. Sorry, but it was quite a bore. Pro Javascript Techniques on the other hand, is gripping. Dives right into modern techniques, writing style, and is filled with practical advice. How wonderful.

So anyway, that’s what my month’s been like. Well worth the break from blogging, I think. (And I haven’t even told you about the movies or sites yet! Well, soon.)

Comments

Sachin Gopalakrishnan 24 Aug 2007 12:00 pm: And i thought this was not exactly a blog!!!
Vinu Yamunan 24 Aug 2007 12:00 pm: Dude, Where’’d you have the energy for all that? Seeing that you cooked, cleaned AND shopped, my guess is that the wife was not in town - true/false? Anyways, good to see you on the blogosphere. Love your music as well as the C&H search pages. Am surprised you have not commented on the ‘‘Asap Utilities Add-in" in your excel related postings… check it out if you haven’’t already! later Vinu
Karthik A 24 Aug 2007 12:00 pm: I felt so too about the last Harry Potter. Like Vinu said, you got the rights to 5 movies after cooking and cleaning on a weekend since your wife’’s not in town…..
sathish 24 Aug 2007 12:00 pm: Anand, a nice set of books.. especially the graphic novels.. I still do not understand the term graphic novel that much.. but, love comics anyway…. I have been reading Osama Tezuka’’s Buddha series - Every time, I see The Dark Knight, my hands itch; but, they are so damn costly in India (atleast the hardback is).. And if you are interested in fantasy - I would suggest Steven Erikson’’s – A Tale of Malazan The book of Fallen I have been reading them with greater ferocity than Tolkien and George R R Martin..Do give it a shot if you are in a mood for fantasy - but, be prepared for long long hours of endless reading..

Sparklines

Wed, 25 Apr 2007 12:00:00 +0000

John Resig has written a Sparklines library. Here’s an example. I wrote that HTTP download speeds not linear 182,315,313,319,314,459,441,445,453,525,567,552,577,587,580,581,590,663,639,658,616,705,720,695,739,750,720,741,803,800,800,818,800,856,796,816,866,841,836,828,861,893,859,905,881,885,946,944,943,984,1003,1012,994,979,977,986,1010,1017,1027,1000,1035,986,1006,1006,996,1022,1003,1053,1046,1061,1002,1064,1014,1039,1061,1023,1076,1081,1019,1064,1072,1089,1101,1069,1128,1125,1092,1155,1170,1067,1094,1082,1178,1211,1154,1169,1133,1161,1193,1167 and that they flatten out over time. A linear line would look like this: 180,190,201,211,221,232,242,252,262,273,283,293,304,314,324,335,345,355,365,376,386,396,407,417,427,438,448,458,468,479,489,499,510,520,530,541,551,561,572,582,592,602,613,623,633,644,654,664,675,685,695,705,716,726,736,747,757,767,778,788,798,808,819,829,839,850,860,870,881,891,901,912,922,932,942,953,963,973,984,994,1004,1015,1025,1035,1045,1056,1066,1076,1087,1097,1107,1118,1128,1138,1148,1159,1169,1179,1190,1200 The little red line here is a sparkline that’s based on real data. John’s javascript converts the data into a graph.

Sparklines were introduced by Edward Tufte.

javascript on S Anand

Mapscii Rendering

Architecture Overview

Key Libraries Used

Step-by-Step Tutorial

Step 1: Understanding Braille Characters for Pixels

Step 2: Creating a Braille Buffer

Step 3: Drawing Lines with Bresenham’s Algorithm

Step 4: Drawing Filled Polygons with Earcut

Step 5: Processing Vector Tiles

Step 6: The Complete Rendering Pipeline

Step 7: Coordinate Transformations

Complete Example: Drawing a Simple Map

Performance Optimizations

Key Takeaways

Patterns for Short Code

1. Data-Driven Design Over Repetitive Conditionals

2. Optional Chaining & Nullish Coalescing

3. Pattern Arrays for Iterative Operations

4. Destructuring for Clarity

5. Arrow Functions with Implicit Returns

6. Combine Related Functions

7. Object Literal Shorthand

8. Remove Dead Code

9. Inline Single-Use Helpers

10. Single-Line Simple Functions

11. Early Returns Over Nested Conditions

12. Spread Operator for String Iteration

13. Concise Module Exports

Summary: Core Principles

Things I Learned - 06 Jul 2025

Breaking mental coding barriers with LLMs

Cyborg scraping

Comments

Programming Minecraft with Websockets

Connect to Minecraft

Subscribe to chat messages

Build structures using chat

Understand Minecraft’s responses

Wait for commands to be done

Comments

Rodland 8 Mar 2021 12:45 am:Thanks for a great tutorial.However when I try to implement the chat listener (server2.js) i get an JSON error from MineCraft (in node.js console):“kryptert økt kreves” = “Needs secure connection”

Auto reloading pages

Codecasting

Watch a demo of how I used it for codecasting

Comments

Javascript arrays vs objects

Comments

Error logging with Google Analytics

Comments

Random quotes generator

Client side scraping for contacts

Comments

Bound methods in Javascript

Heading

Comments

JPath - XPath for Javascript

Comments

Chaining functions in Javascript

Comments

Javascript error logging

Comments

Handling missing pages

Website load distribution using Javascript

Comments

A busy break from blogging

Comments

Sparklines