Coding

I tried 2 experiments. Can I code a visual data story only using LLMs? Does this make me faster? How much? Has GitHub Copilot caught up with Cursor? How far behind is it? Can I recommend it? So I built a visual story for Lech Mazur’s elimination game benchmark (it’s like LLMs playing Survivor) using only the free GitHub Copilot as the AI code editor. SUMMARY: using LLMs and AI code editors make me a bit faster. It took me 7 hours instead of 10-12. But more importantly: ...

Here’s the GitHub Actions file (.github/workflows/deploy.yaml) I use to publish to GitHub pages. name: Deploy to GitHub Pages on: # Run when pushed. Use { branches: [main, master] } to run only on specific branches push: # Allow manual triggering of the workflow workflow_dispatch: # OPTIONAL: Run at a specific cron schedule, e.g. first day of every month at 12:00 UTC (noon) schedule: - cron: "0 12 1 * *" permissions: # To deploy to GitHub Pages pages: write # To verify that deployment originated from the right source id-token: write jobs: # Run as a single build + deploy job to reduce setup time deploy: # Specify the deployment environment. Displays the URL in the GitHub Actions UI environment: name: github-pages url: ${{ steps.deployment.outputs.page_url }} # Run on the latest Ubuntu LTS runs-on: ubuntu-latest \ steps: # Checkout the repository - uses: actions/checkout@v4 # Run whatever commands you want - run: echo '<h1>Hello World</h1>' > index.html # Upload a specific page to GitHub Pages. Defaults to _site - uses: actions/upload-pages-artifact@v3 with: path: . # Deploy the built site to GitHub Pages. The `id:` is required to show the URL in the GitHub Actions UI - id: deployment uses: actions/deploy-pages@v4 This is based on Simon Willison’s workflow and some of my earlier actions. ...

Fake data is usually boring if you analyze it. It’s usually uniform, with no outliers or interesting patterns. If I ask ChatGPT: Generate realistic fake tourism data using these columns: - Age - Nationality - Gender - Income - Booking_Channel - Month - Occupancy_Rate - Travel_Frequency - Spending Run the code and let me download the output as a CSV file. … the output is remarkably boring. Men & women from all countries and ages in every month visit equally. Income and spending are uniformly distributed - and the same pattern holds for all countries and ages. ...

At PyConf Hyderabad, I spoke about uv. It's a package manager for Python. I usually mix live demos into my narrative. So, rather than present with something static like PowerPoint (or Google Slides), I usually use: Front-end: Custom HTML mixed with RevealJS and CodePen. Observable is a good, too. Python: Jupyter Notebooks. Marimo is good, too. Others: Markdown and VS Code for most other things, e.g. SQL. For this talk, I needed to run commands on the shell. I evaluated: ...

Zohaib Rauf suggested using LLMs to spec code and using Cursor to build it. (via Simon Willison). I tried it. It’s promising, but my first attempt failed. I couldn’t generate a SPEC.md using LLMs At first, I started writing what I wanted. This application identifies the drugs, diseases, and symptoms, as well as the emotions from an audio recording of a patient call in a clinical trial. … and then went on to define the EXACT code structure I wanted. So I spent 20 minutes spec-ing our application structure and 20 minutes spec-ing our internal LLM Foundry APIs and 40 minutes detailing every step of how I wanted the app to look and interact. ...

In my Tools in Data Science course, I launched a Project: Automated Analysis. This is automatically evaluated by a Python script and LLMs. I gently encouraged students to hack this - to teach how to persuade LLMs. I did not expect that they’d hack the evaluation system itself. One student exfiltrated the API Keys for evaluation by setting up a Firebase account and sending the API keys from anyone who runs the script. ...

The Gemini documentation is clear: The File API service extracts image frames from videos at 1 frame per second (FPS) and audio at 1Kbps, single channel, adding timestamps every second. These rates are subject to change in the future for improvements in inference. Note: The details of fast action sequences may be lost at the 1 FPS frame sampling rate. Consider slowing down high-speed clips for improved inference quality. Individual frames are 258 tokens, and audio is 32 tokens per second. With metadata, each second of video becomes ~300 tokens, which means a 1M context window can fit slightly less than an hour of video. ...

It's surprisingly easy to clone a voice using F5-TTS: "A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching". Here's a clip of me, saying: I think Taylor Swift is the best singer. I've attended every one of her concerts and in fact, I've even proposed to her once. Don't tell anyone. (Which is ironic since I didn't know who she was until this year and I still haven't seen or heard her.) ...

In 1994, I learnt Perl. It was fantastic. I used it to: 1995: Build CCChat - the unofficial IITM email system and software repository 1999: Build my entire blog from scratch 2000: Author my 2nd year thesis on the Behavioural Aspects of Financial Analysts by analyzing 600MB of IBES data 2002: Analyze where to place the central processing hubs for a bank 2004: Analyze the interest durations of public sector banks 2005: Creating music quizzes 2006: Create my own music search engine (which earned me about $100 a month in Google Ad revenue for a while) 2006: Automated resume filtering 2007: Create custom search engines 2008: Build application launchers In 2006, I was convinced I should stick to Perl over Python. ...

cursor.directory is a catalog of Cursor rules. Since I’ve actively switched over from VS Code to Cursor as my editor, I reviewed the popular rules and came up with this as my list: You are an expert full stack developer in Python and JavaScript. Write concise, technical responses with accurate Python examples. Use functional, declarative programming; avoid classes. Avoid code duplication (iteration, functions, vectorization). Use descriptive variable names with auxiliary verbs as snake_case for Python (is_active, has_permission) and camelCase for JavaScript (isActive, hasPermission). Functions should receive and object and return an object (RORO) where possible. Use environment variables for sensitive information. Write unit tests in pytest for Python and Jest for JavaScript. Follow PEP 8 for Python. Always use type hints in all function signatures. Always write docstrings. Use Google style for Python and JSDoc for JavaScript. Cache slow or frequent operations in memory. Minimize blocking I/O operations with async operations. Only write ESM (ES6) JavaScript. Target modern browsers. Libraries ...

In Excel, using Visual Basic, you can change an image as you scroll. This makes it easy to look at each image and annotate it. This is how I transcribed every Calvin & Hobbes. I used this technique first when typing out the strips during my train rides from Bandra to Churchgate. I had an opportunity to re-apply it recently when we needed to tag hundreds of photographs based on a set of criteria. ...

text-embedding-ada-002 used to give high cosine similarity between texts. I used to consider 85% a reasonable threshold for similarity. I almost never got a similarity less than 50%. text-embedding-3-small and text-embedding-3-large give much lower cosine similarities between texts. For example, take these 5 words: “apple”, “orange”, “Facebook”, “Jamaica”, “Australia”. Here is the similarity between every pair of words across the 3 models: For our words, new text-embedding-3-* models have an average similarity of ~43% while the older text-embedding-ada-002 model had ~85%. ...

I am a fairly good programmer. So, when I see a problem, my natural tendency is to code. I’m trying to break that pattern. Instead, I ask ChatGPT. For example, I asked: Write a compact 1-line Python expression that checks if user.id ends with @gramener.com or @straive.com user.id.endswith(("@gramener.com", "@straive.com")) After 15 years of using Python, I learnt that .endswith() supports tuple suffixes. This has been around since Python 2.5 (released in 2006 – before I knew Python.) The documentation has a tiny sentence in the middle saying “suffix can also be a tuple of suffixes to look for.” ...

Even after 30 years of HTML, I learn new things about it. This Monday morning, I woke up to a mail from Sundeep saying requests for a Data Engineer - AWS/Azure/GCP in our internal fulfilment portal raised an error. My guess was one of these: The “/” in the role is causing a problem. (Developer mistake.) The role exists in one table but not the other. (Recruitment team mistake.) The application wasn’t set up / restarted properly. (IT mistake.) All three were wrong. So I dug deeper. ...

LinkedIn has a page that shows the people who most recently followed you. At first, it shows just 20 people. But as you scroll, it keeps fetching the rest. I’d love to get the full list on a spreadsheet. I’m curious about: What kind of people follow me? Which of them has the most followers? Who are my earliest followers? But first, I need to scrape this list. Normally, I’d spend a day writing a program. But I tried a different approach yesterday. ...

At PyCon Indonesia, I spoke about a project we worked on with the World Mosquito Program. The World Mosquito Program (WMP) modifies mosquitoes with a bacteria – Wolbachia. This reduces their ability to carry deadly viruses. (It makes me perversely happy that we’re infecting mosquitoes now 😉.) Modifying mosquitoes is an expensive process. With a limited set of “good mosquitoes”, it is critical to find the best release points that will help them replicate rapidly. ...

Minecraft lets you connect to a websocket server when you’re in a game. The server can receive and send any commands. This lets you build a bot that you can … (well, I don’t know what it can do, let’s explore.) Minecraft has commands you can type on a chat window. For example, type / to start a command and type setblock ~1 ~0 ~0 grass changes the block 1 north of you into grass. (~ means relative to you. Coordinates are specified as X, Y and Z.) ...

One problem I’ve had in Markdown is rendering a content in columns. On Bootstrap, the markup would look like this: <div class="row"> <div class="col">...</div> <div class="col">...</div> </div> How do we get that into Markdown without writing HTML? On Python, the attribute lists extension lets you add a class. For example: This is some content {: .row} … renders <p class="row">This is some content</p>. But I can’t do that to multiple paragraphs. Nor can I next content, i.e. add a .col inside the .row. ...

With Office 365, PowerPoint supports SVG editing. This is really powerful. It means you can draw in PowerPoint and render it on the web – including as interactive or animated visuals. For example, the SVG in this simulator was created just with PowerPoint. The process is simple. Draw anything. Select any shapes and right-click. Select Save As Picture… and choose SVG. For example, you can use PowerPoint to create Smart Art, export it as SVG, and embed it into a page. See this example on CodePen. ...

Given the blazing speed of Node.js these days, I expected HTML parsing to be faster on Node than on Python. So I compared lxml with htmlparser2 – the fastest libraries on Python and JS in parsing the reddit home page (~700KB). lxml took ~8.6 milliseconds htmlparser2 took ~14.5 milliseconds Looks like lxml is much faster. I’m likely to stick around with Python for pure HTML parsing (without JavaScript) for a while longer. In [1]: from lxml.html import parse In [2]: %timeit tree = parse('reddit.html') 8.69 ms ± 190 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) const { Parser } = require("htmlparser2"); const { DomHandler } = require("domhandler"); const fs = require("fs"); const html = fs.readFileSync("reddit.html"); const handler = new DomHandler(function(error, dom) {}); const start = +new Date(); for (var i = 0; i < 100; i++) { const parser = new Parser(); parser.write(html); parser.end(); } const end = +new Date(); console.log((end - start) / 100); Note: If I run the htmlparser2 code 100 times instead of 10, it only takes 7ms per loop. The more the number of loops, the faster it parses. I guess Node.js optimizes repeated loops. But I’m only interested in the first iteration, since I’ll be parsing files only once.