Coding

At PyConf Hyderabad, I spoke about uv. It's a package manager for Python. I usually mix live demos into my narrative. So, rather than present with something static like PowerPoint (or Google Slides), I usually use: Front-end: Custom HTML mixed with RevealJS and CodePen. Observable is a good, too. Python: Jupyter Notebooks. Marimo is good, too. Others: Markdown and VS Code for most other things, e.g. SQL. For this talk, I needed to run commands on the shell. I evaluated: ...

Zohaib Rauf suggested using LLMs to spec code and using Cursor to build it. (via Simon Willison). I tried it. It’s promising, but my first attempt failed. I couldn’t generate a SPEC.md using LLMs At first, I started writing what I wanted. This application identifies the drugs, diseases, and symptoms, as well as the emotions from an audio recording of a patient call in a clinical trial. … and then went on to define the EXACT code structure I wanted. So I spent 20 minutes spec-ing our application structure and 20 minutes spec-ing our internal LLM Foundry APIs and 40 minutes detailing every step of how I wanted the app to look and interact. ...

In my Tools in Data Science course, I launched a Project: Automated Analysis. This is automatically evaluated by a Python script and LLMs. I gently encouraged students to hack this - to teach how to persuade LLMs. I did not expect that they’d hack the evaluation system itself. One student exfiltrated the API Keys for evaluation by setting up a Firebase account and sending the API keys from anyone who runs the script. ...

The Gemini documentation is clear: The File API service extracts image frames from videos at 1 frame per second (FPS) and audio at 1Kbps, single channel, adding timestamps every second. These rates are subject to change in the future for improvements in inference. Note: The details of fast action sequences may be lost at the 1 FPS frame sampling rate. Consider slowing down high-speed clips for improved inference quality. Individual frames are 258 tokens, and audio is 32 tokens per second. With metadata, each second of video becomes ~300 tokens, which means a 1M context window can fit slightly less than an hour of video. ...

It's surprisingly easy to clone a voice using F5-TTS: "A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching". Here's a clip of me, saying: I think Taylor Swift is the best singer. I've attended every one of her concerts and in fact, I've even proposed to her once. Don't tell anyone. (Which is ironic since I didn't know who she was until this year and I still haven't seen or heard her.) ...

In 1994, I learnt Perl. It was fantastic. I used it to: 1995: Build CCChat - the unofficial IITM email system and software repository 1999: Build my entire blog from scratch 2000: Author my 2nd year thesis on the Behavioural Aspects of Financial Analysts by analyzing 600MB of IBES data 2002: Analyze where to place the central processing hubs for a bank 2004: Analyze the interest durations of public sector banks 2005: Creating music quizzes 2006: Create my own music search engine (which earned me about $100 a month in Google Ad revenue for a while) 2006: Automated resume filtering 2007: Create custom search engines 2008: Build application launchers In 2006, I was convinced I should stick to Perl over Python. ...

cursor.directory is a catalog of Cursor rules. Since I’ve actively switched over from VS Code to Cursor as my editor, I reviewed the popular rules and came up with this as my list: You are an expert full stack developer in Python and JavaScript. Write concise, technical responses with accurate Python examples. Use functional, declarative programming; avoid classes. Avoid code duplication (iteration, functions, vectorization). Use descriptive variable names with auxiliary verbs as snake_case for Python (is_active, has_permission) and camelCase for JavaScript (isActive, hasPermission). Functions should receive and object and return an object (RORO) where possible. Use environment variables for sensitive information. Write unit tests in pytest for Python and Jest for JavaScript. Follow PEP 8 for Python. Always use type hints in all function signatures. Always write docstrings. Use Google style for Python and JSDoc for JavaScript. Cache slow or frequent operations in memory. Minimize blocking I/O operations with async operations. Only write ESM (ES6) JavaScript. Target modern browsers. Libraries ...

In Excel, using Visual Basic, you can change an image as you scroll. This makes it easy to look at each image and annotate it. This is how I transcribed every Calvin & Hobbes. I used this technique first when typing out the strips during my train rides from Bandra to Churchgate. I had an opportunity to re-apply it recently when we needed to tag hundreds of photographs based on a set of criteria. ...

text-embedding-ada-002 used to give high cosine similarity between texts. I used to consider 85% a reasonable threshold for similarity. I almost never got a similarity less than 50%. text-embedding-3-small and text-embedding-3-large give much lower cosine similarities between texts. For example, take these 5 words: “apple”, “orange”, “Facebook”, “Jamaica”, “Australia”. Here is the similarity between every pair of words across the 3 models: For our words, new text-embedding-3-* models have an average similarity of ~43% while the older text-embedding-ada-002 model had ~85%. ...

I am a fairly good programmer. So, when I see a problem, my natural tendency is to code. I’m trying to break that pattern. Instead, I ask ChatGPT. For example, I asked: Write a compact 1-line Python expression that checks if user.id ends with @gramener.com or @straive.com user.id.endswith(("@gramener.com", "@straive.com")) After 15 years of using Python, I learnt that .endswith() supports tuple suffixes. This has been around since Python 2.5 (released in 2006 – before I knew Python.) The documentation has a tiny sentence in the middle saying “suffix can also be a tuple of suffixes to look for.” ...

Even after 30 years of HTML, I learn new things about it. This Monday morning, I woke up to a mail from Sundeep saying requests for a Data Engineer - AWS/Azure/GCP in our internal fulfilment portal raised an error. My guess was one of these: The “/” in the role is causing a problem. (Developer mistake.) The role exists in one table but not the other. (Recruitment team mistake.) The application wasn’t set up / restarted properly. (IT mistake.) All three were wrong. So I dug deeper. ...

LinkedIn has a page that shows the people who most recently followed you. At first, it shows just 20 people. But as you scroll, it keeps fetching the rest. I’d love to get the full list on a spreadsheet. I’m curious about: What kind of people follow me? Which of them has the most followers? Who are my earliest followers? But first, I need to scrape this list. Normally, I’d spend a day writing a program. But I tried a different approach yesterday. ...

At PyCon Indonesia, I spoke about a project we worked on with the World Mosquito Program. The World Mosquito Program (WMP) modifies mosquitoes with a bacteria – Wolbachia. This reduces their ability to carry deadly viruses. (It makes me perversely happy that we’re infecting mosquitoes now 😉.) Modifying mosquitoes is an expensive process. With a limited set of “good mosquitoes”, it is critical to find the best release points that will help them replicate rapidly. ...

Minecraft lets you connect to a websocket server when you’re in a game. The server can receive and send any commands. This lets you build a bot that you can … (well, I don’t know what it can do, let’s explore.) Minecraft has commands you can type on a chat window. For example, type / to start a command and type setblock ~1 ~0 ~0 grass changes the block 1 north of you into grass. (~ means relative to you. Coordinates are specified as X, Y and Z.) ...

One problem I’ve had in Markdown is rendering a content in columns. On Bootstrap, the markup would look like this: <div class="row"> <div class="col">...</div> <div class="col">...</div> </div> How do we get that into Markdown without writing HTML? On Python, the attribute lists extension lets you add a class. For example: This is some content {: .row} … renders <p class="row">This is some content</p>. But I can’t do that to multiple paragraphs. Nor can I next content, i.e. add a .col inside the .row. ...

With Office 365, PowerPoint supports SVG editing. This is really powerful. It means you can draw in PowerPoint and render it on the web -- including as interactive or animated visuals. For example, the SVG in this simulator was created just with PowerPoint. The process is simple. Draw anything. Select any shapes and right-click. Select Save As Picture... and choose SVG. For example, you can use PowerPoint to create Smart Art, export it as SVG, and embed it into a page. See this example on CodePen. ...

Given the blazing speed of Node.js these days, I expected HTML parsing to be faster on Node than on Python. So I compared lxml with htmlparser2 – the fastest libraries on Python and JS in parsing the reddit home page (~700KB). lxml took ~8.6 milliseconds htmlparser2 took ~14.5 milliseconds Looks like lxml is much faster. I’m likely to stick around with Python for pure HTML parsing (without JavaScript) for a while longer. In [1]: from lxml.html import parse In [2]: %timeit tree = parse('reddit.html') 8.69 ms ± 190 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) const { Parser } = require("htmlparser2"); const { DomHandler } = require("domhandler"); const fs = require("fs"); const html = fs.readFileSync("reddit.html"); const handler = new DomHandler(function(error, dom) {}); const start = +new Date(); for (var i = 0; i < 100; i++) { const parser = new Parser(); parser.write(html); parser.end(); } const end = +new Date(); console.log((end - start) / 100); Note: If I run the htmlparser2 code 100 times instead of 10, it only takes 7ms per loop. The more the number of loops, the faster it parses. I guess Node.js optimizes repeated loops. But I’m only interested in the first iteration, since I’ll be parsing files only once.

It’s easy to convert addresses into latitudes and longitudes into addresses in Excel. Here’s the Github project with a downloadable Excel file. This is via Visual Basic code for a GoogleGeocode function that geocodes addresses. Function GoogleGeocode(address As String) As String Dim xDoc As New MSXML2.DOMDocument xDoc.async = False xDoc.Load ("http://maps.googleapis.com/maps/api/geocode/" + _ "xml?address=" + address + "&sensor=false") If xDoc.parseError.ErrorCode <> 0 Then GoogleGeocode = xDoc.parseError.reason Else xDoc.setProperty "SelectionLanguage", "XPath" lat = xDoc.SelectSingleNode("//lat").Text lng = xDoc.SelectSingleNode("//lng").Text GoogleGeocode = lat & "," & lng End If End Function Comments Ryan 8 Jun 2015 9:28 pm: I find this isn’t working and says, Compile Error; User defined type not defined xDoc As New MSXML2.DOMDocument what do I change to fix it? Thank you Richie Lionell 27 Jul 2016 6:40 am: Ryan, Inside the VBE, Go to Tools -> References, then Select Microsoft XML, v6.0 . If that doesn’t work unselect that and select Microsoft XML, v3.0

Github offers Github Pages that let you host web pages on Github. You create these by adding a branch to git called gh-pages, and this is often in addition to the default branch master. I just needed the gh-pages branch. So thanks to YJL, here’s the simplest way to do it. Create the repositoryon github. Create your local repository and git commitinto it. Type git push -u origin master:gh-pages In .git/config, under the [remote "origin"] section, add push = +refs/heads/master:refs/heads/gh-pages The magic is the last :gh-pages.

I just scraped the scientific packages on pypi. Here are the top 50 by downloads. Name Description Size Downloads numpy NumPy: array processing for numbers, strings, records, and objects. 2000000 133076 scipy SciPy: Scientific Library for Python 7000000 33990 pygraphviz Python interface to Graphviz 99000 22828 geopy Python Geocoding Toolbox 32000 18617 googlemaps Easy geocoding, reverse geocoding, driving directions, and local search in Python via Google. 69000 15135 Rtree R-Tree spatial index for Python GIS 495000 14370 nltk Natural Language Toolkit 1000000 12844 Shapely Geometric objects, predicates, and operations 93000 12635 pyutilib.component.doc Documentation for the PyUtilib Component Architecture. 372000 10181 geojson Encoder/decoder for simple GIS features 12000 9407 GDAL GDAL: Geospatial Data Abstraction Library 410000 8957 scikits.audiolab A python module to make noise from numpy arrays 1000000 8856 pupynere NetCDF file reader and writer. 16000 8809 scikits.statsmodels Statistical computations and models for use with SciPy 3000000 8761 munkres munkres algorithm for the Assignment Problem 42000 8409 scikit-learn A set of python modules for machine learning and data mining 2000000 7735 networkx Python package for creating and manipulating graphs and networks 1009000 7652 pyephem Scientific-grade astronomy routines 927000 7644 PyBrain PyBrain is the swiss army knife for neural networking. 255000 7313 scikits.learn A set of python modules for machine learning and data mining 1000000 7088 obspy.seisan SEISAN read support for ObsPy. 3000000 6990 obspy.wav WAV(audio) read and write support for ObsPy. 241000 6985 obspy.seishub SeisHub database client for ObsPy. 237000 6941 obspy.sh Q and ASC (Seismic Handler) read and write support for ObsPy. 285000 6926 crcmod CRC Generator 128000 6714 obspy.fissures DHI/Fissures request client for ObsPy. 1000000 6339 stsci.distutils distutils/packaging-related utilities used by some of STScI’s packages 25000 6215 pyopencl Python wrapper for OpenCL 1000000 6124 Kivy A software library for rapid development of hardware-accelerated multitouch applications. 11000000 5879 speech A clean interface to Windows speech recognition and text-to-speech capabilities. 17000 5809 patsy A Python package for describing statistical models and for building design matrices. 276000 5517 periodictable Extensible periodic table of the elements 775000 5498 pymorphy Morphological analyzer (POS tagger + inflection engine) for Russian and English (+perhaps German) languages. 70000 5174 imposm.parser Fast and easy OpenStreetMap XML/PBF parser. 31000 4940 hcluster A hierarchical clustering package for Scipy. 442000 4761 obspy.core ObsPy - a Python framework for seismological observatories. 487000 4608 Pyevolve A complete python genetic algorithm framework 99000 4509 scikits.ann Approximate Nearest Neighbor library wrapper for Numpy 82000 4368 obspy.imaging Plotting routines for ObsPy. 324000 4356 obspy.xseed Dataless SEED, RESP and XML-SEED read and write support for ObsPy. 2000000 4331 obspy.sac SAC read and write support for ObsPy. 306000 4319 obspy.arclink ArcLink/WebDC client for ObsPy. 247000 4164 obspy.iris IRIS Web service client for ObsPy. 261000 4153 Orange Machine learning and interactive data mining toolbox. 14000000 4099 obspy.neries NERIES Web service client for ObsPy. 239000 4066 pandas Powerful data structures for data analysis, time series,and statistics 2000000 4037 pycuda Python wrapper for Nvidia CUDA 1000000 4030 GeoAlchemy Using SQLAlchemy with Spatial Databases 159000 3881 pyfits Reads FITS images and tables into numpy arrays and manipulates FITS headers 748000 3746 HTSeq A framework to process and analyze data from high-throughput sequencing (HTS) assays 523000 3720 pyopencv PyOpenCV - A Python wrapper for OpenCV 2.x using Boost.Python and NumPy 354000 3660 thredds THREDDS catalog generator. 25000 3622 hachoir-subfile Find subfile in any binary stream 16000 3540 fluid Procedures to study geophysical fluids on Python. 210000 3520 pygeocoder Python interface for Google Geocoding API V3. Can be used to easily geocode, reverse geocode, validate and format addresses. 7000 3514 csc-pysparse A fast sparse matrix library for Python (Commonsense Computing version) 111000 3455 topex A very simple library to interpret and load TOPEX/JASON altimetry data 7000 3378 arrayterator Buffered iterator for big arrays. 7000 3320 python-igraph High performance graph data structures and algorithms 3000000 3260 csvkit A library of utilities for working with CSV, the king of tabular file formats. 29000 3236 PyVISA Python VISA bindings for GPIB, RS232, and USB instruments 237000 3201 Quadtree Quadtree spatial index for Python GIS 40000 3000 ProxyHTTPServer ProxyHTTPServer – from the creator of PyWebRun 3000 2991 mpmath Python library for arbitrary-precision floating-point arithmetic 1000000 2901 bigfloat Arbitrary precision correctly-rounded floating point arithmetic, via MPFR. 126000 2879 SimPy Event discrete, process based simulation for Python. 5000000 2871 Delny Delaunay triangulation 18000 2790 pymc Markov Chain Monte Carlo sampling toolkit. 1000000 2727 PyBUFR Pure Python library to encode and decode BUFR. 10000 2676 collective.geo.bundle Plone Maps (collective.geo) 11000 2676 dap DAP (Data Access Protocol) client and server for Python. 125000 2598 rq RQ is a simple, lightweight, library for creating background jobs, and processing them. 29000 2590 pyinterval Interval arithmetic in Python 397000 2558 StarCluster StarCluster is a utility for creating and managing computing clusters hosted on Amazon’s Elastic Compute Cloud (EC2). 2000000 2521 fisher Fast Fisher’s Exact Test 43000 2503 mathdom MathDOM - Content MathML in Python 169000 2482 img2txt superseded by asciiporn, http://pypi.python.org/pypi/asciiporn 443000 2436 DendroPy A Python library for phylogenetics and phylogenetic computing: reading, writing, simulation, processing and manipulation of phylogenetic trees (phylogenies) and characters. 6000000 2349 geolocator geolocator library: locate places and calculate distances between them 26000 2342 MyProxyClient MyProxy Client 67000 2325 PyUblas Seamless Numpy-UBlas interoperability 51000 2252 oroboros Astrology software 1000000 2228 textmining Python Text Mining Utilities 1000000 2198 scikits.talkbox Talkbox, a set of python modules for speech/signal processing 147000 2188 asciitable Extensible ASCII table reader and writer 312000 2160 scikits.samplerate A python module for high quality audio resampling 368000 2151 tabular Tabular data container and associated convenience routines in Python 52000 2114 pywcs Python wrappers to WCSLIB 2000000 2081 DeliciousAPI Unofficial Python API for retrieving data from Delicious.com 19000 2038 hachoir-regex Manipulation of regular expressions (regex) 31000 2031 Kamaelia Kamaelia - Multimedia & Server Development Kit 2000000 2007 seawater Seawater Libray for Python 2000000 1985 descartes Use geometric objects as matplotlib paths and patches 3000 1983 vectorformats geographic data serialization/deserialization library 10000 1949 PyMT A framework for making accelerated multitouch UI 18000000 1945 times Times is a small, minimalistic, Python library for dealing with time conversions between universal time and arbitrary timezones. 4000 1929 CocoPy Python implementation of the famous CoCo/R LL(k) compiler generator. 302000 1913 django-shapes Upload and export shapefiles using GeoDjango. 9000 1901 sympy Computer algebra system (CAS) in Python 5000000 1842 pyfasta fast, memory-efficient, pythonic (and command-line) access to fasta sequence files 14000 1836 ...