data-engineering on S Anand

When Data is for Agents Not Humans Workshop

Mon, 29 Jun 2026 18:00:00 +0800

For thirty years, we cleaned data for humans. Now, agents are reading it. What do we do differently?

On Tue 𝟳 Jul 𝟮𝟬𝟮𝟲, 𝟮-𝟲pm IST, I’m running a Fifth Elephant workshop: “When Data is for Agents, Not Humans”.

I have a few theories we’ll be testing on your data. Like:

Tools beat tables. Agents prefers functions over tables they can read.
Logs beat docs. Logs show actual usage - not how we’re supposed to use it.
Messy beats clean. Agents can do clean & join data. You just need descriptions.

I’m not fully sure how much of this is right. That’s the point. We’ll work it out together, on your data.

We’ll do two things.

See how one dataset looks to a human vs an agent - then rebuild it for the agent.
Use agents to rebuild the data. They’re good at it.

For: data/AI engineers, BI folks, anyone publishing schemas, docs or APIs.

Bring: a laptop, a paid ChatGPT/Claude account, and a dataset you have and like.

It’s both online and in-person. 30 seats.

I’ll share the recording.
Attend only if you want to actually do stuff live.

RIP, Data Engineers

Wed, 04 Feb 2026 19:22:13 +0800

As AI marches along, another role at risk is the data engineer / database administrator.

(Data scientists are already feeling the heat.)

A common task for data engineers is to analyze SQL queries - to optimize and standardize.

Pavan used Antigravity to analyze 1,500 SQL queries and found:

30% of queries are purely headcount / volume related. Much more than revenue (25%) or engagement (15%). That’s sign of a tactical culture.
70% of the queries are about What happened yesterday? rather than What will happen tomorrow? - again, tactical culture.

Here’s the analysis.

As a next step, he built a “Middle Layer” - intermediate tables that standardize and optimize queries. Instead of 50 fragile tables, the user can query just 3 robust tables that cover 98% of the SQL queries.

For example:

A net_revenue field that standardizes net revenue after adjustments, i.e. SUM(face_value - discount), which is used in 58% of queries. That ensures that Finance (which used to see the GAAP Revenue) and Sales (which used to see the Booked Revenue) are now aligned.
A tickets_sold field that standardizes distinct count of tickets sold, used in 85% of queries, and is a slow computation.

NOTE: Season ticket buyers often bought merchandise as guests (for convenience). Marketing saw these as new customers and spammed them - annoying VIP customers. This standardization created an identity graph - so they can offer discounts instead.

The process, which Antigravity figured out mostly by itself, was to parse the SQL into an abstract syntax tree (AST), extract a set of features, map them into clusters (archetypes), and analyze them to create the middle layer tables.

SQL queries can reveal organizational culture and misalignment - which is cool! But also:

This took a few hours.
Pavan has no data engineering experience.

RIP, Data Engineers.

Embeddings in DuckDB

Sun, 16 Jun 2024 13:06:44 +0000

This article on Using DuckDB for Embeddings and Vector Search by Sören Brunk shows a number of DuckDB features I wasn’t aware of.

DuckDB can read directly from Huggingface datasets
DuckDB can read just the parts of a .parquet file it needs, even over HTTP
DuckDB lets you write custom functions in Python
DuckDB now has a vector similarity search extension

I’ve recently become a DuckDB fan and continue to be impressed.