data-engineering 3

For thirty years, we cleaned data for humans. Now, agents are reading it. What do we do differently? On Tue 𝟳 Jul 𝟮𝟬𝟮𝟲, 𝟮-𝟲pm IST, I’m running a Fifth Elephant workshop: “When Data is for Agents, Not Humans”. I have a few theories we’ll be testing on your data. Like: Tools beat tables. Agents prefers functions over tables they can read. Logs beat docs. Logs show actual usage - not how we’re supposed to use it. Messy beats clean. Agents can do clean & join data. You just need descriptions. ...

As AI marches along, another role at risk is the data engineer / database administrator. (Data scientists are already feeling the heat.) A common task for data engineers is to analyze SQL queries - to optimize and standardize. Pavan used Antigravity to analyze 1,500 SQL queries and found: 30% of queries are purely headcount / volume related. Much more than revenue (25%) or engagement (15%). That’s sign of a tactical culture. 70% of the queries are about What happened yesterday? rather than What will happen tomorrow? - again, tactical culture. Here’s the analysis. ...

This article on Using DuckDB for Embeddings and Vector Search by Sören Brunk shows a number of DuckDB features I wasn’t aware of. DuckDB can read directly from Huggingface datasets DuckDB can read just the parts of a .parquet file it needs, even over HTTP DuckDB lets you write custom functions in Python DuckDB now has a vector similarity search extension I’ve recently become a DuckDB fan and continue to be impressed.