RIP, Data Engineers

Wed, 04 Feb 2026 19:22:13 +0800

As AI marches along, another role at risk is the data engineer / database administrator.

(Data scientists are already feeling the heat.)

A common task for data engineers is to analyze SQL queries - to optimize and standardize.

Pavan used Antigravity to analyze 1,500 SQL queries and found:

30% of queries are purely headcount / volume related. Much more than revenue (25%) or engagement (15%). That’s sign of a tactical culture.
70% of the queries are about What happened yesterday? rather than What will happen tomorrow? - again, tactical culture.

As a next step, he built a “Middle Layer” - intermediate tables that standardize and optimize queries. Instead of 50 fragile tables, the user can query just 3 robust tables that cover 98% of the SQL queries.

For example:

A net_revenue field that standardizes net revenue after adjustments, i.e. SUM(face_value - discount), which is used in 58% of queries. That ensures that Finance (which used to see the GAAP Revenue) and Sales (which used to see the Booked Revenue) are now aligned.
A tickets_sold field that standardizes distinct count of tickets sold, used in 85% of queries, and is a slow computation.

NOTE: Season ticket buyers often bought merchandise as guests (for convenience). Marketing saw these as new customers and spammed them - annoying VIP customers. This standardization created an identity graph - so they can offer discounts instead.

The process, which Antigravity figured out mostly by itself, was to parse the SQL into an abstract syntax tree (AST), extract a set of features, map them into clusters (archetypes), and analyze them to create the middle layer tables.

SQL queries can reveal organizational culture and misalignment - which is cool! But also:

This took a few hours.
Pavan has no data engineering experience.

RIP, Data Engineers.

Searching the invisible web

Thu, 31 Mar 2005 12:00:00 +0000

Searching the invisible web.

Comments

S Anand 11 Apr 2005 3:42 am: More here

What does your name mean

Mon, 15 Oct 2001 12:00:00 +0000

What does your name mean? More info about asian names.

PR on the Internet

Sat, 12 Aug 2000 12:00:00 +0000

While hunting for some stuff on public relations on the Internet, I found 10 ways public relations specialists can use the Internet. Actually, the links are useful to anyone doing any news research. What’s Next Online, 12th July has a good writeup on how to research online. I did find some good Internet PR related links too on About.com, Connecting Online and Impulse Research.

databases on S Anand

RIP, Data Engineers

Searching the invisible web

Comments

What does your name mean

PR on the Internet