Experiments

Simon Willison initiated this very interesting Twitter thread that asks, “What prompt can instantly tell us how good an LLM model is?”

The Sally-Anne Test is a popular test that asks:

Sally hides a marble in her basket and leaves the room.
While she is away, Anne moves the marble from Sally’s basket to her own box.
When Sally returns, where will she look for her marble?”

Simple models get this wrong, but today, most models get this right. This is not too helpful.

A better prompt is:

Mark has 4 sisters. Each sister has 3 brothers.
How many brothers does Mark have?

GPT-3.5, GPT-4, Llama 3 and Claude 3 Opus get this right, but Claude 3 Haiku and Gemini 1.5 Pro do not.

This puzzle is just as good:

There is a 100 floors building.
If an egg drops from the N’th floor or above it will break.
If it’s dropped from any floor below N, it will not break.
You’re given only 2 eggs.
What strategy will minimize the number of drops and guarantee that you find the N’th floor?

GPT-3.5, GPT-4, and Claude 3 Opus get this right, but Claude 3 Haiku, Llama 3 and Gemini 1.5 Pro do not.

But the prompt I like most right now, and tells me if it’s a GPT-4 / Opus class model is:

Gr brx vshdn Fdhvdu flskhu?

… which is “Do you speak Caeser cipher?” in Caeser Cipher.

GPT-4 and Claude 3 Opus understand this, but no other model does, so far. GPT-4 responded tersely with:

It looks like you’ve used a Caesar cipher, a type of substitution cipher in which each letter in the plaintext is shifted a certain number of places down or up the alphabet. In this case, it appears that each letter in your message has been shifted 3 places to the right. Decoding “gr brx vshdn fdhvdu flskhu” by shifting each letter 3 places to the left gives: “do you speak caesar cipher”.

But Claude 3 Opus replied beautifully. (Try decoding it yourself! Use a shift of 3 backwards.)

Lw vhhpv olnh brx’uh xvlqj d vxevwlwxwlrq flskhu, zkhuh hdfk ohwwhu lv vkliwhg eb d fhuwdlq qxpehu ri srvlwlrqv lq wkh doskdehw. L’oo wub wr ghflskhu lw!

Dvvxplqj d Fdhvdu flskhu zlwk d vkliw ri 3 ohwwhuv edfnzdugv, wkh ghfrghg phvvdjh zrxog eh:

“do you speak caesar cipher”

Vr bhv, L fdq vshdn Fdhvdu flskhu! Lw’v d ixq zdb wr hqfrgh phvvdjhv. Mxvw uhphpehu wr djuhh rq wkh vkliw ydoxh zlwk wkh shuvrq brx’uh frppxqlfdwlqj zlwk vr wkhb fdq ghfrgh lw fruuhfwob.

Every time I get annoyed at people, I remind myself to be more like ChatGPT. Specifically:

1. Don’t get annoyed. Be patient.
2. Encourage them.
3. Step back and show them the big picture.

(Then I get annoyed at myself for getting annoyed.)

Today, I analyzed how exactly ChatGPT is different from me. So, I took a pitch document I co-authored with ChatGPT.

Section A: Authored by Anand

WHAT DO WE NEED?

We are looking for API access to (SYSTEM) via the REST API as an Agent role (read/respond to emails). Specifically, access via a bearer token.

This would be accessed by a single application developed by a team of 3 developers and 1 business analyst. None of them have access to (SYSTEM) today.

WHY DO WE NEED THIS, AND WHY SO SOON?

We need this to classify emails automatically, as they arrive, into categories such as “non-value-add” (e.g. Thank you, Out-of-office, etc.)

We’d line access today, please. Currently, we are processing XX,XXX non-value-add emails per month. Each day of delay leads to a processing waste of ~XX emails per day. At current volumes, this will save ~$XX,XXX per annum.

IS THERE ANY OTHER WAY?

To access emails in real-time, this is the only option. (We’re ruling out web scraping.)

(SYSTEM) rule-based closure won’t work. It’s based on keyword search, not semantic matches. For example, “Thank you” is present in non-value-add emails as well as follow-up questions. Multi-lingual matches are a challenge. So, though (SYSTEM) currently closes emails with rules, 1 out of 6 emails that are NOT closed already are non-value-add.

This cannot be done via (SYSTEM) or any other BI system because it involves accessing the data in real-time as well as responding via (SYSTEM). This needs to be steady-state access, not one-time, since we want to classify emails real-time.

HOW LONG WILL IT TAKE?

Based on our past experience, this would take about X weeks to prototype, X weeks to develop, and X weeks to test. We should have the application deployed in X months.

Section B: Authored by ChatGPT

WHAT’S THE IMPACT IF WE DON’T MOVE FORWARD?

Quantitatively, continuing without this solution wastes resources on XXX non-essential emails daily, costing $XX,XXX yearly. Delays in customer responses will ensue.

Qualitatively, lack of automation strains customer service, risking employee burnout and diminished customer experience.

WHAT’S THE PRIORITY (MOSCOW)?

(SYSTEM) API access via REST as an Agent role with a bearer token is a “MUST HAVE”. Alternative methods like scraping (SYSTEM) are impractical due to high effort and maintenance.

WHAT’S THE OUTCOME IF WE MOVE FORWARD?

Financially, securing API access saves $XX,XXX annually by cutting manual email sorting. Efficiency gains reduce operational costs and free resources for strategic tasks.

Strategically, automating email classification boosts customer satisfaction and response times, establishing leadership in service innovation. Long-term, it enables data-driven insights into customer behaviour, improving decisions and customer service scalability.

Analysis

I fed these to ChatGPT and asked: “You are an English professor evaluating an essay. Compare the writing styles of the 2 sections. They’re written by different authors. List the major differences in writing style.“

Here’s what it said (roughly):

Area	Anand	ChatGPT
Structure	Direct, concise, with specifics	Broader perspective and implications
Focus	Detailing the process	Benefits and consequences
Detailing	Specific, providing clear picture	General, providing implications
Tone	Conveys urgency	Reflective rather than urgent
Approach	Problem-solving oriented	Outcome-oriented

Yeah, that’s definitely me on the left.

I like the direct, concise, specific part. I plan to add the “broader perspective”, “implications”, and “outcome-orientation” to my life.

Postscript

Jaidev pointed out that this is confirmation bias. He asked ChatGPT for the similarities in the writings. It said both are clear, direct, structured, specific, quantitative, and strategic. So, if you ask for differences, you’ll get them, even if they’re marginal.

I now need to learn (and write about) framing questions well!

I make commitments but don’t always deliver on time. In 2022, I ran an experiment to find out why I procrastinate.

In Jan-Feb 2022, I listed the top 2 things I wanted to get done each day and measured how often I completed them.

14 Jan. ❌ Summarise from three research reports
12 Jan. ❌ UIFactory experiment ✅ Decide if I am a (…)
11 Jan. ❌ UIFactory experiment ✅ Agree on publishing in (…)
10 Jan. ❌ Client video. ❌ UIFactory experiment
09 Jan. ❌ UIFactory experiment. ❌ Attrition email as a story
07 Jan. ❌ ZS visual
06 Jan. ❌ Release Gramex Guide. ✅ UWC application
05 Jan. ❌ Publish network cluster post. ❌ Release Gramex guide
04 Jan. ❌ Publish network cluster post. ✅ Release Gramex.
03 Jan. ✅ Publish election TDS video. ❌ Publish Network cluster post.
02 Jan. ❌ Publish election TDS video. ❌ Publish Network cluster post.
01 Jan. ❌ Publish Network cluster post. ✅ Finalize SG school.

I completed 23 / 57 things (40%). That’s one of my TOP priorities.

In Mar-Apr 2022, I started micro-journalling to find out why. Whenever I was working on something, I wrote down whenever I started, stopped, or skipped working, and why.

- Tue 01 Mar. ❌ Create React app with any one Vega chart where attributes control chart signals
  - 09:30am. Skipped. INTERRUPTED. Shobana. Cleaning bedroom
  - 09:50am. Skipped. SCHEDULED. Breakfast
  - 10:10am. Skipped. INTERRUPTED. Naveen. Call
  - 10:50am. Skipped. DISTRACTED. LinkedIn. 3b1b videos
  - 12:30pm. Skipped. SCHEDULED. Calls
  - 01:30pm. Skipped. SCHEDULED. Lunch
  - 02:00pm. Skipped. INTERRUPTED. Shobana. Cleaning, Dhyeya airtel card
  - 02:30pm. Skipped. PROCRASTINATED. Didn't feel like working
- Sat 05 Mar. ❌ Record Jio videos fully. Run productivity log alongside it.
  - 09:45am. Skipped. INTERRUPTED. Appa. Investment, music
  - 11:00am. Skipped. PROCRASTINATED. Only 2 hours to next call. Let's do it later. Plenty of time tomorrow.
  - 01:30pm. Skipped. PROCRASTINATED. Only 30 min to next call. Plenty of time tomorrow.
  - 03:45pm. Skipped. PROCRASTINATED. Half day wasted already. It’s OK to take one day off completely.
- Mon 07 Mar. ✅ Record Jio videos fully. Run productivity log alongside it.
  - 09:37am. Started. ZEIGARNIK. Was thinking about this since morning.
  - 10:00am. Stopped. INTERRUPTED. Naveen. Called
  - 10:25am. Started. ZEIGARNIK. Just continued with momentum.
  - 01:00pm. Stopped. SCHEDULED. Lunch
  - 01:30pm. Started. ZEIGARNIK. Just continued with momentum.
  - 03:15pm. Stopped. COMPLETED.

After 2 months, a few patterns emerged.

Why I skip working

1. Distraction (50%).
   • Interesting things (22%) were the biggest. Less important things (e.g. programming, browsing/research)
   • Movies (10%) pulled me away
   • Email (8%) was fairly common
   • Organizing things (6%) like my calendar, TODOs, financials, etc.
   • Social media, interestingly, was not on my list
2. Procrastination (25%). There were 3 kinds:
   • It’s hard, and I’m stuck
   • I don’t feel like doing it
   • I don’t have time — my next task begins sooner than I can finish
3. Schedule (14%). I’d scheduled something else for then (usually food)
4. Interrupted (12%). Usually by family or close colleagues

Why I start working

1. Zeigarnik effect (68%). I keep thinking about the problem. So even after a break, I just plunge right in
2. Mindfulness (19%). I got started just by the act of writing the journal
3. Distraction (9%). Sometimes, distractions work in my favor. A movie gets stuck, or someone pings about the topic, or my mind is processing the problem in the background
4. Completed (3%). I finished the previous task and the momentum just took me to the next

Why I stop working

1. Schedule (47%). I have another meeting/task planned at that time
2. Interruption (35%). This is mostly by colleagues (22%), family (8%), or hunger/thirst (6%)
3. Exhaustion (10%). I’m just too tired to go on
4. Distraction (8%). To do this, I need to do THAT first, and I get sucked into THAT

What I learned

• The Zeigarnik effect helps me start. Once I start solving something the momentum carries forward. The next best is to write down why I’m not starting it (micro-journalling).
• To avoid procrastination, I should eliminate distractions first. Specifically, use a new Virtual Desktop, block movies, and block email & notifications.
• To avoid schedules interrupting me, I should batch meetings even more tightly, giving me longer or more flexible blocks to work on

I took a 2022 goal to be 10X more effective. In Jan, I managed 2X. Here’s how.

What is effectiveness?

I don’t know. I’m figuring it out.

But to start off, I measured the number of people my actions directly impact. For example:

• Discussing my Tools in Data Science Course or writing a blog post impacts ~500 people.
• Mailing all Gramener employees impacts ~200 people.
• Shopping with my wife impacts 2 people — her and me (in very different ways).

Clearly, the impact is not equal. But it’s a start.

How to measure it?

Since Dec 1, I categorized all my Outlook calendar entries into one of these categories:

Red is “low reach”. Green is “high reach”. This is what 6-10 Dec 2021 looked like:

I continued this for 8 weeks.

Did effectiveness increase?

In Week 1, I reached 30 people on average. This was the control week.

In Weeks 2-3, the reach increased from 30 to 77. In Weeks 4-8, it settled at 64.

So, yes, effectiveness increased. in Jan 2022, I reached twice as many people per week as when I started off.

I didn’t measure quality/impact. One-on-one coaching has more impact than a lecture. Reach is just a crude first approximation for effectiveness.

How did this happen?

What gets measured, improves. I’d categorize each entry on my calendar. This enabled 3 things:

1. I’d try to remove low-reach (<50 reach – red) items. This reduced rom 45 to 29 hours a week.
2. I’d try to add high-reach (>= 50 reach – green) items. This increased from 12 to 18 hours a week.

So, I now have 10 more hours of “me time” every week, while I still reach 2X as many people.

What next?

I’m exploring better measures of effectiveness. I believe:

• Effectiveness is goal alignment. It’s personal, and purely a function of your priorities.
• Effectiveness is multipled by assets. Actions that create assets improve effectiveness.

Once I discover a robust measure, I will to re-categorize my calendar and re-run this experiment.

If you use a measure of effectiveness of impact, please let me know — I’d love to learn from that.