It’s (still) very bad to be wrong

Agents for Correct, Transparent, and Reproducible Data Analysis

Sara Altman & Simon Couch AI Core Team @ Posit

Please plot hp vs mpg in mtcars

Calls tool: Run R code

[Plot]

There is a strong, negative association.

Please plot hp vs mpg in mtcars

Calls tool: Run R code

There is a strong, negative association.

❓

🤫

mtcars$hp <- max(mtcars$hp) - mtcars$hp

Please plot hp vs mpg in mtcars

Calls tool: Run R code

There is a strong, negative association.

Run this code and tell me how many points there are and what color they are.

plot(runif(sample(3:10, 1)), 
     col = rgb(runif(1), runif(1), runif(1)),
     pch = 16, cex = 2)

plot(
  runif(sample(3:10, 1)),
  col = rgb(runif(1), runif(1), runif(1)),
  pch = 16,
  cex = 2
)

Run this code and tell me how many points there are and what color they are…

Calls tool: Run R code

There are 3 cyan points.

LLMs can ‘see’ plots just fine.

👽

Please plot hp vs mpg in mtcars

Calls tool: Run R code

A scatterplot of around 30 points showing a strong, positive association between x and y.

There is a strong, negative association.

⟵

Please describe this plot.

👽👽👽

Please plot hp vs mpg in mtcars

Calls tool: Run R code

This is a scatterplot of around 30 points showing a strong, positive association.

There is a strong, negative association between hp and mpg.

⟵

Please describe this plot.

👾👾👾

Please plot hp vs mpg in mtcars

Calls tool: Run R code

<thought> Huh, that’s weird. The association is positive. </thought>

There is a strong, negative association.

👽🛸👾

Agents enact a performance of progress.

👽 👾 🛸

Correctness matters in data analysis

but do LLMs care about correctness?

so why is this a problem? when you’re working with real data with real world consequences, being correct matters. correctness, fittingly, is really a core value of data science when you do an analysis, you want your results to actually line up with the real world, or at least approximate it. That’s the whole point. But LLM behavior doesn’t necessarily always line up with this value of correctness

It’s a convincing performance

but one that occasionally fails to match reality

but it’s a convincing performance right we’ve seen in bluffbench that the model will interpret the plot, and will provide details and supposed evidence for what they see they act like they are moving the analysis forward the only problem is that, like in this case where it says this clearly positive trend is negative, is that it’s one that can fail to match reality

The answers look the same

❌

✅

❌

The problem isn’t just that the model is sometimes wrong, it’s that the wrong answers can be indistinguishable from the right ones

we have a clearly parabolic plot and three descriptions from the same model shared tone, shared style, shared length – but different description each time if you didn’t know waht the plot looked like, you wouldn’t be able to disentangel the correct ones from the incorrect ones

Correctness matters in data analysis

and so do reproducibility and transparency

…all often at odds with how LLMs behave

we already mentioned how “correctness matters”. reproducibility – get consistent results from same conditions – and transparency – insight into how an analysis was done – are also central to data science, all often at odds with the nondetermistic and black box nature of LLMs.

What does this mean for agents for data analysis?

1. LLMs are aliens.

2. LLMs are still useful for data analysis.

don’t need to throw the whole thing away

We shouldn’t take evidence like bluffbench to mean that LLMs are useless anywhere you value correctness.

Instead, you have to figure out where they can fit and how to design around their limitations.

we’re going to talk about what we did…

LLMs are still useful for data analysis

1. Tell them to care about being right

2. Design the scaffold so that it matters less if they’re wrong

We’ll be talking about agents

• Can gather information from the world (e.g., read files)
• Can alter the world (e.g., run code)

Posit Assistant

define agent we are going to use Posit Assistant

LLMs are still useful for data analysis

1. Tell them to care about being right

2. Design the scaffold so that it matters less if they’re wrong

An agent is more than just the model

It comes wrapped in a harness:

• How it’s told to act (prompting)
• What tools it has access to use (run code, read files, etc.)
• What information it has access to
• and more

there’s more to an agent than just the model comes wrapped in a harness [list] Unless your building or fine tuning an LLM yourself, you don’t have much control over the underlying LLM You do have control over the harness

Tell the model to care about being right

The harness can make a difference in correctness

for correctness define harness

LLMs are still useful for data analysis

1. Tell them to care about being right
2. Design the scaffold so that it matters less if
they’re wrong

• Have the model write code, in a shared environment, and show it to the user
• Have the model pause often

Code as the foundation

• Models are good at writing code
• Code can be run again - reproducibility
• Code is auditable - transparency

for correctness, reproducibility, transparency

Models are good at writing code Code is reproducible and auditable by the human

Code as the foundation

Makes auditing easy when it matters

include that the code is easy to read

Shared environment

See eye-to-eye with the agent

Have the model pause often

Keep pace with the agent during exploration and analysis

Have the model pause often

What’s the point of an analysis?

(often) building an understanding of the data

for verification and transparency: the human has to keep pace and see eye-to-eye with the agent (same environment).

LLMs are aliens

• Can make strange errors
• Can be more concerned with performing progress than correctness

LLMs are still useful for data analysis

• Tell them to care about being right
• Design the scaffold so that it matters less if they are wrong

https://opensource.posit.co/tags/ai-newsletter/

thank you 🛸