Test reporters for LLM evaluation

evalthat provides a number of custom testthat testthat::Reporters. These reporters process the results from test files, generating interactive summaries and saving results to persistent files.

EvalProgressReporter is designed for interactive use. Its goal is to give you actionable insights to help you understand the status of your code. This reporter also praises you from time-to-time if all your tests pass. It's the default reporter for test_dir().

EvalCompactProgressReporter is a minimal version of EvalProgressReporter designed for use with single files. It's the default reporter for test_file().

Super classes

testthat::Reporter -> testthat::ProgressReporter -> EvalProgressReporter

Public fields

show_praise

Whether to show praise. Will not be shown, even if TRUE.

res_ok, res_skip, res_warn, res_fail

The same as fields prefixed with n_* in the superclass, but encoded a vector whose sum can be taken to find the n_* values. Enables weights per-expectation.

ctxt_res_ok, ctxt_res_skip, ctxt_res_warn, ctxt_res_fail, ctxt_issues, ctxt_n

The same as those without the ctxt_* prefix, but per-context.

io

The inputs and outputs flagged with input() and output().

Active bindings

res_ok, res_skip, res_warn, res_fail

The same as fields prefixed with n_* in the superclass, but encoded a vector whose sum can be taken to find the n_* values. Enables weights per-expectation.

ctxt_res_ok, ctxt_res_skip, ctxt_res_warn, ctxt_res_fail, ctxt_issues, ctxt_n

The same as those without the ctxt_* prefix, but per-context.

Methods

Public methods

• EvalProgressReporter$is_full()

• EvalProgressReporter$update_io()

• EvalProgressReporter$start_file()

• EvalProgressReporter$update_counts()

• EvalProgressReporter$reset_counts()

• EvalProgressReporter$start_context()

• EvalProgressReporter$show_header()

• EvalProgressReporter$show_status()

• EvalProgressReporter$end_context()

• EvalProgressReporter$save_results()

• EvalProgressReporter$result_summary()

• EvalProgressReporter$add_result()

• EvalProgressReporter$cache_for_grading()

• EvalProgressReporter$start_reporter()

• EvalProgressReporter$end_reporter()

• EvalProgressReporter$end_file()

• EvalProgressReporter$clone()

Inherited methods

• testthat::Reporter$.start_context()
• testthat::Reporter$cat_line()
• testthat::Reporter$cat_tight()
• testthat::Reporter$end_context_if_started()
• testthat::Reporter$end_test()
• testthat::Reporter$local_user_output()
• testthat::Reporter$rule()
• testthat::Reporter$start_test()
• testthat::Reporter$update()
• testthat::ProgressReporter$cr()
• testthat::ProgressReporter$initialize()
• testthat::ProgressReporter$report_issues()
• testthat::ProgressReporter$should_update()
• testthat::ProgressReporter$status_data()

---

Method is_full()

Method overwritting to always return FALSE, as failed evals are not a reason to stop testing in our context.

Usage

EvalProgressReporter$is_full()

---

Method update_io()

Concatenated inputs and outputs flagged with input() and output().

Concatenate inputs and outputs flagged with input() and output().

Usage

EvalProgressReporter$update_io(x, test, type)

Arguments

x

The input or output.

test

The name of the test chunk.

type

One of "input" or "output".

---

Method start_file()

Setup for starting a new test file.

Usage

EvalProgressReporter$start_file(file)

Arguments

file

File name.

---

Method update_counts()

Eval reporters' n_* fields are actually tracked by maintaining vectors of 1s prefixed with res_*. This is to accommodate a future extension where weights can be assigned to each pass/fail.

This function updates the n_* values based on the res_* values.

Usage

EvalProgressReporter$update_counts()

---

Method reset_counts()

Sets the values of res_* fields to numeric() and calls update_counts().

Usage

EvalProgressReporter$reset_counts()

---

Method start_context()

Resets counters and initiates a progress bar.

Usage

EvalProgressReporter$start_context(context)

Arguments

context

Arguments supplied to the evaluation function, as a named list.

---

Method show_header()

Show the header tabulating successes and failures.

Usage

EvalProgressReporter$show_header()

---

Method show_status()

Tabulate successes and failures in the current context.

Usage

EvalProgressReporter$show_status(complete = FALSE, time = 0, pad = FALSE)

Arguments

complete

Logical.

time

Ignored. TODO: remove this?

pad

Logical.

---

Method end_context()

Teardown following the test run.

Usage

EvalProgressReporter$end_context()

---

Method save_results()

Situates the evaluation results in a tibble and saves it to the file eval_file_name/timestamp.rds using qs::qread(). Read individual results with qs::qsave(). Situate evaluation results in a tibble.

Usage

EvalProgressReporter$save_results()

---

Method result_summary()

Usage

EvalProgressReporter$result_summary(timestamp)

Arguments

timestamp

DTTM as format(Sys.time(), "%Y%m%d_%H%M%S").

---

Method add_result()

Append a given expect_*() or grade_*() result to the current context.

Usage

EvalProgressReporter$add_result(context, test, result)

Arguments

context

Context from evaluating().

test

The name of the test block.

result

Result.

---

Method cache_for_grading()

Usage

EvalProgressReporter$cache_for_grading(target)

---

Method start_reporter()

Start the current reporter.

Usage

EvalProgressReporter$start_reporter()

---

Method end_reporter()

Tear down the current reporter.

Usage

EvalProgressReporter$end_reporter()

---

Method end_file()

Overrides the superclass' method to end the context when the file ends since files are reran when running evals.

Usage

EvalProgressReporter$end_file()

---

Method clone()

The objects of this class are cloneable with this method.

Usage

EvalProgressReporter$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.

Super classes

testthat::Reporter -> testthat::ProgressReporter -> evalthat::EvalProgressReporter -> EvalCompactProgressReporter

Methods

Public methods

• EvalCompactProgressReporter$new()

• EvalCompactProgressReporter$start_file()

• EvalCompactProgressReporter$start_reporter()

• EvalCompactProgressReporter$end_reporter()

• EvalCompactProgressReporter$show_status()

• EvalCompactProgressReporter$clone()

Inherited methods

• testthat::Reporter$.start_context()
• testthat::Reporter$cat_line()
• testthat::Reporter$cat_tight()
• testthat::Reporter$end_context_if_started()
• testthat::Reporter$end_test()
• testthat::Reporter$local_user_output()
• testthat::Reporter$rule()
• testthat::Reporter$start_test()
• testthat::Reporter$update()
• testthat::ProgressReporter$cr()
• testthat::ProgressReporter$report_issues()
• testthat::ProgressReporter$should_update()
• testthat::ProgressReporter$status_data()
• evalthat::EvalProgressReporter$add_result()
• evalthat::EvalProgressReporter$cache_for_grading()
• evalthat::EvalProgressReporter$end_context()
• evalthat::EvalProgressReporter$end_file()
• evalthat::EvalProgressReporter$is_full()
• evalthat::EvalProgressReporter$reset_counts()
• evalthat::EvalProgressReporter$result_summary()
• evalthat::EvalProgressReporter$save_results()
• evalthat::EvalProgressReporter$show_header()
• evalthat::EvalProgressReporter$start_context()
• evalthat::EvalProgressReporter$update_counts()
• evalthat::EvalProgressReporter$update_io()

---

Method new()

Sets minimum time to infinity.

Usage

EvalCompactProgressReporter$new(min_time = Inf, ...)

Arguments

min_time

A numeric. Defaults to Inf.

...

Passed on to super$initialize().

---

Method start_file()

Setup for a single file.

Usage

EvalCompactProgressReporter$start_file(name)

Arguments

name

File name.

---

Method start_reporter()

Setup.

Usage

EvalCompactProgressReporter$start_reporter()

---

Method end_reporter()

Teardown.

Usage

EvalCompactProgressReporter$end_reporter()

---

Method show_status()

Show current status.

Usage

EvalCompactProgressReporter$show_status(complete = NULL, time = NULL)

Arguments

complete

Ignored.

time

Ignored–included for compatibility with EvalProgressReporter.

---

Method clone()

The objects of this class are cloneable with this method.

Usage

EvalCompactProgressReporter$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.