Grade an evaluation data frame — grade

If you set the judges argument to a non-NULL value in evaluate(), this function will be evoked automatically.

Usage

grade_queue(x, judges, type = c("pairwise", "score"))

Arguments

judges: A judges() object or NULL to grade the evals yourself.
type: One of "pairwise" or "score". See details for more.

Examples

library(ellmer)

ggplot2 <- evaluate(
  "tests/evalthat/test-ggplot2-graded.R",
    across = tibble(chat = c(
        chat_openai(model = "gpt-4o-mini", echo = FALSE),
        chat_claude(model = "claude-3-5-sonnet-latest", echo = FALSE),
        chat_ollama(model = "qwen2.5-coder:14b", echo = FALSE))
      ),
      repeats = 2
    )
#> Error in tibble(chat = c(chat_openai(model = "gpt-4o-mini", echo = FALSE),     chat_claude(model = "claude-3-5-sonnet-latest", echo = FALSE),     chat_ollama(model = "qwen2.5-coder:14b", echo = FALSE))): could not find function "tibble"

ggplot2
#> Error: object 'ggplot2' not found


ggplot2_graded
#> Error: object 'ggplot2_graded' not found