Skip to contents

The testthat pal

Source: R/doc-pal-testthat.R
pal_testthat.Rd

testthat 3.0.0 was released in 2020, bringing with it numerous changes that were both huge quality of life improvements for package developers and also highly breaking changes.

While some of the task of converting legacy unit testing code to testthat 3e is quite is pretty straightforward, other components can be quite tedious. The testthat pal helps you transition your R package's unit tests to the third edition of testthat, namely via:

  • Converting to snapshot tests

  • Disentangling nested expectations

  • Transitioning from deprecated functions like expect_known_*()

Cost

The system prompt from a testthat pal includes something like 1,000 tokens. Add in (a generous) 100 tokens for the code that's actually highlighted and also sent off to the model and you're looking at 1,100 input tokens. The model returns approximately the same number of output tokens as it receives, so we'll call that 100 output tokens per refactor.

As of the time of writing (October 2024), the default pal model Claude Sonnet 3.5 costs $3 per million input tokens and $15 per million output tokens. So, using the default model, testthat pals cost around $4 for every 1,000 refactored pieces of code. GPT-4o Mini, by contrast, doesn't tend to get many pieces of formatting right and often fails to line-break properly, but does usually return syntactically valid calls to testthat functions, and it would cost around 20 cents per 1,000 refactored pieces of code.

This section includes a handful of examples "from the wild" and are generated with the default model, Claude Sonnet 3.5.

Testthat pals convert expect_error() (and *_warning() and *_message() and *_condition()) calls to use expect_snapshot() when there's a regular expression present:

expect_warning(
  check_ellipses("exponentiate", "tidy", "boop", exponentiate = TRUE, quick = FALSE),
  "\\`exponentiate\\` argument is not supported in the \\`tidy\\(\\)\\` method for \\`boop\\` objects"
)

Returns:

expect_snapshot(
  .res <- check_ellipses(
    "exponentiate", "tidy", "boop", exponentiate = TRUE, quick = FALSE
  )
)

Note, as well, that intermediate results are assigned to an object so as not to be snapshotted when their contents weren't previously tests.

Another example with multiple, redudant calls:

augment_error <- "augment is only supported for fixest models estimated with feols, feglm, or femlm"
expect_error(augment(res_fenegbin, df), augment_error)
expect_error(augment(res_feNmlm, df), augment_error)
expect_error(augment(res_fepois, df), augment_error)

Returns:

expect_snapshot(error = TRUE, augment(res_fenegbin, df))
expect_snapshot(error = TRUE, augment(res_feNmlm, df))
expect_snapshot(error = TRUE, augment(res_fepois, df))

They know about regexp = NA, which means "no error" (or warning, or message):

expect_error(
  p4_b <- check_parameters(w4, p4_a, data = mtcars),
  regex = NA
)

Returns:

expect_no_error(p4_b <- check_parameters(w4, p4_a, data = mtcars))

They also know not to adjust calls to those condition expectations when there's a class argument present (which usually means that one is testing a condition from another package, which should be able to change the wording of the message without consequence):

expect_error(tidy(pca, matrix = "u"), class = "pca_error")

Returns:

expect_error(tidy(pca, matrix = "u"), class = "pca_error")

When converting non-erroring code, testthat pals will assign intermediate results so as not to snapshot both the result and the warning:

expect_warning(
  tidy(fit, robust = TRUE),
  '"robust" argument has been deprecated'
)

Returns:

expect_snapshot(
  .res <- tidy(fit, robust = TRUE)
)

Nested expectations can generally be disentangled without issue:

expect_equal(
  fit_resamples(decision_tree(cost_complexity = 1), bootstraps(mtcars)),
  expect_warning(tune_grid(decision_tree(cost_complexity = 1), bootstraps(mtcars)))
)

Returns:

expect_snapshot({
  fit_resamples_result <- fit_resamples(decision_tree(cost_complexity = 1),
                                        bootstraps(mtcars))
  tune_grid_result <- tune_grid(decision_tree(cost_complexity = 1),
                                bootstraps(mtcars))
})
expect_equal(fit_resamples_result, tune_grid_result)

There are also a few edits the pal knows to make to third-edition code. For example, it transitions expect_snapshot_error() and friends to use expect_snapshot(error = TRUE) so that the error context is snapshotted in addition to the message itself:

expect_snapshot_error(
  fit_best(knn_pca_res, parameters = tibble(neighbors = 2))
)

Returns:

expect_snapshot(
  error = TRUE,
  fit_best(knn_pca_res, parameters = tibble(neighbors = 2))
)

Interfacing manually with the testthat pal

Pals are typically interfaced with via the pal addin. To call the testthat pal directly, use:

pal_testthat <- .init_pal("testthat")

Then, to submit a query, run:

pal_testthat$chat({expr})