| Title: | Testing for R Packages with Multiple Attempts for Noisy Tests |
|---|---|
| Description: | Runs tests using the 'testthat' package but allows for multiple attempts for a single test. This is useful for noisy or flaky tests that generally pass but can fail due to occasional random errors, such as numeric instability or using random data. |
| Authors: | Collin Erickson |
| Maintainer: | Collin Erickson <[email protected]> |
| License: | GPL (>= 3) |
| Version: | 0.1.0 |
| Built: | 2024-11-10 06:15:49 UTC |
| Source: | https://github.com/collinerickson/testthatmulti |
Test that with multiple attempts
ttm(n, expr, verbose = 0)ttm(n, expr, verbose = 0)
n |
Maximum number of attempts |
expr |
Expression to evaluate |
verbose |
Amount that should be printed |
Nothing
set.seed(0) # 1 attempt, all pass ttm(1, { ttm_expect_true(TRUE) ttm_expect_true(1 == 1) ttm_expect_true(all(1:5 == 1:5)) }) # Fails first 10 times, then passes ttm(100, { x <- runif(1) print(x) ttm_expect_true(x < 0.1) }) # Will always fail regardless of number of attempts try({ ttm(3, { ttm_expect_true(1 == 2) }) })set.seed(0) # 1 attempt, all pass ttm(1, { ttm_expect_true(TRUE) ttm_expect_true(1 == 1) ttm_expect_true(all(1:5 == 1:5)) }) # Fails first 10 times, then passes ttm(100, { x <- runif(1) print(x) ttm_expect_true(x < 0.1) }) # Will always fail regardless of number of attempts try({ ttm(3, { ttm_expect_true(1 == 2) }) })
Test that multi: expect equal
ttm_expect_equal( object, expected, ..., tolerance = if (testthat::edition_get() >= 3) testthat::testthat_tolerance(), info = NULL, label = NULL, expected.label = NULL, verbose = 0 )ttm_expect_equal( object, expected, ..., tolerance = if (testthat::edition_get() >= 3) testthat::testthat_tolerance(), info = NULL, label = NULL, expected.label = NULL, verbose = 0 )
object |
Object to check if equal to expected |
expected |
Expected value |
... |
Args passed to testthat::expect_equal() |
tolerance |
Passed to 'testthat::expect_true()'. |
info |
Passed to 'testthat::expect_true()'. |
label |
Passed to 'testthat::expect_true()'. |
expected.label |
Passed to 'testthat::expect_true()'. |
verbose |
Amount of info that should be printed. |
Test result
set.seed(0) # 1 attempt, all pass ttm(1, { ttm_expect_equal(TRUE, TRUE) ttm_expect_equal(1, 1) ttm_expect_equal(1:5, 1:5) }) # Fails first 6 times, then passes ttm(100, { x <- sample(1:6, 1) print(x) ttm_expect_equal(x, 3) }) # Will always fail regardless of number of attempts try({ ttm(3, { ttm_expect_equal(1, 2) }) })set.seed(0) # 1 attempt, all pass ttm(1, { ttm_expect_equal(TRUE, TRUE) ttm_expect_equal(1, 1) ttm_expect_equal(1:5, 1:5) }) # Fails first 6 times, then passes ttm(100, { x <- sample(1:6, 1) print(x) ttm_expect_equal(x, 3) }) # Will always fail regardless of number of attempts try({ ttm(3, { ttm_expect_equal(1, 2) }) })
See 'testthat::expect_true' for details.
ttm_expect_true(object, info = NULL, label = NULL, verbose = 0)ttm_expect_true(object, info = NULL, label = NULL, verbose = 0)
object |
Object to test. |
info |
Passed to 'testthat::expect_true()'. |
label |
Passed to 'testthat::expect_true()'. |
verbose |
Amount of info that should be printed. |
Test result
set.seed(0) # 1 attempt, all pass ttm(1, { ttm_expect_true(TRUE) ttm_expect_true(1 == 1) ttm_expect_true(all(1:5 == 1:5)) }) # Fails first 10 times, then passes ttm(100, { x <- runif(1) print(x) ttm_expect_true(x < 0.1) }) # Will always fail regardless of number of attempts try({ ttm(3, { ttm_expect_true(1 == 2) }) })set.seed(0) # 1 attempt, all pass ttm(1, { ttm_expect_true(TRUE) ttm_expect_true(1 == 1) ttm_expect_true(all(1:5 == 1:5)) }) # Fails first 10 times, then passes ttm(100, { x <- runif(1) print(x) ttm_expect_true(x < 0.1) }) # Will always fail regardless of number of attempts try({ ttm(3, { ttm_expect_true(1 == 2) }) })