Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Time profiling scores #350

Closed
bblodfon opened this issue Jan 22, 2024 · 3 comments
Closed

Time profiling scores #350

bblodfon opened this issue Jan 22, 2024 · 3 comments
Assignees
@bblodfon

Comments

@bblodfon
Copy link
Collaborator

Based on a recent benchmark, investigate/profile timings for scores. Results showed that:

  1. Graf improper > graf proper
  2. Score-ERV > score
  3. Caliba and logloss are distr6-based so are a real bottleneck
@bblodfon bblodfon self-assigned this Jan 22, 2024
@jemus42
Copy link
Member

jemus42 commented Jan 23, 2024 •
edited
Loading

Here's my small benchmark.
Selected measures are a superset of those we use in the survival benchmark for evaluation, I'm including e.g. proper rgaf score with both ERV = TRUE and FALSE here for the sake of comparison.

library(mlr3)
library(mlr3proba)
library(mlr3benchmark)
library(mlr3extralearners)

measures = list(
msr("surv.cindex",      id = "harrell_c",                                 label = "Harrell's C"),
msr("surv.cindex",      id = "uno_c",                 weight_meth = "G2", label = "Uno's C"),

msr("surv.rcll",        id = "rcll",                  ERV = FALSE,        label = "Right-Censored Log Loss"),
msr("surv.rcll",        id = "rcll_erv",              ERV = TRUE,         label = "Right-Censored Log Loss (ERV)"),

msr("surv.logloss",     id = "logloss",               ERV = FALSE,        label = "Log Loss"),
msr("surv.logloss",     id = "logloss_erv",           ERV = TRUE,         label = "Log Loss (ERV)"),

msr("surv.intlogloss",  id = "intlogloss_proper",     ERV = FALSE, proper = TRUE, label = "Integrated Log Loss (Proper)"),
msr("surv.intlogloss",  id = "intlogloss_proper_erv", ERV = TRUE,  proper = TRUE, label = "Integrated Log Loss (Proper, ERV)"),

msr("surv.calib_alpha", id = "caliba",                label = "Van Houwelingen's Alpha"),
msr("surv.dcalib",      id = "dcalib", truncate = 10, label = "D-Calibration (truncated)"),

msr("surv.graf",        id = "graf_proper",       proper = TRUE,   ERV = FALSE, label = "Graf Score (Proper)"),
msr("surv.graf",        id = "graf_proper_erv",   proper = TRUE,   ERV = TRUE,  label = "Graf Score (Proper, ERV)"),
msr("surv.graf",        id = "graf_improper",     proper = FALSE,  ERV = FALSE, label = "Graf Score (Improper)"),
msr("surv.graf",        id = "graf_improper_erv", proper = FALSE,  ERV = TRUE,  label = "Graf Score (Improper, ERV)")
)
names(measures) = mlr3misc::ids(measures)

# Example bmr
bmr = benchmark(benchmark_grid(
  tasks = tsks(c("rats", "gbcs", "grace")),
  learners = lrns(c("surv.ranger", "surv.coxph")),
  resamplings = rsmp("cv", folds = 3)
), store_backends = TRUE)
#> INFO  [10:43:04.581] [mlr3] Running benchmark with 18 resampling iterations
#> INFO  [10:43:04.614] [mlr3] Applying learner 'surv.ranger' on task 'rats' (iter 1/3)
#> INFO  [10:43:04.722] [mlr3] Applying learner 'surv.ranger' on task 'rats' (iter 2/3)
#> INFO  [10:43:04.801] [mlr3] Applying learner 'surv.ranger' on task 'rats' (iter 3/3)
#> INFO  [10:43:04.876] [mlr3] Applying learner 'surv.coxph' on task 'rats' (iter 1/3)
#> INFO  [10:43:04.922] [mlr3] Applying learner 'surv.coxph' on task 'rats' (iter 2/3)
#> INFO  [10:43:04.930] [mlr3] Applying learner 'surv.coxph' on task 'rats' (iter 3/3)
#> INFO  [10:43:04.937] [mlr3] Applying learner 'surv.ranger' on task 'gbcs' (iter 1/3)
#> INFO  [10:43:05.826] [mlr3] Applying learner 'surv.ranger' on task 'gbcs' (iter 2/3)
#> INFO  [10:43:06.696] [mlr3] Applying learner 'surv.ranger' on task 'gbcs' (iter 3/3)
#> INFO  [10:43:07.467] [mlr3] Applying learner 'surv.coxph' on task 'gbcs' (iter 1/3)
#> INFO  [10:43:07.488] [mlr3] Applying learner 'surv.coxph' on task 'gbcs' (iter 2/3)
#> INFO  [10:43:07.499] [mlr3] Applying learner 'surv.coxph' on task 'gbcs' (iter 3/3)
#> INFO  [10:43:07.510] [mlr3] Applying learner 'surv.ranger' on task 'grace' (iter 1/3)
#> INFO  [10:43:08.487] [mlr3] Applying learner 'surv.ranger' on task 'grace' (iter 2/3)
#> INFO  [10:43:09.647] [mlr3] Applying learner 'surv.ranger' on task 'grace' (iter 3/3)
#> INFO  [10:43:10.590] [mlr3] Applying learner 'surv.coxph' on task 'grace' (iter 1/3)
#> INFO  [10:43:10.600] [mlr3] Applying learner 'surv.coxph' on task 'grace' (iter 2/3)
#> INFO  [10:43:10.609] [mlr3] Applying learner 'surv.coxph' on task 'grace' (iter 3/3)
#> INFO  [10:43:10.620] [mlr3] Finished benchmark

bmr
#> <BenchmarkResult> of 18 rows with 6 resampling runs
#>  nr task_id  learner_id resampling_id iters warnings errors
#>   1    rats surv.ranger            cv     3        0      0
#>   2    rats  surv.coxph            cv     3        0      0
#>   3    gbcs surv.ranger            cv     3        0      0
#>   4    gbcs  surv.coxph            cv     3        0      0
#>   5   grace surv.ranger            cv     3        0      0
#>   6   grace  surv.coxph            cv     3        0      0

# benchmark
bm = bench::mark(
  harrell_c             = mlr3benchmark::as_benchmark_aggr(bmr, measures = measures[["harrell_c"]]),
  uno_c                 = mlr3benchmark::as_benchmark_aggr(bmr, measures = measures[["uno_c"]]),
  rcll                  = mlr3benchmark::as_benchmark_aggr(bmr, measures = measures[["rcll"]]),
  rcll_erv              = mlr3benchmark::as_benchmark_aggr(bmr, measures = measures[["rcll_erv"]]),
  logloss               = mlr3benchmark::as_benchmark_aggr(bmr, measures = measures[["logloss"]]),
  logloss_erv           = mlr3benchmark::as_benchmark_aggr(bmr, measures = measures[["logloss_erv"]]),
  intlogloss_proper     = mlr3benchmark::as_benchmark_aggr(bmr, measures = measures[["intlogloss_proper"]]),
  intlogloss_proper_erv = mlr3benchmark::as_benchmark_aggr(bmr, measures = measures[["intlogloss_proper_erv"]]),
  caliba                = mlr3benchmark::as_benchmark_aggr(bmr, measures = measures[["caliba"]]),
  dcalib                = mlr3benchmark::as_benchmark_aggr(bmr, measures = measures[["dcalib"]]),
  graf_proper           = mlr3benchmark::as_benchmark_aggr(bmr, measures = measures[["graf_proper"]]),
  graf_proper_erv       = mlr3benchmark::as_benchmark_aggr(bmr, measures = measures[["graf_proper_erv"]]),
  graf_improper         = mlr3benchmark::as_benchmark_aggr(bmr, measures = measures[["graf_improper"]]),
  graf_improper_erv     = mlr3benchmark::as_benchmark_aggr(bmr, measures = measures[["graf_improper_erv"]]),
  check = FALSE
)
#> Warning: Some expressions had a GC in every iteration; so filtering is
#> disabled.

bm
#> # A tibble: 14 × 6
#>    expression                 min   median `itr/sec` mem_alloc `gc/sec`
#>    <bch:expr>            <bch:tm> <bch:tm>     <dbl> <bch:byt>    <dbl>
#>  1 harrell_c               25.3ms  25.97ms    36.4      13.9MB     5.75
#>  2 uno_c                  46.84ms  49.46ms    19.7      22.2MB     5.92
#>  3 rcll                   66.85ms  73.87ms    13.1      35.5MB     7.48
#>  4 rcll_erv              301.73ms 303.23ms     3.30    112.7MB     3.30
#>  5 logloss                  3.16s    3.16s     0.316   373.7MB     9.81
#>  6 logloss_erv              6.49s    6.49s     0.154   813.4MB    10.5 
#>  7 intlogloss_proper      53.47ms  54.21ms    17.5      39.7MB     5.84
#>  8 intlogloss_proper_erv 237.45ms 238.21ms     4.06    109.7MB     5.42
#>  9 caliba                   2.24s    2.24s     0.447   238.4MB     9.83
#> 10 dcalib                106.54ms 107.35ms     9.06     29.5MB     3.62
#> 11 graf_proper            53.91ms  55.49ms    16.8      39.6MB     5.61
#> 12 graf_proper_erv       244.75ms 251.41ms     3.98    109.7MB     3.98
#> 13 graf_improper         138.05ms 139.77ms     7.00     36.9MB     1.75
#> 14 graf_improper_erv     413.76ms 418.55ms     2.39    104.3MB     2.39

plot(bm, type = "violin")
#> Loading required namespace: tidyr
#> Warning: Groups with fewer than two data points have been dropped.
#> Warning: Groups with fewer than two data points have been dropped.
#> Groups with fewer than two data points have been dropped.

plot(bm, type = "beeswarm")

Created on 2024-01-23 with reprex v2.1.0

Session info 
sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.3.2 (2023-10-31)
#>  os       macOS Sonoma 14.2.1
#>  system   aarch64, darwin20
#>  ui       X11
#>  language (EN)
#>  collate  en_US.UTF-8
#>  ctype    en_US.UTF-8
#>  tz       Europe/Berlin
#>  date     2024-01-23
#>  pandoc   3.1.1 @ /System/Volumes/Data/Applications/RStudio.app/Contents/Resources/app/quarto/bin/tools/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package           * version     date (UTC) lib source
#>  backports           1.4.1       2021-12-13 [1] CRAN (R 4.3.0)
#>  beeswarm            0.4.0       2021-06-01 [1] CRAN (R 4.3.0)
#>  bench               1.1.3       2023-05-04 [1] CRAN (R 4.3.0)
#>  checkmate           2.3.1       2023-12-04 [1] CRAN (R 4.3.1)
#>  cli                 3.6.2       2023-12-11 [1] CRAN (R 4.3.1)
#>  codetools           0.2-19      2023-02-01 [2] CRAN (R 4.3.2)
#>  colorspace          2.1-0       2023-01-23 [1] CRAN (R 4.3.0)
#>  crayon              1.5.2       2022-09-29 [1] CRAN (R 4.3.0)
#>  curl                5.2.0       2023-12-08 [1] CRAN (R 4.3.1)
#>  data.table          1.14.10     2023-12-08 [1] CRAN (R 4.3.1)
#>  dictionar6          0.1.3       2021-09-13 [1] CRAN (R 4.3.0)
#>  digest              0.6.34      2024-01-11 [1] CRAN (R 4.3.1)
#>  distr6              1.8.4       2023-11-13 [1] Github (xoopR/distr6@1854b22)
#>  dplyr               1.1.4       2023-11-17 [1] CRAN (R 4.3.1)
#>  evaluate            0.23        2023-11-01 [1] CRAN (R 4.3.1)
#>  fansi               1.0.6       2023-12-08 [1] CRAN (R 4.3.1)
#>  farver              2.1.1       2022-07-06 [1] CRAN (R 4.3.0)
#>  fastmap             1.1.1       2023-02-24 [1] CRAN (R 4.3.0)
#>  fs                  1.6.3       2023-07-20 [1] CRAN (R 4.3.0)
#>  future              1.33.1      2023-12-22 [1] CRAN (R 4.3.1)
#>  future.apply        1.11.1      2023-12-21 [1] CRAN (R 4.3.1)
#>  generics            0.1.3       2022-07-05 [1] CRAN (R 4.3.0)
#>  ggbeeswarm          0.7.2       2023-04-29 [1] CRAN (R 4.3.0)
#>  ggplot2             3.4.4       2023-10-12 [1] CRAN (R 4.3.1)
#>  globals             0.16.2      2022-11-21 [1] CRAN (R 4.3.0)
#>  glue                1.7.0       2024-01-09 [1] CRAN (R 4.3.1)
#>  gtable              0.3.4       2023-08-21 [1] CRAN (R 4.3.0)
#>  highr               0.10        2022-12-22 [1] CRAN (R 4.3.0)
#>  htmltools           0.5.7       2023-11-03 [1] CRAN (R 4.3.1)
#>  knitr               1.45        2023-10-30 [1] CRAN (R 4.3.1)
#>  labeling            0.4.3       2023-08-29 [1] CRAN (R 4.3.0)
#>  lattice             0.21-9      2023-10-01 [1] CRAN (R 4.3.1)
#>  lgr                 0.4.4       2022-09-05 [1] CRAN (R 4.3.0)
#>  lifecycle           1.0.4       2023-11-07 [1] CRAN (R 4.3.1)
#>  listenv             0.9.0       2022-12-16 [1] CRAN (R 4.3.0)
#>  magrittr            2.0.3       2022-03-30 [1] CRAN (R 4.3.0)
#>  Matrix              1.6-1.1     2023-09-18 [1] CRAN (R 4.3.1)
#>  mlr3              * 0.17.2      2024-01-09 [1] CRAN (R 4.3.1)
#>  mlr3benchmark     * 0.1.6       2023-05-30 [1] CRAN (R 4.3.0)
#>  mlr3extralearners * 0.7.1-9000  2024-01-11 [1] Github (mlr-org/mlr3extralearners@0cbdb72)
#>  mlr3misc            0.13.0-9000 2023-10-17 [1] Github (mlr-org/mlr3misc@43c0ffa)
#>  mlr3pipelines       0.5.0-2     2023-12-08 [1] CRAN (R 4.3.1)
#>  mlr3proba         * 0.5.8       2024-01-19 [1] Github (mlr-org/mlr3proba@930d8b0)
#>  mlr3viz             0.7.0.9000  2023-12-22 [1] Github (mlr-org/mlr3viz@73aa4ec)
#>  munsell             0.5.0       2018-06-12 [1] CRAN (R 4.3.0)
#>  ooplah              0.2.0       2022-01-21 [1] CRAN (R 4.3.0)
#>  palmerpenguins      0.1.1       2022-08-15 [1] CRAN (R 4.3.0)
#>  paradox             0.11.1      2023-03-17 [1] CRAN (R 4.3.0)
#>  parallelly          1.36.0      2023-05-26 [1] CRAN (R 4.3.0)
#>  param6              0.2.4       2023-07-24 [1] Github (xoopR/param6@0fa3577)
#>  pillar              1.9.0       2023-03-22 [1] CRAN (R 4.3.0)
#>  pkgconfig           2.0.3       2019-09-22 [1] CRAN (R 4.3.0)
#>  profmem             0.6.0       2020-12-13 [1] CRAN (R 4.3.0)
#>  purrr               1.0.2       2023-08-10 [1] CRAN (R 4.3.0)
#>  R.cache             0.16.0      2022-07-21 [1] CRAN (R 4.3.0)
#>  R.methodsS3         1.8.2       2022-06-13 [1] CRAN (R 4.3.0)
#>  R.oo                1.25.0      2022-06-12 [1] CRAN (R 4.3.0)
#>  R.utils             2.12.3      2023-11-18 [1] CRAN (R 4.3.1)
#>  R6                  2.5.1       2021-08-19 [1] CRAN (R 4.3.0)
#>  ranger              0.16.0      2023-11-12 [1] CRAN (R 4.3.1)
#>  Rcpp                1.0.12      2024-01-09 [1] CRAN (R 4.3.1)
#>  reprex              2.1.0       2024-01-11 [1] CRAN (R 4.3.1)
#>  RhpcBLASctl         0.23-42     2023-02-11 [1] CRAN (R 4.3.0)
#>  rlang               1.1.3       2024-01-10 [1] CRAN (R 4.3.1)
#>  rmarkdown           2.25        2023-09-18 [1] CRAN (R 4.3.1)
#>  rstudioapi          0.15.0      2023-07-07 [1] CRAN (R 4.3.0)
#>  scales              1.3.0       2023-11-28 [1] CRAN (R 4.3.1)
#>  sessioninfo         1.2.2       2021-12-06 [1] CRAN (R 4.3.0)
#>  set6                0.2.6       2023-10-16 [1] Github (xoopR/set6@a901255)
#>  styler              1.10.2      2023-08-29 [1] CRAN (R 4.3.0)
#>  survival            3.5-7       2023-08-14 [1] CRAN (R 4.3.0)
#>  survivalmodels      0.1.18      2023-11-21 [1] Github (RaphaelS1/survivalmodels@79b913f)
#>  tibble              3.2.1       2023-03-20 [1] CRAN (R 4.3.0)
#>  tidyr               1.3.0       2023-01-24 [1] CRAN (R 4.3.0)
#>  tidyselect          1.2.0       2022-10-10 [1] CRAN (R 4.3.0)
#>  utf8                1.2.4       2023-10-22 [1] CRAN (R 4.3.1)
#>  uuid                1.2-0       2024-01-14 [1] CRAN (R 4.3.1)
#>  vctrs               0.6.5       2023-12-01 [1] CRAN (R 4.3.1)
#>  vipor               0.4.7       2023-12-18 [1] CRAN (R 4.3.1)
#>  withr               3.0.0       2024-01-16 [1] CRAN (R 4.3.1)
#>  xfun                0.41        2023-11-01 [1] CRAN (R 4.3.1)
#>  xml2                1.3.6       2023-12-04 [1] CRAN (R 4.3.1)
#>  yaml                2.3.8       2023-12-11 [1] CRAN (R 4.3.1)
#> 
#>  [1] /Users/Lukas/Library/R/arm64/4.3/library
#>  [2] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
#> 
#> ──────────────────────────────────────────────────────────────────────────────

@bblodfon
Copy link
Collaborator Author

bblodfon commented Feb 14, 2024 •
edited
Loading

  • Giving the task and train_set to graf (and similar scores) does not impact execution time
  • ERV = TRUE takes almost double the time because 1) we need to recalculate the score() (as in ERV = FALSE) and we're calling this again inside score(), which adds overhead (I guess some R6 reasons behind) - can't do anything about this, it's like double-calculating the score 2) KM needs train and prediction which takes some time but this is not the bottleneck really
  • logloss and calib_alpha need to be optimized

@bblodfon bblodfon mentioned this issue Feb 16, 2024
Merged
@jemus42
Copy link
Member

jemus42 commented Feb 22, 2024 •
edited
Loading

Re-ran the small benchmark just for the sake of it with 0.6.0: especially logloss has improved a ton! 🎉

library(mlr3)
library(mlr3proba)
library(mlr3benchmark)
library(mlr3extralearners)

measures = list(
msr("surv.cindex",      id = "harrell_c",                                 label = "Harrell's C"),
msr("surv.cindex",      id = "uno_c",                 weight_meth = "G2", label = "Uno's C"),

msr("surv.rcll",        id = "rcll",                  ERV = FALSE,        label = "Right-Censored Log Loss"),
msr("surv.rcll",        id = "rcll_erv",              ERV = TRUE,         label = "Right-Censored Log Loss (ERV)"),

msr("surv.logloss",     id = "logloss",               ERV = FALSE,        label = "Log Loss"),
msr("surv.logloss",     id = "logloss_erv",           ERV = TRUE,         label = "Log Loss (ERV)"),

msr("surv.intlogloss",  id = "intlogloss_proper",     ERV = FALSE, proper = TRUE, label = "Integrated Log Loss (Proper)"),
msr("surv.intlogloss",  id = "intlogloss_proper_erv", ERV = TRUE,  proper = TRUE, label = "Integrated Log Loss (Proper, ERV)"),

msr("surv.calib_alpha", id = "caliba",                label = "Van Houwelingen's Alpha"),
msr("surv.dcalib",      id = "dcalib", truncate = 10, label = "D-Calibration (truncated)"),

msr("surv.graf",        id = "graf_proper",       proper = TRUE,   ERV = FALSE, label = "Graf Score (Proper)"),
msr("surv.graf",        id = "graf_proper_erv",   proper = TRUE,   ERV = TRUE,  label = "Graf Score (Proper, ERV)"),
msr("surv.graf",        id = "graf_improper",     proper = FALSE,  ERV = FALSE, label = "Graf Score (Improper)"),
msr("surv.graf",        id = "graf_improper_erv", proper = FALSE,  ERV = TRUE,  label = "Graf Score (Improper, ERV)")
)
names(measures) = mlr3misc::ids(measures)

# Example bmr
bmr = benchmark(benchmark_grid(
  tasks = tsks(c("rats", "gbcs", "grace")),
  learners = lrns(c("surv.ranger", "surv.coxph")),
  resamplings = rsmp("cv", folds = 3)
), store_backends = TRUE)
#> INFO  [13:18:20.336] [mlr3] Running benchmark with 18 resampling iterations
#> INFO  [13:18:20.368] [mlr3] Applying learner 'surv.ranger' on task 'rats' (iter 1/3)
#> INFO  [13:18:20.498] [mlr3] Applying learner 'surv.ranger' on task 'rats' (iter 2/3)
#> INFO  [13:18:20.601] [mlr3] Applying learner 'surv.ranger' on task 'rats' (iter 3/3)
#> INFO  [13:18:20.711] [mlr3] Applying learner 'surv.coxph' on task 'rats' (iter 1/3)
#> INFO  [13:18:20.725] [mlr3] Applying learner 'surv.coxph' on task 'rats' (iter 2/3)
#> INFO  [13:18:20.733] [mlr3] Applying learner 'surv.coxph' on task 'rats' (iter 3/3)
#> INFO  [13:18:20.774] [mlr3] Applying learner 'surv.ranger' on task 'gbcs' (iter 1/3)
#> INFO  [13:18:22.820] [mlr3] Applying learner 'surv.ranger' on task 'gbcs' (iter 2/3)
#> INFO  [13:18:24.678] [mlr3] Applying learner 'surv.ranger' on task 'gbcs' (iter 3/3)
#> INFO  [13:18:26.607] [mlr3] Applying learner 'surv.coxph' on task 'gbcs' (iter 1/3)
#> INFO  [13:18:26.619] [mlr3] Applying learner 'surv.coxph' on task 'gbcs' (iter 2/3)
#> INFO  [13:18:26.636] [mlr3] Applying learner 'surv.coxph' on task 'gbcs' (iter 3/3)
#> INFO  [13:18:26.647] [mlr3] Applying learner 'surv.ranger' on task 'grace' (iter 1/3)
#> INFO  [13:18:27.913] [mlr3] Applying learner 'surv.ranger' on task 'grace' (iter 2/3)
#> INFO  [13:18:28.979] [mlr3] Applying learner 'surv.ranger' on task 'grace' (iter 3/3)
#> INFO  [13:18:30.112] [mlr3] Applying learner 'surv.coxph' on task 'grace' (iter 1/3)
#> INFO  [13:18:30.122] [mlr3] Applying learner 'surv.coxph' on task 'grace' (iter 2/3)
#> INFO  [13:18:30.131] [mlr3] Applying learner 'surv.coxph' on task 'grace' (iter 3/3)
#> INFO  [13:18:30.143] [mlr3] Finished benchmark

bmr
#> <BenchmarkResult> of 18 rows with 6 resampling runs
#>  nr task_id  learner_id resampling_id iters warnings errors
#>   1    rats surv.ranger            cv     3        0      0
#>   2    rats  surv.coxph            cv     3        0      0
#>   3    gbcs surv.ranger            cv     3        0      0
#>   4    gbcs  surv.coxph            cv     3        0      0
#>   5   grace surv.ranger            cv     3        0      0
#>   6   grace  surv.coxph            cv     3        0      0

# benchmark
bm = bench::mark(
  harrell_c             = mlr3benchmark::as_benchmark_aggr(bmr, measures = measures[["harrell_c"]]),
  uno_c                 = mlr3benchmark::as_benchmark_aggr(bmr, measures = measures[["uno_c"]]),
  rcll                  = mlr3benchmark::as_benchmark_aggr(bmr, measures = measures[["rcll"]]),
  rcll_erv              = mlr3benchmark::as_benchmark_aggr(bmr, measures = measures[["rcll_erv"]]),
  logloss               = mlr3benchmark::as_benchmark_aggr(bmr, measures = measures[["logloss"]]),
  logloss_erv           = mlr3benchmark::as_benchmark_aggr(bmr, measures = measures[["logloss_erv"]]),
  intlogloss_proper     = mlr3benchmark::as_benchmark_aggr(bmr, measures = measures[["intlogloss_proper"]]),
  intlogloss_proper_erv = mlr3benchmark::as_benchmark_aggr(bmr, measures = measures[["intlogloss_proper_erv"]]),
  caliba                = mlr3benchmark::as_benchmark_aggr(bmr, measures = measures[["caliba"]]),
  dcalib                = mlr3benchmark::as_benchmark_aggr(bmr, measures = measures[["dcalib"]]),
  graf_proper           = mlr3benchmark::as_benchmark_aggr(bmr, measures = measures[["graf_proper"]]),
  graf_proper_erv       = mlr3benchmark::as_benchmark_aggr(bmr, measures = measures[["graf_proper_erv"]]),
  graf_improper         = mlr3benchmark::as_benchmark_aggr(bmr, measures = measures[["graf_improper"]]),
  graf_improper_erv     = mlr3benchmark::as_benchmark_aggr(bmr, measures = measures[["graf_improper_erv"]]),
  check = FALSE
)
#> Warning: Some expressions had a GC in every iteration; so filtering is
#> disabled.

bm
#> # A tibble: 14 × 6
#>    expression                 min   median `itr/sec` mem_alloc `gc/sec`
#>    <bch:expr>            <bch:tm> <bch:tm>     <dbl> <bch:byt>    <dbl>
#>  1 harrell_c               22.6ms   24.3ms     39.1     18.4MB     9.79
#>  2 uno_c                   43.3ms   44.7ms     20.4     26.7MB     7.41
#>  3 rcll                    77.4ms   82.5ms     12.3       47MB     7.03
#>  4 rcll_erv               597.8ms  597.8ms      1.67     126MB     3.35
#>  5 logloss                115.7ms  126.3ms      7.96    69.7MB     1.99
#>  6 logloss_erv              369ms  374.1ms      2.67   167.4MB     4.01
#>  7 intlogloss_proper       64.2ms     74ms     13.8     72.8MB     5.89
#>  8 intlogloss_proper_erv  265.7ms    270ms      3.70   173.4MB     5.56
#>  9 caliba                  98.7ms   99.8ms      9.91    37.6MB     1.98
#> 10 dcalib                 120.6ms  126.3ms      8.03    37.7MB     3.21
#> 11 graf_proper             66.9ms   74.6ms     13.6     72.7MB     7.79
#> 12 graf_proper_erv          272ms  276.4ms      3.62   173.4MB     5.43
#> 13 graf_improper          160.9ms  162.3ms      5.84    69.9MB     3.89
#> 14 graf_improper_erv      450.5ms  452.2ms      2.21   168.1MB     2.21

plot(bm, type = "violin")
#> Loading required namespace: tidyr
#> Warning: Groups with fewer than two data points have been dropped.

plot(bm, type = "beeswarm")

Created on 2024-02-22 with reprex v2.1.0

Session info 
sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.3.2 (2023-10-31)
#>  os       macOS Sonoma 14.2.1
#>  system   aarch64, darwin20
#>  ui       X11
#>  language (EN)
#>  collate  en_US.UTF-8
#>  ctype    en_US.UTF-8
#>  tz       Europe/Berlin
#>  date     2024-02-22
#>  pandoc   3.1.1 @ /System/Volumes/Data/Applications/RStudio.app/Contents/Resources/app/quarto/bin/tools/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package           * version     date (UTC) lib source
#>  backports           1.4.1       2021-12-13 [1] CRAN (R 4.3.0)
#>  beeswarm            0.4.0       2021-06-01 [1] CRAN (R 4.3.0)
#>  bench               1.1.3       2023-05-04 [1] CRAN (R 4.3.0)
#>  checkmate           2.3.1       2023-12-04 [1] CRAN (R 4.3.1)
#>  cli                 3.6.2       2023-12-11 [1] CRAN (R 4.3.1)
#>  codetools           0.2-19      2023-02-01 [2] CRAN (R 4.3.2)
#>  colorspace          2.1-0       2023-01-23 [1] CRAN (R 4.3.0)
#>  crayon              1.5.2       2022-09-29 [1] CRAN (R 4.3.0)
#>  curl                5.2.0       2023-12-08 [1] CRAN (R 4.3.1)
#>  data.table          1.15.0      2024-01-30 [1] CRAN (R 4.3.1)
#>  dictionar6          0.1.3       2021-09-13 [1] CRAN (R 4.3.0)
#>  digest              0.6.34      2024-01-11 [1] CRAN (R 4.3.1)
#>  distr6              1.8.4       2023-11-13 [1] Github (xoopR/distr6@1854b22)
#>  dplyr               1.1.4       2023-11-17 [1] CRAN (R 4.3.1)
#>  evaluate            0.23        2023-11-01 [1] CRAN (R 4.3.1)
#>  fansi               1.0.6       2023-12-08 [1] CRAN (R 4.3.1)
#>  farver              2.1.1       2022-07-06 [1] CRAN (R 4.3.0)
#>  fastmap             1.1.1       2023-02-24 [1] CRAN (R 4.3.0)
#>  fs                  1.6.3       2023-07-20 [1] CRAN (R 4.3.0)
#>  future              1.33.1      2023-12-22 [1] CRAN (R 4.3.1)
#>  future.apply        1.11.1      2023-12-21 [1] CRAN (R 4.3.1)
#>  generics            0.1.3       2022-07-05 [1] CRAN (R 4.3.0)
#>  ggbeeswarm          0.7.2       2023-04-29 [1] CRAN (R 4.3.0)
#>  ggplot2             3.4.4       2023-10-12 [1] CRAN (R 4.3.1)
#>  globals             0.16.2      2022-11-21 [1] CRAN (R 4.3.0)
#>  glue                1.7.0       2024-01-09 [1] CRAN (R 4.3.1)
#>  gtable              0.3.4       2023-08-21 [1] CRAN (R 4.3.0)
#>  highr               0.10        2022-12-22 [1] CRAN (R 4.3.0)
#>  htmltools           0.5.7       2023-11-03 [1] CRAN (R 4.3.1)
#>  knitr               1.45        2023-10-30 [1] CRAN (R 4.3.1)
#>  labeling            0.4.3       2023-08-29 [1] CRAN (R 4.3.0)
#>  lattice             0.22-5      2023-10-24 [1] CRAN (R 4.3.1)
#>  lgr                 0.4.4       2022-09-05 [1] CRAN (R 4.3.0)
#>  lifecycle           1.0.4       2023-11-07 [1] CRAN (R 4.3.1)
#>  listenv             0.9.1       2024-01-29 [1] CRAN (R 4.3.1)
#>  magrittr            2.0.3       2022-03-30 [1] CRAN (R 4.3.0)
#>  Matrix              1.6-5       2024-01-11 [1] CRAN (R 4.3.1)
#>  mlr3              * 0.17.2      2024-01-09 [1] CRAN (R 4.3.1)
#>  mlr3benchmark     * 0.1.6       2023-05-30 [1] CRAN (R 4.3.0)
#>  mlr3extralearners * 0.7.1-9000  2024-02-07 [1] Github (mlr-org/mlr3extralearners@5dd745b)
#>  mlr3misc            0.13.0-9000 2023-10-17 [1] Github (mlr-org/mlr3misc@43c0ffa)
#>  mlr3pipelines       0.5.0-2     2023-12-08 [1] CRAN (R 4.3.1)
#>  mlr3proba         * 0.6.0       2024-02-22 [1] Github (mlr-org/mlr3proba@ed6c351)
#>  mlr3viz             0.7.0.9000  2024-01-25 [1] Github (mlr-org/mlr3viz@0049c4c)
#>  munsell             0.5.0       2018-06-12 [1] CRAN (R 4.3.0)
#>  ooplah              0.2.0       2022-01-21 [1] CRAN (R 4.3.0)
#>  palmerpenguins      0.1.1       2022-08-15 [1] CRAN (R 4.3.0)
#>  paradox             0.11.1      2023-03-17 [1] CRAN (R 4.3.0)
#>  parallelly          1.37.0      2024-02-14 [1] CRAN (R 4.3.1)
#>  param6              0.2.4       2023-07-24 [1] Github (xoopR/param6@0fa3577)
#>  pillar              1.9.0       2023-03-22 [1] CRAN (R 4.3.0)
#>  pkgconfig           2.0.3       2019-09-22 [1] CRAN (R 4.3.0)
#>  profmem             0.6.0       2020-12-13 [1] CRAN (R 4.3.0)
#>  purrr               1.0.2       2023-08-10 [1] CRAN (R 4.3.0)
#>  R.cache             0.16.0      2022-07-21 [1] CRAN (R 4.3.0)
#>  R.methodsS3         1.8.2       2022-06-13 [1] CRAN (R 4.3.0)
#>  R.oo                1.26.0      2024-01-24 [1] CRAN (R 4.3.1)
#>  R.utils             2.12.3      2023-11-18 [1] CRAN (R 4.3.1)
#>  R6                  2.5.1       2021-08-19 [1] CRAN (R 4.3.0)
#>  ranger              0.15.3      2024-02-11 [1] Github (imbs-hl/ranger@81bf7d8)
#>  Rcpp                1.0.12      2024-01-09 [1] CRAN (R 4.3.1)
#>  reprex              2.1.0       2024-01-11 [1] CRAN (R 4.3.1)
#>  RhpcBLASctl         0.23-42     2023-02-11 [1] CRAN (R 4.3.0)
#>  rlang               1.1.3       2024-01-10 [1] CRAN (R 4.3.1)
#>  rmarkdown           2.25        2023-09-18 [1] CRAN (R 4.3.1)
#>  rstudioapi          0.15.0      2023-07-07 [1] CRAN (R 4.3.0)
#>  scales              1.3.0       2023-11-28 [1] CRAN (R 4.3.1)
#>  sessioninfo         1.2.2       2021-12-06 [1] CRAN (R 4.3.0)
#>  set6                0.2.6       2023-10-16 [1] Github (xoopR/set6@a901255)
#>  styler              1.10.2.9000 2024-02-16 [1] Github (r-lib/styler@20c5b37)
#>  survival            3.5-7       2023-08-14 [1] CRAN (R 4.3.0)
#>  survivalmodels      0.1.18      2023-11-21 [1] Github (RaphaelS1/survivalmodels@79b913f)
#>  tibble              3.2.1       2023-03-20 [1] CRAN (R 4.3.0)
#>  tidyr               1.3.1       2024-01-24 [1] CRAN (R 4.3.1)
#>  tidyselect          1.2.0       2022-10-10 [1] CRAN (R 4.3.0)
#>  utf8                1.2.4       2023-10-22 [1] CRAN (R 4.3.1)
#>  uuid                1.2-0       2024-01-14 [1] CRAN (R 4.3.1)
#>  vctrs               0.6.5       2023-12-01 [1] CRAN (R 4.3.1)
#>  vipor               0.4.7       2023-12-18 [1] CRAN (R 4.3.1)
#>  withr               3.0.0       2024-01-16 [1] CRAN (R 4.3.1)
#>  xfun                0.42        2024-02-08 [1] CRAN (R 4.3.1)
#>  xml2                1.3.6       2023-12-04 [1] CRAN (R 4.3.1)
#>  yaml                2.3.8       2023-12-11 [1] CRAN (R 4.3.1)
#> 
#>  [1] /Users/Lukas/Library/R/arm64/4.3/library
#>  [2] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
#> 
#> ──────────────────────────────────────────────────────────────────────────────

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bblodfon bblodfon

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jemus42 @bblodfon