| Title: | Performance Metrics for Classification |
|---|---|
| Description: | Computes the area under the receiver operating characteristic curve (AUC) and the average precision of binary classifiers, and their one-vs-rest values for each class of a multiclass outcome. The AUC is the Mann-Whitney U statistic divided by the number of positive-negative pairs (Hanley and McNeil (1982) <doi:10.1148/radiology.143.1.7063747>), and the average precision is the area under the stepwise precision-recall curve. The computations are implemented in the 'rtemis' 'Rust' crate, which sorts the scores once per problem and reads the inputs without copying them. |
| Authors: | E.D. Gennatas [aut, cre, cph] (ORCID: <https://orcid.org/0000-0001-9280-3609>), Authors of the bundled Rust crates [ctb, cph] (Rust crates compiled into the package; see inst/AUTHORS) |
| Maintainer: | E.D. Gennatas <[email protected]> |
| License: | BSD_3_clause + file LICENSE |
| Version: | 0.1.0 |
| Built: | 2026-11-12 01:38:25 UTC |
| Source: | https://github.com/rtemis-org/rtemis.ml |
Computes the average precision (AP) of a binary classifier: the area under
the stepwise precision-recall curve. Each distinct score is one threshold,
and the cases tied at it enter together. Thresholds are numbered from the
highest score down; with recall and precision at
threshold , , with
.
average_precision(reference, scores)average_precision(reference, scores)
reference |
Integer {0, 1}: Reference label of each case, |
scores |
Numeric: Score of each case, where a higher value indicates
the positive class, such as the predicted probability of the positive
class. A double vector the same length as |
The AP depends on the scores only through their order. The value of a classifier that scores independently of the reference labels is expected to be near the prevalence of the positive class. The inputs are read in place, without copying. Extra memory is 8 bytes per case, and time is O(n log n).
Double: the AP, between 0 and 1. NaN when reference contains no
positive case; 1 when it contains no negative case. Errors as
roc_auc().
average_precision_ovr() for multiclass outcomes, roc_auc() for
the area under the ROC curve.
average_precision(c(0L, 0L, 1L, 1L), c(0.1, 0.4, 0.35, 0.8))average_precision(c(0L, 0L, 1L, 1L), c(0.1, 0.4, 0.35, 0.8))
Computes, for each class of a multiclass outcome, the average precision of
that class against all other classes, as average_precision() does for a
binary outcome. The macro-averaged AP is the mean of the result.
average_precision_ovr(reference, scores)average_precision_ovr(reference, scores)
reference |
Factor: Reference label of each case. |
scores |
Numeric matrix: Score of each case for each class, where a
higher value indicates that class, such as the predicted class
probabilities. A double matrix with one row per case and one column per
level of |
The inputs are read in place, without copying. Extra memory is 8 bytes per case, reused across classes, and time is O(k n log n) for k classes.
Named double vector: the AP of each class, named by the levels of
reference. NaN for a class that is absent. Errors as
roc_auc_ovr().
reference <- factor(c("a", "b", "c", "a")) scores <- matrix( c(0.8, 0.1, 0.1, 0.6, 0.1, 0.7, 0.2, 0.2, 0.1, 0.2, 0.7, 0.2), ncol = 3L ) average_precision_ovr(reference, scores)reference <- factor(c("a", "b", "c", "a")) scores <- matrix( c(0.8, 0.1, 0.1, 0.6, 0.1, 0.7, 0.2, 0.2, 0.1, 0.2, 0.7, 0.2), ncol = 3L ) average_precision_ovr(reference, scores)
Computes the area under the receiver operating characteristic (ROC) curve of a binary classifier: the probability that a randomly chosen positive case receives a higher score than a randomly chosen negative case, with ties counted as one half.
roc_auc(reference, scores)roc_auc(reference, scores)
reference |
Integer {0, 1}: Reference label of each case, |
scores |
Numeric: Score of each case, where a higher value indicates
the positive class, such as the predicted probability of the positive
class. A double vector the same length as |
The AUC depends on the scores only through their order, so any score where a higher value indicates the positive class gives the same result as its predicted probability. The inputs are read in place, without copying. Extra memory is 8 bytes per case, and time is O(n log n).
Double: the AUC, between 0 and 1. NaN when reference contains
only one class or is empty. An error is raised when the lengths differ,
when reference contains a value other than 0 or 1, including NA, or
when scores contains NA or NaN.
roc_auc_ovr() for multiclass outcomes, average_precision() for
the area under the precision-recall curve.
roc_auc(c(0L, 0L, 1L, 1L), c(0.1, 0.4, 0.35, 0.8))roc_auc(c(0L, 0L, 1L, 1L), c(0.1, 0.4, 0.35, 0.8))
Computes, for each class of a multiclass outcome, the AUC of that class
against all other classes, as roc_auc() does for a binary outcome. The
macro-averaged AUC is the mean of the result.
roc_auc_ovr(reference, scores)roc_auc_ovr(reference, scores)
reference |
Factor: Reference label of each case. |
scores |
Numeric matrix: Score of each case for each class, where a
higher value indicates that class, such as the predicted class
probabilities. A double matrix with one row per case and one column per
level of |
The inputs are read in place, without copying. Extra memory is 8 bytes per case, reused across classes, and time is O(k n log n) for k classes.
Named double vector: the AUC of each class, named by the levels of
reference. NaN for a class that is absent or is the only class
present. An error is raised when reference is not a factor or contains
NA, when the dimensions of scores do not match, or when scores
contains NA or NaN.
reference <- factor(c("a", "b", "c", "a")) scores <- matrix( c(0.8, 0.1, 0.1, 0.6, 0.1, 0.7, 0.2, 0.2, 0.1, 0.2, 0.7, 0.2), ncol = 3L ) roc_auc_ovr(reference, scores)reference <- factor(c("a", "b", "c", "a")) scores <- matrix( c(0.8, 0.1, 0.1, 0.6, 0.1, 0.7, 0.2, 0.2, 0.1, 0.2, 0.7, 0.2), ncol = 3L ) roc_auc_ovr(reference, scores)