4. Evaluation metrics and processes

In challenges or in any publication that aims to determine if a particular method improves on the state-of-the-art for a particular task, the question of how to evaluate the methods is extremely important. At the core of any evaluation process, there is one or several evaluation metrics. Different metrics have been proposed and have achieved a more-or-less standard status for all types of tasks, and many studies have been made to examine their behaviour in different circumstances. Moreover, the evaluation process is not limited to the metric itself.

In this chapter, we will start in section 4.1 by providing the necessary definitions for the description of the evaluation process of digital pathology image analysis tasks, and of the common (and less common) evaluation metrics used for detection, classification, and segmentation tasks, as well as for more complex tasks that combine these different aspects. The use of these metrics in digital pathology challenges will then be discussed in section 4.2, and the state-of-the-art of the analysis of their biases and limitations in section 4.3.

We will then present our additional analyses and experiments in section 4.4, and provide recommendations on the choices that can be made when determining the evaluation process for a digital pathology task in section 4.5.

4.1 Definitions

Most of the formal definitions of the metrics used in this section are adapted from several publications and reworked to use the same mathematical conventions. For the detection metrics, we largely rely on the works of Padilla et al. [38], [39]. Most of the classification metrics can be found in Luque et al. [32], while the definitions for the segmentation metrics are given in Reinke et al. [40]. The evaluation process in general described here is largely based on our analysis of the processes described in all the digital pathology challenges referenced in section 4.2.

4.1.1 Evaluation process

The evaluation of an algorithm in an image analysis task starts from a set of “target” samples (usually from expert annotations and considered to be the “ground truth”), associated to a set of predictions from the algorithm. In the final evaluation of an algorithm, the samples would come from the “test set” extracted from the overall dataset, which should be as independent as possible from the training and validation set. The exact nature of the samples will depend on the task.

Digital pathology datasets have a hierarchical nature, with the top-level being the patient. From each patient, different whole-slide images (WSI) may have been acquired. From each of those WSIs, different image patches may have been extracted, and often each of these patches will contain a set of individual objects of interest (as illustrated in Figure 4.1).

In the DigestPath 2019 challenge¹, for instance, two different datasets are described. The “signet ring cell dataset” contains objects (the signet ring cell) in 2000x2000px image patches extracted from WSIs of H&E stained slides of individual patients. In the “colonoscopy tissue segment dataset,” however, WSIs have been directly annotated at the pixel-level, so there are no “patch-level” or “object-level” items.

A metric is thus defined as a function $M = Metric(T,\ P)$ that evaluates a pair of “target” (ground truth) and “predicted” items (e.g. object localization, patient diagnostic class…). Most metrics will produce outputs either in the $\left\lbrack 0,\ 1 \right\rbrack$ or the $\lbrack - 1,\ 1\rbrack$ range, but there are exceptions (such as distance-based metrics, which are typically in $\lbrack 0,\ \infty\lbrack$ ). The aggregation process, on the other hand, describes how the set of metric values $\{ M_{i}\}$ at a given level is reduced to a single score $S$ at a higher level (see Figure 4.2).

With these definitions in mind, we can now turn our attention to the three main “task types” that we previously identified in Chapter 1, as well as their combinations.

In a detection task, there will typically be an object-level annotation. To evaluate such a task, the first step is therefore to find the matching pairs $(T_{i},\ P_{j})$ of objects in the images. The matching criteria are therefore an important factor in the evaluation process. They typically involve a matching threshold, for instance based on the centroid distance or the surface overlap. Once the matching pairs of objects have been found, there are two separate aspects of the results that can be evaluated. First, a partial confusion matrix can be built:

$$\begin{pmatrix} \text{N.A.} & FP = \left| \{ P_{i} \notin M\} \right| \\ FN = \left| {\{ T}_{i} \notin M\} \right| & TP = \left| \{ T_{i} \in M\} \right| \\ \end{pmatrix}$$

With TP the “True Positives” (number of matching pairs), FN the “False Negatives” (number of un-matched ground truth objects), FP the “False Positives” (number of un-matched predicted objects), $M = \left\{ \text{Matches}\left( T_{k},P_{l} \right) \right\}$ and $| \cdot |$ is the cardinality of a set. It should be noted that, at the object-level, there are no countable “True Negatives,” which will limit the available metrics for evaluating the detection score [38].

The other possible aspect of the evaluation is the quality of the matches. This can take different forms depending on what exactly the target output was: bounding box overlap, centroid distance, etc.

In a classification task, the annotation could be at the object level (instance classification), at the pixel level (semantic segmentation), at both levels (instance segmentation and classification), or at any of the “higher” levels in Figure 4.1: patch, WSI, or even patient. The latter cases correspond to “pure” classification tasks, whereas the others are combining classification with detection and/or segmentation. The classification part of the problem can typically be summed up with a confusion matrix, built from pairs of ${(T}_{i},\ P_{i})$ where $T_{i} \in \lbrack 1,\ m\rbrack$ and $m$ is the number of classes in the problem, and $P_{i} = argmax_{c}(\pi_{\text{ic}})$ with $\pi_{\text{ic}}$ a vector of $m$ class probabilities so that $\sum_{c}^{}{\pi_{\text{ic}} = 1}$. The confusion matrix will therefore be a $m \times m$ matrix with $CM_{\text{jk}} = |\left\{ \left( T_{i},P_{i} \right)\ \right|\ T_{i} = j,\ P_{i} = k\}|$.

The distinction between “classification” tasks and “detection” tasks can sometimes be difficult to make. In many digital pathology applications, there will be one “meaningful” class (for instance: nuclei, tumour, etc.), and a “no class” category that includes “all the rest.” In those cases, there will be clear “positive” and “negative” categories, and the confusion matrix will include the TP, FP, FN (and, in this case, countable TNs) of the detection tasks. As we will see further in this chapter, this can be a source of confusion in the definition of the metrics, as some metrics, such as the F1-Score, take a different form if a single “positive” class is considered (as in a detection task) or if two or more classes are considered on equal grounds.

The combination of “detection” and “classification” is instance classification, which can also be seen as multi-class detection. In such problems, the results can be described in a confusion matrix that includes multiple classes and a background or no-instance category. The confusion matrix for an m-class problem would therefore look like:

$$\begin{pmatrix} \text{N.A.} & CM_{01} & \ldots & CM_{0m} \\ CM_{10} & CM_{11} & \ldots & CM_{1m} \\ \vdots & \vdots & \ddots & \vdots \\ CM_{m0} & \ldots & \ldots & CM_{\text{mm}} \\ \end{pmatrix}$$

Where the first line corresponds to falsely positive detections (i.e. detections for any of the classes that correspond to no ground truth object) and the first column to falsely negative detections (i.e. ground truth objects that have no predictions). As in the previously described detection confusion matrix, the top-left corner is uncountable and corresponds to the true negative detections.

In segmentation tasks, the annotations will be at the pixel-level. These annotations can be simply binary (an “annotation mask”), or also contain class labels (“class map”) and/or instance labels (“instance map”). We will therefore have $T = \{ T_{i}\}$ and $P = \{ P_{i}\}$, with $i$ the pixel position. In an instance segmentation case, there will once again need to be a matching criterion to find the matching subsets $T_{k} = \left\{ T_{i}\ \right|\text{\ label}\left( T_{i} \right) = k\}$ and $P_{l} = \left\{ P_{i}\ \right|\text{\ label}\left( P_{i} \right) = l\}$, so that the evaluation can be reduced to several binary “object-level” annotation masks. In semantic segmentation, there is no matching step necessary. Instead, a segmentation metric will often simply be computed per-class, then an average score can be computed. The most common evaluation metrics can be defined from pixel-level confusion matrices, or as distance metrics that compare the contours and/or the centroids of the binary masks.

When all three tasks are combined together, we get instance segmentation and classification. In such problems, each pixel can be associated both to a class and to an instance label. The evaluation of such problems can be quite difficult, as will be discussed through this chapter.

4.1.2 Detection metrics

As mentioned above, detection tasks typically require a matching step before the evaluation, often involving a matching threshold $\mu$. The matching step takes the sets of Target objects $\{ T_{i}\}$ and Predicted objects $\{ P_{i}\}$ and outputs three sets: the true positives, the un-matched false negatives, and the un-matched false positives.

These sets are related to a confidence threshold $\tau$ on the detection probability (which is the raw output of the detection algorithm), that determines if a candidate region is considered as a detected object or not: $P\left( \tau \right) = \{ P_{i};\pi_{i} \geq \tau\}$, with $\pi_{i}$ the detection probability for the object $i$ [38], with $\tau$ usually set to 0.5.

Higher confidence thresholds will yield a smaller set of $P_{i}$, which means that the following relationships are always verified:

$$\tau_{1} < \tau_{2} \Rightarrow \text{TP}_{\tau_{1}} \geq \text{TP}_{\tau_{2}},\ \text{FP}_{\tau_{1}} \geq \text{FP}_{\tau_{2}}\text{\ and\ }\text{FN}_{\tau_{1}} \leq \text{FN}_{\tau_{2}}$$

Detection metrics will therefore be either “fixed-threshold” metrics, bound to a particular choice for the matching and confidence thresholds, or “varying-threshold,” capturing the evolution of the metric as the thresholds are moved.

4.1.2.1 Object Matching

In most cases, object detection outputs include some sort of localisation, which gives an indication of where is the object in the image, and what is its size. In the most precise case, we have the full segmentation of the object, and are therefore in an “instance segmentation” problem. More often, the localisation will be given in the form of a bounding box with a centroid. In the extreme opposite case, there is no localisation at all, and simply an indication of whether an object is present in the image or not. There is then no “matching” step, and the problem is formulated as a binary classification problem, with the “no object” and “object” class being determined at the image patch level.

The matching step generally is “overlap-based” and/or “distance-based.”

Given a pair of target and predicted objects $T_{i}$ and $P_{j}$, represented as a set of pixels belonging to this object (which may come either from bounding boxes or per-pixel instance masks), overlap-based methods will look at the intersection area ($|T_{i} \cap P_{j}|$) and the union area ($|T_{i} \cup P_{j}|$) of the two objects. These two measures can be combined in the intersection over union (IoU), also known as the Jaccard Index [24]:

$$IoU(T_{i},P_{j}) = \frac{|T_{i} \cap P_{j}|}{|T_{i} \cup P_{j}|}$$

A common criterion is to apply a threshold on this IoU, so that a pair of objects is “a match” if $\text{IoU}\left( T_{i},P_{j} \right) \geq \mu_{\text{IoU}}$, with $\mu_{\text{IoU}}$ often set at 0.5 or 0.75 [39]. If $\mu_{\text{IoU}} < 0.5$, it is possible for multiple target objects to be matched to the same predicted objects (and vice-versa), in which case a “maximum IoU” criterion is typically added to the rule (so that the match is the predicted object with the maximum IoU among those that satisfy the threshold condition). In that case, we will therefore have: $\text{Matches}\left( T,P \right) = \left\{ \left( T_{i},P_{j} \right) \right\}$ where all three following conditions are respected:

$$\text{IoU}\left( T_{i},P_{j} \right) \geq \mu_{\text{IoU}},\$$

$$\text{IoU}\left( T_{i},\ P_{j} \right) > IoU\left( T_{i},\ P_{k \neq j} \right)\ \forall P_{k} \in P$$

$$\text{IoU}\left( T_{i},P_{j} \right) > IoU\left( T_{k \neq i},\ P_{j} \right)\forall T_{k} \in T$$

The maximum IoU can also be used as a single criterion, with no threshold attached, in which case any overlap may be detected as a match (this is equivalent to setting $\mu_{\text{IoU}} = 0$).

Another strategy to determine a match relates to the distance between the centroids of the objects. Similarly to the IoU criterion, this will imply a “closest distance” rule, which may be associated with a “maximum distance threshold.” The criteria will therefore be:

$$d\left( T_{i},P_{j} \right) \leq \mu_{\text{dist}}$$

$$d\left( T_{i},P_{j} \right) < d\left( T_{i},\ P_{k \neq j} \right)\ \forall P_{k} \in P$$

$$d\left( T_{i},P_{j} \right) < d\left( T_{k \neq i},\ P_{j} \right)\ \forall T_{k} \in T$$

With $d\left( T_{i},P_{j} \right)$ usually defined as the Euclidian distance between the centroids of $T_{i}$ and $P_{j}$.

The same rule can be adapted to other distance definitions, such as for instance the Hausdorff’s distance (defined in the segmentation metrics below) between the contours of the objects.

4.1.2.2 Fixed-threshold metrics

Fixed-threshold metrics are computed for a specific value of the matching threshold (commonly, $\mu_{\text{IoU}} = 0.5$) and of the confidence threshold (also usually $\tau = 0.5$), which determine the sets TP, FP and FN.

The precision (PRE) of the detection algorithm is the proportion of “positive detections” that are correct, and is defined as:

$$PRE = \frac{\text{TP}}{TP + FP}$$

The recall (REC) is the proportion of “positive target” that have been correctly predicted, and is defined as:

$$REC = \frac{\text{TP}}{TP + FN}$$

The F1-score (F1) is the harmonic mean of the two:

$$F1 = 2 \times \frac{PRE \times REC}{PRE + REC} = 2 \times \frac{\text{TP}}{2 \times TP + FP + FN}$$

4.1.2.3 Varying-threshold metrics

To better characterize the robustness of an algorithm’s prediction, it is sometimes useful to look at how the performance evolves with different values of the confidence and/or matching thresholds.

Decreasing the confidence threshold means being less restrictive on what’s considered a “prediction,” and therefore to more true positives, more false positives and less false negatives. There is thus a monotonic relationship between the confidence threshold and the recall. The relationship between the precision and the confidence threshold, however, is less predictable.

This dynamic relationship between PRE, REC and the confidence threshold can be captured in a “precision-recall curve” (PR curve). An example of a PR curve is shown in Figure 4.3, constructed from synthetic data. As we can see, while the overall trend is for the precision to decrease as the recall increases, the relationship is not monotonic, and the precision has a characteristic “zig-zag” pattern.

To summarize the performance shown by the PR curve, the “area under the PR curve” (AUPRC) can be computed. An ideal detector would have $AUPRC = 1$. A very common approximation of the AUPRC, that is less sensitive to the small oscillations of the precision, is the “Average Precision” (AP). Adapting the formulation from Luque et al. [39], we can express the PR curve as a function $PRE = Pre(REC)$, and the AP is obtained by:

1. Interpolating the PR curve so that $Pre^{\ast}\left( \text{REC} \right) = \max_{k \geq REC}{Pre(k)}$, thus removing the zigzag pattern by replacing each point by the “highest value to its right” (see green line in Figure 4.3).
   
2. Sampling $Pre^{\ast}$ with $n$ equally spaced recall values, so that $AP = \frac{1}{n}\sum_{k = 1}^{n}{Pre^{\ast}(k)}.$

The AP is therefore a varying-threshold metric for the confidence threshold, but a fixed-threshold metric for the matching rule. To also take into account this matching rule variability, the AP can be averaged over different matching threshold values. As an example, in the NuCLS 2021 challenge [3], one of the metrics used to evaluate the detection score is the mean AP using an IoU threshold varying between 0.5 and 0.95, by step of 0.05. If we set $AP@k$ to mean the AP using the matches found with $\mu_{\text{IoU}} \geq k$, then we have:

$$AP@.5:.95 = \frac{1}{10}\sum_{k = 0}^{9}{AP@(0.5 + 0.05k)}$$

4.1.3 Classification metrics

The results of a classification problem with $m$ categories are summarized in a $m \times m$ confusion matrix. We will use here the convention that the rows of that matrix correspond to the “ground truth” class, while the columns correspond to the “predicted” class, and $CM_{\text{ij}}$ corresponds to the number of examples with “true class” $i$ and “predicted class” $j$. It should be noted, however, that some of the classification metrics may be used to simply characterize the agreement between two sets of observations, without one being assumed to be the “truth.” To be useful for that latter purpose, a metric needs to be observer invariant. Practically, this is the case if $\text{Metric}\left( \text{CM} \right) = Metric(CM^{T})$ (or, from the sets of ground truth and predicted observations $T = \left\{ T_{i} \right\},\ P = \{ P_{i}\}$ : $\text{Metric}\left( T,P \right) = Metric(P,\ \ T)$).

Another characteristic of classification metrics is that they can be class-specific or global. A class-specific metric $Metric_{c}$ will evaluate class $c$ as a “positive” class against all others, while global metrics aggregate the class-specific values.

Metrics can also assume an order to the classes or, on the contrary, that they are purely categorical. In the former case, switching the classes in the confusion matrix would lead to a different result, while categorical metrics are class-switch invariant.

Finally, another aspect (that we more thoroughly examine in sections 4.3 and 4.4) is the class imbalance bias that many metrics exhibit. This means that changing the class balance of the dataset (e.g. by changing the sampling method) alters the results.

4.1.3.1 Class-specific metrics

Several per-class metrics can be defined. These use the notions of the True Positives (TP), True Negatives (TN), False Positives (FP) and False Negatives (FN). Using our general notation for the confusion matrix, we can have for class $c$:

$$TP_{c} = CM_{\text{cc}}$$

$$FP_{c} = \sum_{k \neq c}^{}{CM_{\text{kc}}}$$

$$FN_{c} = \sum_{k \neq c}^{}{CM_{\text{ck}}}$$

$$TN_{c} = n - TP_{c} - FP_{c} - FN_{c}$$

An example on a 5 classes problem is shown in Table 4.1.

The sensitivity (SEN), also called recall (REC) or True Positive Rate (TPR) measures the proportion of “True Positives” among all the samples that are “positive” in the Target (i.e. that truly belong to the class $c$):

$$SEN_{c} = \ \frac{TP_{c}}{TP_{c} + FN_{c}}$$

The specificity (SPE), or True Negative Rate (TNR) is conversely the proportion of TN in the negative Target samples:

$$SPE_{c} = \frac{TN_{c}}{TN_{c} + FP_{c}}$$

The precision (PRE) or Positive Predictive Value (PPV) is the proportion of TP in the positive Predicted samples:

$$PRE_{c} = \frac{TP_{c}}{TP_{c} + FP_{c}}$$

The Negative Predictive Value (NPV) is similarly the proportion of TN in the negative Predicted samples:

$$NPV_{c} = \frac{TN_{c}}{TN_{c} + FN_{c}}$$

The False Positive Rate (FPR) is the proportion of FP in the negative Target samples:

$$FPR_{c} = \frac{FP_{c}}{TN_{c} + FP_{c}} = 1 - SPE_{c}$$

The F1-Score is better defined as a detection metric (see section 4.1.2), but it is often used in classification problems as well.

The per-class F1-Score ($F1_{c}$) is defined as the harmonic mean of the precision and sensitivity:

$$F1_{c} = \ 2 \times \frac{PRE_{c} \times SEN_{c}}{PRE_{c} + SEN_{C}}$$

The per-class Geometric Mean ($GM_{c}$) of the SEN and SPE is also sometimes used [32]:

$$GM_{c} = \sqrt{SEN_{c} \times SPE_{c}}$$

4.1.3.2 Global metrics

To get a global classification score that aggregates the results from all classes, several metrics are possible.

The simplest performance measure from the confusion matrix is the accuracy (ACC), which is given by the sum of the elements on the diagonal (the “correct” samples) divided by the total number of samples $n$:

$$ACC = \ \frac{\sum_{i}^{}{CM_{\text{ii}}}}{n}$$

It is also possible to extend the F1-Score so that it becomes a global metric. The macro-averaged F1-Score has been defined in two different ways in the literature [42].

First, as a simple average of the per-class F1-scores. We call this version the simple-averaged F1-Score ($sF1$):

$$sF1 = \frac{1}{m}\sum_{c}^{}{F1_{c}}$$

The other definition first computes the average of the per-class PRE and SEN, before computing the harmonic mean. We will call it the harmonic-averaged F1-Score ($hF1$):

$$hF1 = 2 \times \frac{ΜPRE \times Μ\text{SEN}}{ΜPRE + Μ\text{SEN}}$$

With:

$$ΜPRE = \frac{\sum_{k}^{}{PRE_{k}}}{m}$$

$$ΜSEN = \frac{\sum_{k}^{}{SEN_{k}}}{m}$$

Meanwhile, the micro-averaged F1-Score ($\mu F1$) will first aggregate the TP, FP and FN before computing the micro-precision and micro-recall, and finally the F1-Score. It has been used in challenges such as Gleason 2019, but it should be noted that it is equivalent to the accuracy:

$$\mu PRE = \frac{\sum_{k}^{}{TP_{k}}}{\sum_{k}^{}{(TP_{k} + FP_{k})}} = \frac{\sum_{k}^{}{TP_{k}}}{n} = ACC$$

$$\mu SPE = \frac{\sum_{k}^{}{TP_{k}}}{\sum_{k}^{}{(TP_{k} + \ FN_{k})}} = \ \frac{\sum_{k}^{}{TP_{k}}}{n} = ACC$$

$$\mu F1 = 2 \times \frac{\mu PRE \times \mu SPE}{\mu PRE + \mu SPE} = ACC\$$

Like the F1-Score, the $GM_{c}$ can also be extended as a global metric. It will then be the Geometric Mean (GM) of the per-class $SEN_{c}$:

$$GM = \left( \prod_{c}^{}{SEN_{c}} \right)^{\frac{1}{m}}$$

The GM has zero bias due to class imbalance [32]. It is, however, not observer invariant.

The Matthews Correlation Coefficient (MCC), often called $R_{K}$ in the multiclass case (and sometimes phi coefficient), uses all the terms of the CM and is defined as:

$$MCC = \ \frac{n \times \sum_{k}^{}{CM_{\text{kk}}} - \ \sum_{k}^{}{(\sum_{l}^{}{CM_{\text{lk}}} \times \sum_{l}^{}{CM_{\text{kl}}})}}{\sqrt{n^{2} - \sum_{k}^{}\left( \sum_{l}^{}{CM_{\text{kl}}} \right)^{2}}\sqrt{n^{2} - \sum_{k}^{}\left( \sum_{l}^{}{CM_{\text{lk}}} \right)^{2}}}$$

It is bound between -1 and 1, where 0 means that the target and prediction sets are completely uncorrelated.

Cohen’s kappa is a measure of the agreement between two sets of observations and can be used as a classification metrics to rate the agreement between the target and the predictions. It measures the difference between the observed agreement $p_{o}$ and the agreement expected by chance $p_{e}$, and is defined as [8]:

$$\kappa = \frac{p_{o} - p_{e}}{1 - p_{e}}$$

So that a “perfect agreement” ($p_{o} = 1$) leads to $\kappa = 1$, a perfect disagreement ($p_{o} = 0$) to a negative $\kappa$ (the exact minimum value depends on the distribution of the data). From the confusion matrix, $p_{o}$ and $p_{e}$ are defined as:

$$p_{o} = ACC = \frac{\sum_{i}^{}{CM_{\text{ii}}}}{n}$$

$$p_{e} = \frac{\sum_{i}^{}\left( \sum_{k}^{}{CM_{\text{ik}}}\sum_{k}^{}{CM_{\text{ki}}} \right)}{n^{2}}$$

Cohen’s kappa is also often used for ordered categories, where errors between classes which are “close” together are less penalised.

Let $W$ be a weight matrix ($W$ should verify $w_{\text{ij}} = w_{\text{ji}}$, and $0 \leq w_{\text{ij}} \leq 1$).

We can define the matrix of expected observations from random chance with:

$$e_{\text{ij}} = \frac{\sum_{k}^{}{CM_{\text{ik}}}\sum_{k}^{}{CM_{\text{kj}}}}{n}$$

From that, we define Cohen’s kappa generally with:

$$\begin{matrix} \kappa = 1 - \ \frac{\sum_{i}^{}{\sum_{j}^{}{w_{\text{ij}}CM_{\text{ij}}}}}{\sum_{i}^{}{\sum_{j}^{}{w_{\text{ij}}e_{\text{ij}}}}}\ \\ \end{matrix}$$

The three most common formulations of the weights are:

Unweighted kappa $\mathbf{\kappa_U}$: $w_{\text{ij}} = 1 - \delta_{\text{ij}}$ (using the Kronecker delta), which resolves to the same value as the original definition of the $\kappa$ shown above.

Linear weighted kappa $\mathbf{\kappa_L}$: $w_{\text{ij}} = |i - j|\$

Quadratic weighted kappa $\mathbf{\kappa_Q}$: $w_{\text{ij}} = \left( i - j \right)^{2}$

Cohen’s kappa is bound between -1 and 1, with 0 corresponding to as much agreement as expected by random chance.

4.1.3.3 Varying-threshold metrics

All the classification metrics described thus far are “no threshold” metrics, where the predicted class of a sample is simply taken as the class which has the maximum predicted probability. It can however also be interesting to take into account the “confidence” of a model with regards to its predictions. This information can be captured in a “Receiver Operating Characterstic,” or ROC Curve. Where the detection PR-curve plotted the PRE and REC for varying confidence thresholds, the ROC curve plots the SEN (=REC) and the FPR (=1-SPE).

As the SEN and SPE are “per-class” metrics, so is the ROC a “per-class” visualisation, where the “varying threshold” is used in a “one class vs all” manner. For a class $c$, a threshold $\tau$, and the set of predicted class probabilities vectors $\{ P_{i}\}$ where $\pi_{\text{ic}}$ is the probability that sample $i$ has the class $c$, we can define:

$$TP_{c,\tau} = \left| \{ P_{i}\ ;\ \pi_{\text{ic}} \geq \tau\ \&\ T_{i,c} = 1\} \right|$$

$$FP_{c,\tau} = |{\{ P}_{i}\ \ ;\pi_{\text{ic}} \geq \tau\ \& T_{i,c} = 0\}|$$

$$TN_{c,\tau} = |{\{ P}_{i}\ ;\pi_{\text{ic}} < \tau\ \& T_{i,c} = 0\}|$$

$$FN_{c,\tau} = \left| \{ P_{i}\ ;\pi_{\text{ic}} < \tau\ \&\ T_{i,c} = 1\} \right|$$

From which $SPE_{c,\tau}$ and $SEN_{c,\tau}$ can be computed.

To summarize the information contained in the ROC curve, the Area Under the ROC ($AUROC_{c}$) can be computed for a class, so that an “ideal classifier” for that class would have an $AUROC_{c}$ of 1. As for the F1-Score, we can also define micro-averaged and macro-averaged summaries of the $AUROC$.

The macro-average $MAUROC$ is defined as:

$$MAUROC = \frac{1}{c}\sum_{c}^{}{AUROC_{c}}$$

While the micro-averaged is defined by first summing the TP, FP, TN and FN:

$μTP_{\tau} = \frac{1}{c}\sum_{c}^{}{TP_{c,\tau}}$, etc.

Then computing the resulting $μSPE_{\tau}$, $μSEN_{\tau}$, and finally the $μAUROC$ from those micro-averaged values. An example on synthetic data is presented in Figure 4.4.

A summary of the main characteristics of the metrics discussed here can be found in Table 4.2.

4.1.4 Segmentation metrics

We consider in this section pure “segmentation” tasks that separate a foreground region from a background region. Instance and semantic segmentation will be considered in the “combined metrics” section. Segmentation metrics therefore compare two binary masks in an image. Let $T$ be the set of pixels that belong to the target (“ground truth”) mask, and $P$ the set of pixels that belong to the predicted mask. There are two main categories of segmentation metrics: those that measure the overlap between the two sets, and those that measure a distance between the contours of the $T$ and $P$ masks, labelled $T_{o}$ and $P_{o}$ respectively.

In the following definitions, $| \cdot |$ is the cardinality of the set and $\sim$ indicates all the elements that are not in the set.

4.1.4.1 Overlap metrics

The overlap metrics can also be defined using a “confusion matrix” based on the binary segmentation masks $\mathbf{T}$ and $\mathbf{P}$ (we use here boldface to denote the mask matrices instead of the sets), where $\mathbf{T}_{\mathbf{i}} = 1$ for all $\mathbf{T}_{\mathbf{i}} \in T$, $0$ otherwise. The TP, FP, FN and TN are then defined as:

$$TP = \ \sum_{i}^{}{\mathbf{T}_{\mathbf{i}}*\mathbf{P}_{\mathbf{i}}} = |T \cap P|$$

$$FP = \ \sum_{i}^{}{(1 - \mathbf{T}_{\mathbf{i}})*\mathbf{P}_{\mathbf{i}}} = |\sim T \cap P|$$

$$FN = \ \sum_{i}^{}{\mathbf{T}_{\mathbf{i}}*\left( 1 - \mathbf{P}_{\mathbf{i}} \right)} = |T \cap \sim P|$$

$$TN = \ \sum_{i}^{}{\left( 1 - \mathbf{T}_{\mathbf{i}} \right)*(1 - \mathbf{P}_{\mathbf{i}})} = |\sim T \cap \sim P|$$

The two most commonly used overlap metrics are the Dice Similarity Coefficient (DSC) and the Intersection over Union (IoU), while the most common distance-based metric is Hausdorff’s Distance (HD). The IoU, DSC and HD are illustrated in Figure 4.5.

The IoU is defined as:

$$IoU = \ \frac{|T \cap P|}{|T \cup P|}$$

It is bounded between 0 (no overlap) and 1 (perfect overlap).

Using the confusion matrix notation, it can also be defined as:

$$IoU = \frac{\text{TP}}{TP + FP + FN}$$

The DSC is defined as:

$$DSC = \ \frac{2|T \cap P|}{\left| T \right| + |P|}$$

It is also bounded between 0 and 1, and heavily correlated to the IoU, as the two are linked by the relationship:

$$DSC = \frac{2 \times \ IoU}{1 + IoU}$$

That relationship also means that $DSC \geq IoU$ is always verified.

Like the IoU, we can define it in terms of the confusion matrix values:

$$DSC = \frac{2 \times TP}{2 \times TP + FP + FN}$$

Which shows that it simply corresponds to a “per-pixel” definition of the F1-score.

4.1.4.2 Distance metrics

For Hausdorff’’s distance (HD), only the contours $T_{o}$ and $P_{o}$ are considered. First, the Euclidean distances from all points in $T_{o}$ to the closest point in $P_{o}$ are computed, and the maximum value is taken:

$$h_{\text{TP}} = \max_{t \in T_{o}}{\min_{p \in P_{o}}{|\left| t - p \right||}}$$

Then, the same thing is done starting from all the points in $P_{c}$:

$$h_{\text{PT}} = \max_{p \in P_{o}}{\min_{t \in T_{o}}{|\left| p - t \right||}}$$

Hausdorff’s distance is then defined as the maximum of those two values:

$$HD = max(h_{\text{TP}},h_{\text{PT}})$$

This process is illustrated in Figure 4.6.

As the HD is extremely sensitive to outliers, it is also sometimes preferred to slightly relax the “maximum,” and to replace it with a percentile $P_{\lambda}$:

$$h_{\lambda,TP} = \underset{t \in T_{o}}{P_{\lambda}}{\min_{p \in P_{o}}{|\left| t - p \right||}},\ HD_{\lambda} = max(h_{\lambda,TP},h_{\lambda,PT})$$

With $HD_{95}$ being the most frequent choice.

While a HD of 0 corresponds to a perfect segmentation, its maximum value is unbounded. This makes it a particularly tricky metric to use in aggregates, as it is difficult to assign a value, for instance, to a “missing prediction” (i.e. an image where $P = \varnothing$).

While less commonly used, some other distance-based metrics exist. Reinke et al.’s review [40] notably mention the Average Symmetric Surface Distance (ASSD) and the Normalized Surface Distance (NSD).

The ASSD is defined as:

$$ASSD = \frac{\sum_{t \in T_{o}}^{}{\min_{p \in P_{o}}{|\left| t - p \right||}} + \sum_{p \in P_{o}}^{}{\min_{t \in T_{o}}{|\left| p - t \right||}}}{\left| T_{o} \right| + \left| P_{o} \right|}$$

For each point of the contour, the distance to the other contour is computed. The final value is the average distance computed from both contours. Compared to the HD, it will tend to penalize more a segmentation that is consistently wrong by a small amount than a segmentation that is almost perfect but with a few large errors.

The NSD, meanwhile, is a metric that takes into account the uncertainty of the annotations by first defining “boundary regions” $B_{T}$ and $B_{P}$, which are the set of all pixels within a certain tolerance distance $\delta$ of the contours $T_{o}$ and $P_{o}$:

$$B_{T,\ \delta} = \left\{ b\ ;\min_{t \in T_{o}}\left| b - t \right| < \delta \right\},\ B_{P,\ \delta} = \left\{ b\ ;\min_{p \in P_{o}}{\left| b - p \right| < \delta} \right\}$$

The NSD is then defined as:

$$NSD_{\delta} = \frac{\left| P_{o} \cap B_{T,\ \delta} \right| + \left| T_{o} \cap B_{P,\ \delta} \right|}{\left| P_{o} \right| + \left| T_{o} \right|}$$

This same idea of a “tolerance” could easily be applied to the HD as well by redefining the $h_{\text{TP}}$ and $h_{\text{PT}}$ as:

$$h_{TP,\delta} = \max_{t \in T_{o}}{\min_{b \in B_{P,\delta}}{|\left| t - b \right||}},\ h_{PT,\ \delta} = \max_{p \in P_{o}}{\min_{b \in B_{T,\delta}}{|\left| p - b \right||}}$$

So that the “uncertainty-aware” $HD_{\delta}$ is given by:

$$HD_{\delta} = max(h_{TP,\delta},h_{PT,\delta})$$

4.1.4.3 Segmentation as binary classification

The IoU and DSC metrics, in their “confusion matrix” definitions, are essentially “pixel detection” metrics, which do not take into account the “true negatives” and consider that there is a “positive” class, the foreground, and a “negative” class, the background. But once we have established the confusion matrix, it is also possible to treat the problem as a “binary classification” problem, including the true negatives. The classification metrics defined in 4.1.3 can therefore also be used. The ACC or the MCC, for instance, can easily be computed.

The main difference of this approach is that it does not consider that the “foreground class” is inherently more important than the “background class,” and that the metric should reflect that correctly identifying that a pixel is in the background is just as important as identifying one in the foreground.

A key benefit is that it becomes trivial to extend the metrics to the semantic segmentation case, as adding new classes just means adding new rows and columns to the confusion matrix, but the formula for computing the metrics remains unchanged.

The main drawback of this approach is that, when the segmentation metric is computed on patches extracted from a larger image (for instance, comparing just the masks of a detected object and its matching ground truth object), the result of the metrics becomes highly dependent on the size of the patches, and the amount of background included in it.

Binary segmentation problems are indeed typically not really “two classes” problems, but rather “one class” class problems where we detect one class and put all others in the same bin. The “per-pixel binary classification” approach would therefore only make sense if the problem can actually be defined as a two classes problem.

4.1.5 Metrics for combined tasks

The most common approach for the evaluation of combined tasks (instance segmentation, semantic segmentation, instance classification, instance segmentation and classification) is to combine “basic” metrics to create a new one which attempts to summarize all aspects of the task into a single score.

The Gleason 2019 challenge, for instance, combines a classification $\kappa$ with a micro-averaged and a macro-averaged segmentation F1-Score into a score $S = \kappa + \frac{1}{2}(\mu F_{1} + MF_{1})$. The SNI challenges, meanwhile, combine a simple “binary” DSC (on the whole image and not on separate instances) with a sort of micro-averaged DSC where the numerator and denominator of the DSC are separately aggregated over the set of matching instances. A similar idea is used in the MoNuSeg challenge with the “Aggregated Jaccard Index” (AJI), where the Intersection and the Union are aggregated over the matches.

For the most complex task of instance segmentation and classification, the “Panoptic Quality” (PQ), originally proposed by Kirillov et al. [27] for natural scenes, has recently been introduced to digital pathology by Graham et al. [20].

The PQ considers each class separately. For each class $c$, $T_{c} = \{ t_{i}\}$ is the set of ground truth instances in the class. Given a set of corresponding class predictions $P_{c} = \{ p_{i}\}$, the $PQ_{c}$ is computed in two steps.

First, the matches between ground truth instances and predicted instances are found (using the $\mu_{\text{IoU}} = 0.5$ matching threshold). Using this strict matching rule, each segmented instance in $T_{c}$ and $P_{c}$ can be counted as TP, FP or FN.

Then, the $\mathbf{\text{PQ}}$ of the class $\mathbf{c}$ in the image $\mathbf{i}$ is computed as:

$$PQ_{c,i} = \frac{\sum_{\left( p_{i},t_{j} \right) \in TP}^{}{\text{IoU}\left( p_{i},t_{j} \right)}}{TP + \frac{1}{2}FP + \frac{1}{2}\text{FN}}$$

Which can be decomposed into:

$$RQ_{c,i} = \frac{\text{TP}}{TP + \frac{1}{2}FP + \frac{1}{2}\text{FN}}$$

$$SQ_{c,i} = \frac{\sum_{\left( p_{i},t_{j} \right) \in Matches}^{}{\text{IoU}\left( p_{i},t_{j} \right)}}{\text{TP}}$$

$$PQ_{c,i} = SQ_{c,i} \times RQ_{c,i}$$

With $RQ_{c}$ the “Recognition Quality,” corresponding to the per-object F1-score of the class $c$, and $SQ_{c}$ the “Segmentation Quality,” corresponding to the average IoU of the matching pairs of ground truth and predicted instances. The $\text{RQ}$ is also often referred to as the Detection Quality, and therefore noted as $\text{DQ}$ [43], [20].

4.1.6 Aggregation methods

A critical step to arrive at a final “score” for an algorithm on set of test samples is the aggregation of the per-sample metric(s). Let us therefore go back to the hierarchical representation of digital pathology datasets that we used at the beginning of the chapter (see Figure 4.1).

The aggregation process can happen in different dimensions: across that hierarchical representation, across multiple classes, and across multiple metrics.

4.1.6.1 Hierarchical aggregation

The aggregation across the hierarchical aggregation has two aspects: where are the metrics computed, and how are they averaged?

Object-level metrics would typically be those used in instance segmentation (per-object IoU, HD…). In a “bottom-up” approach, all the object-level measures could be averaged over a patch, then the patch-level measures over a WSI, the WSI-level over a patient, and finally the patient-level measures over the whole dataset. The problem of this approach is that, if the patches are very dissimilar in their object distribution, it may lead to cases where a single error on a patch with few objects is penalized a lot more harshly than an error on a larger patch with a larger population of objects. A more robust approach will therefore generally be to rather aggregate the per-object metrics over a WSI or a patient. It is also possible to directly average the object-level metrics on the entire dataset, skipping intermediate levels entirely. In the digital pathology context, having a patient-level distribution of a metric is however often desirable for a clinically relevant analysis of the results of an algorithm, as the goal will generally be to be able to validate that the performance of an algorithm is good across a wide range of patients. It also makes it possible to use powerful statistical tests to compare different algorithms together, as patients can then be used as paired samples.

Patch-level metrics are very commonly used in all sorts of tasks. It could be a region segmentation metric like the IoU or DSC, an object detection metric like the F1-Score, or a classification metric like the ACC or the MCC. It is once again possible to directly average those results on the entire dataset, or to go through the patient-level first.

Another possibility for many of those metrics is to not compute the metric at the patch level, but to aggregate its base components over a patient or WSI. For instance, a confusion matrix can be built over multiple patches of a WSI, before computing the relevant classification or detection metrics on this aggregated confusion matrix. This reduces the risk of introducing biases from the patch selection and focuses the results on the object of interest of pathology studies: the patient. We therefore find here again the same choices of micro- and macro-averaging that were described in the classification metrics.

4.1.6.2 Multiclass aggregation

When multiple classes are present, detection and segmentation metrics are generally first computed independently for each class before being aggregated together. This aggregation can also happen at any point in the hierarchy of the dataset. For the PQ metric, for instance, some aggregate the $PQ_{c,i}$ at the image patch level as $PQ_{i} = \frac{1}{m}\sum_{c}^{}{PQ_{c,i}}$, then find the global $PQ = \frac{1}{n}\sum_{i}^{}{PQ_{i}}$ [43], [20], [18], while others averages each $PQ_{c}$ separately over all patches as $PQ_{c} = \frac{1}{n}\sum_{i}^{}{PQ_{c,i}}$, then define the global $PQ = \frac{1}{m}\sum_{c}^{}{PQ_{c}}$ [19] (with $n$ the number of images and $m$ the number of classes). As always, the micro- or the macro-averaging approaches can also be considered. In a micro-averaging approach, the metric must first be decomposed into its base components, which are aggregated separately before computing the metric. For most classification or detection metrics, the base components would be the elements of the confusion matrix. For combined metrics such as the PQ or some custom score, then a choice must be made as to what constitutes a “base component.” Practically, almost all publications and challenges use a macro-averaging approach.

4.1.6.3 Multi-metrics aggregation

Single metrics are not capable of fully representing the capabilities of a given model or algorithm. This is true even for simple tasks: as we have seen, different metrics have distinct behaviours and biases, and may therefore miss some important insights about a particular algorithm. A clear example would be the IoU and the HD for segmentation metrics, which give very different information about the same task. As noted in section 4.1.5, combined tasks are often evaluated by creating complex metrics that combine basic detection, classification and segmentation metrics in some ways. Another approach, however, is to compute and report those simple metrics independently, to provide more detailed information about the predictive performance. Computing multiple metrics, however, means that algorithms cannot necessarily be ranked based on a single score, thus making the overall ranking (and therefore the choice of the “best method”) more difficult to obtain.

A possible solution is to compute the ranks separately for the different metrics, then to combine them with, for instance, the “sum of ranks.” As we will discuss below in section 4.4.7, this approach comes with its own pitfalls and challenges.

4.2 Metrics in digital pathology challenges

We examine here the evaluation metrics used in the challenges previously presented in Chapter 3, to get a sense of the common choices made by challenge organizers when determining the evaluation process. We look here mainly at the “basic” metrics (i.e. pure detection, classification or segmentation), and we will note when those metrics are included as part of a more complex evaluation score.

4.2.1 Detection tasks

A summary of the evaluation metrics used in digital pathology detection challenges can be found in Table 4.3. Almost all challenges used the F1-Score as the primary metric for ranking the participants’ submissions. Some challenges reported the PRE and REC separately. The only “challenge” to use varying-threshold metrics is NuCLS 2021. This dataset, however, is presented more as a benchmark for future usage by researchers than as a challenge, despite being listed on the grand-challenge.org website. In the baseline results reported by Amgad et al. [3], the AP using a fixed matching threshold of 0.5 on the IoU, and the mAP using varying matching thresholds from 0.5 to 0.95 are used, so that both the confidence threshold and the IoU threshold are varying in the metric. DigestPath ranks three different metrics separately, then use the average rank as the overall rank for each competing team.

:Table 4.3. Summary of the metric(s) used in digital pathology detection challenges. Bolded values are used for the final ranking.

4.2.2 Classification challenges

A summary of the metrics used in digital pathology classification challenges is presented in Table 4.4. Compared to the detection tasks, there is a lot more diversity in the choices made by challenge organisers. All of the multi-class tasks in these challenges (except NuCLS 2021) have some form of ordering present in their categories. For the binary classification tasks, many challenges are assessed with a positive-class only metric, as shown with CAMELYON 2016 and PatchCamelyon 2019 (AUROC of the metastasis class), DigestPath 2019 (AUROC of the malignant class), HeroHE 2019 (F1-Score, AUROC, SEN and PRE of the HER2-positive class) and PAIP 2020 (F1-Score of the MSI-High class). In the multi-class case, the TUPAC 2016 and PANDA 2020 evaluations use the weighted quadratic kappa, while in MITOS-ATYPIA 2014 a “penalty” system is used for errors of more than one class. In several challenges, however, this ordering is not taken into account at all. This is the case with the Brain Tumour DP 2014, BIOIMAGING 2015 and BACH 2018 challenges, where the simple accuracy is used, and for the C-NMC 2019 challenge, where a weighted macro-averaged $sF1$ is preferred.

These classification tasks make the boundaries between detection, classification and regression tasks sometimes difficult to determine. In HeroHE 2020, for instance, the ranking of the algorithm is based on the “F1-Score of the positive class,” which is a typical detection metric, but the AUROC of the positive class is also computed, which takes the “true negatives” into account and is typically a classification metric. The tasks that are explicitly about predicting a “score” all attempt to incorporate the notion of “distance” to the ground truth into their metric, which brings them closer to a regression task. The use of the quadratic kappa is also clearly related to its popularity in pathology and medical sciences in general, as it makes the results more relatable for medical experts. The danger of that particular metric, however, is that it may give a false sense of “interpretability,” as the number of classes and the target class distribution have a large effect on the perceived performance of an algorithm using that metric.

4.2.3 Segmentation challenges

Almost all segmentation challenge use overlap-based metrics (either the IoU or the DSC) as their main metric for ranking participating algorithms, as shown in Table 4.5. The only challenge to really use a distance-based metric as part of the ranking was the GlaS 2015 challenge, which ranked the per-object average HD and the per-object average DSC (as well as the detection F1 score) separately. PAIP 2021 used the HD as a matching criterion but did not use it for the ranking.

The BACH 2018 challenge is an outlier in this, as they use a custom score that loosely corresponds to a “per-pixel” accuracy, with ordered classes so that “larger” errors are penalized more.

Table 4.5. Summary of the metric(s) used in digital pathology segmentation challenges.

4.3 State-of-the-art of the analyses of metrics

There is a growing amount of literature on the topic of evaluation metrics, within the context of biomedical imaging or in general. Reinke et al. [40] thoroughly examine the pitfalls and limitations of common image analysis metrics in the context of biomedical imaging. Luque et al. [32] examine the impact of class imbalance in binary classification metrics. Their methods and conclusions will be explored more deeply in section 4.3.3, and we extend their analysis in section 4.4.2. Chicco et al. [7] compare the MCC to several other binary classification metrics and find it generally more reliable. Delgado et al. [9] describes the limitations of Cohen’s kappa compared to the Matthews correlation coefficient, and some of their results are discussed below in section 4.3.2. Grandini et al [21] propose an overview of multi-class classification metrics, a topic generally less explored in the literature. Padilla et al. [38], [39] compare object detection metrics and note the importance of precisely defining the metrics used, as the AP, for instance, which can be computed in several ways.

Oksuz et al. [37] focus on object detection imbalance problems. They review the different sampling methods that can help to counteract the foreground-background imbalance to train machine learning algorithms. They identify four classes of such methods: hard sampling (where a subset of positive and negative examples with desired proportions is selected from the whole set), soft sampling (where the samples are weighted so that the background class samples contribution to the loss is diminished), sampling-free (where the architecture of the network, or the pipeline, are adapted to address the imbalance directly in the training with no particular sampling heuristic), and generative (where data augmentation is used also to correct the imbalance by increasing the minority class samples). In section 4.4.1, we also explore the effect of foreground-background imbalance on the evaluation process.

4.3.1 Relations between classification and detection metrics

Many evaluation metrics are related or take similar forms. The most obvious relationship is between the different forms of the F1-Score, which can be used as a detection measure, a classification measure, and a segmentation measure (as the DSC).

A somewhat less obvious relation exists between the F1-Score and unweighted Cohen’s kappa, as identified by Zijdenbos et al. [45]. In a binary classification where we consider a “positive” and a “negative” class, we have the confusion matrix:

$$\begin{pmatrix} \text{TN} & \text{FP} \\ \text{FN} & \text{TP} \\ \end{pmatrix}$$

And $\kappa_{U}$ resolves to:

$$\kappa_{U} = \frac{2 \times \left( \left( TN \times TP \right) - \left( FN \times FP \right) \right)}{\left( TN + FP \right)\left( FP + TP \right) + \left( TN + FN \right)\left( FN + TP \right)}$$

In a detection problem, TN is uncountable, but it can generally be assumed to verify:

$$TN \gg TP,\ FP,\ FN$$

With this assumption, $\kappa_{U}$ for detection can be simplified to:

$$\kappa_{U,d} = \frac{2 \times TN \times TP}{\text{TN}\left( FP + TP \right) + TN(FN + TP)} = \frac{2 \times TP}{2 \times TP + FP + FN} = \frac{2 \times PRE \times REC}{PRE + REC}\$$

Which is the harmonic mean of the PRE and REC, i.e. the detection F1-Score.

Meanwhile, the same exercise applied to the MCC leads us to a “detection” version of:

$$MCC_{d} = \frac{\left( TP \times TN \right) - \left( FP \times FN \right)}{\sqrt{\left( TP + FP \right)\left( TP + FN \right)\left( TN + FP \right)\left( TN + FN \right)}} = \frac{TP \times TN}{\sqrt{\left( TP + FP \right)\left( TP + FN \right)TN^{2}}}$$

$$MCC_{d} = \frac{\text{TP}}{\sqrt{\left( TP + FP \right)\left( TP + FN \right)}} = \sqrt{PRE \times REC}$$

Which is the geometric mean of the PRE and REC (note that this is different from the classification GM we used before, which was the geometric mean of the REC and SPE, not REC and PRE).

While the harmonic mean is typically the preferred averaging methods for ratios (such as the PRE and REC), the geometric mean will penalize less harshly small values for one of the averaged elements. For instance, for $PRE = 0.1$ and $REC = 0.9$, the geometric mean will still be equal to 0.3 while the harmonic mean will drop to 0.18.

4.3.2 Interpretation and problems of Cohen’s kappa

Cohen’s kappa is very popular in digital pathology. It is often associated to an “interpretation” scale such as this one [1]:

• < 0: less disagreement than random chance.

• 0.00-0.20: slight agreement

• 0.21-0.40: fair agreement

• 0.41-0.60: moderate agreement

• 0.61-0.80: substantial agreement

• 0.81-1.00: almost perfect agreement

There is however some debate on the validity of these interpretation, with some authors suggesting that, for medical research in particular, a less optimistic view of the $\kappa$ value should be used [34]. It should also be noted that the kappa values are highly dependent on the weights used, and that this interpretation can therefore be highly misleading if a weighted kappa is used. For instance, in McLean et al.’s study of interobserver agreement in Gleason scoring [35], looking at the unweighted kappa would lead to an interpretation of “slight agreement,” the linear kappa to “slight to fair agreement,” and the quadratic kappa to “fair to moderate agreement.”

This can be problematic when, for instance, Humphrey et al. [22] compare the average unweighted kappa of 0.435 of general pathologists on Gleason scoring from a study [2] with the 0.6-0.7 weighted kappa¹¹ for urologic pathologists from another [1] to find an “enhanced interobserver agreement” for the specialists, instead of using the 0.47-0.64 unweighted kappa which was also reported in the same study, marking an improvement that is still very substantial but not quite as extreme as Humphrey’s paper suggests.

As noted by Delgado et al. [9], in some circumstances Cohen’s kappa can have an inconsistent behaviour, where a worse agreement leads to better results. In general, this can be demonstrated using the $Z_{A}$ family of imbalanced confusion matrices, defined by:

$$Z_{A} = \ \begin{pmatrix} 1 & \cdots & A \\ \vdots & \ddots & \vdots \\ 1 & \cdots & 1 \\ \end{pmatrix}$$

So that $Z_{A,ij} = 1$ everywhere except for $Z_{A,0m} = A$, with $A \geq 0$. Intuitively, increasing $A$ means that at the same time we increase the imbalance, and we reduce the agreement. While we would expect in that case $\kappa$ to be monotically decreasing, Delgado et al. show that it is not the case, and that after the inflexion point at $A = 1 + m\sqrt{m\left( m - 1 \right)}$ the $\kappa$ values start to increase. This can be verified experimentally, as we can see in Figure 4.7.

However, this is unlikely to happen, and in practice the $\kappa_{U}$ in general agrees well with the MCC, as shown in our experiments in section 4.4.5.

4.3.3 Imbalanced datasets in classification tasks

The question of how classification metrics behave with imbalanced datasets was systematically explored by Luque et al. [32] for binary classification. In this section, we will briefly explain their method and their results. In section 4.4.2, we will extend their analysis to some metrics that they did not include, and to multi-class problems.

To analyse the metrics’ behaviour, the first step is to parametrize the confusion matrix so that it becomes a function of the sensitivity of each class, and of the class imbalance. Let $\lambda_{c}$ be the sensitivity of class $c$ and $\pi_{c}$ the proportion of the dataset that is of class $c$, so that $\sum_{c}^{}\pi_{c} = 1$ and $0 \leq \lambda_{c} \leq 1$. As it is a binary classification problem, Luque et al. further define one of the classes as the positive class, and the other as the negative class. We therefore have $\pi_{P} = 1 - \pi_{N}$, and we can then define the balance parameter $\delta = 2\pi_{P} - 1 = 1 - 2\pi_{N}$, so that $\delta = 0$ corresponds to balanced data, $\delta = - 1$ to an “all-negative” dataset, and $\delta = 1$ to an “all-positive” dataset. The binary confusion matrix can be formulated as:

$$CM = \ N\begin{pmatrix} \lambda_{N}\frac{1 - \delta}{2} & \left( 1 - \lambda_{N} \right)\frac{1 - \delta}{2} \\ \left( 1 - \lambda_{P} \right)\frac{1 + \delta}{2} & \lambda_{P}\frac{1 + \delta}{2} \\ \end{pmatrix}$$

With $N$ the total number of samples in the dataset.

The study then shows how different classification metrics behave for different values of $(\delta,\ \lambda_{P},\ \lambda_{N})$. The bias of a metric due to class imbalance can be characterized by looking at the difference between the metric measured for $(\delta,\ \lambda_{P},\ \lambda_{N})$ with the same metric measured at $(0,\lambda_{P},\lambda_{N})$ (i.e. for the same per-class sensitivities in a balanced dataset). Figure 4.8 shows for instance how the ACC behaves with imbalanced datasets. The interpretation of the top row is that, as the balance skews towards one of the two classes, only the performance of that class influences the result of the metric, as the isocontours (shown at 0.1 intervals in the figures) become parallel to either the horizontal or the vertical axis. In the bottom row, we can visualise the bias that it introduces for medium ($\delta = \pm 0.5$) and very high ($\delta = \pm 0.99$) imbalance according to per-class sensitivities. By contrast, the unbiased behaviour of the GM is shown in Figure 4.9.

In Table 4.6, we reproduce the range of bias found in Luque et al. [32] for several classification metrics. It should be noted that, for the MCC, the metric is “normalized” with $MCC_{n} = \frac{1 + MCC}{2}$ so that its value is limited to the same [0, 1] range as the other metrics, to make the comparisons more accurate.

A limitation of their study is that the F1-Score that they used is the $F1_{P}$, the “per-class” F1-score of the “positive” class. This is a very common choice in binary classification problems, but it makes the comparison with the $\text{ACC}$, $\text{GM}$ and $\text{MCC}$ slightly uneven, as the $F1_{P}$ is a class-specific metric while the others are all global metrics. In fact, the $F1_{P}$ here is not really a binary classification metric, but rather a detection metric, as it does not consider a “two classes” problem but rather a “one class-vs-all” problem.

4.3.4 Statistical tests

When looking at results from different algorithms on a given task, it can be very difficult to determine if the difference in metrics between methods can be considered “significant.” Part of the question is related to the various sources of uncertainty that come with the annotation process, and will be discussed more in the following chapters, such as interobserver variability and imperfect annotations. Even if we assume “perfect” annotations, however, it is still necessary to determine if a difference in results may be attributed simply to random sampling, or if we can safely reject that hypothesis.

Let’s consider datasets $D_{i}$, which are sampled from a larger, potentially infinite population $P$, and a performance metric $M(D_{i},\ A_{j})$ computed on the output of an algorithm $A_{j}$ for the dataset $D_{i}$. Ideally, the null hypothesis would be that the samples of $M(D_{i},A_{j})$ for the algorithms that we want to compare come from the same distribution.

Practically, however, we generally have a single test dataset which we want to use for our comparison. Different statistical tests have been proposed to provide some confidence in the detection of significant differences in algorithms, depending on the situation. Dietterich [11], for instance, proposes to use the McNemar test for comparing the performances of two classifiers. McNemar’s test is based on the contingency table between two classifiers:

The statistic computed is:

$$\frac{\left( \left| n_{01} - n_{10} \right| - 1 \right)^{2}}{n_{01} + n_{10}}\ \sim\ \chi_{1}^{2}$$

It follows the $\chi^{2}$ distribution with one degree of freedom under the null hypothesis.

In general, however, the goal is to be able to compare multiple algorithms based on arbitrary evaluation metrics and the same test set. The recommended test for comparing multiple algorithms on the same data is the Friedman non-parametric test [10]–[31]. The Friedman test tells if the null hypothesis that all the tested and dependent samples (in this case the values of the metrics) come from the same distribution can be rejected. To determine the pairwise significance of the differences between algorithms, a post hoc test must be conducted. The Nemenyi post hoc is a recommended choice [10], although some suggest replacing the post hoc by pairwise Wilcoxon signed-rank tests [4]. The Friedman and Nemenyi tests were used in the statistical score we used in our experiments on imperfect annotations [13], [14] (see Chapter 5).

The Friedman test is the non-parametric version of the repeated-measures ANOVA [10]. If we consider a test set $T$ that can be divided into $n$ non-overlapping subsets $S_{i},\ i \in \{ 1,\ 2,\ \ldots,\ n\}$ (for instance, $S_{i}$ may contain all the patches from a patient in a patch classification task, or all the objects from a patch in a detection or instance segmentation task), so that a performance metric $M$ for the $k$ algorithms $A_{j},\ j \in \{ 1,\ 2,\ \ldots,\ k\}$ can be computed as $m_{\text{ij}} = M(S_{i},A_{j})$.

First, the rank of each algorithm is computed on each subset: $r_{\text{ij}} = rank(S_{i},\ A_{j})$, where $\text{rank}\left( S_{i},A_{j} \right) = 1$ for the algorithm where $M\left( S_{i},A_{j} \right) = \min_{j \in \lbrack 1,k\rbrack}{M(S_{i},A_{j})}$. For each algorithm, the ranks average $R_{j}$ is then computed: $R_{j} = \frac{1}{n}\sum_{i \in \left\lbrack 1,n \right\rbrack}^{}r_{\text{ij}}$. If all algorithms are equivalent, then their average ranks should be equal. The statistic used to test if this null hypothesis can be rejected is [10]:

$$S = \frac{12n}{k\left( k + 1 \right)}\left\lbrack \sum_{j = 1}^{k}R_{j}^{2} - \frac{k\left( k + 1 \right)^{2}}{4} \right\rbrack$$

Under the null hypothesis, this statistic follows a $\chi^{2}$ distribution with $k - 1$ degrees of freedom. If the null hypothesis is rejected according to a p-value threshold (typically$\ p < 0.05$), the Nemenyi post hoc can be used to determine which algorithms are significantly different from each other. The Nemenyi test is based on the average rank difference, computed during the Friedman test, between pairs of algorithms. The ranks depend not only on the performance of the two algorithms being considered, but also on the performance of all the other algorithms compared in the Friedman test. This can be problematic, as it can lead to situations where two algorithms are considered to be significantly different if compared as part of one set of algorithms, and not significant if compared as part of another. For this reason, some recommend using the Wilcoxon signed-rank test instead of the Nemenyi for the pairwise analysis (adjusting the significance level to correct for the Type-I error, for instance by dividing the confidence level by the number of compared methods minus one) [10], [4].

Practically, this type of analysis is unfortunately rarely used when comparing deep learning algorithms. At best, challenges will sometimes provide box-plots of the distributions of results for the top teams (as for instance in the ACDC@LungHP 2019 [29] or in the PAIP 2019 [26] challenges), or 95% confidence intervals alongside the average results (as in MoNuSeg 2018 [28]).

4.4 Experiments and original analyses

In this section, we present several experiments that we performed in order to extend these analyses, or to explore more thoroughly some of the most interesting aspects of the behaviour of the metrics. Specifically, we will look at the effects of different types of class imbalance (4.4.1, 4.4.2), the consequences of the fuzzy boundary between detection and classification tasks (4.4.3, 4.4.4), the agreement between the scores given by different classification metrics for the same set of predictions (4.4.5), the biases of common segmentation metrics (4.4.6), the difficulties in multi-metrics aggregation (4.4.7) and, finally, at why the Panoptic Quality should be avoided for instance segmentation and classification problems (4.4.8).

4.4.1 Foreground-background imbalance in detection metrics

As detection tasks are “one class versus all” problems, they are often associated with a very large “foreground-background” imbalance, meaning that the objects of interest are sparsely distributed in the images. This is particularly true in the most popular detection task in digital pathology: mitosis detection.

In the MITOS12 mitosis detection challenge, for instance, the training dataset consists of 35 image patches, each with dimensions of 2048×2048 pixels. There are 226 mitoses in this training set, occupying a total mitosis area of around 135.000 pixels, which corresponds to about 0.09% of the whole area.

The detection process, illustrated in Figure 4.10, typically involves three main phases: first finding “candidate” regions with a “candidate selector,” then classifying each candidate as a positive or negative detection with a “candidate classifier,” then finally merging overlapping regions into detected object instances.

For the first phase, there are two main strategies: either create a very large number of candidates by densely sampling the space of possible bounding boxes, or create a restricted set of candidates, using the image features already in the first phase to reject “obvious” negatives. The main advantage of the first approach is that it generally ensures that no positive object is accidentally rejected in the first phase, and therefore not even presented to the candidate classifier. The main advantage of the second is that it makes the task of the candidate classifier a lot easier.

The overall performance of the whole detection pipeline, however, is not just affected by the performance of the selector, classifier and merger, but also, as in all machine learning tasks, by the selection of the possible data used for the evaluation. In the MITOS12 dataset, the 35 image patches correspond to small portions of 5 WSI, selected because of the presence of several mitosis. When the choice of regions of interest is not random, the distribution of positive & negatives changes with the size of the selected region. Taking a larger region “around” the mitosis that first attracted the attention of the pathologist is likely to increase the proportion of negatives, while taking a narrower region would increase the proportion of positives. Given the sparsity of the objects of interest, the effect can quickly become large.

To quickly simulate how this may affect the results of a detection pipeline, we define a detector with the following characteristics:

• The “candidate selection” step filters out 99.99% of the possible candidates at the pixel level, assuming that we define the set of possible candidates as same-sized patches centred on each pixel of the image (so there are as many possible candidates before selection as there are pixels in the image). This assumes that most possible candidates will be trivially dismissed, which is not unrealistic in a mitosis detection problem, as all pixels that are relatively lighter are certain not to be part of a mitosis (see Figure 4.11).

• The “candidate classifier” step has a 99% specificity and 75% sensitivity rate.

We then look at three scenarios. In the first one, we take the characteristics of the MITOS12 dataset (226 mitosis occupying 0.09% of the total pixel area). In the second, we imagine a more restricted set where a smaller region of interest was considered. We look at what happens if, when restricting the total area by half, we keep about 75% of the mitosis. In the third scenario, we do the opposite and double the total area, and simulate that this includes 150% of the original amount of mitosis. The results presented in Table 4.7 show that, while the recall in all cases remains the same (as it is equal to the sensitivity that we fixed as a parameter of the pipeline), the precision varies as the foreground-background imbalance changes. As proportionally more negative examples are shown, the number of false positives can only increase and the number of true positives can only decrease, so the precision and F1-score can only get worse.

Of course, as long as methods are compared on exactly the same test set, a ranking of the methods based on the F1-Score should still be representative of their relative performance. However, the characteristics of the dataset are still important to keep in mind when interpreting the results, particularly when the same task is measured on different datasets. The MITOS-ATYPIA-14 dataset, for instance, has a much lower ratio of mitosis to total area, with mitosis representing 0.02% of the total area of the patches. A detection pipeline with the same behaviour as the one we simulated on MITOS12 would see its precision fall from 0.55 to 0.18, and its F1-score from 0.63 to 0.29 just by this change of distribution.

It is tempting to see that as the main explanation for the difference in results observed between the two challenges, as the top result for MITOS12 was a 0.78 F1-score, while it was 0.36 for MITOS-ATYPIA-14. There were, however, less participants in the mitosis detection part of the challenge in the 2014 version, and the MITOS12 challenge was also designed with a problematic train/test split, with patches extracted from the same WSI appearing in both parts of the dataset. The difference in the mitosis density is very likely, however, to be a contributing factor.

4.4.2 Imbalanced datasets in classification tasks

4.4.2.1 Extension to other metrics

As noted in section 4.3.3, the analysis of Luque et al. [32] on the effects of imbalanced datasets in classification tasks uses a “detection” F1-Score, which is not a global metric for a classification problem. We therefore reproduce the experiment with the two version of the macro-averaged F1-score, the $sF1$ and the $hF1$. We also add the $\kappa_{\text{Un}}$, normalized similarly as the $MCC_{n}$ (cf. section 4.3.3), for completeness’ sake. Our results are presented in Table 4.8, and a comparison between the biases of the $F1_{P}$, the $hF1$, the $sF1$ and the $MCC_{n}$ is shown in Figure 4.12.

As we can see, the shape of the bias is very similar between the $hF1$ and the $MCC_{n}$. While the $F1_{P}$ is clearly extremely biased with imbalanced datasets and should generally be avoided as a binary classification metric, the use of a macro-averaged version of the metric reduces its bias quite significantly and puts it in the same range as the $MCC$ (see Table 4.6) for the $sF1$, and even lower for the $hF1$. The latter therefore appears to be more robust to class imbalance than the MCC.

4.4.2.2 Extension to multi-class problems

In a multi-class problem, the notion of a “positive” and “negative” class has to be removed. The generic parametrized confusion matrix is therefore, for $m$ classes:

$$CM = N\begin{pmatrix} \lambda_{11}\pi_{1} & \ldots & \lambda_{1\left( m - 1 \right)}\pi_{1} \\ \vdots & \lambda_{\text{ii}}\pi_{i} & \lambda_{i\left( m - 1 \right)}\pi_{i} \\ \lambda_{\left( m - 1 \right)0}\pi_{m - 1} & \lambda_{\left( m - 1 \right)i}\pi_{m - 1} & \lambda_{\left( m - 1 \right)\left( m - 1 \right)}\pi_{m - 1} \\ \end{pmatrix}$$

Where $N$ is the total number of samples, $\pi_{c}$ is the proportion of class $c$ in the dataset, and $\lambda_{\text{ij}}$ is the proportion of class $i$ samples that were misclassified as class $j$, so that $\lambda_{\text{ii}} = SEN_{i}$ and $\sum_{j}^{}\lambda_{\text{ij}} = 1$.

This leads to a parametric space with a very high dimensionality, that is quite complex to explore. For a 3-class problem, we would have two “balance” parameters $\pi_{1}$ and $\pi_{2}$ (which set $\pi_{3} = 1 - \pi_{1} - \pi_{2}$) and six “error distribution” parameters $\lambda_{11},\ \lambda_{12},\ \lambda_{21},\lambda_{22},\ \lambda_{31},\ \lambda_{32}$ (which set $\lambda_{13} = 1 - \lambda_{11} - \lambda_{12}$, etc.). In general, for a $m$ class problem, we would have $m - 1$ balance parameters and $m(m - 1)$ error distribution parameters.

To reduce this dimensionality, we define a single balance parameter $\mathbf{\beta}$ that will vary from 0 (balanced) to 1 (extremely imbalanced). We also define $Β\left( \beta,m \right) = \beta + \frac{1 - \beta}{m}$, and then recursively $\pi_{1} = Β(\beta,m)$ and $\pi_{i} = Β\left( \beta,m - \left( i - 1 \right) \right)\prod_{j = 1}^{i - 1}\left( 1 - Β\left( \beta,m - \left( j - 1 \right) \right) \right)$. An example of the class distributions yielded by this method for a 4-class problem and different $\beta$ values is shown in Figure 4.13.

For the error distribution, we consider three scenarios that allow us to only have as parameters the $m$ class sensitivity values:

• The evenly distributed error (ED) scenario, where we set $\lambda_{\text{ij}} = \frac{1 - \lambda_{\text{ii}}}{m - 1}\ \forall j \neq i$.

• The randomly distributed error (RD) scenario, where we first generate a random “error likelihood” matrix $e$ so that $e_{\text{ii}} = 0$ and $\sum_{j}^{}e_{\text{ij}} = 1$, and the $\lambda_{\text{ij}}$ are then simply computed as $\lambda_{\text{ij}} = e_{\text{ij}}\left( 1 - \lambda_{\text{ii}} \right)\ \forall\ i \neq j$. The error matrix needs to be computed first so that the same “error likelihood” is used when exploring the space of $\lambda_{\text{ii}}$ values.

• The majority bias error (MB) scenario, where the errors are distributed more towards the majority class, so that $\lambda_{\text{ij}} = (1 - \lambda_{\text{ii}})\frac{\pi_{j}}{\sum_{k \neq i}^{}\pi_{k}}$.

The three scenarios are illustrated in Figure 4.14. The biases related to the class imbalance are reported in Table 4.9 for $m = 3$ and $m = 4$, in the three error distribution scenarios. The imbalance bias for the $ACC$ and the $hF1$ increases as the number of classes increases. For the $MCC_{n}$, the main effect seems to be that the negative bias (i.e. the imbalanced result is worse than the balanced result at the same sensitivity values) becomes more important than the positive bias, so the $MCC_{n}$ in general will be lower in a multi-class, imbalanced dataset. The $sF1$ keeps a relatively constant bias. The $\kappa_{\text{Un}}$ sees the range of its bias reduced, but only on the positive side. Finally, the GM remains unbiased.

4.4.2.3 Normalized confusion matrix to counter class imbalance

To counteract the bias induced by class imbalance, the most obvious solution is to compute the metrics on the balanced (or normalized) confusion matrix (NCM). The NCM is computed by simply dividing each element by the sum of the values on the same row:

$$NCM_{\text{ij}} = \frac{CM_{\text{ij}}}{\sum_{k}^{}{CM_{\text{ik}}}}$$

This acts as a “resampling” of the data so that each class has the same number of samples, while keeping the per-class sensitivity values intact. This solution, however, is not without its own drawbacks. To illustrate these, we need to use some real data. The results of the MoNuSAC 2020 challenge can be used in this case: as the prediction maps of four teams are available, it is possible to compute alternate metrics to those reported in the challenge results.

The MoNuSAC challenge is a nuclei instance segmentation and classification challenge. We will focus here on the classification part of the challenge, which has four classes (epithelial, neutrophil, lymphocyte and macrophage). From the published prediction maps, we can compute the classification confusion matrices (based on the matching pairs of detected nuclei, which will therefore vary between the teams), shown in Table 4.10, and the number of per-class TP, FP and FN for the detection task, shown in Table 4.11. A full description of the MoNuSAC dataset is given in Annex A.