Measuring Diagnostic Test Performance Using Imperfect Reference Tests: A Partial Identification Approach

Measuring Diagnostic Test Performance Using Imperfect Reference Tests: A Partial Identification Approach

\

KEYWORDS: Sensitivity, specificity, partial identification, moment inequalities, gold standard bias

\

JEL classification codes: C14, I10

\thispagestyle{empty} \setcounter{page}{1}

Introduction

Diagnostic tests are indispensable in modern clinical decision making. As they are almost never perfect, evaluation of test performance is a common research goal. Test performance studies seek to quantify test accuracy predominantly in the form of sensitivity and specificity, also referred to as performance measures or operating characteristics. Sensitivity (true positive rate) is the probability that a test will return a positive result for an individual who truly has the underlying condition, while specificity (true negative rate) is the probability that a test will produce a negative result for an individual who does not have the underlying condition. Equivalently, one can measure false positive and false negative rates. False negative rate and sensitivity sum to unity, as do specificity and the false positive rate.

Determining sensitivity and specificity for a diagnostic test of interest, referred to as an index test, requires knowledge of the true health status for all participants in the study. The true health status is most often unobservable, so a reference test is commonly used in lieu of it. However, such tests are rarely perfect themselves. When the reference is imperfect, conventional studies only identify “apparent” sensitivity and specificity, or the so-called rates of positive and negative agreement with the reference.\footnote{“Apparent” false negative rate and “apparent” sensitivity sum to unity. Similarly, “apparent” false positive rate and “apparent” specificity sum to unity.} They measure performance with respect to the reference test and not true performance. Hence, “apparent” parameters are typically not of interest. Moreover, the true performance measures are usually only partially identified, as shown in (ref). In other words, there exists a set of parameter values that are consistent with the observed data, called the identified set. The smallest such set under maintained assumptions, or the set that exhausts all information from the data, is known as the sharp identified set. This paper addresses the issue of finding, estimating and doing inference on the points in the sharp identified set for sensitivity and specificity, or equivalently false negative and positive rates, under standard assumptions used in the literature.

I first characterize the sharp joint identified set for the true performance measures without imposing any assumptions on the latent statistical dependence between the index and reference tests conditional on health status, assuming exact or approximate knowledge of the reference test characteristics. The set is a line segment or a union of line segments in $[0,1]^2$, in contrast to rectangular sets following from comparable existing bounds (thibodeau1981evaluating, emerson2018biomarker). The proposed framework allows researchers to layer on additional assumptions regarding the latent dependence to further reduce the size of the set. This is demonstrated through a formalization of an informally stated assumption from the literature which is plausible when the two tests are physiologically related.

Sensitivity and specificity are frequently used to obtain other policy-relevant parameters. I present how the derived identified sets may be utilized to sharply bound prevalence, or the population rate of illness, in populations screened by the index test. Bounds may be markedly narrower than those implied by existing comparable methods, owing to the specific shape of the identified sets. Implied bounds on predictive values, i.e. probabilities that a patient is sick conditional on observing a test result, are discussed in Appendix (ref).

The FDA Statistical Guidance on Reporting Results Evaluating Diagnostic Tests\footnote{Link: \href{https://www.fda.gov/media/71147/download}{https://www.fda.gov/media/71147/download} (Last accessed: 12/25/2022)} requires diagnostic performance studies to report confidence intervals for index test sensitivity and specificity to quantify the statistical uncertainty in the estimates. To conform to the practice, I construct confidence sets for points in the identified set using an inference method based on moment inequalities (romano2014practical). The confidence sets are uniformly consistent in level over a large family of permissible distributions relevant in the application. Namely, they asymptotically cover all points in the identified set uniformly over the family of population distributions with probability of at least $1-\alpha$, where $\alpha$ is the chosen significance level.

Diagnostic test performance studies for rapid COVID-19 tests have a mandated RT-PCR reference test which is known to produce false negative results, and thus pertain to the setting analyzed in the paper.\footnote{Link: \href{https://www.fda.gov/media/137907/download}{https://www.fda.gov/media/137907/download} (Last accessed: 12/25/2022)} fitzpatrick2021buyer emphasize that the false negative rate of the ubiquitous Abbott BinaxNOW COVID-19 Ag2 CARD rapid antigen test may be substantially understated by the reported “apparent” analog due to imperfect reference tests.\footnote{The test held $75\%$ of the COVID-19 antigen test market share in the United States, according to Abbott Laboratories CEO Robert Ford on Q3 2021 Results - Earnings Call Transcript.}

Application of the method to the data from the original Emergency Use Authorization (EUA) performance study, as well as an independent study by shah2021performance bolsters this claim and reveals that bounds can be very informative. Depending on interpretation, the results from both studies suggest that the test may not satisfy the initial FDA requirement for EUA of at least $80\%$ estimated sensitivity, despite fulfilling the criterion of high “apparent” sensitivity, implying the need for alternative testing protocols. Moreover, the estimated false negative rates for symptomatic and asymptomatic patients are up to $3.17$ and $4.59$ times higher than the frequently cited “apparent” false negative rate, warranting further attention. Comparison with existing bounds reveals that the proposed method can provide significant reductions in the size of the identified set for operating characteristics, and consequently in the width of implied bounds on prevalence when the test is used for screening.

The methodological framework developed in the paper offers solutions to two issues in the current research practice guidelines set forth by the FDA Statistical Guidance, as explained by (ref): 1) Inability to measure true test performance in common settings; 2) Inability to compare index and reference test performance. It also addresses the concerns raised in boyko1988reference: “When two tests are strongly suspected of being conditionally dependent, then the performance of one of these tests should probably not be compared with that of the other, unless better methods are developed to sort out the degree of bias caused by reference test errors in the presence of conditional dependence."

Provided replication files allow researchers to directly utilize the findings of the paper to obtain estimates and confidence sets in their own work.\footnote{Available from: \href{https://github.com/obradovicfilip/bounding_test_performance}{https://github.com/obradovicfilip/bounding_test_performance}} Since the method does not require any changes to the data-collection process of standard studies, it can also be readily applied to estimate test performance based on published data, as demonstrated by the application section of the paper.

Broader applicability of the approach is discussed in (ref). It can be used to study features of the joint distribution between a binary outcome and a binary latent variable measured with an imperfect proxy. Illustrative examples include variables such as program participation as indicated by a survey response, or race, imputed using the Bayesian Improved Surname Geocoding (BISG) algorithm. The method is appealing when validation studies measuring proxy misclassification rates exist. The bounds also readily apply whenever one wishes to learn performance of a binary classifier by comparing it to another imperfect classifier or label, rather than the ground truth, which is common in satellite imagery and other remote sensing applications.

Related Literature

gart1966comparison2, staquet1981methodology, and zhou2009statistical show that if the reference and index tests are statistically independent conditional on the true health status, index test sensitivity and specificity are point identified, assuming exactly known reference test performance measures. (see also hui1980estimating) However, vacek1985effect, valenstein1990evaluating, hui1998evaluation and emerson2018biomarker elaborate that conditional independence may frequently be untenable. A salient case is when the two tests are physiologically related, such as when they rely on the same type of sample (e.g. nasal swab or capillary blood) or measure the same quantities (e.g. antibody reaction to tuberculin).

Tests are generally expected to be dependent, but the dependence structure is latent since true status is unobservable. Several authors explore how dependence between the tests may affect the direction of gold standard bias, defined as the difference between the “apparent” and true performance measures. (deneef1987evaluating, boyko1988reference) valenstein1990evaluating concludes that when errors committed by index and reference tests are highly correlated, the “apparent” measures may overstate the true parameters. However, they do not precisely define highly correlated errors, prompting the formalization of the assumption in this paper. Authors focus on the direction of the effects of the conditional dependence, rather than on the magnitude. The purpose is to allow researchers to determine whether their estimates are biased upwards or downwards. However, since the dependence cannot be measured, the practical relevance of these findings is diminished. Additionally, one could argue that the magnitude is perhaps even more important than the direction of the bias.

A formal approach to the issue of unknown bias magnitude is found in thibodeau1981evaluating. Assuming that the reference and index tests are positively correlated, and that the reference is at least as accurate as the index, the author bounds the bias. The framework presented below does not require such assumptions. More recently, emerson2018biomarker sketch an argument for individual bounds on sensitivity and specificity under similar assumptions used in this paper. This study contributes to the literature on gold standard bias by presenting the sharp joint identified set for test performance measures, formalizing and incorporating existing dependence assumptions to further reduce its size, bounding derived parameters of interest, and providing an appropriate uniform inference procedure.

In doing so, it builds upon on existing work on partial identification (manski2003partial, manski2007identification). (ref) revitalizes the general analysis of cross2002regressions in the context of diagnostic test performance measurements. Technical contributions over their work primarily lie within the novel identification findings in Propositions (ref) and (ref), as well as the inference procedure for the points in the identified set that is uniformly consistent in level. The procedure assumes a mild and easily interpretable restriction on the family of population distributions, and relies on existing methods for inference in moment inequality models (andrews2010inference, andrews2012inference, chernozhukov2013intersection, romano2014practical, canay2017practical, bugni2017inference, chernozhukov2019inference, kaido2019confidence, bai2021two).

This paper aligns with a growing body of literature concerning partial identification in medical and epidemiological research. (bhattacharya2012treatment, manski2020bounding, toulis2021estimation, manski2021estimating, ziegler2021binary, stoye2022bounding) It also aims to contribute to the corpus of COVID-19 test performance studies by estimating the true sensitivity and specificity for COVID-19 antigen tests despite reference test imperfections under plausible assumptions. (shah2021performance, pollock2021performance, siddiqui2021implementation)

The remainder of the paper is organized as follows. (ref) provides the identification argument. (ref) discusses identification of prevalence. (ref) explains estimation and inference. (ref) presents confidence and estimated identified sets for the operating characteristics of the COVID-19 antigen test. (ref) indicates uses of the results beyond the context of diagnostic test performance studies. (ref) concludes. Appendix (ref) derives bounds on predictive values. All proofs are collected in Appendix (ref).

Identification

Studies quantifying the performance of a test of interest, also known as an index test, require knowledge of the true health status. Health status is usually unobservable, so it is determined by an alternative test, called the reference test. Even though the reference test should be the best available test for the underlying condition, it is almost always imperfect in practice, giving rise to identification issues. Let $t=1$ and $r=1$ if the index and reference tests, respectively, yield positive results and $t=0$, $r=0$ otherwise. Let $y=1$ denote the existence of the underlying condition we are testing for and $y=0$ the absence of it.\footnote{I interchangeably say that the person is ill when $y=1$ and when $y=0$, that they are healthy. This can be extended to encompass antibody tests with minor semantic changes, since they can also measure if a person has been ill.}

We are interested in learning the sensitivity and specificity of the index test:

which are defined when $P(y=1)\in(0,1)$ in the study population. Equivalently, one can study the false negative and false positive rates, $1-\theta_1$ and $1-\theta_0$. Similarly, define reference test sensitivity $s_1 = P(r=1|y=1)$ and specificity $s_0 = P(r=0|y=0)$. Data collection in test performance studies is commonly done by testing all participants with both the reference and index tests. The observed outcome for each participant is $(t,r)\in\{0,1\}^2$. The data identify the joint probability distribution $P(t,r)$. When $P(r=1)\in(0,1)$, “apparent” sensitivity $\tilde{\theta}_1 = P(t=1|r=1)$, and “apparent” specificity $\tilde{\theta}_0 = P(t=0|r=0)$ are also identified.

It is typically assumed that the reference test is perfect, so that $r=y$. Then $(\tilde{\theta}_1,\tilde{\theta}_0)=(\theta_1,\theta_0)$. This is rarely the case in practice. Generally, $\tilde{\theta}_j\neq\theta_j$ for some $j=0,1$, which referred to as gold standard bias. Interpreting $(\Tilde{\theta}_1,\Tilde{\theta}_0)$ as true performance measures can lead to severely misleading conclusions due to the bias. Alternatively, researchers may explicitly study $(\Tilde{\theta}_1,\Tilde{\theta}_0)$. However, they only measure performance of $t$ with respect to $r$, and not $y$. If one wishes to learn about true performance $(\theta_1,\theta_0)$, then these parameters are not of interest.

Focusing the analysis on binary tests and binary health statuses is standard procedure. FDA Statistical Guidance on Reporting Results Evaluating Diagnostic Tests recognizes only binary reference tests and health statuses, explicitly stating: “A reference standard ... divides the intended use population into only two groups (condition present or absent)." Many tests that yield discrete or continuous test results, such as RT-PCR tests, are reduced to binary tests by thresholding in practice. While the results of this paper can be extended to cases in which ranges of $t$ and $r$ are finite sets, I limit the analysis to the binary setting to conform to current research practice.

The section begins by outlining the formal assumptions used. I then provide the set of parameter values $(\theta_1,\theta_0)$ consistent with the observed data, also known as the identified set, without imposing any assumptions on the statistical dependence between $t$ and $r$. The set is sharp, or the smallest possible under maintained assumptions. For simplicity of exposition, this is first done when $(s_1,s_0)$ are known. I show how an additional assumption on the dependence structure between the two tests can be used to further reduce the size of the sharp identified set. Finally, I allow $(s_1,s_0)$ to be approximately known by assuming $(s_1,s_0)\in\mathcal{S}$, where $\mathcal{S}$ is some known set.

Assumptions

The framework in this paper relies on common assumptions maintained in the literature.

The distribution $P(t,r)$ is a marginal of the joint distribution $P(t,r,y)$. Since $y$ is not observable, $P(t,r,y)$ is not point identified.

The analysis is first done for the simple case when $(s_1,s_0)$ are known exactly. The approach is then generalized in (ref) by assuming $(s_1,s_0)\in\mathcal{S}$, where $\mathcal{S}$ is some known set. Thus, reference test performance needs to be known only approximately. The generalization can also be used to perform sensitivity analyses. Knowledge of $(s_1,s_0)$ or $\mathcal{S}$ is commonly assumed in work dealing with gold standard bias correction, such as gart1966comparison2, thibodeau1981evaluating, staquet1981methodology, and emerson2018biomarker. The current norm of relying on the assumption that the reference test is perfect means that researchers regularly maintain $(s_1,s_0)=(1,1)$, which is a stronger condition since it implies Assumption (ref). The framework hence weakens standard assumptions. However, knowledge of $(s_1,s_0)$ or $\mathcal{S}$ is a crucial identifying assumption and warrants further discussion.

Assumption (ref) is particularly appealing when one has access to a study that identifies performance of $r$. mathews2010estimating and matos2011clinical justify assumed $(s_1,s_0)$ based on such studies. In specific settings, $y$ may indeed be plausibly observable, allowing for identification of $(s_1,s_0)$. However, the procedure needed to observe $y$ may be exceedingly costly, invasive, or have an unsuitably long turnaround time for widespread use in performance measurement. Thus, $r$ is commonly used as a reference for $t$ instead. For example, a neuropathological autopsy report is the only way to pose a definitive diagnosis of Alzheimer's disease, i.e. observe $y$. (suemoto2023autopsy) Studies of novel tests $t$ for Alzheimer's disease frequently use an amyloid positron emission tomography (PET) scan or a clinical diagnosis as $r$, since a neuropathological report may be unobtainable. (budelier2020biomarkers, wang2023bayesian) Autopsy performance studies for $r$ in which $y$ is directly observed post-mortem are available, identifying performance of $r$. (patwardhan2004alzheimer)

Unfortunately, in many cases such a performance study for $r$ does not exist. Nevertheless, tests are often expected to have precisely measured analytical performance measures. Analytical specificity and sensitivity are defined as performance measures obtained based on contrived, rather than clinical samples. This may provide some information on how $r$ will perform in clinical settings. For example, kucirka2020variation and kanji2021false maintain that COVID-19 RT-PCR tests are perfectly specific owing to the absence of cross-reactivity with other pathogens, that is, due to its perfect analytical specificity.\footnote{Specificity on contrived laboratory samples containing other pathogens, but not SARS-CoV-2.}

The commonly maintained assumption $(s_1,s_0)=(1,1)$ has been disputed for a plethora of reference tests. This fact indicates that at least a set $\mathcal{S}$ of more credible values $(s_1,s_0)$ exists for a variety of tests used as $r$. In these cases, the method will yield sharp bounds on $(\theta_1,\theta_0)$. However, if nothing can be credibly assumed about the performance of $r$, one cannot reasonably use it to identify performance of $t$. This is not a novel observation, but it highlights the importance of the assumption. emerson2018biomarker state: “If very little is known about the reference test performance, then it is clear that a comparison to such a reference test is a futile exercise and can provide no information about a new test.” gart1966comparison2 similarly note that when $(s_1,s_0)$ are unknown, both $t\perp\mkern-9.5mu\perp y$ or $t\centernot{\perp\mkern-9.5mu\perp} y$ will generally be consistent with $P(t,r)$. The choice of $(s_1,s_0)$ is context-specific, and should be carefully considered in each study.

I further assume that $s_1>1-s_0$, or that the reference test is reasonable.\footnote{The assumption does not require that both $s_1$ and $s_0$ are high. Indeed, it is possible that either $s_1$ or $s_0$ are close to $0$, but that their sum is higher than $1$.} If $s_1=1-s_0$, one can show that $r\perp\mkern-9.5mu\perp y$, so the test provides no information on $y$. Tests are costly, and any use of such test is not rational. If $s_1<1-s_0$, it would be possible to redefine $r^* = 1-r$, so that $s_1^*=1-s_1$ and $s_0^*=1-s_0$. Now $s^*_1 >1-s^*_0$, since $1-s_1>s_0$.

The assumption is implicitly found in all diagnostic test performance studies measuring sensitivity and specificity, since it is necessary for them to be defined. Assumptions (ref) and (ref) imply that $P(r=1)\in(1-s_0,s_1)$. If the condition fails, at least one of the two assumptions are refuted.

If one additionally maintains that $t\perp\mkern-9.5mu\perp r|y$, then $(\theta_1,\theta_0)$ are point identified (gart1966comparison2, staquet1981methodology, and zhou2009statistical). However, it is well established that conditional independence is generally untenable (valenstein1990evaluating, hui1998evaluation and emerson2018biomarker). Dependence may arise $t$ and $r$ are physiologically related, such as when they rely on the same type of sample or measure the same quantities. For example, tine and Mantoux tests may be dependent since they both rely on the antibody reaction to tuberculin (vacek1985effect), and direct immunoassay and culture swab tests for Group A streptococci may be related since they rely on the same type of sample (valenstein1990evaluating). Since $y$ is unobserved, the dependence structure is latent, and multiple structures may be consistent with the data distribution $P(t,r)$. This leads to a possibly non-singleton set of values $(\theta_1,\theta_0)$ that are consistent with the data, called the identified set. We would first like to learn this set without imposing any assumptions on the statistical dependence structure between $t$ and $r$ conditional on $y$. Additional assumptions on the possible dependence structures may then be used to reduce the size of the identified set, as shown in (ref).

Identified Set for \texorpdfstring{$(\theta_1,\theta_0)$}{.}

The data reveal $P(t,r)$, while probability distributions involving $y$ are not directly observable. Still, $P(r,y)$ can be determined using $(s_1,s_0)$ and $P(t,r)$. I henceforth use $P_{s_1,s_0}$ to denote probability distributions that are derived from observable distributions given $(s_1,s_0)$. All directly observable distributions, such as $P(t,r)$, do not have the subscript. By the law of total probability and $s_1\neq 1-s_0$ from Assumption (ref):

$P_{s_1,s_0}(r,y)$ is then known from $P_{s_1,s_0}(r,y) = P_{s_1,s_0}(r|y)P_{s_1,s_0}(y)$, since $(s_1,s_0)$ fully characterize $P_{s_1,s_0}(r|y)$. To outline the idea of finding the identified set, first note that for $j=0,1$:

Probabilities $P_{s_1,s_0}(t = j,r=k,y=j)$ for $k=0,1$ are unobservable. However, they can be bounded using the knowledge of $P(t,r)$ and $P_{s_1,s_0}(r,y)$. By the properties of probability measures, an upper bound on $P_{s_1,s_0}(t = j,r=k,y=j)$ is $min\Big(P(t = j, r=k), P_{s_1,s_0}(r=k,y=j)\Big)$. To form a lower bound, one can similarly find that $P_{s_1,s_0}(t = j,r=k,y=1-j)\leq min\Big(P(t = j, r=k), P_{s_1,s_0}(r=k,y=1-j)\Big)$ and use:

Note that these coincide with Fréchet-Hoeffding bounds for $P(t=j,y=j|r=k)$ multiplied by $P(r=k)$. Proof of (ref) demonstrates that any pair of values for $P_{s_1,s_0}(t = j,r=0,y=j)$ and $P_{s_1,s_0}(t = j,r=1,y=j)$ within their respective bounds is consistent with the observed data. Hence, sharp bounds on $P_{s_1,s_0}(t=j,y=j)$ are obtained by summing the two individual set of bounds. The sharp bounds for $\theta_j$ then follow directly from (ref) and (ref). Finally, the sharp joint identified set for $(\theta_1,\theta_0)$ is derived using $P(t=1)=\theta_1 P_{s_1,s_0}(y=1)+(1-\theta_0)P_{s_1,s_0}(y=0)$. Observe that no restrictions beyond those set by the distribution $P(t,r)$ are imposed on the latent dependence structure of $t$ and $r$ conditional on $y$.

(ref) revitalizes the general identification result of cross2002regressions. Arguments therein can also be used to derive the identified set for $(\theta_1,\theta_0)$.\footnote{I thank an anonymous referee for bringing this to my attention.} Namely, $P(t|r)$ and $P_{s_1,s_0}(y|r)$ are identified from $P(t,r)$ and $(s_1,s_0)$, yielding the identified set for the vector $\left(E[t|r=j,y=k]\right)_{(j,k)\in\{0,1\}^2}$.\footnote{$P_{s_1,s_0}(y=j|r=1-j) = 0$ when $s_j=1$ for some $j$, which violates the assumptions of cross2002regressions. It is still possible to utilize their results by bounding only $\left(E[t|r=j,y=1],E[t|r=j,y=0]\right)$.} Sharp bounds on $(\theta_1,\theta_0) = (E[t|y=1],1-E[t|y=0])$ follow. This paper relies on a constructive proof approach which can be adapted to accommodate assumptions on the latent dependence structure of $t$ and $r$ conditional on $y$. (ref) introduces a novel formalization of an informally stated assumption from the literature and exploits this feature to further tighten the bounds.

The set $\mathcal{H}_{(\theta_1,\theta_0)}(s_1,s_0)$ is a line segment on $[0,1]^2$ for a given value of reference test operating characteristics $s_1$ and $s_0$. It collapses to a point $(\Tilde{\theta}_1,\Tilde{\theta}_0)$ when $(s_1,s_0)=(1,1)$. emerson2018biomarker sketch an argument for individual bounds on $\theta_j$ as in (ref) and do not discuss sharpness or the joint identified set. (ref) goes further by proving that both individual bounds and the joint identified sets are the smallest possible under the assumptions. (ref) shows that the linear structure of the set $\mathcal{H}_{(\theta_1,\theta_0)}(s_1,s_0)$ is crucial for sharpness of bounds on certain derived policy-relevant parameters, such as the illness rate in populations screened with the test $t$, otherwise known as prevalence. Bounds on prevalence are unnecessarily wide if any pair of values $(\theta_1,\theta_0)$ from their individual bounds is considered feasible, so that the joint identified set is a rectangle $\mathcal{H}_{\theta_1}(s_1,s_0)\times \mathcal{H}_{\theta_0}(s_1,s_0)$. The sharp joint identified set for false negative and false positive rates $(1-\theta_1,1-\theta_0)$ also directly follows from $\mathcal{H}_{(\theta_1,\theta_0)}(s_1,s_0)$. The same will hold for other identified sets for $(\theta_1,\theta_0)$ below.

The identified set for $(\theta_1,\theta_0)$ is sharp. Encountering wide bounds on sensitivity and specificity implies that is not possible to learn the operating characteristics more precisely without additional assumptions that may be untenable, or without changing the reference test. Since the reference test is supposed to be the best available test, researchers and practitioners may have to embrace the ambiguity regarding the index test performance.

(ref) provides sharp identified sets for $(\theta_1,\theta_0)$ when the researcher maintains that the tests have a tendency to wrongly agree for only one or both health statuses.\footnote{One can also define the tendency to correctly disagree for disease status $\bar{y}$ as $P_{s_1,s_0}(t=1-\bar{y}|r=1-\bar{y},y=\bar{y})\leq P_{s_1,s_0}(t=\bar{y}|r=1-\bar{y},y=\bar{y})$. Identified sets that follow can easily be derived symmetrically. thibodeau1981evaluating emphasizes that tests are generally not expected to exhibit negative dependence. However, the formulation may be beneficial in applications described in (ref).} Both sets given $(s_1,s_0)$ are again line segments in $[0,1]^2$. The bounds $[\theta_j^L,\Bar{\theta}_j^U]$, and $[\theta_j^L,\Bar{\Bar{\theta}}_j^U]$ imply that the sets may be reduced in size only from above compared to $\mathcal{H}_{(\theta_1,\theta_0)}(s_1,s_0)$. This can be seen in (ref) below.

It is important to note that the assumption may or may not have identifying power for a given $P(t,r)$ and $(s_1,s_0)$. This is evident in the empirical application. (ref) notes that the assumption effectively halves the size of the estimated identified set in one population, but has no effect in the remaining two. (ref) characterizes sufficient and necessary conditions for the assumption to have identifying power, and provides an easily verifiable necessary condition.

For the purpose of interpreting this result, suppose that tendency to wrongly agree holds for $y=j$. If $s_j=1$, $\{r=1-j,y=j\}$ is a probability zero event and properties of $t$ on it are inconsequential. Having $s_j<1$ is thus necessary for reducing the size of the identified set. Assumption (ref) and $s_j<1$ imply $P_{s_1,s_0}(r=1-j,y=j)>0$. For the assumption to have identifying power we then only need $P(t=j,r=1-j)$ to be “large enough”. The definition of “large enough” is contingent upon $P(r=1)$ and $(s_1,s_0)$. In (ref) below, the threshold is $P(t=j,r=1-j)>2.5\%$ for $j\in\{0,1\}$.

Given that SARS-CoV-2 RT-PCR and rapid antigen swab tests rely on the same type of sample usually taken from the same location (e.g. nasopharynx, nares or oropharynx), it may be plausible to maintain that the two tests have a tendency to wrongly agree. We will use the assumption in the empirical application in (ref). More examples can be found in the literature. hadgu1999discrepant observes that the same assumption is credible for the ligase chain reaction (LCR) and culture tests for Chlamydia trachomatis by the same reasoning. valenstein1990evaluating indicates that when determining the performance of direct immunoassay swab tests for Group A streptococci using a culture as a reference, the tendency to wrongly agree may hold for $y=1$ due to inadequately obtained samples leading to false negatives. The same is suggested for $y=0$. Patients who are ill with viral pharyngitis, but incidentally carry the bacteria elsewhere, may appear falsely positive on both tests. vacek1985effect argues that tine and Mantoux tuberculin tests may have a tendency to wrongly agree for any $y$ as both rely on the antibody reaction to tuberculin.

Imperfect Knowledge of Reference Test Characteristics

For simplicity of exposition, previously derived identified sets for $(\theta_1,\theta_0)$ were presented under the premise that $(s_1,s_0)$ are known exactly. That assumption might be implausible depending on the setting. Researchers may instead prefer to maintain that they do not possess exact, but rather approximate knowledge of $(s_1,s_0)$. I thus relax Assumption (ref) by supposing that we only have knowledge of a set $\mathcal{S}$ that contains true sensitivity and specificity of the reference test.

Assumption (ref) is a weaker form of Assumption (ref), since it is implied by it. Similarly, jointly with Assumption (ref), Assumption (ref) implies that $\forall(s_1,s_0)\in\mathcal{S}:P(r=1)\in(1-s_0,s_1)$. If the condition fails, at least one of the two assumptions is refuted. Compactness of $\mathcal{S}$ is not relevant for identification, but it is utilized in the inference procedure construscted in (ref).

For an element $(s_1,s_0)\in\mathcal{S}$, let the identified set $\mathcal{G}_{(\theta_1,\theta_0)}(s_1,s_0)$ for $(\theta_1,\theta_0)$ be found using (ref) or (ref), depending on which of the discussed assumptions the researcher is willing to maintain. Denote by $\mathcal{G}_{(\theta_1,\theta_0)}(\mathcal{S})$ the corresponding identified set for $(\theta_1,\theta_0)$ when $(s_1,s_0)$ is known to be in $\mathcal{S}$. All values $(\theta_1,\theta_0)$ that are found in at least one set $G$ within a collection of sets $G\in\{\mathcal{G}_{(\theta_1,\theta_0)}(s_1,s_0): (s_1,s_0)\in\mathcal{S}\}$ then constitute $\mathcal{G}_{(\theta_1,\theta_0)}(\mathcal{S})$. In other words, the set $\mathcal{G}_{(\theta_1,\theta_0)}(\mathcal{S})$ contains all values of $(\theta_1,\theta_0)$ that are consistent with the observed data and at least one $(s_1,s_0)\in\mathcal{S}$. We can formally define:

Any set $\mathcal{G}_{(\theta_1,\theta_0)}(s_1,s_0)$ contains only the values of $(\theta_1,\theta_0)$ that are consistent with the observed data and $(s_1,s_0)$. The union of sets $\mathcal{G}_{(\theta_1,\theta_0)}(s_1,s_0)$ over all possible $(s_1,s_0)\in\mathcal{S}$ then only contains the values of $(\theta_1,\theta_0)$ that are consistent with the observed data and at least one $(s_1,s_0)\in\mathcal{S}$. Hence, the identified set $\mathcal{G}_{(\theta_1,\theta_0)}(\mathcal{S})$ is the smallest possible under the maintained assumptions.

The set $\mathcal{S}$ may take different forms. Expected ones include finite sets, line segments or rectangles. In general, within $\mathcal{G}_{(\theta_1,\theta_0)}(\mathcal{S})$ test performance measures $\theta_1$ and $\theta_0$ will no longer necessarily be linearly dependent. The set $\mathcal{G}_{(\theta_1,\theta_0)}(\mathcal{S})$ may not be a line segment in $[0,1]^2$, but rather a union of line segments of positive and bounded slopes. Hence, it will not be rectangular. It is still possible to demonstrate that $r$ is more precise than $t$. As in (ref), it is feasible for all $(s_1,s_0)\in\mathcal{S}$ and $(\theta_1,\theta_0)\in\mathcal{G}_{(\theta_1,\theta_0)}(\mathcal{S})$ to have $(s_1,s_0)>(\theta_1,\theta_0)$ component-wise.

Bounding Prevalence in Screened Populations

Sensitivity and specificity in the performance study population are often extrapolated to other populations and used to identify different parameters of interest. Notable examples are prevalence in a population undergoing screening and predictive values. In this section, I show how the specific structure of the identified set for $(\theta_1,\theta_0)$ helps reduce the width of bounds on prevalence when test $t$ is used for screening. Bounds on predictive values are discussed in Appendix (ref).

Population disease prevalence is both a research- and policy-relevant parameter. Suppose that we are interested in learning the true prevalence in a certain population that is being screened. Identification of prevalence based on results of an imperfect screening test is a standard epidemiological problem. Assume that each individual is tested exactly once using only the test $t$. Maintain that $r$ is not used for screening. Depending on the setting, $t$ may be preferred over $r$ for the purpose due to resource constraints, turnaround time or invasiveness, as explained in (ref). A prominent recent example was the use of antigen testing in university and institutional settings to monitor prevalence during the COVID-19 pandemic.

To make the distinction between the screened and performance study populations explicit, let $Q(t,y)$ denote the probability distribution which generates the data in the screened population. Unlike in the performance study, $r$ is not available, making the use of $Q(t,r,y)$ superfluous. The data alone identify only $Q(t=1)$. As before, $y$ is not observed, and we are interested in learning $Q(y=1)$. Let $(\tau_1,\tau_0) = \left(Q(t=1|y=1),Q(t=0|y=0)\right)$ be the sensitivity and specificity of $t$ among the screened individuals. We can then write the following identities for the two populations:

Exact knowledge of $(\tau_1,\tau_0)$ identifies the prevalence. (gart1966comparison2, diggle2011estimating) In the population of interest it directly follows from (ref):

walter1988estimation, and greenland1996basic explain that knowledge of $(\tau_1,\tau_0)$ is commonly extrapolated from test performance studies. (see also gastwirth1987statistical) That is, researchers maintain the extrapolation assumption: $(\tau_1,\tau_0) =(\theta_1,\theta_0)$, where $(\theta_1,\theta_0)$ are assumed to be identified in a performance study. I follow this practice and generalize (ref) to the case when $(\tau_1,\tau_0) = (\theta_1,\theta_0)$, but $(\theta_1,\theta_0)$ are partially identified. Denote by $\mathcal{G}_{(\theta_1,\theta_0)}(\mathcal{S})$ the identified set for $(\theta_1,\theta_0)$ obtained in a performance study. $\mathcal{S}$ can be a singleton, as when $(s_1, s_0)$ are assumed to be known, in which case I write $\mathcal{G}_{(\theta_1,\theta_0)}(s_1,s_0)$.

The assumption maintains that test performance is identical in the performance study and the screened populations. It does not require $(\theta_1,\theta_0)$ to be known exactly, and it implies that $(\tau_1,\tau_0)\in \mathcal{G}_{(\theta_1,\theta_0)}(\mathcal{S})$. While walter1988estimation state that sensitivity and specificity may readily extrapolate to other populations in many cases, one should be aware that credibility of Assumption (ref) critically depends on the details of the empirical setting. One notable potential threat to its validity is variability of test performance across subpopulations, otherwise known as spectrum effects. For example, it is known that test sensitivity may vary across subpopulations with different illness severity.

When spectrum effects exist and the two populations differ in terms of relevant subpopulation proportions, willis2008spectrum argues that $(\theta_1,\theta_0) = (\tau_1,\tau_0)$ may be implausible. To see this, observe that in this case $(\theta_1,\theta_0)$ and $(\tau_1,\tau_0)$ are weighted averages of sensitivity and specificity across relevant subpopulations, but with different weights. However, if test performance is known for all relevant subpopulations, one could maintain Assumption (ref) and identify prevalence at the subpopulation level using arguments that follow. (see also mulherin2002spectrum)

(ref) extends the identity (ref) to the case when $(\theta_1,\theta_0)$ extrapolate to $Q(t,y)$. Note that it maintains that $(s_1,s_0)$ are known exactly in the performance study. (ref) generalizes the results to the case when $(s_1,s_0)$ are known to lie in $\mathcal{S}$. The resulting bounds on the screened population prevalence $Q(y=1)$ are sharp in the absence of additional data, namely results of other tests such as $r$.

Whenever there exist $(\theta_1,\theta_0)\in\mathcal{G}_{(\theta_1,\theta_0)}(s_1,s_0):\theta_1+\theta_0=1$, prevalence is unidentified as $t$ may not be informative of $y$. Such tests are not useful for screening purposes. When $t$ is informative of $y$, that is $\forall(\theta_1,\theta_0)\in\mathcal{G}_{(\theta_1,\theta_0)}(s_1,s_0):\enskip \theta_1\neq 1- \theta_0$, the importance of the linear structure of $\mathcal{G}_{(\theta_1,\theta_0)}(s_1,s_0)$ for bounding $Q(y=1)$ becomes apparent. Identifying power of the structure can be substantial as highlighted by (ref). For certain $Q(t,y)$, resulting bounds may even point identify $Q(y=1)$ using data only on $t$, despite $(\theta_1,\theta_0)$ being partially identified.

It is important to highlight that Assumption (ref) is refutable. If $t$ is informative of $y$, then it is possible that $\Pi_{s_1,s_0} = \emptyset$. By (ref), that happens if all assumed values $(\tau_1,\tau_0)$ consistent with the assumption $(\tau_1,\tau_0)=(\theta_1,\theta_0)\in\mathcal{G}_{(\theta_1,\theta_0)}(s_1,s_0)$ result in $Q(y=1)\not\in[0,1]$. This would contradict the definition of $Q(y=1)$, implying that $(\tau_1,\tau_0)\neq (\theta_1,\theta_0)$.

Let $\mathcal{G}_{\theta_j}(s_1,s_0)=\{\theta_j:(\theta_1,\theta_0)\in\mathcal{G}_{(\theta_1,\theta_0)}(s_1,s_0)\}$ denote the individual bounds on $\theta_j$ for $j=0,1$. The sets $\mathcal{G}_{\theta_1}(s_1,s_0)$ and $\mathcal{G}_{\theta_0}(s_1,s_0)$ are also referred to as projection bounds on $\theta_1$ and $\theta_0$.

Bounds in (ref) would follow from methods that do not establish the linear structure of $\mathcal{G}_{(\theta_1,\theta_0)}(s_1,s_0)$, such as thibodeau1981evaluating and emerson2018biomarker. (ref) shows that using such methods to bound test performance will yield wider bounds on prevalence in a population screened by an informative $t$. Moreover, depending on $P(t,r)$ and $Q(t)$ the difference in width can be extreme, since $\Pi_{s_1,s_0}$ can be a singleton, while $\Bar{\Pi}_{s_1,s_0}$ is always an infinite set. (ref) illustrates how relying on rectangular identified sets affects the width of prevalence bounds using empirical examples. It compares prevalence bound widths implied by estimated identified sets for $(\theta_1,\theta_0)$ constructed using thibodeau1981evaluating, emerson2018biomarker, and the method described here, for hypothetical screened populations with different $Q(t=1)$.

(ref) generalizes (ref) to the case when $(s_1,s_0)$ are not known exactly. If the shape of $\mathcal{G}_{(\theta_1,\theta_0)}(\mathcal{S})$ was disregarded by assuming that the identified set was a rectangle, bounds $\Bar{\Pi}_\mathcal{S}$ analogous to the ones in (ref) can still be formed, and it would hold that $\Pi_\mathcal{S}\subset \Bar{\Pi}_\mathcal{S}$.

Throughout this section we have assumed that screening is performed once in the population. If screening is done repeatedly, a time series of prevalence bounds can be constructed. When there is selection into testing, bounds on prevalence by stoye2022bounding may be used, for which the bounds on $(\theta_1,\theta_0)$ derived in (ref) are natural inputs.

Estimation and Inference

Identified sets in (ref) can be found when $P(t,r)$ is known. In practice, researchers must use sample data to estimate the identified set and conduct inference. This section demonstrates consistent estimation of the identified set and construction of confidence sets for the points in the identified set that are uniformly consistent in level over a large family of permissible distributions.

Let $W_i = (t_i,r_i)\in\{0,1\}^2$ for $i=1,\hdots,n$ constitute the observed data of $n$ i.i.d observations from the distribution $P(t,r)\in\mathbf{P}$, where $\mathbf{P}$ is a family of categorical distributions with 4 categories. Let $\mathcal{G}_{(\theta_1,\theta_0)}(s_1,s_0)$ denote an arbitrary identified set for $(\theta_1,\theta_0)$ given $(s_1,s_0)$ from any of the propositions above, and $\mathcal{G}_{\theta_j}(s_1,s_0)$ the corresponding identified set for $\theta_j$ with $j=0,1$. Replacing population parameters with their consistent estimators in closed form expressions for $\mathcal{G}_{\theta_j}(s_1,s_0)$ and $\mathcal{G}_{(\theta_1,\theta_0)}(s_1,s_0)$ yields the consistent plug-in estimator of the identified sets (manski1998monotone, tamer2010partial).

Let $\mathbbm{1}\{\cdot\}$ denote the indicator function. Suppose first that $(s_1,s_0)$ are known. $\hat{P}(t=j,r=k)=\frac{\sum_{i=1}^n\mathbbm{1}\{t_i=j,r_i=k\}}{n}$ are consistent estimators of $P(t=j,r=k)$ for all $(j,k)\in\{0,1\}^2$. Combining $\hat{P}(t=j,r=k)$ with the knowledge of $(s_1,s_0)$ yields $\hat{P}_{s_1,s_0}(r=k,y=l)$ for every ${k,l}\in\{0,1\}^2$. The plug-in estimator $\hat{\mathcal{G}}_{\theta_j}(s_1,s_0)$ for the identified set of a single parameter $\theta_j$ follows immediately by inputting $\hat{P}(t=j,r=k)$ and $\hat{P}_{s_1,s_0}(r=k,y=l)$ into the bounds in (ref), or (ref). Consistent estimator $\hat{\mathcal{G}}_{(\theta_1,\theta_0)}(s_1,s_0)$ of the joint identified set for $(\theta_1,\theta_0)$ follows from (ref).

In the case when $(s_1,s_0)$ are only known to be bounded by some compact set $\mathcal{S}$, one can obtain the consistent estimator $\hat{\mathcal{G}}_{(\theta_1,\theta_0)}(\mathcal{S})=\bigcup_{(s_1,s_0)\in\mathcal{S}}\hat{\mathcal{G}}_{(\theta_1,\theta_0)}(s_1,s_0)$. This is done by finding a union of $\hat{\mathcal{G}}_{(\theta_1,\theta_0)}(s_1,s_0)$ over a fine grid of $(s_1,s_0)$ covering $\mathcal{S}$. The procedure requires two nested grid-search algorithms, and the level of coarseness of the two grids can impact computation time.

FDA Statistical Guidance on Reporting Results Evaluating Diagnostic Tests requires all diagnostic performance studies to report confidence intervals for $\theta_1$ and $\theta_0$. I show how one can use the method for inference based on moment inequalities from romano2014practical to form confidence sets that cover the true parameters with at least some pre-specified probability $1-\alpha$ and that are uniformly consistent over a large family of permissible distributions $\mathbf{P}$.

Let $C_n$ be the confidence set of interest and let $\mathbf{\Theta}(P)= \bigcup_{(s_1,s_0)\in\mathcal{S}}\Big(\mathcal{G}_{(\theta_1,\theta_0)}(s_1,s_0)\times\{(s_1,s_0)\}\Big)$ be an identification region for $\theta = (\theta_1,\theta_0,s_1,s_0)$ that depends on $P\in\mathbf{P}$ through $\mathcal{G}_{(\theta_1,\theta_0)}(s_1,s_0)$, and where $\mathcal{S}$ can be a singleton.\footnote{More precisely, we are interested in $C_n$ for the points in $\mathcal{G}_{(\theta_1,\theta_0)}(\mathcal{S})=\bigcup_{(s_1,s_0)\in\mathcal{S}}\mathcal{G}_{(\theta_1,\theta_0)}(s_1,s_0)=\{(\theta_1,\theta_0):\theta\in\mathbf{\Theta}(P)\}$. When $P(t,r)$ is known, whether one defines the identified set as $\mathcal{G}_{(\theta_1,\theta_0)}(\mathcal{S})$ or $\mathbf{\Theta}(P)$ is inconsequential.} Note that $\theta$ includes reference test performance $(s_1,s_0)$. This is done to facilitate convenient definition of moment inequalities that represent the identified set of interest, regardless of whether $(s_1,s_0)$ are known exactly or not. The confidence set $C_n$ should satisfy:

canay2017practical provide an overview of the recent advances in inference based on moment inequalities that are focused on finding $C_n$ in partially identified models. They underline the importance of uniform consistency of $C_n$ in level in these settings. If it fails, it may be possible to construct a distribution of the data $P(t,r)$ such that for any sample size finite-sample coverage probability of some points in the identified set is arbitrarily low. In that sense, inference based on confidence intervals that are consistent only pointwise may be severely misleading in finite samples. To exploit existing inference methods based on moment inequalities to construct $C_n$, the identified set $\mathbf{\Theta}(P)$ must be equivalent to some set $\tilde{\mathbf{\Theta}}(P)$:

where $m_j(W_i,\theta)$ for $j\in J_1\cup J_2$ are the components of a random function $m:\{0,1\}^2\times[0,1]^2\times\mathcal{S}\xrightarrow{\;\;\;}\mathbb{R}^k$ such that $|J_1|+|J_2|=k$. Construction of the confidence set for points in the identified set $\tilde{\mathbf{\Theta}}(P)$ is done by imposing a fine grid over the parameter space $[0,1]^2\times\mathcal{S}$ for $\theta$ and performing test inversion.

Identified sets derived in the previous section are representable by (ref). Focus on the bounds for $\theta_1 $ in (ref) when the tests have the tendency to wrongly agree for $y=1$ for intuition. Observe that there are four values that are all lower bounds on $\theta_1$ given $(s_1,s_0)$. Similarly there are four values that are all upper bounds. One of the lower bounds is trivial: $\theta_1\geq 0$. One upper bound is $\theta_1\leq \left( \frac{P_{s_1,s_0}(r=0,y=1)}{2}+P_{s_1,s_0}(r=1,y=1)\right)\frac{1}{P_{s_1,s_0}(y=1)} = \frac{1+s_1}{2}$. There are no parameters pertaining to the population distribution in the bound. This is a restriction on the parameter space, under which $\theta\in\bigcup_{(s_1,s_0)\in\mathcal{S}}[0,\frac{1+s_1}{2}]\times[0,1]\times\{(s_1,s_0)\}$. With the appropriate parameter space, there are six relevant values for the bounds on $\theta_1$ that depend on parameters of the population distribution, three for the upper and three for the lower bound. Hence, we can represent the bounds on $\theta_1$ using six moment inequalities.

Proposition (ref) implies that we only need to include one additional moment equality to represent the joint identification region $\Bar{\mathcal{H}}_{(\theta_1,\theta_0)}(s_1,s_0)$ for $(\theta_1,\theta_0)$. Then the moment function $\Bar{m}^1(W_i,\theta)$ representing the identified set $\mathbf{\Theta}(P)= \bigcup_{(s_1,s_0)\in\mathcal{S}}\Big(\Bar{\mathcal{H}}_{(\theta_1,\theta_0)}(s_1,s_0)\times\{(s_1,s_0)\}\Big)$ will have $k=7$, where $J_1=\{1,\hdots,6\}$ and $J_2=\{7\}$.

Similarly, it is possible to define moment inequality functions that represent remaining identified sets in Propositions (ref) and (ref). They are found in equations (ref), (ref), and (ref) in Appendix (ref).

romano2014practical, Theorem 3.1 provides sufficient conditions for uniform consistency of confidence sets over a large family of distributions. Assumption (ref) defines a family $\mathbf{P}$ to which the conclusions of Theorem 3.1 apply. This is demonstrated by (ref) below.

The assumption restricts $\mathbf{P}$ to distributions $P(t,r)$ such that all outcomes $(t,r)\in\{0,1\}^2$ have probability that is bounded away from zero. It serves a technical purpose, ensuring that the uniform integrability condition required by romano2014practical, Theorem 3.1 holds. The assumption is easily interpretable and it appears reasonable in the analyzed data, as discussed in (ref).

(ref) enables us to use the method from romano2014practical to construct confidence sets $C_n$ for points $(\theta_1,\theta_0)$ in the identified sets defined by (ref) and (ref) that satisfy (ref) when the relevant family of population distributions conforms to Assumption (ref).

Application - Abbott BinaxNOW COVID-19 Antigen Test

In this section, I apply the developed method to existing study data to provide confidence and estimated identified sets for $(\theta_1,\theta_0)$ of the rapid antigen COVID-19 test with the currently highest market share in the United States - Abbott BinaxNOW COVID-19 Ag2 CARD test.

Template for Developers of Antigen Tests required by the FDA for EUA mandates that the reference for all COVID-19 antigen test studies must be an approved RT-PCR test.\footnote{Link: \href{https://www.fda.gov/media/137907/download}{https://www.fda.gov/media/137907/download} (Last accessed: 12/25/2022)} However, arevalo2020false, kucirka2020variation, drame2020should, huerta2020should, and kanji2021false explain that these tests are imperfectly sensitive. Using them as a reference yields “apparent” and generally not true sensitivity and specificity. Interpreting the results as measures of true performance may be severely misleading. fitzpatrick2021buyer emphasize that the false negative rate of the BinaxNOW test may be substantially understated by the reported “apparent” analog due to imperfect reference tests. They highlight that “apparent” measures can unjustifiably lead the users to believe that the test must have high sensitivity, even when this is not true.

To verify the claim, I revisit the test performance study data from the submitted EUA documentation\footnote{Link: \href{https://www.fda.gov/media/141570/download}{https://www.fda.gov/media/141570/download} (Last accessed: 12/25/2022)}, as well data from an independent study by shah2021performance. I compare the results with the corresponding “apparent” estimates from the original documentation and the instructions for use pamphlet. I also use the data to compare the developed method with existing bounds by thibodeau1981evaluating and emerson2018biomarker, henceforth referred to as comparable methods.

By established notation, $t$ is the antigen test, $r$ is the RT-PCR test and $y$ determines whether the person truly has COVID-19. To construct the confidence sets, I implement the test from romano2014practical denoted by $\phi_n^{RSW2}$ in bai2021two. The test relies on the maximum statistic $T_n = \max\left\{\max_{1\leq j\leq k} \frac{\sqrt{n}\Bar{m}_j}{S_j},0\right\}$, where $\Bar{m}_j =\frac{1}{n} \sum_{i=1}^n m_j(W_i,\theta)$ and $S_j^2 =\frac{1}{n} \sum_{i=1}^n( m_j(W_i,\theta)-\Bar{m}_j)^2$ for a value $\theta$ and components of the appropriate moment function $m_j(W_i,\theta)$ $j=1,\hdots,k$. The testing procedure has two steps: 1) Construction of confidence regions for the moments; 2) Formation of a critical value incorporating information on which moment inequalities are “negative”. I perform test inversion over a fine grid of $10^5$ points for the relevant parameter space for $(\theta_1,\theta_0)$, and additionally over $10$ points over $\mathcal{S}$, where applicable. Following the original paper, I use $500$ bootstrap samples to find the critical values and set $\beta=\alpha/10$. The results do not change significantly with alternative values $\beta=\alpha/5$ and $\beta=\alpha/20$.

Identification Assumptions

Use of comparable and bounds developed in the paper requires a credible set of values $(s_1,s_0)\in\mathcal{S}$ for the reference RT-PCR test. I maintain that $s_0=1$ following kucirka2020variation who do the same, citing perfect analytical specificity.\footnote{Specificity on contrived laboratory samples containing other pathogens, but not SARS-CoV-2.} The same assumption has been used in other existing work, such as manski2020bounding, manski2021estimating kanji2021false, ziegler2021binary, and stoye2022bounding.

In the absence of a perfect gold standard, a conventional diagnostic test performance study cannot identify sensitivity $s_1$ of the RT-PCR tests. Some studies rely on a different set of assumptions to identify the parameter of interest. kanji2021false provide a discordant result analysis of the RT-PCR test used for frontline testing of symptomatic individuals. The authors define discordant results as initially negative RT-PCR findings followed by a positive test result within the incubation period. The negative samples were retested by three alternative RT-PCR assays targeting different genes. If at least two alternative assays yielded positive results, the initial result was considered to be a false negative finding. Assuming perfect specificities of each assay, and perfect sensitivity of the combined testing procedure, they estimate the sensitivity of the used RT-PCR test at $90.3\%$. Perfect specificity is maintained based on perfect analytical specificity. arevalo2020false use data from published studies to estimate sensitivity, defining false negatives as patients who were symptomatic and negative, but subsequently positive on an RT-PCR test within the incubation period. It is implicitly maintained that all initial results must have been false negatives. Three estimates are based on data from the United States. Sensitivity of $90\%$ is the only estimate which is not considered to be at high risk of bias according to the QUADAS-2 tool (whiting2011quadas). Following the two references, I assume that $s_1=0.9$. One should note that assumed $(s_1,s_0)$ are a critical identifying assumption which directly affect the obtained estimates. Appendix (ref) thus discusses robustness of findings to different assumed values of $s_1$.

Since the antigen and RT-PCR test rely on the same type of sample, I maintain that they have a tendency to wrongly agree for $y=1$. Since it is assumed that $s_0=1$, the tendency to wrongly agree for $y=0$ has no identifying power, as explained by (ref). It is thus not maintained.

Data and Results

EUA documentation and the instructions for use outline the initial performance study. The estimates were obtained on a sample of $460$ participants tested within $7$ days of symptom onset. shah2021performance perform the same analysis on an independent sample of $2110$ individuals enrolled at a community testing site. This includes $1188$ symptomatic individuals, of which $929$ were tested within $7$ days of symptom onset. I omit symptomatic individuals tested more than $7$ days after initial symptoms for comparability with the EUA study. I separately analyze the performance on $877$ asymptomatic participants to provide plausible estimates of performance in the absence of symptoms. The data are summarized in (ref). In all three samples, estimates of joint probabilities $\hat{P}(t=j,r=k)$ for $(r,k)\in\{0,1\}^2$ are bounded away from zero. That the distributions which have generated the data lie in a family of distributions satisfying Assumption (ref) is reasonable.

Panels (\subref{figure_EUA:conf_set}), (\subref{figure_shah_sx:conf_set}), and (\subref{figure_shah_asx:conf_set}) of (ref) represent the estimated “apparent” operating characteristics and joint identified sets for $(\theta_1,\theta_0)$, as well as corresponding $95\%$ confidence sets. The joint confidence set for apparent measures is the projection Clopper-Pearson exact confidence set. The areas of the two confidence sets are similar. (ref) show the estimates of projected individual bounds on sensitivity and specificity. The bounds are revealing.

\newgeometry{left=3mm, right=5mm, top=5mm, bottom=12mm}

\restoregeometry

The original EUA was granted based on interim results of the study in which the test exhibited estimated “apparent” sensitivity and specificity of $(91.7\%,100\%)$. Subsequent results of the full study yielded “apparent” operating characteristics estimates of $(84.6\%,98.5\%)$. Public statements and media releases erroneously cite all of the estimates as estimates of true performance.\footnote{For example: \href{https://www.bloomberg.com/press-releases/2020-12-16/abbott-s-binaxnow-covid-19-rapid-test-receives-fda-emergency-use-authorization-for-first-virtually-guided-at-home-rapid-test-u}{https://www.bloomberg.com/press-releases/2020-12-16/abbott-s-binaxnow-covid-19-rapid-test-receives-fda-emergency-use-authorization-for-first-virtually-guided-at-home-rapid-test-u}.} Both the interim and final estimates are reported on the instructions-for-use pamphlet accompanying the test. First two rows of (ref) show that both estimates of “apparent” sensitivity lie strictly above the estimated projected upper bound for true sensitivity in all samples. Hence, sensitivity may be overstated by the “apparent” analog, as fitzpatrick2021buyer suggest. The fifth and sixth rows demonstrate that final estimates of “apparent” specificity are at the estimated projected lower bounds for true specificity. True specificity may be understated.

In (ref): (\subref{figure_EUA:conf_set}), the estimate of “apparent” measures is outside the confidence set for $(\theta_1,\theta_0)$. At the $5\%$ significance level the hypothesis $H_0: (\theta_1,\theta_0)=(84.6\%,98.5\%)$ would be rejected. In other words, under the assumptions, the true sensitivity and specificity are not jointly equal to currently often cited “apparent” values $(84.6\%,98.5\%)$ at the ubiquitous level of significance. The argument for the same value holds in all other samples, as well as for the interim “apparent” estimates $(91.7\%,100\%)$.

Estimated bounds on false negative rate for symptomatic individuals within 7 days of symptom onset are $[20\%,23.9\%]$ in the final EUA study data, which is between $1.3$ and $1.55$ times larger than the corresponding “apparent” estimate of $15.4\%$. Comparison with the often-cited interim estimate of $8.3\%$ reveals that the estimated true false negative rate is between $2.41$ and $2.88$ times larger than the “apparent” analog. Data from shah2021performance yield estimated bounds of $[25.6\%,26.3\%]$ for symptomatic and $[33.1\%,38.1\%]$ for asymptomatic individuals. These estimates suggest that the true false negative rates may be up to $3.17$ and $4.59$ times higher than “apparent” interim analogs for symptomatic and asymptomatic individuals, respectively. Appendix (ref) shows that assuming lower $s_1$ further exacerbates the difference between true and “apparent” false negative rates. It also notes that assuming any $0.9<s_1<1$ lessens the difference. However, the apparent measure is never contained by the set. Hence for any feasible $s_1$, apparent sensitivity overestimates true sensitivity.

hadgu1999discrepant highlights that the errors in measurement of $2.9$ percentage points for sensitivity are significant. The differences I find in this paper between the estimates of “apparent” and true sensitivity are substantially larger under plausible assumptions. The differences vary between $4.6$ and $8.5$ percentage points using the final EUA study data. Results from shah2021performance exacerbate the discrepancies when compared to the final EUA study “apparent” sensitivity to as much $10.9$ percentage points in the symptomatic population and $22.7$ percentage points in the asymptomatic population. Even though the estimates of specificity remain close to the estimates of “apparent” specificity, the findings for sensitivity warrant further attention.

Panels (\subref{figure_EUA:comparison}), (\subref{figure_shah_sx:comparison}), and (\subref{figure_shah_asx:comparison}) of (ref) show estimates of the identified set for $(\theta_1,\theta_0)$ and compare them with estimates obtained using comparable methods. Results are represented graphically in order preserve the specific linear structure of the identified set that is lost through projection. The sharp identified set provides a substantial reduction in size in all three samples over the comparable methods, and can be very informative. Estimates of the identified set do not contain the estimates of “apparent” measures in any of the samples. Owing to the lack of sharpness, bounds estimated using other methods do not necessarily exclude the “apparent” measures. (ref) shows that projected bounds on $\theta_1$ and $\theta_0$ can also be proper subsets of those produced by comparable methods.

Panels (\subref{figure_EUA:prevalence}), (\subref{figure_shah_sx:prevalence}), and (\subref{figure_shah_asx:prevalence}) of (ref) depict the width of prevalence bounds implied by estimates from the three methods when extrapolated to populations screened by the antigen test. The solid line represents bound width given estimates of the identified set for $(\theta_1,\theta_0)$ and (ref) for various hypothetical values of $Q(t=1)$. As previously highlighted, for $Q(t=1)=P(t=1)$ prevalence becomes point-identified, despite $(\theta_1,\theta_0)$ being only partially identified. The remaining lines refer to widths of bounds in (ref), following from estimates obtained by comparable methods which yield rectangular bounds on $(\theta_1,\theta_0)$. The resulting sharp bounds on prevalence are always proper subsets of bounds found via the other two methods. Benefits stemming from the particular shape of the sharp joint identified set are immediate. Even when the projected bounds are not strictly narrower compared to other methods, the identified set can yield substantially narrower bounds on derived parameters, as shown by (ref):(\subref{figure_EUA:prevalence}), (\subref{figure_shah_sx:prevalence}).

Applications Beyond Diagnostic Test Performance

Derived results have applications that extend beyond diagnostic test performance studies. This section offers three illustrative examples, highlighting further utility of the bounds on $(\theta_1,\theta_0)$ and $Q(y=1)$. It also interprets the tendency to wrongly agree in the relevant contexts, and contrasts it with the exclusion restrictions $E[t|r,y]=E[t|r]$ and $E[t|r,y]=E[t|y]$ from cross2002regressions.\footnote{Note that $E[t|r,y]=E[t|y]$ is equivalent to $t\perp\mkern-9.5mu\perp r|y$, which is frequently considered implausible in the context of diagnostic tests, as discussed by (ref).}

In abstraction, suppose $P(t,r)$ is identified and $P(r|y)$ can be identified or credibly bounded, for $(t,r,y)\in\{0,1\}^3$. A salient case is the one in which a validation dataset identifying $P(r|y)$ exists, but it cannot be matched with the dataset identifying $P(t,r)$. This may happen due to legal or privacy concerns, lack of adequate identifiers, or because the two datasets are independent. Sharp bounds on $P(t,y)$ and its features follow from the results. If $(\theta_1,\theta_0)$ are bounded and they extrapolate to other populations where data only on $t$ is available, one can also sharply bound $Q(y=1)$.

Concluding Remarks

This paper derives the smallest possible identified set for sensitivity and specificity of a diagnostic test of interest in standard settings, when the reference test is imperfect. It formalizes an existing assumption on dependence between the reference and the test of interest, and shows how it can further reduce the size of the identified set. Finally, it develops an appropriate uniform inference procedure for the points in the identified set, enabling construction of confidence sets. The study also indicates applicability of the method beyond the context of diagnostic test performance studies.

The framework is proposed as a solution to a ubiquitous problem in diagnostic test performance studies, and it can be directly applied to existing study data to bound true test performance. Doing so demonstrates that a widely used COVID-19 antigen test tends to produce significantly more false negative results than what the currently cited figures suggest. Since other rapid COVID-19 antigen tests may exhibit similar tendencies, these findings warrant further investigation.