Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
73,449 characters · 15 sections · 49 citation commands
Total Error and Variability Measures for the Quarterly Workforce Indicators and LEHD Origin-Destination Employment Statistics in OnTheMap
\RaggedRight \thispagestyle{empty}
\sloppy
\setcounter{page}{1} \singlespace
Keywords: Multiple imputation; Total quality measures; Employment statistics; Earnings statistics; Total survey error; Input noise infusion SDL.
\strut
\doublespacing
We undertake the first comprehensive analysis of the total error and variability for two Longitudinal Employer-Household Dynamics (LEHD) products from the U.S. Census Bureau: the Quarterly Workforce Indicators (QWI), which are public-use tables displayed in QWI Explorer, and the workplace-based LEHD Origin-Destination Employment Statistics (LODES), which are the public-use tables displayed in OnTheMap (OTM) and OnTheMap for Emergency Mangement. The Census Bureau produces these labor market indicators from a comprehensive integrated administrative record system known as the LEHD Infrastructure File System, which is based primarily on the linkage between employers and employees provided by state-regulated Unemployment Insurance (UI) earnings records. The theoretical universe to which these earnings records correspond is all statutory jobs in the economy---private and public (excluding federal employees).\footnote{At the time this evaluation was first undertaken, federal employees were not covered in QWI and LODES, although they are now. }
In principle, the published indicators are subject to errors from coverage, record-level non-response, edit, imputation of item-level missing data, and statistical disclosure limitation (SDL). The SDL error is due to employer-level noise infused before tabulation. By addressing these sources of error in our assessment of total variability, we have created comprehensive total quality measures for these data.\footnote{biemer2010 also identifies sampling, specification, measurement, and data processing errors. The QWI data are not sample-survey based; so, there is no sampling error in the traditional sense. We do, however, model record-level missing data using concepts from sampling theory.}
The five indicators we study are published every quarter in the QWI, stratified by ownership, sub-state geography, detailed industry, worker age, gender, race, ethnicity, and education. The publication tables cross-classify many of these same stratifiers. One of the five indicators, beginning-of-quarter employment, is also the primary tabulation variable in LODES for display in OnTheMap, which is released annually (reference date April $1^{st}$) using many of the same stratifiers as in the QWI, and tabulated at geographies as detailed as the census block. Overall, our comprehensive measures of the total variability of QWI and LODES tabulations for these five critical variables provide substantial evidence that the system is producing reliable data.\footnote{By the standards in acs14, section 13.7, at all levels of stratification, QWI and LODES data are fit for use when the cells have at least 3-9 jobs, with the exact cutoff dependant on the set of tabulation characteristics. Cells with less than three jobs are not released in QWI, where the full set of aggregations is always available. However, they are released in LODES, with additional SDL, to permit construction of arbitrary geographic aggregates that usually contain three or more jobs.}
This study contributes to the growing body of literature on total survey error. biemer2010 defines total survey error as the “accumulation of all errors that may arise in the design, collection, processing, and analysis of survey data." The total error estimates undertaken in this study address errors due to coverage, record-level non-response, edit, imputation, and SDL.
This study also contributes to a recent, if more mature, literature that uses administrative data to evaluate existing surveys, as well as an emerging literature that assesses the total quality of administrative data themselves. See mulryjos2017, reidjos2017, and doi:10.1093/jssam/smy017 for recent examples. Our final assessment adheres closely to the best practices enumerated across many statistical agencies when applied to current data products. See Eurostat2014, blsqual2014, and acs14 for examples of total quality frameworks applied to other statistical products.
The remainder of this paper is organized as follows. Section (ref) provides important background on the LEHD data, including variable sources, definitions, and characteristics related to employers and workers. Section (ref) formally defines the universes, frames, estimands, and estimators that we study for each of the five QWI statistics. Section (ref) demonstrates that the total error bias is zero. Section (ref) provides formal models for estimating total variability and its associated components in a manner that fully incorporates the uncertainty due to the SDL procedures. Section (ref) discusses the detailed results and provides guidance for computing confidence intervals. Section (ref) concludes.
The QWI and LODES are based on the LEHD Infrastructure File System. The original production version of this system is documented in qwi09. Enhancements to the processing of characteristics are further documented in mckinney:green:vilhuber:abowd:2017. We focus on five of the QWI indicators:
In LODES, the primary tabulation variable is $B$ using QWI definitions.\footnote{In order to be consistent with the notation in most of the technical documentation of the LEHD data, QWI, and LODES, we use the compact notation as in qwi09. A crosswalk between the notation used here and that found on the public website can be found at lehdschema410.}
UI earnings records are used to construct a job-based frame for the QWI and LODES. An in-scope job occurs when a worker produces at least one dollar of UI-covered earnings at a non-federal employer in a given quarter. The LEHD Infrastructure File System combines this information with additional survey and administrative data to associate individual characteristics (or features) such as birth date, gender, place of birth, race, ethnicity, and education, as well as workplace characteristics (or features), such as workplace address and North American Industrial Classification System (NAICS) codes to all jobs in the frame. The LEHD Infrastructure File System was developed using model-based edit and imputation procedures. Every missing data element has been multiply-imputed using an integrated set of models described in qwi09. There are 10 implicates for every missing item, subscripted $l=1,\dots,L=10$. The missing data models for most of the variables used in this paper, including birth date, gender, race, ethnicity, education, workplace geography, workplace NAICS, firm age, and firm size, have been substantially improved and modified since the 2009 article was written. Because the LEHD Infrastructure File System is rebuilt every quarter from all historical records, the analysis in this paper incorporates all of those model improvements.
The five worker characteristics used in this paper are birth date, gender, race, ethnicity, and education, each of which is part of an integrated multiple-imputation model. We also evaluate two workplace characteristics; NAICS sector and county. Both of the workplace characteristics have extremely low missing data rates and are not multiply imputed, however, jobs at multi-unit employers must be assigned to one of the employer's establishments. The assignment of a job to an establishment is multiply imputed and will potentially introduce variation across implicates to the extent that a workplace characteristic varies across workplaces (also called establishments) within the same employing firm (UI account). Both processes are documented in mckinney:green:vilhuber:abowd:2017. The statistical disclosure limitation applied to QWI and the workplace component of LODES uses employer-level input noise infusion qwi09,noise2012.
All statistics in this paper are estimated for a given state-year-quarter. All estimation uses the original micro-data from the production system, which incorporates the missing data imputation models and provides values of all micro-data elements with and without SDL applied. We re-estimate the actual QWI statistics, then estimate total variability and its components. The analysis covers the period 1995 through 2016 for the states AK, DC, DE, HI, KY, ND, NH, RI, SD, VT, WV, and WY.\footnote{For simplicity, we include Washington, D.C. when we say “state.” We restricted analysis to a subset of states to address the comments from referees and the editor to incorporate the error due to SDL. Using a subset of states reduces the computational burden. The longer technical paper mckinney:green:vilhuber:abowd:2017 uses data from all available states in the QWI release labeled R2012Q4, which covers 1990:1 through 2012:1, but does not contain an estimate of the contribution of the noise infusion to the total variability, and does not use the formulas developed for this paper. The schema for the QWI as of the R2012Q4 release are described at \url{https://lehd.ces.census.gov/data/schema/v3.5/}. The schema is regularly updated, and can be found at \url{https://lehd.ces.census.gov/data/schema/}. Availability for each state varies both historically and at any point in time, see \href{https://web.archive.org/web/20170803015127/https://lehd.ces.census.gov/doc/QWI_data_notices.pdf}{https://lehd.ces.census.gov/doc/QWI\textunderscore data\textunderscore notices.pdf} for available data for each state.}
Every job in the universe must have completed data for all the publication variables. The LEHD Infrastructure File System has a fully-integrated collection of probability models that generate multiply-imputed values for all missing data items in the system. Most details are supplied in qwi09---in particular, the models for imputing missing demographic and workplace characteristics.\footnote{qwi09 does not document the replacement to the demographic variable imputation methods that were incorporated in 2010. Those methods are documented in mckinney:green:vilhuber:abowd:2017.} The system uses the methods first proposed by rubin:1978,rubin87 and expanded in rodrub2002 for analyses using multiply-imputed missing data.
For each record in the component files of the LEHD data, variables used in the computation of any QWI or LODES statistic are examined by an integrated set of edit and imputation models. The system uses $L=10$ threads for these models. Each thread generates its own posterior predictive samples, which are used to apply the edits and imputations to each record. Calculations are done for all time periods $t$, but for notational convenience, and without loss of generality, we drop the $t$ subscript for most of our discussion below.
The data tabulation system infuses permanent multiplicative noise into the employer-level data used to produce tabular output for QWI and LODES qwi09,noise2012.\footnote{A similar method is now used in many other Census Bureau economic data publications brownetal:2009.} The multiplicative noise factors $\delta_j$ for each establishment $j$ are drawn from a two-sided symmetric ramp distribution centered at unity. The draws from the distribution distort the original input by at least a minimum percentage, and by no more than a maximum percentage. Both of these values are Census confidential. This system is a substantial generalization of the method originally developed by jos1998. The SDL system provides protection to both an employer and all of its workplaces---all establishments for the same employer within a given state have input noise-infusion factors on the same side of unity. In addition, the release statistics are dynamically consistent---the same noise factor is used for an employer or workplace in every quarter of data.\footnote{Most states do not code the workplace or establishment on their UI earnings records. Consequently, it is the employer-level noise-infusion factors that are incorporated into the estimators in this paper. The establishment-level noise factors are only salient when the workplace identifier does not have to be imputed on the UI earnings record. In that case an establishment and a firm are equivalent, and we call them both employers.}
Multiplicative input noise infusion provides confidentiality protection in the following sense. The originally reported values of the tabulation variables are never used in the formation of the magnitudes (employer-level counts and dollar sums) and ratios that are tabulated. The input noise infusion insures that for every micro-data record tabulated, there is a strictly non-zero percentage difference between the value used in tabulation and the true confidential value. Tabulations based upon a small number of establishments (at the limit one) or a small number of employees (at the limit one) contain uncertainty induced by the distribution of the noise factor. This uncertainty limits a user's ability to infer attributes to within a range that is confidential. Finally, the physical location of a workplace is not treated as confidential because it is defined as the location where an employee must report for work, and is therefore public. While the protection system is not formally private in the sense of Dwork2006a, it does satisfy the necessary conditions in Dinur2003 for resistance to database reconstruction attacks. See HaneySIGMOD2017 for a formal privacy analysis of this protection mechanism.
In this section we define the statistical framework for the universes, frames, estimands, and estimators used in the QWI and LODES data analyzed in this paper. Because our focus is the estimation of total error in the spirit of biemer2010, the estimands defined here match the production system at the Census Bureau, but the estimators have been slightly simplified to permit estimation of their variance components.
The theoretical universe for QWI/LODES is all statutory jobs. While this universe is conceptually easy to understand, implementing it in a frame via administrative records only---that is, without independent field work to verify the existence of businesses---is very tricky, as we will discuss in section (ref). Because the universe includes statutory employers whether or not they have positive UI-covered jobs (i.e., statutory employment) in a given quarter, it is possible that a particular sub-universe will have no statutory employment in a particular quarter, but nonetheless be at risk for positive employment because statutory employers exist in that sub-universe. This is a sampling zero. In contrast, some sub-universes will never have any statutory jobs because there are no statutory employers in that sub-universe, and we denote these as structural zeros because the probability of observing positive employment in these sub-universes is zero.
To make all of the sources of uncertainty relative to the complete data estimands clear, we define each component of the micro-data records completely.\footnote{Although these definitions can be found in qwi09, repeating them here helps properly distinguish the consequences of record and item missing data.} For individual $i$ employed by business $j$, a UI-covered statutory job exists in quarter $t$, if $i$ has a UI-earnings at $j$, $w1_{i,j,t}$, with at least $\$1$ of earnings in quarter $t$; that is, $w1_{i,j,t} \ge 1$. In this case we say $m_{i,j,t}=1$, otherwise $m_{i,j,t}=0$. In quarter $t$, there is a vector of characteristics associated with $i$ and $j$, which we call $x_{i,j,t}$. These characteristics include worker features (gender, age, race, ethnicity, and education) and employer features (workplace location, workplace industry, employer size, and employer age). We define the sample space of $x_{i,j,t}$ as $\chi$.
Partitions of $\chi$ are denoted by the set $\{\Omega_k\}$ having the properties
and
Hence, a partition of $\chi$ is a mutually exclusive, exhaustive stratification of the characteristics in the population. Although QWI and OTM use multiple partitions, the properties of all estimators can be understood by examining those defined on an arbitrary partition. When it is clear from the context, we will call one element of this partition “cell $k$,” which means all statutory jobs in the universe that belong in partition $\Omega_k$.
The finite-population estimands of $M_k$, $B_k$, $F_k$, $Z\_W3_k$, and $W1_k$ are defined here for the partition $\{\Omega_k\}$. For the remainder of the paper, the subscript $t$ is suppressed unless it is needed to define a primitive for quarter $t$.
The estimand for $M_k$ is
where $M_{j} = \sum_i m_{i,j}$, $I[A]$ is the indicator function taking the value 1 when $A$ is true (0, otherwise), and $J$ is the number of employers in the universe.
To define the estimand for $B_k$, we define $b_{i,j,t}=1$ when $m_{i,j,t}=1$ and $m_{i,j,t-1}=1$, otherwise $b_{i,j,t}=0$; then,
To define he estimand for $F_k$, we define $f_{i,j,t}=1$ when $m_{i,j,t+1}=1$, $m_{i,j,t}=1$ and $m_{i,j,t-1}=1$, otherwise $f_{i,j,t}=0$; then,
To define the estimand for $Z\_W3_k$, we use $w1_{i,j,t}$ divided by 3, to define average monthly earnings for a statutory job in quarter $t$. Then,
Note that the multiplication by $f_{i,j}$ inside the summation selects the correct employees---those employed for the full quarter.
Finally, the estimand for $W1_k$ is
Note that multiplication by $m_{i,j}$ is not required, since $m_{i,j,t}=1$ if, and only if, $w1_{i,j,t}\ge 1$. There are no rows in the universe where $m_{i,j,t}=0$, by construction.
If the LEHD data contained a record for every statutory job in the universe with no item missing data, then the finite-population estimators for $M_k$, $B_k$, $F$, $Z\_W3_k$, and $W1_k$, in the absence of SDL, would be identical to the estimands in equations ((ref))-((ref)). Because the frame must be constructed dynamically, and because there are missing records and missing items in the file system, and, finally, because the released data are subject to SDL, we carefully construct the estimator appropriate for each estimand in this section.
We cannot tell by observing a single quarter of data whether the absence of employers in cell $k$ means that there were no statutory jobs in that cell for that quarter (sampling zero) or no possibility of jobs in that cell because there were never any employers (structural zero). Therefore, to develop a frame for estimation, we must adopt a definition of the effective universe that spans a broad time period.\footnote{abowd:crepon:kramarz:2001 show how the use of long sequences of mandatory tax information returns, collected in a distinct administrative operation from the data used for the employment statistics, can substitute for independent field work in modeling the birth and death of employers in a dynamic business frame problem similar to the one studied in this paper.} We define the frame to include an employer if there is any evidence of statutory jobs anywhere in $\chi$ for that employer between the earliest quarter in the database and the most recent quarter in the database--1990 to 2016, in this paper. Note that the tabulation quarters in this paper--1995-2016--are a temporally contiguous subset of the universe used to construct the frame. This is also how the employer frame is defined in the production QWI and OTM.
The LEHD program receives two employer-level reports of employment and earnings every quarter. The first is the Quarterly Census of Employment and Wages (QCEW). The second is the state Unemployment Insurance (UI) earnings data. The QWI/LODES employment and earnings measures are constructed from the state UI earnings records, but the QCEW data also provide an employment measure that we combine with the UI records to construct a proper frame. Most statutory jobs are defined by state law. At the state level, any employer with positive statutory employment in the QCEW or positive statutory employment in the UI earnings data, as long as the UI employer appears at least once in the QCEW in the frame window, contributes to the population of jobs.\footnote{We permanently exclude all UI accounts that appear in the UI earnings data but never appear in the QCEW because they are very likely to be duplicates due to identifier mismatch ben2007.} Using this definition of the frame, structural zeros occur in cells $k$ that never have any employers in the period 1990 to 2016. Note that structural zeros limit the sample space, $\chi$, of $x_{i,j,t}$ but are only possible for employer features, not for individual features. For example, a particular county can have a structural zero for a particular NAICS sector because there are no employers in the frame who ever had jobs in that county and NAICS sector. When we report statistics for partitions of the sample space, partitions that are structural zeros are excluded because they have neither employers nor jobs with probability one ex ante. Therefore, there is no error associated with the structural zeros. This is exactly the same as the treatment of geographic areas or NAICS sectors where there are no employer firms in other economic data.
Record-level incomplete data occur because either the QCEW quarterly employment and earnings summaries for the employer or all the job-level UI earnings records for an employer are missing due to failure to file or late filing. The QCEW summary and UI individual earnings data are both collected as part of the administration of the state UI system. They share employer-level identifiers called SEINs in the LEHD data and UI account numbers in the state systems. The statistical problem in determining total frame employment in quarter $t$ is reconciling QCEW employment definitions (measured for the pay period that includes the $12^{th}$ calendar day of the month for the first, second and third months of each quarter) and the available employment definitions in the QWI (first day of the quarter for $B$ and $F$ or any day in the quarter for $M$). The QCEW data are edited following procedures laid out by the BLS, and employment counts generated via the QCEW for month-one of the quarter are most comparable to the $B$ measure in the QWI. There is no employment measure in the QCEW data comparable to $F$ or $M$ in the QWI data. QCEW total quarterly earnings are exactly comparable to $W1$ in the QWI.
To account for records missing from the LEHD system, we form a composite total frame employment measure based on a precedence ordering. We use QCEW month-one employment, if available. If not, we use QCEW month-two or month-three, in that order, if available. If no QCEW data are available, we use UI-defined $B$ employment (applying equation ((ref)) directly to the raw micro-data), if available. If not, we look forward one quarter for $B$, if available. If not, we use $M$ (again applying equation ((ref)) directly to the raw micro-data), which is always available if a UI record exists in quarter $t$. Thus, for the purpose of determining the total frame employment, the UI data substitute for QCEW data when QCEW data are not available in quarter $t$, as long as a QCEW record exists for that employer in at least one other quarter in the frame window (1990-2016). If no QCEW data ever exist in the frame window, the UI earnings records for that employer are discarded from the frame.
For each quarter, the total employment calculated from this composite measure is the finite job population, which we call $N_B$ because it is based on the $B$ definition of employment. The employers with observed UI earnings data in that quarter are the “sampled jobs” from this finite population, $N_{UB}$. Call this set of employers $S_t$. $N_{UB}=\sum_j \sum_i b_{i,j}$ for $j \in S_t$. The weight is $w=\frac{N_B}{N_{UB}}$ and the observed fraction of jobs is $f=\frac{1}{w}$.\footnote{We apologize for the abuse of notation in defining the weight, $w$, and the sampled fraction, $f$, using symbols that also appear with subscripts in the text. Whenever $w$ and $f$ are used without subscripts, they always mean the weight and the sampled fraction.}
As can be seen from the definitions, the weight is the ratio of total composite $B$-based employment for all employers to total composite $B$-employment for employers that appear in both the QCEW and UI earnings data. Using a consistent set of employment reports in both the numerator and the denominator ensures that an employer always receives a positive weight not less than 1. The median employer-level weight is 1.007. The $75^{th}$ percentile employer-level weight is 1.020. The $95^{th}$ percentile employer-level weight is 1.092.
The weight that we construct from the composite $B$-based employment measure is calculated separately each quarter for private and public-sector employers. The variation in weights between employers is due entirely to the private-public classification. Public-sector weights are typically larger than private-sector weights with a median of 1.182. Both Hawaii and DC have unusually large public-sector weights (median of 6.409, max of 7.402). The $95^{th}$ percentile for the public-sector weight is 1.278, while the $95^{th}$ percentile for the private-sector weight is 1.039.
There is a single weight for all private employers and a different single weight for all public-sector employers for each state and quarter. There is no between-employer variance in the weight within the private or public sectors. The weight, thus, assumes that employers, and thus UI earnings records, not found in the LEHD file system are missing completely at random, given the public/private sector of the employer. We recognize that this is a strong assumption. Given the very small fraction of implied missing UI earnings records, on average 1.67%, and the importance of having implementable formulas for certain components of total variability as derived in Section (ref), we think it is a good working approximation.
The public/private status of an employer is permanent (non-time-varying) and never missing. Therefore, we can unambiguously permanently stratify the frame into public and private employers. The statistical analysis that follows is identical for these two strata except that the numerical value of the total job population is different in each stratum. For that reason, we do the theoretical analysis assuming the frame has a single stratum. This keeps the formulas simpler. Aggregation of the stratum-level statistics is via addition for $M$, $B$, $F$, and $W1$ and weighted averages using the proportion of the total job population in each stratum as weights for $Z\_W3$.
The LEHD micro-data also contain the permanent noise-infusion factor, $\delta_j$, for each establishment $j$ and the $L$ implicates from the multiple imputation of missing characteristics. Given $w$, $\delta_j$ and the implicates $l$, we can define the QWI/LODES estimators $\hat{M}_k$, $\hat{B}_k$, $\hat{F}_k$, $\hat{Z}\_W3_k$, and $\hat{W}1_k$ as follows.
where $x_{i,j}^{(l)}$ is the $l^{th}$ implicate of $x_{i,j}$ from the multiple imputation system. Note that $m_{i,j}$ cannot be missing if employer $j$ is in the sampled set $S$ for the quarter being estimated. The upper limit of the summation over $i$ is therefore unaffected by item missing data, and the entire $i$ summation is multiplied by zero when UI records for employer $j$ are missing.
Notice that there is no multiple imputation of the variable $b_{i,j}$, just as there was not for $m_{i,j}$, because it is always possible to define $b_{i,j}$ without reference to any variable except $m_{i,j}$, which is never missing when employer $j$ is in the observed sample. The variation due to item missing data occurs only because the features in $x_{i,j}$, which may be missing, determine whether the job record belongs in cell $k$.
Notice, again, that $f_{i,j}$ is never missing for the same reasons as $b_{i,j}$ and $m_{i,j}$. To define the estimator for $Z\_W3$, we also need an estimator for $F$ that does not have SDL, for use in the denominator:
Then, the estimator for $Z\_W3_k$ is
Notice, again, that $w1_{i,j}$ is never missing when employer $j$ is in the observed sample. Finally, the estimator for $W1_k$ is
Under the maintained assumptions that UI records are missing completely at random and that item missing data imputed via the multiple imputation system are ignorable, we can directly compute the bias in $B$. We evaluate
where $w$ is given, and $\rho$ are the parameters of the SDL system. $E[I[j \in S]|w]=f=\frac{1}{w}$ because of the assumption that UI earnings records are missing completely at random. By design, the SDL random variation is independent of all other sources of error, and $E[\delta_j|\rho]=1$ for all $\rho$ by properties of the symmetric ramp distribution used for the input noise infusion. Finally, $E[I[x_{i,j}^{(l)} \in \Omega_k]]=I[x_{i,j} \in \Omega_k]$, because of the ignorable item missing data assumption and the independence of the SDL random variable. Substituting yields
There is no comparable proof of unbiasedness for $\hat{M}_{k}$, $\hat{F}_{k}$, or $\hat{Z}\_{W3}_{k}$, because the frame can only be constructed using the job definition reflected in $B$ (beginning-of-quarter employment) due to limitations of date information on UI earnings records in comparison with the date information on the QCEW data. In principle, however, the same procedures used to compute $w$ based on $B$ could be used to compute a separate weight based on $W1$ because the QCEW and QWI concepts and dating conventions are identical. The alternative frame would be constructed using methods comparable to those used in Section (ref). In order to keep our analysis of statistics related to $W1$ as useful as possible in understanding the properties of the published QWI data, which use a single weight across all statistics, we do not to re-weight $W1$.
In this section, we exploit the structure of the LEHD data to develop appropriate variance decomposition formulas. We proceed in three steps. First, we show how earnings records missing due to incomplete reporting of employers in the frame introduce simple random sampling uncertainty into the estimators for each QWI statistic. For this component, we exploit the assumption that these records are missing completely at random, leading to the conventional estimand for the variance of finite-population totals and means. We implement the conventional estimator for this estimand. Second, we show how noise infusion introduces a multiplicative random error into the employer-level component of each QWI statistic. The estimand does not have a simple estimator with a closed-form solution; hence, we exploit the independence of the noise-infusion process from all other components of total error to implement a simulation-based estimator. These two components of error would occur even if the data on characteristics of the jobs were complete. Finally, we show how multiple imputation of the missing job characteristics implies the conventional Rubin estimator for the total variation, when implemented with the standard ignorability assumptions. In developing this estimator, we use the first component of variance (due to randomly missing employers) as the within-implicate estimator. The contribution of the noise infusion occurs only at the employer level. We estimate this contribution with a simulation estimator and add it to the between-implicate variance estimator.\footnote{In previous versions of this work, including mckinney:green:vilhuber:abowd:2017, we developed formulas for total variability and its components that also incorporated SDL in the estimators for the variance component estimands that were consistent with the noise infusion developed for the QWI/LODES publication system. This was done to permit release of sub-state total variability statistics based on our formulas. We are grateful to the editors and referees for urging us to incorporate the SDL uncertainty into our overall analysis, and to purge the total variability estimators of SDL components introduced exclusively to meet our original goal of releasing estimates at sub-state levels. The Census Bureau Disclosure Review Board cleared the release of the summary statistics in this paper, which are all aggregated across 12 states, but would not have approved the release of sub-state versions of the statistics because in this paper we replace the required noise-infusion SDL with legacy rounding and cell-size rules. The DRB did allow publication of the SDL component of error, which is an important contribution to data analysis in the presence of non-ignorable disclosure limitation abowd:schmutte:BPEA:2015.}
We develop our analysis of the components of total variability for the statistic $B_k$ first, because that measure corresponds to the central job concept used to create the frame and generate the weights, as described in Section (ref). We distinguish between the variance component estimand and its estimator. Finally, we describe how the estimator is implemented in our variance decomposition. For the other four QWI statistics, we describe only the changes to the components of the estimands and estimators necessary to implement our analysis.
We begin by noting that in the absence of missing employers, disclosure limitation, and missing job characteristics, equation ((ref)) is identical to equation ((ref)), and there is no error due to the statistical processing. As noted in Section (ref), there may still be error in the raw data production, ingestion and curation due to features of the Unemployment Insurance program administration that are not modeled in this paper.
Consider the randomness due to employers in the frame not reporting UI earnings records in a particular quarter. In this case equation ((ref)) becomes:
where the subscript 1 indicates that this equation refers to the randomness induced by the missing UI earnings records; i.e., the first component of variation. Define the complete data estimand for cell proportions as
Then, the finite population estimator for $B_{1k}$ can be rewritten as
and the estimator for the proportion of the jobs in cell $k$ is
Under simple random sampling, implied by the assumption that job records are missing completely at random, the proportions $\hat{P}_{1k}$ are unbiased estimators of $\frac{B_{1k}}{N_B}$, with variance given by cochran:1977:
The conventional unbiased estimator of $V[\hat{P}_{1k}]$ is cochran:1977:
where $(1-f)$ is the finite population correction, and
The second component of variance is due to employer-level input noise infusion. Including SDL in equation ((ref)) yields
where the subscript 2 indicates that this equation includes the randomness induced by missing UI earnings records and SDL. Rewriting equation ((ref)) gives
where $SDL[\hat{B}_{2k}]$ is the component of $\hat{B}_{2k}$ due to SDL. By the independence of $I[j \in S]$ and $\delta_j$, the expectation of $SDL[\hat{B}_{2k}]$ is zero. $V[\hat{B}_{2k}]=V[\hat{B}_{1k}]$ + $V[SDL[\hat{B}_{2k}]]$ + $2Cov[\hat{B}_{1k},SDL[\hat{B}_{2k}]]$. By the same independence, $Cov[\hat{B}_{1k},SDL[\hat{B}_{2k}]]=0$.\footnote{See proof OSM-1 in the supplemental online materials OSM2020.} We are left to estimate $V_2[\hat{B}_{2k}-\hat{B}_{1k}]$=$V_{SDL}[\hat{B}_{2k}]$, where the subscript 2 on the variance indicates that it is the second component of variance for $\hat{B}_k$, i.e., the component due to independent noise infusion.
Decompose $V_{SDL}[\hat{B}_{2k}]$ as
Note that $E[SDL[\hat{B}_{2k}]|I[j \in S]]=0$, so we only need to evaluate $E[V_{SDL}[\hat{B}_{2k}]|I[j \in S]]$. We do this by simulating $V_{SDL}[\hat{B}_{2k}|I[j \in S]]$ and noting that, when job records are missing completely at random, our simulation estimator is unbiased if we use the same weights as are used for $\hat{B}_{1k}$. Let $G$ be the number of simulations and $g$ be the simulation index. For each simulation, and for each employer $j$, draw $\delta_j^{(g)}$ from the symmetric ramp distribution used by the QWI SDL system.\footnote{We used the actual ramp distribution approved for the QWI data, but the parameters of that distribution have never been published because they are Census confidential.} Compute $SDL[\hat{B}_{2k}]^{(g)}$ using $\delta_j^{(g)}$. Then, the unbiased estimator of the component of variance due to SDL in $\hat{B}_{k}$ is
Finally, consider the contribution to total variation arising from multiple imputation of the job features $x_{i,j}$. We begin by noting that the proportions defined in equation ((ref)) are the only place where the job features affect any of the estimators for $\hat{B}_k$ or components of its error $\hat{B}_k-B_k$. For each implicate $l$, define the proportions
Now, substitute $p_{i,j,k}^{(l)}$ into equation ((ref)) to get
Notice that equation ((ref)) is identical to equation ((ref)) before averaging over the $L$ implicates of the multiple imputation. Therefore, we can apply the multiple imputation formulas directly to $\hat{B}_{k}^{(l)}$ yielding the estimator in equation ((ref)) and
For completeness we define the average within-implicate variance as
and the between-implicate variance due to imputation as
In the absence of SDL, implementing equation ((ref)) would be straightforward, given the formulas already derived in this section. The first term, the average within-implicate variance, would be estimated by evaluating the variance component $\hat{V}[\hat{B}_{1k}]$ in equation ((ref)) for each implicate and averaging. The second component would be computed by evaluating the estimator in equation ((ref)) for each implicate $\hat{B}_k^{(l)}$, averaging to obtain $\hat{B}_k$, and substituting directly into the formula in equation ((ref)).
In the presence of multiplicative noise infusion SDL, the answer is not so straightforward. We should use equation ((ref)) to evaluate $\hat{B}_{k}^{(l)}$. Then, use equations ((ref)) and ((ref)) to estimate the within-component of multiple-imputation variance. But, as we showed above, the SDL is applied at the employer-level, and the data on $b_{i,j}$ are never missing. Hence, whether $V_{SDL}[\hat{B}_{2k}]$ is computed for each implicate, or computed once for an arbitrary implicate should not matter. Conceptually, the error in $SDL[\hat{B}_{2k}]$ is a component of total error independent of both the record-level and item-level missing data randomness. We could estimate its contribution by computing the components of variance due to missing UI earnings records and missing job features as described in the paragraph above but with $\delta_j=1$ everywhere, then add our estimate of $V_{SDL}[\hat{B}_{2k}]$ to that. This is the method used in the tables of the paper.
Because $\delta_j$ multiplies the sample selection random variable $I[j \in S]$ and, consequently, the employer-level estimate $\sum_{i=1}^{M_{j}} p_{i,j,k}^{(l)}$, the multiple-imputation variance formulas are affected by its presence in equation ((ref)). The expected value of this interaction is zero for $B$, $M$ and $F$, but positive for $Z\_W3$ and $W1$. Consequently, we present estimates in the supplemental online materials that include $\delta_j$ in the computation of the multiple imputation variance components OSM2020.\footnote{We verified by small-scale simulation that the differences between the text and online supplemental tables were random fluctuations around the expected value of zero for $B$, $M$ and $F$. We also verified that the inflation of the variance components of $Z\_W3$ and $W1$ shown in the online supplemental tables was consistent with the known, but still confidential, parameters of the distribution of $\delta_j$.}
To summarize, in the text tables, columns estimating the average within-implicate component of total variance average the values of $\hat{V}_1[\hat{B}_{1k}^{(l)}]$ over $l=1,..., L=10$ implicates. That is, they substitute $\hat{B}_{1k}^{(l)}$, defined for each implicate using equation ((ref)), for $\hat{B}_k^{(l)}$ in equation ((ref)). In the text tables, columns estimating the between-implicate component of variance due to multiple imputation, substitute $\hat{B}_{1k}^{(l)}$ for $\hat{B}_{k}^{(l)}$ and compute $\hat{B}_k$ with all $\delta_j=1$ in equation ((ref)). In the text tables, columns estimating the between component of total variance due to noise infusion compute $\hat{V}_{SDL}[\hat{B}_{2k}]$ using equation ((ref)) exactly as defined, holding $x_{i,j}^{(l)}=x_{i,j}^{(1)}$ with $G=10$.
In the online supplemental tables, the columns estimating the average within-implicate component of total variance evaluate equation ((ref)) by substituting $\hat{V}[\hat{B}_{1k}]$ evaluated using $\hat{B}_{k}^{(l)}$ for each implicate for $\hat{V}[\hat{B}_{k}^{(l)}]$. In the online supplemental tables, the columns estimating the between-implicate component of variance due to multiple imputation estimate equation ((ref)) exactly as shown in the text. Finally, in the online supplemental tables, columns estimating the between component of total variance due to noise infusion compute $\hat{V}_{SDL}[\hat{B}_{2k}]$ using equation ((ref)) exactly as defined, holding $x_{i,j}^{(l)}=x_{i,j}^{(1)}$ with $G=10$.
In all results, the total variability is estimated as
with the appropriate estimators, as described in the paragraphs above, substituted for $\hat{V}_W[.]$ and $\hat{V}_B[.]$. The coefficient of variation is estimated as
To permit the estimation of approximate confidence intervals, we estimate approximate degrees of freedom using the moment-matching formula from rubschenk86
where $N_{k}=\frac{\hat{B}_{1k}}{w}$ is the observed job count in cell $k$.
The formulas for $\hat{V}[\hat{M}_k]$ and $\hat{V}[\hat{F}_k]$, and their components, are comparable to equation ((ref)); however, unlike for $B$-based jobs, the total populations of $M$- and $F$-based jobs are unknown, as explained in Section (ref). Let $N_M$ and $N_F$ be the unknown total population of $M$- and $F$-based jobs, respectively. We assume that the same fraction $f$ of these jobs are observed as for $B$, and that the missing fraction $(1-f)$ of UI earnings records required to compute $m_{i,j}$ and $f_{i,j}$, respectively, are missing completely at random. In each quarter, then, let
We now enumerate the modifications to equations ((ref))-((ref)) required to estimate components of variance for $\hat{M}_k$ (resp., $\hat{F}_k$). In all equations, substitute $N_M$ (resp., $N_F$) for $N_B$, substitute $m_{i,j}$ (resp., $f_{i,j}$) for $b_{i,j}$, and substitute the symbol $M$ (resp., $F$) for the symbol $B$.
We now explain the modifications to equations ((ref))-((ref)) required to estimate components of variance for $\hat{Z}\_W3_k$. The frame for $Z\_W3$ is the same as the frame for $F$; hence, the total population of $F$-based jobs, $N_F$ substitutes for $N_B$. Notice that the estimand ((ref)) is a sub-population average for monthly earnings of full-quarter employees; hence, we need to build the conventional finite-population variance estimand and estimator. Equations ((ref)) -((ref)) are not needed because the estimator is already in the required ratio form. Replace equation ((ref)) with
We can proceed directly to the replacements for the variance equations. Because we assumed that unreported UI earnings records are missing completely at random, the finite-population variance of a mean from a simple random sample of the proportion $f$ of the $F$-based jobs is given by cochran:1977:
Note that $\hat{F}_{k,noSDL}$ is the estimated number of $F_k$ jobs in cell $k$, and, therefore, $f\hat{F}_{k,noSDL}$ is the number of $F_k$ jobs in the sample. The conventional unbiased estimator for a sub-population mean replaces equation ((ref)) and is given by cochran:1977:
Equation ((ref)) is replaced by its analogue for $\hat{Z}\_W3_{2k}$:
Equation ((ref)) and the analysis producing equation ((ref)) are unchanged after substituting $\hat{Z}W\_3_{1k}$ for $\hat{B}_{1k}$ and $\hat{Z}\_W3_{2k}$ for $\hat{B}_{2k}$. Equation ((ref)) is not needed. Equation ((ref)) is replaced with:
Note that the divisor $\hat{F}_{k,noSDL}$ has been averaged over the implicates. Equations ((ref))-((ref)) require only the substitution of the $Z\_W3$ analogue of the $B$ quantity.
Finally, we explain the modifications to equations ((ref))-((ref)) required to estimate components of variance for $\hat{W}1_k$. The frame for $\hat{W}1_k$ is the same as the frame for $M$; hence, the total population of $M$-based jobs, $N_M$ substitutes for $N_B$. Notice that the estimand ((ref)) is a sub-population total for quarterly earnings of all employees regardless of when they were active during the quarter ($M$-based jobs); hence, we need to build the conventional finite-population variance estimand and estimator. Equations ((ref)) -((ref)) are not needed because the estimator can be derived directly. Replace equation ((ref)) with
We can proceed directly to the replacements for the variance equations. Again, because we assumed that unreported UI earnings records are missing completely at random, the finite-population variance of a total from a simple random sample of the proportion $f$ of the $M$-based jobs is given by cochran:1977:
where the estimated population of $M$-based jobs without SDL is
Note that $\hat{M}_{k,noSDL}$ is the estimated population of $M_k$ jobs in cell $k$, and, therefore, $f\hat{M}_{k,noSDL}$ is the number of $M_k$ jobs in the sample.
The conventional unbiased estimator for a sub-population total replaces equation ((ref)) and is given by cochran:1977:
Equation ((ref)) is replaced by its analogue for $\hat{W}1_{k}$:
Equation ((ref)) and the analysis producing equation ((ref)) are unchanged after substituting $\hat{W}1_{1k}$ for $\hat{B}_{1k}$ and $\hat{W}1_{2k}$ for $\hat{B}_{2k}$. Equation ((ref)) is not needed. Equation ((ref)) is replaced with:
Equations ((ref))-((ref)) require only the substitution of the $W1$ analogue of the $B$ quantity.
We summarize the results for all establishments in the universe in Table (ref) for total employment, $M$; in Table (ref) for beginning-of-quarter employment, $B$; in Table (ref) for full-quarter employment, $F$; in Table (ref) for average monthly earnings of full-quarter employees, $Z\_{W3}$; and Table (ref) for total payroll, $W1$. In all cases, $L=G=10$.
\singlespacing
\includepdf[pages=1,angle=0,scale=1.1,offset=0 -40, addtolist={1,table,Summary of Total Variability of Total Quarterly Employment (M) by Table and Count,tab:emptotall}]{qwitv_tables_20200121CBDRB-FY20-CED002-B0002.pdf} \includepdf[pages=2,angle=0,scale=1.1,offset=0 -40, addtolist={2,table,Summary of Total Variability of Beginning of Quarter Employment (B) by Table and Count,tab:emp}]{qwitv_tables_20200121CBDRB-FY20-CED002-B0002.pdf} \includepdf[pages=3,angle=0,scale=1.1,offset=0 -40, addtolist={3,table,Summary of Total Variability of Full Quarter Employment (F) by Table and Count,tab:empsall}]{qwitv_tables_20200121CBDRB-FY20-CED002-B0002.pdf} \includepdf[pages=4,angle=0,scale=1.1,offset=0 -40, addtolist={4,table,Summary of Total Variability of Average Monthly Earnings (Z\textunderscore W3) by Table and Count,tab:ametot}]{qwitv_tables_20200121CBDRB-FY20-CED002-B0002.pdf} \includepdf[pages=5,angle=0,scale=1.1,offset=0 -40, addtolist={5,table,Summary of Total Variability of Quarterly Payroll (W1) by Table and Count,tab:payroll}]{qwitv_tables_20200121CBDRB-FY20-CED002-B0002.pdf}
\doublespacing
Tables (ref)-(ref) have the same structure. The supplemental online files contain tables A1-A5, which have the same structure as the analogous table in the main text. The supplemental online files also contain additional columns for Tables 1-5. The major row label is the level of QWI tabulation. For example, the row labeled “Age $ \times $ Gender” refers to the collection of tabulations stratified by year, quarter, state, age category, and gender. The data conform to the published QWI schema in lehdschema410, which contains the levels for each of the stratifying variables. The minor row label characterizes the publication cell by its job count size. For Tables (ref) and (ref) the size classes are based on total flow-employment $M$, for Table (ref), on the values of beginning-of-quarter employment $B$, and for Tables (ref) and (ref), the classes are based on full-quarter employment $F$. The complete set of size classes we summarize are:
We report medians rather than averages for most statistics, to avoid the influence of outlier cells on the results. “Median Total Variation” reports the the total variation $\hat{V}_T$ at the median values of each component. This is the overall summary measure of data quality for the five statistics from QWI and LODES studied in this paper. The remaining columns provide additional information for interpreting total variation and attributing components to the various sources. The next three columns report the percent of total variation due to “Within” ($\hat{V}_W$), “Between (Imputation)” ($\hat{V}_B$), and “Between (Noise Infusion)” ($\hat{V}_{SDL}$), evaluated at the median values of each component. The between components are both multiplied by $(L+1)/L$ so that the components sum to one when divided by the median total variation.
We also report three quantiles of the coefficient of variation, defined as a function of the total variance and the QWI estimator in ((ref)). The quantile statistics on the coefficient of variation can be used to assess the proportionate total variation around the published value arising from all sources of error. They can also be used as a portmanteau “fitness for use” assessment of the cells in the associated row, considered as a single table.
We also produced statistics for the margin of error at the median total variation, calculated at the 90% confidence level. We used the approximate degrees of freedom as defined in equation ((ref)) to compute the margin of error. These statistics can be found in the supplemental online files with the same table numbers as in the text. The last column in each table is one-half of the 90% confidence interval width using a $t$-distribution with the indicated degrees of freedom.
We interpret the approximate margins of error as providing evidence about the overall reliability of each statistic for cells that lie in the indicated count range. For example, the median value of $B$ associated with the Age $ \times $ Gender cell in Table (ref), in the +1000 row, is 22,385. The approximate margin of error in that row is 289. Hence, the approximate 90% confidence interval is 22,385 +/- 289. The median coefficient of variation is 0.0074 or 0.74%.
The total variation or the coefficient of variation can be used to summarize the fitness-for-use of the published indicators for total flow-employment, beginning-of-quarter employment, full-quarter employment, average monthly earnings of full-quarter employees, and total quarterly payroll. It is clear from Tables (ref)-(ref) that the coefficient of variation declines monotonically as the number of jobs used in the tabulation value increases for each of the displayed quantiles. Careful attention to the magnitudes of these coefficients of variation reveals that for even the most detailed tables and for the stratifiers associated with the largest noise infusion and between-implicate variance contributions to total variability, the tabulations are very reliable when based on job counts of at least 10, and moderately reliable for job counts of three to nine. This conclusion remains valid even using the very conservative $95^{th}$ percentile of the distribution of the coefficient of variation.\footnote{The American Community Survey uses a median coefficient of variation no greater than 61% as one of its portmanteau fitness-for-use measures for the 1-year tabular summaries acs14.}
By contrast, the percent of total variation due to noise infusion is not a traditional measure of fitness for use. It captures the contribution of the error deliberately introduced to protect the confidentiality of the micro-data. It is a necessary “cost of doing business” in the production of statistics with granularity at the level published in the QWIs and LODES/OnTheMap.
In all of the tables, the total variability declines as the number of jobs in a cell increases, however the components of variation decline at different rates. The between-implicate variation declines at a faster rate than the noise infusion component as the number of jobs in a cell increases. As a result, the noise infusion component accounts for a relatively larger share of the total variability in cells that contain more jobs, but the total variability of those cells is smaller. In cells with many jobs ($100-999$ and $+1000$) the majority of the total variability is due to noise infusion. For example, the median coefficient of variation is 0.0074 or 0.74% in Table (ref) in the row Age $ \times $ Gender $+1000$, which is almost entirely due to noise infusion (95.1%). This implies that the 289 margin of error at the median associated with a 90% confidence level is due almost entirely to the confidentiality protection system, and not to any of the other sources of error.
Education is imputed for the vast majority (about 87%) of individuals in the LEHD data, based on a multistage ignorable missing data model that relies heavily on the sampling properties of the Census 2000 long form and the ACS---specifically that education is missing because the individual was not sampled mckinney:green:vilhuber:abowd:2017. By contrast, worker age and gender are imputed for less than seven percent of the individuals, race and ethnicity are imputed for about 18% of the individuals. Looking closely at the median coefficients of variation for the Age $ \times $ Gender $ \times $ Industry $ \times $ County table in comparison with the Gender $ \times $ Education $ \times $ Industry $ \times $ County table, we see that for every count range, the Age $ \times $ Gender table has less total variation than the Gender $ \times $ Education table and the percentage due to between-implicate variation is less, except for cells with measured zeros. Notice, in particular, that the percent of total variation due to between-implicate variance is substantially similar in all rows labeled $+1000$ when the table includes education, indicating that the high imputation rate for education is a limiting factor. This is not the case for tables including only age and gender or race and ethnicity. One conclusion is, thus, that the total quality of the QWI data could be substantially improved by investing in better education data. However, by conventional measures of general fitness-for-use, the tabulations involving education are of high quality but not as high as the quality of statistics that do not use education.
We have conducted the first comprehensive total error and variability analysis of five major publication variables in the Quarterly Workforce Indicators, including two key employment indicators and the most widely used earnings indicator. The beginning-of-quarter employment variable from QWI is also the primary tabulation variable in the LEHD Origin-Destination Employment Statistics; hence, our analysis is also applicable to workplace tabulations directly from LODES or as displayed in OnTheMap, including OnTheMap for Emergency Management. Tabulations involving 10 or more jobs are very reliable having median coefficients of variation that decline from a worst case of 16% (count range 10-99, detailed tables involving education) to a best case of less than one percent (count range +1000, simple tables involving education). Tabulations based on three to nine jobs represent a transition zone in the sense that the 90% confidence bound does not generally include zero, however a substantial number of cells are not statistically different from zero, especially for tabulations by education. Our analysis further reveals that the very smallest tabulations (estimated zeros and counts of one or two) are not particularly reliable in the sense that they could easily range from zero to three or more; however, the QWI tabulations already suppress estimates of one or two with a flag that warns users of their unreliability and LODES/OnTheMap primarily uses them to build larger aggregates that should be reliable by our measures. Finally, our analysis includes the deliberate error introduced by noise infusion for the purpose of statistical disclosure limitation; hence, our total variation measures promote correct inferences from the published data even in the presence of uncertainty due to SDL.
To the best of our knowledge, no other widely used statistical system based on administrative records has produced a comprehensive total error analysis to which the results in this paper can be compared. As compared to survey-based estimates like those derived from the American Community Survey, for example, the QWI employment and earnings tabulations have accuracy comparable to that of the ACS acs14, even when comparing state and PUMA-level estimates in the ACS to county and core-based statistical areas in the QWI. The LODES/OTM estimates for sub-county geographies and small sub-populations have much lower total error than estimates from the ACS for comparably-sized sub-populations. The ACS margins of error do not account for the uncertainty introduced by the edit, imputation and statistical disclosure limitation systems, whereas ours do. Designed surveys like the ACS deliver statistics on a much broader set of variables and can be used for analyses that are far outside the scope of the QWIs or LODES/OTM. But our analyses demonstrate that the total error of an administrative-records based publishing system that combines data from many sources can compare very favorably with much more expensive survey-based systems for their common domains.
\strut
\singlespacing
\printbibliography