EconBase
← Back to paper

Statistical Inference in Large Multi-way Networks

Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.

101,251 characters · 19 sections · 75 citation commands

Rendered from LaTeX for readability, not typeset faithfully. Citation keys are highlighted; maths is left as source; figures, tables and equation environments are summarised rather than reproduced; unrecognised commands are greyed out so nothing is silently dropped. Email addresses are removed.

Statistical inference in large multi-way networks

\affil[$\dagger$]{{CREST, ENSAE, Institut Polytechnique de Paris, France}} \affil[$\S$]{{ESSEC Business School, France}}

abstractWe propose a new method to estimate structural parameters in multi-way networks while controlling for rich structures of fixed effects. The method is based on a series of classification tasks and is agnostic to both the number and structure of fixed effects. In contrast to full maximum likelihood approaches, our estimator does not suffer from the incidental parameter problem. For sparsely connected networks, it is also computationally faster than PPML. We provide empirical evidence that our estimator yields more reliable confidence intervals than PPML and its bias-correction strategies. These improvements hold even under model misspecification and are more pronounced in sparse settings. While PPML remains competitive in dense, low-dimensional data, our approach offers a robust alternative for multi-way models that scales efficiently with sparsity. The method is applied to study the causal effect of a policy reform on spatial accessibility to health care in France.

{\bf JEL codes} C13 C31 C33 C55

Introduction

Network data, which are becoming available at increasingly granular levels, are receiving a great deal of attention in economics, see Grah_dePa_Book_2020. In this paper, we consider “multi-way” networks that involve interactions between entities of different nature such as importing and exporting countries, buyers and suppliers, teachers and schools, doctors and patients. The strength of the interactions in such networks is commonly measured at disaggregated levels, e.g., industries or products for trade data, consultations and medical procedures for health data, patents and citations for innovation data, etc. Multi-way network data, sometimes referred to as “polyadic”, are indexed by multidimensional indices that represent the relevant dimensions in each case, for instance exporter, importer, product, and time in the trade example.

To model connections in multi-way networks and control for unobserved heterogeneity along various dimensions, recent applied research has gradually considered models with richer structures of fixed effects, involving higher-dimensional interactions. For instance, three-way gravity models, with exporter-year, importer-year, and exporter-importer fixed effects, are common in the modern trade literature.\footnote{Recent studies recognize the economic importance of the sector or product levels, potentially leading to even richer structures of fixed effects, breinlich2024trade and Delb_Dina_WD_2020.} Yet as the structure of fixed effects becomes more complex, maximum likelihood estimators may be plagued by incidental parameter problems, see fernandez2016individual and weidner2021bias. Specifically, as the sample size grows, so too does the number of nuisance parameters representing the fixed effects, possibly creating bias in standard maximum likelihood estimation of the parameters of interest, as first described by Neym_Scot_Ecma_1948.

In this paper, we propose a novel estimator that does not suffer from the incidental parameter problem and handles multi-way gravity models. In the spirit of graham2017econometric, we regard data through the lens of graph theory. The key insight is that certain configurations of outcomes within subgroups of observations, which we call polyads, have identical sufficient statistics for the fixed effects, making their relative likelihood independent of the fixed effects. Our framework differs from graham2017econometric in two important dimensions. First, while Graham models undirected graphs, we use multipartite graphs to model multi-way networks. Second, Graham's network formation model considers only the extensive margin, i.e., the probability that potential links are realized. By contrast, we study the strength of connections in weighted networks, thus modeling both the intensive and extensive margins.

This study is connected to the strand of the gravity literature starting with silva2006log.\footnote{Their seminal paper shows that traditional log-linear OLS estimation suffers from bias under heteroskedasticity, particularly when many flows are zero, which greatly motivated the adoption of Poisson models for gravity. The properties of Poisson pseudo-maximum likelihood estimators have first been investigated by gourieroux1984pseudo in the absence of fixed effects, with one-way fixed effect panel data applications being pioneered by HHG. Recently, chen2024logs argue that log-like transformations can also distort the interpretation of coefficients as percentage effects, since they depend on the units of the outcome.} Our method compares favorably with recent econometric studies along several dimensions. First, it accommodates multi-way models with an arbitrary number of node groups, in contrast to debiasing methods such as fernandez2016individual, jochmans2017two, and weidner2021bias, which are restricted to two- and three-way structures. Second, unlike PPML estimation, our estimator has no incidental parameter problem by construction. Even relative to bias-corrected PPML procedures, our approach remains advantageous: the corrections may themselves be biased in finite samples, as weidner2021bias argue, or computationally prohibitive in large datasets, as in zylkin2024bootstrap. Third, compared with approaches such as charbonneau2012multiple, our estimator better exploits the available variability while remaining computationally feasible, and the convexity of our loss function delivers strong numerical performance relative to more general semiparametric method-of-moments procedures (e.g., jochmans2017two, yang2023three).

Our study is also connected to the literature on discrete choice models for panel and network data (Rasc_1960, Ande_JRSS_1970,Cham_Hand_1984, Hono_Kyri_Ecma_2000, Magn_Ecma_2004). Presenting the conditional likelihood methods used in these settings, the recent review of Dano_Hono_Weid_WP_2025 highlights how identification strategies relate to difference-in-differences approaches. Specifically, they provide the differencing vectors that are valid to identify the parameters of interest.\footnote{Muris_Pakel_WP_2025 follow this approach to study the formation of triadic networks. They introduce a hexad logit estimator that extends the tetrad logit estimator of graham2017econometric.} We proceed the same way for count data and multi-way networks. The polyad estimator can be thought of as a nonlinear version of a difference-in-differences estimator. Contrary to the above cited literature, the polyad method handles count data and recovers both the existence and intensity of relationships. In the special case where the count data is right-censored at one, i.e., where the intensive margin is ignored, the polyad estimator maximizes the same conditional likelihood as graham2017econometric and the studies mentioned in this paragraph .

For researchers working with sparse networks --- i.e., where most potential connections are not realized ---, our approach offers a distinct computational advantage: polyads can be constructed by looping over pairs of edges with strictly positive counts (i.e., realized connections), allowing the estimation procedure to scale with the number of observed relationships rather than the number of potential relationships. This is a significant step forward, since the usage of tetrad-based methods has been limited by its computational cost: the available methods for computing tetrad-based statistics, which work only for two- or three-way models, either (i) require looping over all pairs of edges, including unrealized connections (e.g., graham2017econometric, Muris_Pakel_WP_2025), or (ii) rely on matrix multiplications that remain slow for sparse networks (e.g., jochmans2017two). In particular, as exemplified by our experiments, our computational implementation enables the use of the polyads method on large administrative datasets. See Section (ref) for a more detailed discussion.

We establish consistency and asymptotic normality under mild assumptions, extending the current theoretical framework in two main directions: relaxing sampling assumptions and better exploiting convexity. First, unlike graham2017econometric,graham2020dyadic, we do not require the existence of a distribution to sample the nodes, thus treating the fixed effects as true parameters and revealing fundamental geometrical properties that a graph should satisfy for consistency and asymptotic normality (see Assumptions (ref) and (ref)). Second, we obtain consistency without boundedness assumptions by modifying classical results from newey1994large under the light of convex analysis tools from rockafellar-1970a and asymptotic statistics results from Andersen1982. We also avoid the existence of a limiting risk for asymptotic normality exploring results from victor, which are based on MR1026303, MR1186263.

The practical limitations of our method merit clear statement. First, we do not consider interdependencies between observations beyond those captured by fixed effects and observed covariates. Second, our approach is designed specifically for count data, rather than continuous weights. Third, the method becomes computationally inefficient in dense networks where most potential relationships are realized. Within these constraints, however, our estimator provides a powerful tool for inference in multi-way networks with high-dimensional fixed effects structures.

The remainder of the paper proceeds as follows. Section (ref) introduces the Poisson model and the structure of fixed effects. Section (ref) introduces the polyad estimator. Section (ref) establishes its theoretical properties, demonstrating consistency and asymptotic normality. Section (ref) develops the computational implementation, emphasizing how the algorithm efficiently exploits sparsity. Section (ref) documents the finite-sample properties of the estimator using artificial data and healthcare claims data.

Model assumptions

We consider a count variable $Y_{i_1i_2\cdots i_D}\in\mathbb{N}$ that is indexed by $(i_1, \dots, i_D)\in \mathcal{I} = [n_1] \times \cdots \times [n_D]$. Using the $D$-dimensional index ${\bf i} = (i_1, \dots, i_D) \in \mathcal{I}$, we represent the variable as $Y_{\bf i}$.

Throughout the paper, we think of $Y=(Y_{\bf i})_{\bf i}\in \mathbb{N}^\mathcal{I}$ as a random $D$-partite graph, with the sets $[n_1],[n_2],\dots[n_D]$ representing the nodes of each category, the multidimensional index ${\bf i}$ representing a potential (hyper-)edge of a the graph, and $Y_{{\bf i}}$ being the number of connections along edge ${\bf i}$, see the concrete examples below. We denote by $E$ the set of positive edges, i.e., the set of $D$-dimensional indices ${\bf i}$ such that $Y_{{\bf i}}>0$. The graph is sparse when the data contains many zeros, a case where our method delivers especially good results.

We assume that the dependent variable $Y_{{\bf i}}$ depends on a set of $p$ explanatory variables $X_{\bf i} \in \mathbb{R}^p$ and a set of fixed effects. A level of fixed effect is represented by a proper subset $g$ of $[D]$. By abuse of notation, we set $g({\bf i}) = (i_d : d \in g)$ and the fixed effects for level $g$ are denoted as $\theta_{g({\bf i})}^g \in \mathbb{R}$. The structure of the fixed effects in the model is represented by a collection $\mathcal{G}$ of fixed effects levels. Of particular interest to us is the structure $\mathcal{G}^{{\rm max}}$ consisting of the $D$ subsets of $[D]$ of cardinal $D-1$; in this particular case, each level of fixed effect absorbs the variations of $Y_{\bf i}$ in all but one dimension of ${\bf i}$. The set of all fixed effects is denoted by $\theta^\mathcal{G} = \{\theta^g_\rho : g\in \mathcal{G}, \rho\in g({\bf i}) \}$.

assumptionLet $\beta_\star \in \mathbb{R}^p$ be the parameter of interest. The distribution of $Y = (Y_{\bf i} \in \mathbb{N} : {\bf i}\in\mathcal{I})$ conditionally on $X = (X_{\bf i} \in \mathbb{N} : {\bf i}\in\mathcal{I})$ is $\mathbb{P}_{\beta_\star, \theta^\mathcal{G}}^{Y|X}= \bigotimes_{{\bf i} \in \mathcal{I}} \mathcal{P}(\lambda_{\bf i})$ , where $\mathcal{P}(\lambda_{\bf i})$ is the Poisson distribution with intensity $\lambda_{\bf i}>0$ given by \begin{equation} \ln \lambda_{\bf i} = \beta_\star^\top X_{\bf i} + \sum_{g \in \mathcal{G}} \theta_{g({\bf i})}^g . \end{equation} It follows that the residual $\varepsilon_{{\bf i}}=Y_{\bf i}-\lambda_{\bf i}$ satisfies $\mathbb{E} (\varepsilon_{\bf i} | X) = \mathbb{E} (\varepsilon_{\bf i} |X_{\bf i})=0$. In other words, the explanatory variables $X_{\bf i}$ are assumed to be strongly exogenous.

The log-likelihood of $(\beta, \theta^{\cal G})$ in the model (ref) at the observed graph $y$ is

equation[equation omitted — 281 chars of source]

involves a potentially high number of fixed effects. The MLE estimator of the parameter of interest $\beta_\star$ has been shown to be asymptotically biased for $D\geq 3$, see weidner2021bias as well as the experiments presented in Section (ref).

The following examples show how our framework encompasses gravity models and other classical econometric models.

example[One-way model in panel data] Taking $D= 2$ and $\mathcal{G} = \{ \{1\} \}$ yields the structure of the classical model studied by HHG \[ \ln \lambda_{i_1i_2} = \beta_\star^\top X_{i_1i_2} + \theta_{i_1}^1 \]
example[Two-way model] Taking $D = 2$ and $\mathcal{G} = \mathcal{G}^{{\rm max}} = \{ \{1\}, \{2\} \}$ yields the structure of the standard gravity model studied by silva2006log , i.e., $ \ln \lambda_{i_1i_2} = \beta_\star^\top X_{i_1i_2} + \theta_{i_1}^1 + \theta_{i_2}^2 $. The usual econometric model \[ \ln \lambda_{ij} = \beta_\star^\top X_{ij} + u_i + v_j \] obtains when relabeling the two-dimensional indices $(i_1,i_2)$ as $(i,j)$ and the fixed effects $\theta_{i_1}^1$ and $\theta_{i_2}^2$ as $u_i$ and $v_j$ respectively. In the trade literature, $i$ is an exporter, $j$ is an importer, $X_{i_1i_2}$ is a feature of the dyad (e.g., sharing borders or same language, having a free trade agreements in force).
example[Three-way model] Taking $D = 3$ and $\mathcal{G} = \mathcal{G}^{{\rm max}} = \{ \{1,2\}, \{1,3\}, \{2,3\} \}$ yields the structure of the model studied by weidner2021bias, i.e., $ \ln \lambda_{i_1i_2i_3} = \beta_\star^\top X_{i_1i_2 i_3} + \theta_{i_1i_2}^{1,2} + \theta_{i_1i_3}^{1,3} + \theta_{i_2i_3}^{2,3}$. Relabeling the edges $(i_1,i_2,i_3)$ as $(i,j,t)$ and the fixed effects $(\theta_{i_1i_2}^{1,2},\theta_{i_1i_3}^{1,3}, \theta_{i_2i_3}^{2,3})$ as $(u_{ij},v_{jt},w_{it})$, we obtain the usual econometric formulation \[ \ln \lambda_{ijt} = \beta_\star^\top X_{ijt} + u_{ij} + v_{jt}+w_{it}. \] This model is used in the trade literature in the presence of a time dimension, where $i_1$ is an exporter, $i_2$ is an importer, and $i_3$ is the time. Denoting the indices $i,j,t$ instead of $i_1, i_2,i_3$ yields a notation familiar to economists $\ln\lambda_{ijt}= \beta_\star^\top X_{ijt}+u_{ij}+v_{jt}+w_{it}$.

Estimation of the homophily parameter $\beta_\star$ via polyads

To avoid the incidental parameter problem mentioned above, we first condition the likelihood on a sufficient statistics for the fixed effects given by the degrees of the nodes, see Subsection (ref). This approach has been followed in two-way contexts by charbonneau2012multiple, graham2017econometric and jochmans2018semiparametric. We construct a loss function based on a set of `directions' that generate all the variability in the data for given degrees.

Generalized degrees and polyad transformations

Consider a level of fixed effect $g\in\mathcal{G}$. For $\rho\in g(\mathcal{I})$, the fixed effect $\theta_{\rho}^g$ enters the log-likelihood (ref) only through the quantity \[ \delta^g_\rho(y) = \sum_{{\bf i} \in \mathcal{I} : g({\bf i}) = \rho } y_{\bf i}, \] which we call the degree of $\rho$ relative to the fixed effect level $g$ in the graph $y$. This quantity generalizes the notion of degree used in graham2017econometric, capturing the total number of connections between edges ${\bf i}$ for which $g({\bf i}) = \rho$.\footnote{Consider for instance Example (ref), where the index $i_1,i_2,i_3$ are denoted $i,j,t$ as in many gravity models. The degree of (4,5) relative to the fixed effect level $u_{ij}$ is $\delta^{1,2}_{4,5}(y)=\sum_{t} y_{45t}$.} The set of degrees for the fixed effects level $g \in \mathcal{G}$ is denoted by $\delta^g(y) = \left[ \delta^g_\rho(y) \right]_{\rho \in g(\mathcal{I})}$. The set of all degrees is denoted by $\delta(y)=\left[ \delta^g(y) \right]_{g\in \mathcal{G}}$.

As announced above, we condition the likelyhood on the set of degrees $\delta(Y)$:

equation[equation omitted — 366 chars of source]

Rewriting the conditional likelihood as

eqnarray[eqnarray omitted — 1,145 chars of source]

shows that it does not depend on the fixed effects $\theta^\mathcal{G}$ regardless of their structure $\mathcal{G}$. In the next subsection, we characterize the support of the distribution of the $Y$ conditional on all degrees $\delta(Y)$.

definitionA {\bf polyad} $\xi$ of ${\cal I}=[n_1]\times \cdots\times [n_D]$ is a $2\times D$ matrix \begin{equation*} \xi = \begin{pmatrix} j_1 & j_2 & \cdots & j_D\\ j_1^\prime & j_2^\prime & \cdots & j_D^\prime\\ \end{pmatrix} \end{equation*}where for all $d\in[D]$, $j_d\neq j_d^\prime\in[n_d]$. We denote by $\Xi$ the set of all polyads of ${\cal I}$.

An edge of a polyad $\xi$ is an index ${\bf i}=(i_d)_d\in{\cal I}$ such that $i_d\in\{j_d, j_d^\prime\}$ for all $d\in[D]$. We denote by ${\cal E}(\xi)$ the set of all edges of $\xi$. Any polyad has $|{\cal E}|= 2^D$ edges. In other words, a polyad $\xi$ induces a subgraph of $Y$ made of $2^D$ edges with weights $(y_{{\bf i}})_{{\bf i}\in {\cal E}(\xi)}$.

Polyads as introduced in Definition (ref) are generalizations to the $D$-dimensional framework of tetrads from charbonneau2012multiple, graham2017econometric, jochmans2018semiparametric defined for $D=2$. The total number of polyads, i.e. the size of $\Xi$, is $\prod_{d=1}^D n_d(n_d-1)$. In the upper left corner of Figure (ref) we exemplify a polyad on $D=2$,

definitionLet $\xi\in\Xi$ be a polyad. Let ${\bf i}\in{\cal I}$. The {\bf sign of ${\bf i}$ relative to $\xi$} is defined as \begin{equation*} s_\xi({\bf i}) = \prod_{d=1}^D \left( \mathbf{1}\{i_d = j_d\} - \mathbf{1}\{i_d = j_d'\} \right). \end{equation*}

In particular, $s_\xi({\bf i}) =0$ when ${\bf i}$ is not an edge of $\xi$ and $s_\xi({\bf i})\in\{\pm1\}$ otherwise. The main purpose of the sign function $s_\xi$ is that it gives the signs of the diff-in-diff property stated below. The sign function is represented in the upper right corner of Figure (ref), each edge has a sign associated to it, notice that the signs sum zero for all axis and that the sign of ${\bf j}$ is always 1.

We now define a class of transformations indexed by polyads. These transformations act on $D$-partite graphs with integer edge weights, i.e., on the set $\mathbb{Z}^\mathcal{I}$. Recall that we see the dependent variable $Y \in \mathbb{N}^\mathcal{I}$ as a graph with nonnegative edge weights. This discrepancy plays an important role in the analysis developed below.

definitionLet $\xi\in\Xi$ be a polyad. The polyad transformation $T_\xi: \mathbb{Z}^{\cal I}\rightarrow \mathbb{Z}^{\cal I}$ is defined by $T_\xi(y) = y + s_{\xi}$ where $s_\xi=(s_\xi({\bf i}):{\bf i}\in{\cal I})$ i.e. $T_\xi(y)$ is a graph with weights given for all ${\bf i}\in{\cal I}$ by \begin{equation} T_\xi(y)_{\bf i} = y_{\bf i} +s_\xi({\bf i}). \end{equation}For all $r\in\mathbb{Z}$, $T_\xi^r(y) = y+rs_\xi$.

For any polyad $\xi$, the transformation $T_\xi$ alters only the subgraph of $y$ induced by the polyad $\xi$. In other words, $y'_{\bf i}=y_{\bf i}$ for all ${\bf i}\notin {\cal E}(\xi)$. The proposition below states the polyad transformations preserve degrees and, conversely, that they allow to generate all graphs sharing the same degrees as a given graph.\footnote{The converse result is stated in Proposition (ref) only for $\mathcal{G}=\mathcal{G}^{{\rm max}}$. In the appendix, we consider any fixed effect structure $\mathcal{G}$.}

proposition[Characterization of degree-preserving transformations] Consider any graph $y\in \mathbb{Z}^{\cal I}$ and any polyad $\xi\in\Xi$. If $y'=T_\xi(y)$, then we have \begin{equation} \delta(y')=\delta(y). \end{equation} Conversely, take two graphs $y$ and $y'$ in $\mathbb{Z}^{\cal I}$ having the same degrees, i.e., such that $\delta(y')=\delta(y)$. Suppose furthermore that the structure of fixed effect is $\mathcal{G}^{{\rm max}}$. Then there exists a finite sequence of integers $r_1,\dots,r_m\in\mathbb{Z}$ and finite sequence of polyads such that \begin{equation} y'= T_{\xi_1}^{r_1} \circ \cdots \circ T_{\xi_m}^{r_m} (y). \end{equation}
proofPick a polyad $\xi$, a fixed effect level $g\in\mathcal{G}$, $\rho\in g(\mathcal{I})$, and $y'=T_\xi(y)$. We observe that the equality \begin{equation} \sum_{{\bf i}\in\mathcal{I}:g({\bf i})=\rho} s_\xi({\bf i}) =0 \end{equation} immediately implies \[ \delta^g_{\rho}(y')= \sum_{{\bf i}\in\mathcal{I}:g({\bf i})=\rho} y'_{{\bf i}}= \sum_{{\bf i}\in\mathcal{I}:g({\bf i})=\rho} y_{{\bf i}} + s_\xi({\bf i}) = \sum_{{\bf i}\in\mathcal{I}:g({\bf i})=\rho} y_{{\bf i}} = \delta^g_{\rho}(y), \] and hence the direct part of the proposition. To prove (ref), we consider an edge ${\bf i}$ such that $g({\bf i})=\rho$. If ${\bf i}$ is not an edge of the polyad-induced subgraph, i.e., if ${\bf i}\notin\mathcal{E}(\xi)$, the sign $s_\xi({\bf i})=0$. Consider now the edges that belong to $\mathcal{E}(\xi)$. There exist $2^{D-|g|}\geq 2$ edges ${\bf i}\in \mathcal{E}(\xi)$ such that $g({\bf i})=\rho$, with half of them having $s_\xi({\bf i})=1$ and the other half having $s_\xi({\bf i})=-1$, which yields (ref). The converse part is proved in the Appendix (ref).

According to Proposition (ref), the conditioning set in the likelihood (ref) can be written as \[ \left\{\,Y \,|\, \delta(Y)= \delta(y)\,\right\}= \left\{\, Y= T_{\xi_1}^{r_1} \circ \cdots \circ T_{\xi_m}^{r_m} (y) : m\geq 0, (\xi_1,\dots,\xi_m)\in\Xi^m, (r_1,\dots,r_m)\in\mathbb{Z}^m \,\right\}, \] which can be represented only at a prohibitively high computational cost.

A classification problem on polyads and the associated estimator of $\beta_\star$

Rather than attempting to exploit the data variations within this whole set, we propose to exploit variations in the directions induced by each polyad separately, i.e., to restrict attention to sequences of polyads of length $m=1$. Because the count variable $Y_{\bf i}$ takes nonnegative values, the transformations of the graph $T_\xi^r(Y)$ that do not belong to $\mathbb{N}^\mathcal{I}$ are irrelevant because they occur with zero probability. We thus further restrict the conditioning set. For any polyad $\xi\in\Xi$, we introduce the nonnegative integers $m_\xi(y)$ and $M_\xi(y)$ given by \[ m_\xi(y) = \bigwedge_{{\bf i} : s_\xi({\bf i}) = 1} y_{\bf i} \ \ \ \text{ and }\ \ \ M_\xi(y) = \bigwedge_{{\bf i} : s_\xi({\bf i}) = -1} y_{\bf i}. \] The range of integers $r$ such that $T_\xi^r(y)\in\mathbb{N}^\mathcal{I}$ is $\{-m_\xi(y), \dots, M_\xi(y)\}$. Accordingly, we define the orbit $\mathcal{O}_\xi(y)$ of a graph $y\in\mathbb{N}^\mathcal{I}$ with respect to $\xi$ as

equation[equation omitted — 130 chars of source]

All graphs $y$ in the orbit $\mathcal{O}_\xi(Y)$ of the observed graph have the same degrees as $Y$, $\delta(y)=\delta(Y)$, and coincide with $Y$ except in the subgraph induced by $\xi$, $Y_{\bf i}=y_{\bf i}$ for ${\bf i}\notin\mathcal{E}(\xi)$.

figure[figure omitted — 616 chars of source]

We introduce the loss function at the level of the polyad $\xi$:

equation[equation omitted — 150 chars of source]

A polyad $\xi$ is not informative if the orbit $\mathcal{O}_\xi(Y)$ is a singleton, i.e., if both $m_\xi(Y)$ and $M_\xi(Y)$ are zero. For non-informative polyads, we have $\ell_\xi(y|X,\beta) =0$ for all $\beta \in \mathbb{R}^p$. We can thus restrict attention to informative (or “active”) polyads $\xi$ for which the corresponding orbit $\mathcal{O}_\xi(Y)$ has room for potential variation in the data, i.e., contains at least two distinct elements. Formally, active polyads satisfy: $|\mathcal{O}_\xi(Y)| = m_\xi(Y)+M_\xi(Y)+1\geq 2$. To simplify notations, we denote the transformed graph $T^r_\xi(y)$ by $y^r$, hence $y^r_{\bf i}=y_{\bf i}+r s_\xi({\bf i})$. The same computation as in (ref) yields

eqnarray*[eqnarray* omitted — 569 chars of source]

As already explained, all graphs in the orbit $\mathcal{O}_\xi(Y)$ coincide with $Y$ except in the subgraph induced by the polyad $\xi$. Formally, for ${\bf i}\notin\mathcal{E}(\xi)$, we have $s_\xi({\bf i})=0$ and hence $y_{\bf i}^r=y_{\bf i}$ for all $r$ between $-m_\xi$ and $M_\xi$. In the above sums over edges ${\bf i}$, we can thus restrict attention to edges ${\bf i}\in\mathcal{E}(\xi)$. From (ref), the loss function associated with the polyad $\xi$ is

equation[equation omitted — 254 chars of source]

where the generalized “difference-in-differences” (DiD) operator $\widetilde{X}_\xi$ is defined as

equation[equation omitted — 192 chars of source]

In Example (ref), consider the polyad \[ \xi=

pmatrix[pmatrix omitted — 64 chars of source]

. \] In this two-way example, we get the usual DiD formula: \[ \widetilde{X}_\xi= X_{22}- X_{21} -(X_{12} - X_{11})= X_{22}- X_{21} -X_{12} + X_{11}. \] In Example (ref), consider the polyad \[ \xi=

pmatrix[pmatrix omitted — 70 chars of source]

. \] In this three-way example, we get (the opposite of) a “triple difference” formula:

eqnarray*[eqnarray* omitted — 201 chars of source]
remark[Difference-in-differences (DiD)] For a given polyad $\xi$, we may look at the sign $s_\xi=(s_\xi({\bf i}))_{\bf i}$ as a DiD operator acting on the observed graph $y$ and the observed feature vector $(X_{\bf i})_{\bf i}$ either in an additive or a multiplicative way leading to the polyad transformation $T_\xi(y) = y+ s_\xi$ and the polyad feature $\widetilde{X}_\xi = \bigl< s_\xi, (X_{\bf i})_{\bf i} \bigr>$. The tensor $s_\xi=(s_\xi({\bf i}))_{\bf i}$ provides the sign in a DiD approach.

Finally, we call $\Xi_a$ the set of informative (or active) polyads and form the loss function taking $y = Y$:

equation[equation omitted — 151 chars of source]

Two immediate observations will play an important role in our analysis. First, because the LogSumExp function is convex,\footnote{Recall that the LogSumExp function $\mathbb{R}_{+*}^S\rightarrow\mathbb{R}$\ is given by $\mbox{LSE}(U_0,\cdots,U_{S-1};S) = \ln\left(\sum_{s=0}^{S-1} e^{U_s}\right)$.} the function $\ell_\xi(y | X, \beta)$ is convex in $\beta$ for any polyad $\xi$ and hence the loss function is convex. Second, given two active polyads $\xi$ and $\xi'$, the terms $\ell_\xi(Y|X,\beta)$ and $\ell_{\xi'}(Y|X,\beta)$ in the above some are not independent if the two polyads share at least one edge, i.e., if $|\mathcal{E}(\xi)\cap\mathcal{E}(\xi')|\geq 1$.

We are now in a position to define our loss function and the associated estimator.

definitionGiven the observed graph $Y$, our loss function is $\beta \to \widehat{L}_\Xi(Y | X, \beta)$ and the polyads estimator of the parameter of interest $\beta_\star$ is given by \begin{equation} \widehat{\beta}_\Xi = \operatorname*{arg\,min}_{\beta} \widehat{L}_\Xi(Y | X, \beta). \end{equation}

The above polyad estimator can be interpreted in relation to a logit classification problem. Specifically, given a polyad $\xi$ and the observed orbit $\mathcal{O}_\xi(Y)$, the minimal rank $m_\xi(Y)$ is distributed according to a conditional Logit model, see McFa_Chap_1974. To see this, consider the transformed graph $\underline{y}=T^{-m_\xi(Y)}(Y)$, which can be thought of as the “minimal” graph in the orbit of the observed graph $y$. In the example of Figure (ref), the graph $\underline{y}$ is the second graph among the six graphs shown on the bottom line (that is for $r=-m_\xi(y)=-1$). The observed orbit $\mathcal{O}_\xi(Y)$ can thus be represented as \[ \mathcal{O}_\xi(Y) := \left\{ \, T_\xi^m(\underline{y}): 0\leq m \leq |\mathcal{O}_\xi(Y)|-1 \,\right\}. \] If the polyad $\xi$ is active, the size of the orbit $|\mathcal{O}_\xi(Y)|= m_\xi(Y)+ M_\xi(Y)+1$ is greater than 2. Changing the indices $m=r+m_\xi(Y)$ in (ref) yields the conditional logit structure for the distribution of $m_\xi(Y)$

equation[equation omitted — 333 chars of source]
remark(Analogy with conditional likelihood methods used in panel and network data) For any active polyads $\xi\in\Xi_a$, the tensor $(s_\xi({\bf i}))_{\bf i}$, whose entries belong to $\{-1,0,1\}$, is a “differencing vector” in the sense of Dano_Hono_Weid_WP_2025. It applies linearly to the independent features $X$ to deliver the generalized difference-in-differences term $\widetilde{X}_\xi$. The corresponding transformation $T_\xi$ plays the same role for the count data $Y$. Conditionally on the observed graph $y$ belonging to the orbit $\mathcal{O}_\xi(Y)$, we thus obtain a conditional logit classification problem, with the number of alternatives, $|\mathcal{O}_\xi(Y)|-1$, being polyad-specific.

The conditional logit distribution of $m_\xi(Y)$ yields the following result.

lemmaFor every polyad $\xi$ and $y=(y_{\bf i})_{{\bf i}}$, the function $\beta\to \ell_\xi(y | X, \beta)$ has gradient \begin{equation} \nabla_\beta \ell_\xi(y | X, \beta) = \left( \operatorname{\mathbb{E}}_\beta\left[ m_\xi(Y) |X, Y \in \mathcal{O}_\xi(y) \right] - m_\xi(y) \right) \widetilde{X}_\xi \end{equation} and its Hessian is non-negative since it is given by \begin{equation} \nabla_\beta^2 \ell_\xi(y| X, \beta) = \mathbb{V}_\beta\left[ m_\xi(Y) |X, Y \in \mathcal{O}_\xi(y) \right] \widetilde{X}_\xi\widetilde{X}_\xi^\top \end{equation}
proofDenote by $p_\xi(m;\beta)$ the conditional probability that $m_\xi(Y)=m$ given by (ref). Notice that \begin{eqnarray} \nabla_\beta p_\xi(m;\beta) &=& m p_\xi(m;\beta) \widetilde{X}_\xi- \sum_{m'=0}^{|\mathcal{O}_\xi(y)|-1} m' p_\xi(m;\beta) p_\xi(m';\beta) \widetilde{X}_\xi \nonumber\\ &=& p_\xi(m;\beta) \left\{ m - \operatorname{\mathbb{E}}_\beta\left[ m_\xi(Y) |X, Y \in \mathcal{O}_\xi(y) \right] \right\} \widetilde{X}_\xi. \end{eqnarray} Using $\ell_\xi(y | X, \beta)=-\ln p_\xi(m_\xi(y);\beta)$, we get \[ \nabla_\beta \ell_\xi(y | X, \beta)= \widetilde{X}_\xi \sum_{m=0}^{|\mathcal{O}_\xi(y)|-1} p_\xi(m; \beta) \,(m-m_\xi(y)) , \] which gives (ref). Differentiating the above equality and using (ref) yields \[ \nabla_\beta^2 \ell_\xi(y| X, \beta) = \widetilde{X}_\xi\widetilde{X}_\xi^\top \sum_{m=0}^{|\mathcal{O}_\xi(y)|-1} p_\xi(m; \beta) \,(m-m_\xi(y)) \left\{ m- \operatorname{\mathbb{E}}_\beta\left[ m_\xi(Y) |X, Y \in \mathcal{O}_\xi(y) \right]\right\}, \] and hence (ref).

As mentioned above, the loss function $L(y | X, \beta)$ defined by (ref) is convex in $\beta$. Thanks to Lemma (ref), we can now compute its Hessian \[ \nabla^2_\beta \widehat{L}_\Xi(y | X, \beta) = \sum_{\xi \in \Xi_a} \mathbb{V}_\beta\left[ m_\xi(Y) | Y \in \mathcal{O}_\xi(y) \right] \widetilde{X}_\xi\widetilde{X}_\xi^\top \] and discuss strict convexity.

lemmaThe loss function $\widehat{L}_\Xi(y | X, \beta)$ defined by (ref) is strictly convex if and only if $ \sum_{\xi \in \Xi_a} \widetilde{X}_\xi \widetilde{X}_\xi^\top \succ 0$.

In other words, $\widehat{L}_\Xi(y|X,\beta)$ is strictly convex in $\beta$ if the DiD-features vary in the data.

proofSince $|\Xi_a| < \infty$ for any finite sample, $|\mathcal{O}_\xi(y)| > 1$ for all $\xi\in\Xi_a$ and the shape of the distribution of $m_\xi(Y)$ in (ref), the minimum $\underline{\sigma}^2(\beta) := \min_{\xi\in\Xi_a} \mathbb{V}_\beta\left[ m_\xi(Y) | Y \in \mathcal{O}_\xi(y) \right]$ is strictly positive for all $\beta\in\mathbb{R}^p$. Therefore, for all $\beta\in\mathbb{R}^p$, \[ \nabla^2_\beta L(y | X, \beta) \succeq \underline{\sigma}^2(\beta) \sum_{\xi \in \Xi_a} \widetilde{X}_\xi\widetilde{X}_\xi^\top.\]

Large sample properties of the polyads estimator

In this section we establish consistency and asymptotic normality of the polyads estimator introduced in Section (ref). The concept of an active polyad is central to both our theoretical results and the implementation of the method (see Section (ref)). Recall that a polyad $\xi$ is active when its orbit satisfies $|\mathcal{O}_\xi(Y)| > 1$, meaning it exhibits variation from which the parameters can be identified. This observation motivates normalizing the loss function by the expected number of active polyads and suggests deriving limiting behavior as the number of active polyads increases.

Let $\widehat{N}_a = |\Xi_a|$ denote the observed number of active polyads and let $N_a$ denote its conditional expectation given $X$ under Assumption (ref). Formally,

equation[equation omitted — 261 chars of source]

As mentioned in Remark (ref), $N_a$ plays the role of the average of number of data, we therefore normalize the loss function by $N_a$, defining

equation[equation omitted — 209 chars of source]

In machine learning $\beta\to Q_\Xi(\beta)$ is refered as the risk function.

Recall that $n = \prod_{d=1}^D n_d$ is the size of the graph indexed by $\mathcal{I} = [n_1] \times [n_2] \times \dots \times [n_D]$ and that if $\Xi$ is the set of polyads given by the indices $\mathcal{I}$, then $|\Xi|$ has order $n^2$. Through this section, one is given a sequence $\left(\Xi^{(1)}, \Xi^{(2)}, \dots\right)$ of families of polyads, defined over growing sets of indices. Let $n_d^{(k)}$ be the size of the $d$-th dimension associated with $\Xi^{(k)}$. We impose no restrictions on how each dimension $n_d^{(k)}$ grows, requiring only that $n^{(k)} = \prod_{d=1}^D n_d^{(k)} \to \infty$ --- or, equivalently, $|\Xi^{(k)}|\to\infty$. Notably, our results accommodate short panels where one dimension remains bounded while others diverge. For three-way models, this includes settings where the PPML estimator suffers from the incidental parameter problem weidner2021bias. We also do not assume the existence of a limiting function $Q_\infty$, thus avoiding restrictive assumptions on the asymptotic behavior of fixed effects and covariates. In particular, we do not impose that fixed effects or covariates are drawn from any probability distribution. To circumvent a convoluted notation we use $(k)$ as index in place of $\Xi^{(k)}$, for instance, $\widehat{\beta}_{\Xi^{(k)}}$ is written as $\widehat{\beta}^{(k)}$. Also, notice that associated with each $(k)$ we also have covariates $X^{(k)}$, fixed effects $\theta^{(k)}$ and random variables $Y^{(k)}$.

Finally, unlike graham2017econometric,jochmans2018semiparametric, we do not require the components of $\left[n_d^{(k)}\right]$ to arise from random sampling. This permits a more general interpretation of the data generating process. Consider a two-way model of doctor-patient interactions. Under the framework of graham2017econometric,jochmans2018semiparametric, one assumes the existence of a large population graph containing all doctors and patients, from which patients and doctors are randomly sampled, inducing distributions on both fixed effects and covariates. Our approach is more constructive: for each $(k)$, the distribution of fixed effects and covariates may differ entirely. In the doctor-patient example, our framework accommodates data collection that expands geographically. For instance, initially observing patients and doctors in Paris only, then adding doctors from Marseilles, then adding also patients from Marseilles, and so forth. Imposing a random sampling structure would limit this possibility, as any finite sample could contain doctors from Marseilles with positive probability.

Consistency

The polyads estimator (ref) is defined as the minimizer of a loss function that is itself the sum of possibly non-independent terms. Our consistency result proceeds in two steps. First, we show that the normalized empirical loss $\widehat{Q}^{(k)}$ converges to its expectation $Q^{(k)}$ as $k\to\infty$. Second, we establish regularity of $Q^{(k)}$ around its minimum $\beta_\star$. Note that at no point do we require the existence of a limit risk function $Q_\infty$.

Assumption (ref) controls the approximation between the empirical and expected versions of $\widehat{Q}^{(k)}$. Observe that $\ell_\xi$ and $\ell_{\xi'}$ are independent as long as $\xi$ and $\xi'$ share no edges. Moreover, a polyad $\xi$ contributes to the loss only when it is active. Thus, Assumption (ref) controls the amount of dependence across the terms of $\widehat{Q}^{(k)}$, enabling a law of large numbers for the normalized losses.

assumptionWe assume that, as $k \to \infty$, \begin{equation*} \operatorname{\mathbb{E}}_{\beta_\star, \theta^{(k)}}\left[ \left. \sum_{\xi,\xi' \in \Xi^{(k)}} \mathbf{1}_{ \xi and \xi' are active} \right|X^{(k)}\right] \gg \operatorname{\mathbb{E}}_{\beta_\star, \theta^{(k)}}\left[ \left. \sum_{\xi,\xi' \in \Xi^{(k)}} \mathbf{1}_{ \xi and \xi' are active and |\mathcal{E}(\xi) \cap \mathcal{E}(\xi')| \geq 1 } \right|X^{(k)} \right] \end{equation*}

Assumption (ref) is not restrictive: as $n^{(k)}$ grows, the number of positive entries $Y_{\bf i}^{(k)}$ with disjoint indices should also increase, and therefore, the number of pairs of active polyads sharing no edge should outnumber those sharing at least one edge. Notice that if one assumes a distribution on the fixed effects and on the covariates this assumption is automatically true, as in this case the probability that $\xi$ and $\xi'$ are both active is always upper and lower bounded by a constant, thus it is just a matter of comparing the number of pairs of polyads -- which has order $(n^{(k)})^4$ -- and the number of pairs of polyads sharing an edge -- which has order $(n^{(k)})^3$. This last approach is used by graham2017econometric and jochmans2018semiparametric.

Next, Assumption (ref) ensures enough regularity of $Q^{(k)}$ around the minimum $\beta_\star$ so that it can be identified.

assumptionWe assume that as $k \to \infty$, the polyads estimator $\widehat{\beta}^{(k)}$ defined in (ref) is the unique minimizer of $\widehat{Q}^{(k)}(\beta)$. We assume that there exists $\varepsilon_0>0$ such that \begin{itemize} • there exists $L_\star\in\mathbb{R}$ such that $\inf_k Q^{(k)}(\beta_\star)\geq L_\star$ and for all $\beta\in \beta_\star + 3 \varepsilon_0 B_2$, there exists $L$ satisfying $\sup_k Q^{(k)}(\beta)\leq L$; • for all $0<\varepsilon\leq \varepsilon_0$ there exists $\eta>0$ and $n_0$ such that for all $n\geq n_0$ and all $\beta\in\mathbb{R}^p$, if $\left\|\beta - \beta_\star\right\|_2 = \varepsilon$ then $Q^{(k)}(\beta) - Q^{(k)}(\beta_\star)\geq \eta$. \end{itemize}

We are now in position to state the following consistency result:

theorem[Consistency of the polyads estimator]Grant Assumptions (ref), (ref) and (ref). Then, for almost all $\left(X^{(k)}\right)_k$ and for all $\varepsilon>0$, if $n^{(k)} \to \infty$ as $k\to\infty$, then \[\mathbb{P}_{\beta_\star, \theta^{(k)}}^{\left. Y^{(k)}\right|X^{(k)}}\left[ \left\|\widehat{\beta}^{(k)} - \beta_\star\right\|_2\geq \varepsilon\right]\to 0.\]
proofThe proof can be found in Section (ref). The proof uses the convexity of the loss function as a key ingredient. It is based on a slight modification of Theorem 2.7 of newey1994large that requires revisiting classical results in convex analysis (such as those from Chapter 10 in rockafellar-1970a), as well as some results in asymptotic statistics from newey1994large and Andersen1982.

Asymptotic normality

Before introducing the assumptions and main result on the asymptotic normality of the polyads estimator we first investigate the error $\beta^{(k)} - \beta_\star$. A Taylor approximation yields, for some $\overline{\beta}^{(k)}$ between $\beta_\star$ and $\widehat{\beta}^{(k)}$, \[ \nabla \widehat{Q}^{(k)}\left(\beta_\star\right) - \nabla \widehat{Q}^{(k)}\left(\widehat{\beta}^{(k)}\right) = \nabla^2 \widehat{Q}^{(k)}\left(\overline{\beta}^{(k)}\right) \left(\beta_\star - \widehat{\beta}^{(k)}\right), \] which implies

equation[equation omitted — 219 chars of source]

If $\widehat{\beta}^{(k)}$ is close to $\beta_\star$, $\widehat{Q}^{(k)}$ is a good approximation of $Q^{(k)}$ and $\nabla^2 Q^{(k)}(\beta_\star) \to \Gamma$ for some invertible $\Gamma$ we may approximate \[ \widehat{\beta}^{(k)} - \beta_\star \approx - \Gamma^{-1} \nabla \widehat{Q}^{(k)}\left(\beta_\star\right).\] The last approximation shows that to control the fluctuations of $\widehat{\beta}^{(k)} - \beta_\star$ we need to control the fluctuations of $\nabla \widehat{Q}^{(k)}\left(\beta_\star\right)$, which is not a sum of independent random variables. Following graham2017econometric, jochmans2018semiparametric, we use Hájek projections to approximate $\nabla \widehat{Q}^{(k)}\left(\beta_\star\right)$ by a sum of independent random variables for which classical central limit theorems hold. The following assumption is useful to control the error of this approximation:

assumptionGiven $c \in \mathbb{R}^p$, define $w_\xi = \bigl< \nabla \ell_{\xi}(\beta_\star),c \bigr>$. We assume that, as $k \to \infty$, \begin{equation*} \operatorname{\mathbb{E}}_{\beta_\star, \theta^{(k)}}\left[ \left. \sum_{\xi,\xi' \in \Xi^{(k)}} w_\xi w_{\xi'} \mathbf{1}_{|\mathcal{E}(\xi) \cap \mathcal{E}(\xi')| = 1 } \right|X^{(k)}\right] \gg \operatorname{\mathbb{E}}_{\beta_\star, \theta^{(k)}}\left[ \left. \sum_{\xi,\xi' \in \Xi^{(k)}} |w_\xi w_{\xi'}| \mathbf{1}_{|\mathcal{E}(\xi) \cap \mathcal{E}(\xi')| \geq 2 } \right|X^{(k)}\right]. \end{equation*}

Assumption (ref) bears resemblance to Assumption (ref). Assumption (ref) asks for the active polyads to be distributed among the edges of $Y_{\bf i}$ in a way that no edge ${\bf i}$ has a substantial amount of active polyads $\xi$ such that ${\bf i} \in \mathcal{E}(\xi)$. Meanwhile and up to the weights $w_\xi$, which are zero when $\xi$ is not active, Assumption (ref) looks at the cases where two polyads share one edge and requires that, up to some weights, in most of these cases only one edge is being shared.

Assumption (ref) below is a technical assumption to control the convergence of the Hessian:

assumptionWe assume that \begin{itemize} • $\left(Q^{(k)}\right)_k$ is equicontinuous, i.e., for all $\beta_0\in \mathbb{R}^p$, for all $\varepsilon>0$, there exists $k_1$ and $\eta>0$ such that for all $k\geq k_1$ and all $\beta_1\in \mathbb{R}^p$, if $\left\|\beta_1 - \beta_0\right\|_2\leq \eta$ then $|Q^{(k)}(\beta_0) - Q^{(k)}(\beta_1)|\leq \varepsilon$; • $\nabla^2 Q^{(k)}(\beta_\star) \to \Gamma$ is invertible; • the sequence of Hessian of $\left(Q^{(k)}\right)_k$ is equicontinuous over $B_2(\beta_\star, 1)$. \end{itemize} Moreover, we assume that the fixed effects and the covariates are uniformly bounded as $k \to \infty$.

In order to present the main result we introduce the quantity $\Sigma^{(k)}$, which is the covariance of the Hájek projection of $\nabla \widehat{Q}^{(k)}\left(\beta_\star\right)$ and is given by

equation[equation omitted — 281 chars of source]

where \[ \bar{s}_{{\bf i}}^{(k)} = \sum_{\xi:{\bf i}\in{\cal E}(\xi)} {\mathbb E}_{\beta^*}\left[\nabla \ell_\xi\left(Y^{(k)} | X^{(k)}, \beta_\star\right) \left| X^{(k)}, Y^{(k)}_{\bf i}\right.\right]. \]

Under Assumption (ref), $\Sigma^{(k)}$ will play the role of the asymptotic variance of the polyads estimator if it does not vanish to zero as $k\to+\infty$ and if a third moment condition holds. We gather these last conditions in Assumption (ref) below:

assumptionGiven $c \in \mathbb{R}^p$. We assume that \begin{equation} N_a^{-1}\operatorname{\mathbb{E}}_{\beta_\star, \theta^{(k)}}\left[ \left. \sum_{\xi,\xi' \in \Xi^{(k)}} \mathbf{1}_{ \xi and \xi' are active} \right|X^{(k)}\right] \not\gg \operatorname{\mathbb{E}}_{\beta_\star, \theta^{(k)}}\left[ \left. \sum_{\xi,\xi' \in \Xi^{(k)}} w_\xi w_{\xi'} \mathbf{1}_{|\mathcal{E}(\xi) \cap \mathcal{E}(\xi')| = 1 } \right|X^{(k)}\right]. \end{equation} For $m\in\{2,3\}$, we define $\widehat{S}_m^{(k)} = \sum_{{\bf i}\in{\cal I}} |Z_{\bf i}|^m$ where $Z_{\bf i} = \bigl< \bar s_{\bf i}, \Gamma^{-1}c \bigr>$, $S_m^{(k)} = {\mathbb E}_{\beta_\star}\left[\left.\widehat{S}_m^{(k)}\right|X^{(k)}\right]$ and $V_m^{(k)} = \sum_{\bf i} {\mathbb V}_{\beta_\star}\left[|Z_{\bf i}|^m\left|X^{(k)}\right.\right]$. Assume that exists a sequence $\left(a^{(k)}\right)_k$ such that as $k \to \infty$, $a^{(k)}\to \infty$ and \begin{equation} S_3^{(k)}+ a^{(k)}\sqrt{V_3^{(k)}} \ll \left( S_2^{(k)} - a^{(k)} \sqrt{V_2^{(k)}}\right)^{3/2}. \end{equation}

Assumption (ref) is useful to bound the third moment of the Hájek projection of $\nabla \widehat{Q}^{(k)}\left(\beta_\star\right)$ in a central limit theorem and is not restrictive. It is satisfied as long as the quantities $\bar{s}_{\bf i}$ (and their variances) are not concentrated on a limited number of edges ${\bf i}$ as $k\to\infty$.

theorem[Asymptotic normality of the polyads estimator] Grant Assumptions (ref) to (ref) for some $c\in\mathbb{R}^p$, $c\neq0$. It holds that, for almost all $\left(X^{(k)}\right)_k$, conditionally on $\left(X^{(k)}\right)_k$, if $n^{(k)} \to \infty$ as $k \to \infty$, then \begin{equation*} \frac{\bigl< \widehat{\beta}^{(k)} - \beta_\star,c \bigr>}{\sqrt{c^\top \Sigma^{(k)} c}} \overset{d}{\to} {\cal N}(0,1). \end{equation*}
proofThe proof of Theorem (ref), given in Section (ref), follows from a general lemma presented in Section (ref), adapted from Brunel’s lecture notes victor, which reference MR1026303 and MR1186263. Unlike the classical setup, where the loss is a sum of independent terms, we handle dependent polyads and do not assume the existence of a limiting $Q_\infty$, only a limiting covariance matrix at $\beta_\star$. Hence, we extend the standard asymptotic normality proof under convexity to this dependent setting.

In Theorem (ref), the asymptotic normality is written in terms of $\Sigma^{(k)}$, which cannot be directly evaluated from the sample. In Section (ref) we discuss two alternatives to approximate $\Sigma^{(k)}$ from the sample; in Section (ref) we provide empirical validation that the confidence intervals obtained by each approach are accurate.

Computational implementation

Lemma (ref) provides the tools to solve the optimization problem (ref) defining the polyads estimator $\widehat{\beta}_\Xi$. Since each term in the loss function is convex and we have expressions for both their gradients and their Hessians, we can efficiently solve the minimization problem using Newton's method. However, computing the gradient and Hessian involves summing over all polyads $\xi \in \Xi$, which naively requires looping over $\prod_{d=1}^D n_d (n_d - 1)$ terms, an operation that quickly becomes computationally expensive. The main goal of this section is to reduce this complexity by avoiding unnecessary iterations, which is done characterizing the set of active polyads $\Xi_a$. Besides that, we also discuss the implementation of two approximations of the variance. These approximations are essential to construct confidence intervals. The methods presented in this section are of special interest when $Y$ is sparse. In particular, the computational complexity of our methods outperforms the PPML alternative correia2019ppmlhdfe when the size of $E = \{ {\bf i} : Y_{\bf i} > 0 \}$ is of order smaller than $\sqrt{n}$.

Permutating polyads

A key tool that we explore to obtain efficient computational implementations of the polyads estimator is their invariance to permutations. A permutation of a polyad defined by $({\bf i}, {\bf i}')$ is obtained by flipping some indices between ${\bf i}$ and ${\bf i}'$. For instance, \[ \xi'=

pmatrix[pmatrix omitted — 46 chars of source]

is a permutation of \xi=

pmatrix[pmatrix omitted — 46 chars of source]

on indices d\in\{2,3\}. \] We say that a permutation is odd when an odd number of indices are flipped and even when an even number of indices are flipped. Notice that each polyad $\xi$ has a total of $2^D$ unique permutations, including itself. Lemma (ref) below collects the main tools that will be necessary in this section:

lemmaIt holds that: \begin{enumerate}[(i)] • If ${\bf i} \in \mathcal{E}(\xi)$ for some $\xi$, then ${\bf i}$ belongs to all $2^D$ permutations of $\xi$ and exists exactly one permutation that can be written as $({\bf i}, {\bf i}')$ for some ${\bf i}' \in \mathcal{I}$. • Let $\xi'$ be any permutation of $\xi$. Then $s_{\xi'}({\bf i}) = s_{\xi}({\bf i}), \forall {\bf i}$ iff the permutation is even and $s_{\xi'}({\bf i}) = -s_{\xi}({\bf i}), \forall {\bf i}$ iff the permutation is odd. In particular, for odd permutations $m_{\xi'}(y) = M_\xi(y)$ and $M_{\xi'}(y) = m_\xi(y)$ for all $y \in \mathbb{Z}^\mathcal{I}$ and for even permutations $m_{\xi'}(y) = m_\xi(y)$ and $M_{\xi'}(y) = M_\xi(y)$ for all $y \in \mathbb{Z}^\mathcal{I}$. • If $\xi'$ is a permutation of $\xi$, then $\ell_\xi(y|X,\beta) = \ell_{\xi'}(y|X,\beta)$ for all $\beta \in \mathbb{R}^p$. \end{enumerate}
proofTo see (i), notice that flipping the $d$-th index of ${\bf i}$ with the $d$-th index of ${\bf i}'$ produces a new edge that still belongs to $\mathcal{E}(\xi)$. Repeating this operation for all possible combinations of indices produces all $2^D$ permutations. Also, given ${\bf i}\in\mathcal{E}(\xi)$, to find the unique ${\bf i}'$ such that $({\bf i}, {\bf i}')$ is a permutation of $\xi$ one just needs to flip the $d$-th index of ${\bf i}$ with the $d$-th index of ${\bf i}'$ if and only if $i_d \neq i_d'$. To see (ii), notice that flipping one index changes the sign of $s_\xi({\bf i})$, thus flipping an even number of indices preserves the sign while flipping an odd number of indices changes it. The expressions for $m_{\xi'}(y)$ and $M_{\xi'}(y)$ follow directly from the definition. Finally, (iii) follows observing that $\mathcal{O}_\xi(y) = \mathcal{O}_{\xi'}(y)$ and so \[ \mathbb{P}_\beta( m_\xi(Y) = m_\xi(y) | X, Y \in \mathcal{O}_\xi(y) ) = \begin{cases} \mathbb{P}_\beta( m_{\xi'}(Y) = m_{\xi'}(y) | X, Y \in \mathcal{O}_{\xi'}(y) )\text{, if the permutation is even}\\ \mathbb{P}_\beta( M_{\xi'}(Y) = M_{\xi'}(y) | X, Y \in \mathcal{O}_{\xi'}(y) )\text{, if the permutation is odd} \end{cases}. \] Since the event $M_{\xi'}(Y) = M_{\xi'}(y)$ is equivalent to $m_{\xi'}(Y) = m_{\xi'}(y)$ we are done.

Efficiently gathering all active polyads

A direct conclusion of Lemma (ref) is that if $\xi \in \Xi_a$, then exists a permutation $\xi'$ of $\xi$ such that $i_d < i_d'$ for all $d = 2, \dots, D$ and $m_{\xi'}(y)>0$. If $M_{\xi'} = 0$ then this permutation is unique, but if $M_{\xi'} > 0$ then there are two such permutations, one with $i_1 < i_1'$ and another with $i_1 > i_1'$. Based on this observation we define the following set: \[ \Xi_a^\star = \left\{ \xi = ({\bf i}, {\bf i}') : m_\xi(y) > 0 \text{ and }i_d < i_d' \,\forall d=2,\dots,D \text{ and } (M_\xi(y) = 0 \text{ or } i_1 < i_1') \right\}. \]

The set $\Xi_a^\star$ contains exactly one permutation of each active polyad $\xi \in \Xi_a$ and, by part (iii) of Lemma (ref),

equation[equation omitted — 167 chars of source]

Thus, to solve (ref) it suffices to look at all polyads in $\Xi_a^\star$. The definition of $\Xi_a^\star$ also leads to an efficient method to construct it. Notice that $m_\xi(y)$ it is positive if and only if $y_{\bf i} > 0$ for all ${\bf i}$ with $s_\xi({\bf i}) = 1$. In particular, we need at least $y_{\bf i}$ to be positive to have $m_\xi(y)$ positive. This suggests looping over pairs ${\bf i},{\bf i}' \in E$. The procedure to do it differs slightly depending on the parity of $D$.

First, take $D=2$ and let $i_1 \neq i_1'$ be given. To have $({\bf i}, {\bf i}') \in \Xi_a^\star$ we need to find $i_2 \neq i_2'$ such that $y_{i_1 i_2} \wedge y_{i_1' i_2'} > 0$. Now let $D=3$ and $i_1 \neq i_1'$ be given. We search for $i_2 \neq i_2'$ and $i_3 \neq i_3'$ satisfying $y_{i_1 i_2 i_3} \wedge y_{i_1 i_2' i_3'} \wedge y_{i_1' i_2 i_3'} \wedge y_{i_1' i_2' i_3} > 0$, in particular, $y_{i_1 i_2 i_3} \wedge y_{i_1 i_2' i_3'} > 0$. More generally, given $i_1$ we let \[E_{i_1} = \{ (j_2, \dots, j_D) : y_{i_1j_2\dots j_D} > 0 \},\] it holds that if $({\bf i}, {\bf i}') \in \Xi_a^\star$, then (i) $(i_2, \dots, i_D) \in E_{i_1}$ and $(i_2',\dots, i_D') \in E_{i_1'}$ when $D$ is even; or (ii) $(i_2, \dots, i_D), (i_2', \dots, i_D') \in E_{i_1}$ when $D$ is odd. Thus, one only needs to loop over the pairs $i_1 \neq i_1'$ and, for each of these pairs, over $((i_2, \dots, i_D), (i_2', \dots, i_D')) \in E_{i_1} \times E_{i_1'}$ (if $D$ is even) or $((i_2, \dots, i_D), (i_2', \dots, i_D')) \in E_{i_1} \times E_{i_1}$ (if $D$ is odd). Then, one simply verifies the remaining conditions for $({\bf i}, {\bf i}') \in \Xi_a^\star$. This procedure is summarized in Algorithm (ref).

algorithm[algorithm omitted — 699 chars of source]

The next theorem establishes the computational complexity of constructing $\Xi_a^\star$ using Algorithm (ref).

theoremWhen $D$ is odd assume there exists $c \geq 1$ that $|E_{i_1}| < c\frac{|E|}{n_1}$ for all $i_d \in [n_d]$. Make no assumption if $D$ is even. The set $\Xi_a^\star$ can be computed in $O(|E|^2)$ using Algorithm (ref).
proofFirst, notice that checking for $m_\xi(y)>0$ and $M_\xi(y)=0$ requires checking the values of all $2^D$ edges in $\mathcal{E}(\xi)$. Implementing the sets $E_{i_1}$ as hash tables allows us to check these values in constant time. Thus, we just need to count the number of times the innermost loop is executed. If $D$ is even, the innermost loop is executed $\sum_{i_1 \neq i_1'} |E_{i_1}||E_{i_1'}| = \left( \sum_{i_1} |E_{i_1}| \right)^2 - \sum_{i_1} |E_{i_1}|^2 \leq |E|^2$ times. If $D$ is odd, the innermost loop is executed $\sum_{i_1 \neq i_1'} |E_{i_1}|^2 \leq c \frac{|E|}{n_1} \sum_{i_1 \neq i_1'} |E_{i_1}| \leq c|E|^2$ times. Thus, in both cases the total complexity is $O(|E|^2)$.
remarkIn practice, one not only keep track of the polyads in $\Xi_a^\star$ but also of their corresponding edge values $\{ y_{\bf i} : {\bf i}\in\mathcal{E}(\xi) \}$ and of $\widetilde{X}_\xi$. This precomputation allows us to avoid recomputing these quantities at each iteration of the optimization algorithm. Besides that, implementing $E_{i_1}$ as an ordered list allows the usage of binary search, which although theoretically slower than a hash table, tends to be faster in practice. Another key point is that we do not require all features $X_{\bf i}$ to be precomputed. All that suffices is a function that can map ${\bf i}$ into $X_{\bf i}$, this function will be called $2^D$ times for each active polyad to obtain $\widetilde{X}_\xi$. This is essential to get an efficient implementation of our method, otherwise the computational cost would be at least the cost of computing all features, which is $O(n)$.

Solving the optimization problem

Once the set of polyads $\Xi_a^\star$ is computed, we minimize the loss (ref) using Newton's method. Lemma (ref) provides closed-form expressions for the gradient and Hessian of each $\ell_\xi$, so a Newton step can be computed exactly. When the loss is strictly convex (see Lemma (ref)), Newton's method converges from any initial value $\beta^0$. Algorithm (ref) displays one update step from $\beta^t$ to $\beta^{t+1}$ for $t\geq0$.

algorithm[algorithm omitted — 426 chars of source]

The function EvaluateMoments in Algorithm (ref) must return the expectation and variance of $m_\xi(Y)$ conditioned on $Y \in \mathcal{O}_\xi(y)$ when $Y$ has law parametrized by $\beta = \beta^t$. Notice that from (ref), for all $\beta$,

align*[align* omitted — 376 chars of source]

where \[ v(m,\xi;\beta) = m \,\beta^\top \widetilde{X}_\xi \;-\; \sum_{{\bf i} \in \mathcal{E}(\xi)} \ln \bigl(\underline{y}_{\bf i}^{\,m}! \bigr), \] and $\underline{y}_{\bf i}^{\,m} = y_{\bf i}+(m-m_\xi(y))s_\xi({\bf i})$ . Directly evaluating $v(m,\xi;\beta)$ for each $m$ would require computing log factorials, we avoid this computation by noticing that \[ v(m,\xi;\beta) - v(m-1,\xi;\beta) = \beta^\top \widetilde{X}_\xi + \sum_{{\bf i} : s_\xi({\bf i}) = -1} \ln\!\bigl(y_{\bf i} - (m-1-m_\xi(y))\bigr) - \sum_{{\bf i} : s_\xi({\bf i}) = 1} \ln\!\bigl(y_{\bf i} + (m-m_\xi(y))\bigr), \] so the values $v(m,\xi;\beta) - v(0, \xi; \beta)$ can be computed sequentially by cumulative summation without a log factorial.

algorithm[algorithm omitted — 773 chars of source]

The procedure EvaluateMoments, given in Algorithm (ref), computes the moments $\mu,\sigma^2$ required in Algorithm (ref) using approximately $2^D |\mathcal{O}_\xi(y)|$ operations. Since this cost scales linearly with the orbit size, evaluating all $Y \in \mathcal{O}_\xi(y)$ may become prohibitive when the orbit is large. In practice, whenever $|\mathcal{O}_\xi(y)|$ exceeds a predefined threshold $L$, we approximate the conditional distribution of $m_\xi(Y)$ by restricting the computation to the truncated set \[ m \in \Bigl[ m_\xi(y) - \bigl( L/2 \wedge m_\xi(y) \bigr), \dots, m_\xi(y) + \bigl(L/2 \wedge M_\xi(y)\bigr) \Bigr]. \] This truncation has negligible numerical effects, because the distribution of $m_\xi(Y)$ is concentrated around $m_\xi(y)$, and extreme values contribute essentially nothing to the expectation or variance.

theoremAlgorithm (ref) runs with $O(\widehat{N}_a)$ operations.
proofFollows from the discussion above and observing that Algorithm (ref) requires exactly one loop over all $\xi \in \Xi_a^\star$.

Evaluating the variance

We now discuss two approaches for evaluating the covariance matrix of the polyads estimator. To shorten notation let $\nabla \ell_\xi\left(Y | X, \widehat{\beta}_\Xi\right)$ be denoted by $\nabla \widehat{\ell}_\xi$ and define \[ \widehat{\Gamma} = \nabla^2 L_\Xi\left(Y | X, \widehat{\beta}_\Xi\right). \] In Theorem (ref), the asymptotic normality is written in terms of $\Sigma^{(k)}$, which can not be directly evaluated from the sample. In practice, (ref) suggest to approximate it by $\widehat{\Sigma}$ given by

equation*[equation* omitted — 316 chars of source]

We also implement and empirically verify the performance of another variance estimator. Equation (ref) suggests approximating the covariance of $\left( \nabla^2 \widehat{Q}^{(k)}\left(\overline{\beta}^{(k)}\right) \right) \left( \widehat{\beta}^{(k)} - \beta_\star\right)$ by the expectation of \[ \left( \nabla \widehat{Q}^{(k)}\left(\beta_\star\right)\right) \left( \nabla \widehat{Q}^{(k)}\left(\beta_\star\right)\right)^\top = \left( \widehat{N}_a^{(k)} \right)^{-2}\sum_{\xi, \xi' \in \Xi^{(k)}} \left( \nabla \ell_\xi\left(Y^{(k)} | X^{(k)}, \beta_\star\right) \right) \left( \nabla \ell_{\xi'}\left(Y^{(k)} | X^{(k)}, \beta_\star\right) \right)^\top. \] Notice that if $\xi$ and $\xi'$ share no edges, the expectation of their corresponding term is zero since it is the product of independent quantities with zero mean. This suggests approximating $\Sigma^{(k)}$ by

equation*[equation* omitted — 266 chars of source]

The difference between $\Omega$ and $\Omega'$ is subtle. The next lemma illuminates this difference and provides computationally tractable expressions for $\Omega$ and $\Omega'$.

lemmaLet \[ \mathcal{I}_a = \{{\bf i} : \exists \xi \in \Xi_a \text{ such that } {\bf i} \in \mathcal{E}(\xi) \} \] be the set of all edges that belong to at least one active polyad. It holds that \begin{align} \widehat{\Omega} &= \sum_{{\bf i}\in{\cal I}_a} \left( \sum_{{\bf i}' : ({\bf i}, {\bf i}') \in \Xi_a} 2^D \nabla \widehat{\ell}_{({\bf i}, {\bf i}')} \right) \left( \sum_{{\bf i}' : ({\bf i}, {\bf i}') \in \Xi_a} 2^D\nabla \widehat{\ell}_{({\bf i}, {\bf i}')} \right)^\top\\ & = \sum_{{\bf i}\in{\cal I}_a} \sum_{{\bf i}' : ({\bf i}, {\bf i}') \in \Xi_a} \sum_{{\bf i}” : ({\bf i}, {\bf i}”) \in \Xi_a} \left( \nabla \widehat{\ell}_{({\bf i}, {\bf i}')}\right) \left( \nabla \widehat{\ell}_{({\bf i}, {\bf i}”)}\right)^\top 2^{2D} \nonumber \end{align} and \begin{equation} \widehat{\Omega}' = \sum_{{\bf i}\in{\cal I}_a} \sum_{{\bf i}' : ({\bf i}, {\bf i}') \in \Xi_a} \sum_{{\bf i}” : ({\bf i}, {\bf i}”) \in \Xi_a} \left( \nabla \widehat{\ell}_{({\bf i}, {\bf i}')}\right) \left( \nabla \widehat{\ell}_{({\bf i}, {\bf i}”)}\right)^\top 2^{D + \sum_{d=1}^D \mathbf{1}\{ i_d' \neq i_d”\}}. \end{equation}
proofTo see (ref) we start noticing that if ${\bf i} \not \in \mathcal{I}_a$, then its corresponding term is zero. Thus, the first sum can be taken only over ${\bf i} \in \mathcal{I}_a$. Now recall from Lemma (ref) that if ${\bf i} \in \mathcal{E}(\xi)$ for some $\xi$, then ${\bf i}$ belongs to all $2^D$ permutations of $\xi$ and there exists exactly one permutation that can be written as $({\bf i}, {\bf i}')$ for some ${\bf i}'$. Since each $\ell_\xi$ is invariant by permutation we have \[ \sum_{\xi:{\bf i}\in{\cal E}(\xi)} \nabla \widehat{\ell}_\xi = \sum_{{\bf i}' : ({\bf i}, {\bf i}') \in \Xi_a} 2^D \nabla \widehat{\ell}_{({\bf i}, {\bf i}')} \] and so (ref) is proved. The proof of (ref) is more intricate. Notice that if $\xi, \xi'$ share one edge we can put this edge “in evidence” to obtain a permutation $({\bf i}, {\bf i}')$ of $\xi$ and a permutation $({\bf i}, {\bf i}')$ of $\xi'$. If $\xi, \xi'$ share exactly one edge this representation is unique and a total of $2^D \times 2^D$ permutations ($2^D$ for each polyad in the pair) will be represented by the pair $({\bf i}, {\bf i}'), ({\bf i},{\bf i}'')$. Now assume that the polyads share exactly $m$ edges, in this case the representation is not unique anymore, as there are $m$ possible choices of ${\bf i}$. To avoid double counting we split the $2^D \times 2^D$ permutations evenly between all possible choices of ${\bf i}$, making each one account for $\frac{2^D \times 2^D}{m}$ permutations. It remains to understand, for a given pair $({\bf i}, {\bf i}'), ({\bf i}, {\bf i}'')$, how many edges they share. Notice that if $i_d' = i_d''$, then we can simultaneously flip the $d$-th coordinate of ${\bf i}$ with the $d$-th coordinate of ${\bf i}'$ and ${\bf i}''$ to obtain a new shared edge. It can not be done if $i_d' \neq i_d''$. Thus, the number of shared edges is $m = 2^{\sum_{d=1}^D \mathbf{1}\{ i_d' = i_d''\}}$, which yields \[ \frac{2^D \times 2^D}{ 2^{\sum_{d=1}^D \mathbf{1}\{ i_d' = i_d''\}} } = 2^{D + \sum_{d=1}^D \mathbf{1}\{ i_d' \neq i_d''\}} \] permutations counted for each pair $({\bf i}, {\bf i}'), ({\bf i}, {\bf i}'')$.

Indeed, (ref) and (ref) make explicit the two main differences between $\widehat{\Omega}$ and $\widehat{\Omega}'$. First, $\widehat{\Omega}$ contains duplicates of certain pairs of polyads. A closer inspection of the proof reveals that these duplicates arise precisely on pairs that share strictly more than one edge. This clarifies the role of Assumption (ref) in Theorem (ref): for the projection strategy to be valid, the covariance of the Hájek projection of $\nabla \widehat{Q}^{(k)}(\beta_\star)$ --- which is approximately $\widehat{N}_a^{-2}\mathbb{E}\,\widehat{\Omega}$ --- and the true covariance --- approximately $\widehat{N}_a^{-2}\mathbb{E}\,\widehat{\Omega}'$ --- must converge to each other. Second, computing $\widehat{\Omega}$ is less costly than computing $\widehat{\Omega}'$. For each $\mathbf{i} \in \mathcal{I}_a$, evaluating $\widehat{\Omega}$ requires only a single pass over each $\mathbf{i}'$ such that $(\mathbf{i}, \mathbf{i}') \in\Xi_a$. In contrast, computing $\widehat{\Omega}'$ requires an extra loop over ${\bf i}''$ such that $(\mathbf{i}, \mathbf{i}'') \in\Xi_a$.

We now use equations (ref) and (ref) to obtain an algorithm for computing $\widehat{\Sigma}$ and $\widehat{\Sigma}'$. We need to be able to loop over all ${\bf i} \in \mathcal{I}_a$ and, given ${\bf i}$, to efficiently loop over all ${\bf i}'$ such that $({\bf i}, {\bf i}') \in \Xi_a$. Recall that $\Xi_a^\star$ contains exactly one permutation of each active polyad $\xi \in \Xi_a$, in fact, $\widehat{N}_a = |\Xi_a| = 2^D|\Xi_a^\star|$. One can loop over each $\xi \in \Xi_a^\star$ and compute all permutations of $\xi$. By updating a dictionary containing for each key ${\bf i}$ the corresponding set of ${\bf i}'$s we can easily construct the data structure needed to evaluate the covariances. In practice we also store a pointer to the original $\xi$ so that we can profit from the already evaluated $\widehat{X}_\xi$ and $\{Y_{\bf i} : {\bf i} \in \mathcal{E}(\xi)\}$. The following result gives the computational complexity of computing each variance alternative.

theoremComputing $\widehat{\Sigma}$ requires $O(\widehat{N}_a)$ operations and computing $\widehat{\Sigma}'$ requires $O(\widehat{N}_a m_a )$ operations, where $m_a$ is the maximum over all edges ${\bf i}$ of the number of active polyads $\xi$ such that ${\bf i} \in \mathcal{E}(\xi)$.
proofSince the cost of updating and consulting a dictionary is constant one can construct the dictionary that maps ${\bf i}$ to ${\bf i}'$ in $O(\widehat{N}_a)$. For evaluating $\widehat{\Omega}$ one goes through each key ${\bf i}$ and each set of ${\bf i}'$s once, thus yielding $O(\widehat{N}_a)$. Evaluating $\widehat{\Omega}$ requires a double loop over the ${\bf i}'$s, thus $O(\widehat{N}_a m_a)$. The corresponding $\widehat{\Gamma}$ is just the Hessian of the loss, which is evaluated in $O(\widehat{N}_a)$.

Final cost analysis and practical considerations

As a consequence of this section's discussion, we can provide a complete computational cost analysis of our method:

theoremAssume that $m_a$ is bounded. When $D$ is odd, assume also that there exists $c \geq 1$ that $|E_{i_1}| < c\frac{|E|}{n_1}$ for all $i_d \in [n_d]$. Thus, evaluating $\widehat{\beta}_\Xi$ and estimating its variance requires $O(T|E|^2)$ operations, where $T$ is the number of iterations of Newton's method.
proofFirst notice that constructing $\Xi_a^\star$ has cost $O(|E|^2)$, thus the size of $\Xi_a$, i.e. $\widehat N_a$, must be of order at most $|E|^2$. Each Newton's method update has cost $O(\widehat{N}_a)$ and, under the assumption that $m_a$ is bounded, both variance estimates have cost $O(\widehat{N}_a)$. Thus, the total cost is driven by the number of iterations of Newton's method times $|E|^2$.

In practice, Newton's method converges in less than $10$ iterations, yielding $O(|E|^2)$ operations. This quantity is to be compared with the fast implementation of PPML from correia2019ppmlhdfe, which is $O(n)$. Our analysis suggests that our method is faster when $|E| \ll \sqrt{n}$ and competitive when $|E|$ is of order $\sqrt{n}$. Although being the standard practice when reporting the complexity of algorithms, the big-O notation hides a constant that matters to practitioners. In the next section, we empirically verify that, as predicted by our cost analysis, our method outperforms PPML in terms of computational time when $|E|$ is smaller than $\sqrt{n}$ and remains competitive as $|E|$ grows. Indeed, in our computational setup (see Section (ref)) the running time of our method is shorter than that of PPML as long as $|E| \leq 15 \sqrt{n}$. This is, for example, the case of a bipartite network with $n_1=n_2$ such that the average degree of a node is at most $15$.

remarkWe also note that the improvements developed in this section, and particularly the construction of $\Xi_a^\star$, may be useful for other polyads-based methods. For example, jochmans2017two proposes a generalized method of moments for two-way models with $n_1 = n_2 = \sqrt{n}$. His tetrad-based estimator leverages matrix-multiplication tricks and achieves a computational complexity of order $O(n^{1.1877})$ when using the best available theoretical matrix-multiplication algorithm. In contrast, our $O(|E|^2)$ complexity can yield substantial gains in sparse regimes.

Experiments

We provide experiments comparing our method with PPML and with the analytical debias proposed by zylkin2024bootstrap. We consider artificial and real data. With artificial data we investigate the impact of the incidental parameter problem while knowing the correct value of $\beta_\star$. With real data we display evidence of the incidental parameter bias and show how it may lead towards wrong conclusions in inference. We also discuss the computational time and the effect of sparsity for all methods.

Our polyads estimator is implemented as described in Section (ref), and confidence intervals use the covariance approximation $\widehat{\Sigma}'$ from Section (ref). All reported running times correspond to executions of $25$ seeds in parallel on a server equipped with a 32vCPU Intel Xeon Gold 6444Y and 3 TB of RAM.

Artificial data

We conduct computational experiments using a three-way data-generating process inspired by weidner2021bias; see Example (ref). The dimensions are $n_1 = n_2$ (varied) and $n_3 = 5$ (fixed). The fixed effects $u_{ij}, w_{it}, v_{jt}$ are i.i.d.\ $\mathcal{N}(0,1/16)$, and $\beta_\star = 1$. The covariates $X_{ijt}$ are correlated with the fixed effects and, along the third axis, with their own past values: \[ X_{ijt} =

cases\tfrac{1}{2} X_{ij(t-1)} + w_{it} + v_{jt} + \tfrac{1}{4}\mathcal{N}(0,1), & t>1,\\[3pt] w_{it} + v_{jt} + \tfrac{1}{4}\mathcal{N}(0,1), & t=1.

\] The mean of the ${\bf i}=(i,j,t)$ edge's weight satisfies \[ \mathbb{E}(Y_{ijt}) = \exp\!\left(c + \beta_\star X_{ijt} + u_{ij} + w_{it} + v_{jt}\right), \] where the constant $c$ can be selected to control the density $|E|/n$ of the graph. We generate both Poisson data (satisfying Assumption (ref) with intensity $\lambda_{\bf i} = \lambda_{ijt}$) and non-Poisson data. To obtain non-Poisson outcomes, we generate $Y_{{\bf i}}$ as Negative Binomial via a Gamma--Poisson mixture: the rate of the Gamma controls the variance, while its shape is scaled to match the desired mean $\lambda_{{\bf i}}$. Setting the rate to $\infty$ recovers the Poisson model; for experiments with overdispersion, we take the rate equal to $0.1$. We executed $600$ replications of each configuration.

We compare three estimators: PPML, PPML debiased, and our polyads estimator. For PPML we use the fast implementation of correia2019ppmlhdfe; for PPML debiased we use the analytical correction of weidner2021bias via their Stata package. Our first experiment varies the graph density with \[ |E| \in \{0.02n,\ 0.03n,\ 0.04n,\ 0.05n,\ 0.1n\}, \quad\text{and}\quad n_1 = n_2 \in \{50,100\}. \] Each configuration is replicated $600$ times. Figure (ref) summarizes the results. The first row shows the distributions of the normalized errors $\sqrt{n}\,(\widehat{\beta}_\Xi - \beta_\star)$ at densities 2%, 5%, and 10%. The second row reports, from left to right, the empirical coverage of the 95% confidence interval (which should be close to 95%), the convergence rate, and the running time. We observe:

itemize• Incidental parameter bias. PPML exhibits a clear incidental parameter problem (blue curves shifted to the right), especially at low densities. Our polyads estimator eliminate this bias and the debiased PPML, at larger sample sizes $n$, do not display bias. • Instability at low density. For small $n$ and sparse graphs, PPML and PPML debiased often fail to converge and produce poor coverage. • Coverage issues. Even at larger samples, PPML intervals remain unreliable (best coverage $\approx 90\%$ for $n_1=n_2=100$ and $|E|=0.1n$). The debiased method improves coverage but still undercovers at low densities and $n$, and overcovers at high densities. • Computational efficiency. Our estimator is substantially faster at low densities and remains competitive as density increases.
figure[figure omitted — 270 chars of source]
figure[figure omitted — 289 chars of source]

We next consider a second experiment aimed at evaluating performance in the sparse regime. Here we let $n$ vary from $2\times 10^{5}$ to $32\times 10^{5}$ and choose the constant $c$ so that \[ |E| \approx 4\sqrt{n}, \] thus forcing the density $\frac{|E|}{n}$ to shrink towards zero as the sample size grows. This setting is substantially sparser than the fixed-density designs considered above. Figure (ref) reports the results for the Poisson case and Figure (ref) reports the results for the Negative Binomial case. The qualitative conclusions of both cases are similar and consistent with those found in the low-density examples from Figure (ref), but become even more pronounced under sparsity:

itemize• Estimation error. The first panel presents boxplots of the non-scaled estimation error $\widehat{\beta}_\Xi-\beta_\star$ (conditional on convergence). PPML again displays a pronounced incidental parameter bias, with its median shifted upward across all sample sizes. The debiased PPML estimator reduces, but does not eliminate—this distortion. In contrast, the polyads estimator remains concentrated around $\beta_\star$ for every $n$. More interestingly, as $n$ increases the variance of the PPML debiased approach shrinks, but the incidental parameter bias remains relevant in a way that its coverage starts to decay. • Coverage of the 95% confidence interval. The second panel highlights the severity of the incidental parameter problem in sparse settings. PPML confidence intervals exhibit near-zero coverage throughout: the bias places the estimator far outside the nominal interval in almost all replications. The debiased PPML method improves coverage but still undercovers for smaller $n$, where sparsity is most acute. The polyads estimator, by contrast, maintains coverage close to the nominal 95% level uniformly across all sample sizes. • Running time. The third panel reports computation times. Both PPML and debiased PPML become increasingly expensive as $n$ grows, despite the sparsity of the graph. The polyads estimator directly leverages sparsity and is substantially faster, especially for moderate and large $n$.

Across bias, coverage, computation time, and convergence, the results in this sparse regime reinforce the findings from the low-density experiment: PPML exhibits severe incidental parameter bias and essentially zero inferential validity; the debiased PPML estimator improves upon PPML but continues to suffer from miscoverage under sparsity; the polyads estimator remains accurate, fast, and statistically reliable, even when sparsity increases with sample size.

figure[figure omitted — 290 chars of source]

Comparing Figures (ref) and (ref) we notice that the Negative Binomial case closely match the Poisson one: PPML remains biased, while both the debiased PPML and our polyads estimator remove the bias and achieve better coverage. Thus, although our theory assumes a Poisson model, these results indicate that the polyads estimator is empirically robust to overdispersion and model misspecification.

Real data

We study health insurance claims data that cover the universe of physician consultations in metropolitan France over the years 2016 to 2018.\footnote{More details on data are provided in Appendix (ref).} In May 2017, the fees charged by general practitioners (GPs) belonging to the regulated sector\footnote{The majority of French GPs are subject to fee regulation. In January 2016, only 8.6% of GPs can charge fees in excess of the regulated level.} have increased by 8.7%. Using a difference-in-differences approach at the doctor level, we find that the stronger financial incentives have caused physician activity (as measured by number of visits) to rise by approximately 10% (see Appendix (ref) for details about the considered control groups and specifications, as well as Table (ref)).

Given the strong policy concern about care accessibility, it is important to understand the impact of the reform on potential patients. Have certain categories of patients (patients with chronic diseases, women, elderly patients, patients living in medically undeserved areas, low-income patients, etc.) been disproportionately affected by the reform? Has the reform caused patients to seek care further away from their home? To answer these questions, we need to perform the analysis at the doctor-patient level rather than at the doctor level. Using a three-way model and controlling for dyads fixed effects allows to estimate how the reform has changed the doctor-patient connections. To illustrate the method, we highlight in this subsection the role of two variables: gender and geographic distance (spatial accessibility).

The outcome $Y_{ijt}$ is the number of visits by patient $i$ to doctor $j$ on month $t$. We aggregate data at the city-sex level. The high number of potential patients in each city-sex group makes to the Poisson assumption plausible.\footnote{The aggregate number of consultations in each group is the sum of (possibly heterogeneous) Bernoulli distributions that represent the occurrence of a consultation for all potential patients in the group. This sum follows approximately a Poisson distribution under the conditions exposed in le1960approximation. The assumption that the individual occurrences of a consultation are independent across potential patients is relaxed in galambos1973general and serfling1978some. } The index $i$ (resp. $j$) stands thus for the set of patients (resp. doctors) in a given municipality with given gender. The treatment $T_{j}$ is a binary variable equal to 1 for sector 1 GPs, and to 0 for direct access specialists. The reform has been implemented from May 2017 onward, hence the definition of $\text{Post}_t$, a dummy variable equal to 1 after that date. The model includes three features that account for any possible policy-induced change in homophily preferences as regards gender and spatial dimensions: the interactions between $\text{Post}_t \times T_j$ (as in any difference-in-differences approach) and (i) a dummy variable equals to 1 when patients and doctors have the same sex, (ii) a dummy variable that is equal to 1 when the medical office chosen is located in the same municipality as the patient's home, and (iii) travel time (measured in minutes between the centroids of municipalities).

We thus consider the three-way Poisson model $Y_{ijt}\sim\mathcal{P}(\lambda_{ijt})$ with intensity given by \[\ln\lambda_{ijt}=\big(\beta_\texttt{d} d_{ij}+\beta_\texttt{sc} \mathds{1}\{\text{city}_i=\text{city}_j\}+\beta_\texttt{ss} \mathds{1}\{\text{sex}_i=\text{sex}_j\}\big) \times \text{Post}_t \times T_j +u_{ij}+v_{jt}+w_{it}.\]

We compare three estimators: the standard PPML estimator, the analytical correction of weidner2021bias, and our polyads estimator. Because the full dataset contains $n_1 = 69{,}265$ patient groups, $n_2 = 16{,}941$ doctors, and $n_3 = 34$ months --- corresponding to roughly $n \approx 40$ billion edges and $|E| = 56{,}034{,}015$ --- direct estimation on the full graph is computationally infeasible for all methods. We therefore adopt a subsampling strategy combined with meta-analysis.

For each configuration, we sample a proportion $s \in \{2\%, 3\%, 4\%\}$ of patient groups and the same proportion of doctors, while always retaining all $34$ months\footnote{Notice that we sample the groups of patients, the doctors and then get all edges through the sampled nodes, keeping all times. This sampling strategy is consistent with Assumption (ref) and also with the sampling assumptions in fernandez2016individual, weidner2021bias, graham2017econometric, jochmans2018semiparametric. One could also propose sampling directly the edges and not the nodes, but there is evidence that this procedure can lead to bias, see subsamplegraphs for details.}. Each subsample consists of independent random draws of patient and doctor groups; doctors in the treatment and control groups are sampled independently to ensure comparability. We repeat the procedure independently to obtain $100$ subsamples. The final estimates are obtained via a random-effects meta-analysis using the default implementation in the statsmodels Python library, which is based on the iterated method of paule1982consensus, a refinement of the classical method of dersimonian1986meta.

table[table omitted — 1,622 chars of source]
table[table omitted — 735 chars of source]
figure[figure omitted — 264 chars of source]

The results are the following. First, Table (ref) and Figure (ref) show that debiased PPML has many outliers, mostly on `small' samples (admittedly, not that small since $|E|\approx 20,000$ with the 2% sampling), which results in (very) large confidence intervals and point estimates. As expected, the variance decreases with the subsample size, regardless of the estimation method used.

Second, as reported on Table (ref), the running time of PPML and debiased PPML scales with $n$ while the polyads time scales with $|E|^2$.

Third, a closer look at the 4% subsample coefficients suggests that: (i) PPML estimates of $\beta_d$ yields confidence intervals that do not contain the point estimate of debiased PPML and of Polyads, which may be a sign of incidental parameter bias. This is to be compared with the artificial experiments where PPML yields small variance around a biased point, missing the point estimate. All methods point towards a positive coefficient, implying that after the reform some patients travel further away from home to get healthcare; (ii) as regards the preference coefficient for being treated in the same municipality $\beta_{sc}$, the Polyads estimator yields similar results to the PPML, while debiased PPML gives a different point estimate, although the latter confidence intervals are overall consistent with PPML and Polyads. All methods point towards a negative coefficient, meaning that after the reform patients are more likely to visit a doctor outside their own city ; (iii) as regards gender homophily, we see some difference up to the level of signs. The PPML would lead to estimate a negative impact on the reform on that homophily (this diagnosis holding true regardless of the subsample size considered here), while Polyads and debiased PPML both contain $0$ in their confidence intervals in the $4\%$ sample. Note also that, PPML yields a confidence interval that is close to the lower half of Polyads confidence interval, this may be a sign of the incidental parameter problem as PPML may be giving a narrow variance around a biased point estimate and leading to possibly wrong conclusions (in this case, that gender homophily between patients and doctors decreased).

Code resources

A fast python-based implementation of the polyads method here presented is available at the Github repository lucasresenderc/polyads, available at \href{https://github.com/lucasresenderc/polyads}{https://github.com/lucasresenderc/polyads}.

Acknowledgments

We are grateful to Cl\'ement de Chaisemartin, Laurent Davezies, Xavier D'Haultf\oe uille, Francis Kramarz, and Yannick Guyonvarch for helpful comments. We thank the Agence Nationale de la Recherche for financial support (ANR-23-CE36-0014).

\numberwithin{equation}{section}

\setcounter{equation}{0} \setcounter{table}{0} \setcounter{figure}{0}

\numberwithin{figure}{section} \numberwithin{lemma}{section} \numberwithin{table}{section}