Lagrange multipliers in Maximum likelihood estimations and Least squares problems with Constraints

Lagrange multipliers in Maximum likelihood estimations and Least squares problems with Constraints

Introduction

Maximum Likelihood Estimation (MLE) and Least Squares (LS) problems with constraints arise in a wide range of fields. Examples include constrained regressions (linear/logistic regressions) in machine learning and statistics (e.g., geyer1991constrained,na2025statistical,shapiro2000asymptotics), inverse problems in natural science (PDE-constrained optimization; e.g., de2015numerical), and structural estimations of economic models in economics (e.g., rust1987optimal,su2012constrained). In recent years, deep learning techniques have also been applied to these problems (e.g., Physics-Informed Neural Networks (PINNs); e.g., raissi2019physics,lu2021physics). Motivated by these developments, recent studies have explored increasingly sophisticated numerical algorithms for solving constrained optimization problems.

In constrained optimization, the values of the Lagrange multipliers play a crucial role in the performance of numerical algorithms. The current study demonstrates that we can utilize the property that the values of Lagrange multipliers are close to 0 in large samples of the dataset, provided the distribution is correctly specified (for MLE) or the residuals follow a normal distribution (for LS). The property can be easily derived by relying on the large sample theory in statistics. Note that, even in deep learning applications, where the number of parameters can be larger than the sample size, we can obtain analogous results provided that the generalization performance is good.

The property has several practical implications. First, many constrained optimization algorithms, including the Augmented Lagrangian Method (ALM), Sequential Quadratic Programming (SQP), and Interior Point (IP) method, and recently proposed methods for solving stochastic optimization problem (e.g., na2023inequality,na2025statistical) require specifying initial candidate values for the Lagrange multipliers, and choosing 0 as the initial values is reasonable. The current study presents numerical results supporting the discussions. Second, penalty-based approaches, which convert constrained problems into unconstrained ones by adding penalty terms (e.g., PINNs with soft constraints for inverse problems), are widely used in applications. Our results show that when the true Lagrange multipliers are small, the solution to the penalized problem can be close to the exact constrained solution. This helps explain why penalty-based methods often perform well in practice (cf. toscano2025pinns for the survey of the recent studies on PINNs.).

Related literature

This study is closely connected to recent work on numerical algorithms for constrained stochastic optimization (e.g., berahas2021sequential,berahas2024stochastic,fang2024fully,na2025statistical). Unlike these studies, which consider general constrained stochastic optimization, we focus specifically on MLE and LS problems. This narrower focus allows us to exploit the statistical property that Lagrange multipliers tend to be small under the conditions previously mentioned. The insights obtained here may contribute to the development of more efficient numerical algorithms tailored to these widely used problem classes.

The asymptotic behavior of Lagrange multipliers---specifically, their convergence to zero as the sample size increases---was originally established by aitchison1958maximum in the MLE setting. However, subsequent research primarily examined this property in the context of statistical hypothesis testing, and no previous studies explicitly discussed its implications from the viewpoint of the numerical optimization, to my knowledge. Moreover, prior work typically assumed a classical statistical setting in which the number of observations far exceeds the number of parameters. In contrast, the present study discusses the implications of this property even in high-dimensional settings where the number of parameters may exceed the sample size, a situation common in modern deep learning applications.

Although the present study focuses on how the statistical properties of the Lagrange multipliers inform widely used existing methods, this property is also useful for developing new algorithms for constrained optimization. Building on earlier work in economics, fukasawa2026sequential proposes the Sequential Linearly Constrained (SLC) algorithm, which exploits this property, and showed that the SLC algorithm can be several times faster than alternative approaches in several structural estimation applications in economics.\ifx\@typeset@protect \expandafter\SF@@footnote \else \expandafter\SF@gobble@opt \fi {The key idea is to solve a linearly constrained optimization problem at each iteration. The advantages of the SLC algorithm are twofold. First, it achieves quadratic local convergence in large samples. Second, it can be implemented without explicitly computing the Jacobian of the constraint function---which may be high-dimensional---by relying instead on numerical directional derivatives and Krylov subspace methods. Note that a related algorithm was mentioned in murtagh1982projected, although its convergence properties were not established.} The study also shows that previously proposed methods in economics, including NPL (aguirregabiria2002swapping) and EPL (dearing2025efficient), implicitly utilize the property.

The rest of this paper is organized as follows. Section (ref) derives the statistical properties of the Lagrange multipliers, and Section (ref) discusses their implications for numerical optimization. Section (ref) presents the results of numerical experiments, and Section (ref) concludes. Appendix (ref) provides additional material, including further details and supplementary results for the numerical experiments.

Constrained optimization problem and Lagrange multipliers

We consider the following constrained optimization problem:

Let $\widehat{\theta}$ denote the solution, and let $g\left(\theta\right)\equiv\left(

\right)$. Here, we let $g\left(\theta\right)\equiv\left(g_{i}\left(\theta\right)\right)_{i=1,\cdots,n_{E}+n_{I}}$ where $n_{E}$ and $n_{I}$ denote the number of parameters, and let $\mathcal{A}\left(\widehat{\theta}\right)$ denote the set of active constraints. Let $\widehat{\lambda}$ denote the Lagrange multiplier of the problem, and let $\mathcal{L}\left(\theta,\lambda\right)\equiv Q\left(\theta\right)+\lambda^{T}g\left(\theta\right)$ denote the Lagrangian function. Note that the stochastic optimization problem $\min_{\theta}Q\left(\theta\right)=E_{w}\left[q\left(w;\theta\right)\right]\ s.t.\ g_{E}\left(\theta\right)=0,g_{I}\left(\theta\right)\leq0$ can be interpreted as a case where $N$ is infinite.

Here, we assume twice continuous differentiability of $Q$ and $g$, and we impose the following two standard assumptions concerning the constrained optimization problem:

The KKT conditions for the problem are:

Let $\theta\equiv\left(\theta_{1},\theta_{2}\right)$, where $\theta_{2}\in\mathbb{R}^{n_{A}}$. Note that the divisions of $\theta$ into $\theta_{1}$ and $\theta_{2}$ are arbitrary. $n_{A}$ denotes the number of active constraints. Then, by ((ref)),

holds. Under Assumption (ref), $\left\{ \nabla_{\theta_{2}}g_{i}\left(\widehat{\theta}\right)\right\} _{i\in\mathcal{A}(\widehat{\theta})}$ are linearly independent. Then, $\nabla_{\theta_{2}}g_{\mathcal{A}}\left(\widehat{\theta}\right)$ is nonsingular, and ((ref)) implies:

In the rest of the current paper, we focus on two specific problems:

Values of Lagrange multipliers

We can show that the Lagrange multipliers take values close to 0 under moderate conditions in both the MLE and LS settings. This section clarifies this point. In both cases, define $\theta^{*}\equiv\arg\max_{\theta}E_{w}\left[q\left(w_{i};\theta\right)\right]\ s.t.\ g_{E}\left(\theta\right)=0,g_{I}\left(\theta\right)\leq0$. $\theta^{*}$ can be interpreted as a solution of the constrained optimization problem when an infinite number of observations are available. Here, we assume that $\widehat{\theta}-\theta^{*}=O_{p}\left(N^{-\frac{1}{2}}\right)$ holds (cf. shapiro2014lectures for the $\sqrt{N}$-consistency of the constrained M-estimator.).

Note that, in general, $\nabla_{\theta}Q\left(\widehat{\theta}\right)=0$ must hold for an objective function $Q$ satisfying $Q\left(\theta\right)\geq0$ and $Q\left(\widehat{\theta}\right)=0$, because $\widehat{\theta}$ can be regarded as a local minimizer of $Q$ as long as $g_{E}\left(\widehat{\theta}\right)=0,g_{I}\left(\widehat{\theta}\right)\leq0$. This implies $\widehat{\lambda}=0$ by ((ref)). The discussion below suggests that, under suitable conditions and in large samples, the original objective functions of the MLE and LS can essentially be reformulated in this manner.

Maximum Likelihood Estimation (MLE)

Let $f^{*}\left(w\right)$ be the true distribution of the observation $w$. When the model is correctly specified, $\theta^{*}$ satisfies $f\left(\cdot,\theta^{*}\right)=f^{*}\left(\cdot\right)$. Note that analogous discussions hold, even when $w_{i}$ can be divided into $x_{i}$ and $y_{i}$ and we alternatively consider the conditional likelihood $f\left(y_{i};\theta,x_{i}\right)$, (i.e., $\min_{\theta}-\frac{1}{N}\sum_{i=1}^{N}\ln f\left(y_{i};\theta,x_{i}\right)\ s.t.\ g_{E}\left(\theta\right)=0,g_{I}\left(\theta\right)\leq0$) which usually appear in applications.

Intuition

We can intuitively understand why the the Lagrange multiplier is close to 0 in large samples for the MLE. Here, we consider a setting where the number of samples is infinite. In the setting, as is well known, the MLE is equivalent to finding parameters minimizing the Kullback Leibler Information Criterion (KLIC). Hence, we let $Q\left(\theta\right)=KLIC\left(f^{*};f\left(\cdot;\theta\right)\right)\equiv\int\left[\ln\frac{f^{*}\left(w\right)}{f\left(w|\theta\right)}f^{*}\left(w\right)\right]dw$, and consider the constrained optimization problem $\min_{\theta}Q\left(\theta\right)=KLIC\left(f^{*};f\left(\cdot;\theta\right)\right)\ s.t.\ g_{E}\left(\theta\right)=0,g_{I}\left(\theta\right)\leq0$. By the property of KLIC, $Q\left(\theta\right)=KLIC\left(f^{*};f\left(\cdot;\theta\right)\right)\geq0$ holds for all $\theta$. If the distribution $f\left(w;\theta\right)$ is correctly specified, $Q\left(\widehat{\theta}\right)=KLIC\left(f^{*};f\left(\cdot;\widehat{\theta}\right)\right)=0$ additionally holds. Then, the discussions at the beginning of the current subsection suggests $\nabla_{\theta}Q\left(\widehat{\theta}\right)=0$, and consequently $\widehat{\lambda}=0$. Therefore, the Lagrange multipliers should be equal to zero in the infinite-sample limit under the correct specification.

It is worth noting that, although $KLIC\left(f^{*};f\left(\cdot;\widehat{\theta}\right)\right)=0$ can hold, it does not necessarily imply the redundancy of the constraints. If the constraints are not imposed, $\theta$ satisfying $KLIC\left(f^{*};f\left(\cdot;\theta\right)\right)=0$ may not be unique.

Formal discussion

The following formal discussions largely simplify the discussions in aitchison1958maximum, while considering the possibility of misspecified distribution of $f$. Because the primary focus of the current study is to clarify $\widehat{\lambda}=O_{p}\left(N^{-\frac{1}{2}}\right)$ and the conditions required to derive the result, in contrast to aitchison1958maximum deriving detailed asymptotic distributions of $\left(\widehat{\theta},\widehat{\lambda}\right)$, we take a different approach. Here, we assume finite $Var\left(\nabla_{\theta}\ln f\left(w;\theta^{*}\right)\right)$ and the boundedness of $\nabla_{\theta\theta^{\prime}}\ln f\left(w;\theta\right)$.

First, by differentiating the both sides of $1=\int f\left(w;\theta\right)dw\ \forall\theta$ with regard to $\theta$ at $\theta=\theta^{*}$, we have \[ 0=\int\left(\nabla_{\theta}f\left(w;\theta^{*}\right)\right)dw=\int\left(\nabla_{\theta}\ln f\left(w;\theta^{*}\right)\right)f\left(w;\theta^{*}\right)dw. \] . Then, $\nabla_{\theta}Q\left(\widehat{\theta}\right)=-\frac{1}{N}\sum_{i=1}^{N}\nabla_{\theta}\ln f\left(w_{i};\widehat{\theta}\right)$ can be decomposed into three terms:

Concerning the first term, $\nabla_{\theta}\ln f\left(w_{i};\widehat{\theta}\right)-\nabla_{\theta}\ln f\left(w_{i};\theta^{*}\right)=\left(\nabla_{\theta\theta^{\prime}}\ln f\left(w_{i};\exists\overline{\theta}\right)\right)\left(\widehat{\theta}-\theta^{*}\right)$ holds by the mean value theorem. Then, by $\widehat{\theta}-\theta^{*}=O_{p}\left(N^{-\frac{1}{2}}\right)$ and the boundedness of $\nabla_{\theta\theta^{\prime}}\ln f\left(w;\theta\right)$, the first term is $O_{p}\left(N^{-\frac{1}{2}}\right)$. The second term is also $O_{p}\left(N^{-\frac{1}{2}}\right)$ by the central limit theorem and the assumption of finite $Var\left(\nabla_{\theta}\ln f\left(w;\theta^{*}\right)\right)$. The third term is 0 if $f\left(w;\theta^{*}\right)=f^{*}\left(w\right)$, which is satisfied when the distribution $f$ is correctly specified. Consequently, $\nabla_{\theta}Q\left(\widehat{\theta}\right)=O_{p}\left(N^{-\frac{1}{2}}\right)$ holds if the distribution $f$ is correctly specified.

Note that aitchison1958maximum originally considered a setting where the objective is $\sum_{i=1}^{N}q_{i}\left(\theta\right)$ (i.e., not normalized by $N$), and showed that the Lagrange multiplier $\widetilde{\lambda}$ satisfies $\widetilde{\lambda}=O_{p}\left(N^{\frac{1}{2}}\right)$. In contrast, we consider the objective $\frac{1}{N}\sum_{i=1}^{N}q_{i}\left(\theta\right)$, and $\widehat{\lambda}=O_{p}\left(N^{-\frac{1}{2}}\right)$ holds.

Least Squares (LS)

Intuition

Concerning the LS problem, it is well known that the problem is essentially equivalent to the MLE if the residuals in the LS follow the normal distribution. Consequently, the values of the Lagrange multipliers should be equal to 0 in the infinite-sample limit when the residuals actually follow a normal distribution.

Formal discussion\ifx\@typeset@protect \expandafter\SF@@footnote \else \expandafter\SF@gobble@opt \fi {When the parameters are identified without introducing the constraint (i.e., $\theta^{*}=\min_{\theta}Q(\theta)$ when $N$ is infinite), $\widehat{\lambda}=O_{p}\left(N^{-\frac{1}{2}}\right)$ holds, even when the residuals do not follow the normal distribution. Previous studies utilized the property for statistical hypothetical testings (cf. gourieroux1995statistics, for instance). }

Here, let $f^{*}\left(r\right)$ be the true distribution of the residuals $\left\{ r\left(w,\theta^{*}\right)\right\} _{w\sim\mathcal{P}}$, where $w\sim\mathcal{P}$ implies that the observation $w$ is drawn from $\mathcal{P}$. Let $\sigma$ be the standard deviation of $\left\{ r\left(w,\theta^{*}\right)\right\} _{w\sim\mathcal{P}}$. We impose the assumption of finite $Var\left(\nabla_{\theta}\left(\ln\phi_{\sigma}\left(r\left(w,\theta^{*}\right)\right)\right)\right)$ and the boundedness of $\nabla_{\theta\theta^{\prime}}\ln\phi_{\sigma}\left(r\left(w,\theta\right)\right)$. First, $Q\left(\theta\right)$ can be reformulated as:

Let $\phi_{\sigma}\left(r\right)\equiv\frac{1}{\sqrt{2\pi}\sigma}\exp\left(-\frac{r^{2}}{2\sigma^{2}}\right)$, which corresponds to the density function of the normal distribution with standard deviation $\sigma$. We also define $h_{\sigma}\left(w;\theta\right)\equiv\phi_{\sigma}\left(r\left(w;\theta\right)\right)$. By $\int_{-\infty}^{\infty}\frac{1}{\sqrt{2\pi}\sigma}\exp\left(-\frac{r^{2}}{2\sigma^{2}}\right)dr=1$, \[ 1=\int_{-\infty}^{\infty}\frac{1}{\sqrt{2\pi}\sigma}\exp\left(-\frac{\left(y-\psi\left(x,\theta\right)\right)^{2}}{2\sigma^{2}}\right)dy=\int h_{\sigma}\left(w\equiv\left(x,y\right);\theta\right)dy \] holds for all $x$ and $\theta$. Then, by differentiating the both sides with respect to $\theta$ at $\theta=\theta^{*}$, \[ \int\left(\nabla_{\theta}\ln h_{\sigma}\left(w;\theta^{*}\right)\right)h_{\sigma}\left(w;\theta^{*}\right)dy=0 \] holds for all $x$, which implies \[ \int\left(\nabla_{\theta}\ln h_{\sigma}\left(w;\theta^{*}\right)\right)h_{\sigma}\left(w;\theta^{*}\right)dw=\int\left[\int\left(\nabla_{\theta}\ln h_{\sigma}\left(w;\theta^{*}\right)\right)h_{\sigma}\left(w;\theta^{*}\right)dy\right]dx=0. \]

Then, $\nabla_{\theta}Q\left(\widehat{\theta}\right)=\nabla_{\theta}\left[-\sigma^{2}\frac{1}{N}\sum_{i=1}^{N}\ln h_{\sigma}\left(w_{i};\widehat{\theta}\right)\right.$$\left.+\sigma^{2}\ln\left(\frac{1}{\sqrt{2\pi}\sigma}\right)\right]$ can be decomposed into three terms:

Concerning the first term, $\nabla_{\theta}\ln\left(h_{\sigma}\left(w_{i};\widehat{\theta}\right)\right)-\nabla_{\theta}\ln\left(h_{\sigma}\left(w_{i};\theta^{*}\right)\right)=\left(\nabla_{\theta\theta^{\prime}}\ln\left(h_{\sigma}\left(w_{i};\exists\overline{\theta}\right)\right)\right)\left(\widehat{\theta}-\theta^{*}\right)$ holds by the mean value theorem. Then, by $\widehat{\theta}-\theta^{*}=O_{p}\left(N^{-\frac{1}{2}}\right)$ and the boundedness of $\nabla_{\theta\theta^{\prime}}\ln f\left(w_{i};\exists\overline{\theta}\right)$, the first term is $O_{p}\left(N^{-\frac{1}{2}}\right)$. The second term is also $O_{p}\left(N^{-\frac{1}{2}}\right)$ by the central limit theorem. The third term is 0 if the residual actually follows the normal distribution, i.e., $f^{*}\left(w\right)=h_{\sigma}\left(w;\theta^{*}\right)$. Consequently, $\nabla_{\theta}Q\left(\widehat{\theta}\right)=O_{p}\left(N^{-\frac{1}{2}}\right)$ holds if the residuals follow a normal distribution.

Application in Deep learning

The discussions so far, which establish $\widehat{\lambda}=O_{p}\left(N^{-\frac{1}{2}}\right)$, were based on a classical statistical setting in which the number of free parameters is smaller than the number of observations $N$. However, in deep learning applications, the number of parameters can be in the millions or even billions, and therefore a direct application of the previous arguments may not be appropriate. Nevertheless, we can still show that $\widehat{\lambda}\approx0$ holds by slightly modifying the earlier reasoning. In this subsection, we focus on the LS problem, since PINNs primarily rely on the LS formulation. Note that analogous arguments also apply to the MLE setting.

First, we choose $\theta^{*}\in\arg\min_{\theta}E_{w}\left[\left(y-\psi\left(x;\theta\right)\right)^{2}\right]\ s.t.\ g_{E}\left(\theta\right)=0,g_{I}\left(\theta\right)\leq0$. Note that the minimizer may not be unique, and we select the one closest to $\widehat{\theta}$. Concerning the three terms discussed in Section (ref), the second term becomes close to 0 by the law of large numbers when $N$ is large, and the third term becomes close to zero when the true error distribution is close to normal. Importantly, these arguments continue to hold even when the number of parameters is very large. Concerning the first term, the mean value theorem implies

$\nabla_{\theta}\ln\left(h_{\sigma}\left(w_{i};\widehat{\theta}\right)\right)-\nabla_{\theta}\ln\left(h_{\sigma}\left(w_{i};\theta^{*}\right)\right)=\left(\nabla_{\theta\theta^{\prime}}\ln\left(h_{\sigma}\left(w_{i};\exists\overline{\theta}\right)\right)\right)\left(\widehat{\theta}-\theta^{*}\right)$, where $\overline{\theta}$ lies between $\widehat{\theta}$ and $\theta^{*}$. This suggests that the first term becomes small when if the generalization performance is good and $\widehat{\theta}\approx\theta^{*}$ holds. Consequently, under the conditions that the error distribution is close to normal, the sample size is large, and the generalization performance is good, the values of $\nabla_{\theta}Q\left(\widehat{\theta}\right)$ and $\widehat{\lambda}$ will be close to zero.

Regarding this point, lu2021physics, who proposed a deep learning-based method (PINNs with hard constraints) for solving topology optimization problems, reported a numerical example in which the values of $\widehat{\lambda}$ concentrate near zero in a holography problem in optics (Figure 6 of lu2021physics). The constrained optimization problem they consider is essentially similar to the LS problem (see also Section (ref) of the current paper). Although they did not discuss any mechanisms behind the numerical results, our statistical arguments offer a natural interpretation: the Lagrange multipliers tend to be close to zero when the distribution of the residuals is close to normal.

Mean Squared Residual

In applications of PDE (Partial Differential Equation)-constrained optimizations for solving inverse problems, the problems are sometimes in the following form, which involves an integral:

where $\Omega$ denotes a region and $Area\left(\Omega\right)$ represents its area. For instance, lu2021physics consider a holography problem in physics, that is formulated as a constrained optimization problem of this type. Note that $y\left(x\right)-\psi\left(x;\theta\right)$ corresponds to the difference between a target specified by an analyst and a model-predicted value in the problem. The discussions in Section (ref) imply that the Lagrange multipliers for this problem become equal to zero when the residuals $\left\{ y\left(x\right)-\psi\left(x;\theta\right)\right\} _{x\in\Omega}$ follows a normal distribution.

Implications

Augmented Lagrange Multiplier (ALM) Method

To simplify the discussion, we consider the Augmented Lagrange Multiplier (ALM) method for an equality-constrained problem of the form $\min_{\theta}Q\left(\theta\right)\ s.t.\ g\left(\theta\right)=0$. Let $L_{\rho}(\theta,\lambda)\equiv Q(\theta)+\lambda^{T}g(\theta)+\frac{\rho}{2}\left\Vert g(\theta)\right\Vert ^{2}$ be the augmented Lagrangian function, where $\rho$ is a penalty parameter. In the ALM, the following steps are iterated until convergence given the initial Lagrange multipliers $\lambda_{0}$ $(k=0,1,2,\cdots)$: $\theta_{k+1}=\min_{\theta}L_{\rho}(\theta,\lambda_{k})$, $\lambda_{k+1}=\lambda_{k}+\rho g_{E}(\theta_{k+1})$. As shown in bertsekas1982constrained, there exists a $\overline{\rho_{N}}>0$ such that $\nabla_{\theta}L_{\overline{\rho_{N}}}\left(\widehat{\theta},\widehat{\lambda}\right)>0$ and $\left\Vert \lambda_{k+1}-\widehat{\lambda}\right\Vert \leq\frac{\exists M_{N}}{\rho}\left\Vert \lambda_{k}-\widehat{\lambda}\right\Vert $, i.e., $\left\Vert \lambda_{k}-\widehat{\lambda}\right\Vert \leq\left(\frac{M_{N}}{\rho}\right)^{k}\left\Vert \lambda_{0}-\widehat{\lambda}\right\Vert $ for $\rho\geq\overline{\rho_{N}}$, and choosing $\lambda_{0}$ close to $\widehat{\lambda}$ is desirable to reduce the number of iterations. Note that $\overline{\rho_{N}}$ and $M_{N}$ do not diverge even when $N\rightarrow\infty$, as long as the solution $\min_{\theta}Q\left(\theta\right)=E_{w}\left[q\left(w;\theta\right)\right]\ s.t.\ g\left(\theta\right)=0$ exists.

birgin2014practical have recommended the use of 0 as the initial Lagrange multipliers unless we have prior knowledge on the true value of $\widehat{\lambda}$, although formal justifications were not given. Previous studies utilizing ALM-type algorithms set the initial Lagrange multipliers to 0 in practice (e.g., dener2020training,lu2021physics,basir2023adaptive for physics constrained neural networks, sangalli2021constrained for solving class-imbalanced binary classification). The discussions in Section (ref) suggest that the Lagrange multipliers of the problem ((ref)) are close to 0 under the conditions discussed before. Therefore, choosing $\lambda_{0}=0$ is not only a practical heuristic but also statistically justified.

Sequential Quadratic Programming and Interior Point Method

Sequential Quadratic Programming (SQP) and Interior Point (IP) methods require specifying not only the initial value of $\theta$ but also the initial Lagrange multipliers. Although previous studies have established superlinear local convergence and global convergence properties for these methods, their practical performance can deteriorate when the initial values are far from the true solution. In practice, the initial multipliers $\lambda_{0}=0$ are typically set to zero or chosen as the least-squares solution to the dual infeasibility,\ifx\@typeset@protect \expandafter\SF@@footnote \else \expandafter\SF@gobble@opt \fi {When only the equality constraints $g(\theta)=0$ exists, $\lambda_{0}$ is chosen as the solution of a least-square problem $\min_{\theta}\left\Vert \nabla_{\theta}Q\left(\theta_{0}\right)+\lambda^{T}\left(g\left(\theta_{0}\right)\right)\right\Vert _{2}^{2}$, where $\theta_{0}$ denotes the initial values of $\theta$ specified by an analyst.} often combined with safeguard strategies in certain solvers (e.g., IPOPT; wachter2006implementation), unless prior knowledge about the true multipliers is available.

The results presented in this study suggest that, under the statistical conditions discussed earlier, the true Lagrange multipliers tend to be close to zero. This implies that choosing $\lambda_{0}=0$ may reduce the number of iterations and function evaluations required for convergence. Please note that, unlike the ALM, the initial choice of $\theta$, is also important for the convergence. Nevertheless, initial $\lambda$ far from the true value may make the convergence slower and more unstable. Numerical results in Section (ref) support this observation.

Unconstrained optimization with a penalty term (Soft constraint)

To simplify the discussion, we consider the case in which only equality constraints $g\left(\theta\right)=0$ exist. In some applications, sometimes the following problem is solved instead of the original constrained optimization problem:

$\rho\left\Vert g\left(\theta\right)\right\Vert _{2}^{2}$ represents a penalty term that becomes large when $g\left(\theta\right)$ deviates from zero. Let $\widetilde{\theta}$ denote the solution to the problem ((ref)).

First, $\widehat{\theta}$ is a local minimizer of $\min_{\theta}Q\left(\theta\right)+\widehat{\lambda}g\left(\theta\right)+\rho\left\Vert g\left(\theta\right)\right\Vert ^{2}$ when $\rho$ is large enough (cf. Theorem 17.5 of nocedal2006numerical). Here, $\widehat{\lambda}$ can be regarded as the consistent estimator of $\lambda^{*}=0$, because $\widehat{\lambda}=O_{p}\left(N^{-\frac{1}{2}}\right)$. Thus, by treating the Lagrange multipliers $\lambda$ as nuisance parameters and applying Property 24.8 of gourieroux1995statistics,\ifx\@typeset@protect \expandafter\SF@@footnote \else \expandafter\SF@gobble@opt \fi {The property claims that the extremum estimator $\widehat{\beta}$, which is the solution to $\min_{\beta}\widetilde{Q}$$\left(\widehat{\alpha},\beta\right)$, is a consistent estimator of true $\beta$ if $\widehat{\alpha}$ is a consistent estimator of the true nuisance parameter $\alpha$.} $\widetilde{\theta}$ can be viewed as a consistent estimator of $\theta^{*}$, which is the solution of $\min_{\theta}Q\left(\theta\right)=E_{w}\left[q\left(w;\theta\right)\right]\ s.t.\ g\left(\theta\right)=0$.

In deep learning applications, however, such asymptotic arguments may not hold because the number of parameters can be much larger than the number of observations. Even so, we can generally show that the difference between $\widehat{\theta}$ and $\widetilde{\theta}$ remains small under moderate conditions.

The first\nobreakdash-order optimality conditions for $\widehat{\theta}$ and $\widetilde{\theta}$ imply:

By Taylor's theorem, $\nabla_{\theta}Q\left(\widetilde{\theta}\right)=\nabla_{\theta}Q\left(\widehat{\theta}\right)+\left(\nabla_{\theta\theta^{\prime}}Q\left(\overline{\theta}\right)\right)\left(\widetilde{\theta}-\widehat{\theta}\right)$ and $g\left(\widetilde{\theta}\right)=g\left(\widehat{\theta}\right)+\left(\nabla_{\theta}g\left(\breve{\theta}\right)\right)\left(\widetilde{\theta}-\widehat{\theta}\right)$ holds, where $\overline{\theta}$ and $\breve{\theta}$ lie between $\widehat{\theta}$ and $\widetilde{\theta}$. Then, ((ref)), ((ref)), and $g\left(\widehat{\theta}\right)=0$ imply $\left(\nabla_{\theta\theta^{\prime}}Q\left(\overline{\theta}\right)+2\rho\left(\nabla_{\theta}g\left(\widetilde{\theta}\right)\right)^{T}\left(\nabla_{\theta}g\left(\breve{\theta}\right)\right)\right)\left(\widetilde{\theta}-\widehat{\theta}\right)=\widehat{\lambda}\left(\nabla g\left(\widehat{\theta}\right)\right)$. Assuming the nonsingularity of the matrix $\nabla_{\theta\theta^{\prime}}Q\left(\overline{\theta}\right)+2\rho\left(\nabla_{\theta}g\left(\widetilde{\theta}\right)\right)^{T}\left(\nabla_{\theta}g\left(\breve{\theta}\right)\right)$, we obtain $\widetilde{\theta}-\widehat{\theta}=\left(\nabla_{\theta\theta^{\prime}}Q\left(\overline{\theta}\right)+2\rho\left(\nabla_{\theta}g\left(\widetilde{\theta}\right)\right)^{T}\left(\nabla_{\theta}g\left(\breve{\theta}\right)\right)\right)^{-1}\widehat{\lambda}\left(\nabla g\left(\widehat{\theta}\right)\right)$.

The formula suggests that $\widetilde{\theta}-\widehat{\theta}$ is close to 0 when $\widehat{\lambda}$ is close to 0, unless $\nabla_{\theta\theta^{\prime}}Q\left(\overline{\theta}\right)+2\rho\left(\nabla_{\theta}g\left(\widetilde{\theta}\right)\right)^{T}\left(\nabla_{\theta}g\left(\breve{\theta}\right)\right)$ becomes nonsingular. Hence, small $\left\Vert \widehat{\lambda}\right\Vert $ implies small difference between $\widehat{\theta}$ and $\widetilde{\theta}$.

Since $\widehat{\lambda}$ tends to be close to zero in large samples for MLE and LS under the conditions discussed earlier, the difference between the exact constrained solution and the solution to the penalized unconstrained problem can be very small. This provides a theoretical explanation for why penalty\nobreakdash-based approaches often perform well in practice, even when the penalty parameter $\rho$ is not extremely large.

Although the difference between the two solutions vanishes as $\rho\rightarrow\infty$, it is well known that excessively large values of $\rho$ can lead to numerical instability, making reliable optimization difficult. The results of the present study suggest that penalty\nobreakdash-based methods can still perform well with moderate values of $\rho$ when the true Lagrange multipliers $\widehat{\lambda}$ are small.

Numerical Experiments

This section presents numerical experiments designed to demonstrate that choosing $\lambda_{0}=0$ is effective in the statistical applications discussed in this paper. We consider both the Augmented Lagrange Multiplier (ALM) algorithm and the Interior Point (IP) algorithm. All experiments are implemented in Julia. For the IP method, we use the IPOPT package (v1.13.0) in Julia 1.12.3. Additional details of the experimental setup are provided in Appendix (ref).

Constrained regression problems

We first conduct numerical experiments focusing on constrained regression problems---specifically, linear regression and logistic regression---as in na2025statistical. These correspond to LS and MLE problems, respectively. In both cases, we impose a nonlinear constraint $\left\Vert \theta\right\Vert _{2}^{2}=b$, which is consistent with the data\nobreakdash-generating process.

Table (ref) reports the results for the ALM. For each problem setting, we vary the initial Lagrange multiplier $\lambda_{0}$ and examine its effect on the number of main iterations and the number of objective evaluations. The results clearly indicate that choosing $\lambda_{0}=0$ minimizes both the number of main iterations and the number of objective evaluations.

Table (ref) presents the results for the IP method. We again observe the tendency that $\lambda_{0}=0$ outperforms other choices. Additional numerical results are provided in Appendix (ref), and they exhibit similar patterns.

Dynamic discrete choice model estimation

We next conduct numerical experiments by focusing on dynamic discrete choice (DDC) model estimation, using the bus engine replacement data from rust1987optimal, which has been treated as a benchmark model and dataset in the economics literature (cf. su2012constrained,iskhakov2016comment). Note that estimations of dynamic discrete choice models is closely related to inverse reinforcement learning, which has been extensively considered in recent machine learning studies, as discussed sanghvi2021inverse. In this problem, we estimate the utility parameters of a forward\nobreakdash-looking agent who owns buses and decides when to replace their engines. The objective function corresponds to a likelihood function, and the constraints correspond to equalities derived from the agent’s Bellman equation. As discussed in Appendix (ref), we also impose several inequality constraints to exclude unrealistic parameter domains.

Table (ref) shows the results when we apply the IP. As in the case of the constrained regression problems, the choice of $\lambda_{0}=0$ usually outperforms other values of $\lambda_{0}$. Note that the table also reports the results obtained when we employ the default choice of $\lambda$. The numerical results suggest that the simple choice of $\lambda_{0}=0$ can outperform the default setting, which incorporates several safeguard strategies.

Conclusions

The current study has investigated the property of Lagrange multipliers in constrained Maximum Likelihood Estimation (MLE) and Least Squares (LS) problems---which frequently appear in various applications---from the perspective of numerical optimization. Building on large-sample theory in statistics, we can utilize a property that Lagrange multiplier take values close to zero under some conditions. The property would be useful for further investigation of more efficient numerical optimization methods, and we leave this direction for future research.

Declarations

\paragraph*{Funding}

This study was supported by JSPS KAKENHI Grant Number JP24K22629.