Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
44,406 characters · 21 sections · 51 citation commands
Algorithmic Compliance and Regulatory Loss in Digital Assets
\noindentKeywords: Cryptocurrency, Anti-Money Laundering, Concept Drift, Regulatory Technology, Model Risk
The rapid expansion of cryptocurrency markets has intensified regulatory concern over illicit financial activity conducted through digital assets. Empirical evidence using Bitcoin transaction data suggests that illegal activity has accounted for a substantial share of on-chain usage in some periods, particularly linked to darknet-market flows foley2019sex. In response, regulators, exchanges, and blockchain analytics firms increasingly rely on automated anti-money laundering (AML) systems to monitor transactions at scale. These systems are typically evaluated using static classification metrics such as the area under the receiver operating characteristic curve (ROC-AUC) or the precision-recall area under the curve (PR-AUC). While such metrics summarize ranking performance, they abstract from the decision environment in which AML systems actually operate.
In practice, regulators must select enforcement thresholds that determine which transactions are flagged for investigation. These thresholds reflect asymmetric costs: false negatives allow illicit activity to proceed, while false positives generate investigative burden and compliance costs. As emphasized in the cost-sensitive decision-making literature, optimal classification depends not only on predictive accuracy but also on how misclassification costs enter downstream enforcement decisions elkan2001foundations. Industry accounts of transaction monitoring reinforce that alert volumes, false-positive rates, and threshold governance are first-order operational constraints in AML programs oztas2024transaction. Supervisory expectations have similarly become more explicit: Financial Action Task Force's (FATF) risk-based guidance for virtual assets and Virtual Assest Service Providers (VASP) emphasizes ongoing risk assessment and monitoring, with controls calibrated to evolving typologies and risk exposure fatf2021vasp. Together, these perspectives underscore that AML performance is ultimately determined by how model outputs are translated into enforcement actions, not by ranking accuracy alone.
Yet most empirical evaluations of AML systems implicitly assume stationarity: models are trained and tested as if the underlying data-generating process and the associated optimal enforcement threshold remain stable over time. This assumption is particularly problematic in cryptocurrency markets, where prior work finds that Bitcoin often behaves more like a speculative asset than a medium of exchange and that its pricing dynamics can vary substantially across market conditions, implying that relationships estimated in one period may not remain stable out of sample baek2015bitcoins,baur2018bitcoin,biais2023equilibrium. When feature distributions, label frequencies, or their joint relationship evolve, models trained on historical data may become systematically misaligned with current conditions widmer1996learning,gama2014survey. Crucially, this misalignment need not manifest as declining predictive accuracy. The model risk literature emphasizes that decisions calibrated in one regime can become unreliable as conditions shift danielsson2016model. In our empirical setting, we show that strong static classification performance can co-exist with large deployment losses because fixed enforcement thresholds become miscalibrated as class prevalence and score distributions evolve over time.
This paper addresses that gap by studying the deployment performance of automated AML systems in an evolving crypto transaction network. Using the Elliptic Bitcoin transaction dataset, we move beyond static evaluation and implement forward-looking and rolling deployment designs that more closely mirror real-world regulatory use. We explicitly model regulatory loss as a function of false positives and false negatives and examine how cost-sensitive enforcement thresholds evolve over time. This framework allows us to quantify the economic cost of deploying static decision rules in a non-stationary environment, relative to an infeasible but informative oracle benchmark that re-optimizes thresholds ex-post. We summarize this inefficiency using the deployment gap, defined as realized loss under the historically calibrated threshold minus the oracle loss under contemporaneous re-optimization. Regulatory fragility therefore arises not primarily from declining predictive accuracy, but from miscalibration of decision rules as market conditions and illicit prevalence evolve.
Our results reveal a clear disconnect between static classification performance and realized regulatory outcomes. Random train-test splits suggest strong predictive accuracy, but forward and rolling evaluations uncover pronounced instability in optimal enforcement thresholds and large, persistent excess regulatory losses. These losses are economically meaningful and robust across regulatory cost ratios, often approaching or exceeding a doubling of loss relative to the oracle benchmark. We employ a simple and transparent predictive model to isolate the role of deployment design and regulatory decision rules from model-specific effects. By focusing on deployment loss rather than statistical accuracy alone, this paper contributes to three strands of literature. First, we complement research on concept drift by documenting its economic consequences in a regulatory context. Second, we add to the financial machine learning and model risk literature by showing how conventional evaluation practices can mask deployment fragility. Third, we contribute to the literature on cryptocurrency regulation and regulatory technology by reframing AML effectiveness in terms of loss-based outcomes rather than static classification metrics anagnostopoulos2018fintech,hardjono2021toward.
The remainder of the paper proceeds as follows. Section 2 reviews the related literature. Section 3 describes the data and institutional setting. Section 4 outlines the regulatory decision framework. Section 5 presents the empirical results. Section 6 discusses policy implications and limitations. Section 7 concludes.
This paper relates to four strands of literature: the economics of cryptocurrencies and regulatory technology; model risk in financial decision systems; classifier evaluation and cost-sensitive decision making; and learning under non-stationarity. Our contribution lies in linking these strands through an explicit focus on the economic performance of deployed regulatory decision rules. While cost-sensitive and adaptive detection approaches exist, deployment-centered evidence on the regulatory loss consequences of holding enforcement thresholds fixed under temporal non-stationarity remains limited.
A first strand concerns the regulation of digital asset markets and the feasibility of transaction monitoring. anagnostopoulos2018fintech and hardjono2021toward propose infrastructure for VASP-facing compliance (e.g., identity/key management), highlighting implementation challenges for secure VA ecosystems. foley2019sex estimate that a substantial share of cryptocurrency transactions has historically been associated with illicit activity, underscoring the importance of effective AML systems. Using heuristic clustering on the public ledger, meiklejohn demonstrate that transaction graphs can link clusters of activity to real-world services, providing a foundation for compliance analytics. weber2019aml introduce the Elliptic dataset and benchmark supervised methods for illicit transaction classification, while schnoering2025bitcoin large-scale temporally annotated Bitcoin transaction graphs for broader blockchain analytics. More recent theoretical contributions model Bitcoin pricing biais2023equilibrium and token-based platform finance cong2022token, highlighting how incentives and market structure can shape outcomes in tokenized ecosystems. Early empirical work documents that Bitcoin exhibits strong speculative components and time-varying market behavior baek2015bitcoins,baur2018bitcoin, while evidence such as the Kimchi premium highlights market frictions and speculative trading that can vary across venues and over time eom2021kimchi. In addition, evidence of coordinated price manipulation episodes suggests that crypto markets can experience abrupt shifts in behavior that complicate stable inference gandal2018price. These dynamics imply that the statistical properties of transaction data, including illicit prevalence and observable transaction patterns, may vary substantially over time.
A second strand applies machine learning methods to financial data, with particular attention to model risk and evaluation design. gu2020empirical and kelly2019characteristics show that high-dimensional predictors/ML methods can be competitive in asset pricing tasks, while sirignano2021universal document stable patterns in price formation using deep learning. At the same time, bailey2017probability show that back-test overfitting is pervasive in financial applications, and de2018advances emphasize the importance of robust validation and deployment practices. danielsson2016model formalize model risk and show that decisions calibrated in one regime may become unreliable as conditions shift. We complement this literature by showing that even simple, well-performing classifiers can generate substantial regulatory losses when cost-sensitive decision rules are not dynamically recalibrated.
A third strand concerns how predictive models are evaluated and translated into operational decisions. As class prevalence shifts, conventional ranking summaries can give a distorted picture of operational performance: ROC-based measures may appear stable even when precision and PR performance change materially, because the ROC--PR relationship depends directly on the base rate fawcett2006introduction,hand2009measuring,davis2006relationship. saito2015pr further show that precision--recall summaries are often more informative than ROC-based summaries under severe class imbalance, which closely matches AML settings. More fundamentally, elkan2001foundations demonstrates that optimal classification depends on asymmetric misclassification costs rather than statistical accuracy alone, and nami2018cost develop cost-sensitive fraud detection methods that embed these costs directly into the detection objective. kleinberg2016inherent and corbett2023measure further show that trade-offs between competing objectives are inherent in risk-scoring systems, and that mismeasurement arises when ranking performance is conflated with decision quality. davis2006relationship link precision-recall and ROC curves and emphasize that metric choice materially affects perceived performance under class imbalance. Building on these insights, we examine how cost-sensitive enforcement thresholds evolve over time and show that static deployment rules can generate economically meaningful losses in non-stationary environments.
A fourth strand studies predictive modeling when the data-generating process evolves. widmer1996learning formalize learning under concept drift, emphasizing that models trained on historical data may become systematically misaligned as underlying patterns change. gama2014survey survey concept drift adaptation methods and highlight a fundamental trade-off between stability and adaptability in deployed systems. morenotorres2012datasetshift provide a unifying framework for distinguish changes in covariates, class priors, and the conditional relationship between features and labels across training vs deployment. Recent work also proposes graph neural network architectures for blockchain fraud detection haider2025towards. However, even dynamic modeling approaches do not by themselves resolve the governance problem of deploying cost-sensitive decision rules under drift. Our analysis extends this literature by quantifying the regulatory costs of failing to adapt enforcement thresholds in a rapidly evolving market environment.
Cryptocurrency exchanges and virtual asset service providers are subject to AML obligations analogous to those imposed on traditional financial institutions. In practice, compliance is operationalized through automated risk-scoring systems that assign a probability of illicit activity to each transaction and trigger enforcement actions when scores exceed a pre-specified threshold. Such systems implement a resource allocation rule under asymmetric costs. Compliance teams face regulatory penalties for under-enforcement, operational costs from over-enforcement, and capacity constraints in investigative review. The performance of an AML system is therefore not defined solely by predictive accuracy, but by how effectively it minimizes regulatory loss given these constraints. Our empirical setting mirrors this regulatory deployment problem using transaction-level data from the Bitcoin network.
We use the publicly available Elliptic Bitcoin transaction dataset, which has become a benchmark for studying illicit cryptocurrency activity weber2019aml. Its structure combines transaction-level labels with a time-indexed transaction graph, making it well suited for analyzing how predictive systems perform when deployed sequentially over time.
The dataset offers two key advantages for our purposes. First, it contains transaction-level labels informed by proprietary heuristics and law enforcement intelligence, allowing us to approximate the ground truth used in real-world AML monitoring. While these labels are necessarily imperfect, they reflect the practical reality of AML enforcement, in which compliance decisions rely on noisy ground truth constructed from investigations and heuristic rules rather than definitive ex-post verification. Second, the dataset's discrete temporal structure permits forward-looking and rolling evaluation designs that closely mirror actual deployment conditions.
The dataset consists of a directed transaction graph in which nodes represent Bitcoin transactions and edges represent flows of funds. Each transaction is assigned a discrete time index from 1 to 49. Transactions are labeled as either illicit or licit based on Elliptic's classification procedures. Observations labeled as unknown are excluded from the analysis. After excluding unlabeled observations, the final sample contains 46{,}564 labeled transactions, of which 9.8 percent are illicit. Illicit prevalence declines over time, falling from 14.3 percent in early periods to 5.3 percent in later periods. This time variation in the base rate of illicit activity is economically meaningful because optimal cost-sensitive enforcement thresholds depend mechanically on both predicted risk scores and class prevalence. As a result, shifts in illicit incidence directly affect the regulatory loss associated with fixed enforcement rules. These descriptive dynamics are summarized in Appendix A, Table (ref).
Each transaction is described by 165 engineered features capturing transaction behavior, local network structure, and aggregate flow characteristics. All features are constructed using information available at or before each transaction's time step. No forward-looking information is used in model training or evaluation, preventing look-ahead bias. These features approximate the information set available to blockchain analytics firms and compliance teams at the time of monitoring.
We exclude observations labeled unknown and restrict attention to transactions with definitive licit or illicit labels. Transactions are indexed by discrete time steps $t \in \{1,\dots,49\}$, and all evaluation designs enforce temporal ordering.
Predicted illicit risk is generated using a regularized logistic regression classifier with L2 penalization. Features are standardized using training-sample moments, and the same transformation is applied to the corresponding test data. We report three evaluation protocols. (i) A random split benchmark using a stratified 70/30 split that ignores temporal ordering and serves only as a conventional baseline. (ii) A forward split that trains on transactions with $t \le 34$ and tests on transactions with $t \ge 35$. (iii) A rolling deployment design in which, for each test period $t$, the classifier is trained on the preceding 10 time steps $\{t-10,\dots,t-1\}$ and deployed on transactions observed at $t$; periods with fewer than 50 labeled test transactions are excluded to avoid unstable period-level metrics.
Enforcement thresholds are chosen to minimize the cost-sensitive regulatory loss function described in Section 4. In the forward and rolling designs, the loss-minimizing threshold $\tau^{*}$ is selected using training data only and then held fixed when computing test outcomes. For benchmarking purposes, we also compute an infeasible oracle that re-optimizes $\tau^{*}$ using contemporaneous test data; the oracle is not deployable and serves only to benchmark the cost of holding thresholds fixed under temporal non-stationarity. Regulatory loss is computed in levels as $C_{FN}\times FN + C_{FP}\times FP$ within each test period. Because rolling outcomes are serially correlated across adjacent test periods, uncertainty in average deployment loss and excess loss is assessed using a moving block bootstrap over contiguous test periods, with block lengths 3, 5, and 7 and 5{,}000 bootstrap replications.
We conceptualize automated AML systems as implementing a regulatory decision rule rather than a purely predictive task. At each point in time, transactions generate observable signals summarized by model-based risk scores that regulators or compliance teams must translate into enforcement actions through an enforcement threshold that determines which transactions are flagged for investigation, delayed, or blocked.
The enforcement threshold is a policy instrument chosen under asymmetric costs and operational constraints. False negatives allow illicit activity to proceed and may generate regulatory penalties or reputational harm, while false positives impose investigative burdens, compliance costs, and potential disruption to legitimate users. Given limited investigative capacity, the regulator's objective is not to maximize predictive accuracy, but to minimize expected regulatory loss. The loss-minimizing enforcement threshold depends on three objects: (i) the distribution of risk scores produced by the model, (ii) the underlying prevalence of illicit activity, and (iii) the relative costs of false positives and false negatives. Even when the predictive model itself remains fixed, changes in any of these objects alter the optimal decision rule. In non-stationary environments, the optimal enforcement threshold is therefore inherently time-varying.
Regulatory fragility arises when enforcement thresholds calibrated under past conditions are deployed in evolving environments. When transaction patterns, illicit prevalence, or score distributions shift, static thresholds become misaligned with current conditions, generating excess regulatory loss even when conventional measures of predictive performance, such as ROC-AUC or PR-AUC, remain high. By comparing realized regulatory loss under historically calibrated thresholds to an infeasible benchmark that re-optimizes thresholds ex-post, we isolate the economic cost of static decision rules in a non-stationary environment.
We model automated AML systems as implementing a cost-minimizing regulatory policy. Let $y_i \in \{0,1\}$ denote the true status of transaction $i$, where $y_i = 1$ indicates illicit activity, and let $\hat{p}_i$ denote the predicted probability that transaction $i$ is illicit. A transaction is flagged if $\hat{p}_i \geq \tau$, where $\tau$ is an enforcement threshold.
Following elkan2001foundations, regulatory loss is defined as: \[ L(\tau) = C_{FN} \cdot FN(\tau) + C_{FP} \cdot FP(\tau), \] where $FN(\tau)$ and $FP(\tau)$ denote the number of false negatives and false positives induced by threshold $\tau$, and $C_{FN}$ and $C_{FP}$ represent the relative costs of these errors. This loss function captures regulatory penalties, reputational damage, compliance costs, and investigative burdens. Cost-sensitive objectives are common in fraud detection research and are often implemented by weighting false negatives more heavily than false positives nami2018cost. In AML settings, thresholds are tuned to manage alert volumes under binding investigative capacity, making the economic relevance of false positives and false negatives inherently operational oztas2024transaction. The regulator's objective is to choose the enforcement threshold $\tau$ that minimizes expected loss.
We evaluate regulatory loss under two alternative cost ratios, $C_{FN}/C_{FP} \in \{10, 25\}$. These values are not intended to represent precise or estimated regulatory parameters, which are rarely observable in practice, but rather to capture economically plausible asymmetry between the costs of false negatives and false positives in AML enforcement. The cost-sensitive learning literature emphasizes that enforcement objectives in high-risk environments typically place far greater weight on avoiding false negatives than on minimizing routine compliance costs elkan2001foundations, and the model risk literature highlights that decision-rule miscalibration under changing environments can amplify losses even when predictive accuracy appears stable danielsson2016model. We therefore use moderate and high penalty ratios to assess whether deployment fragility persists across alternative but economically relevant enforcement environments. All core results are reported under both cost ratios, providing a sensitivity analysis with respect to the relative cost of misclassification. Supplementary evidence, including rolling-window summaries and block bootstrap inference under alternative dependence assumptions, is reported in Appendix A and Appendix B.
We treat enforcement thresholds as policy instruments rather than as mechanical outputs of predictive models. The loss-minimizing threshold depends on three objects that may all evolve over time: the distribution of predicted risk scores, the underlying prevalence of illicit activity, and the relative costs of false positives and false negatives. As a result, regulatory fragility can arise through threshold miscalibration even when the underlying predictive model remains fixed and continues to exhibit strong ranking performance.
We compare three evaluation regimes. First, a conventional random train-test split assumes stationarity and provides a benchmark comparable to much of the existing machine learning literature. Second, a forward split trains the model on early periods and evaluates it on later periods, approximating real-world regulatory rollout while eliminating temporal leakage. Third, we implement a rolling deployment design in which, at each time step $t$, the model is trained on a fixed-length historical window ending at $t-1$ and deployed on transactions observed at $t$. In this design, the enforcement threshold is optimized using training data only and then held fixed during deployment, mirroring operational AML systems where retraining and recalibration are costly and therefore infrequent.
To isolate the inefficiency arising from static threshold deployment, we construct an infeasible oracle benchmark that re-optimizes the enforcement threshold using contemporaneous test data. Let $L^{deploy}_t$ denote regulatory loss under the historically calibrated threshold and $L^{oracle}_t$ denote loss under the oracle threshold. We define excess regulatory loss as \[ \text{Excess Loss}_t = L^{deploy}_t - L^{oracle}_t. \] This quantity provides a direct measure of regulatory inefficiency attributable to threshold miscalibration in a changing environment, holding the predictive model fixed. For comparability with prior work, we also report standard classification metrics, including ROC-AUC and PR-AUC fawcett2006introduction,hand2009measuring. However, our primary outcomes are economic rather than statistical: realized regulatory loss, excess regulatory loss relative to the oracle benchmark, deployment-to-oracle loss ratios, and precision among the highest-risk transactions.
We begin by evaluating regulatory performance under deployment, comparing enforcement thresholds calibrated on historical data to an infeasible oracle benchmark that re-optimizes thresholds using contemporaneous information. Figures (ref) and (ref) plot realized regulatory loss over time under two alternative cost ratios, $C_{FN}/C_{FP}\in\{10,25\}$, which serve as distinct but economically plausible enforcement environments rather than calibrated policy parameters.
Across both cost ratios, deployed regulatory loss systematically exceeds oracle loss. Importantly, the deployment gap is highly time-varying. Loss differentials widen sharply during periods of market transition and contract during relatively stable intervals, indicating that excess regulatory loss is state-dependent rather than a mechanical artifact of model choice or evaluation design. Appendix A, Figures (ref) and (ref) plot the deployed-to-oracle loss ratio over time, confirming that deployment inefficiency is episodic rather than uniform. Under $C_{FN}/C_{FP}=10$ (Figure (ref)), deployed loss frequently approaches or exceeds twice the oracle benchmark during transition periods. When the relative cost of false negatives is increased to $C_{FN}/C_{FP}=25$ (Figure (ref)), loss spikes become larger and more persistent, reflecting the heightened sensitivity of enforcement outcomes to threshold miscalibration when under-enforcement is more costly.
Table (ref) summarizes the magnitude of this deployment gap across rolling windows. Mean deployed loss exceeds oracle loss by approximately 69 units under $C_{FN}/C_{FP}=10$ and by 154 units under $C_{FN}/C_{FP}=25$. These average effects mask substantial episodic instability. The aggregate ratio of means (1.51 and 1.75) understates the magnitude of inefficiency observed during peak periods. Mean window-level loss ratios of 1.97 and 2.23 indicate that the average rolling test window incurs approximately 97 and 123 percent more regulatory loss than the oracle benchmark, respectively, with short-lived but severe spikes accounting for a large share of overall deployment loss. The dispersion and right tail of these loss spikes is summarized in Appendix A, Figure (ref). Moving block bootstrap confidence intervals exclude zero excess loss across alternative block lengths, confirming that these patterns are statistically robust rather than driven by sampling noise. Full moving-block bootstrap results across alternative block lengths are reported in Appendix A, Table (ref).
To understand the source of this deployment fragility, we examine the evolution of the loss-minimizing enforcement threshold over time. Figure (ref) plots the optimal threshold $\tau^*$ estimated on rolling training windows for $C_{FN}/C_{FP}=10$. The threshold exhibits pronounced instability, including discrete jumps and sustained drift across adjacent periods, despite the predictive model itself being held fixed. Appendix B, Figures (ref) and (ref) visualize the deployed-versus-oracle threshold miscalibration gap directly.
This instability follows mechanically from the regulatory loss function. For a fixed cost ratio, the loss-minimizing threshold depends on both the prevalence of illicit activity and the distribution of predicted risk scores. Changes in either object alter the trade-off between false positives and false negatives, shifting the optimal enforcement rule even when predictive accuracy remains stable. As shown in Appendix A (Figure (ref)), illicit prevalence declines markedly over time in the rolling test windows. Periods of sharp base-rate decline coincide with downward adjustments in the optimal threshold, as the marginal cost of false positives rises relative to false negatives. Conversely, periods of elevated or stabilizing illicit prevalence are associated with higher and more stable thresholds, consistent with the threshold jumps observed in Figure (ref).
In parallel, shifts in the distribution of predicted risk scores contribute to threshold instability. As score distributions compress or shift over time, reflecting changes in transaction behavior and network structure, the same nominal threshold corresponds to different effective false-positive and false-negative rates. Consistent with this mechanism, periods characterized by greater score dispersion are associated with higher loss-minimizing thresholds and lower deployment loss, whereas compressed score distributions coincide with threshold drops and loss spikes.
Table (ref) summarizes these relationships using rolling-window averages. Windows with lower illicit prevalence and compressed score distributions exhibit systematically lower thresholds and higher deployed-to-oracle loss ratios. These patterns arise directly from the structure of the regulatory loss function under temporal non-stationarity and require no additional estimation. A quantitative summary of deployment loss by illicit prevalence tertile is reported in Appendix B, Table (ref). Supplementary descriptive evidence on base-rate dynamics and rolling-window summaries is reported in Appendix A (Figure (ref); Table (ref)), and the distribution of deployment loss ratios is shown in Appendix A, Figure (ref).
Finally, we contrast these deployment outcomes with conventional static evaluation. Table (ref) compares a random train-test split to a forward, time-respecting split. Under random sampling, the classifier appears highly effective, with PR-AUC of 0.77, ROC-AUC exceeding 0.96, and top-one-percent precision above 0.82. Once temporal ordering is respected, performance deteriorates sharply. PR-AUC falls to 0.36, top-one-percent precision declines to 0.43, and test loss nearly doubles under both cost ratios.
Overall, the results establish three core findings. Static evaluation substantially overstates AML effectiveness. Rolling deployment generates large and statistically robust excess regulatory losses relative to a dynamically recalibrated benchmark. Instability in cost-sensitive enforcement thresholds, driven by changes in base rates and score distributions, is a central mechanism underlying regulatory fragility in evolving crypto-asset markets.
To assess the generality of these findings, Appendix (ref) reports three additional robustness exercises. First, we replicate the rolling deployment analysis using XGBoost to determine whether model capacity resolves the observed fragility. Despite substantially higher predictive accuracy under random evaluation (PR-AUC of 0.978 versus 0.769), forward-split test losses converge across models (7{,}521 for XGBoost versus 7{,}579 for logistic regression), and rolling deployment fragility persists, confirming that the result is a property of the decision rule under non-stationarity rather than of any particular model's predictive limitations (Table (ref) (ref), Figures (ref)--(ref)). Second, we decompose window-level loss ratios using OLS with HAC standard errors to examine the sources of deployment failure. Score dispersion is negatively associated with fragility when measured on contemporaneous test data, but this relationship vanishes when using the training-window dispersion observable at deployment time, indicating that score compression is detectable ex-post but not foreseeable ex-ante (Tables (ref) (ref) (ref), Figures (ref)--(ref)). Third, we demonstrate that feasible threshold recalibration fails to close the deployment gap across all tested window lengths (2 to 7 periods). Short recalibration windows amplify estimation noise and worsen loss, while longer windows provide negligible improvement over the static threshold (Table (ref), Figures (ref)--(ref)). Together, these exercises indicate that deployment fragility is robust to model choice, is not explained by predictable window-level characteristics, and may not be resolved through mechanical threshold updating.
The results reveal a substantial gap between static machine learning performance and realized regulatory outcomes in crypto asset monitoring. Strong performance under conventional evaluation can co-exist with large and persistent regulatory losses once models are deployed in a changing environment. In other words, high AUC or precision does not guarantee effective enforcement.
This finding connects directly to the concept drift literature, which emphasizes that models trained on historical data may become misaligned when the underlying data-generating process evolves widmer1996learning,gama2014survey. In cryptocurrency markets characterized by rapid innovation, shifting participation, and evolving transaction patterns, such drift is not merely a statistical inconvenience. Our evidence suggests that it generates economically meaningful increases in enforcement costs when decision rules remain fixed. Prior research documents substantial regime variation in crypto markets driven by speculative trading, regulatory frictions, and changing market structure baek2015bitcoins,baur2018bitcoin,eom2021kimchi,biais2023equilibrium. We show that such shifts translate into instability in optimal enforcement thresholds and large increases in realized regulatory loss. The economic consequences of regime change are therefore not limited to price dynamics or volatility; they also affect the efficiency of compliance systems.
Methodologically, our findings reinforce broader concerns about model evaluation in finance. A growing literature warns that random train-test splits can produce overly optimistic assessments of model performance when data are temporally dependent or structurally evolving bailey2017probability,de2018advances. We show that this concern is particularly acute in regulatory settings, where random splits obscure deployment fragility that forward and rolling evaluations reveal clearly. A central contribution of the paper is to highlight threshold instability as a distinct channel of regulatory fragility. Building on cost-sensitive learning elkan2001foundations, we show that the loss-minimizing threshold varies substantially over time even when the predictive model is held fixed. This instability provides a mechanism through which otherwise stable models can generate inefficient outcomes, echoing work on algorithmic decision systems showing that policy-relevant performance depends on the chosen objective and the decision rule used to translate scores into actions kleinberg2016inherent,corbett2023measure. More broadly, our findings align with the model risk literature: risk models that appear reliable under backtesting may fail when economic conditions change, generating losses that are foreseeable ex-post but difficult to prevent ex-ante danielsson2016model.
From a regulatory design perspective, these results suggest several cautious implications. First, supervisory assessment of AML systems may benefit from incorporating time-respecting evaluation protocols, such as forward-looking or rolling deployment exercises, alongside conventional static metrics. Second, threshold selection may be more appropriately viewed as an ongoing governance decision rather than a one-time technical calibration, particularly in environments where illicit prevalence and transaction patterns evolve rapidly. Third, evaluating regulatory loss under alternative prevalence or score distribution scenarios may help identify settings in which static enforcement rules are especially fragile. Consistent with industry accounts of transaction monitoring oztas2024transaction, our results suggest that threshold calibration and alert governance are not merely implementation details but first-order determinants of AML effectiveness in fast-changing digital asset environments anagnostopoulos2018fintech.
Several limitations suggest directions for future research. We focus on a single benchmark dataset and a relatively simple predictive model in order to isolate deployment effects; extending the analysis to alternative architectures or more recent datasets would help assess external validity. In addition, we do not model endogenous behavioral responses by illicit actors to enforcement policies. Strategic adaptation could amplify or mitigate the threshold instability documented here. Finally, incorporating network dynamics more explicitly into the regulatory objective may provide further insight into systemic versus idiosyncratic illicit risk within crypto transaction networks. Addressing these questions would deepen our understanding of regulatory fragility in algorithmic enforcement systems and inform the design of more resilient supervisory frameworks.
This paper examines the deployment performance of automated AML systems in evolving cryptocurrency markets and shows that strong static classification metrics substantially overstate real-world regulatory effectiveness. While conventional evaluation suggests high predictive performance, forward-looking and rolling deployment exercises reveal pronounced fragility once enforcement systems operate in non-stationary environments. Optimal enforcement thresholds are highly unstable over time, and when thresholds calibrated on historical data are held fixed during deployment, regulatory losses rise sharply relative to a dynamically recalibrated benchmark. These excess losses are economically meaningful, statistically robust, and persistent across alternative regulatory cost ratios, indicating that deployment fragility is a first-order concern rather than a technical detail.
Our findings clarify that this fragility arises not only from potential declines in predictive accuracy, but from the interaction between temporal non-stationarity and fixed decision rules governing enforcement. Even when the predictive model itself remains unchanged, shifts in illicit prevalence and score distributions render previously optimal thresholds inefficient. By shifting attention from static accuracy metrics to loss-based deployment outcomes, this paper provides a framework for evaluating algorithmic compliance systems in rapidly evolving digital asset markets. More broadly, the results suggest that effective regulatory oversight of automated decision systems requires time-respecting validation, careful governance of enforcement thresholds, and explicit attention to the economic consequences of misclassification.
This research received no external funding. The authors declare that no financial support was provided by public, commercial, or non-profit entities for the conduct of this study.
The authors declare no competing financial or non-financial interests that could have influenced the research reported in this paper.
This study does not involve human participants, human subjects data, or experiments requiring ethical approval. All analyses are conducted using publicly available, anonymized transaction-level data.
The Elliptic Bitcoin transaction dataset is publicly available.
All code used to generate the results in this paper is available from the corresponding author upon reasonable request. Replication scripts are designed to reproduce all tables and figures reported in the paper.