Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
144,951 characters · 25 sections · 91 citation commands
Visual Polarization Measurement Using Counterfactual Image Generation
\thispagestyle{empty}
Political polarization has emerged as a central issue in American politics over the past few decades. Three out of ten Americans now consider polarization one of the most significant challenges facing the country skelley_fuong_2022. When asked to describe the current political climate, the term “divisive” was the most frequent response, and similar terms like “polarized” and “partisan” were among the most common responses by Americans pew2023. As individuals become more deeply rooted in their political identities, the potential for cross-party dialogue, compromise, and effective governance diminishes. Indeed, the impact of polarization extends beyond politics; it affects public policy, social cohesion, and the overall functioning of democratic institutions. Thus, it is important to have precise and systematic measures to study polarization.
Several studies have focused on the verbal content in news articles as a rich source of information to propose measures for media bias and polarization and examine the demand-driven motives for news outlets to create content that matches the partisan preferences of their readers gentzkow2006media, gentzkow2010drives. The research in this domain suggests that the choice of words and framing of issues can reveal accurate information about the speaker or the writer gentzkow2019measuring. The existence of ideological slanting in the choice of words and framing poses important questions about the non-verbal aspect of news content. The non-verbal content conveys meaning through channels other than language, such as facial expression and body language. In news articles, the non-verbal content is often communicated through visual content such as images. Editors often pay great attention to the choice of visual content, as visuals are more memorable, processed more rapidly, and elicit stronger emotional responses than text sullivan1988happy, TownsendKahn2014, blanchard2023extraction.\footnote{A large body of work in visual marketing and eye-tracking data demonstrates the value of visual information wedel_pieters_2007, chandon2009does, wedel2023modeling.} Visuals not only shape engagement but also play a key role in spreading misinformation matatov2022stop and influencing voter perceptions of competence and trustworthiness in politics hoegg2011impact. Moreover, younger generations of consumers are showing a growing preference for news content that relies less on verbal and more on visual elements. Nevertheless, only a few studies have focused on the visual content to study polarization peng2018same, boxell2021slanted, ash2021visual, caprini2023visual, and no prior work has utilized the richness and dimensionality of visual information.
In this paper, we bridge this gap, and develop a framework to measure polarization and slant in visual content. Figure (ref) shows a small sample of images used by {\it CNN} and {\it Fox News} to portray the 2024 presidential nominees, Donald Trump and Kamala Harris. Visually, we can see a more positive portrayal of Donald Trump (Kamala Harris) by {\it Fox News} ({\it CNN}). Our goal in this paper is to propose a method to systematically quantify this form of visual slanting. In particular, we seek to answer the following questions:
There are three key challenges that we need to address to answer these questions satisfactorily. First, we need a formal definition of a metric or parameter of interest that captures polarization in visual content. To address this challenge, we turn to the very structure of the problem: the editor's choice of visual content. An ideologically slanted choice of image intuitively means that a news outlet prefers a more positive (negative) portrayal of a politician from the same (opposite) ideological side, compared to a neutral news outlet. We construct a flexible utility framework that models the editor's choice of images from a set of available options and focus on the smile as a focal feature we use to measure visual polarization given the consistent finding that a smile leads to a more positive portrayal of a politician sulflow2019power. Using a utility framework allows us to compare the utility derived from two images that are on all aspects except the focal feature (e.g., smile) on which the visual polarization is measured. This comparison enables us to build a polarization measure centered around the focal feature of interest. For instance, consider two images of Donald Trump that differ only in whether he is smiling. For each outlet, we can define the difference in the utility from using the image with and without a smile. Intuitively, the greater the variability in this utility difference across outlets, the higher the level of political polarization in visual content. To capture this notion of variability for a given pair of news outlets, we define the difference in this utility difference as the visual polarization measure between any pair of outlets. When one news outlet in the pair is a neutral outlet, the visual polarization measure reflects how slanted the image choice is by the other outlet, allowing us to characterize a visual slant measure. Together, we develop a utility framework that allows us characterize both visual slant and visual polarization.
Our second challenge stems from the richness and high dimensionality of the visual content. The common approach in the literature is to use an off-the-shelf machine learning model that extracts a certain feature (e.g., smile in an image) from the image peng2018same, boxell2021slanted. However, this feature extraction approach relies solely on the selected feature and disregards other information in the image, which can introduce both omitted variable bias and extraction bias in the analysis wei2022unstructured. We address this challenge by employing a generative approach that creates counterfactual versions of the same image with and without the feature of interest. This method enables us to retain all the information in the image and minimizes the degree to which other factors can confound our polarization measure.
The third challenge lies in identifying our polarization parameter from the observed data, which is defined based on the utility functions of news outlets. This challenge arises because these utility functions are not directly identified from the data as we only observe the image selected by each outlet, but not the full choice set available to them. To address this issue, we leverage the variation in image choices across outlets for similar events (e.g., a specific press hearing), under the assumption that the choice set is nearly identical for all outlets covering the same event. For instance, for a similar event, both {\it Fox News} and {\it CNN} likely have access to almost identical sets of images sourced from common providers such as Getty Images or Associated Press, which allows us to assess whether one outlet derives greater utility from a positive portrayal of a politician compared to the other. We then theoretically link the identification of the polarization parameter to a news outlet prediction problem and develop a multi-modal deep learning model for this task. The model is designed to capture both the clustering structure in similar events and the subtle facial features that may reflect outlets' ideological preferences (if any). Finally, the estimates derived from this prediction task allow us to measure polarization at any desired level of granularity.
Together, we build a unified framework that combines the economic structure of the problem with generative models to generate comparable counterfactual images and measure polarization in visual content. The key advantage of our framework in comparison to the traditional regression-based approaches lies in its ability to overcome the potential confounding and mis-estimation of polarization due to information loss from feature extraction. Additionally, our generative strategy extends beyond the study of polarization and can be applied to other contexts involving visual data. Another key benefit of our framework is its ability to provide individual-level measures, enabling us to quantify the heterogeneity in visual polarization across politicians and news outlets.
We apply our framework to a comprehensive dataset comprising over 60,000 images of 30 prominent politicians from both the Republican and Democratic parties across 20 major news outlets over a 10-year span from 2011 to 2021. First, we employ a multi-modal deep learning model to predict the outlet of an image based on its visual information, the textual content of the article, and contextual data about the politician, year, and other relevant factors. We then employ Generative Adversarial Networks (GANs) to create sets of counterfactual images for all politicians with and without a smile. Using the model estimated in the initial step, we measure how the predicted probability of the image belonging to a particular outlet changes between these counterfactual images. Finally, we connect these differences to our visual slant measure and quantify the extent of polarization at both the aggregate and individual levels.
Our results suggest that both Democratic- and Republican-leaning outlets ideologically slant their visual content. Using Reuters as the base neutral news outlet, we find that compared to a neutral image of a Republican politician, a smiling image increases utility for a Republican-leaning news outlet and decreases utility for a Democratic-leaning outlet. Conversely, for Democratic politicians, a smiling image generates lower utility for Republican-leaning outlets and higher utility for Democratic-leaning outlets compared to the neutral image. These results suggest that news outlets exhibit a positive visual slant when covering politicians who share their ideological leanings and a negative visual slant when covering those with opposing views. We perform formal statistical tests and show that the distributions of visual slant are significantly different, highlighting the divide in their portrayal of politicians from either side.
We then examine the heterogeneity of visual slant across news outlets, documenting substantial variation in how politicians are portrayed. For Democratic politicians, USA Today, The New York Times, and The Washington Post exhibit the highest positive visual slant in their favor, while Daily Mail and Fox News display the strongest negative slant. For Republican politicians, the most positive visual slant scores appear in Daily Mail and \textit{Newsmax}, whereas \textit{The Washington Post} and \textit{CNN} show the most negative slant. To quantify overall outlet-specific visual slant, we introduce a new measure, \textit{Conservative Visual Slant (CVS)}, which captures the degree to which an outlet's visual content favors Republican politicians while disadvantaging Democratic ones. {\it CVS} measure calculates the difference between an outlet's visual slant for Republican and Democratic politicians. Based on this measure, we identify \textit{Daily Mail}, \textit{Fox News}, and \textit{Newsmax} as outlets with the highest \textit{CVS} values, while \textit{The Washington Post}, \textit{USA Today}, and \textit{The New York Times} as the outlets the most negative \textit{CVS} values.
We also document significant variation in the degree of polarization across individual politicians. To capture this, we introduce a metric called {\it Overall Visual Polarization (OVP)}, which quantifies the standard deviation of a politician’s visual slant measures across media outlets. A higher {\it OVP} indicates greater dispersion in how different outlets visually present a politician, suggesting a more polarizing figure. On the Republican side, Donald Trump emerges as the most polarizing figure. That is, the gap in the extent of ideological slanting is remarkably large, with the Republican-leaning outlets receiving higher utility from using a positive and smiley portrayal of him compared to Democratic-leaning outlets. On the Democratic side, we find Barack Obama, Bernie Sanders, and Kamala Harris to be among the most polarizing figures who are portrayed very differently in Democratic-leaning and Republican-leaning outlets. Interestingly, Joe Manchin (D) and Susan Collins (R) rank among the least polarizing politicians, reflecting their reputations as moderates within their respective parties. We also observe low polarization levels for Liz Cheney, whose fallout with Donald Trump resulted in increased favorability among liberal outlets and decreased favorability among conservative outlets, ultimately contributing to reduced levels of visual polarization.
Lastly, we conduct a series of tests to validate the measures derived from our algorithm. First, we compare our {\it Conservative Visual Slant (CVS)} measure with existing measures of media slant from the prior literature flaxman2016filter, faris2017partisanship using a series of correlation tests. Second, we compare the performance of our {\it CVS} measure against the visual slant metric proposed by boxell2021slanted and demonstrate that our measure is more effective at capturing variations in external media slant indicators. Finally, we validate our {\it Overall Visual Polarization (OVP)} measure by showing a strong correlation between {\it OVP} scores and the ideological alignment of a politician’s constituency. Together, these validation tests underscore our algorithm’s ability to capture media polarization using images used by news outlets.
In summary, our paper makes several contributions to the literature. Methodologically, we propose a framework that combines economic theory with generative models to provide robust measures of media bias and polarization in visual content. A key innovation of our framework is the use of Generative Adversarial Networks (GANs) in a way consistent with experimentation to generate counterfactual images based on a feature of interest, addressing the bias due to information loss present in social science studies that utilize image data. As such, our framework is general and applicable to all settings where researchers seek to quantify polarization on a given feature across a given set of images and outlets. Substantively, our work demonstrates the existence of ideological slanting in the visual content used by media outlets, with substantial heterogeneity observed across news outlets and politicians.
Our paper relates to the study of political polarization in news media, a phenomenon well documented using text data. groseclose2005measure quantifies media bias by examining the frequency with which different media outlets cite various think tanks, uncovering a persistent liberal bias. Similarly, gentzkow2010drives investigate the factors driving media slant, highlighting the significant roles of consumer preferences and political affiliations by analyzing the alignment of newspaper language with political parties. jensen2012political study the polarization of political discourse by analyzing records of Congressional speech and the Google Ngrams corpus, discovering a notable increase in discourse polarization since the late 1990s. Furthermore, gentzkow2019measuring focus on the linguistic divide in Congressional speeches, showing how Democrats and Republicans increasingly use distinct vocabularies. While these studies predominantly focus on textual data, our work shifts the focus to polarization in visual content by examining how images in news articles contribute to this phenomenon.
More recently, research in this area has expanded from textual to visual content due to advances in computer vision techniques. peng2018same use a two-stage approach with Azure Microsoft to extract facial expressions from 13,000 images of the 2016 election in the first stage and then analyze them through a regression model in the second stage, revealing bias toward politicians aligned with a news outlet's stance. boxell2021slanted expand this by analyzing 70,000 images across more politicians and outlets. caprini2023visual further extend this by generating and analyzing textual descriptions of images alongside news articles using Azure, applying gentzkow2019measuring to show how the alignment of visual and textual bias amplifies polarization. Our study advances previous research in three significant ways. First, we enhance the traditional two-stage approach by addressing its inherent biases and issues with omitted variables, introducing a more robust methodology called Polarization Measurement Using Counterfactual Image Generation (PMCIG) that uses the rich information in the image content to measure the polarization in visual content. Second, our method delivers results that are both precise and available at a lower level of granularity, enabling us to quantify polarization at both the news outlet and politician levels and to track their evolution over the past decade. Third, we leverage a long term dataset spanning from 2011 to 2021, which allows us to capture long-term trends in political coverage that earlier studies may have missed.
From a methodological perspective, our work aligns with the growing trend in social science research that leverages computer vision techniques to analyze unstructured image data. One stream of work uses image data to generate descriptive insights on consumers and firms using novel measurement techniques dew2022letting, liu2020visual. Another stream of work employs a two-stage approach and tries to connect image features to economic/marketing outcomes of interest by first, extracting features from images to create structured data and then applying statistical analysis to these features davenport2017analytics. However, as discussed earlier, this approach has limitations, including potential information loss, endogeneity issues, and oversimplification due to parametric assumptions. Recently, researchers have started paying attention to this problem and proposing potential solutions. wei2022unstructured identify biases in econometric models using machine-learned variables from unstructured data and propose solutions to improve accuracy. singh2023causal introduce the RieszIV estimator, which incorporates high-dimensional unstructured data directly into causal analysis to manage endogeneity. xu2024unstructured propose a debiased embedding framework that integrates representation learning with causal inference, addressing biases inherent in traditional embedding-then-inference frameworks. Additionally, luo2024using employ GANs for Controllable Stimuli Generation (CSG), enabling precise manipulation of image attributes to isolate causal effects. Similarly, li2024product advances AI-driven product design by integrating consumer preferences from internal data and external user-generated content into a generative framework, addressing the limitations of traditional GAN-based approaches. Building on these advancements, our PMCIG method combines GAN-based image manipulation with non-parametric deep learning models to more accurately quantify polarization in an image feature, holding all other features constant.
We collect publicly available data on images of politicians from media websites spanning a 10-year period for the study. Below, we describe our data collection, cleaning, and labeling strategy.
We collect data on 30 politicians and 20 news outlets over a span of ten years, from 2011 to 2021. The set of politicians consists of those who ran for important public offices and/or held important national roles during the ten-year span of 2011--2021 and were commonly searched on Google (based on their popularity on Google searches using Google Trends data). The news outlets used in our study consist of the list of popular outlets that have been used in earlier studies on media polarization; see flaxman2016filter for details. We refer readers to Web Appendix $\S$(ref) for a full list of politicians and outlets.
The data collection is done using the SerpAPI application serpapi. SerpAPI facilitates efficient large-scale image scraping by leveraging Google's image search to extract relevant images and metadata. The process involves generating search queries based on specific criteria, such as targeted news outlets and date ranges, to extract images, article links, titles, and dates.\footnote{For each politician-outlet combination, we use a two-year rolling window that advances by one year at a time, resulting in 10 queries spanning a total of 10 years.} This approach was chosen because it provides a structured, efficient, and reliable interface for large-scale data collection, automating the process while ensuring compliance with Google's data access policies.
Figure (ref) presents an example of a query we use for data collection, where we use “site:" to restrict searches to specific news websites, “before:" and “after:" to filter results by publication date, and exact phrase searches to capture relevant content precisely. For each query, we aim to collect 80 news articles for each politician from each news outlet for the specified period.\footnote{Some queries retrieve fewer articles/images for certain politicians-outlet combinations because the outlet may not have published 80 images for that specific politician.} This information for each image in the query is compiled into a structured data frame with the following fields:
Overall, our data collection strategy gives us a comprehensive data set of 287,275 images for 30 politicians from 20 news outlets.
A crucial step is cleaning the data to ensure that each image contains the face of the politician referred to in the query. We face the following challenges in the data-cleaning step:
To address these challenges, we design a two-phase computer vision framework. We provide a brief overview of this framework here and refer readers to Web Appendix $\S$(ref) for the technical details. In the first phase, we use a series of computer vision models to keep only images with one face presented in them. In the second phase, we build and deploy a face-verification tool to ensure that the face in an image belongs to the intended politician. First, we manually select 20 high-quality, single-face images for each of the 30 politicians. These selected images serve as true labels identifying the correct politician, ensuring a reliable foundation for training the face verification model. Next, we use the trained verification model for each instance in our one-face sample obtained from the first phase to verify that the predicted label for the face is the same as the intended politician. After applying the above data cleaning procedure, we are left with a set of 63,188 images, where each image shows a single face that belongs to the intended politician. Web Appendix $\S$(ref) provides further details on our two-step model and includes a comprehensive table (Figure (ref)) listing the number of images for each politician-outlet combination after the cleaning.
Recall that our objective is to see if and how images of politicians in media outlets can be used to measure political polarization. As such, our goal here is to develop measures of {\it visual slant} and {\it visual polarization} that can be estimated from data on news articles (with images of politicians).
Consider a dataset of news articles, $\mathcal{D}$. Each news article $i$ in this dataset is characterized by a tuple $(X_i, P_i, Y_i, Z_i)$, where $X_i$ represents the features associated with the key aspects of the article, such as its title text, topic, and publication date. $P_i$ refers to the politician who is the focal subject of the article, $Y_i$ is the news outlet that produces the article, and $Z_i$ is the image. $Z_i$ can be interpreted as detailed pixel-level information capturing all the relevant aspects of the image, such as its background, brightness, other objects present, and the facial expression of the politician $P_i$.
Next, we define the image choice problem for a given article $i$ from the perspective of a news editor. Prior research has shown that images can have a significant impact on the extent to which readers engage with and click on news content MatiasEtAl2021. As such, editors must carefully select images for each article, which involves choosing an image that appeals to the target audience and aligns with the editorial stance while ensuring that the visual elements complement the textual content jakesch2022belief. Figure (ref) shows an example news article, highlighting the decision stage where an editor picks an accompanying image for the given article.
Formally, let the editor or news outlet receive utility $U_i$ from producing article $i$ with characteristics $(X_i,P_i,Y_i,Z_i)$, which is defined as follows:
where $u(Z_i, X_i, P_i, Y_i)$ is the deterministic component and represents the editor's expected utility from producing the article, and $\xi_i$ denotes the idiosyncratic error term, which follows an i.i.d. Type 1 Extreme Value distribution.\footnote{We model the news outlet's decision-making using a utility function rather than a conventional profit function, as ideological preferences may drive them to deviate from profit-maximizing choices. For other theoretical models that micro-found agents' decision-making to examine polarization in equilibrium, see gentzkow2010drives, iyer_yoganarasimhan_2021, amaldoss2021media and bondi2023privacy.} This utility function captures all the main considerations of the editor when making image choices, such as how well the image supports the outlet's ideological stance (alignment with editorial policy, i.e., alignment of features of $Z_i$ with $Y_i$), the image's ability to attract and retain readers (reader-engagement), and the aesthetic quality and/or emotional impact of the image (visual appeal).
Intuitively, ideological slanting in images means that the outlet receives a higher (lower) utility from a more positive visual portrayal of a politician from the same (opposite) ideological side compared to a neutral outlet. The key challenge in developing a measure of ideological slant in visual content stems from the ambiguity around the definition of a “positive” portrayal: an image is an unstructured and high-dimensional object, and there are presumably numerous ways for the outlet to choose a more or less positive image. As such, we need to define ideological slanting for a certain image feature. In our analysis, we focus on the presence of smile as the feature of interest because there is consensus that smile is a feature that contributes to a more positive portrayal sulflow2019power. However, our framework is not restricted to this particular feature and can easily extend to other features or sets of features.
Let $T_i$ denote whether the subject in image $Z_i$ is smiling or not. To define the ideological slant measure for feature $T$, we start with a utility difference measure that compares the utility from two articles that are identical in all respects, except for the smile feature. Similar to the causal inference literature, we therefore consider two versions of an image $Z$: $Z(T=1)$ and $Z(T=0)$, where $Z(T=1)$ is the image with a smile and $Z(T=0)$ is the same image without a smile (with a neutral expression). For ease of exposition, let $Z^{(-T)}$ denote all the information in the image $Z$, except the smile feature $T$. This implies that $Z(T=1) \equiv (T=1, Z^{(-T)})$ and $Z(T=0) \equiv (T=0, Z^{(-T)})$, i.e., the two versions of the image are identical in all features other than smile \footnote{The notation assumes that $Z^{(-T)}$ is independent of $T$, meaning all non-smile features remain identical across conditions. This holds if the images are constructed such that only the smile varies, ensuring $Z^{(-T)} \mid T=1 \sim Z^{(-T)} \mid T=0$.}. Using this set of two images, we can define a measure of utility difference for a given news outlet $y$ and politician $p$ with respect to visual feature $T$ as follows:
This utility difference helps isolate the utility increase or decrease for a news outlet that solely comes from the presence of a smile in the image of any given politician. However, this is still not a measure of ideological slant in visual content because we cannot fully attribute this utility difference to ideological preferences. For instance, there can be vertical preferences for the smile feature where all news outlets prefer a smiling image of a politician. As such, the difference in the utility difference $\Delta^T u (p,y)$ across two news outlets helps cancel out the vertical preference for the feature and the remaining difference can be linked to ideological preferences. For example, we expect a news outlet with a higher conservative audience share (e.g., {\it Fox News}) to have a higher $\Delta^T u (p,y)$ for a conservative politician (e.g., Donald Trump) than a news outlet with lower conservative audience share (e.g., {\it CNN}) even if they both have a vertical preference for smiling images. Since this difference is defined for any two news outlets, it measures visual polarization of the two outlets with respect to the feature of interest. However, it is important to note that this is not a measure of visual slant, because both outlets may have slanted preferences. In what follows, we present formal definitions for both visual polarization and visual slant. We first present the formal definition for visual polarization as follows:
This definition of {\it visual polarization}, $\rho^T (p,y_1,y_2)$, is flexible and allows for detailed analysis at the level of individual politicians and news outlets. For example, by examining the above measure for a specific politician like Donald Trump across specific outlets such as {\it CNN} and {\it Fox News}, we can measure the extent to which these outlets are polarized or differentiated in their visual portrayal of Trump. Naturally, for the measure of visual polarization to capture visual slant, we need the news outlet $y_2$ to be neutral. Let $y_n$ denote the neutral news outlet. We can formally define visual slant as follows:
To further clarify the difference between visual polarization and visual slant, we return to the example with the portrayal of Donald Trump in {\it CNN} and {\it Fox News}, but add a neutral news outlet like Reuters. Suppose that the visual polarization between {\it Fox News} and {\it CNN} for Donald Trump is equal to one, i.e., $\rho^T (\text{Donald Trump},\text{Fox News},\text{CNN})=1$. This measure is a composite of both Fox News' preference for a positive portrayal of Trump and {\it CNN}'s preference for a negative portrayal of him. Our visual slant measure helps decompose the visual polarization measure. For example, we may find that there is a positive visual slant for Donald Trump at {\it Fox News} such that $\rho^T_s (\text{Donald Trump},\text{Fox News})=0.4$, but a negative \textit{visual slant} for him at {\it CNN} such that $\rho^T_s (\text{Donald Trump},\text{CNN})=-0.6$. It is easy to verify the following relationship between the two definitions:
Lastly, a notable feature of our measures is that they are defined at a high degree of granularity, which allows us to capture the varying degrees of ideological slant in visual content specific to each outlet or each politician. This specification allows us to consider different types of aggregation:
In this section, we discuss the standard reduced-form approach used to measure polarization in visual content. In $\S$(ref), we describe the two-step reduced-form approach and connect the estimated parameter under this approach to the measure of polarization in visual content as defined in Equation (ref)). Next, in $\S$(ref), we provide a theoretical and conceptual discussion of the drawbacks of this approach.
Recall that our goal is to measure {\it visual polarization}, \( \rho^T (p,y_1,y_2) \), with respect to a focal feature $T$, such as whether the politician in the image is smiling or not. However, this is inherently challenging when dealing with unstructured image data, because unlike the standard causal inference literature, where treatment is observed imbens_rubin_2015, here we do not directly observe $T_i$, i.e., the presence or absence of the treatment (smile in this case) in a given article $i$. As a result, a common approach in social science settings is to use a {\it Two-step Approach}. This approach addresses the high dimensionality and complexity of unstructured data by first extracting meaningful features and then using these features in a structured econometric model. See davenport2017analytics for a general discussion of this approach in the broader social sciences literature, and boxell2021slanted and peng2018same for applications of this approach to the context of image polarization in media.\footnote{Beyond the political polarization context, this two-step approach is now commonly used in marketing research involving images in other settings as well. For instance, unstructured data such as video and audio streams (\( Z \)) are analyzed to extract features like facial expressions (\( T \)), which are then used to study their impact on user engagement (\( Y \)) lu2021larger. In the online labor market, profile pictures (\( Z \)) are examined to identify features such as perceived race or attire (\( T \)) to assess job offer likelihood (\( Y \)), highlighting visual biases in employment opportunities troncoso2022look. Additionally, research on social media posts about e-cigarettes (\( Z \)) extracts demographic features (\( T \)) to study tax policy compliance (\( Y \)), revealing demographic responses to legislative changes anand2024frontiers.} Methodologically, the two-step approach can be outlined as follows:
We now describe how this approach can be used to quantify polarization in visual content with respect to feature \(T\), defined as \(\rho^T (p,y_1,y_2)\) in Equation (ref). Consider a simple example where news outlets from Democratic-leaning outlets like the {\it The New York Times}, {\it CNN}, and {\it BBC} (\(y_1\)) and Republican-leaning outlets like {\it Fox News}, {\it Newsmax}, and {\it Daily Mail} (\(y_2\)) choose images for a politician, say Hillary Clinton (\(p\)). In this context, the first step involves estimating the binary indicator \(\hat{T}\), which denotes whether the focal politician (Hillary Clinton) is smiling in the image (\(\hat{T} = 1\)) or not (\(\hat{T} = 0\)). This can be obtained from standard off-the-shelf emotion recognition models such as Face++ faceplusplus_emotion, Google Vision google_ml_kit\footnote{microsoft_azure_vision’s Face API, a popular model previously used in studies such as boxell2021slanted, caprini2023visual, has been unavailable for use since June 2022 due to updated responsible AI policies microsoft2023responsible.}, or through in-house model training where we first label a sample of the images using human subjects and then train a model to predict the label given the image.
For the second step, we can use a logistic regression where we regress \(\hat{T}_i\) on characteristics \(X_i\) and \(Y_i\). For notational simplicity, consider the case where \(Y\) is a binary variable (e.g., \(Y=1\) for Democratic-leaning outlets, and \(Y=0\) for Republican-leaning outlets). We can write the log odds ratio for the resulting logistic regression as follows:
If the model in the equation above is well-specified, the log odds ratio characterizes the utility difference \(\Delta^T u(p, y)\) between choosing an image with a smile versus one without a smile, as defined in Equation (ref). In that event, we can connect the parameter \(\beta\) to the polarization measure defined in Equation (ref) as follows:
Thus, the coefficient \( \beta \) in the logistic regression model serves as an empirical estimate of the polarization measurement \( \rho^T(p, Y=1, Y=0) \). If \( \beta > 0 \), it suggests that Democratic-leaning outlets (\(Y = 1\)) derive a higher utility from using a smiling image of Hillary Clinton compared to Republican-leaning outlets, implying a greater \(\Delta^T u\) for the Democratic-leaning outlets. Conversely, if \( \beta < 0 \), Republican-leaning outlets (\(Y = 0\)) have a higher utility difference compared to Democratic-leaning outlets, indicating a greater \(\Delta^T u\). Therefore, \( \rho^T(p, Y=1, Y=0) \) can be approximated by \( \beta \), and provides a quantitative measure of the ideological polarization across news outlets.
While the two-step model described in $\S$(ref) is easy to interpret and apply, it relies on the assumption that the regression model is well-specified. However, this assumption can fail due to two natural reasons and lead to incorrect inference: (1) extraction bias, and (2) omitted variable bias. In this section, we discuss these biases and explain how they can manifest in our setting.
{\it Extraction bias} occurs due to imperfections in the feature extraction process, or the first step of the two-step process wei2022unstructured. That is, the true label $T$ may differ from the extracted label \(\hat{T} = f_1(Z)\) as follows:
The error in Equation (ref) can be viewed as a measurement error. It is well-known that systematic dependence of this error on other relevant factors in the model can introduce biases or inconsistencies in the estimates; see Chapter 4 of wooldridge2010econometrics. As such, we decompose this error into two parts, such that $\epsilon_1 = \epsilon_r + \epsilon_e$, where $\epsilon_r$ is the random i.i.d. noise in the measurement process, and $\epsilon_e$ is the extraction error that can be correlated with the other relevant image features contained in \(Z^{(-T)}\). For instance, a machine learning model designed to detect the facial expressions (e.g., whether a person is smiling) may also inadvertently capture brightness levels in the image, as brighter images are often associated with happier expressions. We can rewrite the relationship between $T$ and $\hat{T}$ as:
Then, the predicted feature $\hat{T}$ is a combination of the true feature of interest $T$ (e.g., smile) and a correlated nuisance feature $\epsilon_e$ (e.g., brightness). Thus, when we perform the second stage estimation, the estimated effect ($\beta$) captures the effect of the outlet on the biased measurement $\hat{T}$ rather than the true feature $T$.
Next, omitted variable bias arises from relevant factors omitted from the second step. This issue can arise even if we have the true labels $T$. To illustrate this point, consider the second-step relationship between the true label and covariates:
where $\epsilon_2$ consists of unobserved variables that are not captured by the observed covariates $Y$ and $X$ through the semi-parametric function $f_2$. If this error term is independent of the covariates, we can identify function $f_2$ correctly. However, this is often a very strong assumption given the amount of information in \(Z^{(-T)}\) that is omitted from the model. For example, consider a collection of images by Donald Trump in articles by {\it CNN} and {\it Fox News}, where the smile labels are accurate, i.e., $\hat{T} = T$. Now, consider an image feature, such as the presence of the US flag. It is likely that the US flag is present more often in images from {\it Fox News}, given their higher share of nationalist viewers. At the same time, it is more likely for a politician to smile in formal settings with flags present in the background. In this scenario, estimating the parameters of Equation (ref) will lead to function $f_2$ picking up the association between the presence of the flag and the smile in the image because the presence of the flag is omitted from the model. Given the high-dimensional and unstructured nature of the images, numerous other image features can result in omitted variable bias in our estimates. To characterize the endogenous part of the error term $\epsilon_2$, we rewrite Equation (ref) as follows:
where $\epsilon_0$ is the part that is independent of covariates, and $\epsilon_{ov}$ is the part that is correlated with $Y$ or $X$, which can lead to omitted variable bias. It is worth noting that image features that lead to extraction bias can also lead to omitted variable bias. For example, omitting image features like brightness can lead to omitted variable bias if brightness is correlated with both the actual presence of a smile in the image (not the extracted one) and the news outlet.
Together, if we want to estimate the second step of the two-step model in $\S$(ref) by using $\hat{T}$ as the outcome, the errors in Equation (ref) also appear in the second step equation as follows:
where $\epsilon_{ov}$ leads to omitted variable bias, $\epsilon_e$ leads to extraction bias due to the use of a machine learning model in the first step, and $\epsilon_r$ contributes to higher uncertainty in model estimates. In summary, the information loss resulting from extracting a single feature from an image can introduce both extraction bias and omitted variable bias.
We present a more formal characterization of these biases and their derivations in Web Appendix $\S$(ref). We also provide empirical evidence of how the two-step approach can be biased in estimating visual polarization in our setting in Web Appendix $\S$(ref).
As discussed in $\S$(ref), the two-step approach is subject to information loss since it ignores the high-dimensional information in the image, which in turn can bias the estimated measure of polarization. Therefore, we develop a novel algorithm -- Polarization Measurement Using Counterfactual Image Generation (PMCIG) -- that recovers the polarization measure defined in $\S$(ref) without incurring information loss.
Recall that the {\it visual polarization} measure is defined as (see Definition (ref)): \[\rho^T(p, y_1, y_2) = \Delta^T u(p,y_1) - \Delta^T u(p,y_2),\] where $\Delta^T u (p,y) = \mathbb{E}_{X}\left[u \left( Z(T=1), X,P,Y \right) - u \left(Z(T=0),X,P,Y \right) \mid P = p, Y = y\right]$. Before outlining our approach to measuring visual polarization, we first present a thought experiment of what an ideal experiment to measure $\rho^T(p, y_1, y_2)$ would look like. In order to measure $\Delta^T u (p,y)$ for a given politician-outlet pair $(p,y)$ for a feature/treatment $T$, we need to observe the difference in the editors' utilities (or choice probabilities) for two images that are exactly the same on all dimensions except $T$. Thus, an ideal experiment to measure $\rho^T(p, y_1, y_2)$ would involve presenting the editors of outlets $y_1$ and $y_2$ two images of politician $p$, that are exactly the same on all dimensions except the feature of interest, i.e., one where the politician is smiling (or the feature of interest $T$ is turned on, i.e., $Z(T=1) \equiv (T=1, Z^{(-T)})$) and one where s/he is not smiling (i.e., $Z(T=0) \equiv (T=0, Z^{(-T)})$). Then, the observed difference in relative choice probabilities of the two images ($Z(T=1)$ and $Z(T=0$)) across the two outlets can be linked to polarization measure $\rho^T(p, y_1, y_2)$.
However, our data comes from an observational setting rather than this ideal experiment. Therefore, we only observe realized combinations of $Z$ and $y$. For example, for an image $Z$ with $T=1$, we may see that it was chosen by outlet $y_1$, as in the case where Biden's smiling image was chosen by {\it CNN} in Figure (ref). However, we do not see what would be the probability of this image being chosen by $y_1$ if everything else about it was held the same, with only the feature/treatment of interest turned off (i.e., a non-smiling version of the same image of Biden with $T=0$). Similarly, we also do not see what would have been the likelihood of outlet $y_2$ choosing this image for the two cases: when $T=1$ and when $T=0$. Thus, we only observe one realized combination of $y$-$T$ for an image $Z$, as shown in Table (ref). However, to quantify/measure polarization, we need to be able to reliably estimate/model the three other counterfactual outcomes for each image $Z$ in the data. This limitation is similar in spirit to the well-established unobservability challenge in the potential outcomes framework angrist2009mostly.
As we can see from Table (ref), there are two key challenges that we need to overcome to measure {\it visual polarization}.
The rest of this section is organized as follows. First, in $\S$(ref), we present the overview of our solution to these two challenges. Next, in $\S$(ref), we present our unified algorithm for quantifying visual polarization.
In this section, we present an overview of our solution to the two challenges described earlier. We first present the generative component in $\S$(ref) (to address Challenge 1). We then discuss our identification strategy in $\S$(ref) (to address Challenge 2).
To address the first challenge, we leverage the recent developments in generative image models that allow us to manipulate images such that we can modify one specific aspect of an image (e.g., feature $T$) while keeping everything else constant. Generative models have been employed in several recent studies on image analysis: athey2022smiles use generative models to adjust features such as smiles in profile images to examine their effects on user preferences and economic transactions in a micro-lending platform. ludwig2024machine utilize these models to manipulate facial features and explore how these alterations affect judicial decisions. Finally, in luo2024using, these models create facial images to assess the impact of perceived gender traits on discrimination within online marketplaces.
We define two operators, \(\pi^1\) and \(\pi^0\), which transform an image \(Z\) into either an image where the treatment/feature of interest is turned on \(Z(T=1)\) or one where it is turned off \(Z(T=0)\). The purpose of these operators is to create a set of control and treatment images that only differ in the feature of interest (e.g., the facial expression or smile).\footnote{We can easily extend this operation to a continuous case, where we manipulate the extent to which the feature is activated, e.g., the extent to which the politician smiles.}
For a set of original images, together, these two operators give us pairs of treated and control images.
{\bf Implementation Details:} We now discuss how we generate counterfactual images in our study. For each politician $p$, we first select three neutral images \( \tilde{Z}^{0}_{p1}, \tilde{Z}^{0}_{p2}, \tilde{Z}^{0}_{p3} \) (where $T=0$) from our dataset to ensure a balanced and unbiased representation of the politician.\footnote{Using three instead of one image ensures that the findings are not driven by the peculiarities of a specific image. In principle, it is possible to use a larger/fewer number of images for this task. We choose three because it balances noise reduction with the cost of generating counterfactual images. Focusing on three images also allows us to manually assess the quality of the generated images and avoid cases where the smile looks unnatural or unappealing.} As such, the function $\pi^0(\cdot)$ to generate the neutral image without the smile feature is an identity function. To generate the counterfactual version of each image with a smile, we employ \href{https://www.ailabtools.com/image-editor/}{AILabTools} as our \(\pi^1\) operator; this tool utilizes conditional Generative Adversarial Networks (cGANs) mirza2014conditional and produces high quality and realistic outputs.\footnote{While many such tools are available, we use AILabTools because it allows us to precisely modify the focal images (by adding a smile) while maintaining the authenticity of the other features in the original image.} Thus, for each neutral image, we generate corresponding smiling versions \( \tilde{Z}^{1}_{p1}, \tilde{Z}^{1}_{p2}, \tilde{Z}^{1}_{p3} \), where the only difference is the added attribute (smile). Figure (ref) illustrates how we apply $\pi^1$ to three neutral images of Donald Trump, which in turn gives us three corresponding neutral images of Donald Trump with a smile. We repeat this process for all the 30 politicians in our study to construct the set $\{ (\tilde{Z}^{0}_{p1}, \tilde{Z}^{0}_{p2}, \tilde{Z}^{0}_{p3} ,\tilde{Z}^{1}_{p1}, \tilde{Z}^{1}_{p2}, \tilde{Z}^{1}_{p3}) \}_p$.
We begin with a high-level overview of our identification strategy before formalizing our approach. The core challenge in this task lies in the fact that while we observe the image selected by a news outlet, we do not observe the full set of available choices. Our identification strategy leverages variation from similar events covered by multiple outlets (e.g., press conferences). We argue that in these cases, outlets have access to the same choice set, making the identification task analogous to a news outlet prediction problem: the selected image is the one that provides the highest utility to the outlet compared to all other similar images chosen by other outlets.
Recall that our measure is defined for a politician $p$ and any pair of news outlets $y_1$ and $y_2$. Consider a pair of images $\{ Z^{0}_{p}, Z^{1}_{p}\}$ for politician $p$, where $Z^{0}_{p}$ is the version of the image $Z$ with the feature $T$ turned off and $Z^{1}_{p}$ is the version with the feature $T$ turned on. Then, we can write the sample analogue estimator for the polarization parameter for this pair of images as follows:
where $\mathcal{X}_p$ is the set of all articles featuring politician $p$, so the sample analog estimator takes the average over all these articles featuring politician $p$. Clearly, if the utility function is known, we can directly estimate $\hat{\rho}(p,y_1,y_2)$ by integrating the above equation over all the articles. However, as highlighted in Challenge 2, the underlying utility function is not identified with our data because we do not observe the editors' choice sets of images.
We propose a solution to this identification challenge that directly links the {\it visual polarization} parameter to an outlet-prediction model that can be identified from the data given without observing/identifying utility functions. We now introduce some additional notation to help with this task. In a setting with two outlets $y_1$ and $y_2$, we define $\tilde{Y}$ as a pseudo-variable that denotes the news outlet with he highest utility for producing an article with the content $x$, politician $p$, and image $z$. Formally, we can define $\tilde{Y}$ as follows:
where $\xi_1$ and $\xi_2$ are independent and identically distributed terms that come from the Type 1 Extreme Value distribution. We can link the elements of the right-hand side (RHS) of Equation (ref) to pieces identifiable from the data at hand by re-writing the sample analog estimator for {\it visual polarization} as:
In deriving Equation (ref), we switch two elements (second line), apply $\log(\exp(\cdot))$ transformation (third line), and use a log-odds interpretation that connects the polarization parameter to a news outlet choice problem. As such, Equation (ref) allows us to define the polarization parameter as the difference between two log-odds ratios related to the pseudo-variable $\tilde{Y}$.
We now link the identification of the polarization parameter to an outlet prediction problem (given article and image features). This link depends on the link between the pseudo-variable $\tilde{Y}$ and the actual $Y$ variable that determines the news outlet for an article. The following proposition characterizes the assumption needed for our identification:
The proof of this proposition is simple: under the assumption that the outlet producing an article $i$ has the highest utility from it (compared to other outlets), we have $\tilde{Y} = Y$. Therefore, the identification of log-odds in Equation (ref) becomes equivalent to log-odds for variable $Y$. Let $\hat{g}(y \mid z,x,p)$ denote the probability of an article with image $z$, characteristics $x$, and politician $p$ is produced by news outlet $y$. If the condition in Proposition (ref) is satisfied, we can write {\it visual polarization} as follows:
As such, the critical question is where in our data the condition in Proposition (ref) is satisfied, that is, the production of an article by an outlet implies having the highest utility from producing it compared to other outlets. Such areas in the data satisfy $\tilde{Y} = Y$ and characterize our identifying variation. For that purpose, we focus on important events that are covered by all news outlets, such as important events like press conferences by the politician. For instance, consider different images of Donald Trump during the “Operation Warp Speed” speech in our dataset, as shown in Figure (ref). Despite being captured from the same event, these images present Trump in varying ways, ranging from serious to expressive. We argue that in such events, our assumption that the producing outlet has the highest utility is more reasonable because all outlets have access to the same set of images. Therefore, the news outlet prediction from the set \( \mathcal{Y} \) can potentially mimic the true choice model over the set of images \(\mathcal{Z}\). Later in $\S$(ref), we present results highlighting how our prediction model utilizes this identifying variation in the data.
A notable feature of our data is the presence of such similar patterns that form a clustering structure, where we observe extensive choice variation by outlets within a cluster of very similar events. In what follows, we discuss how we design the structure of our news outlet prediction model such that it will be able to fully exploit the variation in similar events.
{\bf Model Architecture for the Multi-Modal News Outlet Prediction Problem:} We now discuss the estimation of our news outlet prediction model, \( g(y_i \mid Z_i, \boldsymbol{X}_i, P_i; \theta) \). Given the multi-modal nature of inputs (e.g., text, image, categorical variables), we can use any flexible semi-parametric model that can capture complex patterns in the data. However, there are important challenges that we need to address. First, we need to ensure that the model utilizes the variation in similar events to satisfy the condition in Proposition (ref). Second, we need to ensure that our predictive model correctly identifies the link between the use of smile in an image and the outlet. For example, a purely loss-minimizing objective may end up estimating a model that underestimates the strength of the link between smile and the outlet by misattributing its link to features correlated with a smile. Therefore, instead of simply using a naive off-the-shelf prediction model, we impose some structure on the architecture of machine learning models we consider, as illustrated in Figure (ref).
The multi-modal deep learning model has the following architecture:
The final classification layer combines all modalities. The model is trained using the AdamW optimizer loshchilov2017decoupled, which ensures efficient optimization and robust regularization. To train the model, we maximize the following entropy function over the training data:
The resulting model $\hat{g}$ is then used to estimate the polarization parameter using Equation (ref). Please see Web Appendix $\S$(ref) for additional details on implementation and model training.
We now present our algorithm, Polarization Measurement Using Counterfactual Image Generation (PMCIG) in Algorithm (ref). We split our data $\mathcal{D}$ into training and test sets, denoted by $\mathcal{D}_{\text{train}}$ and $\mathcal{D}_{\text{test}}$, respectively. We use the training data to train the deep learning model, and use the estimated model to calculate the polarization measure on the test data. Since our main goal is polarization measurement, this form of cross-fitting helps avoid overfitting bias. The algorithm consists of three steps as described below:
In our empirical application, we use Reuters as the baseline news outlet given its reputation to be a non-partisan news source. However, we can easily change that baseline to any other outlet to obtain polarization measures. To the extent that Reuters is the neutral point zero on the spectrum, we can interpret our polarization estimates as measures of visual slant.
We now present the results from our empirical analysis. First, in $\S$(ref), we provide evidence demonstrating that the counterfactual images generated in the Step 1 of our algorithm are consistent with the original images, preserving all features except the added smile. Next, $\S$(ref), we present results on the performance of our news outlet prediction model from Step 2 of our algorithm. Finally, in $\S$(ref), we share the Step 3 results on estimates of political polarization in visual content, and present findings at both the news outlet and politician levels.
We briefly discuss the validation of the counterfactual images using our cGANs toolkit. It is important to ensure that adding a smile does not systematically change other contextual features of the image. Ensuring this consistency is essential for isolating the effect of the smile without introducing biases from other image characteristics, such as brightness or colorfulness (see $\S$(ref)). To that end, we measure a few other image characteristics (such as brightness and colorfulness) before and after applying the smile operator \( \pi^{1} \) and use the Kolmogorov-Smirnov (K-S) test to compare the distributions of these characteristics for the original and smiley images. Our tests suggest that there are no statistically significant differences between the original and smiley images for other contextual features. This confirms that the operator \( \pi^{1} \) does not introduce significant changes in other characteristics, ensuring that the counterfactual images remain consistent with the original ones. Detailed histograms and a complete analysis of the test results are shown in the Web Appendix $\S$(ref).
In this section, we present the performance of our multi-modal multi-class classification problem for news outlet prediction. As mentioned earlier, we use an 85%-15% split between training and test data. We consider four measures to evaluate the model's predictive performance -- (1) Accuracy, (2) Precision, (3) Recall, and (4) Weighted Cross-Entropy (WCE). Detailed explanations of these metrics are provided in Web Appendix $\S$(ref).
We use the above performance measures to evaluate different versions of our deep learning model on the test data and present the results in Table (ref). Each row shows a more complex model that progressively adds more explanatory variables/features. The first row considers a model that only uses categorical inputs such as politician identity, party affiliation, and date. We then add textual information in the second row and add image features in the final row. Two key points emerge from Table (ref). First, our final outlet prediction model achieves a remarkably good performance of approximately 44%. It is worth emphasizing that a random benchmark only achieves a 5% accuracy (given that there are 20 news labels). Second, we notice that the image information greatly contributes to the predictive accuracy of the model: while the non-image models achieve a maximum accuracy of 16.72%, the multi-modal model achieves an accuracy of 43.81%. This aligns with prior research that highlights the importance of visual data in predictive tasks dzyabura2023leveraging.
We present additional details on the model's performance measures across outlets in Web Appendix (ref). In the rest of this section, we provide some additional interpretation and intuition for the results from the black-box predictive model. In $\S$(ref), we discuss how our model captures the information in a smile, and in $\S$(ref), we examine how outlet predictions shift when we add a smile to an image.
Understanding how a multi-modal model learns and utilizes visual features is essential for interpreting predictions, especially in complex tasks like news outlet classification. However, this can be challenging since visual features are high-dimensional. In particular, it is important to examine whether the model is really utilizing information from facial features and emotions or whether it is simply utilizing the broader contextual information in the image for prediction (such as background color, text, etc.).
We employ a recently developed, popular tool -- Grad-CAM -- to visualize the contribution of specific visual features to the model's predictions by helping us understand the attention patterns of different components in the model selvaraju_etal_2017. Grad-CAM highlights the regions of an image that drive the decision-making process by examining the gradient of the model's prediction function \( g(\cdot) \) with respect to its inputs. This technique is particularly helpful in our architecture, where two different CNN branches—Face-VGG and ResNet101—process facial features and global image context, respectively. We can therefore use Grad-CAM to analyze attention patterns in both CNN branches separately.
As an illustrative example, consider the last image of Donald Trump in Figure (ref), from Washington Post. Figure (ref) shows visualizations for two versions of this image: the original image (top row) and a smile-added version (bottom row) generated using the transformation function \(\pi^1\). The Grad-CAM visualizations illustrate two distinct patterns of model attention. First, the modified ResNet101 heatmaps predominantly focus on broader contextual features, such as the lectern, the logo, and background elements, including the event name (“Operation Warp Speed"). The gradients of \(g(\cdot)\) with respect to the global image embeddings emphasize that ResNet101 captures cues related to the overall spatial and contextual layout of the image, such as positioning and visual saliency. Second, the modified Face-VGG heatmaps display heightened sensitivity to facial features, with particular emphasis on the mouth region. For the original image, the gradient of \(g(\cdot)\) with respect to the facial embeddings shows moderate activation around the mouth, corresponding to a neutral expression. When a smile is introduced, the activation in the mouth region intensifies and extends to other facial areas associated with smile dynamics. This pattern suggests that Face-VGG is able to successfully detect smile-related features and subtle changes in facial expressions, which provides strong evidence in support of including Face-VGG in our network architecture.
In the previous section, we saw that our outlet prediction model can capture the information in the smile/facial features of the focal politician. We now examine how adding a smile to a focal image changes news outlet predictions. As discussed in $\S$(ref), our identifying assumption requires that the dataset contains similar images (e.g., from the same/similar events, as in the case of Operation Warp Speed) from a diverse set of news outlets, where the main difference between the images is the facial expression of the focal politician. As such, our goal in this section is to demonstrate how our multi-modal ML model $g$ not only learns the structure of the comparable images for similar events but also captures the editorial preferences of different news outlets for smiles versus neutral expressions.
To do so, we turn to unsupervised learning techniques. First, we extract embeddings for all images using ResNet101. Each image is represented as a high-dimensional vector $\mathbf{e}_i \in \mathbb{R}^d$, where $d$ denotes the dimensionality of the embedding space. These embeddings capture general visual features of the images, such as texture, color, and background features, without being tied to specific news outlets. To make the embeddings interpretable and suitable for visualization, we reduce their dimensionality to two dimensions using Principal Component Analysis (PCA) and t-Distributed Stochastic Neighbor Embedding (t-SNE), followed by clustering to identify patterns. We present the details of PCA and t-SNE in our context in Web Appendix $\S$(ref).
Figure (ref) shows the t-SNE projection for all images of Donald Trump, grouped into 20 clusters based on their visual similarity. Each cluster captures images with shared contextual or visual features and provides a conceptualization of “similar events". For instance, consider Cluster 18, which is highlighted in Figure (ref). Here, each point represents an image, and the color indicates the conservative score of the news outlet that selected the image, ranging from liberal (blue) to conservative (red). The diversity in news outlets within the cluster suggests that multiple news outlets show images that are similar in structure/background (likely covering similar events). This variation is critical since it allows the outlet prediction model to capture how variation in facial features influences the likelihood of an image being chosen by a given outlet, conditional on the other image features (e.g., background, text, texture) being similar.
Figure (ref) illustrates examples from Cluster 18, showcasing images used by different news outlets, while Figure (ref) displays the corresponding images for the highlighted IDs. Within this cluster, consider the image corresponding to ID 196491 in our dataset by the Washington Post, which is the same image of Donald Trump used in the previous section. In this image, Trump appears unhappy, leaning slightly towards an angry expression. As shown in Figure (ref), this image is initially positioned on the top right-hand side of the plot. Next, we modify the original image (ID 196491) by applying the transformation function \(\pi^1\), which adds a smile to Donald Trump while preserving the contextual elements of the image. After computing the embedding for this modified image, its coordinates in the t-SNE space are updated, as illustrated by the gold dot in the scatter plot. Interestingly, the smile-added version of the original image shifts to the left. Now, the updated position results in a shorter distance to images such as ID 36529 from the Daily Mail and ID 72022 from Newsmax, compared to its distance to the original image ID 196491 by the Washington Post and ID 12736 by ABC News. This demonstrates how the addition of a smile alters the embedding position to align with images that share similar facial expressions, even though the overall context remains unchanged.
Although the t-SNE coordinates provide interpretable insights into how our model works, they do not capture the full complexity of the image features that our model $\hat{g}$ captures. Therefore, in Table (ref), we present our model's outlet predictions for the fourth image from Figure 4 (the one that was originally shown in Washington Post) for two cases -- the original image and smile-added image version. We see substantial differences in the outlet predictions between the original image and the counterfactual version with the smile. Notably, we find that adding a smile substantially increases the predicted probability for Daily Mail (from 2.89% to 20.97%) and Newsmax (from 7.10% to 16.52%), slightly decreases the predicted probability for ABS News (from 2.59% to 1.70%), and substantially decreases the predicted probability for Washington Post (from 30.98% to 3.32%).
In summary, we see that adding a smile systematically shifts the outlet prediction probabilities, consistent with the finding in the prior work in the advertising domain that shows facial expressions and emotions can shape audience engagement by capturing attention and influencing perception teixeira2012emotion. Specifically, we see an increase (decrease) in the predicted probabilities for right-leaning (left-leaning) news outlets when we add a smile to an image of Donald Trump. Nevertheless, this is just a single instance in the data that we use to illustrate the intuition behind how our algorithm works. In the next section, we present more systematic measures to quantify visual polarization.
We now present our main results on visual slant. All results are direct applications of Algorithm (ref) in $\S$(ref). Before presenting the results, we review a few important considerations in applying our algorithm. First, recall that our visual slant parameter requires a neutral outlet denoted by $y_n$ in Definition (ref). We use {\it Reuters} as the baseline outlet $y_n$ to measure visual slant, as {\it Reuters} is a largely neutral and fact-based outlet.\footnote{It is worth emphasizing that one could easily change the choice of base. Although the visual slant measure will change with a different choice, the overall visual polarization between two outlets remains unchanged.} Second, we use a train-test split, and all the polarization and slant measures are shown for the test data $\mathcal{D}_{\emph{test}}$ using the estimates obtained from the training data $\mathcal{D}_{\emph{train}}$. Third, for each politician, we start with three neutral images, generate counterfactual smiling versions of these three images, and use these six images in our polarization and slant measurement for each politician.
This section is organized as follows. In $\S$(ref), we present the overall distribution of visual slant across all outlets and politicians. We then explore the extent of heterogeneity across news outlets in $\S$(ref) to see which outlets exhibit higher levels of visual slant and validate our measure using external measures of media slant. Finally, in $\S$(ref), we document the heterogeneity in visual slant at the politician level.
An important feature of our algorithm is its ability to produce an individual-level measure of visual slant \(\hat{\rho}^T_{i}(p_i, y^{k}_{i}, y^{Reuters}_{i}) \). As such, for each article $i$ featuring politician $p$, we can estimate 19 visual slant measures corresponding to all the outlets other than Reuters. Based on the scores from faris2017partisanship, allsides2024, and flaxman2016filter, we categorize the three most Republican-leaning news outlets as: Fox News, Newsmax, and Daily Mail, and the three most Democratic-leaning news outlets as: Washington Post, CNN, and The New York Times. Figure (ref) shows the distributions of visual slant for Democratic and Republican politicians as featured in Democratic- and Republican-leaning news outlets. If there were no visual slant or polarization, all distributions would be tightly centered around zero. However, we observe that for both Republican- and Democratic-leaning outlets, the individual-level visual slant measures deviate from zero and exhibit significant dispersion. This indicates the presence of a visual slant in these outlets. Additionally, we notice a clear divide in the distributions of Democratic- and Republican-leaning outlets for both Republican and Democratic politicians, pointing to evidence of visual polarization.
Next, we examine if there is a systematic difference between Republican- and Democratic-learning outlets by examining their averages in each panel. We denote the set of Republican and Democratic politicians by $\mathcal{P}_R$ and $\mathcal{P}_D$, respectively. Similarly, we denote the sets of Republican- and Democratic-leaning outlets defined above are denoted by $\mathcal{Y}_R$ and $\mathcal{Y}_D$, respectively. For Republican politicians, in Figure (ref), the distributions of \( \hat{\rho}^T_{i} \) show distinct differences between left- and Republican-leaning news outlets. The visual slant measurement for left-leaning news outlets is -0.45, implying that, on average, smiling images of Republican politicians decrease the utility of a news outlet classified as Democratic-leaning compared to Reuters. In contrast, the visual slant measurement for Republican-leaning news outlets is 0.43, suggesting that, on average, smiling images of Republican politicians significantly increase their utility compared to Reuters. Therefore, for Republican politicians, we can conclude that:
For Democratic politicians, in Figure (ref), the visual slant distribution for Democratic and Republican news outlets also shows clear differences. The average visual slant for Democratic news outlets is 0.43, indicating that, on average, images of smiling Democratic politicians increase the utility of a Democratic-leaning outlet compared to Reuters (neutral baseline). Conversely, the average visual slant for Republican-leaning new outlets is -0.38, suggesting that, on average, smiling images of democratic politicians decrease their utility compared to Reuters. Therefore, for Democratic politicians, we can conclude that:
In Web Appendix $\S$(ref), we employ two statistical tests to analyze these differences: the Kolmogorov-Smirnov (K-S) test and one-sample t-tests. Both confirm that there are significant differences between the distributions and means of Republican- and Democratic-leaning outlets for both Democratic and Republican politicians.
We now document the extent to which visual slant varies across outlets. Figure (ref) shows our visual slant measure for each news outlet ($y^k \in \mathcal{Y}$) for Republican politicians as \( \hat{\rho}^T(p \in \mathcal{P}_R, y^{k}, y^{Reuters}) \)), and for Democratic politicians as \( \hat{\rho}^T(p \in \mathcal{P}_D, y^{k}, y^{Reuters}) \), ranked in bar charts. In Figure (ref), we see that Democratic news outlets have a positive visual slant measurement for Democratic politicians and a negative visual slant for Republican politicians. Conversely, in Figure (ref), we see that Republican news outlets exhibit a positive visual slant measurement for Republican politicians and a negative visual slant measurement for Democratic politicians.
Further, outlets such as Fox News and Daily Mail display a more positive visual slant measurement for Republican politicians, indicating a more favorable portrayal of these politicians. Simultaneously, these outlets show a more negative visual slant measure of Democratic politicians, displaying a less favorable portrayal of these politicians. On the other hand, outlets like CNN, Washington Post, and The New York Times exhibit a more positive visual slant measurement for Democratic politicians and a more negative visual slant for Republican politicians. Interestingly, some outlets, such as the Wall Street Journal and \textit{CBS News}, demonstrate a positive visual slant for both Democratic and Republican politicians. This suggests that their editorial approach may balance portrayals of politicians from both parties, reflecting a more centrist ideological positioning rather than strong partisan alignment.
To establish a single outlet-specific visual slant measurement, we compute the difference between the visual slant measures for Republican and Democratic politicians within each outlet. A larger difference indicates a stronger conservative visual slant. Accordingly, we define our unified measure as the Conservative Visual Slant (CVS) as follows:
Intuitively, this metric captures the degree to which an outlet portrays Republican politicians more favorably compared to Democratic politicians. For example, if a Republican-leaning outlet scores high on this measure, it implies that it emphasizes showing Republican politicians with a smile while depicting Democratic politicians without a smile. Figure (ref) illustrates the conservative visual slant scores for all outlets in our dataset, sorted in increasing order. The results reveal that outlets such as Daily Mail and Fox News demonstrate high conservative visual slant, strongly favoring Republican politicians. On the other hand, outlets like Washington Post, USA Today, and NY Times exhibit a pronounced liberal slant, favoring Democratic politicians. Meanwhile, outlets such as Wall Street Journal, \textit{BBC News}, and \textit{CBS News} exhibit relatively neutral or low levels of slant. Overall, there is significant heterogeneity in visual slant across outlets, providing insights into which outlets are the most and least polarized visually.
Lastly, we perform two validation tests to show how our proposed {\it CVS} measure relates to the existing measures of media slant, and how it performs relative to benchmarks that measure visual slant. We present a brief summary of our validation exercise below and refer readers to Web Appendix $\S$(ref) for a comprehensive analysis of outlet-level visual slant, including individual histograms, statistical summaries, and additional tests to quantify bias and polarization across news outlets.
In summary, these results demonstrate the validity of our approach in comparison to existing benchmark measures of media bias.
We now examine how different politicians are portrayed across news outlets and identify the politicians with the most polarizing depictions in media. Figure (ref) presents the visual slant measures for two prominent figures: Barack Obama and Donald Trump. Consistent with our earlier findings, we observe a clear divide in visual slant scores -- Obama receives more favorable portrayals in Democratic-leaning outlets, while Trump is depicted more positively in Republican-leaning outlets. We extend this analysis to other politicians, with detailed results provided in Web Appendix $\S$(ref).
Next, we develop a measure for the overall extent of visual polarization for each politician. Intuitively, we expect to observe a greater extent of variability in visual slant measures across outlets for a more polarizing politician. As such, we quantify the {\it Overall Visual Polarization (OVP)} for each politician, which is measured by the standard deviation of the polarization across all news outlets as:
Figure (ref) ranks the politicians from most polarized to least polarized based on this criterion. We see that Donald Trump and Barack Obama are the top two politicians with the most visually polarizing portrayal across all outlets. Given that both were the president/presidential candidate for significant chunks of our observation period and at the forefront of multiple polarizing discussions and events, this is understandable. Further, Rand Paul and Ted Cruz, both prominent Republican politicians, also show high levels of polarization, likely attributed to their roles in policy debates and media prominence during key political events. On the Democratic side, figures such as Bernie Sanders, Hillary Clinton, and Kamala Harris display significant polarization, reflecting their leadership roles and ideological positions within the party. Further, on the right end of Figure (ref), we observe politicians with lower {\it OVP} scores. Notably, Susan Collins, Liz Cheney, and Joe Manchin rank among the least polarized, aligning with their reputations for bipartisanship and moderation cnn_2018. These findings have important implications for political strategy, particularly in how politicians shape their election campaigns when deciding whether to mobilize their base or appeal to a broader, centrist electorate.
Finally, we seek to validate our politician-specific {\it OVP} measure by comparing it with external indicators of a politician's level of polarization. However, there exists no widely accepted, politician-specific polarization metric. To overcome this issue, we propose that a politician’s ideological alignment with their primary constituency serves as a meaningful proxy, since more polarizing politicians are likely to perform better in ideologically aligned constituencies and struggle in misaligned ones. To quantify this alignment, we use the politician’s party's success in the 2016 election within their state, by measuring their party's percentage point advantage in that election. We then analyze the correlation between this measure and our {\it OVP} metric. Figure (ref) presents the results from this exercise. We see a strong and significant correlation between the two measures, which further supports the validity of our {\it OVP} measure.
This finding supports the idea that politicians from states with strong partisan leanings tend to adopt more pronounced partisan positions without seeking to appeal to a broad electorate. For example, Bernie Sanders (Vermont) and Ted Cruz (Texas) come from states with clear ideological identities and exhibit high {\it OVP} values, likely reflecting their strong partisan stances and the resulting polarized media portrayals. In contrast, Joe Manchin (West Virginia) and Susan Collins (Maine), who represent states where their party is in the minority, have lower {\it OVP} values, which reflects their moderate positions.
In this paper, we present a framework for measuring slant and polarization in the visual content accompanying news articles. We propose the Polarization Measurement Using Counterfactual Image Generation (PMCIG) algorithm, which quantifies news outlets' preference for slanted imagery -- such as smiling images -- to convey positive (vs. negative) representation of politicians. Our framework combines the economic structure of the problem with generative models to generate comparable counterfactual images and measure polarization in visual content. Notably, our algorithm overcomes the key limitations in traditional descriptive methods due to information loss in the feature extraction phase by using the rich information contained in images.
In our empirical analysis, we apply the PMCIG framework to a decade-long dataset that covers 20 major news outlets and 30 prominent politicians. We identify clear patterns of ideological slanting and political polarization in the visual representation of political figures. We validate our measure of visual slant by demonstrating a high correlation between our measure and the existing measures used for media slant and partisanship. Our framework measures visual slant and polarization with detailed granularity, highlighting differences both at the outlet level and for individual politicians. Among outlets, we find that {\it Daily Mail} and {\it Fox News} display the strongest Republican-leaning visual slant, while {\it Washington Post} and {\it The New York Times} exhibit the strongest Democratic-leaning slant. In contrast, {\it CBS News} and {\it Wall Street Journal} are among the outlets with the lowest overall visual slant. At the individual level, Donald Trump and Barack Obama stand out as the most polarizing figures, whereas Joe Manchin, Liz Cheney, and Susan Collins are among the least polarizing in their visual portrayal across news outlets.
In summary, the PMCIG framework offers a systematic approach to analyze how ideological preferences shape visual content in news media and contribute to polarization. Nevertheless, our paper has limitations that serve as excellent avenues for future research. For instance, our analysis is based on data from the United States, and extending the framework to other regions, such as Europe, could uncover cross-cultural differences in visual polarization. Additionally, while the framework measures visual slant and polarization, it does not examine the downstream impact on individuals' beliefs or behavior, which could be fruitful avenues for future research. Future studies could also apply the PMCIG framework to other forms of media, such as social media or advertising, and investigate the extent of ideological slanting/bias in these settings.
Author(s) have no competing interests to declare.
\setcounter{table}{0} \setcounter{page}{0} \setcounter{figure}{0}