Replicating Pan, Shao, and Xu (2022)
Jinwen Wu
2024-9-16
Pan, Shao, and Xu (2022) investigate whether government-controlled media can shift public attitudes on domestic and foreign policies. They find the effects are significant and remain even when the regime reverses its policy stance. This study is contextualized within China, where the Party-state heavily invests in political propaganda.
This RMarkdown tutorial replicates the core analyses in “How government-controlled media shifts policy attitudes through framing”. The replication, conducted by Jinwen Wu, a predoctoral fellow at Stanford University, is supervised by Professor Yiqing Xu. The replication summarizes the main data analyses from the article; please refer to the original paper for a comprehensive understanding of the ideas presented.
Click the Code button at the top right and select
Show All Code to reveal all codes used in this RMarkdown.
Click Show in paragraphs to reveal the code used to
generate a finding. The original replication files can be downloaded
from here.
The researchers conducted an online survey experiment of 830 respondents in China to assess the effectiveness of government-controlled media in shifting public attitudes through framing. Building upon political psychology literature, the researchers concentrated on how framing affects people’s policy preferences on controversial political discourse about economic development and foreign affairs. The analytical results suggest:
- Government-controlled media effectively shifts public attitudes toward the regime’s new policy positions, regardless of individuals’ predispositions.
- The effect holds for both domestic (state-owned enterprise reform) and foreign policy issues (China-Philippines relations).
- Framing effects are observed in both direct and composite measures of policy attitudes and persist up to 48 hours after exposure.
- The effects may not be heterogeneous across people with different predispositions or characteristics.
1 Conceptual Framework
1.1 Propaganda Changes Public Opinion
Governments invest in propaganda to change people’s attitudes, opinions, and behaviors. The authors summarize four major explanations for how propaganda can achieve these effects.
- First, propaganda serves as massive persuasion, leveraging repetitive messages and presumably authoritative channels to shape public opinion (Chaffee (1977); Lasswell (1927); Jowett and O’Donnell (2018)).
- Second, it can teach the public especially when the information promoted by the propagandists is novel (Prior and Lupia (2008)).
- Third, instead of making logical arguments or sharing selected information, propaganda can activate certain emotions and sentiments that are triggers to manipulate perceptions and subsequent actions (Huang (2015); Wedeen (1998)).
- Fourth, propaganda can serve for agenda-setting, where media not only tells the public what to think but also how to think about a controversy, thereby framing the public discourse (Iyengar (1994); McCombs (2018)).
The effectiveness of propaganda is generally conditional on individuals’ predispositions, characteristics, and the source’s credibility (Gunther and Mughan (2000); Zaller et al. (1992)). However, survey results in this study do not suggest any heterogeneous effects.
1.2 Framing Effects and Moderators
Scholars often use an expectancy-value model to conceptualize the causal path of framing effects; attitudes towards an object, such as a policy or event, are depicted as a function of the weighted evaluation of its attributes (Azjen (1980); Nelson, Clawson, and Oxley (1997)). Framing changes the prominence of issue attributes, thus shifting public attitudes (Chong and Druckman (2007)).
Diverse studies have found that exposure to framed information alters the accessibility and applicability of specific attributes stored in memory by emphasizing particular aspects of an issue, employing specific language, or providing context that highlights certain perspectives. For instance, attitudes toward economic policies, such as state-owned enterprise reform, could depend upon a weighted sum of some attributes like production efficiency and public interest. When the media and policymakers frame a policy discussion around economic impact, attributes such as cost savings, budget deficits, and tax implications become more prominent. The narrative prompts the public to evaluate the policy proposal, as well as the office, based on financial efficiency. It is not surprising that governments can leverage framing effects for political benefits and electoral advantages.
Pan, Shao, and Xu (2022) discuss two key moderators that can impact framing effects: predisposition and political knowledge.
- Predisposition, prior beliefs, and values influence how individuals react to alternative frames by suggesting disconfirming attributes less applicable. Thus, people with strong predispositions are less likely to be swayed by new frames.
- Political knowledge also affects susceptibility to framing, with mixed research results. While less politically knowledgeable individuals may be more influenced by frames, those with greater knowledge could reduce their susceptibility. Yet, some studies also found that when predisposition is controlled for, greater political knowledge can enhance framing effects.
The framing effects dynamic is particularly critical for government-controlled media, where frequent position changes can undermine the source’s credibility, further complicating the effectiveness of new frames. Ineffective framing can undermine public trust and perceptions of the regime’s competence. In modern autocracies, maintaining public confidence in the leader’s competence is crucial for regime stability. Thus, it’s necessary for authoritarian regimes to emphasize media control, investing in propaganda to mold public opinion (Geddes and Zaller (1989); Stockmann (2010)).
1.3 Government-Controlled Media
Media framing influences political attitudes by adjusting the weights of attributes. State-controlled media is a critical, if not indispensable, tool used for political propaganda in authoritarian regimes. The media systems are designed to ensure the regime’s narratives dominate the public discourse, suppress dissent, and maintain control over the population. The state-controlled media such as China’s China Central Television, North Korea’s Korean Central News Agency, and Russia’s Russia Today, are arguably the most powerful news agencies that easily pierce all other domestic competitors. These outlets not only propagate the government’s perspective but also work to discredit the opposition and shape international perceptions.
The authors discuss an interesting question: while public opinion is often considered less consequential in autocracies compared to democracies, why do authoritarians consistently invest a significant portion of the budget in running state-owned agencies?
One hypothesis is that while people may not directly influence policy-making, their conformity and cooperation are essential for achieving desired policy outcomes. It is not uncommon for authoritarian regimes to revise or even reverse their policy stances in response to public dissent. For instance, studies in China have shown that public opposition to certain policies can compel the government to make concessions (Weeks (2008); Weiss (2013)). By preemptively shaping public preferences through propaganda, authoritarian governments can mitigate potential resistance and maintain stability.
2 Research Method
2.1 Case selection
The researchers collaborated with media professionals in China to create six short news video segments that closely resemble the content and style of China’s main television broadcast, Xinwen Lianbo. These segments were embedded in an online survey experiment to measure the effects of government-controlled media on policy preferences. The selection has four major considerations.
First, although exhibiting a gradual decline in people’s media consumption share, television is still found to be “the primary way” for people to consume news in authoritarian regimes.
Second, television propaganda activates multiple sensory channels and elicits strong emotional and physiological responses (Houts et al. (2006); Kensinger and Schacter (2006); Lang, Potter, and Bolls (1999); Okdie et al. (2014)), so the video treatment can generate opinion changes more effectively than text-only propaganda.
Third, the video clips resemble the Xinwen Lianbo format and represent how the Chinese government uses state-controlled media to shift public opinion. On each day from 7 to 7:30 pm, Xinwen Lianbo dedicates 20 to 25 minutes to domestic issues and covers foreign affairs in the remaining time. It has been sponsored by and serving the Communist Party since January 1, 1978. Unlike successful news broadcasts typically seen in free media democracies, which cover various topics ranging from entertainment to sports, Xinwen Lianbo usually solely concentrates on political issues, communicating the regime’s official stance, setting the policy agenda, and framing issues in line with the regime’s interests (Zhao and Guo (2005)).
Fourth, Xinwen Lianbo has a broad audience base and a standardized format that the authors have successfully replicated in their experimental clips. Xinwen Lianbo enjoys a monopoly as virtually all local TV channels are required to broadcast it (Zhao and Guo (2005))). Between 55 to 60 million viewers watch Xinwen Lianbo each evening, which translates to 4.6% of the 1.2 billion people with access to television in China. In this study, the authors incorporate validation questions to check respondents’ prior exposure to Xinwen Lianbo. Therefore, without explicitly telling the respondents that the video prompts the Chinese Communist Party (CCP)’s official policy stance, the distinctive Xinwen Lianbo features in the video clips imply that the information communicated is endorsed by the CCP. By doing so, the researchers hope to reduce social desirability bias and clarify the causal framing effects
2.2 Experiment Flow
The researchers recruited 830 respondents from 26 provinces in China for the experiment. Using a quota sampling strategy, the researchers included participants of diverse genders, ages, and educational backgrounds.
The respondents were first screened for background information and internet connection. Respondents, for whom the videos can be smoothly loaded, were asked to answer questions measuring their predispositions on economic and foreign policy issues. Then, they were randomly assigned to watch two videos that mimic Xinwen Lianbo, each on different policy issues: state-owned enterprise (SOE) reform and China-Philippine relations. Immediately after viewing each video, respondents completed attention checks on basic factual information and fundamental political messages discussed in the video clip.
The researchers record their policy preferences with two composite measures: one about specific policy support, and the other about related policy domains. 48 hours later, respondents received new survey questions on their policy preference for the SOE reform and China-Philippine relations. To ensure the quality of the survey responses, the authors validate 1. attention filter questions 2. background information consistency (the reported birth year in the two rounds of surveys are consistent), and 3. time spent on each video. The filtering process removed 68 survey responses.
The flowchart of the experiment is attached below:
2.3 Hypotheses
The researchers are interested in testing：
H1: Government-Controlled Media Framing: Exposure to the framed narratives will move respondents to the political position endorsed by the government media.
H2: Moderation by Predisposition and Political Knowledge: Framing effects are moderated by respondents’ predisposition or political knowledge.
2.4 Treatments
2.4.1 Domestic Policy - SOE
2.4.2 Foreign Policy - China-Philippines
3 Replicating the Main Findings
The survey experiment contains two waves. In the first stage, the researchers measured respondents’ specific policy support by asking whether they would support or oppose Pro-Market domestic economic policy and Pro-Corporation foreign policy. They also created a composite index for SOE and China-Philippines relations to track respondents’ opinions on the broader policy domains. In the second stage, the researchers were able to reconnect with 353 respondents 48 hours after the first survey (a recontact rate of 63 percent) to test the persistence of the media framing effects.
3.1 Installing Packages
Several R packages are required for subsequent data analysis and visualization. The code chunk below checks for all required libraries and installs the missing ones.
Packages:“haven”, “dplyr”, “stargazer”, “tidyr”, “kableExtra”, “estimatr”, “broom”, “interflex”, “ggplot2”, “modelsummary”, “gridExtra”
install_all <- function(packages) {
installed_pkgs <- installed.packages()[, "Package"]
for (pkg in packages) {
if (!pkg %in% installed_pkgs) {
install.packages(pkg, repos = "http://cran.us.r-project.org")
}
}
}
# Packages to be installed
packages <- c("haven", "dplyr", "stargazer", "tidyr", "kableExtra", "estimatr", "broom", "interflex", "ggplot2", "modelsummary", "gridExtra")
install_all(packages)After installation, call to load the packages. Then, import the
experiment data. The file is in the data folder titled
media.dta.
library(haven)
library(dplyr)
library(stargazer)
library(tidyr)
library(kableExtra)
library(estimatr)
library(broom)
library(interflex)
library(ggplot2)
library(modelsummary)
library(gridExtra)
library(dotwhisker)
data <- read_dta("media.dta")Two sets of outcome variables and the 16 covariates are defined to assist further analysis.
# Define variables
treat <- c("tr1_market", "tr1_state", "tr2_dove", "tr2_hawk")
covar <- c("female", "age", "age_sq", "hschool", "jcollege", "college", "minority", "religious", "unmarried", "work", "work_public", "work_report", "ccp", "vpn", "english", "income", "socialclass")
covarbal <- c("female", "age", "hschool", "jcollege", "college", "minority", "religious", "unmarried", "work", "work_public", "work_report", "ccp", "vpn", "english", "income", "socialclass")
outcome1 <- c("soe_market", "opn_market", "fp_dove", "opn_dove", "soe_pass", "soe_content_market", "soe_content_state", "soe_agree", "fp_pass", "fp_content_hawk", "fp_content_dove", "fp_agree")
outcome2 <- c("w2_soe_market", "w2_opn_market", "w2_fp_dove", "w2_opn_dove", "w2_soe_pass", "w2_soe_content_market", "w2_soe_content_state", "w2_soe_agree", "w2_fp_pass", "w2_fp_content_hawk", "w2_fp_content_dove", "w2_fp_agree")
other <- c("pat", "nat", "eff", "trust", "complain", "election", "discuss")
covariate_variable_mapping <- c(
"female" = "Female",
"age" = "Age",
"hschool" = "High school",
"jcollege" = "Junior college",
"college" = "College or above",
"minority" = "Minority",
"religious" = "Religious",
"unmarried" = "Single",
"work" = "Having worked",
"work_public" = "Public sector worker",
"work_report" = "Being reported to",
"ccp" = "CCP member",
"vpn" = "Having used VPN",
"english" = "English proficiency [1, 5]",
"income" = "Income category [1, 10]",
"socialclass" = "Self-reported social class [1, 10]"
)
outcome_variable_mapping <- c(
"soe_market" = "Agree to market reform (1, 2 or 3)",
"opn_market" = "An index of market-oriented reform",
"soe_agree" = "Agree on policy toward SOE reform is correct (1, 2, or 3)",
"soe_pass" = "Pass SOE manipulation check",
"soe_content_market" = "Identify SOE media content as pro-market",
"soe_content_state" = "Identify SOE media content as pro-state",
"fp_agree" = "Agree on policy toward South China Sea is correct (1, 2, or 3)",
"fp_pass" = "Pass FP manipulation check",
"fp_content_hawk" = "Identify FP media content as hawkish",
"fp_content_dove" = "Identify FP media content as dovish",
"fp_dove" = "An index of dovish foreign policy",
"opn_dove" = "Agree to more cooperation (1, 2, or 3)"
)3.2 Covariate Balance Check
First, the author performed two covaraite balance checks. As shown by the tables below, the randomization is successful.
data$clusters = data$id #cluster at ID for now
var_t1 <- c(covar, "clusters", "tr1")
var_t2 <- c(covar, "clusters", "tr2")
data_pre <- data[, var_t1]
data_pre$tr1 <- factor(data_pre$tr1, levels = c(1, 2, 3), labels = c("Control", "Treatment 1: Market","Treatment 1: State"))
names(data_pre) <- ifelse(names(data_pre) %in% names(covariate_variable_mapping),
covariate_variable_mapping[names(data_pre)],
names(data_pre))
data_pre_2 <- data[, var_t2]
data_pre_2$tr2 <- factor(data_pre_2$tr2, levels = c(1, 2, 3), labels = c("Control", "Treatment 2: Dovish", "Treatment 2: Hawkish"))
names(data_pre_2) <- ifelse(names(data_pre_2) %in% names(covariate_variable_mapping),
covariate_variable_mapping[names(data_pre_2)],
names(data_pre_2))
balance_summary_tr1 <- datasummary_balance(~tr1, data = data_pre, dinm_statistic = "p.value", fmt=2)
balance_summary_tr2 <- datasummary_balance(~tr2, data = data_pre_2, dinm_statistic = "p.value", fmt=2)
balance_summary_tr1| Control (N=274) | Treatment 1: Market (N=277) | Treatment 1: State (N=279) | ||||
|---|---|---|---|---|---|---|
| Mean | Std. Dev. | Mean | Std. Dev. | Mean | Std. Dev. | |
| Female | 0.42 | 0.49 | 0.43 | 0.50 | 0.37 | 0.48 |
| Age | 31.03 | 8.61 | 31.49 | 8.70 | 30.92 | 8.83 |
| age_sq | 1036.42 | 576.09 | 1067.17 | 590.42 | 1033.88 | 587.86 |
| High school | 0.24 | 0.43 | 0.26 | 0.44 | 0.22 | 0.41 |
| Junior college | 0.13 | 0.34 | 0.11 | 0.32 | 0.16 | 0.37 |
| College or above | 0.42 | 0.49 | 0.43 | 0.50 | 0.43 | 0.50 |
| Minority | 0.03 | 0.17 | 0.04 | 0.20 | 0.03 | 0.18 |
| Religious | 0.16 | 0.36 | 0.17 | 0.37 | 0.18 | 0.39 |
| Single | 0.47 | 0.50 | 0.43 | 0.50 | 0.46 | 0.50 |
| Having worked | 0.79 | 0.41 | 0.77 | 0.42 | 0.76 | 0.43 |
| Public sector worker | 0.16 | 0.36 | 0.15 | 0.36 | 0.17 | 0.38 |
| Being reported to | 0.38 | 0.49 | 0.41 | 0.49 | 0.35 | 0.48 |
| CCP member | 0.13 | 0.33 | 0.15 | 0.36 | 0.13 | 0.33 |
| Having used VPN | 0.35 | 0.48 | 0.41 | 0.49 | 0.37 | 0.48 |
| English proficiency [1, 5] | 2.91 | 1.15 | 2.83 | 1.12 | 2.99 | 1.12 |
| Income category [1, 10] | 4.63 | 1.96 | 4.57 | 2.07 | 4.59 | 2.07 |
| Self-reported social class [1, 10] | 5.16 | 2.02 | 5.12 | 2.11 | 5.38 | 2.05 |
Covariate Balance - SOE.
balance_summary_tr2| Control (N=281) | Treatment 2: Dovish (N=273) | Treatment 2: Hawkish (N=276) | ||||
|---|---|---|---|---|---|---|
| Mean | Std. Dev. | Mean | Std. Dev. | Mean | Std. Dev. | |
| Female | 0.37 | 0.48 | 0.45 | 0.50 | 0.39 | 0.49 |
| Age | 31.39 | 8.90 | 31.86 | 8.93 | 30.19 | 8.23 |
| age_sq | 1064.07 | 601.84 | 1094.62 | 595.52 | 979.00 | 550.10 |
| High school | 0.22 | 0.42 | 0.24 | 0.43 | 0.25 | 0.44 |
| Junior college | 0.12 | 0.33 | 0.13 | 0.34 | 0.14 | 0.35 |
| College or above | 0.40 | 0.49 | 0.45 | 0.50 | 0.43 | 0.50 |
| Minority | 0.03 | 0.17 | 0.03 | 0.18 | 0.04 | 0.20 |
| Religious | 0.20 | 0.40 | 0.17 | 0.37 | 0.14 | 0.35 |
| Single | 0.47 | 0.50 | 0.40 | 0.49 | 0.49 | 0.50 |
| Having worked | 0.77 | 0.42 | 0.75 | 0.43 | 0.79 | 0.41 |
| Public sector worker | 0.17 | 0.38 | 0.15 | 0.36 | 0.15 | 0.36 |
| Being reported to | 0.37 | 0.48 | 0.39 | 0.49 | 0.39 | 0.49 |
| CCP member | 0.12 | 0.32 | 0.13 | 0.34 | 0.15 | 0.36 |
| Having used VPN | 0.37 | 0.48 | 0.39 | 0.49 | 0.37 | 0.48 |
| English proficiency [1, 5] | 2.81 | 1.16 | 2.95 | 1.13 | 2.97 | 1.10 |
| Income category [1, 10] | 4.74 | 2.03 | 4.62 | 2.09 | 4.42 | 1.97 |
| Self-reported social class [1, 10] | 5.10 | 2.01 | 5.24 | 2.09 | 5.33 | 2.07 |
Covariate Balance - China-Philippines.
3.3 Treatmehnt Effects on Policy Preferences - SOE
Then, the authors calculated the treatment effects of the Xinwen Lianbo clips on domestic SOE policy. All effects are estimated using regressions, controlling for individual characteristics and provincial fixed effects.
Each of the three assigned videos about SOE takes a pro-market, pro-state, or neutral stance. The neutral video segment only describes the basic facts about the state-owned enterprise, and thus serves as the baseline.
The left panel shows people’s direct policy preference (higher values indicate a more pro-market stance), while the right panel features a standardized index of composite policy preference based on nine questions related to market reform policies. The bars represent 95% confidence intervals (also applied to Figure 3 below).
# Run regressions and store results
results <- list()
for (outcome in outcome1) {
model <- lm_robust(as.formula(paste(outcome, "~", paste(treat, collapse = " + "), "+", paste(covar, collapse = " + "))), clusters = prov, data = data)
model_tidy <- tidy(model)
results[[outcome]] <- model_tidy %>% filter(term %in% treat)
}
# Combine results into a data frame
final_results <- bind_rows(results, .id = "outcome")
# Split data for the two separate plots
plot_data1 <- final_results %>%
filter(outcome %in% c("soe_market", "opn_market")) %>%
mutate(outcome = factor(outcome, levels = c("soe_market", "opn_market")))
plot_data2 <- final_results %>% filter(outcome %in% c("fp_dove", "opn_dove"))
# Define labels
tr.labs <- c("T1: Market", "T1: State", "T2: Dove", "T2: Hawk")
outcome_labels1 <- c("soe_market" = "Agree to Market Reform", "opn_market" = "Index of Market-oriented Reform")
outcome_labels2 <- c("fp_dove" = "Agree to Dovish FP", "opn_dove" = "Index: Support Dovish FP")
# Plot for soe_market and opn_market
p1 <- ggplot(plot_data1, aes(x = term, y = estimate, ymin = estimate - 1.96 * std.error, ymax = estimate + 1.96 * std.error)) +
geom_pointrange() +
facet_wrap(~ outcome, scales = "fixed", labeller = labeller(outcome = outcome_labels1)) +
labs(title = "Treatment Effects on Policy Preferences: Market Reform", x = "Treatment", y = "Coefficients") +
theme_minimal() +
scale_x_discrete(labels = tr.labs) +
geom_hline(yintercept = 0, color = "gray", linetype = "dashed") +
coord_cartesian(ylim = c(-0.6, 0.6)) +
theme(panel.spacing = unit(3, "lines"), # Increase spacing between facets
axis.title = element_text(size = 12), # Increase axis title size
axis.text = element_text(size = 10), # Increase axis text size
strip.text = element_text(size = 10)) # Increase facet label size
p1
Treatment effects on policy preferences: SOE reform. (a) Agree to market reform. (b) Index of market-oriented reform. Replicating Figure 2 in the article.
Framing SOE reform in terms of efficiency (Market condition) shifts respondents’ preferences towards a market-oriented approach. The direct measure shows an approximate 0.3-point shift on a 1-3 scale, and the composite index shows a 0.3 standard deviation shift towards market-oriented policies. Conversely, framing SOE reform in terms of public interest (State condition) decreases market-oriented preferences by roughly 0.3 points (direct measure) and 0.2 standard deviations (composite measure).
Notably, both the Dove and Hawk treatments shift respondents’ preferences towards the state-led approach. The authors suggest that the difference is likely due to the treatment video about foreign policy (FP). Both treatments highlight the state’s central role in national interests. Since the order of receiving either video topic is random, the spillover effects may be reflected.
3.4 Treatment Effects on Policy Preference - China-Philippines
Figure 3 takes a similar format as Figure 2. It presents the average treatment effects on policy preferences regarding China–Philippines relations about the South China Sea disputes. The left panel shows direct policy preferences (higher values indicate a more dovish stance), while the right panel features a standardized index of composite policy preferences based on nine questions related to China-Philippines foreign policies.
p2 <- ggplot(plot_data2, aes(x = term, y = estimate, ymin = estimate - 1.96 * std.error, ymax = estimate + 1.96 * std.error)) +
geom_pointrange() +
facet_wrap(~ outcome, scales = "fixed", labeller = labeller(outcome = outcome_labels2)) +
labs(title = "Treatment Effects on Policy Preferences: Dovish vs Hawkish", x = "Treatment", y = "Coefficients") +
theme_minimal() +
scale_x_discrete(labels = tr.labs) +
geom_hline(yintercept = 0, color = "gray", linetype = "dashed") +
coord_cartesian(ylim = c(-0.6, 0.6)) +
theme(panel.spacing = unit(3, "lines"),
axis.title = element_text(size = 12),
axis.text = element_text(size = 10),
strip.text = element_text(size = 10))
p2
Treatment effects on policy preferences: South China Sea. (a) Agree to more cooperation. (b) Index of dovish foreign policy. Replicating Figure 3 in the article.
Framing China–Philippines relations in terms of mutual prosperity (Dove condition) shifts respondents’ preferences towards a more cooperative strategy by approximately 0.15 points on a 1–3 scale and around 0.19 standard deviation based on the composite measure. Conversely, emphasizing national sovereignty( Hawk condition) decreases support for the dovish approach by about 0.3 points on the direct measure and 0.26 standard deviation on the composite index.
Policy preferences on FP are unaffected by the video content on SOE reform.
3.5 Marginal Treatment Effects by Predisposition
Predisposition does not moderate the effects of government-controlled media. The researchers measured respondents’ predispositions toward economic policies and national sovereignty before exposing them to the video treatments. Regardless of predisposition, exposure to the different frames in our video treatments influences policy preferences.
Estimates above zero on the y-axis in panel (a) indicate a marginal effect favoring market-oriented SOE reform, while below zero indicate support for state-led SOE reform. In panel (b), estimates above zero indicate a marginal effect favoring more dovish FP, while below zero indicates support for more hawkish FP. The stacked histograms at the bottom show the number of respondents under each treatment condition across different predispositions.
ggsave("combined_plot_1.png", combined_plot_1, width = 16, height = 8)
knitr::include_graphics("combined_plot_1.png")
Marginal effects of the treatments by predisposition. (a) Treatment 1 on pro-market. (b) Treatment 2 on pro- cooperation. Replicating Figure 4 in the article.
Panel (a) of Figure 4 demonstrates that respondents’ support for market-oriented SOE reform is influenced by policy frames, regardless of their predispositions towards resource allocation. Exposure to efficiency frames (Market video, dark gray) increases support for market-oriented reform, while exposure to public interest frames (State video, light gray) decreases it. Similarly, panel (b) also supports the findings in Figure 3.
The framing effects are not moderated by respondents’ predispositions.
3.6 Marginal Treatment Effects by Political Knowledge
After controlling for predisposition, political knowledge was found to strengthen the framing effects. Panel (a) depicts the impact of State (light gray) and Market (dark gray) videos on support for market-oriented SOE reform. Positive estimates indicate a shift towards market reform, while negative estimates indicate support for state-led reform. Panel (b) shows the effects of Hawk (light gray) and Dove (dark gray) videos on support for a cooperative foreign policy. Positive estimates suggest a dovish stance and negative estimates suggest a hawkish stance.
ggsave("combined_plot_2.png", combined_plot_2, width = 16, height = 8)
knitr::include_graphics("combined_plot_2.png")
Marginal effects of the treatments by political knowledge. (a) Treatment 1 on pro-market. (b) Treatment 2 on pro-cooperation. Replicating Figure 5 in the article.
Panel (a) indicates that respondents with higher political knowledge are more influenced by the treatments, particularly the video promoting state-led reform. Similarly, panel (b) shows that those with higher political knowledge are more susceptible to the Dove video treatment that advocates for mutual prosperity.
3.7 Persistence of Framing Effects
With the second round of surveys, the authors tested the persistence of the framing effects. The first table offers a broad overview of the characteristics of the 830 respondents representing the urban Chinese population. The Wave 1 （W1） sample includes all the responses. The Wave 2 (W2) sample on the right-hand side consists of all respondents (354 out of the 556) that the authors randomly selected, reconnected, and received answers from 48 hours after the first round survey.
The function below extracts the summary statistics. Call it to generate the summary statistics tables for respondents’ characteristics.
# Function to format summary statistics
summary_statistics <- function(df, variables) {
df %>%
select(all_of(variables)) %>%
summarise_all(list(
mean = ~ round(mean(., na.rm = TRUE), 2),
sd = ~ round(sd(., na.rm = TRUE), 2),
n = ~ sum(!is.na(.))
)) %>%
pivot_longer(everything(), names_to = c("Variable", ".value"), names_pattern = "(.*)_(.*)") %>%
arrange(match(Variable, variables))
}
# Calculate summary statistics
sum_outcome1 <- summary_statistics(data, outcome1)
sum_outcome2 <- summary_statistics(data, outcome2)
sum_covarbal <- summary_statistics(data, covarbal)
sum_covarbal_twowaves <- summary_statistics(data %>% filter(twowaves == 1), covarbal)
# Function to apply variable name mapping
apply_variable_mapping <- function(df, mapping) {
df %>%
mutate(Variable = recode(Variable, !!!mapping))
}
# Apply mapping to the summary statistics
sum_outcome1 <- apply_variable_mapping(sum_outcome1, outcome_variable_mapping)
sum_outcome2 <- apply_variable_mapping(sum_outcome2, outcome_variable_mapping)
sum_covarbal <- apply_variable_mapping(sum_covarbal, covariate_variable_mapping)
sum_covarbal_twowaves <- apply_variable_mapping(sum_covarbal_twowaves, covariate_variable_mapping)
# Combine summary statistics into a single table
combined_summary <- list(
"W1 Outcomes and Checks" = sum_outcome1,
"W2 Outcomes and Checks" = sum_outcome2,
"Covariates" = sum_covarbal,
"Covariates (W2 Sample)" = sum_covarbal_twowaves
)
# Function to create and format the summary table
create_kable <- function(df, title) {
df %>%
select(Variable, everything()) %>%
kable("html", caption = title, col.names = c("Variable", "Mean", "Standard Deviation", "N", "Mean", "Standard Deviation", "N")) %>%
kable_styling(bootstrap_options = c("striped", "hover", "condensed", "responsive"),
full_width = FALSE,
position = "left") %>%
column_spec(1, bold = TRUE) %>%
add_header_above(c(" " = 1, "W1 Summary Statistics" = 3, "W2 Summary Statistics" = 3))
}
# Merge the tables side-by-side
merged_outcome <- bind_cols(sum_outcome1, sum_outcome2 %>% select(-Variable))
merged_covarbal <- bind_cols(sum_covarbal, sum_covarbal_twowaves %>% select(-Variable))
# Create and format merged summary tables
kable_outcome <- create_kable(merged_outcome, "W1 and W2 Outcomes and Checks")
kable_covarbal <- create_kable(merged_covarbal, "Covariates for Main Sample and W2 Sample")
kable_covarbal| Variable | Mean | Standard Deviation | N | Mean | Standard Deviation | N |
|---|---|---|---|---|---|---|
| Female | 0.41 | 0.49 | 830 | 0.41 | 0.49 | 354 |
| Age | 31.15 | 8.71 | 830 | 31.82 | 8.68 | 354 |
| High school | 0.24 | 0.43 | 830 | 0.19 | 0.39 | 354 |
| Junior college | 0.13 | 0.34 | 830 | 0.13 | 0.33 | 354 |
| College or above | 0.43 | 0.49 | 830 | 0.59 | 0.49 | 354 |
| Minority | 0.03 | 0.18 | 830 | 0.04 | 0.19 | 354 |
| Religious | 0.17 | 0.37 | 830 | 0.18 | 0.39 | 354 |
| Single | 0.45 | 0.50 | 830 | 0.40 | 0.49 | 354 |
| Having worked | 0.77 | 0.42 | 830 | 0.82 | 0.38 | 354 |
| Public sector worker | 0.16 | 0.37 | 830 | 0.17 | 0.38 | 354 |
| Being reported to | 0.38 | 0.49 | 830 | 0.47 | 0.50 | 354 |
| CCP member | 0.13 | 0.34 | 830 | 0.15 | 0.36 | 354 |
| Having used VPN | 0.38 | 0.48 | 830 | 0.41 | 0.49 | 354 |
| English proficiency [1, 5] | 2.91 | 1.13 | 830 | 3.13 | 1.08 | 354 |
| Income category [1, 10] | 4.59 | 2.04 | 830 | 4.93 | 1.93 | 354 |
| Self-reported social class [1, 10] | 5.22 | 2.06 | 830 | 5.66 | 1.94 | 354 |
Respondent’s Background.
As shown above, people who participated in the two waves are very similar. Therefore, it is reasonable to infer the persistence of the treatment effect by comparing W1 and W2 estimates.
kable_outcome| Variable | Mean | Standard Deviation | N | Mean | Standard Deviation | N |
|---|---|---|---|---|---|---|
| Agree to market reform (1, 2 or 3) | 2.17 | 0.96 | 830 | 2.22 | 0.96 | 354 |
| An index of market-oriented reform | 0.00 | 1.00 | 823 | 0.13 | 1.03 | 351 |
| An index of dovish foreign policy | 1.58 | 0.74 | 830 | 1.53 | 0.73 | 354 |
| Agree to more cooperation (1, 2, or 3) | 0.00 | 1.00 | 825 | 0.00 | 0.97 | 353 |
| Pass SOE manipulation check | 0.83 | 0.37 | 830 | 0.94 | 0.23 | 354 |
| Identify SOE media content as pro-market | 0.44 | 0.50 | 830 | 0.46 | 0.50 | 354 |
| Identify SOE media content as pro-state | 0.28 | 0.45 | 830 | 0.30 | 0.46 | 354 |
| Agree on policy toward SOE reform is correct (1, 2, or 3) | 2.30 | 0.51 | 794 | 2.26 | 0.51 | 350 |
| Pass FP manipulation check | 0.97 | 0.16 | 830 | 0.89 | 0.31 | 354 |
| Identify FP media content as hawkish | 0.07 | 0.26 | 830 | 0.12 | 0.32 | 354 |
| Identify FP media content as dovish | 0.46 | 0.50 | 830 | 0.50 | 0.50 | 354 |
| Agree on policy toward South China Sea is correct (1, 2, or 3) | 2.46 | 0.53 | 800 | 2.36 | 0.54 | 347 |
Outcomes tracked in W1 and W2.
Replicating Table A3
in the article.
The codes below visualize the results using the same style as Figure 3 and Figure 4. The differences between all four estimates across the two rounds are small after 48 hours.
# Construct formulas for the regression models
formula1 <- as.formula(paste("soe_market ~", paste(treat, collapse = " + "), "+", paste(covar, collapse = " + ")))
formula2 <- as.formula(paste("w2_soe_market ~", paste(treat, collapse = " + "), "+", paste(covar, collapse = " + ")))
# Fit robust linear models with clustering by province
round1 <- lm_robust(formula1, clusters = prov, data = data)
round2 <- lm_robust(formula2, clusters = prov, data = data)
m1_df <-
broom::tidy(round1) %>%
filter(term %in% treat) %>%
mutate(model = "Wave 1", term = factor(term, levels = treat, labels = tr.labs))
m2_df <-
broom::tidy(round2) %>%
filter(term %in% treat) %>%
mutate(model = "Wave 2", term = factor(term, levels = treat, labels = tr.labs))
two_models <- rbind(m1_df, m2_df)
fig1 <- dwplot(two_models,
vline = geom_vline(
xintercept = 0,
colour = "grey60",
linetype = 2
),
dot_args = list(aes(shape = model)),
whisker_args = list(aes(linetype = model))
) +
theme_bw(base_size = 10) +
xlab("Coefficient Estimate") + ylab("") +
ggtitle("Agree to Market Reform") +
theme(
plot.title = element_text(face = "bold", size = 10),
legend.position = c(0.007, 0.01),
legend.justification = c(0, 0),
legend.background = element_rect(colour = "grey80"),
legend.title.align = 0.5
) +
scale_colour_grey(
start = 0.1,
end = 0.1,
name = "Model",
breaks = c("Wave 1", "Wave 2"),
labels = c("Wave 1", "Wave 2")
) +
scale_shape_discrete(
name = "Model",
breaks = c("Wave 1", "Wave 2"),
labels = c("Wave 1", "Wave 2")
) +
guides(
shape = guide_legend(title = "Wave"),
colour = guide_legend(title = "Wave")
)
# Construct formulas for the regression models
formula1 <- as.formula(paste("opn_market ~", paste(treat, collapse = " + "), "+", paste(covar, collapse = " + ")))
formula2 <- as.formula(paste("w2_opn_market ~", paste(treat, collapse = " + "), "+", paste(covar, collapse = " + ")))
# Fit robust linear models with clustering by province
round1 <- lm_robust(formula1, clusters = prov, data = data)
round2 <- lm_robust(formula2, clusters = prov, data = data)
m1_df <-
broom::tidy(round1) %>%
filter(term %in% treat) %>%
mutate(model = "Wave 1", term = factor(term, levels = treat, labels = tr.labs))
m2_df <-
broom::tidy(round2) %>%
filter(term %in% treat) %>%
mutate(model = "Wave 2", term = factor(term, levels = treat, labels = tr.labs))
two_models <- rbind(m1_df, m2_df)
fig2 <- dwplot(two_models,
vline = geom_vline(
xintercept = 0,
colour = "grey60",
linetype = 2
),
dot_args = list(aes(shape = model)),
whisker_args = list(aes(linetype = model))
) +
theme_bw(base_size = 10) +
xlab("Coefficient Estimate") + ylab("") +
ggtitle("Index of Market-oriented Reform") +
theme(
plot.title = element_text(face = "bold", size = 10),
legend.position = c(0.007, 0.01),
legend.justification = c(0, 0),
legend.background = element_rect(colour = "grey80"),
legend.title.align = 0.5
) +
scale_colour_grey(
start = 0.1,
end = 0.1,
name = "Model",
breaks = c("Wave 1", "Wave 2"),
labels = c("Wave 1", "Wave 2")
) +
scale_shape_discrete(
name = "Model",
breaks = c("Wave 1", "Wave 2"),
labels = c("Wave 1", "Wave 2")
) +
guides(
shape = guide_legend(title = "Wave"),
colour = guide_legend(title = "Wave")
)
library(patchwork)
combined_plot <- fig1 + fig2 +
plot_layout(guides = 'collect')+
plot_annotation(
title = "Treatment Effects on Policy Preferences: Market Reform",
theme = theme(
plot.title = element_text(hjust = 0.5, face = "bold", size = 14)
)
)
combined_plot
Treatment Effects for 48 Hours: Market Reform.
# Construct formulas for the regression models
formula1 <- as.formula(paste("fp_dove ~", paste(treat, collapse = " + "), "+", paste(covar, collapse = " + ")))
formula2 <- as.formula(paste("w2_fp_dove ~", paste(treat, collapse = " + "), "+", paste(covar, collapse = " + ")))
# Fit robust linear models with clustering by province
round1 <- lm_robust(formula1, clusters = prov, data = data)
round2 <- lm_robust(formula2, clusters = prov, data = data)
m1_df <-
broom::tidy(round1) %>%
filter(term %in% treat) %>%
mutate(model = "Wave 1", term = factor(term, levels = treat, labels = tr.labs))
m2_df <-
broom::tidy(round2) %>%
filter(term %in% treat) %>%
mutate(model = "Wave 2", term = factor(term, levels = treat, labels = tr.labs))
two_models <- rbind(m1_df, m2_df)
fig1 <- dwplot(two_models,
vline = geom_vline(
xintercept = 0,
colour = "grey60",
linetype = 2
),
dot_args = list(aes(shape = model)),
whisker_args = list(aes(linetype = model))
) +
theme_bw(base_size = 10) +
xlab("Coefficient Estimate") + ylab("") +
ggtitle("Agree to Dovish FP") +
theme(
plot.title = element_text(face = "bold", size = 10),
legend.position = c(0.007, 0.01),
legend.justification = c(0, 0),
legend.background = element_rect(colour = "grey80"),
legend.title.align = 0.5
) +
scale_colour_grey(
start = 0.1,
end = 0.1,
name = "Model",
breaks = c("Wave 1", "Wave 2"),
labels = c("Wave 1", "Wave 2")
) +
scale_shape_discrete(
name = "Model",
breaks = c("Wave 1", "Wave 2"),
labels = c("Wave 1", "Wave 2")
) +
guides(
shape = guide_legend(title = "Wave"),
colour = guide_legend(title = "Wave")
)
# Construct formulas for the regression models
formula1 <- as.formula(paste("opn_dove ~", paste(treat, collapse = " + "), "+", paste(covar, collapse = " + ")))
formula2 <- as.formula(paste("w2_opn_dove ~", paste(treat, collapse = " + "), "+", paste(covar, collapse = " + ")))
# Fit robust linear models with clustering by province
round1 <- lm_robust(formula1, clusters = prov, data = data)
round2 <- lm_robust(formula2, clusters = prov, data = data)
m1_df <-
broom::tidy(round1) %>%
filter(term %in% treat) %>%
mutate(model = "Wave 1", term = factor(term, levels = treat, labels = tr.labs))
m2_df <-
broom::tidy(round2) %>%
filter(term %in% treat) %>%
mutate(model = "Wave 2", term = factor(term, levels = treat, labels = tr.labs))
two_models <- rbind(m1_df, m2_df)
fig2 <- dwplot(two_models,
vline = geom_vline(
xintercept = 0,
colour = "grey60",
linetype = 2
),
dot_args = list(aes(shape = model)),
whisker_args = list(aes(linetype = model))
) +
theme_bw(base_size = 10) +
xlab("Coefficient Estimate") + ylab("") +
ggtitle("Index: Support Dovish FP") +
theme(
plot.title = element_text(face = "bold", size = 10),
legend.position = c(0.007, 0.01),
legend.justification = c(0, 0),
legend.background = element_rect(colour = "grey80"),
legend.title.align = 0.5
) +
scale_colour_grey(
start = 0.1,
end = 0.1,
name = "Model",
breaks = c("Wave 1", "Wave 2"),
labels = c("Wave 1", "Wave 2")
) +
scale_shape_discrete(
name = "Model",
breaks = c("Wave 1", "Wave 2"),
labels = c("Wave 1", "Wave 2")
) +
guides(
shape = guide_legend(title = "Wave"),
colour = guide_legend(title = "Wave")
)
library(patchwork)
combined_plot <- fig1 + fig2 +
plot_layout(guides = 'collect') +
plot_annotation(
title = "Treatment Effects on Policy Preferences: Dovish vs Hawkish",
theme = theme(
plot.title = element_text(hjust = 0.5, face = "bold", size = 14)
)
)
combined_plot
Treatment Effects for 48 Hours: FP.
Respondents exposed to the Market condition consistently supported market-oriented SOE reform, whereas those assigned with the State condition demonstrated reduced support. Similarly, participants who received the Dove treatment preferred cooperation, while those given the Hawk condition did not favor cooperative policies. Due to limited statistical power, permutation tests were conducted to evaluate the effects of Treatment 1 and Treatment 2 against the null hypothesis of no effect on follow-up survey outcomes. The null hypothesis was rejected with high confidence for: comparisons between the State and Control conditions with Treatment 1, the Market and State conditions with Treatment 1, and the Dove and Hawk conditions with Treatment 2, with p-values of 0.001, 0.000, and 0.012, respectively. The treatments had a significant and lasting impact on respondents’ preferences 48 hours post-exposure. Detailed analysis can be found in the Appendix.
4 Implications
The study suggests that authoritarian regimes can effectively shift public attitudes by providing new frames for issues through government-controlled media. Media framing moves respondents to adopt policy positions aligned with those promoted by the government, whether the issue is domestic or foreign policy, and these effects persist up to 48 hours after exposure.
The study finds that framing is effective regardless of predispositions. Both pro-market and pro-state individuals shifted their economic policy preferences alongside the stance presented in the videotape treatment. Similarly, nationalist and anti-nationalist respondents also altered their foreign policy preferences according to the treatment conditions. Those with higher political knowledge are more susceptible to framing effects. Potentially, strategic framing communicated through state-controlled media can help authoritarian regimes achieve mainstream consensus and suppress alternative frames and contextual information.