Fu, Xu, and Zhang (2024) investigate whether “pure” legality—law enforcement stripped of normative and substantive elements—can confer legitimacy to governmental institutions and activities. This study is contextualized within urban China, where the Party-state heavily invests in legal enforcement without accompanying political freedoms or legal checks on power.
This RMarkdown tutorial replicates the core analyses in “Does Legality Produce Political Legitimacy? An Experimental Approach”. The replication, conducted by Jinwen Wu, a predoctoral fellow at Stanford University, is supervised by Professor Yiqing Xu. The replication summarizes the main data analyses from the article; please refer to the original paper for a comprehensive understanding of the ideas presented.
Click the Code
button in the top right and select
Show All Code
to reveal all code used in this RMarkdown.
Click Show
in paragraphs to reveal the code used to
generate a finding. The original replication files can be downloaded
from here.
To isolate the effects of pure legality from other factors, the researchers conduct a multi-arm survey experiment. This design strips out the interfering effects by focusing on legal changes that attempt to restrict private rights and freedoms.The researchers analyzed the online survey experiment of 1,040 urban Chinese respondents in 2021 and found:
Based on the experimental results, the authors argued that: legality, especially professional training for law enforcement, can enhance political legitimacy. These findings challenge the conventional wisdom that legality without substantive rights cannot confer legitimacy.
The author defines “legality” as the quality of being lawful and legal. They distinguish legality from 3 related but distinct notions such as the rule of law, rule by law, and procedural justice. The differences are identified below:
To explore the connection between perceived legitimacy and government actions, the authors bifurcate the outcome variable (“legitimacy”) into two main questions:
This approach aims to capture both people’s policy-specific reactions and overall support for the regime.
The authors outline several pathways through which law might influence perceived legitimacy:
The core contribution of the research is to differentiate the effects of legality’s inherent qualities from those of other factors, such as the protection of substantive rights and freedoms, the mechanism of checks and balances against political power, the enhancement of governmental predictability, and the provision of procedural justice.
The study explores four treatments of governmental behavior that can shape perceived legitimacy: (1) whether the state issues formal legal rules for lower-level agents, (2) whether it publishes these rules to the public, (3) whether it invests in professional training for agents to enforce these rules accurately, and (4) whether agents respond to requests from affected parties to explain their actions.
The authors address how these factors, contingent on the first treatment, influence governance. They found that the third and fourth treatments (abbreviated as Training treatment and Response treatment in later analysis) boost the dependent variable by 0.15 and 0.3 standard deviations, respectively.
Different combinations of the treatments imply distinct lessons. For example, the first (denoted as Law treatment) and the third treatments represent Pure Legality Measurement, the second (denoted as Publication treatment) and the third correspond to the Legal System’s Social Transparency, and the fourth binary variable measures Procedural Justice.
The experiment includes the following four arms. Each respondent is randomized into one of theses four arms and each reads four fact patterns.
The difference between Arm A (Pure Control) and Arm B (Issuance Condition) represents the impact of issuing formal regulations on perceived legitimacy and responses to government actions. The researchers compare the average perceived legitimacy of case variants between Arm C (Pure Treatment) and Arm D (Individually Randomized) to test for spillover effects across fact patterns.
The experiment flowchart is attached below. This multi-arm design allows the researchers to distinguish support for specific government action and diffused (general) trust of the system while at the same time detect potential spillover effects from treatment embedded in fact pattern to another fact pattern.
Each respondent reads all four fact patterns of governmental control measures. The patterns are summarized below: - Regulation of Street-Side Vendors - Regulation of Fireworks Sales - Media Content Review (Television Series) - Online Speech Censorship
This research design allows the researchers to examine different combinations of the four treatments: - Issuance of formal legal rules - Publication of these rules - Professional training of enforcement agents - Responsiveness to private requests for explanation
At the end of each fact pattern, participants assess the legitimacy of the government’s actions. After reviewing all four scenarios, participants evaluate their overall level of trust in the regime.
The research design has three main strengths to bolsters the study’s validity and reliability.
First, the factorial design can isolate the outcome variable, perceived legitimacy, enhancing effects of each phase (issuance, publication, training, response).
Second, the researchers randomize respondents into treatment groups to test their immediate reaction to the four different fact patterns and the overall impact of the law on trust in the regime. The former represents specific support, while the latter reflects diffused trust. Collectively, both variables provide valuable insights into the treatment effects on perceived legitimacy.
Third, the treatment arms help detect and control for spillover effects, ensuring that early reactions do not bias subsequent responses.
The researchers are interested in testing five hypotheses. The difference between Arm A and Arm B reflects:
By comparing specific support responses to combinations of treatments within variations of Arm D, the researchers learn:
H2 (“Strengthening predictability through legality”): Government action is more legitimate when laws are consistently enforced and publicly disclosed.
H3 (“Legality for its inherent qualities”): Government action is more legitimate when enforcement officials receive professional training.
H4 (“Procedural justice”) : Government action is more legitimate when the government responds to private requests for explanation.
H5 (“Diffused trust”): Consistent exposure to richer legal characteristics increases trust in the regime.
The main outcome variables are the respondents’ perceived legitimacy for each enforcement action (specific support) and their trust in the fictional regime (diffused support). Both variables are assessed using a 0-3 scale ranging from “extremely unreasonable/untrustworthy” to “extremely reasonable/trustworthy.” The keybook of all variables in this analysis can be found in the ReadMe.pdf file in the replication folder.
Several R packages are required for the data analysis and visualization. The code chunk below checks for all required packages and installs the missing ones.
Packages: “readr”, “dplyr”, “tidyr”, “janitor”,“summarytools”, “estimatr”, “broom”, “ggplot2”, “glue”, “tibble”, “forcats”,“gridExtra”, “kableExtra”,“patchwork”,“vtable”, “knitr”.
install_all <- function(packages) {
installed_pkgs <- installed.packages()[, "Package"]
for (pkg in packages) {
if (!pkg %in% installed_pkgs) {
install.packages(pkg, repo = 'http://cran.us.r-project.org')
}
}
}
# Packages to be installed
packages <- c(
"readr", "dplyr", "tidyr", "janitor","summarytools",
"estimatr", "broom", "ggplot2", "glue", "tibble", "forcats","gridExtra", "kableExtra","patchwork","vtable", "knitr"
)
install_all(packages)
After installation, call to load the packages. Then, import the
experiment data. The file is in the Replication folder titled
data.csv
.
library(readr)
library(dplyr)
library(tidyr)
library(janitor)
library(estimatr)
library(broom)
library(ggplot2)
library(glue)
library(tibble)
library(forcats)
library(kableExtra)
library(patchwork)
library(vtable)
complete_recoded = read_csv("main.csv")
Before running the regressions, the authors checked covariate balance. The covariates are grouped into two categories - either demographic or attitude covariates. The table offers a broad overview of the characteristics of the 1,040 respondents representing the urban Chinese population.
# Identify covariates used for later regressions.
covariates_demo = c(
"age_demeaned",
"female_numerical",
"edu_numerical",
"income_numerical",
"party_numerical",
"knowledge_correct_total"
)
covariates_attitudes = c(
"attitude_justice_index",
"attitude_nationalism_index",
"attitude_liberalism_index",
"attitude_market_index",
"attitude_regime_index"
)
# Variables used in regressions
treatment_vars = c("transparency",
"training",
"response",
"case_variant_1")
interaction_vars = c(
"transparency * training",
"transparency * response",
"training * response",
"transparency * training * response"
)
# List of variables to calculate summary statistics
variables <- c("age", "female_numerical", "junior_college_numerical",
"high_school_numerical", "college_numerical",
"income", "class_numerical", "knowledge_correct_total",
"ethnicity_minority_numerical", "ccp",
"attitude_justice_index_scaled", "attitude_nationalism_index_scaled",
"attitude_liberalism_index_scaled", "attitude_market_index_scaled",
"attitude_regime_index_scaled")
# Corresponding names for the variables in the table
variable_names <- c("Age", "Female", "Junior College",
"High School", "College or Above",
"Income Category", "Self-Reported Social Class", "Political Knowledge",
"Ethnic Minority", "CCP Member",
"Ideology: Legality", "Ideology: Nationalism",
"Ideology: Liberalism", "Ideology: Market Economy",
"Regime Support")
# Function to generate summary statistics
func_gen_summary_stats <- function(data, var, variable) {
data %>%
summarise(
variable = variable,
count = sum(!is.na(.data[[var]])),
mean = round(mean(.data[[var]], na.rm = TRUE), 2),
sd = round(sd(.data[[var]], na.rm = TRUE), 2),
min = round(min(.data[[var]], na.rm = TRUE), 2),
max = round(max(.data[[var]], na.rm = TRUE), 2),
median = round(median(.data[[var]], na.rm = TRUE), 2)
) %>%
as.data.frame()
}
# Generate summary statistics for each variable
summary_stats <- do.call(rbind, Map(function(var, variable) func_gen_summary_stats(complete_recoded, var, variable), variables, variable_names))
rownames(summary_stats) <- NULL
kable(summary_stats, format = "html") %>%
kable_styling(full_width = F, bootstrap_options = c("striped", "hover", "condensed"))
variable | count | mean | sd | min | max | median |
---|---|---|---|---|---|---|
Age | 1040 | 37.44 | 12.12 | 19.00 | 61.00 | 36.00 |
Female | 1040 | 0.51 | 0.50 | 0.00 | 1.00 | 1.00 |
Junior College | 1040 | 0.14 | 0.34 | 0.00 | 1.00 | 0.00 |
High School | 1040 | 0.24 | 0.43 | 0.00 | 1.00 | 0.00 |
College or Above | 1040 | 0.20 | 0.40 | 0.00 | 1.00 | 0.00 |
Income Category | 1030 | 3.77 | 1.85 | 0.00 | 8.00 | 4.00 |
Self-Reported Social Class | 1040 | 1.26 | 0.70 | 0.00 | 3.00 | 1.00 |
Political Knowledge | 1040 | 2.62 | 1.90 | 0.00 | 5.00 | 3.00 |
Ethnic Minority | 1040 | 0.04 | 0.19 | 0.00 | 1.00 | 0.00 |
CCP Member | 1040 | 0.10 | 0.30 | 0.00 | 1.00 | 0.00 |
Ideology: Legality | 1040 | 0.00 | 1.00 | -3.18 | 2.87 | -0.15 |
Ideology: Nationalism | 1040 | 0.00 | 1.00 | -4.49 | 2.27 | -0.10 |
Ideology: Liberalism | 1040 | 0.00 | 1.00 | -3.59 | 4.18 | -0.14 |
Ideology: Market Economy | 1040 | 0.00 | 1.00 | -3.95 | 3.88 | -0.03 |
Regime Support | 1040 | 0.00 | 1.00 | -4.74 | 2.28 | 0.09 |
Replicating Table 3 in the article.
The researchers then segment the main dataset based on treatment
status. In this data preparation step, the
case_treat_status
variable is recoded to strings.
Additionally, several parameters, such as income and education, are
introduced to group the survey results. By doing so, the authors explore
potential heterogeneous treatment effects.
# Prepare the sub-datasets
case_level_treatments = complete_recoded %>%
select(contains("case_"), contains("attention_check_correct")) %>%
pivot_longer(contains("treat_status"),
names_to = "case1",
values_to = "case_treat_status") %>%
pivot_longer(contains("legitimacy"),
names_to = "case2",
values_to = "case_legitimacy") %>%
filter(substr(case1, 6, 6) == substr(case2, 6, 6)) %>%
select(-case1,-case2) %>%
mutate(case_number = rep(1:4, nrow(complete_recoded))) %>%
mutate(id = rep(1:nrow(complete_recoded), each = 4)) %>%
arrange(case_number, id) %>%
select(-id) %>%
pivot_longer(contains("_attention_check_correct"),
names_to = "case",
values_to = "attention_check_correct") %>%
mutate(case = case_when(
grepl("vendor", case) ~ 1,
grepl("fireworks", case) ~ 2,
grepl("drama", case) ~ 3,
grepl("speech", case) ~ 4
)) %>%
filter(case == case_number) %>%
select(case_treat_status,
case_number,
case_legitimacy,
attention_check_correct) %>%
type_convert()
complete_case_level_long = complete_recoded %>%
select(-contains("case_")) %>%
bind_rows(replicate(3, ., simplify = FALSE)) %>%
cbind(case_level_treatments)
# Create a dataset for case level regressions
lm_data = complete_case_level_long %>%
mutate(
law = case_when(case_treat_status == 1 ~ 0,
TRUE ~ 1),
case_variant_1 = case_when(case_treat_status == 1 ~ 1,
TRUE ~ 0), # case variant 1 dummy
transparency = case_when(case_treat_status %in% c(3, 4, 5, 9) ~ 1,
TRUE ~ 0),
training = case_when(case_treat_status %in% c(4, 5, 6, 7) ~ 1,
TRUE ~ 0),
response = case_when(case_treat_status %in% c(5, 7, 8, 9) ~ 1,
TRUE ~ 0)
) %>%
mutate(case_treat_text = case_when(
case_treat_status == 1 ~ "no law, no publication, no training, no response",
case_treat_status == 2 ~ "written law, no publication, no training, no response",
case_treat_status == 3 ~ "written law, publication, no training, no response",
case_treat_status == 4 ~ "written law, publication, training, no response",
case_treat_status == 5 ~ "written law, publication, training, response",
case_treat_status == 6 ~ "written law, no publication, training, no response",
case_treat_status == 7 ~ "written law, no publication, training, response",
case_treat_status == 8 ~ "written law, no publication, no training, response",
case_treat_status == 9 ~ "written law, publication, no training, response"
))
arm_level_treatment_four_digit_tally = complete_recoded %>%
group_by(arm_level_treat_status_four_digit) %>%
tally()
arm_level_treatment_assignment_tally = complete_recoded %>%
group_by(treatment_text) %>%
tally() %>%
ungroup %>%
mutate(treatment_prob = recode(treatment_text,
"full_control" = "1/6",
"full_no_law" = "1/6",
"full_treat" = "1/6",
"ind_random" = "1/2")) %>%
mutate(treatment_text = recode(treatment_text,
"full_control" = 'Four Counts of Case Variant 2 ("pure control")',
"full_no_law" = 'Four Counts of Case Variant 1 ("rule of man")',
"full_treat" = 'Four Counts of Case Variant 4 ("saturated")',
"ind_random" = 'Four Cases, Individually Randomized ("ind random")'))
## Create data subsets for robustness checks and heterogeneous effects
lm_data_ind_randomized = lm_data %>%
filter(treatment_text == "ind_random")
lm_data_correct_answers_only = lm_data %>%
filter(attention_check_correct == 1)
lm_data_college = lm_data %>%
filter(college_numerical == 1)
lm_data_no_college = lm_data %>%
filter(college_numerical == 0)
lm_data_case_1_street_vendors = lm_data %>%
filter(case_number == 1)
lm_data_case_2_fireworks_sales = lm_data %>%
filter(case_number == 2)
lm_data_case_3_web_series = lm_data %>%
filter(case_number == 3)
lm_data_case_4_online_speech = lm_data %>%
filter(case_number == 4)
lm_data_men = lm_data %>%
filter(female_numerical == 0)
lm_data_women = lm_data %>%
filter(female_numerical == 1)
lm_data_knowledge_high = lm_data %>%
filter(knowledge_correct_total %in% 3:4)
lm_data_knowledge_low = lm_data %>%
filter(knowledge_correct_total %in% 0:2)
lm_data_party_member = lm_data %>%
filter(ccp == 1)
lm_data_non_party_member = lm_data %>%
filter(ccp == 0)
lm_data_regime_high = lm_data %>%
filter(regime_half == 2)
lm_data_regime_low = lm_data %>%
filter(regime_half == 1)
lm_data_justice_high = lm_data %>%
filter(justice_half == 2)
lm_data_justice_low = lm_data %>%
filter(justice_half == 1)
lm_data_income_high = lm_data %>%
filter(income_half == 2)
lm_data_income_low = lm_data %>%
filter(income_half == 1)
The plot below examines the covariate balance in each arm using standardized mean differences. The covariates across the 9 groups are differentiated by treatment assignment. The estimates all cluster near 0, suggesting the randomizations are successful. The numbers can be found at Table A1(a) and Table A1(b) in the appendix.
standardized_data <- lm_data %>%
mutate(across(all_of(variables), ~ (.-mean(.))/sd(.)))
# Calculate the mean of each covariate for the 9 experimental groups
means <- standardized_data %>%
group_by(case_treat_text) %>%
summarize(across(all_of(variables), mean, na.rm = TRUE))
# Reshape data for plotting
means_long <- means %>%
pivot_longer(-case_treat_text, names_to = "Covariate", values_to = "Mean") %>%
mutate(Covariate = factor(Covariate, levels = variables, labels = variable_names))
# Define unique symbols and colors for the 9 groups
symbols <- c(16, 17, 18, 19, 20, 21, 22, 23, 24)
# Create the plot with adjusted legend position, layout, and colors
ggplot(means_long, aes(x = Mean, y = Covariate, shape = as.factor(case_treat_text))) +
geom_point(size = 3) +
scale_shape_manual(values = symbols) +
labs(title = "Covariate Balance",
x = "Standardized Mean Differences",
y = "") +
theme_minimal() +
theme(legend.position = "bottom",
legend.box = "horizontal",
legend.title = element_blank(),
legend.text = element_text(size = 8),
legend.key.size = unit(1, "lines"),
legend.spacing = unit(0.2, "lines"),
legend.margin = margin(0, 0, 0, 0)) +
guides(shape = guide_legend(ncol = 2)) +
scale_x_continuous(limits = c(-1, 1))
Figure 2 demonstrates how the four government actions — law issuance, law publication, enforcement official training, and responsiveness — shape people’s approval of the four legal changes. The control condition is the Arm A and represented by Case Variant 1 (Law, No Publication, No Training, No Response).
# Graphical Set-ups
caption_effect = "Treatment Effect on Perceived Legitimacy"
caption_effect_diffused = "Treatment Effect on Diffused Trust"
# colors ----
alpha = 0.05
size_pt = 2
size_conf_line = 1
font_plot = "Times New Roman"
ci_width = 0.15
ci_alpha = 1
color_pt = "#DD1A23"
theme_paper = function() {
theme_minimal(base_size = 16, base_family = "Times New Roman") %+replace%
theme(
axis.text.x = element_text(size = 15, color = "black",
margin = margin(20, 0, 0, 0)),
axis.text.y = element_text(size = 15, color = "black",
hjust = 1,
margin = margin(20, 20, 20, 20)),
axis.title.y = element_text(size = 12, angle = 90,
margin = margin(0, 5, 0, 0)),
axis.ticks = element_line(),
axis.ticks.length = unit(.25, "cm"),
panel.grid = element_line(
colour = "#c7c9c7",
size = 0.1,
linetype = "dashed"
),
legend.position = "none",
legend.title = element_blank(),
panel.border = element_rect(colour = "black", fill = NA, size = 1),
plot.margin = margin(40, 50, 20, 50),
# plot.title.position = "plot",
plot.title = element_text(
size = 20,
hjust = 0.5,
margin = margin(0, 0, 30, 0)
),
plot.subtitle = element_text(size = 11, hjust = 0.49,
margin=margin(0, 0, 30, 0), color="#7f7f7f"),
plot.caption = element_text(
hjust = 1,
color = "#A0A0A0",
size = 8,
lineheight = 1.2,
margin = margin(10, 0, 0, 0)
),
complete = TRUE
)}
theme_line = function () {
theme_minimal(base_size = 12, base_family = "Avenir") %+replace%
theme(
axis.text.x = element_text(size = 20,
margin = margin(0, 0, 0, 0)),
axis.text.y = element_text(size = 20, hjust = 1,
margin = margin(0, 0, 0, 0)),
axis.title = element_blank(),
panel.grid = element_line(
colour = "#8b8c8b",
size = 0.4,
linetype = "dashed"
),
legend.position = "none",
legend.title = element_blank(),
# panel.border = element_rect(colour = "black", fill = NA, size = 1),
plot.margin = margin(30, 50, 30, 50),
plot.title.position = "plot",
plot.title = element_text(
size = 18,
hjust = 0.5,
margin = margin(0, 0, 20, 0)
),
plot.subtitle = element_text(size = 11, hjust = 0.49,
margin=margin(0, 0, 30, 0), color="#7f7f7f"),
plot.caption = element_text(
hjust = 1,
color = "#A0A0A0",
size = 8,
lineheight = 1.2,
margin = margin(10, 0, 0, 0)
),
complete = TRUE
)
}
func_plot_treatment_effects = function(model_number = 1,
df = lm_data,
df_captions) {
if (model_number == 1) {
plot_order = treatment_vars
} else if (model_number == 2) {
plot_order = c(treatment_vars, covariates_demo)
} else if (model_number == 3) {
plot_order = c(treatment_vars, covariates_demo, covariates_attitudes)
} else if (model_number == 4) {
plot_order = c(treatment_vars, interaction_vars)
} else if (model_number == 5) {
plot_order = c(treatment_vars, interaction_vars, covariates_demo)
} else if (model_number == 6) {
plot_order = c(treatment_vars,
interaction_vars,
covariates_demo,
covariates_attitudes)
}
#' @param model_number An integer (1-6) specifying the model specification to use.
#' 1: Treatment variables only.
#' 2: Treatment variables and demographic covariates.
#' 3: Treatment variables, demographic covariates, and attitude covariates.
#' 4: Treatment variables and interaction variables.
#' 5: Treatment variables, interaction variables, and demographic covariates.
#' 6: Treatment variables, interaction variables, demographic covariates, and attitude covariates.
fmla = as.formula(glue("case_legitimacy ~ {paste(plot_order, collapse = ' + ')}"))
lm_obj = summary(lm_robust(fmla,
clusters = id,
df))
# Fit the linear model with robust standard errors clustered by id
lm_obj_df = lm_obj$coefficients %>%
as.data.frame() %>%
rownames_to_column(var = "variable") %>%
clean_names() %>%
filter(variable != "(Intercept)") %>%
mutate(variable = gsub(":", " * ", variable)) %>%
mutate(variable = fct_relevel(variable, plot_order)) %>%
mutate(
variable = recode(
variable,
"transparency" = "Publication",
"training" = "Training",
"response" = "Response",
"case_variant_1" = "No Law",
"transparency * training" = "Pub * Train",
"transparency * response" = "Pub * Resp",
"training * response" = "Train * Resp",
"transparency * training * response" = "Pub * Train * Resp",
"age_demeaned" = "age (demeaned)",
"female_numerical" = "female",
"edu_numerical" = "education (8-pt)",
"income_numerical" = "income (9-pt)",
"party_numerical" = "party member",
"knowledge_correct_total" = "knowledge (5-pt)",
"attitude_justice_index" = "rule of law (5-pt)",
"attitude_nationalism_index" = "nationalism (5-pt)",
"attitude_liberalism_index" = "liberalism (5-pt)",
"attitude_market_index" = "market economy (5-pt)",
"attitude_regime_index" = "regime support (5-pt)"
)
)
p = ggplot(lm_obj_df, aes(x = variable, y = estimate), alpha = 0.7) +
geom_point(size = size_pt) +
theme_paper() +
geom_errorbar(
aes(ymin = ci_lower,
ymax = ci_upper),
width = ci_width,
alpha = ci_alpha,
size = size_conf_line
) +
geom_hline(yintercept = 0,
linetype = "dashed",
alpha = 0.8) +
labs(y = "", x = "")
if (model_number == 1) {
p = p +
scale_y_continuous(limits = c(-0.25, 0.5))
} else {
p = p + scale_y_continuous(limits = c(-1, 1))
}
if (!is.na(df_captions)) {
p = p +
labs(title = glue("{df_captions}"))
}
return(p)
}
p = mapply(
func_plot_treatment_effects,
model_number = c(rep(1, 5)),
df = list(
lm_data,
lm_data_case_1_street_vendors,
lm_data_case_2_fireworks_sales,
lm_data_case_3_web_series,
lm_data_case_4_online_speech
),
df_captions = list(
NA,
"Fact Pattern 1: Street Vendors",
"Fact Pattern 2: Fireworks Sales",
"Fact Pattern 3: Web Series",
"Fact Pattern 4: Forum Posting"
),
SIMPLIFY = FALSE
)
p1 = p[[1]] +
labs(y = caption_effect)
ggsave(glue("output/01_treatment_effect.png"),
p1,
width = 12,
height = 6)
knitr::include_graphics("output/01_treatment_effect.png")
Replicating Figure 2 in the article.
As shown by the graph above, only Training and Response treatment significantly enhanced the perceived legitimacy, aligning with hypotheses H3 and H4. On a 0-3 scale, government training of enforcement officials increased perceived legitimacy by an average of 0.1 points (equivalent to 0.15 SD), while responsiveness to affected citizens raised it by over 0.2 points (equivalent to 0.3 SD). In contrast, neither the issuance nor the publication of formal laws and rules had any significant impact, contrary to hypotheses H1 and H2.
Next, the authors zoom in on how the treatment effects shape the dependent variable across all four fact patterns.
p2_text = ggplot(data.frame(l = caption_effect,
x = 1, y = 1)) +
geom_text(aes(x, y, label = l),
angle = 90,
size = 11,
family = font_plot) +
theme_void() +
coord_cartesian(clip = "off")
p2 = p2_text + (p[[2]] + p[[3]]) / (p[[4]] + p[[5]]) +
plot_layout(widths = c(1, 25)) +
plot_annotation(theme = theme(plot.margin = margin(rep(30, 4))))
ggsave(
glue("output/02_by_fact_pattern.png"),
p2,
width = 17,
height = 12
)
knitr::include_graphics("output/02_by_fact_pattern.png")
Replicating Figure 3 in the article.
Figure 3 supports the finding in Figure 2 that in all four scenarios, neither law issuance nor publication affected perceived legitimacy. However, both training and responsiveness increased legitimacy, except for training in the web series fact pattern, with responsiveness boosting legitimacy approximately twice as much as training.
The researchers use a four-arm experimental design to control for potential spillover effects. Intuitively, the treatment assigned in the previous fact pattern may have left an impression of the government on the respondents, thus impacting their answers to the next fact pattern. For example, if the government responded to people’s requests about the restriction of street-side vendors but not to media censorship, respondents might relate their answers to the latter scenario with the previous one and view the government’s action differently than if they were only seeing one fact pattern. The plot below compares the average level of perceived legitimacy across the four arms. The dotted lines connect the groups that received the same treatment set for all four fact patterns. If spillover contamination were present, there would be noticeable inconsistencies or unexpected changes between different arms.
## mean by case variant ----
lm_data_by_case_variant = lm_data %>%
# break the sample by variants "with response" and "without response"
mutate(case_treat_text_ex_resp = gsub("(.*)\\,.*", "\\1", case_treat_text)) %>%
mutate(case_treat_text_ex_resp = gsub(", ", "\n", case_treat_text_ex_resp)) %>%
mutate(case_treat_text_ex_resp = gsub("transparency", "publication", case_treat_text_ex_resp)) %>%
# break the sample by arm x variant (11 combinations)
mutate(case_treat_text_arm = glue("arm: {treatment_text}\n{case_treat_text}")) %>%
mutate(case_treat_text_arm = gsub(", ", "\n", case_treat_text_arm)) %>%
mutate(case_treat_text_arm = gsub(": ", "\n", case_treat_text_arm)) %>%
mutate(case_treat_text_arm = tools::toTitleCase(case_treat_text_arm)) %>%
mutate(case_treat_text_arm = gsub("Arm\nFull_no_law", "Arm A\n(No Law)", case_treat_text_arm)) %>%
mutate(
case_treat_text_arm = gsub(
"Arm\nInd_random",
"Arm D\n(Case-Level R)",
case_treat_text_arm
)
) %>%
mutate(
case_treat_text_arm = gsub(
"Arm\nFull_control",
"Arm B\n(Opaque Law)",
case_treat_text_arm
)
) %>%
mutate(
case_treat_text_arm = gsub(
"Arm\nFull_treat",
"Arm C\n(Full Combination)",
case_treat_text_arm
)
) %>%
mutate(case_treat_text_arm = factor(
case_treat_text_arm,
levels = c(
"Arm A\n(No Law)\nNo Law\nNo Publication\nNo Training\nNo Response",
"Arm B\n(Opaque Law)\nWritten Law\nNo Publication\nNo Training\nNo Response",
"Arm C\n(Full Combination)\nWritten Law\nPublication\nTraining\nResponse",
"Arm D\n(Case-Level R)\nNo Law\nNo Publication\nNo Training\nNo Response",
"Arm D\n(Case-Level R)\nWritten Law\nNo Publication\nNo Training\nNo Response",
"Arm D\n(Case-Level R)\nWritten Law\nPublication\nNo Training\nNo Response",
"Arm D\n(Case-Level R)\nWritten Law\nNo Publication\nTraining\nNo Response",
"Arm D\n(Case-Level R)\nWritten Law\nPublication\nTraining\nNo Response",
"Arm D\n(Case-Level R)\nWritten Law\nNo Publication\nNo Training\nResponse",
"Arm D\n(Case-Level R)\nWritten Law\nPublication\nNo Training\nResponse",
"Arm D\n(Case-Level R)\nWritten Law\nNo Publication\nTraining\nResponse",
"Arm D\n(Case-Level R)\nWritten Law\nPublication\nTraining\nResponse"
)
))
lm_data_by_case_variant_obj_df = lm_data_by_case_variant %>%
group_by(case_treat_text_arm, case_treat_text, treatment_text) %>%
summarize(
mean = mean(case_legitimacy),
ci_lower = mean(case_legitimacy) - qt(1 - 0.05 / 2, (n() - 1)) *
sd(case_legitimacy) / sqrt(n()),
ci_upper = mean(case_legitimacy) + qt(1 - 0.05 / 2, (n() - 1)) *
sd(case_legitimacy) / sqrt(n()),
n = n()
) %>%
ungroup
size_text_arm = 8
p3 = ggplot(lm_data_by_case_variant_obj_df,
aes(x = case_treat_text_arm,
y = mean)) +
geom_point(aes(shape = case_treat_text),
size = size_pt) +
theme_paper() +
geom_errorbar(
aes(ymin = ci_lower,
ymax = ci_upper),
width = ci_width,
alpha = ci_alpha,
size = size_conf_line
) +
theme_paper() +
scale_shape_manual(values = seq(0, 15)) +
scale_y_continuous(limits = c(1, 3)) +
geom_vline(xintercept = c(1.5, 2.5, 3.5),
linetype = "dotted") +
theme(
axis.text.x = element_blank(),
legend.position = "bottom",
legend.text = element_text(size = 13),
legend.key.height = unit(0.7, "cm"),
legend.key.width = unit(1.5, "cm")
) +
guides(shape = guide_legend(ncol = 3)) +
# add labels
annotate(
"text",
x = 1,
y = 2.75,
label = "Arm A",
family = font_plot,
size = size_text_arm
) +
annotate(
"text",
x = 2,
y = 2.75,
label = "Arm B",
family = font_plot,
size = size_text_arm
) +
annotate(
"text",
x = 3,
y = 2.75,
label = "Arm C",
family = font_plot,
size = size_text_arm
) +
annotate(
"text",
x = 8,
y = 2.75,
label = "Arm D",
family = font_plot,
size = size_text_arm
) +
annotate(
"text",
x = 1,
y = 2.6,
label = "(No Law)",
family = font_plot,
size = 5
) +
annotate(
"text",
x = 2,
y = 2.6,
label = "(Opaque Law)",
family = font_plot,
size = 5
) +
annotate(
"text",
x = 3,
y = 2.6,
label = "(Full Combo)",
family = font_plot,
size = 5
) +
annotate(
"text",
x = 8,
y = 2.6,
label = "(Case-Level Randomization)",
family = font_plot,
size = 5
) +
annotate(
geom = "curve",
x = 1,
y = lm_data_by_case_variant_obj_df$mean[[1]],
xend = 4,
yend = lm_data_by_case_variant_obj_df$mean[[4]],
curvature = .7,
linetype = "dashed",
color = "#808080"
) +
annotate(
geom = "curve",
x = 2,
y = lm_data_by_case_variant_obj_df$mean[[2]],
xend = 5,
yend = lm_data_by_case_variant_obj_df$mean[[5]],
curvature = .4,
linetype = "dashed",
color = "#808080"
) +
annotate(
geom = "curve",
x = 3,
y = lm_data_by_case_variant_obj_df$mean[[3]],
xend = 12,
yend = lm_data_by_case_variant_obj_df$mean[[12]],
curvature = -.08,
linetype = "dashed",
color = "#808080"
) +
labs(y = "Average Perceived Legitimacy", x = "")
ggsave(
glue("output/03_by_variant.png"),
p3,
width = 18,
height = 6
)
knitr::include_graphics("output/03_by_variant.png")
Replicating
Figure 1A in the article.
The average perceived legitimacy levels of respondents who received the same treatment set exhibit a relatively consistent pattern and do not show erratic differences. The error bars in the plot overlap across the different arms. Results from the three pairs of comparable groups show no deviations that would suggest contamination. In addition, the researchers tested regression models with treatment interaction terms shown in Table A5 and did not found spillover effects.
In addition to enhancing the legitimacy of specific government actions, the researchers suggests that “investments in various aspects of law also boosted the legitimacy of the fictional regime as a whole.” The code below shows the impact of each treatment on trust in the fictional regime, referred to as “diffused trust” in later analysis.
## diffused trust mean ----
diffused_trust = complete_recoded %>%
group_by(treatment_text) %>%
summarize(
mean = mean(w_trust),
ci_lower = mean(w_trust) - qt(1 - 0.05 / 2, (n() - 1)) *
sd(w_trust) / sqrt(n()),
ci_upper = mean(w_trust) + qt(1 - 0.05 / 2, (n() - 1)) *
sd(w_trust) / sqrt(n()),
n = n()
) %>%
ungroup %>%
mutate(
label = recode(
treatment_text,
"full_no_law" = glue("4 x no law (n = {n})"),
"full_control" = glue("4 x control (n = {n})"),
"ind_random" = glue("4 x individually randomized (n = {n})"),
"full_treat" = glue("4 x treat (n = {n})")
)
) %>%
mutate(
order = recode(
treatment_text,
"full_no_law" = 1,
"full_control" = 2,
"ind_random" = 3,
"full_treat" = 4
)
) %>%
mutate(label = fct_reorder(label, rev(order)))
## diffused trust treatment effect ----
diffused_lm = tidy(lm_robust(w_trust ~ treatment_text, data = complete_recoded))
diffused_lm_obj_df = diffused_lm %>%
mutate(term = gsub("treatment_text", "", term)) %>%
filter(term != "(Intercept)") %>%
mutate(term = fct_relevel(term, c(
"full_no_law", "ind_random", "full_treat"
))) %>%
mutate(
term = recode(
term,
"full_no_law" = "Arm A\n(No Law)",
"full_treat" = "Arm C\n(Full Combination)",
"ind_random" = "Arm D\n(Case-Level Randomization)"
)
) %>%
mutate(term = factor(
term,
levels = c(
"Arm A\n(No Law)",
"Arm C\n(Full Combination)",
"Arm D\n(Case-Level Randomization)"
)
))
# Plot
p5 = ggplot(diffused_lm_obj_df, aes(x = term, y = estimate), alpha = 0.7) +
geom_point(size = size_pt) +
theme_paper() +
geom_errorbar(
aes(ymin = conf.low,
ymax = conf.high),
width = ci_width,
alpha = ci_alpha,
size = size_conf_line
) +
geom_hline(yintercept = 0,
linetype = "dashed",
alpha = 0.8) +
labs(y = caption_effect_diffused, x = "") +
scale_y_continuous(limits = c(-1, 2.2))
ggsave(
glue("output/05_diffused.png"),
p5,
width = 12,
height = 6
)
knitr::include_graphics("output/05_diffused.png")
Replicating Figure 4 in the article.
Consistent with hypothesis H5, respondents in the Arm C(Pure Treatment) rated the regime as 0.7 points more trustworthy, translating to a 0.4 SD increase relative to the Arm B (Issuance Condition) baseline. According to the experiment result, a systemically more legalistic and procedurally just regime is perceived as more legitimate than those that did not invest in legality or did so only inconsistently.
The researchers examine how treatment effects vary based on respondent characteristics such as income, education, and their initial levels of regime support.
Figure 5(a) demonstrates that professional training and government response consistently increased legitimacy across nearly all participant subgroups, with no significant differences in effect size.
Similarly, the effects on diffused trust are consistent across subgroups, as shown in Figure 5(b). The robustness of these results suggests that the findings are generalizable beyond the sample of urban Chinese Internet users.
# hetero ----
# education, income, regime support, justice
func_plot_hetero_reg = function(model_number = 1,
df = lm_data,
label,
plot_number) {
if (model_number == 1) {
plot_order = treatment_vars
} else if (model_number == 2) {
plot_order = c(treatment_vars, covariates_demo)
} else if (model_number == 3) {
plot_order = c(treatment_vars, covariates_demo, covariates_attitudes)
} else if (model_number == 4) {
plot_order = c(treatment_vars, interaction_vars)
} else if (model_number == 5) {
plot_order = c(treatment_vars, interaction_vars, covariates_demo)
} else if (model_number == 6) {
plot_order = c(treatment_vars,
interaction_vars,
covariates_demo,
covariates_attitudes)
}
fmla = as.formula(glue("case_legitimacy ~ {paste(plot_order, collapse = ' + ')}"))
lm_obj = tidy(lm_robust(fmla,
clusters = id,
df)) %>%
mutate(label = label,
plot_number = plot_number)
return(lm_obj)
}
hetero_regs = do.call(
rbind,
mapply(
func_plot_hetero_reg,
model_number = 1,
plot_number = rep(1:4, each = 2),
df = list(
lm_data_income_high,
lm_data_income_low,
lm_data_college,
lm_data_no_college,
lm_data_regime_high,
lm_data_regime_low,
lm_data_justice_high,
lm_data_justice_low
),
label = list(
"Above Median",
"Below Median",
"College",
"No College",
"Above Median",
"Below Median",
"Above Median",
"Below Median"
),
SIMPLIFY = FALSE
)
)
func_plot_hetero = function(index, title) {
plot_df = hetero_regs %>%
filter(plot_number == index &
!grepl("Intercept", term)) %>%
mutate(term = fct_relevel(term, treatment_vars)) %>%
mutate(
term = recode(
term,
"transparency" = "Publication",
"training" = "Training",
"response" = "Response",
"case_variant_1" = "No Law"
)
)
p = ggplot(plot_df,
aes(
x = term,
y = estimate,
group = label,
shape = label
),
alpha = 0.7) +
geom_point(size = size_pt, position = position_dodge(width = 0.5)) +
theme_paper() +
geom_errorbar(
aes(ymin = conf.low,
ymax = conf.high),
width = ci_width,
alpha = ci_alpha,
size = size_conf_line,
position = position_dodge(width = 0.5)
) +
geom_hline(yintercept = 0,
linetype = "dashed",
alpha = 0.8) +
labs(y = "", x = "", title = title) +
scale_y_continuous(limits = c(-0.25, 0.5)) +
theme(legend.position = "top",
legend.text = element_text(size = 18)) +
scale_shape_manual(values = c(9, 16))
return(p)
}
hetero_plots = mapply(
index = 1:4,
title = c(
"A. By Income",
"B. By Education",
"C. By Predisposition: Regime Support",
"D. By Predisposition: Legality"
),
func_plot_hetero,
SIMPLIFY = FALSE
)
p4 = (hetero_plots[[1]] + hetero_plots[[2]]) / (hetero_plots[[3]] + hetero_plots[[4]])
p4_text = ggplot(data.frame(l = caption_effect,
x = 1, y = 1)) +
geom_text(aes(x, y, label = l),
angle = 90,
size = 11,
family = font_plot) +
theme_void() +
coord_cartesian(clip = "off")
p4 = p4_text + p4 +
plot_layout(widths = c(1, 25)) +
plot_annotation(theme = theme(plot.margin = margin(rep(30, 4))))
ggsave(
glue("output/04_hetero.png"),
p4,
width = 19,
height = 12
)
knitr::include_graphics("output/04_hetero.png")
Replicating
Figure 5(a) in the article.
# diffused hetero ----
func_plot_hetero_diffused_reg = function(model_number = 1,
df = lm_data,
label,
plot_number) {
fmla = as.formula(glue("w_trust ~ treatment_text"))
lm_obj = tidy(lm_robust(fmla,
clusters = id,
df)) %>%
mutate(label = label,
plot_number = plot_number)
return(lm_obj)
}
hetero_diffused_regs = do.call(
rbind,
mapply(
func_plot_hetero_diffused_reg,
model_number = 1,
plot_number = rep(1:4, each = 2),
df = list(
complete_recoded %>% filter(income_half == 2),
complete_recoded %>% filter(income_half == 1),
complete_recoded %>% filter(college_numerical == 1),
complete_recoded %>% filter(college_numerical == 0),
complete_recoded %>% filter(regime_half == 2),
complete_recoded %>% filter(regime_half == 1),
complete_recoded %>% filter(justice_half == 2),
complete_recoded %>% filter(justice_half == 1)
),
label = list(
"Above Median",
"Below Median",
"College",
"No College",
"Above Median",
"Below Median",
"Above Median",
"Below Median"
),
SIMPLIFY = FALSE
)
)
func_plot_diffused_hetero = function(index, title) {
plot_df = hetero_diffused_regs %>%
filter(plot_number == index &
!grepl("Intercept", term)) %>%
mutate(term = gsub("treatment_text", "", term)) %>%
filter(term != "(Intercept)") %>%
mutate(term = fct_relevel(term, c(
"full_no_law", "ind_random", "full_treat"
))) %>%
mutate(
term = recode(
term,
"full_no_law" = "Arm A\n(No Law)",
"full_treat" = "Arm C\n(Full Combination)",
"ind_random" = "Arm D\n(Case-Level Randomization)"
)
) %>%
mutate(term = factor(
term,
levels = c(
"Arm A\n(No Law)",
"Arm C\n(Full Combination)",
"Arm D\n(Case-Level Randomization)"
)
))
p = ggplot(plot_df,
aes(
x = term,
y = estimate,
group = label,
shape = label
),
alpha = 0.7) +
geom_point(size = size_pt, position = position_dodge(width = 0.5)) +
theme_paper() +
geom_errorbar(
aes(ymin = conf.low,
ymax = conf.high),
width = ci_width,
alpha = ci_alpha,
size = size_conf_line,
position = position_dodge(width = 0.5)
) +
geom_hline(yintercept = 0,
linetype = "dashed",
alpha = 0.8) +
labs(y = "", x = "", title = title) +
scale_y_continuous(limits = c(-2, 2)) +
theme(
legend.position = "top",
legend.text = element_text(size = 18),
axis.text.x = element_text(size = 14)
) +
scale_shape_manual(values = c(9, 16))
return(p)
}
diffused_hetero_plots = mapply(
index = 1:4,
title = c(
"A. By Income",
"B. By Education",
"C. By Predisposition: Regime Support",
"D. By Predisposition: Legality"
),
func_plot_diffused_hetero,
SIMPLIFY = FALSE
)
# Plot
p6 = (diffused_hetero_plots[[1]] + diffused_hetero_plots[[2]]) / (diffused_hetero_plots[[3]] + diffused_hetero_plots[[4]])
p6_text = ggplot(data.frame(l = caption_effect_diffused,
x = 1, y = 1)) +
geom_text(aes(x, y, label = l),
angle = 90,
size = 11,
family = font_plot) +
theme_void() +
coord_cartesian(clip = "off")
p6 = p6_text + p6 +
plot_layout(widths = c(1, 25)) +
plot_annotation(theme = theme(plot.margin = margin(rep(30, 4))))
ggsave(
glue("output/06_diffused_hetero.png"),
p6,
width = 19,
height = 12
)
knitr::include_graphics("output/06_diffused_hetero.png")
Replicating Figure 5(b) in the article.
With a survey experiment of 1,040 respondents representing urban Chinese, the study shows that pure legality can enhance political legitimacy, even when it does not limit the regime’s power or protect rights. This effect may be driven by professional training that ensures consistent law implementation. Surprisingly, investment in legality can boost legitimacy even without publishing legal rules in China. The generalizability of these findings beyond China remains uncertain. The researchers encourage future studies to investigate whether these patterns hold in other regions.