# descriptives 97 completed study ```r topic condition treatment N <char> <char> <char> <int> 1: policy persuadeHarris llm 26 2: policy persuadeHarris video 17 3: policy persuadeTrump llm 27 4: policy persuadeTrump video 27 ``` no. of messages: most people ended when the stop button showed up (i.e., 7 messages in total). One person had 65 messages! ```r N 7 9 11 13 15 19 23 27 65 36 7 2 1 1 1 1 2 1 ``` vote ```r vote N prop <char> <int> <num> 1: Kamala Harris 47 0.48453608 2: Donald Trump 39 0.40206186 3: I would not vote, bu... 4 0.04123711 4: I would not vote out... 3 0.03092784 5: Prefer not to say 3 0.03092784 6: Jill Stein 1 0.01030928 ``` top issues ```r topic perpolchoice N Nprop <char> <char> <int> <num> 1: policy Economy 39 0.40206186 2: policy Healthcare 9 0.09278351 3: policy Immigration 7 0.07216495 4: policy Democracy and election integrity 6 0.06185567 5: policy Abortion rights 5 0.05154639 6: policy Climate change 4 0.04123711 7: policy Racial equity and social justice 4 0.04123711 8: policy Education 3 0.03092784 9: policy LGBTQ rights 3 0.03092784 10: policy Leadership style 3 0.03092784 11: policy Role of federal government 3 0.03092784 12: policy Social security and medicare 3 0.03092784 13: policy Crime 2 0.02061856 14: policy Foreign policy and international relations 2 0.02061856 15: policy Taxation 2 0.02061856 16: policy Cybersecurity and foreign interference in US elections 1 0.01030928 17: policy Gun policy 1 0.01030928 ``` ![[1728931364.png]] ![[1728931376.png]] ![[1728931388.png]] # results ## figures ![[1728931687.png]] ![[1728931703.png]] ## models - outcome: lean/preference - treated: video (0), llm (1) - conditionC: proTrump (-0.5), proHarris (0.5) - `*_Z` variables: z-score variables ### specification 1: interaction model ```r > m0 <- feols(lean_bidentrump_2 ~ treated * conditionC * lean_bidentrump_1Z, d0) > summ(m0, estimate = est_string) term result <char> <char> 1: (Intercept) b = 47.21 [42.19, 52.23], p < .001 2: treated b = -3.32 [-10.03, 3.39], p = .328 3: conditionC b = -9.84 [-19.88, 0.21], p = .055 # video treatment was effective 4: lean_bidentrump_1Z b = 36.02 [31.05, 41.00], p < .001 5: treated × conditionC b = 3.74 [-9.68, 17.16], p = .581 # video more effective than llm? 6: treated × lean_bidentrump_1Z b = 4.63 [-2.09, 11.34], p = .175 7: conditionC × lean_bidentrump_1Z b = -5.09 [-15.05, 4.86], p = .312 8: treated × conditionC × lean_bidentrump_1Z b = 5.94 [-7.50, 19.37], p = .382 ``` ### specification 2: main effect model - if condition is proHarris, lean/preference variable is recoded: 100-x ```r > m1 <- feols(lean_bidentrump_2_v2 ~ treated * conditionC * lean_bidentrump_1_v2Z, d0) > summ(m1, estimate = est_string) term result <char> <char> 1: (Intercept) b = 53.14 [47.98, 58.30], p < .001 2: treated b = -2.49 [-9.27, 4.29], p = .467 # video is more effective than llm?? 3: conditionC b = -2.67 [-12.99, 7.65], p = .609 4: lean_bidentrump_1_v2Z b = 36.22 [31.21, 41.22], p < .001 5: treated × conditionC b = 5.20 [-8.36, 18.76], p = .448 6: treated × lean_bidentrump_1_v2Z b = 4.65 [-2.10, 11.40], p = .175 7: conditionC × lean_bidentrump_1_v2Z b = -5.12 [-15.13, 4.88], p = .312 8: treated × conditionC × lean_bidentrump_1_v2Z b = 5.97 [-7.54, 19.48], p = .382 ``` ## models - outcome: vote intentions/likelihood ```r # among all participants > m2 <- feols(vote_chance_2 ~ conditionC * vote_chance_1Z * treated, d0) > summ(m2, estimate = est_string) term result <char> <char> 1: (Intercept) b = 84.01 [80.69, 87.33], p < .001 2: conditionC b = 0.52 [-6.12, 7.16], p = .877 3: vote_chance_1Z b = 20.85 [17.50, 24.19], p < .001 4: treated b = 0.25 [-4.18, 4.67], p = .913 5: conditionC × vote_chance_1Z b = -8.19 [-14.88, -1.49], p = .017 6: conditionC × treated b = 2.99 [-5.87, 11.84], p = .505 # llm more effective 7: vote_chance_1Z × treated b = 1.72 [-2.76, 6.19], p = .448 8: conditionC × vote_chance_1Z × treated b = 1.17 [-7.78, 10.13], p = .795 ``` subset participants who were assigned to the condition that's consistent with their initial lean/preference - (Harris supporters in proHarris condition) or (Trump supporters in proTrump condition) ```r > m2_votemore <- feols(vote_chance_2 ~ conditionC * vote_chance_1Z * treated, d0[vote_more == 1]) > summ(m2_votemore, estimate = est_string) term result <char> <char> 1: (Intercept) b = 83.49 [79.53, 87.46], p < .001 2: conditionC b = -8.54 [-16.48, -0.60], p = .036 3: vote_chance_1Z b = 25.40 [20.90, 29.89], p < .001 4: treated b = 2.42 [-2.67, 7.51], p = .342 5: conditionC × vote_chance_1Z b = 9.12 [0.12, 18.11], p = .047 6: conditionC × treated b = 15.58 [5.40, 25.77], p = .004 # llm more effective at getting people to vote 7: vote_chance_1Z × treated b = -6.61 [-12.17, -1.05], p = .021 8: conditionC × vote_chance_1Z × treated b = -22.51 [-33.63, -11.39], p < .001 ``` subset participants who were assigned to condition that's inconsistent with their initial lean - (Harris supporters in proTrump condition) or (Trump supporters in proHarris condition) ```r > m2_votemoreno <- feols(vote_chance_2 ~ conditionC * vote_chance_1Z * treated, d0[vote_more == 0]) > summ(m2_votemoreno, estimate = est_string) term result <char> <char> 1: (Intercept) b = 80.22 [75.47, 84.98], p < .001 2: conditionC b = 5.65 [-3.85, 15.16], p = .237 3: vote_chance_1Z b = 22.59 [18.33, 26.85], p < .001 4: treated b = 3.17 [-2.85, 9.19], p = .294 5: conditionC × vote_chance_1Z b = -22.70 [-31.22, -14.18], p < .001 6: conditionC × treated b = -4.12 [-16.16, 7.92], p = .494 # llm no more effective than video at geting people to vote 7: vote_chance_1Z × treated b = 2.43 [-3.30, 8.17], p = .397 8: conditionC × vote_chance_1Z × treated b = 18.85 [7.37, 30.32], p = .002 ``` 4-way interaction model to see if the two conditionC x treated effects above are significantly different (yes!) ```r > m2_votemore_interact <- feols(vote_chance_2 ~ conditionC * vote_chance_1Z * treated * vote_more, d0) > summ(m2_votemore_interact, estimate = est_string) term result <char> <char> 1: (Intercept) b = 80.22 [75.82, 84.63], p < .001 2: conditionC b = 5.65 [-3.15, 14.46], p = .205 3: vote_chance_1Z b = 22.59 [18.64, 26.53], p < .001 4: treated b = 3.17 [-2.40, 8.75], p = .261 5: vote_more1 b = 3.27 [-2.83, 9.37], p = .289 6: conditionC × vote_chance_1Z b = -22.70 [-30.59, -14.81], p < .001 7: conditionC × treated b = -4.12 [-15.27, 7.03], p = .464 8: vote_chance_1Z × treated b = 2.43 [-2.88, 7.75], p = .365 9: conditionC × vote_more1 b = -14.20 [-26.39, -2.00], p = .023 10: vote_chance_1Z × vote_more1 b = 2.81 [-3.39, 9.01], p = .370 11: treated × vote_more1 b = -0.75 [-8.52, 7.02], p = .849 12: conditionC × vote_chance_1Z × treated b = 18.85 [8.22, 29.47], p = .001 13: conditionC × vote_chance_1Z × vote_more1 b = 31.82 [19.42, 44.21], p < .001 # sig 14: conditionC × treated × vote_more1 b = 19.71 [4.17, 35.25], p = .014 15: vote_chance_1Z × treated × vote_more1 b = -9.04 [-16.99, -1.09], p = .026 16: conditionC × vote_chance_1Z × treated × vote_more1 b = -41.36 [-57.25, -25.46], p < .001 ``` # attrition chi-square and fisher's tests ```r condition treatment n_start n_finish prop_finish <char> <char> <int> <int> <num> 1: persuadeHarris llm 33 26 78.78788 2: persuadeHarris video 24 17 70.83333 3: persuadeTrump llm 32 27 84.37500 4: persuadeTrump video 30 27 90.00000 persuadeHarris persuadeTrump llm 78.78788 84.375 video 70.83333 90.000 X-squared = 0.4292, df = 1, p-value = 0.5124 Fisher's exact test for count data: p-value = 0.3148 ``` predict attrition status (1: attrit, 0: finished) ```r > summ(feglm(attrit ~ condition, raw, family = binomial), estimate = est_string) term result <char> <char> 1: (Intercept) b = -1.12 [-1.73, -0.52], p < .001 2: conditionpersuadeTrump b = -0.79 [-1.75, 0.17], p = .108 > summ(feglm(attrit ~ treatment, raw, family = binomial), estimate = est_string) term result <char> <char> 1: (Intercept) b = -1.49 [-2.11, -0.86], p < .001 2: treatmentvideo b = 0.00 [-0.93, 0.94], p = .994 > summ(feglm(attrit ~ treatment * condition, raw, family = binomial), estimate = est_string) term result <char> <char> 1: (Intercept) b = -1.31 [-2.16, -0.47], p = .002 2: treatmentvideo b = 0.42 [-0.80, 1.65], p = .498 3: conditionpersuadeTrump b = -0.37 [-1.66, 0.91], p = .568 4: treatmentvideo × conditionpersuadeTrump b = -0.94 [-2.91, 1.04], p = .353 > summ(feglm(attrit ~ lean_bidentrump_1, raw, family = binomial)) term result <char> <char> 1: (Intercept) b = -3.13 (0.72) [-4.54, -1.72] p < .001 2: lean_bidentrump_1 b = 0.01 (0.01) [-0.01, 0.03] p = .507 > summ(feglm(attrit ~ condition * lean_bidentrump_1 * treatment, raw, family = binomial)) term result <char> <char> 1: (Intercept) b = -23.88 (36.12) [-94.67, 46.91] p = .508 2: conditionpersuadeTrump b = 22.27 (36.13) [-48.54, 93.08] p = .538 3: lean_bidentrump_1 b = 0.22 (0.37) [-0.49, 0.94] p = .541 4: treatmentvideo b = 20.75 (36.15) [-50.11, 91.61] p = .566 5: conditionpersuadeTrump × lean_bidentrump_1 b = -1.69 (410.95) [-807.13, 803.75] p = .997 6: conditionpersuadeTrump × treatmentvideo b = -240.93 (40212.58) [-79056.14, 78574.29] p = .995 7: lean_bidentrump_1 × treatmentvideo b = -0.21 (0.37) [-0.92, 0.51] p = .574 8: conditionpersuadeTrump × lean_bidentrump_1 × treatmentvideo b = 3.88 (574.96) [-1123.03, 1130.78] p = .995 ```