# descriptives
no. of participants
- started survey: 777
- finished all questions leading up to treatment/chat: 128
- completed survey: 128
distribution of political orient
```r
orient N
<num> <int>
1: 1 10
2: 2 29
3: 3 17
4: 4 32
5: 5 17
6: 6 14
7: 7 9
```
![[1745524027.png]]
![[1745523564.png]]
# randomization checks and attrition analysis
- no significant chi.square tests (randomization to 2 conditions, 2 strategies, 3 models seem fine)
```r
# initial assignment
strategy baseline nofacts
condition model
persuadeCarney deepseek/deepseek-chat-v3-0324 54 64
meta-llama/llama-4-maverick 57 69
openai/gpt-4.1 80 56
persuadePoilievre deepseek/deepseek-chat-v3-0324 67 65
meta-llama/llama-4-maverick 57 68
openai/gpt-4.1 83 57
# completed study
strategy baseline nofacts
condition model
persuadeCarney deepseek/deepseek-chat-v3-0324 7 10
meta-llama/llama-4-maverick 9 12
openai/gpt-4.1 14 7
persuadePoilievre deepseek/deepseek-chat-v3-0324 7 11
meta-llama/llama-4-maverick 15 12
openai/gpt-4.1 14 10
```
## among those who completed
there are differences in **pre-treatment variables**
- those assigned to llama model or gpt4 were less likely to vote than those assigned to deepseek
```r
r
gt; feols(vote_chance_1 ~ condition + model + strategy, data = d2, se = "HC1")
OLS estimation, Dep. Var.: vote_chance_1
Observations: 128
Standard-errors: Heteroskedasticity-robust
Estimate Std. Error t value Pr(>|t|)
(Intercept) 94.03425 3.95499 23.776093 < 2.2e-16 ***
conditionpersuadePoilievre 1.58553 5.00221 0.316967 0.7518060
modelmeta-llama/llama-4-maverick -14.47103 5.43669 -2.661736 0.0088122 ** # vs deepseek
modelopenai/gpt-4.1 -11.03695 5.66103 -1.949637 0.0534951 . # vs deepseek
strategynofacts -5.70182 5.22858 -1.090511 0.2776196
```
but not significant difference for candidate preference (pre-treatment)
```r
rgt; feols(lean_carneypoilievre_1 ~ condition + model + strategy, data = d2, se = "HC1")
OLS estimation, Dep. Var.: lean_carneypoilievre_1
Observations: 128
Standard-errors: Heteroskedasticity-robust
Estimate Std. Error t value Pr(>|t|)
(Intercept) 45.432231 8.77758 5.175939 8.9547e-07 ***
conditionpersuadePoilievre 5.515555 6.58032 0.838190 4.0355e-01
modelmeta-llama/llama-4-maverick -0.905967 8.51229 -0.106430 9.1541e-01
modelopenai/gpt-4.1 -11.476572 8.40870 -1.364845 1.7479e-01
strategynofacts -3.257528 6.60992 -0.492824 6.2302e-01
```
some difference for political orientation
```r
rgt; feols(orient ~ condition + model + strategy, data = d2, se = "HC1")
OLS estimation, Dep. Var.: orient
Observations: 128
Standard-errors: Heteroskedasticity-robust
Estimate Std. Error t value Pr(>|t|)
(Intercept) 4.045857 0.388052 10.426061 < 2.2e-16 ***
conditionpersuadePoilievre 0.423655 0.297893 1.422173 0.157507
modelmeta-llama/llama-4-maverick -0.079263 0.379498 -0.208864 0.834900
modelopenai/gpt-4.1 -0.691993 0.387013 -1.788038 0.076232 .
strategynofacts -0.534799 0.297126 -1.799909 0.074326 .
```
no difference for other variables: age, interest in politics, ai trust/use, political involvement, importance of considering candidate's policy positions, social conservatism, economic conservatism
# use of factual claims and rationality
no. of factual claims used in AI messages
```python
┌──────────┬──────────────────┐
│ strategy ┆ n_factual_claims │
│ --- ┆ --- │
│ str ┆ f64 │
╞══════════╪══════════════════╡
│ baseline ┆ 3.879167 │
│ nofacts ┆ 1.596154 │
└──────────┴──────────────────┘
shape: (12, 4)
┌──────────┬───────────────────┬────────────────────────────────┬──────────────────┐
│ strategy ┆ condition ┆ llm ┆ n_factual_claims │
│ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ str ┆ str ┆ f64 │
╞══════════╪═══════════════════╪════════════════════════════════╪══════════════════╡
│ baseline ┆ persuadeCarney ┆ deepseek/deepseek-chat-v3-0324 ┆ 4.185185 │
│ baseline ┆ persuadeCarney ┆ meta-llama/llama-4-maverick ┆ 3.914286 │
│ baseline ┆ persuadeCarney ┆ openai/gpt-4.1 ┆ 3.75 │
│ baseline ┆ persuadePoilievre ┆ deepseek/deepseek-chat-v3-0324 ┆ 3.2 │
│ baseline ┆ persuadePoilievre ┆ meta-llama/llama-4-maverick ┆ 4.185185 │
│ baseline ┆ persuadePoilievre ┆ openai/gpt-4.1 ┆ 3.829787 │
│ nofacts ┆ persuadeCarney ┆ deepseek/deepseek-chat-v3-0324 ┆ 1.0 │
│ nofacts ┆ persuadeCarney ┆ meta-llama/llama-4-maverick ┆ 1.5 │
│ nofacts ┆ persuadeCarney ┆ openai/gpt-4.1 ┆ 1.0 │
│ nofacts ┆ persuadePoilievre ┆ deepseek/deepseek-chat-v3-0324 ┆ 1.5 │
│ nofacts ┆ persuadePoilievre ┆ meta-llama/llama-4-maverick ┆ 1.95 │
│ nofacts ┆ persuadePoilievre ┆ openai/gpt-4.1 ┆ 1.5 │
└──────────┴───────────────────┴────────────────────────────────┴──────────────────┘
```
"use of rationality" in AI messages
```python
shape: (2, 2)
┌──────────┬─────────────┐
│ strategy ┆ rationality │
│ --- ┆ --- │
│ str ┆ f64 │
╞══════════╪═════════════╡
│ baseline ┆ 78.65019 │
│ nofacts ┆ 55.595918 │
└──────────┴─────────────┘
shape: (12, 4)
┌──────────┬───────────────────┬────────────────────────────────┬─────────────┐
│ strategy ┆ condition ┆ llm ┆ rationality │
│ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ str ┆ str ┆ f64 │
╞══════════╪═══════════════════╪════════════════════════════════╪═════════════╡
│ baseline ┆ persuadeCarney ┆ deepseek/deepseek-chat-v3-0324 ┆ 82.857143 │
│ baseline ┆ persuadeCarney ┆ meta-llama/llama-4-maverick ┆ 82.083333 │
│ baseline ┆ persuadeCarney ┆ openai/gpt-4.1 ┆ 81.339286 │
│ baseline ┆ persuadePoilievre ┆ deepseek/deepseek-chat-v3-0324 ┆ 72.321429 │
│ baseline ┆ persuadePoilievre ┆ meta-llama/llama-4-maverick ┆ 76.59322 │
│ baseline ┆ persuadePoilievre ┆ openai/gpt-4.1 ┆ 76.982143 │
│ nofacts ┆ persuadeCarney ┆ deepseek/deepseek-chat-v3-0324 ┆ 49.894737 │
│ nofacts ┆ persuadeCarney ┆ meta-llama/llama-4-maverick ┆ 58.75 │
│ nofacts ┆ persuadeCarney ┆ openai/gpt-4.1 ┆ 55.178571 │
│ nofacts ┆ persuadePoilievre ┆ deepseek/deepseek-chat-v3-0324 ┆ 53.295455 │
│ nofacts ┆ persuadePoilievre ┆ meta-llama/llama-4-maverick ┆ 61.914894 │
│ nofacts ┆ persuadePoilievre ┆ openai/gpt-4.1 ┆ 52.625 │
└──────────┴───────────────────┴────────────────────────────────┴─────────────┘
```