Show each rater 2 "message-evaluation" pair
- tell them: an AI has been asked to objectively evaluate the veracity/accuracy of some text. It was asked to "rate the overall veracity of the paragraph on a scale of 0 (completely inaccurate) to 100 (completely accurate). For scores lower than 100, provide a very brief explanation of the inaccuracies in the paragraph."
- read the text and the AI's evaluation. then indicate the extent to which you agree with the AI's evaluation (0 to 100)
- also tell the rater the user's top priority - e.g., economy/integrity
how many raters
- 160 message
- 30 raters per message (4800)
- each rater rates 2 messages, so 2400 participants
Perplexity fact-check prompt
> You are a professional fact-checker, and you excel at objectively evaluating the veracity/accuracy of information. Rate the overall veracity of the paragraph on a scale of 0 (completely inaccurate) to 100 (completely accurate). For scores lower than 100, provide a very brief explanation of the inaccuracies in the paragraph.
![[1727291081.png]]
summary statistics for each of four bins, 16 cells (10 AI messages per cell)
```r
# pfc: perplexity fact-check
# pfc_tile: bins 1, 2, 3, 4
pfc_tile condition topic N pfc_min pfc_mean pfc_median pfc_max
<int> <char> <char> <int> <int> <num> <num> <int>
1: 1 persuadeHarris personality 10 25 50.5 50.0 70
2: 1 persuadeHarris policy 10 0 46.2 60.0 75
3: 1 persuadeTrump personality 10 0 30.3 24.0 75
4: 1 persuadeTrump policy 10 0 36.4 40.0 75
5: 2 persuadeHarris personality 10 80 85.0 85.0 90
6: 2 persuadeHarris policy 10 80 83.5 80.0 90
7: 2 persuadeTrump personality 10 80 83.8 81.5 90
8: 2 persuadeTrump policy 10 80 85.0 85.0 90
9: 3 persuadeHarris personality 10 90 91.5 90.0 95
10: 3 persuadeHarris policy 10 90 91.9 91.0 95
11: 3 persuadeTrump personality 10 90 90.5 90.0 95
12: 3 persuadeTrump policy 10 90 92.2 91.0 95
13: 4 persuadeHarris personality 10 95 99.3 100.0 100
14: 4 persuadeHarris policy 10 95 99.3 100.0 100
15: 4 persuadeTrump personality 10 95 99.0 100.0 100
16: 4 persuadeTrump policy 10 97 99.5 100.0 100
```