Prompted [google gemini](https://openrouter.ai/google/gemini-flash-1.5) to make two evaluations
- `llm_opinion_vs_claim`: whether llm thinks it's mostly opinion (0) or mostly factual claim (100)
- `llm_accuracy`: whether llm thinks it's mostly inaccurate (0) or mostly inaccurate (100)
# validations with different datasets
## jenny 90 headlines
only 3 headlines where LLM thinks it's definitely an opinion (llm_opinion_vs_claim = 0)
```r
llm_opinion_vs_claim N
<int> <int>
1: 0 3
2: 10 3
3: 20 23
4: 50 4
5: 60 7
6: 80 39
7: 90 9
8: 100 2
```
6 headlines where llm_opinion_vs_claim <= 10 (LLM thinks it's likely to be opinion)
```r
content llm_opinion_vs_claim llm_accuracy
<char> <int> <int>
1: Opinion | People are wrecking their bodies trying to 'detox' from the Covid vaccine 0 0
2: Ep. 1731 Stunning Vaccine Info Emerges - The Dan Bongino Show 0 0
3: Jab Or Journalism?! 0 0
4: The culprit in COVID and vaccine harm: Micro blood clots 10 10
5: DOD/Fauci Caught In Massive Scandal Hiding Death And Damage From COVID Vaccines 10 10
6: CDC: Pregnant Women Are Dropping Likes Flies - News Punch 10 10
```
different measures of misinformation classification in jenny's data (1: misinfo, 0: not misinfo), compared with LLM evaluations
```r
classification llm_opinion_vs_claim llm_accuracy
<num> <num> <num>
1: 0.0 77.82609 80.43478
2: 0.5 52.63158 50.52632
3: 1.0 26.40000 17.20000
classification_agree llm_opinion_vs_claim llm_accuracy
<int> <num> <num>
1: 0 70.46154 71.69231
2: 1 26.40000 17.20000
mis_bin llm_opinion_vs_claim llm_accuracy
<num> <num> <num>
1: 0.0 77.95918 80.20408
2: 0.5 43.88889 40.55556
3: 1.0 27.39130 18.69565
```
![[1737501120.png]]
## tucker data 245 headlines
`modal_fact_checker_rating`: human fact-checker evaluation
```python
┌───────────────────────────┬──────────────────────┬──────────────┐
│ modal_fact_checker_rating ┆ llm_opinion_vs_claim ┆ llm_accuracy │
│ --- ┆ --- ┆ --- │
│ str ┆ f64 ┆ f64 │
╞═══════════════════════════╪══════════════════════╪══════════════╡
│ False/Misleading ┆ 28.793103 ┆ 23.103448 │
│ No Mode ┆ 31.818182 ┆ 25.454545 │
│ Could Not Determine ┆ 50.0 ┆ 47.5 │
│ True ┆ 68.136646 ┆ 65.807453 │
└───────────────────────────┴──────────────────────┴──────────────┘
```
Opinion (0) or claim (100)? LLM rated 8 headlines as being opinion (llm_opinion_vs_claim = 0).
```r
llm_opinion_vs_claim N
<int> <int>
1: 0 8
2: 10 4
3: 20 77
4: 30 3
5: 50 9
6: 60 15
7: 70 5
8: 80 105
9: 90 13
10: 100 6
```
Recode
- 0: false/misleading
- 0.5: no mode, could not determine
- 1: true
```r
fact_check_rating modal_fact_checker_rating N
<num> <char> <int>
1: 0.0 False/Misleading 58
2: 0.5 Could Not Determine 4
3: 0.5 No Mode 22
4: 1.0 True 161
```
```r
fact_check_rating llm_opinion_vs_claim llm_accuracy
<num> <num> <num>
1: 0.0 28.79310 23.10345
2: 0.5 34.61538 28.84615
3: 1.0 68.13665 65.80745
```
![[1737503326.png]]
## crowdtangle 900 headlines
Only gave LLM the headline without the lede (so it's more like a tweet).
201 headlines were definitely considered opinions.
```r
llm_opinion_vs_claim N
<int> <int>
1: 0 201
2: 5 1
3: 10 20
4: 20 182
5: 30 32
6: 50 63
7: 60 56
8: 70 34
9: 80 233
10: 90 54
11: 100 14
```
model: gemini
![[1737498663.png]]
model: nova
![[1737520273.png]]
only headlings that LLM thinks is definitely not opinion (llm_opinion_vs_claim > 0)
![[1737498924.png]]
only headlines with fact-checker ratings
![[1737498450.png]]
## community notes 3000 english tweets
684 tweets are considered opinion.
```r
llm_opinion_vs_claim N
<int> <int>
1: 0 684
2: 5 1
3: 10 128
4: 20 1095
5: 30 219
6: 40 4
7: 50 202
8: 60 190
9: 70 70
10: 80 320
11: 90 37
12: 100 54
```
2156 notes were classifed as misinformed/misleading by community raters
```r
classification N
<char> <int>
1: MISINFORMED_OR_POTENTIALLY_MISLEADING 2156
2: NOT_MISLEADING 848
```
```r
misleading llm_opinion_vs_claim llm_accuracy
<num> <num> <num>
1: 0 29.89387 26.06722
2: 1 30.28525 25.41512
```
Correlations are quite bad, but hard to know if it's because posts that are identified for fact-checking are more likely to be misleading to begin with...
```r
claim_v_opinion llm_accuracy
<char> <num>
1: claim\n(llm_opinion_vs_claim>50) 60.46945
2: opinion\n(llm_opinion_vs_claim<=50) 15.57008
```
![[1737502631.png]]
examples
```r
llm_accuracy < 10
content llm_opinion_vs_claim llm_accuracy
<char> <int> <int>
1: Why so mad bro? Avi wants to be famous for this, lets help him. 0 0
2: Trying to think of the last time a major candidate for anything has refused to release cash on hand numbers 20 0
3: A trillion dollars for Ukraine, Israel, and invaders, but nothing, not even the crumbs, are left for YOU! 20 0
4: BREAKING: Jordan Peterson has been found dead by an apparent suicide at the age of 61. 50 0
5: The memes know 0 0
6: 60% of population in Germany dont speak German.Actually sad when you live in the country. 50 0
7: How is AOC worth $29 million?! Her salary is $155,000 per year. 50 0
8: Type “Nig” and let your phone complete the sentence. DONT CHEAT! \U{01faf5}🏽📷 0 0
9: You have said the actual truth 0 0
10: how ironic 0 0
40 <= llm_accuracy <= 60
content llm_opinion_vs_claim llm_accuracy
<char> <int> <int>
1: 28 straight wins in conference play for 😤 The longest streak in SEC history 👏 50 50
2: Here's Donald trump, saying ""laws allow a baby to be born...in the ninth month. It is wrong."" Show this to people who question President Biden's fitness. Stop giving trump a pass, FFS. 20 50
3: Israel just shut down Al Jazeera and raided its offices. Journalism like this is the reason why: 50 50
4: Baby in the oven? This reminds me of when Israeli terrorists burned a Palestinian baby, Ali Dawabshe, then danced and sang ""Ali's on the grill"" to taunt his family. 20 50
5: Cameraman at the World Cup dropping the smoothest transition of all time 50 50
6: An update from this week’s meeting with the Reigate Business Guild, joined by the Police, to discuss and work to address crime in the Town 50 50
7: China and Russia aren’t the ones sending the National Guard to beat college kid’s sculls into submission. 50 50
8: one million ETH bridged to — and it's still day one 50 50
9: Late Major Mukund Varadarajan is a hero. This nation stands on the foundation of men like him and the families they leave behind. 20 50
10: The $TRENCHAI Distribution Phase 1 is now live. 💚 Users have 24 hours to check their eligibility and claim their rewards. More information: 50 50
llm_accuracy > 90
content llm_opinion_vs_claim llm_accuracy
<char> <int> <int>
1: Who agrees we deserve a government that prioritizes the well-being of American citizens just as much as it supports other nations like Ukraine? 10 100
2: Today, on Transgender Day of Visibility, I have a simple message to all trans Americans: I see you. You are made in the image of God, and you're worthy of respect and dignity. 20 100
3: Is England a real country 50 100
4: Feels like the genre of ""ARPG"" is starting to mean a lot of different things. Much like ""Souls-like"" and ""Rogue-like"", I wonder if we could normalize ""Diablo-like"" for ARPGs that follow the Diablo formula... 20 100
5: Gary Neville: ""i know Arsenal have conceded the fewest goals and kept the most clean sheets but why is Ben White still being played at Right back when Arteta bought him from Brighton as a Central Defender, Why?"" 20 100
6: Olivia Rodrigo has concluded the GUTS World Tour. 100 100
7: Bronny James at halftime: 22 points 4 rebounds 3 assists 66% FG +18 100 100
8: I am in a wheelchair due to an accident that resulted in a spinal cord injury. Even though I have a disability, I try many things. I would be happy if I could bring courage and energy to everyone! 20 100
9: David Hogg, Parkland shooting survivor and co-founder of March for our Lives, calls for America’s youth to step up and fight against gun violence. ""They tell us to run, we are done running away from this issue."" Follow along for live updates: 80 100
10: DELUSION MUST END': Professional wrestler and influencer makes his opinion known after a women's Olympic fighter dropped out a match just seconds into it after being forced to compete against a biological man. 20 100
```
# descriptives for mohsen elite tweets (no ground truth)
225 elites, 11,126 tweets
## opinion vs claim
```r
llm_opinion_vs_claim N N_perc
<num> <num> <num>
1: 0 2083 18.722 # LLM classifed many as definitely opinion
2: 5 1 0.009
3: 10 260 2.337
4: 20 3789 34.055
5: 30 1302 11.702
6: 40 7 0.063
7: 50 1096 9.851
8: 60 570 5.123
9: 70 485 4.359
10: 75 1 0.009
11: 80 1224 11.001
12: 90 184 1.654
13: 100 124 1.115
```
Example tweets (randomly sampled)
```
llm_opinion_vs_claim == 0 (definitely opinion)
[1] "I am Spartacus"
[2] "What a complete disaster that endangers citizens of Portland!"
[3] "Tracks."
[4] "DRILL, BABY DRILL. That’s how you lower energy costs."
[5] "Hey, Congress, did you all approve this? Asking for millions of friends. 😳"
[6] "One more victory on the road to Houston ..Go horns!🤘"
[7] "Couldn't have said it better to a complete idiot who knows zero about international politics."
[8] "congrats to and the amazing CNN teams behind these informative town halls last night!"
[9] "Huge news, y'all. I mean, it shouldn't have even come to this. Not even close. Ms. Watts deserved compassionate healthcare, not prosecution. BansOffOurBodies"
[10] "Vivek Ramaswamy is one sick boy. Taylor Swift needs no “artificially propped up”"
llm_opinion_vs_claim == 50 (opinion and claim?)
[1] "I sat down with activist Hebel Morales to discuss Cuban-American relations on today’s Firelight Chat. Marianne2024"
[2] "And I would add Aruba to this…go to the State Dept Human Trafficking report and look at Aruba"
[3] "Everyone should have access to safe, stable, and affordable housing. Yet rising home prices and high-interest rates continue to widen the affordability gap for first-generation homeowners. I was proud to cut the ribbon on our latest housing investment, Providence Gardens."
[4] "Join me at The Bookery in Manchester, NH on Saturday, January 13. More info at"
[5] "In South Carolina campaigning for 😀🇺🇸"
[6] "And just like that, winter is here. Please avoid traveling today if possible and stay safe and warm!"
[7] "Congratulations to President Steve Schulz on his upcoming retirement. He has devoted over 40 years to education in Iowa & improving skills-based learning opportunities. Thank you for your dedication & cheers to your next chapter beginning in May!"
[8] "TUNE IN: I am speaking on the floor of the wishing Senator well. Today is his last day with us in the chamber for this session, as he prepares for a 9-month deployment to Kuwait. Live here:"
[9] "Thank you, Mariska Hargitay, for bravely sharing your story. To change the culture of sexual violence and abuse across our country, we have to be willing to talk about it. Survivors deserve to have their voices heard."
[10] "The 2023 HolidayImpactPrize has raised $6.3 million and recruited hundreds of volunteers for four life-changing nonprofits! Your donations are making a difference – let’s finish strong! Join by Jan. 31 🎁"
llm_opinion_vs_claim == 100 (definitely claim)
[1] "Today's Classic Doonesbury: \"\"Mickey D in the mix.\"\""
[2] "King Center Press Conference: Passing of Dexter Scott King via"
[3] "Today we celebrate and honor the life and legacy of Dr. Martin Luther King Jr."
[4] "And catch Stanley Kramer’s work/"
[5] "In case you missed it, from CNBC: \"\"In Congress and in class: Rep. Don Beyer works toward master’s degree in AI\"\""
[6] "Jimmy Buffet lives still in South Texas"
[7] "Rev. Dr. Martin Luther King Jr."
[8] "On this day in 1991, Big Bend Ranch State Natural Area was opened to the public."
[9] "New all endorse different candidates in ncpol"
[10] "🐮🥛Did you know - NY is the 5th largest producer of milk in the US? nationalmilkday"
```
## accuracy
```r
llm_accuracy N N_perc
<num> <num> <num>
1: 0 3042 27.341
2: 10 82 0.737
3: 20 2655 23.863
4: 25 1 0.009
5: 30 286 2.571
6: 40 38 0.342
7: 50 861 7.739
8: 60 96 0.863
9: 70 1457 13.095
10: 75 60 0.539
11: 80 1400 12.583
12: 90 489 4.395
13: 95 20 0.180
14: 100 639 5.743
```
```r
claim_v_opinion llm_accuracy
<char> <num>
1: claim\n(llm_opinion_vs_claim>50) 75.90997
2: opinion\n(llm_opinion_vs_claim<=50) 28.66011
```
![[1737514181.png]]
Example tweets
- llm_opinion_vs_claim > 50 (2588 such tweets)
```
llm_accuracy < 10
[1] "‘Not survive’ = murder"
[2] "the voting population has doubled since Jim Hunt was last on the ballot in 1996"
[3] "6 Tech’s and 2 ejections"
[4] "Before ruling on the defendant's request for a postponement to attend his mother-in-law's funeral, the court would like to ask Mr. Trump a question: what was his late mother-in-law's name?\"\""
[5] "TODAY is the last day to apply for, renew, or change your health insurance marketplace plan for 2024 coverage. Visit today to review your coverage options and GetCovered."
[6] "He didn't explain the gold bars, the cash or the free Mercedes."
[7] "@jaynordlinger corrects me that the original was \"\"between Albania and Zimbabwe.\"\""
[8] "Tonight, dropped bombs on Yemen without Congressional approval."
[9] "Do You Know What The Enumerated Powers Are?"
[10] "we have not had 5 straight NCAA trips since he left"
40 ≤ llm_accuracy ≤ 60
[1] "A major snowstorm is coming to New Yorkers living near the Great Lakes this weekend. Please take steps now to prepare."
[2] "An update to yesterday's clue, “College consortium that will automatically admit the top 10% of New York high school classes as part of a State of the State plan to expand access to higher education”"
[3] "@NBCNews projecting Trump will win Iowa as polls predicted so now it’s a race between Desantis and Haley for 2nd - and how big Trump’s margin of victory will be"
[4] "In Iowa and New Hampshire, Republicans quietly court their angry white majority"
[5] "Well..I would pause on your comment for a bit..some universities appear to be falling short on being good examples for young people seeking to learn. harvard plagiarism"
[6] "@NBCNews projects that Ron Desantis will come in 2nd in Iowa behind winner Donald Trump who will end up with more than 50% of the vote - Nicki Haley comes in 3rd. Vivak Ramaswamy dropped out and endorsed Trump."
[7] "New @CarolinaJournal @JohnLockeNC article on @JudgeBobOrr attempts to get the NC Supreme Court to mandate the creation of the most gerrymandered map in U.S. History ncpol ncga"
[8] "Washington Post: Falling inflation, rising growth give U.S. the world’s best recovery"
[9] "Today, the Senate will pass a clean extension of government funding to avert a needless shutdown and give appropriators more time to finish their work."
[10] "In 1984 Walter Mondale, who'd served four years as VP, won Iowa with 49% of the vote. Gary Hart ran a surprising second with 17%. Eight days later, a surging Hart defeated Mondale by ten points in New Hampshire. Yes, Hart eventually fell short, but he made a real race of it."
llm_accuracy > 90
[1] "Today we celebrate the life & legacy of Dr. Martin Luther King Jr. and recommit ourselves to his unfinished work - combating racial prejudice, promoting equality & peace, and building stronger communities. The Dream endures and the March continues. MLKDay MLK2024"
[2] "Thank you to for inviting and me to discuss the importance of investing in school infrastructure. When we invest in school infrastructure, we invest in the success and well-being of our students and school staff."
[3] "Rev. Dr. Martin Luther King Jr."
[4] "PRE-BOARD OF COUNTY COMMISSION MEETING - January 18, 2024"
[5] "International Holocaust Remembrance Day is a moment to mourn one of the darkest moments in history. We will never forget the millions of Jews and persecuted people who were murdered by the Nazis. We have a duty to condemn antisemitism and speak out against all forms of hate."
[6] "Democrat-run Chicago."
[7] "Remaking the World of Arturo Schomburg Tickets, Wed, Jan 24, 2024 at 1:30 PM | Eventbrite"
[8] "In a 2012 essay, I struggled with the problem: how do you follow the rule, \"\"no speaking ill of the dead,\"\" when the dead person was both very important and very wicked?"
[9] "Israeli Public Figures Sign Letter Against ‘The Dehumanization of Palestinians and Israelis’"
[10] "In SacramentoCounty and across America, law enforcement officers put their lives on the line to protect our communities. On National Law Enforcement Appreciation Day and every day, I am grateful for their service and sacrifice."
```