220728_061630 prereg results

- [prereg](https://docs.google.com/document/d/1MzvJf3dtW9oGKcD3Feyisthh4O0pGf6E42--QXi-NV4/edit) - coding - condition (0/1): control-interest vs treatment-accuracy - repetition (-0.5/0.5): novel vs repeated - veracity (-0.5/0.5): false vs true - models with covariates (11 in total) - numeric covariates are z-scored - categorical covariates are dummies - see exploratory [[220729_114133 forest - doubly robust estimators|causal forest doubly robust estimator results]] # hypothesis 2 - false headlines only (no significant interaction effect) model: `accuracy_rating ~ repetition * condition` (false headlines only) Frequentist model - singular fit. Interpret Bayesian model. ```r # frequentist model m2 <- lmer(truth ~ repetitionc * conditiond + (1 + repetitionc | id) + (1 + conditiond | item), d2f) summaryh(m2) > summaryh(m2) # singular term results 1: (Intercept) b = 3.20, SE = 0.13, t(155) = 24.04, p < .001, r = 0.89 2: repetitionc b = 0.08, SE = 0.07, t(299) = 1.14, p = .254, r = 0.07 # no repetition effect for control condition 3: conditiond b = 0.14, SE = 0.14, t(275) = 1.01, p = .312, r = 0.06 4: repetitionc:conditiond b = −0.13, SE = 0.09, t(329) = −1.49, p = .138, r = 0.08 # no interaction but correct sign # bayesian version Population-Level Effects: Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS Intercept 3.20 0.14 2.93 3.47 1.00 3105 6666 repetitionc 0.08 0.07 -0.06 0.22 1.00 19074 23304 conditiond 0.14 0.14 -0.14 0.41 1.00 2739 5912 repetitionc:conditiond -0.13 0.09 -0.30 0.04 1.00 19290 24353 # no interaction but correct sign ``` ## controlling for 11 covariates leads to significant interaction effect See row 21 below. Effect of repetition of truth ratings (false headlines) decreased for treatment group, b = -0.26, SE = 0.10, t(301) = -2.68, p = .008.. ```r m2cov <- lmer(truth ~ repetitionc * (conditiond + age + gender + role + tenuredays + domain + region + country + aot + covid_concern + conspiracy + gap_days) + (1 + repetitionc | id) + (1 + conditiond | item), d2f) > summaryh(m2cov) # converged, but singular because of the varying condition slopes for item term results 1: (Intercept) b = 6.47, SE = 0.92, t(259) = 7.06, p < .001, r = 0.40 2: repetitionc b = −0.02, SE = 0.73, t(256) = −0.02, p = .982, r = 0.001 3: conditiond b = −0.28, SE = 0.12, t(257) = −2.37, p = .018, r = 0.15 # 4: age b = −0.05, SE = 0.06, t(256) = −0.80, p = .425, r = 0.05 5: genderMale b = 0.11, SE = 0.11, t(256) = 1.02, p = .307, r = 0.06 6: roleTeam Leader b = −4.04, SE = 0.94, t(256) = −4.29, p < .001, r = 0.26 7: roleTeammate b = −3.52, SE = 0.89, t(256) = −3.94, p < .001, r = 0.24 8: tenuredays b = −0.10, SE = 0.07, t(256) = −1.43, p = .153, r = 0.09 9: domainHi-Tech b = −0.02, SE = 0.34, t(256) = −0.07, p = .944, r = 0.004 10: domainMedia & Ent b = 0.20, SE = 0.29, t(256) = 0.70, p = .486, r = 0.04 11: domainProfessional Services / Industry b = −0.53, SE = 0.35, t(256) = −1.50, p = .135, r = 0.09 12: domainSocial Media b = 0.05, SE = 0.27, t(256) = 0.18, p = .856, r = 0.01 13: regionIN b = 0.80, SE = 0.29, t(256) = 2.75, p = .006, r = 0.17 14: regionLATAM b = −0.18, SE = 0.35, t(256) = −0.52, p = .605, r = 0.03 15: regionSEA b = 0.08, SE = 0.44, t(256) = 0.18, p = .860, r = 0.01 16: countryPH b = 0.14, SE = 0.39, t(256) = 0.37, p = .712, r = 0.02 17: aot b = −0.40, SE = 0.06, t(256) = −6.86, p < .001, r = 0.39 # 18: covid_concern b = 0.01, SE = 0.06, t(256) = 0.20, p = .840, r = 0.01 19: conspiracy b = 0.14, SE = 0.05, t(256) = 2.58, p = .010, r = 0.16 # 20: gap_days b = 0.02, SE = 0.07, t(256) = 0.34, p = .736, r = 0.02 21: repetitionc:conditiond b = −0.26, SE = 0.10, t(301) = −2.68, p = .008, r = 0.15 # significant effect (treatment condition rate headlines as less true) 22: repetitionc:age b = −0.001, SE = 0.05, t(257) = −0.03, p = .975, r = 0.002 23: repetitionc:genderMale b = −0.01, SE = 0.08, t(257) = −0.17, p = .868, r = 0.01 24: repetitionc:roleTeam Leader b = −0.04, SE = 0.75, t(256) = −0.05, p = .960, r = 0.003 25: repetitionc:roleTeammate b = −0.16, SE = 0.71, t(256) = −0.22, p = .827, r = 0.01 26: repetitionc:tenuredays b = 0.11, SE = 0.05, t(256) = 2.04, p = .043, r = 0.13 27: repetitionc:domainHi-Tech b = 0.45, SE = 0.27, t(256) = 1.65, p = .100, r = 0.10 28: repetitionc:domainMedia & Ent b = 0.35, SE = 0.23, t(258) = 1.50, p = .134, r = 0.09 29: repetitionc:domainProfessional Services / Industry b = 0.28, SE = 0.28, t(257) = 1.00, p = .317, r = 0.06 30: repetitionc:domainSocial Media b = 0.48, SE = 0.22, t(258) = 2.21, p = .028, r = 0.14 31: repetitionc:regionIN b = 0.05, SE = 0.23, t(258) = 0.21, p = .837, r = 0.01 32: repetitionc:regionLATAM b = −0.10, SE = 0.28, t(257) = −0.34, p = .735, r = 0.02 33: repetitionc:regionSEA b = 0.47, SE = 0.35, t(257) = 1.32, p = .187, r = 0.08 34: repetitionc:countryPH b = −0.74, SE = 0.31, t(257) = −2.39, p = .018, r = 0.15 35: repetitionc:aot b = 0.005, SE = 0.05, t(257) = 0.11, p = .912, r = 0.007 36: repetitionc:covid_concern b = 0.07, SE = 0.05, t(256) = 1.48, p = .141, r = 0.09 37: repetitionc:conspiracy b = −0.06, SE = 0.04, t(256) = −1.48, p = .141, r = 0.09 38: repetitionc:gap_days b = 0.02, SE = 0.05, t(256) = 0.42, p = .678, r = 0.03 term results # bayesian version fitting... ``` ### marginal effects with p<.05 ```r term contrast estimate std.error statistic p.value 1: conditiond dY/dX -0.234 0.115 -2.040 0.041 # treatment group rate false headlines as less true 2: role Team Leader - SOM -4.031 0.925 -4.358 0.000 3: role Teammate - SOM -3.494 0.878 -3.979 0.000 4: region IN - EU 0.792 0.287 2.764 0.006 5: aot dY/dX -0.397 0.057 -6.996 0.000 # higher AOT people rate false headlines as less true 6: conspiracy dY/dX 0.147 0.052 2.824 0.005 # conspiratorial thinking -> false headlines rated as more true ``` - see [[220729_114133 forest - doubly robust estimators|causal forest doubly robust estimator results]] # hypothesis 1 - all headlines - only CONTROL group Since we find a significant `repetition * condition` effect above with the covariate model, we fit the model with only data in the control group. model: `accuracy_rating ~ veracity * repetition` (control group only) Only significant effect of **veracity**, not repetition. ```r # frequentist model - converged but singular fit > m1 <- lmer(truth ~ veracityc * repetitionc + (1 + veracityc * repetitionc | id) + (1 + repetitionc | item), d2c) > summaryh(m1) term results 1: (Intercept) b = 3.51, SE = 0.10, t(118) = 34.72, p < .001, r = 0.95 2: veracityc b = 0.63, SE = 0.12, t(70) = 5.30, p < .001, r = 0.54 3: repetitionc b = 0.04, SE = 0.05, t(35) = 0.77, p = .444, r = 0.13 4: veracityc:repetitionc b = −0.02, SE = 0.10, t(31) = −0.23, p = .816, r = 0.04 # bayesian version... ``` ## controlling for 11 covariates - didn't change the results ```r # frequentist model - converged but singular fit > m1cov <- lmer(truth ~ veracityc * (repetitionc + age + gender + role + tenuredays + domain + region + country + aot + covid_concern + conspiracy + gap_days) + (1 + veracityc * repetitionc | id) + (1 + repetitionc | item), d2c, + control = lmerControl(optimizer = "bobyqa", optCtrl = list(maxfun = 2e4))) > summaryh(m1cov) term results 1: (Intercept) b = 2.76, SE = 0.76, t(76) = 3.64, p < .001, r = 0.39 2: veracityc b = 0.65, SE = 0.68, t(78) = 0.95, p = .344, r = 0.11 3: repetitionc b = 0.04, SE = 0.05, t(35) = 0.77, p = .447, r = 0.13 4: age b = −0.11, SE = 0.08, t(75) = −1.42, p = .160, r = 0.16 5: genderMale b = −0.03, SE = 0.18, t(75) = −0.18, p = .856, r = 0.02 6: roleTeammate b = 0.81, SE = 0.37, t(75) = 2.22, p = .029, r = 0.25 7: tenuredays b = 0.16, SE = 0.11, t(75) = 1.36, p = .178, r = 0.15 8: domainHi-Tech b = −0.50, SE = 0.65, t(75) = −0.77, p = .443, r = 0.09 9: domainMedia & Ent b = −0.58, SE = 0.58, t(75) = −1.00, p = .320, r = 0.11 10: domainProfessional Services / Industry b = −0.22, SE = 0.65, t(75) = −0.33, p = .739, r = 0.04 11: domainSocial Media b = −0.37, SE = 0.56, t(75) = −0.65, p = .520, r = 0.07 12: regionIN b = 0.99, SE = 0.38, t(75) = 2.59, p = .011, r = 0.29 13: regionLATAM b = 0.36, SE = 0.40, t(75) = 0.89, p = .375, r = 0.10 14: regionSEA b = 0.25, SE = 0.73, t(75) = 0.34, p = .739, r = 0.04 15: countryPH b = 0.14, SE = 0.68, t(75) = 0.21, p = .832, r = 0.02 16: aot b = −0.18, SE = 0.09, t(75) = −1.85, p = .069, r = 0.21 17: covid_concern b = 0.12, SE = 0.08, t(75) = 1.39, p = .169, r = 0.16 18: conspiracy b = 0.15, SE = 0.07, t(75) = 1.96, p = .054, r = 0.22 19: gap_days b = −0.02, SE = 0.11, t(75) = −0.14, p = .885, r = 0.02 20: veracityc:repetitionc b = −0.02, SE = 0.10, t(31) = −0.23, p = .822, r = 0.04 # no interaction 21: veracityc:age b = 0.11, SE = 0.07, t(75) = 1.54, p = .127, r = 0.17 22: veracityc:genderMale b = −0.12, SE = 0.16, t(75) = −0.75, p = .456, r = 0.09 23: veracityc:roleTeammate b = −0.50, SE = 0.33, t(75) = −1.52, p = .132, r = 0.17 24: veracityc:tenuredays b = 0.08, SE = 0.10, t(75) = 0.82, p = .416, r = 0.09 25: veracityc:domainHi-Tech b = 0.24, SE = 0.58, t(75) = 0.41, p = .686, r = 0.05 26: veracityc:domainMedia & Ent b = −0.03, SE = 0.52, t(75) = −0.06, p = .956, r = 0.006 27: veracityc:domainProfessional Services / Industry b = 0.93, SE = 0.58, t(75) = 1.61, p = .113, r = 0.18 28: veracityc:domainSocial Media b = 0.31, SE = 0.51, t(75) = 0.61, p = .545, r = 0.07 29: veracityc:regionIN b = 0.25, SE = 0.34, t(76) = 0.73, p = .467, r = 0.08 30: veracityc:regionLATAM b = 0.34, SE = 0.36, t(76) = 0.96, p = .340, r = 0.11 31: veracityc:regionSEA b = 0.21, SE = 0.66, t(75) = 0.33, p = .744, r = 0.04 32: veracityc:countryPH b = −0.13, SE = 0.60, t(75) = −0.21, p = .834, r = 0.02 33: veracityc:aot b = 0.30, SE = 0.08, t(76) = 3.55, p < .001, r = 0.38 # interaction 34: veracityc:covid_concern b = 0.05, SE = 0.08, t(75) = 0.66, p = .511, r = 0.08 35: veracityc:conspiracy b = 0.08, SE = 0.07, t(75) = 1.25, p = .216, r = 0.14 36: veracityc:gap_days b = −0.09, SE = 0.10, t(75) = −0.94, p = .352, r = 0.11 term results ``` ### marginal effects with p<.05 ```r > mutate_if(m1cov_mfx_sum, is.numeric, round, 3)[p.value < .05, .(term, contrast, estimate, std.error, statistic, p.value)] term contrast estimate std.error statistic p.value 1: veracityc dY/dX 0.635 0.113 5.599 0.000 # true headlines rated as more true 2: role Teammate - Team Leader 0.813 0.366 2.220 0.026 # teamates rate headlines as more true than team leaders 3: region IN - EU 0.990 0.382 2.592 0.010 # indians rated headlines as more true than europeans ``` ## refit hypothesis 1 model with CONTROL & TREATMENT group ```r # frequentist model - converged but singular m11 <- lmer(truth ~ veracityc * repetitionc + (1 + veracityc * repetitionc | id) + (1 + repetitionc | item), d2) > summaryh(m11) term results 1: (Intercept) b = 3.62, SE = 0.07, t(168) = 49.13, p < .001, r = 0.97 2: veracityc b = 0.66, SE = 0.11, t(55) = 6.20, p < .001, r = 0.64 # 3: repetitionc b = 0.02, SE = 0.03, t(47) = 0.58, p = .567, r = 0.08 4: veracityc:repetitionc b = 0.03, SE = 0.06, t(43) = 0.47, p = .639, r = 0.07 # bayesian version Population-Level Effects: Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS Intercept 3.62 0.07 3.47 3.76 1.01 319 514 veracityc 0.65 0.11 0.43 0.87 1.01 472 982 repetitionc 0.02 0.03 -0.04 0.07 1.00 3272 2987 veracityc:repetitionc 0.03 0.06 -0.08 0.13 1.00 3883 2932 ``` ### controlling for 11 covariates ```r # frequentist model - converged but singular fit > m11cov <- lmer(truth ~ veracityc * (repetitionc + age + gender + role + tenuredays + domain + region + country + aot + covid_concern + conspiracy + gap_days) + (1 + veracityc * repetitionc | id) + (1 + repetitionc | item), d2, + control = lmerControl(optimizer = "bobyqa", optCtrl = list(maxfun = 2e4))) > summaryh(m11cov) term results 1: (Intercept) b = 5.87, SE = 0.80, t(259) = 7.31, p < .001, r = 0.41 2: veracityc b = −0.88, SE = 0.67, t(268) = −1.31, p = .193, r = 0.08 3: repetitionc b = 0.02, SE = 0.03, t(47) = 0.57, p = .570, r = 0.08 4: age b = 0.004, SE = 0.05, t(257) = 0.08, p = .939, r = 0.005 5: genderMale b = 0.06, SE = 0.09, t(257) = 0.68, p = .496, r = 0.04 6: roleTeam Leader b = −3.21, SE = 0.83, t(257) = −3.87, p < .001, r = 0.23 7: roleTeammate b = −2.86, SE = 0.79, t(257) = −3.63, p < .001, r = 0.22 8: tenuredays b = −0.06, SE = 0.06, t(257) = −1.04, p = .298, r = 0.06 9: domainHi-Tech b = 0.09, SE = 0.29, t(257) = 0.30, p = .767, r = 0.02 10: domainMedia & Ent b = 0.19, SE = 0.26, t(257) = 0.75, p = .453, r = 0.05 11: domainProfessional Services / Industry b = −0.17, SE = 0.31, t(257) = −0.55, p = .581, r = 0.03 12: domainSocial Media b = 0.13, SE = 0.24, t(257) = 0.53, p = .599, r = 0.03 13: regionIN b = 0.75, SE = 0.25, t(257) = 2.97, p = .003, r = 0.18 14: regionLATAM b = 0.03, SE = 0.31, t(257) = 0.09, p = .932, r = 0.005 15: regionSEA b = 0.09, SE = 0.39, t(257) = 0.23, p = .819, r = 0.01 16: countryPH b = 0.22, SE = 0.34, t(257) = 0.66, p = .512, r = 0.04 17: aot b = −0.28, SE = 0.05, t(257) = −5.48, p < .001, r = 0.32 18: covid_concern b = 0.03, SE = 0.05, t(257) = 0.48, p = .632, r = 0.03 19: conspiracy b = 0.16, SE = 0.05, t(257) = 3.35, p < .001, r = 0.20 20: gap_days b = −0.009, SE = 0.06, t(257) = −0.16, p = .872, r = 0.01 21: veracityc:repetitionc b = 0.03, SE = 0.05, t(43) = 0.52, p = .608, r = 0.08 # no interaction 22: veracityc:age b = 0.08, SE = 0.04, t(257) = 1.89, p = .060, r = 0.12 23: veracityc:genderMale b = −0.09, SE = 0.08, t(257) = −1.18, p = .239, r = 0.07 24: veracityc:roleTeam Leader b = 1.60, SE = 0.69, t(257) = 2.31, p = .022, r = 0.14 # 25: veracityc:roleTeammate b = 1.23, SE = 0.66, t(257) = 1.87, p = .063, r = 0.12 26: veracityc:tenuredays b = 0.15, SE = 0.05, t(257) = 3.04, p = .003, r = 0.19 # 27: veracityc:domainHi-Tech b = 0.22, SE = 0.24, t(257) = 0.91, p = .361, r = 0.06 28: veracityc:domainMedia & Ent b = 0.05, SE = 0.21, t(257) = 0.23, p = .815, r = 0.01 29: veracityc:domainProfessional Services / Industry b = 0.79, SE = 0.26, t(257) = 3.05, p = .003, r = 0.19 # 30: veracityc:domainSocial Media b = 0.30, SE = 0.20, t(257) = 1.49, p = .138, r = 0.09 31: veracityc:regionIN b = 0.10, SE = 0.21, t(257) = 0.47, p = .638, r = 0.03 32: veracityc:regionLATAM b = 0.34, SE = 0.26, t(257) = 1.33, p = .186, r = 0.08 33: veracityc:regionSEA b = 0.18, SE = 0.32, t(257) = 0.55, p = .582, r = 0.03 34: veracityc:countryPH b = −0.13, SE = 0.28, t(257) = −0.44, p = .658, r = 0.03 35: veracityc:aot b = 0.24, SE = 0.04, t(257) = 5.55, p < .001, r = 0.33 # 36: veracityc:covid_concern b = 0.09, SE = 0.05, t(257) = 2.06, p = .040, r = 0.13 # 37: veracityc:conspiracy b = −0.01, SE = 0.04, t(257) = −0.29, p = .774, r = 0.02 38: veracityc:gap_days b = −0.06, SE = 0.05, t(257) = −1.25, p = .211, r = 0.08 term results ``` ### marginal effects with p<.05 ```r term contrast estimate std.error statistic p.value 1: veracityc dY/dX 0.653 0.105 6.211 0.000 2: role Team Leader - SOM -3.210 0.829 -3.872 0.000 3: role Teammate - SOM -2.860 0.787 -3.634 0.000 4: region IN - EU 0.753 0.254 2.968 0.003 5: aot dY/dX -0.279 0.051 -5.480 0.000 6: conspiracy dY/dX 0.156 0.047 3.354 0.001 ```