# analysis parameters - total N: 32888 (pilot campaign) - only retweets - 8 days: Oct 17 to Oct 24 - each day has different no. of active users => different no. of users included in analysis - active: >= 1 tweet on that day; ANY kind of tweet, not limited to tweets with rated domains - domain rating (9 in total): fc, mbfc_min, mbfc_bias, afm_min etc. - winsorization: .85, .90, .95, .99, 1.00 (no winsorization) - outcomes: summed, count, fraction badness - pre-campaign/time0 value for the outcome (`t0`): Sept 23 to Oct 4 (same as original analysis) - thresholds: 40-80 (steps of 5) - standard errors: HC1 or cluster on block # daily active users from from Oct 17 to Oct 24 31537 users (95.89%) active on those 8 days. About 28k users are active on any given day and these 28k users are included in the analysis for each day. If they didn't retweet anything with rated domains on that day, they are assigned a value of 0 for that day. ```r # unique active users per day date n_unique 1: 2021-10-17 28142 2: 2021-10-18 28766 3: 2021-10-19 28798 4: 2021-10-20 28671 5: 2021-10-21 28674 6: 2021-10-22 28420 7: 2021-10-23 27750 8: 2021-10-24 27620 # most users are active every day total_active_days n_users perc 1: 8 21892 69.42 # 69% of 31537 users are active EVERY DAY 2: 7 3749 11.89 3: 6 1996 6.33 4: 5 1209 3.83 5: 4 900 2.85 6: 3 722 2.29 7: 2 606 1.92 8: 1 463 1.47 ``` # models `t1 ~ condition * t0` ```r # summed and count badness models feglm(t1 ~ conditionC * t0LC | block, dt1, family = "quasipoisson") # SEs cluster on block feglm(t1 ~ conditionC * t0LC | block, dt1, family = "quasipoisson", vcov = "HC1") # fraction badness models feols(t1 ~ conditionC * t0LC | block, dt1) # SEs cluster on block feols(t1 ~ conditionC * t0LC | block, dt1, vcov = "HC1") ``` # results - condition estimate ## distribution of t- and p-values for condition estimate Analyses aren't independent! But just FYI. ![[_ 7.png]] % significant effects each day for different outcomes (sorted) ```r outcome day perc_significant 1: sum 2021-10-22 54.07 2: sum 2021-10-20 45.19 3: sum 2021-10-23 34.32 4: sum 2021-10-21 33.21 5: sum 2021-10-18 32.96 6: sum 2021-10-17 8.77 7: sum 2021-10-19 0.00 8: sum 2021-10-24 0.00 9: count 2021-10-23 32.91 10: count 2021-10-17 23.42 11: count 2021-10-20 21.77 12: count 2021-10-22 21.14 13: count 2021-10-18 17.09 14: count 2021-10-21 14.94 15: count 2021-10-19 0.00 16: count 2021-10-24 0.00 17: frac 2021-10-20 14.56 18: frac 2021-10-23 11.01 19: frac 2021-10-24 5.06 20: frac 2021-10-22 4.43 21: frac 2021-10-21 4.30 22: frac 2021-10-17 2.41 23: frac 2021-10-19 2.28 24: frac 2021-10-18 1.01 outcome day perc_significant ``` ## strongest negative effects (treatment group shared less bad stuff) Top 15, separately for summed, count fraction, summed analyses. - `outcome`: summed, count, fraction badness - `dv`: different measures of domain quality/badness - `estimate`: coefficient - `statistic`: t-value - `day`: day of the campaign the analysis was performed on - `n_users_active`: no. of users of were active on that day - `n_users_exclude`: no. of users excluded from analysis (because all users in the block have 0 as the outcome value, so the entire block has to be excluded) ```r outcome dv threshold estimate se p statistic winsorize se_type day n_users_active n_users_exclude 1: sum afm_min 80 -0.061 0.017 0.000 -3.509 0.85 clustblock 2021-10-20 28671 2440 2: sum afm_min 75 -0.060 0.017 0.001 -3.464 0.85 clustblock 2021-10-20 28671 2440 3: sum afm 80 -0.060 0.017 0.001 -3.446 0.85 clustblock 2021-10-20 28671 2440 4: sum afm_bias 80 -0.060 0.018 0.001 -3.386 0.85 clustblock 2021-10-20 28671 2440 5: sum afm_min 70 -0.059 0.017 0.001 -3.379 0.85 clustblock 2021-10-20 28671 2440 6: sum afm_rely 80 -0.058 0.017 0.001 -3.374 0.85 clustblock 2021-10-20 28671 2440 7: sum afm 75 -0.058 0.017 0.001 -3.342 0.85 clustblock 2021-10-20 28671 2440 8: sum afm_bias 75 -0.059 0.018 0.001 -3.334 0.85 clustblock 2021-10-20 28671 2440 9: sum afm_min 80 -0.061 0.018 0.001 -3.326 0.85 hc1 2021-10-20 28671 2440 10: sum afm_min 65 -0.058 0.017 0.001 -3.322 0.85 clustblock 2021-10-20 28671 2440 11: sum afm_min 75 -0.060 0.018 0.001 -3.281 0.85 hc1 2021-10-20 28671 2440 12: sum afm_min 60 -0.058 0.018 0.001 -3.280 0.85 clustblock 2021-10-20 28671 2440 13: sum afm 80 -0.060 0.018 0.001 -3.268 0.85 hc1 2021-10-20 28671 2440 14: sum afm_rely 75 -0.057 0.017 0.001 -3.266 0.85 clustblock 2021-10-20 28671 2440 15: sum afm_bias 70 -0.058 0.018 0.001 -3.252 0.85 clustblock 2021-10-20 28671 2440 16: count fc 55 -0.108 0.036 0.003 -2.987 0.85 clustblock 2021-10-23 27750 9430 17: count fc 65 -0.105 0.036 0.003 -2.967 0.85 clustblock 2021-10-23 27750 9254 18: count fc 70 -0.105 0.036 0.003 -2.967 0.85 clustblock 2021-10-23 27750 9254 19: count mbfc_min 70 -0.048 0.016 0.003 -2.955 0.85 clustblock 2021-10-21 28674 1297 20: count mbfc_min 75 -0.048 0.016 0.003 -2.955 0.85 clustblock 2021-10-21 28674 1297 21: count mbfc_min 80 -0.048 0.016 0.003 -2.955 0.85 clustblock 2021-10-21 28674 1297 22: count afm_min 65 -0.079 0.027 0.003 -2.931 0.90 clustblock 2021-10-23 27750 4953 23: count fc 65 -0.112 0.038 0.003 -2.930 0.90 clustblock 2021-10-23 27750 9254 24: count fc 70 -0.112 0.038 0.003 -2.930 0.90 clustblock 2021-10-23 27750 9254 25: count afm_bias 65 -0.084 0.029 0.004 -2.919 0.90 clustblock 2021-10-23 27750 5427 26: count fc 50 -0.106 0.036 0.004 -2.917 0.85 clustblock 2021-10-23 27750 9443 27: count fc 55 -0.113 0.039 0.004 -2.910 0.90 clustblock 2021-10-23 27750 9430 28: count afm_bias 65 -0.079 0.027 0.004 -2.909 0.85 clustblock 2021-10-23 27750 5427 29: count mbfc_fact 70 -0.046 0.016 0.004 -2.906 0.85 clustblock 2021-10-21 28674 1320 30: count mbfc_fact 75 -0.046 0.016 0.004 -2.906 0.85 clustblock 2021-10-21 28674 1320 31: frac fc 40 -0.001 0.000 0.004 -2.897 0.85 hc1 2021-10-22 28420 13190 32: frac fc 40 -0.001 0.000 0.004 -2.868 0.85 clustblock 2021-10-22 28420 13190 33: frac fc 40 -0.001 0.000 0.004 -2.847 0.90 hc1 2021-10-22 28420 13190 34: frac mbfc 75 -0.002 0.001 0.005 -2.819 0.85 hc1 2021-10-20 28671 1558 35: frac mbfc 80 -0.002 0.001 0.005 -2.819 0.85 hc1 2021-10-20 28671 1558 36: frac fc 40 -0.001 0.000 0.005 -2.796 0.90 clustblock 2021-10-22 28420 13190 37: frac fc 40 -0.001 0.000 0.005 -2.785 0.95 hc1 2021-10-22 28420 13190 38: frac mbfc 75 -0.002 0.001 0.006 -2.771 0.85 clustblock 2021-10-20 28671 1558 39: frac mbfc 80 -0.002 0.001 0.006 -2.771 0.85 clustblock 2021-10-20 28671 1558 40: frac mbfc_min 70 -0.002 0.001 0.006 -2.745 0.85 hc1 2021-10-20 28671 1526 41: frac mbfc_min 75 -0.002 0.001 0.006 -2.745 0.85 hc1 2021-10-20 28671 1526 42: frac mbfc_min 80 -0.002 0.001 0.006 -2.745 0.85 hc1 2021-10-20 28671 1526 43: frac fc 40 -0.001 0.000 0.006 -2.730 0.95 clustblock 2021-10-22 28420 13190 44: frac mbfc_min 70 -0.002 0.001 0.007 -2.701 0.85 clustblock 2021-10-20 28671 1526 45: frac mbfc_min 75 -0.002 0.001 0.007 -2.701 0.85 clustblock 2021-10-20 28671 1526 outcome dv threshold estimate se p statistic winsorize se_type day n_users_active n_users_exclude ``` ## strongest positive effects (treatment group shared more bad stuff) Effects are generally weaker than the negative effects above. ```r outcome dv threshold estimate se p statistic winsorize se_type day n_users_active n_users_exclude 1: sum fc 40 0.064 0.057 0.263 1.120 1.00 clustblock 2021-10-24 27620 6249 2: sum fc 40 0.064 0.058 0.273 1.096 1.00 hc1 2021-10-24 27620 6249 3: sum fc 45 0.060 0.056 0.284 1.071 1.00 clustblock 2021-10-24 27620 6249 4: sum fc 45 0.060 0.057 0.295 1.047 1.00 hc1 2021-10-24 27620 6249 5: sum fc 50 0.056 0.055 0.311 1.014 1.00 clustblock 2021-10-24 27620 6249 6: sum mbfc_bias 50 0.026 0.026 0.313 1.008 1.00 hc1 2021-10-24 27620 1193 7: sum mbfc_bias 45 0.026 0.026 0.315 1.005 1.00 hc1 2021-10-24 27620 1193 8: sum mbfc_bias 40 0.026 0.026 0.316 1.002 1.00 hc1 2021-10-24 27620 1193 9: sum fc 50 0.056 0.056 0.321 0.992 1.00 hc1 2021-10-24 27620 6249 10: sum mbfc_bias 50 0.026 0.026 0.322 0.990 1.00 clustblock 2021-10-24 27620 1193 11: sum mbfc_bias 45 0.026 0.026 0.324 0.986 1.00 clustblock 2021-10-24 27620 1193 12: sum mbfc_bias 40 0.026 0.026 0.325 0.983 1.00 clustblock 2021-10-24 27620 1193 13: sum mbfc_bias 55 0.024 0.025 0.337 0.961 1.00 hc1 2021-10-24 27620 1193 14: sum fc 40 0.057 0.059 0.339 0.957 1.00 clustblock 2021-10-20 28671 6852 15: sum fc 55 0.052 0.055 0.341 0.953 1.00 clustblock 2021-10-24 27620 6249 16: count afm_rely 40 0.143 0.074 0.052 1.948 1.00 clustblock 2021-10-24 27620 18032 17: count fc 45 0.180 0.093 0.053 1.936 1.00 clustblock 2021-10-24 27620 13746 18: count fc 40 0.209 0.108 0.053 1.933 1.00 clustblock 2021-10-24 27620 15596 19: count fc 40 0.209 0.110 0.058 1.900 1.00 hc1 2021-10-24 27620 15596 20: count fc 45 0.180 0.095 0.058 1.896 1.00 hc1 2021-10-24 27620 13746 21: count afm_rely 40 0.143 0.080 0.074 1.788 1.00 hc1 2021-10-24 27620 18032 22: count afm_rely 40 0.120 0.068 0.077 1.767 0.99 clustblock 2021-10-24 27620 18032 23: count fc 55 0.103 0.059 0.080 1.749 1.00 clustblock 2021-10-24 27620 8413 24: count fc 50 0.102 0.059 0.081 1.744 1.00 clustblock 2021-10-24 27620 8417 25: count fc 60 0.102 0.059 0.083 1.733 1.00 clustblock 2021-10-24 27620 8399 26: count afm_rely 40 0.114 0.067 0.090 1.697 0.99 clustblock 2021-10-23 27750 18157 27: count fc 40 0.104 0.061 0.091 1.691 0.99 clustblock 2021-10-24 27620 15596 28: count fc 55 0.103 0.061 0.094 1.674 1.00 hc1 2021-10-24 27620 8413 29: count fc 50 0.102 0.061 0.095 1.669 1.00 hc1 2021-10-24 27620 8417 30: count fc 60 0.102 0.061 0.097 1.659 1.00 hc1 2021-10-24 27620 8399 31: frac fc 60 0.002 0.001 0.006 2.754 0.99 hc1 2021-10-24 27620 8399 32: frac fc 55 0.002 0.001 0.006 2.750 0.99 hc1 2021-10-24 27620 8413 33: frac fc 50 0.002 0.001 0.006 2.743 0.99 hc1 2021-10-24 27620 8417 34: frac fc 65 0.002 0.001 0.007 2.687 0.99 hc1 2021-10-24 27620 8346 35: frac fc 70 0.002 0.001 0.007 2.687 0.99 hc1 2021-10-24 27620 8346 36: frac fc 60 0.002 0.001 0.008 2.671 0.99 clustblock 2021-10-24 27620 8399 37: frac fc 55 0.002 0.001 0.008 2.667 0.99 clustblock 2021-10-24 27620 8413 38: frac fc 50 0.002 0.001 0.008 2.661 0.99 clustblock 2021-10-24 27620 8417 39: frac fc 65 0.002 0.001 0.009 2.606 0.99 clustblock 2021-10-24 27620 8346 40: frac fc 70 0.002 0.001 0.009 2.606 0.99 clustblock 2021-10-24 27620 8346 41: frac fc 60 0.002 0.001 0.012 2.504 1.00 hc1 2021-10-24 27620 8399 42: frac fc 55 0.002 0.001 0.012 2.501 1.00 hc1 2021-10-24 27620 8413 43: frac fc 50 0.002 0.001 0.013 2.495 1.00 hc1 2021-10-24 27620 8417 44: frac fc 60 0.002 0.001 0.014 2.470 1.00 clustblock 2021-10-24 27620 8399 45: frac fc 55 0.002 0.001 0.014 2.467 1.00 clustblock 2021-10-24 27620 8413 outcome dv threshold estimate se p statistic winsorize se_type day n_users_active n_users_exclude ``` ```r # which outcomes/dvs have the most significant condition effect (p < .05) > dttemp[p < .05, .N, keyby = .(term, outcome, dv)] term outcome dv N 1: conditionC sum afm 244 2: conditionC sum afm_bias 222 3: conditionC sum afm_min 258 4: conditionC sum afm_rely 258 5: conditionC sum fc 170 6: conditionC sum mbfc 128 7: conditionC sum mbfc_bias 151 8: conditionC sum mbfc_fact 129 9: conditionC sum mbfc_min 129 10: conditionC count afm 54 11: conditionC count afm_bias 66 12: conditionC count afm_min 117 13: conditionC count afm_rely 99 14: conditionC count fc 106 15: conditionC count mbfc 182 16: conditionC count mbfc_bias 99 17: conditionC count mbfc_fact 156 18: conditionC count mbfc_min 158 19: conditionC frac afm 28 20: conditionC frac afm_bias 48 21: conditionC frac afm_min 53 22: conditionC frac afm_rely 25 23: conditionC frac fc 93 24: conditionC frac mbfc 26 25: conditionC frac mbfc_bias 24 26: conditionC frac mbfc_fact 30 27: conditionC frac mbfc_min 29 term outcome dv N ``` # results - condition-t0 interaction Weak positive interaction estimates? ![[_ 8.png]] % significant effects each day for different outcomes (sorted) ```r outcome day perc_significant 1: sum 2021-10-23 59.38 2: sum 2021-10-21 33.21 3: sum 2021-10-24 5.19 4: sum 2021-10-18 4.20 5: sum 2021-10-17 1.11 6: sum 2021-10-19 0.00 7: sum 2021-10-20 0.00 8: sum 2021-10-22 0.00 9: count 2021-10-23 37.34 10: count 2021-10-17 37.22 11: count 2021-10-21 33.54 12: count 2021-10-20 18.73 13: count 2021-10-24 14.94 14: count 2021-10-18 12.53 15: count 2021-10-22 0.76 16: count 2021-10-19 0.00 17: frac 2021-10-19 14.05 18: frac 2021-10-23 6.96 19: frac 2021-10-24 6.84 20: frac 2021-10-20 6.20 21: frac 2021-10-17 2.78 22: frac 2021-10-18 2.78 23: frac 2021-10-21 1.27 24: frac 2021-10-22 1.14 outcome day perc_significant ```