# descriptives - 29 campaign days: 2022-06-10 to 2022-07-09 (but the campaigns are still running) - 17049 users (15873 were active at least one day) - 8561 control (7968 active at least one day) - 8488 treatment (7905 active at least one day) - 54 adgroups - 27 control (no ads shown to them) - 27 treatment - **only 3913 bad tweets included in the analysis below!** (Andrew's biolab cluster had about 13k tweets - many were shared by the 100k friends instead of the 17k users in the experiment) - too few tweets - many days & blocks were excluded in the fixed-effects models because of only 0 outcomes within blocks or days - effective sample size is more like 6.9k! ## reach Each dot is an adgroup's reach for one day. Sorted by descending adgroup mean reach. Reach decreased over time across time (color bar): brighter colors (later campaign days) are generally at the bottom (lower reach) for all adgroups. ![[1661531161.png]] - see [[220826_153411 ITT separately for each ad group]] # results - winsorize 99% - added IO classifier score as covariate (at least one covariate required for the the two-stage least-squares bootstrapping model below to run) - outcome: count of bad tweets (Andrew's biolab cluster results) ## intent-to-treat models **Negative** treatment effects (see `conditiont` row) in both models below. ### overall model (only block fixed effect) - outcomes (no. of bad tweet counts are summed across campaign days for each user) - excluded users who weren't active at least on 1 day throughout the campaign (about 1.2k users) ```r > feglm(bad_tweet_count_win ~ condition + classifier_score | block, cluster = "block", data = dt_overall_active, family = "quasipoisson") NOTE: 1,746 fixed-effects (8,961 observations) removed because of only 0 outcomes. GLM estimation, family = quasipoisson, Dep. Var.: bad_tweet_count_win Observations: 6,912 Fixed-effects: block: 1,177 Standard-errors: Clustered (block) Estimate Std. Error t value Pr(>|t|) conditiont -0.017229 0.053649 -0.321141 7.4816e-01 classifier_score 2.022803 0.305501 6.621266 5.4031e-11 *** ``` ### daily model (with block + day fixed effects) - excluded users who weren't active, separately for each day ```r > feglm(bad_tweet_count_win ~ condition + classifier_score | block + sub_campaign, cluster = "block", data = dt_daily_active, family = "quasipoisson") NOTE: 1,746/14 fixed-effects (270,160 observations) removed because of only 0 outcomes. GLM estimation, family = quasipoisson, Dep. Var.: bad_tweet_count_win Observations: 78,239 Fixed-effects: block: 1,177, sub_campaign: 13 Standard-errors: Clustered (block) Estimate Std. Error t value Pr(>|t|) conditiont -0.012431 0.055616 -0.223520 8.2317e-01 classifier_score 1.993348 0.303777 6.561884 7.9468e-11 *** ``` ## two-stage least-squares with bootstrapped SEs Sign of treatment effects (see `D` row) in models below varies... ### overall model (only block fixed effect) ```r GLM estimation, family = quasipoisson, Dep. Var.: Y Observations: 6,912 Fixed-effects: block: 1,177 Standard-errors: Custom Estimate Std. Error t value Pr(>|t|) x1 2.025225 0.220867 9.169431 < 2.2e-16 *** # IO classifier score D -0.039452 0.448131 -0.088036 0.92985 # treatment effect stage1_resids -1.602109 1.660080 -0.965079 0.33455 ``` ### daily model (with block + day fixed effects) ```r GLM estimation, family = quasipoisson, Dep. Var.: Y Observations: 78,239 Fixed-effects: block: 1,177, sub_campaign: 13 Standard-errors: Custom Estimate Std. Error t value Pr(>|t|) x1 1.997030 0.229263 8.710653 < 2.2e-16 *** # IO classifier score D 0.293663 0.542109 0.541705 0.588023 # POSITIVE treatment effect?!? stage1_resids -1.677723 0.887698 -1.889969 0.058766 . ``` Positive treatment effect in the model above? If we refit model to include only first 14 days of campaigns where reach is higher, then treatment effect is negative again ```r > results GLM estimation, family = quasipoisson, Dep. Var.: Y Observations: 78,239 Fixed-effects: block: 1,177, sub_campaign: 13 Standard-errors: Custom Estimate Std. Error t value Pr(>|t|) x1 1.997445 0.229263 8.712465 < 2.2e-16 *** # IO classifier score D -0.111823 0.542109 -0.206274 0.83658 # NEGATIVE treatment effect stage1_resids -1.272236 0.887698 -1.433185 0.15181 ```