# Overview

“以下面的虚拟实验为例。我们想研究做作业对学习效率的影响。这个实验对150个学生进行了AB随机分组，75个学生要求做作业，而另外75个学生不做作业，在一个月后用一个包含20道题的标准化考试来衡量效果，预期得分提升一个点。最后的结果是不做作业的那一组平均得分15.6分，而做作业的组，平均得分17.3，提高了1.7分。统计检验在97.5%的显著性要求下通过。如果分析就到此为止也没有什么问题，但是优秀的分析师会对平均提高1.7分进行进一步的解读。比如，在这个实验中，结合样本的均方差，组间差异平均提高1.7分意味着两组有88%的重合，或者说实验组随机挑选一个学生，这个学生有58.4%的概率比对照组随机挑选的一个学生的得分要高（称之为probability of superiority）。此外，如果想要在实验组中获得比对照组多一个学生的成绩更好，我们平均需要让10.6个学生做作业，换句话说，如果有100个学生都严格执行了做作业的学习过程，相比他们不做作业的学习过程，平均来说有9.4个学生会得到更好的考试结果。这样的解读会让主管或者业务方更好地理解实验的应用意义。”

# Definitions

## Effect size

In statistics, an effect size is a number measuring the strength of the relationship between two variables in a statistical population, or a sample-based estimate of that quantity…Examples of effect sizes include … mean difference.

In medical education research studies that compare different educational interventions, effect size is the
magnitude of the difference between groups.

Thus, effect size can refer to the raw difference between group means, or absolute effect size, as well as standardized measures of effect, which are calculated to transform the effect to an easily understood scale.

## Cohen’s d

Cohen’s d is an effect size used to indicate the standardised difference between two means.

# How

Overlap: overlapping coefficient (OVL):

Probability of superiority: common language effect size (CL):

Number Needed to Treat(NNT):

TODO: 至于例子中这个9.4, 我直观感觉应该是$100 \times \frac{1}{10.6}=9.433962$，但是有点绕不过来Orz

Cohen’s $U_3$:

# And…

## Probability of superiority or AUC…

（另外一提，在ROC-AUC这篇文章中介绍的计算AUC的最快的方法，是WXG 的算法岗面试题）

# 附录:CER&EER&NNT

Event rate: The proportion of patients in a group in whom the event is observed. Thus, if out of 100 patients, the event is observed in 27, the event rate is 0.27 or 27%. Control event rate (CER) and experimental event rate (EER) are used to refer to this in control and experimental groups of patients respectively. An example might help.

Treatment 总人数 疼痛减少50%以上的人数 疼痛没有减少50%以上的人

Experimental event rate (EER, event rate with ibuprofen): 22/40 = 0.55 or 55%

Control event rate(CER, event rate with placebo): 7/40 = 0.18 or 18%

Absolute risk increase or reduction (EER-CER): 0.55 - 0.18 = 0.37 or 37%

NNT (1/(EER-CER)):1/(0.55 - 0.18) = 2.7