Permutation Tests

Hypothesis tests, also referred to as statistical tests, are used in the design of experiments to measure the effect of changes on experimental units.

Experiments are structured in terms of null and alternative hypotheses. A null hypothesis claims that the changes made to the baseline are not an improvement, while alternative hypothesis is the opposite. The data is then randomized to remove bias from the results, and the differences between the null and alternative hypotheses are then measured.

In this dataset, I compare students at different painting schools, and test if student outcomes improve when compared against the first school. I permute (shuffle) the treatment assignments of the data, and re-calculate the test statistic. After some sufficient number of permutations, an approximate test statistic distribution is determined. This distribution approximates all possible test statistic values we could have seen under the null hypothesis. We can then use this distribution to obtain probabilities associated with different mean-difference values.

A p-value represents the probability of obtaining the observed values, assuming the null hypothesis is true. If the p-value is less than or equal to the significance level, we reject the null hypothesis; the outcome is said to be statistically significant.

https://artarriaga.shinyapps.io/Painters