The binomial test is a useful statistical tool used to determine if the observed distribution of binary outcomes matches expected probabilities. It’s commonly used when there are two possible outcomes or categories for a dataset. In this article, we will explain how to perform a binomial test in Python, focusing on the use of the Scipy library.

Performing a Simple Binomial Test

Consider an experiment where you flip a coin 10 times and it lands on heads 7 times. You want to know whether this coin is fair. A binomial test can provide an answer to this.

Here’s how you perform a binomial test using the scipy.stats.binom_test function:

from scipy.stats import binom_test

# Number of successes
x = 7

# Number of trials
n = 10

# Expected probability of success
p = 0.5  # We expect a fair coin

# Perform the test
p_value = binom_test(x, n, p, alternative='two-sided')

print(f"P-value: {p_value}")

The ‘two-sided’ alternative hypothesizes that the probability of success is not equal to p. If the p-value is less than 0.05, we reject the null hypothesis that the coin is fair. In this case, the p-value is greater than 0.05, so we do not reject the null hypothesis.

More Complicated Scenarios

The binomial test can also be applied to more complex situations. For instance, let’s consider a case where we are running an A/B test on a website. We show 1000 users version A of a webpage and 1200 users version B. In version A, 500 users clicked on our link, while in version B, 700 users clicked on our link. We can use a binomial test to determine if version B is significantly better than version A:

from scipy.stats import binom_test

# Number of successes
x = 700

# Number of trials
n = 1200

# Expected probability of success
# The success rate of version A
p = 500 / 1000

# Perform the test
p_value = binom_test(x, n, p, alternative='greater')

print(f"P-value: {p_value}")

Here, we used the ‘greater’ alternative hypothesis because we are interested in whether version B is better than version A. If the p-value is less than 0.05, we can conclude that version B leads to a significantly higher click rate.

Considerations and Caveats

The binomial test makes several assumptions:

1. Independence of trials: Each trial (e.g., coin flip, user interaction) must be independent.
2. Fixed number of trials: The number of trials must be fixed in advance.
3. Two possible outcomes: Each trial can only result in one of two possible outcomes.

If any of these assumptions is violated, the binomial test may not be appropriate.

Additionally, it’s worth mentioning that while p-values give us an indication of the statistical significance, they do not measure the size or importance of the effect. In our A/B testing example, while we may conclude that version B leads to a higher click rate, the binomial test does not tell us by how much the click rate is increased.

Conclusion

In this article, we covered how to perform a binomial test in Python using the scipy.stats.binom_test function. The binomial test is a powerful tool that allows us to test hypotheses about the probability of success in a binary outcome experiment. However, as with all statistical methods, it’s important to ensure the assumptions of the test are met and to interpret the results in the context of the specific experiment or study.