Statistics Archives - Locally Optimistic

A man with striped shirt and glasses with piles of papers on each side marking up a test.

Against A/B Tests

In Data ScienceTags Statistics, Testing, Web AnalyticsDecember 4, 2018

The notion of an A/B test is premised on the fundamentally flawed assumption that there exists one version of some treatment that is better on average for all users. Analytics practitioners should reject the assumptions of homogeneity and start designing systems that allow for (and encourage) non-binary outcomes of tests.

In the past few weeks, two really interesting articles about non-standard interpretations of A/B tests have been published. One, from Uber’s engineering blog, is about calculating “quantile treatment effects” and

…

A man and a woman walking along a trail laughing.

A chat about A/B testing

In Data ScienceTags A/B Testing, StatisticsAugust 26, 2018

Ilan Man

A hypothetical tech company just completed an A/B test of two experiences, A (the test) and B (the control). The test was set up properly and executed successfully. The following

…

A person in a factory looking into the window of a "black box" machine that's turning gears and other parts into a big gift box with a bow on it.

The Blacker the Box

In Data ScienceTags Data Science, Machine Learning, StatisticsJuly 29, 2018

Michael Kaminsky

There has been a lot of discussion in the data science community about the use of black-box models, and there is lots of really fascinating ongoing research into methods, algorithms, and tools to

…