Notes on Robbins’ SPRT

The mixture sequential probability ratio test is an extension of Wald’s SPRT that allows us to reduce the worst case performance by integrating the likelihood ratio over a family of alternative values. The resultant test guarantees our frequentist false alarm rate.
2023-06-30

Notes on Wald’s SPRT

Wald’s sequential probability ratio test is a sequential procedure that allows one to make observations sequentially. The procedure terminates as soon as the evidence is sufficient to decide between \(H_0\) and \(H_1\), and can result in runtime savings over traditional “fixed horizon” approaches common to A/B testing.
2023-05-20

Quantifying the CUPED Improvement with Binomial Responses

CUPED is a method for reducing the variance of an estimator by leveraging another correlated variable. Many tout the importance and impact of such a method on speeding up A/B tests, but CUPED is not a panacea, and here I show some cases where it provides little value.
2023-04-06

Data Quality Control

A Requirement for Data for Data Science and Analytics Users

Data quality is important for modern analytics and data science teams. No one treats it that way. Here, I enumerate some ways to think about and monitor data quality.
2023-02-19

Analysis Question Without Answers

Reflections on the difficulty of assigning a value to a product feature

Sometimes a question sounds simple but like some fractal of suffering as one gets closer and closer one finds things are underdetermined, too open to subjective interpretation, and no result will serve as an appropriate answer.
2023-02-05

Multivariate Tests on the Web

Factorial and orthogonal array designs have a place in marketing experimentation, despite their unpopularity amongst some in the A/B testing world. Here, we get into some of the efficiency and philosophy behind these experimental designs.
2023-01-18

Marketing Data Science BOM

A ChatGPT-Inspired Bill of Materials for Marketing Data Science Projects

What makes a marketing data science project successful? What is required to make this set of tools work for you? Much has been dedicated to techniques for individual models, feature engineering, et cetera, yet general requirements often fail to include all of the details. Here, I discuss some things critical to a general set of applications.
2023-01-15

Alpha Spending for Group Sequential Trials

Group sequential designs offer analysts a lot of flexibility in that a test can be monitored with multiple analyses, and it can be stopped early when the test is likely to end up futile or already has strong evidence that the null hypothesis isn’t correct. This is my notes on these types of designs.
2023-01-06

Notes on Hsu's Method for Conversion Data

Hsu's method is a way to compare treatments to the best of the remaining treatments all while preserving the overall type 1 error rate. The result is confidence intervals that tell you if a particular treatment is dominated by another or if the treatment is among the best. Here are my notes on the procedure.
2022-12-29

Notes on Dunnett's Method for Conversion Data

Dunnett's method is a way to compare any number of treatments against a common control, all while preserving the overall type 1 error rate. This is a far more efficient approach than using Bonferroni corrections or doing a series of A/B tests. Here are my notes on the procedure.
2022-12-28

Your A/B Test May Be Three Different Tests

People tend to think of A/B tests in terms of the simple test of a challenger over a control, but the data we collect often finds itself in three distinct uses that we often fail to plan for. Here, I discuss this and ways to address it.
2022-12-23

A/B Testing: Oops I got A in my B!

Sometimes failed randomization can put people into both a control and challenger treatments, and just counting them for both or ignoring result in different conclusions…
2022-12-03

Pandemic Christmas Tree 2020

A colorful Christmas tree that’s baby-safe! Trying out Twinkly lights!
2020-12-25

You Probably Don’t Want To Pick The Winner

People often want an experiment to pick the winner, but the winner is significantly better than the next best option, and it can be very expensive to detect what could be a very small difference. Here I look at other options for analyzing A/B/n tests that can be more efficient.
2020-09-27

Analysts Have Technical Debt, And It Will Kill You

Technical debt is well-ish understood by developers, and care is taken to make sure errors rarely make it into production. Yet analysts have to deal with technical debt too, and analysts are all-too-comfortable ignoring errors as a kind of statistical noise. However, these errors do not come from a nice distribution and can have devastating effects on an analysis.
2020-09-15

DoE Fireside Chat on OCEAN

Anson and I sat down with Doug Montgomery to discuss design and analysis of experiments in an online setting. Anson and I also wore Hawaiian shirts!
2020-08-19

Lighting a Christmas Tree 2019

Decorating the Christmas tree, 2019 edition!
2019-12-25

Wilkinson-Rogers Notation: A Formula by any Other Name

The formula environment in R and implemented in Python’s statsmodels package is a powerful and expressive way to describe models and transforms to data. It’s often described as Wilkinson’s notation or Wilkinson-Rogers notation. Here, I break down how it works.
2019-07-15

Python Package Importing

It turns out importing packages in an editable way breaks depending on how the packages were defined. It’s the kind of bug that, after driving one insane, makes one write about it to hopefully save anyone else the same trouble.
2019-02-08

Suffix Trees for Log Aggregation

Email logs follow a standard in the same way toddlers follow a preschool teacher. Here, I use suffix trees to try to learn new patterns in log data.
2019-01-06

Lighting a Christmas Tree 2018

Deck the halls with the results of my Christmas tree model! And learn how live trees fail to behave as cones! Wow!
2018-12-25

Lighting a Christmas Tree

Let’s bring math to bear on decorating our Christmas tree! Here, I derive some equations and calculators to help get nice uniform lighting on a tree! And we kick it up a notch (do people still remember Emeril?) and create some nice patterns.
2017-12-25

Taco Model

A taco festival, personal rating, Yelp data, machine learning, and finding the best tacos.
2017-10-24

Fast Updating for Optimal Linear Model and Generalized Linear Model Experimental Designs

When things get big, refitting a GLM becomes too expensive, and instead you need to rely on some update functions that are in an expensive out-of-print Springer book or strewn across paywalled papers. Here, I derive (or motivate the derivation) of some useful updating identities for use in linear algebra for linear and generalized linear models.
2017-01-22

Modeling Carryover Effects in Industrial Settings

Sometimes a test can leave lingering effects on the experimental unit. Here I try a simple approach to model this phenomenon and then use genetic algorithms to try to find a good design to mitigate as much as possible the impact of the carryover effects.
2016-05-01

Testing for the Best of Times Without Knowing the Worst of Times

I was asked to answer a question about failure times when we had no data and the prospect of tremendous losses if we took a bad approach. Here’s my attempt to formulate some plan in such an environment.
2016-04-25

Binomial Proportion Confidence Intervals are Ugly and Messy

Binomial confidence intervals will let you down and hurt you. Here, I look at how, when, and why, and look at alternative formulations of the intervals that have better behavior.
2016-04-21
No matching items