from Hacker News

Good Experiment, Bad Experiment

by xtacy on 2/7/20, 5:33 PM with 4 comments

• by mwexler on 2/8/20, 4:30 PM

  This is philosophically interesting, but little of this really is about _running_ a good experiment. Perhaps it's about choosing which ones to run?

  Kohavi's book will probably provide much more value than this kind of abstract post. See https://experimentguide.com/ for more details.

• by daveFNbuck on 2/8/20, 4:21 PM

  > Good experiments define success up-front.

  This is absolutely critical. If you're not defining success up-front, you're not running an experiment. You're just doing a staged rollout. You can use the data to craft whatever story you want for most changes.

• by Nzen on 2/8/20, 4:21 PM

  tl;dr former Slack director of product outlines twelve principles for a/b type experiments with features, products, whatever. (ex Good experiments involve analyzing the results as opposed to 'ship/kill?') Each principle has a couple of sentences of elaboration and calls out the opposite style to avoid.