from Hacker News

Does A/B testing lead to crappy products?

by wmorein on 3/2/09, 5:04 PM with 4 comments

• by joe_the_user on 3/2/09, 8:16 PM

  Are crappy products unprofitable?

  He says Hollywood keeps making sequels even though they turn out to be bad. But does he show that Hollywood loses money making bad sequels?

  IF bad inevitably translated to unprofitable, Hollywood would have a strong incentive to make good movies. If movie goers flock to movies that are just a mediocre mishmash of the themes they've seen before, then Hollywood has every reason to keep putting them out.

• by aaronblohowiak on 3/2/09, 5:43 PM

  From TFA: "Thus, my answer to the title question is that no, A/B testing does NOT lead to crappy products, but poor decision-making around data can absolutely lead to it"

  A more accurate title would have been "Metrics don't replace strategy."

• by suhail on 3/2/09, 7:43 PM

  Metrics aren't wrong, it's how you perceive the results of what you're testing. If you make too many assumptions, it's your fault.

  There's too many variables involved for metrics to be the final solution towards success, part of building a startup is taking risks and your intelligence to come out on top.