Significant Difference and Break-even Point
https://gyazo.com/b6394d8392afea51c79d6fdaedc8a45f
2 has significant difference but not beyond [the break-even point
Whether or not the field finds value in this situation depends on the field.
Any system is costly to maintain.
If the customer doesn't pay that cost, it won't be maintained.
Let's quickly and inexpensively verify "are there paying customers?"
This is the Lean startup concept
Paying customers" = "people who value the product enough to want to continue using it even if they have to pay for it."
Even if there is a significant difference, it's as if there isn't enough to pay for it.
On the other hand, scientific thinking tends to ignore 3, "a small percentage of subjects are worthwhile"
It can be ignored simply because the sample size is too small to make a significant difference.
When "one in a hundred has a significant effect."
If there were 100 subjects, it would be "only one was effective".
sequel
Significant differences, gains/losses and variance
relevance
Who pays the maintenance costs?
---
This page is auto-translated from /nishio/有意差と損益分岐点 using DeepL. If you looks something interesting but the auto-translated English is not good enough to understand it, feel free to let me know at @nishio_en. I'm very happy to spread my thought to non-Japanese readers.