A click is not automatically an incremental visit.
CTR was the number that won the meeting. It never told me who would have come back anyway.
Talk to usCTR tells part of the story
Click-through rate shows who responded to the notification. It does not show who would have visited anyway. Loyal readers often return through search, email, or habit. Without a comparison group, you cannot tell which visits the Nudge actually created.
How holdouts work
- 01
Reserve a small holdout group
A random subset of eligible people does not receive the Nudge.
- 02
Measure both groups
Track qualified visits for exposed people and holdout people over the same window.
- 03
Compare outcomes
The difference between groups is the estimated incremental effect, with assumptions shown.
What we measure today
Page view
Did they open a page after the Nudge? Early signal. We show this with a comparison group, not as a trophy CTR.
Qualified visit
Did they stay long enough to count? Primary outcome when the sample is large enough and the window is closed.
Not measured yet
Ad value, registrations, newsletter signups, and purchases are not in the product today. We will not print them as proof.
Honest measurement
A holdout is an estimate, not a guarantee. We claim lift only when both groups are large enough and the window is closed. Until then the report says collecting, insufficient, or directional, never an invented percentage.
Know when to nudge.
Know when a notification is worth sending.
The useful question is who should not get this one.