How to Prove What Your Website Changes Are Worth
Written by RetainTee Team on November 3, 2026
A hotel launches a new offer pop-up in March. Direct bookings rise in April. The pop-up gets the credit.
But April also had Easter, a new Google Ads campaign and a competitor that raised its prices. Any of those could explain the rise. Without a fair comparison, you can’t tell what the pop-up did. You might keep something that does nothing, or remove something that works.
There are two ways to get a fair answer. An A/B test measures one change. A holdout group measures everything together.
Why before-and-after comparisons mislead
Hotel demand moves with the season, holidays, events, flight prices and your own marketing. Comparing this month with last month mixes all of that in with your change. Comparing with the same month last year mixes in a year of other changes.
A fair test compares two groups of visitors over the same days. The only difference between the groups is the change you want to measure.
A/B tests: one change at a time
In an A/B test, each visitor is placed in group A or group B at random. Group A sees the current page. Group B sees the change. Both groups visit in the same weeks, see the same prices and come from the same mix of channels. If group B books more, the change is the likely reason.
A few rules make the result worth trusting:
- Pick the main metric before you start. For a hotel, that is usually completed bookings or booking revenue per visitor. Clicks and views are useful clues, but they are not the goal.
- Plan how long to run it. Run whole weeks, so weekday and weekend visitors are both included. A hotel website with modest traffic may need several weeks to reach a clear answer.
- Don’t stop early because it looks good. Results swing in the first days. If you check every morning and stop on a good day, you will often crown a winner that isn’t one. Either wait for the planned end date or use a method built for checking as you go.
- Keep each visitor in one group. A guest who sees version A on Monday and version B on Thursday muddles both groups.
- Test one idea at a time. If you change the headline, the offer and the button together, a win won’t tell you which part worked.
What to test first on a hotel website
Start with changes that sit close to the booking decision:
- The wording and position of your booking button.
- A direct-booking benefit on room pages, against no benefit.
- An offer with a countdown, against the same offer without one.
- Guest reviews next to the room list, against no reviews.
- An exit-intent message, against no message.
Each test should answer a question you would act on. If no result would change what you do, skip the test.
Holdout groups: what all of it is worth
A/B tests measure one change. After a year, though, you have many widgets, offers and messages running at once. The owner or general manager asks a simpler question: what is all of this worth?
A holdout group answers it. A small share of visitors, often 5 to 10 percent, is set aside. They see your website without any of the added widgets. Everyone else sees everything. Over the following weeks and months, you compare bookings per visitor between the two groups.
The difference is the combined effect of your website work, measured the same fair way as an A/B test. It also captures how widgets affect each other, which single tests miss.
A holdout has a cost. The visitors in it don’t see your best messages. Keep the group small, keep it running long enough to give a clear answer, and keep the same visitors in it so their experience stays consistent.
Report results honestly
When you share a result, include:
- The dates, the groups and the share of traffic in each.
- The main metric for each group, with a range of likely values, not only one number.
- Whether the result is clear or still uncertain.
- What you will do next.
“No clear difference” is still a useful result. It tells you a change doesn’t earn its space on the page, or that you need a bolder idea.
Count real bookings
A test is only as good as the booking data behind it. Make sure it counts completed bookings from the booking engine, not clicks on a “Book now” button. Check that revenue is compared in one currency, and decide up front how you treat cancellations.
Then connect the result to the rest of your marketing. Booking attribution shows which channel each booking came from, and return on ad spend shows whether your Google Ads and Meta budgets pay back. Together with test results, they show the website’s real share of your direct bookings.
How RetainTee does this
RetainTee runs A/B tests on its widgets and can keep a standing holdout group of visitors who see no widgets at all. Results count bookings from the booking engine, and the monthly client report shows them next to booking attribution and return on ad spend from Google Ads and Meta.