What should I be testing?
A 4-step guide to unlocking a step change in results
This guide is specifically for early-stage startups that need help finding traction or later-stage businesses that are trying to reach new audiences or markets.
All startups know unlocking growth is all about testing. But even with the right A/B test tools in place, a clear testing roadmap, and weekly experiments, many startups aren’t any closer to their goals.
The #1 question founders ask me is: “Am I testing the right things?”
One founder I recently worked with had been trying to unlock growth for her functional medicine startup for over two years. She hired and had to fire multiple marketing team members. She tried email, Google ads, SEO, and Facebook. She felt like she had tested everything but learned nothing, and revenue had been flat for months.
This is the worst feeling: out of money, out of time, and feeling hopeless.
The first question I asked her was: “Is your testing roadmap designed to unlock a step change?”
What is a Step Change?
Expectations for the results of a testing roadmap are very different from reality.
Many startups expect to set up the roadmap and see predictable, exponential growth month-over-month. Eventually, great companies achieve this type of growth. But first, every company goes through the reality of choppy performance.
What actually happens is growth is flat for a while until something clicks and then performance jumps—we call these “step changes” because they look like steps on a graph.
Early-stage startups who are struggling to find traction and later-stage startups who need to pivot to new audiences should focus on finding step changes in results rather than making smaller optimizations. Why?
There’s a high degree of uncertainty about who the product is for and what value it unlocks. Focusing too narrowly on optimizing one part of the experience could be a waste of time if you’re testing in the wrong area.
At this stage, marketing efficiency is quite poor. Smaller improvements will take too long to achieve a sustainable business model.
Discovery vs. Optimization
In practice, looking for a step change in performance requires a “discovery” mindset.
Both processes share the same underlying goal (accelerating growth), and some of the same mechanics (A/B tests, testing roadmaps), but the test ideas and learning cycles look different.
I learned this the hard way when I switched from leading marketing at Caviar, a $300M+ brand, to leading marketing at Realm, a startup with <$1M in revenue.
The three mistakes I made and see my clients making:
Test ideas are too scattered. Startups test a bunch of assorted tactics, such as testing one Google campaign, a few Facebook ads, and some edits to the signup flow. This doesn’t work because most test ideas fail—they don’t drive growth. If you’re failing at many different things at one time, those failures don’t add up to a clear answer on where to focus next.
Imagine you’re playing a game of Clue, but instead of guessing three things (person, weapon, room), you have to guess 10 without a scorecard to keep track of what you’re learning. That’s what this approach feels like.The testing roadmap is focused on tactics rather than strategy. Startups test tweaks, such as A/B testing the headline copy on a landing page or changing the color of an ad. This doesn’t work because in discovery mode, you need to unlock big learnings about who your product is for and what value it offers. Focusing on minor changes might unlock a win here and there, but you’re not answering your big questions.
Imagine you’re playing that same game of Clue, and you start in the library. You decide to try every combination of person and weapon “in the library” before moving on to the next room. You’re wasting valuable time while your opponents are off learning faster than you.Tests only focus on ads and landing pages. Startups relegate growth testing to marketing, keeping everything about the product experience the same. This can work if you already have strong evidence that the product experience works for a particular cohort of customers (i.e., happy paying customers who share similar traits). But you can often get to the winning combination much faster if you test versions of the whole user experience, including the purchase or sign-up flow and the product itself.
Imagine you’re playing that same game of Clue but only making guesses about the weapon. (You would NEVER win Clue with this strategy!)
How to Shift Your Testing Roadmap to Focus on Discovery
If you’re a founder or a marketing leader who needs to 5x the performance of your funnel, don’t fall into the trap of optimizing a funnel that isn’t yet working.
Follow these 4 steps to shift to a discovery mindset and build a testing roadmap that helps you discover who your product is for and what value it unlocks.
1. Be realistic about how much you need to improve.
Do you need to improve CAC by 23x? 5x? 10x?
The incremental approach works if you only need to improve by 20% or 50% in a year. But if you have a big gap, you likely don’t have product-market-channel fit and you need to be in discovery mode.
For startups with a CAC 5x more expensive than is sustainable, I recommend taking significant swings across 3 core elements.
Positioning: Do people understand your product and why it’s better than the alternatives? Are you going after the right audience? [Read more about positioning here]
User Experience: Is the way people buy your product now how they want to buy it? Are you clearly articulating the value throughout the experience?
Marketing Channels: Are you reaching the right audience? What do your ads communicate? Do the landing pages line up with the ads?
Sidebar: The rest of this article will focus on startups needing a step change of ~5x. For startups that need 10x, the best answer is often to blow up the whole funnel and start from 1st principles. Go back to your customer hypotheses and ask yourself:
What is the problem my product is best at solving?
What are my leading hypotheses about who (1) feels this pain point deeply, (2) loves all the things my product is best at, and (3) doesn’t mind the things my product isn’t great at?
What would the ideal user experience look like for this customer? Where does my current
Often, you’ll end up with 2–3 combinations of a target persona x use case that your product can solve. I recommend investing in a minimum viable product experience tailored to each and testing them to see if you can get a stronger signal on where to double down.
2. Identify your leading hypotheses about what drives customer behavior.
Don’t focus on generic hypotheses, like “Customers are motivated by price” or “Customers don’t trust us.” Dig deeper with questions like:
What problem are they trying to solve? When did they become aware this was a problem?
What is happening in their life that has prompted them to start seeking a solution?
Why are they feeling so frustrated with their current situation?
What are the workarounds they’re using now? Why aren’t they working?
How this works in practice
Back to the functional medicine founder I mentioned earlier. When we looked at her data, we saw pockets of conversion and retention, but overall, the funnel was inefficient. The product was being marketed broadly to anyone experiencing chronic symptoms. Based on the commonalities among those who converted and retained, we identified 3 key customer hypotheses.
Customers are only willing to pay for an out-of-pocket option once they have tried the traditional medicine options covered by their insurance.
Customers are more willing to buy if motivated by a life-stage event, e.g., trying to conceive, post-partum, or menopause.
Customers are symptom-centric and are more willing to buy if they believe and understand that this product works for someone with their exact symptoms.
3. Build a roadmap of divergent ideas.
At this point, you understand which part of the experience to focus on as well as which customer motivations you’re tapping into or barriers you’re reducing. Now is when you start to come up with ideas.
The #1 mistake I see people make is focusing on the ideas from the beginning. Why doesn’t this work? When you’re in discovery mode, you can test thousands of ideas. You can spend years testing ideas and making no progress. Starting with the customer problem ensures that even if (when) your ideas fail, you’ve learned something valuable about who your product is for and the value it offers.
Let’s bring this to life a little more: Startups A and B are the same size and can each run 20 tests a month.
Startup A generates a list of test ideas and prioritizes them based on impact and effort.
They run 5 tests on lifecycle emails, 5 on influencer ads, 5 on landing pages, and 5 on pricing.
At the end of the month, they found 1 win in each category.
They improved gross margin by 20%.
Startup B identifies a challenge with bringing the right traffic to their website.
They run 20 tests, all focused on attracting the right customer.
5 tests focus on higher-income leads on Meta, 5 tests focus on testing a new persona on Meta, 5 on targeting the current persona Google, and 5 focus on a new creative strategy to help self-quality leads on Meta.
They found multiple wins within the new persona and new creative strategy categories and improved gross margin by 20%.
Same result but now they have a clear point of view on what to focus on next month and how to expand upon what worked.
Fast-forward two months. Startup B will be exponentially ahead of Startup A because they not only found some successes but also significantly advanced their understanding of who their product was for.
How this works in practice
The functional medicine founder needed to 5x her monthly gross profits to become profitable and stay in business. We looked at the numbers and determined this would need to be a combination of a change in pricing strategy and a reduction in CAC.
The first thing we needed to do was improve the quality of traffic coming to the site from Meta by focusing on the right audience, who would be motivated to take the next step in their health journey.
We used the audience hypothesis mentioned above to design the roadmap and structured it in 2 phases:
Phase 1: Ads. Testing ads and landing pages targeted at different personas to find the most valuable audience.
Phase 2: Purchase Funnel. Testing the sign-up flow, sales call, and product packaging (description, pricing) to maximize results from the newly refined target.
With this structure in place, we generated 20+ ideas for each theme. We scored the ideas based on their potential to unlock a 2–5x improvement. We weeded out the incremental ideas and discussed how to execute the higher-effort ideas with less work.
4. Create a learning engine.
With the roadmap in place, you can slide back into execution mode. I typically recommend one month of testing around a specific customer hypothesis. At the end of the month, determine if there’s enough signal to keep iterating or if you need to revisit your hypothesis.
How this works in practice
For this functional medicine business, we first set up distinct ad groups for our ~6 customer personas to test them against one another.
Set up 6 ad groups on Meta, each with the same budget.
Included 1 creative for each, using the format that had worked previously (UGC-style video).
Sent them to landing pages that spoke directly to the segment and pain points.
Looked for step change differences in quiz starts/impressions.
Why this metric? Creating custom quizzes for each was a lot of work, so we wanted to quickly hone in on the top 2 personas out of the 6 we were testing.
We ran this for ~2 weeks and then narrowed in on 2 new audience segments. Since we were looking for a minimum of a 2x to conversion rate we didn’t need to wait as long to get a read.
From there, she tested different creative hooks for each segment and went to work on her user experience (quizzes, sales calls).
The result?
In 3 months, she 3x’ed her monthly revenue, after it being flat for over a year.
Summary
If you feel stuck in an endless loop of testing and not getting results, take a step back.
Do you need to make a step change to your funnel, or will incremental improvements get you close enough to your goal?
If the former, what are your hypotheses about why prospective customers don’t buy your product? What do you know about customers who purchased from you? What motivated them to try your product?
What tests can you design to help you answer those questions?
In a discovery roadmap, the interim milestone is not results but conclusively determining what’s worth doubling down on and what’s not. The faster you can hone in on where to focus, the closer you’ll get to unlocking that significant change in performance.
Questions or comments? I’d love to hear from you.
Love this article, you can’t go wrong by starting with a customer problem.