How to Run High-Impact Marketing Experiments: A 7-Step Process

Want the full story? Listen to my complete interview with Sundar Swaminathan, where we discuss how Uber built a data-obsessed culture and the secrets behind their marketing experimentation success.

We all know experimentation is important, but here’s a shocking stat: 90% of marketing experiments fail. Not just fall short of expectations - completely fail. That’s enough to make anyone hesitate before running their next test.

But here’s the thing: those “failures” are actually the secret to breakthrough success. As Sundar Swaminathan, former head of US/Canada performance marketing analytics at Uber, shared with me, “You have to be in such a psychologically safe environment that allows you to fail 90% of the time and people still think that’s a good thing.”

Today, I’m breaking down the exact 7-step process that successful companies use to run high-impact marketing experiments - the same process that helped Uber launch massive wins like Uber Eats (which started as a simple experiment with drivers carrying pre-packaged meals).

Step 1: Start with Customer Insights

The foundation of any successful experiment isn’t your brilliant idea - it’s understanding your customer’s actual problems. As Sundar emphasizes, “Our job as marketers, as product people, as just people building companies is to solve customer problems, but they should be the ones telling us the problem.”

This means:

• Conducting customer interviews
• Analyzing support tickets
• Reviewing user behavior data
• Gathering feedback from front-line teams

Don’t skip this step. The biggest experiment failures often come from solutions searching for problems, rather than addressing real customer pain points.

Step 2: Generate Multiple Hypotheses

Once you understand the customer problem, resist the urge to jump straight to your first solution. Instead, generate multiple hypotheses about potential solutions.

For example, when Uber noticed poor pickup experiences around the DC arena, they could have:

• Created designated pickup spots
• Added more driver education
• Implemented geo-fencing
• Adjusted pricing during events
• Modified the app interface for high-traffic areas

The key is to avoid getting emotionally attached to any single solution. As Sundar notes, “If you go into it with ‘I’m gonna try to find things that work,’ it’s just not [going to succeed]… You go into this with the idea that you know absolutely what’s going to work. If that’s the case, then why even experiment on it?”

Step 3: Prioritize Ruthlessly

With limited resources, you can’t test everything. This is where many companies stumble - they try to run too many experiments simultaneously or chase low-impact wins.

Sundar recommends spending about an hour each week on prioritization, working closely with stakeholders to identify:

• Potential impact size
• Resource requirements
• Technical complexity
• Strategic alignment
• Learning value

Remember: it’s better to run one high-impact experiment well than ten mediocre tests poorly.

Step 4: Design the Experiment

This is where the science meets the art. A well-designed experiment needs:

Clear Success Metrics:

• Primary metric (what must move)
• Secondary metrics (what should not break)
• Guard rails (what cannot be compromised)

Clean Methodology:

• Random assignment
• Appropriate sample size
• Control for external factors
• Clear start/end criteria

Documentation:

• Hypothesis statement
• Expected outcomes
• Test parameters
• Analysis plan

As Sundar shares from his Uber experience, “We had that we would try to correlate and validate this with [multiple data sources]… The data was just there.”

Step 5: Execute Flawlessly

Even the best-designed experiment can fail due to poor execution. Key considerations include:

Technical Implementation:

• QA testing
• Monitoring systems
• Backup plans

Timeline Management:

• Clear milestones
• Regular check-ins
• Defined decision points

Stakeholder Communication:

• Progress updates
• Early warning signals
• Expectation setting

“The only way you get promoted is if you make impact,” Sundar notes. “Well, you can only make impact if your recommendations get absorbed, if they get implemented, people go take actions on them.”

Step 6: Analyze Rigorously

Analysis isn’t just about determining whether your hypothesis was correct. It’s about extracting maximum learning value from the experiment.

Sundar recommends spending 3-4 days on analysis, including:

Primary Analysis:

• Statistical significance
• Effect size
• Confidence intervals

Secondary Analysis:

• Segment performance
• Unexpected effects
• Interactive effects

Meta Analysis:

• Pattern recognition
• Comparative results
• Future implications

“Don’t ask an analyst to fudge the numbers,” Sundar warns. “It’s just shitty for an analyst because you’re put in a really crappy situation.”

Step 7: Close the Loop

The final step is often overlooked but crucial: ensuring learnings drive action. This includes:

Documentation:

• Key findings
• Methodology details
• Future implications

Communication:

• Stakeholder updates
• Team learnings
• Executive summaries

Action Planning:

• Next steps
• Resource allocation
• Timeline commitments

“Follow up a week later,” Sundar advises. “Be like, ‘Hey, I did this amazing work, what happened with it?’”

Building an Experimentation Culture

Remember: this process isn’t just about running better experiments - it’s about building a culture of experimentation. As Sundar emphasizes, “We don’t need to prove experimentation works… What you need to prove is do we have the right culture to make experimentation work at our company.”

This means:

• Celebrating learning from failures
• Investing in data infrastructure
• Training teams in experimental thinking
• Creating psychological safety
• Rewarding data-driven decisions

The most successful companies don’t just run experiments - they build experimentation into their DNA. As Sundar notes from his Uber experience, “I saw so many times decisions being made by the most junior people who just had really good data, really good evidence, a really good story to supersede a more senior person.”

Start small, but start today. Pick one customer problem, generate multiple hypotheses, and run your first well-designed experiment. Remember: even if it “fails,” you’re still ahead of 90% of companies who never try.

Want to hear more insights from Sundar Swaminathan, including how Uber built their legendary data culture and made multi-million dollar decisions through experimentation? Listen to the full interview here.