3 Non-negotiable Filters To Pass Before You Experiment

“Do you use any framework to state the experiment?”

“How do you keep track of different experiments?”

I have been running experiments since 2018. They ranged from A/B tests on large sample audiences, to sequential MVPs and everything in between down to process experiments within the team.

I took more Losses than Wins, but that’s how you learn.

Here are some non-negotiables…

1️⃣ Define the Dilemma

When I first started doing product experiments, I thought I just needed a bunch of cool experiment ideas and enough time to run them.

I should’ve known better.

Netflix and Booking.com boasted running 100’s of experiments a day, and I thought, “I can do that too.”

I had enough traffic to get meaningful results, but that’s not the point.

When you know a problem, there’s a dilemma, an inability to make progress.

Without understanding the crux of the dilemma, everything else you do is garbage.

Example:

prodmgmt.world customers often complain about company culture.

They can now find the right things to do with the product, but it’s hard in their org.

It’s not frameworks. Missing is a safe place to apply them.

Other people see the tangle differently. Examples follow.

2️⃣ Define the Mechanism of Change

But it’s not enough to know the dilemma.

You can still be powerless before it.

And so the second part of this is knowing what realistically can resolve it, and why.

A lot of bad experiments take the shape of “mess around and find out”.

Knowing how your intervention will work is important. Stop fantasising.

Be a Bayesian thinker.

Why would something that has never worked work here?

Has something like this worked elsewhere?

What is the mechanism by which this experiment will solve the dilemma?

Faced with this question, most will stutter, and so have I, many times.

You should go back and examine your plans.

Example:

Marty Cagan wrote “Empowered” for the execs at such companies. A book can spread quickly. Since the Bible, this method has worked. It’s a precedent.

GoPractice said: “We’ll help you practise. You can use this skill elsewhere, but you won’t feel impostor syndrome in your current role.” They added a practical element to courses to make them more effective.

3️⃣ Define the Conditions for the Decision

This trips me up to this day.

What measure will tell you that the experiment “works”?

How does Marty decide if he needs to do more if he considers “Empowered” an experiment? Is the book “working”?

Importantly, revenue won’t tell him that. Revenue will tell if he found the right tangle to solve, and that people believe in the method that he devised.

But how will he know if the culture is shifting?

So I want you to define:

• What leading metric can tell you if you’re having an effect on the dilemma?

• At what thresholds will you say Stop, Pivot or Continue?

It’s not easy, but I hope this helps a bit.