Made in collaboration with Everton Guilherme.
How I convinced my team to run a fake door experiment (and the results).
Back in the craziest year of our lives, 2020, in the middle of an effort to lower the interest rates we offer to Nubank's customers today, I was helping my PM to define what the next product hypotheses to be tested should be. Due to the engineering complexity, business prioritisation and greater opportunity, we decided to move forward with: adding the possibility of paying your bills with your credit card via boleto. (If you're not from Brazil: "what is boleto?")
And them, my PM and I started to discuss our discovery agenda and the conversation was a little bit like:
- "Ok, to assess this feature and validate our hypothesis we need to know...."
And then what followed was a list of more than 50 questions that we wrote on our product assessment document as a way to say "God help us we know nothing". But, as always, we wanted to validate it quickly so we wouldn't waste a large amount of engineering or other investment before knowing if customers would want it or not.
Normally, the basics of a product strategy are: knowing about the product economics, portfolio vision, business goals and to know everything about the competitive arena. Is there a way for us to tackle almost single part of this strategy in a fast way? Maybe.
Enter fake door test
"What if add a button with the option that we want to develop, but when the customers clicks on it, they'll see a 'Thank you for your interest!' screen?".
And the team was like:
"Aren't you supposed to advocate for our customers, Amanda?"
I swear I heard that.
But I wasn't trying to sell them a dark pattern test and I argued back with 05 important points:
(01) It depends on how we approach the solution. We can design a success screen asking if they want to be the firsts to use that feature. It will be good for them and we'll have a list of possible beta testers more susceptible to bugs and giving honest feedback.
(02) The population of the test will be small and random.
(03) After the experiment, we'll run a qualitative research to understand more deeply how they felt and reacted to a fake door test.
(04) We can check the conversion metrics on the selected entry point (and map others events on screen) and help the business team to better shape the economics.
(05) I give you my word that we will never do this again if it fails somehow and we should write down together what failure and success means.
So, we did it.
And I started designing the experiment. As the feature was a new payment method, the entry point would have to be the same (Pagar/Pay), but the next screen (when our customers have to choose how they want to pay), we added our fake button: "pagar boleto no cartão de crédito" (pay with credit card).
After choosing credit card as a payment method, the next screen showed the message that the feature wasn't yet ready and asked them if they would like to be the first to use it.
And the results were:
(01) A better understanding of which users would be willing to pay their boletos with their credit cards. From the 120k customers on the rollout, 77k accessed the screen containing the fake door test, from those, 9.7k clicked on the CTA "pay with your credit card" and finally 3.6k clicked on "I'm interested" button to join the wait-list, and 2k clicked on the "Not Interested" button (the other customers either got back to the previous screen, or closed the flow).
With that, considering only the customers that actually took an action after engaging with the painted door, 65% are interested and 35% are not interested in paying boleto via credit card.
(02) A waitlist for better qualitative surveys and a refinement of our usage profile hypothesis. We interviewed customers that were participating in the painted-door test, both customers that joined the waiting list and those that didn't seem interested in the feature. And when talking to them, we understood that:
(02.01) Our main affected population/who would be interested in this feature are those seeking for benefits (rewards, promotions), people who want to concentrate all their spend in one place (one credit card), and people who have no money in their account at the time of payment.
(02.02) Thanks to that insight, we decided to focus our value proposition on customers that wanted to concentrate all their spend in one credit card.
(03) A deeper understanding of how our users react to a fake door test. While interviewed customers did not feel frustrated by the fake door experiment, most of them did not remember going through it and in multiple occasions expressed different intents than those collected by the fake door.
Low friction, highly forgettable.
I also learned some things about fake door test. Here's what you need to know:
(01) We know that customers will forget about the test/screen. How can we make it less forgettable? Adding the pin code as a confirmation of the "fake" button can be a solution.
(02) The screen where we ask our customers if they are interested in being the first to use the product was originally design with a "I'm interested" and a "Not now" button. The data scientists raised an important question by the time: "we should compare who's interested and who's not. By letting they choose "not now", we'll never know if they were interest or not".
And that was it!
The final feature is still :wip: but once it's finished I'll come back to show you how I designed the complete flow and experience :)
Here's a spoiler:
