By Booking.com, Flo PMs
13 practical cases
To conduct experiments, it is not necessary to know statistics. You need to understand the basic principles of testing.
Even the CEO won’t be able to argue with you if your hypothesis is confirmed by the experiment.
Even the CEO won’t be able to argue with you if your hypothesis is confirmed by the experiment.
A/B TESTING FOR PRODUCT MANAGERS
Hands-on experience
Students love practicing with us
- 3000+alumni since 2020
- 1200+product challenges
- 4.7/5average course review
The course gave me the knowledge and skills to effectively design and analyze AB tests. I am very glad that I can immediately apply the acquired knowledge in my work. During the course, we stopped the crookedly launched AV tests, choose the metrics, counted the sample sizes, and launched them with a clear conscience and in accordance with all the rules :)
The course helped me structure my knowledge and fill in gaps in basic concepts, and that helped to establish communication between the PM and the analyst, now we "speak the same language."
The structure of the course is a balance of theory and practice. I liked that everything is explained with simple examples and reinforced with “think about it” questions, sometimes controversial ones, which not only helps to absorb the A/B testing methodology but also, once again, trains critical thinking.
The course helped me structure my knowledge and fill in gaps in basic concepts, and that helped to establish communication between the PM and the analyst, now we "speak the same language."
The structure of the course is a balance of theory and practice. I liked that everything is explained with simple examples and reinforced with “think about it” questions, sometimes controversial ones, which not only helps to absorb the A/B testing methodology but also, once again, trains critical thinking.
Learning is most engaging to me when combined with an immediate practical application of newly gained knowledge and skills, especially when results come along quickly. When our team faced the challenge of dealing with conversion metrics, I enrolled in ProductDo "A/B Testing for PMs" course as I had to learn on the go. The course provided me with a fundamental understanding of running experiments on digital products and highlighted possible errors and mistakes we could have fronted otherwise. This course covers topics like: when to use A/B experiments, confidence intervals, statistical test and power, making a decision on an A/B test, and many more. Thanks, Konstantin Grabar and Andrew Mende, for the awesome course. Below is our humble observation of the very first A/B testing, which came as an experiment itself.
The course helped me get rid of the inferiority complex 🤣. It showed that in fact, despite the strict mathematical apparatus, everything is done through a product hypothesis, and at the forefront, we have not the testing process, but the product and its metrics.
I got a specific track with the instrumentation that I use.
I know the methods, approaches and, most importantly, the solutions used in top companies (the review of services on the market was especially interesting). Now I can confidently design tests and understand how to interpret them. And most importantly, it gave me a way, an understanding of how to go further.
I got a specific track with the instrumentation that I use.
I know the methods, approaches and, most importantly, the solutions used in top companies (the review of services on the market was especially interesting). Now I can confidently design tests and understand how to interpret them. And most importantly, it gave me a way, an understanding of how to go further.
I am thrilled to announce that I have recently completed an A/B Testing course and received my certificate of completion. This course has provided me with the knowledge and skills to effectively design and analyze A/B tests, and I am eager to apply what I have learned in my work as a product manager.
A/B testing is a powerful tool for optimizing products and improving user experience, and I am grateful for the opportunity to deepen my understanding of this method. The course was expertly designed, with a balance of theory and practical examples, making the concepts easy to understand and apply.
I would like to take this opportunity to express my heartfelt gratitude to Andrew Mende and Konstantin Grabar. Their passion for teaching and commitment to excellence were evident in every aspect of the course, from the well-structured lessons to the interactive quizzes and hands-on projects. Their guidance and support have been invaluable, and I am grateful for the opportunity to learn from such talented and knowledgeable instructors.
Completing this course has been a major accomplishment for me, and I am confident that the knowledge and skills I have acquired will be of great benefit in my career. I look forward to continuing my professional development in the field of A/B testing and contributing to the growth and success of the products I manage.
Thank you again to the creators of the course ProductDo for their dedication and expertise. I am proud to have completed the A/B Testing course and to have received my certificate of completion.
A/B testing is a powerful tool for optimizing products and improving user experience, and I am grateful for the opportunity to deepen my understanding of this method. The course was expertly designed, with a balance of theory and practical examples, making the concepts easy to understand and apply.
I would like to take this opportunity to express my heartfelt gratitude to Andrew Mende and Konstantin Grabar. Their passion for teaching and commitment to excellence were evident in every aspect of the course, from the well-structured lessons to the interactive quizzes and hands-on projects. Their guidance and support have been invaluable, and I am grateful for the opportunity to learn from such talented and knowledgeable instructors.
Completing this course has been a major accomplishment for me, and I am confident that the knowledge and skills I have acquired will be of great benefit in my career. I look forward to continuing my professional development in the field of A/B testing and contributing to the growth and success of the products I manage.
Thank you again to the creators of the course ProductDo for their dedication and expertise. I am proud to have completed the A/B Testing course and to have received my certificate of completion.
I want to thank ProductDo t4eam for this ability to dig deeper into A/B experimenting, and I especially thank Andrew Mende and Konstantin Grabar for this amazing simulator. I liked it a lot and I'm a bit sad it's over now. I'd love to continue our communication :)
For whom is this simulator?
PMs with basic knowledge
A / B testing is an advanced topic: not all operating project managers can calculate the duration of the experiment, so this is an opportunity for you to stand out.
Those who want to work in big companies
A/B tests are likely to be used in big companies, so a good understanding of this tool will be your advantage as a potential employee.
Those who rely on precise data
Experimentation is the only way to see how a certain change affects the product. If you are closer to mathematics than visionary, A/B tests will become your main tool.
Your superpower after the simulator
PM Skill
Before
After the simulator
Formulating hypotheses
In general terms
Comes up with strong wording
& target metric
& target metric
Defining statistical power
Heard something about it
Can find the balance between power and duration of a test
Interpreting the results
Uses data of the online calculator
Can come up with a detailed interpretation and draw next steps
Managing experiments backlog
Prioritizes experiments just like regular tasks
Understands how to take uncertainty into account
Knowledge of statistics
Remembers the basic concepts
Cannot be fooled by randomness
How are our simulators different from others?
Leaders are expected to produce results, not theoretical ideas. Therefore, the goal of any course is to turn knowledge into a skill by solving many tasks and problems.
Students also note:
Students also note:
- structural learning approach (from simple to complex)
- specific tools
- useful additional resources
- original articles of different levels of complexity
Our key value is immersive tasks that are indistinguishable from working with a real product. While solving them, you live 10+ years of experience in a concentrated story.
What our students like:
What our students like:
- scenario game and immersion into exciting cases
- dialog format with mistake analysis, detailed feedback
- сability to work at your convenience at any time
Our products are designed for PMs who want to grow: students solve the same practical problems that experienced PMs face every day in international companies. The main goal when choosing cases is to show that products and projects can be very different. It's hard to understand if you've been working on one product for 5 years and take courses that repeat your existing knowledge.
What's cool:
What's cool:
- you can immediately apply these skills at work
- you will have a clear structural view
- you will get unlimited access to the materials
We teach you the things we ourselves use every day: for more than seven years, we've been building real products at Booking.com, Wolt, Flo.
What the student can get from it:
Also, we do not hire external instructors or lecturers, we are with our students from beginning to end.
What the student can get from it:
- practice at real tasks and fuckups of the international companies
- learn from the experience of global PMs and ask questions like “how does it work there?”
- learn how to build your career and get a job offer from a big company
Also, we do not hire external instructors or lecturers, we are with our students from beginning to end.
Who are the authors and mentors?
- Andrew MendeSr. Product Manager Data
@ Booking.com, Netherlands- Master of Business Administration (MBA)
- 11 years in Product Management
- Launched products in Yandex (search verticals)
- Trained 1,000+ PMs on various edu-platforms
Teaches: analytics, A/B tests, ML for PMs
- Konstantin GrabarProduct Owner Data
@ Flo, Lithuania- 10 years in Product Management
- Developer and Data analyst in the past
- created and launched A/B test system in Flo
Teaches: data, A/B tests, platform products
How does learning with the simulator work?
Course plan
Throughout 8 chapters of the simulator, you'll immerse yourself in real product situations and apply A/B experimentation to help the team advance with their product.
The most important thing is to apply experiments where they are really needed. This is the most accurate but at the same time expensive way to test product hypotheses.
You will learn:
→ What is an A/B test
→ Product hypothesis as the basis of the experiment
→ Alternative ways of making decisions
→ The Magic of Randomization
→ Pros and cons of А/B experiments
You will learn:
→ What is an A/B test
→ Product hypothesis as the basis of the experiment
→ Alternative ways of making decisions
→ The Magic of Randomization
→ Pros and cons of А/B experiments
Yesterday the conversion was 1.5%, and today it is 3.2%, although we did not change anything. How randomness manifests itself in data, and why it is important for a product to have skills in working with random variables.
Что ты узнаешь:
→ What is a sample and general population
→ How to interpret observations
→ How many observations are enough
→ What is a confidence interval
→ How to use CI to make decisions
Что ты узнаешь:
→ What is a sample and general population
→ How to interpret observations
→ How many observations are enough
→ What is a confidence interval
→ How to use CI to make decisions
For a product manager, A/B tests are like a car: you don’t need to know the injector or catalyst device thoroughly to get to the office. But you need to understand how the brake differs from the throttle, where to turn the headlights on, and what the braking distance depends on. Here we will look at the parameters that the PM controls.
You will learn about:
→ Statistical test as it is
→ Statistical hypothesis of the experiment
→ Interpreting the results
→ Type I and II errors
→ Calculating test power in practice
You will learn about:
→ Statistical test as it is
→ Statistical hypothesis of the experiment
→ Interpreting the results
→ Type I and II errors
→ Calculating test power in practice
The first practical task the Product Manager has to solve is to define how long should the A/B experiment run in order to be able to make reliable conclusions. And it's not only relevant for the tests you are preparing to launch. Actually, the first quick estimation of the required duration should be a part of the hypothesis prioritization process too.
You will learn:
→ What is the statistical power of the experiment
→ How to calculate the duration of the experiment
→ What is observed p_value
→ How PMs calculate test duration in practice
You will learn:
→ What is the statistical power of the experiment
→ How to calculate the duration of the experiment
→ What is observed p_value
→ How PMs calculate test duration in practice
Here you will learn why clicks or CTR are almost always bad as target metrics, what is cannibalization, and how it affects the experiment results.
You will learn about:
→ Target metrics of A/B tests
→ Self-fulfilling prophecies and other mistakes when choosing a target metric
→ Supporting metrics
→ Control metrics, "accidental" discoveries, and false positive results
→ Procedure of making a decision after the experiment was finished
You will learn about:
→ Target metrics of A/B tests
→ Self-fulfilling prophecies and other mistakes when choosing a target metric
→ Supporting metrics
→ Control metrics, "accidental" discoveries, and false positive results
→ Procedure of making a decision after the experiment was finished
Not all metrics are convenient for A/B testing. Some of them are much easier to work with just because of the technical parameters of their distribution.
You will learn:
→ КHow statistical noise affects test power
→ What metric distributions suit the experiments best
→ What to do if the target metric has not optimal distribution
→ What other metric properties can be of importance in experimenting
You will learn:
→ КHow statistical noise affects test power
→ What metric distributions suit the experiments best
→ What to do if the target metric has not optimal distribution
→ What other metric properties can be of importance in experimenting
The next step is to figure out how to make a decision based on the results of an A/B test. This may not be as obvious as it seems at first glance. Here you will learn about a possible decision-making algorithm and practice on industry examples.
You will learn about:
→ Decision-making procedure after the experiment
→ Life cycle of the experiment
→ Possible actions if the results are inconsistent
→ Following steps after the experiment is over
You will learn about:
→ Decision-making procedure after the experiment
→ Life cycle of the experiment
→ Possible actions if the results are inconsistent
→ Following steps after the experiment is over
Finally, we will see how all the topics studied are combined in the design and conduct of the experiment. We will discuss what tools can be used in different situations.
You will learn about:
→ Life cycle of the experiment
→ Checklist of the experiment starting
→ What experiments look like in code
→ Monitoring experiment state
→ Working with one hypothesis at a time, and why no more
You will learn about:
→ Life cycle of the experiment
→ Checklist of the experiment starting
→ What experiments look like in code
→ Monitoring experiment state
→ Working with one hypothesis at a time, and why no more
Pricing
€450 €349
- 200+ practical challenges
- 5 real industry use cases
- PM checklists to use at work
- Unlimited access
Our students work at
- TikTok
- Gett.
- DECATHLON
- Revolut
- Microsoft
- Booking.com
- <epam>
- Acronis
Still not sure?
→ Checkout more student reviews on LinkedIn
→ Try a free simulator lesson - 60 minutes of practice, more than 20 actual tasks that give an idea of working as a PM in an international company. It's also a quick and easy way to understand how the ProductDo simulator teaches.
→ Try a free simulator lesson - 60 minutes of practice, more than 20 actual tasks that give an idea of working as a PM in an international company. It's also a quick and easy way to understand how the ProductDo simulator teaches.
We'll send you a link to the simulator. One click - and you're on the first task.
Answering your questions
No, you don't have to do it. We give you an introductory theory with lots of additional materials, and the tasks are built with a smooth increase in complexity. But if you have zero experience, it will take more time and effort.
Of course! All our graduates immediately become VPs and CEOs in top companies :) Jokes aside, our main goal when choosing cases and examples is to show that products and projects can be very different.
Not everything is an online store, not everything fits into the AARRR metrics, and not everything uses Yandex.Metrics, etc. But this is very difficult to understand if you have been working on one product for five years and taking courses that repeat your existing knowledge.
This is all complicated by the mechanism of our brain which always understands everything. There is a great book about this from a Nobel laureate, a must-read for every PM.
Therefore, in our courses, we try to break your ideal world where the product is always a "website", users tell their dreams on custom dev, and prioritization is a rational choice according to the formula. We talk about products that are not a website at all, such as a chat service or a taxi platform, show several cases when the PM tells users what they need, and not vice versa, and share the choice mistakes we have encountered.
Doing so, we create contradictions in your head, that later will turn into knowledge, and then, into a visual experience, e.g., you become "well-watched." And then you are able to convert this knowledge into skills, and in about seven years, into a VP or CEO role in some big company.
Not everything is an online store, not everything fits into the AARRR metrics, and not everything uses Yandex.Metrics, etc. But this is very difficult to understand if you have been working on one product for five years and taking courses that repeat your existing knowledge.
This is all complicated by the mechanism of our brain which always understands everything. There is a great book about this from a Nobel laureate, a must-read for every PM.
Therefore, in our courses, we try to break your ideal world where the product is always a "website", users tell their dreams on custom dev, and prioritization is a rational choice according to the formula. We talk about products that are not a website at all, such as a chat service or a taxi platform, show several cases when the PM tells users what they need, and not vice versa, and share the choice mistakes we have encountered.
Doing so, we create contradictions in your head, that later will turn into knowledge, and then, into a visual experience, e.g., you become "well-watched." And then you are able to convert this knowledge into skills, and in about seven years, into a VP or CEO role in some big company.
No, the simulator and additional materials will be available to you forever, so go at your own pace. Please note that, as in life, you cannot go back and "live" the task again: all your progress will be preserved in its original form.
We help by giving you a bunch of practical tasks and sharing our mistakes and experience. We have already built so many products that with our examples we can help you formulate your area (more technical or more payment related to ML / AI, or sales, etc.). We believe that a good PM chooses their favorite product themselves and does not need to be led by the hand.
Yes, students who have successfully passed the course may generate a certificate и and proudly add it to their LinkedIn profile to Licenses & certifications, or on Facebook in the Schools section.
Yet do keep in mind that upon applying to a product company, certificates do not really increase the chances of being hired, пat least in our practice, it has never happened. If a resume mentions a certain course, then this may raise questions about what the candidate really knows right now. Therefore, you need to solve problems and develop skills, and not chase certificates. Fortunately, our courses have enough practice.
Yet do keep in mind that upon applying to a product company, certificates do not really increase the chances of being hired, пat least in our practice, it has never happened. If a resume mentions a certain course, then this may raise questions about what the candidate really knows right now. Therefore, you need to solve problems and develop skills, and not chase certificates. Fortunately, our courses have enough practice.
You can be sure of the quality of the materials and the experience of our mentors. We've been teaching for many years, and we have thousands of happy PMs. Take a look at the student reviews, they are real and have social media links.
If we cannot not find a suitable program for you, we will fully refund your payment. No questions asked.
If we cannot not find a suitable program for you, we will fully refund your payment. No questions asked.
- Andrew, ProductDoWe hope we answered all the questions. No? Contact us via any messenger plug-in at the bottom right, or drop us a line at support@productdo.io, and we'll help you.