LLM & Evals
for Product Managers

Everyone talks about LLM magic, but it all starts with a prompt. This lesson is about talking to a model so it solves a business problem instead of just making conversation.

In this lesson:
→ What makes a prompt good, and why it's not "just text"
→ What temperature is and how to control it
→ Why the same prompt gives different answers, and what to do about it
→ How to check whether the model handles the task: your first eval

What it looks like:
→ Write, test and improve a prompt for a specific task
→ Case: replacing manual support ticket tagging with AI

A prompt that works is only the start. Next comes the actual work: finding where the model fails and pushing accuracy to a level you'd ship.

In this lesson:
→ Few-shot: how examples in the prompt lift accuracy
→ How to read eval results and find the categories the model gets wrong
→ Why narrowing the task often beats writing a longer prompt
→ How to decide what to measure when one category matters most to the business

What it looks like:
→ Improve the prompt step by step and watch what happens to accuracy
→ Case: getting ticket classification to a level you could launch

LLMs are good at open conversation, but they can also answer in a strict, predictable format. That's what turns a model from a chatbot into a piece of your product other services can work with.

In this lesson:
→ Why products need structured output and where it matters
→ How to write a JSON schema the model will follow
→ Why good field descriptions and examples beat a longer prompt
→ How domain expertise turns into instructions for an LLM

What it looks like:
→ Write a schema, run it on real queries, fix what doesn't line up
→ Case: search that turns a user's question into catalogue filters

There are a lot of models and they're all different. This lesson is about choosing between them with numbers instead of marketing pages, and knowing upfront what a feature will cost.

In this lesson:
→ What you're actually paying for: tokens, input and output
→ How instruct models differ from reasoning models, and when you need the latter
→ Why a model's size and its type are two separate things
→ When self-hosting an open-source model makes sense, and when it costs more than you think

What it looks like:
→ Work out what a feature costs and compare models on the same dataset
→ Case: picking a model for ticket classification, weighing accuracy against the cost of an error

One big prompt or a chain of small ones? This lesson is about sorting out a messy stream of incoming messages, and why splitting a task into steps usually beats writing longer instructions.

In this lesson:
→ When a single prompt stops being enough
→ What routing is and how it works in a product
→ Why every step in a pipeline needs its own eval
→ How an explicit pipeline differs from a reasoning model

What it looks like:
→ Build a pipeline from several prompts and check each step separately
→ Case: one assistant handling both support questions and product search

Until now, evals compared the model's answer to a label. Here there's nothing to compare against: one request can have many correct answers. This lesson is about measuring quality when there's no single right answer.

In this lesson:
→ How to write quality criteria people won't read differently
→ What LLM-as-judge is and when you can let a model do the scoring
→ How to check whether the judge is right often enough to rely on
→ What data drift is, and why a good eval stops being good over time

What it looks like:
→ Write criteria, run the eval, and check it against human labels
→ Case: scoring the AI stylist's dialogs and taking the check into production

An LLM product talks to users freely, and that's its strength. But the same freedom means it can respond to a provocation, leak its instructions, or wander into topics where your company has no business giving advice.

In this lesson:
→ Where the lines are: off-limits topics, grey areas, and requests that look risky but aren't
→ How teams actually defend: from keyword checks to a dedicated pipeline step
→ Why safety testing needs two different datasets
→ Why the bar here is higher than for quality

What it looks like:
→ Tune the assistant to decline gracefully without losing accuracy on normal requests
→ Case: locking down the AI stylist against provocations and prompt extraction

The course ends here, but a few things stay with you. We pull it all together and hand over what you'll actually use afterwards.

In this lesson:
→ A PM's checklist: situations, a glossary, and the checks to run before launching an LLM feature. Lives in Notion, duplicates to your workspace in one click
→ A LinkedIn-ready certificate
→ A discount on other ProductDo simulators
→ What to learn next so AI isn't your only strong skill

What it looks like:
→ Look back at the whole path: from a first prompt to evals and safety
→ Grab the checklist and decide where to go next

Highlight business impact: Explain how AI-prototyping knowledge will help you build smarter products that solve real customer problems and drive growth.
Emphasize upskilling ROI: Show that investing in your skills now means faster delivery, better decisions, and fewer costly mistakes down the road.
Connect to company goals: Align the course benefits with your company’s strategy, whether it’s innovation, efficiency, or entering new markets.
Stress team advantage: Point out how your new AI-prototyping skills will uplift your whole team’s expertise and collaboration.
Assure minimal downtime: Reassure your employer the course is hands-on, flexible, and won’t disrupt your current workload.
Simplify payment: Let them know we can issue an invoice if needed, making the process easy for company reimbursement.

Contact Sales

The cases are really real and reflect the diversity of products in the industry: not everyone has a "website" or a "store". In the "Fundamentals" course, we cover the clothes rental start-up; in the "Tech for PM" simulator, we will look at a taxi ordering app (like Uber), a music service (like Spotify), and a food delivery app. In the simulator "A/B testing" we will dive into experiments on the Meditation service, and in "Product Discovery" we will analyze the business of renting clothes.

The more cases you solve (and not just read/watch), the more experienced a product manager you become. There is no other way.

One of the distinctive features of ProductDo is that all our courses are packaged in a simulator format. We can say that this is an important element of our product vision. The simulator as a learning tool has many wonderful properties:
- it requires constant active participation of the student. In order to move forward, you need to choose some action or answer a question every time.
- it allows you to give detailed feedback on any action: You can justify the preferred option with links and examples
- it allows you to move at your own pace and does not burden those who do not really like to work in a group with unnecessary communication.

Usually, they teach like this: theory → practice. If you have mastered the theory well, then you do the practice 100% or so. We have a completely different methodology. We start with a practical task that needs to be solved (and if the course suits you, then you don't know how to do it yet). Only when you have tried to solve the problem, we bring up some theory behind it. We believe that the cycle: trying to think independently → making a decision → feedback on a decision is a very effective way to master new material, especially for those who have already reached some professional level.

If you want to try the simulator format, take a free lesson. It contains questions from different areas of product management. At the same time, you can evaluate which skills you already have enough pumped, and which can be further enhanced.

We help by giving you a bunch of practical tasks and sharing our mistakes and experiences. We have already built so many products that with our examples we can help you formulate your area (more technical or more payment, related to ML / AI, or sales, etc.). We believe that a good PM chooses their favorite product themselves and does not need to be led by the hand.

The entire history of the simulator with comments and answers is saved. Access to the materials remains forever.

Yes, students who have successfully passed the course may generate a certificate and proudly add it to their LinkedIn profile to Licenses & Certifications, or upload it to Facebook in the Schools section.

Yet do keep in mind that upon applying to a product company, certificates do not really increase the chances of being hired, at least in our practice. If a resume mentions a certain course, then this may raise questions about what the candidate really knows right now. Therefore, you need to solve problems and develop skills, and not chase certificates. Fortunately, our courses have enough practice.

You can be sure of the materials' quality and our mentors' experience. We've been teaching for many years, and we have thousands of happy PMs. Feel free to take a look at the student reviews for verification.

If you passed less than 30% of tasks of the simulator you could refund your money in full.