Building AI into Your Product Strategy
Working with ML Engineering Teams
5 min read
ML engineers think differently than traditional software engineers. Understanding their world helps you collaborate effectively.
The PM-ML Communication Gap
| What You Say | What They Hear | Better Way to Say It |
|---|---|---|
| "Can we make this smarter?" | Vague, undefined request | "Can we improve precision from 85% to 90%?" |
| "Why is this taking so long?" | They're slow | "What are the blockers? How can I help unblock?" |
| "Users want better results" | Undefined success | "Users report 20% of recommendations are irrelevant" |
| "Just use AI for this" | Naively simple | "Would ML be appropriate here? What would it take?" |
Key ML Terms for PMs
Model Metrics
| Term | Definition | Why It Matters |
|---|---|---|
| Precision | Of all predicted positives, how many were correct? | High precision = fewer false alarms |
| Recall | Of all actual positives, how many did we catch? | High recall = fewer missed cases |
| F1 Score | Balance of precision and recall | Overall model quality |
| Accuracy | Overall correctness | Can be misleading with imbalanced data |
PM Rule: You usually can't maximize both precision AND recall. Know which matters more for your use case.
Data Concepts
| Term | Definition | Why It Matters |
|---|---|---|
| Training data | Data used to teach the model | Quality in = quality out |
| Validation data | Data to tune the model | Prevents overfitting |
| Test data | Data for final evaluation | Never seen during training |
| Data drift | When real data changes over time | Model performance degrades |
| Ground truth | The correct labels | Defines what "right" means |
Development Stages
| Stage | What Happens | Typical Duration |
|---|---|---|
| Data exploration | Understand what data exists | 1-2 weeks |
| Feature engineering | Create inputs for the model | 2-4 weeks |
| Model training | Teach the model | 1-2 weeks |
| Evaluation | Test model quality | 1 week |
| Deployment | Put into production | 1-2 weeks |
| Monitoring | Track ongoing performance | Ongoing |
What ML Teams Need from PMs
1. Clear Problem Definition
Bad: "We need AI to improve search"
Good:
- "Users abandon search when no results match their intent"
- "Success = users click on a result within top 5"
- "Current baseline: 60% click-through on top 5"
- "Target: 80% click-through on top 5"
2. Labeled Data (or Help Getting It)
ML models need examples of "correct" answers.
How you can help:
- Provide historical data with outcomes
- Set up labeling processes
- Define edge cases and how to handle them
- Prioritize data quality initiatives
3. Realistic Timelines
ML development is iterative, not linear.
| Phase | What Can Go Wrong | Buffer |
|---|---|---|
| Data preparation | Data is messier than expected | 2x time |
| Model development | First model doesn't work | 3-5 iterations |
| Production deployment | Integration challenges | 2x time |
4. Tolerance for Uncertainty
ML can't guarantee outcomes. Help stakeholders understand:
- "We'll aim for 90% accuracy but may land at 85%"
- "First version will be MVP, we'll iterate"
- "We need production data to truly optimize"
Questions to Ask in ML Kickoffs
About data:
- "What data do we need that we don't have?"
- "How much labeled data exists?"
- "What's our data refresh strategy?"
About approach:
- "Are we using pre-trained models or training from scratch?"
- "What's the simplest baseline we can compare against?"
- "What are the biggest technical risks?"
About success:
- "What accuracy can we realistically expect?"
- "How will we know if the model is degrading?"
- "What's our rollback plan?"
Red Flags in ML Projects
| Red Flag | What It Means | Action |
|---|---|---|
| "We need more data" (repeatedly) | Fundamental problem with approach | Re-evaluate if ML is right |
| "The model works in testing but not production" | Data drift or training issues | Investigate data pipeline |
| No baseline comparison | Can't prove ML adds value | Establish baseline first |
| "Just a few more weeks" | Scope creep or stuck | Set hard deadline, ship MVP |
| Accuracy keeps changing | Unstable model | Review evaluation methodology |
Setting Up ML Projects for Success
Pre-Kickoff Checklist
- Problem defined with measurable success criteria
- Data inventory completed (what exists, what's missing)
- Baseline established (current solution performance)
- Timeline includes buffer for iteration
- Stakeholders aligned on uncertainty
During Development
- Weekly syncs on progress and blockers
- Review intermediate results (don't wait for final)
- Adjust scope based on learnings
- Document decisions and trade-offs
Post-Launch
- Monitoring dashboards in place
- Alert thresholds defined
- Feedback loop established
- Retraining schedule planned
Key Takeaway
ML teams are partners, not implementers. Invest in understanding their constraints and providing clear requirements. The best AI products come from PM-ML collaboration, not handoffs.
Up next: Test your understanding with Module 2 Quiz. :::