The Definitive Guide to AI ML Product Management

The actionable guide you need to become, operate and thrive as a AI product manager

The winter age for AI is finally over, and today AI is an idea whose time has come.

While we can't accurately predict how AI is going to affect human lives in the next 10 years, we can safely say that more and more products will have AI as their core or one of the parts.

Now is the time for product managers, builders, and entrepreneurs to learn "How to build AI products" to leverage this path-breaking technology to build products that your users will love and businesses will thrive on.

This loooong article attempts to help product managers, leaders, and aspiring product managers become confident in leading AI & ML products.

This is a quick index, of what all we are going to cover in this detailed (4000+ words) guide:

1. Setting the context (How to build with AI rather than how to use AI)

2. Why it's important to learn AI

3. How to learn the fundamentals of AI ML

4. Understanding use cases of current AI ML: Recommendations, Ranking, Classification, Regression, Clustering, Anamolies, Creation.

5. How to approach AI ML product management - User & Business Centricity

6. Understanding the AI ML product development life cycle

7. Challenges and Nuances

8. Choosing models and training data. Challenges of modeling.

9. Feedback loop

10. Deploying AI Products & AI tech stack

11. Dealing with the in-deterministic challenge

12. Working with AI ML engineers and data scientists

13. Ethical Considerations: Racism, Privacy, Biases, Training data.

14. How to get a job as an AI product manager

15. Additional reading

Setting the context

This article focuses on How to approach building AI/ML products as a product manager, and not how to use AI/ML tools to be efficient at product management. If you are looking for later, this article on How to use ChatGPT as a product manager can help you.

By building AI/ML products, I mean how to lead products like Perplexity, ChatGPT, NotionAI, HeyGen, etc. as a product manager, or how product management happens at AI companies.

This article and research wouldn't have been possible without the works of Lenny Rachitsky, Aakash Gupta, Ken Norton, Marily Nika, Marty Cagan, and other inspiring product people, especially their posts on understanding product management processes at multiple companies.

Why it is important to learn AI/ML Product Management

Understanding AI & ML product management will very soon become an essential skill for any product manager. Whether your company wants to be an AI company or not, this technology is going to affect how your users solve their problems and how your company operates in a competitive environment.

So rather than playing the catchup game later, it’s adviced to play proactively.

To seize this opportunity and build AI products you need to understand how product management should be done with these kinds of products.

While the fundamentals of product management like customer centricity, agility, and business outcome orientation remain the same, the technical complexity, product development cycle, and customer experience could be different for AI-based products and need to be learned.

It appears even Bill Gates would go ahead and build something in AI (only if he was young) 👇🏽

Learning the fundamentals of AI

The first step towards getting used to the AI world is to help yourself get confident with the common AI vocabulary.

This could be one of the most overwhelming steps for people, especially if you are from a non-tech background. Please remember that having a solid understanding of the basics will take you a very long way in your journey, and it might look difficult initially but with enough persistence and focus things will get easier and you'll gain confidence.

I would suggest the following resources to understand the fundamentals of AI and ML:

1. AI for everyone by Angrew NG: Great for understanding fundamentals details and all-around understand of AI applications.

2. Introduction to AI with Python by Harvard: Great for understanding the technical details and AI principles.

3. Basics of ML for Product Managers by Duke: Talks about fundamentals modeling and how to approach AI product management.

4. For the adventurous, here is a definitive GenAI learning guide.

Doing these courses might look like a lot of work and time to spend in this world of instant gratification, please understand that if you need to become a confident AI product manager, you need to invest this time to unlock your true potential.

Also given the wide applications of AI and its impact, it's worth spending this time to start on a confident note.

But before you start your journey of technicalities, do finish this article to gain the right perspective.

Understanding the use-case of current AI systems

Most product managers make the mistake of thinking like developers while approaching to build AI products and getting into the deep nuances of technology very early on. Always remember that no matter how sophisticated a technology is, your job as a product manager is to find its use cases, identify what customer problems can it solve, and how it can help you achieve business outcomes.

The current AI systems offer the following 7 high-level solutions or capabilities.

Recommendations: AI systems suggest items or actions to users based on their preferences, behavior, and historical data. Common examples like product recommendations on Amazon based on your browsing history and music recommendations on Spotify.

Rankings: AI algorithms sort items in a list based on relevance or importance. Search engines like Google rank web pages based on relevance to the query and social media platforms like LinkedIn prioritize posts in a feed based on user engagement.

Classification: AI models assign labels to items based on their features. For example email filtering into spam or not spam, identifying diseases from medical images or records.

Regression: AI models predict continuous values based on input data. Helpful in price prediction like estimating housing prices based on features like locations and size, and predicting future sales based on historical data at companies like housing, 99acres etc.

Clustering: AI systems group similar items based on their features without predefined labels. Can be helpful in customer segmentation based on purchase behavior or other attributes.

Anomalies: AI detects unusual patterns or outliers in data. Provides applications like fraud detection in banking transactions, network security during unusual traffic activity etc.

Creation: The most popular AI use case right now! Also called Gen AI or Generative AI. Here AI generates new content or solutions based on learned patterns. Examples like ChatGPT for text and image creation, Dall-E, mid-journey for image creation etc. Eleven labs, PlayHT for voice.

These 7 abilities of AI should act as your cheat sheet when you think about using AI to solve customer problems.

How to approach AI product management

The key is to have a fundamental understanding of what is possible with AI (7 capabilities above), and then start with understanding your user problems.

One way is to go through the customer stories & use cases section of AI companies like OpenAI, Perplexity, Amazon AWS etc, and try to understand how different companies are using AI to solve their customer problems.

The other way is to look at your customer journey and identify what current pain points can you solve with AI.

For example, Notion understood that most people face writer's block while starting a document or note, hence introducing Notion AI to create quick drafts easily.

Similarly, Stripe came across the fact that their customers and support team struggle a lot in identifying the right piece of information from their huge developer documentation, so they created a GPT-powered chatbot, on top of their documentation to make the process easier.

Actionable for you:

1. Make a list of top AI companies and identify what problems they are solving, and how the clients are using their services.

2. Identify the moments in your customer journey and product flow where you can use AI to make life easier for your customers.

This approach will make sure that you are creating value for the customers & your business.

Understanding the AI product development cycle

A typical product management cycle looks like this:

Discovery > Definition > Design > Development > Delivery > Distribution

However, While creating an AI product, the product managers need to take care of a few additional steps:
- Creating a model (or choosing an existing one),
- Acquiring the training data for the model, and
- Altering the model based on performance and accuracy.
- Collaborating with more stakeholders like the data scientists and ML engineers.

You also need to keep in mind that like AI products need to iterated rapidly and need strong feedback looks to remain relevant. In the later section, we will learn exactly how a AI product manager can do the same.

Nuances of building AI products

As an AI product manager, you need to ask yourself a few important questions:

1. Which is the right model for my use-case?

Different problems can be solved by different kinds of algorithms and models. You need to consider factors such as the scale of the model, quality of training data it needs, the cost of using that model and its performance for your use case, before finalizing on the model.

Remember that some models need fine-tuning with human intervention with rewards and penalties feedback. So your model choice should align with your needs and resources.

Here is an exhaustive list of models from HuggingFace.

2. How would you find data for training?

Your model will need enough data before it can start giving you accurate results. You have multiple ways to gather this data, including:

1. Public datasets from sources like government websites (data.gov, data.gov.in), Kaggle, or even scrapping the internet (do check the website policies before doing this).

2. Partnering with companies for data: Advertisement companies, web beacons, and other sources of data can be enabled through partnerships.

3. Your own data generated from click-streams, UGC, documentation, interviews, customer calls, support tickets, product usage, sales, or user behaviour.

4. Generating data through the same or different models.

Stripe uses their own development codebase and documentation to train the data, while OpenAI has been trained on 575 GB to 40 TB of data, most of which is sourced through web scraping.

Once you get a hold of data you also need to convert the data in a consumable format for your model. (yes most of a data-engineers time go into sorting, structuring and cleaning data).

Another unique challenge in building AI products is:

3. How do you keep your model current and evolving?

Because your initial training data will get older, you will need to take conscious steps to keep the model updated with the right inflow of feedback data.

Most AI products like chatGPT, Whimsical, Perplexity, etc, will ask you about the feedback of AI-generated results and might take that response to keep their results accurate. As AI generated outcomes could be non-deterministic, you need to build a strong feedback loop to understand if the results align with customer needs or not.

For example, when one of the mentors at HelloPM and Director of Product Management at Meesho, Gagan Mahajan, was creating a product ranking engine, they made sure to do aggressive A/B testing and kept on measuring conversion rates and customer satisfaction to fine-tune various variables of the model.

You can also keep on retraining your models on newly acquired data from multiple resources (primary being usage data) like OpenAI does it..

Deploying AI products

Deploying AI tools needs a different mindset than the traditional applications, because of the need of intensive computing and data requirements.

In the last two years, the AI ops ecosystem has rapidly evolved. A typical AI deployment life-cycle looks like this:

Model Development > Model Training > Evaluation > Deployment > Monitoring & Performance Check.

Here are various tools used for each stage:

Please note that as a product manager, you don't need to go into details of each tool, just a high-level understanding of your company's tech stack is enough.

Model Development:

1. Jupyter Notebooks: An interactive environment for developing and testing machine learning
   models.

2. TensorFlow/PyTorch: Popular frameworks for building and training deep learning models.

3. scikit-learn: A versatile library for traditional machine learning algorithms.

Model Training:

1. Google Colab: Provides free access to GPUs for training models.

2. AWS SageMaker: Offers scalable machine learning services on Amazon Web Services.

3. Azure Machine Learning: Provides cloud-based training environments on Microsoft Azure.

Model Evaluation:

1. MLflow: Tracks experiments, records results, and compares performance metrics.

2. TensorBoard: Visualizes training metrics and model performance.

Model Deployment:

1. TensorFlow serving: Pushing ML models to production, offers flexible deployments.

2. Docker/Kubernetes: For containerizing and orchestrating the applications and their management.

3. SageMaker/Google AI platform/Azure ML: Deploying ML models, maintaining their life-cycle on the shoulders of 3 infra-giants.

4. Open Neural Network Exchange (ONNX): Interoperability between various AI frameworks, ensuring models can be easily deployed across different platforms.

5. Jenkins/Ansible/Terraform: For CI/CD, and monitoring cloud infra and performance.

Dealing with the in-deterministic challenge

Building Generative AI (GenAI) products presents a unique challenge due to the inherently indeterministic nature of generative models. These models, such as GPT-3, GANs, and VAEs, can produce varied outputs given the same input, which can be both a strength and a challenge.

To solve this problem, product managers can implement following practices:

1. Set user expectations right: Make sure you inform the user about the variability, interministic nature and variable accuracy of output for the same input.

ChatGPT informs us about its variable nature with a tiny warning at the bottom 👇🏽:

2. Reinforcement Learning from Human Feedback (RLHF): Train models using RLHF to fine-tune them based on preferred outputs, which can help in reducing unwanted variability.

Humans can manually provide different kinds of feedback to the outputs generated.

3. Temperature and Top-k Sampling: Adjust the model’s sampling parameters like temperature and top-k/top-p (nucleus) sampling to control the randomness of outputs.

4. Output Regularization: Use techniques to penalize or filter out highly variable outputs that do not meet specific criteria.

5. Output Filtering and Ranking: Implement post-processing steps to filter and rank outputs based on predefined criteria, ensuring only the most suitable results are presented to users. (Super helpful)

6. A/B Testing: Regularly perform A/B testing to evaluate different model settings and configurations, identifying the best trade-offs between creativity and consistency. (as discussed above)

7. User Feedback Loops: Collect user feedback on the outputs and iteratively improve the model and its deployment based on this feedback.

8. Build Hybrid Systems: Combine generative models with deterministic systems to create a balance. For example, use rule-based systems to validate and refine the outputs of generative models.

Working with ML, Data, and AI Engineers:

Now, this is an important consideration for every product manager: You should know how to do right stakeholder management with these critical players of your team.

Always remember: “Product Management is a team sport.” The most important variable for your success is your team.

If you are like most product managers, who can’t understand the detailed technical nuances of underlying AI/ML technologies (and most of the tech terms and tools I mentioned above, looked alien to you), then right stakeholder management is your chance to shine.

Here is the general flow that I recommend:

Understand the responsibilities > set expectations right > enable cross-functional collaborations > monitor performance & share timely feedback.

You need to start by understanding the roles and responsibilities of these folks to set the expectations right:

Data Scientists: Focus on data analysis, creating models, and deriving insights from data.

ML Engineers: Specialize in developing, training, and optimizing machine learning models, and often work on integrating these models into production.

AI Engineers: Work on building and maintaining AI systems, which can include both software engineering and the implementation of ML models.

It is also advised, to ask questions, show curiosity and spend some time to get yourself acquainted with the vocabulary and tools that your engineering counterparts use.

A major mistake that product managers make, is that they treat engineering stakeholders largely as execution arms only. Engineers and Designers are great problem solvers too. If you involve them earlier in the process, during your product discovery part, they might bring better ideas, ensure feasibility/usability from the start, and also end up being more motivated to pursue the idea.

Ethical Considerations while building AI products

AI is an extremely powerful technology, and as Uncle Ben from Spider-Man says:

“With great power comes great responsibility”

You need to set up the right guard rails to make sure you are operating under ethical considerations.

Here are my tips on understanding common avenues where AI can go wrong, and how you might address them as an AI product manager:

Racism

AI systems can worsen racial biases if not managed carefully, such as facial recognition misidentifying people of color or predictive policing unfairly targeting minority communities.

In 2020, a study revealed that facial recognition technology used by several major tech companies, including IBM, Amazon, and Microsoft, had significantly higher error rates in identifying people of color compared to white individuals. This prompted companies like IBM to halt their facial recognition programs, emphasizing the need for ethical considerations in AI development.

As a product manager, it’s crucial to ensure diversity in the development team (if you are large enough) and the training data. Regular audits of AI outputs and incorporating fairness constraints in algorithms can help mitigate racial biases. Engaging with diverse communities for feedback can also ensure the AI system respects and acknowledges racial differences fairly.

Privacy

AI products often rely on vast amounts of personal data, raising significant privacy concerns. Misuse or unauthorized access to this data can lead to breaches of privacy, exposing sensitive user information.

Recently in 2023, a significant privacy breach occurred with an AI-driven health app that exposed sensitive user health data. The app collected detailed health information to provide personalized advice, but inadequate security measures led to the data being accessible to unauthorized third parties.

Implementing robust data governance policies, anonymizing data, obtaining explicit consent from users, and adhering to regulations like GDPR and CCPA are essential steps to protect user privacy.

As a Product manager, you should also prioritize transparency, clearly communicating how data is used and giving users control over their information. Incorporating privacy by design principles from the outset can help mitigate these risks.

If you are interested to know more about how to create privacy-first products, do reach these 7 principles of privacy by design.

Biases

Biases in AI can arise from the training data or the algorithms themselves, leading to unfair and discriminatory outcomes. Bias can manifest in many forms, including gender, age, and socioeconomic status.

To counteract biases, you as a product manager should advocate for diverse and representative training datasets. You should also implement bias detection and correction mechanisms, conduct thorough testing across different demographic groups, and continuously monitor AI outputs for unfair biases. Implementing feedback loops where users can report biased outcomes can also help in continuously improving the system.

Training Data

Bad input => Bad Output.

The quality and representativeness of training data are critical in building effective and fair AI systems. Poor quality or unrepresentative data can lead to inaccurate models and biased outcomes. You need to make well-informed decisions about acquiring, diversifying, and cleaning these training data-sets.

Collaborating with data scientists to preprocess and clean data, identifying potential gaps, and augmenting datasets with underrepresented groups are key practices.

Additionally, leveraging synthetic data to fill in gaps can also help improve the robustness of AI models (as mentioned above). Continuous updating and validation of the training data are also essential to maintain the relevance and accuracy of the AI system.

How to become an AI product manager?

I am writing another detailed & actionable write on becoming an AI product manager, which involves resources, skills, and actionable tips to reach out to the right companies, and create proof of work in the field.

👉🏽 If you want to access it for free, do subscribe to this substack of ours, to not miss it when I release it.

But for now, my advice to people who want to break into AI product management would:

1. Focus on understanding user pain points and desires, and how AI as a technology can help them ease that. Obsessively go through user journeys of multiple products and make notes on how AI can optimize them.

2. Get your AI fundamentals right: If you are in it for the long term, make sure you do some of the courses I have recommended at the top. You can also consider this actionable program that we offer at HelloPM (Made by people who are working at companies like Adobe, BlinkIT, Walmart, Meesho, Apple & Meta).

3. Build & Practice: You can’t learn by reading or watching videos. You need to get your hands-dirty. Go direct, learn by doing, and build your proof of work.

4. Build a network with other AI PMs: A Network can not only unlock work opportunities but also help you learn by bouncing ideas off each other. Attend events, send cold emails, appreciate people’s work, and join a community to meet and connect with the right people.

Additional reading

Here are a few articles I would recommend for further reading:

1. How perplexity builds products: An insightful blog right from the leaders who are building an amazing AI product.

2. How to approach AI product management: Packed write-up by Marty Cagan and Marily Nika on dealing with 4 product risks as an AI product manager.

3. Resources for AI product management.

Got feedback or need help? Let me know in the comments.