The discipline of experimentation in the era of Product Management

In a constantly evolving digital world, where user expectations are continuously changing and competition is fierce, orchestrating an optimal digital experience is essential to delight users and increase profitability. Product Managers and CRO (conversion rate optimization) experts share these goals and are continuously seeking to maximize the effectiveness of their strategies.

Pascal Davis, Senior Product Manager, and Charlaine Sintes, Manager of Analytics and Conversion at Converteo, examine the role of data and experimentation in Product Discovery.

How can conducted tests help product strategies?

Pascal Davis:
The most innovative digital products result from rigorous experimentation. By adopting product analytics and applying the scientific method and statistical inference, Product Managers can objectively determine which projects to implement or abandon. A/B testing is essential for validating hypotheses, reducing risks, and making informed decisions.

The most effective and impactful Product teams make “Data Discovery” an indispensable and essential practice. Leaders like Jeff Bezos (Amazon), Brian Chesky (Airbnb), and Ted Sarandos (Netflix) established rigorous experimentation principles from the beginning, emphasizing scientific validation during the discovery phase before scaling in the delivery phase.

Charlaine Sintes:
Product Managers can benefit from CRO and experimentation strategies. This enhances the product roadmap by aligning features with user and business needs while providing a clearer perspective on what works and what doesn’t. The scientific method deployed in CRO should be adopted and validated by product teams to solidify and streamline innovation, allowing the backlog to be driven by outcomes rather than outputs. Tests should be prioritized and framed by SMART KPIs (Specific, Measurable, Achievable, Relevant, and Time-bound).

What common obstacles exist to the adoption of CRO methodologies, and how can they be overcome?

Charlaine Sintes:
The rigorous and systematic integration of experimentation into Product Ops requires a deliberately ROI-driven or data-driven approach. The main obstacles to this approach often stem from cultural resistance and a lack of specialized resources. Communicating tangible benefits, such as improved KPIs and better ROI, as well as investing in training and skill development, are essential to overcoming these challenges.

Pascal Davis:
Too often, the primary obstacle is organizational. To simplify, CRO resides on the marketing side with advanced analytical skills but limited scope (action on web UX client-side). Product teams, with a broader scope (server-side), often lack expertise in A/B testing, even though these resources exist elsewhere in the organization. This creates an opportunity for collaboration and synergy between CRO and Product Management.

What tools are essential for improving CRO strategies, and how are they used?

Charlaine Sintes:
Tools such as AB Tasty, Kameleoon, Dynamic Yield, and Contentsquare are crucial for quantifying the impact of initiatives and adjusting actions based on their return on investment. The transition from client-side to server-side transforms the ability of teams to conduct strategic testing, enhancing security, reducing latency, and increasing performance.

This evolution allows for more precise personalization for various audiences, thereby increasing the relevance of user experiences. At the same time, CRO tools and methodologies are spreading across different experience and performance levers. From UX optimization (client-side), we are evolving towards middleware solutions, such as tests on product catalogs and content, to application experiences (server-side).

Pascal Davis:
To innovate more effectively, we need to expand the playground for CRO. Established server-side tools as well as next-generation solutions, such as Amplitude, LaunchDarkly, Eppo, or Statsig, target product teams, DevOps, and MLOps with feature flagging and progressive rollouts. These platforms allow for the integration of the final phase of discovery into delivery and enable CRO strategies outside the marketing silo.

What advantages do generative AI tools bring to CRO and product management?

Pascal Davis:
CRO strategies, combined with generative AI tools and behavioral analytics platforms, represent a significant evolution in the “data-driven” management of digital products. They not only enable rapid responses to market needs but also anticipate user expectations, thereby ensuring a sustainable competitive advantage. Like in other fields, AI will facilitate the acceleration of experimentation programs by reducing the marginal cost of incremental testing—the strategic challenge remains to experiment more and better.

All solutions are in the process of adding ‘AI’ features to their roadmaps to enhance the productivity of CRO and Product teams during the most time-consuming phases of an AB Test cycle (detailed analysis and creation of variations, for example). What may not be immediately evident is that experimentation tools are just as essential for the creation of GenAI products as GPUs are. For instance, at OpenAI or Anthropic, absolutely every feature and code release goes through the Statsig experimentation platform: everything is tested and validated through statistical inference.

Charlaine Sintes:
Generative AI also offers transformative possibilities for content personalization, predicting user behaviors, and real-time interactions, thereby increasing engagement. It plays a crucial role in developing transnational CRO approaches, allowing for the deployment of optimized and personalized strategies for different international markets by facilitating, for example, real-time translations or image adaptations.

Pascal Davis:
In summary, to innovate effectively, it is essential to thoroughly understand your market, excel at discovery, and learn well. Since the advent of the scientific method and modern advancements in statistics, coupled with recent developments in cognitive and behavioral sciences, it is clear that a culture of experimentation is indispensable for creating and optimizing digital products, which are inherently improvable. Beyond tools, resources, and methods, it is leadership that alone can encourage and embody a culture of experimentation.

Take the example of Satya Nadella, who says: “Some call it ‘rapid experimentation,’ we call it ‘hypothesis testing.’ Instead of saying ‘I have an idea,’ if you said ‘I have a new hypothesis, let’s test it, let’s see if it is valid, let’s ask ourselves how quickly we can validate it.’ And if it’s not valid, let’s move on to the next one.” This statement may seem obvious, but it represents a significant cultural shift, especially in our country. It is very important to promote the alignment of CRO and Product teams for mutual cultural understanding, but it is even more critical to promote experimentation as a strategic imperative to ambitious executive committees (COMEX).