AI vocal agent and customer relationship: the Sharlie case by Converteo

David Guede, partner data / AI and agentic expert at Converteo, specializes in deploying AI architectures in production. He supports companies in industrializing intelligent agents, transforming complex business processes into real levers for performance and competitive advantage.

Key takeaways

• Sosh has deployed “Sharlie”, a new-generation vocal agent based on an architecture of 30 specialized agents orchestrated to guarantee a fluid, natural, and personalized dialogue.
• Processes designed for humans must be reinvented for AI, as the literal obedience of an LLM when faced with overly constrained procedures generates rigid and artificial interactions.
• Securing this probabilistic agent relies on three essential pillars: mass simulation via a virtual customer, real-time monitoring by an “LLM as a judge”, and the integration of strict semantic safeguards.
• The management of the solution is evolving from traditional IT to semantic performance management, where the AI is supervised, reviewed, and corrected continuously, just like a human advisor.

Designing the first European vocal agent from end to end, capable of natural dialogue with its customers without predictive menus or rigid scripts: this is the bold challenge we took up with Sosh. Launched in mid-March, “Sharlie” inaugurates a new generation of 100% AI call centers dedicated to care excellence. Its goal: to offer total availability and unprecedented fluid dialogue for the brand’s customers. A strong promise underpinned by a technical feat: how to entrust your customer relationship to a probabilistic technology without risking hallucinations or a breakdown in tone at any second?

Relational excellence through multi-agent orchestration

What the user perceives as a single interlocutor is actually a complex orchestration. Rather than a single agent following a linear script, Sharlie relies on a multi-agent architecture: 30 specialized agents (billing, claims, plan management…) linked to the Orange IT system pass the context along according to the natural logic of the conversation. The innovative technology, also called voice to voice or full duplex (OpenAI realtime), allows providing an expert and personalized response to each request, without ever imposing a rigid path on the customer.

The first lesson is crucial: processes designed for humans cannot be transplanted as they are into an AI. A human advisor appropriates a procedure to make it fluid; an LLM, through its literal obedience, produces a rigid and artificial interaction if it is too constrained. For Sosh, we reinvented this business logic so that the AI could move away from a mechanical response loop and engage in a natural dialogue with its interlocutor.

How to secure the customer experience with a probabilistic agent?

Technical obstacles (latency, context management, model stability) are mastered today. The real challenge lies elsewhere: validating a vocal agent is unlike anything CIOs have known until now: the priority is not only the success of the task, but the quality of the interaction experienced by the subscriber.

Deterministic software is tested by use case. An LLM, however, is probabilistic: it can answer accurately 99 times and slip up on the hundredth. How do you anticipate a semantic unforeseen event… that you cannot anticipate?

To guarantee an irreproachable experience, we deployed three pillars:

• The virtual customer: capable of simulating thousands of variations of the same intent.
• The “LLM as a judge”: to monitor semantic quality in real time and ensure that each response perfectly matches the brand’s requirement.
• Semantic safeguards: while addressing a customer informally might be acceptable, praising a competitor’s offer triggers an immediate kill switch.

Managing semantic performance daily

With vocal agents, the nature of monitoring changes radically. Traditional indicators (availability, response time) become secondary to semantic coherence, hallucination detection, and compliance with the tone of voice.

In production, Sharlie is managed like a human advisor: its conversations are reviewed, its deviations identified, and its learnings injected into the backlog. We are no longer managing a frozen IT project, but a constantly evolving non-human collaborator. This transition from IT to semantic performance management is the key to transforming AI into a sustainable and trustworthy asset for the brand.

• How to choose your LLM architecture and stick to it in an ecosystem that changes every month?
• How to organize the large-scale validation of a probabilistic agent?
• How to monitor semantic quality in production with an “LLM as a judge”?
• How to govern a vocal agent that evolves continuously, between IT and business teams?