Job Description

Sensitivity Label: General
AI Agent Solution Specialist
Job Description
This role supports both AI-enabled and human-assisted customer interactions by configuring no-code AI workflows, monitoring performance, and analyzing customer interaction data to drive continuous improvement, operational efficiency, and positive customer outcomes within a contact center environment.
What You’ll Do
AI Agent Journey Design & Configuration
• Design, build, and continuously evolve no-code AI agent journeys, including conversation flows, decision logic, and end-to-end user experiences.
• Configure intents, prompts, business rules, and escalation paths using intuitive tools, enabling scalable content management without direct coding.
• Collaborate with product owners, engineers, and business stakeholders to translate requirements into scalable, no-code agent experiences.
• Ensure all agent journeys align with governance, security, and responsible AI practices across the agent development lifecycle.
Performance Monitoring & Analytics
• Monitor and benchmark AI agent performance across journeys (accuracy, containment, resolution rate, user satisfaction), applying simulation-driven thinking to real-world scenarios.
• Analyze interaction logs and journey analytics to identify drop-offs, failure patterns, and optimization opportunities, feeding insights into continuous improvement loops.
• Design and maintain advanced search and query frameworks (speech and text, pattern-based logic) to enable automated analysis and topic identification in customer interactions.
• Build and maintain advanced speech and text queries (including pattern-based logic) to monitor both human and AI-assisted interactions.
• Generate actionable insights from interaction data and communicate findings to CX stakeholders, supporting data-driven decision making.
AI Agent Quality Management
• Own the end-to-end quality management framework for AI agents — defining quality standards, evaluation criteria, scoring rubrics, and pass/fail thresholds across all deployed agent journeys.
• Conduct systematic conversation reviews and audits of AI agent interactions, scoring responses for accuracy, tone, compliance, escalation appropriateness, and resolution quality.
Sensitivity Label: General
•
Develop and maintain a structured QA scorecard tailored to AI agent interactions, incorporating dimensions such as intent recognition accuracy, hallucination detection, knowledge retrieval relevance, and conversation coherence.
• Identify recurring quality defects, failure modes, and edge cases through interaction sampling and trend analysis — distinguishing between prompt-level, knowledge-level, and integration-level root causes.
• Establish and run calibration sessions with cross-functional stakeholders (product, engineering, CX) to ensure consistent quality evaluation standards across agent deployments.
• Track quality metrics over time (QA pass rate, critical defect rate, regression frequency) and report trends to leadership with clear improvement recommendations.
Training Feedback & AI Engineer Collaboration
• Translate quality findings into structured, actionable feedback for AI Agent Engineers — providing specific examples, annotated conversation logs, and clear descriptions of expected vs. actual agent behavior.
• Maintain a prioritized defect and improvement backlog informed by QA findings, categorized by severity, frequency, and customer impact — collaborating with engineers to drive resolution.
• Participate in regular feedback loops with engineering, reviewing prompt refinements, knowledge base updates, and guardrail adjustments to validate that quality issues are resolved without introducing regressions.
• Develop and curate a library of gold-standard conversation examples and failure-case annotations that serve as training references for prompt tuning, knowledge curation, and agent behavior calibration.
• Contribute to the design of automated evaluation pipelines by defining test scenarios, expected outputs, and quality assertions that engineers can integrate into CI/CD workflows.
• Support the creation of regression test suites by documenting resolved defects as repeatable test cases, ensuring fixed issues do not resurface across agent updates.
• Partner with engineers during post-deployment reviews to assess whether agent updates have improved quality metrics, using before-and-after analysis of QA scores and interaction outcomes.
Testing, Experimentation & Continuous Improvement
• Test and validate AI agent behavior through structured experimentation (A/B testing, edge case validation), ensuring quality, compliance, and responsible AI standards.
• Investigate incidents and unexpected agent behavior, conducting root-cause analysis in non-deterministic AI systems.
• Contribute to the evolution of self-improving, generative agent systems by leveraging real-world interactions and feedback loops.
Sensitivity Label: General
What You’ll Bring
• Passion for working at the frontier of AI products, especially in generative AI and agent-based systems.
• Language proficiency in English, French, Spanish (written and spoken).
• High ownership mindset with the ability to operate autonomously, navigate ambiguity, and drive meaningful outcomes.
• Strong analytical and problem-solving skills, with the ability to interpret complex interaction data and translate insights into action.
• Excellent verbal and written communication skills, with the ability to clearly convey findings to both technical and non-technical stakeholders.
• Ability to manage multiple priorities independently in a deadline-driven environment, while collaborating effectively across cross-functional teams.
• Strong planning, organizational, and time-management skills.
• A quality-first mindset — methodical attention to detail in reviewing AI agent outputs, with the discipline to maintain consistent evaluation standards across high volumes of interactions.
• Comfort operating in the feedback loop between quality evaluation and engineering execution — able to articulate what’s wrong, why it matters, and what good looks like.
Nice to Have
• Experience with quality assurance and training in a contact center or BPO environment.
• Experience supporting contact center technologies, particularly speech analytics and AI-driven interaction platforms.
• Experience building AI-powered products, particularly with LLMs, conversational AI, or autonomous agents.
• Hands-on experience with AI agent evaluation frameworks, including conversation scoring, automated testing, and regression analysis.
• Familiarity with prompt engineering and knowledge base curation as levers for improving agent quality.
• Experience creating QA rubrics or scorecards for conversational AI or chatbot deployments.

Job Tags

Similar Jobs