CrawlJobs Logo

Manager, Agent Evaluation

comcastadvertising.com Logo

Comcast Advertising

Location Icon

Location:
United States , Washington D.C.

Category Icon

Job Type Icon

Contract Type:
Not provided

Salary Icon

Salary:

183063.62 - 274595.42 USD / Year

Job Description:

The Agent Evaluation team is responsible for testing whether AI agents return the correct and expected responses. We build the framework, metrics, and test cases that validate agent behavior, accuracy, and reliability before release. Our goal is to ensure agents perform consistently and meet product and user expectations. The Manager, Agent Evaluation will lead the team responsible for building and scaling the evaluation framework that tests whether AI agents return accurate, reliable, and expected responses across real-world scenarios.

Job Responsibility:

  • Lead and grow a team focused on agent and model evaluation
  • Define the strategy, roadmap, and standards for agent testing and validation
  • Oversee development of metrics, benchmarks, and testing frameworks to measure response quality, accuracy, safety, and performance
  • Ensure evaluation coverage aligns with product, UX, and business requirements
  • Partner closely with Product, Engineering, Research, and Platform teams to integrate evaluation into the development lifecycle
  • Drive experimentation and continuous improvement of evaluation methodologies
  • Establish reporting mechanisms to clearly communicate evaluation results and trade-offs to leadership
  • Implement best practices for model versioning, monitoring, and release validation
  • Stay current with advancements in LLMs, AI agents, and evaluation techniques

Requirements:

  • Strong foundation in machine learning fundamentals and applied ML systems
  • Hands-on experience with model and agent evaluation methodologies
  • Familiarity with LLMs, AI agents, and prompt-driven systems
  • Proficiency in Python and modern ML frameworks (e.g., PyTorch, TensorFlow)
  • Experience defining metrics, benchmarks, and experimentation frameworks
  • Solid understanding of MLOps practices, including model versioning, monitoring, and CI/CD
  • Ability to collaborate effectively with product, platform, and research teams
  • Clear communicator of technical trade-offs, evaluation insights, and results
  • Master's Degree
  • 5-7 Years Relevant Work Experience
What we offer:
  • Paid Time off
  • Physical Wellbeing benefits
  • Financial Wellbeing benefits
  • Emotional Wellbeing benefits
  • Life Events + Family Support benefits

Additional Information:

Job Posted:
February 13, 2026

Employment Type:
Fulltime
Work Type:
On-site work
Job Link Share:

Looking for more opportunities? Search for other job offers that match your skills and interests.

Briefcase Icon

Similar Jobs for Manager, Agent Evaluation

Senior Product Manager, AI Agents

This role owns AI research, messaging, and context—spanning both the user experi...
Location
Location
United States
Salary
Salary:
187000.00 - 250000.00 USD / Year
apollo.io Logo
Apollo.io
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5+ years in product management
  • 2+ years experience launching AI/ML new products and scaling existing products
  • Track record of shipping AI features that drove measurable business outcomes
  • Experience with LLM-powered applications, prompt engineering, evaluation frameworks, and model selection tradeoffs
  • Comfortable working in Python/SQL to analyze data, prototype prompts, and evaluate outputs
  • Understanding of LLM architectures, RAG pipelines, agent frameworks, and inference optimization
  • Obsession with quality over speed
  • GTM or sales tech experience (strongly preferred)
  • Familiarity with sales workflows, prospecting tools, or CRM systems
  • Understanding of why sales teams are skeptical of AI tools and what it takes to earn their trust
Job Responsibility
Job Responsibility
  • Develop and execute a strategic roadmap for AI research, messaging, and context capabilities
  • Enhance Apollo's AI research agents to surface actionable insights from the web
  • Define how AI understands each user's business
  • Own AI-powered messaging tools that create personalized, context-aware emails at scale
  • Build and scale evaluation infrastructure across accuracy, relevance, clarity, and tone
  • Partner with engineering, design, prompt writers, and sales to deliver cohesive AI experiences
What we offer
What we offer
  • Equity
  • Company bonus or sales commissions/bonuses
  • 401(k) plan
  • At least 10 paid holidays per year
  • Flex PTO
  • Parental leave
  • Employee assistance program and wellbeing benefits
  • Global travel coverage
  • Life/AD&D/STD/LTD insurance
  • FSA/HSA and medical, dental, and vision benefits
  • Fulltime
Read More
Arrow Right

AI Engineering Manager - Internal AI Agent

We are looking for an AI Engineering Manager to drive Mirakl's internal AI trans...
Location
Location
France , Paris
Salary
Salary:
Not provided
mirakl.com Logo
Mirakl
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 7+ years of experience in AI/ML or software engineering
  • Proven track record building AI agents using LLMs, RAG, MCP and related technologies
  • Strong technical proficiency in Python and multiple programming languages, with architectural design experience
  • Production deployment expertise - you've shipped AI solutions to real users
  • Technical pragmatism - ability to match the right technology to the use case
  • Curiosity and continuous learning - you stay current with AI/ML trends
  • 1+ years experience as a Lead or management roles (team management or technical leadership)
  • Strong leadership skills - you inspire and develop high-performing engineering teams
  • Cross-functional stakeholder management - you build relationships and excel at working with all organizational levels & functions
  • Strong communication & presentation skills - in both English and French
Job Responsibility
Job Responsibility
  • Partner closely with Mirakl teams & leadership to identify & prioritize opportunities, redesign workflows around AI agents, and drive adoption at scale
  • Lead and mentor a team of cross-functional AI engineers, defining your team’s roadmap to support strategic AI initiatives
  • Build advanced Mirakl-specific AI agents centrally, owning the complete delivery cycle from discovery to production deployment and operations
  • Foster organization-wide AI adoption by animating internal communities, providing self-service tools, training & support to empower teams as autonomous AI builders
  • Establish & scale technical standards & stack to ensure secure, compliant & high-quality deliverables across all internal AI projects
  • Explore emerging AI paradigms, evaluate new tools and technologies, and maintain active technology watch
Read More
Arrow Right

AI Engineering Manager - Internal AI Agent

We are looking for an AI Engineering Manager to drive Mirakl's internal AI trans...
Location
Location
France
Salary
Salary:
Not provided
mirakl.com Logo
Mirakl
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 7+ years of experience in AI/ML or software engineering
  • Proven track record building AI agents using LLMs, RAG, MCP and related technologies
  • Strong technical proficiency in Python and multiple programming languages, with architectural design experience
  • Production deployment expertise - you've shipped AI solutions to real users
  • Technical pragmatism - ability to match the right technology to the use case
  • Curiosity and continuous learning
  • 1+ years experience as a Lead or management roles (team management or technical leadership)
  • Strong leadership skills
  • Cross-functional stakeholder management
  • Strong communication & presentation skills - in both English and French
Job Responsibility
Job Responsibility
  • Partner closely with Mirakl teams & leadership to identify & prioritize opportunities, redesign workflows around AI agents, and drive adoption at scale
  • Lead and mentor a team of cross-functional AI engineers, defining your team’s roadmap to support strategic AI initiatives
  • Build advanced Mirakl-specific AI agents centrally, owning the complete delivery cycle from discovery to production deployment and operations
  • Foster organization-wide AI adoption by animating internal communities, providing self-service tools, training & support to empower teams as autonomous AI builders
  • Establish & scale technical standards & stack to ensure secure, compliant & high-quality deliverables across all internal AI projects
  • Explore emerging AI paradigms, evaluate new tools and technologies, and maintain active technology watch
Read More
Arrow Right

AI Engineering Manager - Internal AI Agent

We are looking for an AI Engineering Manager to drive Mirakl's internal AI trans...
Location
Location
France , Bordeaux
Salary
Salary:
Not provided
mirakl.com Logo
Mirakl
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 7+ years of experience in AI/ML or software engineering
  • Proven track record building AI agents using LLMs, RAG, MCP and related technologies
  • Strong technical proficiency in Python and multiple programming languages, with architectural design experience
  • Production deployment expertise - you've shipped AI solutions to real users
  • Technical pragmatism - ability to match the right technology to the use case
  • Curiosity and continuous learning
  • 1+ years experience as a Lead or management roles (team management or technical leadership)
  • Strong leadership skills
  • Cross-functional stakeholder management
  • Strong communication & presentation skills - in both English and French
Job Responsibility
Job Responsibility
  • Partner closely with Mirakl teams & leadership to identify & prioritize opportunities, redesign workflows around AI agents, and drive adoption at scale
  • Lead and mentor a team of cross-functional AI engineers, defining your team’s roadmap to support strategic AI initiatives
  • Build advanced Mirakl-specific AI agents centrally, owning the complete delivery cycle from discovery to production deployment and operations
  • Foster organization-wide AI adoption by animating internal communities, providing self-service tools, training & support to empower teams as autonomous AI builders
  • Establish & scale technical standards & stack to ensure secure, compliant & high-quality deliverables across all internal AI projects
  • Explore emerging AI paradigms, evaluate new tools and technologies, and maintain active technology watch
Read More
Arrow Right

Assistant Construction Project Manager

This role focuses on managing a range of Private Residential projects, from priv...
Location
Location
United Kingdom , London
Salary
Salary:
28000.00 - 38000.00 GBP / Year
https://brandonjames.co.uk Logo
Brandon James
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Holds a degree in Project Management or an equivalent qualification
  • Aspiring to achieve chartership in the future
  • Strong communication skills, both written and verbal
  • Keen interest in the field of high-end private residential construction
  • Able to effectively support senior team members in project management tasks
Job Responsibility
Job Responsibility
  • Assist in the setup and governance of high-end private residential projects
  • Monitor project processes, ensuring compliance and efficiency
  • Conduct due diligence and quality assurance checks
  • Assist in financial monitoring and progress reporting of projects
  • Participate in project audits and post-project evaluations
What we offer
What we offer
  • 25 Days holiday + Bank holidays
  • Hybrid working
  • Pension contribution
  • APC Support
  • Clear progression pathway
  • Supportive culture
  • Internal training programmes
  • Flexible working conditions
  • Birthday off
  • Company phone and laptop
  • Fulltime
Read More
Arrow Right

Unit Business Risk & Compliance Agent

You could think that we have supernatural powers, but the truth is that our team...
Location
Location
Canada , Richmond
Salary
Salary:
19.37 CAD / Hour
https://www.ikea.com Logo
IKEA
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • You have previous experience in the Health and Safety and Security sector and/or Safety and Security experience within a Retail environment
  • You’re knowledgeable of relevant safety standards and regulations, security processes, tools and working methods
  • You’re energized by the implementation of safeguards that bring value to the business and protect the financial and moral position
  • you can ensure the integrity of safety and security systems, guidelines and documentation
  • You know how to conduct a risk assessment and implement the hierarchy of controls
  • You have good communication and documentation skills in dealings with various levels of management
  • You think and work in a risk-based way (i.e. Evaluate trade-offs between potential costs and benefits and acts accordingly)
  • You have good analytical and numerical skills
Job Responsibility
Job Responsibility
  • Promote risk management in the unit, informing and sharing expertise in order to develop risk-aware decision taking in relation to unit goals and unit business plan
  • Support co-workers, by providing expertise, in acting in accordance with Ingka Risk & Compliance Rules and Local legislation on Health Safety and Security to secure a safe environment for customers and co-workers
  • Promote and ensure completion of trainings needed and facilitate for unit employees
  • Support a Risk & Compliance culture by utilizing systems to detect, analyze and reduce business loss and financial impact
  • Ensure the reporting of relevant figures for co-workers, customer and visitor incidents to establish progress and areas for improvement
What we offer
What we offer
  • Wellness days (in addition to your vacation days!)
  • Extended health, dental, and vision coverage (for you and your family)
  • RRSP with IKEA contribution matching options
  • Eligibility for our annual IKEA bonus incentive plan
  • Flexible spending account
  • Life insurance
  • Merchandise and restaurant discounts (plus free drinks and different healthy meal options in the co-worker restaurant, where available)
  • Parental leave
  • Bereavement leave
  • Employee assistance program (that helps you support your mental, physical, and financial wellbeing)
  • Fulltime
Read More
Arrow Right
New

Indirect Sales Manager

At Vodafone, we’re not just shaping the future of connectivity for our customers...
Location
Location
Portugal , Lisboa
Salary
Salary:
Not provided
vodafone.com Logo
Vodafone
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's degree
  • 3 years of experience in sales
  • Management and training of sales representatives
  • Knowledge of Vodafone's commercial offer
  • User-level IT skills, Telecommunications, Information Technologies
  • Basic knowledge of retail
  • Knowledge of existing Telecommunications Equipment/Services in the Market
  • Market knowledge
  • Basic knowledge of business management and accounting
Job Responsibility
Job Responsibility
  • Monitor the Agent's business and their salespeople, and approve deals and exceptions
  • Follow up on results with each Agent to ensure the achievement and constant improvement of results
  • Implement Sales Plans (annual and quarterly) and campaigns launched by Vodafone
  • Collect and share information about competitors and customer needs
  • Ensure a prompt response to all complaints related to Agents and trigger actions to guarantee high customer satisfaction
  • Ensure an appropriate response – in time and content – to all customer requests – complaints, proposals, clarifications, and supplies
  • Ensure compliance with commitments made to the customer regarding commercial conditions and account management
  • Provide adequate support to each of the Agents in your portfolio, particularly in responding to requests and visiting the Agent's premises (for both result evaluation and business monitoring and development)
  • Ensure that Agents have the appropriate information/training on the entire Vodafone offer, providing on-the-job training to the Agent's staff whenever necessary
  • Monitor the Agent's management model, contributing to its improvement through the implementation of best practices
What we offer
What we offer
  • Hybrid Work Model - Flexible hybrid work model with 8-10 in-office days per month, managed by team leaders
  • Vodafone Products and Services - Employees get a mobile phone, free communication plan, data card, and various discounts on services and products
  • Recognition - Recognition programs for innovative, creative, high-potential employees and exemplary behaviors
  • Health and Well-being - Well-being Program offers nutrition and psychological consultations, webinars, workshops, and discounts on various services and products
  • Learning - Access to Communities of Practice and a customizable digital training platform with high-quality content (namely Harvard Business Publishing, Skillsoft and Speexx)
  • Local and International Mobility - Internal recruitment with local and international rotation opportunities across departments and roles
  • Fulltime
Read More
Arrow Right

Senior Staff Machine Learning Engineer

Help design our AI platform and develop our next generation of machine learning ...
Location
Location
United States , San Francisco
Salary
Salary:
216500.00 - 324500.00 USD / Year
gofundme.com Logo
GoFundMe
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 9+ years of hands-on experience in machine learning engineering, AI development, software engineering, or related fields
  • Experience emphasizing secure, large-scale, distributed system design, AI/ML pipeline development, and implementation
  • Extensive experience designing, developing, and operating scalable backend systems
  • Experience applying software engineering best practices such as domain-driven design, event-driven architectures, and microservices
  • Deep expertise in agentic workflows, AI evaluation solutions, prompt management, and secure AI development and testing practices
  • Strong knowledge of relational and document-based databases, data storage paradigms, and efficient RESTful API design
  • Experience establishing robust CI/CD pipelines, automated testing (unit and integration), and deployment practices
  • Strong leadership skills, including effective planning and management of complex projects, mentoring of team members, and fostering a collaborative, high-performing engineering culture
  • Excellent communicator, able to articulate complex technical concepts clearly to both technical and non-technical stakeholders
  • Bachelor's degree in Computer Science, Software Engineering, or a related technical field (preferred)
Job Responsibility
Job Responsibility
  • Design and implement AI platforms to enable scalable and secure access to LLMs from multiple model providers for diverse use cases
  • Design and implement agentic workflows, agentic tool ecosystems, and LLM prompt management solutions
  • Design, build, and optimize scalable model training, fine tuning, and inference pipelines, ensuring robust integration with production systems
  • Influence technical strategy and approach to developing embedding stores, vector databases, and other reusable assets
  • Lead initiatives to streamline ML and AI workflows, improve operational efficiency, and establish standardized procedures to achieve consistent, high-quality results across our AI systems
  • Design and develop backend services and RESTful APIs using Python and FastAPI, integrating seamlessly with ML pipelines and services
  • Take operational responsibility for team-owned services, including performance monitoring, optimization, troubleshooting, and participation in an on-call rotation
  • Collaborate with both technical and non-technical colleagues, including data and applied scientists, software engineers, product managers, and business stakeholders, to deliver reliable and scalable ML-driven products
  • Coach and mentor fellow ML engineers, promoting a culture of collaboration, continuous improvement, and engineering excellence within the team
  • Employ a diverse set of tools and platforms including Python, AWS, Databricks, Docker, Kubernetes, FastAPI, Terraform, Snowflake, Coralogix, and GitHub to build, deploy, and maintain scalable, highly available machine learning infrastructure
What we offer
What we offer
  • Competitive pay
  • Comprehensive healthcare benefits
  • Financial assistance for things like hybrid work, family planning
  • Generous parental leave
  • Flexible time-off policies
  • Mental health and wellness resources
  • Learning, development, and recognition programs
  • Fulltime
Read More
Arrow Right