CrawlJobs Logo

Qualitative Evaluation Engineer

lumalabs.ai Logo

Luma AI

Location Icon

Location:
United States , Palo Alto

Category Icon
Category:
IT - Software Development

Job Type Icon

Contract Type:
Not provided

Salary Icon

Salary:

Not provided
Save Job
Save Icon
Apply Position

Job Description:

Luma is pushing the boundaries of generative AI, building tools that redefine how visual content is created. We’re seeking a candidate to help shape and scale the way we understand, measure, and improve model performance. In this role, you’ll partner with researchers, engineers, and technical artists to evaluate our models against real-world creative use cases, design frameworks that capture qualitative nuance, and identify actionable insights that guide development. This is not a checkbox metrics role - it's about building evaluative systems that match the complexity of human perception, creativity, and intention.

Job Responsibility:

  • Evaluate generative model performance across diverse tasks, prompts, and modalities
  • Identify key failure modes, regression patterns, and edge cases that impact product quality
  • Develop and maintain qualitative evaluation frameworks that are scalable and reusable
  • Collaborate closely with technical artists and engineers to align evaluations with model capabilities and target use cases
  • Translate high-level product goals into concrete evaluative criteria
  • Lead qualitative studies, side-by-side comparisons, and human-in-the-loop evaluation efforts
  • Provide detailed feedback that informs model fine-tuning, dataset curation, and product UX
  • Stay informed about emerging evaluation standards in generative AI and creative tools

Requirements:

  • Master’s degree or higher in Cognitive Science, Human-Computer Interaction (HCI), Design Research, Psychology, Media Studies, or a related field
  • 5+ years of experience in product evaluation, UX research, model testing, or similar roles that involve structured qualitative assessment
  • Deep familiarity with creative workflows and real-world use cases for generative models (e.g., animation, filmmaking, digital art, VFX)
  • Strong systems thinking and the ability to define abstract qualities (like believability, identity retention, or scene coherence) in clear evaluative terms
  • Experience working cross-functionally with engineers, researchers, and creatives
  • Excellent written communication skills and the ability to synthesize nuanced judgments into clear, actionable insights

Nice to have:

  • Background in motion, visual effects, or storytelling pipelines
  • Experience evaluating AI-generated media (video, images, 3D)
  • Prior work on building internal tools for qualitative data collection or scoring
  • Familiarity with prompt engineering and reference-based input methods

Additional Information:

Job Posted:
January 13, 2026

Employment Type:
Fulltime
Work Type:
Hybrid work
Icon
Luma AI - All Job Offers
Job Link Share:
Copy Link Icon
Facebook Icon Twitter Icon LinkedIn Icon

Looking for more opportunities? Search for other job offers that match your skills and interests.

Briefcase Icon

Similar Jobs for Qualitative Evaluation Engineer

New

Senior Lead AI Engineer

Omio is building the future of travel. We’re moving from manual, rule-based syst...
Location
Location
Singapore , Singapore
Salary
Salary:
Not provided
foodlabs.com Logo
FoodLabs & Atlantic Labs
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Minimum 10+ years of experience in software engineering, developing complex models and algorithms
  • Proven track record in designing, implementing, and deploying production-grade AI solutions at scale
  • Strong communication and presentation skills, with the ability to influence and collaborate effectively with non-technical stakeholders
  • Self-motivated and capable of working independently, driving initiatives with minimal supervision
  • Prior experience deploying scalable AI and LLM-based solutions for real-time, high-performance systems is highly desirable
  • Experience with diverse model evaluation techniques (quantitative and qualitative) and an iterative approach to improving AI system performance and user outcomes
  • Expertise in building AI applications using large language models such as OpenAI, Claude, Gemini, LLaMA
  • Experience with LLM orchestration frameworks like LangChain, LangGraph, vLLM, LMDeploy
  • Strong programming skills in Java, Python, and SQL
  • Familiarity with data preprocessing, feature engineering, model evaluation, MLOps, and LLMOps best practices
Job Responsibility
Job Responsibility
  • Develop AI solutions leveraging LLMs to improve productivity and deliver strong business impact
  • Lead the end-to-end development lifecycle from ideation to deployment of AI-powered solutions across various domains
  • Build scalable AI systems that support Omio’s global expansion goals
  • Act as an evangelist for AI adoption by demonstrating clear value to stakeholders
  • Collaborate with Business, Product, and Engineering teams to integrate AI into workflows and drive adoption
  • Present models, results, and systems to both technical and non-technical audiences, including C-level stakeholders
  • Fulltime
Read More
Arrow Right

Sustainable transport economist

We aim to recruit a junior/intermediate consultant with expertise in transport e...
Location
Location
Italy , Milan
Salary
Salary:
Not provided
trt.it Logo
TRT TRASPORTI E TERRITORIO SRL
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Master degree or PhD in Economics, Engineering, Statistics or equivalent
  • Solid theoretical knowledge and/or previous work experience in transport policy, energy policy and economics studies
  • Familiarity with quantitative methods
  • Interest in studying and exploring innovative topics
  • Problem solving attitude
  • Excellent verbal and written communication skills (minute taking, drafting reports and delivering presentations)
  • A very good command of English (reading, writing and speaking)
  • Readiness for travel in EU and outside EU
Job Responsibility
Job Responsibility
  • Develop qualitative and quantitative analysis
  • Support the drafting and publication of reports
  • Source and write scientific articles
  • Contribute to communication, information and dissemination tasks within the projects
What we offer
What we offer
  • Possibility to have flexible working arrangements
  • Fulltime
Read More
Arrow Right

UX Researcher, Qualitative

Our UX Research team is designing for the broad spectrum of human needs, which r...
Location
Location
United States , Bellevue
Salary
Salary:
164000.00 - 227000.00 USD / Year
meta.com Logo
Meta
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor’s degree with 10+ years of relevant experience in user experience, applied research and/or product research and development or a Master’s degree and 8+ years relevant experience, or PhD and 5+ years relevant experience
  • Experience conducting In-Depth Interviews or Focus Groups and Concept Testing or Usability Testing
  • Interest in and experience executing hands-on, primary research
  • Experience translating research findings into strategic narratives
Job Responsibility
Job Responsibility
  • Work closely with product and business teams to identify research topics
  • Act as a thought leader in the domain of research, while advocating for the people who could use our products
  • Design and execute end-to-end custom primary research using a wide variety of methods
  • Design studies that address both user behavior and attitudes
  • Work independently and autonomously
  • Effectively manage and prioritize research plans through ambiguous and fast-changing environments, align and efficiently execute critical insights and work with a large group of stakeholders
  • Communicate results and illustrate suggestions in compelling and unique ways
  • Work cross-functionally with design, product management, content strategy, engineering and marketing
  • Generate insights that both fuel ideation and evaluate designs
What we offer
What we offer
  • bonus
  • equity
  • benefits
  • Fulltime
Read More
Arrow Right

UX Researcher, Mixed Methods

Our UX Research team is designing for the broad spectrum of human needs, which r...
Location
Location
United States , Bellevue
Salary
Salary:
164000.00 - 227000.00 USD / Year
meta.com Logo
Meta
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor’s degree with 10+ years of relevant experience in user experience, applied research and/or product research and development or a Master’s degree and 8+ years relevant experience, or PhD and 5+ years relevant experience
  • Relevant experience within user experience, applied research setting and/or product research and development
  • Experience conducting In-Depth Interviews or Focus Groups and Concept Testing or Usability Testing
  • Experience coding with R, SQL, STATA, SPSS or equivalent
  • Experience applying statistical analysis methods such as Regressions, ANOVA, and T-Tests
  • Interest in and experience executing hands-on, primary research
  • Experience translating research findings into strategic narratives
Job Responsibility
Job Responsibility
  • Work closely with product and business teams to identify research topics
  • Act as a thought leader in the domain of research, while advocating for the people who could use our products
  • Design and execute end-to-end custom primary research using a wide variety of methods
  • Design studies that address both user behavior and attitudes
  • Effectively manage and prioritize research plans through ambiguous and fast-changing environments, align and efficiently execute critical insights and work with a large group of stakeholders
  • Communicate results and illustrate suggestions in compelling and interesting ways
  • Work cross-functionally with design, product management, content strategy, engineering and marketing
  • Generate insights that both fuel ideation and evaluate designs
What we offer
What we offer
  • bonus
  • equity
  • benefits
  • Fulltime
Read More
Arrow Right

UX Researcher, Qualitative

Our UX Research team is designing for the global spectrum of human needs, which ...
Location
Location
United States , Menlo Park
Salary
Salary:
103000.00 - 166000.00 USD / Year
meta.com Logo
Meta
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor’s degree with 4+ years of relevant experience within user experience, applied research setting and/or product research and development, Master’s degree and 2+ years relevant experience, or a PhD and relevant experience
  • Experience conducting In-Depth Interviews or Focus Groups and Concept Testing or Usability Testing
  • Interest in and experience executing hands-on, primary research
  • Experience translating research findings into strategic narratives
Job Responsibility
Job Responsibility
  • Work closely with product and business teams to identify research topics
  • Act as a thought leader in the domain of research, while advocating for the people who could use our products
  • Design and execute end-to-end custom primary research using a wide variety of methods
  • Design studies that address both user behavior and attitudes
  • Effectively manage and prioritize research plans through ambiguous and fast-changing environments, align and efficiently execute critical insights and work with a large group of stakeholders
  • Communicate results and illustrate suggestions in compelling and creative ways
  • Work cross-functionally with design, product management, content strategy, engineering and marketing
  • Generate insights that both fuel ideation and evaluate designs
What we offer
What we offer
  • bonus
  • equity
  • benefits
Read More
Arrow Right

User Experience Researcher, Qualitative

Our UX Research team is designing for businesses of all shapes and sizes, which ...
Location
Location
United States , Bellevue
Salary
Salary:
141000.00 - 192000.00 USD / Year
meta.com Logo
Meta
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor’s degree with 4+ years of relevant experience within user experience, applied research setting and/or product research and development, Master’s degree and 2+ years relevant experience, or a PhD and relevant experience
  • Experience conducting In-Depth Interviews or Focus Groups and Concept Testing or Usability Testing
  • Interest in and experience executing hands-on, primary research
  • Experience translating research findings into strategic narratives
Job Responsibility
Job Responsibility
  • Work closely with product and business teams to identify research topics
  • Act as a thought leader in the domain of research, while advocating for the people who could use our products
  • Design and execute end-to-end custom primary research using a wide variety of methods
  • Design studies that address both user behavior and attitudes
  • Effectively manage and prioritize research plans through ambiguous and fast-changing environments, align and efficiently execute critical insights and work with a large group of stakeholders
  • Communicate results and illustrate suggestions in compelling and novel ways
  • Work cross-functionally with design, product management, content strategy, engineering and marketing
  • Generate insights that both fuel ideation and evaluate designs
What we offer
What we offer
  • bonus
  • equity
  • benefits
  • Fulltime
Read More
Arrow Right

Ux Researcher, Mixed Methods

Our UX Research team is designing for the broad spectrum of human needs, which r...
Location
Location
United States , Bellevue
Salary
Salary:
141000.00 - 192000.00 USD / Year
meta.com Logo
Meta
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor’s degree with 7+ years of relevant experience in user experience, applied research and/or product research and development or a Master’s degree and 5+ years relevant experience, or PhD and 2+ years relevant experience
  • Relevant experience within user experience, applied research setting and/or product research and development
  • Experience conducting In-Depth Interviews or Focus Groups and Concept Testing or Usability Testing
  • Experience coding with R, SQL, STATA, SPSS or equivalent
  • Experience applying statistical analysis methods such as Regressions, ANOVA, and T-Tests
  • Interest in and experience executing hands-on, primary research
  • Experience translating research findings into narratives
  • Proven experiences in a fast-moving organization, open to learning new research methods, committed to high-quality and rigorous research, and focused on influencing the future of Meta
  • Experience of consistently working under your own initiative, seeking feedback and input where appropriate
Job Responsibility
Job Responsibility
  • Work closely with product and business teams to identify research topics
  • Act as a thought leader in the domain of research, while advocating for the people who could use our products
  • Design and execute end-to-end custom primary research using a wide variety of methods
  • Design studies that address both user behavior and attitudes
  • Effectively manage and prioritize research plans through ambiguous and fast-changing environments, align and efficiently execute critical insights and work with a large group of stakeholders
  • Communicate results and illustrate suggestions in compelling ways
  • Work cross-functionally with design, product management, content strategy, engineering and marketing
  • Generate insights that both fuel ideation and evaluate designs
What we offer
What we offer
  • bonus
  • equity
  • benefits
  • Fulltime
Read More
Arrow Right

Hf+uxd engineer

The HF+UXD engineer will serve as an individual contributor to Diabetes Care pro...
Location
Location
United States , Andover
Salary
Salary:
45.00 - 50.00 USD / Hour
gomillenniumsoft.com Logo
MillenniumSoft Inc
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Human Factors & User Experience Tools: Drive understanding of the human-system interface through the creation of user profiles, environmental characterization, workflow mapping, task analysis, functional modeling, and use cases / scenarios
  • Utilize anthropometry, biomechanics, and human capability data to support, analyze, and optimization system design and ergonomics
  • User Needs/Requirements & User Risk Assessment: Distill research insights into themes that inform user needs, user risk, and user interface design through processes such as affinization or KJ Analysis to ensure traceability
  • Have working understanding of the requirements management process that focuses on the creation of user needs and stakeholder requirements as part of a cross-functional team
  • Evaluate Use Error as part of use-related risk analysis (aEURA, uFMEA, and/or Hazard Analysis) and contribute to mitigation efforts in partnership with medical affairs and cross-functional teams
  • Design of User Interfaces: Support design and development of user interfaces which include but are not limited to devices, software, packaging, instructions, and training
  • Conduct product evaluations, brainstorming sessions, competitive analysis, and other usability engineering methodologies with internal teams and external vendors
  • Utilize low to high fidelity prototyping methods to enable quick turn customer centric design loops that power Agile processes and decision making
  • Generative Research & Front-End Innovation: Participate as a member of the experience teams to conduct generative research including but not limited to observational research (contextual inquiry and ethnography), competitive usability assessment as required
  • Formative Evaluation, Design Validation, and Summative Human Factor Design Validation: Design, plan, execute, analyze, and report using qualitative and quantitative research as part of formative study evaluation which may include but are not limited to heuristic evaluation, expert review, simulated use studies, and actual use studies
Job Responsibility
Job Responsibility
  • Usability Planning & Execution: Collaborate with cross functional core teams to ensure human factors and usability plans are appropriately scaled
  • Execute human factors and user experience design deliverables by working with minimal supervision, managing the work of external vendors, and consulting with project teams to ensure quality deliverables
  • Support the Diabetes Care team in execution of deliverables associated with closing gaps in project usability files
  • Fulltime
Read More
Arrow Right
Jobs by Category
Art and Entertainment Jobs Banking Jobs Call Center Jobs Construction Jobs Consulting Jobs Customer Service Jobs Education Jobs Energy Jobs Finance Jobs Gastronomy Jobs Health and Beauty Jobs Hospitality and Tourism Jobs Human Resources Jobs Insurance Jobs IT - Administration Jobs IT - Software Development Jobs Legal Jobs Logistics Jobs Manufacturing Jobs Marketing, Digital Marketing, SEO, SEM Jobs Media Jobs Nursing Jobs Office Administration Jobs Pharmacy Jobs Physical Work Jobs Public Relations Jobs Public Sector Jobs Purchasing Jobs Quality Control Jobs Real Estate Jobs Research and Development Jobs Sales Jobs
Job Positions Jobs by Country
United States Jobs United Kingdom Jobs India Jobs Canada Jobs Australia Jobs Germany Jobs Ireland Jobs Poland Jobs
Company
Blog About Us Contact
Follow Us
Facebook
X (Twitter)
LinkedIn
Instagram
CrawlJobs Group
ITFlashcards AllDevBlogs
© 2026 CrawlJobs Ltd. All Rights Reserved
Terms & Conditions Privacy Policy