CrawlJobs Logo

Research Scientist, Human AI Interaction

joinhandshake.com Logo

Handshake

Location Icon

Location:
United States , San Francisco, CA, New York, NY

Category Icon

Job Type Icon

Contract Type:
Not provided

Salary Icon

Salary:

350000.00 - 420000.00 USD / Year

Job Description:

As a Research Scientist, Human–AI Interaction, you will play a pivotal role in defining how AI systems support real human work by leading research at the intersection of Human–Computer Interaction (HCI), Large Language Models (LLMs), and task-level benchmarking. You will operate at the frontier of human-centered AI evaluation, with a focus on understanding what people actually do to accomplish meaningful work—and how AI systems change, accelerate, or reshape that activity. Your research will define jobs-to-be-done benchmarks, comparative evaluation frameworks, and empirical methods for measuring human effort, time, quality, and outcomes when working with AI copilots. Additionally, the Handshake AI platform is an interface used by thousands of the top subject matter experts in the world to evaluate AI systems, and offers numerous interesting HCI / HITL-AI research questions that will drive large business impact. You’ll set research direction, establish standards for measuring human activity in AI-mediated workflows, publish papers and open-source code, and lead the development of rigorous, scalable benchmarks that connect human work, AI assistance, and real economic value.

Job Responsibility:

  • Lead high-impact research on jobs-to-be-done benchmarks for AI systems, including: Defining task taxonomies grounded in real professional and economic activities
  • Identifying what constitutes meaningful task completion, quality, and success
  • Translating qualitative work understanding into measurable, repeatable benchmarks
  • Develop methods to measure human activity in AI-mediated workflows
  • Design benchmarks to assess AI-as-a-collaborator/copilot, rather than autonomous agents / basic Q&A
  • Design and run empirical studies of how people use AI to solve tasks, including: Controlled experiments and field studies measuring task performance
  • Instrumentation for capturing fine-grained interaction traces and outcomes
  • Drive strategy for professional-domain AI benchmarks, focusing on: Understanding domain-specific workflows (e.g., analysis, writing, planning, coordination)
  • Grounding benchmark design in how work is actually performed, not idealized tasks
  • Build and prototype AI systems and evaluation infrastructure to support research and Data production, including: LLM-powered copilots and experimental tools used for task-level measurement
  • Benchmark harnesses that evaluate both model behavior and human outcomes
  • Data pipelines for analyzing human–AI interaction at scale
  • The human-in-the-loop experience for Handshake fellows to produce effective evaluations and training data for frontier models, through structured UI/UX interactions with these models
  • Collaborate closely with User Experience Research (UXR) to: Leverage deep qualitative insights into real user behavior and workflows
  • Translate ethnographic and observational findings into formal research constructs
  • Publish and present research that advances the field of human-centered AI benchmarking, with an expectation of regular contributions to top-tier venues such as CHI (Conference on Human Factors in Computing Systems), and related HCI and AI conferences

Requirements:

  • PhD or equivalent experience in Human–Computer Interaction, Computer Science, Cognitive Science, or a related field, with a strong emphasis on empirical evaluation of interactive AI/LLM systems
  • 3+ years of academic or industry research experience post-PhD, including leadership on complex research initiatives and analyzing data from a real AI product
  • Strong publication record, with demonstrated impact in top-tier AI (NeurIPS, ICML, ICLR, ACL) and HCI (CHI) venues
  • Deep expertise in experimental design and measurement, particularly for: Task performance and human activity
  • Comparative evaluation frameworks
  • Mixed-methods research grounded in real-world behavior
  • Strong technical and coding skills, including: Python and data analysis / ML tooling
  • Experience building experimental systems and benchmark infrastructure
  • Familiarity working with LLM APIs, agent frameworks, or AI-assisted tooling
  • Proven ability to define and lead research agendas that connect human work, AI capability, and business or economic impact
  • Strong collaboration skills, especially working across research, engineering, product, and UXR teams

Nice to have:

  • Experience developing benchmarks or evaluation frameworks for human–AI systems or AI-assisted productivity tools
  • Prior work on copilot-style systems, agentic workflows, or automation of professional tasks
  • Familiarity with workplace studies, CSCW, or socio-technical systems research
  • Contributions to open-source tools, datasets, or benchmarks related to task-level evaluation
  • Interest in how AI reshapes labor, productivity, and the future of work
What we offer:
  • Equity in a fast-growing company
  • 401(k) match, competitive compensation, financial coaching
  • Paid parental leave, fertility benefits, parental coaching
  • Medical, dental, and vision, mental health support, $500 wellness stipend
  • $2,000 learning stipend, ongoing development
  • Internet, commuting, and free lunch/gym in our SF office
  • Flexible PTO, 15 holidays + 2 flex days, winter #ShakeBreak where our whole office closes for a week
  • Team outings & referral bonuses

Additional Information:

Job Posted:
February 20, 2026

Employment Type:
Fulltime
Work Type:
On-site work
Job Link Share:

Looking for more opportunities? Search for other job offers that match your skills and interests.

Briefcase Icon

Similar Jobs for Research Scientist, Human AI Interaction

AI Research Scientist, Robotics

The ideal Research Scientist candidate will use their skills in system design an...
Location
Location
United States , Redmond
Salary
Salary:
154000.00 - 217000.00 USD / Year
meta.com Logo
Meta
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's degree in Computer Science, Computer Engineering, relevant technical field, or equivalent practical experience
  • Currently has or is in the process of obtaining a PhD degree in the field of Artificial Intelligence, Robotics, Computer Vision, Machine Learning, Language, a related field, or equivalent practical experience
  • Experience with any of the following research areas: robotics, motion planning, embodied AI, human-robot interaction, sim-to-real transfer, learning from demonstration, reinforcement learning, dexterous manipulation, digital agents, vision language models, computer vision, egocentric perception, and/or LLMs
  • Experience in relevant robotics related research areas, such as: VLM, robot learning, reinforcement learning, imitation learning, action-conditioned world models, task and motion planning, sim-to-real transfer robotic control, manipulation, navigation, or generally embodied AI
Job Responsibility
Job Responsibility
  • Perform fundamental and applied research to push the scientific and technological frontiers of embodied artificial intelligence
  • Invent/improve novel data-driven paradigms for robotics, leveraging a variety of modalities (images, video, text, audio, tactile, etc)
  • Investigate paradigms that can deliver a spectrum of embodied behaviors - from simulated characters to real robots, and from short-horizon, low-level to long-horizon, high-level intelligence
  • Develop algorithms based on state-of-the-art machine learning and neural network methodologies
  • Define, build and benchmark new functionalities needed for the next generation of AI
  • Conduct research towards long-term product goals while identifying intermediate milestones
  • Plan and execute novel research based on long-term objectives of the organization
What we offer
What we offer
  • bonus
  • equity
  • benefits
Read More
Arrow Right

AI Research Scientist, Robotics

At Meta, we’re building the future of human connection and the technology that e...
Location
Location
United States , Redmond
Salary
Salary:
219000.00 - 301000.00 USD / Year
meta.com Logo
Meta
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's degree in Computer Science, Computer Engineering, relevant technical field, or equivalent practical experience
  • PhD degree in the field of Artificial Intelligence, Robotics, Computer Vision, Machine Learning, Language, a related field, or equivalent practical experience
  • Experience with any of the following research areas: robotics, motion planning, embodied AI, human-robot interaction, sim-to-real transfer, learning from demonstration, reinforcement learning, dexterous manipulation, digital agents, vision language models, computer vision, egocentric perception, and/or Large Language Models
  • 5+ years of industry experience in relevant robotics related research areas, such as: Vision Language Models robot learning, reinforcement learning, imitation learning, action-conditioned world models, task and motion planning, sim-to-real transfer robotic control, manipulation, navigation, or generally embodied AI
Job Responsibility
Job Responsibility
  • Perform fundamental and applied research to push the scientific and technological frontiers of embodied artificial intelligence
  • Invent/improve novel data-driven paradigms for robotics, leveraging a variety of modalities (images, video, text, audio, tactile, etc.)
  • Investigate paradigms that can deliver a spectrum of embodied behaviors - from simulated characters to real robots, and from short-horizon, low-level to long-horizon, high-level intelligence
  • Develop algorithms based on state-of-the-art machine learning and neural network methodologies
  • Define, build and benchmark new functionality needed for the next generation of AI
  • Conduct research towards long-term product goals while identifying intermediate milestones
  • Lead, plan, and execute novel research based on long-term objectives of the organization
What we offer
What we offer
  • bonus
  • equity
  • benefits
Read More
Arrow Right

Senior Applied Scientist

Microsoft is a company where innovators come to collaborate, envision what can b...
Location
Location
United States , Redmond
Salary
Salary:
119800.00 - 234700.00 USD / Year
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 4+ years related experience (e.g., statistics predictive analytics, research)
  • OR Master's Degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 3+ years related experience (e.g., statistics, predictive analytics, research)
  • OR Doctorate in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 1+ year(s) related experience (e.g., statistics, predictive analytics, research)
  • OR equivalent experience
  • 4+ years of experience in statistics, predictive analytics and research
  • Ability to meet Microsoft, customer and/or government security screening requirements
  • This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter
Job Responsibility
Job Responsibility
  • Collaborate with AI researchers and audio signal processing experts to design and build end-to-end ML systems, tuned for human-human and human-AI interactions
  • Design and develop ML pipelines involving data cleaning, feature engineering, model training, and evaluation
  • Work across the product lifecycle from prototyping to shipping production-grade code optimized for performance and memory and updating the deployed models based on A/B testing
  • Remain up to date with the latest advancements, trends and research and contribute towards our IP portfolio
  • Research and develop synthetic data generation strategies
  • Proactively follow state of the art research and share latest work, write papers, attend conferences and share knowledge in the wider team
  • Fulltime
Read More
Arrow Right

Senior Applied Scientist

We are developing the Intelligent Conversation and Communications Cloud (IC3) to...
Location
Location
United States , Redmond
Salary
Salary:
119800.00 - 234700.00 USD / Year
https://www.microsoft.com/ Logo
Microsoft Corporation
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's Degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 4+ years related experience (e.g., statistics predictive analytics, research)
  • OR Master's Degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 3+ years related experience (e.g., statistics, predictive analytics, research)
  • OR Doctorate in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND 1+ year(s) related experience (e.g., statistics, predictive analytics, research)
  • OR equivalent experience
  • Ability to meet Microsoft, customer and/or government security screening requirements are required for this role
  • These requirements include but are not limited to the following specialized security screenings: Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter
Job Responsibility
Job Responsibility
  • Collaborating with AI researchers and audio signal processing experts to design and build end-to-end ML systems, tuned for human-human and human-AI interactions
  • Design and develop ML pipelines involving data cleaning, feature engineering, model training, and evaluation
  • Work across the product lifecycle from prototyping to shipping production-grade code optimized for performance and memory and updating the deployed models based on A/B testing
  • Remain up to date with latest advancements, trends and research and contribute towards our IP portfolio
  • Fulltime
Read More
Arrow Right

Principal Research Scientist

We are hiring a Principal Research Scientist to join the Wikimedia Foundation’s ...
Location
Location
United States
Salary
Salary:
164924.00 - 251398.00 USD / Year
wikimediafoundation.org Logo
Wikimedia Foundation
Expiration Date
March 08, 2026
Flip Icon
Requirements
Requirements
  • A track record of scholarly publications and service to the research and scientific communities including but not limited to human-computer interaction and computational social science communities
  • 3 or more years of strategic technical leadership in large matrixed organizations or their equivalent in academia, with a proven ability to counsel senior leadership
  • Ability to distill scientific nuance into high-impact, actionable clarity for non-technical stakeholders
  • Ability to manage a complex portfolio and mentor contributors while pivoting between high-level strategy and technical details
  • Advanced proficiency in AI, NLP or ML frameworks, with experience auditing algorithms for bias and accuracy OR mastery of mixed-method research specifically applied to online community governance and human system interaction
  • Proven ability to perform high velocity audits of research to identify methodological flaws, Movement and organizational opportunities and risks, and insights
  • PhD degree and a minimum of five years of work experience in a related field
Job Responsibility
Job Responsibility
  • Maintaining a fluent and real-time command of the global research landscape to inform the conversations on Wikipedia quality and integrity
  • Leading a strategic research portfolio on knowledge integrity
  • Mentoring individual contributors to execute high-impact scientific projects
  • Translating complex research findings into actionable strategy and recommendations for the Head of Research, Legal, and Communications teams
  • Delivering rapid, authoritative technical vetting of external research to identify organizational risks and scientific opportunities under tight deadlines
  • Driving research advocacy and public engagement in knowledge integrity research. domain, representing the Foundation’s scientific perspective to global stakeholders and the public
  • Cultivating strategic relationships within the scientific community and standards organizations, translating Wikimedia research findings into industry-wide best practices
  • Fulltime
!
Read More
Arrow Right

Applied Data Scientist

As an Applied Data Scientist on our Insights team, you will help pioneer the nex...
Location
Location
United States
Salary
Salary:
150000.00 - 200000.00 USD / Year
cresta.com Logo
Cresta
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 3+ years building and shipping models for real-world business applications, ideally in NLP and LLM-based systems
  • Strong proficiency in Python and standard ML / data tooling (e.g., SQL, data pipelines, experiment frameworks)
  • World-class first principles thinking and ML intuition
  • Ability to turn ambiguous product asks into crisp problem statements, eval specs, metrics, and hypotheses
  • Experience working directly with customers or internal stakeholders to understand constraints, explain tradeoffs, and iterate on solutions
  • Comfort working with design-partner style engagements where requirements evolve rapidly and you’re expected to co-create the solution
  • Track record of building evaluation suites that go beyond single scalar metrics to capture reliability, safety, and qualitative user experience
  • Strong written and verbal communication skills
  • able to clearly explain complex technical work to both engineers and non-technical partners
Job Responsibility
Job Responsibility
  • Co-develop new capabilities with a small number of high-impact enterprise customers along with our product, engineering, and design teams
  • using their real workflows and constraints as your testbed
  • Communicate effectively across all levels of the organization
  • Plan and run short, focused design-partner engagements (days to weeks) where you ship early versions, collect structured feedback, and iterate quickly
  • Generalize learnings from each design partner into reusable, productized capabilities rather than one-off bespoke models
  • Partner with domain experts to curate high-quality eval guidelines and datasets for domains such as CSAT prediction and outcome prediction (across both human<>human and human<>AI interactions)
  • Use the best tools + models for the job (simple and interpretable where it matters, sophisticated where it can drive outsized value
  • Write clear specs and experiment reports that make tradeoffs and assumptions explicit
  • Stay close to the research frontier in ML/AI, LLMs, and evals, translating promising ideas into pragmatic, shippable improvements
  • Where applicable, help translate your solutions into publications, whitepapers, technical blogs, etc.
What we offer
What we offer
  • Comprehensive medical, dental, and vision coverage with plans to fit you and your family
  • Flexible PTO to take the time you need, when you need it
  • Paid parental leave for all new parents welcoming a new child
  • Retirement savings plan to help you plan for the future
  • Remote work setup budget to help you create a productive home office
  • Monthly wellness and communication stipend to keep you connected and balanced
  • In-office meal program and commuter benefits provided for onsite employees
  • Offers Equity
  • Fulltime
Read More
Arrow Right

Senior AI Engineer (Agents)

We are looking for an experienced and exceptional Senior AI Engineer (Agents) to...
Location
Location
Singapore
Salary
Salary:
Not provided
workato.com Logo
Workato
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor's degree in Computer Science, or a related field, or equivalent practical experience
  • 5+ years in backend software development using modern programming languages (e.g., Python (strongly preferred!), Golang or Java)
  • Demonstrated experience building production AI systems including chatbots, virtual assistants, and automated support agents using LLMs (OpenAI, Anthropic, open-source models)
  • Expertise in natural language understanding (NLU) and intent classification for customer query interpretation, entity extraction, and conversation flow management
  • Expertise in building knowledge bases and FAQ systems with dynamic content retrieval and self-learning capabilities from support interactions
  • Experience implementing multi-channel support automation across chat, email, voice, and messaging platforms with consistent context handling
  • Deep knowledge of REST API design and integration patterns
  • Experience working with PostgreSQL and ClickHouse, or similar relational and analytical databases
  • Strong understanding of software architecture, scalability, security, and system design
Job Responsibility
Job Responsibility
  • Design and implement advanced AI/ML systems with a focus on LLMs, AI Agents, and retrieval-augmented generation (RAG) architectures
  • Build conversational AI interfaces that handle multi-turn customer interactions, maintain context across sessions, and seamlessly escalate to human agents when necessary
  • Build production-grade AI pipelines for data processing, model training, fine-tuning, and serving at scale
  • Implement feedback loops and continuous learning systems that incorporate customer satisfaction metrics, agent corrections, and conversation outcomes to improve model performance over time
  • Create analytics dashboards and reporting tools to track automation effectiveness, identify common customer pain points, and measure key performance indicators like resolution time, containment rate, and customer satisfaction scores
  • Lead technical initiatives for AI system integration into existing products and services
  • Collaborate with data scientists and ML researchers to implement and productionize new AI approaches and models
Read More
Arrow Right

AI Engineer

Prolific is not just another player in the AI space – we are the architects of t...
Location
Location
United Kingdom
Salary
Salary:
Not provided
prolific.com Logo
Prolific
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • A strong track record of designing, building, and shipping AI-powered features that have solved real business problems
  • An inquisitive and experimental mindset, with a passion for tackling ambiguous problems that may not have established solutions
  • Deep practical knowledge of the modern AI engineering stack (e.g., RAG, vector databases, prompt engineering, fine-tuning techniques, and evaluation frameworks)
  • The ability to thoughtfully balance cutting-edge research with the practical constraints of production reliability and scalability
  • Deep expertise in one, or both of the following areas: Software Engineering & Systems Design: Expertise in Python and experience building scalable services (e.g., using FastAPI, Django). You have a strong command of system design, infrastructure, and CI/CD pipelines, and you're comfortable taking a feature from concept to deployment
  • Machine Learning Engineering & Research: A comprehensive background in machine learning, with deep experience in ML frameworks (e.g., PyTorch, TensorFlow, Hugging Face) and a strong grasp of the underlying theory behind the models and techniques you use
Job Responsibility
Job Responsibility
  • Architect & Build: Lead the design and implementation of production-grade AI systems that enhance how human feedback is collected, validated, and utilised across our platform
  • Solve Core Challenges: Develop sophisticated systems for fraud detection, quality assurance, and intelligent participant-to-task matching using techniques like vector search and advanced profile representation
  • Innovate with LLMs: Create and deploy AI agents and systems with immediate product impact - from basic automated data tagging, to advanced features like LLM-based evaluation judges, all the way to experimental ideas such as synthetic personas
  • Drive Technical Strategy: Provide technical leadership and mentorship on the practical application of AI/LLM techniques, defining best practices for everything from prompt engineering to fine-tuning and RAG
  • Ensure Reliability: Implement and own LLM observability and evaluation systems to ensure our AI features are reliable, performant, and continuously improving
  • Collaborate Cross-Functionally: Work closely with data scientists, platform engineers, and research teams to build cohesive, end-to-end solutions that push the boundaries of human-AI interaction
What we offer
What we offer
  • competitive salary
  • benefits
  • remote working
  • impactful, mission-driven culture
Read More
Arrow Right