Human Evaluator Jobs

Secure Every Identity, from AI to Human Identity is the key to unlocking the potential of AI. Okta secures AI by building the trusted, neutral infrastructure that enables organizations to safely embrace this new era. This work requires a relentless drive to solve complex challenges with real-world stakes. We are looking for builders and owners who operate with speed and urgency and execute with excellence. This is an opportunity to do career-defining work. We're all in on this mission. If you are too, let's talk. About the Role: Okta is looking for a highly skilled Globalization Engineer to be the technical backbone of our global digital operations. You will be responsible for owning, building, and optimizing the automation and technology stack that powers our localization team. This is a hands-on role for an experienced technical expert who is passionate about building robust, scalable systems. You will leverage your deep expertise in APIs, AI, and internationalization to solve complex technical challenges and drive efficiency. As a subject matter expert, you will also act as a key technical consultant to internal teams, helping them prepare content for a global audience. What you'll be doing: Build and Own the Localization Automation Framework: Design, develop, and maintain the core automation workflows for our content lifecycle using agentic workflows, Python and APIs, connecting our TMS with content sources, repositories, and other internal tools. Spearhead AI and Technology Integration: Research, evaluate, and implement cutting-edge localization technologies, including AI/LLM-based solutions and advanced TMS features, to improve the quality, speed, and cost-effectiveness of our workflows. Provide Technical Oversight and Systems Integrity: Act as the authority for resolving high-impact architectural failures. Conduct deep-dive root cause analysis on complex integration issues, file parsing conflicts, and system-wide bottlenecks to ensure uninterrupte

At Lyft, our purpose is to serve and connect. We aim to achieve this by cultivating a work environment where all team members belong and have the opportunity to thrive. The Safety and Customer Care (SCC) team at Lyft manages over 1.7 million monthly human and AI interactions and serves as Lyft's primary direct touchpoint with riders and drivers. We handle critical infrastructure that powers both human associates and AI agents to make riders and drivers feel safe and comfortable while riding or driving with Lyft, transforming every support interaction into a moment of genuine connection. Agentic AI is at the center of how we scale that mission. We fine-tune and align open-source models, build AI-powered support agents, and develop end-to-end AI agents for safety case management, systems that reason over complex, high-stakes cases and drive them to resolution. SCC brings together ML, data, backend, and product engineers alongside data scientists and operations partners to transform these systems. As a Machine Learning Engineer on the SCC team, you will fine-tune and align models and build AI Agents that power how riders and drivers get help. Your work spans the full loop: post-training open-source models for our domain, composing them into multi-step agents, and building the evaluation that proves they are safe to ship in a customer-facing, safety-critical setting. Post-train and adapt open-source LLMs for SCC use cases using SFT, LoRA, and preference-tuning methods (RLHF, RLAIF, RLVR). Design and build AI-powered support agents and end-to-end agents for safety case management using LangGraph or equivalent agentic frameworks. Own the evaluation data flywheel, offline and online, that defines what "good" looks like and build benchmarks for the team to hill-climb. Turn interaction feedback into training data and learning signals, closing the data flywheel that continuously improves the models. Responsibilities: Conduct literature review and build post-training fra

About Bolna Bolna is a YC-backed voice AI orchestration platform built for the Indian market—powering multilingual, vernacular voice agents across Hindi, Hinglish, Tamil, and 10+ languages at sub-500ms latency across collections, recruitment, sales, and e-commerce use cases. We are an orchestration layer, not a model company: our moat is outcome-labelled vernacular data, rigorous evaluation infrastructure, and a growing taxonomy of how Indian enterprise voice AI fails in production. Why This Role Exists Product decisions at Bolna increasingly hinge on rigorous, code-mixed-aware data analysis—and not just one kind. On one side, there is model and evaluation rigor: LLM benchmarking for post-call intelligence, ASR/WER evaluation, inter-rater reliability on human-labelled calls, and routing and latency economics. On the other, there is product and growth insight: understanding where self-serve users drop off in their journey, what patterns emerge across lakhs of monthly calls, and which use cases and configurations are actually working. Both currently sit with the Head of Product alongside strategy and roadmap ownership. We need a dedicated analyst to own the execution and recurring cadence across both-freeing product leadership to act on findings rather than produce them. What You’ll Do Model and Evaluation Analysis LLM and model benchmarking: Run structured comparisons across model providers such as Sarvam, DeepSeek, Gemini, and Claude variants for tasks including post-call extraction and LLM-as-judge scoring. Evaluate cost, accuracy, fill rate, and TTR, with particular attention to Hinglish and code-mixed content. Evaluation infrastructure: Build and maintain LLM-as-judge pipelines using tools such as DeepEval, design and track evaluation metrics, and run inter-rater reliability analysis such as Krippendorff’s alpha across human call reviewers. Golden dataset creation: Support the construction of golden datasets for ASR and transcript labelling, including flagging co

Secure Every Identity, from AI to Human Identity is the key to unlocking the potential of AI. Okta secures AI by building the trusted, neutral infrastructure that enables organizations to safely embrace this new era. This work requires a relentless drive to solve complex challenges with real-world stakes. We are looking for builders and owners who operate with speed and urgency and execute with excellence. This is an opportunity to do career-defining work. We're all in on this mission. If you are too, let's talk. The Agent Gateway Team The Agent Gateway team owns the identity-aware infrastructure that connects enterprise AI agents to the tools, data, and services their organizations authorize. Every call from Claude, Agentforce, Codex, and internal/homegrown agents to a resource flows through us. We enforce authorization, isolate credentials, mint the right token per target, and produce the audit trails that security teams rely on. We are early in a rapidly evolving space. The standards for agent identity (MCP, OAuth token exchange, DCR) are being built under our feet. Our roadmap includes hardening the data plane for on-premises customer deployments, extending policy semantics beyond tool-level allowlists, adding native support for Agent-to-Agent brokered delegation, and scaling to tenants with thousands of virtual MCP servers. The Senior Software Engineer Opportunity Okta is looking for a Senior Software Engineer to help build the Agent Gateway. You will own features and components across the data and control planes, turning technical designs and product requirements into reliable production systems. Working alongside staff engineers, you will implement token exchange, request routing, credential resolution, and policy evaluation as agent identity specifications evolve. This is a hands-on software development role at the intersection of product, security, and infrastructure. You will ship services that handle agentic traffic reliably and performantly at scale. Wha

Secure Every Identity, from AI to Human Identity is the key to unlocking the potential of AI. Okta secures AI by building the trusted, neutral infrastructure that enables organizations to safely embrace this new era. This work requires a relentless drive to solve complex challenges with real-world stakes. We are looking for builders and owners who operate with speed and urgency and execute with excellence. This is an opportunity to do career-defining work. We're all in on this mission. If you are too, let's talk. The Agent Gateway Team The Agent Gateway team owns the identity-aware infrastructure that connects enterprise AI agents to the tools, data, and services their organizations authorize. Every call from Claude, Agentforce,, Codex, and internal/homegrown agents to a resource flows through us. We enforce authorization, isolate credentials, mint the right token per target, and produce the audit trails that security teams rely on. We are early in a rapidly evolving space. The standards (MCP, XAA, ID JAG, DCR for agents,CIMD) are being built under our feet. Our roadmap includes hardening the data plane for on-premises customer deployments, extending policy semantics beyond tool-level allowlists, adding native support for Agent-to-Agent brokered delegation, XAA and scaling to tenants with thousands of virtual MCP servers. The Staff Software Engineer Opportunity Okta is looking for a Staff Software Engineer to serve as a technical anchor for the Agent Gateway team. You will own critical parts of the data and control planes and drive architectural design as the MCP and agent identity specs evolve. In this role, you will work close to the metal on request routing, token exchange, credential resolution, and policy evaluation. You will work equally close to the identity control plane on config bundles, tenant fanout, and operational rollouts. You are expected to make sharp technology choices, prototype with agentic tooling, and turn rough product ideas into product

Secure Every Identity, from AI to Human Identity is the key to unlocking the potential of AI. Okta secures AI by building the trusted, neutral infrastructure that enables organizations to safely embrace this new era. This work requires a relentless drive to solve complex challenges with real-world stakes. We are looking for builders and owners who operate with speed and urgency and execute with excellence. This is an opportunity to do career-defining work. We're all in on this mission. If you are too, let's talk. Position Description: You will proactively construct and own an AI-forward research roadmap for key product pillars at Okta, operating at the intersection of Identity, Security, and Artificial Intelligence. As a Staff-level individual contributor, you will drive both qualitative and quantitative methodologies to define user experiences for non-deterministic AI agents, automated workflows, and enterprise platforms. Additionally, you leverage agentic tools and AI workflows internally to scale qualitative coding, synthesis, and research operations across the organization. In this role, you’ll get to: Drive Product-shaping Research: Lead foundational and evaluative research to inform strategic and tactical product decisions. Execute Balanced Mixed-Methods: Drive qualitative depth (contextual inquiry, cognitive walkthroughs) and quantitative rigor (large-scale surveys, behavioral telemetry, statistical modeling) to evaluate AI products. Pioneer AI Research Operations: Integrate AI tools, prompt engineering, and agentic research workflows into your daily practice to accelerate quality research outputs. Lead Cross-Discipline Collaboration: Facilitate research to drive cross-discipline clarity, executive alignment, and decision-making across complex security and admin workflows. Solve Complex Problems: Bring stakeholders together across multiple teams to scope problems and collaborate on research in ways that nurture a culture of curiosity and learning. Cultivate E

Secure Every Identity, from AI to Human Identity is the key to unlocking the potential of AI. Okta secures AI by building the trusted, neutral infrastructure that enables organizations to safely embrace this new era. This work requires a relentless drive to solve complex challenges with real-world stakes. We are looking for builders and owners who operate with speed and urgency and execute with excellence. This is an opportunity to do career-defining work. We're all in on this mission. If you are too, let's talk. About the Role As Senior Product Manager, Policies, you own product strategy and execution for Okta's authentication policy roadmap. The Policies team owns how Okta evaluates risk and context to decide when, how, and whether a user, machine, or agentic identity is allowed to authenticate and stay authenticated. This includes Application Sign-On Policy, Global Session Policy, Okta Account Management Policy (OAMP), and Device Signal Collection Policy as well as the underlying Policy Framework, Policy API, and Auth Pipeline that power them. Policies is foundational to the entire Okta product portfolio as it's the decision layer that sits between every authentication event and the security experience a customer wants for their end users. You'll work closely with engineering, design, other product teams, and cross-functional stakeholders across Okta's platform to ensure policy capabilities evolve strategically, including how policy changes are rolled out and managed at scale (change management / identity operations). What You'll Do Own roadmap and prioritization for your product area, balancing customer requests, platform needs, and market/strategic differentiation Collaborate with design and engineering to define, scope, and ship policies that have best-in-class user experience while meeting the security needs of Okta customers Drive product direction for the Policy Framework, Policy API, and Auth Pipeline that underpin every policy evaluation acros

About Ramp Ramp is building the smart infrastructure for finance teams, embedded in the transaction flow of every dollar a business spends. We automate how over $200B in annualized spend flows in and out of 70,000+ companies: authorizing payments, flagging risk, categorizing spend, and closing books. The problems are high-stakes, data-dense, and unforgiving. We hire people with high agency and high urgency. We look for slope over intercept. We care less about where you trained and more about what you’ve built. At Ramp, everyone is a builder who owns problems end to end and makes consequential decisions that shape the outcome. The median Ramp customer saves 5% and grows revenue 16% in their first year – far in excess of businesses operating without Ramp. We believe every ambitious company deserves the same. If you want to build systems that directly shape how companies move and manage billions, Ramp is the place to do it. About the Role As the Product Manager for Tax, you will own the vision, strategy, and roadmap for Ramp’s next generation of AI-powered tax products. You will build solutions to automate tax calculation, validation, and filing for our existing suite of products (card, bill pay, banking, etc), for both domestic and international use cases. You’ll also be accountable for **expanding** Stack - our agentic accounting product - into the tax automation domain, to automate the work of CPAs. You’ll partner closely with engineering, design, data, and cross-functional stakeholders (Finance, Legal, Security, CX, Sales) to build intelligent experiences that help customers save money, reduce risk, and eliminate manual work. This role blends AI-native product development (LLM-powered workflows, evaluation systems, human-in-the-loop design) and deep domain expertise in domestic/international tax and CPA’s workflows. You will define and execute a strategy that will unlock new markets for Ramp and help us save our customers even more time and money. Please note that

Secure Every Identity, from AI to Human Identity is the key to unlocking the potential of AI. Okta secures AI by building the trusted, neutral infrastructure that enables organizations to safely embrace this new era. This work requires a relentless drive to solve complex challenges with real-world stakes. We are looking for builders and owners who operate with speed and urgency and execute with excellence. This is an opportunity to do career-defining work. We're all in on this mission. If you are too, let's talk. The Sr Director of Global Deal Strategy Opportunity Okta is seeking a Sr Director of Global Deal Strategy to lead a high-impact global function at the center of our commercial engine. This leader will shape how Okta structures, evaluates, and executes strategic and complex transactions, while building the operating model, governance, and team required to scale with the business. This role is equal parts strategist, operator, and trusted business partner. You will work closely with Sales leadership and cross-functional partners across Finance, Legal, Revenue, Product, and our internal systems and technology team to improve deal quality, accelerate velocity, and ensure commercial decisions align with Okta’s growth, profitability, and risk objectives. What you’ll be doing Strategic Deal Leadership Lead Okta’s global deal strategy vision, operating model, and engagement framework across regions, segments, and routes to market. Serve as a trusted advisor to Sales leadership on strategic, competitive, and non-standard deals. Act as a senior escalation point on high-value, high-risk, and business-critical transactions, balancing growth, customer outcomes, profitability, and risk. Commercial Governance & Deal Structuring Establish and evolve global guardrails for pricing, discounting, approvals, exception handling, and non-standard commercial structures, in alignment with our Pricing & Packaging team. Partner with Finance, Legal, Revenue, Product, and

Secure Every Identity, from AI to Human Identity is the key to unlocking the potential of AI. Okta secures AI by building the trusted, neutral infrastructure that enables organizations to safely embrace this new era. This work requires a relentless drive to solve complex challenges with real-world stakes. We are looking for builders and owners who operate with speed and urgency and execute with excellence. This is an opportunity to do career-defining work. We're all in on this mission. If you are too, let's talk. You will proactively construct and own an AI-forward research roadmap for key product pillars at Okta, operating at the intersection of Identity, Security, and Artificial Intelligence. As a Staff-level individual contributor, you will drive both qualitative and quantitative methodologies to define user experiences for non-deterministic AI agents, automated workflows, and developer platforms. Additionally, you leverage agentic tools and AI workflows internally to scale qualitative coding, synthesis, and research operations across the organization. In this role, you’ll get to: Drive AI Agent Research: Lead foundational and evaluative research on autonomous AI agents and the security thereof. Execute Balanced Mixed-Methods: Drive qualitative depth (contextual inquiry, cognitive walkthroughs) and quantitative rigor (large-scale surveys, behavioral telemetry, statistical modeling) to evaluate AI products. Pioneer AI Research Operations: Integrate AI tools, prompt engineering, and agentic research workflows into your daily practice to accelerate quality research outputs. Lead Cross-Discipline Collaboration: Facilitate research to drive cross-discipline clarity, executive alignment, and decision-making across complex security and admin workflows. Solve Complex Problems: Bring stakeholders together across multiple teams to scope problems and collaborate on research in ways that nurture a culture of curiosity and learning. Cultivate Empathy: Design mechanisms to bui

Secure Every Identity, from AI to Human Identity is the key to unlocking the potential of AI. Okta secures AI by building the trusted, neutral infrastructure that enables organizations to safely embrace this new era. This work requires a relentless drive to solve complex challenges with real-world stakes. We are looking for builders and owners who operate with speed and urgency and execute with excellence. This is an opportunity to do career-defining work. We're all in on this mission. If you are too, let's talk. The Okta Privileged Access Management (PAM) is an identity-centric approach to a common and critical privileged access use case. Our elegant Zero Trust architecture is purpose-built for the modern cloud and helps customers solve challenging security and operations pain points at scale. We're looking for a Senior level Platform Engineer to join a team of highly skilled and talented team players who are proud of what they own and deliver. Our elite team is fast, creative, and flexible; with a weekly release cycle and individual ownership, we expect great things from our engineers and reward them with stimulating new projects, new technologies, and the chance to have significant equity in a company that is changing the cloud computing landscape forever. What you’ll do Leverage cutting-edge AI pair-programmers and LLMs (such as Copilot and Claude) to accelerate the development of secure, enterprise-grade Privileged Access Management (PAM) products. Work with engineering teams to design, develop and deliver cloud-based infrastructure projects on a modern tech stack (Kubernetes/EKS, RDS, DynamoDB, Kinesis, MKS, Redis, OpenSearch, Docker, Terraform on AWS) Drive evaluation, development, and rollout of microservices Operate, support, and upgrade shared services and frameworks. Scale these as their usage invariably grows along with Okta's business. Evaluate and scale existing systems to meet specialized requirements and support Okta’s future business ne

Secure Every Identity, from AI to Human Identity is the key to unlocking the potential of AI. Okta secures AI by building the trusted, neutral infrastructure that enables organizations to safely embrace this new era. This work requires a relentless drive to solve complex challenges with real-world stakes. We are looking for builders and owners who operate with speed and urgency and execute with excellence. This is an opportunity to do career-defining work. We're all in on this mission. If you are too, let's talk. The role At Okta, we believe AI will fundamentally transform how design, research, and product development happen. We are looking for a Senior Design Operations Program Manager, Agentic Design to lead the operationalization and enablement of internal AI agents, plug-ins, and automated workflows across our Design team — reimaging how designers create in the AI era through internal AI agents. In this role, you will bridge the gap between AI tooling capabilities and practical design ops execution. You will partner closely with design, product management and design systems to govern, evaluate, deploy, and scale AI-driven design capabilities — making our 60+ person design org dramatically faster, more consistent, and highly innovative. This is a tactical, execution-focused builder role for an operations leader who is deeply curious about AI tools, understands modern UX workflows, and excels at change management, team enablement, and operationalizing new frameworks. What you'll do AI & Agentic Design Enablement Serve as the operational connective tissue driving AI adoption, evaluation, and workflow integration across Product Design, UX Research, and Content Design. Manage the intake, evaluation, and approval process for design plug-ins, Model Context Protocol (MCP) integrations, and generative AI tools (e.g., custom content design assistants, design-to-code agents). Partner with internal engineering teams to operationalize, ship, and drive adopt

Secure Every Identity, from AI to Human Identity is the key to unlocking the potential of AI. Okta secures AI by building the trusted, neutral infrastructure that enables organizations to safely embrace this new era. This work requires a relentless drive to solve complex challenges with real-world stakes. We are looking for builders and owners who operate with speed and urgency and execute with excellence. This is an opportunity to do career-defining work. We're all in on this mission. If you are too, let's talk. Okta is seeking a Staff Analyst – People Technology to join our Business Solutions Team. In this critical role, you will partner closely with the People Technology leadership and play a key role in our Workday transformation efforts. As part of a cross-functional team, you will collaborate with stakeholders across People, Payroll, IT, Finance, and international offices to optimize and scale our HR systems and processes. This role is ideal for someone who is passionate about solving complex business challenges, driving innovation through automation, and delivering a best-in-class user experience. Job Duties and Responsibilities Key Responsibilities Own the end-to-end architecture and technical design of Workday Extend applications, integrations, and supporting technology solutions. Define scalable, secure, resilient, and maintainable application architecture, integration architecture, data flows, API strategies, and technical standards . Architect and lead the development of complex Workday Extend applications using TypeScript/JavaScript, Workday APIs, and Extend frameworks. Architect enterprise Workday integrations using Workday Studio, EIB, Core Connectors, RaaS, REST APIs, SOAP Web Services, and Workday Web Services . Evaluate business requirements and determine the appropriate solution across Workday configuration, Extend, integrations, APIs, AI, cloud services, and third-party platforms . Establish architectural patterns and reusable frameworks fo

Secure Every Identity, from AI to Human Identity is the key to unlocking the potential of AI. Okta secures AI by building the trusted, neutral infrastructure that enables organizations to safely embrace this new era. This work requires a relentless drive to solve complex challenges with real-world stakes. We are looking for builders and owners who operate with speed and urgency and execute with excellence. This is an opportunity to do career-defining work. We're all in on this mission. If you are too, let's talk. About the Team The Professional Services R&D team is a new, dynamic group at the forefront of innovation within Okta. Our mission is to design and build reusable, scalable assets and tools that empower our delivery teams and partners. By making customer engagements more efficient, streamlined, and cost-effective, we directly contribute to our customers' success and accelerate their time-to-value with Okta. This is a unique opportunity to join a strategic team from the ground up and shape the future of Okta's professional services. Position Summary As the Architect for the Professional Services R&D team, you will be the principal technical leader responsible for the high-level design and technical strategy of our entire portfolio of reusable assets. You will define the architectural vision and standards that ensure our solutions are robust, scalable, and secure. This role requires a blend of deep technical expertise, strategic thinking, and strong collaboration skills to guide the engineering team in building a cohesive ecosystem of tools on technologies like Java, React, Node, Python, and .NET. Responsibilities Define and own the end-to-end technical architecture for all reusable assets and solutions developed by the R&D team. Establish architectural standards, patterns, and best practices to guide the development team. Evaluate and select appropriate technologies, frameworks, and platforms to meet project requirements and long-term business goa

Every day, tens of millions of people come to Roblox to explore, create, play, learn, and connect with friends in 3D immersive digital experiences– all created by our global community of developers and creators. At Roblox, we’re building the tools and platform that empower our community to bring any experience that they can imagine to life. Our vision is to reimagine the way people come together, from anywhere in the world, and on any device. We’re on a mission to connect a billion people with optimism and civility, and looking for amazing talent to help us get there. A career at Roblox means you’ll be working to shape the future of human interaction, solving unique technical challenges at scale, and helping to create safer, more civil shared experiences for everyone. As a Senior Software Engineer on the Foundation AI organization, you will sit at the epicenter of our foundation model efforts. While the research world is focused on architecture, you will be the architect of the data flywheel that makes VideoGen and 3DGen possible. You aren't just building pipelines; you are building the infrastructure that defines how our models perceive and generate virtual worlds in three dimensions and across time. In this role, you will partner directly with our AI researchers to advance beyond experimental datasets and into the realm of dynamic, high-fidelity data synthesis and evaluation. You will bridge the gap between research prototypes working locally to scaling for millions of users. You will design, implement, and scale robust, high-performance infrastructure to crawl, create, curate, store, and serve the massive datasets required for these models. We are seeking accomplished software engineers with a passion for data, experience building large distributed systems, and a commitment to writing high-quality, well-tested code to solve complex data challenges at scale. Your contributions will ensure that our foundation models receive the highest quality dat