Human Evaluator in Canada

Software is eating the world, but AI is eating software. We live in unprecedented times – AI has the potential to exponentially augment human intelligence. Every person will have a personal tutor, coach, assistant, personal shopper, travel guide, and therapist throughout life. As the world adjusts to this new reality, leading platform companies are scrambling to build LLMs at billion scale, while large enterprises figure out how to add it to their products. To make them safe, aligned and actually useful, these models need human evaluation and reinforcement learning through human feedback (RLHF) during pre-training, fine-tuning, and production evaluations. This is the main innovation that’s enabled ChatGPT to get such a large headstart among competition. At Scale, our products include the Generative AI Data Engine, SGP, Donovan, and others that power the most advanced LLMs and generative models in the world through world-class RLHF, human data generation, model evaluation, safety, and alignment. The data we are producing is some of the most important work for how humanity will interact with AI. At the foundation of these products is the Platform Engineering team. In this role, you will lead the design and development of core data storage, streaming, caching, and indexing platforms and underlying systems. You’ll also get widespread exposure to the forefront of the AI race as Scale sees it in enterprises, startups, governments, and large tech companies. You will: Drive the architecture, design, implementation, and reliability of our foundational data platforms and systems, working closely with stakeholders and internal customers to understand and refine requirements. Collaborate with cross-functional teams to define, design, and deliver new features. Proactively identify opportunities for, and driving improvements to, current programming practices, including process enhancements and tool upgrades. Present technical information to teams and stakeholders, providing

Secure Every Identity, from AI to Human Identity is the key to unlocking the potential of AI. Okta secures AI by building the trusted, neutral infrastructure that enables organizations to safely embrace this new era. This work requires a relentless drive to solve complex challenges with real-world stakes. We are looking for builders and owners who operate with speed and urgency and execute with excellence. This is an opportunity to do career-defining work. We're all in on this mission. If you are too, let's talk. About the Role: Okta is looking for a highly skilled Globalization Engineer to be the technical backbone of our global digital operations. You will be responsible for owning, building, and optimizing the automation and technology stack that powers our localization team. This is a hands-on role for an experienced technical expert who is passionate about building robust, scalable systems. You will leverage your deep expertise in APIs, AI, and internationalization to solve complex technical challenges and drive efficiency. As a subject matter expert, you will also act as a key technical consultant to internal teams, helping them prepare content for a global audience. What you'll be doing: Build and Own the Localization Automation Framework: Design, develop, and maintain the core automation workflows for our content lifecycle using agentic workflows, Python and APIs, connecting our TMS with content sources, repositories, and other internal tools. Spearhead AI and Technology Integration: Research, evaluate, and implement cutting-edge localization technologies, including AI/LLM-based solutions and advanced TMS features, to improve the quality, speed, and cost-effectiveness of our workflows. Provide Technical Oversight and Systems Integrity: Act as the authority for resolving high-impact architectural failures. Conduct deep-dive root cause analysis on complex integration issues, file parsing conflicts, and system-wide bottlenecks to ensure uninterrupte

At Lyft, our purpose is to serve and connect. We aim to achieve this by cultivating a work environment where all team members belong and have the opportunity to thrive. The Safety and Customer Care (SCC) team at Lyft manages over 1.7 million monthly human and AI interactions and serves as Lyft's primary direct touchpoint with riders and drivers. We handle critical infrastructure that powers both human associates and AI agents to make riders and drivers feel safe and comfortable while riding or driving with Lyft, transforming every support interaction into a moment of genuine connection. Agentic AI is at the center of how we scale that mission. We fine-tune and align open-source models, build AI-powered support agents, and develop end-to-end AI agents for safety case management, systems that reason over complex, high-stakes cases and drive them to resolution. SCC brings together ML, data, backend, and product engineers alongside data scientists and operations partners to transform these systems. As a Machine Learning Engineer on the SCC team, you will fine-tune and align models and build AI Agents that power how riders and drivers get help. Your work spans the full loop: post-training open-source models for our domain, composing them into multi-step agents, and building the evaluation that proves they are safe to ship in a customer-facing, safety-critical setting. Post-train and adapt open-source LLMs for SCC use cases using SFT, LoRA, and preference-tuning methods (RLHF, RLAIF, RLVR). Design and build AI-powered support agents and end-to-end agents for safety case management using LangGraph or equivalent agentic frameworks. Own the evaluation data flywheel, offline and online, that defines what "good" looks like and build benchmarks for the team to hill-climb. Turn interaction feedback into training data and learning signals, closing the data flywheel that continuously improves the models. Responsibilities: Conduct literature review and build post-training fra

Secure Every Identity, from AI to Human Identity is the key to unlocking the potential of AI. Okta secures AI by building the trusted, neutral infrastructure that enables organizations to safely embrace this new era. This work requires a relentless drive to solve complex challenges with real-world stakes. We are looking for builders and owners who operate with speed and urgency and execute with excellence. This is an opportunity to do career-defining work. We're all in on this mission. If you are too, let's talk. The Agent Gateway Team The Agent Gateway team owns the identity-aware infrastructure that connects enterprise AI agents to the tools, data, and services their organizations authorize. Every call from Claude, Agentforce, Codex, and internal/homegrown agents to a resource flows through us. We enforce authorization, isolate credentials, mint the right token per target, and produce the audit trails that security teams rely on. We are early in a rapidly evolving space. The standards for agent identity (MCP, OAuth token exchange, DCR) are being built under our feet. Our roadmap includes hardening the data plane for on-premises customer deployments, extending policy semantics beyond tool-level allowlists, adding native support for Agent-to-Agent brokered delegation, and scaling to tenants with thousands of virtual MCP servers. The Senior Software Engineer Opportunity Okta is looking for a Senior Software Engineer to help build the Agent Gateway. You will own features and components across the data and control planes, turning technical designs and product requirements into reliable production systems. Working alongside staff engineers, you will implement token exchange, request routing, credential resolution, and policy evaluation as agent identity specifications evolve. This is a hands-on software development role at the intersection of product, security, and infrastructure. You will ship services that handle agentic traffic reliably and performantly at scale. Wha

Secure Every Identity, from AI to Human Identity is the key to unlocking the potential of AI. Okta secures AI by building the trusted, neutral infrastructure that enables organizations to safely embrace this new era. This work requires a relentless drive to solve complex challenges with real-world stakes. We are looking for builders and owners who operate with speed and urgency and execute with excellence. This is an opportunity to do career-defining work. We're all in on this mission. If you are too, let's talk. Position Description: You will proactively construct and own an AI-forward research roadmap for key product pillars at Okta, operating at the intersection of Identity, Security, and Artificial Intelligence. As a Staff-level individual contributor, you will drive both qualitative and quantitative methodologies to define user experiences for non-deterministic AI agents, automated workflows, and enterprise platforms. Additionally, you leverage agentic tools and AI workflows internally to scale qualitative coding, synthesis, and research operations across the organization. In this role, you’ll get to: Drive Product-shaping Research: Lead foundational and evaluative research to inform strategic and tactical product decisions. Execute Balanced Mixed-Methods: Drive qualitative depth (contextual inquiry, cognitive walkthroughs) and quantitative rigor (large-scale surveys, behavioral telemetry, statistical modeling) to evaluate AI products. Pioneer AI Research Operations: Integrate AI tools, prompt engineering, and agentic research workflows into your daily practice to accelerate quality research outputs. Lead Cross-Discipline Collaboration: Facilitate research to drive cross-discipline clarity, executive alignment, and decision-making across complex security and admin workflows. Solve Complex Problems: Bring stakeholders together across multiple teams to scope problems and collaborate on research in ways that nurture a culture of curiosity and learning. Cultivate E

Secure Every Identity, from AI to Human Identity is the key to unlocking the potential of AI. Okta secures AI by building the trusted, neutral infrastructure that enables organizations to safely embrace this new era. This work requires a relentless drive to solve complex challenges with real-world stakes. We are looking for builders and owners who operate with speed and urgency and execute with excellence. This is an opportunity to do career-defining work. We're all in on this mission. If you are too, let's talk. The Okta Privileged Access Management (PAM) is an identity-centric approach to a common and critical privileged access use case. Our elegant Zero Trust architecture is purpose-built for the modern cloud and helps customers solve challenging security and operations pain points at scale. We're looking for a Senior level Platform Engineer to join a team of highly skilled and talented team players who are proud of what they own and deliver. Our elite team is fast, creative, and flexible; with a weekly release cycle and individual ownership, we expect great things from our engineers and reward them with stimulating new projects, new technologies, and the chance to have significant equity in a company that is changing the cloud computing landscape forever. What you’ll do Leverage cutting-edge AI pair-programmers and LLMs (such as Copilot and Claude) to accelerate the development of secure, enterprise-grade Privileged Access Management (PAM) products. Work with engineering teams to design, develop and deliver cloud-based infrastructure projects on a modern tech stack (Kubernetes/EKS, RDS, DynamoDB, Kinesis, MKS, Redis, OpenSearch, Docker, Terraform on AWS) Drive evaluation, development, and rollout of microservices Operate, support, and upgrade shared services and frameworks. Scale these as their usage invariably grows along with Okta's business. Evaluate and scale existing systems to meet specialized requirements and support Okta’s future business ne

Secure Every Identity, from AI to Human Identity is the key to unlocking the potential of AI. Okta secures AI by building the trusted, neutral infrastructure that enables organizations to safely embrace this new era. This work requires a relentless drive to solve complex challenges with real-world stakes. We are looking for builders and owners who operate with speed and urgency and execute with excellence. This is an opportunity to do career-defining work. We're all in on this mission. If you are too, let's talk. The role At Okta, we believe AI will fundamentally transform how design, research, and product development happen. We are looking for a Senior Design Operations Program Manager, Agentic Design to lead the operationalization and enablement of internal AI agents, plug-ins, and automated workflows across our Design team — reimaging how designers create in the AI era through internal AI agents. In this role, you will bridge the gap between AI tooling capabilities and practical design ops execution. You will partner closely with design, product management and design systems to govern, evaluate, deploy, and scale AI-driven design capabilities — making our 60+ person design org dramatically faster, more consistent, and highly innovative. This is a tactical, execution-focused builder role for an operations leader who is deeply curious about AI tools, understands modern UX workflows, and excels at change management, team enablement, and operationalizing new frameworks. What you'll do AI & Agentic Design Enablement Serve as the operational connective tissue driving AI adoption, evaluation, and workflow integration across Product Design, UX Research, and Content Design. Manage the intake, evaluation, and approval process for design plug-ins, Model Context Protocol (MCP) integrations, and generative AI tools (e.g., custom content design assistants, design-to-code agents). Partner with internal engineering teams to operationalize, ship, and drive adopt

Secure Every Identity, from AI to Human Identity is the key to unlocking the potential of AI. Okta secures AI by building the trusted, neutral infrastructure that enables organizations to safely embrace this new era. This work requires a relentless drive to solve complex challenges with real-world stakes. We are looking for builders and owners who operate with speed and urgency and execute with excellence. This is an opportunity to do career-defining work. We're all in on this mission. If you are too, let's talk. Here is the updated summary for the Okta Device Access (ODA) and Okta Verify (OV) space, adapted to match the collaborative, ownership-focused, and candidate-centric tone of your PAM example: In this role, you will work as a trusted cross-functional partner on our Okta Device Access (ODA) and Okta Verify (OV) products, and have the opportunity to influence product and design decisions through your work. You will own the UX roadmap for ODA and OV, working closely with your design, product, and engineering partners to ensure that your work is aligned to the highest priority needs of the endpoint identity area, informing both long and near-term decisions. You’ll provide mentorship and coaching to more junior team members, helping raise the quality bar and impact of the team. ODA and OV’s goal is to develop products that enable Okta customers to secure their workforce right from physical device bootup, providing frictionless, phishing-resistant, and passwordless authentication. In this role, you’ll get to: Grow and develop a research practice aimed at understanding the needs of the OPA and OV teams, collaborate with stakeholders to craft research questions and execute research Scope, prioritize, organize, and manage research initiatives across these teams. Exercise flexibility in research methodology (as appropriate for the product space and timeline) by designing and utilizing generative studies or evaluative research, appropriately. Facilitate cross-dis

Software is eating the world, but AI is eating software. We live in unprecedented times – AI has the potential to exponentially augment human intelligence. Every person will have a personal tutor, coach, assistant, personal shopper, travel guide, and therapist throughout life. As the world adjusts to this new reality, leading platform companies are scrambling to build LLMs at billion scale, while large enterprises figure out how to add it to their products. To make them safe, aligned and actually useful, these models need human eval and reinforcement learning through human feedback (RLHF) during pre-training, fine-tuning, and production evaluations. This is the main innovation that’s enabled ChatGPT to get such a large headstart among competition. At Scale, our products include the Generative AI Data Engine, SGP, Donovan, and others that power the most advanced LLMs and generative models in the world through world-class RLHF, human data generation, model evaluation, safety, and alignment. The data we are producing is some of the most important work for how humanity will interact with AI. At the foundation of these products is the Platform Engineering team. In this role, you will support the design and development of shared platforms used across Scale. This includes designing our foundational data platforms and lifecycle, architecting Scale’s core cloud infrastructure and orchestration stack, and redefining how engineers develop, build, test, and deploy software at Scale. You’ll also get widespread exposure to the forefront of the AI race as Scale sees it in enterprises, startups, governments, and large tech companies. You will: Drive the design, and implementation of our foundational platforms and systems, working closely with stakeholders and internal customers to understand and refine requirements. Collaborating with cross-functional teams to define, design, and deliver new features. Proactively identifying opportunities for, and driving improvements to, current p

Software is eating the world, but AI is eating software. We live in unprecedented times – AI has the potential to exponentially augment human intelligence. Every person will have a personal tutor, coach, assistant, personal shopper, travel guide, and therapist throughout life. As the world adjusts to this new reality, leading platform companies are scrambling to build LLMs at billion scale, while large enterprises figure out how to add it to their products. To make them safe, aligned and actually useful, these models need human eval and reinforcement learning through human feedback (RLHF) during pre-training, fine-tuning, and production evaluations. This is the main innovation that’s enabled ChatGPT to get such a large headstart among competition. At Scale, our products include the Generative AI Data Engine, SGP, Donovan, and others that power the most advanced LLMs and generative models in the world through world-class RLHF, human data generation, model evaluation, safety, and alignment. The data we are producing is some of the most important work for how humanity will interact with AI. At the foundation of these products is the Identity Engineering team. In this role, you will help support the design and development of core software systems specifically focused on identity, access management, authorization, and authentication. You’ll also get widespread exposure to the forefront of the AI race as Scale sees it in enterprises, startups, governments, and large tech companies. You will: Drive the design, and implementation of our identity infrastructure to ensure secure authentication and authorization across enterprise systems. Build software for authentication mechanisms such as Single Sign-On (SSO), Multi-Factor Authentication (MFA), and federated identity solutions (SAML, OAuth, OpenID Connect). Build software for authorization mechanisms such as Relation-based access control (ReBAC), Attribute-based access control (ABAC), Role-based access cont

Software is eating the world, but AI is eating software. We live in unprecedented times – AI has the potential to exponentially augment human intelligence. Every person will have a personal tutor, coach, assistant, personal shopper, travel guide, and therapist throughout life. As the world adjusts to this new reality, leading platform companies are scrambling to build LLMs at billion scale, while large enterprises figure out how to add it to their products. To make them safe, aligned and actually useful, these models need human eval and reinforcement learning through human feedback (RLHF) during pre-training, fine-tuning, and production evaluations. This is the main innovation that’s enabled ChatGPT to get such a large headstart among competition. At Scale, our products include the Generative AI Data Engine, SGP, Donovan, and others that power the most advanced LLMs and generative models in the world through world-class RLHF, human data generation, model evaluation, safety, and alignment. The data we are producing is some of the most important work for how humanity will interact with AI. At the foundation of these products is the Identity Engineering team. In this role, you will help support the design and development of core software systems specifically focused on identity, access management, authorization, and authentication. You’ll also get widespread exposure to the forefront of the AI race as Scale sees it in enterprises, startups, governments, and large tech companies. You will: Drive the design, and implementation of our identity infrastructure to ensure secure authentication and authorization across enterprise systems. Build software for authentication mechanisms such as Single Sign-On (SSO), Multi-Factor Authentication (MFA), and federated identity solutions (SAML, OAuth, OpenID Connect). Build software for authorization mechanisms such as Relation-based access control (ReBAC), Attribute-based access control (ABAC), Role-based access cont

Software is eating the world, but AI is eating software. We live in unprecedented times – AI has the potential to exponentially augment human intelligence. Every person will have a personal tutor, coach, assistant, personal shopper, travel guide, and therapist throughout life. As the world adjusts to this new reality, leading platform companies are scrambling to build LLMs at billion scale, while large enterprises figure out how to add it to their products. To make them safe, aligned and actually useful, these models need human eval and reinforcement learning through human feedback (RLHF) during pre-training, fine-tuning, and production evaluations. This is the main innovation that’s enabled ChatGPT to get such a large headstart among competition. At Scale, our products include the Generative AI Data Engine, SGP, Donovan, and others that power the most advanced LLMs and generative models in the world through world-class RLHF, human data generation, model evaluation, safety, and alignment. The data we are producing is some of the most important work for how humanity will interact with AI. At the foundation of these products is the Platform Engineering team. In this role, you will support the design and development of shared platforms used across Scale. This includes designing our foundational data platforms and lifecycle, architecting Scale’s core cloud infrastructure and orchestration stack, and redefining how engineers develop, build, test, and deploy software at Scale. You’ll also get widespread exposure to the forefront of the AI race as Scale sees it in enterprises, startups, governments, and large tech companies. You will: Drive the design, and implementation of our foundational platforms and systems, working closely with stakeholders and internal customers to understand and refine requirements. Collaborating with cross-functional teams to define, design, and deliver new features. Proactively identifying opportunities for, and driving improvements to, current p

About the Team: Tubi's Internal Tools team is at the forefront of AI integration, developing everything from developer resources to production-grade AI for business operations. We are the group responsible for turning AI from an experiment into an operating capability: training, infrastructure, developer agents, and AI-powered business systems. Engineers operate with high ownership and autonomy, collaborating on shared architectural decisions and AI infrastructure. What You'll Do: Own systems end to end — design them, build them, and support them in production. Lead the projects you own: sequence the work, decide what lands first, and set technical direction for the engineers working with you. Sit with the people who use what you build, and turn what you learn there into a system. Design the service boundaries, contracts and schema evolution that let our platforms grow without breaking the teams depending on them. Make our AI systems dependable in production: evaluation harnesses, human approval steps before an agent acts, retries that handle a model returning something unexpected, and cost tracking that tells you what a task costs before you run it. Build what other engineers build on — agent skills, tool and MCP integrations, shared libraries — and raise the bar through code review, design discussion and mentoring. Spot the platform work nobody has asked for yet, make the case for it, and build it. Your Background: 5+ years of professional experience building and operating production systems, from design through production ownership. A system you designed and can walk us through end to end — where its boundaries sit, what constrained it, and what you chose against. Strong programming proficiency in a statically typed language such as Rust, Go, C++, Java, Kotlin, C#, or TypeScript. Production Rust is a plus rather than a requirement. You have owned a service in production: you wrote the runbooks, you knew what it cost, and you were the one paged when it broke. Expe

About Scale AI At Scale, our mission is to develop reliable AI systems for the world's most important decisions. Our products provide the high-quality data and full-stack technologies that power the world's leading models, and help enterprises and governments build, deploy, and oversee AI applications that deliver real impact. Scale Frontier Data is the organization behind the training and evaluation data that frontier labs depend on. We build the systems, tooling, and expert workflows that turn hard human expertise into signals that models can learn from, across reasoning, coding, agentic tool use, and domain expertise. Reinforcement learning environments are now the center of gravity for that work: the difference between a model that demos well and a model that reliably completes long-horizon work is almost always the quality of the environments and reward signals it was trained against. Responsibilities As a Staff Software Engineer, RL Environments, you'll own the technical foundation for how Scale builds, runs, verifies, and delivers RL environments at scale. An RL environment is a real piece of software: a containerized world with real dependencies, real state, real tools, and a grader that has to be correct even when the agent is creative about breaking it. Building one is a full-stack engineering problem. Building thousands of them reproducibly, cheaply, with trustworthy reward signals and throughput measured in millions of rollouts is a systems problem that very few people have solved. You'll work on both. You'll design the platform: sandboxed execution, environment packaging and versioning, rollout orchestration, trajectory capture, verifier frameworks, and the authoring surfaces that let engineers and domain experts produce environments without reinventing infrastructure each time. And you'll go deep on the environments themselves by instrumenting real applications, designing task suites that expose specific capability gaps, and building graders that

About Scale AI At Scale AI, our mission is to accelerate the development of AI applications. For 8 years, Scale has been the leading AI data foundry, helping fuel the most exciting advancements in AI, including generative AI, defense applications, and autonomous vehicles. With our recent Series F round, we’re accelerating the abundance of frontier data to pave the road to Artificial General Intelligence (AGI) and building upon our prior model evaluation work with enterprise customers and governments to deepen our capabilities and offerings for public and private evaluations. About Data Engine Our Generative AI Data Engine powers the world’s most advanced LLMs and generative models through world-class RLHF (Reinforcement Learning with Human Feedback), human data generation, model evaluation, safety, and alignment. The data we produce is some of the most critical work for how humanity will interact with AI. About Our FDE Team Generating high-quality data is the core problem our business solves. We aim to make producing and delivering high-quality data seamless and efficient for operators and customers. Our Team is building customer and operator-specific infrastructure to provide high-quality data with low turnaround time. You'll be exposed to the cutting edge of the Generative AI industry while directly interfacing with the leading model-building organizations in the space, including the top AI research labs and government agencies. Join us in shaping the future of Artificial General Intelligence. As a Forward Deployed Engineer, you'll be at the forefront of providing the critical data infrastructure that powers the most advanced AI models, directly influencing how humanity interacts with AI. You will work with the world’s leading AI companies and government agencies to solve their most complex AI data-related problems. Responsibilities: Drive Impact: Directly contribute to the advancement of AI by delivering critical data solutions for leading AI innovators and