Human Evaluator in United States

Become a part of our caring community Most AI engineering jobs are a thin wrapper around a model API. This role is different. We build the platform that transforms millions of clinical documents into trusted, actionable data. Our systems use large language models (LLMs) to read medical records, extract structured facts, answer complex questions with citations back to the source document, and route ambiguous cases to human experts for review. Our users make decisions that impact real healthcare outcomes, so “good enough” is not good enough. Building AI systems that are accurate, reliable, auditable, and scalable is at the core of this role. As a Senior AI Applied Engineer, you will design, build, deploy, and operate production AI systems used at scale within one of the largest health insurers in the United States. You will own solutions end-to-end, from user experience and APIs to model orchestration, evaluation frameworks, infrastructure, and production operations. Why Join Us Build production AI systems where LLMs are in the critical path, not just demos or proofs of concept. Work on extraction, retrieval, agentic workflows, and human-review systems that process real healthcare data at scale. Own projects end-to-end across frontend, backend, AI orchestration, infrastructure, deployment, and operations. Solve challenging problems around accuracy, explainability, traceability, and reliability in regulated environments. Ship quickly in a small, high-impact team that embraces AI-assisted development and rigorous quality standards. Build systems that continuously improve through expert feedback, evaluations, and human-in-the-loop workflows. Key Responsibilities Design, develop, and deploy full-stack AI-powered application

Become a part of our caring community Every large organization is making critical decisions today about how it will leverage AI over the next decade. Few have leaders who can both define that vision and demonstrate its viability through hands-on engineering. This role requires both. We build the platform that transforms millions of clinical documents into trusted, actionable data. Our systems use large language models (LLMs) to read medical records, extract structured facts, answer complex questions with citations to source documents, and route difficult cases to human experts. These capabilities support decisions that impact real healthcare outcomes for members. As a Principal AI Applied Engineer, you will define the technical strategy, architectural standards, and long-term vision for AI-enabled products across the organization. You will influence enterprise-wide decisions regarding AI platforms, model strategies, engineering standards, and technology investments while remaining deeply hands-on in prototyping, experimentation, architecture, and software development. This is the highest-level individual contributor role within the AI Applied Engineering organization. Success requires exceptional technical depth, organizational influence, strategic thinking, and the ability to translate emerging AI capabilities into scalable, reliable, and responsible production systems. Why Join Us Shape the long-term AI architecture and engineering direction for a large enterprise healthcare organization. Influence how AI-enabled products are designed, built, evaluated, deployed, and governed across multiple teams. Drive strategic decisions involving models, vendors, platforms, infrastructure, and shared capabilities. Prototype and validate emerging technologies before the organization invests at scale.</

Every day, tens of millions of people come to Roblox to explore, create, play, learn, and connect with friends in 3D immersive digital experiences– all created by our global community of developers and creators. At Roblox, we’re building the tools and platform that empower our community to bring any experience that they can imagine to life. Our vision is to reimagine the way people come together, from anywhere in the world, and on any device. We’re on a mission to connect a billion people with optimism and civility, and looking for amazing talent to help us get there. A career at Roblox means you’ll be working to shape the future of human interaction, solving unique technical challenges at scale, and helping to create safer, more civil shared experiences for everyone. As a Senior Software Engineer within the Creator Organization, you will develop innovative full-stack solutions that define the future of Roblox’s Content Understanding Platform. This platform processes billions of pieces of content—spanning 3D models, audio files, text, video, and entire experiences—extracting structured information about their meaning, context, and relationships. The Content Understanding Team develops cutting-edge AI models, advanced computer vision systems, and highly scalable backend platforms to power search, discovery, and moderation across Roblox. Your contributions will enable seamless asset discovery, automate moderation at scale, and drive transformative generative AI tools that reshape how millions of creators and users engage with Roblox. You Will: Solve full-stack challenges to improve how AI, creators, and users describe and get along with content, including images, 3D models, audio, text, and video. Craft and build scalable pipelines for training, evaluating, and deploying machine learning models to support content annotation and discovery. Develop robust backend systems to power real-time search, discovery, and powerful generative AI features. Blend innovat

At ClickUp, we're building the future of work: the first truly converged AI workspace unifying tasks, docs, chat, calendar, and enterprise search, all supercharged by context-driven AI. We are an AI-native company. Every team member is expected to leverage AI daily, and we evaluate AI fluency as part of our hiring process. Join us and help redefine what's possible. 🚀 We’re hiring a highly influential product analytics leader who can turn ambiguous questions into sharp insight, scalable measurement, and recommendations that directly shape what we build. This person will partner closely with Product, Engineering, Design, and Growth to raise the bar on decision quality and establish a more AI-native analytics operating model. Mission Drive the product insight agenda by helping ClickUp make faster, smarter product decisions through rigorous analysis, strong product judgment, and AI-enabled analytics workflows. What You'll Do Own the product analytics agenda across product usage, activation, feature adoption, retention, and expansion, and translate open-ended business questions into structured analyses and clear recommendations Partner with Product, Engineering, and Design to define success metrics early, improve instrumentation quality, and ensure important product surfaces are measurable from launch Build reusable analysis frameworks, semantic layers, metric definitions, and self-serve resources that help product teams answer routine questions faster and more consistently Apply AI-first methods across the analytics workflow, using large language models, coding agents, and automation for tasks like query drafting, QA, validation, documentation, and first-pass synthesis while keeping human judgment at the center of final recommendations Design and interpret experiments, observational analyses, and trend investigations, including situations where data is incomplete or traditional experimentation is not feasible Surface meaningful patterns in behavioral, subscription, and

Every day, tens of millions of people come to Roblox to explore, create, play, learn, and connect with friends in 3D immersive digital experiences– all created by our global community of developers and creators. At Roblox, we’re building the tools and platform that empower our community to bring any experience that they can imagine to life. Our vision is to reimagine the way people come together, from anywhere in the world, and on any device. We’re on a mission to connect a billion people with optimism and civility, and looking for amazing talent to help us get there. A career at Roblox means you’ll be working to shape the future of human interaction, solving unique technical challenges at scale, and helping to create safer, more civil shared experiences for everyone. As a Senior Enterprise Security Engineer, you will play a critical role in executing Roblox’s Enterprise Security Strategy. You will design, deploy, and manage security solutions to protect Roblox’s corporate infrastructure and ensure secure, compliant operations across the organization. Working closely with Corporate Engineering and Trust & Safety teams, you will translate business requirements into robust security implementations that enable secure productivity while mitigating risk. You will be reporting directly to the Senior Manager of Enterprise Security Engineering. You'll partner with security professionals across the Information Security organization, and work cross-functionally with teams throughout Roblox to drive security initiatives that scale with our business. You will: Evaluate and implement security technologies and vendor solutions to ensure alignment with enterprise security requirements, compliance standards, and overall risk management strategy Lead and drive initiatives across core security domains, including Endpoint Security, SaaS Security, Identity & Access Management (IAM), Agentic AI Governance, and Supply Chain Security. Collaborate closely with IT, engin

Every day, tens of millions of people come to Roblox to explore, create, play, learn, and connect with friends in 3D immersive digital experiences– all created by our global community of developers and creators. At Roblox, we’re building the tools and platform that empower our community to bring any experience that they can imagine to life. Our vision is to reimagine the way people come together, from anywhere in the world, and on any device. We’re on a mission to connect a billion people with optimism and civility, and looking for amazing talent to help us get there. A career at Roblox means you’ll be working to shape the future of human interaction, solving unique technical challenges at scale, and helping to create safer, more civil shared experiences for everyone. As a Principal Security Software Engineer in the Enterprise Security team, you will advance Roblox's Enterprise Security strategy by building the systems and integrations that protect Roblox's corporate infrastructure. Where traditional security engineers evaluate and deploy vendor solutions, you will design and build production-grade security software - Identity and Access governance, policy enforcement engines, and security integrations that scale with Roblox's business. You'll partner with security professionals across InfoSec and work cross-functionally with Corporate Engineering, DevOps, and Product teams to drive security initiatives. You will: Build identity and access systems : Design, implement, and own integrations across Roblox's IAM ecosystem, including SSO federation, SCIM provisioning/deprovisioning pipelines, OAuth 2.0 authorization servers, and token lifecycle management. Lead security automation : Develop production-quality tools and services that enforce security policies at scale, replace manual workflows, and surface actionable signals Drive secure-by-design implementations : Partner with Corporate Engineering, DevOps, and product teams to embed security controls in

Every day, tens of millions of people come to Roblox to explore, create, play, learn, and connect with friends in 3D immersive digital experiences– all created by our global community of developers and creators. At Roblox, we’re building the tools and platform that empower our community to bring any experience that they can imagine to life. Our vision is to reimagine the way people come together, from anywhere in the world, and on any device. We’re on a mission to connect a billion people with optimism and civility, and looking for amazing talent to help us get there. A career at Roblox means you’ll be working to shape the future of human interaction, solving unique technical challenges at scale, and helping to create safer, more civil shared experiences for everyone. As a member of the Infrastructure Foundation Hardware Engineering team, you will play a key role in enabling our mission to deliver a reliable, high-performing, and cost-efficient infrastructure that powers the world’s play. In this specialized role, you will be the technical lead for our GPU and AI accelerator ecosystem. You will be responsible for the full lifecycle of GPU hardware, from initial architectural evaluation and firmware qualification to large-scale fleet integration and performance tuning. You will ensure that Roblox’s massive-scale rendering and ML workloads run on the most optimized and stable hardware possible. You Will: Architect & Prototype: Prototype next-generation GPU-accelerated hardware platforms, ensuring seamless integration between high-density compute nodes, high-speed interconnects (NVLink/PCIe Gen5/6), and system firmware. GPU Optimization: Drive the integration, performance testing, and debugging of GPUs in our fleet, focusing specifically on hardware-level optimizations, driver tuning, and thermal/power management. Validation & Certification: Develop and execute rigorous evaluation and stress-testing strategies for GPU-heavy server platforms to ensur

Every day, tens of millions of people come to Roblox to explore, create, play, learn, and connect with friends in 3D immersive digital experiences– all created by our global community of developers and creators. At Roblox, we’re building the tools and platform that empower our community to bring any experience that they can imagine to life. Our vision is to reimagine the way people come together, from anywhere in the world, and on any device. We’re on a mission to connect a billion people with optimism and civility, and looking for amazing talent to help us get there. A career at Roblox means you’ll be working to shape the future of human interaction, solving unique technical challenges at scale, and helping to create safer, more civil shared experiences for everyone. As a member of the Infrastructure Foundation Hardware Engineering team, you will help develop and validate next-generation server platforms that power a reliable, high-performing, and cost-efficient infrastructure at scale. You will work across platform bring-up, firmware qualification, hardware validation, fleet integration, and performance optimization to support large-scale production deployments. You Will: Bring-up & Sustaining: Drive key aspects of the hardware development lifecycle, including feasibility studies, hardware bring-up, validation, deployment, and ongoing production support. Platform Optimization: Perform platform integration, performance characterization, and system-level debugging across compute infrastructure, focusing on hardware optimization, driver tuning, and thermal/power efficiency. Hardware Validation: Develop and execute rigorous evaluation and stress-testing strategies for server platforms to ensure reliability and performance under production-scale workloads. Firmware & Fleet Enablement: Support BIOS/BMC firmware qualification, hardware health monitoring, and automation tooling for firmware deployment and lifecycle management. Vendor & Cross-Functi

Every day, tens of millions of people come to Roblox to explore, create, play, learn, and connect with friends in 3D immersive digital experiences– all created by our global community of developers and creators. At Roblox, we’re building the tools and platform that empower our community to bring any experience that they can imagine to life. Our vision is to reimagine the way people come together, from anywhere in the world, and on any device. We’re on a mission to connect a billion people with optimism and civility, and looking for amazing talent to help us get there. A career at Roblox means you’ll be working to shape the future of human interaction, solving unique technical challenges at scale, and helping to create safer, more civil shared experiences for everyone. The Consumer App Foundation team is looking for a Senior Software Engineer to improve Roblox performance across our supported platforms. Our platform is used by over 150M people daily. You will work primarily in cross-platform C++ Roblox game engine code, partnering with teams across the company to identify bottlenecks, ship measurable improvements, and prevent regressions over time. Our team is responsible for the end-to-end performance of the Roblox client on Android, iOS, macOS, Windows, Xbox, Playstation and TV. We tackle the platform’s most difficult technical constraints, ranging from speed and responsiveness to efficiency and stability, to ensure a high-performance experience across all of our devices and platforms. You will: Own the end-to-end performance of the Roblox client across all platforms directly moving the needle on our key metrics: TTI, app size, battery usage and more. Build the benchmarking, profiling, and telemetry systems required to catch regressions early and continue scaling our application even further. Collaborate closely with stakeholders across Engineering, Product, and Design to evaluate technical tradeoffs, align on company-wide priorities, and

Every day, tens of millions of people come to Roblox to explore, create, play, learn, and connect with friends in 3D immersive digital experiences– all created by our global community of developers and creators. At Roblox, we’re building the tools and platform that empower our community to bring any experience that they can imagine to life. Our vision is to reimagine the way people come together, from anywhere in the world, and on any device. We’re on a mission to connect a billion people with optimism and civility, and looking for amazing talent to help us get there. A career at Roblox means you’ll be working to shape the future of human interaction, solving unique technical challenges at scale, and helping to create safer, more civil shared experiences for everyone. About the role: Roblox Studio is the creation engine behind millions of immersive 3D games built by creators around the world. We are entering a major platform transition, evolving Studio into an AI-Native IDE where intelligent systems plan, act, validate, and iterate alongside creators to amplify their output. We are seeking a Technical Director, Applied AI to own the technical direction of the agentic AI systems embedded deeply into Roblox Studio. This is a hands-on individual-contributor role operating at the intersection of platform engineering, AI systems, and developer experience, where you will set the technical direction, design the architecture, and write the code. You will: Design and build the agentic AI systems at the core of the AI-Native IDE (planning, execution, validation, evaluation) so they are production-grade and reliable. Decide how Studio uses coding models, agents, retrieval, tool calling, and context management across the Assistant and the broader creator workflow. Own the quality bar and evaluation strategy , standing up quantitative and qualitative eval pipelines, including human-in-the-loop, so we ship with confidence. Optimize these systems for latency, throughpu

Every day, tens of millions of people come to Roblox to explore, create, play, learn, and connect with friends in 3D immersive digital experiences– all created by our global community of developers and creators. At Roblox, we’re building the tools and platform that empower our community to bring any experience that they can imagine to life. Our vision is to reimagine the way people come together, from anywhere in the world, and on any device. We’re on a mission to connect a billion people with optimism and civility, and looking for amazing talent to help us get there. A career at Roblox means you’ll be working to shape the future of human interaction, solving unique technical challenges at scale, and helping to create safer, more civil shared experiences for everyone. The Infrastructure Compute Site Reliability Engineering mission is to own and manage the successful operation of our underlying cell infrastructure system, along with elements of service discovery, secrets management and related software layers. We’re looking for a skilled Senior Site Reliability Engineer with strong programming skills to help us build Roblox's private cloud, productionize our growing Kubernetes-based infrastructure, and institute reliability best practices across the Roblox Compute team. You will: Design and Develop systems & libraries that promote fault-tolerance and resilience, automate much of the management and lifecycle of our clusters, and ensure systems are observable. Promote and Institute reliability best practices across the Infra Compute group, drive common reliability initiatives. Provides collaborative technical reviews and operational guidance to strengthen system reliability. Build, Automate and Standardize process automation to create a "golden path" of tooling and platform support that powers the fundamental Roblox ecosystem. Create Tooling that provides production guardrails, by evaluating release candidate capacity with load testing tooling before de

Every day, tens of millions of people come to Roblox to explore, create, play, learn, and connect with friends in 3D immersive digital experiences– all created by our global community of developers and creators. At Roblox, we’re building the tools and platform that empower our community to bring any experience that they can imagine to life. Our vision is to reimagine the way people come together, from anywhere in the world, and on any device. We’re on a mission to connect a billion people with optimism and civility, and looking for amazing talent to help us get there. A career at Roblox means you’ll be working to shape the future of human interaction, solving unique technical challenges at scale, and helping to create safer, more civil shared experiences for everyone. As a Senior Privacy Engineer on the Security and Privacy GRC team, you will shepherd and grow Roblox's Privacy Review Program, the technical and process backbone that ensures privacy is engineered into every product before it ships. You will join the Security and Privacy GRC team, reporting to the Sr Engineering Manager for Privacy Governance and Operations. This is a hands-on privacy engineering role: you'll set standards for privacy-enhancing technologies, act as the go-to SME for policy-as-code, and partner closely with Product teams, Trust & Safety, Legal, and Regulatory Compliance as part of Roblox's broader cross-functional privacy program. You will: Shepherd and evolve the Privacy Review Program, designing consistent intake workflows, evaluation criteria, and documentation to systematically assess privacy risk across new products, features, and infrastructure changes. Develop standards and guidelines that enable product teams to adopt privacy-enhancing technologies (PETs) such as differential privacy, k-anonymity, data minimization, and secure computation, with clear guidance on trade-offs and implementation patterns. Act as the Privacy SME for policy-as-code, translating privacy

Every day, tens of millions of people come to Roblox to explore, create, play, learn, and connect with friends in 3D immersive digital experiences– all created by our global community of developers and creators. At Roblox, we’re building the tools and platform that empower our community to bring any experience that they can imagine to life. Our vision is to reimagine the way people come together, from anywhere in the world, and on any device. We’re on a mission to connect a billion people with optimism and civility, and looking for amazing talent to help us get there. A career at Roblox means you’ll be working to shape the future of human interaction, solving unique technical challenges at scale, and helping to create safer, more civil shared experiences for everyone. The Consumer App Foundation team is looking for a Senior Software Engineer to improve Roblox performance across our supported platforms. Our platform is used by over 150M people daily. You will work primarily in cross-platform C++ Roblox game engine code, partnering with teams across the company to identify bottlenecks, ship measurable improvements, and prevent regressions over time. Our team is responsible for the end-to-end performance of the Roblox client on Android, iOS, macOS, Windows, Xbox, Playstation and TV. We tackle the platform’s most difficult technical constraints, ranging from speed and responsiveness to efficiency and stability, to ensure a high-performance experience across all of our devices and platforms. You will: Own the end-to-end performance of the Roblox client across all platforms directly moving the needle on our key metrics: TTI, app size, battery usage and more. Build the benchmarking, profiling, and telemetry systems required to catch regressions early and continue scaling our application even further. Collaborate closely with stakeholders across Engineering, Product, and Design to evaluate technical tradeoffs, align on company-wide priorities, and

Every day, tens of millions of people come to Roblox to explore, create, play, learn, and connect with friends in 3D immersive digital experiences– all created by our global community of developers and creators. At Roblox, we’re building the tools and platform that empower our community to bring any experience that they can imagine to life. Our vision is to reimagine the way people come together, from anywhere in the world, and on any device. We’re on a mission to connect a billion people with optimism and civility, and looking for amazing talent to help us get there. A career at Roblox means you’ll be working to shape the future of human interaction, solving unique technical challenges at scale, and helping to create safer, more civil shared experiences for everyone. As a User Researcher on the Economy team, reporting directly to the Director of Design Strategy & Operations, you will generate actionable qualitative and quantitative insights that shape core economic systems, developer monetization workflows, and immersive brand experiences. Embedded directly on product teams, you will conduct generative and evaluative research centered on how creators build businesses, how brands interact on the platform, and how users participate in virtual economies. You will tackle high-priority features, solve complex behavioral puzzles, and help translate multi-faceted economic infrastructure into intuitive, trustworthy, and scalable experiences. Partnering closely with Product, Product Design, Engineering, Data Science, and Policy, you will ensure user needs directly inform platform decisions while driving mutual value across players, creators, and brands. You will: Execute end-to-end mixed-methods research projects to understand creator monetization, payouts, advertising, and commerce workflows, translating abstract user behaviors into clear product feature recommendations. Conduct foundational research to uncover key user pain points, mental models, and behav

About the Team The Strategic Initiatives & Operations team is in need of a Technical Program Manager (TPM) to streamline our processes, including full safety governance and integration of various safety research and mitigations into our ChatGPT, API, and any frontier models. This role is critical for driving safe deployment of our new models, synthesizing inputs from multiple stakeholders, ranging across research, product, engineering, legal and policy, and ensuring all the risks are effectively and properly monitored, mitigated or resolved. About the Role As a TPM, you will be responsible for critical tasks ranging from tracking safety research progress and risk tables to overseeing the quality of human data campaigns – acting as the connective tissue to enhance the deployment of OpenAI’s safety system. Additionally, you will create and execute a compute roadmap for your team to ensure that our top priorities are resourced while taking advantage of new opportunities to make key safety research discoveries. Your primary focus will be to ensure our models are qualified for safe deployment. This role is based in San Francisco, CA. We use a hybrid work model of 3 days in the office per week and offer relocation assistance to new employees. In this role, you will: Manage key risk areas and corresponding stakeholders. Keep track of a stack of existing and future mitigations for every major product and model deployment. Standardize the lifecycle of risk assessment, setting safety bars, consolidating inputs from multiple stakeholders across research, product, engineering, legal and policy, pre-launch safety reviews and post-launch followup. Manage pre-launch safety reviews. Share launch calendars and key safety practices and evaluations with our key parter (i.e. Microsoft). Develop comprehensive documentation for all the safety work, including metrics, evaluations, and progress tracking across multiple teams within OpenAI. Help with publishing and open sourcing safety