Human Evaluator in Bengaluru

About Bolna Bolna is a YC-backed voice AI orchestration platform built for the Indian market—powering multilingual, vernacular voice agents across Hindi, Hinglish, Tamil, and 10+ languages at sub-500ms latency across collections, recruitment, sales, and e-commerce use cases. We are an orchestration layer, not a model company: our moat is outcome-labelled vernacular data, rigorous evaluation infrastructure, and a growing taxonomy of how Indian enterprise voice AI fails in production. Why This Role Exists Product decisions at Bolna increasingly hinge on rigorous, code-mixed-aware data analysis—and not just one kind. On one side, there is model and evaluation rigor: LLM benchmarking for post-call intelligence, ASR/WER evaluation, inter-rater reliability on human-labelled calls, and routing and latency economics. On the other, there is product and growth insight: understanding where self-serve users drop off in their journey, what patterns emerge across lakhs of monthly calls, and which use cases and configurations are actually working. Both currently sit with the Head of Product alongside strategy and roadmap ownership. We need a dedicated analyst to own the execution and recurring cadence across both-freeing product leadership to act on findings rather than produce them. What You’ll Do Model and Evaluation Analysis LLM and model benchmarking: Run structured comparisons across model providers such as Sarvam, DeepSeek, Gemini, and Claude variants for tasks including post-call extraction and LLM-as-judge scoring. Evaluate cost, accuracy, fill rate, and TTR, with particular attention to Hinglish and code-mixed content. Evaluation infrastructure: Build and maintain LLM-as-judge pipelines using tools such as DeepEval, design and track evaluation metrics, and run inter-rater reliability analysis such as Krippendorff’s alpha across human call reviewers. Golden dataset creation: Support the construction of golden datasets for ASR and transcript labelling, including flagging co

Secure Every Identity, from AI to Human Identity is the key to unlocking the potential of AI. Okta secures AI by building the trusted, neutral infrastructure that enables organizations to safely embrace this new era. This work requires a relentless drive to solve complex challenges with real-world stakes. We are looking for builders and owners who operate with speed and urgency and execute with excellence. This is an opportunity to do career-defining work. We're all in on this mission. If you are too, let's talk. Okta is seeking a Staff Analyst – People Technology to join our Business Solutions Team. In this critical role, you will partner closely with the People Technology leadership and play a key role in our Workday transformation efforts. As part of a cross-functional team, you will collaborate with stakeholders across People, Payroll, IT, Finance, and international offices to optimize and scale our HR systems and processes. This role is ideal for someone who is passionate about solving complex business challenges, driving innovation through automation, and delivering a best-in-class user experience. Job Duties and Responsibilities Key Responsibilities Own the end-to-end architecture and technical design of Workday Extend applications, integrations, and supporting technology solutions. Define scalable, secure, resilient, and maintainable application architecture, integration architecture, data flows, API strategies, and technical standards . Architect and lead the development of complex Workday Extend applications using TypeScript/JavaScript, Workday APIs, and Extend frameworks. Architect enterprise Workday integrations using Workday Studio, EIB, Core Connectors, RaaS, REST APIs, SOAP Web Services, and Workday Web Services . Evaluate business requirements and determine the appropriate solution across Workday configuration, Extend, integrations, APIs, AI, cloud services, and third-party platforms . Establish architectural patterns and reusable frameworks fo

Secure Every Identity, from AI to Human Identity is the key to unlocking the potential of AI. Okta secures AI by building the trusted, neutral infrastructure that enables organizations to safely embrace this new era. This work requires a relentless drive to solve complex challenges with real-world stakes. We are looking for builders and owners who operate with speed and urgency and execute with excellence. This is an opportunity to do career-defining work. We're all in on this mission. If you are too, let's talk. About the Team The Professional Services R&D team is a new, dynamic group at the forefront of innovation within Okta. Our mission is to design and build reusable, scalable assets and tools that empower our delivery teams and partners. By making customer engagements more efficient, streamlined, and cost-effective, we directly contribute to our customers' success and accelerate their time-to-value with Okta. This is a unique opportunity to join a strategic team from the ground up and shape the future of Okta's professional services. Position Summary As the Architect for the Professional Services R&D team, you will be the principal technical leader responsible for the high-level design and technical strategy of our entire portfolio of reusable assets. You will define the architectural vision and standards that ensure our solutions are robust, scalable, and secure. This role requires a blend of deep technical expertise, strategic thinking, and strong collaboration skills to guide the engineering team in building a cohesive ecosystem of tools on technologies like Java, React, Node, Python, and .NET. Responsibilities Define and own the end-to-end technical architecture for all reusable assets and solutions developed by the R&D team. Establish architectural standards, patterns, and best practices to guide the development team. Evaluate and select appropriate technologies, frameworks, and platforms to meet project requirements and long-term business goa

Secure Every Identity, from AI to Human Identity is the key to unlocking the potential of AI. Okta secures AI by building the trusted, neutral infrastructure that enables organizations to safely embrace this new era. This work requires a relentless drive to solve complex challenges with real-world stakes. We are looking for builders and owners who operate with speed and urgency and execute with excellence. This is an opportunity to do career-defining work. We're all in on this mission. If you are too, let's talk. About Okta: Okta is the leading independent provider of identity solutions for enterprises. With a deep commitment to innovation and customer success, we power and secure digital transformation for the world's biggest brands. We're scaling rapidly, and we’re seeking talented individuals to join us in enabling secure and seamless digital experiences. One of the most important components of our operations is an efficient financial system, including seamless processes for the Record-to-Report (R2R) cycle. As a NetSuite R2R Analyst, your role will support and strengthen Okta's financial systems and processes to ensure accuracy, efficiency, and scalability. This is a potential Contract to Hire position. Roles & Responsibilities: Act as the subject matter expert (SME) for NetSuite functionality in Record-to-Report (R2R) processes, maintaining workflows, configurations, and data integrity to enhance efficiency and compliance (e.g., SOX, GAAP). Lead or participate in R2R-related projects, including implementations, migrations, process improvements, customizations, upgrades, and integrations with NetSuite’s financial modules. Support month-end, quarter-end, and year-end close processes by ensuring timely reconciliations, reporting, general ledger accuracy, consolidations, and inter company accounting. Partner with finance, IT, and external vendors to provide NetSuite support, resolve issues, streamline processes, evaluate automation opportunities, and deliver us

Secure Every Identity, from AI to Human Identity is the key to unlocking the potential of AI. Okta secures AI by building the trusted, neutral infrastructure that enables organizations to safely embrace this new era. This work requires a relentless drive to solve complex challenges with real-world stakes. We are looking for builders and owners who operate with speed and urgency and execute with excellence. This is an opportunity to do career-defining work. We're all in on this mission. If you are too, let's talk. Secure Every Identity, from AI to Human Identity is the key to unlocking the potential of AI. Okta secures AI by building the trusted, neutral infrastructure that enables organizations to safely embrace this new era. This work requires a relentless drive to solve complex challenges with real-world stakes. We are looking for builders and owners who operate with speed and urgency and execute with excellence. This is an opportunity to do career-defining work. We're all in on this mission. If you are too, let's talk. The Digital Platform Team The Digital Platform team is a global organization distributed across the US and India, responsible for the integration and infrastructure layer that connects Adobe Experience Cloud to the rest of Okta's marketing technology, and analytics stack. The team owns Adobe Experience Platform (AEP) architecture, data collection infrastructure, AEM platform services, and the integration portfolio — including the MCP server architecture, LLM provider integrations, and AI agent infrastructure that power AI-native marketing experiences across the company. The team partners closely with Marketing, IT, DevSecOps, and Adobe to extend platform capabilities, govern data, and deliver scalable AI-driven experiences to the GTM organization. The work is strategic and product-driven: anticipating capability gaps, evaluating new tools and integrations, and building the connective tissue that keeps the broader ecosystem unified and well-governed

For over 20 years, Smartsheet has empowered teams to manage work seamlessly and scale solutions smarter. Now, in our most ambitious chapter yet, we are uniting human teams with AI agents. By orchestrating the work agents do best, automating manual tasks and uncovering insights at scale, we create the space for people to focus on what truly matters: judgment, creativity, and big thinking. That is magic at work, and it’s what we show up for every day. Smartsheet is looking for a Sr. Principal Data Scientist to set the technical direction for the ML models and AI sub-agents that drive growth, monetization, efficiency, and retention across the customer lifecycle. You’ll architect the systems, define the modeling and evaluation standards, and shape the roadmap for how Smartsheet uses applied ML and agentic AI to serve millions of customers. The data is unusually rich: petabyte-scale execution data spanning two decades of how real work gets done. You are a recognized technical leader who moves fluidly between architecture, prototype, and production; raises the bar for the data scientists around you; and partners with senior product and engineering leaders to make decisions that compound across the org. You will work primarily with Product and Engineering and will be a part of Smartsheet’s Business Intelligence team. This full-time position reports to the Director of Data Science and is based in Smartsheet’s Bengaluru, India office. You Will: Set the technical direction for how Smartsheet uses applied ML and agentic AI across the customer lifecycle — what gets built, what doesn’t, and what good looks like Architect the systems behind the sub-agents: how they ground themselves in evidence, how risk and confidence are calibrated, how decisions are evaluated, and how all of it stays safe and trustworthy at scale Define the modeling, evaluation, and experimentation standards the team follows — from offline metrics to online rollout to production monitoring Personally ship the

At Snowflake, we are powering the era of the agentic enterprise. To usher in this new era, we seek AI-native thinkers across every function who are energized by the opportunity to reinvent how they work. You don’t just use tools; you possess an innate curiosity, treating AI as a high-trust collaborator that is core to how you solve problems and accelerate your impact. We look for low-ego individuals who thrive in dynamic and fast-moving environments and move with an experimental mindset — who rapidly test emerging capabilities to discover simpler, more powerful ways to deliver results. At Snowflake, your role isn't just to execute a function, but to help redefine the future of how work gets done. We are hiring a Staff Software Engineer for our Frontier Security AI team. Snowflake's Frontier Security AI teams develop production-grade LLM applications, intelligent agents, AI infrastructure, and evaluation systems for enterprise customers — products that must meet a high bar for quality, security, reliability, and efficiency while operating over sensitive data at large scale. In this role, you will lead the design and development of our Agentic Harness and agent evaluation platform, working across product, infrastructure, applied AI, security, and modeling teams to take new capabilities from prototype to dependable customer value. AS A STAFF SOFTWARE ENGINEER AT SNOWFLAKE, YOU WILL: Architect and build the Agentic Harness that executes complex, multi-step AI workflows across models, tools, data, and services. Design stable interfaces for tool execution, context construction, state management, memory, permissions, retries, fallbacks, and human review. Own agent quality end to end by building evaluation harnesses, representative datasets, automated graders, experiment pipelines, and release gates. Convert ambiguous reports such as "the agent feels worse" into measurable failure modes, reproducible tests, and durable fixes. Analyze production agent trajectories to identif

For over 20 years, Smartsheet has empowered teams to manage work seamlessly and scale solutions smarter. Now, in our most ambitious chapter yet, we are uniting human teams with AI agents. By orchestrating the work agents do best, automating manual tasks and uncovering insights at scale, we create the space for people to focus on what truly matters: judgment, creativity, and big thinking. That is magic at work, and it’s what we show up for every day. You Will: Data Architecture and Design: Designing and overseeing the architecture of scalable and reliable data platforms, including data pipelines, storage solutions, and processing systems Data Modelling and Management:Developing and implementing data models, ensuring data quality, and establishing data governance policies Data Pipeline Development: Building and optimising data pipelines for ingesting, processing, and transforming large datasets from various sources Performance Optimisation: Identifying and resolving performance bottlenecks in data pipelines and systems, ensuring efficient data retrieval and processing Technology Evaluation and Innovation: Staying abreast of emerging data technologies and exploring opportunities for innovation to improve the organisation’s data infrastructure Troubleshooting and Problem Solving: Diagnosing and resolving complex data-related issues, ensuring the stability and reliability of the data platform Data Security and Compliance: Implementing data security measures, ensuring compliance with data governance policies, and protecting sensitive data Perform other duties as assigned You Have: Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field. 10+ years of experience in data engineering or a similar role. Enterprise SaaS software solutions with high availability and scalability Solution handling large scale structured and unstructured data from varied data sources Experience in building and maintaining data platform systems such as distributed compute,

For over 20 years, Smartsheet has empowered teams to manage work seamlessly and scale solutions smarter. Now, in our most ambitious chapter yet, we are uniting human teams with AI agents. By orchestrating the work agents do best, automating manual tasks and uncovering insights at scale, we create the space for people to focus on what truly matters: judgment, creativity, and big thinking. That is magic at work, and it’s what we show up for every day. Smartsheet is looking for a highly technical Sr. Manager to lead our India-based Service Desk and IT Automation operations. This is a working-manager role: you'll personally execute technical work — scripting, automation builds, escalated ticket resolution — while also leading a team. If you're looking for a purely strategic or delegation-only leadership role, this isn't it. You'll manage a team spanning Service Desk, Tier 1 SOC support, and IT Automation Engineering, driving the technical maturity of our global support infrastructure through automation, scripting, deep systems integration, and increasingly, AI-augmented workflows. This role reports to the Director of End User Systems, with close, day-to-day partnership across US-based Desktop Support managers. For the right candidate, this role has a clear path to Director as we continue scaling our India GCC — with growth into that level anticipated within 12–18 months. This is also a strong fit for a Director-level leader currently at a SaaS startup who's looking to move into a larger, enterprise SaaS organization — and who is open to a step down in title now in exchange for a step up in scale, platform, and long-term trajectory. You Will IT Automation & Architecture Drive the IT automation function, identifying repetitive Tier 1 tasks and owning their automation end-to-end — from scoping through build, testing, and measurement Evaluate and build AI-augmented automation where it meaningfully reduces toil (e.g. intelligent ticket triage, SOC alert summarisation, AI

For over 20 years, Smartsheet has empowered teams to manage work seamlessly and scale solutions smarter. Now, in our most ambitious chapter yet, we are uniting human teams with AI agents. By orchestrating the work agents do best, automating manual tasks and uncovering insights at scale, we create the space for people to focus on what truly matters: judgment, creativity, and big thinking. That is magic at work, and it’s what we show up for every day. We are seeking a seasoned and strategic Senior Product Manager II to join our Product Organization. You will be instrumental in developing new capabilities that address the advanced project management needs of our most valuable and strategic customers, driving meaningful change in their organizations. You will be focused on delivering a completely new Gantt View experience. You will work closely with a dedicated engineering team based in Bangalore and report to the US-based Group Product Manager for Core Views experiences. You Will: Product Strategy & Vision: Define, prioritize, and drive the future roadmap for project management capabilities, ensuring alignment with customer needs, company objectives, and industry best practices. Feature Delivery: Own the end-to-end delivery of a new, scalable Gantt View experience, ensuring a compelling, high-quality customer experience that sets a new bar for Smartsheet. Cross-Functional Partnership: Work in close partnership with design, engineering, user research, data science, and analytics teams to create, test, and launch proposed new capabilities and validate hypotheses. Data-Driven Decision Making: Leverage internal and external data to guide experimentation, make informed product decisions, and evaluate the success of launches. Executive Alignment: Regularly present product strategy, concepts, health metrics, and roadmap updates to senior leadership to educate, influence, and align stakeholders across the company. Product Leadership: Operate effectively at both strategic a

For over 20 years, Smartsheet has empowered teams to manage work seamlessly and scale solutions smarter. Now, in our most ambitious chapter yet, we are uniting human teams with AI agents. By orchestrating the work agents do best, automating manual tasks and uncovering insights at scale, we create the space for people to focus on what truly matters: judgment, creativity, and big thinking. That is magic at work, and it’s what we show up for every day. Smartsheet is looking for a highly technical Sr. Manager to lead our India-based Service Desk and IT Automation operations. This is a hands-on, working manager role — you will be personally executing technical work (scripting, automation builds, escalated ticket resolution) and leading a team, not just overseeing one. If you're looking for a purely strategic or delegation-only leadership role, this isn't it. You will manage a team spanning Service Desk, Tier 1 SOC support, and IT Automation Engineering — driving the technical maturity of our global support infrastructure through automation, scripting, deep systems integration, and increasingly, AI-augmented workflows (e.g. intelligent ticket triage, SOC alert summarisation, AI-assisted knowledge base drafting). This role reports directly to the Director of End User Systems , with close, cooperative partnership with US-based Desktop Support managers. You Will IT Automation & Architecture Support the IT automation function, driving initiatives that reduce repetitive work and scale operations efficiently Identify repetitive Tier 1 tasks and drive end-to-end automation from identification through build, testing, and measurement Evaluate and build AI-augmented automation where it reduces toil — e.g. intelligent ticket triage, SOC alert summarisation, AI-assisted knowledge base drafting Mentor team on automation best practices and review technical designs before deployment Team Leadership & Execution Manage India-based Service Desk team with daily coordination with US

Job Details: Job Description: Intel's Design Quality and Reliability organization is seeking an AI Platform Engineer to architect and build an enterprise-grade AI platform for mission-critical engineering work. This platform will enable Intel engineers to analyze complex design, qualification, and reliability data; automate engineering workflows; access organizational knowledge; and make faster, evidence-based decisions throughout the product lifecycle. The successful candidate will combine strong software engineering fundamentals with expertise in AI-native and agentic development. They will be highly proficient with Agentic AI coding assistants and able to use these tools responsibly to accelerate architecture, implementation, testing, debugging, and documentation. This role requires close collaboration with Design, Quality and Reliability, Product Engineering, Manufacturing, IT, Information Security, and other Intel stakeholders. Responsibilities 1. Architect and develop Intel's reusable AI platform for Design Quality and Reliability. 2. Build AI agents and workflows for engineering data analysis, qualification planning, risk assessment, knowledge retrieval, reporting, and process automation. 3. Apply Agentic AI coding assistants to accelerate software development while maintaining rigorous engineering review and validation. 4. Integrate AI capabilities with Intel engineering databases, quality-management systems, internal APIs, spreadsheets, documentation repositories, and workflow tools. 5. Develop production-grade backend services, APIs, data pipelines, model gateways, and agent-orchestration components. 6. Establish shared platform capabilities for identity, access control, tool authorization, memory, observability, evaluation, and auditability. 7. Implement human approval, deterministic validation, and rollback controls for consequential engineering actions. 8.

About Bazaarvoice At Bazaarvoice, we create smart shopping experiences. Through our expansive global network, product-passionate community & enterprise technology, we connect thousands of brands and retailers with billions of consumers. Our solutions enable brands to connect with consumers and collect valuable user-generated content, at an unprecedented scale. This content achieves global reach by leveraging our extensive and ever-expanding retail, social & search syndication network. And we make it easy for brands & retailers to gain valuable business insights from real-time consumer feedback with intuitive tools and dashboards. The result is smarter shopping: loyal customers, increased sales, and improved products. The problem we are trying to solve : Brands and retailers struggle to make real connections with consumers. It's a challenge to deliver trustworthy and inspiring content in the moments that matter most during the discovery and purchase cycle. The result? Time and money spent on content that doesn't attract new consumers, convert them, or earn their long-term loyalty. Our brand promise : closing the gap between brands and consumers. Founded in 2005, Bazaarvoice is headquartered in Austin, Texas with offices in North America, Europe, Asia and Australia. It’s official: Bazaarvoice is a Great Place to Work in the US , Australia, India, Lithuania, France, Germany and the UK! The Human Resources function is responsible for providing human resources (HR) support to a major business unit or functional area. This includes any issues relating to talent acquisition, compensation and benefits, employee relations, training and development, and HR operations. The Analytics, Modeling, & Reporting area is responsible for analyzing and reporting on data sets for a particular area to evaluate, recommend and support the implementation of business strategies and operational processes. The Human Resources Information System (HRIS) and Reporting focus specializes

About Bolna Bolna is Voice AI infrastructure built for India - and now for the world. We help businesses deploy intelligent voice agents that can call, converse, and convert in any language, at scale. From collections to customer support to sales, our agents handle millions of conversations so humans don’t have to. We’re a YC F25 company, backed by General Catalyst, with 1,050+ paying customers and growing fast. Our team of ~25 is based in Bengaluru. The Role Every voice AI agent Bolna deploys makes real-time judgment calls - when to speak, when to go silent, when a customer is done talking, when to hand off. We’re building automated systems to grade these calls at scale, using LLMs as judges of call quality. But before you trust a model’s judgment, you verify it against a human’s. That’s this role. You’ll listen to real calls, annotate what actually happened, and check whether our automated systems - LLM-as-judge evals and quantitative signal detection - got it right. It’s precise, high-attention work, and it sits right at the center of how we know our voice agents are actually working. This is an internship role for someone early in their career who wants hands-on exposure to how a voice AI company builds trust in its own AI. What You’ll Do Annotation Listen to and annotate real customer calls - transcription review, issue tagging, labeling - using tools like Label Studio Follow (and help sharpen) annotation guidelines for a multilingual environment (Hindi, English, Hinglish, ) Verifying LLM-as-Judge Evaluations For calls flagged by our automated eval pipeline, verify whether the model’s call was actually correct - for example, confirming whether a detected barge-in (agent/customer talking over each other) genuinely happened by listening to the audio Mark agreements and disagreements clearly, with reasoning, so we can measure and improve model accuracy over time All tools needed for this will be provided Verifying Quantitative Measures Check system-flagged quantit

Everpure (NYSE: P) has evolved from storage pioneer to data platform, closing fiscal 2026 with $3.7 billion in revenue, its first billion-dollar quarter, and accelerating growth into FY27. Our strategic agenda spans the companies defining the next era of technology - hyperscalers, AI labs, the AI hardware supply chain, data platform providers, and the broader AI ecosystem. This type of work—work that changes the world—is what the tech industry was founded on. So, if you're ready to seize the endless opportunities and leave your mark, come join us. THE ROLE We’re looking for an experienced Recruiter to join our growing Talent Acquisition team in Bangalore. The role will focus on hiring for the Go-To-Market roles (sales, presales, CX) across the region. This role offers a unique opportunity to influence hiring strategy in a fast-growing, highly competitive environment. In this role, you will: Act as a trusted Talent Acquisition partner to business leaders, developing and executing hiring strategies aligned to business priorities. Own end-to-end recruiting process, including intake alignment, sourcing strategy, screening, interview panel design, decision-making, offer strategy, and onboarding alignment. Build strong relationships with hiring managers and stakeholders, advising them on market trends, talent availability, and hiring trade-offs. Deliver an exceptional candidate experience from first contact through offer acceptance. Develop and execute sourcing strategies for highly specialized and competitive talent. Proactively source, engage, and maintain strong, diverse pipelines of candidates. Use recruiting metrics and insights to drive continuous improvement in hiring outcomes and efficiency. Partner closely with HR Business Partners, Total Rewards, Finance, and Interview Coordination to ensure smooth, timely hiring decisions. Be an ambassador for Everpure WHAT YOU’LL NEED TO BRING TO THIS ROLE… Strong Recruiting expertise with a focus on quality,