Director of Quality Assurance - AI Products

Key Responsibilities

• QA Strategy & Governance: Design and implement comprehensive testing strategies for generative AI products, LLMs, and agentic workflows to ensure production-grade quality and safety. 

• AI-Driven Frameworks: Architect and build scalable, AI-driven testing frameworks that evaluate accuracy, consistency, bias, and robustness. 

• Test Stream Generation: Develop and integrate automated test stream generation pipelines to simulate complex user behaviors, conversational dialogues, and edge cases. 

• Tooling Integration: Utilize and configure modern AI testing tools (such as Claude API for test stream generation, deep evaluation frameworks, and self-healing test automation). 

• Model Validation: Establish golden datasets, monitor system outputs for hallucinations, and validate data compliance.

• Cross-Functional Leadership: Partner with Product and Engineering to embed quality guardrails early in the SDLC and define acceptance criteria for AI features. 

Qualifications & Skills

• Experience: 10+ years of progressive QA/Software Engineering experience, with at least 3-5 years explicitly focused on AI/ML applications and LLM validation.

• Frameworks & Tech Stack: Advanced proficiency in building custom testing frameworks (e.g., using Python, Pytest, Playwright) and integrating Large Language Models (LLMs) like Claude for test generation and evaluation.

• AI & Automation Expertise: Deep understanding of AI evaluation techniques, including hallucination detection, prompt injection/security testing, bias analysis, and continuous monitoring. 

• Strategic Mindset: Proven ability to define long-term QA strategy, scale testing teams, and transition traditional testing paradigms to AI-first architectures.