Senior Quality Assurance Engineer
About the role
The Senior Quality Assurance Engineer will be responsible for owning and executing the quality strategy for Pearson's AI Support Assistant, an LLM-powered, context-aware support tool integrated across digital learning platforms. The role focuses on testing conversational AI quality, validating RAG pipelines, integration and end-to-end testing, and building scalable quality monitoring and automation infrastructure. The position involves close collaboration with ML/AI engineers, product management, and platform teams to ensure the assistant delivers accurate, reliable, and seamless support experiences to millions of users. Candidates should have significant experience in software quality engineering, particularly with AI/ML systems and LLM-based applications, as well as strong technical skills in automation, scripting, and distributed systems testing. Pearson offers competitive salary, comprehensive benefits, flexible working arrangements, and opportunities for professional development within a global EdTech leader.
Must have
- 5+ years in software quality engineering, with at least 2 years testing AI/ML systems, conversational AI, chatbots, or NLP-driven products
- Hands-on experience testing LLM-based applications — prompt evaluation, response grading, factual accuracy measurement, and regression testing for non-deterministic outputs
- Hands-on experience validating Retrieval-Augmented Generation (RAG) pipelines, including retrieval accuracy, context grounding, and relevance validation
- Experience testing semantic search and vector-based retrieval systems (embeddings, similarity scoring, ranking relevance)
- Experience with LLM Evaluator frameworks and LLM-as-judge methodologies for automated, scalable scoring of model outputs
- Experience building evaluation harnesses for LLM outputs, including automated scoring and human-in-the-loop review pipelines
- Strong experience with API testing (REST) and integration testing across distributed systems
- Proficiency with test automation frameworks (Selenium, Playwright, Cypress, or similar) and CI/CD pipelines
- Strong scripting/programming ability (e.g., Python, JavaScript, or Java) to build custom test tooling, evaluation harnesses, and data pipelines
- Experience testing across multiple browsers, devices, and platforms — including mobile web
- Solid understanding of session/local storage, client-side state management, and front-end data contracts
- Familiarity with e-commerce or EdTech platforms — checkout flows, entitlements, user provisioning
Nice to have
- Experience with LMS platforms (e.g., Canvas, Blackboard, Moodle) and LTI integrations
- Familiarity with MCP (Model Context Protocol) or similar AI tool-integration patterns
- Knowledge of accessibility testing (WCAG 2.1 AA) for embedded chat interfaces
- Experience with performance/load testing tools (k6, Locust, JMeter)
- Familiarity with Jira, Confluence, and Agile workflows
Technologies
- Accessibility
- Agile
- API Testing
- CI/CD
- Confluence
- Conversational AI
- Cypress
- Data Pipelines
- E-commerce
- Embeddings
- Java
- JavaScript
- Jira
- JMeter
- k6
- LLM
- Machine Learning
- MCP
- NLP
- Performance Testing
- Playwright
- Python
- RAG
- REST
- Selenium
- Semantic Search
- Test Automation
- User Research
- WCAG
Highlights
The role is hybrid, with flexible working arrangements. Pearson offers a competitive salary, performance-based bonus, comprehensive health benefits, generous PTO, and an annual learning and development budget. Reasonable accommodations are available for applicants with disabilities. Pearson is an Equal Opportunity Employer and participates in E-Verify.
Jobs across Sri Lanka, watched for you.Colombo, Sri Lanka