SENIOR QA ENGINEER (AI)
About the role
The Senior QA Engineer (AI) role is responsible for ensuring the quality, reliability, security, and performance of AI-enabled solutions throughout their lifecycle. The position involves designing and implementing risk-based test strategies, performing various types of testing (functional, integration, regression, system, accessibility, performance, security), and developing repeatable AI evaluation methods. The engineer will test advanced AI features such as RAG, embeddings, vector search, prompts, guardrails, and AI agents, as well as identify AI-specific risks and build automated regression suites integrated into CI/CD pipelines. Collaboration with cross-functional teams and mentoring of QA engineers are also key aspects of the role. The ideal candidate will have a strong background in software testing and automation, hands-on experience with AI technologies, and knowledge of responsible AI practices. The position is based in Colombo, Sri Lanka, within the Information Technology – Systems Delivery business unit.
Must have
- Strong background in software testing, test planning, execution, automation and defect management.
- Hands-on experience with RAG, vector search, embeddings, prompt engineering, grounding and guardrails.
- Experience testing agentic AI solutions, including tool use, memory, orchestration, permissions and failure recovery.
- Knowledge of AI evaluation datasets, benchmarks, regression suites and production monitoring.
- Ability to evaluate non-deterministic AI outputs through automated metrics and structured human review.
- Understanding of AI risks, responsible AI controls, security and data privacy.
- Experience with secure testing practices, including SAST, DAST and SCA.
- Familiarity with Agile methodologies such as Scrum or Kanban.
- Bachelor's degree in Computer Science, Engineering, Information Systems Management or a related field, or equivalent experience.
Nice to have
- Automation frameworks for web, mobile or API testing.
- Azure DevOps or Jenkins, SQL and Postman.
- JMeter, Jira, Xray and WCAG accessibility standards.
- ISTQB certification or relevant AI/ML training.
- Insurance or financial-services industry experience.
Technologies
- Accessibility
- Agentic AI
- Agile
- API Testing
- Azure DevOps
- CI/CD
- DAST
- Embeddings
- Generative AI
- Istqb
- Jenkins
- Jira
- JMeter
- Kanban
- Machine Learning
- Postman
- Prompt Engineering
- RAG
- Responsible AI
- SAST
- Scrum
- SQL
- Vector Search
- WCAG
Jobs across Sri Lanka, watched for you.Colombo, Sri Lanka