SDET - Test Automation AI (Agentic & LLM Systems) - Perm - Circa £85K - 2 days a week in Glasgow

SDET - Test Automation AI (Agentic & LLM Systems) - Perm - Circa £85K - 2 days a week in Glasgow

** DUE TO BACKGROUND CHECKS, WE CAN ONLY ACCEPT APPLICANTS WHO HAS 3 YEARS UK ADDRESS HISTORY **

Role Overview

We are seeking an SDET - Test Automation AI (Agentic & LLM Systems) to define and implement assurance approaches for AI-enabled systems.
The role focuses on ensuring that AI solutions are reliable, robust, explainable, secure, and fit for purpose across their full lifecycle.

The AI Assurance Engineer will assure both:

• Probabilistic components (data, models, and AI outputs)
• Deterministic components (software, integrations, and infrastructure)

and will embed assurance into automated, end-to-end delivery pipelines.

This role requires a strong understanding of how to assure AI systems holistically, rather than deep specialism in a single discipline.

Key Responsibilities

AI Assurance Strategy

• Define and implement an AI assurance approach aligned to business risk and regulatory expectations.
• Provide assurance coverage across the full AI system lifecycle (design, build, deploy, operate).
• Work with engineering, data, and product teams to embed quality and risk controls early.

Probabilistic Component Assurance

• Design validation approaches for:
  • Data quality and bias
  • Model and prompt behaviour
  • Output accuracy, relevance, and consistency
• Implement evaluation methods for:
  • Drift and instability
  • Hallucination and error patterns
• Support human-in-the-loop review where required.

Deterministic Component Assurance

• Assure non-AI system elements including:
  • Application logic and workflows
  • APIs and integrations
  • Security and access controls
• Design and execute:
  • Functional testing
  • Non-functional testing (performance, resilience, scalability)
  • Security and data protection validation

Automation & E2E Assurance

• Design & build automated assurance for AI systems.
• Integrate assurance into CI/CD and deployment pipelines.
• Implement regression and quality gates across data, models, and orchestration workflows.
• Maintain an end-to-end assurance pipeline from input data through to system outputs.

Operational AI & Observability

• Support monitoring and observability for AI-enabled systems in production.
• Analyse operational signals such as:
  • Latency and failures
  • Behaviour changes
  • Performance degradation
• Contribute to incident analysis and continuous improvement of AI services.

Governance, Risk & Reporting

• Define and track AI quality and risk metrics (accuracy, robustness, explainability).
• Support compliance with:
  • Data protection and privacy requirements
  • Responsible AI principles
• Produce clear assurance evidence for technical and non-technical stakeholders.

Required Skills & Experience

Core

• Strong software engineering background (Python or similar).
• Experience building automated test or validation frameworks.
• Experience working with complex distributed or cloud-based systems.

AI & Probabilistic Systems

• Understanding of:
  • Data quality and bias
  • Model behaviour and non-deterministic outputs
  • Prompt-based or agent-based systems
• Experience validating correctness, consistency, and relevance of AI outputs.

Deterministic Systems & Non-Functional Testing

• Experience testing:
  • APIs and workflows
  • Cloud services
• Knowledge of:
  • Performance testing
  • Security testing
  • Resilience and failure handling

Operational AI (MLOps / AIOps Awareness)

• Familiarity with:
  • Model lifecycle management
  • CI/CD for AI systems
  • Monitoring and drift detection
• Understanding of production risks associated with AI systems.