Principal Data Platform Engineer

Principal Engineer – Data Platform

Location: Remote (UK-based; Security Clearance Required)

A UK Government Security Check (SC) clearance is required for this role. If you don’t currently hold SC clearance, we can support you in applying, provided you have lived and worked in the UK for a minimum of 5 years.

As a Principal Data Platform Engineer, you will define and drive the technical vision, architecture, and evolution of data platforms for a government client. You will operate at a systems and organisational level, shaping how data is ingested, processed, governed, and consumed across multiple teams and services.

This role goes beyond hands-on engineering — it requires deep technical authority, cross-team influence, and strategic decision-making to ensure platforms are secure, scalable, resilient, and aligned to long-term business outcomes.

Key Responsibilities

Define and own the technical strategy for data platform architecture, balancing short-term delivery with long-term scalability and maintainability.

Lead the design of complex, distributed data systems, including batch and real-time processing at scale.

Establish platform-wide standards, patterns, and best practices for data engineering, reliability, and security.

Act as a technical authority and mentor, guiding senior and lead engineers across teams.

Drive architecture decisions across multiple squads, ensuring alignment with enterprise and government standards.

Partner with architects, product leaders, and stakeholders to translate ambiguous requirements into scalable technical solutions.

Ensure high availability, fault tolerance, and low-latency performance across critical data services.

Champion engineering excellence, including TDD, BDD, observability, and continuous delivery practices.

Oversee data modelling standards and governance, ensuring consistency, lineage, and integrity across systems.

Lead cross-team incident analysis and root cause resolution, driving systemic improvements.

Evaluate and introduce new technologies, tools, and frameworks to evolve platform capabilities.

Provide technical leadership in client engagements, influencing decisions and managing senior stakeholder expectations.

Ensure operational excellence, including resilience planning, cost optimisation, and platform sustainability.

Required Experience & Competencies

Non-Technical Competencies

Operating effectively in high ambiguity and incomplete problem spaces

Influencing senior stakeholders and cross-functional teams

Navigating complex client environments and competing priorities

Leading through technical authority rather than hierarchy

Driving alignment across multiple teams and domains

Continuously evaluating and adopting emerging technologies strategically

Technical Competencies

Extensive experience designing and building large-scale, distributed data platforms

Deep expertise in Java-based systems and modern backend architectures

Strong experience with streaming and event-driven architectures (e.g., Kafka, Kinesis)

Experience with large-scale data processing frameworks (e.g., Spark, Flink, Kafka Streams)

Expertise in cloud-native architectures, ideally on AWS (including Lambda, S3, managed services)

Strong understanding of data pipeline design, orchestration, and optimisation

Experience with data modelling at scale, including schema design and evolution

Expertise in relational and NoSQL database design and performance tuning

Proven track record of implementing CI/CD, observability, and platform reliability practices

Strong experience with DevOps and Infrastructure as Code (IaC)

Ability to define and enforce engineering standards across teams

Desirable Skills

Advanced expertise in data modelling strategies and trade-offs across systems

Experience with search and indexing platforms (Elasticsearch, OpenSearch, Solr)

Strong knowledge of CI/CD ecosystems (e.g., Jenkins, GitLab CI, Drone)

Experience with infrastructure automation tools (Terraform, Ansible, Packer)

Exposure to data governance, security, and compliance frameworks in regulated environments

Qualifications

Degree in Computer Science or a related field (or equivalent experience)

Typically 10+ years of experience in software/data engineering, with significant time spent in architecture or principal-level roles