124 Lead Data Engineer jobs in the United Kingdom

Role Overview- SC Cleared or Eligible only

Salary- 70k -80k per annum

We are seeking an accomplished Lead Data Engineer (SFIA Level 5) to provide technical leadership, direction, and strategy across complex data engineering initiatives. This role involves leading teams to deliver resilient, scalable, and standards-driven data solutions, while engaging with senior stakeholders to translate requirements into robust data products and services.

The Lead Data Engineer will be responsible for setting engineering standards, embedding best practices, and ensuring delivery aligns with strategic principles. The role also carries line management responsibilities, helping to build capability within the engineering community.

Key Responsibilities

• Provide technical leadership and set the direction for engineering teams and communities, ensuring delivery adheres to data standards and strategic principles.
• Engage with senior stakeholders to define requirements for complex and sensitive data products, pipelines, and platforms.
• Oversee delivery and user testing of requirements across teams.
• Lead the redevelopment of existing data journeys, enhancing performance, resilience, and scalability, while adapting to evolving systems, tools, and platforms.
• Translate user requirements and data designs into effective, reusable, and standards-compliant data solutions.
• Define, deliver, and embed engineering standards across multiple platforms, keeping them updated and ensuring adherence across teams.
• Champion data validation methods and standards to ensure data quality from source to consumption.
• Standardise code, product, and service quality across engineering teams by implementing best practices and governance.
• Design and promote reusable metadata libraries and cross-business standards in collaboration with other technical disciplines.
• Partner with Lead Data Engineers, Data Architects, and Technologists to design and develop innovative solutions.
• Continually improve data engineering, automation, and scaling capabilities.
• Provide leadership and mentoring, helping to develop the skills and capabilities of the Data Engineer community.
• Carry out line management responsibilities, supporting the growth and development of Data Engineers.
• Ensure alignment and collaboration with teams across Digital Group, Data & Analytics, and Data Practice for consistency and scalability.

Skills & Experience Required

Practitioner Level:

• Proven ability to lead development and delivery of complex data pipelines using big data technologies (Apache Kafka, Spark, or similar) and cloud platforms (AWS, Azure).
• Technical leadership experience, setting team direction and ensuring adherence to data standards and strategic principles.
• Strong programming/coding expertise across Python, SQL, Proc SQL, and cloud technologies (Azure stack, AWS).
• Experience in designing and executing data/code quality assurance processes, troubleshooting, and resolving processing issues.
• Ability to apply standards and tools to design, code, test, correct, and document programs/scripts from agreed specifications.
• Knowledge of data modelling, with experience leading strategies for delivery to wider communities.

Expert Level:

• Proven track record leading the development of ETL pipelines for large, complex, or high-volume datasets, with effective delegation and oversight.

Advanced knowledge of data engineering standards, with the ability to establish, maintain, and enforce them across teams.

Role Overview- SC Cleared or Eligible only

Salary- 70k -80k per annum

We are seeking an accomplished Lead Data Engineer (SFIA Level 5) to provide technical leadership, direction, and strategy across complex data engineering initiatives. This role involves leading teams to deliver resilient, scalable, and standards-driven data solutions, while engaging with senior stakeholders to translate requirements into robust data products and services.

The Lead Data Engineer will be responsible for setting engineering standards, embedding best practices, and ensuring delivery aligns with strategic principles. The role also carries line management responsibilities, helping to build capability within the engineering community.

Key Responsibilities

• Provide technical leadership and set the direction for engineering teams and communities, ensuring delivery adheres to data standards and strategic principles.
• Engage with senior stakeholders to define requirements for complex and sensitive data products, pipelines, and platforms.
• Oversee delivery and user testing of requirements across teams.
• Lead the redevelopment of existing data journeys, enhancing performance, resilience, and scalability, while adapting to evolving systems, tools, and platforms.
• Translate user requirements and data designs into effective, reusable, and standards-compliant data solutions.
• Define, deliver, and embed engineering standards across multiple platforms, keeping them updated and ensuring adherence across teams.
• Champion data validation methods and standards to ensure data quality from source to consumption.
• Standardise code, product, and service quality across engineering teams by implementing best practices and governance.
• Design and promote reusable metadata libraries and cross-business standards in collaboration with other technical disciplines.
• Partner with Lead Data Engineers, Data Architects, and Technologists to design and develop innovative solutions.
• Continually improve data engineering, automation, and scaling capabilities.
• Provide leadership and mentoring, helping to develop the skills and capabilities of the Data Engineer community.
• Carry out line management responsibilities, supporting the growth and development of Data Engineers.
• Ensure alignment and collaboration with teams across Digital Group, Data & Analytics, and Data Practice for consistency and scalability.

Skills & Experience Required

Practitioner Level:

• Proven ability to lead development and delivery of complex data pipelines using big data technologies (Apache Kafka, Spark, or similar) and cloud platforms (AWS, Azure).
• Technical leadership experience, setting team direction and ensuring adherence to data standards and strategic principles.
• Strong programming/coding expertise across Python, SQL, Proc SQL, and cloud technologies (Azure stack, AWS).
• Experience in designing and executing data/code quality assurance processes, troubleshooting, and resolving processing issues.
• Ability to apply standards and tools to design, code, test, correct, and document programs/scripts from agreed specifications.
• Knowledge of data modelling, with experience leading strategies for delivery to wider communities.

Expert Level:

• Proven track record leading the development of ETL pipelines for large, complex, or high-volume datasets, with effective delegation and oversight.

Advanced knowledge of data engineering standards, with the ability to establish, maintain, and enforce them across teams.

• Your title will be Lead Data Engineer
• Marco, Lead Analytics Engineer, is the hiring manager
• We work Mon-Thu in our office in Chancery Lane, London, Fri from anywhere

>£140,000 + equity 

Walk around the average office and you’ll see people’s days taken up by emails, Slack and meetings instead of real work.

People in client facing roles - think estate agents, insurance brokers, recruiters - feel this pain most acutely. Instead of meeting clients, they spend hours doing admin. Following up. Scheduling meetings, then taking notes on them. Answering questions they’ve been asked a thousand times. Sorting through the mess that is their inbox.

We’ve built an AI executive assistant that looks at all your emails, messages and meetings, and uses that knowledge to answer your email, schedule meetings, take next steps from meetings and organise your inbox.

Since launching in April 2024, we've gone from $0 to $8m in ARR and raised a 30m Series B from top SF investors.

We’re very intentional about adding new people. We think a small team of exceptional people working hard at a problem they care about will always beat a larger, more unfocused team. That means you’ll need to bring an intensity to this role that might not be asked at other companies. But it also means you will be fast tracked into more senior roles and responsibilities far earlier.

We also believe in hiring people who want ownership and autonomy in their work, and giving it to them. Instead of just being handed tickets, you’ll own our data infrastructure, proactively suggesting improvements, including tools we use, and how data is modelled and moved between locations.

In short, you’ll own Fyxer AI’s end-to-end data platform—powering both our technical infrastructure and all commercial marketing analytics.

• Maintain and improve data pipelines between our warehouse (BigQuery) and multiple data sources—including our production database, Posthog, and the full suite of SaaS tools used by our GTM and marketing teams (HubSpot, Zoominfo, Stripe, Intercom, Customer.io, and more).
• Build new, robust integrations to ingest and model marketing and revenue data—transforming messy, multi-channel customer and campaign data into well-structured, analytics-ready tables.
• Optimise our data transformation process (dbt, SQL), ensuring not just performance and ease of querying, but also that marketing and sales teams can accurately track campaign ROI, spend attribution, lead scoring, funnel conversion, and churn risk.
• Make architecture decisions that ensure our infrastructure is scalable and secure, but also designed to empower marketing analytics—enabling experimentation, rapid reporting, and actionable GTM insights.
• Work directly with stakeholders in marketing, sales, customer success and product to define the key commercial metrics and marketing KPIs we need to monitor—then build and maintain the data models to deliver reliable, real-time reporting.
• Collaborate with analysts and business users to ensure all data, especially marketing data, is modelled in the right format for easy dashboarding, self-service analytics, and deep-dive analysis.
• Drive innovation across our data stack—automate processes, experiment with new approaches, and find ways for data to proactively surface the signals that help our marketing and GTM teams grow faster.

• You’ve worked at an early stage tech company (<100 people) as an analytics engineer or similar
• You have an expert level understanding of SQL and a modern ELT stack. If you don’t have both of these, you shouldn’t apply.
• You are excellent at transforming raw, and often complex, data into production tables for use for visualization and analysis
• You can interface with both technical and non-technical stakeholders and recommend a data structure for their area that is intuitive and easy to use.
• Urgency and intensity in your work

Broadly, we use a fairly typical ELT stack. It’s not a requirement to have worked with every tool we use, but the more the better!

• BigQuery as our data warehouse
• Metabase for data visualization
• Fivetran to pipe raw data from third party tools (eg Stripe, which we use for billing) into our data warehouse
• dbt hosted on Github Actions for transformation of raw data into production tables ready for consumption
• Census for reverse ETL
• Third party tools we interested in analysing, such as Posthog, Stripe, Hubspot, Intercom, and Meta

1. Submit your CV (no need for a cover letter)
2. An initial call with someone from the Fyxer AI team to review your experience and motivation for joining (15 mins)
3. Meet our Co Founder, Archie - (30 minutes)
4. Technical Take Home Test
5. Hiring Manager interview - Onsite (45 minutes)
6. Meet Tom - COO (30 minutes)

Salary: £100,000 - £120,000 - Must be based in the UK

Please note: The hiring manager is on annual leave from the 20th October to 3rd November.

Midnite is a next-generation betting platform that is built for today’s fandom. We are a collective of engineers and designers who all share a passion for building the best sportsbook & casino experience possible, allowing our fans to feel closer to the games they love through the rush of winning money. 

Unlike the alternatives, Midnite doesn't feel like a website built two decades ago. Instead, it's a cutting-edge creation, designed and constructed from the ground up with the latest technologies. Crafting an experience that's truly intuitive, immersive, and immediately understandable is no walk in the park, but we thrive on the challenge. We believe we're on the brink of creating something truly awesome. 

We’re looking for a Lead Data Engineer to drive the next phase of our data strategy at Midnite. This is a hands-on leadership role where you’ll set the technical direction, own the design and scalability of our data infrastructure, and ensure the team delivers high-quality, impactful solutions.

You’ll work across the full data lifecycle from ingestion and modelling to orchestration, monitoring, and analytics enablement, while also mentoring engineers and shaping engineering best practices. As a lead, you’ll partner with our leadership team to make sure our data function not only delivers but also drives strategic decision-making.

Python, Docker, Dagster, dbt, Fivetran, Apache Iceberg, Snowflake, S3, Glue, ECS, and Omni. We’re constantly evolving our stack and welcome input from engineering leaders on how we can improve scalability, reliability, and efficiency.

As Lead Data Engineer, you’ll be both a technical expert and a team leader. You’ll:

• Set technical standards and drive adoption of best practices across the team.
• Mentor and coach engineers, raising the bar on quality and delivery.
• Collaborate closely with senior stakeholders to align data initiatives with business priorities.
• Champion innovation, evaluating new tools, platforms, and methodologies.

• Own the technical strategy for data engineering, ensuring our stack scales with the business.
• Design, maintain, and evolve robust data pipelines and architecture to support low latency batch use cases.
• Oversee the implementation of data models and frameworks that support analytics, and business intelligence.
• Drive engineering best practices across testing, monitoring, version control, and automation.
• Lead code reviews, enforce quality standards, and ensure technical debt is managed proactively.
• Manage and mentor engineers, supporting career development and creating a culture of excellence.
• Stay ahead of industry trends, introducing tools and methods that future-proof the data platform.

• 7+ years in data engineering, with at least 2+ years in a lead or equivalent role.
• Proven track record of designing and scaling data platforms in a high-growth or start-up environment.
• Strong expertise in Python and SQL, with deep experience in orchestration frameworks (Dagster, Airflow, Prefect).
• Advanced knowledge of data modelling and architecture (Kimball dimensional modelling, Data Vault etc).
• Hands-on experience with dbt, modern data warehouses, and AWS.
• Demonstrated ability to mentor and develop engineers.

• Experience with Snowflake.
• Experience with Apache Iceberg.
• Experience with infrastructure-as-code (Terraform preferred).
• Experience embedding observability and monitoring in data systems.
• Previous experience building and leading data teams in a scale-up environment.

• Shape our future: Play a key role in our team's success, where your voice matters, and you'll have a direct impact on shaping Midnite's future.
• Connect and unwind: Take part in our quarterly gatherings where our community comes together to bond and have fun.
• Comprehensive health coverage: Look after your well-being with our outstanding zero-excess health insurance plan, which includes optical and dental coverage.
• Simplify life: Take advantage of our nursery salary sacrifice scheme, allowing you to conveniently pay your child's nursery fees straight from your paycheck.
• Work-life balance: Enjoy 25 paid holidays a year, plus generous paid maternity, paternity, and adoption leave, supporting you during life's most important moments.
• Productive home office: We provide everything you need for a comfortable and ergonomic home setup, ensuring you're as productive as possible.
• Flexible working: We embrace flexible working, allowing you to adjust your schedule when life's unexpected moments arise.
• Latest tech made easy: With our salary sacrifice schemes, you can upgrade to the latest gadgets, household items, and mobile tech without the upfront cost.
• Exclusive perks: Enjoy a wide range of discounts on retailers, groceries, and subscriptions, making life a little more affordable.
• Grow with us: Expand your skills through internal and external learning opportunities while benefiting from access to mentorship programs that support your development.
• Transparent compensation: We provide competitive pay with clear team bandings and salary grids, ensuring that salary discussions are simple and fair. 
• Constructive feedback: We foster a transparent culture, encouraging individual feedback and review sessions to help everyone improve.
• Work from anywhere: Whether it's a cosy cottage in the Cotswolds or anywhere else, enjoy the freedom of working remotely.

Ready to join a team that's leading the way in reshaping the future of insurance? Here at esure Group, we are on a mission to revolutionise insurance for good!

We’ve been providing Home and Motor Insurance since 2000, with over 2 million customers trusting us to keep them covered through our esure and Sheilas’ Wheels brands. With a bold dedication for digital innovation, we're transforming the way the industry operates and putting customers at the heart of everything we do.

Having completed our recent multi-year digital transformation, we’re now leveraging advanced technology and data-driven insights alongside exceptional service, to deliver personalised experiences that meet our customers ever-changing needs today and in the future.

We are currently looking for a Lead Data Engineer to join our forward thinking Data department.

You’ll work in a team of data engineers, analytics engineers, data scientists and AI specialists to design and evolve scalable data platforms and modern data products that enable self-service analytics, advanced modelling, and AI-driven decision-making across our insurance business.

What you’ll do:

• Lead on data engineering projects, managing a sub- team within engineering, building out our data warehouse & pipelines
• Collaborate effectively across data science, analytics, product, engineering and commercial business teams
• Build and support esure’s data products within our industry leading platform
• Work with product managers, data & AI engineers to deliver technical solutions for our most pressing data problems including GenAI applications
• Integrate data from a variety of sources, assuring that they adhere to data quality and technical standards. Migrating legacy systems into a modern scalable platform
• Creating frameworks and processes for data pipelines across the data and analytics platform
• Improve data engineering processes and roll these out across our team and wider data community
• Set coding standards across the platform, naming conventions across data products, championing and enforcing both.
• Work with architects on best design for data products, evaluating and experimenting with new data tools & supporting ML & AI infrastructure and workflows

What we’d love you to bring:

• Strategic thinker, aligning multiple workstreams to deliver a scalable, high-quality data platform.
• Develops the data function roadmap with thought leadership on tooling, technology, and guidelines.
• Product-minded leader, solving customer and business data challenges.
• Culture builder, driving continuous improvement and operational excellence.
• Deep expertise in data compliance frameworks, cost management, and platform optimisation.
• Strong hands-on experience with modern cloud data warehouses (Databricks, Snowflake, AWS), SQL, Spark, Airflow, Terraform.
• Advanced Python skills with orchestration tooling; solid experience in CI/CD (Git, Jenkins).
• Proven track record in data modelling, batch/real-time integration, and large-scale data engineering.
• Exposure to deploying Generative AI in production environments.

What’s in it for you?:

• Competitive salary that reflects your skills, experience and potential.
• Discretionary bonus scheme that recognises your hard work and contributions to esure’s success.
• 25 days annual leave, plus 8 flexible days and the ability to buy and sell further holiday.
• Our flexible benefits platform is loaded with perks to choose from, so you can build a personal toolkit to support your health, wellbeing, lifestyle, and finances.
• Company funded private medical insurance for qualifying colleagues.
• Fantastic discounts on our insurance products! 50% off for yourself and spouse/partner and 10% off for direct family members.
• We’ll elevate your career with hands-on training, mentoring, access to our exclusive academies, regular career conversations, and expert partner resources.
• Driving good in the world couldn’t be more important to us. Our colleagues can use 2 volunteering days per year to support their local communities.
• Join our internal networks and communities to connect, learn, and share ideas with likeminded colleagues.
• We’re a proud supporter of the ABI’s ‘Make Flexible Work’ campaign and welcome you to ask about the flexibility you need. Our hybrid working approach also puts you in the driving seat of how and where you do your best work.
• And much more; See a full overview of our benefits here Reward and benefits | Esure Group PLC

We are committed to creating an inclusive and diverse workplace where everyone feels valued, respected, and empowered. We celebrate individuality and create spaces where unique backgrounds and experiences can come together. We believe that diverse perspectives drive innovation, in turn enabling us to better serve our customers, community and build a stronger organisation. Our commitment to inclusion extends to every part of our business, from hiring practices to professional growth opportunities, ensuring equal access and support for all.

Lead Data Engineer (Databricks) - Leeds

(Lead Data Engineer, Team Lead, Technical Lead, Senior Data Engineer, Data Engineer, Python, PySpark, SQL, Big Data, Databricks, R, Machine Learning, AI, Agile, Scrum, TDD, BDD, CI / CD, SOLID principles, Github, Azure DevOps, Jenkins, Terraform, AWS CDK, AWS CloudFormation, Azure, Lead Data Engineer, Team Lead, Technical Lead, Senior Data Engineer, Data Engineer)

Our client is a global innovator and world leader with one of the most recognisable names within technology. They are looking for a Lead Data Engineer with significant Databricks experience as well as leadership responsibility to run an exceptional Agile engineering team and provide technical leadership through coaching and mentorship.

We are seeking a Lead Data Engineer capable of leading client delivery, ensuring the highest standards. This will include working with architects, creating automated tests, instilling a culture of continuous improvement and setting standards for the team. You will be responsible for building a greenfield modern data platform using cutting-edge technologies, architecting big data solutions and developing complex enterprise data ETL and ML pipelines and projections.

The successful candidate will have strong Python, PySpark and SQL experience, possess a clear understanding of databricks, as well as a passion for Data Science (R, Machine Learning and AI). Database experience with SQL and No-SQL – Aurora, MS SQL Server, MySQL is expected, as well as significant Agile and Scrum exposure along with SOLID principles. Continuous Integration tools, Infrastructure as code and strong Cloud Platform knowledge, ideally with AWS is also key.

We are keen to hear from talented Lead Data Engineer candidates from all backgrounds.

This is a truly amazing opportunity to work for a prestigious brand that will do wonders for your career. They invest heavily in training and career development with unlimited career progression for top performers.

Location: Leeds

Salary: £55k - £70k + Pension + Benefits

To apply for this position please send your CV to Nathan Warner at Noir Consulting.

NOIRUKTECHREC

NOIRUKREC

• Design, build, and maintain scalable big data pipelines and infrastructure.
• Develop and optimize ETL/ELT processes for data ingestion and transformation.
• Manage and enhance data lakes and data warehouses.
• Ensure data quality, integrity, and reliability.
• Implement data governance and security best practices.
• Lead and mentor a team of data engineers.
• Collaborate with data scientists and analysts to meet their data needs.
• Evaluate and adopt new data technologies and tools.