124 Lead Data Engineer jobs in the United Kingdom
Lead Data Engineer
Posted today
Job Viewed
Job Description
We're the world's leading data, insights, and consulting company; we shape the brands of tomorrow by better understanding people everywhere.
Kantar's Profiles division is home to the world's largest audience network.
With access to 170m+ people in over 100 global markets, we offer unrivalled global reach with local relevancy. Validated by industry leading anti-fraud technology, Kantar's Profiles Audience Network delivers the most meaningful data with consistency, accuracy, and accountability – all at speed and scale.
Job Details
Location:
Reading or London, UK
Please note:
You must have the right to work in the UK. We are unable to offer visa sponsorship for this role.
At Kantar, we help the world's leading brands make better decisions through data. We're now looking for an experienced
Lead Data Engineer
to join our team and help us elevate the quality and integrity of our global consumer panels.
These panels—more than 50 worldwide—are the foundation of our data-driven insights. Your mission? To build and evolve the data ecosystem, pipelines and data feed applications that ensure our panels remain robust, fraud-resistant, and ready to power world-class analytics.
What You'll Be Working On
- Designing and maintaining scalable, low-latency data pipelines that fuel our Data Science and AI models.
- Integrating data from diverse sources (databases, APIs, external sets) into a unified ecosystem.
- Collaborating with Data Scientists, Analysts, and Engineers to refine panellist quality and detect fraud.
- Innovating on our cloud infrastructure (AWS and Azure) to support new datasets and smarter decisioning.
- Automating manual processes and optimising data delivery for speed and reliability.
- Leading, managing and mentoring data engineers and junior engineers and contributing to a culture of technical excellence.
What You'll Bring
- 5+ years of experience in Data Engineering, with strong skills in data warehousing, BI, and big data.
- Team leadership and technical leadership
- Proficiency in Python and/or R, SQL, and CI/CD tools.
- Experience with AWS (preferred) or Azure, and tools like DBT and PowerBI.
- A problem-solving mindset with a passion for agile, iterative development.
- Ability to translate business needs into technical solutions and communicate clearly with stakeholders.
Our Tech Stack
- AWS (current environment)
- Azure (future direction)
- Redshift, Postgres, DBT, PowerBI
This is a high-impact role with visibility across our Operational, Engineering, and Data Science teams. If you're ready to shape the future of data at Kantar, we'd love to hear from you.
Apply now and be part of a team that's redefining how data drives decisions.
Country
United Kingdom
Why join Kantar?
We shape the brands of tomorrow by better understanding people everywhere. By understanding people, we can understand what drives their decisions, actions, and aspirations on a global scale. And by amplifying our in-depth expertise of human understanding alongside ground-breaking technology, we can help brands find concrete insights that will help them succeed in our fast-paced, ever shifting world.
And because we know people, we like to make sure our own people are being looked after as well. Equality of opportunity for everyone is our highest priority and we support our colleagues to work in a way that supports their health and wellbeing. While we encourage teams to spend part of their working week in the office, we understand no one size fits all; our approach is flexible to ensure everybody feels included, accepted, and that we can win together. We're dedicated to creating an inclusive culture and value the diversity of our people, clients, suppliers and communities, and we encourage applications from all backgrounds and sections of society. Even if you feel like you're not an exact match, we'd love to receive your application and talk to you about this job or others at Kantar.
Lead Data Engineer
Posted 9 days ago
Job Viewed
Job Description
Role Overview- SC Cleared or Eligible only
Salary- 70k -80k per annum
We are seeking an accomplished Lead Data Engineer (SFIA Level 5) to provide technical leadership, direction, and strategy across complex data engineering initiatives. This role involves leading teams to deliver resilient, scalable, and standards-driven data solutions, while engaging with senior stakeholders to translate requirements into robust data products and services.
The Lead Data Engineer will be responsible for setting engineering standards, embedding best practices, and ensuring delivery aligns with strategic principles. The role also carries line management responsibilities, helping to build capability within the engineering community.
Key Responsibilities
- Provide technical leadership and set the direction for engineering teams and communities, ensuring delivery adheres to data standards and strategic principles.
- Engage with senior stakeholders to define requirements for complex and sensitive data products, pipelines, and platforms.
- Oversee delivery and user testing of requirements across teams.
- Lead the redevelopment of existing data journeys, enhancing performance, resilience, and scalability, while adapting to evolving systems, tools, and platforms.
- Translate user requirements and data designs into effective, reusable, and standards-compliant data solutions.
- Define, deliver, and embed engineering standards across multiple platforms, keeping them updated and ensuring adherence across teams.
- Champion data validation methods and standards to ensure data quality from source to consumption.
- Standardise code, product, and service quality across engineering teams by implementing best practices and governance.
- Design and promote reusable metadata libraries and cross-business standards in collaboration with other technical disciplines.
- Partner with Lead Data Engineers, Data Architects, and Technologists to design and develop innovative solutions.
- Continually improve data engineering, automation, and scaling capabilities.
- Provide leadership and mentoring, helping to develop the skills and capabilities of the Data Engineer community.
- Carry out line management responsibilities, supporting the growth and development of Data Engineers.
- Ensure alignment and collaboration with teams across Digital Group, Data & Analytics, and Data Practice for consistency and scalability.
Skills & Experience Required
Practitioner Level:
- Proven ability to lead development and delivery of complex data pipelines using big data technologies (Apache Kafka, Spark, or similar) and cloud platforms (AWS, Azure).
- Technical leadership experience, setting team direction and ensuring adherence to data standards and strategic principles.
- Strong programming/coding expertise across Python, SQL, Proc SQL, and cloud technologies (Azure stack, AWS).
- Experience in designing and executing data/code quality assurance processes, troubleshooting, and resolving processing issues.
- Ability to apply standards and tools to design, code, test, correct, and document programs/scripts from agreed specifications.
- Knowledge of data modelling, with experience leading strategies for delivery to wider communities.
Expert Level:
- Proven track record leading the development of ETL pipelines for large, complex, or high-volume datasets, with effective delegation and oversight.
Advanced knowledge of data engineering standards, with the ability to establish, maintain, and enforce them across teams.
Lead Data Engineer
Posted today
Job Viewed
Job Description
Role Overview- SC Cleared or Eligible only
Salary- 70k -80k per annum
We are seeking an accomplished Lead Data Engineer (SFIA Level 5) to provide technical leadership, direction, and strategy across complex data engineering initiatives. This role involves leading teams to deliver resilient, scalable, and standards-driven data solutions, while engaging with senior stakeholders to translate requirements into robust data products and services.
The Lead Data Engineer will be responsible for setting engineering standards, embedding best practices, and ensuring delivery aligns with strategic principles. The role also carries line management responsibilities, helping to build capability within the engineering community.
Key Responsibilities
- Provide technical leadership and set the direction for engineering teams and communities, ensuring delivery adheres to data standards and strategic principles.
- Engage with senior stakeholders to define requirements for complex and sensitive data products, pipelines, and platforms.
- Oversee delivery and user testing of requirements across teams.
- Lead the redevelopment of existing data journeys, enhancing performance, resilience, and scalability, while adapting to evolving systems, tools, and platforms.
- Translate user requirements and data designs into effective, reusable, and standards-compliant data solutions.
- Define, deliver, and embed engineering standards across multiple platforms, keeping them updated and ensuring adherence across teams.
- Champion data validation methods and standards to ensure data quality from source to consumption.
- Standardise code, product, and service quality across engineering teams by implementing best practices and governance.
- Design and promote reusable metadata libraries and cross-business standards in collaboration with other technical disciplines.
- Partner with Lead Data Engineers, Data Architects, and Technologists to design and develop innovative solutions.
- Continually improve data engineering, automation, and scaling capabilities.
- Provide leadership and mentoring, helping to develop the skills and capabilities of the Data Engineer community.
- Carry out line management responsibilities, supporting the growth and development of Data Engineers.
- Ensure alignment and collaboration with teams across Digital Group, Data & Analytics, and Data Practice for consistency and scalability.
Skills & Experience Required
Practitioner Level:
- Proven ability to lead development and delivery of complex data pipelines using big data technologies (Apache Kafka, Spark, or similar) and cloud platforms (AWS, Azure).
- Technical leadership experience, setting team direction and ensuring adherence to data standards and strategic principles.
- Strong programming/coding expertise across Python, SQL, Proc SQL, and cloud technologies (Azure stack, AWS).
- Experience in designing and executing data/code quality assurance processes, troubleshooting, and resolving processing issues.
- Ability to apply standards and tools to design, code, test, correct, and document programs/scripts from agreed specifications.
- Knowledge of data modelling, with experience leading strategies for delivery to wider communities.
Expert Level:
- Proven track record leading the development of ETL pipelines for large, complex, or high-volume datasets, with effective delegation and oversight.
Advanced knowledge of data engineering standards, with the ability to establish, maintain, and enforce them across teams.
Lead Data Engineer
Posted 20 days ago
Job Viewed
Job Description
- Your title will be Lead Data Engineer
- Marco, Lead Analytics Engineer, is the hiring manager
- We work Mon-Thu in our office in Chancery Lane, London, Fri from anywhere
>£140,000 + equity
What are we building?Walk around the average office and you’ll see people’s days taken up by emails, Slack and meetings instead of real work.
People in client facing roles - think estate agents, insurance brokers, recruiters - feel this pain most acutely. Instead of meeting clients, they spend hours doing admin. Following up. Scheduling meetings, then taking notes on them. Answering questions they’ve been asked a thousand times. Sorting through the mess that is their inbox.
We’ve built an AI executive assistant that looks at all your emails, messages and meetings, and uses that knowledge to answer your email, schedule meetings, take next steps from meetings and organise your inbox.
How has it been going?Since launching in April 2024, we've gone from $0 to $8m in ARR and raised a 30m Series B from top SF investors.
What do we value?We’re very intentional about adding new people. We think a small team of exceptional people working hard at a problem they care about will always beat a larger, more unfocused team. That means you’ll need to bring an intensity to this role that might not be asked at other companies. But it also means you will be fast tracked into more senior roles and responsibilities far earlier.
We also believe in hiring people who want ownership and autonomy in their work, and giving it to them. Instead of just being handed tickets, you’ll own our data infrastructure, proactively suggesting improvements, including tools we use, and how data is modelled and moved between locations.
What will I do?In short, you’ll own Fyxer AI’s end-to-end data platform—powering both our technical infrastructure and all commercial marketing analytics.
- Maintain and improve data pipelines between our warehouse (BigQuery) and multiple data sources—including our production database, Posthog, and the full suite of SaaS tools used by our GTM and marketing teams (HubSpot, Zoominfo, Stripe, Intercom, Customer.io, and more).
- Build new, robust integrations to ingest and model marketing and revenue data—transforming messy, multi-channel customer and campaign data into well-structured, analytics-ready tables.
- Optimise our data transformation process (dbt, SQL), ensuring not just performance and ease of querying, but also that marketing and sales teams can accurately track campaign ROI, spend attribution, lead scoring, funnel conversion, and churn risk.
- Make architecture decisions that ensure our infrastructure is scalable and secure, but also designed to empower marketing analytics—enabling experimentation, rapid reporting, and actionable GTM insights.
- Work directly with stakeholders in marketing, sales, customer success and product to define the key commercial metrics and marketing KPIs we need to monitor—then build and maintain the data models to deliver reliable, real-time reporting.
- Collaborate with analysts and business users to ensure all data, especially marketing data, is modelled in the right format for easy dashboarding, self-service analytics, and deep-dive analysis.
- Drive innovation across our data stack—automate processes, experiment with new approaches, and find ways for data to proactively surface the signals that help our marketing and GTM teams grow faster.
- You’ve worked at an early stage tech company (<100 people) as an analytics engineer or similar
- You have an expert level understanding of SQL and a modern ELT stack. If you don’t have both of these, you shouldn’t apply.
- You are excellent at transforming raw, and often complex, data into production tables for use for visualization and analysis
- You can interface with both technical and non-technical stakeholders and recommend a data structure for their area that is intuitive and easy to use.
- Urgency and intensity in your work
Broadly, we use a fairly typical ELT stack. It’s not a requirement to have worked with every tool we use, but the more the better!
- BigQuery as our data warehouse
- Metabase for data visualization
- Fivetran to pipe raw data from third party tools (eg Stripe, which we use for billing) into our data warehouse
- dbt hosted on Github Actions for transformation of raw data into production tables ready for consumption
- Census for reverse ETL
- Third party tools we interested in analysing, such as Posthog, Stripe, Hubspot, Intercom, and Meta
- Submit your CV (no need for a cover letter)
- An initial call with someone from the Fyxer AI team to review your experience and motivation for joining (15 mins)
- Meet our Co Founder, Archie - (30 minutes)
- Technical Take Home Test
- Hiring Manager interview - Onsite (45 minutes)
- Meet Tom - COO (30 minutes)
Lead Data Engineer
Posted 20 days ago
Job Viewed
Job Description
Salary: £100,000 - £120,000 - Must be based in the UK
Please note: The hiring manager is on annual leave from the 20th October to 3rd November.
Why Midnite?Midnite is a next-generation betting platform that is built for today’s fandom. We are a collective of engineers and designers who all share a passion for building the best sportsbook & casino experience possible, allowing our fans to feel closer to the games they love through the rush of winning money.
Unlike the alternatives, Midnite doesn't feel like a website built two decades ago. Instead, it's a cutting-edge creation, designed and constructed from the ground up with the latest technologies. Crafting an experience that's truly intuitive, immersive, and immediately understandable is no walk in the park, but we thrive on the challenge. We believe we're on the brink of creating something truly awesome.
What will you do?We’re looking for a Lead Data Engineer to drive the next phase of our data strategy at Midnite. This is a hands-on leadership role where you’ll set the technical direction, own the design and scalability of our data infrastructure, and ensure the team delivers high-quality, impactful solutions.
You’ll work across the full data lifecycle from ingestion and modelling to orchestration, monitoring, and analytics enablement, while also mentoring engineers and shaping engineering best practices. As a lead, you’ll partner with our leadership team to make sure our data function not only delivers but also drives strategic decision-making.
Our Tech StackPython, Docker, Dagster, dbt, Fivetran, Apache Iceberg, Snowflake, S3, Glue, ECS, and Omni. We’re constantly evolving our stack and welcome input from engineering leaders on how we can improve scalability, reliability, and efficiency.
Leadership & CollaborationAs Lead Data Engineer, you’ll be both a technical expert and a team leader. You’ll:
- Set technical standards and drive adoption of best practices across the team.
- Mentor and coach engineers, raising the bar on quality and delivery.
- Collaborate closely with senior stakeholders to align data initiatives with business priorities.
- Champion innovation, evaluating new tools, platforms, and methodologies.
- Own the technical strategy for data engineering, ensuring our stack scales with the business.
- Design, maintain, and evolve robust data pipelines and architecture to support low latency batch use cases.
- Oversee the implementation of data models and frameworks that support analytics, and business intelligence.
- Drive engineering best practices across testing, monitoring, version control, and automation.
- Lead code reviews, enforce quality standards, and ensure technical debt is managed proactively.
- Manage and mentor engineers, supporting career development and creating a culture of excellence.
- Stay ahead of industry trends, introducing tools and methods that future-proof the data platform.
- 7+ years in data engineering, with at least 2+ years in a lead or equivalent role.
- Proven track record of designing and scaling data platforms in a high-growth or start-up environment.
- Strong expertise in Python and SQL, with deep experience in orchestration frameworks (Dagster, Airflow, Prefect).
- Advanced knowledge of data modelling and architecture (Kimball dimensional modelling, Data Vault etc).
- Hands-on experience with dbt, modern data warehouses, and AWS.
- Demonstrated ability to mentor and develop engineers.
- Experience with Snowflake.
- Experience with Apache Iceberg.
- Experience with infrastructure-as-code (Terraform preferred).
- Experience embedding observability and monitoring in data systems.
- Previous experience building and leading data teams in a scale-up environment.
- Shape our future: Play a key role in our team's success, where your voice matters, and you'll have a direct impact on shaping Midnite's future.
- Connect and unwind: Take part in our quarterly gatherings where our community comes together to bond and have fun.
- Comprehensive health coverage: Look after your well-being with our outstanding zero-excess health insurance plan, which includes optical and dental coverage.
- Simplify life: Take advantage of our nursery salary sacrifice scheme, allowing you to conveniently pay your child's nursery fees straight from your paycheck.
- Work-life balance: Enjoy 25 paid holidays a year, plus generous paid maternity, paternity, and adoption leave, supporting you during life's most important moments.
- Productive home office: We provide everything you need for a comfortable and ergonomic home setup, ensuring you're as productive as possible.
- Flexible working: We embrace flexible working, allowing you to adjust your schedule when life's unexpected moments arise.
- Latest tech made easy: With our salary sacrifice schemes, you can upgrade to the latest gadgets, household items, and mobile tech without the upfront cost.
- Exclusive perks: Enjoy a wide range of discounts on retailers, groceries, and subscriptions, making life a little more affordable.
- Grow with us: Expand your skills through internal and external learning opportunities while benefiting from access to mentorship programs that support your development.
- Transparent compensation: We provide competitive pay with clear team bandings and salary grids, ensuring that salary discussions are simple and fair.
- Constructive feedback: We foster a transparent culture, encouraging individual feedback and review sessions to help everyone improve.
- Work from anywhere: Whether it's a cosy cottage in the Cotswolds or anywhere else, enjoy the freedom of working remotely.
Lead Data Engineer - Reigate
Posted today
Job Viewed
Job Description
Ready to join a team that's leading the way in reshaping the future of insurance? Here at esure Group, we are on a mission to revolutionise insurance for good!
We’ve been providing Home and Motor Insurance since 2000, with over 2 million customers trusting us to keep them covered through our esure and Sheilas’ Wheels brands. With a bold dedication for digital innovation, we're transforming the way the industry operates and putting customers at the heart of everything we do.
Having completed our recent multi-year digital transformation, we’re now leveraging advanced technology and data-driven insights alongside exceptional service, to deliver personalised experiences that meet our customers ever-changing needs today and in the future.
Job DescriptionWe are currently looking for a Lead Data Engineer to join our forward thinking Data department.
You’ll work in a team of data engineers, analytics engineers, data scientists and AI specialists to design and evolve scalable data platforms and modern data products that enable self-service analytics, advanced modelling, and AI-driven decision-making across our insurance business.
What you’ll do:
- Lead on data engineering projects, managing a sub- team within engineering, building out our data warehouse & pipelines
- Collaborate effectively across data science, analytics, product, engineering and commercial business teams
- Build and support esure’s data products within our industry leading platform
- Work with product managers, data & AI engineers to deliver technical solutions for our most pressing data problems including GenAI applications
- Integrate data from a variety of sources, assuring that they adhere to data quality and technical standards. Migrating legacy systems into a modern scalable platform
- Creating frameworks and processes for data pipelines across the data and analytics platform
- Improve data engineering processes and roll these out across our team and wider data community
- Set coding standards across the platform, naming conventions across data products, championing and enforcing both.
- Work with architects on best design for data products, evaluating and experimenting with new data tools & supporting ML & AI infrastructure and workflows
What we’d love you to bring:
- Strategic thinker, aligning multiple workstreams to deliver a scalable, high-quality data platform.
- Develops the data function roadmap with thought leadership on tooling, technology, and guidelines.
- Product-minded leader, solving customer and business data challenges.
- Culture builder, driving continuous improvement and operational excellence.
- Deep expertise in data compliance frameworks, cost management, and platform optimisation.
- Strong hands-on experience with modern cloud data warehouses (Databricks, Snowflake, AWS), SQL, Spark, Airflow, Terraform.
- Advanced Python skills with orchestration tooling; solid experience in CI/CD (Git, Jenkins).
- Proven track record in data modelling, batch/real-time integration, and large-scale data engineering.
- Exposure to deploying Generative AI in production environments.
What’s in it for you?:
- Competitive salary that reflects your skills, experience and potential.
- Discretionary bonus scheme that recognises your hard work and contributions to esure’s success.
- 25 days annual leave, plus 8 flexible days and the ability to buy and sell further holiday.
- Our flexible benefits platform is loaded with perks to choose from, so you can build a personal toolkit to support your health, wellbeing, lifestyle, and finances.
- Company funded private medical insurance for qualifying colleagues.
- Fantastic discounts on our insurance products! 50% off for yourself and spouse/partner and 10% off for direct family members.
- We’ll elevate your career with hands-on training, mentoring, access to our exclusive academies, regular career conversations, and expert partner resources.
- Driving good in the world couldn’t be more important to us. Our colleagues can use 2 volunteering days per year to support their local communities.
- Join our internal networks and communities to connect, learn, and share ideas with likeminded colleagues.
- We’re a proud supporter of the ABI’s ‘Make Flexible Work’ campaign and welcome you to ask about the flexibility you need. Our hybrid working approach also puts you in the driving seat of how and where you do your best work.
- And much more; See a full overview of our benefits here Reward and benefits | Esure Group PLC
We are committed to creating an inclusive and diverse workplace where everyone feels valued, respected, and empowered. We celebrate individuality and create spaces where unique backgrounds and experiences can come together. We believe that diverse perspectives drive innovation, in turn enabling us to better serve our customers, community and build a stronger organisation. Our commitment to inclusion extends to every part of our business, from hiring practices to professional growth opportunities, ensuring equal access and support for all.
Lead Data Engineer (Databricks) - Leeds
Posted 2 days ago
Job Viewed
Job Description
Lead Data Engineer (Databricks) - Leeds
(Lead Data Engineer, Team Lead, Technical Lead, Senior Data Engineer, Data Engineer, Python, PySpark, SQL, Big Data, Databricks, R, Machine Learning, AI, Agile, Scrum, TDD, BDD, CI / CD, SOLID principles, Github, Azure DevOps, Jenkins, Terraform, AWS CDK, AWS CloudFormation, Azure, Lead Data Engineer, Team Lead, Technical Lead, Senior Data Engineer, Data Engineer)
Our client is a global innovator and world leader with one of the most recognisable names within technology. They are looking for a Lead Data Engineer with significant Databricks experience as well as leadership responsibility to run an exceptional Agile engineering team and provide technical leadership through coaching and mentorship.
We are seeking a Lead Data Engineer capable of leading client delivery, ensuring the highest standards. This will include working with architects, creating automated tests, instilling a culture of continuous improvement and setting standards for the team. You will be responsible for building a greenfield modern data platform using cutting-edge technologies, architecting big data solutions and developing complex enterprise data ETL and ML pipelines and projections.
The successful candidate will have strong Python, PySpark and SQL experience, possess a clear understanding of databricks, as well as a passion for Data Science (R, Machine Learning and AI). Database experience with SQL and No-SQL – Aurora, MS SQL Server, MySQL is expected, as well as significant Agile and Scrum exposure along with SOLID principles. Continuous Integration tools, Infrastructure as code and strong Cloud Platform knowledge, ideally with AWS is also key.
We are keen to hear from talented Lead Data Engineer candidates from all backgrounds.
This is a truly amazing opportunity to work for a prestigious brand that will do wonders for your career. They invest heavily in training and career development with unlimited career progression for top performers.
Location: Leeds
Salary: £55k - £70k + Pension + Benefits
To apply for this position please send your CV to Nathan Warner at Noir Consulting.
NOIRUKTECHREC
NOIRUKREC
Be The First To Know
About the latest Lead data engineer Jobs in United Kingdom !
Lead Data Engineer, Machine Learning

Posted 6 days ago
Job Viewed
Job Description
Our impact is rooted in improving the communities where our employees, customers, and audiences live and work. We have a rich tradition of giving back and ensuring our employees have the opportunity to serve their communities. We champion an inclusive culture and strive to attract and develop a talented workforce to create and deliver a wide range of content reflecting our world.
Comcast NBCUniversal has announced its intent to create a new publicly traded company ('Versant') comprised of most of NBCUniversal's cable television networks, including USA Network, CNBC, MSNBC, Oxygen, E!, SYFY and Golf Channel along with complementary digital assets Fandango, Rotten Tomatoes, GolfNow, GolfPass, and SportsEngine. The well-capitalized company will have significant scale as a pure-play set of assets anchored by leading news, sports and entertainment content. The spin-off is expected to be completed during 2025.
Job Description
Our Media Group portfolio is a powerhouse collection of consumer-first brands, supported by media industry leaders, Comcast, NBCUniversal, and Sky. When you join our team, you'll work across our dynamic portfolio including Peacock, NOW, Fandango, SkyShowtime, Showmax, and TV Everywhere, powering streaming across more than 70 countries globally. And the evolution doesn't stop there. With unequaled scale, our teams make the most out of every opportunity to collaborate and learn from one another. We're always looking for ways to innovate faster, accelerate our growth, and consistently offer the very best in consumer experience. But most of all, we're backed by a culture of respect. We embrace authenticity and inspire people to thrive.
As part of the Media Group Decision Sciences team, the Lead Data Engineer will be responsible for creating a connected data ecosystem that unleashes the power of our streaming data. We gather data from across all customer/prospect journeys in near real-time, to allow fast feedback loops across territories; combined with our strategic data platform, this data ecosystem is at the core of being able to make intelligent customer and business decisions.
In this role, the Lead Data Engineer will share responsibilities in the development and maintenance of optimised and highly available data pipelines that facilitate deeper analysis and reporting by the business, as well as support ongoing operations related to the Media Group data ecosystem.
Responsibilities include, but are not limited to:
+ Help manage a high-performance team of Data Engineers.
+ Contribute to and help lead team in design, build, testing, scaling and maintaining data pipelines from a variety of source systems and streams (internal, third party, cloud based, etc.), according to business and technical requirements.
+ Deliver observable, reliable and secure software, embracing "you build it you run it" mentality, and focus on automation and GitOps.
+ Continually work on improving the codebase and have active participation and oversight in all aspects of the team, including agile ceremonies.
+ Take an active role in story definition, assisting business stakeholders with acceptance criteria.
+ Work with Principal Engineers and Architects to share and contribute to the broader technical vision.
+ Develop and champion best practices, striving towards excellence and raising the bar within the department.
+ Develop solutions combining data blending, profiling, mining, statistical analysis, and machine learning, to better define and curate models, test hypothesis, and deliver key insights.
+ Operationalise data processing systems (dev ops).
Qualifications
+ Extensive relevant experience in Data Engineering.
+ Experience of near Real Time & Batch Data Pipeline development in a similar Big Data Engineering role.
+ Programming skills in one or more of the following: Python, Java, Scala, R, SQL and experience in writing reusable/efficient code to automate analysis and data processes.
+ Experience in processing structured and unstructured data into a form suitable for analysis and reporting with integration with a variety of data metric providers ranging from advertising, web analytics, and consumer devices.
+ Experience implementing scalable, distributed, and highly available systems using Google Cloud.
+ Hands on programming experience of the following (or similar) technologies: Apache Beam, Scio, Apache Spark, and Snowflake.
+ Experience in progressive data application development, working in large scale/distributed SQL, NoSQL, and/or Hadoop environment.
+ Build and maintain dimensional data warehouses in support of BI tools.
+ Develop data catalogs and data cleanliness to ensure clarity and correctness of key business metrics.
+ Experience building streaming data pipelines using Kafka, Spark or Flink.
+ Data modelling experience (operationalising data science models/products) a plus.
+ Bachelors' degree with a specialisation in Computer Science, Engineering, Physics, other quantitative field or equivalent industry experience.
Desired Characteristics
+ Experience with graph-based data workflows using Apache Airflow.
+ Experience building and deploying ML pipelines: training models, feature development, regression testing.
+ Strong Test-Driven Development background, with understanding of levels of testing required to continuously deliver value to production.
+ Experience with large-scale video assets.
+ Ability to work effectively across functions, disciplines, and levels.
+ Team-oriented and collaborative approach with a demonstrated aptitude, enthusiasm and willingness to learn new methods, tools, practices and skills.
+ Ability to recognise discordant views and take part in constructive dialogue to resolve them.
+ Pride and ownership in your work and confident representation of your team to other parts of NBCUniversal.
As part of our selection process, external candidates may be required to attend an in-person interview with an NBCUniversal employee at one of our locations prior to a hiring decision. NBCUniversal's policy is to provide equal employment opportunities to all applicants and employees without regard to race, color, religion, creed, gender, gender identity or expression, age, national origin or ancestry, citizenship, disability, sexual orientation, marital status, pregnancy, veteran status, membership in the uniformed services, genetic information, or any other basis protected by applicable law.
If you are a qualified individual with a disability or a disabled veteran and require support throughout the application and/or recruitment process as a result of your disability, you have the right to request a reasonable accommodation. You can submit your request to
Lead Data Engineer - Big Data Platforms
Posted 15 days ago
Job Viewed
Job Description
The ideal candidate will possess a Bachelor's or Master's degree in Computer Science, Engineering, or a related quantitative field, with a minimum of 7 years of professional experience in data engineering. Extensive hands-on experience with distributed big data technologies such as Apache Spark, Hadoop, Kafka, and cloud-based data services (e.g., AWS EMR, Azure Databricks, Google Cloud Dataflow) is essential. Proficiency in SQL and NoSQL databases, as well as strong programming skills in Python or Scala, are required. You should have a deep understanding of data warehousing concepts, data modeling, and ETL/ELT best practices. Experience in designing and implementing robust data governance and data quality frameworks is highly desirable. Strong leadership qualities, with the ability to mentor junior engineers, guide technical strategy, and collaborate effectively with cross-functional teams (data scientists, analysts, business stakeholders), are paramount.
This role offers a unique opportunity to shape the future of data architecture within a dynamic and growing company. You will be instrumental in architecting solutions that handle vast amounts of data, unlock new insights, and drive business value. The position involves a hybrid working model, allowing for flexibility while fostering team collaboration. You will be based in our state-of-the-art offices in London, England, UK , working with cutting-edge technologies and solving complex data challenges. Join a team committed to excellence and innovation in the heart of the tech industry.
Key Responsibilities:
- Design, build, and maintain scalable big data pipelines and infrastructure.
- Develop and optimize ETL/ELT processes for data ingestion and transformation.
- Manage and enhance data lakes and data warehouses.
- Ensure data quality, integrity, and reliability.
- Implement data governance and security best practices.
- Lead and mentor a team of data engineers.
- Collaborate with data scientists and analysts to meet their data needs.
- Evaluate and adopt new data technologies and tools.