234 Lead Data Engineer jobs in London

Ready to join a team that's leading the way in reshaping the future of insurance? Here at esure Group, we are on a mission to revolutionise insurance for good!

We’ve been providing Home and Motor Insurance since 2000, with over 2 million customers trusting us to keep them covered through our esure and Sheilas’ Wheels brands. With a bold dedication for digital innovation, we're transforming the way the industry operates and putting customers at the heart of everything we do.

Having completed our recent multi-year digital transformation, we’re now leveraging advanced technology and data-driven insights alongside exceptional service, to deliver personalised experiences that meet our customers ever-changing needs today and in the future.

We are currently looking for a Lead Data Engineer to join our forward thinking Data department.

You’ll work in a team of data engineers, analytics engineers, data scientists and AI specialists to design and evolve scalable data platforms and modern data products that enable self-service analytics, advanced modelling, and AI-driven decision-making across our insurance business.

What you’ll do:

• Lead on data engineering projects, managing a sub- team within engineering, building out our data warehouse & pipelines
• Collaborate effectively across data science, analytics, product, engineering and commercial business teams
• Build and support esure’s data products within our industry leading platform
• Work with product managers, data & AI engineers to deliver technical solutions for our most pressing data problems including GenAI applications
• Integrate data from a variety of sources, assuring that they adhere to data quality and technical standards. Migrating legacy systems into a modern scalable platform
• Creating frameworks and processes for data pipelines across the data and analytics platform
• Improve data engineering processes and roll these out across our team and wider data community
• Set coding standards across the platform, naming conventions across data products, championing and enforcing both.
• Work with architects on best design for data products, evaluating and experimenting with new data tools & supporting ML & AI infrastructure and workflows

What we’d love you to bring:

• Strategic thinker, aligning multiple workstreams to deliver a scalable, high-quality data platform.
• Develops the data function roadmap with thought leadership on tooling, technology, and guidelines.
• Product-minded leader, solving customer and business data challenges.
• Culture builder, driving continuous improvement and operational excellence.
• Deep expertise in data compliance frameworks, cost management, and platform optimisation.
• Strong hands-on experience with modern cloud data warehouses (Databricks, Snowflake, AWS), SQL, Spark, Airflow, Terraform.
• Advanced Python skills with orchestration tooling; solid experience in CI/CD (Git, Jenkins).
• Proven track record in data modelling, batch/real-time integration, and large-scale data engineering.
• Exposure to deploying Generative AI in production environments.

What’s in it for you?:

• Competitive salary that reflects your skills, experience and potential.
• Discretionary bonus scheme that recognises your hard work and contributions to esure’s success.
• 25 days annual leave, plus 8 flexible days and the ability to buy and sell further holiday.
• Our flexible benefits platform is loaded with perks to choose from, so you can build a personal toolkit to support your health, wellbeing, lifestyle, and finances.
• Company funded private medical insurance for qualifying colleagues.
• Fantastic discounts on our insurance products! 50% off for yourself and spouse/partner and 10% off for direct family members.
• We’ll elevate your career with hands-on training, mentoring, access to our exclusive academies, regular career conversations, and expert partner resources.
• Driving good in the world couldn’t be more important to us. Our colleagues can use 2 volunteering days per year to support their local communities.
• Join our internal networks and communities to connect, learn, and share ideas with likeminded colleagues.
• We’re a proud supporter of the ABI’s ‘Make Flexible Work’ campaign and welcome you to ask about the flexibility you need. Our hybrid working approach also puts you in the driving seat of how and where you do your best work.
• And much more; See a full overview of our benefits here Reward and benefits | Esure Group PLC

We are committed to creating an inclusive and diverse workplace where everyone feels valued, respected, and empowered. We celebrate individuality and create spaces where unique backgrounds and experiences can come together. We believe that diverse perspectives drive innovation, in turn enabling us to better serve our customers, community and build a stronger organisation. Our commitment to inclusion extends to every part of our business, from hiring practices to professional growth opportunities, ensuring equal access and support for all.

Lyst is a global Fashion Tech company and premium shopping app, founded in London in 2010 and catering to over 200M shoppers per year. We offer our customers the largest assortment of premium & luxury fashion items & products in one place, via an assortment of 8.5M+ items from over 17,000 of the world’s leading brands. We are a scale-up business with a current team size of c. 160 people in London, combining an agile mentality with a proven business model and over a decade of experience. This provides a balance between foundations and structure, and autonomy and pace.

At Lyst we obsess over the customer, providing a search & discovery experience which offers inspiration, fulfilment, and personalisation. We believe that fashion is amazing but shopping for fashion often isn’t, and use our technology, data and creativity to bring more joy, greater choice and fewer fails. Our mission is to help fashion shoppers make better decisions and help fashion partners find better audiences as the category-leading destination for every fashion shopper. Lyst has raised over $160m from leading investors including Accel, Balderton, Molten Ventures, Fidelity International, and LVMH.

We’re looking for a Lead Data Platform Engineer to join our central Analytics Engineering team, and help drive the Data Engineering discipline within the Data Platform chapter. This is a great opportunity for someone who wants to have impact from day one and make their mark as an expert in a high performing, collaborative team.

We record nearly 100 million lines of data daily, and we are looking for a senior engineer to support and maintain a data platform that is scalable, reliable and enables data-led decision making business-wide.

As a data engineer within the central data platform function, you will maintain the data ingestion infrastructure in AWS for data owners across Lyst to leverage and to ensure the data created is of high quality. You will identify, design and implement improvements to these processes, including data contracts and testing. This role will work closely with our engineers, analytics engineers and analysts to help us successfully deliver our data platform and data strategy.

Key responsibilities:

• Maintain our data ingestion infrastructure, and liaise with Engineering data owners on the infrastructure and standards
• Identify, design and implement process improvements for data ingestion and testing
• Create documentation and guidelines on building reliable, scalable data pipelines
• Own our data contract capabilities, and develop guidelines for data owners
• Advocate data quality across the organisation
• Support data owners on data ingestion, for new and existing data pipelines
• Work closely with Analytics Engineering to bridge the gap between data creation and transformation
• Support in delivering our data strategy
• We work mainly in Python and SQL, running on a range of AWS technologies such as S3, EKS, SQS, Firehose, Kinesis & Lambda and non-AWS tools such as Snowflake, dbt, Looker, CircleCI, Docker, Terraform & GitHub.

Requirements

• Experience with Python, AWS data tooling, and software fundamentals (SQL, Git)
• Experience with data processing & testing in the cloud, with a large amount of data
• Excellent Python knowledge and experience and up to date on best Python practices, as well as experience writing efficient SQL
• Experience working on full data engineering projects, from design to implementation
• Experience iterating & refactoring existing code as well as writing new code
• Communication: You are able to communicate clearly and be humble when sharing ideas and solutions. You can communicate with both technical and non-technical stakeholders
• You strive to write clear, well documented, tested code, and look to improve existing code and ways of working. When you find an opportunity, you’re are excited to go for it
• (Bonus) Experience with data contract implementation

Things that matter to us:

• You are curious at heart and like to take ownership of something to make it better
• You are detail oriented
• You are a team player and communicate with your peers and other stakeholders in the company on a day to day basis
• Being confronted with a difficult or strange problem makes you feel like a detective that wants to crack the mystery
• You have a sense of ownership over products, features and services the team looks after

Benefits

• Our Ways of Working: We come into the office between 2-4 days a week and we're always in on Tuesdays and Thursdays . We strongly believe that in-person collaboration and time spent together as a team allows us to deliver high impact work, but the flexibility to also work remotely supports our diverse team.
• Time Off : In addition to the 8 statutory bank holidays, you will receive 29 holidays per year. Lyst’s holiday year runs from 1 April to 31 March.
• Competitive Family Leave Package : This includes Enhanced Family Leave for those eligible, paid Time off for Dependents and Support for Fertility Treatment & Loss.
• Clothing Benefit: We want you to enjoy using the Lyst app and site as much as our customers, so we provide you with a clothing allowance to use on Lyst every year. This starts at £250 when you join and increases up to £,000 with your length of service
• Private Healthcare: Our healthcare provider is Vitality. Your health is important to us which is why we offer all employees a comprehensive healthcare scheme from the day you start.
• Training Allowance : We’re big on continuous learning and growth, so all employees are currently entitled to an annual training allowance of £1 00. This can be used to attend conferences, industry events, training courses and to purchase resources.
• Pension Scheme: Our pension provider is The People’s Pension. We offer a minimum employee contribution of 5% and 3% employer contribution.
• Eye Tests and Vouchers: Employees can make a saving on their eye test and glasses through our chosen provider. You’ll receive a free eye test every year and a discount towards glasses.
• Cycle-to-Work Scheme : Lyst will purchase a bicycle from your chosen retailer, you will then receive a voucher to pick up your bicycle from them.
• Transport Season Ticket Loan : Employees can apply for an interest free season ticket loan to support your travel to work.
• Social Events : Frequent company wide social events including Christmas & summer parties, sports days, themed drinks, quizzes, cook alongs, as well as smaller team socials. We also have plenty of interest based groups such as football, running club, book club, culinary and more.

We want to build a world where fashion works for everyone, and we want teams that are just as inclusive. Diversity and inclusion is an integral part of our culture at Lyst. We recognise and celebrate the value and impact diversity brings to our company and are committed to ensuring this is a consistent focus, for which we are held to account. We are committed to treating all applicants fairly and equally, and encourage candidates from all backgrounds to apply for this role. We are happy to talk about flexible working arrangements.

Don’t meet every single requirement? Studies have shown that women and people of color are less likely to apply to jobs unless they meet every single qualification. At Lyst we are dedicated to building a diverse, inclusive and authentic workplace, so if you’re excited about this role but your past experience doesn’t align perfectly with every qualification in the job description, we encourage you to apply anyway. You may be just the right candidate for this or other roles.

Lead AWS Data Engineer / Architect - Databricks - London

I'm working with a globally renowned financial services client that are looking for a seasons Data professional. My client are seen as leaders and pioneers within their relative field but also, a very well known house hold name. They have operations in near enough every country and despite having such a huge presence globally, they still pride themselves on having their employees at the heart of every bit of success.
This has lead them onto winning multiple awards such as being named in the top 100 best companies to work for along side many other outstanding accolades.

This is a hands on technical role. The successful applicant will be building and maintaining AWS Data pipelines and infrastructure. This is while working with cross functional teams to design best in Class data solutions across the business. As a business, they are very mature within their data services however, there is always room for improvements so we're looking for an expert to not look to be involved in new processes but equally, help to reverse engineer the existing platform to increase efficiency and performance.

This is a salaried position paying up to 130k as a base salary. It's hybrid working with 2/3 days in office, central London. My client pride themselves on having their employees at the heart of everything that they do and their success has come from their outstanding workforce. They like to return the favour by offering unparalleled career progression opportunities along side training courses and certifications.

Key Requirements

• Proven experience with AWS services and tools.
• Strong knowledge of data modeling and ETL processes.
• Proficiency in programming languages such as Python or SQL.
• Excellent problem-solving skills with a proactive approach.
• Ability to communicate effectively within a team.

If you are a skilled and driven AWS Data Engineer looking to make an impact, get in touch ASAP as interviews are already taking place. Don't miss out!

Key Skills: AWS, Data, Architecture, Data Engineering, Data Warehousing, Data Lakes, Databricks, Glue, Pyspark, Athena, Python, SQL, Machine Learning, London

AWS Data Engineer - VP Level

Location: London (Canary Wharf)
Salary: (phone number removed) - (phone number removed)

A leading financial institution is seeking a Senior AWS Data Engineer to join a greenfield initiative focused on transforming market data access and utilization. This VP-level role offers strategic influence, technical leadership, and cross-functional collaboration across business units.

Key Responsibilities:
* Design and maintain scalable data pipelines, warehouses, and lakes using AWS technologies.
* Implement robust data governance, quality, and security practices.
* Collaborate with data scientists to deploy machine learning models.
* Lead or contribute to strategic planning, risk management, and stakeholder engagement.
* Drive innovation through advanced analytics and research-based problem solving.

To be successful you should have:

• 10 years hands-on experience in AWS data engineering technologies, including Glue, PySpark, Athena, Iceberg, Databricks, Lake Formation, and other standard data engineering tools.
• Previous experience in implementing best practices for data engineering, including data governance, data quality, and data security.
• Proficiency in data processing and analysis using Python and SQL.
• Experience with data governance, data quality, and data security best practices.
• Strong knowledge of market data and its applications.

Some other highly valued skills may include:

• Experience with other data engineering tools and technologies.
• Knowledge of machine learning and data science concepts.
• Familiarity with Barclays' data strategy and practices.

Please send me your CV is you meet these requirements and we can then arrange call