218 Senior Data Engineer jobs in London
Big Data Engineer
Posted 13 days ago
Job Viewed
Job Description
RELOCATION ASSISTANCE: Relocation assistance may be available
CLEARANCE TYPE: UK-Highest Level of Government Clearance
TRAVEL: Yes, 10% of the Time
**Salary: £77,400 - £116,000**
**Define Possible at Northrop Grumman UK**
At Northrop Grumman UK, our mission is to solve the most complex challenges by shaping the technology and solutions of tomorrow. We call it Defining Possible.
This mind-set goes beyond our customer solutions; it's the foundation for your career development and the impact we have within the community. So, what's your possible?
**Opportunity:**
This is more than just a job; it's a mission.
As a Big Data Systems Engineer, you will be responsible for designing, implementing, and maintaining scalable big data systems that support our data science, machine learning, and AI workloads. You will work closely with cross-functional teams to ensure seamless data integration, security, and compliance with GDPR and privacy regulations. Your expertise in scripting, troubleshooting, and integration testing will be essential in optimizing our data pipelines and orchestration processes.
Our UK Cyber & Intelligence business combines modern software development approaches with a rich heritage and experience in the Defence and security sectors. Our customers have complex and sensitive data and information requirements requiring a mission partner who quickly understands the context, delivering and sustaining a portfolio of challenging technology projects at scale and pace, supporting them through an ambitious digital transformation programme.
If you are looking for a career with meaning where you can make a difference to the nation's security and support critical missions, then look no further.
_"My purpose; to lead a team of engineers with the brightest minds, to push the boundaries and define possible together."_
**Role responsibilities:**
+ Scripting and Automation: Develop and maintain scripts for automating data processes and workflows.
+ Troubleshooting: Diagnose and resolve technical issues related to big data systems, data pipelines, and integrations.
+ Integration Testing: Conduct thorough testing to ensure seamless integration of new tools and technologies into existing systems.
+ Linux Internals: Utilize in-depth knowledge of Linux internals to optimize performance and reliability of big data infrastructure.
+ Network Protocols: Apply understanding of TCP/IP and OSI models in the design and troubleshooting of networked systems.
+ Data Pipeline Support: Manage and optimize data pipelines and orchestration tools such as NiFi and Airflow.
+ Scalable System Design: Design scalable big data systems with a focus on security, GDPR compliance, and privacy
+ Hybrid Cloud Management: Leverage hybrid cloud experience to manage on-premise and AWS cloud resources effectively
+ Data Science Support: Support data science, machine learning, and AI workloads using tools like Jupyter, Spacy, Transformers, and NLTK.
+ Big Data Platforms: Utilize big data NoSQL engines and platforms such as Hive, Impala, and Elasticsearch for data storage and processing
+ BI and Visualization: Implement and support business intelligence and visualization tools like Tableau, Kibana, and PowerBI to provide actionable insights.
**We are looking for:**
+ Strong troubleshooting skills and the ability to diagnose and resolve complex technical issues.
+ Experience with integration testing and ensuring seamless tool integration.
+ In-depth knowledge of Linux internals and system administration.
+ Understanding of TCP/IP and OSI models.
+ Hands-on experience with data pipeline tools like NiFi and Airflow.
+ Proven ability to design scalable big data systems with a focus on security and GDPR compliance.
+ Hybrid cloud experience, specifically with on-premise and AWS cloud environments.
+ Familiarity with data science, machine learning, and AI tools such as Jupyter, Spacy, Transformers, and NLTK.
+ Experience with big data NoSQL engines/platforms such as Hive, Impala, and Elasticsearch.
+ Proficiency with business intelligence and visualization tools like Tableau, Kibana, and PowerBI.
+ Excellent communication and collaboration skills.
Preferred Qualifications:
+ Certification in AWS or other cloud platforms.
+ Experience with additional data orchestration tools.
+ Familiarity with other big data tools and technologies.
+ Previous experience in a similar role within a dynamic and fast-paced environment.
Experience in Cloudera, Hadoop, Cloudera Data Science Workbench (CDSW), Cloudera Machine Learning (CML) would be highly desirable.
**Work Environment:**
+ Full time on-site presence required
**If you don't meet every single requirement, we still encourage you to apply.**
Sometimes, people hesitate to apply because they can't tick every box. We encourage you to apply if you believe the role will suit you well, even if you don't meet all the criteria. You might be exactly who we are looking for, either for this position or for our opportunities at Northrop Grumman UK. We are on an exciting growth trajectory and growing our teams across the UK.
**Security clearance:**
You must hold the highest level of UK Government security clearance. Our requirement team is on hand to answer any questions and we will guide you through the process: .
**Benefits:**
We can offer you a range of flexible working options to suit you, including optional compressed working schedule with every other Friday off. Our benefits including private health care, cash health plan, holiday buy and sell, career development opportunities and performance bonuses. For a comprehensive list of benefits, speak to our recruitment team.
**Why join us?**
+ **A mission to believe in** **-** Every day we contribute to building a more secure and connected world, expanding our reach from land, sea, and air to space and cyberspace. From engineering data and intelligence solutions, to developing maritime navigation and control systems and innovating command and control systems for the UK and NATO, what we do together matters.
+ **A place to belong and thrive** **-** Every voice matters at our table meaning you can bring your authentic self to work. From our Employee Resource Groups backed by thousands of employees to our partnerships with the Association For Black and Minority Ethnic Engineers, Forces Transition Group, Mind, and Women in Defence - we are passionate about growing and supporting our inclusive community where everyone can belong.
+ **Your career, your way** - Shape your career journey with diverse roles, mentorship, and development opportunities that fuel your curiosity, channel your expertise, and nurture your passion. Looking for flexibility? Balance your professional career with your personal life through our health and wellbeing benefits, discount schemes, and investment in your future development. Speak to our team to find the balance that's right for you.
**Ready to apply?**
**Yes** - Submit your application online. Your application will be reviewed by our team and we will be in touch.
**Possibly, I'd like to find out more** **about this role** - Reach out to our team for more information and support: .
**No, I don't think this role is right for me** - Our extensive UK growth means we have exciting, new opportunities opening all the time. Speak to our team to discuss your career goals.
Northrop Grumman is committed to hiring and retaining a diverse workforce, and encourages individuals from all backgrounds and all abilities to apply and consider becoming a part of our diverse and inclusive workforce.
Data Engineer
Posted today
Job Viewed
Job Description
Data Engineer
I am working with a rapidly growing, category-defining technology company that is transforming how industrial organisations operate and compete through deep data integration, generative AI, and tailored enterprise workflows.
As part of their continued expansion, they are looking to bring on a Data Engineer to lead client delivery and embed their platform into global operations. This is a strategic, high-impact role where you will work directly with senior stakeholders to solve complex business challenges and drive measurable outcomes through data and AI.
You will join a collaborative, mission-driven team of elite technologists and business leaders, working at the intersection of cutting-edge technology and real-world impact.
As part of this role, you will be responsible for:
- Executing complex data integration projects using PySpark and distributed technologies
- Designing and implementing scalable generative AI workflows using modern AI infrastructure
- Collaborating with cross-functional teams to ensure successful delivery and adoption
- Driving continuous improvement and innovation across client engagements
To be successful in this role, you will have:
- Experience working in data engineering or data integration
- Strong technical skills in Python or PySpark
- Exposure to generative AI platforms or interest in building AI-powered workflows
- Ability to work closely with clients and lead delivery in fast-paced environments
- Exposure to Airflow, Databricks or DBT would be beneficial
Some of the package details include:
- Salary up to 75,000 per year
- Bonus and share options
- Vibrant central London office, working closely with the wider tech function (3 days in office, 2 from home)
This is just a brief overview of the role. For the full details, simply apply with your CV and I'll be in touch to discuss further.
Data Engineer
Posted today
Job Viewed
Job Description
Data Engineer
Posted 2 days ago
Job Viewed
Job Description
Data Engineer - London / Nationwide (Hybrid) - SC Eligible - 60,000
A leading digital consultancy is looking to hire a Data Engineer to support the expansion of its Data & AI team. This is an excellent opportunity for someone with up to 3 years of experience (or a strong academic background) to gain hands-on exposure to cloud technologies and grow in a collaborative, fast-paced environment.
What You'll Be Doing:
- Building and maintaining scalable data pipelines across Azure and GCP
- Working with tools such as Azure Data Factory, Snowflake, BigQuery, and Cloud Functions
- Supporting data warehousing and integration across diverse industry projects
- Collaborating with data professionals and contributing to cross-functional delivery teams
- Assisting in reporting and data visualisation (Power BI / Looker - nice to have)
What We're Looking For:
- 0-3 years of experience in a data engineering or similar role (graduates welcome)
- Proficiency in SQL and Python
- Exposure to any of the following:
- Azure Data Factory
- GCP BigQuery
- Snowflake
- Cloud Functions
- Strong communication and problem-solving skills
- Must be eligible for SC (Security Check) clearance (UK residency required)
What's On Offer:
- 60,000 base salary
- Hybrid working (London office base)
- Clear progression plan and technical mentorship
- Involvement in real-world data and digital transformation programmes
- Supportive, forward-thinking team culture
Data Engineer
Posted 2 days ago
Job Viewed
Job Description
Contract Role: Data Engineer - Azure Migration Project
Location: Hybrid (2 days/week onsite in London)
Contract Length: 6 months
Day Rate: 450/day (Outside IR35)
Start Date: ASAP
Sector: Consultancy
Overview
We are recruiting on behalf of a consultancy client for an experienced Data Engineer to support a strategic data migration project . The role involves migrating legacy data systems to a modern Azure-based architecture, with a focus on performance, scalability, and reliability.
Responsibilities
- Design and implement robust data migration pipelines using Azure Data Factory, Synapse Analytics, and Databricks
- Develop scalable ETL processes using PySpark and Python
- Collaborate with stakeholders to understand legacy data structures and ensure accurate mapping and transformation
- Ensure data quality, governance, and performance throughout the migration lifecycle
- Document technical processes and support knowledge transfer to internal teams
Required Skills
- Strong hands-on experience with Azure Data Factory, Synapse, Databricks, PySpark, Python, and SQL
- Proven track record in delivering data migration projects within Azure environments
- Ability to work independently and communicate effectively with technical and non-technical stakeholders
- Previous experience in consultancy or client-facing roles is advantageous
Data Engineer
Posted 3 days ago
Job Viewed
Job Description
Job Description:
Our tech stack
- AWS is a significant growth area for us with a diverse and growing capability and we are looking for a Data Engineer with experience in AWS cloud technologies for ETL pipeline, data warehouse and data lake design/building and data movement.
- AWS data and analytics services (or open-source equivalent) such as EMR, Glue, RedShift, Kinesis, Lambda, DynamoDB.
Essential requirements
- You thrive in problem-solving and analytical thinking
- You enjoy collaborating with multiple stakeholders in a fast-paced environment
- Experience in the design and deployment of production data pipelines from ingestion to consumption within a big data architecture, using Java, Python, Scala, Spark, SQL.
- Experience performing tasks such as writing scripts, extracting data using APIs, writing SQL queries etc.
- Experience in processing large amounts of structured and unstructured data, including integrating data from multiple sources through ingestion and curation functions on AWS cloud using AWS native or custom programming
Data Engineer
Posted 6 days ago
Job Viewed
Job Description
- We are seeking a skilled Data Engineer to design and develop robust data pipelines that support a cloud-based Data Analytics platform. This role plays a key part in transforming the data landscape and enhancing the Data Warehouse Engineering function.
- The successful candidate will contribute to a range of initiatives focused on improving data quality, accessibility, and performance.
- Design and maintain data pipelines, including error-handling, automation, and performance optimisation.
- Resolve data queries by tracing issues to source systems.
- Define and deliver cloud-based data analytics platform requirements using modern EDW tools and practices.
- Maintain clear and accurate team and technical documentation.
- Monitor and improve data quality, consistency, and compliance with standards.
- Apply operational procedures, security protocols, and support frameworks to meet service levels.
- Recommend development tools to improve delivery efficiency and value.
- Document systems, solutions, and processes for database and data engineering deliveries.
- Contribute to IT strategy and architecture development.
- Ensure solution designs comply with relevant policies, standards, and governance.
- Maintain strategic oversight across data engineering components.
- Identify and manage risks and issues proactively.
- Conduct technical quality checks with leads to ensure standards are met.
- Promote lean practices by challenging inefficiencies.
- Support the strategic direction of the IT function with a proactive and positive approach.
Be The First To Know
About the latest Senior data engineer Jobs in London !
Data Engineer
Posted 9 days ago
Job Viewed
Job Description
Data Engineer
I am working with a fast-growing, technology driven organisation that is redefining how data is used to drive decision making across regulated industries. Through advanced data engineering, cloud technologies and a strong focus on compliance they are enabling smarter, faster and more secure business operations.
As part of their continued growth, they are looking to bring on a Data Engineer to join their Data team. This is a high-impact, delivery focused role where you will work across business units to build scalable data solutions and support enterprise-wide transformation.
You will be part of a collaborative, forward thinking team that values innovation, quality, and continuous improvement.
In this role, you will be responsible for:
- Designing and delivering data solutions using Microsoft technologies
- Building and maintaining ETL pipelines across complex data environments
- Collaborating with Architects, Engineers, and cross-functional teams to develop solutions that meet business needs
- Participating in Agile ceremonies and contributing to code reviews
- Creating documentation and supporting the development lifecycle
To be successful in this role, you will have:
- Experience working on the Microsoft Platform e.g. SQL Server, SSIS, SSRS
- Strong data warehousing and data modelling experience
- Experience creating ETL pipelines (primarily using SSIS)
- Experience working in regulated environments
- Power BI experience for reporting would be beneficial
Some of the package details include:
- Salary of up to 60,000
- Fully remote working
- 25 days annual leave plus bank holiday
- Discretionary bonus scheme
- Supportive, inclusive culture with a focus on professional development
This is just a brief overview of the role. For the full details, simply apply with your CV and I'll be in touch to discuss it further.
Data Engineer
Posted 10 days ago
Job Viewed
Job Description
Contract Role: Data Engineer - Azure Migration Project
Location: Hybrid (2 days/week onsite in London)
Contract Length: 6 months
Day Rate: 450/day (Outside IR35)
Start Date: ASAP
Sector: Consultancy
Overview
We are recruiting on behalf of a consultancy client for an experienced Data Engineer to support a strategic data migration project . The role involves migrating legacy data systems to a modern Azure-based architecture, with a focus on performance, scalability, and reliability.
Responsibilities
- Design and implement robust data migration pipelines using Azure Data Factory, Synapse Analytics, and Databricks
- Develop scalable ETL processes using PySpark and Python
- Collaborate with stakeholders to understand legacy data structures and ensure accurate mapping and transformation
- Ensure data quality, governance, and performance throughout the migration lifecycle
- Document technical processes and support knowledge transfer to internal teams
Required Skills
- Strong hands-on experience with Azure Data Factory, Synapse, Databricks, PySpark, Python, and SQL
- Proven track record in delivering data migration projects within Azure environments
- Ability to work independently and communicate effectively with technical and non-technical stakeholders
- Previous experience in consultancy or client-facing roles is advantageous
Data Engineer
Posted 13 days ago
Job Viewed
Job Description
Data Engineer
We’re looking for a Data Engineer with strong Python skills and experience in event-driven systems to join our growing data team. This isn’t your typical “pipelines-for-analysts” role -you’ll be building real-time systems that power applications, tooling, and commercial products across the business.
What you’ll be doing:
-Build and maintain event-driven data pipelines that power Citywire’s Catalyst platform.
-Design resilient, fault-tolerant workflows using AWS services such as Lambda, Kinesis, SQS, DynamoDB Streams, and EventBridge.
-Implement processors that ensure data consistency across DynamoDB, PostgreSQL (Aurora), OpenSearch, and BigQuery.
-Modernise legacy batch processes into stream-first architectures.
-Build and integrate APIs to enable smooth publishing and consumption of events across systems.
-Collaborate with engineers on greenfield and existing projects, balancing speed with resilience.
-Take ownership of key pipelines and services, ensuring reliability, performance, and scalability.
-Share best practices and mentor others in event-driven data engineering.
What we’re looking for:
-Technical Skills: Proven experience in data engineering or backend development, with solid Python skills and hands-on use of AWS event-driven services.
-Event-Driven Knowledge: Understanding of DLQs, retries, buffering, idempotency, and resilient design patterns.
-Cloud & CI/CD Experience: Familiarity with Terraform, Git-based workflows, and cloud-native deployments.
-Database Skills: Experience with SQL and NoSQL databases such as PostgreSQL, DynamoDB, or OpenSearch.
-Problem-Solver: Comfortable working in Linux environments and confident debugging logs, scripts, and production issues.
-Additional Skills: Exposure to Kafka, Spark, or dbt Core, with an interest in domain-driven data contracts.
Meet Citywire
We cover - and connect - all sides of the $100 trillion global asset management industry - through our news, events and insights.
At Citywire, we uphold a culture rooted in honesty, integrity , and fairness , where every voice is valued and heard. Our culture promotes constructive dialogue and collaboration on a global scale.
Join the team at the Heart of Wealth.
Our perks:
-Generous holiday entitlement: Start with 25 days per annum, increasing to 28 days after three years' service, and 30 days after five years' service, in addition to bank holidays.
-Flexible working options.
-£480 annual allowance for well-being activities or gym memberships, with assistance available for monthly or annual costs.
-Eye-test and glasses allowance.
-Competitive private pension scheme.
-Critical illness cover and group life assurance from day one of employment.
-Well-being support: Access to an independent Employee Assistance Programme, available 24/7.
-Cycle to work scheme and annual travel card loans.
-Techscheme: Purchase the latest tech through our employer scheme, spreading the cost over 12 months with National Insurance savings.
-After two years of continuous service, access group income protection, private medical, and dental insurance.
Citywire is an equal opportunities employer.