248 Senior Data Engineer jobs in London
Big Data Engineer
Posted 13 days ago
Job Viewed
Job Description
RELOCATION ASSISTANCE: Relocation assistance may be available
CLEARANCE TYPE: UK-Highest Level of Government Clearance
TRAVEL: Yes, 10% of the Time
**Salary: £77,400 - £116,000**
**Define Possible at Northrop Grumman UK**
At Northrop Grumman UK, our mission is to solve the most complex challenges by shaping the technology and solutions of tomorrow. We call it Defining Possible.
This mind-set goes beyond our customer solutions; it's the foundation for your career development and the impact we have within the community. So, what's your possible?
**Opportunity:**
This is more than just a job; it's a mission.
As a Big Data Systems Engineer, you will be responsible for designing, implementing, and maintaining scalable big data systems that support our data science, machine learning, and AI workloads. You will work closely with cross-functional teams to ensure seamless data integration, security, and compliance with GDPR and privacy regulations. Your expertise in scripting, troubleshooting, and integration testing will be essential in optimizing our data pipelines and orchestration processes.
Our UK Cyber & Intelligence business combines modern software development approaches with a rich heritage and experience in the Defence and security sectors. Our customers have complex and sensitive data and information requirements requiring a mission partner who quickly understands the context, delivering and sustaining a portfolio of challenging technology projects at scale and pace, supporting them through an ambitious digital transformation programme.
If you are looking for a career with meaning where you can make a difference to the nation's security and support critical missions, then look no further.
_"My purpose; to lead a team of engineers with the brightest minds, to push the boundaries and define possible together."_
**Role responsibilities:**
+ Scripting and Automation: Develop and maintain scripts for automating data processes and workflows.
+ Troubleshooting: Diagnose and resolve technical issues related to big data systems, data pipelines, and integrations.
+ Integration Testing: Conduct thorough testing to ensure seamless integration of new tools and technologies into existing systems.
+ Linux Internals: Utilize in-depth knowledge of Linux internals to optimize performance and reliability of big data infrastructure.
+ Network Protocols: Apply understanding of TCP/IP and OSI models in the design and troubleshooting of networked systems.
+ Data Pipeline Support: Manage and optimize data pipelines and orchestration tools such as NiFi and Airflow.
+ Scalable System Design: Design scalable big data systems with a focus on security, GDPR compliance, and privacy
+ Hybrid Cloud Management: Leverage hybrid cloud experience to manage on-premise and AWS cloud resources effectively
+ Data Science Support: Support data science, machine learning, and AI workloads using tools like Jupyter, Spacy, Transformers, and NLTK.
+ Big Data Platforms: Utilize big data NoSQL engines and platforms such as Hive, Impala, and Elasticsearch for data storage and processing
+ BI and Visualization: Implement and support business intelligence and visualization tools like Tableau, Kibana, and PowerBI to provide actionable insights.
**We are looking for:**
+ Strong troubleshooting skills and the ability to diagnose and resolve complex technical issues.
+ Experience with integration testing and ensuring seamless tool integration.
+ In-depth knowledge of Linux internals and system administration.
+ Understanding of TCP/IP and OSI models.
+ Hands-on experience with data pipeline tools like NiFi and Airflow.
+ Proven ability to design scalable big data systems with a focus on security and GDPR compliance.
+ Hybrid cloud experience, specifically with on-premise and AWS cloud environments.
+ Familiarity with data science, machine learning, and AI tools such as Jupyter, Spacy, Transformers, and NLTK.
+ Experience with big data NoSQL engines/platforms such as Hive, Impala, and Elasticsearch.
+ Proficiency with business intelligence and visualization tools like Tableau, Kibana, and PowerBI.
+ Excellent communication and collaboration skills.
Preferred Qualifications:
+ Certification in AWS or other cloud platforms.
+ Experience with additional data orchestration tools.
+ Familiarity with other big data tools and technologies.
+ Previous experience in a similar role within a dynamic and fast-paced environment.
Experience in Cloudera, Hadoop, Cloudera Data Science Workbench (CDSW), Cloudera Machine Learning (CML) would be highly desirable.
**Work Environment:**
+ Full time on-site presence required
**If you don't meet every single requirement, we still encourage you to apply.**
Sometimes, people hesitate to apply because they can't tick every box. We encourage you to apply if you believe the role will suit you well, even if you don't meet all the criteria. You might be exactly who we are looking for, either for this position or for our opportunities at Northrop Grumman UK. We are on an exciting growth trajectory and growing our teams across the UK.
**Security clearance:**
You must hold the highest level of UK Government security clearance. Our requirement team is on hand to answer any questions and we will guide you through the process: .
**Benefits:**
We can offer you a range of flexible working options to suit you, including optional compressed working schedule with every other Friday off. Our benefits including private health care, cash health plan, holiday buy and sell, career development opportunities and performance bonuses. For a comprehensive list of benefits, speak to our recruitment team.
**Why join us?**
+ **A mission to believe in** **-** Every day we contribute to building a more secure and connected world, expanding our reach from land, sea, and air to space and cyberspace. From engineering data and intelligence solutions, to developing maritime navigation and control systems and innovating command and control systems for the UK and NATO, what we do together matters.
+ **A place to belong and thrive** **-** Every voice matters at our table meaning you can bring your authentic self to work. From our Employee Resource Groups backed by thousands of employees to our partnerships with the Association For Black and Minority Ethnic Engineers, Forces Transition Group, Mind, and Women in Defence - we are passionate about growing and supporting our inclusive community where everyone can belong.
+ **Your career, your way** - Shape your career journey with diverse roles, mentorship, and development opportunities that fuel your curiosity, channel your expertise, and nurture your passion. Looking for flexibility? Balance your professional career with your personal life through our health and wellbeing benefits, discount schemes, and investment in your future development. Speak to our team to find the balance that's right for you.
**Ready to apply?**
**Yes** - Submit your application online. Your application will be reviewed by our team and we will be in touch.
**Possibly, I'd like to find out more** **about this role** - Reach out to our team for more information and support: .
**No, I don't think this role is right for me** - Our extensive UK growth means we have exciting, new opportunities opening all the time. Speak to our team to discuss your career goals.
Northrop Grumman is committed to hiring and retaining a diverse workforce, and encourages individuals from all backgrounds and all abilities to apply and consider becoming a part of our diverse and inclusive workforce.
Data Engineer
Posted 5 days ago
Job Viewed
Job Description
Overview
GlaxoSmithKline (GSK) is a science-led global healthcare company with a special purpose: to help people do more, feel better, live longer. We are on an audacious journey to impact the health of 2.5 billion people over the next decade. Our R&D division is at the forefront of this mission, dedicated to the discovery and development of groundbreaking vaccines and medicines. We are transforming the landscape of medical research by integrating cutting-edge science and technology and harnessing the power of genetics and new data. By fostering a collaborative environment that unites the talents of our people, we are revolutionizing R&D to pre-empt and defeat diseases. Join us in our commitment to uniting science, technology, and talent to get ahead of disease together.
Position Summary
At GSK we see a world in which advanced applications of Machine Learning and AI will allow us to develop novel therapies to existing diseases and to quickly respond to emerging or changing diseases with personalized drugs, driving better outcomes at reduced cost with fewer side effects. It is an ambitious vision that will require the development of products and solutions at the cutting edge of Machine Learning and AI. We’re looking for a highly skilled Backend Engineer to help us make this vision a reality.
The ideal candidate will have a track record of shipping data products derived from complex sources, responsible for the process from conceptual data pipelines to production scale. We have a commitment to quality, so the person in this role will be able to use modern cloud tooling and techniques to deliver reliable data pipelines and continuously improve them.
This role requires a passion for solving challenging problems aligned to exciting Artificial Intelligence and Machine Learning applications. Educational or professional background in the biological sciences is a plus but is not necessary; passion to help therapies for new and existing diseases, and a pattern of continuous learning and development is mandatory.
Key Responsibilities
- Build data pipelines using modern data engineering tools on Google Cloud: Python, Spark, SQL, BigQuery, Cloud Storage.
- Ensure data pipelines meet the specific scientific needs of data consuming applications.
- Responsible for high quality software implementations according to best practices, including automated test suites and documentation.
- Develop, measure, and monitor key metrics for all tools and services and consistently seek to iterate on and improve them.
- Participate in code reviews, continuously improving personal standards as well as the wider team and product.
- Liaise with other technical staff and data engineers in the team and across allied teams, to build an end-to-end pipeline consuming other data products.
Minimum Requirements
- 2+ years of data engineering experience with a Bachelors degree in a relevant field (including computational, numerate or life sciences), or equivalent experience.
- Cloud experience (e.g. Google Cloud preferred).
- Strong skills with industry experience in Python and SQL.
- Unit testing experience (e.g. pytest).
- Knowledge of agile practices and able to perform in agile software development environments.
- Strong experience with modern software development tools / ways of working (e.g. git/GitHub, DevOps tools for deployment).
Preferred Requirements
- Demonstrated experience with biological or scientific data (e.g. genomics, transcriptomics, proteomics), or pharmaceutical industry experience.
- Bioinformatics expertise, familiarity with large scale bioinformatics datasets.
- Experience using Nextflow pipelines.
- Knowledge of NLP techniques and experience of processing unstructured data, using vector stores, and approximate retrieval.
- Familiarity with orchestration tooling (e.g. Airflow or Google Workflows).
- Experience with AI/ML powered applications.
- Experience with Docker or containerized applications.
Why GSK?
Uniting science, technology and talent to get ahead of disease together.
GSK is a global biopharma company with a special purpose – to unite science, technology and talent to get ahead of disease together – so we can positively impact the health of billions of people and deliver stronger, more sustainable shareholder returns – as an organisation where people can thrive. We prevent and treat disease with vaccines, specialty and general medicines. We focus on the science of the immune system and the use of new platform and data technologies, investing in four core therapeutic areas (infectious diseases, HIV, respiratory/ immunology and oncology).
Our success absolutely depends on our people. While getting ahead of disease together is about our ambition for patients and shareholders, it’s also about making GSK a place where people can thrive. We want GSK to be a place where people feel inspired, encouraged and challenged to be the best they can be. A place where they can be themselves – feeling welcome, valued, and included. Where they can keep growing and look after their wellbeing. So, if you share our ambition, join us at this exciting moment in our journey to get Ahead Together.
Inclusion at GSK
GSK is an Equal Opportunity/Affirmative Action Employer. All qualified applicants will receive equal consideration for employment without regard to race, color, national origin, religion, sex, pregnancy, marital status, sexual orientation, gender identity/expression, age, disability, genetic information, military service, covered/protected veteran status or any other federal, state or local protected class.
If you need any adjustments in the recruitment process, please get in touch with our Recruitment team ( ) to further discuss this today.
Important notice to employment businesses/agencies
GSK does not accept referrals from employment businesses and/or employment agencies in respect of the vacancies posted on this site. All employment businesses/agencies are required to contact GSK's commercial and general procurement/human resources department to obtain prior written authorization before referring any candidates to GSK. The obtaining of prior written authorization is a condition precedent to any agreement (verbal or written) between the employment business/ agency and GSK. In the absence of such written authorization being obtained any actions undertaken by the employment business/agency shall be deemed to have been performed without the consent or contractual agreement of GSK. GSK shall therefore not be liable for any fees arising from such actions or any fees arising from any referrals by employment businesses/agencies in respect of the vacancies posted on this site.
Please note that if you are aUS Licensed Healthcare Professional or Healthcare Professional as defined by the laws of the state issuing your license, GSK may be required to capture and report expenses GSK incurs, on your behalf, in the event you are afforded an interview for employment. This capture of applicable transfers of value is necessary to ensure GSK’s compliance to all federal and state US Transparency requirements. For more information, please visit GSK’s Transparency Reporting For the Record site.
Data Engineer
Posted today
Job Viewed
Job Description
Data Engineer
Croydon (Must be London-based, able to work from site in Croydon/London)
SC Clearance required / or eligible for clearance
12 months
Umbrella only
The required skills are:
- Attention to detail and ability to follow defined processes
- Drive and commitment to learn new technical concepts quickly
- Familiarity with Agile & DevOps ways of working
- Familiarity with and experience of using UNIX Knowledge of CI toolsets
- Good client facing skills and problem solving aptitude
- DevOps knowledge of SQL Oracle DB Postgres ActiveMQ Zabbix Ambari Hadoop Jira Confluence BitBucket ActiviBPM Oracle SOA Azure SQLServer IIS AWS Grafana Oracle BPM Jenkins Puppet CI and other cloud technologies.
All profiles will be reviewed against the required skills and experience. Due to the high number of applications we will only be able to respond to successful applicants in the first instance. We thank you for your interest and the time taken to apply!
Data Engineer
Posted 2 days ago
Job Viewed
Job Description
Data Engineer
Location: Hammersmith - Hybrid after probation
Salary: 38,000 -45,000
Join Our Team as a Data Engineer!
About Us : Join a forward-thinking organisation dedicated to revolutionising retail through innovative solutions and operations. We leverage physical spaces to test new concepts and pilot initiatives, helping brands and retailers thrive in a dynamic marketplace.
What You'll Do :
As our Data Engineer, you will play a pivotal role in the day-to-day management and maintenance of our data infrastructure, driving our large-scale programmes in the USA. Your primary responsibilities will include :
- Design & Development: Create, develop, and maintain scalable data pipelines and ETL processes across various systems.
- Data Integration: Build and manage seamless data integrations from diverse sources, including POS systems, e-commerce platforms, RFID sensors, IoT devices, and CRM systems.
- Data optimisation: Develop and optimise data models to support reporting, analytics, and machine learning initiatives.
- Data Quality Assurance: Ensure data accuracy, consistency, security, and compliance with U.S. data privacy standards (CCPA, HIPAA).
- Monitoring & Troubleshooting: Monitor, troubleshoot, and enhance data pipelines to maximise reliability and performance.
- Collaboration: Work closely with teams to deliver datasets that empower insight and decision-making.
- Strategic Contribution: Contribute to our data strategy by evaluating and implementing modern data tools and practises.
- Documentation: Document data flows, schemas, and processes to support knowledge sharing and scalability.
About You :
To thrive in this role, you will need :
- Strong knowledge of Python and APIs for effective data integration.
- Familiarity with cloud platforms, preferably Azure (AWS, GCP experience is also valuable).
- Experience managing large and complex datasets, with a strong command of SQL for cloud-based environments (Fabric, Snowflake, BigQuery, Redshift, etc.).
- A solid understanding of data modelling techniques (star schema, data vault, dimensional modelling).
- Proficiency in Excel-based data workflows for various Agile Retail projects.
- Hands-on experience with data pipeline orchestration tools (Airflow, dbt, Prefect, or similar).
Benefits:
- Unlimited holiday
- Annual Wellbeing Allowance
- Flexible work culture
- Monthly socials and events
- Complimentary snack bar
- Employer pension contribution
If you're a data enthusiast ready to innovate and drive impactful solutions in retail, we want to hear from you!
Office Angels is an employment agency and business. We are an equal-opportunities employer who puts expertise, energy and enthusiasm into improving everyone's chance of being part of the workplace. We respect and appreciate people of all ethnicities, generations, religious beliefs, sexual orientations, gender identities, abilities and more. By showcasing talents, skills and unique experiences in an inclusive environment, we help individuals thrive. If you require reasonable adjustments at any stage, please let us know and we will be happy to support you.
Office Angels acts as an employment agency for permanent recruitment and an employment business for the supply of temporary workers. Office Angels UK is an Equal Opportunities Employer.
By applying for this role your details will be submitted to Office Angels. Our Candidate Privacy Information Statement explaining how we will use your information is available on our website.
Data Engineer
Posted 3 days ago
Job Viewed
Job Description
Data Engineer
12 months
Remote
470 per day inside IR35 - Umbrella only
Active SC clearance and eligible candidates will be considered
Project description
Our client playing a crucial role in the ESN project in 2025, which aims to modernize and enhance communication for frontline emergency services in the UK. This initiative will greatly enhance the capabilities of emergency services, allowing them to share vital data and information quickly and securely, improving response times and overall public safety.
Required Skills
- Data Stage, RedShift, QuickSight, S3
- data migration/ ETL both batch and real time
- data warehouse development
- DevSecOps
- Java
- SQL
- relational databases
- GitHub/lab experience
Nice to have:
- Data quality
- xml
- AWS Data Specialty certification
All profiles will be reviewed against the required skills and experience. Due to the high number of applications we will only be able to respond to successful applicants in the first instance. We thank you for your interest and the time taken to apply!
Data Engineer
Posted 9 days ago
Job Viewed
Job Description
My client within Venture capital is looking for a data engineer to join their team.
The role will be working maintaining & modernising data pipelines.
Requirements
- Python
- Azure, CI/CD
- Airflow, DBT
- Docker
- GitHub, Azure DevOps
- LLM Experience desirable
Contract: 12 Months
Rate: (Apply online only) Via Umbrella
Location: London - 3 days per week in the office.
If this role is of interest, please reply with your up-to-date CV and I will be in touch to discuss further.
Thanks,
Please click here to find out more about our Key Information Documents. Please note that the documents provided contain generic information. If we are successful in finding you an assignment, you will receive a Key Information Document which will be specific to the vendor set-up you have chosen and your placement.
To find out more about Huxley, please visit (url removed)
Huxley, a trading division of SThree Partnership LLP is acting as an Employment Business in relation to this vacancy | Registered office | 8 Bishopsgate, London, EC2N 4BQ, United Kingdom | Partnership Number | OC(phone number removed) England and Wales
Data Engineer
Posted 9 days ago
Job Viewed
Job Description
If you are an experienced Data Engineer with excellent communication skills and a proven track record of working on large scale data enablement projects, we have a new contract we would like to discuss with you.
Please note this role requires onsite attendance once a week and has been deemed inside IR35.
Requirement:
- Experience in Azure Synapse, ETL, Pyspark, SQL , data Modelling and Data Bricks.
- Design and develop Azure Pipelines including data transformation and data cleansing
- Document source-to-target mappings
- Re-engineer manual data flows to enable scaling and repeatable use
- Build accessible datasets for distribution and analysis
- Development of Azure Pipelines for transforming data
- A scalable meta-data driven ingestion and transformation framework
- Aligning transformation pipelines and datasets with Purview
- Experience of working with Azure Data Lakes, Data warehousing, and pipelines and Storage accounts
- Experience in building and managing Pipelines including: building data interfaces to source systems, combining and transforming data into appropriate storage formats
- Experience identifying and resolving issues in databases, data processes, data products and services.
- Proficiency in T-SQL and Python to develop automation
Be The First To Know
About the latest Senior data engineer Jobs in London !
Data Engineer
Posted 11 days ago
Job Viewed
Job Description
Contract Role: Data Engineer - Azure Migration Project
Location: Hybrid (2 days/week onsite in London)
Contract Length: 6 months
Day Rate: 450/day (Outside IR35)
Start Date: ASAP
Sector: Consultancy
Overview
We are recruiting on behalf of a consultancy client for an experienced Data Engineer to support a strategic data migration project . The role involves migrating legacy data systems to a modern Azure-based architecture, with a focus on performance, scalability, and reliability.
Responsibilities
- Design and implement robust data migration pipelines using Azure Data Factory, Synapse Analytics, and Databricks
- Develop scalable ETL processes using PySpark and Python
- Collaborate with stakeholders to understand legacy data structures and ensure accurate mapping and transformation
- Ensure data quality, governance, and performance throughout the migration lifecycle
- Document technical processes and support knowledge transfer to internal teams
Required Skills
- Strong hands-on experience with Azure Data Factory, Synapse, Databricks, PySpark, Python, and SQL
- Proven track record in delivering data migration projects within Azure environments
- Ability to work independently and communicate effectively with technical and non-technical stakeholders
- Previous experience in consultancy or client-facing roles is advantageous
Data Engineer
Posted 11 days ago
Job Viewed
Job Description
Contract Role: Data Engineer - Azure Migration Project
Location: Hybrid (2 days/week onsite in London)
Contract Length: 6 months
Day Rate: 450/day (Outside IR35)
Start Date: ASAP
Sector: Consultancy
Overview
We are recruiting on behalf of a consultancy client for an experienced Data Engineer to support a strategic data migration project . The role involves migrating legacy data systems to a modern Azure-based architecture, with a focus on performance, scalability, and reliability.
Responsibilities
- Design and implement robust data migration pipelines using Azure Data Factory, Synapse Analytics, and Databricks
- Develop scalable ETL processes using PySpark and Python
- Collaborate with stakeholders to understand legacy data structures and ensure accurate mapping and transformation
- Ensure data quality, governance, and performance throughout the migration lifecycle
- Document technical processes and support knowledge transfer to internal teams
Required Skills
- Strong hands-on experience with Azure Data Factory, Synapse, Databricks, PySpark, Python, and SQL
- Proven track record in delivering data migration projects within Azure environments
- Ability to work independently and communicate effectively with technical and non-technical stakeholders
- Previous experience in consultancy or client-facing roles is advantageous
Data Engineer
Posted 11 days ago
Job Viewed
Job Description
Contract Role: Data Engineer - Azure Migration Project
Location: Hybrid (2 days/week onsite in London)
Contract Length: 6 months
Day Rate: 450/day (Outside IR35)
Start Date: ASAP
Sector: Consultancy
Overview
We are recruiting on behalf of a consultancy client for an experienced Data Engineer to support a strategic data migration project . The role involves migrating legacy data systems to a modern Azure-based architecture, with a focus on performance, scalability, and reliability.
Responsibilities
- Design and implement robust data migration pipelines using Azure Data Factory, Synapse Analytics, and Databricks
- Develop scalable ETL processes using PySpark and Python
- Collaborate with stakeholders to understand legacy data structures and ensure accurate mapping and transformation
- Ensure data quality, governance, and performance throughout the migration lifecycle
- Document technical processes and support knowledge transfer to internal teams
Required Skills
- Strong hands-on experience with Azure Data Factory, Synapse, Databricks, PySpark, Python, and SQL
- Proven track record in delivering data migration projects within Azure environments
- Ability to work independently and communicate effectively with technical and non-technical stakeholders
- Previous experience in consultancy or client-facing roles is advantageous