3,026 Senior Data Engineer jobs in the United Kingdom

SpAItial is pioneering the development of a frontier 3D foundation model, pushing the boundaries of AI, computer vision, and spatial computing. Our mission is to redefine how industries, from robotics and AR/VR to gaming and movies, generate and interact with 3D content.

We're looking for individuals who are bold, innovative, and driven by a passion for pushing the boundaries of what's possible. You should thrive in an environment where creativity meets challenge and be fearless in tackling complex problems. Our team is built on a foundation of dedication and a shared commitment to excellence, so we value people who take immense pride in their work and place the collective goals of the team above personal ambition. As a part of our startup, you'll be at the forefront of the AI revolution in 3D technology, and we want you to be excited about shaping the future of this dynamic field. If you're ready to make an impact, embrace the unknown, and collaborate with a talented group of visionaries, we want to hear from you.

Responsibilities

• Be the first Software Engineer with a focus on data in a dynamic deep-tech ML startup, unblocking a high standard of execution across the company.
• Architecting and building our core data infrastructure for managing large-scale ML training datasets (e.g., Apache Iceberg, Parquet).
• Develop cloud-based data processing pipelines , that ingest and compute auxiliary metadata signals on image, video and 3D data (PySpark, Airflow).
• Develop a data serving strategy for training ML models, including data loaders, caching, etc.
• Generate tooling to help in the ML lifecycle , including evaluation, training data inspection, model versioning, experiment tracking, etc.
• Ensure code quality and maintainability by conducting code reviews and promoting best coding practices.
• Collaborate with team members to uphold best practices and improve the long-term health of the codebase.

• 3 years full-time professional experience, committing code to a production environment.
• Proficiency in large-scale data processing (e.g. Spark, Cloud SQL.), and large-scale data systems (Iceberg, Parquet, .)
• Proficiency in cloud platforms (e.g. AWS, GCP, Azure).
• Proficiency in the Python ecosystem and its best practices
• Experience in CI/CD (e.g. CircleCI).

• Familiarity and enthusiasm about AI-based coding solutions (e.g. Cursor, Windsurf, etc.)
• Familiarity with ML concepts and frameworks (PyTorch).
• Experience in large-scale processing of multimodal computer vision data (images, videos, captioning, etc) for ML purposes.
• Experience in Structure-from-Motion for large-scale 3D reconstruction of image data.

Overview

Senior Big Data Engineer at Gaming Innovation Group (GiG) – Department: Technology. Location: Madrid.

As part of the Big Data Team, you will work with cutting edge technologies to collate data from multiple sources in real time within GiG. You will support an event-driven microservice Lambda architecture and contribute to the platform’s objectives, enabling other data technology stakeholders such as Business Intelligence, Data Science and Quality Assurance teams. You will maintain and add new features to a platform that ingests up to 5000 messages per second across products such as Lottery, Sports and Casino. Reporting to the Lead Big Data Engineer and supported by the Big Data Architect to follow best practices, you’ll deliver world-class product increments that support our Tier 1 Platform.

Key Responsibilities

• Participate in all agile scrum meetings such as daily standups, refinement sessions, retro sessions, etc.
• Maintain the data platform by daily reconciliations, data reloads, addressing support tickets, etc.
• Propose ideas to improve existing products and services.
• Implement bug fixes and enhancements within the data platform.
• Perform code reviews for other engineers.
• Take ownership of releases where necessary.
• Communicate with stakeholders to ensure information is transmitted accurately to the right audience.
• Take ownership of complex, large-scale initiatives, including data migration projects, ensuring successful planning, execution and delivery.

Skills, Knowledge & Expertise

• You’re really awesome at:
• Object oriented programming (Java)
• Data modelling using any database technologies
• ETL processes and experience with Apache Spark or Apache NiFi
• Applied understanding of CI/CD in change management
• Dockerised applications
• Used distributed version control systems
• Excellent team player
• Meticulous and passionate about your work

• You’re also good at:
• Functional programming (Python, Scala)
• Analytical mindset
• Low latency databases such as ClickHouse
• Bachelor’s Degree in Computer Science or equivalent
• Collaboration in scoping exercises
• Mentoring skills
• Experience with Linux and Windows environments

Job Benefits

• Great career development opportunities
• 100% remote or hybrid working model
• International Health Insurance
• Health and Wellbeing Package (350 EUR per year)
• Birthday Day Off
• Me Time - 1 day off per year
• Lunches in the office for free

• Mid-Senior level

• Full-time

• Engineering and Information Technology

• Gambling Facilities and Casinos

Referrals increase your chances of interviewing at Gaming Innovation Group by 2x.

Overview

Join to apply for the Big Data Engineer - R role at Northrop Grumman UK.

UK CITIZENSHIP REQUIRED FOR THIS POSITION: Yes

RELOCATION ASSISTANCE: Relocation assistance may be available

CLEARANCE TYPE: UK-Highest Level of Government Clearance

TRAVEL: Yes, 10% of the Time

Salary: £77,400 - £116,000

Define Possible at Northrop Grumman UK

At Northrop Grumman UK, our mission is to solve the most complex challenges by shaping the technology and solutions of tomorrow. We call it Defining Possible. This mindset goes beyond our customer solutions; it’s the foundation for your career development and the impact we have within the community. So, what’s your possible?

Opportunity

This is more than just a job; it’s a mission.

As a Big Data Systems Engineer, you will be responsible for designing, implementing, and maintaining scalable big data systems that support our data science, machine learning, and AI workloads. You will work closely with cross-functional teams to ensure seamless data integration, security, and compliance with GDPR and privacy regulations. Your expertise in scripting, troubleshooting, and integration testing will be essential in optimizing our data pipelines and orchestration processes.

Our UK Cyber & Intelligence business combines modern software development approaches with a rich heritage and experience in the Defence and security sectors. Our customers have complex and sensitive data and information requirements requiring a mission partner who quickly understands the context, delivering and sustaining a portfolio of challenging technology projects at scale and pace, supporting them through an ambitious digital transformation programme.

If you are looking for a career with meaning where you can make a difference to the nation’s security and support critical missions, then look no further.

“My purpose; to lead a team of engineers with the brightest minds, to push the boundaries and define possible together.”

• Scripting and Automation: Develop and maintain scripts for automating data processes and workflows.
• Troubleshooting: Diagnose and resolve technical issues related to big data systems, data pipelines, and integrations.
• Integration Testing: Conduct thorough testing to ensure seamless integration of new tools and technologies into existing systems.
• Linux Internals: Utilize in-depth knowledge of Linux internals to optimize performance and reliability of big data infrastructure.
• Network Protocols: Apply understanding of TCP/IP and OSI models in the design and troubleshooting of networked systems.
• Data Pipeline Support: Manage and optimize data pipelines and orchestration tools such as NiFi and Airflow.
• Scalable System Design: Design scalable big data systems with a focus on security, GDPR compliance, and privacy.
• Hybrid Cloud Management: Leverage hybrid cloud experience to manage on-premise and AWS cloud resources effectively.
• Data Science Support: Support data science, machine learning, and AI workloads using tools like Jupyter, Spacy, Transformers, and NLTK.
• Big Data Platforms: Utilize big data NoSQL engines and platforms such as Hive, Impala, and Elasticsearch for data storage and processing.
• BI and Visualization: Implement and support business intelligence and visualization tools like Tableau, Kibana, and PowerBI to provide actionable insights.

• Strong troubleshooting skills and the ability to diagnose and resolve complex technical issues.
• Experience with integration testing and ensuring seamless tool integration.
• In-depth knowledge of Linux internals and system administration.
• Understanding of TCP/IP and OSI models.
• Hands-on experience with data pipeline tools like NiFi and Airflow.
• Proven ability to design scalable big data systems with a focus on security and GDPR compliance.
• Hybrid cloud experience, specifically with on-premise and AWS cloud environments.
• Familiarity with data science, machine learning, and AI tools such as Jupyter, Spacy, Transformers, and NLTK.
• Experience with big data NoSQL engines/platforms such as Hive, Impala, and Elasticsearch.
• Proficiency with business intelligence and visualization tools like Tableau, Kibana, and PowerBI.
• Excellent communication and collaboration skills.

• Certification in AWS or other cloud platforms.
• Experience with additional data orchestration tools.
• Familiarity with other big data tools and technologies.
• Previous experience in a similar role within a dynamic and fast-paced environment.

Experience in Cloudera, Hadoop, Cloudera Data Science Workbench (CDSW), Cloudera Machine Learning (CML) would be highly desirable.

• Full time on-site presence required

If you don't meet every single requirement, we still encourage you to apply. Sometimes, people hesitate to apply because they can't tick every box. We encourage you to apply if you believe the role will suit you well, even if you don’t meet all the criteria. You might be exactly who we are looking for, either for this position or for our opportunities at Northrop Grumman UK. We are on an exciting growth trajectory and growing our teams across the UK.

You must hold the highest level of UK Government security clearance. Our requirement team is on hand to answer any questions and we will guide you through the process:

We can offer you a range of flexible working options to suit you, including optional compressed working schedule with every other Friday off. Our benefits including private health care, cash health plan, holiday buy and sell, career development opportunities and performance bonuses. For a comprehensive list of benefits, speak to our recruitment team.

• A mission to believe in – Every day we contribute to building a more secure and connected world, expanding our reach from land, sea, and air to space and cyberspace. From engineering data and intelligence solutions, to developing maritime navigation and control systems and innovating command and control systems for the UK and NATO, what we do together matters.
• A place to belong and thrive – Every voice matters at our table meaning you can bring your authentic self to work. From our Employee Resource Groups backed by thousands of employees to our partnerships with the Association For Black and Minority Ethnic Engineers, Forces Transition Group, Mind, and Women in Defence – we are passionate about growing and supporting our inclusive community where everyone can belong.
• Your career, your way - Shape your career journey with diverse roles, mentorship, and development opportunities that fuel your curiosity, channel your expertise, and nurture your passion. Looking for flexibility? Balance your professional career with your personal life through our health and wellbeing benefits, discount schemes, and investment in your future development. Speak to our team to find the balance that’s right for you.

Yes – Submit your application online. Your application will be reviewed by our team and we will be in touch.

Possibly, I’d like to find out more – Reach out to our team for more information and support:

No, I don’t think this role is right for me – Our extensive UK growth means we have exciting, new opportunities opening all the time. Speak to our team to discuss your career goals.

Northrop Grumman is committed to hiring and retaining a diverse workforce, and encourages individuals from all backgrounds and all abilities to apply and consider becoming a part of our diverse and inclusive workforce.

Requisition ID: R

• Category: Information Technology

• Location: London, London, United Kingdom

• Clearance Type: Highest Level of Government Clearance

• Telecommute: No- Teleworking not available for this position

• Travel Required: Yes, 10% of the Time

• Relocation Assistance: Relocation assistance may be available

• Positions Available: 1

Your Opportunity to Define Possible . Our Opportunity t o Deliver the Nation’s Security . Together .

Role clearance type : Must already hold highest level of government clearance.

Location: London

About Your Opportunity:

As a Big Data Systems Engineer, you will be responsible for designing, implementing, and maintaining scalable big data systems that support our data science, machine learning, and AI workloads. You will work closely with cross-functional teams to ensure seamless data integration, security, and compliance with GDPR and privacy regulations. Your expertise in scripting, troubleshooting, and integration testing will be essential in optimizing our data pipelines and orchestration processes.

• On-site presence required a minimum of 4 days per week.

Your Benefits:

• Flexible working schedules - we offer flexible and hybrid working arrangements. Talk to us at the application stage about any scheduling preferences you may have.

• Flexible Benefits Package – choose which NGUKL benefits you want to satisfy your personal needs. Core Benefits provided for you are Healthcare, Dental, Life Assurance and Pension. Benefits you can flex include Critical Illness Cover, Health Cash Plan, and Health Assessments.

• Employee Incentive Programme – exceptional performance is recognized through our annual incentive programme which is awarded to top performers who excel.

• Career Development – opportunity for ongoing professional development and career growth opportunities.

Your Responsibilities :

• Scripting and Automation: Develop and maintain scripts for automating data processes and workflows.

• Troubleshooting: Diagnose and resolve technical issues related to big data systems, data pipelines, and integrations.

• Integration Testing: Conduct thorough testing to ensure seamless integration of new tools and technologies into existing systems.

• Linux Internals: Utilize in-depth knowledge of Linux internals to optimize performance and reliability of big data infrastructure.

• Network Protocols: Apply understanding of TCP/IP and OSI models in the design and troubleshooting of networked systems.

• Data Pipeline Support: Manage and optimize data pipelines and orchestration tools such as NiFi and Airflow.

• Scalable System Design: Design scalable big data systems with a focus on security, GDPR compliance, and privacy.

• Hybrid Cloud Management: Leverage hybrid cloud experience to manage on-premise and AWS cloud resources effectively.

• Data Science Support: Support data science, machine learning, and AI workloads using tools like Jupyter, Spacy, Transformers, and NLTK.

• Big Data Platforms: Utilize big data NoSQL engines and platforms such as Hive, Impala, and Elasticsearch for data storage and processing.

• BI and Visualization: Implement and support business intelligence and visualization tools like Tableau, Kibana, and PowerBI to provide actionable insights.

Your Experience :

• Proven experience in scripting languages such as Python, Bash, or similar.

• Strong troubleshooting skills and the ability to diagnose and resolve complex technical issues.

• Experience with integration testing and ensuring seamless tool integration.

• In-depth knowledge of Linux internals and system administration.

• Understanding of TCP/IP and OSI models.

• Hands-on experience with data pipeline tools like NiFi and Airflow.

• Proven ability to design scalable big data systems with a focus on security and GDPR compliance.

• Hybrid cloud experience, specifically with on-premise and AWS cloud environments.

• Familiarity with data science, machine learning, and AI tools such as Jupyter, Spacy, Transformers, and NLTK.

• Experience with big data NoSQL engines/platforms such as Hive, Impala, and Elasticsearch.

• Proficiency with business intelligence and visualization tools like Tableau, Kibana, and PowerBI.

• Excellent communication and collaboration skills.

Preferred Experience:

• Certification in AWS or other cloud platforms.

• Experience with additional data orchestration tools.

• Familiarity with other big data tools and technologies. (Hadoop, Cloudera)

• Previous experience in a similar role within a dynamic and fast-paced environment.

Your Future Team :

The teams in this space work on some of the toughest challenges that are key to our client’s success, as such they carry a heavy load and require the ability to be able to dip in and share the workload. Success in this area requires flexibility in tasking, adapting to the problem at hand and picking up the skills along the way. This work is predominantly on client site and will vary between integrating with development and test teams, as well as taking on standalone tasks when needed.

We believe that creating a team that values diversity and fosters inclusion is essential to great performance. We know the best ideas come from diversity of thought, background, perspective, culture, gender, race, age and many other elements. We welcome candidates from all backgrounds and particularly from communities currently under-represented within our industry.We treat everyone with respect and foster safe and inclusive environments.

About Our Responsibilities:

Our customers operate in unique environments which offer new and exciting challenges every day, cultivating a place where you can learn and thrive, working alongside the best minds in industry. We’ll give you space to develop your career, where your ideas can shape the future of our dynamic business.

We promote collaboration to achieve more than we could imagine, together. And within a respectful and inspirational environment, we value what you say and do.

How to Apply:

Interested in our opportunity?

Yes – then simply submit your application online. Your application will be reviewed by one of our expert recruiters who’ll then respond advising you of the outcome and next steps for successful candidates.

Possibly, I’d like to find out more – then connect direct with , where one of our recruitment business partners will be happy to support you with any enquires.

Background checks and potentially security clearance form part of the recruitment process, our team will inform you of the procedures when required.

Northrop Grumman UK:

Work with a global brand that makes a real contribution to our nation’s security and future. At Northrop Grumman UK, the brightest minds come together to push the boundaries and Define Possible. As leaders in the digital transformation of Aerospace, Defence and Intelligence we are providing ground-breaking outcomes for our customers.

UK Cyber & Intelligence Business :

Our UK Cyber & Intelligence business combines modern software development approaches with a rich heritage and experience in the Defence and security sectors. Our customers have complex and sensitive data and information requirements requiring a mission partner who quickly understands the context, delivering and sustaining a portfolio of challenging technology projects at scale and pace, supporting them through an ambitious digital transformation programme.

Find out more :

#LI-CJ1

#LI-ONSITE

Northrop Grumman is committed to hiring and retaining a diverse workforce, and encourages individuals from all backgrounds and all abilities to apply and consider becoming a part of our diverse and inclusive workforce.

Overview

Big Data QA Engineer needed by a leading Telco client to work on transformation projects. You will develop and execute test scripts and assist with the development and maintenance of smoke, performance, functional and regression tests to ensure code behaves as designed. This is a long-term contract position.

• Develop and execute test scripts for big data solutions.
• Assist with the development and maintenance of smoke, performance, functional and regression tests.
• Ensure data processing and analytics components function as designed through thorough testing.

• Exposure to big data testing.
• SQL Server.
• Python.
• Experience in the Telco industry.

• Seniority level: Mid-Senior level
• Employment type: Contract
• Location: London (hybrid/work-from-home options available)
• Rate: Up to £380/day
• Duration: 6 months +

If you are interested in this Big Data QA Engineer position and meet the above requirements, please apply.

Overview

Big Data Systems Engineer
Highest Level of Clearance
Full time onsite

You will be responsible for designing, implementing, and maintaining scalable big data systems that support our data science, machine learning, and AI workloads. You will work closely with cross-functional teams to ensure seamless data integration, security, and compliance with GDPR and privacy regulations. Your expertise in scripting, troubleshooting, and integration testing will be essential in optimizing our data pipelines and orchestration processes.

• Scripting and Automation: Develop and maintain scripts for automating data processes and workflows.
• Troubleshooting: Diagnose and resolve technical issues related to big data systems, data pipelines, and integrations.
• Integration Testing: Conduct thorough testing to ensure seamless integration of new tools and technologies into existing systems.
• Linux Internals: Utilize in-depth knowledge of Linux internals to optimize performance and reliability of big data infrastructure.
• Network Protocols: Apply understanding of TCP/IP and OSI models in the design and troubleshooting of networked systems.
• Data Pipeline Support: Manage and optimize data pipelines and orchestration tools such as NiFi and Airflow.
• Scalable System Design: Design scalable big data systems with a focus on security, GDPR compliance, and privacy.
• Hybrid Cloud Management: Leverage hybrid cloud experience to manage on-premise and AWS cloud resources effectively.
• Data Science Support: Support data science, machine learning, and AI workloads using tools like Jupyter, Spacy, Transformers, and NLTK.
• Big Data Platforms: Utilize big data NoSQL engines and platforms such as Hive, Impala, and Elasticsearch for data storage and processing.
• BI and Visualization: Implement and support business intelligence and visualization tools like Tableau, Kibana, and PowerBI to provide actionable insights.

• Strong troubleshooting skills and the ability to diagnose and resolve complex technical issues.
• Experience with integration testing and ensuring seamless tool integration.
• In-depth knowledge of Linux internals and system administration.
• Understanding of TCP/IP and OSI models.
• Hands-on experience with data pipeline tools like NiFi and Airflow.
• Proven ability to design scalable big data systems with a focus on security and GDPR compliance.
• Hybrid cloud experience, specifically with on-premise and AWS cloud environments.
• Familiarity with data science, machine learning, and AI tools such as Jupyter, Spacy, Transformers, and NLTK.
• Experience with big data NoSQL engines/platforms such as Hive, Impala, and Elasticsearch.
• Proficiency with business intelligence and visualization tools like Tableau, Kibana, and PowerBI.
• Excellent communication and collaboration skills.

• Certification in AWS or other cloud platforms.
• Experience with additional data orchestration tools.
• Familiarity with other big data tools and technologies.
• Previous experience in a similar role within a dynamic and fast-paced environment.

Please send your latest CV.

Due to the nature and urgency of this post, candidates holding or who have held high level security clearance in the past are most welcome to apply. Please note successful applicants will be required to be security cleared prior to appointment which can take up to a minimum 18 weeks. LA International is a HMG approved ICT Recruitment and Project Solutions Consultancy, operating globally from the largest single site in the UK as an IT Consultancy or as an Employment Business & Agency depending upon the precise nature of the work, for security cleared jobs or non-clearance vacancies, LA International welcome applications from all sections of the community and from people with diverse experience and backgrounds.

Award Winning LA International, winner of the Recruiter Awards for Excellence, Best IT Recruitment Company, Best Public Sector Recruitment Company and overall Gold Award winner, has now secured the most prestigious business award that any business can receive, The Queens Award for Enterprise: International Trade, for the second consecutive period.

Overview

Direct message the job poster from Intuition IT – Intuitive Technology Recruitment

The following qualifications indicate the skills and experience sought for this role in Big data technologies and real-time data processing.

• Experience in Big data technologies, real-time data processing platforms (e.g., Spark Streaming) is advantageous.
• Consistently demonstrates clear and concise written and verbal communication.
• A history of delivering against agreed objectives.
• Ability to multi-task and work under pressure.
• Demonstrated problem solving and decision-making skills.
• Excellent analytical and process-based skills, e.g., process flow diagrams, business modelling.

• Mid-Senior level

• Contract

• Information Technology

• Staffing and Recruiting

Company: Sidetrade

Type: Full-time

Location Type: Hybrid

Location: Birmingham, England, United Kingdom

Indulge your passion for high-availability software and performance enhancement as part of our dynamic team. Embrace the challenge, embrace the excitement – become a DevOps Engineer and thrive! Shape the future of AI-powered Order-to-Cash at Sidetrade today. Join us in creating innovative solutions that redefine the industry!

Sidetrade is a fast-growing international software company that is transforming the Order-to-Cash process for global enterprises. Its AI-powered SaaS platform digitizes the financial customer journey, empowering CFOs to secure and accelerate cash flow generation. Recognized as a Leader in Gartner's Magic Quadrant for two consecutive years, Sidetrade fosters a culture of innovation, collaboration, and customer-centricity from its headquarters in Europe and North America.

The R&D team comprises experienced tech professionals who share a deep passion for technology. Together, they are dedicated to developing cutting-edge software solutions that drive the transformation of our customers' work processes. We provide comprehensive training, coaching, resources, and mentorship to empower every team member's growth and nurture their success.