Explore exciting Spark job opportunities in Big Data Solutions Architect (DE, Spark Architecture)
Posted today
Job Viewed
Job Description
Solutions Architect (Professional Services) Spark Expert
role at
Databricks .nLocation: LondonnWe have 9 new open positions based in our London office. As a Solutions Architect in our Professional Services team, you will work with clients on short to medium-term engagements on big data challenges using the Databricks platform. You will provide data engineering, data science, and cloud technology projects requiring integration with client systems, training, and other technical tasks to help customers get the most value out of their data. RSAs are billable and complete projects according to specifications with excellent customer service. You will report to the regional Manager/Lead.nReq ID: CSQ326R185nWe are looking for someone who can contribute to a variety of customer engagements and help drive successful adoption of Databricks solutions.nResponsibilities
You will work on a variety of impactful customer technical projects which may include designing and building reference architectures, creating how-to guides, and productionizing customer use cases.nWork with engagement managers to scope a variety of professional services work with input from the customer.nGuide strategic customers as they implement transformational big data projects, including end-to-end design, build, and deployment of big data and AI applications.nConsult on architecture and design; bootstrap or implement customer projects to support understanding, evaluation and adoption of Databricks.nProvide escalated support for customer operational issues.nCollaborate with the Databricks technical team, Project Manager, Architect and Customer team to ensure delivery of technical components to meet customer needs.nCoordinate with Engineering and Databricks Customer Support to provide product and implementation feedback and guide rapid resolution for engagement-specific issues.nQualifications
Proficient in data engineering, data platforms, and analytics with a strong track record of successful projects and knowledge of industry best practices.nComfortable writing code in Python or Scala.nWorking knowledge of two or moreCloud ecosystems (AWS, Azure, GCP) with expertise in at least one.nDeep experience with distributed computing with Apache Spark and knowledge of Spark runtime internals.nFamiliarity with CI/CD for production deployments.nWorking knowledge of MLOps.nExperience designing and deploying performant end-to-end data architectures.nExperience with technical project delivery - managing scope and timelines.nDocumentation and white-boarding skills.nExperience working with clients and managing conflicts.nBuild skills in technical areas supporting deployment and integration of Databricks-based solutions.nTravel is required up to 10%, more at peak times.nDatabricks Certification.nAbout Databricks
Databricks is the data and AI company. More than 10,000 organizations worldwide rely on the Databricks Data Intelligence Platform to unify and democratize data, analytics and AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the creators of Lakehouse, Apache Spark, Delta Lake and MLflow.nBenefits
Databricks offers comprehensive benefits and perks. For specific details on the benefits offered in your region, please visit the designated benefits page.nOur Commitment to Diversity and Inclusion
Databricks is committed to fostering a diverse and inclusive culture where everyone can excel. We ensure hiring practices are inclusive and meet equal employment opportunity standards.nCompliance
If access to export-controlled technology or source code is required for performance of job duties, it is within Employer's discretion whether to apply for a U.S. government license for such positions, and Employer may decline to proceed with an applicant on this basis alone.
#J-18808-Ljbffrn
Java Software Engineer Specialist - AWS & Spark
Posted today
Job Viewed
Job Description
We’re looking for a Java Software Engineer Specialist to join the Billing and Funding team to help unleash the potential of every business.nAre you ready to make your mark? Then you sound like a Worldpayer.nOverview
Billing and Funding within Worldpay has built a pricing and billing engine using big-data tech. We are near the end of the first phase of re‑engineering our core payments platform to be more cost effective, scalable and cloud ready. The next phase involves moving to the cloud, making our software cloud native, infrastructure code, and using CI/CD to increase delivery speed.nWhat you’ll own
Design and build highly scalable, loosely coupled, event-driven, distributed systems.nWrite Java applications using a variety of persistence including databases, S3 and Kafka.nExperience building production applications using AWS and EMR would be beneficial.nLead by example and establish industry best practices across software development, TDD/BDD, DevOps & cloud.nGuide the teams on adopting DevOps best practices, tools and technologies.nFoster an engineering mindset within the team and establish best practices for software development.nWhat you bring
Technical Skills (Must)nExperience in functional programming using Java 8 or highernSQL and NoSQL databasesnKnowledge of building scalable applications in AWSnKnowledgeable on development best practices, technologies and toolsnTechnical Skills (nice to have)nKafkanKnowledge of EMR ServerlessnKnowledgeable on service architecturenKnowledge of current architectures and approaches such as Event Driven Design, Domain Driven Design, Event Sourcing & CQRSnGeneralnExperience of leading change and establishing engineering best practice in technical teamsnAccountability for a production service and contributions to its buildnGreat communication skills (written and verbal)nKnowledgeable on BDD / TDD practices and have worked with these approachesnWorldpay perks
Competitive salary and benefitsnParental leave policynGlobal recognition platformnVirgin Pulse accessnGlobal employee assistance programnEmployee wellbeing and lifestyle supportnApplication steps
Application steps explain how to apply, what we’ll ask, and the interview process. You’ll typically upload your resume and a cover letter.
#J-18808-Ljbffrn
Solutions Architect (Professional Services) Spark Expert
Posted today
Job Viewed
Job Description
Solutions Architect (Professional Services) Spark ExpertnLondon, United KingdomnREQ ID: CSQ326R389nLocation: LondonnWe have 9 NEW open positions based in our London office.nAs a Solutions Architect in our Professional Services team, you will work with clients on short to medium-term customer engagements on their big data challenges using the Databricks platform. You will provide data engineering, data science, and cloud technology projects which require integrating with client systems, training, and other technical tasks to help customers get the most value out of their data. RSAs are billable and know how to complete projects according to specifications with excellent customer service. You will report to the regional Manager/Lead.nThe impact you will have:nYou will work on a variety of impactful customer technical projects which may include designing and building reference architectures, creating how-to's and productionalizing customer use casesnWork with engagement managers to scope a variety of professional services work with input from the customernGuide strategic customers as they implement transformational big data projects, 3rd party migrations, including end-to-end design, build and deployment of industry-leading big data and AI applicationsnConsult on architecture and design; bootstrap or implement customer projects which lead to a customer's successful understanding, evaluation and adoption of Databricks.nProvide an escalated level of support for customer operational issues.nYou will work with the Databricks technical team, Project Manager, Architect and Customer team to ensure the technical components of the engagement are delivered to meet customer's needs.nWork with Engineering and Databricks Customer Support to provide product and implementation feedback and to guide rapid resolution for engagement specific product and support issues.nWhat we look for:nProficient in data engineering, data platforms, and analytics with a strong track record of successful projects and in-depth knowledge of industry best practicesnComfortable writing code in either Python or ScalanWorking knowledge of two or more common Cloud ecosystems (AWS, Azure, GCP) with expertise in at least onenDeep experience with distributed computing with Apache Spark and knowledge of Spark runtime internalsnFamiliarity with CI/CD for production deploymentsnWorking knowledge of MLOpsnDesign and deployment of performant end-to-end data architecturesnExperience with technical project delivery - managing scope and timelines.nDocumentation and white-boarding skills.nExperience working with clients and managing conflicts.nBuild skills in technical areas which support the deployment and integration of Databricks-based solutions to complete customer projects.nTravel is required up to 10%, more at peak times.nDatabricks Certification an advantagenAbout Databricks
Databricks is the data and AI company. More than 10,000 organizations worldwide — including Comcast, Condé Nast, Grammarly, and over 50% of the Fortune 500 — rely on the Databricks Data Intelligence Platform to unify and democratize data, analytics and AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache Spark, Delta Lake and MLflow. To learn more, follow Databricks on Twitter, LinkedIn and Facebook.nBenefitsnAt Databricks, we strive to provide comprehensive benefits and perks that meet the needs of all of our employees. For specific details on the benefits offered in your region, please visit Commitment to Diversity and Inclusion
At Databricks, we are committed to fostering a diverse and inclusive culture where everyone can excel. We take great care to ensure that our hiring practices are inclusive and meet equal employment opportunity standards. Individuals looking for employment at Databricks are considered without regard to age, color, disability, ethnicity, family or marital status, gender identity or expression, language, national origin, physical and mental ability, political affiliation, race, religion, sexual orientation, socio-economic status, veteran status, and other protected characteristics.nCompliance
If access to export-controlled technology or source code is required for performance of job duties, it is within Employer's discretion whether to apply for a U.S. government license for such positions, and Employer may decline to proceed with an applicant on this basis alone.
#J-18808-Ljbffrn
Big Data Engineer
Posted 11 days ago
Job Viewed
Job Description
RELOCATION ASSISTANCE: Relocation assistance may be available
CLEARANCE TYPE: UK-Highest Level of Government Clearance
TRAVEL: Yes, 10% of the Time
**Salary: £77,400 - £116,000**
**Define Possible at Northrop Grumman UK**
At Northrop Grumman UK, our mission is to solve the most complex challenges by shaping the technology and solutions of tomorrow. We call it Defining Possible.
This mind-set goes beyond our customer solutions; it's the foundation for your career development and the impact we have within the community. So, what's your possible?
**Opportunity:**
This is more than just a job; it's a mission.
As a Big Data Systems Engineer, you will be responsible for designing, implementing, and maintaining scalable big data systems that support our data science, machine learning, and AI workloads. You will work closely with cross-functional teams to ensure seamless data integration, security, and compliance with GDPR and privacy regulations. Your expertise in scripting, troubleshooting, and integration testing will be essential in optimizing our data pipelines and orchestration processes.
Our UK Cyber & Intelligence business combines modern software development approaches with a rich heritage and experience in the Defence and security sectors. Our customers have complex and sensitive data and information requirements requiring a mission partner who quickly understands the context, delivering and sustaining a portfolio of challenging technology projects at scale and pace, supporting them through an ambitious digital transformation programme.
If you are looking for a career with meaning where you can make a difference to the nation's security and support critical missions, then look no further.
_"My purpose; to lead a team of engineers with the brightest minds, to push the boundaries and define possible together."_
**Role responsibilities:**
+ Scripting and Automation: Develop and maintain scripts for automating data processes and workflows.
+ Troubleshooting: Diagnose and resolve technical issues related to big data systems, data pipelines, and integrations.
+ Integration Testing: Conduct thorough testing to ensure seamless integration of new tools and technologies into existing systems.
+ Linux Internals: Utilize in-depth knowledge of Linux internals to optimize performance and reliability of big data infrastructure.
+ Network Protocols: Apply understanding of TCP/IP and OSI models in the design and troubleshooting of networked systems.
+ Data Pipeline Support: Manage and optimize data pipelines and orchestration tools such as NiFi and Airflow.
+ Scalable System Design: Design scalable big data systems with a focus on security, GDPR compliance, and privacy
+ Hybrid Cloud Management: Leverage hybrid cloud experience to manage on-premise and AWS cloud resources effectively
+ Data Science Support: Support data science, machine learning, and AI workloads using tools like Jupyter, Spacy, Transformers, and NLTK.
+ Big Data Platforms: Utilize big data NoSQL engines and platforms such as Hive, Impala, and Elasticsearch for data storage and processing
+ BI and Visualization: Implement and support business intelligence and visualization tools like Tableau, Kibana, and PowerBI to provide actionable insights.
**We are looking for:**
+ Strong troubleshooting skills and the ability to diagnose and resolve complex technical issues.
+ Experience with integration testing and ensuring seamless tool integration.
+ In-depth knowledge of Linux internals and system administration.
+ Understanding of TCP/IP and OSI models.
+ Hands-on experience with data pipeline tools like NiFi and Airflow.
+ Proven ability to design scalable big data systems with a focus on security and GDPR compliance.
+ Hybrid cloud experience, specifically with on-premise and AWS cloud environments.
+ Familiarity with data science, machine learning, and AI tools such as Jupyter, Spacy, Transformers, and NLTK.
+ Experience with big data NoSQL engines/platforms such as Hive, Impala, and Elasticsearch.
+ Proficiency with business intelligence and visualization tools like Tableau, Kibana, and PowerBI.
+ Excellent communication and collaboration skills.
Preferred Qualifications:
+ Certification in AWS or other cloud platforms.
+ Experience with additional data orchestration tools.
+ Familiarity with other big data tools and technologies.
+ Previous experience in a similar role within a dynamic and fast-paced environment.
Experience in Cloudera, Hadoop, Cloudera Data Science Workbench (CDSW), Cloudera Machine Learning (CML) would be highly desirable.
**Work Environment:**
+ Full time on-site presence required
**If you don't meet every single requirement, we still encourage you to apply.**
Sometimes, people hesitate to apply because they can't tick every box. We encourage you to apply if you believe the role will suit you well, even if you don't meet all the criteria. You might be exactly who we are looking for, either for this position or for our opportunities at Northrop Grumman UK. We are on an exciting growth trajectory and growing our teams across the UK.
**Security clearance:**
You must hold the highest level of UK Government security clearance. Our requirement team is on hand to answer any questions and we will guide you through the process: .
**Benefits:**
We can offer you a range of flexible working options to suit you, including optional compressed working schedule with every other Friday off. Our benefits including private health care, cash health plan, holiday buy and sell, career development opportunities and performance bonuses. For a comprehensive list of benefits, speak to our recruitment team.
**Why join us?**
+ **A mission to believe in** **-** Every day we contribute to building a more secure and connected world, expanding our reach from land, sea, and air to space and cyberspace. From engineering data and intelligence solutions, to developing maritime navigation and control systems and innovating command and control systems for the UK and NATO, what we do together matters.
+ **A place to belong and thrive** **-** Every voice matters at our table meaning you can bring your authentic self to work. From our Employee Resource Groups backed by thousands of employees to our partnerships with the Association For Black and Minority Ethnic Engineers, Forces Transition Group, Mind, and Women in Defence - we are passionate about growing and supporting our inclusive community where everyone can belong.
+ **Your career, your way** - Shape your career journey with diverse roles, mentorship, and development opportunities that fuel your curiosity, channel your expertise, and nurture your passion. Looking for flexibility? Balance your professional career with your personal life through our health and wellbeing benefits, discount schemes, and investment in your future development. Speak to our team to find the balance that's right for you.
**Ready to apply?**
**Yes** - Submit your application online. Your application will be reviewed by our team and we will be in touch.
**Possibly, I'd like to find out more** **about this role** - Reach out to our team for more information and support: .
**No, I don't think this role is right for me** - Our extensive UK growth means we have exciting, new opportunities opening all the time. Speak to our team to discuss your career goals.
Northrop Grumman is committed to hiring and retaining a diverse workforce, and encourages individuals from all backgrounds and all abilities to apply and consider becoming a part of our diverse and inclusive workforce.
Data Solutions Architect (Modern Big Data)
Posted 18 days ago
Job Viewed
Job Description
We are seeking a highly experienced and visionary Data Solutions Architect (Modern Big Data) to join our Data & AI practice. The successful candidate will bring extensive expertise in architecting and delivering modern big data platforms that are scalable, reliable, and business-aligned. This role is pivotal in enabling clients to harness the power of streaming data, data lakes, lakehouses, and advanced analytics platforms, while guiding them on their data modernisation journeys.
As a trusted advisor, you will collaborate with executives, stakeholders, and technical teams to define modern big data strategies, design cloud-native architectures, and implement industry-leading best practices. You will thrive in a fast-paced, evolving technology environment, continuously expanding your knowledge to ensure NTT DATA and our clients remain leaders in data-driven innovation.
**What you'll be doing:**
**Primary Responsibilities:**
+ Client Engagement & Delivery
+ Solution Design & Implementation
+ Modernisation & Transformation
+ Thought Leadership & Knowledge Sharing
+ Collaboration & Leadership
**Business Relationships:**
+ Client Partners
+ Practice Leaders and Members
+ Peer-level relationships within client organisations up to Head of Data Engineering, Chief Data Architect, CIO, and CDO level
**What experience you'll bring:**
**Key Competencies:**
+ Demonstrated expertise in designing and implementing modern big data architectures for large enterprises.
+ Strong consulting values with the ability to engage effectively with senior client stakeholders.
+ Hands-on experience across the data lifecycle: ingestion, storage, transformation, governance, analytics, and consumption.
+ Deep understanding of data product strategies, Data Mesh, and Data Fabric concepts.
+ Ability to align data strategies with business objectives, driving measurable value.
+ Excellent analytical, problem-solving, and communication skills.
+ Proven experience leading data modernisation initiatives across multiple sectors.
**Technical Expertise:**
+ Proficiency with big data technologies: Apache Spark, Kafka, Confluent, Databricks, Unity Catalog.
+ Strong knowledge of cloud ecosystems (AWS, Azure, GCP) and infrastructure-as-code (Terraform, CloudFormation).
+ Hands-on expertise with data lake/lakehouse architectures and modern streaming solutions.
+ Experience with SQL, NoSQL, ETL/ELT pipelines, and programming languages (Python, R, Java).
+ Familiarity with BI, data visualisation, and DevOps principles.
+ Strong understanding of data governance, security, and compliance frameworks (GDPR, CCPA, HIPAA).
+ Exposure to machine learning and AI integration within modern data platforms.
**Experience, Qualifications:**
+ Experience: Minimum 8-12 years in data architecture, engineering, or consulting, with at least 4+ years in modern big data solution architecture.
+ Education: University degree required.
+ Preferred: BSc/MSc in Computer Science, Data Engineering, or related field.
+ Relevant certifications in Databricks, Kafka, or cloud platforms highly desirable.
**Measures of Success:**
+ Delivery of modern big data solutions aligned to client business objectives.
+ High client satisfaction and repeat engagements.
+ Leadership in client transformation journeys to cloud-native data platforms.
+ Contribution to practice growth through assets, accelerators, and thought leadership.
**Who we are:**
We're a business with a global reach that empowers local teams, and we undertake hugely exciting work that is genuinely changing the world. Our advanced portfolio of consulting, applications, business process, cloud, and infrastructure services will allow you to achieve great things by working with brilliant colleagues, and clients, on exciting projects.
Our inclusive work environment prioritises mutual respect, accountability, and continuous learning for all our people. This approach fosters collaboration, well-being, growth, and agility, leading to a more diverse, innovative, and competitive organisation. We are also proud to share that we have a range of Inclusion Networks such as: the Women's Business Network, Cultural and Ethnicity Network, LGBTQ+ & Allies Network, Neurodiversity Network and the Parent Network.
For more information on Diversity, Equity and Inclusion please click here: Creating Inclusion Together at NTT DATA UK | NTT DATA ( we'll offer you:**
We offer a range of tailored benefits that support your physical, emotional, and financial wellbeing. Our Learning and Development team ensure that there are continuous growth and development opportunities for our people. We also offer the opportunity to have flexible work options.
You can find more information about NTT DATA UK & Ireland here: are an equal opportunities employer. We believe in the fair treatment of all our employees and commit to promoting equity and diversity in our employment practices. We are also a proud Disability Confident Committed Employer - we are committed to creating a diverse and inclusive workforce. We actively collaborate with individuals who have disabilities and long-term health conditions which have an effect on their ability to do normal daily activities, ensuring that barriers are eliminated when it comes to employment opportunities. In line with our commitment, we guarantee an interview to applicants who declare to us, during the application process, that they have a disability and meet the minimum requirements for the role. If you require any reasonable adjustments during the recruitment process, please let us know. Join us in building a truly diverse and empowered team.
Back to search Email to a friend Apply now
Data Engineer
Posted 3 days ago
Job Viewed
Job Description
Overview
GlaxoSmithKline (GSK) is a science-led global healthcare company with a special purpose: to help people do more, feel better, live longer. We are on an audacious journey to impact the health of 2.5 billion people over the next decade. Our R&D division is at the forefront of this mission, dedicated to the discovery and development of groundbreaking vaccines and medicines. We are transforming the landscape of medical research by integrating cutting-edge science and technology and harnessing the power of genetics and new data. By fostering a collaborative environment that unites the talents of our people, we are revolutionizing R&D to pre-empt and defeat diseases. Join us in our commitment to uniting science, technology, and talent to get ahead of disease together.
Position Summary
At GSK we see a world in which advanced applications of Machine Learning and AI will allow us to develop novel therapies to existing diseases and to quickly respond to emerging or changing diseases with personalized drugs, driving better outcomes at reduced cost with fewer side effects. It is an ambitious vision that will require the development of products and solutions at the cutting edge of Machine Learning and AI. We’re looking for a highly skilled Backend Engineer to help us make this vision a reality.
The ideal candidate will have a track record of shipping data products derived from complex sources, responsible for the process from conceptual data pipelines to production scale. We have a commitment to quality, so the person in this role will be able to use modern cloud tooling and techniques to deliver reliable data pipelines and continuously improve them.
This role requires a passion for solving challenging problems aligned to exciting Artificial Intelligence and Machine Learning applications. Educational or professional background in the biological sciences is a plus but is not necessary; passion to help therapies for new and existing diseases, and a pattern of continuous learning and development is mandatory.
Key Responsibilities
- Build data pipelines using modern data engineering tools on Google Cloud: Python, Spark, SQL, BigQuery, Cloud Storage.
- Ensure data pipelines meet the specific scientific needs of data consuming applications.
- Responsible for high quality software implementations according to best practices, including automated test suites and documentation.
- Develop, measure, and monitor key metrics for all tools and services and consistently seek to iterate on and improve them.
- Participate in code reviews, continuously improving personal standards as well as the wider team and product.
- Liaise with other technical staff and data engineers in the team and across allied teams, to build an end-to-end pipeline consuming other data products.
Minimum Requirements
- 2+ years of data engineering experience with a Bachelors degree in a relevant field (including computational, numerate or life sciences), or equivalent experience.
- Cloud experience (e.g. Google Cloud preferred).
- Strong skills with industry experience in Python and SQL.
- Unit testing experience (e.g. pytest).
- Knowledge of agile practices and able to perform in agile software development environments.
- Strong experience with modern software development tools / ways of working (e.g. git/GitHub, DevOps tools for deployment).
Preferred Requirements
- Demonstrated experience with biological or scientific data (e.g. genomics, transcriptomics, proteomics), or pharmaceutical industry experience.
- Bioinformatics expertise, familiarity with large scale bioinformatics datasets.
- Experience using Nextflow pipelines.
- Knowledge of NLP techniques and experience of processing unstructured data, using vector stores, and approximate retrieval.
- Familiarity with orchestration tooling (e.g. Airflow or Google Workflows).
- Experience with AI/ML powered applications.
- Experience with Docker or containerized applications.
Why GSK?
Uniting science, technology and talent to get ahead of disease together.
GSK is a global biopharma company with a special purpose – to unite science, technology and talent to get ahead of disease together – so we can positively impact the health of billions of people and deliver stronger, more sustainable shareholder returns – as an organisation where people can thrive. We prevent and treat disease with vaccines, specialty and general medicines. We focus on the science of the immune system and the use of new platform and data technologies, investing in four core therapeutic areas (infectious diseases, HIV, respiratory/ immunology and oncology).
Our success absolutely depends on our people. While getting ahead of disease together is about our ambition for patients and shareholders, it’s also about making GSK a place where people can thrive. We want GSK to be a place where people feel inspired, encouraged and challenged to be the best they can be. A place where they can be themselves – feeling welcome, valued, and included. Where they can keep growing and look after their wellbeing. So, if you share our ambition, join us at this exciting moment in our journey to get Ahead Together.
Inclusion at GSK
GSK is an Equal Opportunity/Affirmative Action Employer. All qualified applicants will receive equal consideration for employment without regard to race, color, national origin, religion, sex, pregnancy, marital status, sexual orientation, gender identity/expression, age, disability, genetic information, military service, covered/protected veteran status or any other federal, state or local protected class.
If you need any adjustments in the recruitment process, please get in touch with our Recruitment team ( ) to further discuss this today.
Important notice to employment businesses/agencies
GSK does not accept referrals from employment businesses and/or employment agencies in respect of the vacancies posted on this site. All employment businesses/agencies are required to contact GSK's commercial and general procurement/human resources department to obtain prior written authorization before referring any candidates to GSK. The obtaining of prior written authorization is a condition precedent to any agreement (verbal or written) between the employment business/ agency and GSK. In the absence of such written authorization being obtained any actions undertaken by the employment business/agency shall be deemed to have been performed without the consent or contractual agreement of GSK. GSK shall therefore not be liable for any fees arising from such actions or any fees arising from any referrals by employment businesses/agencies in respect of the vacancies posted on this site.
Please note that if you are aUS Licensed Healthcare Professional or Healthcare Professional as defined by the laws of the state issuing your license, GSK may be required to capture and report expenses GSK incurs, on your behalf, in the event you are afforded an interview for employment. This capture of applicable transfers of value is necessary to ensure GSK’s compliance to all federal and state US Transparency requirements. For more information, please visit GSK’s Transparency Reporting For the Record site.
Data Engineer
Posted 1 day ago
Job Viewed
Job Description
Data Engineer
12 months
Remote
470 per day inside IR35 - Umbrella only
Active SC clearance and eligible candidates will be considered
Project description
Our client playing a crucial role in the ESN project in 2025, which aims to modernize and enhance communication for frontline emergency services in the UK. This initiative will greatly enhance the capabilities of emergency services, allowing them to share vital data and information quickly and securely, improving response times and overall public safety.
Required Skills
- Data Stage, RedShift, QuickSight, S3
- data migration/ ETL both batch and real time
- data warehouse development
- DevSecOps
- Java
- SQL
- relational databases
- GitHub/lab experience
Nice to have:
- Data quality
- xml
- AWS Data Specialty certification
All profiles will be reviewed against the required skills and experience. Due to the high number of applications we will only be able to respond to successful applicants in the first instance. We thank you for your interest and the time taken to apply!
Be The First To Know
About the latest Spark Jobs in London !
Data Engineer
Posted 9 days ago
Job Viewed
Job Description
Data Engineer
I am working with a leading, AI driven organisation that supports its clients' marketing campaigns, expanding its data engineering capabilities and looking for a Data Engineer to join their team. This is a fantastic opportunity to join an award-winning tech business where you will have the opportunity to work at the forefront of data architecture, building distributed systems and leveraging Gen AI tools to accelerate development and improve quality.
You will be encouraged to explore new ways to utilise modern technologies and will be positioned to help mentor other members of the technical team whilst leading innovative technical projects.
As part of this role, you will be responsible for some of the following areas:
- Design and build distributed data pipelines using languages such as Spark, Scala, and Java
- Collaborate with cross-functional teams to deliver user-centric solutions
- Lead on the design and development of relational and non-relational databases
- Apply Gen AI tools to boost development speed and code quality
- Maintain and optimise large-scale data collection processes
- Support the deployment of machine learning models into production
To be successful in the role you will have:
- Creating scalable ETL jobs using Scala and Spark
- Strong understanding of data structures, algorithms, and distributed systems
- Experience working with orchestration tools such as Airflow
- Familiarity with cloud technologies (AWS or GCP)
- Hands-on experience with Gen AI tools for coding and debugging
This is a remote-first role with flexibility to work from anywhere in the UK. Some of the benefits included with the role are listed below:
- Salary of up to 140,000
- Unlimited holiday allowance
- Fully covered private medical care
- Enhanced parental leave policies
- Bespoke training and career development plans
- Regular socials, volunteer days, and a birthday day off
This is just a brief overview of the role. For the full information, simply apply with your CV and I'll be in touch to discuss further. Interviews are already underway, so don't miss out - APPLY now! To do so, please email me at (url removed) or call directly on (phone number removed).
Data Engineer
Posted 9 days ago
Job Viewed
Job Description
£55000 GBP
Onsite WORKING
Location: Central London, Greater London - United Kingdom Type: Permanent
Data Engineer - Generative AI
Hybrid (2-3 days on-site) - UK-based
£5,000- 5,000 + Benefits
Join a growing international team at the forefront of Generative AI and data innovation. You'll be working within a developing, cross-functional environment, building and maintaining high-quality,.
WHJS1_UKTJ
Data Engineer
Posted today
Job Viewed
Job Description
Data Engineer , specialising in designing and optimising scalable data pipelines, and cloud-based data solutions. This position requires expertise in Python, SQL, dpt and experience with Snowflake. If you have industry experience working with Retail and CPG this could be the role for you!nOur client is the leading collaborative commerce platform that connects CPG brands to real-time sales and inventory data from 40+ retailers and distributors. We serve 7,000+ brands representing over $2.5T in retail sales across 250K+ stores, working with industry leaders like Nestlé, and KraftHeinz. This business believes that teamwork drives innovation. By uniting talented, motivated creators from diverse backgrounds, they are tackling one of the worlds most pressing challenges: making the food supply more sustainable, resilient, and efficient. If youre passionate about solving meaningful problems while building a company youll be proud of, wed love to hear from you.nAbout the positionnAs a Data Engineer, youll play a vital role in turning complex, messy datasets into reliable, actionable insights that drive smarter decisions for our customers. Youll design and optimise scalable data pipelines, ensure data quality, and help evolve our platform to handle the growing demands of the global food supply chain.nIn this role, you wont just build pipelines, youll help shape the future of our platform. From performance tuning and architecture improvements to enabling new customer capabilities, your impact will extend far beyond day-to-day operations.nThis is an evolving position with ample opportunity for growth. Whether your background is in startups or larger organisations, youll thrive here if you enjoy ownership, impact, and the chance to define your role as the business scales.nWhat youll work onnDesign, build, and optimise scalable data pipelines and workflows.nCollect, transform, and normalise datasets to ensure quality and reliability.nCollaborate with teams across the business to align solutions with real-world business needs.nContribute to the evolution of the business data platform by driving improvements in performance and architecture.nSigns of a great candidate for the rolenData oriented:
You think in systems and live by the mantra garbage in, garbage out.nBusiness-focused:
You design robust pipelines that align with stakeholder needs and drive meaningful outcomes.nTechnical expertise:nAdvanced
Python
skills for data engineering tasks.nExperience building and maintaining web scraping pipelines.nStrong
SQL
skills, with expertise in performance tuning.nStrong proficiency with
dbt
for data transformations.nHands-on experience with Apache Airflow or Prefect.nProficiency with GitHub, GitHub Actions, and CI/CD pipelines.nNice to have:nExperience with GCP (BigQuery, Dataflow, Composer, Pub/Sub) or AWS.nFamiliarity with Snowflake.nKnowledge of object-oriented programming (Kotlin, Java, or C#).nRetail data experience (sales, inventory, or related datasets).nWhat would make you a great fit:nCollaboration:
You believe the best results come from working together. You share ideas, pitch in, and elevate those around you.nGrit:
Youre curious, self-driven, and unafraid to roll up your sleeves. You get the job done even when the path isnt clear and adapt quickly when things change.nPeople:
You stay close to those we serve. Listening, learning, and building what matters most.nFeedback:
You see it as fuel. You give it with care, take it with humility, and use it to level up.nIngenuity:
You solve problems with creativity and speed. You look for ways to streamline, automate, or improve without being asked.nWe are committed to transparency, diversity, and meritocracy, fostering an environment where every team member is empowered to make an impact, grow personally, and advance in their career. We invite you to join us
not just to take on a role, but to help shape a company youre proud to be part of.nApplicants must be authorized to work for any employer in the U.K. or Poland. The business is unable to sponsor or take over sponsorship of an employment Visa at this time.
TPBN1_UKTJn