CrawlJobs Logo

Data Engineer - AWS, PySpark, DevOps

barclays.co.uk Logo

Barclays

Location Icon

Location:
India , Bengaluru ▼

Category Icon
Category:
IT - Software Development

Job Type Icon

Contract Type:
Not provided

Salary Icon

Salary:

Not provided
Save Job
Save Icon
Apply Position

Job Description:

You will be responsible for supporting the successful delivery of Location Strategy projects to plan, budget, agreed quality and governance standards. You'll spearhead the evolution of our digital landscape, driving innovation and excellence. You will harness cutting-edge technology to revolutionise our digital offerings, ensuring unparalleled customer experiences. Purpose of the role: To build and maintain the systems that collect, store, process, and analyse data, such as data pipelines, data warehouses and data lakes to ensure that all data is accurate, accessible, and secure.

Job Responsibility:

  • Support the successful delivery of Location Strategy projects to plan, budget, agreed quality and governance standards
  • Spearhead the evolution of our digital landscape, driving innovation and excellence
  • Harness cutting-edge technology to revolutionise our digital offerings, ensuring unparalleled customer experiences
  • Build and maintenance of data architectures pipelines that enable the transfer and processing of durable, complete and consistent data
  • Design and implementation of data warehoused and data lakes that manage the appropriate data volumes and velocity and adhere to the required security measures
  • Development of processing and analysis algorithms fit for the intended data complexity and volumes
  • Collaboration with data scientist to build and deploy machine learning models

Requirements:

  • Hands on experience in pyspark and strong knowledge on Dataframes, RDD and SparkSQL
  • Hands on Experience in developing, testing and maintaining applications on AWS Cloud
  • Strong hold on AWS Data Analytics Technology Stack (Glue, S3, Lambda, Lake formation, Athena)
  • Design and implement scalable and efficient data transformation/storage solutions using Snowflake
  • Experience in Data ingestion to Snowflake for different storage format such Parquet, Iceberg, JSON, CSV etc
  • Experience in using DBT (Data Build Tool) with snowflake for ELT pipeline development
  • Experience in Writing advanced SQL and PL SQL programs
  • Hands On Experience for building reusable components using Snowflake and AWS Tools/Technology
  • Should have worked at least on two major project implementations

Nice to have:

  • Exposure to data governance or lineage tools such as Immuta and Alation is added advantage
  • Experience in using Orchestration tools such as Apache Airflow or Snowflake Tasks is added advantage
  • Knowledge on Abinitio ETL tool is a plus
  • Ability to engage with Stakeholders, elicit requirements/ user stories and translate requirements into ETL components
  • Ability to understand the infrastructure setup and be able to provide solutions either individually or working with teams
  • Good knowledge of Data Marts and Data Warehousing concepts
  • Resource should possess good analytical and Interpersonal skills
  • Implement Cloud based Enterprise data warehouse with multiple data platform along with Snowflake and NoSQL environment to build data movement strategy
What we offer:
  • Competitive holiday allowance
  • Life assurance
  • Private medical care
  • Pension contribution
  • Modern workspaces, collaborative areas, and state-of-the-art meeting rooms
  • Facilities include wellness rooms, on-site cafeterias, fitness centers, and tech-equipped workstations
  • Hybrid working

Additional Information:

Job Posted:
January 07, 2026

Employment Type:
Fulltime
Work Type:
Hybrid work
Icon
Barclays - All Job Offers
Job Link Share:
Copy Link Icon
Facebook Icon Twitter Icon LinkedIn Icon

Looking for more opportunities? Search for other job offers that match your skills and interests.

Briefcase Icon

Similar Jobs for Data Engineer - AWS, PySpark, DevOps

Software Engineer (Data Engineering)

We are seeking a Software Engineer (Data Engineering) who can seamlessly integra...
Location
Location
India , Hyderabad
Salary
Salary:
Not provided
nstarxinc.com Logo
NStarX
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 4+ years in Data Engineering and AI/ML roles
  • Bachelor’s or Master’s degree in Computer Science, Data Science, or a related field
  • Python, SQL, Bash, PySpark, Spark SQL, boto3, pandas
  • Apache Spark on EMR (driver/executor model, sizing, dynamic allocation)
  • Amazon S3 (Parquet) with lifecycle management to Glacier
  • AWS Glue Catalog and Crawlers
  • AWS Step Functions, AWS Lambda, Amazon EventBridge
  • CloudWatch Logs and Metrics, Kinesis Data Firehose (or Kafka/MSK)
  • Amazon Redshift and Redshift Spectrum
  • IAM (least privilege), Secrets Manager, SSM
Job Responsibility
Job Responsibility
  • Design, build, and maintain scalable ETL and ELT pipelines for large-scale data processing
  • Develop and optimize data architectures supporting analytics and ML workflows
  • Ensure data integrity, security, and compliance with organizational and industry standards
  • Collaborate with DevOps teams to deploy and monitor data pipelines in production environments
  • Build predictive and prescriptive models leveraging AI and ML techniques
  • Develop and deploy machine learning and deep learning models using TensorFlow, PyTorch, or Scikit-learn
  • Perform feature engineering, statistical analysis, and data preprocessing
  • Continuously monitor and optimize models for accuracy and scalability
  • Integrate AI-driven insights into business processes and strategies
  • Serve as the technical liaison between NStarX and client teams
What we offer
What we offer
  • Competitive salary and performance-based incentives
  • Opportunity to work on cutting-edge AI and ML projects
  • Exposure to global clients and international project delivery
  • Continuous learning and professional development opportunities
  • Competitive base + commission
  • Fast growth into leadership roles
  • Fulltime
Read More
Arrow Right

Data Engineer

At Ivy Partners, we are a Swiss consultancy firm helping businesses navigate the...
Location
Location
Portugal , Lisboa
Salary
Salary:
Not provided
ivy.partners Logo
IVY Partners
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Over 4 years of professional experience working as a Data Engineer or in a related field
  • Solid hands-on experience with AWS cloud services
  • Practical experience using Databricks and developing data solutions with PySpark
  • Strong command of Python and SQL for data processing, automation, and analysis
  • Familiar with DevOps principles, including CI/CD, monitoring, and infrastructure as code
Job Responsibility
Job Responsibility
  • Build and manage scalable data pipelines and infrastructure
  • Utilize AWS, Databricks, PySpark, and SQL to ensure reliable, high-quality, and efficient data processing
  • Partner with cross-functional teams to enable data-informed decisions across the organization
  • Implement DevOps methodologies to streamline deployment, monitoring, and automation of data solutions
  • Support and enhance standards for data governance, system performance, and security best practices
What we offer
What we offer
  • Caring for Our Staff | Providing a supportive environment where everyone is valued, with training and progression opportunities both in Switzerland and globally
  • Fostering Trust | Working with us means building a relationship based on transparency, professionalism, and commitment
  • Encouraging Innovation | We merge technology and creativity to drive impactful digital transformations
  • Taking Responsibility | Our collective action is at the heart of our operations, aiming to make a positive impact
  • Fulltime
Read More
Arrow Right

Graduate Data Engineer

As a Graduate Data Engineer, you will build and maintain scalable data pipelines...
Location
Location
United Kingdom , Marlow
Salary
Salary:
Not provided
srgtalent.com Logo
SRG
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Degree in Computer Science, Engineering, Mathematics, or similar, or similar work experience
  • Up to 2 years of experience building data pipelines at work or through internships
  • Can write clear and reliable Python/PySpark code
  • Familiar with popular analytics tools (like pandas, numpy, matplotlib), big data frameworks (like Spark), and cloud services (like Palantir, AWS, Azure, or Google Cloud)
  • Deep understanding of data models, relational and non-relational databases, and how they are used to organize, store, and retrieve data efficiently for analytics and machine learning
  • Knowledge about software engineering methods, including DevOps, DataOps, or MLOps is a plus
  • Master's degree in engineering (such as AI/ML, Data Systems, Computer Science, Mathematics, Biotechnology, Physics), or minimum 2 years of relevant technology experience
  • Experience with Generative AI (GenAI) and agentic systems will be considered a strong plus
  • Have a proactive and adaptable mindset: willing to take initiative, learn new skills, and contribute to different aspects of a project as needed to drive solutions from start to finish, even beyond the formal job description
  • Show a strong ability to thrive in situations of ambiguity, taking initiative to create clarity for yourself and the team, and proactively driving progress even when details are uncertain or evolving
Job Responsibility
Job Responsibility
  • Build and maintain data pipelines, leveraging PySpark and/or Typescript within Foundry, to transform raw data into reliable, usable datasets
  • Assist in preparing and optimizing data pipelines to support machine learning and AI model development, ensuring datasets are clean, well-structured, and readily usable by Data Science teams
  • Support the integration and management of feature engineering processes and model outputs into Foundry's data ecosystem, helping enable scalable deployment and monitoring of AI/ML solutions
  • Engaged in gathering and translating stakeholder requirements for key data models and reporting, with a focus on Palantir Foundry workflows and tools
  • Participate in developing and refining dashboards and reports in Foundry to visualize key metrics and insights
  • Collaborate with Product, Engineering, and GTM teams to align data architecture and solutions, learning to support scalable, self-serve analytics across the organization
  • Have some prompt engineering experience with large language models, including writing and evaluating complex multi-step prompts
  • Continuously develop your understanding of the company's data landscape, including Palantir Foundry's ontology-driven approach and best practices for data management
Read More
Arrow Right

Data Engineering Lead

Embark on an exciting journey into the realm of software product development wit...
Location
Location
India
Salary
Salary:
Not provided
3pillarglobal.com Logo
3Pillar Global
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 8+ years of experience in Data Engineering or related field, including 2+ years in a lead role
  • Expert-level proficiency with AWS data services (e.g., Glue, EMR, Lambda, Redshift, S3, Kinesis, Step Functions)
  • Strong Python skills for data processing, automation, and pipeline development
  • Experience building batch and streaming pipelines (Spark, PySpark, Kafka, Kinesis, etc.)
  • Strong SQL expertise and experience with relational and NoSQL databases
  • Hands-on experience with IaC (Terraform, CloudFormation, CDK)
  • Familiarity with DevOps tools for CI/CD (e.g., GitHub Actions, GitLab CI, Jenkins)
  • Understanding of data modeling, data warehousing concepts, and distributed systems
  • Fulltime
Read More
Arrow Right

DevOps Engineer

BioCatch is the leader in Behavioral Biometrics, a technology that leverages mac...
Location
Location
Israel , TLV
Salary
Salary:
Not provided
biocatch.com Logo
BioCatch
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 5+ Years of Experience: Demonstrated experience as a DevOps professional, with a strong focus on big data environments, or Data Engineer with strong DevOps skills
  • Data Components Management: Experiences managing and designing data infrastructure, such as Snowflake, PostgreSQL, Kafka, Aerospike, and Object Store
  • DevOps Expertise: Proven experience creating, establishing, and managing big data tools, including automation tasks. Extensive knowledge of DevOps concepts and tools, including Docker, Kubernetes, Terraform, ArgoCD, Linux OS, Networking, Load Balancing, Nginx, etc.
  • Programming Skills: Proficiency in programming languages such as Python and Object-Oriented Programming (OOP), emphasizing big data processing (like PySpark). Experience with scripting languages like Bash and Shell for automation tasks
  • Cloud Platforms: Hands-on experience with major cloud providers such as Azure, Google Cloud, or AWS
Job Responsibility
Job Responsibility
  • Data Architecture Direction: Provide strategic direction for our data architecture, selecting the appropriate componments for various tasks. Collaborate on requirements and make final decisions on system design and implementation
  • Project Management: Manage end-to-end execution of high-performance, large-scale data-driven projects, including design, implementation, and ongoing maintenance
  • Cost Optimization: Monitor and optimize cloud costs associated with data infrastructure and processes
  • Efficiency and Reliability: Design and build monitoring tools to ensure the efficiency, reliability, and performance of data processes and systems
  • DevOps Integration: Implement and manage DevOps practices to streamline development and operations, focusing on infrastructure automation, continuous integration/continuous deployment (CI/CD) pipelines, containerization, orchestration, and infrastructure as code. Ensure scalable, reliable, and efficient deployment processes
  • Fulltime
Read More
Arrow Right

Data Test Engineer

We are looking for a skilled Data Test Engineer who can design, build, and valid...
Location
Location
India , Chennai
Salary
Salary:
Not provided
optisolbusiness.com Logo
OptiSol Business Solutions
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 4+ years of experience in Data Engineering and Data/ETL Testing
  • Strong expertise in writing and optimizing SQL queries (joins, subqueries, window functions, performance tuning)
  • Proficiency in Python or PySpark for data transformation and automation
  • Hands-on experience with ETL tools such as Azure Data Factory, Talend, SSIS, or Informatica
  • Familiarity with cloud platforms, preferably Azure
  • AWS or GCP is a plus
  • Experience working with data lakes, data warehouses (Snowflake, BigQuery, Redshift), and modern data platforms
  • Knowledge of version control systems (Git), issue tracking tools (JIRA), and Agile methodologies
  • Exposure to data testing frameworks like Great Expectations, DBT tests, or custom validation tools
  • Experience integrating data testing into CI/CD pipelines
Job Responsibility
Job Responsibility
  • Design, develop, and maintain robust ETL/ELT pipelines to process large volumes of structured and unstructured data using Azure Data Factory, PySpark, and SQL-based tools
  • Collaborate with data architects and analysts to understand transformation requirements and implement business rules correctly
  • Develop and execute complex SQL queries to validate, transform, and performance-tune data workflows
  • Perform rigorous data validation including source-to-target mapping (S2T), data profiling, reconciliation, and transformation rule testing
  • Conduct unit, integration, regression, and performance testing for data pipelines and storage layers
  • Automate data quality checks using Python and frameworks like Great Expectations, DBT, or custom-built tools
  • Monitor data pipeline health and implement observability through logging, alerting, and dashboards
  • Integrate testing into CI/CD workflows using tools like Azure DevOps, Jenkins, or GitHub Actions
  • Troubleshoot and resolve data quality issues, schema changes, and pipeline failures
  • Ensure compliance with data privacy, security, and governance policies
What we offer
What we offer
  • Competitive salary aligned with industry standards
  • Hands-on experience with enterprise-scale data platforms and cloud-native tools
  • Opportunities to work on data-centric initiatives across AI, analytics, and enterprise transformation
  • Access to internal learning accelerators, mentorship, and career growth programs
  • Flexible work culture, wellness initiatives, and comprehensive health benefits
  • Fulltime
Read More
Arrow Right

Data Architect

Embark on an exciting journey into the realm of data engineering and architectur...
Location
Location
Canada
Salary
Salary:
Not provided
3pillarglobal.com Logo
3Pillar Global
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Assessments of existing data components, Performing POCs, Consulting to the stakeholders
  • Proposing end to end solutions to an enterprise's data specific business problems, and taking care of data collection, extraction, integration, cleansing, enriching and data visualization
  • Ability to design large data platforms to enable Data Engineers, Analysts & scientists
  • Strong exposure to different Data architectures, data lake & data warehouse
  • Define tools & technologies to develop automated data pipelines, write ETL processes, develop dashboard & report and create insights
  • Continually reassess current state for alignment with architecture goals, best practices and business needs
  • DB modeling, deciding best data storage, creating data flow diagrams, maintaining related documentation
  • Taking care of performance, reliability, reusability, resilience, scalability, security, privacy & data governance while designing a data architecture
  • Apply or recommend best practices in architecture, coding, API integration, CI/CD pipelines
  • Coordinate with data scientists, analysts, and other stakeholders for data-related needs
Job Responsibility
Job Responsibility
  • Translate business requirements into data requests, reports and dashboards.
What we offer
What we offer
  • Flexible work environment (office, home, or blend)
  • Remote-first approach
  • Part of a global team, learning from top talent
  • Well-being focus (fitness offerings, mental health plans, generous time off)
  • Accelerated career growth and development opportunities across projects, offerings, and industries
  • Equal opportunity employer
  • Values like Intrinsic Dignity and Open Collaboration
  • Fulltime
Read More
Arrow Right

Data Architect

Embark on an exciting journey into the realm of data engineering and architectur...
Location
Location
India , Noida
Salary
Salary:
Not provided
3pillarglobal.com Logo
3Pillar Global
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Translate business requirements into data requests, reports and dashboards.
  • Strong Database & modeling concepts with exposure to SQL & NoSQL Databases
  • Strong data architecture patterns & principles, ability to design secure & scalable data lakes, data warehouse, data hubs, and other event-driven architectures
  • Expertise in designing and writing ETL processes in Python / Java / Scala
  • Understanding of Hadoop framework - Exposure to PySpark, Spark, Storm, HDFS, Hive
  • Strong hands-on experience with either Databricks or Snowflake
  • experience with both is desirable.
  • Knowledge of Master Data management and related tools
  • Strong exposure to data security and privacy regulations (GDPR, HIPAA) and best practices
  • Skilled in ensuring data accuracy, consistency, and quality
Job Responsibility
Job Responsibility
  • Work closely with business leaders and information management teams to define and implement a roadmap on data architecture, data management, business intelligence or analytics solutions.
  • Define a reference architecture for our customers.
  • Support our clients to take control of their data and get value out of it.
  • Fulltime
Read More
Arrow Right
Jobs by Category
Art and Entertainment Jobs Banking Jobs Call Center Jobs Construction Jobs Consulting Jobs Customer Service Jobs Education Jobs Energy Jobs Finance Jobs Gastronomy Jobs Health and Beauty Jobs Hospitality and Tourism Jobs Human Resources Jobs Insurance Jobs IT - Administration Jobs IT - Software Development Jobs Legal Jobs Logistics Jobs Manufacturing Jobs Marketing, Digital Marketing, SEO, SEM Jobs Media Jobs Nursing Jobs Office Administration Jobs Pharmacy Jobs Physical Work Jobs Public Relations Jobs Public Sector Jobs Purchasing Jobs Quality Control Jobs Real Estate Jobs Research and Development Jobs Sales Jobs
Job Positions Jobs by Country
United States Jobs United Kingdom Jobs India Jobs Canada Jobs Australia Jobs Germany Jobs Ireland Jobs Poland Jobs
Company
Blog About Us Contact
Follow Us
Facebook
X (Twitter)
LinkedIn
Instagram
CrawlJobs Group
ITFlashcards AllDevBlogs
© 2026 CrawlJobs Ltd. All Rights Reserved
Terms & Conditions Privacy Policy