CrawlJobs Logo

Data Lake SME

https://www.hpe.com/ Logo

Hewlett Packard Enterprise

Location Icon

Location:
India , Bangalore

Category Icon

Job Type Icon

Contract Type:
Employment contract

Salary Icon

Salary:

Not provided

Job Description:

We are looking for an experienced Data Lake / ETL Engineer with 7+ years of expertise in designing, developing, and managing large-scale data ingestion, transformation, and analytics pipelines. The role involves building scalable and secure data lake platforms, enabling business insights through efficient ETL/ELT frameworks, and ensuring data quality, performance, and governance across the enterprise ecosystem.

Job Responsibility:

  • Design and implement data ingestion pipelines for structured, semi-structured, and unstructured data
  • Develop and manage ETL/ELT processes for large-scale data processing
  • Optimize storage and retrieval strategies across on-prem and cloud-based data lakes
  • Integrate data from multiple sources (databases, APIs, streaming platforms)
  • Implement real-time and batch processing using Apache Spark, Kafka, or Flink
  • Support metadata management, data lineage, and cataloging
  • Tune queries and pipelines for high performance and cost efficiency
  • Implement partitioning, indexing, and caching strategies for large datasets
  • Automate routine ETL/ELT workflows for reliability and speed
  • Ensure compliance with data governance, privacy, and regulatory standards (GDPR, HIPAA, etc.)
  • Implement encryption, masking, and role-based access control (RBAC)
  • Collaborate with cybersecurity teams to align with Zero Trust and IAM policies
  • Partner with data scientists, analysts, and application teams for analytics enablement
  • Provide L2/L3 support for production pipelines and troubleshoot failures
  • Mentor junior engineers and contribute to best practices documentation

Requirements:

  • 7+ years of experience in data engineering, ETL/ELT development, or data lake management
  • Strong expertise in ETL tools (Informatica, Talend, dbt, SSIS, or similar)
  • Hands-on experience with big data ecosystems: Hadoop, Spark, Hive, Presto, Delta Lake, or Iceberg
  • Proficiency with SQL, Python, or Scala for data processing and transformation
  • Experience with cloud data platforms (AWS Glue, Redshift, Azure Synapse, GCP BigQuery)
  • Familiarity with workflow orchestration tools (Airflow, Temporal, Oozie)

Nice to have:

  • Exposure to real-time data streaming (Kafka, Kinesis, Pulsar)
  • Knowledge of data modeling (Kimball/Inmon), star schema, and dimensional modeling
  • Experience with containerized deployments (Docker, Kubernetes)
  • Informatica/Talend/dbt certifications
What we offer:
  • Health & Wellbeing
  • Personal & Professional Development
  • Unconditional Inclusion

Additional Information:

Job Posted:
October 08, 2025

Employment Type:
Fulltime
Work Type:
On-site work
Job Link Share:

Looking for more opportunities? Search for other job offers that match your skills and interests.

Briefcase Icon

Similar Jobs for Data Lake SME

Pre-Sales Solution Engineer

We are seeking a highly skilled Enterprise focused Pre-Sales Solution Engineer t...
Location
Location
Salary
Salary:
Not provided
lakefs.io Logo
LakeFS
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Bachelor’s degree in Computer Science, Engineering, or a related technical field
  • 5+ years of experience in a technical pre-sales or solution engineering role, preferably in the software industry
  • Excellent verbal and written communication skills, with the ability to deliver technical presentations to both technical and non-technical audiences
  • Hands-on experience with Kubernetes and containerized environments
  • Experience leading client workshops, sales enablement sessions, and technical training
  • Ability to work independently as well as collaboratively across sales, product, and engineering teams
  • Excellent problem-solving skills and a creative approach to architecting solutions
  • Experience in technical sales of open source products is preferred
  • Strong understanding and hands-on experience in the Data & AI ecosystem, including: Data lake and data warehouse technologies (e.g., S3, Delta Lake, Iceberg, Hive, Glue, Snowflake)
  • Data processing frameworks (e.g., Spark, Databricks, Flink, Trino, Presto)
Job Responsibility
Job Responsibility
  • Own the technical win from discovery through demo, architecture design, pilot/POC, security review, and a clean handoff to Customer Success
  • Design and present tailored demos that showcase branch‑based workflows, reproducible experiments, schema‑safe changes, and instant data rollback
  • Scope and execute pilots with clear success criteria
  • create sample repos, notebooks, and automation (Spark/Databricks jobs, Airflow DAGs, CI pipelines) that prove value quickly
  • Build reference architectures for lakeFS OSS and Cloud across AWS/Azure/GCP
  • document IAM roles, private networking, scaling, GC/performance tuning, and disaster recovery
  • Answer deep technical questions as a lakeFS SME
  • Handle RFPs and security questionnaires
  • map controls to customer requirements and recommend compliant deployment patterns
  • Partner with sales to quantify business impact (risk reduction, developer velocity, storage efficiency) and co‑create the ROI/TCO narrative with champions
Read More
Arrow Right

Senior Bigdata Engineer

The Applications Development Senior Programmer Analyst is an intermediate level ...
Location
Location
India , Pune
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 8 - 10 years of relevant experience
  • Experience in systems analysis and programming of software applications
  • Experience in managing and implementing successful projects
  • Working knowledge of consulting/project management techniques/methods
  • Ability to work under pressure and manage deadlines or unexpected changes in expectations or requirements
  • Programming Languages: Python, PySpark
  • Data Lake Table Format: Apache Iceberg
  • Data Orchestration: Apache Airflow
  • Data Visualization: Tableau
  • Big Data Processing: Apache Spark
Job Responsibility
Job Responsibility
  • Conduct tasks related to feasibility studies, time and cost estimates, IT planning, risk technology, applications development, model development, and establish and implement new or revised applications systems and programs to meet specific business needs or user areas
  • Monitor and control all phases of development process and analysis, design, construction, testing, and implementation as well as provide user and operational support on applications to business users
  • Utilize in-depth specialty knowledge of applications development to analyze complex problems/issues, provide evaluation of business process, system process, and industry standards, and make evaluative judgement
  • Recommend and develop security measures in post implementation analysis of business usage to ensure successful system design and functionality
  • Consult with users/clients and other technology groups on issues, recommend advanced programming solutions, and install and assist customer exposure systems
  • Ensure essential procedures are followed and help define operating standards and processes
  • Serve as advisor or coach to new or lower level analysts
  • Has the ability to operate with a limited level of direct supervision
  • Can exercise independence of judgement and autonomy
  • Acts as SME to senior stakeholders and /or other team members
What we offer
What we offer
  • Equal employment opportunity
  • Fulltime
Read More
Arrow Right

Senior Cloud Data Architect

As a Senior Cloud Architect, your role will focus on supporting users, collabora...
Location
Location
Spain , Barcelona
Salary
Salary:
Not provided
https://www.allianz.com Logo
Allianz
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Strong expertise in Azure cloud infrastructure, Data & AI technologies, and data platform management, with proficiency in Azure Synapse Analytics, Azure Machine Learning, Azure Data Lake, and Informatica Intelligent Data Management Cloud (IDMC)
  • Proven experience in modern Data Warehouse architectures (e.g., Lakehouse) and integrating machine learning models and AI capabilities using Azure services like Cognitive Services and Azure Bot Service for predictive analytics and automation
  • In-depth knowledge of data security and compliance practices using Azure AD, Azure Key Vault, and Informatica’s data governance tools, focusing on data privacy and regulatory standards
  • Expertise in optimizing resource usage, performance, and costs across Azure services and IDMC, leveraging tools like Azure Cost Management and Azure Monitor, and skilled in ETL/ELT tools and advanced SQL
  • Proficiency in data integration, machine learning, and generative AI from an architectural perspective, with hands-on experience in Python, SQL, Spark/Scala/PySpark, and container solutions like Docker and Kubernetes
  • Experience with CI/CD pipelines (e.g., GitHub Actions, Jenkins), microservices architectures, and APIs, with knowledge of architecture frameworks like TOGAF or Zachman, adept at managing multiple priorities in fast-paced environments, and excellent communication and presentation skills
  • Over 5 years of experience in cloud architecture focusing on Data & AI infrastructure, particularly in Azure, with expertise in building scalable, secure, and cost-effective solutions for data analytics and AI/ML environments.
Job Responsibility
Job Responsibility
  • Define and prioritize new functional and non-functional capabilities for the cloud-based data platform, ensuring alignment with business needs and Allianz's security, compliance, privacy, and architecture standards
  • Act as the platform SME for both potential and existing users, guiding them in the architecture of scalable, high-performance Data & AI solutions
  • Provide leadership and product guidance to engineering teams during the design, development, and implementation of new platform capabilities
  • Ensure all solutions meet defined quality standards and acceptance criteria
  • Work with stakeholders to co-create data solutions, optimizing business models and identifying opportunities for improved data usage
  • Lead the evaluation and selection of technologies and partners to implement data analytics use cases, focusing on proofs of concept and prototypes
  • Stay up to date with emerging trends in Data, Analytics, AI/ML, and cloud technologies
  • Leverage open-source technologies and cloud tools to drive innovation and cost-efficiency
  • Prepare materials for management briefings and public events
  • Represent the team in technical discussions, particularly regarding architecture and platform capabilities.
What we offer
What we offer
  • Hybrid work model which recognizes the value of striking a balance between in-person collaboration and remote working incl. up to 25 days per year working from abroad
  • Rewarding performance through company bonus scheme, pension, employee shares program, and multiple employee discounts
  • Career development and digital learning programs to international career mobility
  • Flexible working, health and wellbeing offers (including healthcare and parental leave benefits)
  • Support for balancing family and career and helping employees return from career breaks with experience that nothing else can teach.
  • Fulltime
Read More
Arrow Right

Solution Architect

The Solution Architect role involves driving the architectural transformation fo...
Location
Location
Ireland , Dublin
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Significant experience in Data modeling, Data lineage analysis, Operational reporting, preferably in a global organization
  • Proven architecture experience in solutioning of horizontally scalable, highly available, highly resilient data distribution platforms
  • Proficient in message queuing, stream processing, and highly scalable ‘big data’ data stores
  • Advanced working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases
  • Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement
  • Strong analytic skills related to working with unstructured datasets
  • Extensive experience with Data Integration patterns
  • Extensive experience with Real/Near Real time streaming patterns
  • Strong background in Data Management, Data Governance, Transformation initiatives preferred
  • Preferred Experience/Familiarity with one or more of these tools: Big data platforms - Hadoop, Apache Kafka, Relational SQL, NoSQL, and Cloud Native databases - Postgres, Cassandra, Snowflake, Experience with data pipeline and orchestration tools - Azkaban, Luigi, or Airflow, Experience with stream-processing engines - Apache Spark, Apache Storm, or Apache Flink and ETL tools - Talend, Ab Initio, Experience with Data Analytics/visualization tools - Looker, Mode, or Tableau
Job Responsibility
Job Responsibility
  • Re-engineering the interaction of incoming and outgoing data flows from the Core Accounts DDA platform to Reference Data platforms, Data Warehouse, Data Lake as well as other local reporting systems which consume data from Core Accounts
  • Drive data architecture and roadmap for eliminating non-strategic point-to-point connections and batch handoffs
  • Define canonical data models for key entities and events related to Customer, Account, Core DDA in line with the Data Standards
  • Assess opportunities to simplify/rationalize/refactor the existing database schemas paving way for modularization of the existing stack
  • Provide technical guidance to Data Engineers responsible for designing an Operational Data Store for intra-day and end-of-day reporting
  • Implementing data strategies and developing logical and physical data models
  • Formulate efficient approach to rationalize and formulate strategy to migrate reports
  • Build and nurture a strong engineering organization to deliver value to internal and external clients
  • Acts as SME to senior stakeholders in business, operations, and technology divisions across upstream and downstream Organizations
  • Monitor and control all phases of development process and analysis, design, construction, testing, and implementation as well as provide user and operational support on applications to business users
What we offer
What we offer
  • Competitive base salary (annually reviewed)
  • Hybrid working model (up to 2 days working at home per week)
  • Additional benefits supporting you and your family
  • Fulltime
Read More
Arrow Right
New

Data Engineer

The Amgen India Digital Technology & Innovation Data Engineer for Clinical Data ...
Location
Location
India , Hyderabad
Salary
Salary:
Not provided
amgen.com Logo
Amgen
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • Master's degree / Bachelor's degree and 5 to 9 years [Job Code’s Discipline and/or Sub-Discipline]
  • Advanced skills in SQL, Python, and R languages programing
  • AWS cloud technology and databricks data lake technology stacks
  • Proficient skills and knowledge on common AI/Machine Learning technologies.
  • Data modeling skills, and software development lifecycle knowledge and standard processes
  • Learning ability of new technology in the information field
  • Skill of using DevOps CI/CD tools, such Git, Jenkins and front UI Visualization technology
Job Responsibility
Job Responsibility
  • Design, develop, and deploy data pipeline for clinical domain dataset
  • As an infrastructure programmer, continuously develop and support Data Scientist R-Platform and integration with various technology (Kubernetes Container, HashiCorp Vault, SAS Storage, and Data Science Work Bench)
  • Design and build various reusable program components using innovative technology (NLP, AI, Python, R, etc) to transform and harmonize clinical dataset for insight generation
  • Collaborate with Data Architects, Business SME’s, and Data Scientists to capture the business requirement and translate into Agile product backlog
  • Serve as primary data engineer to manage and support AWS, Databricks, RStudio platform, and cloud AI based system production DevOps
  • Align to best practices for coding, testing, and designing reusable code/component
  • Explore new tools and technologies that will help streamline data pipeline and add new durable capability for clinical development
  • Participate in sprint planning meetings and provide estimations on technical implementation
  • Collaborate and communicate effectively with the product teams
What we offer
What we offer
  • competitive and comprehensive Total Rewards Plans that are aligned with local industry standards
Read More
Arrow Right
New

Solutions Architect

Ampa (home of Shakespeare Martineau, Mayo Wynne Baxter, Marrons and further bran...
Location
Location
United Kingdom , Flexible
Salary
Salary:
Not provided
ampa.co.uk Logo
Ampa - Legal & Professional Services
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • A minimum of 3 years demonstrable track record of successful solution design and delivery of complex technical and service solutions
  • Experience of solutions architecture is essential
  • Certified in and experienced with one or more Architecture Frameworks
  • Evaluating technical opportunities and providing innovative solutions
  • A demonstrable track record in the successful design and delivery of complex solutions
  • Working with various roles, capabilities and skills within an organisation providing the delivery capability needed to ensure successful business outcomes
  • The ability to drive through innovations and best practices from concept to implementation
  • Development techniques
  • Domain-driven design, Scrum/Agile, API Design Principles
  • Significant previous experience with Microsoft Power Platform
Job Responsibility
Job Responsibility
  • Ensures the effective design & delivery of solutions and systems both on-prem and in the cloud, ensuring high-level analysis of requirements and existing solutions
  • Working closely with Enterprise Architecture and technical SME, the Solutions Architect stays informed of current and emerging technologies ensuring innovative solutions
  • The Technology Solutions Architect will form part of a wider architecture landscape working collaboratively with teams across Technology, Data, Security and Business domains
  • Owning the Solutions roadmap, the Solutions Architect is pivotal in supporting change initiatives and plays a key role as part of the Technical Design Authority
  • Commensurate with this responsibility, this role is responsible for the assurance of and where necessary the production of solution design documentation whether this be internal or through partner suppliers
  • Having responsibility for the architecture of all solutions delivered to the business, both from internal and external suppliers
  • Facilitating the handover from development and pilot into supported production, ensuring that appropriate documentation and support models are in place
  • Own architectural blueprint for our solutions
  • Defining and monitoring solutions to ensure benefits are realised
  • Having responsibility for leading key strategic changes and influencing the business
What we offer
What we offer
  • Agile working
  • Blended approach to where and how we work
  • Genuine flexibility
  • Award-winning development
  • Multi-brand experiences
  • Diverse expertise
  • Modern tech
  • Culture that encourages you to think boldly, be yourself, and shape a career that excites you
  • Fulltime
Read More
Arrow Right

Senior Python Pyspark Engineer

The Applications Development Senior Programmer Analyst is an intermediate level ...
Location
Location
India , Pune
Salary
Salary:
Not provided
https://www.citi.com/ Logo
Citi
Expiration Date
Until further notice
Flip Icon
Requirements
Requirements
  • 8 - 10 years of relevant experience
  • Experience in systems analysis and programming of software applications
  • Experience in managing and implementing successful projects
  • Working knowledge of consulting/project management techniques/methods
  • Ability to work under pressure and manage deadlines or unexpected changes in expectations or requirements
  • Programming Languages:Python, PySpark
  • Data Lake Table Format: Apache Iceberg
  • Data Orchestration:Apache Airflow
  • Data Visualization: Tableau
  • Big Data Processing: Apache Spark
Job Responsibility
Job Responsibility
  • Conduct tasks related to feasibility studies, time and cost estimates, IT planning, risk technology, applications development, model development, and establish and implement new or revised applications systems and programs to meet specific business needs or user areas
  • Monitor and control all phases of development process and analysis, design, construction, testing, and implementation as well as provide user and operational support on applications to business users
  • Utilize in-depth specialty knowledge of applications development to analyze complex problems/issues, provide evaluation of business process, system process, and industry standards, and make evaluative judgement
  • Recommend and develop security measures in post implementation analysis of business usage to ensure successful system design and functionality
  • Consult with users/clients and other technology groups on issues, recommend advanced programming solutions, and install and assist customer exposure systems
  • Ensure essential procedures are followed and help define operating standards and processes
  • Serve as advisor or coach to new or lower level analysts
  • Has the ability to operate with a limited level of direct supervision.
  • Can exercise independence of judgement and autonomy.
  • Acts as SME to senior stakeholders and /or other team members.
  • Fulltime
Read More
Arrow Right
New

Swedish Language Expert & Complaints Associate

Are you fluent in Swedish? Would you like to fire up your career by working as a...
Location
Location
Greece , Athens
Salary
Salary:
1450.00 EUR / Month
https://www.randstad.com Logo
Randstad
Expiration Date
December 30, 2026
Flip Icon
Requirements
Requirements
  • Eligible to work in Greece
  • An exceptional level in the Swedish Language
  • Advanced English skills
  • Excellent verbal & written communication skills
  • PC literacy
Job Responsibility
Job Responsibility
  • Manage incoming requests & ensure an utmost customer satisfaction
  • Resolve customer inquiries regarding customer suggestions
  • Provide accurate, valid & complete information by using the right methods/tools
  • Build sustainable relationships of trust through open & interactive communication
  • Promote additional features relevant to the client’s new services & training material
  • Verify and correct machine-generated translations of answers to customer queries
  • Understand and translate the meaning and tone of written words from one language to another
  • Recognize and provide feedback on issues found in the email
  • Fulltime
Read More
Arrow Right