Lead Data Engineer

• Lead the migration of legacy Java-based data pipelines to Databricks
• design and maintain scalable data pipelines using Databricks and Spark
• Create, maintain, and update data models
• build infrastructure for optimal data extraction, transformation, and loading (ETL)
• Analyse and optimize Databricks jobs and queries
• monitor and tune Databricks environments for scalability and reliability
• Address data-related technical issues
• assist teams with data transformation workloads
• perform root cause analysis to identify improvement opportunities
• Implement best practices for efficient data processing and storage
• develop processes that support data transformation, workload management, data structures, and metadata
• Mentor a team of data engineers
• serve as the go-to person for Databricks-related queries
• conduct Proof of Concepts (POCs) to demonstrate Databricks capabilities

• Proficiency in Python / PySpark
• Proven experience with Databricks and Spark, including building and optimizing data pipelines
• Strong data modelling experience and advanced SQL knowledge
• Experience with Delta Lake and performance analysis in Databricks
• Knowledge of message queuing, stream processing, and scalable big data stores
• Experience with Talend is preferable
• Must have worked with US based clients in onsite/off shore delivery model
• Strong verbal and written communication, technical articulation, listening, and presentation skills are essential
• Should have proven analytical and problem-solving skills
• Demonstrated expertise in prioritization, time management, and stakeholder management (both internal and external) is necessary
• Should be a quick learner, self-starter, proactive, and an effective team player
• Must have experience working under tight deadlines within a matrix organizational structure

Experience with Talend is preferable