Senior Kafka Engineer

• Identify and rectify Kafka messaging issues within justified time
• Work with the business and IT team to understand business problems and design, implement, and deliver an appropriate solution using Agile methodology across the larger program
• Work independently to implement solutions on multiple platforms (DEV, QA, UAT, PROD)
• Provide technical direction, guidance, and reviews to other engineers working on the same project
• Administer distributed Kafka clusters in Dev, QA, UAT, and PROD environments and troubleshoot performance issues
• Implement and debug subsystems/microservices and components
• Follow an automate-first/automate-everything philosophy
• Hands-on in programming languages

• Deep understanding of Confluent Kafka: Thorough knowledge of Kafka concepts like producers, consumers, topics, partitions, brokers, and replication mechanisms
• Programming language proficiency: Primarily Java or Scala, with potential for Python depending on the project
• System design and architecture: Ability to design robust and scalable Kafka-based data pipelines, considering factors like data throughput, fault tolerance, and latency
• Data management skills: Understanding of data serialization formats like JSON, Avro, and Protobuf, and how to manage data schema evolution
• Monitoring and troubleshooting: Familiarity with tools to monitor Kafka cluster health, identify performance bottlenecks, and troubleshoot issues
• Cloud integration: Experience deploying and managing Kafka on cloud platforms like AWS, Azure, or GCP
• Distributed systems concepts

Kafka Streams API (optional): Knowledge of Kafka Streams for real-time data processing within the Kafka ecosystem