Aws Data Engineer

Collaborate with Product teams ensuring that raw data is cleansed and transformed and useable by downstream consumers (ML Engineers, BI analytics). Assist and advise on the re-development and modernisation of end-to-end ETL pipelines and introduce new technologies where appropriate in a real-time streaming environment dealing with very large data volumes. Working with AWS cloud infrastructure (specifically SQS, SNS, Redshift, OpenSearch, Athena, Kinesis, AWS code pipeline) to develop, innovate and maintain data flowing through various queues and data warehouses. Working with a variety of data repository platforms (including SQL stores such as Oracle), as well as implementing data visualisation and network analysis (e.g GraphdB). Maintain and ‘productionise’ machine learning and AI models. Assist in the creation of next generation data ingestion platforms – sourcing data using Webscrapes, APIs, Email, and flat file (FTP) methods. Understanding conflict resolution methods and assist subject matter experts in the debugging on data ingestion and managing overall feed uptimes across a large set of data collectors. Create and maintain detailed documentation and functional design specifications including data flows and data conversion. Provide technical information to assist in the development of client facing product documentation. Adhere to change management protocols and version control. Present advanced technical designs with non-technical stakeholders.