Are you passionate about big data, automation, and logging systems? We are seeking a talented AWS Glue Data Platform Engineer to lead our Central Log Collector project. In this role, you’ll be at the forefront of the mission to centralize and enhance security logs by creating robust log processes and automating AWS Glue jobs.
What You’ll Do:
- Log File Creation & Ingestion: Design and implement processes to create, collect, and ingest log files in various formats (JSON, Text, Dotlog) from multiple systems.
- AWS Glue Job Automation: Develop, automate, and maintain AWS Glue jobs to efficiently transform log data into optimized parquet format.
- Data Integration: Gather logs from diverse sources including on-prem data centers, Azure, and AWS, ensuring a unified, centralized logging system.
- Stakeholder Collaboration: Work closely with system owners and cross-functional teams to gather requirements and continuously improve our data platform.
What We’re Looking For:
- Proven experience in automating AWS Glue jobs and building data pipelines.
- Strong expertise in processing and managing various log file formats.
- A solid background in the AWS ecosystem and big data tools.
- Proficiency in Python/Scala and SQL.
- Experience with infrastructure automation and containerization technologies.
- Excellent communication skills to collaborate effectively with system owners and technical teams.
- A passion for innovation and continuous improvement in data engineering practices.
Ready to contribute to a scalable and secure logging infrastructure? Apply now!
Darwin Recruitment is acting as an Employment Business in relation to this vacancy.
Eszter Hlavati