About the job
DeepLight AI is a premier consultancy specializing in artificial intelligence and data solutions, recognized for our robust implementation of intelligent enterprise systems across diverse sectors, particularly in financial services and banking. Our team expertly merges advanced data science, statistical modeling, AI/ML technologies, workflow automation, and systems integration with a profound understanding of intricate business operations.
We are on the lookout for a talented AWS Glue Data Engineer to enhance our Data Factory Squad. In this role, you will take charge of migrating source systems into our Lakehouse ingestion zone, focusing on the development of scalable ingestion pipelines, optimizing performance, and ensuring adherence to architectural and data assurance standards.
The ideal candidate will possess substantial experience in financial services, with a strong command of AWS Glue, PySpark, and ETL pipeline development.
Your key responsibilities as the AWS Glue Data Engineer will encompass:
- Data Ingestion Development
- Creating and implementing AWS Glue jobs for Bronze layer ingestion using established standards and templates.
- Employing appropriate loading techniques based on source requirements (CDC, full load, delta, snapshot).
- Designing and executing historical loading processes to integrate legacy data into the Lakehouse.
- Performance Optimization
- Enhancing Glue job performance (DPU allocation, parallelization, partitioning) in line with industry best practices.
- Collaborating with platform teams to ensure alignment in tooling and optimization efforts.
- Migration & Automation
- Proactively migrating source tables to the Bronze layer, initially utilizing manual methods with standards/templates, and subsequently leveraging AI-driven acceleration.
- Ensuring jobs are version-controlled and that production deployment is automated using Git and Terraform.
- Governance & Monitoring
- Establishing connectivity with source systems within the CDP in partnership with source system owners.
- Guaranteeing compliance with data contracts and effective monitoring of jobs.
- Preparing documentation and facilitating handover to operational support teams.
- Collaboration
- Working closely with the Data Architect to establish ingestion patterns and standards.
- Coordinating with the Data Assurance Lead to ensure quality checks across all jobs.
- Partnering with platform engineers for tooling and optimization enhancements.
Requirements
We expect you to have:
- Expertise in AWS Glue, PySpark, and ETL pipeline development;
- In-depth knowledge of Lakehouse architecture and Medallion design principles;
- Familiarity with CDC, delta loads, and historical data ingestion strategies;
- 5+ years of experience in data engineering roles, with a focus on...

