About the job
We are looking for a talented and experienced Senior Data Engineer to join our dynamic Data Engineering team in Dubai. The successful candidate will play a pivotal role in designing, constructing, and enhancing scalable data pipelines, while also facilitating advanced analytics and machine learning applications. This position demands a deep understanding of big data technologies, data processing frameworks, and feature engineering to effectively support machine learning models in a live production setting.
Key Responsibilities
- Collect and assess business and technical requirements for effective data solutions.
- Conduct Exploratory Data Analysis (EDA) to reveal data patterns and assess data quality.
- Design, develop, and maintain high-performance, scalable data pipelines.
- Ingest both structured and unstructured data from various sources.
- Transform and process extensive datasets using PySpark and Python.
- Apply feature engineering techniques to enhance machine learning models.
- Optimize Spark jobs for peak performance and cost efficiency.
- Ensure the quality, integrity, and security of data across all pipelines.
- Collaborate closely with Data Scientists, Analytics Teams, and Delivery Leads.
- Participate in an Agile environment alongside cross-functional teams.
- Effectively communicate with stakeholders, providing valuable technical insights.
- Utilize Git for version control and manage CI/CD workflows.
Required Skills & Expertise
- Proven experience in Python and PySpark.
- In-depth knowledge of Apache Spark (performance tuning and optimization).
- Comprehensive understanding of the Hadoop ecosystem.
- Advanced skills in SQL.
- Experience in building end-to-end data and ML pipelines.
- Strong grasp of data modeling and warehousing concepts.
- Experience with feature engineering for ML applications.
- Familiarity with Git and version control systems.
- Exposure to cloud platforms (AWS/Azure/GCP) is advantageous.
Qualifications
- Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
- 8–10+ years of experience in Data Engineering / Big Data.
- Experience in the banking or financial services sector is preferred.
Soft Skills
- Exceptional problem-solving and analytical skills.
- Strong communication and stakeholder management abilities.
- Capability to work both independently and collaboratively within a team.
- Adaptability to fast-paced Agile workflows.
