About the job
As a Senior Data Engineer specializing in Machine Learning (ML) and Artificial Intelligence (AI), you will be instrumental in developing and sustaining data pipelines tailored for ML/AI applications. Your efforts will focus on managing extensive datasets, including unstructured and semi-structured information, to ensure our analytics and machine learning initiatives are grounded in high-quality data.
Your Responsibilities
- Design, implement, and optimize data pipelines specifically for ML/AI scenarios, ensuring effective handling of large-scale datasets.
- Construct feature pipelines and feature stores that promote data reusability and consistency for machine learning models.
- Work closely with Data Scientists and ML Engineers to clarify data needs for model training, validation, and deployment.
- Guarantee that data quality, lineage, and governance adhere to the standards necessary for AI/ML applications.
- Facilitate MLOps practices by integrating data pipelines into model training, monitoring, and deployment processes.
- Utilize distributed processing frameworks (e.g., Spark, Databricks, Azure Synapse) for scalable ML data processing.
