About the job
Join Our Innovative Team as a Data Engineer!
This position plays a vital role in crafting and enhancing our data architecture to meet evolving business demands. You'll be at the forefront of designing new data pipelines, maintaining platforms that facilitate data streaming, and ensuring efficient batch and real-time data loading and visualization.
Your Responsibilities Will Include:
- Continuously improving our existing data platform.
- Creating processes for data ingestion from Kafka, APIs, and databases utilizing AWS MSK Connect.
- Developing and maintaining real-time data processing applications with frameworks such as Spark Structured Streaming and Kafka Streams.
- Transforming data streams to optimize usability.
- Engaging in data modeling practices adhering to industry standards like Inmon, Kimball, and Data Vault.
- Ensuring data quality through consistency and accuracy checks.
- Staying updated on the latest technologies and methodologies to enhance our data platform.
- Applying a problem-solving mindset to troubleshooting and incident management.
- Taking full ownership of assigned projects and tasks while collaborating effectively within a team.
- Documenting processes and conducting knowledge-sharing sessions.
What We Seek in You:
Essential Qualifications:
- Proven experience with modern cloud database technologies such as Snowflake.
- Proficiency in orchestrating data pipelines using Airflow.
- Hands-on experience with AWS Glue.
- Familiarity with Apache Iceberg.
- Strong skills in SQL and data integration tools.
- Proficiency in programming languages such as Python or Scala.
- Knowledge of AWS services including S3, Lambda, API Gateways, DMS, RDS.
- Development experience in both Microsoft and Linux/Cloud environments.
- Exceptional analytical and problem-solving skills.
