pridelogic logopridelogic logo

Senior Data Engineer (Python/PySpark/AWS Glue/Amazon Athena/SQL/Apache Airflow)

pridelogicRemote job
Remote Full-time

Clicking Apply Now takes you to AutoApply where you can tailor your resume and apply.


Experience Level

Mid to Senior

Qualifications

Required qualificationsA minimum of 4 years of hands-on experience in data engineering. Extensive experience with Apache Airflow, AWS Glue, PySpark, and Python-based data pipelines. Strong command of SQL and experience with PostgreSQL in live operational environments. Solid understanding of cloud-native data workflows, preferably on AWS, as well as pipeline observability including metrics, logging, tracing, and alerting. Demonstrated experience managing pipelines end-to-end: from design and implementation to testing, deployment, monitoring, and iteration. Preferred qualificationsExperience optimizing Snowflake performance (e.g., warehouses, partitions, clustering, query profiling) and cost efficiency. Experience with real-time or near-real-time processing techniques (e.g., streaming ingestion, incremental models, CDC). Familiarity with backend TypeScript frameworks (e.g., NestJS) is a significant advantage. Experience with data quality frameworks, contract testing, or schema management (e.g., Great Expectations, dbt tests, OpenAPI/Protobuf/Avro). A background in developing internal developer platforms or data platform components (such as connectors, SDKs, CI/CD processes for data).

About the job

We are on the lookout for an exceptional Data Engineer, a technical leader who thrives on challenges and excels in coding. If you are the person who:

  • Acts as the definitive technical authority within your team

  • Solves complex technical problems with ease

  • Delivers intricate features at a remarkable speed

  • Writes code that exemplifies best practices and clarity

  • Is dedicated to enhancing the overall quality of the codebase

Then we want to hear from you!
We are not looking for just anyone; we want developers who are confident in their skills and have proven their excellence.

What you will be responsible for:

  • Designing, optimizing, and expanding data pipelines and infrastructure leveraging Python, TypeScript, Apache Airflow, PySpark, AWS Glue, and Snowflake.

  • Creating, operationalizing, and monitoring data ingestion and transformation workflows including DAGs, alerting mechanisms, retries, SLAs, lineage, and cost management.

  • Partnering with platform and AI/ML teams to streamline ingestion, validation, and real-time compute workflows; contributing towards the development of a feature store.

  • Incorporating pipeline health metrics into engineering dashboards to ensure complete visibility and observability.

  • Modeling data and executing efficient, scalable transformations within Snowflake and PostgreSQL.

  • Establishing reusable frameworks and connectors to standardize internal data publishing and consumption.

About pridelogic

pridelogic is a pioneering technology firm that specializes in delivering innovative data solutions. Our team is driven by a commitment to excellence and a passion for crafting top-tier software that empowers organizations to harness the full potential of their data. We foster a collaborative and inclusive environment where every team member's skills are valued.

Similar jobs

Browse all companies, explore by city & role, or SEO search pages.

Tailoring 0 resumes

We'll move completed jobs to Ready to Apply automatically.