As a data pipeline specialist, you will be responsible for designing and implementing efficient data pipelines using airflow, python, and various data storage solutions.key responsibilitiesdata pipeline development:design and build dags (taskflow, dynamic dags, deferrable operators) to manage data flow between systems.develop python-based etl/elt pipelines for apis, storage, kafka, and databases.operational excellence:ensure smooth operation of airflow in azure/kubernetes with blue/green or canary releases.quality assurance:implement data quality checks, unit tests, and ci/cd dag validation to ensure reliability and accuracy.real-time processing:build near–real-time event pipelines with proper schema management to support business decision-making.performance optimization:model sql/blob datasets with optimal partitioning and retention to improve query performance.incident management:drive observability (metrics, logs, alerts, slas) and lead incident reviews to minimize downtime.security best practices:enforce security best practices: iam, secrets, rbac, pii handling, data contracts to protect sensitive information.infrastructure management:build ci/cd pipelines, containerize with docker, and use terraform/helm for infrastructure as code.collaboration:collaborate with android/backend teams on data interfaces and documentation to ensure seamless integration.requirementsexperience: 8+ years in data/backend engineering with strong python skills.airflow experience: 2+ years with airflow 2.x and advanced operators/hooks/scheduler tuning.data storage: experience with reliable large-scale etl/elt (batch + streaming) solutions like bigquery/redshift/snowflake.sql and data modeling: strong sql and data modeling skills with experience in production-ready debugging, tuning, and monitoring.security and compliance: knowledge of rbac, secrets management, oauth/oidc, and api gateways to ensure secure data exchange.communication and documentation: strong communication and documentation skills to collaborate effectively with cross-functional teams.