The sr data and analytics engineer architects end-to-end data solutions, designing and implementing highly available and fault-tolerant batch and streaming data pipelines for structured and unstructured data using on-prem and cloud platforms. This role defines best practices for robust data pipeline development, data profiling, monitoring, and alerting, leads the evaluation of new technologies, leads the organization's metrics, reports, and semantic layer development, and acts as a subject matter expert for enterprise data domain. With full autonomy, this role will mentor other engineers and act as a subject matter expert, making architectural decisions that shape the organization’s data strategy.
activities:
* architect end-to-end data solutions and products for complex, cross-functional business problems.
* define best practices for pipeline development, profiling, and monitoring.
* architect cross-domain data pipelines optimized for latency, cost, and auditability.
* lead the design and governance of scalable, data architecture, standards-compliant database specifications that align with medallion architecture principles.
* implement rbac to ensure secure access across layers and enable ci/cd pipelines for consistent schema deployment.
* drive ci/cd practices for data workflows; establish deployment playbooks and rollback strategies.
* build shared observability dashboards and alert systems across business units.
* define best practices for robust data pipeline development, data profiling, monitoring, alerting, and data health.
* establish best practices for data security and compliance within the data ecosystem in collaboration with security teams.
* define best practices and technical standards for the data engineering discipline.
* establish documentation standards for data models, curated datasets, and platform workflows. Drive lineage tracking and architectural diagrams across the medallion layers.
* define and enforce backup and recovery plans to ensure resilience and auditability across mission-critical data assets
* lead development of automated data health solutions.
* ensure data is consistently structured for various uses, including kpi reports and advanced analytics.
* lead the implementation of enterprise-wide data quality frameworks, integrating with data catalog like alation.
* with cross-functional stakeholders to ensure quality, consistency, and extensibility of data pipelines across collection, enrichment, and consumption layers.
* own the technical direction for significant data domains or initiatives.
* act as a subject matter expert for specific enterprise data domains (e.g., customer data, supply chain data).
* drive improvements to the data and analytics strategy by evaluating architectural patterns and identifying gaps across ingestion, transformation, orchestration, and reporting.
* collaborate with stakeholders to ensure proposed solutions align with enterprise goals and future-state architecture.
* lead the development and validation of disaster recovery and business continuity plans for data systems.
* ensure resilient architecture across medallion layers with automated recovery processes that minimize data loss and downtime in orchestration and transformation flows.
requirements:
* snowflake / cloud db
* python
* airflow o prefect
* data warehouse / data warehousing
* sql
* etl / elt
* data visualization, dashboarding
* dbt, data modeling & ci/cd
* + 5 yearse in leading design, development and optimization of end-to-end data collection, transformation, and analysis workflows
* expertise across data ecosystem and stack including technologies like advanced sql, python, snowflake, gcp, informatica, dbt labs, prefect, airflow, alation, collibra.
* design and implementation of highly available and fault-tolerant batch and streaming data pipelines using cloud platforms like gcp, snowflake.
* experience architecting complex, end-to-end data solutions. Provide guidance on format standardization and schema evolution. Enable compatibility with cross-platform data movement using iceberg where applicable.
* advanced competence level in defining best practices for robust data pipeline development, data profiling, monitoring, alerting, and data health.
#j-18808-ljbffr