Job description
as a data engineer at capgemini, you will be responsible for designing and developing scalable data pipelines using azure databricks. You will integrate data from various sources to ensure data quality and consistency.
key responsibilities:
* design and develop scalable data pipelines using azure databricks.
* integrate data from various sources, ensuring data quality and consistency.
* write and optimize pl/sql queries for data extraction, transformation, and loading.
* develop scripts using pyspark and python for data processing and analysis.
* monitor the performance of data pipelines and queries.
* work closely with data scientists, analysts, and other stakeholders to understand data requirements.
requirements:
* proficiency in azure databricks and cloud-based technologies.
* strong understanding of data integration and etl processes.
* ability to write and optimize complex sql queries.
* experience with pyspark and python programming.
benefits:
* opportunity to work on large-scale data projects.
* collaborative and dynamic work environment.
* continuous learning and professional development opportunities.