Overview
shell scripting/python developer
responsibilities
* design and develop robust and scalable etl processes to move and transform large volumes of data from various sources.
* build and maintain data pipelines on client cloud platform (gcp) using services such as dataflow, cloud functions, bigquery, and cloud storage.
* automate workflows and processes using shell scripting and python.
* write and optimize complex sql queries for data manipulation and analysis in relational (e.g., postgresql, mysql) and nosql databases.
* monitor, debug, and troubleshoot data pipelines to ensure their proper functioning and performance.
* document implemented processes, architectures, and data flows.
skills and qualifications
* required:
* proven experience in etl development and/or data engineering.
* knowledge of client cloud platform (gcp), especially with data services like bigquery and dataflow.
* proficiency in python for data manipulation and script development (knowledge of libraries like pandas, apache beam is a plus).
* strong skills in shell scripting for task automation in linux/unix environments.
* extensive experience working with sql (e.g., postgresql, sql server, mysql) and nosql databases. Ability to write complex and optimized queries.
* solid understanding of data modeling and data warehousing principles.
* excellent problem-solving skills and attention to detail.
* preferred:
* client cloud certification (e.g., professional data engineer).
* experience with workflow orchestration tools such as apache airflow or cloud composer.
* knowledge of agile methodologies (scrum/kanban).
* familiarity with container technologies like docker and kubernetes.
* ability to communicate effectively in english (oral and written).
#j-18808-ljbffr