Responsibility
- Develop and maintain ETL processes to extract, transform, and load data from various sources to target data repositories.
- Work closely with the Data Architect to understand data models and ensure that ETL processes align with the overall data architecture.
- Monitor and maintain data infrastructure to ensure data quality and accuracy, including implementing data security and privacy policies.
- Provide technical support to end-users to help them access and utilize data effectively.
- Work collaboratively with other members of the data team to ensure the effective and efficient use of data within the organization.
- Develop and maintain documentation of ETL processes and data infrastructure
Qualification
- Strong SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working with a variety of databases.
- Experience with dbt (data build tool) for data transformation, testing, and documentation.
- Experience building and optimizing 'big data' data pipelines, architectures and data sets on Databricks platform.
- Build processes supporting data transformation, data structures, metadata, dependency and workload management.
- A successful history of manipulating, processing and extracting value from large disconnected datasets.
- Experience supporting and working with cross-functional teams in a dynamic environment.
- Experience with object-oriented/object function scripting languages: Python, Java, Scala, C#, etc.
- Experience with Apache Airflow for workflow orchestration and pipeline monitoring.
- Experience in using Databricks, PySpark for big data processing.
- Previous work designing/managing data structures within a data warehouse.
- Experience in data stream processing and its processor is a plus.
- Experience developing application eg. web application, mobile application, REST web services is a plus.