End-to-End Data Processing Pipeline on Azure
- Designed and implemented a data pipeline using Azure services to extract, transform, and load data into Azure Data Lake Storage.
- Developed an extraction process from a Git repository using Azure Data Factory, storing raw CSV data in a designated storage container.
- Utilized Azure Databricks and Spark for preprocessing and transforming data, enhancing data quality and processing efficiency.
- Implemented Azure Key Vault to securely manage sensitive information and credentials.
- Established a data warehouse using Azure Synapse Analytics to enable advanced querying and reporting.
- Created interactive dashboards in Power BI to visualize data insights, facilitating informed decision-making.
- Planned future extensions to enhance pipeline scalability and automation.