Azure

End-to-End Data Processing Pipeline on Azure

-  Designed and implemented a data pipeline using Azure services to extract, transform, and load data into Azure Data Lake Storage.

-  Developed an extraction process from a Git repository using Azure Data Factory, storing raw CSV data in a designated storage container.

-  Utilized Azure Databricks and Spark for preprocessing and transforming data, enhancing data quality and processing efficiency.

-  Implemented Azure Key Vault to securely manage sensitive information and credentials.

-  Established a data warehouse using Azure Synapse Analytics to enable advanced querying and reporting.

-  Created interactive dashboards in Power BI to visualize data insights, facilitating informed decision-making.

-  Planned future extensions to enhance pipeline scalability and automation.