Google Cloud : Building Batch Data Pipelines on Google Cloud
-
Learned to design, build, and orchestrate robust, scalable batch data pipelines on Google Cloud Platform (GCP) to process large datasets efficiently
-
Gained hands-on experience selecting the right serverless and managed compute services for batch processing, leveraging both open-source frameworks and cloud-native tools
-
Developed expertise in monitoring, managing, and scheduling complex ETL/ELT workflows using orchestration and integration platforms
-
Strengthened understanding of batch processing architecture and data pipeline optimization for reliability, scalability, and performance
​​
Tools & Techniques:
-
Cloud Storage – data lake for ingesting, staging, and storing raw data at scale
-
Dataproc (Managed Apache Spark) – distributed batch processing and analytics using Spark clusters
-
Dataflow (Apache Beam) – serverless, unified platform for batch and stream ETL/ELT pipelines
-
Cloud Data Fusion – graphical interface for building and managing data integration workflows
-
Cloud Composer (Managed Apache Airflow) – orchestration service for scheduling and monitoring pipeline dependencies
-
ETL/ELT pipeline design – architecture patterns, workflow optimization, and best-practice implementation
-
Google Cloud Skills Boost / Qwiklabs – hands-on labs for real-world application and deployment within GCP