Data Engineering

A Beginner Guide to Airflow Components

Introduction In today’s data-driven world, efficient management and orchestration of data workflows are essential for organizations to extract insights and drive decision-making. Airflow, an open-source platform, has emerged as a popular choice among data engineers for its powerful capabilities in automating and monitoring workflows. In this comprehensive guide, we will explore the various components of …

A Beginner Guide to Airflow Components Read More »

How To Build A Real-time Data Pipeline For An Online Store Using Apache Beam, Pub/Sub, and SQL

This article is a step-by-step guide to building a real-time data pipeline for a virtual online store(which we will create too) to perform analytics on it later. It is fascinating to see how malleable our data is becoming. Nowadays we have tools to convert highly nested and complex log data to simple rows format, tools to …

How To Build A Real-time Data Pipeline For An Online Store Using Apache Beam, Pub/Sub, and SQL Read More »