From Blueprint to Build - The Design and Planning Phase in Data Engineering

Oscar Garcia @ozkary 9/29/2025 Loading read time...

Overview

The design and planning phase of a data engineering project is crucial for laying out the foundation of a successful and scalable solution. This phase ensures that the architecture is strategically aligned with business objectives, optimizes resource utilization, and mitigates potential risks.

Data Engineering Process Fundamentals

Follow this GitHub repo during the presentation: (Give it a star)

👉 https://github.com/ozkary/data-engineering-mta-turnstile

Read more information on my blog at:

👉 https://www.ozkary.com/2023/03/data-engineering-process-fundamentals.html

YouTube Video

Video Agenda

In this session, we embark on the next chapter of our data journey, delving into the critical Design and Planning Phase. As we transition from discovery to design, we'll unravel the intricacies of:

System Design and Architecture:

Understanding the foundational principles that shape a robust and scalable data system.

Data Pipeline and Orchestration:
Uncovering the essentials of designing an efficient data pipeline and orchestrating seamless data flows.

Source Control and Deployment:
Navigating the best practices for source control, versioning, and deployment strategies.

CI/CD in Data Engineering:
Implementing Continuous Integration and Continuous Deployment (CI/CD) practices for agility and reliability.

Docker Container and Docker Hub:
Harnessing the power of Docker containers and Docker Hub for containerized deployments.

Cloud Infrastructure with IaC:
Exploring technologies for building out cloud infrastructure using Infrastructure as Code (IaC), ensuring efficiency and consistency.

Why Join:

Gain insights into designing scalable and efficient data systems.
Learn best practices for cloud infrastructure and IaC.
Discover the importance of data pipeline orchestration and source control.
Explore the world of CI/CD in the context of data engineering.
Unlock the potential of Docker containers for your data workflows.

Some of the technologies that we will be covering:

Cloud Infrastructure
Data Pipelines
GitHub and Actions
VSCode
Docker and Docker Hub
Terraform

Presentation

Data Engineering Overview

A Data Engineering Process involves executing steps to understand the problem, scope, design, and architecture for creating a solution. This enables ongoing big data analysis using analytical and visualization tools.

Topics

Importance of Design and Planning
System Design and Architecture
Data Pipeline and Orchestration
Source Control and CI/CD
Docker Containers
Cloud Infrastructure with IaC

Follow this project: Give a star

👉 Data Engineering Process Fundamentals

Importance of Design and Planning

Foundational Areas

Designing the data pipeline and technology specifications like flows, coding language, data governance and tools
Define the system architecture like cloud services for scalability, data platform
Source control and deployment automation with CI/CD
Using Docker containers for environment isolation to avoid deployment issues
Infrastructure automation with Terraform or cloud CLI tools
System monitor, notification and recovery

Data Engineering Process Fundamentals - Design and Planning

System Design and Architecture

In a system design, we need to clearly define the different technologies that should be used for each area of the solution. It includes the high-level system architecture, which defines the different components and their integration.

The design outlines the technical solution, including system architecture, data integration, flow orchestration, storage platforms, and data processing tools. It focuses on defining technologies for each component to ensure a cohesive and efficient solution.
A system architecture is a critical high-level design encompassing various components such as data sources, ingestion resources, workflow orchestration, storage, transformation services, continuous ingestion, validation mechanisms, and analytics tools.

Data Engineering Process Fundamentals - System Architecture

Data Pipeline and Orchestration

A data pipeline is basically a workflow of tasks that can be executed in Docker containers. The execution, scheduling, managing and monitoring of the pipeline is referred to as orchestration. In order to support the operations of the pipeline and its orchestration, we need to provision a VM and data lake, and monitor cloud resources.

This can be code-centric, leveraging languages like Python, SQL
Or a low-code approach, utilizing tools such as Azure Data Factory, which provides a turn-key solution
Monitor services enable us to track telemetry data to support operational requirements
Docker Hub, GitHub can be used for the CI/CD process and deployed our code-centric solutions
Scheduling, recovering from failures and dashboards are essentials for orchestration
Low-code solutions , like data factory, can also be used

Data Engineering Process Fundamentals - Data Pipeline

Source Control - CI/CD

Implementing source control practices alongside Continuous Integration and Continuous Delivery (CI/CD) pipelines is vital for facilitating agile development. This ensures efficient collaboration, change tracking, and seamless code deployment, crucial for addressing ongoing feature changes, bug fixes, and new environment deployments.

Systems like Git facilitates effective code and configuration file management, enabling collaboration and change tracking.
Platforms such as GitHub enhance collaboration by providing a remote repository for sharing code.
CI involves integrating code changes into a central repository, followed by automated build and test processes to validate changes and provide feedback.
CD automates the deployment of code builds to various environments, such as staging and production, streamlining the release process and ensuring consistency across environments.

Data Engineering Process Fundamentals - GitHub CI/CD

Docker Container and Docker Hub

Docker proves invaluable for our data pipelines by providing self-contained environments with all necessary dependencies. With Docker Hub, we can effortlessly distribute pipeline images, facilitating swift and reliable provisioning of new environments.

Docker containers streamline the deployment process by encapsulating application and dependency configurations, reducing runtime errors.
Containerizing data pipelines ensures reliability and portability by packaging all necessary components within a single container image.
Docker Hub serves as a centralized container registry, enabling seamless image storage and distribution for streamlined environment provisioning and scalability.

Data Engineering Process Fundamentals - Docker

Cloud Infrastructure with IaC

Infrastructure automation is crucial for maintaining consistency, scalability, and reliability across environments. By defining infrastructure as code (IaC), organizations can efficiently provision and modify cloud resources, mitigating manual errors.

Define infrastructure configurations as code, ensuring consistency across environments.
Easily scale resources up or down to meet changing demands with code-defined infrastructure.
Reduce manual errors and ensure reproducibility by automating resource provisioning and management.
Track infrastructure changes under version control, enabling collaboration and ensuring auditability.
Track infrastructure state, allowing for precise updates and minimizing drift between desired and actual configurations.

Data Engineering Process Fundamentals - Terraform

Summary

The design and planning phase of a data engineering project sets the stage for success. From designing the system architecture and data pipelines to implementing source control, CI/CD, Docker, and infrastructure automation with Terraform, every aspect contributes to efficient and reliable deployment. Infrastructure automation, in particular, plays a critical role by simplifying provisioning of cloud resources, ensuring consistency, and enabling scalability, ultimately leading to a robust and manageable data engineering system.

Upcoming Talks:

Join us for subsequent sessions in our Data Engineering Process Fundamentals series, where we will delve deeper into specific facets of data engineering, exploring topics such as data modeling, pipelines, and best practices in data governance.

This presentation is based on the book, Data Engineering Process Fundamentals, which provides a more comprehensive guide to the topics we'll cover. You can find all the sample code and datasets used in this presentation on our popular GitHub repository Introduction to Data Engineering Process Fundamentals.

Thanks for reading! 😊 If you enjoyed this post and would like to stay updated with our latest content, don’t forget to follow us. Join our community and be the first to know about new articles, exclusive insights, and more!

👍 Originally published by ozkary.com

Overview The design and planning phase of a data engineering project is crucial for laying out the foundation of a successful and scalable ...

Read Article →

Unlock the Blueprint: Design and Planning Phase - Data Engineering Process Fundamentals

Oscar Garcia @ozkary 2/14/2024 Loading read time...

Overview

In this technical presentation, we embark on the next chapter of our data journey, delving into the critical Design and Planning Phase.

Data Engineering Process Fundamentals

Follow this GitHub repo during the presentation: (Give it a star)

👉 https://github.com/ozkary/data-engineering-mta-turnstile

Read more information on my blog at:

👉 https://www.ozkary.com/2023/03/data-engineering-process-fundamentals.html

YouTube Video

Video Agenda

System Design and Architecture:

Understanding the foundational principles that shape a robust and scalable data system.

Data Pipeline and Orchestration:
Uncovering the essentials of designing an efficient data pipeline and orchestrating seamless data flows.

Source Control and Deployment:
Navigating the best practices for source control, versioning, and deployment strategies.

CI/CD in Data Engineering:
Implementing Continuous Integration and Continuous Deployment (CI/CD) practices for agility and reliability.

Docker Container and Docker Hub:
Harnessing the power of Docker containers and Docker Hub for containerized deployments.

Cloud Infrastructure with IaC:
Exploring technologies for building out cloud infrastructure using Infrastructure as Code (IaC), ensuring efficiency and consistency.

Key Takeaways:

Gain insights into designing scalable and efficient data systems.
Learn best practices for cloud infrastructure and IaC.
Discover the importance of data pipeline orchestration and source control.
Explore the world of CI/CD in the context of data engineering.
Unlock the potential of Docker containers for your data workflows.

Some of the technologies that we will be covering:

Cloud Infrastructure
Data Pipelines
GitHub and Actions
VC Code
Docker and Docker Hub
Terraform

Presentation

Data Engineering Overview

Topics

Importance of Design and Planning
System Design and Architecture
Data Pipeline and Orchestration
Source Control and CI/CD
Docker Containers
Cloud Infrastructure with IaC

Follow this project: Give a star

👉 Data Engineering Process Fundamentals

Importance of Design and Planning

Foundational Areas

Designing the data pipeline and technology specifications like flows, coding language, data governance and tools
Define the system architecture like cloud services for scalability, data platform
Source control and deployment automation with CI/CD
Using Docker containers for environment isolation to avoid deployment issues
Infrastructure automation with Terraform or cloud CLI tools
System monitor, notification and recovery

Data Engineering Process Fundamentals - Design and Planning

System Design and Architecture

The design outlines the technical solution, including system architecture, data integration, flow orchestration, storage platforms, and data processing tools. It focuses on defining technologies for each component to ensure a cohesive and efficient solution.
A system architecture is a critical high-level design encompassing various components such as data sources, ingestion resources, workflow orchestration, storage, transformation services, continuous ingestion, validation mechanisms, and analytics tools.

Data Engineering Process Fundamentals - System Architecture

Data Pipeline and Orchestration

This can be code-centric, leveraging languages like Python, SQL
Or a low-code approach, utilizing tools such as Azure Data Factory, which provides a turn-key solution
Monitor services enable us to track telemetry data to support operational requirements
Docker Hub, GitHub can be used for the CI/CD process and deployed our code-centric solutions
Scheduling, recovering from failures and dashboards are essentials for orchestration
Low-code solutions , like data factory, can also be used

Data Engineering Process Fundamentals - Data Pipeline

Source Control - CI/CD

Systems like Git facilitates effective code and configuration file management, enabling collaboration and change tracking.
Platforms such as GitHub enhance collaboration by providing a remote repository for sharing code.
CI involves integrating code changes into a central repository, followed by automated build and test processes to validate changes and provide feedback.
CD automates the deployment of code builds to various environments, such as staging and production, streamlining the release process and ensuring consistency across environments.

Data Engineering Process Fundamentals - GitHub CI/CD

Docker Container and Docker Hub

Docker containers streamline the deployment process by encapsulating application and dependency configurations, reducing runtime errors.
Containerizing data pipelines ensures reliability and portability by packaging all necessary components within a single container image.
Docker Hub serves as a centralized container registry, enabling seamless image storage and distribution for streamlined environment provisioning and scalability.

Data Engineering Process Fundamentals - Docker

Cloud Infrastructure with IaC

Define infrastructure configurations as code, ensuring consistency across environments.
Easily scale resources up or down to meet changing demands with code-defined infrastructure.
Reduce manual errors and ensure reproducibility by automating resource provisioning and management.
Track infrastructure changes under version control, enabling collaboration and ensuring auditability.
Track infrastructure state, allowing for precise updates and minimizing drift between desired and actual configurations.

Data Engineering Process Fundamentals - Terraform

Summary

Thanks for reading.

Send question or comment at Twitter @ozkary

👍 Originally published by ozkary.com

Overview The design and planning phase of a data engineering project is crucial for laying out the foundation of a successful and scalable ...

Read Article →

Data Engineering Process Fundamentals - Design and Planning Exercise

Oscar Garcia @ozkary 4/22/2023 Loading read time...

Having laid a strong design foundation, it's time to embark on a hands-on exercise that's crucial to our data engineering project's success. Our immediate focus is on building the essential cloud resources that will serve as the backbone for our data pipelines, data lake, and data warehouse. Taking a cloud-agnostic approach ensures our implementation remains flexible and adaptable across different cloud providers, enabling us to leverage the advantages of multiple platforms or switch providers seamlessly if required. By completing this step, we set the stage for efficient and effective coding of our solutions. Let's get started on this vital infrastructure-building journey.

ozkary-data-engineering-design-planning-docker-terraform

Cloud Infrastructure Planning

Infrastructure planning is a critical aspect of every technical project, laying the foundation for successful project delivery. In the case of a Data Engineering project, it becomes even more crucial. To support our project's objectives, we need to carefully consider and provision specific resources:

VM instance: This serves as the backbone for hosting our data pipelines and orchestration, ensuring efficient execution of our data workflows.
Data Lake: A vital component for storing various data formats, such as CSV or Parquet files, in a scalable and flexible manner.
Data Warehouse: An essential resource that hosts transformed and curated data, providing a high-performance environment for easy access by visualization tools.

Infrastructure automation, facilitated by tools like Terraform, is important in modern data engineering projects. It enables the provisioning and management of cloud resources, such as virtual machines and storage, in a consistent and reproducible manner. Infrastructure as Code (IaC) allows teams to define their infrastructure declaratively, track it in source control, version it, and apply changes as needed. Automation reduces manual efforts, ensures consistency, and enables infrastructure to be treated as a code artifact, improving reliability and scalability.

ozkary-data-engineering-terraform

Infrastructure Implementation Requirements

When using Terraform with a any cloud provider, there are several key artifacts and considerations to keep in mind for successful configuration and security. Terraform needs access to the cloud account where it can provision the resources. The account token or credentials can vary based on your cloud configuration. For our purpose, we will use a GCP (Google) project to build our resources, but first we need to install the Terraform dependencies for the development environment.

Install Terraform

To install Terraform, open a bash terminal and run the commands below:

Download the package file
Unzip the file
Copy the Terraform binary from the extracted file to the /usr/bin/ directory
Verify the version

$ wget https://releases.hashicorp.com/terraform/1.3.7/terraform_1.3.7_linux_amd64.zip
$ unzip terraform_1.1.2_linux_amd64.zip
$ cp terraform /usr/bin/
$ terraform -v

We should get an output similar to this:

Terraform v1.3.7
on linux_amd64

Configure a Cloud Account

Create a Google account. Here

Create a new project
Make sure to keep track of the Project ID and the location for your project

Create a service account

In the left-hand menu, click on "IAM & Admin" and then select "Service accounts"
Click on the "Create Service Account" button at the top of the page
Enter a name for the service account and an optional description
Then add the BigQuery Admin, Storage Admin, Storage Object Admin as roles for our service account and click the save button.

ozkary gcp roles

Enable IAM APIs by clicking the following links:
- IAM-API
- IAM-credential-API

Authenticate the VM or Local environment with GCP

In the left navigation menu (GCP), click on "IAM & Admin" and then "Service accounts"
Click on the three verticals dots under the action section for the service name you just created
Then click Manage keys, Add key, Create new key. Select JSON option and click Create
Move the key file to a system folder

$ mkdir -p $HOME/.gcp/ 
$ mv ~/Downloads/{xxxxxx}.json ~/.gcp/{acc_credentials}.json

install the SDK and CLI Tools
- Follow the instruction here

Validate the installation and login to GCP with the following commands

$ echo 'export GOOGLE_APPLICATION_CREDENTIALS="~/.gcp/{acc_credentials}.json"' >> ~/.bashrc
$ export GOOGLE_APPLICATION_CREDENTIALS="~/.gcp/{acc_credentials}.json"
$ gcloud auth application-default login

Follow the login process on the browser

Review the Code

👉 Clone this repo or copy the files from this folder

Terraform uses declarative configuration files written in a domain-specific language (DSL) called HCL (HashiCorp Configuration Language). It provides a concise and human-readable syntax for defining resources, dependencies, and configurations, enabling us to provision, modify, and destroy infrastructure in a predictable and reproducible manner.

At a minimum, we should define a variables file, which contains the cloud provider information and a resource file which define what kind of resources should be provision on the cloud. There could be a file for each resource or a single file can define multiple resources.

Variables File

The variables file is used to define a set of variables that can be used on the resource file. This allows for the use of those variables in one more more resource configuration files. The format looks as follows:

locals {
  data_lake_bucket = "mta_data_lake"
}

variable "project" {
  description = "Enter Your GCP Project ID"
  type = string
}

variable "region" {
  description = "Region for GCP resources. Choose as per your location: https://cloud.google.com/about/locations"
  default = "us-east1"
  type = string
}

variable "storage_class" {
  description = "Storage class type for your bucket. Check official docs for more info."
  default = "STANDARD"
  type = string
}

variable "stg_dataset" {
  description = "BigQuery Dataset that raw data (from GCS) will be written to"
  type = string
  default = "mta_data"
}

variable "vm_image" {
  description = "Base image for your Virtual Machine."
  type = string
  default = "ubuntu-os-cloud/ubuntu-2004-lts"
}

Resource File

The resource file is where we define what should be provisioned on the cloud. This is also the file where we need to define the provider specific resources that need to be created.

terraform {
  required_version = ">= 1.0"
  backend "local" {}  # Can change from "local" to "gcs" (for google) or "s3" (for aws), if you would like to preserve your tf-state online
  required_providers {
    google = {
      source  = "hashicorp/google"
    }
  }
}

provider "google" {
  project = var.project
  region = var.region
  // credentials = file(var.credentials)  # Use this if you do not want to set env-var GOOGLE_APPLICATION_CREDENTIALS
}

# Data Lake Bucket
# Ref: https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/storage_bucket
resource "google_storage_bucket" "data-lake-bucket" {
  name          = "${local.data_lake_bucket}_${var.project}" # Concatenating DL bucket & Project name for unique naming
  location      = var.region

  # Optional, but recommended settings:
  storage_class = var.storage_class
  uniform_bucket_level_access = true

  versioning {
    enabled     = true
  }

  lifecycle_rule {
    action {
      type = "Delete"
    }
    condition {
      age = 15  // days
    }
  }

  force_destroy = true
}

# BigQuery data warehouse
# Ref: https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/bigquery_dataset
resource "google_bigquery_dataset" "stg_dataset" {
  dataset_id = var.stg_dataset
  project    = var.project
  location   = var.region
}

# VM instance
resource "google_compute_instance" "vm_instance" {
  name          = "mta-instance"
  project       = var.project
  machine_type  = "e2-standard-4"
  zone          = var.region

  boot_disk {
    initialize_params {
      image = var.vm_image
    }
  }

  network_interface {
    network = "default"

    access_config {
      // Ephemeral public IP
    }
  }
}

This Terraform file defines the infrastructure components to be provisioned on the Google Cloud Platform (GCP). It includes the configuration for the Terraform backend, required providers, and the resources to be created.

The backend section specifies the backend type as "local" for storing the Terraform state locally. It can be changed to "gcs" or "s3" for cloud storage if desired.
The required_providers block defines the required provider and its source, in this case, the Google Cloud provider.
The provider block configures the Google provider with the project and region specified as variables. It can use either environment variable GOOGLE_APPLICATION_CREDENTIALS or the credentials file defined in the variables.
The resource blocks define the infrastructure resources to be created, such as a Google Storage Bucket for the data lake, Google BigQuery datasets for staging and production, and a Google Compute Engine instance named "mta-instance" with specific configuration settings.

Overall, this Terraform file automates the provisioning of a data lake bucket, BigQuery datasets, and a virtual machine instance on the Google Cloud Platform.

How to run it!

Refresh service-account's auth-token for this session

$ gcloud auth application-default login

Set the credentials file on the bash configuration file
- Add the export line and replace filename-here with your file

$ echo export GOOGLE_APPLICATION_CREDENTIALS="${HOME}/.gcp/filename-here.json" >> ~/.bashrc && source ~/.bashrc

Open the terraform folder in your project
Initialize state file (.tfstate) by running terraform init

$ cd ./terraform
$ terraform init

Check changes to new infrastructure plan before applying them

It is important to always review the plan to make sure no unwanted changes are showing up.

👉 Get the project id from your GCP cloud console and replace it on the next command

$ terraform plan -var="project=<your-gcp-project-id>"

Apply the changes

This provisions the resources on the cloud project.

$ terraform apply -var="project=<your-gcp-project-id>"

(Optional) Delete infrastructure after your work, to avoid costs on any running services

$ terraform destroy

Terraform Lifecycle

ozkary-data-engineering-terraform-lifecycle

GitHub Action

In order to be able to automate the building of infrastructure with GitHub, we need to define the cloud provider token as a secret with GitHub. This can be done by following the steps from this link:

👉 Configure GitHub Secrets

Once the secret has been configured, we can create a build action script with the cloud provider secret information as shown with this GitHub Action workflow YAML file:


name: Terraform Deployment

on:
  push:
    branches:
      - main

jobs:
  deploy:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout repository
      uses: actions/checkout@v2

    - name: Set up Terraform
      uses: hashicorp/setup-terraform@v1

    - name: Terraform Init
       env:        
        GOOGLE_APPLICATION_CREDENTIALS:  ${{ secrets.GOOGLE_APPLICATION_CREDENTIALS }}
      run: |
        cd Step2-Cloud-Infrastructure/terraform
        terraform init

    - name: Terraform Apply
      run: |
        cd path/to/terraform/project
        terraform apply -auto-approve

Conclusion

With this exercise, we gain practical experience in using tools like Terraform to automate the provisioning of resources, such as VM, a data lakes and other components essential to our data engineering system. By following cloud-agnostic practices, we can achieve interoperability and avoid vendor lock-in, ensuring our project remains scalable, cost-effective, and adaptable to future requirements.

Next Step

After building our cloud infrastructure, we are now ready to talk about the implementation and orchestration of a data pipeline and review some of the operational requirements that can enable us to make decisions.

Coming Soon!

👉 [Data Engineering Process Fundamentals - Data Pipeline and Orchestration]

Thanks for reading.

Send question or comment at Twitter @ozkary

👍 Originally published by ozkary.com

Having laid a strong design foundation, it's time to embark on a hands-on exercise that's crucial to our data engineering project...

Read Article →