From Raw Data to Analytics: The Modern Data Layer Architecture

Overview

This presentation is part of the Data Engineering Process Fundamentals series, focusing on the essential architectural components—the Data Lake and the Data Warehouse—and defining their respective roles in a modern analytics ecosystem.

• Follow this GitHub repo during the presentation: (Star the project to follow and get updates)

👉 GitHub Repo

• Data engineering Series:

👉 Blog Series

YouTube Video

Video Agenda

Agenda:

1. Introduction to Data Engineering:

2. Brief overview of the data engineering landscape and its critical role in modern data-driven organizations.

3. Operational Data

4. Understanding Data Lakes:

5. Explanation of what a data lake is and its purpose in storing vast amounts of raw and unstructured data.

6. Exploring Data Warehouses:

7. Definition of data warehouses and their role in storing structured, processed, and business-ready data.

8. Comparing Data Lakes and Data Warehouses:

9. Comparative analysis of data lakes and data warehouses, highlighting their strengths and weaknesses.

10. Discussing when to use each based on specific use cases and business needs.

11. Integration and Data Pipelines:

12. Insight into the seamless integration of data lakes and data warehouses within a data engineering pipeline.

13. Code walkthrough showcasing data movement and transformation between these two crucial components.

14. Real-world Use Cases:

15. Presentation of real-world use cases where effective use of data lakes and data warehouses led to actionable insights and business success.

16. Hands-on demonstration using Python, Jupyter Notebook and SQL to solidify the concepts discussed, providing attendees with practical insights and skills.

17. Q&A and Hands-on Session:

18. An interactive Q&A session to address any queries.

Conclusion:

This session aims to equip attendees with a strong foundation in data engineering, focusing on the pivotal role of data lakes and data warehouses. By the end of this presentation, participants will grasp how to effectively utilize these tools, enabling them to design efficient data solutions and drive informed business decisions.

This presentation will be accompanied by live code demonstrations and interactive discussions, ensuring attendees gain practical knowledge and valuable insights into the dynamic world of data engineering.

Supporting Materials Reminder

Subsequent Sessions: Join us for future sessions in our Data Engineering Process Fundamentals series, where we will build a data pipeline and delve deeper into topics like orchestration and governance.

Resources: This presentation is based on the book, Data Engineering Process Fundamentals, and all supporting code and examples are available on our popular GitHub repository.

Presentation

Data Engineering Overview

A Data Engineering Process involves executing steps to understand the problem, scope, design, and architecture for creating a solution. This enables ongoing big data analysis using analytical and visualization tools.

Topics

• Data Lake and Data Warehouse
• Discovery and Data Analysis
• Design and Infrastructure Planning
• Data Lake - Pipeline and Orchestration
• Data Warehouse - Design and Implementation
• Analysis and Visualization

Follow this project: Give a star

👉 Data Engineering Process Fundamentals

Operational Data

Operational data is often generated by applications, and it is stored in transactional relational databases like SQL Server, Oracle and NoSQL (JSON) databases like MongoDB, Firebase. This is the data that is created after an application saves a user transaction like contact information, a purchase or other activities that are available from the application.

Features:

• Application support and transactions
• Relational data structure and SQL or document structure NoSQL
• Small queries for case analysis

Not Best For:

• Reporting system
• Large queries
• Centralized Big Data system

Data Lake - Analytical Data Staging

A Data Lake is an optimized storage system for Big Data scenarios. The primary function is to store the data in its raw format without any transformation. Analytical data is the transaction data that has been extracted from a source system via a data pipeline as part of the staging data process.

Features:

• Store the data in its raw format without any transformation
• This can include structure data like CSV files, unstructured data like JSON and XML documents, or column-base data like parquet files
• Low Cost for massive storage power
• Not Designed for querying or data analysis
• It is used as external tables by most systems

Data Warehouse - Analytical Data

A Data Warehouse is a centralized storage system that stores integrated data from multiple sources. The system is designed to host and serve Big Data scenarios with lower operational cost than transaction databases, but higher costs than a Data Lake. This system host the Analytical Data that has been processed and is ready for analytical purposes.

Data Warehouse Features:

• Stores historical data in relational tables with an optimized schema, which enables the data analysis process
• Provides SQL support to query the data
• It can integrate external resources like CSV and parquet files that are stored on Data Lakes as external tables
• The system is designed to host and serve Big Data scenarios. It is not meant to be used as a transactional system
• Storage is more expensive
• Offloads archived data to Data Lakes

Discovery - Data Analysis

During the discovery phase of a Data Engineering Process, we look to identify and clearly document a problem statement, which helps us have an understanding of what we are trying to solve. We also look at our analytical approach to make observations about at the data, its structure and source. This leads us into defining the requirements for the project, so we can define the scope, design and architecture of the solution.

• Download sample data files
• Run experiments to make observations
• Write Python scripts using VS Code or Jupyter Notebooks
• Transform the data with Pandas
• Make charts with Plotly
• Document the requirements

Design and Planning

The design and planning phase of a data engineering project is crucial for laying out the foundation of a successful system. It involves defining the system architecture, designing data pipelines, implementing source control practices, ensuring continuous integration and deployment (CI/CD), and leveraging tools like Docker and Terraform for infrastructure automation.

• Use GitHub for code repo and for CI/CD actions
• Use Terraform is an Infrastructure as Code (IaC) tool that enables us to manage cloud resources across multiple cloud providers
• Use Docker containers to run the code and manage its dependencies

Data Lake - Pipeline and Orchestration

A data pipeline is basically a workflow of tasks that can be executed in Docker containers. The execution, scheduling, managing and monitoring of the pipeline is referred to as orchestration. In order to support the operations of the pipeline and its orchestration, we need to provision a VM and data lake, and monitor cloud resources.

• This can be code-centric, leveraging languages like Python
• Or a low-code approach, utilizing tools such as Azure Data Factory, which provides a turn-key solution
• Monitor services enable us to track telemetry data
• Docker Hub, GitHub can be used for the CI/CD process

Data Warehouse - Design and Implementation

In the design phase, we lay the groundwork by defining the database system, schema model, and technology stack required to support the data warehouse’s implementation and operations. In the implementation phase, we focus on converting conceptual data models into a functional system. By creating concrete structures like dimension and fact tables and performing data transformation tasks, including data cleansing, integration, and scheduled batch loading, we ensure that raw data is processed and unified for analysis. Create a repeatable and extendable process.

Data Warehouse - Data Analysis

Data analysis is the practice of exploring data and understanding its meaning. It involves activities that can help us achieve a specific goal, such as identifying data dimensions and measures, as well as data analysis to identify outliers, trends, and distributions.

• We can accomplish these activities by writing code using Python and Pandas, SQL, Visual Studio Code or Jupyter Notebooks.
• What's more, we can use libraries, such as Plotly, to generate some visuals to further analyze data and create prototypes.

Data Analysis and Visualization

Data visualization is a powerful tool that takes the insights derived from data analysis and presents them in a visual format. While tables with numbers on a report provide raw information, visualizations allow us to grasp complex relationships and trends at a glance.

• Dashboards, in particular, bring together various visual components like charts, graphs, and scorecards into a unified interface that can help us tell a story
• Use tools like PowerBI, Looker, Tableau to model the data and create enterprise level visualizations

Conclusion

Both data lakes and data warehouses are essential components of a data engineering project. The primary function of a data lake is to store large amounts of operational data in its raw format, serving as a staging area for analytical processes. In contrast, a data warehouse acts as a centralized repository for information, enabling engineers to transform, process, and store extensive data. This allows the analytical team to utilize coding languages like Python and tools such as Jupyter Notebooks, as well as low-code platforms like Looker Studio and Power BI, to create enterprise-quality dashboards for the organization.

Upcoming Talks:

Join us for subsequent sessions in our Data Engineering Process Fundamentals series, where we will delve deeper into specific facets of data engineering, exploring topics such as data modeling, pipelines, and best practices in data governance.

This presentation is based on the book, Data Engineering Process Fundamentals, which provides a more comprehensive guide to the topics we'll cover. You can find all the sample code and datasets used in this presentation on our popular GitHub repository Introduction to Data Engineering Process Fundamentals.

Thanks for reading! 😊 If you enjoyed this post and would like to stay updated with our latest content, don’t forget to follow us. Join our community and be the first to know about new articles, exclusive insights, and more!

• Google Developer Group
• GitHub
• Twitter
• YouTube
• BlueSky

👍 Originally published by ozkary.com

Overview This presentation is part of the Data Engineering Process Fundamentals series, focusing on the essential architectural components—...

Read Article →

Use Remote Dev Container with GitHub Codespaces

As Software Engineers, we usually work with multiple projects in parallel. This forces us to configure our work stations with multiple software development tools, which eventually leaves our workstation performing poorly. To overcome this problem, we often use virtual machine (VM) instances that run in our workstations or a cloud provider like Azure. Setting up those VMs also introduces some overhead into our software development process. As engineers, we want to be able to accelerate this process by using a remote development environment provider like GitHub Codespaces.

What is GitHub Codespaces?

GitHub Codespaces is a cloud hosted development environment that is associated to a GitHub repository. Each environment or Dev Container is cloud hosted in a Docker container with the core dependencies that are required for that project. The container is hosted on a VM running Ubuntu Linux. The hardware is also configurable. It starts with a minimum of 2 cores, 8 GB of RAMs and 32 GB of storage, which should be a good foundation to run small projects. In addition, the hardware resources can be increased up to 32 cores, 64 GB RAM and 128 GB of storage which matches a very good workstation configuration.

👍 There are monthly quotas for using the remote environments of 120 hrs for personal accounts, and 180 hrs for the PRO account.

How to use Codespaces?

Codespaces leverages the Secure Shell (SSH) protocol, which provides a secure channel between client and server.  This protocol is used to provide remote access to resources like VMs that are hosted on cloud platforms. This protocol makes it possible for browsers and IDE tools like VS Code to connect remotely and manage the projects.

When using the browser, the VS Code Browser IDE is loaded. We can also use a local installation of VS Code or any IDE that support SSH. The development process works the same way as if running locally with the exception that the files are hosted remotely, and we can also use a terminal window to execute build commands within the VM space.

How to start a project with GitHub Codespaces

We can start a Codespaces environment right from GitHub. Open a repo on GitHub and click on the Code tab and then click the Code button from the toolbar. This opens up the options to create a new Codespaces, connect to an existent one, or even configure your Codespaces resources, more on this later. 

👍 You can use this repo if you do not have one https://github.com/ozkary/Data-Engineering-Bootcamp

When you add a new environment, GitHub essentially provisions a VM on Azure. It loads a Docker image with some of the dependencies of your project. For example, if your code is .Net Core or a TypeScript with React project, a Docker image with those dependencies is built and provisioned into the VM.

👍 The Docker images are preconfigured. We can also build a custom image to meet specific requirements.

Once the environment is provisioned, we can open the project using any of the options listed on the image below. I prefer to use my local VS Code instance, as I often have all the tools needed to work on my projects. Once the project is open on VS Code, the project connection is cached, and we only need to open VS Code again to load the remote project. The browser feature is also very useful, so do take it for a spin and see how you like it.

Use a Terminal to Manage the Project

When the project is open remotely, we can run common activities like adding additional dependencies, building and debugging the project. Since the environment is running on Ubuntu, we can open a terminal window on VS Code. This enables us to run the CLI commands that we need in order to manage our project. 

In the case of Web projects, we can run the project remotely using our browser. Even though the project runs remotely on the VM, port forwarding is used for secured remote access, so we can open our local browser and load the app. We can see the forwarded ports for our application on the ports tab of VS Code.

Managing your Codespaces Instance

In some cases, we may see some performance issue on our remote environment. If this is the case, we need to inspect the current instance configuration and if possible upgrade the resources. Since this is an Ubuntu instance, we can use the terminal to run commands like lscpu to check the current configuration like cpu’s and memory. We can also use the Codespaces command toolbar and change the machine type, which provides a quick shortcut to change the machine type or configure the container. 

The Dev Container can also be customized by making changes to the devcontainer.json file. Additional customization can be done by building a custom Docker image to meet specific development environment requirements.

👍 When the Dev Container is changed, the VM requires a re-start, which is done automatically

Summary

By leveraging the use of remote managed development environments, software engineers can save time by not having to work on a development environment configuration, we can instead use GitHub Codespaces to quickly provision Dev Containers that get us up and running in a short time, thus allowing us to focus on our development tasks instead of environment managing tasks.

Thanks for reading.

Send question or comment at Twitter @ozkary

Originally published by ozkary.com

As Software Engineers, we usually work with multiple projects in parallel. This forces us to configure our work stations with multiple softw...

Read Article →

Visual Studio Code Online - Quick Intro

Visual Studio Code (VSCode) Online is a browser hosted IDE for software development purposes. It works similarly as the full version of VSCode.  You can access VSCode Online by visiting https://vscode.dev.  

After the IDE is loaded on your browser, you can connect to any GitHub repo, including repos from other services. As the project loads, you are able to interact with the files associated to the project. These files can be JavaScript, TypeScript, CSharp or any other programming language associated to the project.

As a developer, you are able to browse the files, make edits commit and push back the changes to your repo. In addition, you can debug, do code comparison or load other add-ons to enable other development activities.

This service is not meant to replace your development environment, but is an additional tool to enable your work. Do take a look, and let me know what you think by sending my a message at Twitter @ozkary

Take a look at this video for a quick demo of the tool.

Send question or comment at Twitter @ozkary

Originally published by ozkary.com

Visual Studio Code (VSCode) Online is a browser hosted IDE for software development purposes. It works similarly as the full version of VSCo...

Read Article →

Request a code review with Visual Studio 2012 Premium

Before any code is check-in, a developer can request a code review using Visual Studio. The steps for a code review request and response are as follows:

Send a review request:  (developer)

Do not check-in your code

• On Team Explorer Home, click on Request Code Review

o   Enter the name of the reviewer

o   Enter the following information

§  Task name

§  Area Path

§  Description about the change (Same as you would use for a check-in)

o   Click on Submit request

This creates a Code review request task and a shelve-set which is used by the reviewer to look at the code changes.

Suspend work while waiting on response (developer)

While waiting for a code review response, the developer can suspend the ongoing work and move on to another task. This can be done as follows:

• On Team Explorer Home In progress work items & changes

o   Select Suspend & Shelve

o   Press Suspend

This creates a shelve-set and the developer can move on to a different task. Once the review is complete, the developer will be able to select the suspended work and resume work based on the feedback from the reviewer.

Response to a review request: (reviewer)

• On Team Explorer Home, My Work

o   Click on  Code Reviews and  Requests

o   Open the pending reviews

This shows all the files that have been modified. The reviewer can now do the following:

• Do a code compare with the previous version
• Add comments in the areas where a change was made
• Set the status of the review request according to the comments
• Press Send and Finish

This closes the review request task and creates another code review response task for the developer to read the comments and resume work.

Only after the code review has passed, the developer should be able to check-in the code and update the task status. 

Before any code is check-in, a developer can request a code review using Visual Studio. The steps for a code review request and response ar...

Read Article →

What is the difference between ObjectSet and EntityCollection?

When working with the Entity framework, you may see that there are two types of collection created, ObjectSet and EntityCollection. Many of us get a bit confused as of why there are two types of collections and about their use. I will try to describe my understanding of their differences:

ObjectSet Definition from MSDN:

public class ObjectSet<TEntity> : ObjectQuery<TEntity>,

        IObjectSet<TEntity>, IQueryable<TEntity>, IEnumerable<TEntity>,

        IQueryable, IEnumerable 

where TEntity : class

ObjectSet provides access to entities/tables of a database.  ObjectSet is IQueryable which means that when you use this reference with a LINQ query, the expression is changed to the SQL expression for the provider (i.e. Entity classes mapping to SQL Server). This is how LINQ works for the LINQ to Entities expressions to be able to select records from the database.

EntityCollection Definition from MSDN:

public sealed class EntityCollection<TEntity> : RelatedEnd, 

        ICollection<TEntity>, IEnumerable<TEntity>, IEnumerable, IListSource 

where TEntity : class

EntityCollection is a collection type which represents “many” entities. When we use LINQ to query an EntityCollection, we are using LINQ to Objects to retrieve the “in-memory” objects in the collection, and no query is executed in the database.  The records for this collection are the ones associated to a main entity as a property and are loaded using the Load method which uses an ObjectSet to load the records from the database. Look at this pseudocode:

VehicleMake: IQueryable{                //represents the ObjectSet

        VehicleModels:IEnumerable      //represents an EntityCollection populated by Load()

}

VehicleModels:IQueryable{// ObjectSet which gets the records from the database

}

og-bit.com

When the collection is a property is an EntityCollection.

I hope this is able to help some of you.

When working with the Entity framework, you may see that there are two types of collection created, ObjectSet and EntityCollection. Many of...

Read Article →

Fluent Interface to Simplify an API

The mail goal of an API is to facilitate the use of commands and attributes to achieve certain functionality.  Over time, an API may have evolved and become somewhat obscure on its usage which reduces its usability and increases the implementation time.

To clean/simplify an API you can apply a fluent interface to it. This allows you to modify the API with a more descriptive interface which improves its usability. At the same time, this preserves the current API interface and eliminates the risk of introducing bugs to code that is already implemented.  To get a better understanding, let‘s take a look at an order system. To create an order, an API often uses the following code:

//customer and order detail

Customer cust = new Customer() { CustomerId = 12900 };

OrderItem item1 = new OrderItem { Name = "Pen", PricePerUnit = .99, Quantity = 10, Sku = "1675" };

OrderItem item2 = new OrderItem { Name = "Notepad", PricePerUnit = 2.99, Quantity = 10, Sku = "5456" };

OrderItem item3 = new OrderItem { Name = "Marker", PricePerUnit = 1.99, Quantity = 10, Sku = "9836" };

//standard API to create the order

OrderHeader oh = new OrderHeader();

oh.Customer = cust;

oh.AddItem(item1);

oh.AddItem(item2);

oh.AddItem(item3);

oh.Complete();

oh.Send();

That code sample is a simple API order system that is creating an order request with three items. You may argue that the code is really simple to follow, and there is no need to refactor, and I agree with you. However, we should think of a more complex API which may require you to add so many lines of code and call methods that are not that descriptive. With that in mind, let’s look now at how we can make the order creation a bit fluent:

//Fluent API

OrderHeader.CreateOrder()

.WithCustomer(cust)

.WithItems(item1,item2,item3)

.CompleteAndSend();

You can see that the code reads more like a sentence or language than actual method calls. It is descriptive and easier to use which is our main goal.  So how does the code for this look?  Well, let’s see now:

/// <summary>

/// defines a domain specific interface

/// </summary>

interface FluentOrderHeader

{

FluentOrderHeader WithItems(params OrderItem[] item);

FluentOrderHeader WithCustomer(Customer cust);

void CompleteAndSend();

}

/// <summary>

/// partial class to expose fluent api

/// </summary>

partial class OrderHeader : FluentOrderHeader

{

public static FluentOrderHeader CreateOrder()

{

return new OrderHeader();

}

public FluentOrderHeader WithItems(params OrderItem[] items)

{

foreach (OrderItem item in items)

{

   this.AddItem(item);

}

return this;

}

public FluentOrderHeader WithCustomer(Customer cust)

{

this.Customer = cust;

return this;

}

public void CompleteAndSend()

{

this.Complete();

this.Send();

}

}

We first define the interface with the methods that can help our API become fluent.  You should note that each call returns the instance reference. This allows for the chaining of the methods.  The next step is to create a partial class of the OrderHeader that implements our interface. This allows us to isolate the changes to the partial class without making any modifications to the current code. This new class handles the implementation changes using the existent API.

In my opinion, a fluent interface should be used to simplify obscure APIs which are often used in several projects. If the API is only use by one system, the effort to adapt a fluent interface may not be necessary. We need to keep in mind that a fluent implementation requires a bit more thinking, so it can actually provide a descriptive and fluent set of methods which allows any developer to easily follow it. Do not confuse this with just method chaining. The main different between method chaining and fluent interface is that with the latter we are trying to define a domain specific language that tries to target a specific task. Method chaining just facilitates the use of APIs by returning a reference, so it can be used in subsequent calls.

og-bit.com
CodeProject

The mail goal of an API is to facilitate the use of commands and attributes to achieve certain functionality.  Over time, an API may have ev...

Read Article →

Extend web control with an extension method

When using Label and TextBox controls, we often have the need to reset the text value and/or hide the control during a post back. A common example is when using a Label control to display messages to the user. We only need to set the message once after a postback. We may then need to reset the content of that message and not display the control all together on subsequent postbacks.  

To avoid repeating the same lines of codes, we can extent the web controls by adding an extension method that would handle that for us.

public static class ogUIExtensions

{

       

       /// <summary>

       /// resets the text control properties and hide

       /// </summary>

       /// <param name="ctrl"></param>

       public static void ResetAndHide(this WebControl ctrl)

       {

            ctrl.Visible = false;

            (ctrl as ITextControl).Text = string.Empty; 

       }

}

This snippet extends the WebControl class by adding a method that basically sets the Visible property to false (hide) and sets the Text property of the control to an empty string. The WebControl class has not Text property, so we need to upcast to the ITextControl interface to have access to that property. Both Label and TextBox controls implement the ITextControl interface.

We can use this extension method in this way:

Label lbl = new Label();

lbl.ResetAndHide();

I hope it helps.

og-bit.com

When using Label and TextBox controls, we often have the need to reset the text value and/or hide the control during a post back. A common...

Read Article →

Select/UnSelect all items from CheckBoxList, DropDownList, ListBox with an Extension Method

The CheckBoxList, DropDownList and ListBox controls have an items collection based on the ListItemCollection class. To select/unselect the items on the list, we can iterate through the items in the collection and set the selected attribute to true or false.

The following code snippet shows an extension method that implements a common solution for all these controls. The method allows us to extend the functionality of these controls, so we are able to set the selected attribute with just one call.


public static class ogUIExtensions
    {
        /// <summary>
        /// extension method for all ListItemCollection controls to set the selected state
        /// </summary>
        /// <param name="list"></param>
        /// <param name="state"></param>
        public static void SelectAll(this ListItemCollection list, bool state)
        {          
            foreach (ListItem item in list)
            {
                item.Selected = state;
            }
        }
    }


Whenever, we need to set the selected state of any of these controls, we just need to call the method as shown below:

CheckBoxList ckList = new CheckBoxList();
ckList.Items.SelectAll(true); //checks all the items
ckList.Items.SelectAll(false);//unchecks all the items

The same can be done for ListBox and DropDownList controls.
Hope it helps.

The CheckBoxList, DropDownList and ListBox controls have an items collection based on the ListItemCollection class. To select/unselect the i...

Read Article →