Data Engineering Process Fundamentals - Discovery Exercise

In this discovery exercise lab, we review a problem statement and do the analysis to define the scope of work and requirements.

Problem Statement

In the city of New York, commuters use the Metropolitan Transportation Authority (MTA) subway system for transportation. There are millions of people that use this system every day; therefore, businesses around the subway stations would like to be able to use Geofencing advertisement to target those commuters or possible consumers and attract them to their business locations at peak hours of the day.

Geofencing is a location based technology service in which mobile devices’ electronic signal is tracked as it enters or leaves a virtual boundary (geo-fence) on a geographical location. Businesses around those locations would like to use this technology to increase their sales.

The MTA subway system has stations around the city. All the stations are equipped with turnstiles or gates which tracks as each person enters or leaves the station. MTA provides this information in CSV files, which can be imported into a data warehouse to enable the analytical process to identify patterns that can enable these businesses to understand how to best target consumers.

Analytical Approach

Dataset Criteria

We are using the MTA Turnstile data for 2023. Using this data, we can investigate the following criteria:

• Stations with the high number of exits by day and hours
• Stations with high number of entries by day and hours

Exits indicates that commuters are arriving to those locations. Entries indicate that commuters are departing from those locations.

Data Analysis Criteria

The data can be grouped into stations, date and time of the day. This data is audited in blocks of fours hours apart. This means that there are intervals of 8am to 12pm as an example. We analyze the data into those time block intervals to help us identify the best times both in the morning and afternoon for each station location. This should allow businesses to target a particular geo-fence that is close to their business.

In the discovery process, we take a look at the data that is available for our analysis. We are using the MTA turnstiles information which is available at this location:

👉 New York Metropolitan Transportantion Authority Turnstile Data

We can download a single file to take a look at the data structure and make the following observations about the data:

Observations

• It is available in weekly batches every Sunday
• The information is audited in blocks of fours hours apart
• The date and time field are on different columns
• The cumulative entries are on the ENTRIES field
• The cumulative exits are on the EXITS field
• This data is audited in blocks of fours hours apart

Field Description

Name	Description
C/A	Control Area (A002) (Booth)
UNIT	Remote Unit for a station (R051)
SCP	Subunit Channel Position represents an specific address for a device (02-00-00)
STATION	Represents the station name the device is located at
LINENAME	Represents all train lines that can be boarded at this station. Normally lines are represented by one character. LINENAME 456NQR repersents train server for 4, 5, 6, N, Q, and R trains.
DIVISION	Represents the Line originally the station belonged to BMT, IRT, or IND
DATE	Represents the date (MM-DD-YY)
TIME	Represents the time (hh:mm:ss) for a scheduled audit event
DESc	Represent the “REGULAR” scheduled audit event (Normally occurs every 4 hours). Audits may occur more that 4 hours due to planning, or troubleshooting activities. Additionally, there may be a “RECOVR AUD” entry: This refers to missed audit that was recovered.
ENTRIES	The cumulative entry register value for a device
EXIST	The cumulative exit register value for a device

Data Example

The data below shows the entry/exit register values for one turnstile at control area (A002) from 09/27/14 at 00:00 hours to 09/29/14 at 00:00 hours

C/A	UNIT	SCP	STATION	LINENAME	DIVISION	DATE	TIME	DESC	ENTRIES	EXITS
A002	R051	02-00-00	LEXINGTON AVE	456NQR	BMT	09-27-14	00:00:00	REGULAR	0004800073	0001629137
A002	R051	02-00-00	LEXINGTON AVE	456NQR	BMT	09-27-14	04:00:00	REGULAR	0004800125	0001629149
A002	R051	02-00-00	LEXINGTON AVE	456NQR	BMT	09-27-14	08:00:00	REGULAR	0004800146	0001629162

Conclusions

Based on observations, the following conclusions can be made:

• Merge the DATE and TIME columns and create a date time column, CREATED
• The STATION column is a location dimension
• The CREATED column is the datetime dimension to enable the morning and afternoon timeframes
• The ENTRIES column is the measure for entries
• The EXITS column is the measure for exits
• A gate can be identified by using the C/A, SCP and UNIT columns

Requirements

These observations can be used to define technical requirements that can enable us to deliver a successful project.

• Define the infrastructure requirements to host the technology
  • Automate the provisioning of the resources using Terraform
  • Deploy the technology on a cloud platform
• Define the data orchestration process
  • On the original pipeline, load the initial data for 2023
  • Create a data pipeline that runs every week after a new file has been published
  • Copy the unstructured CSV files into a Data Lake
• Define a well-structured and optimized model on a Data Warehouse
  • Keep the source code for the models under source control
  • Copy the data into the Data Warehouse
  • Allow access to the Data Warehouse, so visualization tools can consume the data.
• Create Data Analysis dashboard with the following information
  • Data Analysis dashboard
  • Identify the time slots for morning and afternoon analysis
  • Look at the distribution by stations
  • Look at the daily models
  • Look at the time slot models

Review the Code

In order to do our data analysis, we need to first download some sample data by writing a Python script. We can the analyze this data by writing some code snippets and use the power of the Python Pandas library. We can also use Jupyter Notebooks to quickly manipulate the data and create some charts that can help us as baseline requirements for the final visualization dashboard.

👉 Clone this repo or copy the files from this folder

Download a CSV File from the MTA Site

With this Python script (mta_discovery.py), we download a CSV file with the URL of http://web.mta.info/developers/data/nyct/turnstile/turnstile_230318.txt. The code creates a data stream to download the file in chunks to avoid any timeouts. We append the chunks into a local compressed file to reduce the size of the file. In order to reuse this code, we use the command line parser, so we can pass as parameters the URL.

import os
import argparse
from time import time
from pathlib import Path
import pandas as pd


def read_local(file_path: str) -> Path:
    """
        Reads a local file
        Args:
            file_path:  local file            
    """
    print(F'Reading local file {file_path}')
    df_iter = pd.read_csv(file_path, iterator=True,compression="gzip", chunksize=10000) 
    if df_iter:        
        for df in df_iter:
            try:                                
                print('File headers',df.columns)                                
                print('Top 10 rows',df.head(10))            
                break
            except Exception as ex:
                print(f"Error found {ex}")
                return

        print(f"file was loaded {file_path}")        
    else:
        print(F"failed to read file {file_path}")

def write_local(df: pd.DataFrame, folder: str, file_name: str) -> Path:
    """
        Write DataFrame out locally as csv file
        Args:
            df: dataframe chunk
            folder: the download data folder
            file_name: the local file name
    """

    path = Path(f"{folder}")
    if not os.path.exists(path):
        path.mkdir(parents=True, exist_ok=True)

    file_path = Path(f"{folder}/{file_name}")

    if not os.path.isfile(file_path):
        df.to_csv(file_path, compression="gzip")
        print('new file')
    else:
        df.to_csv(file_path, header=None, compression="gzip", mode="a")    
        print('chunk appended')

    return file_path

def etl_web_to_local(url: str, name: str) -> None:
    """
       Download a file    
       Args:
            url : The file url
            name : the file name

    """
    print(url, name)      

    # skip an existent file
    path = f"../data/"
    file_path = Path(f"{path}/{name}.csv.gz")
    if os.path.exists(file_path):
            read_local(file_path)            
            return

    df_iter = pd.read_csv(url, iterator=True, chunksize=10000) 
    if df_iter:      
        file_name = f"{name}.csv.gz"    
        for df in df_iter:
            try:                                                
                write_local(df, path, file_name)
            except StopIteration as ex:
                print(f"Finished reading file {ex}")
                break
            except Exception as ex:
                print(f"Error found {ex}")
                return

        print(f"file was loaded {file_path}")        
    else:
        print("dataframe failed")

def main_flow(params: str) -> None:
    """
        Process a CSV file from a url location with the goal to understand the data structure
    """    
    url = params.url  
    prefix = params.prefix

    try:
        start_index = url.index('_')
        end_index = url.index('.txt')
        file_name = F"{prefix}{url[start_index:end_index]}"
        # print(file_name)
        etl_web_to_local(url, file_name)
    except ValueError:
        print("Substring not found")


if __name__ == '__main__':

    os.system('clear')    
    parser = argparse.ArgumentParser(description='Process CSV data to understand the data')
    parser.add_argument('--url', required=True, help='url of the csv file')
    parser.add_argument('--prefix', required=True, help='the file prefix or group name')
    args = parser.parse_args()

    print('running...')
    main_flow(args)
    print('end')

Analyze the Data

With some sample data, we can now take a look at the data and make some observations. There are a few ways to approach the analysis. We could create another Python script and play with the data, but this will require to run the script from the console after every code change. A more productive way is to use Jupyter Notebooks. This tools enables us to edit and run code snippets in cells without having to run the entire script. This is a friendlier analysis tool that can help us focus on the data analysis instead of coding and running the script. In addition, once we are good with our changes, the notebook can be exported into a Python file. Let's look at that file discovery.ipynb:

import os
import argparse
from time import time
from pathlib import Path
import pandas as pd 

# read the file and display the top 10 rows
df = pd.read_csv('../data/230318.csv.gz', iterator=False,compression="gzip")
df.head(10)

# Create a new DateTime column and merge the DATE and TIME columns
df['CREATED'] =  pd.to_datetime(df['DATE'] + ' ' + df['TIME'], format='%m/%d/%Y %H:%M:%S')
df = df.drop('DATE', axis=1).drop('TIME',axis=1)
df.head(10)

# Aggregate the information by station and datetime
df["ENTRIES"] = df["ENTRIES"].astype(int)
df["EXITS"] = df["EXITS"].astype(int)
df_totals = df.groupby(["STATION","CREATED"], as_index=False)[["ENTRIES","EXITS"]].sum()
df_totals.head(10)

df_station_totals = df.groupby(["STATION"], as_index=False)[["ENTRIES","EXITS"]].sum()
df_station_totals.head(10)

# Show the total entries by station, use a subset of data
import plotly.express as px
import plotly.graph_objects as go

df_stations =  df_station_totals.head(25)
donut_chart = go.Figure(data=[go.Pie(labels=df_stations["STATION"], values=df_stations["ENTRIES"], hole=.2)])
donut_chart.update_layout(title_text='Entries Distribution by Station', margin=dict(t=40, b=0, l=10, r=10))
donut_chart.show()

# Show the data by the day of the week
df_by_date = df_totals.groupby(["CREATED"], as_index=False)[["ENTRIES"]].sum()
day_order = ['Sun', 'Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat']
df_by_date["WEEKDAY"] = pd.Categorical(df_by_date["CREATED"].dt.strftime('%a'), categories=day_order, ordered=True)
df_entries_by_date =  df_by_date.groupby(["WEEKDAY"], as_index=False)[["ENTRIES"]].sum()
df_entries_by_date.head(10)

bar_chart = go.Figure(data=[go.Bar(x=df_entries_by_date["WEEKDAY"], y=df_entries_by_date["ENTRIES"])])
bar_chart.update_layout(title_text='Total Entries by Week Day')
bar_chart.show()

How to Run it!

With an understanding of the code and tools, let's run the process.

Requirements

👉 Install Python, Pandas and Jupyter notebook

👉 Install Visual Studio Code

👉 Clone this repo or copy the files from this folder

Follow these steps to run the analysis

• Download a file to look at the data
  • This should create a gz file under the ../data folder

$ python3 mta_discovery.py --url http://web.mta.info/developers/data/nyct/turnstile/turnstile_230318.txt

Run the Jupyter notebook (dicovery.ipynb) to do some analysis on the data.

• Load the Jupyter notebook to do analysis
  • First start the Jupyter server from the terminal

$ jupyter notebook

• See the URL on the terminal and click it to load it on the browser
  • Click the discovery.ipynb file link
• Or open the file with VSCode and enter the URL when prompted from a kernel url
• Run every cell from the top down as this is required to load the dependencies

The following images show Jupyter notebook loaded on the browser or directly from VSCode.

Jupyter Notebook loaded on the browser

Using VSCode to load the data and create charts

Show the total entries by station using a subset of data using VSCode

Next Step

👉 Data Engineering Process Fundamentals - Design and Planning

Thanks for reading.

Send question or comment at Twitter @ozkary

👉 Originally published by ozkary.com

In this discovery exercise lab, we review a problem statement and do the analysis to define the scope of work and requirements. Problem S...

Read Article →

Data Engineering Process Fundamentals - Discovery

Introduction

In this series of Data Engineering Process Fundamentals, we explore the Data Engineering Process (DEP) with key concepts, principles and relevant technologies, and explain how they are being used to help us deliver solutions. The first step, and important to never skip, in this process is the Discovery step.

During the discovery step of a Data Engineering Process, we look to identify and clearly document a problem statement, which helps us have an understanding of what we are trying to solve. We also look at our analytical approach to make observations about at the data, its structure and source. This leads us into defining the requirements for the project, so we can define the scope, design and architecture of the solution.

Problem Statement

A Problem Statement is a description of what it is that we are trying to solve. As part of the problem statement, we should provide some background or context on how the data is processed or collected. We should also provide a specific description of what the data engineering process is looking to solve by taking a specific approach to integrate the data. Finally, the objective and goals should also be described with information about how this data will be made available for analytical purposes.

Analytical Approach

The Analytical Approach is a systematic method to observe the data and arrive to insights from it. It involves the use of different techniques, tools and frameworks to make sense of the data in order to arrive to conclusions and actionable requirements.

Dataset Criteria

A Dataset Criteria technique refers to the set of characteristics used to evaluate the data, so we can determine the quality and accuracy of it.

In the data collection process, we should identify the various sources that can provide us with accurate and complete information. Data cleaning and preprocessing needs to be done to identify and eliminate missing values, invalid data and outliers that can skew the information. In addition, we should understand how this data is available for the ongoing integration. Some integrations may require a batch process integration at a scheduled interval. Others may require a real-time integration and/or a combination of batch and real-time processing.

Exploratory Data Analysis

We should conduct exploratory data analysis to understand the structure, patterns and characteristics of the data. We need to make observations about the data, identify the valuable fields, create statistical summaries, and run some data profiling to identify trends, metrics and correlations that are relevant to the main objective of the project.

Tools and Framework

Depending on the size and budget of the organization, the solution can be built with lots of coding and integration, or instead a low-code turn-key solution that provides enterprise quality resources could be used instead. Regardless of the approach, a programming language like Python is a popular programming language for data science and engineers, and it is always applicable. The Python Pandas library is great for data manipulation and analysis. Jupyter notes with Python scripts is great for experiments and discovery.

To run our Python scripts and Jupyter notebooks, we can use Visual Studio Code (VSCode), which is cross-platform Integrated Development Environment (IDE) tool. This tool also enables the integration with source control and deployments platforms like GitHub, so we can maintain version control and automate the deployment and test of our code changes.

To orchestrate the pipelines, we often use a workflow framework like Apache Airflow, Prefect. To host the data, we use data lakes (blob storage) and a relational data warehouse. For data modeling, incremental data and continuous test and data ingestion, Apache Spark or gbt cloud are used.

For the final data analysis and visualization, we could use tools like Looker, PowerBI and Tableau. These are tools that can connect to a data warehouse and consume the data models, so they can visualize in ways that enable stakeholders to make decisions based on the story provided by the data.

Requirements

Requirements refer to the needs, capabilities and constraints that are needed to deliver a data engineering solution. They should outline the project deliverables that are required to meet the main objectives. The requirements should include data related areas like:

• Sources and integration
• Modeling and transformation
• Quality and validation
• Storage and infrastructure
• Processing and Analytics
• Governance and Security
• Scalability and performance
• Monitoring

Design

A data engineering design is the actual plan to build the technical solution. It includes the system architecture, data integration, flow and pipeline orchestration, the data storage platform, transformation and management, data processing and analytics tooling. This is the area where we need to clearly define the different technologies that should be used for each area.

System Architecture

The system architecture is a high-level design of the solution, its components and how they integrate with each other. This often includes the data sources, data ingestion resources, workflow and data orchestration resources and frameworks, storage resources, data services for data transformation and continuous data ingestion and validation, and data analysis and visualization tooling.

Data Pipelines

A data pipeline refers to a series of connected tasks that handles the extract, transform and load (ETL) as well as the extract, load and transform (ELT) operations and integration from a source to a target storage like a data lake or data warehouse.

The use of ETL or ELT depends on the design. For some solutions, a flow task may transform the data prior to loading it into storage. This approach tends to increase the amount of python code and hardware resources used by the hosting environment. For the ELT process, the transformation may be done using SQL code and the data warehouse resources, which often tend to perform great for big data scenarios.

Data Orchestration

Data orchestration refers to the automation, management and coordination of the data pipeline tasks. It involves the scheduling, workflows, monitoring and recovery of those tasks. The orchestration ensures the execution of those tasks, and it takes care of error handling, retry and the alerting of problems in the pipeline.

Summary

The data engineering discovery process involves defining the problem statement, gathering requirements, and determining the scope of work. It also includes a data analysis exercise utilizing Python and Jupyter Notebooks ,or other tools, to extract valuable insights from the data. These steps collectively lay the foundation for successful data engineering endeavors.

Exercise - Hands-on Use Case

Since we now understand the discovery step, we should be able to put that into practice. Let’s move on to a hands-on use case and see how we apply those concepts.

👉 Data Engineering Process Fundamentals - Discovery Exercise

Thanks for reading.

Send question or comment at Twitter @ozkary

👍 Originally published by ozkary.com

Introduction In this series of Data Engineering Process Fundamentals, we explore the Data Engineering Process (DEP) with key concepts, prin...

Read Article →

Data Engineering Process Fundamentals

Introduction

Data Engineering is changing constantly. From cloud data platforms and pipeline automation to data streaming and visualizations tools, new innovations are impacting that way we build today’s data and analytical solutions.

In this series of Data Engineering Process Fundamentals, we explore the Data Engineering Process (DEP) with key concepts, principles and relevant technologies, and explain how they are being used to help us deliver the solution. We discuss concepts and take on a real use case where we execute an end-to-end process from downloading data to visualizing the results.

The end-goal of this series is to take us thru a process in which we deliver an architecture, which facilitates the ongoing analysis of big data via analytical and visualization tools. In the following images, we can get a preview of what we will be delivering as we execute each step of the process.

Data Engineering - Architecture

Data Engineering - Analysis Results

Data Engineering Process

A Data Engineering Process follows a series of steps that should be executed to properly understand the problem statement, scope of work, design and architecture that should be used to create the solution. Some of these steps include the following:

👉 Join this newsletter to receive updates Sign up here

• Discovery
  • Problem Statement
  • Data Analysis
  • Define the Requirements and Scope of Work
  • Discovery Exercise
• Design and Planning
  • Design Approach
  • System Architecture
  • Cloud Engineering and Automation
  • Design Exercise

• Data Orchestration and Operations
  • Pipeline Orchestration
    • Batch Processing
  • Workflow Automation
  • Deployment, Schedules and Monitoring
• Data Warehouse and Modeling
  • Data modeling
  • Data Warehouse Design
  • Continuous Integration
• Data Analysis and Visualization
  • Analyze the data
  • Visualization Concepts
  • Create a Dashboard
    • Provide answers to the problem statement
• Streaming Data
  • Data Warehouse Integration
  • Real-time dashboard

Concepts

What is Data Engineering?

Data Engineering is the practice of designing and building solutions by integrating, transforming and consolidating various data sources into a centralized and structured system, Data Warehouse, at scale, so the data becomes available for building analytics solutions.

What is a Data Engineering Process?

A Data Engineering Process (DEP) is the sequence of steps that engineers should follow in order to build a testable, robust and scalable solution. This process starts really early on with a problem statement to understand what the team is trying to solve. It is then followed with data analysis and requirements discovery, which leads to a design and architecture approach, in which the different applicable technologies are identified.

Operational and Analytical data

Operational data is often generated by applications, and it is stored in transactional databases like SQL Server, CosmosDB, Firebase and others. This is the data that is created after an application saves a user transaction like contact information, a purchase or other activities that are available from the application. This system is not design to support Big Data query scenarios, so the reporting system should not be overloading its resources with large queries.

Analytical data is the transaction data that has been processed and optimized for analytical and visualization purposes. This data is often processed via Data Lakes and stored on Data Warehouse.

Data Pipelines and Orchestration

Data Pipelines are used to orchestrate and automate workflows to move and process the transactional into Data Lakes and Data Warehouse. The pipelines execute repeatable Extract Transform and Load (ETL) or Extract Load and Transform (ELT) processes that can be triggered by a schedule or a data event.

Data Lakes

A Data Lake is an optimized storage system for Big Data scenarios. The primary function is to store the data in its raw format without any transformation. This can include structure data like CSV files, unstructured data like JSON and XML documents, or column-base data like parquet files.

Data Warehouse

A Data Warehouse is a centralized storage system that stores integrated data from multiple sources. This system stores historical data in relational tables with an optimized schema, which enables the data analysis process. This system can also integrate external resources like CSV and parquet files that are stored on Data Lakes as external tables. The system is designed to host and serve Big Data scenarios. It is not meant to be used as a transactional system.

Data Batch Processing

Batch Processing is a method often used to run high-volume, repetitive data jobs. It is usually scheduled during certain time windows that do not impact the application operations, as these processes are often used to export the data from transactional systems. A batch job is an automated software task that may include one or more workflows. These workflows can often run without supervision, and they are monitored by other tools to ensure that the process is not failing.

Streaming Data

Streaming Data is a data source that sends messages with small content but with high volume of messages in real-time. This data often comes from Internet-of-things (IoT) devices, manufacturing equipment or social media sources, often producing a high volume of information per second. This information is often captured in aggregated time windows and then store in a Data Warehouse, so it can be combined with other analytical data. It can also be sent to monitoring and/or real-time systems to show the current system KPI or any type of variance in the system.

Next Step

👉 Data Engineering Process Fundamentals - Discovery

Thanks for reading.

Send question or comment at Twitter @ozkary

👍 Originally published by ozkary.com

Introduction Data Engineering is changing constantly. From cloud data platforms and pipeline automation to data streaming and visualization...

Read Article →

GitHub Codespaces Quick Review

This is a quick review about GitHub Codespaces, which you can load right on the browser.

In this video, we talk about creating a Codespaces instance from a GitHub Repository. We load a GitHub project using the VM instance that is provisioned for us when a GitHub Codespace is added to the repo. To edit the project files, the browser loads a VS Code online version of the IDE, which then uses the SSH protocol to virtualize the code from the VM onto our browser. Since we are basically using a VM, we can open a terminal from the browser session and run NPM and Git commands. All these commands are executed on the VM. GitHub Codespaces are a quick way to provision a VM instance without the complexity of manually building on a cloud account.

Thanks for reading.

Send question or comment at Twitter @ozkary

Originally published by ozkary.com

This is a quick review about GitHub Codespaces, which you can load right on the browser. In this video, we talk about creating a Codes...

Read Article →

React Suspense Fallback Keeps Rendering When Using Lazy-Loaded Routes

When a React application is loading a new lazy-loaded route or a component, it is a common practice to show a loading component with an animation to provide feedback to the user. In React, we use the Suspense component to handle this scenario, but there are times when the fallback component never unloads, and it keeps rendering on the browser.

When a Suspense fallback component never unloads, it is because there must be a child component that keeps rendering. This causes the Suspense component to continue to run, thinking that a child component in still in suspense or loading state.  This gives us the wrong sense that the Suspense component is misbehaving, when it is in fact the child components that is at fault. For a deeper understanding, let’s take a look at a real example.

Suspense with Declarative Routes

To demonstrate this problem, let’s first take a look at an implementation of a React application that loads some route information in a declarative approach, which is simple enough and does not introduce any rendering problems. We should also notice how the components are lazy-loaded to help us do code splitting for performance improvements on the load time and enable us to trigger the Suspense component automatically.

👍 This is a typical approach when creating apps with a simple routing structure.

Suspense with a Router Component

Let’s now take a look at a more complex scenario where we need to load the route information from a JSON configuration file. This introduces a variation on the process by adding the routes using a function. 

This code change introduces new behavior, and it causes the component to re-render multiple times. We can trace that by adding a console.log operation before returning the content. If we look at the browser console, we should see the output of our console.log call. From the application standpoint, this behavior is noticeable because the fallback component continues to show on the browser as the Suspense component does not detect that the child component is done rendering. Now that we understand the problem, how should we correct it?

👍 A child component continues to be in suspense until it stops rendering.

Adding State Management

To correct this behavior, we should clearly understand the root cause. Since we introduce a dynamic way to load the routes, the component has no way to understand its current state. It only knows that some data is being loaded every time it calls the function, and the data seems new or different. To avoid this, we need to add state management to the component, which is a React best practice when writing data driven components. Let’s refactor our code to see how state management can make a difference.

By looking at our new implementation, we can see that the route collection is now managed in a state variable. We also use an useEffect hook to load the data. This enables us to initiate the state of the component with some valid data. We also track the route collection as a dependency, and since there are no changes to the data, the state does not change, and the component completes its suspense state thus allowing the app to complete its rendering process.

Conclusion

The React Suspense component is a great feature to provide feedback to users while a component is rendering. When we add dynamic data to the application, we should understand that this impacts the state management process, which is used to signal when a component is in a suspense state. Depending on how nested your components are, a child component can continue to render, causing the Suspense fallback UI to continue to load non-stop. 

👍 To diagnose a component, you can add a suspense component around it, and we should notice only that component content to continue to use the suspense fallback animation.

Thanks for reading.

Send question or comment at Twitter @ozkary

Originally published by ozkary.com

When a React application is loading a new lazy-loaded route or a component, it is a common practice to show a loading component with an anim...

Read Article →

Use Remote Dev Container with GitHub Codespaces

As Software Engineers, we usually work with multiple projects in parallel. This forces us to configure our work stations with multiple software development tools, which eventually leaves our workstation performing poorly. To overcome this problem, we often use virtual machine (VM) instances that run in our workstations or a cloud provider like Azure. Setting up those VMs also introduces some overhead into our software development process. As engineers, we want to be able to accelerate this process by using a remote development environment provider like GitHub Codespaces.

What is GitHub Codespaces?

GitHub Codespaces is a cloud hosted development environment that is associated to a GitHub repository. Each environment or Dev Container is cloud hosted in a Docker container with the core dependencies that are required for that project. The container is hosted on a VM running Ubuntu Linux. The hardware is also configurable. It starts with a minimum of 2 cores, 8 GB of RAMs and 32 GB of storage, which should be a good foundation to run small projects. In addition, the hardware resources can be increased up to 32 cores, 64 GB RAM and 128 GB of storage which matches a very good workstation configuration.

👍 There are monthly quotas for using the remote environments of 120 hrs for personal accounts, and 180 hrs for the PRO account.

How to use Codespaces?

Codespaces leverages the Secure Shell (SSH) protocol, which provides a secure channel between client and server.  This protocol is used to provide remote access to resources like VMs that are hosted on cloud platforms. This protocol makes it possible for browsers and IDE tools like VS Code to connect remotely and manage the projects.

When using the browser, the VS Code Browser IDE is loaded. We can also use a local installation of VS Code or any IDE that support SSH. The development process works the same way as if running locally with the exception that the files are hosted remotely, and we can also use a terminal window to execute build commands within the VM space.

How to start a project with GitHub Codespaces

We can start a Codespaces environment right from GitHub. Open a repo on GitHub and click on the Code tab and then click the Code button from the toolbar. This opens up the options to create a new Codespaces, connect to an existent one, or even configure your Codespaces resources, more on this later. 

👍 You can use this repo if you do not have one https://github.com/ozkary/Data-Engineering-Bootcamp

When you add a new environment, GitHub essentially provisions a VM on Azure. It loads a Docker image with some of the dependencies of your project. For example, if your code is .Net Core or a TypeScript with React project, a Docker image with those dependencies is built and provisioned into the VM.

👍 The Docker images are preconfigured. We can also build a custom image to meet specific requirements.

Once the environment is provisioned, we can open the project using any of the options listed on the image below. I prefer to use my local VS Code instance, as I often have all the tools needed to work on my projects. Once the project is open on VS Code, the project connection is cached, and we only need to open VS Code again to load the remote project. The browser feature is also very useful, so do take it for a spin and see how you like it.

Use a Terminal to Manage the Project

When the project is open remotely, we can run common activities like adding additional dependencies, building and debugging the project. Since the environment is running on Ubuntu, we can open a terminal window on VS Code. This enables us to run the CLI commands that we need in order to manage our project. 

In the case of Web projects, we can run the project remotely using our browser. Even though the project runs remotely on the VM, port forwarding is used for secured remote access, so we can open our local browser and load the app. We can see the forwarded ports for our application on the ports tab of VS Code.

Managing your Codespaces Instance

In some cases, we may see some performance issue on our remote environment. If this is the case, we need to inspect the current instance configuration and if possible upgrade the resources. Since this is an Ubuntu instance, we can use the terminal to run commands like lscpu to check the current configuration like cpu’s and memory. We can also use the Codespaces command toolbar and change the machine type, which provides a quick shortcut to change the machine type or configure the container. 

The Dev Container can also be customized by making changes to the devcontainer.json file. Additional customization can be done by building a custom Docker image to meet specific development environment requirements.

👍 When the Dev Container is changed, the VM requires a re-start, which is done automatically

Summary

By leveraging the use of remote managed development environments, software engineers can save time by not having to work on a development environment configuration, we can instead use GitHub Codespaces to quickly provision Dev Containers that get us up and running in a short time, thus allowing us to focus on our development tasks instead of environment managing tasks.

Thanks for reading.

Send question or comment at Twitter @ozkary

Originally published by ozkary.com

As Software Engineers, we usually work with multiple projects in parallel. This forces us to configure our work stations with multiple softw...

Read Article →

API CORS Support with Azure API Management

In a Single Page Applications architecture, APIs are used for the application integration with a backend service. When both the SPA and API are hosted on the same domain name, the integration is simple, and the client application can call the APIs directly. When the SPA and API are hosted on different domains, we have to create Cross Domain Access Sharing CORS policies to authorize the client app to access the API. Otherwise, the client application is blocked from calling the APIs.

In cloud platform scenarios, the API is accessible via a gateway, which it is often used to protect the access to internal APIs by enforcing security policies like CORS and IP whitelisting. A very common use case is illustrated below:

On this diagram, we have an SPA application hosted in the app.ozkary.com domain. The app needs to integrate with an internal API that is not available via a public domain. To enable the access to the API, a gateway is used to accept inbound public traffic. This gateway is hosted on a different domain name, api.services.com. Right away, we can expect to have a cross domain problem, which we have to resolve. On the gateway, we can apply policies to allow an inbound request to reach the internal API.

To show a practical example, we need to first review what is CORS and why it is important for security purposes. We can then talk about why we should use an API gateway and how to configure policies to protect an API.

What is CORS?

Cross-Origin Resource Sharing is a security feature supported by modern browsers to enable applications hosted on a particular domain to access resources hosted on different domains. In this case, the resource that needs to be shared is an API via a web request. The way a browser enforces this process is by creating a preflight request to the server before actually sending the request. This is the process to check if the client application is authorized to make the actual request.

When the app is ready to make a request to the API, the browser sends an OPTIONS request to the API, a preflight request. If the cross-origin server has the correct CORS policy, an HTTP 200 status is returned. This authorizes the browser to send the actual request with all the request information and headers.

For the cross-origin server to be configured properly, the policies need to include the client origin or domain, the web methods and headers that should be allowed. This is very important because it protects the API from possible exploits from unauthorized domains. It also controls the operation that can be called. For example, a GET request may be allowed, but a POST request may not be allowed. This level of configuration helps with the authorization of certain operations, like read only, at the domain level. Now that we understand the importance of CORS, let's look at how we can support this scenario using an API gateway.

What is Azure API Management?

The Azure API Management service is a reverse proxy gateway that manages the integration of APIs by providing the routing services, security and other infrastructure and governance resources. It provides the management of cross-cutting concerns like security policies, routing, document (XML, JSON) transformation, logging in addition to other features. An APIM can host multiple API definitions which are accessible via an API suffix or route information. Each API definition can have multiple operations or web methods. As an example, our service has a telemetry and audit API. Each of those APIs have two operations to GET and POST information.

• api.services.com/telemetry
  • GET, POST operations
• api.services.com/audit
  • GET or POST operations

For our purpose, we can use the security features of this gateway to enable the access of cross-origin client applications to internal APIs that are only accessible via the gateway. This can be done by adding policies at the API level or to each API operation. Let's take a look at what that looks like when we are using the Azure Portal.

We can add the policy for all the operations, or we can add it to each operation. Usually, when we can create an API, all the operations should have the same policies. For our case, we apply the policy at the API level, so all the operations are covered under the same policy. But exactly, what does this policy looks like? Let's review our policy and understand what is really doing.

For our integration to work properly, we need to configure the following information:

• Allow the app.ozkary.com domain to call the API by using the allowed-origins policy. This shows as the access-control-allow-origin header on the request response.
• Allow the OPTIONS, GET AND POST HTTP methods by using the allowed-methods policy. This shows as the access-control-allow-methods header on the request response.
• Allow the headers Content-Type and x-requested-by by using the allowed-headers policy. This shows as the access-control-allow-headers header on the request response.

Note: The request response can be viewed using the browser development tools, network tab.

This policy governs the cross-origin apps and what operations the client can use. As an example, if the client application attends to send a PUT or DELETE operation, it will be blocked because those methods are not defined in the CORS policy. It is also important to note, that we could use a wildcard character (*) for each policy, but this essentially indicates that any cross-origin app can make any operation call. Therefore, there is really no security, which is not a recommended approach. The use of wildcards should be done only during the development effort, but it should not be used in production environments.

After Adding the Policy CORS Does not Work

In some cases, even when the policy is configured correctly, we may notice that the policy is not taking effect, and the API request is not allowed. When this is the case, we should look at the policy configuration in all the levels. In Azure APIM, there are three levels of configuration:

• All APIs - One policy to all the API definitions
• All Operations - All the operations under one API definition
• Each Operation - One specific operation

We may think that the configuration at the operation level should take precedence, but this is not the case if there is a <base/> entry, as this indicates to use the parent configuration and apply it to this policy. To help prevent problems like this, make sure to review the high level configurations, and if necessary remove the <base/> entry at the operation level.

Conclusion

A CORS policy is a very important security configuration when dealing with applications and APIs that are hosted in different domains. As resources are shared across domain, it is important to clearly define what cross-origin clients can access the API. When using a cloud platform to protect an API, we can use an API gateway to help us manage the cross-cutting concerns like a CORS policy. This helps us minimizes risks and provides enterprise quality infrastructure.

Send question or comment at Twitter @ozkary

Originally published by ozkary.com

In a Single Page Applications architecture, APIs are used for the application integration with a backend service. When both the SPA and API...

Read Article →