From Passive Dashboards to Active Agents: Real-Time Reasoning over Data Streams

Overview

Dashboards are effective at showing us that something is breaking, but they usually rely on a human to watch the screen and decide what to do next. In this session, we will look at how to take a standard real-time telemetry dashboard and make it autonomous.

We will walk through a practical implementation using an Angular frontend and a Node.js server. We’ll look at how the system leverages a relational database for persistence and a Redis in-memory cache to handle high-frequency and short volume data feeds. From there, we incorporate an AI Agent that follows three core principles: perceiving the data stream through a sliding window, reasoning against statistical control limits, and acting by sending real-time analysis back to the user. This session is focused on bridging the gap between raw data streams and automated decision-making.

From Passive Dashboards to Active Agents: Real-Time Reasoning over Data Streams

🚀 Featured Open Source Projects

Explore these curated resources to level up your engineering skills. If you find them helpful, a ⭐️ is much appreciated!

🏗️ Data Engineering

Focus: Real-world ETL & MTA Turnstile Data

🤖 Artificial Intelligence

Focus: LLM Patterns and Agentic Workflows

📉 Machine Learning

Focus: Introduction to machine learning

💡 Contribute: Found a bug or have a suggestion? Open an issue! and be part of the open source project.

🔗 Review the repo used for this presentation:

Tech Stack

YouTube Video

👍 Subscribe to the channel to get notify on new events!

📅 Agenda

• The Real-Time Feed: Monitoring Device Telemetry

An introduction to the live system where devices emit high-frequency data for quality monitoring and 3-sigma control limit oversight.

• System Architecture: From Ingestion to Persistence

A deep dive into the technical stack, mapping out the data journey through Node.js, Redis in-memory caching, and relational database storage.

• The Human in the Loop: Cognitive Limitations

Exploring the “Passive Monitoring” challenge—why relying on human interpretation of real-time alerts creates a bottleneck in process control.

• The AI Agent: Applying the 3 Principles

Implementing the “Active Observer” using the three core pillars of AI Agents: Perception (the window), Reasoning (the limits), and Action (the analysis).

• The Intelligent Journey: Summary & Advantages

Recapping our transition from a passive monitor to a smart system and discussing the advantages of automated, agentic data interpretation.

⭐ Why Attend?

The industry is moving beyond simple “Chat” interfaces and into the realm of Agentic Observability. By attending this session, you will see a practical blueprint for integrating intelligence directly into a high-velocity data stack.

👥 Who Is This For?

• Junior Developers: Learn the fundamentals of real-time streaming, how to manage state in Node.js, and how to interact with AI APIs in a professional environment.
• Senior Engineers & Architects: See a robust architectural pattern for integrating Redis, relational databases, and AI Agents while maintaining system stability and security.
• Decision Makers (VPs/Directors): Understand the ROI of “Active Monitoring”—how AI Agents can act as a force multiplier for your engineering teams by automating the first layer of data interpretation.
• Quality & Reliability Engineers: Explore how to digitize the control limit logic you already use into an autonomous 24/7 “Digital Twin.”

Presentation

The Real-Time Feed Challenge

High-velocity telemetry defines the “digital pulse” of modern industrial systems.

• Telemetry Streams: Industrial devices emit continuous telemetry (Temperature, Sound, Humidity). These data points require sub-second processing to maintain operational stability.
• Control Limits: Quality engineers rely on statistical boundaries to define “normal” operation. Detecting drift is the critical first step in identifying risks before failure occurs.

Scalable System Architecture

A multi-layered stack designed to bridge the gap between ingestion and intelligence.

• Redis Cache: Manages the “Live State” or Digital Twin. Provides sub-millisecond access for immediate AI perception.
• Relational Data Warehouse: Powers the historical persistence layer for long-term trend analysis and compliance auditing.
• Node Controller: The orchestration hub. Manages telemetry ingestion, state updates, and the execution of the agentic reasoning loop.

The “Human in the Loop” Trap

Passive observability relies on human interpretation, creating a critical bottleneck.

• Cognitive Overload: Humans struggle to interpret hundreds of concurrent streams, missing subtle patterns.
• Alert Fatigue: Constant threshold violations desensitize responders, causing critical 3σ violations to be ignored.
• Response Latency: The time required for a human to interpret a dashboard often exceeds the window for effective corrective action.

The 3 Principles of AI Agents

Automating observability requires a system that can perceive, reason, and act independently.

1. Perception: Maintaining a stateful sliding window of the Digital Twin to understand temporal context.
2. Reasoning: Evaluating the stream against statistical 3σ limits and physical engineering constraints.
3. Action: Closing the loop by emitting real-time narratives or triggering autonomous safety protocols.

The Intelligent Journey: Comparison

Feature	Passive Monitor	Smart Monitor
Interpretation	Requires human "eyes-on-glass"	Autonomous, semantic interpretation
Response	Reactive to simple thresholds	Proactive identification of drift
Reliability	High risk of missed signals	Intelligent filtering of noise
Data Context	Disconnected historical vs. live	Stateful perception of "Live Device"
Intervention	Significant human latency	Automated safety & audit narratives

🌟 Let’s Connect & Build Together

Thanks for reading! 😊 If you enjoyed these resources, let’s stay in touch! I share deep-dives into AI/ML patterns and host community events here:

• GDG Broward: Join our local dev community for meetups and workshops.
• Global AI Events: Join Global AI Events.
• LinkedIn: Let’s connect professionally! I share insights on engineering.
• GitHub: Follow my open-source journey and star the repos you find useful.
• YouTube: Watch step-by-step tutorials on the projects listed above.
• BlueSky / X / Twitter: Daily tech updates and quick engineering tips.

👉 Originally published at ozkary.com

Overview Dashboards are effective at showing us that something is breaking, but they usually rely on a human to watch the screen and decide...

Read Article →

Live Dashboards: Boosting App Performance with Real-Time Integration

Overview

Dive into the future of web applications. We're moving beyond traditional API polling and embracing real-time integration. Imagine your client app maintaining a persistent connection with the server, enabling bidirectional communication and live data streaming. We'll also tackle scalability challenges and integrate Redis as our in-memory data solution.

• Follow this GitHub repo during the presentation: (Give it a star)

👉 https://github.com/ozkary/Realtime-Apps-with-Nodejs-Angular-Socketio-Redis

YouTube Video

Video Agenda

This presentation explores strategies for building highly responsive and interactive live dashboards. We'll delve into the challenges of traditional API polling and demonstrate how to leverage Node.js, Angular, Socket.IO, and Redis to achieve real-time updates and a seamless user experience.

• Introduction:

  • Understanding telemetry data and the importance to monitor the data
  • Challenges of traditional API polling for real-time data.
  • Design patterns to enhance an app with minimum changes

• Traditional Solution Architecture

  • SQL Database Integration.
  • Restful API
  • Angular and Node.js Integration

• Real-Time Integration with Web Sockets

  • Database Optimization Challenges
  • Introduction to Web Sockets for bidirectional communication.
  • Implementing Web Sockets in a Web application.
  • Handling data synchronization and consistency.

• Distributed Caching with Redis:

  • Benefits of in-memory caching for improving performance and scalability.
  • Integrating Redis into your Node.js application.
  • Caching strategies for distributed systems.

• Case Study: Building a Live Telemetry Dashboard

  • Step-by-step demonstration of the implementation.
  • Performance comparison with and without optimization techniques.
  • User experience benefits of real-time updates.

• Benefits and Considerations

  • Improved dashboard performance and responsiveness.
  • Reduced server load and costs.
  • Scalability and scalability considerations.
  • Best practices for implementing real-time updates.

Why Attend:

Gain a deep understanding of real-time data integration for your Web application.

Presentation

Telemetry Data Story

Devices send telemetry data via API integration with SQL Server. There are inherit performance problems with a disk-based database. We progressively enhance the system with minimum changes by adding real-time integration and an in-memory cache system.

Database Integration

Solution Architecture

• Disk-based Storage
• Web apps and APIs query database to get the data
• Applications can do both high reads and writes
• Web components, charts polling back-end database for reads

Let’s Start our Journey

• Review our API integration and talk about concerns
• Do not refactor everything
• Enhance to real-time integration with sockets
• Add Redis as the distributed cache
• Add the service broker strategy to sync the data sources
• Centralized the real-time integration with Redis

RESTful API Integration

Applied Technologies

• REST API Written with Node.js
• TypeORM Library Repository
• Angular Client Application with Plotly.js Charts
• Disk-based storage – SQL Server
• API Telemetry (GET, POST) route

Use Case

• IoT devices report telemetry information via API
• Dashboard reads that most recent data only via API calls which queries the storage service
• Polling the database to get new records

Project Repo (Star the project and follow) https://github.com/ozkary/Realtime-Apps-with-Nodejs-Angular-Socketio-Redis

Database Optimization and Challenges

Slow Queries on disk-based storage

• Effort on index optimization
• Database Partition strategies
• Double-digit millisecond average speed (physics on data disks)

Simplify data access strategies

• Relational data is not optimal for high data read systems (joins?)
• Structure needs to be de-normalized
• Often views are created to shape the data, date range limit

Database Contention

• Read isolation levels (nolock)
• Reads competing with inserts

Cost to Scale

• Vertical and horizontal scaling up on resources
• Database read-replicas to separate reads and writes
• Replication workloads/tasks
• Data lakes and data warehouse

Real-Time Integration

What is Socket.io, Web Sockets?

• Enables real-time bidirectional communication.
• Push data to clients as events take place on the server
• Data streaming
• Connection starts as HTTP is them promoted to Web Sockets

Additional Technologies -Socket.io (Signalr for .Net) for both client and server components

Use Case

• IoT devices report telemetry information via sockets. All subscribed clients get the information as an event which updates the dashboard

Demo

• Update both server and client to support Web sockets
• Use device demo tool to connect and automate the telemetry data to the server

Distributed Cache Strategy

Why Use a Cache?

• Data is stored in-memory
• Sub-millisecond average speed
• Cache-Aside Pattern
  • Read from cache first (cache-hit) fail over to database (cache miss)
  • Update cache on cache miss
• Write-Through
  • Write to cache and database
  • Maintain both systems updated
• Improves app performance
• Reduces load on Database

Application Changes

• Changes are only done on the server
• No changes on client-side

Redis and Socket.io Integration

What is Redis?

• Key-value store, keys can contain strings (JSON), hashes, lists, sets, & sorted sets
• Redis supports a set of atomic operations on these data types (available until committed)
• Other features include transactions, publish/subscribe, limited time to live -TTL
• You can use Redis from most of today's programming languages using libraries

Use Case

• As application load and data frequency increases, we need to use a cache for performance. We also need to centralize the events, so all the socket servers behind a load balancer can notify the clients. Update both storage and cache

Demo

• Start Redis-cli on Ubuntu and show some inserts, reads and sync events.
  • sudo service redis-server restart
  • redis-cli -c -p 6379 -h localhost
  • zadd table:data 100 "{data:'100'}“
  • zrangebycore table:data 100 200
  • subscribe telemetry:data

Summary: Boosting Your App Performance

When your application starts to slow down due to heavy read and writes on your database, consider moving the read operations to a cache solution and broadcasting the data to your application via a real-time integration using Web Sockets. This approach can significantly enhance performance and user experience.

Key Benefits

• Improved Performance: Offloading reads to a cache system like Redis reduces load on the database.
• Real-Time Updates: Using Web Sockets ensures that your application receives updates in real-time, with no need for manual refreshes.
• Scalability: By reducing the database load, your application can handle more concurrent users.
• Efficient Resource Utilization: Leveraging caching and real-time technologies optimizes the user of server resources, leading to savings and better performance.

We've covered a lot today, but this is just the beginning!

If you're interested in learning more about building cloud data pipelines, I encourage you to check out my book, 'Data Engineering Process Fundamentals,' part of the Data Engineering Process Fundamentals series. It provides in-depth explanations, code samples, and practical exercises to help in your learning.

Thanks for reading.

Send question or comment at Twitter @ozkary 👍 Originally published by ozkary.com

Overview Dive into the future of web applications. We're moving beyond traditional API polling and embracing real-time integration. Ima...

Read Article →

Improve App Performance with In-Memory Cache and Real-Time Integration

In the presentation, we discuss some of the performance problems that exists when using an API to SQL Server integration on a high transaction systems with thousands of concurrent clients and several client tools that are used for statistical analysis.

Telemetry Data Story

Devices send telemetry data via API integration with SQL Server. These devices can send thousands of transactions every minute.  There are inherit performance problems with a disk-based database when there are lots of writes and reads on the same table of a database. 

To manage the performance issues, we start by moving away from a polling system into a real-time integration using Web Sockets. This enables the client application to receive events on a bidirectional channel, which in turns removes the need to have to poll the APIs at a certain frequency.

To continue to enhance the system, we introduce the concept of an enterprise in-memory cache, Redis. The in-memory cache can be used to separate the reads and writes operations from the storage engine. 

At the end, we take a look at a Web farm environment with a load balancer, and we discuss the need to centralize the socket messages using Redis Publish and Subscribe feature. This enables all client with a live connection to be notified of the changes in real-time.

Database Optimization and Challenges

Slow Queries  on disk-based storage

• Effort on index optimization
• Database Partition strategies
• Double-digit millisecond  average speed (physics on data disks)

Simplify data access strategies

• Relational data is not optimal for high data read systems (joins?)
• Structure needs to be de-normalized
• Often views are created to shape the data, date range limit

Database Contention

• Read isolation levels (nolock)
• Reads competing with inserts

Cost to Scale

• Vertical and horizontal scaling up on resources
• Database read-replicas to separate reads and writes
• Replication workloads/tasks
• Data lakes and data warehouse

What is Socket.io, WebSockets?

Enables real-time bidirectional communication.

Push data to clients as events take place on the server

Data streaming

Connection starts as HTTP is them promoted to WebSockets 

Why Use a Cache?

• Data is stored in-memory
• Sub-millisecond average speed
• Cache-Aside Pattern
• Read from cache first (cache-hit) fail over to database (cache miss)
• Update cache on cache miss
• Write-Through
• Write to cache and database
•  Maintain both systems updated
• Improves app performance
• Reduces load on Database

What is Redis?

• Key-value store, keys can contain strings (JSON), hashes, lists, sets, & sorted sets
• Redis supports a set of atomic operations on these data types (available until commited)
• Other features include transactions, publish/subscribe, limited time to live -TTL 
• You can use Redis from most of today's programming languages (Libs)

Code Repo

https://github.com/ozkary/Realtime-Apps-with-Nodejs-Angular-Socketio-Redis

Send question or comment at Twitter @ozkary

Originally published by ozkary.com

In the presentation, we discuss some of the performance problems that exists when using an API to SQL Server integration on a high transacti...

Read Article →

Web Development with Nodejs NPM NVM

When building web applications using JavaScript frameworks like React or Angular, there are tools that enable the installation of these frameworks and aid in the development process. These tools are essential to be able to build, test and manage some dependencies in our projects. It is those package dependencies that we continuously need to keep up with the latest update. To update those packages, we use the NPM CLI tool, which runs on the NodeJS runtime environment.

When we need to update a package, we may face issues that a package or a new version of that package is not supported (see error below) for the current version of Node.js and that we need to update to a new version. In this article, we discuss the tools that are used to manage the software development process and how to best update NodeJS using the command line interface (CLI).

npm WARN notsup Unsupported engine for create-react-app@5.0.0: wanted: {"node":">=14"} (current: {"node":"12.16.1","npm":"6.14.4"}) npm WARN notsup Not compatible with your version of node/npm: create-react-app@5.0.0

This error message indicates that a particular required version of Node.js is not in the system and node version 14 is a dependency.

What is Node.js?

Node.js is a cross-platform JavaScript runtime environment. It is used by software engineers to build server-side and client-side web applications.  When building client applications with popular frame works like React, Angular and others, Node.JS provides the runtime environment for the JavaScript applications to run. It also enables the build and test tools that are used during the implementation effort of those applications.

JavaScript's applications are made of several libraries or packages that can be added to the project. Those libraries are mostly refereed as packages, and to install them, developers use a CLI tool to install and configure them.

What is NPM?

Node Package Manager (NPM) is a tool that runs on the NodeJS runtime environment. It comes with the installation of Node.js. Its purpose is to download and install packages for a particular project. Those packages and respective versions are tracked on a JSON file on the root folder of the project. With NPM, we can also install other CLI tools that can be specific for scaffolding startup codebase for a particular JavaScript framework. Some examples include, but not limited to, are: yeoman, create-react-app, angular CLI.

NPM has many commands, but the install command is the most basic and most important one, as this is the one that enables us to install and update packages. Let’s look at some of these commands:

Command	Description
$ npm install package-name  –save	Installs a package latest version and saves the reference the package.json file
$ npm install package-name	Installs a package but does not store any reference information
$ npm update package-name	Updates a package with a new release. NPM decides what version to select
$ npm install package-name@latest	To have better control on what version to install, we can provide the version number or latest release flag right after the package name, separated by the @ character
$ npm install -h	Shows help information on running the install command
$ npm run script-name	Runs a script command defined on the package.json for build, test, starting the project
$ npm install -g npm@next	This command is used to install the next version of NPM. The -g flag should be used to install this globally in the system

 

What is package JSON?

Package.json is a metadata file which host all the project related information like project name, licensing, authors, hosting, location and most importantly information to track project dependencies and scripts to run.

When installing NPM packages to a project, the information is saved on a file at the root of the project, package.json. This file maintains project information and all the package dependencies.  By tracking the package dependencies, a development environment can be easily created it. The developers only need to clone the repo or codebase and use NPM to download all the dependencies by typing the following command from the root folder of the project:

$ npm install

 

*Note:  package.json must exist in the same folder location where this command is typed

The script area of the package.json file provide commands that can be used to create production quality builds, test plan execution, coding standard validation and running the application. These are essential command for the day-to-day development operations and integration with CICD tools like GitHub Actions or Jenkins.

Keep Node.js Updated

With a better understanding of Node.js and the purpose of NPM for the development process, we can now discuss how to deal with situations when NPM fails to install a package because our Node.js installation is behind a few versions, and we need to upgrade it.

What is NVM?

The best way to update Node.js is by using another CLI tool, Node Version Manager (NVM). This tool enables us to manage multiple versions of Node.js in our development workspace. It is not a required tool, but it is useful because it enables us to upgrade and test the application to the latest releases, which can help us identify compatibility issues with the new runtime or NPM packages. It also enables us to downgrade to previous version to help us verify when a feature started to break.

To install NVM on Linux, we can run the following command:

$ curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.34.0/install.sh | bash

Once the tool is installed, we can run a few commands to check the current Node.js version, install a new one and switch between versions. Let us review those commands:

Command	Description
$ nvm version	Shows the selected node version
$ nvm –version	Shows the nvm cli version
$ nvm ls	Lists all the Node.js versions installed
$ nvm use version-number	Selects a Node.js version to use
$ nvm install version-number	Installs a Node.js version

 

To install a new version of Node.js, we can use the install command. This downloads and install a new version. After the version is installed, the environment should default to the new version of Node. If the environment was encountering the unsupported Node.js version error, we can run the NPM command that failed again, and since the new version is installed, it should be able to install the new package.

Conclusion

When working with JavaScript frameworks like React or Angular, the Node.js runtime environment must be installed and kept with the latest version.  When new NPM packages need to be installed on our projects, we need to make sure that they are compatible with the current version of the runtime environment, Node.js. If this is not the case, the NPM package fails to install, and we need to update the runtime with the next or latest version. Using tools like NPM and NVM, we can manage the different versions of packages and runtime respectively.  Understanding the purpose of these tools and how to use them is part of the web development process, and it should help us keep the development environment up to date.

Have you used these CLI tools before? Do you like to use visual tools instead?

Send question or comment at Twitter @ozkary

Originally published by ozkary.com

When building web applications using JavaScript frameworks like React or Angular, there are tools that enable the installation of these fra...

Read Article →

Nodejs CORS Handling No Access-Control-Allow-Origin

When accessing API calls from another domain or port number, we have to be mindful of the possible CORS policy restrictions.  CORS stands for Cross-Origin Resource Sharing. This is used to prevent access from one domain to another without having the policy to enable that access.  A common use case is one an application hosted on one domain tries to access the API hosted on a remote domain. When the calling domain is not whitelisted for access, an error is raised and the call is denied.

Access Error

To illustrate this problem, we can look at an app hosted on our localhost but with a different port number. We should notice that port number makes the policy applicable even when the domain is the same.

Access to XMLHttpRequest at 'http://localhost:5000/api/data' from origin 'http://localhost:8080' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.

When making a CORS request to a different domain, there would be two requests made to the server, the preflight and actual requests. For each of these requests, the server must respond with the Access-Control-Allow-Origin header set with the name of the domain of origin (calling app) or a wildcard ‘*’ to allow all domains. Wild card are a bit too open, so this is typically not used for secured apps.

Preflight Request

A preflight or OPTIONS (HTTP verb) request is created by the browser before the actual request (PUT, POST) is sent for a resource in different domains. The goal is to have the browser and server validate that the other domain has access to that particular resource. This is done by setting the Access-Control-Allow-Origin header with the client host domain.

Actual Request

Once the preflight request has a response with the corresponding headers and HTTP 200 status, the browser sends the actual request. For this request, the server also checks the CORS policies and adds the Access-Control-Allow-Origin header with the client host domain.

NodeJS Middleware

Now that we understand about this security concern, we can take a look at how we can build a middleware with NodeJS to help us manage this integration.

Let’s start by defining a common NodeJS Express application. When we build the bootstrap code for the server, we can include a middleware module (auth-cors) which is our implementation to handle the CORS concerns. This is done by loading the module using the require directive and loading the module script relative to where the loading script is found. Let’s review the code to see how we can do that.

const express = require("express"); // middleware for cors const cors = require("./middleware/auth-cors"); const PORT = process.env.port || config.port; const app = express(); // initialize modules app.use(bodyParser.json()); app.use(   bodyParser.urlencoded({     extended: true   }) ); // add the cors middleware to the http pipeline app.use(cors); // start the server listener app.listen(PORT, () => console.log(`Listening on ${PORT}`));

After we load the CORS module, we can associate it with the application by calling app.use(cors). This is a way to tell the server application to pass the context references to each module in the HTTP  pipeline.

Now that we understand how to include the middleware module, we can take a look at the actual implementation below. We start by exporting the module definition and implementing the RequestHandler interface contract which enables the module to receive the HTTP pipeline context with the request, response, and next references. The next context enables the code to continue the pipeline request execution to complete. This is the context that all middleware modules should return to chain the other middleware modules in the pipeline.

module.exports = (req, res, next) => {   if (req.headers.origin) {     // add the remote domain or only the environment     // listed on the configuration     res.setHeader(       "Access-Control-Allow-Origin",       process.env.domain || req.headers.origin     );   }   // Request methods you wish to allow   // remove the methods you wish to block   res.setHeader(     "Access-Control-Allow-Methods",     "GET, POST, OPTIONS, PUT, PATCH, DELETE"   );   // Request headers you wish to allow   res.setHeader(     "Access-Control-Allow-Headers",     "Origin,Authorization,X-Requested-With,content-type,Accept"   );   // Set to true if you need the website to include cookies in the requests sent   // to the API (e.g. in case you use sessions)   res.setHeader("Access-Control-Allow-Credentials", true);   if ("OPTIONS" === req.method) {     return res.sendStatus(200);   }   return next(); // Pass to next layer of middleware };

What is the module really doing?

The module basically intercepts the request from the HTTP pipeline to add the headers that can enable a client application to send a request from another domain. 

The Access-Control-Allow-Origin header is added to the response header to include the remote domain. This is the area where we can whitelist some domains and not allow others. In this example, we are just adding the remote domain which should not be the normal case. The same approach is taken to Allow Methods and Headers that our application supports. For example, to block the Delete method, we do not add that value in the header. 

The other interesting header to include or exclude based on your application requirements is the  Access-Control-Allow-Credentials header. This is used for cases when you want the client application to send back credential information in the header. This is useful for session management state and authorization token information.

Summary

It is important to understand that we run an application and APIs on different domains, we have some security concerns that we need to plan for. This can enable our apps to safely use remote APIs and prevent malicious attacks to our servers. With NodeJS, building middleware components to handle this concern is fairly simple. Most of the understanding is around how the HTTP protocol works to enable us to handle this integration. 

Please let me know your feedback, and  I will add clarification to the article.

Thanks for reading

🖐 .NET Web API Implementation for CORS

Originally published by ozkary.com

When accessing API calls from another domain or port number, we have to be mindful of the possible CORS policy restrictions.  CORS stands...

Read Article →