kade.im
NextJS All-In-One AI streaming App for Edge Device

NextJS All-In-One AI streaming App for Edge Device

표시
Project Years
2024
Tags
AICHIP
NPU
VisionAI
Skills
k8s
Docker
Typescript
NextJS
Node
GStreamer

Introduce

  • All in one (Client and Server integrated one container) app for edge streaming with NPU
notion image
 

Specific features

  • Used unofficial Gstreamer API for Node Js
  • Client and Server is integrated, Edge computing effective
 

Results

notion image
 
 

Struggles

Issues Arising During Pipeline and Frame Transmission
  • Handling Image Frame: Various formats have been tested, such as RGB raw and Uint8Array, but mismatched image dimensions can result in corrupted outputs like the ones below.
    • Python Solution: These issues can be efficiently handled using OpenCV.
    • JavaScript Challenges: Due to a lack of suitable libraries, the image format used in transmission had to be altered to ensure compatibility.
      • Transformation Process:base64 buffer → Uint8Array (R, G, B) → JPEG → **Blob (video/webm; codecs=vp8)**
notion image
  • Common Problems:
    • Mismatched resolutions.
    • Using incorrect channel formats, such as RGB with 4 channels.
 
 

Key Learnings and Insights

 
This project provided valuable insights into Edge AI Computing, particularly in integrating client and server into a single containerized application for real-time video processing.
  1. Exploring the Feasibility of Edge AI Computing
      • By integrating the server and frontend into one container, I measured the performance impact and evaluated the feasibility of Edge AI Computing.
      • This approach reduced network latency and improved real-time inference performance, making it a viable solution for low-latency applications.
  1. Optimizing AI Processing for Streaming Workflows
      • Unlike traditional AI video processing, which involves handling static images or pre-recorded videos, I discovered the efficiency and importance of streaming libraries.
      • Using GStreamer with Node.js allowed real-time AI inference on streaming data, demonstrating its advantages over batch processing.
  1. Challenges in Image Frame Transmission and Format Compatibility
      • Learned how different image formats (RGB raw, Uint8Array, JPEG, Blob) affect transmission efficiency and output quality.
      • Addressed format mismatches by designing a custom transformation pipeline to ensure proper frame processing.
      • Recognized that JavaScript(Typescript) lacks robust image handling libraries compared to Python (OpenCV), requiring additional transformations for reliable video transmission.
This project deepened my understanding of real-time AI streaming, pipeline optimization, and efficient Edge AI deployment, paving the way for further advancements in low-latency, high-performance AI applications.