Loading Project...

Introduce

• All in one (Client and Server integrated one container) app for edge streaming with NPU

Specific features

• Used unofficial Gstreamer API for Node Js
• Client and Server is integrated, Edge computing effective

Results

Struggles

Issues Arising During Pipeline and Frame Transmission

• Handling Image Frame: Various formats have been tested, such as RGB raw and Uint8Array, but mismatched image dimensions can result in corrupted outputs like the ones below.
  • Python Solution: These issues can be efficiently handled using OpenCV.
  • JavaScript Challenges: Due to a lack of suitable libraries, the image format used in transmission had to be altered to ensure compatibility.
    • Transformation Process:base64 buffer → Uint8Array (R, G, B) → JPEG → **Blob (video/webm; codecs=vp8)**

• Common Problems:
  • Mismatched resolutions.
  • Using incorrect channel formats, such as RGB with 4 channels.

Key Learnings and Insights

This project provided valuable insights into Edge AI Computing, particularly in integrating client and server into a single containerized application for real-time video processing.

1. Exploring the Feasibility of Edge AI Computing
   • By integrating the server and frontend into one container, I measured the performance impact and evaluated the feasibility of Edge AI Computing.
   • This approach reduced network latency and improved real-time inference performance, making it a viable solution for low-latency applications.
2. Optimizing AI Processing for Streaming Workflows
   • Unlike traditional AI video processing, which involves handling static images or pre-recorded videos, I discovered the efficiency and importance of streaming libraries.
   • Using GStreamer with Node.js allowed real-time AI inference on streaming data, demonstrating its advantages over batch processing.
3. Challenges in Image Frame Transmission and Format Compatibility
   • Learned how different image formats (RGB raw, Uint8Array, JPEG, Blob) affect transmission efficiency and output quality.
   • Addressed format mismatches by designing a custom transformation pipeline to ensure proper frame processing.
   • Recognized that JavaScript(Typescript) lacks robust image handling libraries compared to Python (OpenCV), requiring additional transformations for reliable video transmission.

This project deepened my understanding of real-time AI streaming, pipeline optimization, and efficient Edge AI deployment, paving the way for further advancements in low-latency, high-performance AI applications.