Google recently presented MediaPipe graphs for browsers, enabled by WebAssembly and accelerated by the XNNPack ML Inference Library. As previously demonstrated on mobile (Android, iOS), MediaPipe graphs allow developers to build and run machine-learning (ML) pipelines, to achieve complex tasks.

MediaPipe graphs are visualized in a browser environment with MediaPipe Visualizer, a dedicated web application that allows developers to build a graph consisting of nodes representing different machine learning or other processing tasks. The following figure corresponds to the running of the MediaPipe face detection example in the visualizer.

As is apparent from the graph, the face detection application transforms input frames (input_frames_gpu) into output frames (output_frames_gpu) through a series of transformations including the conversion of incoming frames into image tensors (TfLiteTensor), posterior processing by a TFLite model for face detection, and overlaying of annotations on the output video.

The visualized graph matches the facing text which contains a description of the nodes’ content and the expected processing to realize. The MediaPipe Visualizer will react in real-time to changes made within the editor in order to maintain the correspondence between text and graph. The configuration of the previous TFLite model is for instance as follows:

# Converts the transformed input image on GPU into an image tensor stored as a # TfLiteTensor. node { calculator: “TfLiteConverterCalculator” input_stream: “IMAGE:transformed_input_video_cpu” output_stream: “TENSORS:image_tensor”

Google created four demos (Edge Detection, Face Detection, Hair Segmentation, Hand Tracking) to be run in the browser.

The browser-enabled version of MediaPipe graphs is implemented by compiling the C++ source code to WebAssembly using Emscripten, and creating an API for all necessary communications back and forth between JavaScript and C++. Required demo assets (ML models and auxiliary text/data files) are packaged as individual binary data packages, to be loaded at runtime.

To optimize for performance, MediaPipe’s browser version leverages the GPU for image operations whenever possible, and resort to the lightest (yet accurate) available ML models. The XNNPack ML Inference Library is additionally used in connection with the TensorflowLite inference calculator (TfLiteInferenceCalculator), resulting in an estimated 2-3x speed gain in most of applications.

Google plans to improve MediaPipe’s browser version and give developers more control over template graphs and assets used in the MediaPipe model files. Developers are invited to follow the Google Developer twitter account.

MediaPipe is a cross-platform framework for mobile devices, workstations and servers, and supports GPU acceleration. MediaPipe is available under the MIT open source license. Contributions and feedback are welcome and may be provided via the GitHub project.

The post Google’s MediaPipe Machine Learning Framework Web-Enabled with WebAssembly appeared first on Artificial Intelligence.

Roughly a year ago, Google open-sourced MediaPipe, a framework for building cross-platform AI pipelines consisting of fast inference and media processing (like video decoding). Basically, it’s a quick and dirty way to perform object detection, face detection, hand tracking, multi-hand tracking, hair segmentation, and other such tasks in a modular fashion, with popular machine learning frameworks like Google’s own TensorFlow and TensorFlow Lite.

MediaPipe could previously be deployed to desktop, mobile devices running Android and iOS, and edge devices like Google’s Coral hardware family, but it’s increasingly making its way to the web courtesy WebAssembly, a portable binary code format for executable programs, and XNNPack ML Inference Library, an optimized collection of floating-point AI inference operators. On the graphics and rendering side, MediaPipe now automatically taps directly into WebGL, a JavaScript API for rendering interactive 2D and 3D graphics within any compatible web browser, such that the browser creates a virtual machine at runtime that executes instructions very quickly.

An API facilitates communications between JavaScript and C++, allowing users to change and interact with MediaPipe graphs directly using JavaScript. And all the requisite demo assets, including AI models and auxiliary text and data files, are packaged as individual binary data packages to be loaded at runtime.

“Since everything runs directly in the browser, video never leaves the user’s computer and each iteration can be immediately tested on a live webcam stream (and soon, arbitrary video),” explained MediaPipe team members Michael Hays and Tyler Mullen in a blog post.

Google leveraged the above-listed components to integrate preview functionality into a web-based visualizer — a sort of workspace for iterating over MediaPipe flow designs. The visualizer, which is hosted at viz.mediapipe.dev, enables developers to inspect MediaPipe graphs (frameworks for building machine learning pipelines) by pasting a graph code into the editor tab or uploading a file to the visualizer. Users can pan around and zoom into the graphical representation using a mouse and scroll wheel, and the visualization reacts to changes made within the editor in real time.

Hays and Mullen note that currently, web-based MediaPipe support is limited to the demo graphs supplied by Google. Developers must edit one of the template graphs — they can’t provide their own from scratch or add or alter assets. TensorFlow Lite inference isn’t supported, and the graph’s computations must be run on a single processor thread.

A lack of compute shaders — routines compiled for high-throughput accelerators — available for the web is to blame for this last limitation, which Hays, Mullen, and team attempted to work around by using graphic cards for image operations where possible and the lightest-weight possible versions of all AI models. They plan to “continue to build upon this new platform” and to provide developers with “much more control” over time.

The post Google brings cross-platform AI pipeline framework MediaPipe to the web appeared first on Artificial Intelligence.