TensorFlow.js: Architecture and applications

In a paper published last month, Google developers explained the design, API, and implementation of TensorFlow.js, the JavaScript implementation of TensorFlow. TensorFlow.js was first introduced at the TensorFlow Dev Summit 2018. It is basically the successor of deeplearn.js, which was released in August 2017, and is now named as TensorFlow.js Core.

Google’s motivation behind creating TensorFlow.js was to bring machine learning in the hands of web developers who generally do not have much experience with machine learning. It also aims at allowing experienced ML users and teaching enthusiasts to easily migrate their work to JS.

The TensorFlow.js architecture

TensorFlow.js, as the name suggests, is based on TensorFlow, with a few exceptions specific to the JS environment. This library comes with the following two sets of APIs:

• The Ops API facilitates lower-level linear algebra operations such as matrix, multiplication, tensor addition, and so on.
• The Layers API, similar to the Keras API, provide developers high-level model building blocks and best practices with emphasis on neural networks.

Source: TensorFlow.js

TensorFlow.js backends

In order to support device-specific kernel implementations, TensorFlow.js has a concept of backends. Currently it supports three backends: the browser, WebGL, and Node.js. The two new rising web standards, WebAssembly and WebGPU, will also be supported as a backend by TensorFlow.js in the future.

To utilize the GPU for fast parallelized computations, TensorFlow.js relies on WebGL, a cross-platform web standard that provides low-level 3D graphics APIs. Among the three TensorFlow.js backends, the WebGL backend has the highest complexity.

With the introduction of Node.js and event-driven programming, the use of JS in server-side applications has grown over time. Server-side JS has full access to the filesystem, native operating system kernel, and existing C and C++ libraries. In order to support the server-side use cases of machine learning in JavaScript, TensorFlow.js comes with a Node.js backend that binds to the official TensorFlow C API using the N-API.

As a fallback, TensorFlow.js provides a slower CPU implementation in plain JS. This fallback can run in any execution environment and is automatically used when the environment has no access to WebGL or the TensorFlow binary.

Current applications of TensorFlow.js

Since its launch, TensorFlow.js have seen its applications in various domains. Here are some of the interesting examples the paper lists:

Gestural Interfaces

TensorFlow.js is being used in applications that take gestural inputs with the help of webcam. Developers are using this library to build applications that translate sign language to speech translation, enable individuals with limited motor ability control a web browser with their face, and perform real-time facial recognition and pose-detection.

Research dissemination

The library has facilitated ML researchers to make their algorithms more accessible to others. For instance, the Magenta.js library, developed by the Magenta team, provides in-browser access to generative music models. Porting to the web with TensorFlow.js has increased the visibility of their work with their audience, namely musicians.

Desktop and production applications

In addition to web development, JavaScript has been used to develop desktop and production applications. Node Clinic, an open source performance profiling tool, recently integrated a TensorFlow.js model to separate CPU usage spikes by the user from those caused by Node.js internals.

Another example is, Mood.gg Desktop, which is a desktop application powered by Electron, a popular JavaScript framework for writing cross-platform desktop apps. With the help of TensorFlow.js, Mood.gg detects which character the user is playing in the game called Overwatch, by looking at the user’s screen. It then plays a custom soundtrack from a music streaming site that matches with the playing style of the in-game character.

Read the paper, Tensorflow.js: Machine Learning for the Web and Beyond, for more details.

TensorFlow.js 0.11.1 releases!

Emoji Scavenger Hunt showcases TensorFlow.js

16 JavaScript frameworks developers should learn in 2019