Being able to deploy machine learning applications at the edge is the key to unlocking a multibillion-dollar market. TinyML is the art and science of producing machine-learning models frugal enough to work at the edge, and it’s seeing rapid growth.

Is it $61 billion and 38.4% compound annual growth rate (CAGR) by 2028 or $43 billion and 37.4% CAGR by 2027? Depends on which report outlining the growth of edge computing you choose to go by, but in the end it is not that different.

What matters is that edge computing is booming. There is growing interest by vendors, and ample coverage, for good reason. Although the definition of what constitutes edge computing is a bit fuzzy, the idea is simple. It is about taking compute out of the data center and bringing it as close to where the action is as possible.

Whether it’s stand-alone Internet-of-things sensors, devices of all kinds, drones, or autonomous vehicles, there’s one thing in common. Increasingly, data generated at the edge are used to feed applications powered by machine learning models. There’s just one problem: machine learning models were never designed to be deployed at the edge. Not until now, at least. Enter TinyML.

Tiny machine learning (TinyML) is broadly defined as a fast-growing field of machine-learning technologies and applications including hardware, algorithms, and software capable of performing on-device sensor data analytics at extremely low power, typically in the mW range and below, hence enabling a variety of always-on use-cases and targeting battery-operated devices.

This week, the inaugural TinyML EMEA Technical Forum is taking place, and it was a good opportunity to discuss with some key people in this domain. ZDNet caught up with Evgeni Gousev from Qualcomm, Blair Newman from Neuton, and Pete Warden from Google.

Hey Google
Pete Warden wrote the world’s only mustache-detection image-processing algorithm. He also was the founder and chief technology officer of startup Jetpac. He raised a Series A from Khosla Ventures, built a technical team, and created a unique data product that analyzed the pixel data of more than 140 million photos from Instagram and turned them into in-depth guides for more than 5,000 cities around the world.

Jetpac was acquired by Google in 2014, and Warden has been a Google Staff Research Engineer since. Back then, Warden was feeling pretty good about himself for being able to fit machine-learning models in 2 megabytes.

That was until he found some of his new Google colleagues had a 13 kilobyte model that they were using to recognize wake words running on always-on digital signal processor
on Android devices. That way the main CPU wasn’t burning battery listening out for “that” wake word—Hey Google.

“That really blew my mind, the fact that you could do something actually really useful in that smaller model. And it really got me thinking about all of the other applications that might be possible if we can run especially all these new machine-learning, deep-learning approaches” Warden said.

Although Warden is oftentimes credited by his peers as having kickstarted the TinyML subdomain of machine learning, he is quite modest about it. Much of what he did, he acknowledges, was based off things others were already working on: “A lot of my contribution has been helping publicize and document a bunch of these engineering practices that have emerged,” he said.

The post Machine Learning at the Edge: TinyML Is Getting Big appeared first on Artificial Intelligence.

Online learning platform EdX; Google’s open-source machine learning platform, TensorFlow; and HarvardX have put together a certification program to train tech professionals to work with tiny machine learning (TinyML). The program is meant to support this specialized segment of development that can include edge computing with smart devices, wildlife tracking, and other sensors. The program comprises a series of courses that can be completed at home.

The idea is to scale machine learning to function in small form, edge devices that use far less power than desktop computers and have limited storage and processing capacity, says Anant Agarwal, CEO of EdX, which was founded by MIT and Harvard. That can include devices that operate on batteries, such as remote sensors, microphones, and cameras set up in the wilderness.

Agarwal says machine learning is transforming the world with such developments as speech recognition, but the early stages of making the technology work posed a challenge. “It was a hog,” he says. “It was a memory hog; it was a computation hog. It was very expensive to run machine learning, but machine learning could do amazing things.”

The capabilities of machine learning can be limited though by access and availability of robust networks with supporting resources. Devices might always not have such connections, Agarwal says. Smartphones and tablets can leverage machine learning because they connect with computers running in the cloud. That type of access might not be feasible in every environment, he says. “This is where TinyML comes in.”

Google got involved to support the certificate program, in part because it may lead to more developers using its TensorFlow machine learning platform, says Josh Gordon, developer advocate on TensorFlow. “One of the goals, in addition to an open source framework, is we care a lot about the developer community,” he says. “We’re hoping that as more people learn how to use the software they will contribute to new examples and applications in the space.” Gordon describes TinyML as greenfield territory that is waiting to be explored. “We’re interested in seeing what types of projects the students come up with,” he says.

TinyML is meant to run machine learning when the footprint of the hardware is literally tiny, Agarwal says, potentially opening the door for new IT ecosystems and more edge computing. “When the device is small, it has to consume very low power and doesn’t have a huge link to the cloud,” he says. For instance, a motion sensor tied to a camera in the wilderness could be triggered to record leopards passing by. “There’s no way you can have a big computer server there with huge batteries to run it,” Agarwal says. “You don’t have a huge internet connection to transmit the data to the cloud where it can be processed. All your computation has to happen right there.”

More support for the development of TinyML could lead to more embedded devices that operate on little power and bandwidth, he says. “This is the Internet of Things in its most compelling form.”

There is already momentum for such innovation, he says, as more sensors in buildings, infrastructure, vehicles, and personal devices record and compute. The data streams those devices produce must still be turned into actionable intelligence, which can be performed through TinyML, Agarwal says.

He sees ways for TinyML to support multiple industries, such as energy companies with sensors that monitor pipelines, aircraft makers that have sensors on actuators on planes, and the technology behind self-driving cars.

The certification course is taught by Google engineers from the TensorFlow group and Harvard professors, Agarwal says, and can be completed within a few months. The pervasive nature of machine learning and AI could make this program useful to many types of engineers, he says, whether they operate in IT, software, hardware, devices, or sensors. “They might find it useful in terms of learning about applications of TinyML,” Agarwal says. “Others may find it useful in terms of how to develop for these applications.”

The post Google, Harvard, and EdX Team Up to Offer TinyML Training appeared first on Artificial Intelligence.

Many of us are familiar with the concept of machine learning as it pertains to neural networks. But what about TinyML?

Surging Interest in TinyML

TinyML refers to the machine learning technologies on the tiniest of microprocessors using the least amount of power (usually in mW range and lower) while aiming for maximized results.

With the proliferation of IoT devices, big names like Renesas and Arm have taken a vested interest in TinyML—for instance, with Arm’s recent expansion of its AI portfolio with new machine learning and neural processing IP and Renesas’ release of its TinyML platform, Qeexo AutoML, which does not require code nor expertise in ML. 

Other companies have zeroed in on partnerships that will help them exaggerate the utility of TinyML. Eta Compute and Edge Impulse recently announced their partnership in which they’ll combine the strengths of Eta Compute’s neural sensor processor, the ECM3532, with Edge Impulse’s tinyML platform. With an eye on battery capacity—a difficult point to work around in TinyML—this partnership hopes to accelerate the time-to-market of machine learning in billions of low-powered IoT products. 

Another way we can assess the progress of TinyML is to reflect on the tinyML Summit, which took place earlier this year. Several of the presentations at the conference illustrate the key concepts of machine learning at the smallest level. 

Reflections on the tinyML Summit

In February, AAC contributor Luke James forecasted the high aims for the 2020 tinyML Summit, which would, as in years past, spotlight developments in TinyML. The summit published presentations online and explored a number of categories pertaining to TinyML: hardware (dedicated integrated circuits), systems, algorithms and software, and applications.

Here are a few noteworthy presentations as they relate to design engineers.

Model Compression

Two of the presenters at the conference brought the realities of tinyML into focus by discussing a device we all have: mobile phones. In their discussion of “model compression,” MIT researcher Yujun Lin explained that typical machine learning devices, such as cell phones, have approximately 8 GB of RAM while microcontrollers have approximately 100 KB to 1 MB of RAM. Because microcontrollers have weight and activation constraints, they necessitate model compression.

The concept is to shrink the pre-trained large models into smaller ones without losing accuracy. This can be achieved in processes like pruning and deep compression. Pruning parses out synapses and neurons, resulting in ten times fewer connections. Deep compression takes pruning a step further with quantization (fewer bits per weight) and a technique known as “Huffman Encoding.” 

The researchers suggested that by combining a concept known as neural-hardware architecture search with non-expert usage into the neural network, we can improve AI-geared hardware. The VP and lab director of Samsung’s Advanced Institute of Technology, Changkyu Choi, went into further detail on deep model compression, but his focus was on acceleration toward on-sensor AI. 

Deep Reinforcement Learning

Another expert, Hoi-Jun Yoo, the ICT endowed chair professor from the engineering school at KAIST (Korea Advanced Institute of Science and Technology) spoke about the importance of deep reinforcement learning (DRL) accelerators within the deep neural network (DNN).

In his discussion, he points out that “software and hardware co-optimization for DNN training is necessary for low-power and high-speed accelerators in the same way it brought a dramatic increase in the performance of DNN inference accelerators.”

Yoo also explains that DRL is an essential factor in TinyML because it enables continuous decision-making in a low-power, “unknown environment,” or an environment in which labeled data is difficult to capture. 

DNNs for Always-on AI for Battery-Powered Devices

Another company, Syntiant, showcased one of their devices, the NDP100 neural decision processor (NDP), to discuss a broader concept: the value of deep learning over algorithmic genius. Dr. Stephen Bailey, CTO of Syntiant, explained that the magic of the company’s NDP, an always-on and “listening” device, is its deep neural networks (DNN)—continuing Yoo’s discussion on DNNs. 

The Syntiant NDP feeds acoustic features to a large DNN (no need for cascading or energy gating) and trains the DNN with large data sets and wide-ranging augmentation. Beyond its noise immunity, the NDP100 is extremely small in size (1.4 mm x 1.8 mm) and consumes less than 140 μW. 

Since the summit, Syntiant has also released the NDP101, which is said to couple computation power and memory to exploit “the vast inherent parallelism of deep learning and computing at only required numerical precision.” Syntiant says that these features improve efficiency by 100 times compared to the stored program architectures you’d see in CPUs and DSPs. 

Smaller Devices Call for Compressed Machine Learning

The hardware requirements for machine learning in larger systems are similar for TinyML in small IoT. But sometimes, the stakes are higher because of the device’s small size: accuracy, latency, and power consumption. As smaller IoT devices hit the market, engineers may increasingly dabble in TinyML, familiarizing themselves with concepts like deep neural networks, model compression, and deep reinforcement learning. 

The post TinyML: When Small IoT Devices Call for Compressed Machine Learning appeared first on Artificial Intelligence.