As larger neural networks with more layers and nodes are considered, reducing their storage and computational cost becomes critical, especially for some real-time applications such as online learning and incremental learning

In addition, recent years witnessed significant progress in virtual reality, augmented reality, and smart wearable devices, creating challenges in deploying deep learning systems to portable devices with limited resources (e.g. memory, CPU, energy, bandwidth).

Here are a few methods that are part of all compression techniques:

Parameter Pruning And Sharing

• Reducing redundant parameters which are not sensitive to the performance
• Robust to various settings
• Redundancies in the model parameters are explored and the uncritical yet redundant ones are removed

Low-Rank Factorisation

• Uses matrix decomposition to estimate the informative parameters of the deep convolutional neural networks

Transferred/Compact Convolutional Filters

• Special structural convolutional filters are designed to reduce the parameter space and save storage/computation

Knowledge Distillation

• A distilled model is used to train a more compact neural network to reproduce the output of a larger network

Now let’s take a look at a few papers that introduced novel compression models:

1.Deep Neural Network Compression with Single and Multiple Level Quantization

In this paper, the authors propose two novel network quantization approaches single-level network quantization (SLQ) for high-bit quantization and multi-level network quantization (MLQ).

The network quantization is considered from both width and depth level.

2.Efficient Neural Network Compression

In this paper the authors proposed an efficient method for obtaining the rank configuration of the whole network. Unlike previous methods which consider each layer separately, this method considers the whole network to choose the right rank configuration.

3.3LC: Lightweight and Effective Traffic Compression

3LC is a lossy compression scheme developed by the Google researchers that can be used for state change traffic in distributed machine learning (ML) that strikes a balance between multiple goals: traffic reduction, accuracy, computation overhead, and generality. It combines three techniques — value quantization with sparsity multiplication, base encoding, and zero-run encoding.SEE ALSO

DEVELOPERS CORNER

WHAT IS DATAOPS? THINGS AROUND IT THAT YOU NEED TO KNOW

4.Universal Deep Neural Network Compression

This work for the first time, introduces universal DNN compression by universal vector quantization and universal source coding. In particular, this paper examines universal randomised lattice quantization of DNNs, which randomises DNN weights by uniform random dithering before lattice quantization and can perform near-optimally on any source without relying on knowledge of its probability distribution.

5.Compression using Transform Coding and Clustering

The compression (encoding) approach consists of transform and clustering with great encoding efficiency, which is expected to fulfill the requirements towards the future deep model communication and transmission standard. Overall, the framework works towards light weight model encoding pipeline with uniform quantization and clustering has yielded great compression performance, which can be further combined with existing deep model compression approaches towards light-weight models.

6.Weightless: Lossy Weight Encoding

The encoding is based on the Bloomier filter, a probabilistic data structure that saves space at the cost of introducing random errors. The results show that this technique can compress DNN weights by up to 496x; with the same model accuracy, this results in up to a 1.51x improvement over the state-of-the-art.

7.Adaptive Estimators Show Information Compression

The authors developed more robust mutual information estimation techniques, that adapt to hidden activity of neural networks and produce more sensitive measurements of activations from all functions, especially unbounded functions. Using these adaptive estimation techniques, they explored compression in networks with a range of different activation functions. 

8.MLPrune: Multi-Layer Pruning For Neural Network Compression

It is computationally expensive to manually set the compression ratio of each layer to find the sweet spot between size and accuracy of the model. So,in this paper, the authors propose a Multi-Layer Pruning method (MLPrune), which can automatically decide appropriate compression ratios for all layers.

Large number of weights in deep neural networks make the models difficult to be deployed in low memory environments. The above-discussed techniques achieve not only higher model compression but also reduce the compute resources required during inferencing. This enables model deployment in mobile phones, IoT edge devices as well as “inferencing as a service” environments on the cloud.

The post 8 NEURAL NETWORK COMPRESSION TECHNIQUES FOR ML DEVELOPERS appeared first on Artificial Intelligence.

Almost all organisations are adopting emerging technologies such as machine learning and data science. These machine learning frameworks are meant for the developers who work using Java language. In this article, we list you 8 machine learning frameworks for Java developers.

1| Apache SAMOA

Apache Scalable Advanced Massive Online Analysis (SAMOA) is a distributed streaming machine learning framework which contains a programming abstraction for distributed streaming machine learning algorithms. It provides a collection of distributed streaming algorithms for the most common data mining and machine learning tasks such as classification, clustering, and regression. Apache SAMOA enables the development of new machine learning algorithms without dealing with the complexity of underlying streaming processing engines as well as provides extensibility in integrating new SPEs into the framework.

2| AMIDST ToolBox

AMIDST is an open source Java toolbox for scalable probabilistic machine learning with a special focus on streaming data. It allows specifying probabilistic graphical models with latent variables and temporal dependencies. AMIDST provides tailored parallel and distributed implementations of Bayesian parameter learning for batch and streaming data. This processing is based on flexible and scalable message passing algorithms. The features of this toolbox include probabilistic graphical models, scalable inference, data streams, large-scale data, extensible and interoperability.

3| Apache Mahout

Apache Mahout is a distributed linear algebra framework and mathematically expressive Scala DSL which is designed to quickly implement the machine learning algorithms. This framework mainly focuses on clustering, classification, and filtering. Running any application which uses Mahout will require installing a binary or source version and setting the environment.

4| Datumbox

The Datumbox machine learning framework is an open-source framework written in Java which allows the rapid development of machine learning and statistical applications. The main focus of the framework is to include a large number of machine learning algorithms & statistical methods and to be able to handle large-sized datasets.

The framework currently supports performing multiple parametric & non-parametric statistical tests, calculating descriptive statistics on censored & uncensored data, performing ANOVA, cluster analysis, dimension reduction, regression analysis, time series analysis, sampling and calculation of probabilities from the most common discrete and continues Distributions. In addition, it provides several implemented algorithms including Max Entropy, Naive Bayes, SVM, Bootstrap Aggregating, Adaboost, Kmeans, Hierarchical Clustering, Dirichlet Process Mixture Models, Softmax Regression, Ordinal Regression, Linear Regression, Stepwise Regression, PCA, etc.

5| ELKI

ELKI is an open source data mining software written in Java. The focus of ELKI is research in algorithms, with an emphasis on unsupervised methods in cluster analysis and outlier detection. It aims at providing a large collection of highly parameterizable algorithms, in order to allow easy and fair evaluation and benchmarking of algorithms. In ELKI, data mining algorithms and data management tasks are separated and allow for an independent evaluation. This separation makes ELKI unique among data mining frameworks like Weka or Rapidminer and frameworks for index structures like GiST.

6| Encog

Encog is a pure Java/C# machine learning framework which is created in 2008 to support genetic programming, NEAT/HyperNEAT, and other neural network technologies. This framework supports a variety of advanced algorithms, as well as support classes to normalize and process data. Machine learning algorithms such as Support Vector Machines, Neural Networks, Bayesian Networks, Hidden Markov Models, Genetic Programming and Genetic Algorithms are supported. Most Encog training algorithms are multi-threaded and scale well to multicore hardware.

7| Neuroph

Neuroph is an open source, lightweight Java neural network framework to develop common neural network architectures. It contains well designed, open source Java library with a small number of basic classes which correspond to basic NN concepts. This framework also has a nice GUI neural network editor to quickly create Java neural network components.

8| Smile

Smile (Statistical Machine Intelligence and Learning Engine) is a fast and comprehensive machine learning, NLP, linear algebra, graph, interpolation, and visualization system in Java and Scala. It covers every aspect of machine learning with neat interfaces, including classification, regression, clustering, association rule mining, feature selection, manifold learning, multidimensional scaling, genetic algorithms, missing value imputation, efficient nearest neighbour search, etc.

The post 8 Machine Learning Frameworks Java Developers Must Try In 2019 appeared first on Artificial Intelligence.