Sparse Networks from Scratch: Faster Training without Losing Performance This paper demonstrates the possibility of what is called sparse learning: accelerated training of deep neural networks that maintain sparse weights throughout training while achieving dense performance levels.
This paper accomplishs this by developing sparse momentum, an algorithm which uses exponentially smoothed gradients (momentum) to identify layers and weights which reduce the error efficiently.
Sparse momentum redistributes pruned weights across layers according to the mean momentum magnitude of each layer.
Within a layer, sparse momentum grows weights according to the momentum magnitude of zero-valued weights.
The paper demonstrates state-of-the-art sparse performance on MNIST, CIFAR-10, and ImageNet, decreasing the mean error by a relative 8%, 15%, and 6% compared to other sparse algorithms.
The PyTorch code for this paper can be found HERE.
Trading via Image Classification The art of systematic financial trading evolved with an array of approaches, ranging from simple strategies to complex algorithms all relying, primary, on aspects of time-series analysis.
Recently, after visiting the trading floor of a leading financial institution, AI researchers from J.
Morgan noticed that traders always execute their trade orders while observing images of financial time-series on their screens.
This work is built upon the success in image recognition and examine the value in transforming the traditional time-series analysis to that of image classification.
The researchers created a large sample of financial time-series images encoded as candlestick (Box and Whisker) charts and label the samples following three algebraically-defined binary trade strategies.
Using the images, the researchers trained over a dozen machine-learning classification models and find that the algorithms are very efficient in recovering the complicated, multiscale label-generating rules when the data is represented visually.
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods Integration of vision and language tasks has seen a significant growth in the recent times due to surge of interest from multi-disciplinary communities such as deep learning, computer vision, and natural language processing.
This survey focuses on ten different vision and language integration tasks in terms of their problem formulation, methods, existing datasets, evaluation measures, and comparison of results achieved with the corresponding state-of-the-art methods.
Where is the Information in a Deep Neural Network?.Whatever information a Deep Neural Network has gleaned from past data is encoded in its weights.
How this information affects the response of the network to future data is largely an open question.
In fact, even how to define and measure information in a network is still not settled.
This paper introduces the notion of Information in the Weights as the optimal trade-off between accuracy of the network and complexity of the weights, relative to a prior.
Depending on the prior, the definition reduces to known information measures such as Shannon Mutual Information and Fisher Information, but affords added flexibility that enables us to relate it to generalization, via the PAC-Bayes bound, and to invariance.
This relation hinges not only on the architecture of the model, but surprisingly on how it is trained.
A Selective Overview of Deep Learning Deep learning has arguably achieved tremendous success in recent years.
In simple words, deep learning uses the composition of many nonlinear functions to model the complex dependency between input features and labels.
While neural networks have a long history, recent advances have greatly improved their performance in computer vision, natural language processing, etc.
From the statistical and scientific perspective, it is natural to ask: What is deep learning?.What are the new characteristics of deep learning, compared with classical methods?.What are the theoretical foundations of deep learning?.To answer these questions, this paper introduces common neural network models (e.
, convolutional neural nets, recurrent neural nets, generative adversarial nets) and training techniques (e.
, stochastic gradient descent, dropout, batch normalization) from a statistical point of view.
Augmenting Self-attention with Persistent Memory Transformer networks have lead to important progress in language modeling and machine translation.
These models include two consecutive modules, a feed-forward layer and a self-attention layer.
The latter allows the network to capture long term dependencies and are often regarded as the key ingredient in the success of Transformers.
Building upon this intuition, this paper proposes a new model that solely consists of attention layers.
More precisely, the Facebook AI researchers augment the self-attention layers with persistent memory vectors that play a similar role as the feed-forward layer.
Thanks to these vectors, we can remove the feed-forward layer without degrading the performance of a transformer.
The evaluation shows the benefits brought by our model on standard character and word level language modeling benchmarks.
Sign up for the free insideBIGDATA newsletter.