Two presentations at JSAI 2019

We are going to give two research talks at the Annual Conference of the Japanese Society for Artificial Intelligence (JSAI), Niigata, held from June 4th to 7th, 2019. We will open our booth at the exhibition hall and give orientation for recruiting summer interns, engineers, and engineering managers. Please feel free to stop by our booth.

1. Reducing the computations in ConvRNNs We will introduce a method to reduce the computational load of convolutional recursion network (ConvRNN) by 20–40% while keeping the performance degradation as small as 1–2%.

2. Final sample batch normalization We will present a new batch normalization method specialized for quantized, low-bit networks, designed by taking advantage of the behavior found in quantized networks.

1. Reducing the computations in ConvRNNs

Official Information: Presentation information Presentation Date: Wed. Jun 5, 2019 4:40 PM - 5:00 PM Presentation Venue: Room H (303+304 Small meeting rooms) Presentation Number: International Session：[2H4-E-2-05]

D. Vazhenina and A. Kanemura, “Reducing the number of multiplications in convolutional recurrent neural networks (ConvRNNs),” Annual Conference of Japanese Society for Artificial Intelligence (JSAI), Niigata, Japan, 2019. Deep learning has been applied to spatio-temporal modeling, such as human activity recognition (Fig. 1) and weather forecasting (Fig. 2), and expected to be useful for e.g., predicting future events. The most influential model for this purpose is ConvRNNs (convolutional recurrent neural networks) [1], which was an innovation that combines the temporal prediction ability of RNNs and efficient image processing with convolutional networks (Fig. 3). One technical issue in ConvRNN is that it requires huge computation, leading to difficulties in operating it in real-world applications such as video content description and activity recognition.

In our JSAI paper, we report that it is possible to reduce the computation load of ConvRNNs. The technique we employed was to replace convolutional operations in recurrent hidden connections with the Hadamard product to reduce the number of parameters and multiplications. We evaluate our proposal using the task of next video frame prediction and the Moving MNIST dataset. The proposed method requires 38% less multiplications and 21% less parameters compared to the fully convolutional counterpart. In price of the reduced computational complexity, the performance measured by for structural similarity index measure (SSIM) decreased by about 1.5%. We continue research and development to make ConvRNNs more practical to be used in various situations like in web apps or embedded systems. Please refer for details to our paper to be published on the JSAI website. JSAI2019 Proceedings

Figure 1: Image sequences of human actions (from the KTH dataset).

Figure 2: Forecast of the rainfall intensity in a local region based on radar echo maps (from the supplementary material of [1]).

Figure 3: Architecture of a ConvRNN classifier (adopted and modified from Figure 2 of [2]).

[1] S. Xingjian, Z. Chen, H. Wang, D. Yeung, W. Wong, and W. Woo. Convolutional LSTM network: A machine learning approach for precipitation nowcasting. In Neural Information Processing Systems (NIPS), 2015.

[2] Nicolas Ballas, Li Yao, Chris Pal, and Aaron Courville. Delving deeper into convolutional networks for learning video representations. International Conference on Learning Representations (ICLR) , 2016.

2. Final sample batch normalization

Presentation Date： Wed. Jun 5, 2019 4:40 PM - 5:00 PM Presentation Venue： Room A (2F Main hall A) Presentation Number： International Session：[2A4-E-2-05]

J. Nicholls and A. Kanemura, “Final sample batch normalization for quantized neural networks,” Annual Conference of Japanese Society for Artificial Intelligence (JSAI), Niigata, Japan, 2019. Batch normalization is a core invention in deep learning, along with ReLU, and this technique achieved below 5% top-5 error, improving over the winner of the most recent ImageNet competition of the time [3]. Almost all modern deep learning models employ batch normalization. When we performed research for making compact neural networks, we encountered a strange behavior of batch normalization for quantized networks that was not observed for full-precision networks. This discovery led us to find a new technique on batch normalization.

In our recent work to be presented in JSAI 2019, we have found a nice technique, final batch normalization or BN-final, to further improve batch normalization for quantized neural networks. Instead of using population statistics, BN-final uses the batch statistics of only the final batch to create the fixed parameters for inference. We undertook a range of experiments including classification and object detection tasks, to verify the usefulness of BN-final. The figure below illustrates this, for the final accuracy; we can see that BN-final performed better in all of the four datasets and was most effective for the CIFAR-10 dataset, where the accuracy improvement was improved about 9 points (9%) from 78% to 87% (Fig. 4). We also find that the test accuracy is smoother and has greater value during throughout training. Please refer for details to our paper to be published on the JSAI website. [JSAI2019 Proceedings]

Figure 4: BN-final (blue) outperforms BN-pop (green), the standard batch normalization method.

[3] S. Ioffe and C. Szegedy, “Batch normalization accelerating deep network training by reducing internal covariate shift”, International Conference on International Conference on Machine Learning (ICML), Lille, France, 2015.

Announcement of company exhibition at JSAI conference

As a conference sponsor, we will set up a booth at the venue from June 4th to June 7th. At the booth, we will introduce our latest information and job opportunities (including summer interns, engineers, and engineering managers). Please stop by our booth during your participation. We are looking forward to your visit.

Venue

TOKI MESSE, Niigata Convention Center Exhibition Hall A：1F 展示ホールB　A-８

Date and time

June 4th (Tue), 13:00～17:00 June 5th (Wed), 9:00～17:00 June 6th (Thu), 9:00～17:00 June 7th (Fri), 9:00～15:00

Summer interns Wanted!

We are looking for students who are interested in Deep learning for LeapMind's summer internship project 2019!

Notice for international students:

You can apply to "LeapMind Summer Internship Program" if you already have visa eligibility to work as an intern in Japan. Even if you don’t have visa eligibility, you can apply to the regular internship program, which is open all year around. If you are interested, please check here. LeapMind is looking for motivated students for the summer internship program. At LeapMind, we aim to implement deep learning in every object and to realize a more convenient and rich society. To this end, we develop open source software Blueoil for creating low-bit compact neural networks and design processors for deep learning which operate with low power consumption. Would you like to join Blueoil development, processor design, or other projects, and be a part of bringing advanced technology to society?