September 27, 2022

Summer Internship 2022 – Ryu – Exploring Binary Neural Networks

Summer internship


こんにちは。LeapMind インターン運営担当です。 本日は、2022年サマーインターンに参加されたインターン生の成果発表について記事にまとめたいと思います。

Hello, this is LeapMind Internship Administration Team. Today, we would like to share the presentations of our interns who participated in the 2022 Summer Internship Program.

はじめまして。LeapMindでインターンをしていた劉です。 夏休みの2ヶ月間、Network Development Kit (NDK)チームに参加させていただきました。 この記事ではインターン期間中の取り組みをはじめ、社内の環境や雰囲気など、インターンを通して得られたこと、感じたことを総合的に書かせていただきたいと思います。

Hello, I am Ryu, an intern at LeapMind. I joined the Network Development Kit (NDK) team for two months as an intern during the summer vacation. In this article, I would like to write a comprehensive report on what I gained and felt through my internship, including the project I worked on and the work environment and atmosphere within the company.

インターン期間中の取り組み / The project I worked on during my internship.

今回のインターンの課題はBinary Neural Networkに関する新しい論文[1]の再現実験をするというものでした。この課題にインターン生2人で取り組みました。論文では、重みパラメータとアクティベーションの両方をそれぞれ1bitにするという大胆な量子化を提案しています。それでいて、ImageNetを用いた分類問題を始め、CNNが得意とするさまざまな課題で量子化前と同程度の性能を出せることを論文では主張しています。ニューラルネットワークの量子化という分野において精度の低下は常に頭を悩ませてきた問題であるため、「量子化前と同程度の性能」をたったの1bitずつで実現できるのであれば、それは驚くべきことであり、今後のLeapMindの製品に役立つことも期待できます。

The project for this internship was to experiment to reproduce a new research paper on Binary Neural Networks [1]. The two interns including me worked on this assignment together. The research paper proposes a bold quantization scheme in which both the weight parameter and the activation are set to 1 bit. Despite this, they claim that they can deliver the same level of performance as before quantization in various tasks that CNNs are good at, such as classification problems using ImageNet. Since accuracy degradation has always been a major problem in the field of neural network quantization, if "the same level of performance as before quantization" with only 1 bit each could be realized, it is a remarkable achievement and will be useful for LeapMind's future products.


Although these motivations led us to start this reproduction experiment, it turned out to be more difficult than expected. The source code has not been provided in the paper. Neither are hyperparameters like learning rate or necessary information for reproduction, such as optimizers, provided. Therefore, we had to infer these from experience and other implementations. We cannot underestimate the difficulty of hyperparameter search, however, this much effort will often be necessary for the reproduction of other papers. The most difficult part of reproducing this research paper was not knowing the shape of the neural network.


Let me give you one specific example. The figure below shows the network proposed by the paper with bit width annotations.



The red dotted line indicates 4 bits, but we could not figure out from the paper why it should be 4 bits or what kind of structure makes it 4 bits. In the end, we gave up on making this part 4-bit in the reproduction experiment and searched for architecture as close as possible to the description in the paper, being careful not to impair the executability of the hardware.


Although the accuracy was gradually improved through these trial and error processes, we were not able to generate "the same level of performance as before quantization," as stated in the paper, during the internship program. Although we failed to reproduce the paper completely, I believe that we were able to produce a good level of accuracy as a result of quantization with 1-bit weights and activation. Above all, I am very happy that I was able to propose a good accuracy architecture through discussions with other members of the group.

開発環境 / Development environment


In university, there are not many opportunities to develop in teams on assignments or in research, especially when there are dozens of people involved. At LeapMind, internal source code is managed in a single repository. When we write code, submit a pull request, and once it meets readability and other criteria, it is merged. This code review makes the source code very legible and allows us to find changes quickly if they are simple changes. Submitting pull requests to be reviewed and reviewing code written by others was a great learning and refreshing experience for me.


LeapMind uses several tools to streamline research and development, including Bazel and Neptune. Many of these tools seem to be useful for university research, and I plan to continue using them for my development even after my internship is over. I feel that the reason why many of these tools are used is because of the corporate culture, which is characteristic of a venture company, of not fearing changes in the environment and eagerly adopting new technologies.

社内の雰囲気・環境/Company culture and environment


I was amazed at the flexibility of the working style: flextime, no core hours, and full remote availability. I was also able to take vacations when I wanted to, and I never felt uncomfortable taking time off, including among the employees. I was also able to go to the office whenever I wanted to and work in a comfortable environment with two monitors.

COVID-19の影響もあり出勤する方が少なく、社員の方と対面でコミュニケーションを取る機会が非常に限られていた点は好みが分かれる部分だと思います。それでも、ランチ会やCoffee Breakを通じて、オンラインで多くの社員の方との会話の機会を確保していただきました。経歴や趣味など個性的な方も多く、会話していて楽しかったです。また、技術的に分からないことがあっても気軽に質問することができ、詳しい方に助けてもらうことができました。社員の方の技術力の高さにも驚かされました。

Due to COVID-19, few people came to the office, and the opportunities for face-to-face communication with employees were very limited, which I think is something that people may have different preferences for. Nevertheless, through lunch meetings and Coffee Breaks, I was able to have opportunities to talk with many employees online. I enjoyed talking with many unique people with unique backgrounds and interests. Also, I was able to easily ask questions about technical issues and was able to get help from people who were familiar with the subject matter. I was also amazed at the high technical skills of the employees.

最後に / Conclusion


The two months went by very quickly. It was a topic that I had always been interested in, and I was able to work on it with great interest until the very end. I still remember how excited I got when we got the highest score ever as a result of the experiment, and I am very happy to be an intern at LeapMind. Thank you very much for all your support!

参考文献 / References

[1] Nie, G., Xiao, L., Zhu, M., Chu, D., Shen, Y., Li, P., ... & Chen, B. (2022). Binary Neural Networks as a general-propose compute paradigm for on-device computer vision. arXiv preprint arXiv:2202.03716.