<h2 id="my-summer-internship-experience-at-leapmind">My summer internship experience at LeapMind.</h2>
<p>Shibuya Scramble Crossing, the most iconic Tokyo city landscape. Every minute hundreds of people seamlessly pass by, eventually disappearing in an endless crowd. Could you count them all? Could you track them all down? Well, I can. And I learned this secret skill during my summer internship at LeapMind Inc.</p>
<h2 id="multi-object-tracking">Multi-object tracking</h2>
<p>The task is simple: given a video, track all its objects as they move. You can think of it more atomically: first, detect objects within a single frame, and then link those objects in between frames. Former is simply object detection, whereas the latter is called object re-identification. Ideally, we would like to process frames in an online manner, that is, without looking into future frames while processing the current one. This is a pretty reasonable and desired assumption, especially when it comes to live videos (think of CCTV footage).</p>
<p>As one might expect, recent deep learning models enable multi-object tracking – and <a href="https://motchallenge.net/results/MOT20/?det=Private">they are fairly good</a> at it. Unfortunately, they do not necessarily excel at speed. It is challenging to run a live demo smoothly, and a networking delay while streaming the video from a camera to a GPU makes it all hardly feasible. Overcoming the speed issue was a goal of my internship.</p>
<p><img src="/images/posts/en/news/how-i-became-a-spy/7442f705-image1.gif" alt="Image: Multi-object tracking"></p>
<p>A live object tracking demo running at 70 FPS, recorded at LeapMind office, Shibuya, Tokyo.</p>
<h2 id="accelerating">Accelerating</h2>
<p>We focused on a one state-of-the art multi-object tracking method. Having understood, analyzed and identified its bottleneck, we were able to run a live demo in average almost 5-times faster than a baseline – at 100 FPS at a peak. Noteworthy, we did not suffer from a significant accuracy drop.
Despite an outstanding performance, it’s not the end of further improvements. The model we ended up with can be potentially <a href="https://medium.com/@joel_34050/quantization-in-deep-learning-478417eab72b">quantized</a> (compressed) and embedded on a hardware using LeapMind technology.</p>
<p><img src="/images/posts/en/news/how-i-became-a-spy/08d7ac54-joniak4-660x440.png" alt="Image:Presentation"></p>
<h2 id="leapmind-is-a-place-to-work">LeapMind is a place to work</h2>
<p>I spent five weeks at LeapMind as an intern, and I did enjoy it. I was working with professionals on a challenging project that won’t be archived once I leave. The atmosphere here kept me always motivated and I had no problems communicating – LeapMind is fully bilingual.
Company response to COVID-19 is also worth praising, since it went entirely remote (although one can visit the office if necessary). Last but not least, the salary is competitive, even if you’re an intern.</p>
<p>LeapMind is a place to work!</p>
<p>Written by Przemyslaw Joniak, LeapMind Summer 2020 Intern.</p>

How I became a spy

<p>こんにちは。LeapMind インターン運営担当です。 本日は、2022年サマーインターンに参加されたインターン生の成果発表について記事にまとめたいと思います。</p>
<p>Hello, this is LeapMind Internship Administration Team. Today, we would like to share the presentations of our interns who participated in the 2022 Summer Internship Program.</p>
<p>はじめまして。LeapMindでインターンをしていた劉です。
夏休みの2ヶ月間、Network Development Kit (NDK)チームに参加させていただきました。
この記事ではインターン期間中の取り組みをはじめ、社内の環境や雰囲気など、インターンを通して得られたこと、感じたことを総合的に書かせていただきたいと思います。</p>
<p>Hello, I am Ryu, an intern at LeapMind.
I joined the Network Development Kit (NDK) team for two months as an intern during the summer vacation.
In this article, I would like to write a comprehensive report on what I gained and felt through my internship, including the project I worked on and the work environment and atmosphere within the company.</p>
<h2 id="インターン期間中の取り組み--the-project-i-worked-on-during-my-internship">インターン期間中の取り組み / The project I worked on during my internship.</h2>
<p>今回のインターンの課題はBinary Neural Networkに関する新しい論文[1]の再現実験をするというものでした。この課題にインターン生2人で取り組みました。論文では、重みパラメータとアクティベーションの両方をそれぞれ1bitにするという大胆な量子化を提案しています。それでいて、ImageNetを用いた分類問題を始め、CNNが得意とするさまざまな課題で量子化前と同程度の性能を出せることを論文では主張しています。ニューラルネットワークの量子化という分野において精度の低下は常に頭を悩ませてきた問題であるため、「量子化前と同程度の性能」をたったの1bitずつで実現できるのであれば、それは驚くべきことであり、今後のLeapMindの製品に役立つことも期待できます。</p>
<p>The project for this internship was to experiment to reproduce a new research paper on Binary Neural Networks [1]. The two interns including me worked on this assignment together. The research paper proposes a bold quantization scheme in which both the weight parameter and the activation are set to 1 bit. Despite this, they claim that they can deliver the same level of performance as before quantization in various tasks that CNNs are good at, such as classification problems using ImageNet. Since accuracy degradation has always been a major problem in the field of neural network quantization, if "the same level of performance as before quantization" with only 1 bit each could be realized, it is a remarkable achievement and will be useful for LeapMind's future products.</p>
<p>こうしたモチベーションによって始まった今回の再現実験ですが、予想以上に難航してしまいます。論文にソースコードは提供されていません。学習率のようなハイパーパラメータや、オプティマイザのような再現に必要な情報も提供されていません。そのため、これらは過去の経験や他の実装から推測することになります。ハイパーパラメータ探索の大変さを見くびることはできませんが、ここまでの苦労であれば他の論文の再現でも必要となることは多いでしょう。本論文の再現において最も苦労したのは、ニューラルネットワークの形が分からないという点でした。</p>
<p>Although these motivations led us to start this reproduction experiment, it turned out to be more difficult than expected. The source code has not been provided in the paper. Neither are hyperparameters like learning rate or necessary information for reproduction, such as optimizers, provided. Therefore, we had to infer these from experience and other implementations. We cannot underestimate the difficulty of hyperparameter search, however, this much effort will often be necessary for the reproduction of other papers. The most difficult part of reproducing this research paper was not knowing the shape of the neural network.</p>
<p>具体的に1つ例を挙げましょう。下の図は論文が提案するネットワークにビット幅の注釈をつけたものです。</p>
<p>Let me give you one specific example. The figure below shows the network proposed by the paper with bit width annotations.</p>
<p><img src="/images/posts/news/summer-internship-2022-ryu-exploring-binary-neural-networks/f05c0373-ryu-san-360x440.png" alt="具体イメージ"></p>
<p>赤の破線は4bitと書かれていますが、なぜ4bitとするのか、どういう構造で4bitになっているのかを論文から読み取ることができませんでした。結局、再現実験ではここを4bitにすることを諦め、ハードウェアでの実行可能性を損ねないように注意しながら、論文の記述になるべく近いアーキテクチャを探索しました。</p>
<p>The red dotted line indicates 4 bits, but we could not figure out from the paper why it should be 4 bits or what kind of structure makes it 4 bits. In the end, we gave up on making this part 4-bit in the reproduction experiment and searched for architecture as close as possible to the description in the paper, being careful not to impair the executability of the hardware.</p>
<p>こうした試行錯誤によって精度は次第に改善されていったものの、論文の言うような「量子化前と同程度の性能」をインターン期間中に出すことはできませんでした。その意味で論文の完全な再現には失敗したものの、重みとアクティベーション両方を1bitにした量子化の結果としてはまずまずな精度を得ることができたと思います。とりわけ、他のメンバーとのディスカッションを通じて、良い精度のアーキテクチャを自ら提案できたことはとても嬉しく思います。</p>
<p>Although the accuracy was gradually improved through these trial and error processes, we were not able to generate "the same level of performance as before quantization," as stated in the paper, during the internship program. Although we failed to reproduce the paper completely, I believe that we were able to produce a good level of accuracy as a result of quantization with 1-bit weights and activation. Above all, I am very happy that I was able to propose a good accuracy architecture through discussions with other members of the group.</p>
<h2 id="開発環境--development-environment">開発環境 / Development environment</h2>
<p>大学の課題や研究において、チームで開発することは多くありません。数十人規模ともなるとなおさらです。LeapMindでは、社内のソースコードをほぼ1つのレポジトリで管理しています。コードを書いたらプルリクエストを出し、読みやすさ等の評価をクリアするとマージされます。このコードレビューのおかげで、ソースコードはとても読みやすく、簡単な変更であればすぐに変更箇所を見つけ出すことができました。実際にプルリクエストを出してレビューしてもらったり、他の人の書いたコードをレビューしたりすることは、私にとって学びの多い新鮮な経験になりました。</p>
<p>In university, there are not many opportunities to develop in teams on assignments or in research, especially when there are dozens of people involved. At LeapMind, internal source code is managed in a single repository. When we write code, submit a pull request, and once it meets readability and other criteria, it is merged. This code review makes the source code very legible and allows us to find changes quickly if they are simple changes. Submitting pull requests to be reviewed and reviewing code written by others was a great learning and refreshing experience for me.</p>
<p>LeapMindでは、Bazel、Neptuneなど、研究開発を効率化するツールをいくつも使用しています。これらのツールは大学の研究に役立ちそうなものも多く、インターン終了後も自分の開発に役立てていきたいと考えています。このようなツールが多く使われている背景には、環境の変化を恐れず新しい技術を貪欲に取り入れていく、ベンチャー企業らしいとも言える風土があると感じます。</p>
<p>LeapMind uses several tools to streamline research and development, including Bazel and Neptune. Many of these tools seem to be useful for university research, and I plan to continue using them for my development even after my internship is over. I feel that the reason why many of these tools are used is because of the corporate culture, which is characteristic of a venture company, of not fearing changes in the environment and eagerly adopting new technologies.</p>
<h2 id="社内の雰囲気環境company-culture-and-environment">社内の雰囲気・環境/Company culture and environment</h2>
<p>フレックスタイム、コアタイムなし、フルリモート可という働き方の自由さにまず驚かされました。休暇も基本的に自由に取ることができ、社員の方も含め休みづらさといったものは全く感じませんでした。オフィスに出勤したいときには出勤することもでき、モニター2台の快適な環境で作業することができました。</p>
<p>I was amazed at the flexibility of the working style: flextime, no core hours, and full remote availability. I was also able to take vacations when I wanted to, and I never felt uncomfortable taking time off, including among the employees. I was also able to go to the office whenever I wanted to and work in a comfortable environment with two monitors.</p>
<p>COVID-19の影響もあり出勤する方が少なく、社員の方と対面でコミュニケーションを取る機会が非常に限られていた点は好みが分かれる部分だと思います。それでも、ランチ会やCoffee Breakを通じて、オンラインで多くの社員の方との会話の機会を確保していただきました。経歴や趣味など個性的な方も多く、会話していて楽しかったです。また、技術的に分からないことがあっても気軽に質問することができ、詳しい方に助けてもらうことができました。社員の方の技術力の高さにも驚かされました。</p>
<p>Due to COVID-19, few people came to the office, and the opportunities for face-to-face communication with employees were very limited, which I think is something that people may have different preferences for. Nevertheless, through lunch meetings and Coffee Breaks, I was able to have opportunities to talk with many employees online. I enjoyed talking with many unique people with unique backgrounds and interests. Also, I was able to easily ask questions about technical issues and was able to get help from people who were familiar with the subject matter. I was also amazed at the high technical skills of the employees.</p>
<h2 id="最後に--conclusion">最後に / Conclusion</h2>
<p>あっという間の2ヶ月間でした。もともと興味のあるテーマでしたが、最後まで興味を持って取り組むことができました。実験の結果、過去最高のスコアが出たときの興奮は今でも忘れません。LeapMindでインターンをすることができて本当に良かったと思っています。大変お世話になりました。</p>
<p>The two months went by very quickly. It was a topic that I had always been interested in, and I was able to work on it with great interest until the very end. I still remember how excited I got when we got the highest score ever as a result of the experiment, and I am very happy to be an intern at LeapMind. Thank you very much for all your support!</p>
<h2 id="参考文献--references">参考文献 / References</h2>
<p>[1] Nie, G., Xiao, L., Zhu, M., Chu, D., Shen, Y., Li, P., ... &#x26; Chen, B. (2022). Binary Neural Networks as a general-propose compute paradigm for on-device computer vision. arXiv preprint arXiv:2202.03716.</p>

Summer Internship 2022 – Ryu – Exploring Binary Neural Networks

<p>こんにちは。LeapMind インターン運営担当です。 本日は、2022年サマーインターンに参加されたインターン生の成果発表について記事にまとめたいと思います。</p>
<h2 id="my-summer-internship-experience-at-leapmind">My summer internship experience at LeapMind</h2>
<p>Hello, my name is Nishio and I participated in the 2022 summer internship program.
For a little less than two months, I worked with the NDK team on an experiment to reproduce a paper of binary neural networks.
The paper I replicated is “Binary Neural Networks as a general-propose compute paradigm for on-device computer vision” [1].
The “BiNealNet” proposed in this paper is claimed to outperform 8-bit quantization methods in the speed-vs-accuracy tradeoff for classification, detection, segmentation, super-resolution, and matching.</p>
<h2 id="implementation-overview">Implementation Overview</h2>
<p>BiNealNet is one of the binary neural networks that achieves high accuracy and computational efficiency for various tasks by using the following two techniques:</p>
<ul>
<li>Binarize a ResNet block by enlarging its channels.</li>
<li>Injecting auxiliary parameters in binaryconvolution and binary activation.</li>
</ul>
<p>Binarizing ResNet blocks allows for easy extension to various tasks, and the larger number of output channels in the block increases the capacity of the network. In addition, auxiliary parameters also improved the expressiveness of the network.</p>
<p>Binarization of ResNet blocks is realized by two methods: binarization of convolution weight parameters and binarization of activation.</p>
<p>Weight parameters of convolution are binarized by
Wb = Sign ( Tanh( Wf ) )
Wf Wb
Where Wb is the value of the binarized weight and Wf is the full precision weight, and are arbitrary parameters.
Sign function calculated the gradient using STE [2]. is a channel-wise parameter. is implemented so that the shape could be selected from one-, two-, or four-dimensional tensors according to the paper.</p>
<p>Activation is binarized by
Ab = Sign ( Htanh( PReLU( Af + b0 ) + b1 ) )
Af = Ab
where Ab is the value of the binarized activation and Af is the full precision activation, , b0, b1 are channel-wise, and is a scalar arbitrary parameter.
Htanh backpropagation was implemented as follows [3]:</p>
<p><img src="/images/posts/news/summer-internship-2022-nishio-go-beyond-the-accuracy-of-the-float-model-with-binary-neural-networks/1b73ab39-gazou-.png" alt="数式"></p>
<p>This implementation reduces the mismatch in the gradient of the backpropagation of the sign function.</p>
<h2 id="results-of-reproduction-experiments-with-cifar10">Results of Reproduction Experiments with CIFAR10</h2>
<p>The implemented BiNealNet was validated using CIFAR10, a typical classiﬁcation task.
The maximum number of training epochs was set to 400 and the batch size to 128, starting with a learning rate of 0.1, and then decaying using the learning rate scheduler with SGDR [4].</p>
<p>Validated ResNet accuracy with full precision activation and weights as a baseline, yielding 0.9479 as validation prediction accuracy.
Verified whether the prediction accuracy is equivalent to a full precision network without degradation when weights or activation are binarized. Thereafter, when binarizing weights and activations, the number of channels was set to 1.5 times the original ResNet, following the paper [1].</p>
<p>First, only the weight parameters were binarized to obtain a validation prediction accuracy of 0.9485. When binarizing the weights, the weight parameters of the first convolution were not binarized. (The same applies thereafter.)
We then binarized the activation and obtained a validation prediction accuracy of 0.922.
Finally, when both the weight parameters and activation were binarized, we obtained 0.911 as the validation prediction accuracy.</p>
<p>Full precision (baseline)
0.9479
Binarize Weight
0.9485
Binarize Activation
0.922
Binarize Weight &#x26; Binarize Activation
0.9112</p>
<p>Binarization of activation caused degradation of prediction accuracy and did not produce the results as in the paper.</p>
<p>For further verification, we changed the following three points.</p>
<ul>
<li>Change the kernel size of the convolution layer of downsampling from 1 to 3.</li>
<li>Clip the negative slope of PReLU so that it is not less than 0.</li>
<li>Move scaling parameters of and to after the convolution.</li>
<li>The initial earning rate of new auxiliary parameters is reduced by a factor of 10.</li>
</ul>
<p>In the original ResNet, the kernel size of the downsampling layer is 1, but it is known empirically that a kernel size of 1 is inaccurate when the convolution weights are binarized.</p>
<p>Kernel size of downsampling = 1
0.9112
Kernel size of downsampling = 3
0.9214</p>
<p>Based on this result, the kernel size of the downsampling layer was subsequently set to 3.</p>
<p>Next, the negative slope of PReLU was verified by replacing it with a function that was clipped so that the negative slope of PReLU was not less than zero. This was because BiNealNet implicitly assumed that the value of the negative slope of PReLU was non-negative.</p>
<p>Normal PReLU function
0.9214
Clipped PReLU function
0.9235</p>
<p>Thereafter, the clipped PReLU was used in the activation binarization.</p>
<p>Thirdly, the position of the scaling factor was verified. The scaling factors for the activation and convolution weights were moved after the convolution operation. This makes all the inputs to the convolution operation 1bit. Accuracy was significantly reduced.</p>
<p>Scaling before conv
0.9235
Scaling after conv
0.9076</p>
<p>Finally, we considered reducing the learning rate of the added arbitrary parameter to make the change smaller.</p>
<p>Same initial LR (0.1)
0.9235
Lower initial LR for new parameters (0.01)
0.9102</p>
<h2 id="summary-and-future-outlook">Summary and Future Outlook</h2>
<p>The BiNealNet paper was ambiguous in many of its descriptions, which was a barrier to validation. We attempted to replicate the paper by experimenting with a variety of patterns, but ultimately ended up with a validation prediction accuracy that was 0.0244 points lower than the full precision network for the CIFAR10 dataset.
Within this summer internship, I also worked on an experiment in object detection using BiNealNet and MSCOCO dataset, but I will not go into details.</p>
<p>BiNealNet is claimed to have produced accuracy comparable to 8-bit models in segmentation, super-resolution, and matching, so we would like to verify its effectiveness for tasks other than classiﬁcation and object detection.</p>
<h2 id="leapmind-is-a-great-place-to-work">LeapMind is a great place to work!</h2>
<p>I worked at LeapMind for less than two months as an intern and had a very enjoyable and productive time. My mentor and other team members were very helpful and polite with code reviews, and I learned a lot from them.
They were also very helpful in discussing technical difficulties with me to deepen my understanding. The company encouraged working remotely, but it was great that I could come to the office if I wanted.
LeapMind is a great place to work!</p>
<h2 id="references">References</h2>
<p>[1] G. Nie et al., arXiv [cs.CV] (2022), (available at <a href="http://arxiv.org/abs/2202.03716">http://arxiv.org/abs/2202.03716</a>).
[2] M. Courbariaux et al., arXiv [cs.LG] (2016), (available at <a href="http://arxiv.org/abs/1602.02830">http://arxiv.org/abs/1602.02830</a>).
[3] Z. Liu et al., arXiv [cs.CV] (2018), (available at <a href="http://arxiv.org/abs/1808.00278">http://arxiv.org/abs/1808.00278</a>).
[4] I. Loshchilov and F. Hutter, arXiv [cs.LG] (2016), (available at <a href="http://arxiv.org/abs/1608.03983">http://arxiv.org/abs/1608.03983</a>).</p>

Summer Internship 2022 – Nishio – Go beyond the accuracy of the Float model with Binary Neural Networks.

<p>Hello and welcome to my LeapMind Summer Intern summary post!</p>
<h2 id="about-myself">About Myself</h2>
<p>I'm a German-born, Spanish-raised computer scientist with a passion for Neural Networks and Natural Language Processing. I'm a master student at the University of Tokyo and got the opportunity to join LeapMind for a summer itern program. Feel free to follow me on <a href="https://github.com/mariomeissner">GitHub</a> or <a href="https://twitter.com/mario_meissner">Twitter</a>! I also have my own <a href="https://mariomeissner.github.io/">blog</a> where I (very) occasionally write some stuff.</p>
<p>Let's start by giving some background about the topic at hand before I show you the results I obtained!</p>
<h2 id="the-lottery-ticket-hypothesis">The Lottery Ticket Hypothesis</h2>
<p><img src="/images/posts/en/news/summer-internship-2021-mario-finding-lottery-tickets-via-gradient-descent/b7f71fe6--2021-09-24-17.50.45-660x350.png" alt="Image: The Story of the Lottery Ticket Hypothesis: Hidden Networks"></p>
<p>Model pruning is a method to reduce the number of active weights in a neural network, for the purpose of decreasing model size and induce stronger regularization. A recent research direction called <a href="https://arxiv.org/abs/1803.03635">The Lottery Ticket</a> Hypothesis talks about how it is possible to find subnetworks that perform equally well (or even better!) than their full-sized counterparts, while also converging faster. This idea has prompted a number of follow-up works on pruning and lottery tickets and their relationship with the nature of how neural networks learn.</p>
<h2 id="finding-supermasks">Finding Supermasks</h2>
<p>One of the most notorious findings that spawned from the concept of Lottery Tickets is the supermask. This mightly creature is capable of pruning a randomly initialized model in such a way that it achieves full-sized model performance, but without ever changing the weights. Let that sink in for a moment! You randomly initialize the weights of a model, and then apply a certain mask, which zeroes-out a large amount of the weights. Without any weight training, this model performs extremely well on the test set of a certain task.</p>
<p>So what is this black magic? How do we obtain these supermasks? The answer is: gradient descent! Instead of calculating the gradients with respect to the weights, we compute the gradient on the mask and take a step towards a mask that reduces the target loss. Although applying a mask sounds like a non-differentiable task, via some tricks this is a totally possible. The algorithm that trains these masks is called edge-popup. Refer to the <a href="https://arxiv.org/abs/1911.13299">paper</a> and <a href="https://github.com/allenai/hidden-networks">its code</a> for details!</p>
<h2 id="supermasks-for-binarized-models">Supermasks for Binarized Models</h2>
<p>The <a href="https://arxiv.org/abs/2103.09377">Multi-Prize Lottery Ticket Hypothesis</a> paper talks about how we can find very robust binarized models via the supermask method. Models with binarized weights and activations are notoriously difficult to train, and here at LeapMind we wished to investigate the potential of this method to help us improve our own models.</p>
<h2 id="what-i-discovered">What I Discovered</h2>
<p><img src="/images/posts/en/news/summer-internship-2021-mario-finding-lottery-tickets-via-gradient-descent/6b400e0a--2021-09-24-8.36.36-660x350.png" alt="Image: Reproducing their results"></p>
<p>Just like in the Multi-Prize Lottery paper, a fully binarized lottery model remains 10%~20% below in accuracy against a standard weight-trained float model, especially for larger datasets such as ImageNet. However, comparing a binarized lottery model against a weight-trained binarized model is a different story! Here we can see extremely competitive results depending on the model configuration and dataset.</p>
<p>However, an important realization can be made to further improve results. The idea of freezing the weights and performing only mask-optimization is interesting from a theoretical point of view and helps these research papers get published, but when it comes down to the practical benefit, there is no reason for us to restrain ourselves in such a way! Why not combining the power of supermasks with the old-fashioned but well-known weight-training?</p>
<p>It turns out that supermask training in combination with normal weight-training acts like a strong regularizer, helping to improve generalization and avoid learning spurious correlations. Overall, we obtain much better training curves when training an unfrozen model with supermask training.</p>
<p>Additionally, assuming the existence of a powerful sparse-multiplication library, we can speed-up inference time of these models due to the fact that the weights are masked and thus very sparse.</p>
<h2 id="conclusions">Conclusions</h2>
<p>This summer internship was extremely exciting and I learned a ton of things. First and foremost, I got to dive deep into the theoretical guts of model pruning, lottery tickets and edge-popup supermask training. An additional, and in my opinion even more valuable experience was to see firsthand how a large, cutting-edge deep learning repository is organized and maintained. I learned how to <a href="https://techblog.leapmind.io/blog/boost-your-ai-research-with-hydra-and-lightning-templates/">structure complex research experiments</a> aiming for clarity and reproducibility. This will be my biggest takeaway for sure!</p>

Summer Internship 2021 – Mario – Finding Lottery Tickets via Gradient Descent!

<p>Hi there! My name is Oscar. I’m a summer intern at LeapMind and a Taiwanese student from Waseda University. I had an incredible time interning at this rapidly growing, multinational startup and would love to share my journey with you!</p>
<h2 id="making-deep-learning-prevalent">Making Deep Learning Prevalent</h2>
<p>This original title from LeapMind caught my eye during my internship hunting. With IoT becoming more common, it is refreshing to see this new concept of bringing deep learning to various devices. After learning about the company’s mission of embedded deep learning and its multilingual, open working environment, I knew that an internship at LeapMind is going to be unique for me. The experience of working at a Tokyo startup that designs hardware and software for deep learning is just an opportunity I cannot miss!</p>
<h2 id="internship-project-optical-flow-estimation-with-blueoil">Internship Project: Optical Flow Estimation with Blueoil</h2>
<p>The goal of my internship is developing a DNN for optical flow estimation with Blueoil and quantizing the network for edge devices to run. Blueoil is an open-source software from LeapMind that helps me apply quantize-aware training and convert a trained network into an executable program. By running this program on a CPU or FPGA-based device, embedded deep learning is achieved! 😎</p>
<p>I worked on this project with an intern partner, Inoue san, and my mentor, Iizuka san. My main task is designing the network for optical flow estimation using TensorFlow and verifying its performance after quantize-aware training. After surveying multiple related works and considering their implementation costs and compatibilities with Blueoil, we decided to implement and quantize FlowNetS, a famous DNN for learning optical flow, published in 2015.</p>
<p>This task might look simple at first glance. All I have to do is reproduce a DNN and apply quantize-aware training to it. In reality, there are numerous constraints and challenges:</p>
<p>In Blueoil, only a subset of TensorFlow operations in FlowNetS can be converted for edge devices, so I must replace many parts of the network with original architectures.
Blueoil uses extreme low-bit quantization (1-bit and 2-bit), which can significantly weaken the network's performance. Therefore, I need to consider different speed-accuracy trade-offs, e.g., which part of the network to quantize.
The dataset for optical flow estimation is large (30GB), and the total training time takes more than 2 weeks. It usually takes around 2 days to tell whether the network is learning effectively or not.</p>
<p><img src="/images/posts/en/news/deep-learning-internship-at-leapmind/4f216710-img_7065.jpg" alt="Image: Presentation"></p>
<h2 id="results--what-i-learned">Results &#x26; What I learned</h2>
<p>Despite all the difficulties mentioned above, I was able to overcome them thanks to the support and advice from Inoue san, Iizuka san, and multiple full-time members. In the end, I developed an original DNN, LmFlowNetS, and reduced its inference time by 53.14% on a CPU-based device with quantization! Inoue san and I even gave a live demo of this network during our final presentation. Through this experience, in addition to technical skills, I have also learned the following:</p>
<ul>
<li><strong>Take the initiative and don't hesitate to ask the experts.</strong></li>
<li><strong>Communicate and listen actively.</strong></li>
<li><strong>Observe carefully and experiment boldly.</strong></li>
</ul>
<p>To sum up, I had a rewarding experience working at LeapMind. The vibrant, open company culture really keeps me motivated every day. If you're interested in gaining an experience similar to mine, please don't hesitate to apply!</p>
<p>Our Source Code: [<a href="https://github.com/OscarWang114/blueoil/tree/lm-flownet-s">https://github.com/OscarWang114/blueoil/tree/lm-flownet-s</a>]
Final Presentation Slides: [<a href="https://www.slideshare.net/leapmind/final-presentation-optical-flow-estimation-with-dl-183441802">https://www.slideshare.net/leapmind/final-presentation-optical-flow-estimation-with-dl-183441802</a>]</p>
<p><img src="/images/posts/en/news/deep-learning-internship-at-leapmind/f405f06d-img_7117.jpg" alt="Image: Presentation"></p>
<h2 id="bonus-what-i-love-at-leapmind">Bonus: What I love at LeapMind</h2>
<p><strong>Open culture🙌</strong>: Everyone at LeapMind is very approachable and transparent. I don’t need to hesitate to ask anyone questions, even the CTO.
<strong>Shuffle lunch😋</strong>: This regular event divides everyone randomly into groups for lunch. It’s the perfect chance to socialize with people from other teams!
<strong>Technology💻</strong>: I love the free FPGA boards for development and the object detection live demo at the office entrance!</p>

News