<p>We are going to give two research talks at <a href="https://www.ai-gakkai.or.jp/jsai2019/en/">the Annual Conference of the Japanese Society for Artificial Intelligence (JSAI)</a>, Niigata, held from June 4th to 7th, 2019.
We will open our booth at the exhibition hall and give orientation for recruiting summer interns, engineers, and engineering managers. Please feel free to stop by our booth.</p>
<p><strong><a href="#1-reducing-the-computations-in-convrnns">1. Reducing the computations in ConvRNNs</a></strong>
We will introduce a method to reduce the computational load of convolutional recursion network (ConvRNN) by 20–40% while keeping the performance degradation as small as 1–2%.</p>
<p><strong><a href="#2-final-sample-batch-normalization">2. Final sample batch normalization</a></strong>
We will present a new batch normalization method specialized for quantized, low-bit networks, designed by taking advantage of the behavior found in quantized networks.</p>
<h2 id="1-reducing-the-computations-in-convrnns">1. Reducing the computations in ConvRNNs</h2>
<p><strong>Official Information</strong>: <a href="https://confit.atlas.jp/guide/event/jsai2019/subject/2H4-E-2-05/tables?cryptoId=">Presentation information</a>
<strong>Presentation Date</strong>: Wed. Jun 5, 2019 4:40 PM - 5:00 PM
<strong>Presentation Venue</strong>: Room H (303+304 Small meeting rooms)
<strong>Presentation Number</strong>: International Session：[2H4-E-2-05]</p>
<p>D. Vazhenina and A. Kanemura, “Reducing the number of multiplications in convolutional recurrent neural networks (ConvRNNs),” Annual Conference of Japanese Society for Artificial Intelligence (JSAI), Niigata, Japan, 2019.
Deep learning has been applied to spatio-temporal modeling, such as human activity recognition (Fig. 1) and weather forecasting (Fig. 2), and expected to be useful for e.g., predicting future events. The most influential model for this purpose is ConvRNNs (convolutional recurrent neural networks) [1], which was an innovation that combines the temporal prediction ability of RNNs and efficient image processing with convolutional networks (Fig. 3). One technical issue in ConvRNN is that it requires huge computation, leading to difficulties in operating it in real-world applications such as video content description and activity recognition.</p>
<p>In our JSAI paper, we report that it is possible to reduce the computation load of ConvRNNs. The technique we employed was to replace convolutional operations in recurrent hidden connections with the Hadamard product to reduce the number of parameters and multiplications. We evaluate our proposal using the task of next video frame prediction and the Moving MNIST dataset. The proposed method requires 38% less multiplications and 21% less parameters compared to the fully convolutional counterpart. In price of the reduced computational complexity, the performance measured by for structural similarity index measure (SSIM) decreased by about 1.5%. We continue research and development to make ConvRNNs more practical to be used in various situations like in web apps or embedded systems. Please refer for details to our paper to be published on the JSAI website. <a href="https://confit.atlas.jp/guide/event-img/jsai2019/2H4-E-2-05/public/pdf?type=in">JSAI2019 Proceedings</a></p>
<p><img src="/images/posts/en/event/fa63899a-kth-dataset.jpg" alt="/images/posts/en/event/fa63899a-kth-dataset.jpg" title="Image sequences of human actions">
Figure 1: Image sequences of human actions (from the <a href="https://www.nada.kth.se/cvap/actions/">KTH dataset</a>).</p>
<p><img src="/images/posts/en/event/5a0e125f-radar-echo.jpg" alt="/images/posts/en/event/5a0e125f-radar-echo.jpg" title="Forecast of the rainfall intensity in a local region based on radar echo maps">
Figure 2: Forecast of the rainfall intensity in a local region based on radar echo maps (from the supplementary material of [<a href="https://papers.nips.cc/paper_files/paper/2015/hash/07563a3fe3bbe7e3ba84431ad9d055af-Abstract.html">1</a>]).</p>
<p><img src="/images/posts/en/event/dd804dbb-convrnn_final-v2.jpg" alt="/images/posts/en/event/dd804dbb-convrnn_final-v2.jpg" title="Architecture of a ConvRNN classifier">
Figure 3: Architecture of a ConvRNN classifier (adopted and modified from Figure 2 of [<a href="https://arxiv.org/pdf/1511.06432.pdf">2</a>]).</p>
<p>[1] S. Xingjian, Z. Chen, H. Wang, D. Yeung, W. Wong, and W. Woo. Convolutional LSTM network: A machine learning approach for precipitation nowcasting. In Neural Information Processing Systems (NIPS), 2015.</p>
<p>[2] Nicolas Ballas, Li Yao, Chris Pal, and Aaron Courville. Delving deeper into convolutional networks for learning video representations.  International Conference on Learning Representations (ICLR) , 2016.</p>
<h2 id="2-final-sample-batch-normalization">2. Final sample batch normalization</h2>
<p><strong>Presentation Date</strong>： Wed. Jun 5, 2019 4:40 PM - 5:00 PM
<strong>Presentation Venue</strong>： Room A (2F Main hall A)
<strong>Presentation Number</strong>： International Session：[2A4-E-2-05]</p>
<p>J. Nicholls and A. Kanemura, “Final sample batch normalization for quantized neural networks,” Annual Conference of Japanese Society for Artificial Intelligence (JSAI), Niigata, Japan, 2019.
Batch normalization is a core invention in deep learning, along with ReLU, and this technique achieved below 5% top-5 error, improving over the winner of the most recent ImageNet competition of the time [<a href="https://arxiv.org/abs/1502.03167">3</a>]. Almost all modern deep learning models employ batch normalization. When we performed research for making compact neural networks, we encountered a strange behavior of batch normalization for quantized networks that was not observed for full-precision networks. This discovery led us to find a new technique on batch normalization.</p>
<p>In our recent work to be presented in JSAI 2019, we have found a nice technique, final batch normalization or BN-final, to further improve batch normalization for quantized neural networks. Instead of using population statistics, BN-final uses the batch statistics of only the final batch to create the fixed parameters for inference. We undertook a range of experiments including classification and object detection tasks, to verify the usefulness of BN-final. The figure below illustrates this, for the final accuracy; we can see that BN-final performed better in all of the four datasets and was most effective for the CIFAR-10 dataset, where the accuracy improvement was improved about 9 points (9%) from 78% to 87% (Fig. 4). We also find that the test accuracy is smoother and has greater value during throughout training. Please refer for details to our paper to be published on the JSAI website. [<a href="https://confit.atlas.jp/guide/event-img/jsai2019/2A4-E-2-05/public/pdf?type=in">JSAI2019 Proceedings</a>]</p>
<p><img src="/images/posts/en/event/f9762b4e-quantizedbncompare.png" alt="/images/posts/en/event/f9762b4e-quantizedbncompare.png" title="BN-final (blue) outperforms BN-pop (green), the standard batch normalization method">
Figure 4: BN-final (blue) outperforms BN-pop (green), the standard batch normalization method.</p>
<p>[<a href="https://arxiv.org/abs/1502.03167">3</a>] S. Ioffe and C. Szegedy,  “Batch normalization accelerating deep network training by reducing internal covariate shift”, International Conference on International Conference on Machine Learning (ICML), Lille, France, 2015.</p>
<h3 id="announcement-of-company-exhibition-at-jsai-conference">Announcement of company exhibition at JSAI conference</h3>
<p>As a conference sponsor, we will set up a booth at the venue from June 4th to June 7th. At the booth, we will introduce our latest information and job opportunities (including summer interns, engineers, and engineering managers). Please stop by our booth during your participation. We are looking forward to your visit.</p>
<h3 id="venue">Venue</h3>
<p><a href="https://www.tokimesse.com/">TOKI MESSE, Niigata Convention Center</a>
Exhibition Hall A：1F 展示ホールB　A-８</p>
<h3 id="date-and-time">Date and time</h3>
<p>June 4th (Tue), 13:00～17:00
June 5th (Wed), 9:00～17:00
June 6th (Thu), 9:00～17:00
June 7th (Fri), 9:00～15:00</p>
<h2 id="summer-interns-wanted">Summer interns Wanted!</h2>
<p>We are looking for students who are interested in Deep learning for LeapMind's summer internship project 2019!</p>
<p>Notice for international students:</p>
<p>You can apply to "LeapMind Summer Internship Program" if you already have visa eligibility to work as an intern in Japan.
Even if you don’t have visa eligibility, you can apply to the regular internship program, which is open all year around. If you are interested, please check here.
LeapMind is looking for motivated students for the summer internship program. At LeapMind, we aim to implement deep learning in every object and to realize a more convenient and rich society. To this end, we develop open source software Blueoil for creating low-bit compact neural networks and design <a href="https://monoist.itmedia.co.jp/mn/articles/1902/28/news048.html">processors</a> for deep learning which operate with low power consumption. Would you like to join Blueoil development, processor design, or other projects, and be a part of bringing advanced technology to society?</p>

Two presentations at JSAI 2019

<p><strong>August 1, 2023 – Tokyo Japan,</strong> <a href="https://leapmind.io/en/">LeapMind Co., Ltd.,</a> a leading creator of the standard in edge artificial intelligence (AI) announced today that it has achieved an industry-leading power efficiency of 107.8 TOPS/W*1, 2 with “Efficiera”, an ultra-low power AI accelerator IP using the company's core technology, “Extremely low bit quantization”. When comparing this to similar devices in today’s market, 1 ~ 5 TOPS/W in the case of GPUs and about 20 TOPS/W for AI accelerators for edge devices, this means that Efficiera performance would be approximately 5~100 times higher than these devices at the same power consumption.</p>
<h1 id="hardware-challenges-with-increasing-size-of-ai-tasks">Hardware challenges with increasing size of AI tasks</h1>
<p>In recent years, as the execution of AI tasks has become practical on various devices, AI tasks have become larger and more complex year by year. According to the analysis by OpenAI*3, the amount of AI computation used has dramatically increased at a rate of 2x in 3-4 months since 2012 (Moore's Law is 2x in 2 years), and it is worth preparing for the systems far beyond current capabilities. The analysis also notes that the processing of such AI tasks has been driven by custom hardware such as GPUs and/or more chips in parallel. This could indicate two technical challenges for the practical application of edge AI. First, as AI tasks become increasingly complex and larger in scale, the computational performance required during AI training and inference is much higher than before. The second is the challenge regarding the power consumption of devices with higher-performance AI chips processing complex AI tasks. The power available for an AI chip to process AI tasks stably in smartphones and AR glasses is less than 1W and around 25W for autonomous driving, where advanced AI processing is required. This means, edge AI also requires high power efficiency.（Figure 1)</p>
<p><img src="/images/posts/news/230801/01.png" alt="Diagram of AI tasks and performance requirements">
Figure1　AI task and required performance</p>
<p><img src="/images/posts/news/230801/02-en.png" alt="Power Efficiency Comparison Chart">
Reference：Power efficiency comparison (Based on in-house research)</p>
<h1 id="low-bit-quantization-technology-and-a-solution-for-enabling-edge-ai-with-efficiera">Low-bit quantization technology, and a solution for enabling edge AI with “Efficiera”</h1>
<p>Deep learning-based computations are usually performed with 32 bit floating point (FP32) to ensure accuracy. However, this method involves repeated matrix operations (sum-of-products operations) and requires a lot of computational resources, and several important requirements must be met in both software and hardware to perform AI tasks on edge devices, where memory and power are limited and real-time execution is required. The followings are some of the key requirements to be met.
First of all, software requirements include lightweight models, model accuracy comparable to FP32, and model conversion to maximize the efficiency of AI accelerator execution. Second, hardware requirements include low power consumption, high performance for executing various AI tasks, less memory bandwidth, and less memory usage.</p>
<p>To enable edge AI, all of these requirements must be addressed simultaneously, but a general approach is to first use a technique called quantization to lightweight the model. Lightweight 8 bit quantization is commonly used for AI models running on edge devices such as smartphones, but even with this, the AI tasks viable for implementation are still limited to some tasks such as object detection and image processing of static images, from the standpoint of power consumption and heat dissipation. In order to implement various AI tasks, the challenge is that the power efficiency and hardware performance to support many of AI tasks are not sufficient for the scale of those tasks.</p>
<p>In response, LeapMind aims to enable practical AI on various devices, and to solve both software and hardware requirements with our strength, “Extremely low bit quantization” technology. As a part of our effortson the software side, we use this “Extremely low bit quantization” to reduce the size of the model, achieving an ultra-low bit quantization of 1-bit weight x 2-bit activation, and our patented proprietary quantization technology has brought us success in low-bit quantization with minimal degradation in accuracy. In addition, a model converter dedicated to "Efficiera" optimizes computing efficiency and memory transfer to maximize hardware execution efficiency. Furthermore, we have been developing the AI accelerator "Efficiera" as hardware for the efficient execution of Extremely low bit quantization models. Our Extremely low bit quantization technology enables to perform matrix operations with simple logic operations, thus providing high arithmetic efficiency in a small circuit area. Since the DRAM memory transfer data during inference is 1 or 2 bits, the data size is equivalent to that of the data compression, reducing memory usage and bandwidth. By using Extremely low bit quantization technology, it is now possible to improve power efficiency, arithmetic efficiency, and area efficiency while minimizing the accuracy degradation caused by quantization.</p>
<p>Matsuda, Soichi, Chief Executive Officer of LeapMind, says  “As one of the few companies in the world that specialize in research and development of quantization technology as a method of AI technology and in the design of semiconductors for AI accelerators developed, we have been engaged in research and development for many years through joint development with major companies with the aim of achieving the practical application of AI. The balancing of high power efficiency and AI task processing performance with AI accelerators is essential for the implementation of AI in society. And today, we have achieved a power efficiency of 107.8 TOPS/W, far exceeding that of conventional GPUs and NPUs in terms of power efficiency. I believe that we have achieved an important milestone in our efforts to bring AI to the public more broadly.  We will continue to strive to solve our customers' problems and pursue research and development to contribute to the evolution of AI technology”.</p>
<p><strong>Please refer to LeapMind website below for the details of Efficiera.</strong>
<a href="https://leapmind.io/en/products/efficiera-ip/">https://leapmind.io/en/products/efficiera-ip/</a></p>
<p><strong>Appendix</strong>
*1  Estimated values are based on logic synthesis results using Cadence Genus Synthesis Solution. Conditions are as follows.
・Calculated based on IP stand-alone (Not including power of other subsystems such as DRAMs, main bus processors, etc.)
・Process: 7nm
・Clock frequency：533MHz
・Power supply voltage: Center
・Temperature: 25℃
・MAC utilization: 75.8%
*2  TOPS/W: Processing performance per watt（TOPS: tera operations per second）This indicates power efficiency. The higher this number, the greater the amount of computation to be processed with low-power consumption.
*3 <a href="https://openai.com/research/ai-and-compute">https://openai.com/research/ai-and-compute</a></p>
<h2 id="about-leapmind-inc">About LeapMind Inc.</h2>
<p>Founded in 2012, LeapMind has been providing solutions mainly to the consumer electronics, automotive, and manufacturing industries, as one of the few companies in the world that provide “Extremely low bit quantization”  technology to downsize AI and proprietary AI accelerator semiconductor design, under the corporate mission of "To create innovative devices with machine learning and make them available everywhere.” In order to provide foundational technologies to enable the next generation of information devices, we are committed to the development of both the software and hardware necessary to create such devices.LeapMind's business is highly acclaimed and has raised a cumulative total of 4.99 billion yen in funding (as of June 2023). For more information, please visit corporate website (<a href="https://leapmind.io/en/">https://leapmind.io/en/</a>)</p>
<p>Head office: Shibuya Dogenzaka Sky Building 3F, 28-1 Maruyama-cho, Shibuya-ku, Tokyo 150-0044
Representative: Soichi Matsuda, CEO
Established: December 2012
URL：<a href="https://leapmind.io/en/">https://leapmind.io/en/</a></p>
<p>*LeapMind, Efficiera and logo are trademarks or registered trademarks of LeapMind Inc.
*The information in the press release is current as of the date of announcement.</p>
<p>Media Contact
Marketing and Communication Group, LeapMind Inc.
Phone: 813-6696-6267
Email: <a href="mailto:pr@leapmind.io">pr@leapmind.io</a></p>

LeapMind’s Ultra Low-Power AI accelerator IP “Efficiera”
Achieved industry-leading power efficiency of 107.8 TOPS/W

<p>January 31, 2023 - LeapMind Inc. (Shibuya-ku, Tokyo; CEO: Soichi Matsuda), a creator of the standard in edge AI, has celebrated its 10th anniversary of founding last December, and is pleased to announce that Mr. Takayoshi USHIKU has been appointed as a Technical Advisor since January, with the aim of further penetrating deep learning technology into the market.</p>
<h2 id="background">Background</h2>
<p>Since founded in 2012, LeapMind has been developing both software and hardware by using deep learning quantization technology for aiming machine learning into practical use. After 10 years since its founding, we’re seeing that LeapMind’s technology is steadily approaching the end-products that benefit people's lives; however, at the same time, feel that there are still more rooms to expand the understanding of deep learning technology usage, and needs more research and development and activities to raise awareness.
Mr.Ushiku is a computer vision and natural language processing researcher and has been dedicated to development of machine learning and related areas; written numerous papers and has been cited in international papers more than 5,000 times*1, currently teaching at multiple universities, serves as a member of many expert committees, including The Japanese Society for Artificial Intelligence. Also, since 2018, he has been active as a Principal Investigator at OMRON SINIC X Corporation since its establishment, to connect social issues to next-generation innovative technologies.
As a technical advisor, Mr. Ushiku will provide advice on the company's technology strategy for the next 10 years, and will also be involved in supporting the strengthening of our R&#x26;D organization.
Combining Mr. Ushiku's research with our technology related to image processing AI, we will strengthen technology research and development, accelerate commercialization, and promote the realization of social implementation of edge AI.</p>
<p>*1 : Google Scholar search result</p>
<p><img src="/images/posts/news/7912/5e0a0e5e-ushiku_insert.png" alt="Board member of LeapMind Inc." title="Board member of LeapMind Inc.">
(Board member of LeapMind Inc. From right CEO Matsuda, Mr. Ushiku, CTO Tokunaga)</p>
<h2 id="career-of-takayoshi-ushiku">Career of Takayoshi USHIKU</h2>
<ul>
<li>2013 Research Fellow, Japan Society for the Promotion of Science, Microsoft Research Redmond Intern</li>
<li>2014 PhD, Graduate School of Information Science and Technology, The University of Tokyo
Research Scientist, NTT Communication Science Laboratories</li>
<li>Research Scientist, NTT Communication Science Laboratories</li>
<li>2016 Associate Professor, Department of Mechano-Informatics, Graduate School of Information Science and Technology, The University of Tokyo</li>
<li>2018 Principal Investigator, OMRON SINIC X Corporation</li>
<li>2019 Chief Research Officer, Ridge-i Co., Ltd.</li>
<li>2022 Chief Executive Officer, Nine Bulls, LLC</li>
</ul>
<p>The research interests lie in cross-media understanding through machine learning, mainly for computer vision and natural language processing. His research results have received various awards; ACM Multimedia Grand Challenge Special Prize in 2011, ACM Multimedia Open Source Software Competition Honorable Mention in 2017, and NVIDIA Pioneering Research Awards in 2017 and 2018.</p>
<h2 id="comment-from-technical-advisor-mr-yopshitaka-ushiku">Comment from Technical Advisor Mr. Yopshitaka Ushiku</h2>
<p>I have thought for a long time that LeapMind is a standard-bearer in efforts to popularize neural networks on embedded devices in Japan. When it comes to enriching people's lives, simply making PCs and smartphones smarter is not enough, and it is a very difficult task to make even smarter those embedded devices which are already useful. On the other hand, because it is difficult, the meaning of performing neural network processing on embedded devices is extremely high, and the idea of ​​”using embedded technology to spread it widely and enrich people's lives” is quite important and meaningful.
It may sound grandiose but I want to support technological innovation with the big ambition of “run by the embedding on a large-scale base model, putting a huge network on the SoC which was thought to be “absolutely unworkable with embedding.” 　</p>
<h2 id="comment-from-cto-hiroyuki-tokunaga">Comment from CTO Hiroyuki Tokunaga</h2>
<p>LeapMind is currently developing an accelerator for neural networks, but the future of the highly intelligent computers would not be a simple extension of today's technology, and at some point it will require some kind of technological leap. In order to respond to such technological evolution, I am very pleased to have Mr. Ushiku as an advisor who has been involved in leading-edge research and development. His participation is definitely a great reassurance to draw our future picture.</p>
<h2 id="about-leapmind">About LeapMind</h2>
<p>LeapMind Inc. was founded in 2012 with the corporate philosophy of "bringing new devices that use machine learning to the world". Total investment in LeapMind to date has reached 4.99 billion yen (as of May 2021). The company's strength is in extremely low bit quantization for compact deep learning solutions. It has a proven track record of achievement with over 150 companies, centered in manufacturing including the automobile industry. It is also developing its Efficiera semiconductor IP, based on its experience in the development of both software and hardware.</p>
<p>Head office: Shibuya Dogenzaka Sky Building 3F, 28-1 Maruyama-cho, Shibuya-ku, Tokyo 150-0044
Representative: Soichi Matsuda, CEO
Established: December 2012
URL：<a href="https://leapmind.io/en/">https://leapmind.io/en/</a></p>
<p>*LeapMind, Efficiera and logo are trademarks or registered trademarks of LeapMind Inc.</p>

【Press release】Yoshitaka Ushiku appointed as Technical Advisor

<h2 id="my-summer-internship-experience-at-leapmind">My summer internship experience at LeapMind.</h2>
<p>Shibuya Scramble Crossing, the most iconic Tokyo city landscape. Every minute hundreds of people seamlessly pass by, eventually disappearing in an endless crowd. Could you count them all? Could you track them all down? Well, I can. And I learned this secret skill during my summer internship at LeapMind Inc.</p>
<h2 id="multi-object-tracking">Multi-object tracking</h2>
<p>The task is simple: given a video, track all its objects as they move. You can think of it more atomically: first, detect objects within a single frame, and then link those objects in between frames. Former is simply object detection, whereas the latter is called object re-identification. Ideally, we would like to process frames in an online manner, that is, without looking into future frames while processing the current one. This is a pretty reasonable and desired assumption, especially when it comes to live videos (think of CCTV footage).</p>
<p>As one might expect, recent deep learning models enable multi-object tracking – and <a href="https://motchallenge.net/results/MOT20/?det=Private">they are fairly good</a> at it. Unfortunately, they do not necessarily excel at speed. It is challenging to run a live demo smoothly, and a networking delay while streaming the video from a camera to a GPU makes it all hardly feasible. Overcoming the speed issue was a goal of my internship.</p>
<p><img src="/images/posts/en/news/how-i-became-a-spy/7442f705-image1.gif" alt="Image: Multi-object tracking"></p>
<p>A live object tracking demo running at 70 FPS, recorded at LeapMind office, Shibuya, Tokyo.</p>
<h2 id="accelerating">Accelerating</h2>
<p>We focused on a one state-of-the art multi-object tracking method. Having understood, analyzed and identified its bottleneck, we were able to run a live demo in average almost 5-times faster than a baseline – at 100 FPS at a peak. Noteworthy, we did not suffer from a significant accuracy drop.
Despite an outstanding performance, it’s not the end of further improvements. The model we ended up with can be potentially <a href="https://medium.com/@joel_34050/quantization-in-deep-learning-478417eab72b">quantized</a> (compressed) and embedded on a hardware using LeapMind technology.</p>
<p><img src="/images/posts/en/news/how-i-became-a-spy/08d7ac54-joniak4-660x440.png" alt="Image:Presentation"></p>
<h2 id="leapmind-is-a-place-to-work">LeapMind is a place to work</h2>
<p>I spent five weeks at LeapMind as an intern, and I did enjoy it. I was working with professionals on a challenging project that won’t be archived once I leave. The atmosphere here kept me always motivated and I had no problems communicating – LeapMind is fully bilingual.
Company response to COVID-19 is also worth praising, since it went entirely remote (although one can visit the office if necessary). Last but not least, the salary is competitive, even if you’re an intern.</p>
<p>LeapMind is a place to work!</p>
<p>Written by Przemyslaw Joniak, LeapMind Summer 2020 Intern.</p>

How I became a spy

外観検査向け異常検知手法に関する論文紹介: EfficientAD

<p>LeapMindの花元です。</p>
<p>異常検知、してますか？</p>
<p>学習済み深層学習モデルを使って外観検査向け異常検知をしたいけど、良い感じの特徴量が得られなくて困ること、長い人生で、たまにはありますよね。今日はそんなときに使える外観検査向け異常検知の手法である<a href="https://arxiv.org/abs/2111.07677">FastFlow</a>のアルゴリズムを紹介したいと思います。</p>
<h2 id="背景">背景</h2>
<p><a href="https://techblog.leapmind.io/blog/20220511-hanamoto-survey_on_ad_model/">以前の記事</a>で紹介したとおり、近年、外観検査向け異常検知の手法として、画像の特徴を得るために学習済み深層学習モデルを用いる手法が多く提案されています。これらの手法では、訓練データから学習済み深層学習モデルを用いて特徴量を取得し、推論時に対象となる画像から得られた特徴量が、訓練データにおける特徴量の分布からどの程度離れているかを、何らかの基準に基づいて算出します。例えば、<a href="https://arxiv.org/abs/2011.08785">PaDiM</a>では「何らかの基準」としてマハラノビス距離を用いますが、これは訓練データから得られる特徴量の分布が多変量正規分布に従うことを仮定しています。経験的に、多くのデータセットに対してこの仮定を置いて問題ないことが分かっていますが、訓練データの特徴量の分布が多変量正規分布から大きく外れるようなデータセットの場合、うまく動かない場合があります。</p>
<p>うまく動かない可能性があるデータセットの例として、訓練データに複数種類の画像がある場合が挙げられます。例えば訓練データに「赤りんご」と「洋梨」があり、「和梨」を異常として検知したいケースを考えます。この場合、訓練データに「赤りんご」と「洋梨」の2種類があるため、得られる特徴量の分布に多峰性が生じると考えられます。この場合、マハラノビス距離は訓練データの平均と共分散のみを用いて算出されるため、訓練データにほとんど出現しなかったような特徴量（例えば和梨）であったとしても距離が小さい（＝正常である）と計算されてしまうことがあります（図1）。</p>
<p><img src="/images/posts/techblog/20230120-introduction_to_fastflow/mahalanobis.png" alt="mahalanobis">
図1: マハラノビス距離を使った異常検知がうまくいく例（左図）とうまくいかない例（右図）。左図では、トレーニング画像にりんごのみを用いているとする。左図のように訓練データから得られた特徴量の分布が正規分布に近い場合、推定された確率密度関数が訓練データの分布に近い形になるため、高い異常検知性能が得られることが期待できる。一方で、右図のように、訓練データから得られた特徴量の分布に多峰性があるなど、正規分布から離れている場合、推定された確率密度関数が訓練データの分布と離れた形になり、高い異常検知性能が得られない場合がある。</p>
<p><a href="https://arxiv.org/abs/2111.07677">FastFlow</a>では、この問題に対処するために、Normalizing Flowと呼ばれる技術を用いて、訓練時に訓練データから得られた特徴量の分布を、扱いやすい分布（正規分布）に変換できるようなニューラルネットワークを作っていきます。</p>
<h2 id="アルゴリズム">アルゴリズム</h2>
<p><a href="https://arxiv.org/abs/2111.07677">FastFlow</a>は</p>
<ul>
<li>画像の特徴抽出</li>
<li>2D Flow Model</li>
</ul>
<p>の2点から構成されます。これらを順に説明していきます。</p>
<h3 id="画像の特徴抽出">画像の特徴抽出</h3>
<p><a href="https://arxiv.org/abs/2111.07677">FastFlow</a>では、他の学習済み深層学習モデルを用いる手法と同様に、様々な種類のバックボーンを特徴抽出器として用いることができます。論文中ではViTやResNetを用いて実験を行っています。</p>
<h3 id="2d-flow-model">2D Flow Model</h3>
<p>2D Flowでは画像を特徴抽出器に入力して得られる特徴量の空間から、その分布が扱いやすくなる分布（FastFlowでは正規分布）へ変換する写像<span class="math math-inline"><span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>f</mi></mrow><annotation encoding="application/x-tex">f</annotation></semantics></math></span></span>を学習します。ここで、画像を特徴抽出器に入力して得られる特徴量の空間<span class="math math-inline"><span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>X</mi></mrow><annotation encoding="application/x-tex">X</annotation></semantics></math></span></span>、変換後の空間を<span class="math math-inline"><span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>Z</mi></mrow><annotation encoding="application/x-tex">Z</annotation></semantics></math></span></span>とすると、これらの確率密度関数には以下の等式が成り立ちます。</p>
<div class="math math-display"><span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mtable rowspacing="0.16em" columnspacing="1em"><mtr><mtd class="mtr-glue"></mtd><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow><msub><mi>p</mi><mi>X</mi></msub><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo><mo>=</mo><msub><mi>p</mi><mi>Z</mi></msub><mo stretchy="false">(</mo><mi>z</mi><mo stretchy="false">)</mo><mi mathvariant="normal">∣</mi><mi>det</mi><mo>⁡</mo><mo stretchy="false">(</mo><mfrac><mrow><mi mathvariant="normal">∂</mi><mi>z</mi></mrow><mrow><mi mathvariant="normal">∂</mi><mi>x</mi></mrow></mfrac><mo stretchy="false">)</mo><mi mathvariant="normal">∣</mi></mrow></mstyle></mtd><mtd class="mtr-glue"></mtd><mtd class="mml-eqn-num"></mtd></mtr></mtable><annotation encoding="application/x-tex">\begin{equation}
 p_X(x) = p_Z(z)|\det(\frac{\partial z}{\partial x} )|
\end{equation}</annotation></semantics></math></span></div>
<p>上式の対数は以下の通り計算されます。</p>
<div class="math math-display"><span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mtable rowspacing="0.16em" columnspacing="1em"><mtr><mtd class="mtr-glue"></mtd><mtd><mstyle scriptlevel="0" displaystyle="true"><mtable rowspacing="0.25em" columnalign="right left" columnspacing="0em"><mtr><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow><mi>log</mi><mo>⁡</mo><msub><mi>p</mi><mi>X</mi></msub><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow><mrow></mrow><mo>=</mo><mi>log</mi><mo>⁡</mo><msub><mi>p</mi><mi>Z</mi></msub><mo stretchy="false">(</mo><mi>z</mi><mo stretchy="false">)</mo><mo>+</mo><mi>log</mi><mo>⁡</mo><mi mathvariant="normal">∣</mi><mi>det</mi><mo>⁡</mo><mo stretchy="false">(</mo><mfrac><mrow><mi mathvariant="normal">∂</mi><mi>z</mi></mrow><mrow><mi mathvariant="normal">∂</mi><mi>x</mi></mrow></mfrac><mo stretchy="false">)</mo><mi mathvariant="normal">∣</mi></mrow></mstyle></mtd></mtr><mtr><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow></mrow></mstyle></mtd></mtr><mtr><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow></mrow></mstyle></mtd></mtr><mtr><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="true"><mrow><mrow></mrow><mo>=</mo><mi>log</mi><mo>⁡</mo><msub><mi>p</mi><mi>Z</mi></msub><mo stretchy="false">(</mo><msub><mi>f</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo><mo stretchy="false">)</mo><mo>+</mo><mi>log</mi><mo>⁡</mo><mi mathvariant="normal">∣</mi><mi>det</mi><mo>⁡</mo><mo stretchy="false">(</mo><mfrac><mrow><mi mathvariant="normal">∂</mi><msub><mi>f</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo></mrow><mrow><mi mathvariant="normal">∂</mi><mi>x</mi></mrow></mfrac><mo stretchy="false">)</mo><mi mathvariant="normal">∣</mi></mrow></mstyle></mtd></mtr></mtable></mstyle></mtd><mtd class="mtr-glue"></mtd><mtd class="mml-eqn-num"></mtd></mtr></mtable><annotation encoding="application/x-tex">\begin{equation}
\begin{split}
\log p_X(x) &#x26;= \log p_Z(z) + \log |\det(\frac{\partial z}{\partial x} )| \\\\\\
&#x26;=  \log p_Z(f_\theta(x)) + \log |\det(\frac{\partial f_\theta(x)}{\partial x} )|
\end{split}
\end{equation}</annotation></semantics></math></span></div>
<p>ここで、<span class="math math-inline"><span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>θ</mi></mrow><annotation encoding="application/x-tex">\theta</annotation></semantics></math></span></span>は2D Flow Modelのパラメータとします。学習時には、写像<span class="math math-inline"><span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>f</mi><mi>θ</mi></msub></mrow><annotation encoding="application/x-tex">f_\theta</annotation></semantics></math></span></span>を用いて得られた分布と目標とする分布（FastFlowでは正規分布）の距離（KL Divergence）を最小化するように学習します。この際、写像<span class="math math-inline"><span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>f</mi><mi>θ</mi></msub></mrow><annotation encoding="application/x-tex">f_\theta</annotation></semantics></math></span></span>の<span class="math math-inline"><span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>θ</mi></mrow><annotation encoding="application/x-tex">\theta</annotation></semantics></math></span></span>に関する微分とヤコビアン（<span class="math math-inline"><span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>det</mi><mo>⁡</mo><mo stretchy="false">(</mo><mfrac><mrow><mi mathvariant="normal">∂</mi><msub><mi>f</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo></mrow><mrow><mi mathvariant="normal">∂</mi><mi>x</mi></mrow></mfrac><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\det(\frac{\partial f_\theta(x)}{\partial x})</annotation></semantics></math></span></span>）を計算する必要が生じます。写像<span class="math math-inline"><span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>f</mi><mi>θ</mi></msub></mrow><annotation encoding="application/x-tex">f_\theta</annotation></semantics></math></span></span>の<span class="math math-inline"><span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>θ</mi></mrow><annotation encoding="application/x-tex">\theta</annotation></semantics></math></span></span>に関する微分は、写像<span class="math math-inline"><span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>f</mi><mi>θ</mi></msub></mrow><annotation encoding="application/x-tex">f_\theta</annotation></semantics></math></span></span>をニューラルネットワークで構築すれば、自動微分により計算することが可能です。一方で、ヤコビアンを愚直に計算しようとすると、行列の大きさを<span class="math math-inline"><span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>n</mi><mo>×</mo><mi>n</mi></mrow><annotation encoding="application/x-tex">n \times n</annotation></semantics></math></span></span>とした際に<span class="math math-inline"><span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>O</mi><mo stretchy="false">(</mo><msup><mi>n</mi><mn>3</mn></msup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">O(n^3)</annotation></semantics></math></span></span>の計算量を要してしまいます。これは計算効率が悪くなってしまうため、ヤコビアンが自明になるように<span class="math math-inline"><span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>f</mi><mi>θ</mi></msub></mrow><annotation encoding="application/x-tex">f_\theta</annotation></semantics></math></span></span>の構造を設定したものに<a href="https://arxiv.org/pdf/1410.8516.pdf">NICE</a>と呼ばれるネットワークがあり、FastFlowではこのネットワークを2D Flow Modelに採用しています。ネットワークの構造を以下に示します。</p>
<p><img src="/images/posts/techblog/20230120-introduction_to_fastflow/fastflow.png" alt="2D Flow Model">
図2: 2D Flow Modelの構造（論文より引用）</p>
<h2 id="性能比較">性能比較</h2>
<p>論文中では、MVTec ADを対象とした実験で、バックボーンにWide-ResNet50-2を用いた際の平均AUROCが99.3であり、これまでのSoTAである<a href="https://techblog.leapmind.io/blog/20220701-introduction_to_patchcore/">PatchCore</a>を、バックボーンにWide-ResNet50-2を用いて動作させた際のAUROCの99.1を上回っていると主張されています。<a href="https://github.com/openvinotoolkit/anomalib">ANOMALiB</a>による再実装に基づく実験では、FastFlowの平均AUROCが96.3、PatchCoreの平均AUROCが98.0となっており性能が逆転していますが、SoTAに近い異常検知性能が得られているとは言えそうです。</p>
<p>現在のSoTAとされるPatchCoreと性能の点で大きく異なる点は、トレーニング時間と推論時間です。FastFlowでは、トレーニング時にニューラルネットワークのパラメータを学習させる必要があります。一方で、PatchCoreでは特徴ベクトルを抽出し、貪欲法に基づき特徴ベクトルのサブセットを抽出するだけでトレーニングが終了するため、一般にトレーニング時間はPatchCoreの方が速いと言えます。一方で、推論時においては、FastFlowはニューラルネットワークの計算をすることになるため、実行環境にこの計算を速く行う仕組みがあれば（例えばGPUを搭載している）速く推論することが可能です。一方で、PatchCoreでは推論時に、推論ごとに膨大な特徴ベクトルの中からk個の最近傍を得る必要があり、比較的遅くなります。実際、論文中では、FastFlowは、GPUが搭載されている環境において、特徴抽出後の処理時間が画像1枚あたり100ms程度、PatchCoreより速いと主張されています。従って、トレーニング時間と推論時間のどちらを重視するのか、また、推論する際の実行環境に応じて使い分けるのが良いと考えられます。</p>
<h2 id="まとめ">まとめ</h2>
<p>この記事では、学習済み深層学習モデルを用いた外観検査向け異常検知手法であるFastFlowを紹介しました。この手法では、Normalizing Flowと呼ばれる、解析的に表現することが難しい特徴量の分布を解析的に表現することが容易な分布（FastFlowにおいては正規分布）へと変換するニューラルネットワークを用いることで、正常画像における特徴量の分布が複雑な場合においても異常検知の精度を高めることを図っています。</p>
<p>ところで、LeapMindでは異常検知モデルだったり、異常検知とは全く関係ないモデルだったりを開発する機械学習エンジニアを絶賛募集中です！興味のある方は以下からぜひご応募ください！</p>
<p><a href="https://leapmind.io/careers/">https://leapmind.io/careers/</a></p>

外観検査向け異常検知手法に関する論文紹介: FastFlow

<h2 id="概要">概要</h2>
<p>LeapMindの高橋です。現在Efficieraで提供する極小量子化された物体検出(Object Detection)モデルを開発しています。</p>
<p>物体検出モデルはComputer Visionの中でもメジャーな応用で注目度が高い一方、少し調べてみると「アンカーボックス」や「特徴マップピラミッド」など物体検出タスクに特有の概念がいくつも登場するため、とっつきにくい印象を持つかもしれません。</p>
<p>今回取り上げるAnchor Matchingはそんな物体検出モデルに特有な概念の一つで、ここ3年程で研究が特に発展してきたものです。Anchor Matchingについてそれほど日本語の記事はヒットしないのですが、これが分かれば物体検出モデルの全体像も把握しやすいのではないかと思い取り上げます。</p>
<h2 id="物体検出モデルの概略">物体検出モデルの概略</h2>
<p>Anchor Matchingを導入するために、物体検出モデルについて簡単におさらいしましょう。</p>
<p>物体検出タスクは入力画像に含まれる物体の座標(localization)と種類(classification)を同時に推定し、後処理を行い検出結果としてバウンディングボックス:(center_x, center_y, width, height, score, class_id)のリストを出力します。</p>
<p>CNNによる典型的なODモデルのネットワーク構造は以下の3つのサブモジュールから構成されています。</p>
<ol>
<li>Backbone: ResNet50などで入力画像から特徴量を抽出する。</li>
<li>Neck: Backboneから複数スケールの特徴マップを得て変換する。</li>
<li>Head: Neckで得た各特徴マップごとに最終的なバウンディングボックスの出力に合わせて変換する。</li>
</ol>
<p>図示すると次のようになります。</p>
<p><img src="/images/posts/techblog/20220824-shtakahashi-anchor_matching/fpn.png" alt="FPN">
(図1. 文献2より加工して引用。Feature Pyramid Networks (FPNs)を採用した典型的なODモデルの例。)</p>
<p>図1の左下オレンジに囲われた入力画像から上方向への処理がBackboneに相当します。上に行くことにダウンサンプリングされ空間スケールが粗い特徴量が得られます。次にNeckモジュールでは、Feature Pyramid Networks (FPNs)や類似したNeck構造を採用した場合、Backboneの各空間解像度の特徴量とNeckモジュールから得られた特徴量を組み合わせて、複数スケールからなる特徴マップピラミッドを構成します。最後にHeadモジュールでNeckから得られた特徴マップごとに最終的な出力をします。</p>
<p>Headから得られる最終的なネットワークの出力は、(batch_size, num_predictors, head_dim)となります。</p>
<p>ここでは、1つのバウンディングボックスに対応する出力のことをpredictorと呼び、その数をnum_predictorsとします。また、head_dimはclassificationとlocalizationに必要な情報、つまりCOCO datasetであればクラス数の80次元に加えてboxを表現する4次元が必要になります。
加えて、例えば以下で紹介するYOLOv2/v3の場合には、ネットワークの予測と正解間のIoUを推定するobjectivenessと呼ばれるパラメータが導入されているためhead_dim=85となります。</p>
<p>今回はAnchor Matchingを取り上げるため、以下Anchor-basedのモデルの場合のみを考えます。
Anchor-basedモデルでは各Predictorには対応するanchor boxが事前に定義されています。
図示したものが以下の図2になります。</p>
<p><img src="/images/posts/techblog/20220824-shtakahashi-anchor_matching/anchor.png" alt="Anchor"></p>
<p>(図2.) 2x2のcell(特徴マップの一領域)に設定されたanchor boxesの例。</p>
<p>図2は簡単のために2x2のcell(特徴マップの一領域)があり、左上のcellに3つの縦長/横長/正方形のanchorを描画しました。実際にはすべてのcellに同様のanchorが設定されています。
実際のモデル、例えば以下のように、512x512の入力画像が与えられたとき、cellの数はP3:12288 P4:3072, P5: 768で合計16128個、それぞれに3つのanchor boxを割り当てるとnum_predictorsは16128*3=48384個になります。</p>
<pre><code class="hljs language-python">image_width, image_height = <span class="hljs-number">512</span>, <span class="hljs-number">512</span>
num_anchors_per_cell = <span class="hljs-number">3</span>
p3_stride, p4_stride, p5_stride = <span class="hljs-number">8</span>, <span class="hljs-number">16</span>, <span class="hljs-number">32</span>
p3_num_predictor =
  (image_width // p3_stride)
  * (image_height // p3_stride)
  * num_anchors_per_cell <span class="hljs-comment"># 36864</span>
p4_num_predictor = 
  (image_width // p4_stride)
  * (image_height // p4_stride)
  * num_anchors_per_cell <span class="hljs-comment"># 9216</span>
p5_num_predictor =
  (image_width // p5_stride)
  * (image_height // p5_stride)
  * num_anchors_per_cell <span class="hljs-comment"># 2304</span>
num_predictors = 
  p3_num_predictor
  + p4_num_predictor
  + p5_num_predictor <span class="hljs-comment"># 48384</span>
</code></pre>
<p>どの物体とどのpredictorを対応させるかは自明ではありません。この問題をAnchor Matching、あるいはAnchor AssignmentやLabel Assignmentと呼びます。</p>
<h2 id="anchor-matching">Anchor Matching</h2>
<p>Anchor Matchingのアルゴリズムは、与えられたバウンディングボックスのリストをモデルのpredictorに割り当てます。
各predictorは次の3つの状態のいずれかになります。</p>
<ul>
<li>positive: predictorに特定の物体が割り当てられている</li>
<li>negative: predictorにいずれの物体も割り当てられていない</li>
<li>ignore: predictorにいずれの物体も割り当てられないがnegativeとして学習しない</li>
</ul>
<p>では具体的にどのようなアルゴリズムでマッチングを行うかみていきましょう。</p>
<h3 id="yolov2yolov3">YOLOv2/YOLOv3</h3>
<p>YOLOv2/YOLOv3では、それぞれの物体に対して各feature mapごとにpredictorを最も適した1つだけ割り当てます。このアルゴリズムを我々はMax Matching / Max Matcherと呼んでいます。</p>
<p>具体的には次の2つの条件から最適な1つを選びます。</p>
<ol>
<li>cellに対応するピクセルに物体のcx, cyが含まれる</li>
<li>1で該当したcell内のanchorのうち物体とのIoUが最も高い</li>
</ol>
<p>擬似コードに書き直すと次のようになります。</p>
<pre><code>Max Matcherによる単一の物体に対する割り当て
class MaxMatcher(torch.nn.Module):
  def __init__(self):
    super().__init__()
    self.strides = torch.Tensor(
        [8, 16, 32], dtype=torch.long
    )
    self.anchors = torch.Tensor(
      [[4.0, 4.0], [6.0, 3.0], [3.0, 6.0]],
      dtype=torch.float,
    )
    self.num_anchors = 3
    self.image_width, self.image_height = 512, 512

  def forward(self, gt_label):
    positive_predictors = []
    cx, cy, width, height, class_id = gt_label
    for stride in strides:
      cell_in_row = (image_width // stride)
      cell_index_x = cx // stride
      cell_index_y = cy // stride
      cell_index =
        cell_index_x
        + cell_index_y * cell_in_row
      anchor_gt_ious = compute_ious(
        self.anchors, gt_label[:4]
      )
      anchor_index = torch.argmax(anchor_gt_ious)
      predictor_index = 
        cell_index * self.num_anchors + anchor_index
      positive_predictors.append(predictor_index)
    return positive_predictors
</code></pre>
<p>cell_indexが条件1に対応し、 anchor_indexが条件2に対応します。</p>
<p>上の擬似コードでは省かれていますが、もしpositive_predictors以外が推論したバウンディングボックスと物体のIoUが設定した閾値以上だった場合は、そのpredictorはnegativeからignoreに割り当てなおします。</p>
<h4 id="max-matchingの欠点">Max Matchingの欠点</h4>
<p>YOLOv2/YOLOv3は偉大なモデルですが、Anchor Matchingの観点からみるといくつか欠点があります。</p>
<ol>
<li>同じpredictorに複数のgt_boxが該当する場合に、学習時に1つのgt_boxしかないものとして学習される</li>
<li>各特徴マップごとに1つしか、正例/負例の不均衡(Imbalance Problems)が生じ学習が非効率的になる</li>
</ol>
<p>ところで、YOLOv2/YOLOv3でなぜ暗黙のうちにMax Matchingが採用されているかというと、出力のcenter_x, center_yにsigmoid nonlinearityを適用しているため、表現できるboxの中心座標がcellの内部に限られているためです。以下で紹介するRetinaNetにはこのような制約がありません。</p>
<h3 id="retinanet">RetinaNet</h3>
<p>RetinaNetは最も有名な物体検出モデルの一つです。論文の題が"Focal Loss for Dense Object Detection"とあるように、Classification部分にCross Entropy Lossから新規に提案したFocal Lossに変更して、YOLOv3などにあったobjectivenessパラメータを導入せずlocalization+classificationのみのlossで高精度のモデルを学習できることを明らかにしました。Focal Lossの詳細には立ち入りませんが、Cross Entropyと比べて正例/負例の不均衡を損失関数の重み付けを自動的に適切に行うことで、不均衡問題を緩和しています。</p>
<p>Anchor Matching部分はRetinaNet論文の主題ではないのですが、RetinaNetではYOLOv2/v3とは異なりanchor boxと物体間のIoUがそれぞれの閾値を基準にpredictorをpositive/ignore/negativeに割り当てるアルゴリズムを採用しています。閾値に基づいた割り当て方法なので、我々はこれをThresholdMatcherと呼んでいます。</p>
<p>Pythonの疑似コードで書いてみると次のようになります。</p>
<pre><code>Threshold Matcherによる単一の物体に対する割り当て
class ThresholdMatcher(torch.nn.Module):
  def __init__(
    self,
    positive_threshold=0.5,
    negative_threshold=0.4
  ):
    super().__init__()
    self.positive_threshold=0.5
    self.negative_threshold=0.4
    self.strides = torch.Tensor(
      [8, 16, 32], dtype=torch.long
    )
    self.anchors = torch.Tensor(
      [[4.0, 4.0], [6.0, 3.0], [3.0, 6.0]],
      dtype=torch.float
    )
    self.image_width, self.image_height = 512, 512
    self.anchor_boxes = generate_anchor_boxes(
      self.strides,
      self.anchors,
      self.image_width,
      self.image_height
    )

  def forward(self, gt_label):
    anchor_gt_ious = compute_ious(
      self.anchor_boxes, gt_label
    )
    positive_predictors = torch.nonzero(
      anchor_gt_ious >= self.positive_threshold
    )
    negative_predictors = torch.nonzero(
      anchor_gt_ious &#x3C;= self.negative_threshold
    )
    ignore_predictors = torch.nonzero(
      torch.logical_and(
        anchor_gt_ious &#x3C; self.positive_threshold,
        anchor_gt_ious > self.negative_threshold,
      )
    )
    return positive_predictors, negative_predictors, ignore_predictors
</code></pre>
<p>実装は比較的簡単で、predictorのanchorをあらかじめ用意し、それとgtとの間のIoUを計算し条件式で分類します。</p>
<p>論文中ではこの点に対する言及はないのですが、1つの物体に対して各特徴マップごとに1つしか正例がないYOLOv3のMaxMatcherに比べて、正例/負例の不均衡がAnchor Matchingの面からみて緩和されている、という点は指摘できるでしょう。</p>
<h4 id="threshold-matcherの欠点">Threshold Matcherの欠点</h4>
<p>Threshold MatcherはMax Matcherと比較して、1つの物体に複数のpredictorを割り当てるため、不均衡問題は緩和されました。しかし、次の問題を指摘することができます。</p>
<ol>
<li>大きい物体に対してはpositive predictorが多くなるが、小さい物体に対しては少なくなる。</li>
<li>閾値の設定が恣意的で精度への影響が大きい。</li>
</ol>
<p>こうした問題をAnchor Matchingのアルゴリズムから解決したものがこれから紹介するATSSになります。</p>
<h3 id="adaptive-training-sample-selection-atss">Adaptive Training Sample Selection (ATSS)</h3>
<p>RetinaNetが公開されて以降、YOLO系とは異なった1段階モデルが数多く提案されるようになりました。そのうちの一つがFCOS: Fully Convolutional One-Stage Object Detectionというもので、1段階モデルとして既存のモデルから大幅な精度向上を達成しました。</p>
<p>今回取り上げる、Adaptive Training Sample Selection (適応的学習サンプル選択)論文ではRetinaNetとFCOSの精度の差異がどの要素によって生じているか実験から明らかにし、これがAnchor Matchingによって生じていると議論しました。</p>
<blockquote>
<p>After applying those universal improvements, these are only two differences between the anchor-based RetinaNet (#A=1) and the anchor-free FCOS. One is about the classification sub-task in detection, i.e., the way to define positive and negative samples. Another one is about the regression sub-task, i.e., the regression starting from an anchor box or an anchor point.</p>
<p>意訳) これらの普遍的な改善手法を適用した結果、anchor-basedなRetinaNetとanchor-freeなFCOSには2つのみ違いがあります。1つが分類のサブタスクについてで、すなわちpositive sampleとnegative sampleをどのように定義するかです。もう1つが回帰のサブタスクで、回帰をanchor boxから行うかanchor pointから行うかという違いです。</p>
<p>...</p>
<p>According to these experiments conducted in a fair way, we indicate that the essential difference between one-stage anchor-based detectors and center-based anchorfree detectors is actually how to define positive and negative training samples, which is important for current object detection and deserves further study.</p>
<p>意訳) 公正に行われたこれらの実験によれば、1段階anchor-based検出器とcenter-based anchor-free検出器の違いはpositiiveとnegativeな学習サンプルをどのように適用するかで、これは現在の物体検出モデルにとって重要でさらなる研究が求められています。</p>
</blockquote>
<p>論文ではここからアルゴリズムの導入に入ります。ATSSのアルゴリズムは必ずしも直感的に導かれたものではないですが、論文にも書かれている次の5つの条件を満たすという観点から天下り式に設計されたものです。</p>
<ol>
<li>anchor boxとgt boxの中央座標の距離を用いること</li>
<li>anchor-gt間のIoUを用いること</li>
<li>anchorの中央座標が物体の内側に含まれていること</li>
<li>物体の大きさによってpositive predictorの数が大きく影響されないこと</li>
<li>ハイパーパラメタが少ないこと</li>
</ol>
<p>実際のアルゴリズムは擬似コードでは簡潔に書きにくかったので、ここではアルゴリズムの手続きを書きました。</p>
<p><strong>ATSSのアルゴリズム</strong></p>
<ol>
<li><code>すべてのanchor-gt間のIoUを計算する</code></li>
<li><code>1と同様にanchor-gt間の中心座標の距離を計算する</code></li>
<li><code>L個の特徴マップについて、それぞれのgtについて中心座標が最も近いK個のanchor boxを取得する</code></li>
<li><code>3で得たK*L個のanchor boxのIoUを計算し、その平均+標準偏差を閾値とする</code></li>
<li><code>IoUが閾値以上のanchor-gtペアについて、predictorに物体を割り当てる</code></li>
<li><code>5でpositiveに割り当てられたもののうちanchor boxの中央が物体の内側に含まれていないものを省く</code></li>
<li><code>predictorに複数の物体が割り当てられた場合、IoUが最も高い物体を割り当てる。</code></li>
</ol>
<p>登場するハイパーパラメタはKのみで論文中で最適値はK=9とされているもののK>=5であればモデルの精度に与える影響は軽微だと報告されています。</p>
<p>ResNet-101をbackboneとしたモデルでは、RetinaNetにATSSを導入することでMS COCO test-devでAPが39.1->43.6と4.5ptの大幅な上昇を達成したと記述されています。
推論時の計算量を増加させずに学習時のコストのみでこれだけ向上でき、これはモデル開発に大きなインパクトがあります。</p>
<h2 id="終わりに">終わりに</h2>
<p>物体検出は非常にモデル設計の自由度が高いタスクです。Anchor Matchingはそのうちの一要素で、効率的にモデルを学習させるために重要な要素だとの認識が近年高まってきました。
直近ではProbabilistic Anchor Assignment (PAA)、YOLO-Fで導入されたUniform Matching、最適輸送問題として定式化したOptimal Transport Assignment (OTA)などが提案されており刺激的な領域です。また機会があれば、これらを紹介します。</p>
<p>ところでLeapMindでは物体検出モデルを始めとした、Computer Vision分野のモデル開発する機械学習エンジニアを絶賛募集中です！興味のある方は以下からぜひご応募ください！</p>
<p><a href="https://leapmind.io/careers/">https://leapmind.io/careers/</a></p>
<h2 id="文献">文献</h2>
<ol>
<li>Qiang Chen, Yingming Wang, Tong Yang, Xiangyu Zhang, Jian Cheng and Jian Sun. You Only Look One-level Feature. CVPR 2021.</li>
<li>Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, Serge Belongie. Feature Pyramid Networks for Object Detection. CVPR 2017.</li>
<li>Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal Loss for Dense Object Detection. ICCV 2017.</li>
<li>Joseph Redmon and Ali Farhadi. YOLO9000: Better, Faster, Stronger. CVPR 2017.</li>
<li>Joseph Redmon and Ali Farhadi. YOLOv3: An Incremental Improvement. arXiv:1804.02767 2018.</li>
<li>Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. FCOS: Fully Convolutional One-Stage Object Detection. ICCV 2019.</li>
<li>Shifeng Zhang, Cheng Chi, Yongqiang Yao, Zhen Lei, and Stan Z. Li. Bridging the Gap Between Anchor-based and Anchor-free Detection via Adaptive Training Sample Selection. CVPR 2020.</li>
</ol>

The New Standard for Edge AI

Efficiera IP

Efficiera Models

Efficiera SDK

CLIENTS

EVENTS

NEWS

TECH BLOG