<p><strong>August 1, 2023 – Tokyo Japan,</strong> <a href="https://leapmind.io/en/">LeapMind Co., Ltd.,</a> a leading creator of the standard in edge artificial intelligence (AI) announced today that it has achieved an industry-leading power efficiency of 107.8 TOPS/W*1, 2 with “Efficiera”, an ultra-low power AI accelerator IP using the company's core technology, “Extremely low bit quantization”. When comparing this to similar devices in today’s market, 1 ~ 5 TOPS/W in the case of GPUs and about 20 TOPS/W for AI accelerators for edge devices, this means that Efficiera performance would be approximately 5~100 times higher than these devices at the same power consumption.</p>
<h1 id="hardware-challenges-with-increasing-size-of-ai-tasks">Hardware challenges with increasing size of AI tasks</h1>
<p>In recent years, as the execution of AI tasks has become practical on various devices, AI tasks have become larger and more complex year by year. According to the analysis by OpenAI*3, the amount of AI computation used has dramatically increased at a rate of 2x in 3-4 months since 2012 (Moore's Law is 2x in 2 years), and it is worth preparing for the systems far beyond current capabilities. The analysis also notes that the processing of such AI tasks has been driven by custom hardware such as GPUs and/or more chips in parallel. This could indicate two technical challenges for the practical application of edge AI. First, as AI tasks become increasingly complex and larger in scale, the computational performance required during AI training and inference is much higher than before. The second is the challenge regarding the power consumption of devices with higher-performance AI chips processing complex AI tasks. The power available for an AI chip to process AI tasks stably in smartphones and AR glasses is less than 1W and around 25W for autonomous driving, where advanced AI processing is required. This means, edge AI also requires high power efficiency.（Figure 1)</p>
<p><img src="/images/posts/news/230801/01.png" alt="Diagram of AI tasks and performance requirements">
Figure1　AI task and required performance</p>
<p><img src="/images/posts/news/230801/02-en.png" alt="Power Efficiency Comparison Chart">
Reference：Power efficiency comparison (Based on in-house research)</p>
<h1 id="low-bit-quantization-technology-and-a-solution-for-enabling-edge-ai-with-efficiera">Low-bit quantization technology, and a solution for enabling edge AI with “Efficiera”</h1>
<p>Deep learning-based computations are usually performed with 32 bit floating point (FP32) to ensure accuracy. However, this method involves repeated matrix operations (sum-of-products operations) and requires a lot of computational resources, and several important requirements must be met in both software and hardware to perform AI tasks on edge devices, where memory and power are limited and real-time execution is required. The followings are some of the key requirements to be met.
First of all, software requirements include lightweight models, model accuracy comparable to FP32, and model conversion to maximize the efficiency of AI accelerator execution. Second, hardware requirements include low power consumption, high performance for executing various AI tasks, less memory bandwidth, and less memory usage.</p>
<p>To enable edge AI, all of these requirements must be addressed simultaneously, but a general approach is to first use a technique called quantization to lightweight the model. Lightweight 8 bit quantization is commonly used for AI models running on edge devices such as smartphones, but even with this, the AI tasks viable for implementation are still limited to some tasks such as object detection and image processing of static images, from the standpoint of power consumption and heat dissipation. In order to implement various AI tasks, the challenge is that the power efficiency and hardware performance to support many of AI tasks are not sufficient for the scale of those tasks.</p>
<p>In response, LeapMind aims to enable practical AI on various devices, and to solve both software and hardware requirements with our strength, “Extremely low bit quantization” technology. As a part of our effortson the software side, we use this “Extremely low bit quantization” to reduce the size of the model, achieving an ultra-low bit quantization of 1-bit weight x 2-bit activation, and our patented proprietary quantization technology has brought us success in low-bit quantization with minimal degradation in accuracy. In addition, a model converter dedicated to "Efficiera" optimizes computing efficiency and memory transfer to maximize hardware execution efficiency. Furthermore, we have been developing the AI accelerator "Efficiera" as hardware for the efficient execution of Extremely low bit quantization models. Our Extremely low bit quantization technology enables to perform matrix operations with simple logic operations, thus providing high arithmetic efficiency in a small circuit area. Since the DRAM memory transfer data during inference is 1 or 2 bits, the data size is equivalent to that of the data compression, reducing memory usage and bandwidth. By using Extremely low bit quantization technology, it is now possible to improve power efficiency, arithmetic efficiency, and area efficiency while minimizing the accuracy degradation caused by quantization.</p>
<p>Matsuda, Soichi, Chief Executive Officer of LeapMind, says  “As one of the few companies in the world that specialize in research and development of quantization technology as a method of AI technology and in the design of semiconductors for AI accelerators developed, we have been engaged in research and development for many years through joint development with major companies with the aim of achieving the practical application of AI. The balancing of high power efficiency and AI task processing performance with AI accelerators is essential for the implementation of AI in society. And today, we have achieved a power efficiency of 107.8 TOPS/W, far exceeding that of conventional GPUs and NPUs in terms of power efficiency. I believe that we have achieved an important milestone in our efforts to bring AI to the public more broadly.  We will continue to strive to solve our customers' problems and pursue research and development to contribute to the evolution of AI technology”.</p>
<p><strong>Please refer to LeapMind website below for the details of Efficiera.</strong>
<a href="https://leapmind.io/en/products/efficiera-ip/">https://leapmind.io/en/products/efficiera-ip/</a></p>
<p><strong>Appendix</strong>
*1  Estimated values are based on logic synthesis results using Cadence Genus Synthesis Solution. Conditions are as follows.
・Calculated based on IP stand-alone (Not including power of other subsystems such as DRAMs, main bus processors, etc.)
・Process: 7nm
・Clock frequency：533MHz
・Power supply voltage: Center
・Temperature: 25℃
・MAC utilization: 75.8%
*2  TOPS/W: Processing performance per watt（TOPS: tera operations per second）This indicates power efficiency. The higher this number, the greater the amount of computation to be processed with low-power consumption.
*3 <a href="https://openai.com/research/ai-and-compute">https://openai.com/research/ai-and-compute</a></p>
<h2 id="about-leapmind-inc">About LeapMind Inc.</h2>
<p>Founded in 2012, LeapMind has been providing solutions mainly to the consumer electronics, automotive, and manufacturing industries, as one of the few companies in the world that provide “Extremely low bit quantization”  technology to downsize AI and proprietary AI accelerator semiconductor design, under the corporate mission of "To create innovative devices with machine learning and make them available everywhere.” In order to provide foundational technologies to enable the next generation of information devices, we are committed to the development of both the software and hardware necessary to create such devices.LeapMind's business is highly acclaimed and has raised a cumulative total of 4.99 billion yen in funding (as of June 2023). For more information, please visit corporate website (<a href="https://leapmind.io/en/">https://leapmind.io/en/</a>)</p>
<p>Head office: Shibuya Dogenzaka Sky Building 3F, 28-1 Maruyama-cho, Shibuya-ku, Tokyo 150-0044
Representative: Soichi Matsuda, CEO
Established: December 2012
URL：<a href="https://leapmind.io/en/">https://leapmind.io/en/</a></p>
<p>*LeapMind, Efficiera and logo are trademarks or registered trademarks of LeapMind Inc.
*The information in the press release is current as of the date of announcement.</p>
<p>Media Contact
Marketing and Communication Group, LeapMind Inc.
Phone: 813-6696-6267
Email: <a href="mailto:pr@leapmind.io">pr@leapmind.io</a></p>

LeapMind’s Ultra Low-Power AI accelerator IP “Efficiera”
Achieved industry-leading power efficiency of 107.8 TOPS/W

<p>January 31, 2023 - LeapMind Inc. (Shibuya-ku, Tokyo; CEO: Soichi Matsuda), a creator of the standard in edge AI, has celebrated its 10th anniversary of founding last December, and is pleased to announce that Mr. Takayoshi USHIKU has been appointed as a Technical Advisor since January, with the aim of further penetrating deep learning technology into the market.</p>
<h2 id="background">Background</h2>
<p>Since founded in 2012, LeapMind has been developing both software and hardware by using deep learning quantization technology for aiming machine learning into practical use. After 10 years since its founding, we’re seeing that LeapMind’s technology is steadily approaching the end-products that benefit people's lives; however, at the same time, feel that there are still more rooms to expand the understanding of deep learning technology usage, and needs more research and development and activities to raise awareness.
Mr.Ushiku is a computer vision and natural language processing researcher and has been dedicated to development of machine learning and related areas; written numerous papers and has been cited in international papers more than 5,000 times*1, currently teaching at multiple universities, serves as a member of many expert committees, including The Japanese Society for Artificial Intelligence. Also, since 2018, he has been active as a Principal Investigator at OMRON SINIC X Corporation since its establishment, to connect social issues to next-generation innovative technologies.
As a technical advisor, Mr. Ushiku will provide advice on the company's technology strategy for the next 10 years, and will also be involved in supporting the strengthening of our R&#x26;D organization.
Combining Mr. Ushiku's research with our technology related to image processing AI, we will strengthen technology research and development, accelerate commercialization, and promote the realization of social implementation of edge AI.</p>
<p>*1 : Google Scholar search result</p>
<p><img src="/images/posts/news/7912/5e0a0e5e-ushiku_insert.png" alt="Board member of LeapMind Inc." title="Board member of LeapMind Inc.">
(Board member of LeapMind Inc. From right CEO Matsuda, Mr. Ushiku, CTO Tokunaga)</p>
<h2 id="career-of-takayoshi-ushiku">Career of Takayoshi USHIKU</h2>
<ul>
<li>2013 Research Fellow, Japan Society for the Promotion of Science, Microsoft Research Redmond Intern</li>
<li>2014 PhD, Graduate School of Information Science and Technology, The University of Tokyo
Research Scientist, NTT Communication Science Laboratories</li>
<li>Research Scientist, NTT Communication Science Laboratories</li>
<li>2016 Associate Professor, Department of Mechano-Informatics, Graduate School of Information Science and Technology, The University of Tokyo</li>
<li>2018 Principal Investigator, OMRON SINIC X Corporation</li>
<li>2019 Chief Research Officer, Ridge-i Co., Ltd.</li>
<li>2022 Chief Executive Officer, Nine Bulls, LLC</li>
</ul>
<p>The research interests lie in cross-media understanding through machine learning, mainly for computer vision and natural language processing. His research results have received various awards; ACM Multimedia Grand Challenge Special Prize in 2011, ACM Multimedia Open Source Software Competition Honorable Mention in 2017, and NVIDIA Pioneering Research Awards in 2017 and 2018.</p>
<h2 id="comment-from-technical-advisor-mr-yopshitaka-ushiku">Comment from Technical Advisor Mr. Yopshitaka Ushiku</h2>
<p>I have thought for a long time that LeapMind is a standard-bearer in efforts to popularize neural networks on embedded devices in Japan. When it comes to enriching people's lives, simply making PCs and smartphones smarter is not enough, and it is a very difficult task to make even smarter those embedded devices which are already useful. On the other hand, because it is difficult, the meaning of performing neural network processing on embedded devices is extremely high, and the idea of ​​”using embedded technology to spread it widely and enrich people's lives” is quite important and meaningful.
It may sound grandiose but I want to support technological innovation with the big ambition of “run by the embedding on a large-scale base model, putting a huge network on the SoC which was thought to be “absolutely unworkable with embedding.” 　</p>
<h2 id="comment-from-cto-hiroyuki-tokunaga">Comment from CTO Hiroyuki Tokunaga</h2>
<p>LeapMind is currently developing an accelerator for neural networks, but the future of the highly intelligent computers would not be a simple extension of today's technology, and at some point it will require some kind of technological leap. In order to respond to such technological evolution, I am very pleased to have Mr. Ushiku as an advisor who has been involved in leading-edge research and development. His participation is definitely a great reassurance to draw our future picture.</p>
<h2 id="about-leapmind">About LeapMind</h2>
<p>LeapMind Inc. was founded in 2012 with the corporate philosophy of "bringing new devices that use machine learning to the world". Total investment in LeapMind to date has reached 4.99 billion yen (as of May 2021). The company's strength is in extremely low bit quantization for compact deep learning solutions. It has a proven track record of achievement with over 150 companies, centered in manufacturing including the automobile industry. It is also developing its Efficiera semiconductor IP, based on its experience in the development of both software and hardware.</p>
<p>Head office: Shibuya Dogenzaka Sky Building 3F, 28-1 Maruyama-cho, Shibuya-ku, Tokyo 150-0044
Representative: Soichi Matsuda, CEO
Established: December 2012
URL：<a href="https://leapmind.io/en/">https://leapmind.io/en/</a></p>
<p>*LeapMind, Efficiera and logo are trademarks or registered trademarks of LeapMind Inc.</p>

【Press release】Yoshitaka Ushiku appointed as Technical Advisor

<h2 id="my-summer-internship-experience-at-leapmind">My summer internship experience at LeapMind.</h2>
<p>Shibuya Scramble Crossing, the most iconic Tokyo city landscape. Every minute hundreds of people seamlessly pass by, eventually disappearing in an endless crowd. Could you count them all? Could you track them all down? Well, I can. And I learned this secret skill during my summer internship at LeapMind Inc.</p>
<h2 id="multi-object-tracking">Multi-object tracking</h2>
<p>The task is simple: given a video, track all its objects as they move. You can think of it more atomically: first, detect objects within a single frame, and then link those objects in between frames. Former is simply object detection, whereas the latter is called object re-identification. Ideally, we would like to process frames in an online manner, that is, without looking into future frames while processing the current one. This is a pretty reasonable and desired assumption, especially when it comes to live videos (think of CCTV footage).</p>
<p>As one might expect, recent deep learning models enable multi-object tracking – and <a href="https://motchallenge.net/results/MOT20/?det=Private">they are fairly good</a> at it. Unfortunately, they do not necessarily excel at speed. It is challenging to run a live demo smoothly, and a networking delay while streaming the video from a camera to a GPU makes it all hardly feasible. Overcoming the speed issue was a goal of my internship.</p>
<p><img src="/images/posts/en/news/how-i-became-a-spy/7442f705-image1.gif" alt="Image: Multi-object tracking"></p>
<p>A live object tracking demo running at 70 FPS, recorded at LeapMind office, Shibuya, Tokyo.</p>
<h2 id="accelerating">Accelerating</h2>
<p>We focused on a one state-of-the art multi-object tracking method. Having understood, analyzed and identified its bottleneck, we were able to run a live demo in average almost 5-times faster than a baseline – at 100 FPS at a peak. Noteworthy, we did not suffer from a significant accuracy drop.
Despite an outstanding performance, it’s not the end of further improvements. The model we ended up with can be potentially <a href="https://medium.com/@joel_34050/quantization-in-deep-learning-478417eab72b">quantized</a> (compressed) and embedded on a hardware using LeapMind technology.</p>
<p><img src="/images/posts/en/news/how-i-became-a-spy/08d7ac54-joniak4-660x440.png" alt="Image:Presentation"></p>
<h2 id="leapmind-is-a-place-to-work">LeapMind is a place to work</h2>
<p>I spent five weeks at LeapMind as an intern, and I did enjoy it. I was working with professionals on a challenging project that won’t be archived once I leave. The atmosphere here kept me always motivated and I had no problems communicating – LeapMind is fully bilingual.
Company response to COVID-19 is also worth praising, since it went entirely remote (although one can visit the office if necessary). Last but not least, the salary is competitive, even if you’re an intern.</p>
<p>LeapMind is a place to work!</p>
<p>Written by Przemyslaw Joniak, LeapMind Summer 2020 Intern.</p>

How I became a spy

<p>こんにちは。LeapMind インターン運営担当です。 本日は、2022年サマーインターンに参加されたインターン生の成果発表について記事にまとめたいと思います。</p>
<p>Hello, this is LeapMind Internship Administration Team. Today, we would like to share the presentations of our interns who participated in the 2022 Summer Internship Program.</p>
<p>はじめまして。LeapMindでインターンをしていた劉です。
夏休みの2ヶ月間、Network Development Kit (NDK)チームに参加させていただきました。
この記事ではインターン期間中の取り組みをはじめ、社内の環境や雰囲気など、インターンを通して得られたこと、感じたことを総合的に書かせていただきたいと思います。</p>
<p>Hello, I am Ryu, an intern at LeapMind.
I joined the Network Development Kit (NDK) team for two months as an intern during the summer vacation.
In this article, I would like to write a comprehensive report on what I gained and felt through my internship, including the project I worked on and the work environment and atmosphere within the company.</p>
<h2 id="インターン期間中の取り組み--the-project-i-worked-on-during-my-internship">インターン期間中の取り組み / The project I worked on during my internship.</h2>
<p>今回のインターンの課題はBinary Neural Networkに関する新しい論文[1]の再現実験をするというものでした。この課題にインターン生2人で取り組みました。論文では、重みパラメータとアクティベーションの両方をそれぞれ1bitにするという大胆な量子化を提案しています。それでいて、ImageNetを用いた分類問題を始め、CNNが得意とするさまざまな課題で量子化前と同程度の性能を出せることを論文では主張しています。ニューラルネットワークの量子化という分野において精度の低下は常に頭を悩ませてきた問題であるため、「量子化前と同程度の性能」をたったの1bitずつで実現できるのであれば、それは驚くべきことであり、今後のLeapMindの製品に役立つことも期待できます。</p>
<p>The project for this internship was to experiment to reproduce a new research paper on Binary Neural Networks [1]. The two interns including me worked on this assignment together. The research paper proposes a bold quantization scheme in which both the weight parameter and the activation are set to 1 bit. Despite this, they claim that they can deliver the same level of performance as before quantization in various tasks that CNNs are good at, such as classification problems using ImageNet. Since accuracy degradation has always been a major problem in the field of neural network quantization, if "the same level of performance as before quantization" with only 1 bit each could be realized, it is a remarkable achievement and will be useful for LeapMind's future products.</p>
<p>こうしたモチベーションによって始まった今回の再現実験ですが、予想以上に難航してしまいます。論文にソースコードは提供されていません。学習率のようなハイパーパラメータや、オプティマイザのような再現に必要な情報も提供されていません。そのため、これらは過去の経験や他の実装から推測することになります。ハイパーパラメータ探索の大変さを見くびることはできませんが、ここまでの苦労であれば他の論文の再現でも必要となることは多いでしょう。本論文の再現において最も苦労したのは、ニューラルネットワークの形が分からないという点でした。</p>
<p>Although these motivations led us to start this reproduction experiment, it turned out to be more difficult than expected. The source code has not been provided in the paper. Neither are hyperparameters like learning rate or necessary information for reproduction, such as optimizers, provided. Therefore, we had to infer these from experience and other implementations. We cannot underestimate the difficulty of hyperparameter search, however, this much effort will often be necessary for the reproduction of other papers. The most difficult part of reproducing this research paper was not knowing the shape of the neural network.</p>
<p>具体的に1つ例を挙げましょう。下の図は論文が提案するネットワークにビット幅の注釈をつけたものです。</p>
<p>Let me give you one specific example. The figure below shows the network proposed by the paper with bit width annotations.</p>
<p><img src="/images/posts/news/summer-internship-2022-ryu-exploring-binary-neural-networks/f05c0373-ryu-san-360x440.png" alt="具体イメージ"></p>
<p>赤の破線は4bitと書かれていますが、なぜ4bitとするのか、どういう構造で4bitになっているのかを論文から読み取ることができませんでした。結局、再現実験ではここを4bitにすることを諦め、ハードウェアでの実行可能性を損ねないように注意しながら、論文の記述になるべく近いアーキテクチャを探索しました。</p>
<p>The red dotted line indicates 4 bits, but we could not figure out from the paper why it should be 4 bits or what kind of structure makes it 4 bits. In the end, we gave up on making this part 4-bit in the reproduction experiment and searched for architecture as close as possible to the description in the paper, being careful not to impair the executability of the hardware.</p>
<p>こうした試行錯誤によって精度は次第に改善されていったものの、論文の言うような「量子化前と同程度の性能」をインターン期間中に出すことはできませんでした。その意味で論文の完全な再現には失敗したものの、重みとアクティベーション両方を1bitにした量子化の結果としてはまずまずな精度を得ることができたと思います。とりわけ、他のメンバーとのディスカッションを通じて、良い精度のアーキテクチャを自ら提案できたことはとても嬉しく思います。</p>
<p>Although the accuracy was gradually improved through these trial and error processes, we were not able to generate "the same level of performance as before quantization," as stated in the paper, during the internship program. Although we failed to reproduce the paper completely, I believe that we were able to produce a good level of accuracy as a result of quantization with 1-bit weights and activation. Above all, I am very happy that I was able to propose a good accuracy architecture through discussions with other members of the group.</p>
<h2 id="開発環境--development-environment">開発環境 / Development environment</h2>
<p>大学の課題や研究において、チームで開発することは多くありません。数十人規模ともなるとなおさらです。LeapMindでは、社内のソースコードをほぼ1つのレポジトリで管理しています。コードを書いたらプルリクエストを出し、読みやすさ等の評価をクリアするとマージされます。このコードレビューのおかげで、ソースコードはとても読みやすく、簡単な変更であればすぐに変更箇所を見つけ出すことができました。実際にプルリクエストを出してレビューしてもらったり、他の人の書いたコードをレビューしたりすることは、私にとって学びの多い新鮮な経験になりました。</p>
<p>In university, there are not many opportunities to develop in teams on assignments or in research, especially when there are dozens of people involved. At LeapMind, internal source code is managed in a single repository. When we write code, submit a pull request, and once it meets readability and other criteria, it is merged. This code review makes the source code very legible and allows us to find changes quickly if they are simple changes. Submitting pull requests to be reviewed and reviewing code written by others was a great learning and refreshing experience for me.</p>
<p>LeapMindでは、Bazel、Neptuneなど、研究開発を効率化するツールをいくつも使用しています。これらのツールは大学の研究に役立ちそうなものも多く、インターン終了後も自分の開発に役立てていきたいと考えています。このようなツールが多く使われている背景には、環境の変化を恐れず新しい技術を貪欲に取り入れていく、ベンチャー企業らしいとも言える風土があると感じます。</p>
<p>LeapMind uses several tools to streamline research and development, including Bazel and Neptune. Many of these tools seem to be useful for university research, and I plan to continue using them for my development even after my internship is over. I feel that the reason why many of these tools are used is because of the corporate culture, which is characteristic of a venture company, of not fearing changes in the environment and eagerly adopting new technologies.</p>
<h2 id="社内の雰囲気環境company-culture-and-environment">社内の雰囲気・環境/Company culture and environment</h2>
<p>フレックスタイム、コアタイムなし、フルリモート可という働き方の自由さにまず驚かされました。休暇も基本的に自由に取ることができ、社員の方も含め休みづらさといったものは全く感じませんでした。オフィスに出勤したいときには出勤することもでき、モニター2台の快適な環境で作業することができました。</p>
<p>I was amazed at the flexibility of the working style: flextime, no core hours, and full remote availability. I was also able to take vacations when I wanted to, and I never felt uncomfortable taking time off, including among the employees. I was also able to go to the office whenever I wanted to and work in a comfortable environment with two monitors.</p>
<p>COVID-19の影響もあり出勤する方が少なく、社員の方と対面でコミュニケーションを取る機会が非常に限られていた点は好みが分かれる部分だと思います。それでも、ランチ会やCoffee Breakを通じて、オンラインで多くの社員の方との会話の機会を確保していただきました。経歴や趣味など個性的な方も多く、会話していて楽しかったです。また、技術的に分からないことがあっても気軽に質問することができ、詳しい方に助けてもらうことができました。社員の方の技術力の高さにも驚かされました。</p>
<p>Due to COVID-19, few people came to the office, and the opportunities for face-to-face communication with employees were very limited, which I think is something that people may have different preferences for. Nevertheless, through lunch meetings and Coffee Breaks, I was able to have opportunities to talk with many employees online. I enjoyed talking with many unique people with unique backgrounds and interests. Also, I was able to easily ask questions about technical issues and was able to get help from people who were familiar with the subject matter. I was also amazed at the high technical skills of the employees.</p>
<h2 id="最後に--conclusion">最後に / Conclusion</h2>
<p>あっという間の2ヶ月間でした。もともと興味のあるテーマでしたが、最後まで興味を持って取り組むことができました。実験の結果、過去最高のスコアが出たときの興奮は今でも忘れません。LeapMindでインターンをすることができて本当に良かったと思っています。大変お世話になりました。</p>
<p>The two months went by very quickly. It was a topic that I had always been interested in, and I was able to work on it with great interest until the very end. I still remember how excited I got when we got the highest score ever as a result of the experiment, and I am very happy to be an intern at LeapMind. Thank you very much for all your support!</p>
<h2 id="参考文献--references">参考文献 / References</h2>
<p>[1] Nie, G., Xiao, L., Zhu, M., Chu, D., Shen, Y., Li, P., ... &#x26; Chen, B. (2022). Binary Neural Networks as a general-propose compute paradigm for on-device computer vision. arXiv preprint arXiv:2202.03716.</p>

Summer Internship 2022 – Ryu – Exploring Binary Neural Networks

<p>こんにちは。LeapMind インターン運営担当です。 本日は、2022年サマーインターンに参加されたインターン生の成果発表について記事にまとめたいと思います。</p>
<h2 id="my-summer-internship-experience-at-leapmind">My summer internship experience at LeapMind</h2>
<p>Hello, my name is Nishio and I participated in the 2022 summer internship program.
For a little less than two months, I worked with the NDK team on an experiment to reproduce a paper of binary neural networks.
The paper I replicated is “Binary Neural Networks as a general-propose compute paradigm for on-device computer vision” [1].
The “BiNealNet” proposed in this paper is claimed to outperform 8-bit quantization methods in the speed-vs-accuracy tradeoff for classification, detection, segmentation, super-resolution, and matching.</p>
<h2 id="implementation-overview">Implementation Overview</h2>
<p>BiNealNet is one of the binary neural networks that achieves high accuracy and computational efficiency for various tasks by using the following two techniques:</p>
<ul>
<li>Binarize a ResNet block by enlarging its channels.</li>
<li>Injecting auxiliary parameters in binaryconvolution and binary activation.</li>
</ul>
<p>Binarizing ResNet blocks allows for easy extension to various tasks, and the larger number of output channels in the block increases the capacity of the network. In addition, auxiliary parameters also improved the expressiveness of the network.</p>
<p>Binarization of ResNet blocks is realized by two methods: binarization of convolution weight parameters and binarization of activation.</p>
<p>Weight parameters of convolution are binarized by
Wb = Sign ( Tanh( Wf ) )
Wf Wb
Where Wb is the value of the binarized weight and Wf is the full precision weight, and are arbitrary parameters.
Sign function calculated the gradient using STE [2]. is a channel-wise parameter. is implemented so that the shape could be selected from one-, two-, or four-dimensional tensors according to the paper.</p>
<p>Activation is binarized by
Ab = Sign ( Htanh( PReLU( Af + b0 ) + b1 ) )
Af = Ab
where Ab is the value of the binarized activation and Af is the full precision activation, , b0, b1 are channel-wise, and is a scalar arbitrary parameter.
Htanh backpropagation was implemented as follows [3]:</p>
<p><img src="/images/posts/news/summer-internship-2022-nishio-go-beyond-the-accuracy-of-the-float-model-with-binary-neural-networks/1b73ab39-gazou-.png" alt="数式"></p>
<p>This implementation reduces the mismatch in the gradient of the backpropagation of the sign function.</p>
<h2 id="results-of-reproduction-experiments-with-cifar10">Results of Reproduction Experiments with CIFAR10</h2>
<p>The implemented BiNealNet was validated using CIFAR10, a typical classiﬁcation task.
The maximum number of training epochs was set to 400 and the batch size to 128, starting with a learning rate of 0.1, and then decaying using the learning rate scheduler with SGDR [4].</p>
<p>Validated ResNet accuracy with full precision activation and weights as a baseline, yielding 0.9479 as validation prediction accuracy.
Verified whether the prediction accuracy is equivalent to a full precision network without degradation when weights or activation are binarized. Thereafter, when binarizing weights and activations, the number of channels was set to 1.5 times the original ResNet, following the paper [1].</p>
<p>First, only the weight parameters were binarized to obtain a validation prediction accuracy of 0.9485. When binarizing the weights, the weight parameters of the first convolution were not binarized. (The same applies thereafter.)
We then binarized the activation and obtained a validation prediction accuracy of 0.922.
Finally, when both the weight parameters and activation were binarized, we obtained 0.911 as the validation prediction accuracy.</p>
<p>Full precision (baseline)
0.9479
Binarize Weight
0.9485
Binarize Activation
0.922
Binarize Weight &#x26; Binarize Activation
0.9112</p>
<p>Binarization of activation caused degradation of prediction accuracy and did not produce the results as in the paper.</p>
<p>For further verification, we changed the following three points.</p>
<ul>
<li>Change the kernel size of the convolution layer of downsampling from 1 to 3.</li>
<li>Clip the negative slope of PReLU so that it is not less than 0.</li>
<li>Move scaling parameters of and to after the convolution.</li>
<li>The initial earning rate of new auxiliary parameters is reduced by a factor of 10.</li>
</ul>
<p>In the original ResNet, the kernel size of the downsampling layer is 1, but it is known empirically that a kernel size of 1 is inaccurate when the convolution weights are binarized.</p>
<p>Kernel size of downsampling = 1
0.9112
Kernel size of downsampling = 3
0.9214</p>
<p>Based on this result, the kernel size of the downsampling layer was subsequently set to 3.</p>
<p>Next, the negative slope of PReLU was verified by replacing it with a function that was clipped so that the negative slope of PReLU was not less than zero. This was because BiNealNet implicitly assumed that the value of the negative slope of PReLU was non-negative.</p>
<p>Normal PReLU function
0.9214
Clipped PReLU function
0.9235</p>
<p>Thereafter, the clipped PReLU was used in the activation binarization.</p>
<p>Thirdly, the position of the scaling factor was verified. The scaling factors for the activation and convolution weights were moved after the convolution operation. This makes all the inputs to the convolution operation 1bit. Accuracy was significantly reduced.</p>
<p>Scaling before conv
0.9235
Scaling after conv
0.9076</p>
<p>Finally, we considered reducing the learning rate of the added arbitrary parameter to make the change smaller.</p>
<p>Same initial LR (0.1)
0.9235
Lower initial LR for new parameters (0.01)
0.9102</p>
<h2 id="summary-and-future-outlook">Summary and Future Outlook</h2>
<p>The BiNealNet paper was ambiguous in many of its descriptions, which was a barrier to validation. We attempted to replicate the paper by experimenting with a variety of patterns, but ultimately ended up with a validation prediction accuracy that was 0.0244 points lower than the full precision network for the CIFAR10 dataset.
Within this summer internship, I also worked on an experiment in object detection using BiNealNet and MSCOCO dataset, but I will not go into details.</p>
<p>BiNealNet is claimed to have produced accuracy comparable to 8-bit models in segmentation, super-resolution, and matching, so we would like to verify its effectiveness for tasks other than classiﬁcation and object detection.</p>
<h2 id="leapmind-is-a-great-place-to-work">LeapMind is a great place to work!</h2>
<p>I worked at LeapMind for less than two months as an intern and had a very enjoyable and productive time. My mentor and other team members were very helpful and polite with code reviews, and I learned a lot from them.
They were also very helpful in discussing technical difficulties with me to deepen my understanding. The company encouraged working remotely, but it was great that I could come to the office if I wanted.
LeapMind is a great place to work!</p>
<h2 id="references">References</h2>
<p>[1] G. Nie et al., arXiv [cs.CV] (2022), (available at <a href="http://arxiv.org/abs/2202.03716">http://arxiv.org/abs/2202.03716</a>).
[2] M. Courbariaux et al., arXiv [cs.LG] (2016), (available at <a href="http://arxiv.org/abs/1602.02830">http://arxiv.org/abs/1602.02830</a>).
[3] Z. Liu et al., arXiv [cs.CV] (2018), (available at <a href="http://arxiv.org/abs/1808.00278">http://arxiv.org/abs/1808.00278</a>).
[4] I. Loshchilov and F. Hutter, arXiv [cs.LG] (2016), (available at <a href="http://arxiv.org/abs/1608.03983">http://arxiv.org/abs/1608.03983</a>).</p>

Summer Internship 2022 – Nishio – Go beyond the accuracy of the Float model with Binary Neural Networks.

<p>September 16, 2022 - LeapMind Inc. (Shibuya-ku, Tokyo; CEO: Soichi Matsuda), a creator of the standard in edge AI have announced that it has been granted their first overseas patent in Taiwan. The patent is awarded for the invention related to neural network circuits, edge devices and neural network operation method, which is part of our core technology, extremely low-bit quantization used for the flagship product, ultra-low power AI interface accelerator IP “Efficiera®”. It is a pursued to the utmost limit technology of the weight reduction method of deep learning model and has already been granted in Japan in June, 2021.</p>
<p><strong>Patent number:</strong> No. I773245</p>
<p><strong>Date of patent registration:</strong> August 1st, 2022</p>
<p><strong>Title of the invention:</strong> Neural Network Circuits, Edge Devices and Neural Network Operation Methods</p>
<p><strong>Patentee:</strong> LeapMind Inc.</p>
<p>LeapMind has been working to acquire patents on ultra-low quantization technology and Efficiera® in Japan and overseas, and has obtained seven patents in Japan, including this. LeapMind will continuously be committed to strive in advanced technology development by acquiring patents and to accelerate the social implementation of AI.</p>
<p><strong>List of LeapMind’s patent in Japan</strong>
<img src="/images/posts/en/news/7900/9bc7c488-eng.png" alt="List of patents granted by LeapMind to date." title="List of patents granted by LeapMind to date."></p>
<h2 id="about-leapmind">About LeapMind</h2>
<p>LeapMind Inc. was founded in 2012 with the corporate philosophy of "bringing new devices that use machine learning to the world". Total investment in LeapMind to date has reached 4.99 billion yen (as of May 2021). The company's strength is in extremely low bit quantization for compact deep learning solutions. It has a proven track record of achievement with over 150 companies, centered in manufacturing including the automobile industry. It is also developing its Efficiera semiconductor IP, based on its experience in the development of both software and hardware.</p>
<p>Head office: Shibuya Dogenzaka Sky Building 3F, 28-1 Maruyama-cho, Shibuya-ku, Tokyo 150-0044
Representative: Soichi Matsuda, CEO
Established: December 2012
URL:<a href="https://leapmind.io/en/">https://leapmind.io/en/</a></p>
<p>*LeapMind, Efficiera and logo are trademarks or registered trademarks of LeapMind Inc.</p>
<p>Media Contact
Marketing and Communication Group, Public Relations Officer, LeapMind Inc.
Phone: 813-6696-6267
Email: <a href="mailto:pr@leapmind.io">pr@leapmind.io</a></p>

LeapMind 1st-patents in Taiwan for its low bit quantization technology

<h2 id="focusing-to-visual-inspection-edge-ai-enables-fast-and-secured-manufacturing-dxtraining-with-only-a-few-normal-data-all-ai-operations-completes-on-edge-devices">Focusing to visual inspection, edge AI enables fast and secured manufacturing DX
‐Training with only a few normal data, all AI operations completes on edge devices‐</h2>
<p>May 25, 2022– Tokyo Japan, LeapMind Co., Ltd., today announced a new Deep Learning model “Efficiera Anomaly Detection model” based on its ultra-low power consumption AI inference accelerator IP "Efficiera®" and will be available from June, 2022. This specializes in visual inspection for the manufacturing industry and contributes to reducing security risk and labor shortage at the manufacturing site.</p>
<p><img src="/images/posts/en/news/7683/68cfde0b-ad-imageeng.png" alt="Image of the use of the Efficiera Anomaly Detection Model." title="Image of the use of the Efficiera Anomaly Detection Model."></p>
<h2 id="features-of-efficiera-anomaly-detection-model">Features of “Efficiera Anomaly Detection model”</h2>
<h3 id="training-completes-with-only-normal-data">•Training completes with only normal data</h3>
<p>Annotation work is not required which is often outsourced, no necessity of concluding non-disclosure agreement.
Avoids the need of large amount of defective data, and training completes just in a few seconds which enabling rapid introduction of AI to the manufacturing site</p>
<h3 id="completes-both-inference-and-training-on-small-edge-device-with-fpga">•Completes both inference and training on small edge device with FPGA</h3>
<p>No need of send image data outside the company, the risk of information leakage is reduced, and complicated non-disclosure agreements are not required which could lead to labor savings
Can be used in environment without network</p>
<h3 id="re-training-can-be-done-without-an-ai-engineer">•Re-training can be done without an AI engineer</h3>
<p>Visualization of abnormal areas by heat map.</p>
<p>AI training results can be saved and reloaded (redo and version management) easily on site which makes it easy to use to those companies that have difficulty in securing IT human resources.</p>
<p>The needs of automation in the manufacturing industry has been growing due to the chronic understaffing problem. AI usage is brought to the attention for visual inspection for quality check, but the challenges are the shortage of IT human resources and high cost of equipment. Other challenges are collecting defective datas for AI training and the complication of high-mix low-volume production due to the diversification demand and this may be cases where the support of AI engineers is required for AI retraining due to a change in the inspection target. Efforts to deal with these issues are also becoming a hurdle to adoption.</p>
<p>LeapMind has been developing an AI inference accelerator IP “Efficiera®” which can embed deep learning technology into various edge devices to popularize the devices with deep learning technology to the world to make people’s lives convenient. A newly developed “Effciera Anomaly Detection model” lowers the hurdle to introduce AI to visual inspection by using its own unique “Extremely low bit quantization” technology. The key feature of this deep learning model is that both inference and training can be completed on small edge devices with FPGA. Comparing the case using cloud AI, it has an advantage that there is no need to send inspection target images from the device itself which reduces the risk of data breach, contributes to reduce broadband infrastructure and cloud service usage fee. Also, compared to the case of using on-premises GPU server, it resolves concerns of the equipment cost and the installation space. The training can be completed in a few seconds only with a few dozens of normal product images and no necessity of collecting defective datas. This requires no annotation work and makes it possible to detect unexpected abnormalities and lowers the risk of only detecting trained defect targets. This model visualizes the part judged by AI as abnormal with a heating map and retraining operation is very simple so that it makes on-site adjustment easy without the support of AI engineers even on high-mix low-volume production lines. Since the training result is saved in a few MB of data and can be used for inference later, it is easy to redo the wrong training. This will help boost DX in the manufacturing industry even for companies that have difficulty in securing IT human resources or companies that have manufacturing bases in rural areas where broadband networks cannot be used.</p>
<p><img src="/images/posts/news/7679/2ef74f6d-.png" alt="visual inspection using the &#x22;Efficiera&#x22; anomaly detection model.">
Image : visual inspection using the "Efficiera" anomaly detection model. Abnormal parts of defective products can be seen at a glance by visualization with a heat map</p>
<p>Based on our vision “To provide key technologies to bring next-generation information devices into reality”, we will continue to strive to the realization of more convenient lives for people by realizing edge AI into practical use.</p>
<h2 id="about-efficiera">About Efficiera®</h2>
<p>Efficiera is an ultra-low power AI inference accelerator IP specialized for CNN inference processing that runs as a circuit on FPGA or ASIC devices. The "extremely low bit quantization" technology minimizes the number of quantization bits to 1 - 2 bits, maximizing the power and area efficiency of convolution, which accounts for most of the inference processing, without the need for advanced semiconductor manufacturing processes or special cell libraries. By using this product, deep learning functions can be incorporated into a variety of edge devices, including consumer electronics such as home appliances, industrial equipment such as construction machinery, surveillance cameras, broadcasting equipment, as well as small machines and robots that are constrained by power, cost, and heat dissipation, which has been technically difficult in the past. LeapMind provides three Deep Learning models - Efficiera Object Detection model, Efficiera Noise Reduction model and Efficiera Anomaly Detection model. Visit product website at <a href="https://leapmind.io/products/efficiera-ip/">https://leapmind.io/products/efficiera-ip/</a></p>
<h2 id="about-leapmind">About LeapMind</h2>
<p>LeapMind Inc. was founded in 2012 with the corporate mission "To create innovative devices with machine learning and make them available everywhere." Total investment in LeapMind to date has reached 4.99 billion yen (as of May 2021). The company's strength is in extremely low bit quantization for compact deep learning solutions. It has a proven track record of achievement with over 150 companies, many of which are centered in manufacturing, including the automobile industry. It is also developing its Efficiera semiconductor IP, based on its experience in the development of both software and hardware</p>
<p>Head office: Shibuya Dogenzaka Sky Building 3F, 28-1 Maruyama-cho, Shibuya-ku, Tokyo 150-0044
Representative: Soichi Matsuda, CEO
Established: December 2012
URL: <a href="https://leapmind.io/en/">https://leapmind.io/en/</a></p>
<p>*LeapMind, Efficiera and logo are trademarks or registered trademarks of LeapMind Inc.
*The information in the press release is current as of the date of announcement.</p>
<p>Media Contact
Efficiera Division, Marketing and Communication Team, LeapMind Inc.
Phone: 813-6696-6267
Email: <a href="mailto:pr@leapmind.io">pr@leapmind.io</a>　</p>

【Press release】LeapMind announces “Efficiera® Anomaly Detection model”

<h2 id="adopted-on-maxell-frontiers-new-image-recognition-solution-ixam-vision-series">Adopted on Maxell Frontier’s new image recognition solution "iXAM Vision series"</h2>
<p>February 22, 2022– Tokyo Japan, LeapMind Co., Ltd. (Shibuya, Tokyo; CEO: Soichi Matsuda), a leading supplier of the standard in edge artificial intelligence (AI), today announced that have concluded a commercial license agreement of LeapMind’s ultra-low power AI inference accelerator IP "Efficiera" with Maxell Frontier Co., Ltd. (Yokohama, Kanagawa; President: Hiroyuki Ota), the group company of Maxell, Ltd.
Efficiera has been adopted as a technology to enhance AI processing performance in their new image recognition solutions "iXAM Vision series" announced by Maxell Frontier today and scheduled to be launched in October 2022.</p>
<p>This collaboration of LeapMind’s "Extremely low bit quantization," which supports the application of deep learning technology on edge devices with restricted power and space, and Maxell Frontier which has the proven track record of over 30 years will boost the AI performance into the next stage of their image recognition systems.
Together with “Efficiera” IP, LeapMind is providing Efficiera deep learning models “Object detection” and “Anomaly Detection”. The big features of Efficiera are space-saving, the compactness, and of the deep learning model are easy installation to edge devices and re-training. With those features, it is possible to shorten the development time of AI functioned edge devices.
LeapMind will continue to strive to popularize edge AI and aim to solve social issues through deep learning technology.</p>
<h2 id="about-ixam-vision-series-of-maxell-frontier">About “iXAM Vision series” of Maxell Frontier</h2>
<p>iXAM series lines-up two types of unit products, 1) “iXAM Vision Engine” - the intelligent image processing unit, 2) “iXAM Vision Sensor” - the intelligent camera, and “iXAM Vision Meister” - the AI image inspection application development tool, which runs on the unit products. These are scheduled to be launched in October 2022.
Maxell Frontier’s product release:<a href="https://ssl4.eir-parts.net/doc/6810/ir_material21/177469/00.pdf">https://ssl4.eir-parts.net/doc/6810/ir_material21/177469/00.pdf (Japanese)</a></p>
<h2 id="about-efficiera">About Efficiera</h2>
<p>Efficiera is an ultra-low power AI inference accelerator IP specialized for CNN inference processing that runs as a circuit on FPGA or ASIC devices. The "extremely low bit quantization" technology minimizes the number of quantization bits to 1 - 2 bits, maximizing the power and area efficiency of convolution, which accounts for most of the inference processing, without the need for advanced semiconductor manufacturing processes or special cell libraries. By using this product, deep learning functions can be incorporated into a variety of edge devices, including consumer electronics such as home appliances, industrial equipment such as construction machinery, surveillance cameras, broadcasting equipment, as well as small machines and robots that are constrained by power, cost, and heat dissipation, which has been technically difficult in the past. Visit product website at <a href="https://leapmind.io/products/efficiera-ip/">https://leapmind.io/products/efficiera-ip/</a></p>
<h2 id="deep-learning-model-for-efficiera">Deep learning model for Efficiera</h2>
<p>The Efficiera deep learning model is a trained model that can be used on edge devices. LeapMind develops andprovides trained models of Object detection, Anomaly Detection and AI Image processing.</p>
<p>Object Detection model:
The position information of the detection target can be acquired from the image. Can be applied to a wide range of applications by combining with various logic of processing the position information.</p>
<p>Anomaly Detection model:
Realize the real time visual inspection on edge device by AI. Easily complete the AI training on edge devices even the format of the inspection target changes.</p>
<p>AI Image Processing model:
Operates in real time on edge devices to reduce the noise and improve the image quality of noisy images.</p>
<h2 id="about-maxcell-frontier">About Maxcell Frontier</h2>
<p>Head office: YBP North Square III, 134 Godo-cho, Hodogaya-ku, Yokohama, Kanagawa 240-0005
Representative: Hiroyuki Ota, President
Established: April 2020
URL:<a href="https://www.frontier.maxell.co.jp">https://www.frontier.maxell.co.jp</a></p>
<h2 id="about-leapmind">About LeapMind</h2>
<p>LeapMind Inc. was founded in 2012 with the corporate mission "To create innovative devices with machine learning and make them available everywhere." Total investment in LeapMind to date has reached 4.99 billion yen (as of May 2021). The company's strength is in extremely low bit quantization for compact deep learning solutions. It has a proven track record of achievement with over 150 companies, many of which are centered in manufacturing, including the automobile industry. It is also developing its Efficiera semiconductor IP, based on its experience in the development of both software and hardware
Head office: Shibuya Dogenzaka Sky Building 3F, 28-1 Maruyama-cho, Shibuya-ku, Tokyo 150-0044
Representative: Soichi Matsuda, CEO
Established: December 2012
URL: <a href="https://leapmind.io/en/">https://leapmind.io/en/</a></p>
<p>*LeapMind, Efficiera and logo are trademarks or registered trademarks of LeapMind Inc.
*The information in the press release is current as of the date of announcement.</p>
<p>Media Contact
Efficiera Division, Marketing and Communication Team, LeapMind Inc.
Phone: 813-6696-6267
Email:<a href="mailto:pr@leapmind.io">pr@leapmind.io</a></p>
<p>Product Contact
Efficiera Division, Business Development Team, LeapMind Inc.
Phone: 813-6696-6267
Email:<a href="mailto:business@leapmind.io">business@leapmind.io</a>
Product information:<a href="https://leapmind.io/products/efficiera-ip/">https://leapmind.io/products/efficiera-ip/</a></p>

【Press Release】LeapMind announces license agreement for AI accelerator “Efficiera” with Maxell Frontier

<h2 id="achieving-high-quality-image-like-a-smartphone-with-edge-camera-by-ai-noise-reduction">Achieving high quality image like a smartphone with edge camera by AI noise reduction</h2>
<p>January 31, 2022– Tokyo Japan, LeapMind Co., Ltd., a leading creator of the standard in edge artificial intelligence (AI), today announced an AI image processing model which operates in real time on edge device. This AI image processing model uses the company's ultra-low power consumption AI inference accelerator IP "Efficiera" to operate in real time on edge devices to reduce the noise and improve the image quality of noisy images. This solves the challenges of performance and image quality accuracy that have been raised in the past, and achieved high performance, high image quality, and weight reduction.</p>
<h2 id="challenges">Challenges</h2>
<p>Performance challenge: difficult on edge device to operate in real-time with the high calculation cost on AI image processing
Image quality accuracy challenge: Extremely low bit quantization for AI image processing that inputs and outputs the images has degradation in image quality (reproduction of color and resolution, etc.).</p>
<h2 id="solutions-and-achievements">Solutions and achievements</h2>
<p>The company's extreme low bit quantization technology has made it lighter, and by combining it with the performance scalability of Efficiera v2, it became possible to operate in real time even with a video camera.
The company's Pixel embedding*¹ technology has achieved high image quality and image quality comparable to that of commonly used 32-bit floating point models. As a result, image quality can be improved by AI without the need for high-sensitivity sensors and large lenses, which are the high cost components. And high image quality like smartphones is possible by AI to those image quality oriented cameras, which makes industrial cameras (security cameras, inspections cameras, etc.) improve the accuracy of object recognition and inspection by improving the image quality even in low-light conditions or those video cameras which cannot take sufficient exposure time.</p>
<p>This model will be available as an evaluation version from February 2022. To obtain, please contact us at <a href="mailto:business@leapmind.io">business@leapmind.io</a>.</p>
<p>“As far as we searched, we are the world’s first in bringing this Image processing AI model into the product by low bit quantization technology. This model is a product that can be put into practical use only because of LeapMind focusing on both hardware and software development and we believe that it shows the new value of extremely low bit quantization.” says Hiroyuki Tokunaga, CTO of LeapMind.</p>
<p>*¹Pixel embedding: A method of encoding an input image into 2-bit data of multiple channels</p>
<h2 id="achieving-image-quality-comparable-to-the-32-bit-floating-point-model-with-the-ultra-small-quantization-12-bit-model">Achieving image quality comparable to the 32-bit floating point model with the ultra-small quantization (1,2 bit) model</h2>
<p>Input image Left: ISO51200, 1/800sec, F4.0, Right: ISO102400, 1/320sec, F8.0
<img src="/images/posts/news/7528/ca32098b-input.png" alt="Conventional images with noise." title="Conventional images with noise."></p>
<p>Output image: 32 bit floating point model
<img src="/images/posts/news/7528/4804c6b3-fp32.png" alt="Image of 32-bit floating-point model with slightly reduced noise." title="Image of 32-bit floating-point model with slightly reduced noise."></p>
<p><strong>Output image: LeapMind Extremely low bit quantization model</strong>
<img src="/images/posts/news/7528/ec6fe060-efficieranr.png" alt="Image of the LeapMind micro-quantisation model with reduced noise." title="Image of the LeapMind micro-quantisation model with reduced noise."></p>
<h2 id="features">Features</h2>
<ul>
<li>Raw-to-raw noise reduction model for raw input and raw output
　　- Minimum impact to Existing image processing pipeline due to Raw-to-raw</li>
<li>Deep learning-based noise reduction processing
　　- State-of-the-art NR algorithm optimized for embedded applications</li>
<li>Provides a retrainable trained model
　　- Optimize the model by retraining sensor-specific noise in addition to trained noise</li>
<li>Light weight processing for real-time operation
　　- Supports real-time operation with video cameras due to Efficiera v2's performance scalability</li>
</ul>
<p><img src="/images/posts/news/7528/b5a83b3a-nr_technical_image.png" alt="Deep learning-based noise reduction processing image." title="Deep learning-based noise reduction processing image."></p>
<h2 id="about-efficiera">About Efficiera</h2>
<p>Efficiera is an ultra-low power AI inference accelerator IP specialized for CNN inference processing that runs as a circuit on FPGA or ASIC devices. The "extremely low bit quantization" technology minimizes the number of quantization bits to 1 - 2 bits, maximizing the power and area efficiency of convolution, which accounts for most of the inference processing, without the need for advanced semiconductor manufacturing processes or special cell libraries. By using this product, deep learning functions can be incorporated into a variety of edge devices, including consumer electronics such as home appliances, industrial equipment such as construction machinery, surveillance cameras, broadcasting equipment, as well as small machines and robots that are constrained by power, cost, and heat dissipation, which has been technically difficult in the past. Visit product website at <a href="https://leapmind.io/products/efficiera-ip/">https://leapmind.io/products/efficiera-ip/</a></p>
<h2 id="about-leapmind">About LeapMind</h2>
<p>LeapMind Inc. was founded in 2012 with the corporate mission "To create innovative devices with machine learning and make them available everywhere." Total investment in LeapMind to date has reached 4.99 billion yen (as of May 2021). The company's strength is in extremely low bit quantization for compact deep learning solutions. It has a proven track record of achievement with over 150 companies, many of which are centered in manufacturing, including the automobile industry. It is also developing its Efficiera semiconductor IP, based on its experience in the development of both software and hardware</p>
<p>Head office: Shibuya Dogenzaka Sky Building 3F, 28-1 Maruyama-cho, Shibuya-ku, Tokyo 150-0044
Representative: Soichi Matsuda, CEO
Established: December 2012
URL: <a href="https://leapmind.io/en/">https://leapmind.io/en/</a></p>
<p>LeapMind, Efficiera and logo are trademarks or registered trademarks of LeapMind Inc.
The information in the press release is current as of the date of announcement.</p>
<p>Media Contact
Efficiera Division, Marketing and Communication Team, LeapMind Inc.
Phone: 813-6696-6267
Email: <a href="mailto:pr@leapmind.io">pr@leapmind.io</a></p>
<p>end</p>

【Press release】LeapMind Announces AI Image processing model which operates on edge devices

<p>Enhance product for wider-range application based on initial market introduction, results and evaluation</p>
<p>November 30, 2021 – Tokyo Japan, LeapMind Co., Ltd., a leading creator of the standard in edge artificial intelligence (AI), today announced version 2 (hereinafter: v2) of the ultra-low power AI inference accelerator IP Efficiera, scheduled to be available in December 2021. Efficiera is highly valued for its power saving, high performance, space saving and performance scalability features. Efficiera v2’s improved features and benefits expand the range of applications by providing wider coverage of performance with broader product availability while maintaining the circuit scale of the minimum configurations. These enhancements are based on learnings from the initial product introduction and market evaluation.</p>
<p>"Last year, we officially launched the commercial version of v1, and many companies evaluated Efficiera, and by the end of September 2021, we signed license agreements with eight domestic companies. Our corporate motto is concretely feeling that the "spreading to the world new devices that use machine learning " that we have set as our corporate philosophy is steadily progressing through the provision of v1. We will continue to strive to popularize AI through further technological innovation and product lineup expansion." said Soichi Matsuda, CEO of LeapMind.</p>
<p>Efficiera is an ultra-low power AI inference accelerator IP specialized for convolutional neural network (CNN) inference processing that runs as a circuit on FPGA or application-specific integrated circuit (ASIC) devices. The ultra-small quantization technology minimizes the number of quantization bits to 1 - 2 bits, maximizing the power and area efficiency of convolution that accounts for most of inference processing without the need for advanced semiconductor manufacturing processes or special cell libraries. By using this product, deep learning functions can be incorporated into a variety of edge devices, including consumer electronics such as home appliances, industrial equipment such as construction machinery, surveillance cameras, broadcasting equipment, as well as small machines and robots that are constrained by power, cost, and thermals — all of which have been technically difficult in the past.</p>
<p>"Since the official release of v1, we have aimed to develop the world's most power efficient DNN accelerator. We have strengthened the design and verification method and development process. We developed v2 so that it can respond to adoption in ASICs and application-specific standard products (ASSPs). We are also developing an inference learning model on the deep learning side to maximize the benefits of ultra-small quantization technology. The greatest strength of LeapMind is that we can provide a technique to master these two wheels,” said Dr. Hiroyuki Tokunaga, Director and CTO of LeapMind.</p>
<p>By improving design and verification methodology and reviewing the development process, product quality can be applied not only to FPGA but also to ASIC and ASSP. LeapMind starts to provide a model development environment, Network Development Kit (NDK) which enables users to develop deep learning models for Efficiera, which has not been done previously.</p>
<h2 id="other-notable-specifications-and-features-of-efficiera-v2">Other Notable Specifications and Features of Efficiera V2</h2>
<p>Concept of Efficiera v2
<img src="/images/posts/news/7432/0be4b840-efficiera2%E3%82%B3%E3%83%B3%E3%82%BB%E3%83%97%E3%83%88%E5%9B%B3-660x371.png" alt="Concept of Efficiera v2" title="Concept of Efficiera v2"></p>
<h3 id="hardware-features">Hardware features</h3>
<ul>
<li>Performance scalability up to 48 times by multiplexing MAC array + multi-core.</li>
<li>v2 allows you to triple the number of MAC arrays in the Convolution pipeline to v1 (x1, x4 selectable), and expands performance scalability by allowing you to select up to 4 cores.</li>
<li>Hardware execution of Skip connection and Pixel Embedding in addition to convolution and quantization.</li>
<li>Resources equivalent to Efficiera v1 were used with the same configuration and analyzed to understand execution time and assess if additional hardware functions are required.</li>
</ul>
<h3 id="integration-into-soc">Integration into SoC</h3>
<ul>
<li>AMBA AXI interface</li>
<li>AMBA AXI continued to be used as the interface with the external and the interface when viewed as a black box is the same as before however it is now easier to migrate from the existing design.</li>
<li>Single clock domain</li>
</ul>
<h3 id="target-frequency-in-fpga">Target frequency in FPGA</h3>
<ul>
<li>The operating frequency of the FPGA is the same as before and although it depends on the device, about 150 to 250 MHz can be expected.</li>
<li>256 GOP/s @ 125MHz (1 core)</li>
<li>Up to 12 TOP/s @ 250MHz (2 cores)</li>
<li>Provided as encrypted RTL</li>
</ul>
<h3 id="network-development-kit-ndk">Network Development Kit (NDK)</h3>
<ul>
<li>Package of code and information which is required to create ultra-small quantization DL model for Efficiera.</li>
<li>Immediate use is possible for developers working with deep learning model for GPU</li>
<li>DL framework supports PyTorch and TensorFlow 2</li>
<li>The learning environment is a GPU-equipped Linux server</li>
<li>The inference environment is a device equipped with Efficiera</li>
<li>Provide support from LeapMind</li>
</ul>
<p>LeapMind is welcoming trials and feedback from all interested parties, including system on chip (SoC) vendors and end-user product designers. To obtain Efficiera v2, please contact us at <a href="mailto:business@leapmind.io">business@leapmind.io</a>. For more product information, please visit: <a href="https://leapmind.io/products/efficiera-ip/">https://leapmind.io/products/efficiera-ip/</a>.</p>
<h2 id="about-leapmind">About LeapMind</h2>
<p>LeapMind Inc. was founded in 2012 with the corporate mission "To create innovative devices with machine learning and make them available everywhere." Total investment in LeapMind to date has reached 4.99 billion yen (as of May 2021). The company's strength is in extremely low bit quantization for compact deep learning solutions. It has a proven track record of achievement with over 150 companies, many of which are centered in manufacturing, including the automobile industry. It is also developing its Efficiera semiconductor IP, based on its experience in the development of both software and hardware</p>
<p>Head office: Shibuya Dogenzaka Sky Building 5F, 28-1 Maruyama-cho, Shibuya-ku, Tokyo 150-0044
Representative: Soichi Matsuda, CEO
Established: December 2012
URL: <a href="https://leapmind.io/en/">https://leapmind.io/en/</a></p>
<p>LeapMind, Efficiera and logo are trademarks or registered trademarks of LeapMind Inc.
The information in the press release is current as of the date of announcement.</p>
<p>Media Contact
Efficiera Division, Marketing and Communication Team, LeapMind Inc.
Phone: 813-6696-6267
Email: <a href="mailto:pr@leapmind.io">pr@leapmind.io</a></p>

News