<p>Here at LeapMind we are working on making deep learning empowered intelligent IoT devices a widespread reality. One of the topics our R&#x26;D is most concerned about is the compression of deep learning models. In this blog post I would like to talk about why this is important and give an overview of the literature on this topic...</p>
<p>Nicolas Loerbroks' full article on the "State of the Art in Compressing Deep Convolutional Neural Networks" can be found on the Medium platform.Please take a look: <a href="https://medium.com/@nicolas_19145/state-of-the-art-in-compressing-deep-convolutional-neural-networks-cfd8c5404f22">https://medium.com/@nicolas_19145/state-of-the-art-in-compressing-deep-convolutional-neural-networks-cfd8c5404f22</a></p>

State of the Art in Compressing Deep Convolutional Neural Networks

<p>Since last decade we live in a world which is in an accelerated change, specially in technology. Around end of the last century it was obvious that internet will be the backbone of the modern era. And beginning of this century with outstanding development of modern digital technologies we soon entered in a mobile first world. But that didn’t even last very long. And as google put it in IO 2017, we’re already shifting from Mobile First to an AI First world.</p>
<h2 id="what-is-ai">What is AI</h2>
<p>Artificial Intelligence or AI is a term first coined in 1943 by McCullouch and Pitt’s by formal design for <a href="https://en.wikipedia.org/wiki/Turing-complete">Turing-complete</a> “<a href="https://en.wikipedia.org/wiki/Artificial_intelligence#CITEREFRussellNorvig2009">Artificial Neuron</a>”. It is an intelligent system that can somehow mimic the operation of the biological neuron, which is the key to any intelligent biological bodies. The attempt to mimic the operational behavior of a biological neuron in intelligent beings like human ultimately leads to modern-day artificial intelligence. Now it is a cutting edge field for research and seems to be the future of technology according to many experts including Stephen Hawking, Bill Gates &#x26; Elon Mask etc. Today, almost all big shot in IT industry are moving towards AI one way or another. But this is not a recent thing.</p>
<p><img src="/images/posts/en/techblog/artificial-intellegence-an-opportunity-for-developing-nations/ml_position-300x207.png" alt="Fig. 1 : Discipline for Machine Learning">
Fig. 1 : Discipline for Machine Learning</p>
<p>From the early ages, in computational science, statistical approach was applied to processing large data. Besides these statistical analysis it was soon found that with conventional approach sometime it is not possible to program every possible combination of state to get a desired outcome. For example, classification problem in a very large data set or optimizing a multi-variable non-linear system etc. The solution is found to be all along in nature. Conscious being like us has the ability to hunch or guess the correct results without being able to explain the process. It seems like there is some kind of black box in our brain that can figure out the problem by its own and find the solution. We call it learning. In computer science similar algorithm were developed to give the computer program the ability to learn without being explicitly programmed, hence Machine Learning. And this gives birth of Artificial Intelligence.</p>
<p>Today machine learning consist of many domain like information theory, statistics etc. By going one step further, implementing operational behavior of biological neuron a new system emerged named perception which represent a single neuron. By combining multiple perception we have a network of artificial neuron which is called Artificial Neural Network. Using mathematical tricks from Statistical Physics to achieve the behavior understood from Neuroscience the artificial neural network soon became the foundation of modern days Artificial Intelligent systems.</p>
<h2 id="deep-neural-network">Deep Neural Network</h2>
<p>In recent AI development, Deep Neural Network is probably the most common term. By adding multiple neurons connected with each other called a neural network and arranging them in many layers gives us a deep neural network.</p>
<p>Deep neural network is used mostly for classification tasks like image recognition, image processing, natural language processing etc. For example, Deep learning, the most popular term for image processing, is a deep neural network which learns though convolution and back propagation and optimize the network to improve the outcome hence learning over training.</p>
<p><img src="/images/posts/en/techblog/artificial-intellegence-an-opportunity-for-developing-nations/deep_neural_network-2.png" alt="Fig 2 : Deep Neural Network">
Fig 2 : Deep Neural Network</p>
<h2 id="machine-intelligence">Machine Intelligence</h2>
<p>When we think about machine intelligence it always feels like a some thing from the Sci-Fi movie. A super intelligent software that can understand human voice and understand not only the command but also the intent behind it or the conceptual idea of the communication with human. Although it started with movies but it is already a reality right now. The digital assistants in our phone like <a href="https://www.apple.com/siri/">Siri</a>, <a href="https://assistant.google.com/">Google Assistant</a> or <a href="https://www.microsoft.com/en-us/cortana">Cortana</a> is closer than ever to make that fiction to reality. Using cloud infrastructure and with help of recent IoT revolution devices like <a href="https://en.wikipedia.org/wiki/Amazon_Echo">Amazon Echo</a>, <a href="https://madeby.google.com/home/">Google Home</a> or upcoming <a href="https://www.apple.com/homepod/">Apple HomePod</a> bring AI assistant from personal phone to our home for everyone. The natural language interface of all these product is simply outstanding and also being more and more natural and more feature rich over time.</p>
<p><img src="/images/posts/en/techblog/artificial-intellegence-an-opportunity-for-developing-nations/Apple-HomePod-vs-Home-and-Echo-e1496825501356-840x460.jpg" alt="Fig 3 : Digital Home Assistant powered by AI">
Fig 3 : Digital Home Assistant powered by AI</p>
<p>But the story began a lot earlier when almost nobody, if not, knew about AI but also nobody actually cared. The idea of intelligent machines is almost three-quarter of a century old, even it was rejected by technical community as being the future and known to be as impractical and very inefficient due to mostly hardware limitations for providing the necessary computation for AI system. But overtime with recent accelerated progress in digital electronics now the necessary computation power is in our reach. With GPU Computing and specially designed chip for complex processing required. Although, now AI is in its early stages, it need a huge amount of computation power to train a neural network for a particular task within a reasonable timeframe. No wonder, only the IT giants like Google, Facebook, Apple, Amazon etc. as they have invested tremendous compute power behind AI development. Another part of the coin in AI development is data. We need quality data with proper annotation to train a neural network correctly. The more diverse data we have the more accurate results likely to be achieved. This is why Google, Facebook are the leader in AI research as they have the most diverse dataset. All of their business models are based on collecting data.</p>
<p>Now, artificial neural network is far from to be in capable of doing things a biological brain like us can do. Yes it can learn but artificial neural network can not do anything. Today's AI are strictly task specific. We can train the network to do one task and it might perform admirably but if we want to do something else the system completely fails. It need to retrain for the new task and that will destroy the ability to do the previous task. Such kind of AI is called shallow AI.</p>
<p>But our target is to achieve human level intelligence. A system that can learn new things without destroying previous training. Which is called General Artificial Intelligence. Although many great mind expressed concern about such system that it can hold potential danger to human itself. Such general AI can improve itself and soon enough it will exceed human intelligence which might be a problem controlling such system. But this is not a near concern for a long time.</p>
<h2 id="what-we-do-with-ai">What we do with AI</h2>
<p>The early AI system being used is most probably the 'Recommendation System' in Amazon or 'Suggested Ad' in Google Search or Facebook to recommend Ad or interesting posts showed in our feed, those are the early implementation of AI system still being oblivious to rest of us. Google translator may be one of the most significant application that brings AI in the main stream. Without using the direct hardcoded language map technique google approaches the translation between languages with machine learning. By learning from same article written by different languages google translator system can learn from the language itself and produce translation for any text. It started a bit of low accuracy, which is almost very funny, but with training over time now google translator are outstandingly performing piece of software. It extends from text-based translation to image and voice also. Soon there should also be video or live streaming.</p>
<p>Than with Deep Neural Network a new era of image processing begins. Image recognition, object recognition in image, generating captions etc. are now possible with exceptional accuracy. Using these image processing technique AI is being implemented. IBM Watson, started with winning the famous game show "<a href="https://www.jeopardy.com/">Jeopardy</a>"  is now helping medical diagnosis faster and more accurately than a human doctor.  Watson also help financial analysis and better market prediction that any previous models. With AlphaGo Google Brain project was able to beat the best human Go player with a high margin. Beating human in chess by computer "<a href="https://en.wikipedia.org/wiki/Deep_Blue_(chess_computer)">Deep Blue</a>" came nearly two decades ago but now it is considered to be very naive compared to Go in-game complexity. Deep learning also enables promising progress in <a href="https://en.wikipedia.org/wiki/Natural_language_processing">Natural Language</a> Processing (NLP). Recently, Skype enabled a feature to live translate between languages directly from speech. Almost all discipline of science is touched by AI one way or another. From the physics experiment to the fight against cybercrime and cyberterrorism almost every thing is using AI.</p>
<p>May be the latest sensation on AI development is the self-driving car from Tesla. Started as a dream by Elon Mask Tesla self-driving quickly showed the prospect of self-driving automatic vehicle. By enabling self-driving capabilities in vehicles the fatality due to road accident can be practically zero (0). Recent study shows the major cause of accident is human error. Inattentiveness, dosing off during driving, fatigue are the primary cause of accident. By using AI system already the driving condition of the driver is being monitored and warning system is being integrated in all modern car. Soon there will be fully automated car and even further the whole road network will be fully automated that will improve the quality of road and transport.</p>
<p><img src="/images/posts/en/techblog/artificial-intellegence-an-opportunity-for-developing-nations/6TcxiNL-900x450.jpg" alt="Fig 4 : Self-driving car">
Fig 4 : Self-driving car</p>
<h2 id="a-golden-opportunity-for-developing-nation">A golden opportunity for developing nation</h2>
<p>So far, AI systems looks like the future in every domain of science and technology. But AI research is still very costly. Specially the training part. For training we need data. Lots of lots of data. Collecting data and preparing for neural network is time-consuming and costly operation. But the main problem for a developing nation would be the computing hardware required for a practical training. If often takes days, weeks even months to train a neural network with huge amount of data. Providing electricity for such long time uninterruptedly is one of the major challenge for developing nation. Many developing nation has poor electrical infrastructure and distribution system and often suffer from power-cut for long time. So, how can AI be the golden opportunity than?</p>
<p>The prospect of an AI system is often limited by our imagination. The good news in AI research is the resources are open source. Even we can get the pre-trained models from the internet. Pre-trained models means we don't need a large infrastructure and tons of data to train a neural network from scratch. We can start with pre-trained models and use our own data for further training in relatively low volume to achieve comparable results. Such training save us lots and lots of training time. With cloud computing we don't event need to buy expensive hardware for training. Amazon EC2 cloud infrastructure provides platform-as-a-service (PAAS) for us customizable ready systems on demand for such operations with relatively very low price.</p>
<p>All developing nation suffers in many things major for health, education or urban development. Let me mention three of them as follows:</p>
<h3 id="health-care">Health Care</h3>
<p>A developing nation often suffers from lack of good quality doctors. The health care sector probably the highest impact in society. Often it has very poor service quality. The main problem of this is surprising low number of doctors for the general population. In a govt. general hospital a doctor sometimes have to attain hundreds of patient. It's true even it sounds impossible. Because of this, the quality of service go down. Continuous overload of work often cause fatigue in doctors themselves which affect their quality of service. Lack of proper skilled personnel in diagnostic centers often cause the available manpower to push to its limit for long time and results in fatigue in their judgement and care.  In case of emergency of natural disaster the situation gets worse. Is such event a lot of people need to be attained within a very short time but lack of manpower makes it impossible.</p>
<p>By already we have AI systems that can diagnostic faster &#x26; accurately than most of human counterpart. Such system can be used as a first line of defense. As the volume of the patients is very high, an AI system can make the primary investigation for general cases and if necessary consult with specialist doctors if necessary. AI system can also be used to identify the pattern in sickness throughout the city and might suggest any severe problem in health care system or can detect an upcoming epidemic in a very early stages.</p>
<p>Of course AI can not replace the human doctors. We're not there yet. But it can be a game changer for general cases as it can attain way more patients without any fatigue and the accuracy is not subject to the condition of the doctors themselves.</p>
<h3 id="transportation">Transportation</h3>
<p>Another major problem area for a developing nations is transportation. Due to poor transportation system often the traffic is suffered horribly. Large grid lock and traffic jam is one of the common problem faced by urban areas of a developing nation. With a little bit of infrastructure  development AI system can improve both the transport network and driver condition. For transport network with an AI system it is very easy to find the congestion point and find a redirection for the traffic to balance the load, without which the cascading effect can virtually grid lock the whole system. Accident detection system onboard a vehicle powered by AI can warn the driver beforehand to avoid a probable accident. If accident do happen AI system can warn proper authorities for immediate action. On board AI system already been used to detect driver condition, it can warn the driver if the wheel is not held or if the driver is drowsy or drunk or not in a condition for safe driving. In extreme cases it can even stop and prevent the driver from driving the vehicle.</p>
<p>The AI assisted automatic protection systems are costly but it is very much affordable. Govt. can impose policies that all vehicle must have such system. Even a subsidy can be provided if applicable. AI system doesn't fatigue overtime, it doesn't sleep or the performance doesn't decrease over time or due to over work. Such systems can be very much change the fatality rate and the number of road accidents in a developing nation.</p>
<h3 id="industry-automation">Industry Automation</h3>
<p>The main development stream of a developing country consist of economic development and commercial industry are the backbone of that. It also contribute to the national economy by export to international market. When exporting to international market the industry must maintain the international quality check. But in developing nation the necessary expertise is not easy to find. So, often experts are brought from abroad that may interpreted as cost overhead and often be avoided. This ultimately results in poor quality and sometime accidents. But with a one time implement cost of a AI system for safety check or quality control can help industry to maintain much higher production quality.</p>
<h3 id="power-distribution">Power Distribution</h3>
<p>Power production and distribution throughout the nation is a challenging job. Often developing nation can not or not ignore to maintain, update or upgrade the power generation equipments. This results in power failure every now and then. But recent study already proved that an AI system can dramatically improve the load balancing and ensure optimal distribution. AI system can also predict unwanted power outage based on demand balance the grid accordingly. With one time implementation of such system can improve the power distribution situation a lot and also may give a window to focus on the more important development to improve the overall system.</p>
<h3 id="ai-research">AI Research</h3>
<p>Yes, the training part of the AI system can make a hole in our pocket. But on the other hand AI research is a very much possible. Academic study focused on AI will produce young graduate enthusiast that can contribute AI research in relatively smaller scale and join the mainstream development giants. Sooner of later AI will be everywhere, developing nation has a true golden opportunity for train young minds to be prepared to join that future. The need of good AI engineer is the new hot cake of the IT industry. In both hardware development for custom AI chip and software for new Neural Network Architecture needs good engineers worldwide. Developing nation can contribute to that growing manpower. It is certainly possible, they are doing it in many other sectors. Small startups and entrepreneurship are getting more and more welcomed in young generation. They can work on one particular sector of AI and combining such efforts can ultimately gives us a much better solution all together.</p>
<p>The big companies in the developing nation can invest in AI research &#x26; development. Recently, in India TATA Industry invested a huge amount of money to AI research. It might not gives any results right away but evidently the investment will paid off eventually. And that will benefit the whole nation in the long run. AI can practically affect in every developing sectors. With proper planning and effort this will ensure the future for the country from developing to a developed nation.</p>
<p>When Apollo 11 mission lunched by NASA in 1961, it not only change the technology frontier of USA but it actually start a culture in society. It put USA to the map of progress for decades to come. People got interested in science and technology every than before. If put a significant mark on the history. AI research can be the "Apollo" mission for this century, not to reach the moon but to reach the future.</p>

Tech Blog