Deep Learning Internship at LeapMind

Hi there! My name is Oscar. I’m a summer intern at LeapMind and a Taiwanese student from Waseda University. I had an incredible time interning at this rapidly growing, multinational startup and would love to share my journey with you!

Making Deep Learning Prevalent

This original title from LeapMind caught my eye during my internship hunting. With IoT becoming more common, it is refreshing to see this new concept of bringing deep learning to various devices. After learning about the company’s mission of embedded deep learning and its multilingual, open working environment, I knew that an internship at LeapMind is going to be unique for me. The experience of working at a Tokyo startup that designs hardware and software for deep learning is just an opportunity I cannot miss!

Internship Project: Optical Flow Estimation with Blueoil

The goal of my internship is developing a DNN for optical flow estimation with Blueoil and quantizing the network for edge devices to run. Blueoil is an open-source software from LeapMind that helps me apply quantize-aware training and convert a trained network into an executable program. By running this program on a CPU or FPGA-based device, embedded deep learning is achieved! 😎

I worked on this project with an intern partner, Inoue san, and my mentor, Iizuka san. My main task is designing the network for optical flow estimation using TensorFlow and verifying its performance after quantize-aware training. After surveying multiple related works and considering their implementation costs and compatibilities with Blueoil, we decided to implement and quantize FlowNetS, a famous DNN for learning optical flow, published in 2015.

This task might look simple at first glance. All I have to do is reproduce a DNN and apply quantize-aware training to it. In reality, there are numerous constraints and challenges:

In Blueoil, only a subset of TensorFlow operations in FlowNetS can be converted for edge devices, so I must replace many parts of the network with original architectures. Blueoil uses extreme low-bit quantization (1-bit and 2-bit), which can significantly weaken the network's performance. Therefore, I need to consider different speed-accuracy trade-offs, e.g., which part of the network to quantize. The dataset for optical flow estimation is large (30GB), and the total training time takes more than 2 weeks. It usually takes around 2 days to tell whether the network is learning effectively or not.

Results & What I learned

Despite all the difficulties mentioned above, I was able to overcome them thanks to the support and advice from Inoue san, Iizuka san, and multiple full-time members. In the end, I developed an original DNN, LmFlowNetS, and reduced its inference time by 53.14% on a CPU-based device with quantization! Inoue san and I even gave a live demo of this network during our final presentation. Through this experience, in addition to technical skills, I have also learned the following:

• Take the initiative and don't hesitate to ask the experts.
• Communicate and listen actively.
• Observe carefully and experiment boldly.

To sum up, I had a rewarding experience working at LeapMind. The vibrant, open company culture really keeps me motivated every day. If you're interested in gaining an experience similar to mine, please don't hesitate to apply!

Our Source Code: [https://github.com/OscarWang114/blueoil/tree/lm-flownet-s] Final Presentation Slides: [https://www.slideshare.net/leapmind/final-presentation-optical-flow-estimation-with-dl-183441802]

Bonus: What I love at LeapMind

Open culture🙌: Everyone at LeapMind is very approachable and transparent. I don’t need to hesitate to ask anyone questions, even the CTO. Shuffle lunch😋: This regular event divides everyone randomly into groups for lunch. It’s the perfect chance to socialize with people from other teams! Technology💻: I love the free FPGA boards for development and the object detection live demo at the office entrance!