How I became a spy

My summer internship experience at LeapMind.

Shibuya Scramble Crossing, the most iconic Tokyo city landscape. Every minute hundreds of people seamlessly pass by, eventually disappearing in an endless crowd. Could you count them all? Could you track them all down? Well, I can. And I learned this secret skill during my summer internship at LeapMind Inc.

Multi-object tracking

The task is simple: given a video, track all its objects as they move. You can think of it more atomically: first, detect objects within a single frame, and then link those objects in between frames. Former is simply object detection, whereas the latter is called object re-identification. Ideally, we would like to process frames in an online manner, that is, without looking into future frames while processing the current one. This is a pretty reasonable and desired assumption, especially when it comes to live videos (think of CCTV footage).

As one might expect, recent deep learning models enable multi-object tracking – and they are fairly good at it. Unfortunately, they do not necessarily excel at speed. It is challenging to run a live demo smoothly, and a networking delay while streaming the video from a camera to a GPU makes it all hardly feasible. Overcoming the speed issue was a goal of my internship.

A live object tracking demo running at 70 FPS, recorded at LeapMind office, Shibuya, Tokyo.

Accelerating

We focused on a one state-of-the art multi-object tracking method. Having understood, analyzed and identified its bottleneck, we were able to run a live demo in average almost 5-times faster than a baseline – at 100 FPS at a peak. Noteworthy, we did not suffer from a significant accuracy drop. Despite an outstanding performance, it’s not the end of further improvements. The model we ended up with can be potentially quantized (compressed) and embedded on a hardware using LeapMind technology.

LeapMind is a place to work

I spent five weeks at LeapMind as an intern, and I did enjoy it. I was working with professionals on a challenging project that won’t be archived once I leave. The atmosphere here kept me always motivated and I had no problems communicating – LeapMind is fully bilingual. Company response to COVID-19 is also worth praising, since it went entirely remote (although one can visit the office if necessary). Last but not least, the salary is competitive, even if you’re an intern.

LeapMind is a place to work!

Written by Przemyslaw Joniak, LeapMind Summer 2020 Intern.