I Can 'C' Clearly Now ...

I was intending to move on from the Tidy The Toys challenge. However, after the announcement that this year's competition has been delayed I decided to revisit it. 

The main issue I had was with the block detection code. Rather than go for the sensible option of using OpenCV, like most competitors, I decided I would write my own in Python. 

So I did. And it worked very well. If I enter the RGB values of the blocks and give an accuracy value, the robot can find the blocks and move towards them. 

However - although I optimised the Python code as much as I could, the best I could manage with the block recognition was about 3-4 frames per second. 

I could live with this - but it meant moving fairly slowly towards the blocks. The other issue I had was that, although the camera processing and the distance sensing are running in separate threads, the contention between the two meant that the distance readings were also delayed and so I would tend to overrun the optimum distance between the robot and the block in order to deploy the lifter. To overcome this, after getting close to the block - I then switch off the camera processing and then just use the distance sensor to reverse to the correct distance. 

So when Pi Wars 2021 was delayed, and I realised I had a bit more time to improve things, I decided I had two choices - the sensible option of learning OpenCV and doing it properly; or the not so sensible option of rewriting my block detection code in 'C++' and interfacing to that from the main Python code. 

So obviously I chose the latter option. I know how to program in C++ pretty well, so figured I would give it a go. 

The tricky bit was how to call C++ code from Python. I looked at the options and there were a lot of them. SWIG and Boost.Python were a couple that seemed to cover both C and C++. 

Another library that interested me was pybind11. This seems to allow large C++ libraries to be wrapped for access with Python. However it did seem quite complicated to use for such a simple use case. 

In the end I decided to drop C++ and just write a simple function in C. This could be built into a shared library and then the Python ctypes library could be used to convert the Python types to their C equivalent and call the function. 

So I converted the block detection code to C. Sometimes it's easier to just rewrite from scratch - but I was interested in the performance improvements, so I tried to keep the implementation as close to the Python code as possible. I then wrote the Python wrapper to call this new C function. 

Getting the Python wrapper to work with the C code probably took the longest time, although this was down to a few rookie errors matching up the structure types.

I wrote a bit of test code to find the three coloured blocks in the image shown at the top of this article at different resolutions, made sure the results matched, and compared the timings. The table below shows the results:

Timings searching for red, green and blue blocks

This shows a massive performance gain. Interestingly the timings show it's slower to process a 162x121 image than a 324x242 image. This is due to some of my optimisation code that tries to reduce the number of pixels it needs to scan in the image. Rounding issues mean that it can actually scan more pixels in a smaller image, but I know the issues there so can select the best resolution according to how big the blocks are.  

In this test, for finding the blocks, the 80x60 image is more than enough, although I can see there's room for improvement. 

In fact, it gets better. In the tidy the toys challenge, I only need to look for one coloured block at a time. By only searching for one colour in the image gives further performance improvements: 

Timings searching for single block

Although, whilst the Python code is greatly improved by only searching for a single colour, the C code is not so affected. 

And I can see again, that the smaller image actually takes longer, so I should be able to tweak the resolution to optimise performance further. (Writing the performance tests for this blog post has proved very useful!). 

Playing around with the Python / C interface has been quite a learning curve, but the performance gains are such that, going forward, I can now consider writing other parts of the code in C, should they prove to be too slow in Python.