What We’re Actually Doing: Software Development

The Robotic-Assisted Wood Waste Removal team is creating a robot that can help collect fallen plant matter on forest floors. This should prevent small branches and other debris from igniting and starting dangerous forest fires. While this concept is fairly straightforward, the software-building process is complex.

Our project's software engineers have worked to update the code as needed. Recently, the code was upgraded so that it can run on the new platform. The wood waste removal software allows the robot to visually follow (also known as tracking) a human target who guides it in collecting fallen wood waste. Several algorithms used to do this (called tracking algorithms) have been tested in the lab so far. A method called re-identification (usually used to find a specific target person over more than one distinct camera [1]) was tested and found not to be successful. Multiple object tracking (used to identify and find the path of more than one target from a video [2]) was a similar story as it didn't have good compatibility with this application.

The solution to the tracking approach needed to be comprehensive. Ideally, it would track targets as well as re-identify them in a single path of software processes. In addition, the engineers had to thoroughly check the code for stability and measure performance. The code has been tested using simulations in the lab.

In the newest update of the project software, a person can set themselves as a tracking target with just a gesture, thanks to the gesture recognition module. Currently, this gesture is a raised hand, but other gestures (and even voice recognition) could be used in the future. This yields better results than the targets having a unique identification based on their individual bodies, like before. Gesture recognition may be familiar to people who have used it with devices such as iPhones [3]. Figure 1 shows an actual sample of gesture recognition [4]. This was a major and very promising step forward. In the future, the team will test how the code works in the field. This will allow the Robotic-Assisted Wood Waste Removal team to put the robot into its intended use, and start making our communities safer from forest fires.

Figure 1

A way to visualize software gesture recognition of an open palm [4]. In the original source, this was achieved with Python for MediaPipe, but the same principles apply to the Robotic-Assisted Wood Waste Removal Project [4]

Sources:

1. Ye, M., Shen, J., Lin, G., Xiang, T., Shao, L., & Hoi, S. C. H. (2022). Deep Learning for Person Re-Identification: A Survey and Outlook. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(6), 2872–2893. https://doi.org/10.1109/TPAMI.2021.3054775

2. Luo, W., Xing, J., Milan, A., Zhang, X., Liu, W., & Kim, T.-K. (2021). Multiple object tracking: A literature review. Artificial Intelligence, 293, 103448. https://doi.org/10.1016/j.artint.2020.103448

3. Lee, H.-C., Shih, C.-Y., & Lin, T.-M. (2013). Computer-Vision Based Hand Gesture Recognition and Its Application in Iphone. Advances in Intelligent Systems and Applications, 2, 487–497. https://doi.org/10.1007/978-3-642-35473-1_49

4.  Smith, N. & De Luca, N. (2022). Lab Notes: MediaPipe with Python for Gesture Recognition. Growth Acceleration Partners. https://www.growthaccelerationpartners.com/blog/lab-notes-mediapipe-with-python-for-gesture-recognition