Biology subjects

Mihailidis, A.

Publications and source records attributed to Mihailidis, A..

8 recordsLinked to original sources

bioRxiv2023

Sequential image classification of human-robot walking environments using temporal neural networks

Robotic prosthetic legs and exoskeletons require real-time and accurate estimation of the walking environment for smooth transitions between different locomotion mode controllers. However, previous studies have mainly been limited to static image classification, therein ignoring the temporal dynamics of human-robot locomotion. Motivated by these limitations, here we developed several state-of-the-art temporal convolutional neural networks (CNNs) to compare the performances between static vs. sequential image classification of real-world walking environments (i.e., level-ground terrain, incline stairs, and transitions to and from stairs). Using our large-scale image dataset, we trained a number of encoder networks such as VGG, MobileNetV2, ViT, and MobileViT, each coupled with a temporal long short-term memory (LSTM) backbone. We also trained MoViNet, a new video classification model designed for mobile and embedded devices, to further compare the performances between 2D and 3D temporal deep learning models. Our 3D network outperformed all the hybrid 2D encoders with LSTM backbones and the 2D CNN baseline model in terms of classification accuracy, suggesting that network architecture can play an important role in performance. However, although our 3D neural network achieved the highest classification accuracy, it had disproportionally higher computational and memory storage requirements, which can be disadvantageous for real-time control of robotic leg prostheses and exoskeletons with limited onboard resources.

bioengineering↗

bioRxiv2023

AI-powered smart glasses for sensing and recognition of human-robot walking environments

Environment sensing and recognition can allow humans and robots to dynamically adapt to different walking terrains. However, fast and accurate visual perception is challenging, especially on embedded devices with limited computational resources. The purpose of this study was to develop a novel pair of AI-powered smart glasses for onboard sensing and recognition of human-robot walking environments with high accuracy and low latency. We used a Raspberry Pi Pico microcontroller and an ArduCam HM0360 low-power camera, both of which interface with the eyeglass frames using 3D-printed mounts that we custom-designed. We trained and optimized a lightweight and efficient convolutional neural network using a MobileNetV1 backbone to classify the walking terrain as either indoor surfaces, outdoor surfaces (grass and dirt), or outdoor surfaces (paved) using over 62,500 egocentric images that we adapted and manually labelled from the Meta Ego4D dataset. We then compiled and deployed our deep learning model using TensorFlow Lite Micro and post-training quantization to create a minimized byte array model of size 0.31MB. Our system was able to accurately predict complex walking environments with 93.6% classification accuracy and had an embedded inference speed of 1.5 seconds during online experiments using the integrated camera and microcontroller. Our AI-powered smart glasses open new opportunities for visual perception of human-robot walking environments where embedded inference and a low form factor is required. Future research will focus on improving the onboard inference speed and miniaturization of the mechatronic components.

bioengineering↗

bioRxiv2023

Efficient Visual Perception of Human-Robot Walking Environments Using Semi-Supervised Learning

Convolutional neural networks trained using supervised learning can improve visual perception for human-robot walking. These advances have been possible due to large-scale datasets like ExoNet and StairNet - the largest open-source image datasets of real-world walking environments. However, these datasets require vast amounts of manually annotated data, the development of which is time consuming and labor intensive. Here we present a novel semi-supervised learning system (ExoNet-SSL) that uses over 1.2 million unlabelled images from ExoNet to improve training efficiency. We developed a deep learning model based on mobile vision transformers and trained the model using semi-supervised learning for image classification. Compared to standard supervised learning (98.4%), our ExoNet-SSL system was able to maintain high prediction accuracy (98.8%) when tested on previously unseen environments, while requiring 35% fewer labelled images during training. These results show that semi-supervised learning can improve training efficiency by leveraging large amounts of unlabelled data and minimize the size requirements for manually annotated images. Future research will focus on model deployment for onboard real-time inference and control of human-robot walking.

bioengineering↗

bioRxiv2023

Preliminary Development of a Robotic Hip-Knee Exoskeleton with 3D-Printed Backdrivable Actuators

Robotic exoskeletons can provide powered locomotor assistance and rehabilitation to persons with mobility impairments due to aging and/or physical disabilities. Here we present the preliminary development and systems integration of T-BLUE - a modular, bilateral robotic hip-knee exoskeleton with 3D-printed backdriveable actuators. We retrofitted commercially available passive postoperative orthoses with open-source 3D-printed actuators to minimize cost and improve accessibility. The actuators are of quasi-direct drive design with high-torque density brushless DC motors and low gearing (15:1 transmission ratio) for low output impedance and high backdrivability, therein allowing for energy-efficient and dynamic human-robot physical interaction and legged locomotion. The modular design allows the exoskeleton to be customized and adapted to different users (e.g., persons with lateral vs. bilateral mobility impairments) and different hip-knee joint configurations. The goals of this preliminary study were to describe our experience with regards to the repeatability of the open-source 3D-printed actuators in engineering practice and the feasibility of integrating the actuators into wearable robotics hardware. This qualitative research serves as a first step towards using the robotic exoskeleton to support the development and testing of novel controller designs and rehabilitation protocols for different locomotor activities of daily living. We are especially interested in populations that could benefit from partial locomotor assistance such as older adults and/or persons with osteoarthritis. Future research will involve benchtop testing to quantitatively evaluate the actuator performance in terms of dynamics and energy-efficiency.

bioengineering↗

bioRxiv2023

Development and Mobile Deployment of a Stair Recognition System for Human-Robot Locomotion

Environment sensing and recognition can improve the safety and autonomy of human-robot locomotion, especially during transitions between environmental states such as walking to and from stairs. However, accurate and real-time perception on edge devices with limited computational resources is an open problem. Here we present the development and mobile deployment of StairNet - a vision-based automated stair recognition system powered by deep learning. Building on ExoNet - the largest open-source dataset of egocentric images of real-world walking environments - we designed a new dataset specifically for stair recognition with over 515,000 images. We then developed a lightweight and efficient convolutional neural network for image classification, which accurately predicted complex stair environments with 98.4% accuracy. We also studied different model compression and optimization methods and deployed our system on several mobile devices running a custom-designed iOS application with onboard accelerators using CPU, GPU, and/or NPU backend computing. Of the designs that we tested, our highest performing system showed negligible reductions in classification accuracy due to the model conversion for mobile deployment and achieved an inference time of 2.75 ms on an iPhone 11. The high speed and accuracy of the StairNet system on edge devices opens new opportunities for autonomous control and planning of robotic prosthetic legs, exoskeletons, and other assistive technologies for human locomotion.

bioengineering↗

bioRxiv2023

Continuous Prediction of Leg Kinematics during Walking using Inertial Sensors, Smart Glasses, and Embedded Computing

Unlike traditional hierarchical controllers for robotic leg prostheses and exoskeletons, continuous systems could allow persons with mobility impairments to walk more naturally in real-world environments without requiring high-level switching between locomotion modes. To support these next-generation controllers, we developed a new system called KIFNet (Kinematics and Image Fusing Network) that uses lightweight and efficient deep learning models to continuously predict the leg kinematics during walking. We tested different sensor fusion methods to combine kinematics data from inertial sensors and computer vision data from smart glasses and found that adaptive instance normalization achieved the lowest RMSE predictions for knee and ankle joint kinematics. We also deployed our model on an embedded device. Without inference optimization, our model was 20 times faster than the previous state-of-the-art and achieved 20% higher prediction accuracies, and during some locomotor activities like stair descent, decreased RMSE up to 300%. With inference optimization, our best model achieved 125 FPS on an NVIDIA Jetson Nano. These results demonstrate the potential to build fast and accurate deep learning models for continuous prediction of leg kinematics during walking based on sensor fusion and embedded computing, therein providing a foundation for real-time continuous controllers for robotic leg prostheses and exoskeletons.

bioengineering↗

bioRxiv2022

Simulation of Energy Regeneration in Human Locomotion for Efficient Exoskeleton Actuation

Backdriveable actuators with energy regeneration can improve the efficiency and extend the battery-powered operating times of robotic lower-limb exoskeletons by converting some of the otherwise dissipated energy during negative mechanical work into electrical energy. However, previous related studies have focused on steady-state level-ground walking. To better encompass real-world community mobility, here we developed a feedforward human-exoskeleton energy regeneration system model to simulate energy regeneration and storage during other daily locomotor activities. Data from inverse dynamics analyses of 10 healthy young adults walking at variable speeds and slopes were used to calculate the negative joint mechanical power and work (i.e., the mechanical energy theoretically available for electrical energy regeneration). These human joint mechanical energetics were then used to simulate backdriving a robotic exoskeleton and regenerating energy. An empirical characterization of the exoskeleton device was carried out using a joint dynamometer system and an electromechanical motor model to calculate the actuator efficiency and to simulate energy regeneration. Our performance calculations showed that regenerating energy at slower walking speeds and decline slopes could significantly extend the battery-powered operating times of robotic lower-limb exoskeletons (i.e., up to 99% increase in total number of steps), therein improving locomotor efficiency.

bioengineering↗

bioRxiv2022

Stair Recognition for Robotic Exoskeleton Control using Computer Vision and Deep Learning

Computer vision can be used in robotic exoskeleton control to improve transitions between different locomotion modes through the prediction of future environmental states. Here we present the development of a large-scale automated stair recognition system powered by convolutional neural networks to recognize indoor and outdoor real-world stair environments. Building on the ExoNet database - the largest and most diverse open-source dataset of wearable camera images of walking environments - we designed a new computer vision dataset, called StairNet, specifically for stair recognition with over 515,000 images. We then developed and optimized an efficient deep learning model for automatic feature engineering and image classification. Our system was able to accurately predict complex stair environments with 98.4% classification accuracy. These promising results present an opportunity to increase the autonomy and safety of human-exoskeleton locomotion for real-world community mobility. Future work will explore the mobile deployment of our automated stair recognition system for onboard real-time inference.

bioengineering↗