Exploring Visual Proprioception in Robotic Manipulation: Can CNNs and ViTs Achieve Accurate Single-Camera Perception?

On April 20, 2025, Sahara Sheikholeslami and Ladislau Bölöni published Latent Representations for Visual Proprioception in Inexpensive Robots, exploring the use of machine learning models like CNNs, VAEs, and ViTs to estimate joint positions from a single camera image in cost-effective robots. Their research demonstrated the feasibility of achieving accurate visual proprioception with limited data, validated through experiments on a 6-DoF robot.

The research investigates visual proprioception in robotic manipulation using a single external camera image, focusing on inexpensive robots operating in unstructured environments. It explores various latent representations, including CNNs, VAEs, ViTs, and fiducial markers, employing fine-tuning techniques for limited data. Experiments evaluate the accuracy of these approaches on an inexpensive 6-DoF robot, demonstrating potential for precise proprioception without costly sensors.

Visual recognition has become integral to modern robotics, enabling machines to interact with their environments with increasing precision. Recent research has focused on improving pose estimation—the ability to determine an object’s position and orientation in space—using advanced visual recognition techniques. This study evaluates various methods for estimating the pose of a robotic arm in real-time, comparing their accuracy, computational efficiency, and robustness under different conditions. The research highlights the importance of reliable visual recognition systems in robotics, particularly as robots are increasingly deployed in dynamic and unstructured environments.

Methodology

The study was conducted using a UR5 robotic arm equipped with an RGB-D camera, which captures both color and depth information. The robot’s movements were recorded in real-time using the Robot Operating System (ROS), a popular framework for robotics research. Three primary methods were tested: Convolutional Neural Networks (CNNs), ARUCO Markers, and Hybrid Approaches.

CNNs are a deep learning approach trained to recognize patterns in visual data and predict the robotic arm’s pose. ARUCO markers use predefined visual markers to calculate position and orientation, providing consistent performance under low-light conditions and computational efficiency. Hybrid approaches combine elements of both CNNs and ARUCO markers to leverage their respective strengths.

The algorithms were tested across a range of scenarios, including varying lighting conditions, occlusions, and dynamic backgrounds, to assess their performance in real-world applications.

Key Findings

Each method demonstrated distinct advantages and trade-offs. CNNs showed exceptional accuracy in complex environments, particularly when trained on diverse datasets. However, they required significant computational resources and were sensitive to training biases. ARUCO markers provided consistent performance under low-light conditions and were computationally efficient. Their reliance on visible markers limited their applicability in scenarios where markers could be occluded or absent.

Hybrid approaches offered a balance between accuracy and efficiency, with the potential to adapt to changing environments by switching between methods based on real-time conditions. The study revealed that the choice of method significantly impacted the system’s ability to handle dynamic environments. CNNs excelled in scenarios with moving objects but struggled with unexpected or ambiguous visual data. In contrast, ARUCO markers proved more reliable in static environments but were less effective in highly dynamic settings.

Conclusion

This research underscores the importance of selecting appropriate pose estimation methods based on specific application requirements and constraints. While CNNs represent a powerful tool for complex scenarios, marker-based systems offer reliability in controlled environments. Hybrid approaches provide flexibility, adapting to varying conditions by combining the strengths of both methodologies.

As robotics continues to evolve, the development of robust pose estimation techniques will remain crucial for advancing automation and enabling robots to operate effectively in diverse settings.