Demo

Vid. 1 Control the arm remotely to pick up items. 0 s~ 20 s: The control effect without smart golve 20 s~ 57 s: Better control effect with a smart golve

Project Objective

XArm is a gesture-controlled robotic arm project with the goal of creating a mechanical arm that can synchronously mimic human hand movements.

The project’s features:

  • multidimensional hand data acquisition
  • wireless control system
  • robotic arm posture calculation and inverse kinematics

XArm collects hand movement and posture data through image recognition and a sensor glove. After algorithmic analysis, sensors data is converted into motion commands for the robotic arm, which are then wirelessly used to control the movement of the robotic arm.

System Architecture

The XArm mainly consists of 4 modules:

  1. Image-based hand gesture recognition (leapmotion).
  2. Wireless nine-axis IMU and hand gesture recognition glove (smart glove).
  3. Sensor data gather & inverse kinematics for movement commands calculation (control center).
  4. Six-axis wireless robotic arm (XArm)
Fig. 1 System architecture

Hardware

Software

Smart glove and XArm MCU was developed by C, a ESP32 was used as MCU and WiFi module in these parts. Control center was developed by python and was running on a labtop. The posture calculation and inverse kinematics was also written by python and running in control center. The image-based hand gesture recognition module was connecting to control center via USB.
The main design mainly consists of 3 modules:

  1. smart glove: Collect nine-axis posture and gesture. Smart glove was designed as an optional part of the XArm, when glove was not online, the control_center will do it best to calculate hand posture from image recognition. And the glove can be online at any time, control center will automatically detect the glove and taking it into the sensors system even the XArm is running.
  2. control center: Control whole system，gather all info of various sensors，calculate posture of robotic arm. Solve movement commands from sensor data and robotic arm posture via inverse kinematics.
  3. XArm: XArm is a six-axis robotic arm featuring an ESP32 as the wireless control center and MCU. Within this MCU, status reporting, command reception and verification, and execution are all handled on a single core. To provide a real-time control user experience, we employ a TDD system to schedule and manage these tasks, resulting in a seamless control experience.