Soft Robotic Dynamic In-Hand Pen Spinning

Carnegie Mellon University

Abstract

Dynamic in-hand manipulation remains a challenging task for soft robotic systems that have demonstrated advantages in safe compliant interactions but struggle with high-speed dynamic tasks. In this work, we present SWIFT, a system for learning dynamic tasks using a soft and compliant robotic hand. Unlike previous works that rely on quasi-static actions and precise object models, the proposed system learns to spin a pen through trial-and-error using only real-world data without requiring explicit prior knowledge of the pen's physical attributes. With self-labeled trials sampled from the real world, the system discovers the set of pen grasping and spinning primitive parameters that enables a soft hand to spin a pen robustly and reliably. After 130 sampled actions, SWIFT achieves 100 % success rate across three pens with different weights and weight distributions, demonstrating the system's generalizability and robustness to changes in object properties. The results highlight the potential for soft robotic end-effectors to perform dynamic tasks including rapid in-hand manipulation. We also demonstrate that SWIFT generalizes to spinning tools with different shapes and weights such as a brush and a screwdriver which we spin with 10/10 and 5/10 success rates respectively

Soft robot hardware

We tackle the problem of in-hand dynamic pen spinning with MOE, a soft robotic end-effector. Our MOE is made up of three tendon-driven soft robot fingers, each driven by two servos controlling four tendons. The left figure shows the MOE unactuated and actuated by the tendons. The servos pull the tendons to bend the finger in perpendicular planes, and combining the servo motions can actuate each finger tip of MOE hand to reach locations on its semi-hemisphere workspace. The video on the right shows the motions that can be achieved by each finger.

Description of the SVG

Method Overview

There are 4 main stages for each iteration k: 1) During grasping and resetting, the robot arm moves the MOE hand to a target grasp location following a specific grasping location gk . 2) The robot arm then moves the MOE hand to the pre-spin configuration, where the MOE fingers execute the parameterized action. 3) An RGB- D camera records the trial, and we apply masks from SAM-v2 to create segmented point cloud. We then apply other post-processing on the point cloud to get rotation and displacement state of the pen. 4) Lastly, the system evaluates the objective function with observed states of the pen and updates the action parameters with CMA-ES

Description of the SVG

Resetting and CMA-ES updates

(Left) The system is reset by a robot arm picking the pen from the work surface. The fixture on the table offers more consistency for the MOE-hand to grasp onto the pen.(Right) Shows the CMA-ES updates where more successes are found over generations. The action that produced the highest reward over all the generations are chosen as the final action

All Evaluations

We evaluate the performance of our system on pens with different center of gravities. We also evaluate the system using commonly seen objects, such as a paint brush and a screwdriver.

Action parameters

The action parameters are composed of three parts: 1. A grasping position instructing the robot where to pick the pen, 2. A set of 6 target angles for the servos on the robot fingers to reach, and 3. An amount of delay in seconds for the robot to wait between spinning and catching motions. The catching motion is simplified to be the reverse motion of the spin motion.

State estimation pipeline

The state estimation starts with finding the salient markers in the view. The center of the markers are then sent to Segment Anything v2 to track the motion of the pen. Using the RGB-D camera, the point clouds of the trial are filtered by the mask and a bounding box around the MOE-finger tips. Finally, PCA is applied to find the amount of rotation that the pen has taken during each trial. The reward function encourages more rotation while penalizes every frame where the pen is too far from the fingers.

BibTeX

Coming soon