Development of Annulus-Object Random Bin Picking System based on

Rapid Establishment of RGB-D Images

BO-RUI ZHU1, JIN-SIANG SHAW1, SHIH-HAO LEE2

National Taipei University of Technology,

Taipei,

TAIWAN

limited to fixed actions of the past. Production lines increasingly require the recognition and grasping of objects in

complex environments, emphasizing quick setup and stability. In this paper, a rapidly constructed eye-hand system for

robotic arm grasping, which enables fast and efficient object manipulation, particularly for stacking, is introduced.

Initially, images were captured using a camera to generate extensive datasets from a limited number of images.

Objects were subsequently segmented and categorized using deep learning networks for object detection and instance

segmentation. Three-dimensional position information was obtained from an RGB-D camera. Finally, object poses

were determined based on plane normal vectors, and gripping positions were manually marked. This reduced the time

required for grab-point identification, model training, and pose localization. Based on experimental results, the

grasping procedure proposed in this paper is suitable for various object-grasping scenarios. It achieved impressive

picking success rates of 96% for unstacked annular objects and 90.86% for random bin annular objects, respectively.

Received: May 21, 2023. Revised: December 22, 2023. Accepted: January 16, 2023. Published: February 28, 2024.

Object grasping is an important research topic in the

field of robotics. Random sorting (RBP) is an important

research direction for robotic arm grasping, and it

generally needs to be applied in the industry. Although

this action is simple for humans, it is challenging for a

robotic arm. Owing to the random stacking of objects,

companies have actively invested in this development.

laser measurement method was employed to generate

deep learning convolutional neural networks (CNN) for

sampling and evaluation, such as that proposed by

simulated an object model to obtain a synthetic dataset.

Finally, the object was cut out using the R-CNN to

obtain the object point cloud information. The

algorithm calculated the gripping position, [5]. The

above goals were to obtain correct identification and

clamping stability and to achieve a high clamping

success rate. However, these methods require the

creation of a large number of images and labeled data;

therefore, they are relatively time-consuming and

costly. However, in certain applications, setting up the

object was preset, and the center point of the object was

expected to be the clamping position. The desired

object was cropped from the RGB image using deep

learning, and the 3D position of the object was

obtained through algorithms and camera depth

information. Subsequently, the posture and position of

the object relative to the camera were obtained from the

3D information of the object, and the information

obtained by the camera was converted into the

information required by the robotic arm, such that the

(GGCNN) for grasp visualization, [6] or [7], where a

hybrid deep structure of visual and tactile sensing using

a reference rectangle method to identify graspable

regions of an image was proposed. Various methods,

used the powerful learning ability of large

convolutional networks to predict the global grasp of a

complete image of an object, [8]. ResNet50 was used

for feature extraction, [9]. Although the accuracy of

ResNet50 is good, its speed is low. We chose a lighter

Inception model architecture and only used depth

In the follow-up, we used a calculation method

Before deep learning, apart from data collection,

image annotation is the most time-consuming task.

Humans can easily perceive the location and type of

objects. However, before entering deep learning, first,

the objects must be circled, and the machines need to

be trained on the object types. This process is called

data annotation. To expedite object labeling, the

aforementioned data collection method can be applied

for expediting image labeling. The same method can

be used to copy the frame selection object frame, on

could only be recognized when the occluded range was

and number of annotations required for the image, as

shown in Figure 9.

Fig. 9: Schematic of image annotation

3.2.3 Object Recognition

In deep learning object recognition, we used the Mask

R-CNN proposed in [17]. Mask R-CNN is a very

Therefore, in this study, the center point of the object

Fig. 10: The common grab detection process

3.3.2 Grasping Pose Evaluation of Objects

Evaluating the power tape in the box and the posture

of the object, which is related to the gripping and

success rate of gripping, is necessary. The scattered

and unstacked objects usually have two attitudes: lying

flat or standing, as shown in Figure 11. Among these,

Thus, the robotic arm could not grasp the standing

object at a position relative to the center point of the

object. Therefore, if the object was standing or its

center point was unrecognizable, the object could not

be grasped, and a mechanical arm had to be used. The

arm pushes the object down to a flat state, re-identifies

it, and clamps it.

the objects are inclined to a certain angle, as shown in

Figure 12. The attitude angle of the object to the camera

angle.

information was used to determine the object pose. First,

the objects cut by the Mask R-CNN instance were

divided into four regions, as shown in Figure 13. Then,

we extracted seven 3D spatial information points from

each of the four regions and one 3D spatial information

point from each of the three regions.

󰇜and set the plane normal vector as 

 󰇛󰇜.

Figure 14. Then, the center point of the object was set as



 

   (4)

Fig. 14: Schematic of normal vector angle calculation

Fig. 15: Object Plane Quadrant Angle

through three pre-set positions, and finally, the arm was

allowed to perform the grasping action.

From the experiments, we verified the results and

training time of deep learning using Inception.

arm grabbed them until all the objects were clamped or

their image could not be recognized. When the

the correct and incorrect attitude evaluation, failure to

grasp, and misplacement errors caused by incomplete

was accomplished using the correct posture. In several

successful. The majority of gripping failure instances

calculating the position and attitude and insufficient

the misjudgment of attitude. In addition, the small

incomplete cutting of the instance. Even if the grasping

To improve the stability of the abovementioned

In this study, robotic arms, computer vision, and deep

learning were integrated to develop a simplified and

faster-to-implement grasping system compared to the

time-consuming and intricate stacking grasping systems.

The focus was on mass-produced objects. In the

Furthermore, in contrast to other studies, our approach

identified. In this study, we focused on a single-

object type, which limits its applicability. Our

goal was to enable the robotic arm to handle

various objects that could be randomly stacked

within the grasping area without compromising

the success rate.

accuracy by employing a more stable RGB-D

camera, thus enhancing the overall attitude

relying on force feedback to determine whether

the gripper has contacted the intended object. By

analyzing the force data, the robotic arm can

assess if a successful grip has been achieved or if

it should return to the grasping waiting area for

image recognition.