Automation of hydraulic excavators is valuable due to their potential applications in hazardous environments or remote locations, such as radioactive-contaminated areas. Manipulator position sensing is a key issue in the study of hydraulic excavator automation.
A neural network-based computer vision system was designed using MathWorks’ MATLAB neural network toolbox and used to estimate the boom, arm, and bucket cylinder displacements of an excavator manipulator during a grading operation simulation. A computer ran the excavator simulation, and a webcam, which was connected to the computer, took snapshots of the excavator manipulator animation displayed on a secondary screen. The webcam took screenshots of the manipulator at different positions during a grading operation. Those images were then down-sampled and used to train the neural network. The researchers from Volvo Construction Equipment and The University of Alabama then compared the manipulator positions estimated by the neural network-based computer vision system with the actual values.
The neural network-based computer vision system consists of three main subsystems: excavator manipulator simulation model, image acquisition subsystem, and neural network subsystem. The excavator manipulator simulation model was developed in Simulink. The simulation model consists of three main subsystems: hydraulic subsystem, kinematic subsystem, and a proportional integral (PI) controller, as shown in Figure 1.
The hydraulic subsystem and the kinematic subsystem were modeled using SimHydraulics and SimMechanics toolboxes in Simulink, respectively. Three independent PI controllers were applied to control the excavator manipulator during the grading process.
The image acquisition subsystem consists of four main hardware components: a computer that runs the grading operation simulation, a secondary monitor that visualizes the manipulator grading operation, a webcam to take snapshots of the manipulator animation, and an aluminum frame to support the webcam and the secondary screen.
For control of the webcam, the MATLAB image acquisition toolbox was used. The screenshots were down-sampled using a code developed in MATLAB to reduce the image resolution.
The grading operation simulation was visualized using the SimMechanics visualization function. The viewpoint for the manipulator animation was chosen to simulate the real situation in which a camera is installed on the excavator cab moving together with the cab such that the relative position between the manipulator anchor and the camera is always fixed. An exact image is shown in Figure 2.
In the experiment, the manipulator model simulated a 10-second grading operation in which the bucket tip moved inward along a level path. To create sample images, a snapshot was taken at every 0.01 second during the grading cycle simulation, and 80 images were taken for each position by repeating the process 80 times. A total of 80,080 sample images were created using the sample image creation program in MATLAB.
The neural network subsystem consists of three independent neural networks developed using the neural network toolbox in MATLAB. Each neural network was used to estimate displacement of one of the hydraulic cylinders: boom, arm, and bucket, independently.
Following the training process, the trained neural networks were used to estimate the displacements of the hydraulic cylinders in the manipulator during the simulation of a grading operation. The results of the cylinder displacement estimation during the simulation were compared with the actual values.
The simulation results were obtained for three different illumination conditions. The estimation errors for the three cylinders were small compared to the dimension of the excavator manipulator. The relatively low estimation errors show the capability of the neural networks to estimate the position of the excavator manipulator for feedback control applications.
Although the proposed methodology was demonstrated with a single manipulator system, the same principle and algorithm can be applied to other excavator models with different shapes and dimensions. Future work will involve the creation of a larger set of sample images under various illumination conditions with different backgrounds for training the neural networks. Then the estimated manipulator cylinder displacement values will be used for feedback control of the hydraulic cylinders to automate the grading operations.
This article is based on SAE technical paper 2016-01-8122 authored by Jiaqi Xu and Hwan-Sik Yoon of The University of Alabama; and Jae Y. Lee and Seonggon Kim of Volvo Construction Equipment. The paper will be presented at the SAE 2016 Commercial Vehicle Engineering Congress.