Abstract

Automatic guided vehicles (AGVs) and autonomous mobile robots (AMRs) are now ubiquitous in industrial manufacturing environments. These systems all must possess a similar set of core capabilities, including navigation, obstacle avoidance, and localization. However, there are few standard methods to evaluate the capabilities and limitations of these systems in a way that is comparable. In this article, a standard test method is presented that can be used to evaluate these capabilities and can be easily scaled and augmented according to the characteristics of the system under test. The test method can be configured in a variety of ways to exercise different capabilities, all using a common test apparatus to ease test setup and increase versatility. For each test configuration, conditions are specified with respect to the a priori knowledge provided to the system (e.g., boundary or obstacle locations) and the obstacles in the environment. Robustness of system capabilities is evaluated by purposefully introducing misalignment between the characteristics of the physical and virtual environments (e.g., providing representations of obstacles in the system’s map when they are not physically present). Example test performance data from an AMR are provided. The goal of this work is to provide a common method to characterize the performance of mobile systems in industrial environments that is easily comparable and communicated for both commercial and developmental purposes. This work is driven by existing standards and those in development by the ASTM F45 Committee on Driverless Automatic Guided Industrial Vehicles and will influence the development of new standards within the committee.

Issue Section:
Technical Papers
Keywords:
automatic guided vehicles, autonomous mobile robots, navigation, obstacle avoidance, localization, standard test methods, evaluation, benchmarking

References

1.
Safety Standard for Driverless, Automatic Guided Industrial Vehicles and Automated Functions of Manned Industrial Vehicles
, ANSI/ITSDF B56.5-2012 (
Washington, DC
:
Industrial Truck Standards Development Foundation
,
2012
).
2.
Robots and Robotic Devices–Safety Requirements–Non-medical Personal Care Robot
, ISO 13482:2014 (
Geneva, Switzerland
:
International Organization for Standardization
,
2014
).
3.
Schneier
 
M.
 and
Bostelman
 
R.
,
Literature Review of Mobile Robots for Manufacturing, NISTIR 8022
(
Gaithersburg, MD
:
U.S. Department of Commerce, National Institute of Standards and Technology
,
2015
).
4.
Standard Terminology for Driverless Automatic Guided Industrial Vehicles
, ASTM F3200-18a (
West Conshohocken, PA
:
ASTM International
, approved April 1,
2018
). https://doi.org/10.1520/F3200-18A
5.
OMRON Corporation “
LD Series Mobile Robots/Lineup
,” OMRON Industrial Automation,
2019
. http://web.archive.org/web/20191029180403/http://www.ia.omron.com/products/family/3664/lineup.html
6.
America in Motion “
America in Motion - Our Products
,” America in Motion: AGV Systems, 2019. http://web.archive.org/web/20191029180601/https://www.weareaim.com/Products.aspx
8.
Seegrid “
Platform - Seegrid
,” Seegrid,
2019
. http://web.archive.org/web/20191029180928/https://seegrid.com/platform/
10.
Tzafestas
 
S. G.
, “
Mobile Robot Control and Navigation: A Global Overview
,”
Journal of Intelligent & Robotic Systems
 
91
, no. 
1
(July
2018
):
35
–
58
. https://doi.org/10.1007/s10846-018-0805-9
11.
Hoy
 
M.
,
Matveev
 
A. S.
, and
Savkin
 
A. V.
, “
Algorithms for Collision-Free Navigation of Mobile Robots in Complex Cluttered Environments: A Survey
,”
Robotica
 
33
, no. 
3
(March
2015
):
463
–
497
. https://doi.org/10.1017/S0263574714000289
12.
Pandey
 
A.
,
Parhi
 
D. R.
, and
Pandey
 
S.
, “
Mobile Robot Navigation and Obstacle Avoidance Techniques
,”
International Robotics & Automation Journal
 
2
, no. 
3
(May
2017
):
96
–
105
.
13.
Ceballos
 
N. D. M.
,
Valencia
 
J. A.
, and
Ospina
 
N. L.
, “
Quantitative Performance Metrics for Mobile Robots Navigation
,” in
Mobile Robots Navigation
, ed.
Barrera
 
A.
 (
Rijeka, Croatia
:
InTech
,
2010
).
14.
Bostelman
 
R.
,
Hong
 
T.
, and
Cheok
 
G.
, “
Navigation Performance Evaluation for Automatic Guided Vehicles
,” in
2015 IEEE International Conference on Technologies for Practical Robot Applications (TePRA)
(
Piscataway, NJ
:
Institute of Electrical and Electronics Engineers
,
2015
).
15.
Standard Test Method for Grid-Video Obstacle Measurement
, ASTM F3265-17 (
West Conshohocken, PA
:
ASTM International
, approved July 15,
2017
). https://doi.org/10.1520/F3265-17
16.
Norton
 
A.
 and
Yanco
 
H.
, “
Preliminary Development of a Test Method for Obstacle Detection and Avoidance in Industrial Environments
,” in
Autonomous Industrial Vehicles: From the Laboratory to the Factory Floor
, ed.
Bostelman
 
R.
 and
Messina
 
E.
 (
West Conshohocken, PA
:
ASTM International
,
2016
),
23
–
40
. https://doi.org/10.1520/STP159420150059
17.
Yoon
 
S.
 and
Bostelman
 
R.
, “
Analysis of Automatic through Autonomous–Unmanned Ground Vehicles (A-UGVs) Towards Performance Standards
,” in
2019 IEEE International Symposium on RObotic and SEnsors Environments (ROSE)
(
Piscataway, NJ
:
Institute of Electrical and Electronics Engineers
,
2019
).
18.
Standard Test Method for Navigation: Defined Area
, ASTM F3244-17 (
West Conshohocken, PA
:
ASTM International
, approved July 1,
2017
). https://doi.org/10.1520/F3244-17
19.
Standard Practice for Recording Environmental Effects for Utilization with A-UGV Test Methods
, ASTM F3218-17 (
West Conshohocken, PA
:
ASTM International
, approved July 1,
2017
). https://doi.org/10.1520/F3218-17
20.
Standard Practice for Recording the A-UGV Test Configuration
, ASTM F3327-18 (
West Conshohocken, PA
:
ASTM International
, approved July 1,
2018
). https://doi.org/10.1520/F3327-18
21.
Madhavan
 
R.
,
Bostelman
 
R.
,
Kootbally
 
Z.
,
Lakaemper
 
R.
,
Gupta
 
S. K.
, and
Balakirsky
 
S.
, “
Smart, Flexible, and Safety Industrial Mobile Robots: Performance Evaluation & Standardization Efforts
,” in
2011 ITEA Annual Technology Review Conference
(
Fairfax, VA
:
International Test and Evaluation Association
,
2011
).
22.
Huang
 
H.-M.
,
Pavek
 
K.
,
Novak
 
B.
,
Albus
 
J.
, and
Messina
 
E.
, “
A Framework for Autonomy Levels for Unmanned Systems (ALFUS)
,” in
Proceedings of the AUVSI’s Unmanned Systems North America Conference
(
Arlington, VA
:
Association for Unmanned Vehicle Systems International
,
2005
).
23.
Huang
 
H.-M.
,
Messina
 
E.
,
Jacoff
 
A.
,
Wade
 
R.
, and
McNair
 
M.
, “
Performance Measures Framework for Unmanned Systems (PerMFUS): Models for Contextual Metrics
,” in
Proceedings of the 2010 Performance Metrics for Intelligent Systems Workshop
(
Gaithersburg, MD
:
National Institute of Standards and Technology
,
2010
).
24.
Bostelman
 
R.
 and
Messina
 
E.
, “
A-UGV Capabilities
,” in
2019 Third IEEE International Conference on Robotic Computing (IRC)
(
Piscataway, NJ
:
Institute of Electrical and Electronics Engineers
,
2019
).
25.
Gill
 
J. S.
,
Tomaszewski
 
M.
,
Jia
 
Y.
,
Pisu
 
P.
, and
Krovi
 
V. N.
,
Evaluation of Navigation in Mobile Robots for Long-Term Autonomy in Automotive Manufacturing Environments, SAE Technical Paper
(
Warrendale, PA
:
SAE International
,
2019
).
26.
Open Source Robotics Foundation “
ROS.org, Powering the World’s Robots
,” ROS.org. http://web.archive.org/web/20191029181839/https://www.ros.org/
27.
Open Source Robotics Foundation “
amcl - ROS Wiki
,” ROS.org. http://web.archive.org/web/20191029181943/http://wiki.ros.org/amcl
28.
Open Source Robotics Foundation “
move_base, ROS Wiki
,” ROS.org. http://web.archive.org/web/20191029182024/http://wiki.ros.org/move_base
29.
Durrant-Whyte
 
H.
 and
Bailey
 
T.
, “
Simultaneous Localization and Mapping: Part I
,”
IEEE Robotics & Automation Magazine
 
13
, no. 
2
(June
2006
):
99
–
110
. https://doi.org/10.1109/MRA.2006.1638022
This content is only available via PDF.
All rights reserved. This material may not be reproduced or copied, in whole or part, in any printed, mechanical, electronic, film, or other distribution and storage media, without the written consent of ASTM International.
You do not currently have access to this content.