Abstract
The use of flexible and autonomous robotic systems is a possible solution for automation in dynamic and unstructured industrial environments. Pick and place robotic applications are becoming common for the automation of manipulation tasks in an industrial context. This context requires the robot to be aware of its surroundings throughout the whole manipulation task, even after accomplishing the gripping action. This work introduces the deep post gripping perception framework, which includes post gripping perception abilities realized with the help of deep learning techniques, mainly unsupervised learning methods. These abilities help robots to execute a stable and precise placing of the gripped items while respecting the process quality requirements. The framework development is described based on the results of a literature review on post gripping perception functions and frameworks. This results in a modular design using three building components to realize planning, monitoring and verifying modules. Experimental evaluation of the framework shows its advantages in terms of process quality and stability in pick and place applications.
Issue Section:
Research Papers
References
1.
IFR
, 2018
, Robots and the Workplace of the Future, Technical Report, International Federation of Robotics Frankfurt
, Germany.2.
Lammer
, U. A.
, 2017
, “Funktionsvereinigung in der Lagertechnik
,” PhD thesis
, Technical University of Munich
, Munich, Germany
.3.
Winkelhake
, U.
, 2017
, Die Digitale Transformation Der Automobilindustrie: Treiber - Roadmap - Praxis
, Springer Vieweg, Wiesbaden, Germany
.4.
Salehi
, V.
, and Wang
, S.
, 2017
, “Using Point Cloud Technology for Process Simulation in the Context of Digital Factory Based on a Systems Engineering Integrated Approach
,” 21st International Conference on Engineering Design (ICED 17) Vol 3: Product Services and Systems Design
, Vancouver, British Columbia, Canada
.5.
Salehi
, V.
, 2019
, “Development of An Agile Concept for MBSE for Future Digital Products Through the Entire Life Cycle Management Called Munich Agile MBSE Concept (MAGIC)
,” Comput.-Aided Design Appl.
, 17
(1
), pp. 147
–166
. 10.14733/cadaps.2020.147-1666.
Salehi
, V.
, and Wang
, S.
, 2019
, “Munich Agile MBSE Concept (MAGIC)
,” Proc. Design Soc.: Int. Conf. Eng. Design
, 1
(1
), pp. 3701
–3710
. 10.1017/dsi.2019.3777.
Kurrek
, P.
, Zoghlami
, F.
, Jocas
, M.
, Stoelen
, M.
, and Salehi
, V.
, 2020
, “Q-model: An Artificial Intelligence Based Methodology for the Development of Autonomous Robots
,” ASME J. Comput. Inform. Sci. Eng.
, 20
(6
), p. 061009
. 10.1115/1.40469928.
Kurrek
, P.
, Zoghlami
, F.
, Jocas
, M.
, Stoelen
, M.
, and Salehi
, V.
, 2020
, “Reinforcement Learning Lifecycle for the Design of Advanced Robotic Systems
,” 3rd IEEE International Conference on Industrial Cyber-Physical Systems
, Tampere, Finland
.9.
Saloky
, T.
, and Seminsky
, J.
, 2005
, Artificial Intelligence and Machine Learning. KEGA research project, Virtual Program Modules of AI Systems
.10.
Russell
, S.
, and Norvig
, P.
, 2009
, Artificial Intelligence: A Modern Approach
§, 3rd ed, Prentice Hall Press
, Upper Saddle River, NJ
.11.
Ghosh
, D.
, Olewnik
, A.
, and Lewis
, K.
, 2017
, “Application of Feature-Learning Methods Toward Product Usage Context Identification and Comfort Prediction
,” ASME J. Comput. Inf. Sci. Eng.
, 18
(1
), p. 011004
. 10.1115/1.403743512.
Shaw
, S. B.
, and Singh
, A. K.
, 2014
, “A Survey on Cloud Computing
,” 2014 International Conference on Green Computing Communication and Electrical Engineering (ICGCCEE)
, Coimbatore, India
, pp. 1
–6
.13.
Steder
, B.
,2013
, “Feature-Based 3D Perception for Mobile Robots
,” PhD thesis
, Uni Freiburg
, Freiburg, Germany
.14.
Zoghlami
, F.
, Kurrek
, P.
, Jocas
, M.
, Masala
, G.
, and Salehi
, V.
, 2019
, “Usage Identification of Anomaly Detection in An Industrial Context
,” Proc. Design Soc.: Int. Conf. Eng. Design
, 1
(1
), pp. 3761
–3770
. 10.1017/dsi.2019.38315.
Kurrek
, P.
, Jocas
, M.
, Zoghlami
, F.
, Stoelen
, M.
, and Salehi
, V.
, 2019
, “AI Motion Control a Generic Approach to Develop Control Policies for Robotic Manipulation Tasks
,” Proc. Design Soc.: Int. Conf. Eng. Design
, 1
(1
), pp. 3561
–3570
. 10.1017/dsi.2019.36316.
Jocas
, M.
, Kurrek
, P.
, Zoghlami
, F.
, Gianni
, M.
, and Salehi
, V.
, 2019
, “AI-Based Learning Approach with Consideration of Safety Criteria on Example of a Depalletization Robot
,” Proc. Design Soc.: Int. Conf. Eng. Design
, 1
(1
), pp. 2041
–2050
. 10.1017/dsi.2019.21017.
Zhang
, H.
, Long
, P.
, Zhou
, D.
, Qian
, Z.
, Wang
, Z.
, Wan
, W.
, Manocha
, D.
, Park
, C.
, Hu
, T.
, Cao
, C.
, Chen
, Y.
, Chow
, M.
, and Pan
, J.
, 2016
, “Dorapicker: An Autonomous Picking System for General Objects
,” IEEE International Conference on Automation Science and Engineering (CASE)
, Fort Worth, TX
, pp. 721
–726
.18.
Poss
, C.
, Irrenhauser
, T.
, Prueglmeier
, M.
, Goehring
, D.
, Zoghlami
, F.
, Salehi
, V.
, and Ibragimov
, O.
, 2019
, “Enabling Robot Selective Trained Deep Neural Networks for Object Detection Through Intelligent Infrastructure
,” Proceedings of the 2019 4th International Conference on Automation, Control and Robotics Engineering, CACRE2019, Association for Computing Machinery
, Shenzhen, China
.19.
Poss
, C.
, Irrenhauser
, T.
, Prueglmeier
, M.
, Goehring
, D.
, Zoghlami
, F.
, and Salehi
, V.
, 2019
, “Perception Based Intelligent Materialhandling in Industrial Logistics Environments
,” Proceedings of the 2019 11th International Conference on Computer and Automation Engineering, ICCAE 2019
, Perth, Australia
, ACM, pp. 146
–151
.20.
Poss
, C.
, Mlouka
, O. B.
, Irrenhauser
, T.
, Prueglmeier
, M.
, Goehring
, D.
, Zoghlami
, F.
, and Salehi
, V.
, 2019
, “Robust Framework for Intelligent Gripping Point Detection
,” IECON 2019 – 45th Annual Conference of the IEEE Industrial Electronics Society
, Lisbon, Portugal
, Vol. 1
, pp. 717
–723
.21.
Schuster
, M. J.
, Okerman
, J.
, Nguyen
, H.
, Rehg
, J. M.
, and Kemp
, C. C.
, 2010
, “Perceiving Clutter and Surfaces for Object Placement in Indoor Environments
,” 2010 10th IEEE-RAS International Conference on Humanoid Robots
, Nashville, TN
, pp. 152
–159
.22.
Jiang
, Y.
, Lim
, M.
, Zheng
, C.
, and Saxena
, A.
, 2012
, “Learning to Place New Objects in a Scene
,” Int. J. Robot. Res.
, 31
(9
), pp. 1021
–1043
. 10.1177/027836491243878123.
Teodorescu
, C. S.
, Vandenplas
, S.
, Depraetere
, B.
, Anthonis
, J.
, Steinhauser
, A.
, and Swevers
, J.
, 2016
, “A Fast Pick-and-place Prototype Robot: Design and Control
,” IEEE Conference on Control Applications (CCA2016)
, Buenos Aires, Argentina
, pp. 1414
–1420
.24.
Rengervé
, A. D.
, Hirel
, J.
, Andry
, P.
, Quoy
, M.
, and Gaussier
, P.
, 2011
, “On-line Learning and Planning in a Pick-and-place Task Demonstrated Through Body Manipulation
,” IEEE International Conference on Development and Learning (ICDL)
, Genova, Italy
, Vol. 2
, pp. 1
–6
.25.
Correll
, N.
, Bekris
, K. E.
, Berenson
, D.
, Brock
, O.
, Causo
, A.
, Hauser
, K.
, Okada
, K.
, Rodriguez
, A.
, Romano
, J. M.
, and Wurman
, P. R.
, 2016
, “Analysis and Observations From the First Amazon Picking Challenge
,” IEEE Transactions on Automation Science and Engineering
, 15
(1
), pp. 172
–188
. 10.1109/TASE.2016.260052726.
Patil
, G. G.
, 2013
, “Vision Guided Pick and Place Robotic Arm System Based on SIFT
,” Int. J. Sci. Eng. Res.
, 4
(12
), pp. 242
–248
.27.
Cosgun
, A.
, Hermans
, T.
, Emeli
, V.
, and Stilman
, M.
, 2011
, “Push Planning for Object Placement on Cluttered Table Surfaces
,” 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems
, San Francisco, CA
, pp. 4627
–4632
.28.
Ongaro
, F.
, Yoon
, C.
, Abayazid
, M.
, Oh
, S. H.
, Gracias
, D. H.
, and Misra
, S.
, 2016
, “Control of Untethered Soft Grippers for Pick-and-place Tasks
,” 6th IEEE International Conference on Biomedical Robotics and Biomechatronics (BioRob)
, Singapore
, pp. 299
–304
.29.
Kootbally
, Z.
, Kramer
, T.
, Schlenoff
, C.
, and Gupta
, S.
, 2017
, “Overview of an Ontology-Based Approach for Kit Building Applications
,” pp. 520
–525
.30.
Altan
, D.
, and Sariel
, S.
, 2020
, “What Went Wrong?: Identification of Everyday Object Manipulation Anomalies
,”. ArXiv, abs/2001.09084, preprint submitted to Elsevier
.31.
Kothari
, C. R.
, 2004
, Research Methodology : Methods & Techniques
, New Age International (P) Ltd, New Delhi, India
.32.
Blessing
, L. T.
, and Chakrabarti
, A.
, 2009
, DRM, A Design Research Methodology
, Springer-Verlag London, London, UK
.33.
Council
, D.
, 2007
, Eleven Lessons: Managing Design in Eleven Global Brands. A Study of the Design Process
. Technical Report, Design Council.34.
Edsinger
, A.
, and Kemp
, C. C.
, 2006
, “Manipulation in Human Environments
,” 2006 6th IEEE-RAS International Conference on Humanoid Robots
, Genova, Italy
, pp. 102
–109
.35.
Yamashita
, T.
, Godler
, I.
, Takahashi
, Y.
, Wada
, K.
, and Katoh
, R.
, 1991
, “Peg-and-Hole Task by Robot With Force Sensor: Simulation and Experiment
,” Proceedings IECON’91: 1991 International Conference on Industrial Electronics, Control and Instrumentation
, Kobe, Japan
, Vol. 2
, pp. 980
–985
.36.
Bo
, C.
, Hu
, H.
, Xu
, J.
, Liu
, Z.
, and Chai
, P.
, 2017
, “Classification Learning Method for PCB Insertion Holes Based on Shape Context
,” Third IEEE International Conference on Computer and Communications (ICCC)
, Chengdu, China
, pp. 1933
–1937
.37.
Wang
, P.
, Qin
, Z.
, Xiong
, Z.
, Lu
, J.
, Xu
, D.
, Yuan
, X.
, and Liu
, C.
, 2015
, “Robotic Assembly System Guided by Multiple Vision and Laser Sensors for Large Scale Components
,” IEEE International Conference on Robotics and Biomimetics (ROBIO)
, Zhuhai, China
, pp. 1735
–1740
.38.
Karako
, Y.
, Moriya
, T.
, Abe
, M.
, Shimakawa
, H.
, Shirahori
, S.
, and Saitoh
, Y.
, 2017
, “A Practical Simulation Method for Pick-and-place with Vacuum Gripper
,” 56th Annual Conference of the Society of Instrument and Control Engineers of Japan (SICE)
, Kanazawa, Japan
, pp. 1351
–1356
.39.
Cappelleri
, D. J.
, Krishnan
, G.
, Kim
, C.
, Kumar
, V.
, and Kota
, S.
, 2010
, “Toward the Design of a Decoupled, Two-Dimensional, Vision-Based μN Force Sensor
,” ASME J. Mech. Rob.
, 2
(2
), p. 021010
. 10.1115/1.400109340.
Nozu
, K.
, and Shimonomura
, K.
, 2018
, “Robotic Bolt Insertion and Tightening Based on In-Hand Object Localization and Force Sensing
,” 2018 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM)
, Auckland, New Zealand
, pp. 310
–315
.41.
Hayakawa
, S.
, Yamada
, Y.
, and Tsuchida
, N.
, 2004
, “Shaft Insertion for Moving Object by Using Robot Manipulator With One Dimensional PSDs Sensor
,” Proceedings of the 2004 IEEE International Conference on Control Applications
, Taipei, Taiwan
, pp. 21118
–21123
.42.
De Gregorio
, D.
, Zanella
, R.
, Palli
, G.
, Pirozzi
, S.
, and Melchiorri
, C.
, 2019
, “Integration of Robotic Vision and Tactile Sensing for Wire-Terminal Insertion Tasks
,” IEEE Trans. Auto. Sci. Eng.
, 16
(2
), pp. 585
–598
. 10.1109/TASE.2018.284722243.
Erkent
, O.
, Shukla
, D.
, and Piater
, J.
, 2017
, “Visual Task Outcome Verification Using Deep Learning
,” 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
, Vancouver, BC, Canada
, pp. 4821
–4827
.44.
Yuan
, P.
, Zhong
, Y.
, and Yuan
, Y.
, 2017
, “Faster R-CNN With Region Proposal Refinement
”.45.
Krizhevsky
, A.
, Sutskever
, I.
, Hinton
, G. E.
, Pereira
, F.
, Burges
, C. J. C.
, Bottou
, L.
, and Weinberger
, K. Q.
, 2012
, Imagenet Classification With Deep Convolutional Neural Networks
, Curran Associates Inc.
, New York City, NY
, pp. 1097
–1105
.46.
Zhang
, Z.
, Liu
, J.
, Wang
, X.
, Zhao
, Q.
, Zhou
, C.
, Tan
, M.
, Pu
, H.
, Xie
, S.
, and Sun
, Y.
, 2017
, “Robotic Pick-and-Place of Multiple Embryos for Vitrification
,” IEEE Rob. Auto. Lett.
, 2
(2
), pp. 570
–576
. 10.1109/LRA.2016.264036447.
Komati
, B.
, Kudryavtsev
, A.
, Clévy
, C.
, Laurent
, G.
, Tamadazte
, B.
, Agnus
, J.
, and Lutz
, P.
, 2016
, “Automated Robotic Microassembly of Flexible Optical Components
,” IEEE International Symposium on Assembly and Manufacturing (ISAM)
, Fort Worth, TX
, pp. 93
–98
.48.
Harada
, K.
, Tsuji
, T.
, Nagata
, K.
, Yamanobe
, N.
, Onda
, H.
, Yoshimi
, T.
, and Kawai
, Y.
, 2012
, “Object Placement Planner for Robotic Pick and Place Tasks
,” IEEE/RSJ International Conference on Intelligent Robots and Systems
, Algarve, Portugal
, pp. 980
–985
.49.
Toris
, R.
, Kent
, D.
, and Chernova
, S.
, 2015
, “Unsupervised Learning of Multi-hypothesized Pick-and-Place Task Templates Via Crowdsourcing
,” 2015 IEEE International Conference on Robotics and Automation (ICRA)
, Seattle, WA
, pp. 4504
–4510
.50.
Cortes
, C.
, and Vapnik
, V.
, 1995
, “Support-Vector Networks
,” Mach. Learn.
, 20
(3
), pp. 273
–297
.51.
Baumgartl
, J.
, 2016
, “Schnelle, konservative Greif- und Ablageplanung von unbekannten Objekten
”. PhD thesis
, Universit ̈ut Bayreuth
, Bayreuth, Germany
.52.
Balakirsky
, S.
, and Kootbally
, Z.
, 2014
, An Ontology Based Approach to Action Verification for Agile Manufacturing
, Springer International Publishing
, Cham
, pp. 201
–217
.53.
Kootbally
, Z.
, Schlenoff
, C.
, Weisman
, T.
, Balakirsky
, S.
, Kramer
, T.
, and Pietromartire
, A.
, 2014
, A Simulated Sensor-Based Approach for Kit Building Applications
, Springer International Publishing
, Cham
, pp. 241
–257
.54.
Hochreiter
, S.
, and Schmidhuber
, J.
, 1997
, “Long Short-Term Memory
,” Neural Comput.
, 9
(8
), pp. 1735
–1780
. 10.1162/neco.1997.9.8.173555.
Chandola
, V.
, Banerjee
, A.
, and Kumar
, V.
, 2009
, “Anomaly Detection: A Survey
,” ACM Comput. Surv.
, 41
(3
), pp. 1
–72
. 10.1145/1541880.154188256.
Pimentel
, M. A. F.
, Clifton
, D. A.
, Clifton
, L. A.
, and Tarassenko
, L.
, 2014
, “A Review of Novelty Detection
,” Signal Process.
, 99
(1
), pp. 215
–249
. 10.1016/j.sigpro.2013.12.02657.
Redmon
, J.
, and Farhadi
, A.
,2018
, “Yolov3: An Incremental Improvement
,” Preprint available online at arXiv:1804.02767
.58.
Schölkopf
, B.
, Williamson
, R. C.
, Smola
, A. J.
, Shawe-Taylor
, J.
, Platt
, J. C.
, Solla
, S. A.
, Leen
, T. K.
, and Müller
, K.
, 2000
, Support Vector Method for Novelty Detection
, MIT Press
, Cambridge, MA
, pp. 582
–588
.59.
Zoghlami
, F.
, Kurrek
, P.
, Jocas
, M.
, Masala
, G.
, and Salehi
, V.
, 2020
, “Unsupervised Pose Anomaly Detection for Dynamic Robotic Environments
,” Third IEEE International Conference on Industrial Cyber-Physical Systems
, Tampere, Finland
.60.
Schlegl
, T.
, Seeböck
, P.
, Waldstein
, S. M.
, Schmidt-Erfurth
, U.
, and Langs
, G.
, 2017
, “Unsupervised Anomaly Detection With Generative Adversarial Networks to Guide Marker Discovery
,” 25th International conference on Information Processing in Medical Imaging (IPMI 2017)
, Boone, NC
.Copyright © 2020 by ASME
You do not currently have access to this content.
