[1]金寿祥,周宏平,姜洪喆,等.采摘机器人视觉系统研究进展[J].江苏农业学报,2023,(02):582-595.[doi:doi:10.3969/j.issn.1000-4440.2023.02.033]
　JIN Shou-xiang,ZHOU Hong-ping,JIANG Hong-zhe,et al.Research progress on visual system of picking robot[J].,2023,(02):582-595.[doi:doi:10.3969/j.issn.1000-4440.2023.02.033]
点击复制

采摘机器人视觉系统研究进展()
分享到：

江苏农业学报[ISSN:1006-6977/CN:61-1281/TN]

卷:
期数:
2023年02期
页码:
582-595
栏目:
综述
出版日期:
2023-04-30

文章信息/Info

Title:
Research progress on visual system of picking robot
作者:
金寿祥; 周宏平; 姜洪喆; 孙梦梦
(南京林业大学机械电子工程学院，江苏南京210037)
Author(s):
JIN Shou-xiang; ZHOU Hong-ping; JIANG Hong-zhe; SUN Meng-meng
(School of Mechanical and Electronic Engineering, Nanjing Forestry University， Nanjing 210037， China)
关键词:
视觉数据; 采摘点识别; 三维定位; 深度学习
Keywords:
visual data; picking point identification; three dimensions positioning; deep learning
分类号:
TP242.6+2
DOI:
doi:10.3969/j.issn.1000-4440.2023.02.033
文献标志码:
A
摘要:
视觉是采摘机器人感知外部环境的重要手段之一。采摘机器人的采摘效率与准确率很大程度上受到其视觉子系统识别与定位性能影响。近年来，研究者围绕采摘机器人的视觉系统开展了大量研究。伴随人工智能深度学习的发展，机器人视觉研究取得了较大进步，采摘机器人进入了实际应用阶段。本文针对采摘机器人视觉系统研究现状，主要从视觉数据获取方法、采摘点识别技术和采摘点定位技术3个方面进行总结分析，指出了当前采摘机器人采摘点识别与定位过程中面临的一些挑战，最后对未来采摘机器人视觉系统的发展进行了展望。
Abstract:
Vision is one of the important means for picking robots to perceive the external environment. The picking efficiency and accuracy of the picking robots are largely affected by the recognition and positioning performance of its vision subsystem. In recent years, a lot of researches have been carried out regarding the visual system of picking robots. With the development of artificial intelligence deep learning, research of robot vision has made great progress, which opened the stage of practical application of picking robots. In view of the research status of the picking robots visual system, this paper summarized and analyzed from three aspects, such as visual data acquisition method, picking point recognition technology and picking point positioning technology, and pointed out some challenges faced by the current picking robots in the process of picking point identification and positioning. Finally, the development of visual system for picking robots in the future was prospected.

参考文献/References:

[1]张文翔, 张兵园, 贡 宇, 等. 果蔬采摘机器人机械臂研究现状与展望[J]. 中国农机化学报, 2022,43(9): 232-237.
[2]WANG Z H, XUN Y, WANG Y K, et al. Review of smart robots for fruit and vegetable picking in agriculture[J]. International Journal of Agricultural and Biological Engineering, 2021,14(6): 33-54.
[3]VERBIEST R, RUYSEN K, VANWALLEGHEM T, et al. Automation and robotics in the cultivation of pome fruit: where do we stand today?[J]. Journal of Field Robotics, 2021,38(4): 513-531.
[4]BAI Y H, GUO Y X, ZHANG Q, et al. Multi-network fusion algorithm with transfer learning for green cucumber segmentation and recognition under complex natural environment[J]. Computers and Electronics in Agriculture, 2022,194: 1-27.
[5]姚成胜,肖雅雯,杨一单. 农业劳动力转移与农业机械化对中国粮食生产的关联影响分析[J]. 农业现代化研究, 2022,43(2): 1-15.
[6]MONTOYA-CAVERO L, DAZ DE LEN TORRES R, GMEZ-ESPINOSA A, et al. Vision systems for harvesting robots: produce detection and localization[J]. Computers and Electronics in Agriculture, 2022,192: 1-27.
[7]李会宾,史云. 果园采摘机器人研究综述[J]. 中国农业信息, 2019,31(6): 1-9.
[8]刘妤,刘洒,杨长辉,等. 基于双目立体视觉的重叠柑橘空间定位[J]. 中国农业科技导报, 2020,22(9): 104-112.
[9]JUN J, KIM J, SEOL J, et al. Towards an efficient tomato harvesting robot: 3D perception, manipulation, and end-effector[J]. IEEE Access, 2021,9: 17631-17640.
[10]LV J D, WANG Y J, NI H M, et al. Method for discriminating of the shape of overlapped apple fruit images[J]. Biosystems Engineering, 2019,186: 118-129.
[11]刘振宇,丁宇祺. 自然环境中被遮挡果实的识别方法研究[J]. 计算机应用研究, 2020,37(S2): 333-335.
[12]何斌,张亦博,龚健林,等. 基于改进YOLOv5的夜间温室番茄果实快速识别[J]. 农业机械学报, 2022,53(5): 1-10.
[13]雷旺雄,卢军. 葡萄采摘机器人采摘点的视觉定位[J]. 江苏农业学报, 2020,36(4): 1015-1021.
[14]陈志健,伍德林,刘路,等. 复杂背景下油茶果采收机重叠果实定位方法研究[J]. 安徽农业大学学报, 2021,48(5): 842-848.
[15]汪杰,陈曼龙,李奎,等. 基于HSV与形状特征融合的花椒图像识别[J]. 中国农机化学报, 2021,42(10): 180-185.
[16]杨帆,李鹏飞,刘庚,等. 橘子采摘机器人目标识别定位方法与实验研究[J]. 西安理工大学学报, 2018,34(4): 460-467.
[17]王瑾,王瑞荣,李晓红. 番茄采摘机器人目标识别方法研究[J]. 江苏农业科学, 2021,49(20): 217-222.
[18]赵立新,邢润哲,白银光,等. 深度学习在目标检测的研究综述[J]. 科学技术与工程, 2021,21(30): 12787-12795.
[19]包晓敏,王思琪. 基于深度学习的目标检测算法综述[J]. 传感器与微系统, 2022,41(4): 5-9.
[20]杨长辉,刘艳平,王毅,等. 自然环境下柑橘采摘机器人识别定位系统研究[J]. 农业机械学报, 2019,50(12): 14-22.
[21]傅隆生,冯亚利,ELKAMIL T,等. 基于卷积神经网络的田间多簇猕猴桃图像识别方法[J]. 农业工程学报, 2018,34(2): 205-211.
[22]TANG Y C, CHEN M Y, WANG C L, et al. Recognition and localization methods for vision-based fruit picking robots: a review[J]. Frontiers in Plant Science, 2020,11: 1-17.
[23]魏宏飞,张季萌. 基于CMOS传感器的采摘机器人计算机视觉系统研究[J]. 农机化研究, 2022,44(12): 221-224.
[24]JIAO Y H, LUO R, LI Q W, et al. Detection and localization of overlapped fruits application in an apple harvesting robot[J]. Electronics, 2020,9(6): 1-14.
[25]MALIK M H, ZHANG T, LI H, et al. Mature tomato fruit detection algorithm based on improved HSV and watershed algorithm[J]. IFAC-PapersOnLine, 2018,51(17): 431-436.
[26]牛晗,伍希志. 基于大津算法连通域的松果多目标识别定位[J]. 江苏农业科学, 2021,49(15): 193-198.
[27]马帅,张艳,周桂红,等. 基于改进YOLOv4模型的自然环境下梨果实识别[J]. 河北农业大学学报, 2022,45(3): 105-111.
[28]XIONG J T, LIU Z, LIN R, et al. Green grape detection and picking-point calculation in a night-time natural environment using a charge-coupled device (CCD) vision sensor with artificial illumination[J]. Sensors, 2018,18(4): 1-17.
[29]毕松,高峰,陈俊文,等. 基于深度卷积神经网络的柑橘目标识别方法[J]. 农业机械学报, 2019,50(5): 181-186.
[30]ZHOU H Y, WANG X, AU W, et al. Intelligent robots for fruit harvesting: recent developments and future challenges[J]. Precision Agriculture, 2022,23(5): 1856-1907.
[31]刘妤,刘洒,杨长辉,等. 基于双目立体视觉的重叠柑橘空间定位[J]. 中国农业科技导报, 2020,22(9): 104-112.
[32]邹朋朋,张滋黎,王平,等. 基于共线向量与平面单应性的双目相机标定方法[J]. 光学学报, 2017,37(11): 244-252.
[33]杨皓天,万腾. 葡萄采摘机械臂的双目定位与抓取精度研究[J]. 农机化研究, 2022,44(12): 49-54.
[34]LING X, ZHAO Y S, GONG L, et al. Dual-arm cooperation and implementing for robotic harvesting tomato using binocular vision[J]. Robotics and Autonomous Systems, 2019,114: 134-143.
[35]庞超凡. 基于双目视觉小金桔果实的识别及定位采摘研究[D]. 郑州：河南农业大学, 2021.
[36]曹春卿,张吴平,李富忠,等. 自然场景下多目标苹果识别定位融合算法研究[J]. 湖北农业科学, 2022,61(7): 145-151.
[37]陈炎,杨丽丽,王振鹏. 双目视觉的匹配算法综述[J]. 图学学报, 2020,41(5): 702-708.
[38]LI Y J, FENG Q C, LI T, et al. Advance of target visual information acquisition technology for fresh fruit robotic harvesting: a review[J]. Agronomy, 2022,12(6): 1-19.
[39]FU L S, GAO F F, WU J Z, et al. Application of consumer RGB-D cameras for fruit detection and localization in field: a critical review[J]. Computers and Electronics in Agriculture, 2020,177: 1-12.
[40]GONGAL A, AMATYA S, KARKEE M, et al. Sensors and systems for fruit detection and localization: a review[J]. Computers and Electronics in Agriculture, 2015,116: 8-19.
[41]SUN Q X, CHAI X J, ZENG Z K, et al. Noise-tolerant RGB-D feature fusion network for outdoor fruit detect[J]. Computers and Electronics in Agriculture, 2022,198: 1-13.
[42]赵辉,李浩,岳有军,等. 基于RGB-D相机的矮砧苹果识别与定位[J]. 计算机工程与设计, 2020,41(8): 2278-2283.
[43]刘景娜. 基于Kinect的移动式番茄生长信息采集系统的研制[D]. 南京：南京农业大学, 2020.
[44]孙宝霞,郑镇辉,胡文馨,等. 基于RGB-D的龙眼实时检测与定位方法[J]. 林业工程学报, 2022,7(3): 150-157.
[45]张勤,陈建敏,李彬,等. 基于RGB-D信息融合和目标检测的番茄串采摘点识别定位方法[J]. 农业工程学报, 2021,37(18): 143-152.
[46]彭孝东,时磊,何静,等. 消费级RGB-D相机在农业领域应用现状与发展趋势[J]. 中国农机化学报, 2022,43(4): 206-215.
[47]彭育辉,江铭,马中原,等. 汽车自动驾驶关键技术研究进展[J]. 福州大学学报(自然科学版), 2021,49(5): 691-701.
[48]罗玉涛,秦瀚. 基于稀疏彩色点云的自动驾驶汽车3D目标检测方法[J]. 汽车工程, 2021,43(4): 492-500.
[49]GEN-MOLA J, GREGORIO E, GUEVARA J, et al. Fruit detection in an apple orchard using a mobile terrestrial laser scanner[J]. Biosystems Engineering, 2019,187: 171-184.
[50]MNDEZ V, PREZ-ROMERO A, SOLA-GUIRADO R, et al. In-field estimation of orange number and size by 3D laser scanning[J]. Agronomy, 2019,9(12): 1-18.
[51]TANG J, JIANG F G, LONG Y, et al. Identification of the yield of camellia oleifera based on color space by the optimized mean shift clustering algorithm using terrestrial laser scanning[J]. Remote Sensing, 2022,14(3): 1-18.
[52]温玉维,邓长勇,曾德培,等. 三维激光扫描仪在电力工程实测实量中的应用[J]. 测绘通报, 2021(10): 163-167.
[53]TSOULIAS N, PARAFOROS D S, XANTHOPOULOS G, et al. Apple shape detection based on geometric and radiometric features using a LiDAR laser scanner[J]. Remote Sensing, 2020,12(15): 1-18.
[54]WU Y T, WANG Y Y, ZHANG S W, et al. Deep 3D object detection networks using LiDAR data: a review[J]. IEEE Sensors Journal, 2021,21(2): 1152-1171.
[55]JIA W K, ZHANG Y, LIAN J, et al. Apple harvesting robot under information technology: a review[J]. International Journal of Advanced Robotic Systems, 2020,17(3): 1-16.
[56]ZHUANG J J, HOU C J, TANG Y, et al. Computer vision-based localisation of picking points for automatic litchi harvesting applications towards natural scenarios[J]. Biosystems Engineering, 2019,187: 1-20.
[57]JIAO Y H, LUO R, LI Q W, et al. Detection and localization of overlapped fruits application in an apple harvesting robot[J]. Electronics, 2020,9(6): 1-14.
[58]宁政通,罗陆锋,廖嘉欣,等. 基于深度学习的葡萄果梗识别与最优采摘定位[J]. 农业工程学报, 2021,37(9): 222-229.
[59]ZHUANG J J, LUO S M, HOU C J, et al. Detection of orchard citrus fruits using a monocular machine vision-based method for automatic fruit picking applications[J]. Computers and Electronics in Agriculture, 2018,152: 64-73.
[60]柳长源,赖楠旭,毕晓君. 基于深度图像的球形果实识别定位算法[J]. 农业机械学报, 2022,53(10): 228-235.
[61]LIN G C, TANG Y C, ZOU X J, et al. Fruit detection in natural environment using partial shape matching and probabilistic Hough transform[J]. Precision Agriculture, 2020,21(1): 160-177.
[62]LIU G X, NOUAZE J C, TOUKO P L, et al. YOLO-Tomato: a robust algorithm for tomato detection based on YOLOv3[J]. Sensors, 2020,20(7): 1-20.
[63]高梦圆,马双宝,董玉婕,等. 基于实例分割苹果采摘机器人视觉定位与检测[J]. 江苏农业科学, 2022,50(3): 201-208.
[64]BENAVIDES M, CANTN-GARBN M, SNCHEZ-MOLINA J A, et al. Automatic tomato and peduncle location system based on computer vision for use in robotized harvesting[J]. Applied Sciences, 2020,10(17): 1-21.
[65]刘芳,刘玉坤,林森,等. 基于改进型YOLO的复杂环境下番茄果实快速识别方法[J]. 农业机械学报, 2020,51(6): 229-237.
[66]ZHONG Z, XIONG J T, ZHENG Z H, et al. A method for litchi picking points calculation in natural environment based on main fruit bearing branch detection[J]. Computers and Electronics in Agriculture, 2021,189: 1-11.
[67]YU Y, ZHANG K l, YANG L, et al. Fruit detection for strawberry harvesting robot in non-structural environment based on Mask-RCNN[J]. Computers and Electronics in Agriculture, 2019,163: 1-9.
[68]杨长辉,刘艳平,王毅,等. 自然环境下柑橘采摘机器人识别定位系统研究[J]. 农业机械学报, 2019,50(12): 14-22.
[69]黄彤镔,黄河清,李震,等. 基于YOLOv5改进模型的柑橘果实识别方法[J]. 华中农业大学学报, 2022,41(4): 170-177.
[70]TANG Y F, ZHANG Y W, ZHU Y. A research on the fruit recognition algorithm based on the multi-feature fusion[C]. Harbin: ICMCCE, 2020.
[71]ZHANG C L, ZHANG K F, GE L Z, et al. A method for organs classification and fruit counting on pomegranate trees based on multi-features fusion and support vector machine by 3D point cloud[J]. Scientia Horticulturae, 2021,278: 1-9.
[72]雷欢,焦泽昱,马敬奇,等. 基于多特征融合与SVM的苹果品种快速识别算法[J]. 自动化与信息工程, 2020,41(4): 13-17.
[73]WU J G, ZHANG B H, ZHOU J, et al. Automatic recognition of ripening tomatoes by combining multi-feature fusion with a bi-layer classification strategy for harvesting robots[J]. Sensors, 2019,19(3): 1-22.
[74]汪杰,陈曼龙,李奎,等. 基于HSV与形状特征融合的花椒图像识别[J]. 中国农机化学报, 2021,42(10): 180-185.
[75]WU G, LI B, ZHU Q B, et al. Using color and 3D geometry features to segment fruit point cloud and improve fruit recognition accuracy[J]. Computers and Electronics in Agriculture, 2020,174: 1-8.
[76]杨晓静,张福东,胡长斌. 机器学习综述[J]. 科技经济市场, 2021(10): 40-42.
[77]JIANG T, GRADUS J L, ROSELLINI A J. Supervised machine learning: a brief primer[J]. Behavior Therapy, 2020,51(5): 675-687.
[78]郑太雄,江明哲,冯明驰. 基于视觉的采摘机器人目标识别与定位方法研究综述[J]. 仪器仪表学报, 2021,42(9): 28-51.
[79]雷欢,焦泽昱,马敬奇,等. 基于多特征融合与SVM的苹果品种快速识别算法[J]. 自动化与信息工程, 2020,41(4): 13-17.
[80]ALZUBI J, NAYYAR A, KUMAR A. Machine learning from theory to algorithms: an overview[C]. Bangalore, INDIA: NCCI, 2018.
[81]LUO L F, TANG Y C, LU Q H, et al. A vision methodology for harvesting robot to detect cutting points on peduncles of double overlapping grape clusters in a vineyard[J]. Computers in Industry, 2018,99: 130-139.
[82]余凯,贾磊,陈雨强,等. 深度学习的昨天、今天和明天[J]. 计算机研究与发展, 2013,50(9): 1799-1804.
[83]GUO Y M, LIU Y, OERLEMANS A, et al. Deep learning for visual understanding: a review[J]. Neurocomputing, 2016,187: 27-48.
[84]李旭,李振海,杨海滨,等. 基于Faster R-CNN网络的茶叶嫩芽检测[J]. 农业机械学报, 2022,53(5): 217-214.
[85]ZHENG C, CHEN P F, PANG J, et al. A mango picking vision algorithm on instance segmentation and key point detection from RGB images in an open orchard[J]. Biosystems Engineering, 2021,206: 32-54.
[86]WANG P, NIU T, HE D J. Tomato young fruits detection method under near color background based on improved Faster R-CNN with attention mechanism[J]. Agriculture, 2021,11(11): 1-13.
[87]李章维,胡安顺,王晓飞. 基于视觉的目标检测方法综述[J]. 计算机工程与应用, 2020,56(8): 1-9.
[88]XU Z B, HUANG X P, HUANG Y, et al. A real-time Zanthoxylum target detection method for an intelligent picking robot under a complex background, based on an improved YOLOv5s architecture[J]. Sensors, 2022,22(2): 1-15.
[89]何斌,张亦博,龚健林,等. 基于改进YOLO v5的夜间温室番茄果实快速识别[J]. 农业机械学报, 2022,53(5): 201-208.
[90]杨福增,雷小燕,刘志杰,等. 基于CenterNet的密集场景下多苹果目标快速识别方法[J]. 农业机械学报, 2022,53(2): 265-273.
[91]MONTOYA-CAVERO L, TORRES R D D L, GMEZ-ESPINOSA A, et al. Vision systems for harvesting robots: produce detection and localization[J]. Computers and Electronics in Agriculture, 2022,192: 1-27.
[92]LI T, FENG Q C, QIU Q, et al. Occluded apple fruit detection and localization with a frustum-based point-cloud-processing approach for robotic harvesting[J]. Remote Sensing, 2022,14(3): 1-18.
[93]熊棣文,孔文斌,冯洋. 在树柑桔果实识别与定位技术发展现状及展望[J]. 中国南方果树, 2021,50(2): 185-190.
[94]BENAVIDES M, CANTN-GARBN M, SNCHEZ-MOLINA J A, et al. Automatic tomato and peduncle location system based on computer vision for use in robotized harvesting[J]. Applied Sciences, 2020,10(17): 1-21.
[95]LI J H, TANG Y C, ZOU X J, et al. Detection of fruit-bearing branches and localization of litchi clusters for vision-based harvesting robots[J]. IEEE Access, 2020,8: 117746-117758.
[96]罗陆锋,邹湘军,熊俊涛,等. 自然环境下葡萄采摘机器人采摘点的自动定位[J]. 农业工程学报, 2015,31(2): 14-21.
[97]任亚婧,张宁宁,徐媛媛,等. 基于视觉识别的成熟苹果识别及采摘定位系统[J]. 现代电子技术, 2021,44(11): 73-77.
[98]RONG J C, DAI G L, WANG P B. A peduncle detection method of tomato for autonomous harvesting[J]. Complex & Intelligent Systems, 2021,8(4): 2955-2969.
[99]LI Y T, HE L Y, JIA J M, et al. In-field tea shoot detection and 3D localization using an RGB-D camera[J]. Computers and Electronics in Agriculture, 2021,185: 1-12.
[100]王芳,崔丹丹,李林. 基于深度学习的采摘机器人目标识别定位算法[J]. 电子测量技术, 2021,44(20): 162-167.
[101]李瑞龙. 自动驾驶场景下的三维目标检测技术研究[D]. 长春：中国科学院大学(中国科学院长春光学精密机械与物理研究所), 2022.
[102]齐锐丽. 基于机器视觉花椒目标识别与定位技术研究[D]. 汉中：陕西理工大学, 2020.
[103]周俊,刘锐,张高阳. 基于立体视觉的水果采摘机器人系统设计[J]. 农业机械学报, 2010,41(6): 158-162.
[104]胡小平,左富勇,谢珂. 微装配机器人手眼标定方法研究[J]. 仪器仪表学报, 2012,33(7): 1521-1526.
[105]KLAUS H S, GERD H. Optimal hand-eye calibration[C]. Beijing: IEEE, 2006.
[106]卜令昕. 结构化果园苹果收获机器人关键技术研究[D]. 杨凌：西北农林科技大学, 2021.
[107]LI D H, SUN X X, ELKHOUCHLAA H, et al. Fast detection and location of longan fruits using UAV images[J]. Computers and Electronics in Agriculture, 2021,190: 1-15.

备注/Memo

备注/Memo:
收稿日期：2022-05-20 基金项目：国家自然科学基金青年科学基金项目（32102071）；江苏省农业科技自主创新基金项目[CX(20)3040]；江苏省高等学校自然科学研究项目（21KJB220013）作者简介：金寿祥（2000-），男，安徽安庆人，硕士研究生，主要研究方向为图像识别、目标检测技术。（E-mail）sx@njfu.edu.cn 通讯作者：周宏平，（E-mail）hpzhou@nifu.edu.cn
更新日期/Last Update: 2023-05-12