Challenges for Future Robotic Sorters of Mixed Industrial Waste: A Survey

To achieve recycling of mixed industrial waste toward an advanced sustainable society, waste sorting automation through robots is crucial and urgent. For this purpose, a robot is required to recognize the category, shape, pose, and condition of different waste items and manipulate them according to the category to be sorted. This survey considers three potential difficulties in the sorting automation: 1) End-effector: to robustly grasp and manipulate different waste items with dirt and deformations; 2) Sensor: to recognize the category, shape, and pose of existing objects to be manipulated and the wet and dirty conditions of their surfaces; and 3) Planner: to generate feasible and efficient sequences and trajectories. This survey includes 76 references to studies related to automatic waste sorting and 159 references to worldwide waste recycling attempts. This pioneering investigation reveals the possibility and limitations of conventional systems; thus, providing insights on open issues and potential technologies to achieve a robot-incorporated sorter for the chaotic mixed waste is one of its contributions. This paper further presents a system design policy for readers and discusses future advanced sorters, thereby contributing to the field of robotics and automation. Note to Practitioners—Most automated sorting systems operate for limited target waste items. This study is motivated by the automation of mixed industrial waste treatment facilities using advanced robotic sorters. Emerging advances and increasing functionalities of robot system components will widen system applicability and increase use cases in the chaotic mixed industrial waste domain. This paper surveys the research conducted to date, discusses open issues and potential approaches, and presents user guides that provide practitioners with a system design policy. The user guides created according to the strengths and weaknesses of each system configuration provide future researchers and developers with a useful a priori design policy that has been thus far validated on efficiency, quality, productivity, and reliability. A question-and-answer style guide and a sorting-target-aware previous study reference list allows users to find the desired system configuration, including the investigated components according to their purpose.

Generally, in current treatment facilities for mixed industrial waste, large amounts of unsorted recyclable waste are gathered at a collection site (Fig. 1 (a)) and manually sorted into designated boxes (Fig. 1 (b)) or conveyor lanes based on their categories (Fig. 1 (c) and (d)).It is not realistic to treat all industrial waste in the factory or office where it is generated; therefore, we assume an automated system at an outside waste treatment facility that gathers it together from various companies.In other words, robotic recycling systems are thought of being located in a recycling plant, not in the same industrial plant where the waste is generated.The process for limited categories of waste that have a low degree of mixing is easy to mechanize or robotize.In contrast, for highly mixed waste, sorting is very difficult with a single dedicated machine or robot system.Therefore, a combined system that includes both dedicated machines and robots together with few human workers is required, which is shown in Fig. 2.

A. Background and Motivation
The following procedure can be considered for an assumed sorting system using both dedicated machines and robots that can handle all types of collected waste items while being helped by a small number of human workers.In the first stage, a semi-automatic multi-robot picking system capable of  a multi-modal sensing (the blue part of Fig. 2) must untangle long items (e.g.plastic pipes, vinyl ropes, and strings) and remove explosives (e.g.lithium-ion batteries, small electric fans, and electronic cigarettes) from densely gathered waste items that have been unloaded from trucks onto the ground.In the second stage, with a dedicated vibration machine, the manipulability by robots based on their movable range and payload of the robots is considered; then, the vibrating screen removes large, long, and heavy objects that exceed the allowed range from the waste items brought by an evacuation loader.In the third stage, using multi-modal sensors, global sensing (e.g.recognition at the category level, object image segmentation, and grasp point calculation) and local sensing (e.g.recognition at the material level and estimation of grasping state) are performed on several relatively small and lightweight waste items.Finally, a semi-automatic multi-robot module completes the agile sorting for all remaining moving objects (the red part of Fig. 2).The major challenges in developing the components of robotic sorting for mixed industrial waste are two-fold: 1) the wider variety of waste items that are densely gathered and/or move on a conveyor must be manipulated in an agile manner, 2) short lifecycle objects that are dirty on the surface, deformed, and/or damaged must be robustly recognized.A wider range of previous studies that can relate to this topic were investigated.Therefore, this survey considers three potential difficulties of the sorting automation pertaining to the robotic components: 1) End-effector: to robustly grasp and manipulate different waste items with dirt and deformation; 2) Sensor: to recognize the category, shape, and pose of existing objects to be manipulated and the wet and dirty conditions on their surfaces; and 3) Planner: to generate feasible and efficient sequences, grasps, and trajectories.After investigating the current technologies regarding the components, we discuss the advanced modules including these components, which rely on multi-modal sensing and semi-automatic manipulation with multiple robots and human workers.

B. Search and Collection Strategy
To collect the related articles, we first searched the ACM Digital Library, ASME Digital Collection, IEEE Xplore, MDPI, SAGE Journals, ScienceDirect, Springer Link, and Taylor & Francis Online using the keywords "robot" and "waste" and ("sort" or "recycle").We also searched using Google Scholar1 and discovered listed articles published in other databases.Table I contains the list of the web databases and their URLs, and Fig. 3 shows the number of articles for each web database used in this survey.After examining the abstracts and titles of all articles obtained from each database, we chose the related articles among them.Fig. 4 shows the waste domains described in the articles found, which include Not Defined (ND), Municipal (urban) waste, Waste electrical and electronic equipment (WEEE), Construction and demolition (CND) waste, Nuclear waste, Litter, Underwater waste, Household waste, Consumer and industrial (CNI) waste, Biomedical waste, Space waste, and Floating waste.
Furthermore, material-or product-specific waste sorting was also researched.Likewise, these were divided based on the target domain.They are included in this survey because of related techniques that exist to sort solid waste.For example, some existing technologies can sort aluminum scraps [74], [75], plastics [22], [39], [76], beverage containers [77], and batteries [78], [79], [80].These objects are also often included in mixed industrial waste.
Finally, this survey includes 76 references (hereinafter, referred to as sorter group) to related studies on automatic waste sorting and 159 references (hereinafter, referred to as recycling group) to worldwide waste recycling attempts.The former is a list of articles on sensing and manipulation technologies applied toward waste sorting with dedicated machines or robots in the target domains.The latter, in addition to the former list, includes articles on waste sorting technologies used in non-target domains and surveys or reviews of existing recycling management activities around the world.The following sections include the cited references and also consider their further citations (that is, more recent papers citing the selected ones).
To the best of our knowledge, this survey is the first attempt to organize the related technologies and insights for future robotic sorters related to the domain of mixed industrial waste.This paper considers waste treatment on an industrial scale, which includes household and municipal waste.The three primary contributions of this survey are as follows: 1) Similar to this study, Bogue et al. [25], Gibson [12], and Lubongo et al. [39] enumerated several practical sorting machines to outline what current waste-specific systems can manipulate (e.g.sort and disassemble) the limited target waste.Sarc et al. [16] discussed the role and limitation of robotic-based and wastespecific sorting systems with relevance to business models and data tools.Compared with these studies, this paper concentrates on difficulties originating from chaotic mixed waste sorting scenes.This paper broadly covers conventional hardware (e.g.end-effector, sensor, and integrated system) and software (e.g.planner and controller) configurations and further addresses their potential use in the ever-more chaotic and difficult-tosort situations of mixed industrial waste (e.g.densely gathered and moving objects).2) Other two review articles provide more concrete details of components integrated in the sorting systems developed thus far.Cui et al. [21] discussed the criteria and principles of mechanical separation processes with dedicated machines for WEEE sorting tasks.Gundupalli et al. [14] reviewed different approaches of physical processes, industrial sensors, and dedicated actuators, as well as control and autonomy related issues in automated sorting and recycling of source-separated municipal solid waste items.Unlike these articles, this paper discusses the role of future robotic sorters for mixed industrial waste that can possibly eliminate the persisting issues of previously developed dedicated machines.The user guides created according to the strengths and weaknesses of each system configuration provide future researchers and developers with a useful a priori design policy.A question-and-answer style guide and a sorting-target-aware previous study reference list allows users to find the desired system configuration, including the investigated components according to their purpose.Finally, based on these investigations and clarifications, emergent issues are comprehensively discussed to further improve the robotic technologies for chaotic mixed waste treatment.In the following section, we briefly explore the history of the recent well-studied approaches and technologies based on waste sorting automation.Section III introduces and classifies the end-effectors, sensors, and planners used and developed for waste manipulation operations thus far.Thereafter, Section IV discusses the roles of the current dedicated machines and future robotic sorters.Finally, Section V concludes this survey.

A. A Brief History
Waste sorting using robots was first attempted around the 1990s [84].As shown in Fig. 5, it appears that few studies were initiated from 1990 and were actively studied until around 2011.
In the 1990s and later, Ward et al. [49] constructed several mockups of teleoperated robots for radiation exposure reduction that can manipulate objects in sites that manufacture nuclear materials.Customized end-effectors or tools were equipped according to the application (i.e. the workplace and target objects).They used force and torque sensors to check the values for the safe operations and used cameras only for viewing the remote workspaces.Similarly, Holliday et al. [84] demonstrated an automated robotic workcell equipped with multiple sensors for hazardous waste characterization.Glass et al. [85] tackled the problem of collision-free inverse kinematics of manipulators to perform waste management tasks.Prassler et al. [69] proposed an office waste cleanup mobile robot without robot arms.Caldwell et al. [50] developed a pneumatic muscle actuator driven manipulator rig without end-effectors that could be teleoperated for nuclear waste retrieval operations.Karlsson et al. [86] concentrated on a vision feature fusion approach with multiple vision systems for classification of electrical motors for recycling.No studies targeted CNI waste.
From the 2000s, Cui et al. [21] discussed mechanical separation processes for fine particles of WEEE, which was only beginning at that time.Another featured topic is plastic recycling.Ahmad et al. [76] presented an automatic identification and sorting method of plastic waste items.They sorted the plastic materials based on optical identification of fluorescence signatures of dyes, incorporated in these materials in trace concentrations prior to product manufacturing.The identified objects were arranged in a line on the conveyor and sorted by operating the air jet for ejection to the appropriate bins.One breakthrough came with ZenRobotics.ZenRobotics company, based in Helsinki, Finland, was founded in 2007.In 2009, ZenRobotics Heavy Picker was developed for sorting CND waste.The system is equipped with gripper arms and can sort out contaminants and recyclables from mixed waste streams with the help of deep-learning.
In contrast, several developed countries have proposed concepts and taken initiatives to achieve innovative manufacturing processes, as represented by Industry 4.0 [87] that started in 2011, Industrial Internet Consortium [88] established in 2014, and Made-in-China 2025 [89] that started in 2015.Because of this future innovation in the manufacturing industry, the automation of recycling may have also attracted considerable attention.The combined paradigm Recycling 4.0 Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.was discussed in [90].Combining the emergence of practical technologies, such as ZenRobotics' sorting system with this social background, numerous technologies related to waste sorters have appeared, as described in the following sections.
China and India have overwhelmingly large populations compared to other countries.European countries and other 2 https://github.com/johan/world.geo.jsonAsian countries (excluding China and India) are 3 and 0 in the sorter group and 9 and 4 in the recycling group, respectively, which are comparable to China and India.Other regions include the United States and Canada in North America.

III. CURRENT ROBOT CONFIGURATIONS
To enable sorting by robots, the end-effector, sensor, and planner must be meticulously designed.According to [12], the currently installed robotic sorting systems in waste treatment facilities perform the following procedures.First, after the system distinguishes between materials, a robotic arm is activated to pick the targeted items.The robotic arm can use a gripper shaped like a human hand or a suction cup to pluck items off the conveyor.The robot often has a deltastyle configuration, with three arms connected in parallel at the base.However, the ability of the sorting system used for limited target objects like colored bins, clear bags, and papers in small sizes must be extended.Therefore, for a wide range of articles included in the sorter group, the current studies regarding the end-effector, sensor, and planner are investigated in this section.
1) Mechanism: Fig. 8 shows the end-effector categories used for the waste sorting operations and the number of articles.Most end-effectors are not different from the general robotic grippers and hands.The designs used for the sorter can be classified into six types: Two-finger, Vacuum, Multiple grippers, Broom, Combined, and Multi-finger.Combined shows the gripper with both a two-fingered gripper and a suction gripper.Currently, the two-fingered gripper is the one primarily used for the end-effector attached on the robot arm to manipulate waste items.It must be easy to control such that several moving waste items on a conveyor can be swiftly handled.The two-fingered grippers can be easily controlled compared with multi-fingered and combined robot hands.
2) Material: It is desirable that the material and component used for grippers is robust to unseen objects without known properties like three-dimensional shapes, friction characteristics, and deformation characteristics.Chin et al. [102] used soft materials for the two fingers of a robot hand to grasp several types of garbage to be sorted.These soft robotic technologies [103], [104], [105] are expected to improve the robustness of the gripper for various object shapes.Jamming grippers [106], [107] can possibly be an effective approach, but the durability of the soft membrane is a critical issue.Sasatake et al. [73] proposed using a broom to manipulate garbage with unknown shapes, with the same trajectory as humans.Combined grippers comprising both a suction cup and two fingers [93], [108] are acceptable devices for handling a wider variety of waste items.
3) Manipulation Strategy Specific to Waste Sorting: To achieve an agile robotic sorter for a huge volume of waste, previous studies sorted items on a conveyor using suction grippers for quick grasping and manipulation [4], [109].To only move objects to a desired position, such as brooming or non-prehensile manipulation, grasping or in-hand manipulation is not essential.Controlling the suction grippers is easy compared to multi-fingered and combined robot hands.Graspless [110], [111], prehensile pushing [112], and non-prehensile manipulation [113], [114] methods, such as the push-and-drop technique for waste items [77], have not been applied in real waste treatment facilities thus far.Huang et al. [115] proposed a nonprehensile manipulation method for mobile robots to perform waste cleanup, but it was not applied to a real environment.Therefore, the feasibility of nonprehensile manipulation is still untested, notwithstanding that such operations using a two-fingered or suction gripper is a reasonable method of agile manipulation.In Section IV, the grasping ability of the four types of grippers is compared, including two each of different two-fingered and pneumatic grippers.

B. Sensors and Recognition
This section summarizes and classifies sensors with robust recognition methods used for sorting applications.The following subsections describe sensing technologies in terms of measurement principle, learning-free recognition methods, learning-based recognition methods, and wastesorting-specific recognition tasks.
2) Learning-Free Recognition: DL-free methods use handcrafted features obtained by the sensors.Huang et al. [137] used a 3-dimensional (3D) line camera and laser beam to obtain the position and 3D shapes of target metal objects.In addition to these, they detected the object edges from generated images to estimate the bounding box; then, the geometrical center and the particle sizes of the objects could be approximately determined.Gundupalli et al. [139] proposed to classify and sort the recyclables using the thermal imagingbased technique.The groups, metal, PCB, plastic, and glass, were classified based on extracted features comprising the mean intensity, standard deviation, and image sharpness.Xiao et al. [140] proposed a system that makes use of height maps and near-infrared (NIR) hyperspectral images to locate the region of interest of objects and to do online statistic pixel-based classification in contours.Two types of features in a hyperspectral image were extracted; a scale-sensitive algorithm was used to identify amplitude features, and a scale-insensitive algorithm was used to identify trend features.Rapolti et al. [60] proposed a system that categorizes the components based on the materials' spectral signature using a hyperspectral image.The four components, silicon chips, fiberglass, resin, and a mixture of fiberglass and copper, were categorized based on principal component analysis applied to preprocessed images.Localizing, estimating poses, and reconstructing shapes can be achieved based on the DLfree methods, but the classification task in a mixed waste situation, where objects composed of various materials coexist, is difficult by such three traditional approaches.
3) Learning-Based Recognition: Recent studies concentrate more on utilizing a recognition system using DL with RGBD images.The DL-based methods using RGBD cameras release operators in recycling facilities from programming and teaching and help the sustainable development according to environmental changes.Generally, massive training datasets are required for DL-based vision systems because of the numerous model parameters that must be optimized [143].With recent decreases in product lifecycles, unseen waste items frequently appear at the recycling facilities.Therefore, the training dataset must be promptly updated with new waste Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
Despite the several ideas explored, the predominant datasets were built by humans using bounding boxes or polygonal masks [162], [163], [164].Domain adaptation involves adapting machine learning models across domains.This is motivated by the challenge of the test and training datasets falling from different data distributions because of some factors [165].Domain adaptation is a specific scenario in transfer learning that can be used to effectively remove domain differences.
For the waste sorting, Kiyokawa et al. [77] and Koskinopoulou et al. [166] handled the data augmentation problem with domain adaptation for a self-collected wasteimage dataset such that it can be adapted to a real wastesorting problem.In addition to data augmentation, using existing datasets can possibly increase the sizes of the datasets.Table III lists the existing public datasets.Several large and small specific datasets are available in Kaggle. 3As of April 2022, by searching using the keywords "waste" and "garbage" in Kaggle, 155 and 75 datasets were found, respectively.
4) Recognition Tasks Specific to Waste Sorting: Precisely sensing conveyed objects is a critical challenge.Cowley et al. [167] used an RGBD sensor (Microsoft, Kinect) and tackled tracking on a moving object by maintaining a track state that includes both object position and velocity with Kalman filtering.Liu et al. [168] proposed a method for deblurring the images showing conveyed waste items.If the blur can be removed from images, the aforementioned DL-based recognition technologies can be applied for both the motionless objects and the conveyed waste items.Using an RGBD sensor (Intel, RealSense), Wong et al. [169] challenged DL-based moving object recognition with prediction for robotic grasping and manipulation and succeeded with recognition using YOLACT [170], long short-term memory (LSTM)-based moving position prediction, and convolutional neural network (CNN)-based grasping point prediction.However, recognizing conditions of object surfaces is a persisting issue to be researched in the future.The robot must consider the possibilities of grasping and manipulation based on the identified surface condition while robustly tracking the moving object.
In summary, technologies have been developed for global and local sensing, as shown in Fig. 2. The local sensing technologies, such as material identification, have been installed in conventional dedicated machines, and their applicability to waste items has been satisfactorily verified.However, in robotic sorters, global sensing, such as recognition of object categories, regions, and shapes that can be grasped and operated, is a challenge.In Section IV, a preliminary verification of employing RGBD sensors, which have been extensively used in robot manipulation research, in waste applications was conducted.

C. Planning and Execution
Generally, previous planners relied on selective compliance assembly robot arm (SCARA), Delta, or Multiple degrees of freedom (DoFs) robots.We describe the technology related to planning and execution phases by dividing it into two sections: one is for SCARA or Delta robot, which is a type of industry-specific robot, and the other is for multiple DoFs robot, which are relatively flexible for various environments.
1) SCARA or Delta Robot: A critical issue is that the sorting robot must manipulate carrying objects utilizing a motion planner such as a motion planning tool named MoveIt [171].A well-known method to sort moving objects on the conveyor is first-in first-out (FIFO) using a SCARA [172].Several timeminimum plans of path or trajectory for parallel link robots that handle moving objects have been generated in previous studies [173], [174], [175].Chen et al. [176] proposed to segment the sorting area based on the assumed maximum velocity of a robot to reduce the computational load of the robot's velocity planning.Furthermore, they presented a dynamic prediction method regarding workpiece picking positions considering all possible positions of the robot and the workpiece.Han et al. [177] developed a dynamic programming-based optimal pick-and-place algorithm that outperforms the existing state-of-the-art methods, including FIFO.To break through the traditional pick-and-place operations, Chen et al. [178] used a SCARA robot and planned a robot throwing trajectory for solid waste handling.The generation method of time-optimal pick-and-throw trajectories for a SCARA robot was proposed by [179].Similarly, Raptopoulos et al. [180] replaced the usual pick-and-place process with a much faster pick-and-toss process for delta sorting robots.
2) Multiple DoFs Robot: Besides SCARA and delta (parallel link) robots, various dynamic object manipulation planners for a robot of several DoFs have been discussed.Cowley et al. [167] used a mobile and dual-armed robot (Willow Garage, PR2) and developed a pick-and-place planner for dynamic objects.Based on the sensing of dynamic objects in the previous section, they divided the manipulation planning into grasp recording and selection, trajectory planning with ARA* [181], and pick and place action execution.Similarly, Menon et al. [182] used PR2 and presented a heuristic kinodynamic motion planning that can generate smooth trajectories to pick moving objects.The generated trajectories can be matched with object velocity throughout the grasping motion while being feasible with respect to joint torques and velocity limits.
Gundupalli et al. [183] proposed a pick-and-place sequence planner primarily based on the time spent on moving the end-effector to the object and then to the bin.Ku et al. [98] tackled dynamic grasping planning by locating the end-effector relative to the target object and optimizing robot kinematics parameters based on the sorting efficiency.[184] and [169] constructed DL-based object motion prediction along with feasible and efficient grasp configuration frameworks.According to the predicted object state and planned grasp, a single robot arm moves adaptively.They used the same robot arm (Robotiq, UR5).[184] proposed a learning-based generation of smooth adaptive trajectories using RGBD image features, and [169] used MoveIt to generate trajectories.[185] tried to make a UR5 robot to learn moving object manipulation with a reinforcement learning method.Although combined planners between task, grasp, and trajectory have been extensively discussed thus far [186], [187], [188], the combined planner for sorting moving and densely gathered various waste items is an open issue.
Saravanan et al. [189] considered the minimization of traveling time and total energy and the maximization of manipulability to plan the trajectory for a robotic sorter with payload constraints.Currently, to remove the manual teaching process and hard-coded programs, several studies have used reinforcement learning [190] and active learning [191] methods to enable robots to learn sorting motions for static objects.To carefully manipulate the tangled and densely cluttered mixed waste items, such as in Fig. 1 (c) and (d), bulk picking planners for contaminated deformable objects might be necessary for mixed industrial waste sorting.

IV. TOWARD FUTURE ROBOTIC SORTERS
This section discusses the potential sorting system for the future, possibilities of current technologies in terms of sensing and grasping, and technologies expected to be introduced.cannot be fully automated.We can classify the unloading site into two categories because the varieties of the waste items are largely different depending on the carrying-out source.

A. Potential Design According to Level of Chaotic Situation
For example, almost all waste items are packed in bags or relatively large waste items containing only a few types, as shown in scenes (a) and (b).In contrast, scenes (c) and (d) include not only different bags but also tangled long items, several small items, and deformable sheets.
To achieve a waste sorting system that can be applied for a situation with these large variations, the aforementioned combined system is required with dedicated machines and robots based on the advantages of using robots while being helped by a small number of human workers.As shown in Fig. 2, the procedure mentioned in Section I might be applied to handle all types of collected waste items shown in Fig. 10.

1) Re-Identifying Mature and Off-the-Shelf Technologies:
To further re-examine the currently developed technologies, the depth sensing (Fig. 11) and currently developed endeffectors (Fig. 12) were applied for some samples extracted from the mixed industrial waste items to verify if the sensor can be used to obtain deformed waste items and if the end-effectors can be used to grasp the different-shaped waste items.For the robot grasping and manipulation of mixed waste, classifying the graspable and manipulable category using a color camera is important to accurately obtain the shapes of various objects through a depth sensor and to robustly grasp the different shapes.Therefore, we applied several grippers and RGBD sensors for different waste items to verify their sensing and grasping potential.
Our scope in this section does not include sensors such as industrial-grade hyperspectral sensors and laser sensors, and we will not apply any off-the-shelf robust shape estimation methods to improve the performance of original sensing.
In the sensing experiment, sensors were installed such that the object placement table and imaging direction were perpendicular to each other, and the images were captured from a height of 615 mm.Acquiring relatively low-resolution ranged images with these inexpensive depth sensors for robots does not sacrifice the performance of robot manipulation [195], [196].The results indicate that it is difficult to capture shapes of semi-transparent objects with all sensors.Moreover, using Azure Kinect DK (manufactured by Microsoft Corporation) , 4which is based on the time-of-flight method, almost all object Fig. 11.Case study on sensing through four different RGBD sensors for waste examples of seven different categories.The sensors were installed such that the object placement table and imaging direction were perpendicular to each other, and the images were captured from a height of 615 mm.Fig. 12. Case studies on grasping with four different end-effectors for waste examples of seven different categories.In the grasping experiment, a gripper attached to a robot arm approached directly from above a target object placed on the table in a random pose and deformed state.Thereafter, it was considered successful if the grasp could be applied, the arm could lift straight up, and the grasp could be maintained for five seconds or longer.• indicates if it was successful six or more times out of 10 attempts, and indicates if it was successful eight or more times.The empty squares indicate the failed pairs.
shapes could be captured, whereas other sensors could not detect some of the object shapes.We must carefully choose the sensor based on the measurement principle.
In the grasping experiment, four grippers were used: a parallel-jaw gripper (Robotiq, Hand-E), soft gripper (NITTA, SOFTmatics), balloon hand (CONVUM, SGB), and multisuction gripper.Each gripper attached to a robot arm approached directly from above an object placed on the table in a random pose and deformed state.Thereafter, we assumed that it was successful if the grasp could be applied, the object could be lifted straight up, and the grasp could be maintained for five seconds or longer.When the number of successes is six or more out of 10 attempts, it is indicated by •, and if it is eight or more times out of 10 attempts, it is indicated by .The empty squares indicate the failed pairs.The results indicate that it is difficult to grasp plate-shaped and large and/or heavy items with the gripper, even when it has two or more fingers.In contrast, the suction grippers demonstrated successful grasping of plate-shaped items but failed to grasp strings, hoses, small cylinders, stacked containers, voids, bags, and heavy items in the ten trials.
2) Role of Robots Against Dedicated Machines and Humans: This section clarifies the role of the robotic sorter in the combined system shown in Fig. 2 to differentiate the role, job, and duty with the dedicated machines and human operators.The table in Fig. 14 lists the sorting-targetaware references categorized into the domains (e.g.Municipal, WEEE, CND, Nuclear, Litter, Underwater, Household, CNI, Biomedical, Space, and Floating) with descriptions regarding the features of manipulation, sensing, and planning.To avoid the list becoming too large, the table does not describe articles that deal only with recognition modules for various sensors (e.g. the articles that present learning-based recognizers using UAVs, UUVs, and mobile robots are excluded from this list because of their large numbers) and focuses on articles discussing methods that have relatively high feasibility for actual sorting operations.We read all the selected articles and extracted several references in which either the target material or the target object was clearly indicated or the details of the proposed method were described.Fig. 15 and Fig. 16 illustrate user guides that make it easier for readers to design manipulation and sensing system configurations, respectively.Note that these user guides will not be always general in the future and must be updated without fail based on the progress of research and development technology and changes in social background in the future.To divide the objective of users into different situations related to the required manipulation system configuration, Fig. 15 asks four major questions: Sorting target is limited and already specified?,Densely gathered?, Heavy or large?, and Deformable, thin, long, or small?If the answer is Yes to the first question, the readers can start finding a desired configuration from the reference list shown in Fig. 14.If the answer is No or if we cannot find any related study in the list, we proceed to the second question.Thereafter, the readers can follow the arrows as they answer the questions and will end up with one of six different configurations: Semi-automatic module, Multi-robot (arm) module, Multiple end-effectors with tool changers, Multiple suction grippers, Robot hand, and Suction gripper.
Similarly, in the situations related to the required sensing system configuration, Fig. 16 asks three major questions: Sorting target is limited and already specified?,Transparent?, and Densely gathered, deformable, or contaminated?If the answer is No or if we cannot find any related study in the list, we proceed to the second question.In the same way, following the arrows when answering the questions, the reader will end up with one of the four configurations included in the reference list: Semi-automatic and DL-based multi-modal recognizer, Learning-free multi-modal recognizer, DL-based visual recognizer, and Learning-free visual recognizer.
To completely remove the target waste items that are often left behind even if a dedicated machine and robot are used, one promising method is to construct a qualified human-robot collaboration system [197], [198], [199], [200], [201] through interactions using ambiguous linguistic descriptions [202] to thoroughly sort the waste items.We believe that human-robot collaboration is essential for establishing safer work conditions for recyclable waste sorting, e.g. by detecting dangerous waste items.In this collaboration, the concern regarding human workload can be alleviated with a power assisting robot [203].Considering the big picture, establishing a human-robot collaboration workflow that can maximize efficiency, profit, safety, and work quality in waste sorting is an essential requirement to address the critical waste sorting problem in an equitable and just manner.Given that only a small portion of the recyclable waste is currently recycled, a human-robot collaboration has the potential to enforce the introduction of robot-based sorting technologies.
Regarding the operations executed after sensing and grasping, Fig. 13 shows the operations to be executed by future robotic sorters.Fig. 13 (a) and (b) show humanlike dual-arm operations for large and heavy objects.Fig. 13 (c) and (d) can be used to remove entanglements caused by the string wrappings or stacking multiple objects in the scene.To agilely handle densely cluttered waste items, the robot might push conveyed objects, as in Fig. 13 (e).Dualarmed robotic grasp and toss of an object on the conveyor is a reasonable approach to swiftly manipulate the objects proposed by Bombile et al. [204].As mentioned before in Section II-A, numerous telemanipulated robotic sorters have been developed to avoid manual nuclear and hazardous waste retrieval [49], [50], [84].Vision-based shared control technologies for telemanipulation have been developed in simulation [205].Rahal et al. [206] attempted robotic cutting using the shared-control approach.
In these semi-automatic frameworks (e.g.teleoperated robots, collaborative robots, and assistive robots), multifingered hands are required for the end-effector of future robotic sorters to outperform the adaptability of two-fingered grippers and suction grippers.In general, although automatic control of the multi-fingered hand is difficult, the hand with a similar physical structure to humans makes it easier to remotely operate and can be taught easily.To automatically control the multi-fingered hand for moving object manipulation, the robot hand must track a moving object to prepare for subsequent grasping, and it naturally changes the hand pose to generate an optimal pre-grasp to avoid postgrasp adjustments [207].

B. Challenges to Developing Global Recycling
The advanced technologies used to enhance the availability of the robotic sorter are automated disassembling to change the waste of assembly products into a recyclable state [37], [80], [208], [209], [210], [211], [212], [213], autonomous mobile robots capable of manipulating waste items on the ground [214], [215] equipped with a function to handle Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
As discussed in previous section, the sorting of mixed industrial requires an integrated system with new components such as sensing using multiple modalities, semi-Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.automatic operation, and object manipulation by multiple robots, in addition to the components of sensors, grippers, and planners.We believe that it is better to actively promote the diversion of technologies that have been researched and developed in other domains.Research promotion in this direction will expand the possibility that the results of future research on this mixed industrial waste sorting can be conversely diverted for sorting systems in other domains, and a synergistic effect can be expected.The problem setting of densely gathered mixed waste can be applied to target in other domains.For example, in situations other than CNI, which is the subject of this paper, such as municipal, litter, underwater, household, and floating, it is difficult to assume a priori what type of waste will be present; thus, a similar situation with many densely gathered mixed waste items can be assumed.Additionally, in environments such as WEEE, CND, nuclear, biomedical, and space, where the possible target waste can be narrowed down within a certain range, the robust and prompt picking operation of densely gathered and moving objects can be one of the required specifications, and a similar problem can be set.Densely gathered and mixed situations are still considered to be a difficult problem and are regarded as unfeasible in the previous studies; thus, they are still not sufficiently addressed.Therefore, there is a possibility that the technology can be transferred to other domains if similar problem settings are anticipated.
In this context, in addition to the aforementioned modules that consist of semi-automation with multiple robots, the components for the modules: multi-modal sensors, robustly applicable grippers, and integrated planners, are necessary.Improving the fusion systems of multiple sensor data [221], [222] and/or multiple features of sensors [86] is a promising approach to replace the customized sensing systems individually prepared according to different sorting workplaces.Modular [223] and/or multifunctional [224], [225] endeffectors can possibly be used for handling a large variety of waste.The usage of an automated tool changing system (e.g.SMARTSHIFT 5 ) and airbag-equipped gripper to ensure the safety of humans at the workspace of a robot [226] are anticipated for future sorting systems.An integrated planner for more dexterous navigation, grasping, and manipulation must be developed and researched in future studies.Currently, research for the planner is often conducted on individuals and segmented planning problems; however, in practice, each plan interacts with other planning strategies and outcomes, which necessitates the integration of planners for a more general-purpose sorting system.

V. CONCLUSION
To achieve recycling of mixed industrial waste items toward an advanced sustainable society, waste sorting automation by robots is crucial and urgent.For this purpose, a robot is required to recognize the categories, poses, and conditions of different waste items and manipulate them based on the category to be sorted.This survey was organized around the following three potential difficulties in sorting automation components: 1) End-effector: to robustly grasp and manipulate different waste items with dirt and deformation; 2) Sensor: to recognize the category, shape, and pose of existing objects to be manipulated and the wet and dirty conditions on their surfaces; and 3) Planners: to generate feasible and efficient sequences and trajectories.This survey included 76 references to related studies on automatic waste sorting and 159 references to worldwide waste recycling attempts.
In summary, the possibility and limitations of conventional system configurations were summarized; thus, providing insights on open issues and potential technologies to achieve a robot-incorporated system to sort chaotic mixed waste items is one of this paper's contributions.Based on the investigations and organizations, we created user guides to show a system configuration design policy for readers and discussed emergent issues to be solved toward identifying advanced future robotic sorters; this is another contribution of this paper.
Robotics and automation for handling mixed industrial waste is an emerging research field, but one that is expected to grow rapidly in the coming years as more researchers seek to create robots that can actively help toward a sustainable society in the future.

ACKNOWLEDGMENT
The waste samples used in this paper were obtained with the cooperation of DAIEI KANKYO Company Ltd., Japan.The authors would like to thank Makoto Yamada and Sana Ohashi of DAIEI KANKYO Company Ltd., for their cooperation in the investigations of actual manual sorting environments.

Fig. 1 .
Fig. 1.Images showing current processes of sorting mixed industrial waste.(a) Collected waste items are unloaded from a truck, (b) manual sorting of the mixed industrial waste from the ground, (c) collected waste items on a conveyor, and (d) manual sorting at the conveyor.

Fig. 2 .
Fig.2.Overview of an assumed system combining dedicated machines, robots, and humans working for sorting mixed industrial waste.The thick black arrows represent the flow of waste items in the series of processes.

Fig. 5 .
Fig. 5. Publication years of the articles in the two groups.The vertical axis shows the number of articles.

Fig. 6 .
Fig. 6.Number of articles on issues specific to each country.The country names of recycling groups are arranged in descending order from the left in the same manner.Articles that mention the target country explicitly in the title or text were counted.

Fig. 7 .
Fig. 7. Mapping counted countries of Fig. 6 (recycling group).White-colored areas indicate the countries where country-specific articles related to waste recycling could not be obtained.

Fig. 8 .
Fig. 8. End-effector types used for sorting operations.Combined shows the gripper with both two-fingered and suction grippers.

Fig. 10 .
Fig. 10.Different scenes of mixed industrial waste items unloaded from a track, as shown in Fig. 1 (a).The varieties of waste items are largely different depending on the carrying-out source.For example, almost all waste items are packed in bags, as shown in scene (a).(b) shows a scene with many objects of similar shapes and of the same category.Scenes (c) and (d) include long and several other small items not contained in a bag.

Fig. 10
Fig.10shows example scenes of mixed industrial waste items unloaded from a track, as shown in Fig.1 (a).As shown in the figure, because the variety of the collected waste items in terms of the shape and material is quite large, most cases

Fig. 13 .
Fig. 13.Required operations for the future robotic sorter.(a) and (b) show human-like dual-arm operations for large and heavy objects.(c) and (d) can be used to remove entanglements caused by string wrappings or multiple objects stacked in the scene.To agilely handle densely cluttered waste items, the robot can be made to push conveyed objects like (e).

Fig. 15 .
Fig.15.User guide for designing an unknown target object-oriented feasible manipulation system configuration.

Fig. 16 .
Fig.16.User guide for designing an unknown target object-oriented feasible sensing system configuration.

TABLE I LIST
OF WEB DATABASES WITH URLS USED FOR SEARCHING ARTICLES IN THIS SURVEY.THE ITEMS ARE ARRANGED INALPHABETICAL ORDER

TABLE II SURVEY
AND REVIEW ARTICLES PUBLISHED THUS FAR.THIS LIST IS ARRANGED IN ORDER OF THE PUBLICATION YEAR Fig. 3. Numbers of articles from each web database used for this survey.The database names of the recycling groups are arranged in descending order from the left, and the databases with the same number are in alphabetical order.

TABLE III EXISTING
PUBLIC DATASETS THAT CAN BE USED FOR WASTE SORTING VISION SYSTEMS