Sensor-Based Activity Recognition and Performance Assessment in Climbing: A Review

In the past decades, a number of technological developments made it possible to continuously collect various types of sport activity data in an unobtrusive way. Machine learning and analytical methods have been applied to flows of sensor data to predict the conducted sport activity as well as to calculate key performance indicators. In that scenario, researchers started to be interested in leveraging pervasive information technologies for sport climbing, thus allowing, in day-to-day climbing practice, the realization of systems for automatic assessment of a climber’s performance, detection of injury risk factors, and virtual coaching. This article surveys recent research works on the recognition of climbing activities and the evaluation of climbing performance indicators, where data have been acquired with accelerometers, cameras, force sensors, and other types of sensors. We describe the main types of sensors and equipment adopted for data acquisition, the techniques used to extract relevant features from sensor data, and the methods that have been proposed to identify the activities performed by a climber and to calculate key performance indicators. We also present a classification taxonomy of climbing activities and of climbing performance indicators, with the aim to unify the existing work and facilitate the comparison of methods. Moreover, open problems that call for new approaches and solutions are here discussed. We conclude that there is considerable scope for further work, particularly in the application of recognition techniques to problems involving various climbing activities. We hope that this survey will assist in the translation of research effort into intelligent environments that climbers will benefit from.


I. INTRODUCTION
Rock climbing is considered to be born in the 19th century with alpine mountaineering in northern England and the Italian Dolomites. It developed as a sport for the general public in the 1950s. Today, various types of climbing are practiced all around the world [1]. The last few decades have seen a growing popularity of climbing both as a recreational activity and as a competitive sport. For example, in 2018, the Association of British Climbing Walls reported that climbing was one of the fastest growing sports in the UK, with The associate editor coordinating the review of this manuscript and approving it for publication was Eyuphan Bulut . 15-20% annual growth among the population [2]. Similar reports in the United States show that the number of climbing enthusiasts and climbing gyms continue to grow [3]. These trends have contributed to the debut of sport climbing at the Tokyo 2020 Olympic Games (held in the Summer of 2021).
The rising importance of sport of climbing is also reflected in a growing interest in it by the scientific community. The review by Woollings et al. [4] presented the existing literature on risk factors and prevention measures in climbing. A review of studies that have investigated the characteristics of skilled climbing performance can be found in the work of Orth et al. [5]. Stien et al. [6] conducted a mini systematic review of climbing-specific tests and procedures for predicting performance and measuring training effects. Finally, the physiological aspects [7], [8] and the psychological determinants [9] of climbing performance have been studied.
Monitoring sport activities by using ubiquitous computing technologies has become popular in the recent past. Commercial devices are effective for tracking the general levels of physical activity for sports such as running, cycling, and swimming (e.g., [10], [11], [12], and [13]). In contrast, the sport of climbing has not received much support in terms of portable devices and sensor-based electronics. A handful of studies have so far developed initial prototypes for automatic analysis of climbing activities. Notable examples are the use of a body-mounted sensing system for climbing performance monitoring [14], [15], [16], and the exploration of wrist-worn devices for automatic detection of the climbed route [17]. Thanks to decreasing cost of devices with various sensors and novel information and communication technologies, a wealth of applications for day-to-day use in climbing practice can be built. Examples include applications for improving climbers' technical skills by suggesting new and interesting routes to climb, applications for providing climbers with effective and personalized training programs, virtual climbing coaches that give frequent and detailed feedback to climbers as it is received by professional athletes, automatic detection of risk injury factors, as well as usage analytics for climbing gym managers.
Automatic, rapid, and reliable recognition of human activities is an important building block of activity monitoring applications [18]. Historically, research on sensor-based activity recognition has been driven by the intensive research effort toward creating intelligent pervasive environments for ambient assisted living [19]. Methods of activity recognition have since been ported to a number of diverse application domains, including security, healthcare, and different types of intelligent environments. This survey is meant as a starting point for new projects, in order to get an overview of existing algorithms and systems, before designing new systems that will be an integral part of intelligent climbing environments.
This article presents an overview of the studies produced in recent years that made use of different types of sensors and techniques to automatically recognize a climbing activity or assess a climbing performance indicator. More specifically, we investigate research that analyzed data flows acquired from sensors that are either worn by climbers, embedded into the climbing equipment, or placed on fixed locations in the climbing environment. We present the main sensor types adopted for data acquisition, such as optoelectronic systems, inertial measurement units, and force sensors. We describe the various techniques that have been considered to pre-process sensor data, extract features from sensor data relevant to climbing activity recognition, and analytical and machine learning methods used to detect and recognize the activities performed by climbers and assess their performance. The climbing activities and performance indicators we consider are those derived from the analysis of the studies in relevant papers, and we organize them into two taxonomies. Considered climbing activities include, for example, traction, postural regulation, and hold gripping. The performance indicators found in the literature we examined include, for example, fluency, endurance, control, and stability.
While reviews similar to ours have been written for sports in general, treating, among others, basketball, soccer, tennis, and swimming (see, for example, [20], [21], and [22]), this is to our knowledge one of the first specific reviews focused on climbing. We note that we do not include in this review studies that have focused on climbing motions by parsing a climber's body area or skeletal information, as the state-of-the-art of this niche research area has already been covered in a recent review by Richter et al. [23]. Moreover, we do not include here the literature that has described tests and procedures for measuring climbing-related physiological and motor characteristics of a climber, such as those included in the review by Stien et al. [6]. We hope that this review offers a useful summary of existing work concerning climbing activity recognition and climbing performance assessment and will motivate more focused and complementary applied research that will prove useful in practice.
The main contributions of this work are: (i) a taxonomy of climbing activities, (ii) a taxonomy of climbing performance indicators, (iii) an overview of the state-of-the-art research in the context of activity recognition and performance assessment in climbing, (iv) an overview of sensors and work flow of sensor data analysis, and (v) a list of challenges and opportunities in the domain.
The rest of this paper is organized as follows. In Section II we introduce the main concepts related to climbing. Section III presents the methodology for identifying all relevant papers. In Sections IV and V we present, respectively, a proposed classification taxonomy of climbing activities and performance indicators found in the reviewed literature. Section VI presents the main types of sensors that have been used for the collection of climbing data. Section VII describes a work flow for climbing activity recognition and performance assessment. In Section VIII we present some of the research challenges and future opportunities to improve the development of systems for automatic climbing activity recognition and performance assessment. Finally, we conclude this paper in Section IX.

II. CLIMBING IN A NUTSHELL
Climbing is a collective term for many sub-disciplines each having its own distinctions in terms of the type of climbing surface, use of protection, and tactics used to ascend [24]. Climbing venues can be found both outdoors (e.g., on natural cliffs or mountain rock walls) and indoors in climbing gyms. A climbing gym usually provides a large number of different climbing routes on artificial walls, often constructed from plywood and synthetic holds. These holds can differ considerably in size and shape (see, for example, the description of basic holds in [25]), and can be assembled in various positions and orientations.
Popular types of climbing are bouldering, which is practiced on low rock formations and with just a crash pad to protect the climber in case of a fall, and sport climbing, where the climber ascends along a predefined route using a rope and bolts that are pre-placed in the rock, and a second person (''belayer'') holds the rope to prevent the climber from hitting the ground in case of a fall. Based on how the rope is used, sport climbing is differentiated into lead climbing, where the rope is carried up by the climber who has to clip it to the wall, and top rope climbing, where the climber is tied on a rope that is anchored at the top of the route. A particular competitive discipline is speed climbing, where the goal is to climb as quickly as possible a standardized route in a top rope style without falling. Ice climbing is a discipline that involves an ascend on ice or hard snow formations with the assistance of crampons and ice tools. To name but a few other types of outdoor climbing, we have: traditional (shorten trad), a discipline that follows a strict principle that all protection must be placed in the rock by hand and be removable without damaging the rock; aid climbing, where the climber is permitted to use gear to aid their ascend; and deep water solo, where the water (e.g., sea, or lake) at the base of a climb serves to protect against injury from falls.
Climbing typically requires the use of a range of equipment to protect a climber against the consequences of a fall. The use of a climbing rope is essential in top rope and lead climbing disciplines. One end of the rope is tied to a climber's harness, normally worn around their pelvis and hips, while the belayer passes the other end of the rope through a belay device. A belay device uses friction to control how much rope passes and when to stop it. In lead climbing, a rope must be clipped to the wall by quickdraws installed every two or three meters during the climber's ascent. Examples of a climber's harness and a quickdraw, instrumented with sensing devices, are depicted in Table 4 (middle column). Other types of equipment that facilitate climbing motion are powder chalk to remove perspiration, worn in a chalk bag attached to the back of the harness, specialized climbing shoes, and belay gloves, among others.
To declare the difficulty of a climbing route, several grading systems are used around the world. Commonly used scales include the Yosemite Decimal System and the French Numerical System, among others. The Yosemite Decimal System is primarily used in the United States and Canada to rate walks, hikes, and climbs in five classes of increasing technical difficulty, the fifth being subdivided with a decimal notation. It also has indicators of the length of the route and of the quality of the protection available on it. The French Numerical System is dedicated to climbing only and rates a climb according to the overall technical difficulty and strenuousness of the route. Grades start at level 1 (easiest climb) and there is no maximum level, as the scale is open-ended. For a broader overview of the various systems, the reader is directed to the report on comparative grading scales, climber descriptors, and ability grouping [26].

III. RESEARCH FOCUS AND METHOD
The focus of this review is on activity recognition and performance assessment during climbing, by using data obtained from sensors that are either worn by climbers, integrated into climbing equipment or placed on fixed locations in the climbing environment. To identify the studies presented in this article, we used a systematic review technique in accordance with the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guidelines [27].
A. SEARCH QUERY Papers were searched in six different databases: Web of Science, Scopus, PubMed, IEEE Xplore, Science Direct, and ACM Digital Library. The search was conducted on July 31, 2022, and included a filter to select studies which were published in English after 2005. We limited the search to journals and conference papers. The search was performed using relevant search terms formed by two groups of keywords related to (i) activity recognition and (ii) performance assessment in climbing, and also excluding unrelated keywords. The search strategy included the query on study title: ''climb'' or ''climber'' or ''climbing'' and not ''stair'' and not ''robot'' and not ''fish'' and not ''spider'' and not ''gait'' and not ''ladder'', and the query on full study text: ''activity'' OR ''action'' OR ''move'' OR ''state'' OR ''session'' OR ''performance'' OR ''skill''. We also used wildcard (*) where supported to broaden the search for words starting or ending with the keyword. The references of the selected studies and found review papers were further checked in order to include relevant works inadvertently omitted from the keyword-based search. The search strings used are given in the appendix (Table 9).

B. INCLUSION AND EXCLUSION CRITERIA
To be included in our review, the study needed to use at least one sensor, either mounted at a fixed location in the (indoor or outdoor) climbing environment, embedded in the climbing equipment, or worn on the body. Additionally, collected data had to be used to calculate at least one indicator or climbing performance or recognize at least one climbing activity. Studies that focused on measurement of physiological responses during climbing or biomechanical description of a climbing-related activity were excluded. Review articles were also excluded. Finally, all papers needed to be published after 2005 in order to avoid inclusion of out-of-date technologies and research.  were excluded. Additional three studies were found through the reference lists. Finally, 43 studies were considered eligible to be included in this review. A summary of the studies in terms of sensing approach and scope is given in Table 1. The PRISMA flow chart is provided in the appendix (Fig. 3).

IV. CLIMBING ACTIVITIES TAXONOMY
In essence, climbing is a sport in which people use their arms and legs to reach and grip holds while moving the body upward on either a natural or artificial rock formation. In addition to dynamic phases, in which a climber proceeds along the route, climbing also involves static moments, when a climber rests or plans the following sequence of movements.
Although no unique definition for the notion of activity exists in the literature, common to the use of this notion is its characterization in terms of an agent (e.g., a person) that performs it, and spatio-temporal properties (e.g., the location where the activity takes place, the time the activity starts, the time the activity finishes, and the duration of the activity [68]). It is also common to represent activities in a hierarchy at different levels of granularity [69] and to understand them as aggregations of actions, which may, in turn, be aggregations of atomic operations [70]. A classical example is a taxonomy of the so-called Instrumental Activities of Daily Living (IADL) [71], which have been proposed to describe various living scenarios. Such activity modeling is devoted to formally describing the activities of interest and providing a basis for their recognition.
To the best of our knowledge, a unifying way to organize the activities characterizing the climbing domain has not yet been proposed. We believe that the lack of such a view resulted in the presence of ambiguous terms in the literature on climbing activity recognition we examined, leading to misunderstandings of some concepts. As an example, in the paper by Ladha et al., [15], the terms ''episode'' and ''activity'' are used to describe essentially the same notion, i.e., a full ascent of a climber. The same authors use the term ''limb movement'' to refer to movements of a hand while reaching or adjusting on a hold. By contrast, Ebert et al. [55] use the term ''rest period'' to denote the same hand adjustment movement. Boulanger et al. [16] use the terms ''activity'' and ''state'' interchangeably, while Tonoli et al. [43] use the term ''event'' to denote the same concept.
Hence, we examined the 43 selected papers with the goal of defining a taxonomy of climbing activities that provides terminology and a set of concepts that, at various levels of granularity, denote the activities that are relevant to be recognized. The view we propose is shown in Fig. 1, where solid rectangles denote activities of climbing whose automatic recognition has been addressed in the literature we examined, while dashed rectangles represent activities whose recognition, to our knowledge, has not been addressed yet.
Our view is inspired by the notable work on activity recognition in indoor climbing by Boulanger and colleagues [16], extending the previous work presented in [58]. In these studies, the authors proposed a method to automatically recognize five main climbing activities, i.e., postular regulation, traction, immobility, hold change and hold gripping. In several follow-up studies, e.g., [35] and [56], the authors used the proposed activity recognition approach in order to assess climbing performance (see Section V). We have extended their scheme with activities found in the other papers we have analyzed.
As it can be seen in Table 2, the various authors generally focused on a small subset of climbing activities. Climbing encompasses more than the activities related to the pure ascent and possible falls. Other related activities are belaying, visual inspection of the route before the ascent (route previewing), lowering of a climber back to the ground, and pulling the rope down after the ascent is finished. Ascending is a comprehensive activity that is composed of activities occurring between the starting point and the highest point of an ascent. In traction, a climber's pelvis (i.e., the center of mass (COM)) moves usually upward using at least one limb while the remaining limbs are gripped to the wall holds. Ascending not only involves continuous traction, but several studies have also emphasized the importance of more or less static activities, such as postural regulation and immobility. Postural regulation occurs when a climber's limbs are immobile, while some movement of the pelvis is allowed to gain stability on the climbing wall. Immobility is characterized by the absence of movement of all limbs and pelvis. A portion of climbing time devoted to immobility is spent by a climber resting at the so-called aid-points [72]. In addition to temporarily recovering from fatigue, at these points, a climber has the opportunity to look around to find the path to follow along the route. Stationarity, also referred to as plateau in [29], differs from immobility in allowing movements of limbs. While a climber is stationary, he can chalk hands (i.e., apply magnesium carbonate to the hands to remove perspiration and thus reduce slipping), shake the limbs to give them relief, or clip the rope into a quickdraw.
A climber has different types of interactions with the climbing surface using his upper and lower limbs. Seifert et al. [30] described three interaction activities related to ice climbing, namely, climbers typically swing their axes and kick their crampons when the ice is dense without any holes, and hook their axes when the ice is hollow. In rock climbing, two principal types of hold interaction activities can be distinguished: hold change and hold gripping. The former FIGURE 1. Taxonomy of climbing activities. Solid rectangles denote activities of climbing whose automatic recognition is addressed in the literature we examined. Dashed rectangles represent activities whose recognition, to our knowledge, has not yet been addressed. corresponds to a limb transition between holds. This activity may also involve limb adjustments to reach a comfortable position or prepare for the next traction. On the other hand, the gripping activity corresponds to a limb holding firmly to a hold. For clarity, in [16], the authors distinguished between the gripping of a hold, which happens when the global position of a climber on the wall is immobile (i.e., hold gripping), and hold use which refers to gripping a hold when a climber's COM is in motion.
As already mentioned, visual inspection of the route to determine the sequence of required movements and functional properties of the holds may occur before the climb (route previewing) or during the climb (route finding). Visual inspection of the route may enhance climbing performance as a climber can mentally plan and rehearse movements of hands and feet as well as identify places for rest and chalking hands [72]. An interesting example of a study where route previewing activity data is analyzed is that of Seifert et al. [56], where they presented a method to determine the visual strategy of a climber. They distinguished between four strategies as defined in [54], i.e., fragmentary strategy, ascending strategy, zigzagging strategy and sequence-of-blocks.
An ascent is considered successful if a climber reaches the top of a route without falling. Indeed, a fall in climbing is very common when a climber tries to climb a route at the limit of their ability [42].

V. CLIMBING PERFORMANCE INDICATORS TAXONOMY
The skill of climbers is often expressed through the highest degree of difficulty they have achieved by climbing routes. Successfully ascending a route, of a given difficulty grade, on the first attempt and without prior information or rehearsal is known as the climber's on-sight ability level. Conversely, ascending a route of a certain difficulty level, after having practiced or studied the climbing route, is known as a climber's red-point ability level.
Difficulty grades are often used to categorise climbers into ability groups [73] such as ''intermediate'', ''advanced'' and ''expert''. However, route difficulty scales (see Section II) do not provide a complete mean of assessing how climbing skills are related to route difficulty [26]. Continuous measurement of performance parameters at different stages of a climb is more important and informative than a single result. Thus, alternative methods for the quantification of climbing performance have been proposed to assess various aspects of climbing performance.
In the remaining of this section, we describe what we have identified as the most relevant indicators that quantify an aspect of a climber's performance, we group them into six categories as follows: indicators of fluency; indicators of exploration; indicators of core abilities: power, stability, control, and endurance; indicators of hold-limb contact performance; indicators of route previewing and route finding performance; and indicators of variability of body movement coordination patterns. Table 3 shows identified indicators classified according to this taxonomy. We enter into further details regarding the computation of these indicators in Section VII-F. The following subsections describe indicators associated with each category in detail.

A. INDICATORS OF FLUENCY
Climbing fluency has been widely cited even if it has not been consistently defined [74]. For example, Sibella et al. [32] defined fluency as the efficiency of the path which a climber took through the route. Similarly, Zampagni et al. [39] took the COM-to-wall distance and the planar displacement of the COM as indicators of fluency. Such notions of fluency are based on spatial aspects of the performance, but do not take into account the temporal aspects of performance, e.g., the pauses a climber may take for route finding or postural regulation. Some other studies focused on temporal indicator of fluency. For example, in their study on the influence of anxiety on climbing performance, Nieuwenhuys et al. [53] measured the total ascent duration and the total duration of periods of immobility (immobile time) and traction (mobile time). The relationship between periods of mobility to immobility, i.e., the immobility-to-mobility ratio, was assessed in the study by Orth et al. [61], while Rochat et al. [62] measured the immobility ratio as a temporal indicator that refers to the percentage of ascent duration spent in immobility. In an iceclimbing study [30], average plateau duration was measured as the total duration of periods of stationarity. Namely, the plateau was defined as less than 0.15 m of vertical displacement at hips for longer than 30 s. Examples of studies that integrate both spatial and temporal aspects into a single indicator of climbing fluency are those by Seifert et al. [29], [30] and Pansiot et al. [14].

B. INDICATORS OF EXPLORATION
The indicators of exploration have been proposed to measure a climber's ability to exploit the properties of the climbing environment to move upward. For example, in a rockclimbing study [16] and in a ice-climbing study [30], the exploration index has been defined as the ratio between the number of exploratory and performatory activities of a climber. In the first one, the authors calculated for all limbs the number of exploratory activities, i.e., hold changes without the hold being used to move upward, and the number of performatory activities, whereby the hold was used for traction. In the second one, the exploratory activities included tool swinging and crampon kicking, while the performatory activities included hole hooking with an axe or crampon, when the ice is soft, or tool swinging and crampon kicking that led to definitive anchorage when the ice is dense. On the other hand, Nieuwenhuys et al. [53] measured the duration and number of fixations (a set of gaze coordinates restricted to a particular area) during the route finding activity. They distinguished two types of fixations: exploratory fixations -made during immobility, and performatory fixations -made during traction. Core abilities that every climber needs to possess and develop, according to Ladha et al. [15], are power -the ability to transfer isometric strength into a movement, stabilitythe ability to remain composed while holding onto holds, speed -timing observation that indicates a climber's ability to find the path and resist fatigue, and control -the ability to transition smoothly between holds. The authors derived corresponding indicators from the data collected by two wrist-worn sensors. They found that combined evaluation of the core abilities correlated to scores a group of climbers achieved in a competition. The authors also acknowledged that their sensor system was unsuited to measure endurance, i.e., the ability to maintain prolonged effort. On the other hand, Pansiot et al. [14] derived a measure of endurance from data obtained using an ear-worn sensor. They used high-frequency head shaking motion as an indicator of the tiredness of a climber. The authors suggested that a low value of the indicator suggests high resistance to fatigue, however, the limitation of their approach was that a low value could also indicate that a climber spent considerable time in stationarity.

D. INDICATORS OF HOLD-LIMB CONTACT PERFORMANCE
Mechanical aspects of climbing performance on a hold have been measured in sport climbing [45], [50], bouldering [47], and speed climbing [44], [51]. Specifically, in [46] and [48], the authors analyzed performance parameters on a climbing hold with a curved surface. In these studies, various measures have been extracted from contact force-time curves, i.e., magnitude of contact force, contact time, the smoothness factor, impulse, the friction coefficient, and the Hausdorff dimension. Among these, the Hausdorff dimension, representing the entropy of the geometry of contact force-time signal, was identified as the most important performance measure by Fuss et al. [45], as it was found to correlate highly with all the other measures. Pressure systems have been applied in [39], [49], and [66] to provide insights into foot loads. For example, the authors in [49] and [66] proposed an insole with pressure-sensitive foil that was able to capture if a climber was applying enough pressure corresponding to his weight. On the other hand, Zampagni et al. [39] estimated the amount of weight redistribution between the two legs during the double support phase.

E. INDICATORS OF ROUTE PREVIEWING AND ROUTE FINDING PERFORMANCE
It is well established that athletes do not possess a better visual system than non-athletes, and the difference lies in how they use their eyes [75]. In climbing, visual inspection of the route before the climb may enhance climbing performance, as the climber has the opportunity to identify crux sections and points for resting and chalking hands. Route previewing is also an essential part of the 'on-sight' climbing competition format, where a climber is permitted a timed inspection of the route before attempting to ascend without prior physical practice. While ascending a route, a climber makes pauses to inspect the route to determine the following sequence of movements. Using eye movement tracking technology, the authors in [36], [53], and [56] calculated various indicators that describe how a climber inspects a route during route previewing and route finding, respectively. For example, in [36], the authors calculated the complexity of a climber's visual search strategy during route preview using entropy measure. In [53], fixation locations were coded as corresponding either to ''handhold'', ''wall'', ''hand'' or ''other''. In the same study, the authors characterized a climber's technique for inspecting the route during route finding using the following indicators: number of fixations, total and average fixation duration by location, and search rate, i.e., the total number of fixations divided into the sum of fixation durations across all fixation locations.

F. INDICATORS OF VARIABILITY OF BODY MOVEMENT COORDINATION PATTERNS
Several studies proposed methods for assessing variability of a climber's body movement coordination patterns during an ascent. For example, Seifert et al. [67] identified clusters of full-body movement patterns over time by taking into account the combination of four limbs and pelvis orientations. On the other hand, the authors in [63] performed a time-motion analysis taking into consideration neck and hip rotations around the vertical axis. They calculated relative duration (in percentages of ascent duration) of each neck-hip coordination pattern, i.e., in-phase, no-phase, anti-phase. In ice-climbing studies [29], [30], [33], the variability of movement was assessed by calculating the range and variability of angles between the horizontal plane and the line formed by the position of the tip of the left and right ice tools (upper-limb coordination pattern) and the line formed by the lowest position of the left and right crampons (lower-limb coordination pattern).

VI. TYPE OF SENSORS
In this section, we present the three sensing modalities that have been used for climbing activity recognition or performance assessment: external, embedded, and body-worn. External sensors are installed in a fixed, strategic location in the environment. Embedded sensors are integrated into climbing equipment with which the climber interacts. For example, they can be incorporated into a harness or a quickdraw. Body-worn sensors are portable sensing devices that are attached directly to the climber's body, possibly organized in a body sensor network. An advantage of body-worn sensors is that monitoring can be done at any place.
In our classification, external sensors are those that pose significant requirements on existing infrastructures such as calibrated camera setups. In contrast to these, body-worn sensors are suitable for long-term data collection during standard climbing sessions, in indoor and outdoor conditions. External and embedded sensors can be complementary to body-worn sensors as they can provide additional information regarding the ascent in an unobtrusive way. In Table 4 there are some examples of these three types of sensors, which are described in the following subsections. Table 5 gives details (sensors and instrumentation, sensor placement) about different systems that have been used for collecting climbing data for identifying activities and assessing performance indicators. For sensor data to be useful, effective algorithms are required which can extract meaningful information on performed climbing activities and performance (see Section VII).

A. EXTERNAL SENSORS
External sensors typically do not require to be in direct contact with the climbers, although they may be required to wear markers that reflect light back to the sensors. Examples of studies where external sensors have been used are the ones adopting video cameras and camera-based motion capturing systems. 3D optoelectronic motion capture (OMC) systems are often considered as the gold standard for motion analysis due to the high accuracy of marker position measurement [76]. However, optoelectronic-based approaches have several limitations for widespread use such as high cost, the time and skill needed for the subject's sensorization, and the limited calibration volume within which the analyses can be performed [22]. An example of a study which used OMC system is that of Sibella et al. [32], in which the climber's body was modeled as a series of nine rigid body segments, i.e., forearms, upper arms, trunk/head, thighs and legs, using 12 reflective markers. They assessed climbing fluency from a climber's 3D body center of mass coordinates calculated frame by frame from the positions of the markers. On the other hand, Seifert et al. [63] used an HD camera to calculate climbing fluency based on digitized hip trajectory. In several studies the authors reported using digitizing software for key points detection and tracking [29], [30], [33].

B. EMBEDDED SENSORS
Embedded sensors include force transducers, altimeters, and accelerometers, among others. Measurement of forces applied to a climbing hold was done by Fuss and colleagues in studies involving bouldering [47], sport climbing [45], and speed climbing [44]. If the instrumentation of climbing holds is not possible -for example, in outdoor climbing -insole pressure system can provide insight into vertical foot loads [39], [66]. Another embedded sensor is the sensor-enhanced quickdraw used in [28] and [41] for activity recognition in a climbing gym. It is a quickdraw with a small accelerometer sensor attached to the strip in its central part. In a study conducted on natural outdoor routes, Tonoli et al. [42] showed that the activity of falling can be accurately detected using an accelerometer and an altimeter embedded into the climber's harness.

C. BODY-WORN SENSORS
Instances of body-worn sensors are accelerometers, inertial measurement units (IMUs), which include a gyroscope (also known as angular velocity sensor) and sometimes a magnetometer, and eye movement tracking glasses. These sensors are often assembled with a data processing unit into custom-made prototype sensing platforms, which are designed as small and robust. Recent advances in eye movement registration technologies have allowed to record eye tracking data of a climber without impeding his perceptual or physical behaviour. This equipment usually consists of a pair of glasses augmented with miniaturised cameras mounted around the frames and a recording device to store a video footage (top right of Table 4). The accuracy of present eye movement tracking systems is typically within one degree of visual angle.

VII. WORK FLOW OF CLIMBING ACTIVITY RECOGNITION AND PERFORMANCE ASSESSMENT
In this section we present a range of computational techniques that have been applied for activity recognition and performance assessment in the surveyed literature. Building a system for activity and performance monitoring in climbing involves a series of steps that go from data acquisition and labeling to activity recognition and performance assessment. Fig. 2 shows the overall process. The majority of works on climbing activity recognition employs a HAR model which consists of three parts: data segmentation, feature extraction, and training and classification. First, the segmentation procedure uses a fixed-length sliding window to divide the sensor signal into different segments; subsequently the segments are manipulated in order to extract features (e.g., the average sensor measurement in a particular time window) that can be profitably exploited to recognize the activities; finally a machine learning classifier is used to label each segment with an activity, according to the values of the features. On the other hand, some researches, e.g. [16] and [55], do not segment the signal by using the sliding window approach but instead use a threshold-based procedure that directly identifies the start-and end-data points of the activity of interest. A summary of the analysed machine learning and analytical approaches for activity recognition is shown in the Appendix (Table 10). In general, indicators of overall climbing performance such as those of fluency and body movement coordination patterns are derived from the entire sequence of ascent data while activity-specific indicators such as plateau duration and the exploration index are calculated by using the output of the activity recognition model. In the following subsections, each of the steps of the overall process shown in Fig. 2 is presented.

A. DATA ACQUISITION AND LABELING
This step corresponds to data collection using sensors that are either worn by the climber, attached to the climbing equipment, or installed in the climbing infrastructure (see Section VI). Generally, sensors need to be calibrated before collecting the data for their proper functioning. For example, in [56], before eye tracking recording, climbers were asked to fixate upon three calibration points of set locations positioned within the scene of view. In the study by Orth et al. [61], climbers were asked to stand still with extended arms to the side for 10 s before each climb in order to calibrate the orientation of the body-worn IMUs. Several studies using cameras adopted a calibration frame consisting of vertical and horizontal ropes with marks every 1 m [29], [63]. Various software tools offer support for sensor calibration. Examples from the surveyed literature include iMotions software 1 for eye tracking glasses and Simi Motion software 2 for digitizing the position of each marker on video images.
Raw data at a high-frequency rate is usually produced by the sensors and then transmitted and stored on a platform where the analysis takes place. This platform can be a local computer or a remote server. The data can be copied from a body-worn device to a local computer through a wired connection such as USB or wirelessly via Bluetooth. The data can then be transferred to a server via an Internet connection. For example, Ladha et al. [15] developed a wrist-worn sensing platform that included an accelerometer and a memory 1 https://imotions.com/ 2 http://www.simi.com/en/ card to store the data recorded during a climbing session. Following the session, the data was then downloaded to a computer via a USB connector before being uploaded to a remote server. Alternatively, periodically transmitting the data to an analysis platform offers the advantage that data analysis can be performed as soon as data is available instead of waiting until the end of the climbing session; this is especially important for virtual coaching and safety monitoring applications.
In order to learn activity patterns, machine learning algorithms require training data. It is worth mentioning that significant effort may be required to annotate the training data acquired by multiple sensors with the correct activity label. This problem was mitigated in the study by Boulanger et al. [16], in which video recordings were manually annotated and synchronized with sensors by finding the optimal correlation between the sensor acceleration and the video-tracking-based acceleration. However, the authors cautioned that the wrong estimation of the delay between the frame-based manual annotations and the acceleration could have an adverse effect on the performance of classification.

B. PRE-PROCESSING
Raw sensor data often require pre-processing such as noise removal and separating the gravitational (low-frequency) component from the body acceleration (high-frequency) component.
In signal processing, a filter removes the undesirable component from the signal (for more information see, for example, [77]). Moving average is a simple filtering technique for reducing noise. For instance, this approach has been applied in [14], where acceleration data along the three dimensions were smoothed using a 50 ms time window. As an alternative, a Savitzky-Golay filter with a window of 140 ms was applied in [32] for smoothing the 3D coordinates of the COM. In a study on falling recognition by Tonoli et al. [42], the authors developed a method based on Kalman filter [78] which embeds a mechanism for filtering noise from altitude and acceleration measurements. Ivanova et al. [28], [41] generated a digital low-pass filter with a cut-off frequency of 0.25 Hz to separate the movement component from the gravitational component in each time series of triaxial acceleration signals.
The gravity component can also be separated from the acceleration signal using the gyroscope and magnetometer readings of an IMU. This procedure requires estimating the orientation of the sensor in the ground reference frame. Examples of popular and freely available algorithms for estimating sensor based three sensor information sources (i.e., accelerometer, gyroscope, magnetometer) are Madgwick's [79] and Direction Cosine Matrix (DCM) [80].

C. SEGMENTATION
The data segmentation step determines the segments of the pre-processed sensor data streams that are likely to include information about activities. From the literature [81], [82], VOLUME 10, 2022 FIGURE 2. Overall work flow of climbing data analysis for activity recognition and performance assessment.
[83], [84], some of the methods to tackle the problem of segmentation are: fixed-size sliding window, dynamically varied time window, energy-based segmentation, and semantic approach, which incorporates an ontological model to represent relationships for the derivation of complex activities.
From the reviewed literature on climbing, a typical way to perform segmentation is by using a sliding (or moving) window approach, by means of which the whole data sequence is subdivided into smaller time windows of fixed length; classifiers are then applied separately to each window (see Section VII-E). A range of window sizes has been used in the included studies, ranging from less than one second [17] to 10 s [43]. One challenge of the windowing technique is to select an appropriate size for the time window, which decides how often the features are extracted (see Section VII-D), thereby influencing the classifier performance. As an example, to recognize hold gripping from accelerometer signals, window sizes of 280 ms [17], 750 ms [55] and 5 s [15] have been used. In addition to duration, another important parameter of a sliding window procedure is the overlap between successive window frames. The overlapping technique is used to address the problem that segmentation can rarely exactly match the beginning or end of an activity. Two consecutive windows have usually between 50% [43] and 95% [41] of data in common.

D. FEATURE EXTRACTION AND SELECTION
Activity recognition often relies on features that represent statistical or mathematical quantities derived from the temporal course of the sensor data (time-domain features) or their frequency course (frequency-domain features). In order to derive frequency-domain features, a window of sensor data must first be transformed into the frequency domain, typically using a fast Fourier Transform (FFT).
An example of a study that used time-domain features is the one of Bonfitto et al. [43] where a set of 30 features have been defined to tackle falling recognition using an accelerometer and an altimeter on a climber's harness. These features have been derived from time windows of three types of signals i.e., the altitude variation, the (raw) acceleration, and a pre-processed acceleration signal. On the other hand, Ivanova et al. [41] used a combination of 60 time-and frequency-domain features for recognizing rope pulling. In contrast to these approaches, the studies of Kosmalla et al. [17] and Ebert et al. [55] required only one feature for hold gripping recognition.
While in these studies, the authors used domain knowledge to extract features, the study of Ladha et al. [15] employed a feature learning approach. In [15], the 3D data of both wrists obtained by segmentation are first concatenated into a unified representation. Then, a feature learning approach based on Restricted Boltzmann Machines (RBM) [85] is employed to calculate the feature vectors associated with time windows. At the end of the learning, the activation probabilities of the hidden units of the RBM are retained as feature representations for each time window.
The set of extracted features is sometimes first screened in order to identify those features that are the most informative and discriminating. For example, a trial-and-error approach was used in [43] to identify the best 23 features among the 30 initial ones. The resulting features are typically represented as a numerical array, called the feature vector, and used as an input to the classifier (see Section VII-E).

E. TRAINING AND CLASSIFICATION
The classification approaches range from simple thresholdbased methods to more advanced algorithms, such as machine learning (ML) algorithms, which can associate patterns in input features with each activity. In this section, the different classification approaches we have found in the analysed literature are presented within several sub-categories.
Before entering into the classification techniques, we first discuss the evaluation methods. The classical approach to evaluate the accuracy of an activity recognition system is using k-fold cross-validation (CV) [86]. The parameter k, whose value is typically less or equal to ten, determines the number of groups into which the dataset is split for training and testing the classifier. In [15] and [17], the authors split the dataset into groups taking also into account the information about the climber who produced records. Namely, whenever a dataset used for testing contained recordings of a certain climber, all the records from this climber were removed from the training dataset. In this way the authors investigated whether their recognition method is user independent, i.e., whether body of foreign user data is sufficient to accurately identify the target user's activity. In general, the overall accuracy of the classification is calculated as the average proportion of correctly classified windows (or samples forming the windows) in each cycle. A more detailed description of the system performance can be given by measuring precision, recall, and specificity. Precision captures the ability of the classifier to correctly identify instances of a certain activity class. Recall and specificity represent true positive and true negative rates, respectively.

1) THRESHOLD-BASED CLASSIFICATION
In threshold-based classification, a derived feature is usually compared against a predefined threshold or a set of thresholds to determine whether a particular activity has been performed. A learning step may be applied to find the thresholds that attain the highest classification accuracy while producing the least number of false identifications. For example, in order to choose two thresholds for identifying the start and end of each hold gripping occurrence in sensor signals, Kosmalla et al. [17] ran an iterative procedure on a portion of training data. A range of values was assigned to each threshold; they considered the best combination of thresholds to be the pair of values which resulted in the smallest difference between the actual number and the detected number of hold gripping.
The threshold-based approach has been applied to differentiate between static and dynamic activity, such as hold gripping and hold change, and immobility and traction. To differentiate between hold gripping and hold change, features derived from data produced by wrist-worn accelerometers [15], [17] or IMUs [55] have been used. For example, in [15], they calculated the so-called short-term energy from a window of acceleration data, i.e., the inverse of the Euclidean norm, while in [55] they calculated the sum of the acceleration standard deviations along each of the axes. When the energy was lower than the threshold, the window was classified as hold gripping. To differentiate between immobility and traction, a threshold-based approach has been applied to the hip acceleration signal [30]. From this signal the authors derived vertical hip displacement over time; stationarity corresponded to a sequence where the displacement was less than 0.15 m for a duration longer than 30 s.
The potential of the threshold-based approach for ascending classification has been demonstrated in the study by Ebert et al. [55]. The authors used the wrists' acceleration along the vertical axis [55]. They applied a threshold of 0 g to the mean of vertical acceleration generated from a window of 750 ms duration to identify if the corresponding arm was pointing upwards or downwards, whereby a value greater than 0 g indicated an upwards direction. The ascending activity was considered to begin from the first window when both arms were pointing upwards and finish as soon as both hands were pointing downwards for multiple windows.
Using an accelerometer and an altimeter attached to a climber's harness, Tonoli et al. [42] developed a threshold-based approach to recognizing falls. Their framework exploited a Kalman-filter-based model [78] to obtain estimates of the state vector variables, i.e., altitude, vertical velocity, and vertical acceleration. Once the estimates were available, a lower-bound threshold was applied to energy density estimated from the velocity vector to determine whether a fall happened.
The threshold-based approach has also been applied to identify visual fixations from gaze position data based on which insights into a climber's route previewing and route finding performance have been obtained. For example, in [56], the authors used a moving window that spans consecutive gaze points, determined by the minimum fixation duration of 90 ms; a window was classified as a fixation if the spatial dispersion calculated as the sum of the difference between the points' maximum and minimum x and y value, i.e., (max(x) − min(x)) + (max(y) − min(y)), did not exceed a threshold of 100 pixels.

2) HIERARCHICAL CLASSIFICATION
In hierarchical classification, a binary decision tree is constructed. It is generally handcrafted based on domain knowledge. At each node, a binary decision is made depending on the input feature, leading to either the final classification or to the following decision node.
The hierarchical classification scheme has been used to classify five climbing activities, i.e., traction, postural regulation, hold gripping, hold change, and immobility [16]. The same approach has been instrumental in assessing performance indicators, i.e., route previewing skills [56], variability of body movement coordination patterns [35], exploration index [16], [57], and temporal indicators of fluency [57]. The proposed scheme used threshold rules which were applied to cumulative sums of the log-likelihood ratio depending on the norms of acceleration and angular velocity from IMUs located on the left and right wrists, left and right feet, and hip. In addition to threshold-based rules, a Gamma distribution model was used to make the classification decision at each sensor. This decision indicated whether a limb or hip was immobile or mobile at a given time. These decisions from individual sensor nodes were combined in a binary decision tree in order to differentiate between the five activities. As an example, a climber was considered to regulate his posture when his limbs were immobile while his hip was mobile.

3) MACHINE LEARNING CLASSIFIERS
After extracting and selecting appropriate features from the time windows, a machine learning (ML) algorithm tries to learn a mapping from the input feature vector to the output activity class (training phase). Once the algorithm is trained, it can predict the unknown class of a new unseen input feature vector. Some classifiers, instead of outputting a predicted class, produce a numerical output, such as the class probabilities, from which the predicted class is determined, e.g., by selecting the one with the highest probability.
ML classifiers that have been tested for climbing activity recognition are decision tree, logistic regression, k-nearest neighbor (kNN), convolutional neural network (CNN), AdaBoost, random forest, and artificial neural network (ANN). Differently from hierarchical classification, in this VOLUME 10, 2022 case the decision tree is automatically built by a learning algorithm. Ladha et al. [15] applied several machine learning algorithms to the ascending activity recognition problem. They found that logistic regression outperformed other investigated classifiers, i.e., kNN and a decision tree classifier. Another study where the machine learning approach has been applied is that of Bonfitto et al. [43]. Their study demonstrated that falling can be differentiated with a high success rate and very few false positive instances using ANN.

4) DISCUSSION
This subsection has presented an overview of different techniques used to classify climbing activities. This range of techniques is summarized in Table 10. Given that the classification performance results have not been presented in each study, we could not perform a quantitative comparison of individual classifiers. In this section, we discuss factors, such as ease of development and real-time execution, that along with accuracy should influence the choice of a classifier. We briefly summarize the different techniques, providing information about the potential strengths and weaknesses of each approach.
The threshold-based approach has often been exploited given its simplicity and ability to effectively differentiate between static and dynamic activity. This approach typically uses features that are derived from fundamental knowledge about how some activity will produce a distinctive sensor signal, and it is usually easy to develop. In order to differentiate between a larger set of activities, it is required to use more advanced classification techniques which take one or more features as input.
The hierarchical classification scheme can distinguish between a range of activities based on multiple binary decision nodes. Promising results have been obtained by using this approach in one study [16] and further work is required to establish whether it is applicable to other climbing activity classification problems. In comparison to the threshold-based classification, the hierarchical approach can take a longer time to develop given that the exact parameters for making a decision at each node are obtained by examination and analysis of data in the training phase. Both threshold-based and hierarchical classification are generally executed with minimal computational resources and are therefore suited for real-time applications.
Classification approaches based on machine learning algorithms such as logistic regression, random forest, and k-nearest neighbors are also simple to develop and can be used to classify one or more activities. Similarly, artificial neural networks are a powerful approach, which demonstrated high levels of accuracy for fall recognition [43], though generally, they can be slow to train and difficult to implement. Real-time execution of ML algorithms may be slower than previously considered approaches due to the mapping of a feature vector to a set of class labels. The higher the dimensionality of the feature space is, the more computationally intensive the classification is. Moreover, the degree of overlap between consecutive sliding windows, which are routinely used in this case, is subject to a trade-off. Namely, the smaller the overlap the less frequently the subsequent stages of activity recognition are executed, which reduces the computational load, but also the less precisely the activity borders can be defined.
Finally, we note that there are many methods such as support vector machines, fuzzy logic, or hidden Markov models, which have been shown to be effective for a wide range of activity classification problems from sensor data (e.g., see [86] and [87] and references therein), but have not yet been tested in climbing studies. With the limited number of studies on climbing activity recognition, there is a considerable need for further work to establish the suitability of the different techniques for various climbing activity classification problems.

F. PERFORMANCE ASSESSMENT
This subsection describes computational methods for deriving performance indicators found in the examined literature (see Section V). Some performance indicators are computed on the sensor signal corresponding to the entire ascent, whereas others are derived from signal segments corresponding to a specific climbing activity. Examples of the former type include indicators of fluency and variability of body movement coordination patterns, while indicators of exploration, control, and stability, among others, require sensor data to be segmented according to the activity of interest. A correspondence relationship between a set of performance indicators and climbing activities is shown in Table 6.

1) INDICATORS OF FLUENCY
As already indicated (see Section V-A), climbing fluency has been assessed by spatial, temporal, and spatial-temporal indicators. The geometric index of entropy (GIE) has often been applied to quantify the spatial aspect of fluency. Moreover, external sensing approaches (e.g., motion capture system in [32] or frontal camera in [63]) have been used in these works. For example, in [32], the authors first calculated the position of the center of mass (COM) as the weighted average of the positions of the center of mass of nine body segments, then, the GIE was computed by taking the natural logarithm of two times the length of the pattern traveled by the COM divided by the perimeter of the convex hull around that path. When assessing temporal indicators of fluency, the ascent is segmented by identifying stationarity segments (e.g., for assessing average plateau duration in [30]), immobility and traction (e.g., for estimating immobility-tomobility ratio in [61]. Spatial-temporal indicators have been assessed by calculating the jerk (also called jolt) coefficient, i.e., the derivative of the acceleration with respect to time. Usually, a body-worn IMU sensor has been used to collect the acceleration data. In addition to calculating jerk from the hip acceleration data, Seifert et al. [64] calculated jerk coefficient from the hip angular velocity data. They observed a high correlation between the two jerk coefficients.

2) EXPLORATION
The exploration indicators have been derived after identifying hold change and traction activity segments in the recorded acceleration measurements by sensors placed on wrists and pelvis. Consequently, in [16], the authors considered a hold change to be performatory if it was performed in parallel with traction, otherwise, it was regarded as exploratory. By counting the frequency of exploratory and performatory hold changes, they calculated the exploration index as the ratio between the two quantities. Exploration has also been assessed through the duration of exploratory and performatory activities [56], [57].

3) INDICATORS OF CORE ABILITIES: POWER, STABILITY, SPEED, CONTROL, AND ENDURANCE
Indicators of core abilities have been assessed from segmented acceleration signals of wrists [15], [55] and head [14]. For instance, in [15] and [55], the segmentation consisted of identifying the ascent within a recording, using the sensor data of both hands, followed by identifying hold gripping segments. It is worth noting that these works are based on the assumption that a signal is composed of alternate hold gripping and hold change segments, thus, not accounting for other hand activities that may occur during an ascent, such as limb shaking or chalking. In order to assess a climber's ability to remain composed while gripping a hold (i.e., stability) and ability to smoothly move the hand in transitions between holds (i.e., control), Ebert et al. [55] used the mean and variance of identified segments i.e., a high value of these features in hold gripping segments indicated a lack of stability, while a low variance of these features in hold change segments implied a good level of control in hold transitions. On the other hand, in [15], the authors assessed stability by calculating the inverse of the variance of the first derivative of the jerk coefficient based on hold gripping segments. Speed of ascent has been assessed by the number of hold changes per second [15], ascent duration [14], and average velocity [66]. Endurance has been computed in [14] as the mean jerk coefficient of ascent acceleration data recorded from an ear-worn sensor.

4) INDICATORS OF HOLD-LIMB CONTACT PERFORMANCE
As we have seen, a number of indicators characterizing a climber's contact with a hold have been considered (Table 3). In a comprehensive study by Fuss et al. [45], the authors extracted a number of parameters from the 3-axial force-time signal returned by the transducers such as the contact time, the mean force (amplitude of the signal), the tangential and normal forces, a parabolic curve of the same impulse. As a next step, from these measures, the authors derived the indicators such as friction coefficient, the Hausdorff dimension, the impulse, the smoothness factor, etc. For instance, the smoothness factor was calculated by dividing the body weight by the mean of the absolute difference between the force-time signal and the parabolic curve of the same impulse. In several studies where pressure sensors were used [39], [49], the authors computed the loading force from the pressure values as a function of time. Then, the force signal was pre-processed to filter out the noise (see Section VII-B. From such preprocessed signal, Zampagni et al. [39] estimated the foot load by taking the amplitude (peak-to-peak) oscillations of the vertical force oscillations under both feet, while Balas et al. [49] took the sum of force-time integral for the left and right foot.

5) INDICATORS OF ROUTE PREVIEWING AND ROUTE FINDING PERFORMANCE
In the reviewed literature, route previewing and route finding performance have been often assessed by the duration and frequency of fixations, which are recognized from the activity gaze data, taking also into account the area of interest (AOI) such as a hold or the climbing wall [53], [56]. Interesting work is that of van Knobelsdorff et al. [36] in which they used a first-order Markov model to calculate probabilities of fixation transitions from one AOI to another. Based on this matrix of transitions the authors calculated gaze transition entropy during route previewing.

6) INDICATORS OF VARIABILITY OF BODY MOVEMENT COORDINATION PATTERNS
Detection of body movement coordination patterns has usually been performed from orientation data collected during an ascent by body-worn IMU sensors. As a pre-processing step, some works reduced noise or pre-computed the sensor directions in the Earth reference frame from recorded data (see Section VII-B). Clusters of coordination patterns have typically been explicitly defined (e.g. neck-hip coordination patters [63], and inter-limb coordination patterns [29]). Alternatively, Seifert et al. [67] used k-means clustering method to identify clusters of orientation data points corresponding to full body movement orientation patterns.

VIII. RESEARCH CHALLENGES AND OPPORTUNITIES
In this section, we highlight some of the challenges, as well as opportunities, related to aspects that need to be improved to have fully operational, reliable, and automatic systems in day-to-day climbing practice.

A. ACTIVITY RECOGNITION AND PERFORMANCE ASSESSMENT METHODS
Although a variety of methods with which many climbing activities can be recognized and assessed have been proposed and validated in the reviewed literature, still there is a considerable need for future work in this area. In Tables 7 and 8 we provide, respectively, suggestions for further work regarding those activities whose recognition has been addressed in the reviewed literature and those whose recognition has not been addressed yet.
Reviewed approaches for identifying hold gripping occurrences in accelerometry data have been unable to differentiate between hold gripping, limb shaking for relief, and hand chalking activities (see Section IV). Additional machine learning could be applied to recognize and eliminate non-hold gripping activities, for example, by exploiting the fact that limb shaking and chalking often occur with the arm downward. Such an approach would result in better hold gripping detection, which would in turn give a more accurate assessment of the activity-related performance indicators ( Table 6).
Assessment of a crucial aspect of performance such as endurance (resilience to fatigue) remains a challenging task. Although the results of Pansiot et al. [14] evidenced some ability of the proposed indicator to capture endurance, their analysis revealed a shortcoming in dealing with the climbs where a climber took long periods of resting during an ascent. To overcome this issue, additional machine learning could be applied to recognize and discard periods of stationarity. Moreover, the approach of Pansiot et al. has been evaluated using a small number of climbers and routes. Therefore, further work is required to develop and also validate indicators of endurance by collecting a large dataset of climbs. Furthermore, as Schmidt et al. [89] pointed out, more research is required to develop procedures for analyzing data at multiple levels of coordination such as body-gaze movement coordination.
In contrast to rock climbing, less attention has been given to the study of ice climbing activities in the surveyed literature. Notably, the activities of interaction with the climbing surface, i.e., tool swinging, hole hooking, and crampon kicking, have so far been only visually assessed from the video footage [30]. Automatic recognition of these activities would allow also automatic assessment of exploration indicators (see Section V-B).

B. SENSOR DATA COLLECTION
For an extensive collection of sensor data in the climbing domain, there are a few important challenges to be addressed. One of them is the development of lower-priced sensing devices for more extensive instrumentation of the climbing facilities. That would allow, for example, to measure contact forces at many footholds and handholds. Towards this goal, Bauer et al. [90] recently showed that route instrumentation with one-and two-dimensional force sensors can be a cost-efficient alternative to standard six-dimensional force sensors. On the other hand, Iguma et al. [91] showed early results that suggest that a 3D motion capture system can be exploited to simultaneously collect motion and force measurements. Another challenging problem is to develop procedures for collecting and aggregating data from multiple sensing modalities such as IMU sensors, cameras, and eye tracking glasses [89].
To the best of our knowledge, one set of publicly available climbing data has so far been released [91]. This dataset contains labeled 2D skeleton time series obtained from video recordings of a large set of speed climbing performances. Given that sensor data collection and labeling is laborious VOLUME 10, 2022 TABLE 8. Recommendations for future work concerning climbing activities whose recognition has not been addressed in the reviewed literature. and time-consuming, greater availability of public data for climbing activity recognition in the future could facilitate the development of new algorithms and methods for climbing applications.

C. APPLICATIONS
From the perspective of applications in the climbing domain, there is still significant scope for further work.
The exploitation of sensor data gives recommender systems exciting new application opportunities in the sports domain [92]. With the adoption of wearable sensors in climbing practice, large quantities of data could be recorded during training sessions. As an example, automatic recognition of a gym route which a climber successfully ascended [17], as well as assessment of various performance parameters (e.g. control, stability, speed, etc.) could be exploited by a climbing recommender systems [93] to suggest training routes that will assist in addressing weaknesses. It would be of further interest to investigate if such virtual training recommendations provide comparable benefits to that of a professional and committed coach as pointed out by Ladha et al. [15].
Activity recognition techniques could be used to build systems for securing climbers' safety in a climbing gym. Such systems could assist in injury and accident prevention by detecting potentially risky situations. For example, a climber may suffer injuries during a fall as a result of the rope overtightening (a consequence of hard belaying), potentially causing a strong impact into the wall. In a preliminary study, Munz et al. [88] hypothesized that fall duration and distance, among other features, would allow quantifying the softness of the belaying technique. Methods for fall recognition developed in [42] and [43] could be instrumental in developing these system. Moreover, future work should focus on the automatic assessment of belaying as another important factor for injury prevention.
Several studies have so far developed first prototype systems for automated climbing performance assessment [14], [15], [16], [41]. Utilizing such a system represents an important building block for digital cockpits for improving levels of climbing activity by providing and visualizing statistics to the user that are related to various aspects of performance (see Table 3).
It should be also noted that future research should focus on the development of real-time algorithms for climbing activity recognition. In climbing monitoring systems developed by Ladha et al. [15] and Kosmalla et al. [17] sensor data from a wristband are processed off-line on an analysis platform after climbing is finished. With real-time sensor data processing, the knowledge about the climbing activity would instead be immediately available, thus, for example, enabling a virtual climbing coach to adapt to the climber and make suggestions that would fit into the climber's current mode of training.

D. PRIVACY PROTECTION AND ACCEPTABILITY
For complex software systems, such as those described in Section VIII-C, protecting the privacy of sensitive user data produced by sensors is becoming a necessity with the new EU Data Protection regulations [94]. Moreover, a lack of users' trust in personal data privacy may reduce acceptance of the technology [95]. Recently, novel tools, guidelines, and frameworks have been developed to help application developers conform to established principles of software engineering for protecting users' privacy [96].
In addition to privacy preservation, several other factors relating to climbers' acceptance of technology have been identified [97], [98]. As an example, Mencarini et al. [97] concluded that wearable devices designed for climbers should support rather than substitute the competencies of expert climbers or help beginners acquire new abilities; they should be reliable, easy to carry, and not obtrusive of the flow of climbing activity.

IX. CONCLUSION
In this paper, we have surveyed the state-of-the-art of research on using sensors to collect climbing data, with the primary  focus on their usage for the recognition of climbing activities and the assessment of a climber's performance. We have presented a taxonomy of climbing activities emerging from the examined research works (Section IV), which we believe will help to perform new research in information and com-munication technologies for the sport of climbing, both for the recognition of activities that have not been addressed yet and for understanding the overall structure and relationship between different activities. We have presented a taxonomy of the main indicators of a climber's performance (Section V), VOLUME 10, 2022 and the techniques to practically compute them. We have offered a comprehensive view of the various types of sensors being used in the climbing domain for data collection (Section VI): external sensors (located in the climbing environment), embedded sensors (incorporated into the climbing equipment), and body-worn sensors (put on by a climber). We have also identified the key steps of the standard work flow by means of which the collected raw sensor data can be manipulated and analyzed in order to classify activities and derive performance indicators, using machine learning and/or analytical models (Section VII). We have also presented some of the research challenges and opportunities to advance the field (Section VIII). Based on the reviewed literature, the application of sensing technologies, along with data analysis methods, represents a great opportunity of providing technology tools that climbers can greatly benefit from.

APPENDIX PRISMA FLOW DIAGRAM AND SEARCH QUERY
See Figure 3 and Table 9.

APPENDIX ACTIVITY RECOGNITION APPROACHES
See Table 10.
MARINA ANDRIĆ received the Diploma degree in mathematics from the Faculty of Mathematics, University of Belgrade, Serbia, and the Ph.D. degree in computer, decision and system science from the IMT School for Advanced Studies, Lucca, Italy. She is currently a Research Assistant at the Faculty of Computer Science, Free University of Bozen-Bolzano, Italy. Her current research interests include pervasive computing systems, machine learning, and data analysis for sports applications. FLORIANO ZINI received the Ph.D. degree in computer science. He is currently an Academic Technologist at Smart Data Factory, the Technology Transfer Laboratory, Faculty of Computer Science, Free University of Bozen-Bolzano. Over the past ten years, his research activity has focused on the design, development, and evaluation of mobile and web-based information systems for personalized health care. Personalization is achieved through the application of human-computer interaction and artificial intelligence methodologies and algorithms, particularly machine learning. He has authored about 60 scientific papers on e-health, personalized information systems, grid computing, multiagent systems, and evolutionary computing. According to Google Scholar, his H-index is 21. Open Access funding provided by 'Libera Università di Bolzano' within the CRUI CARE Agreement VOLUME 10, 2022