Standards for the Integration of Online Laboratories With Learning Management Systems

Laboratory work is essential in most applied science and engineering specific coursers. The continuous advances in Information and Communication Technologies be able the use of online laboratories (OLs). Their use is increased due to the advantages they offer compared to conventional laboratories, as unfortunately verified during Covid-19 pandemic. OLs and Learning Management Systems (LMSs) are key technologies in the learning process field. Normally, the integration of OLs with LMSs is carried out using proprietary or ad-hoc solutions. Furthermore, there are standards that can be used for this integration, such as: LTI, IEEE P92741.1 (xAPI), SCORM and IEEE1876. This work elaborates a survey which has been provided to the experts of online laboratories for them to complete. Gathered information allowed authors to diagnose the level of knowledge and use of the main standards for creating and integrating OLs. In fact, the analysis of this information confirms that there is a lack of criteria for selecting one standard against the others and how use it in the online laboratory development. This work gives guidelines for selecting the most appropriate standard according to desired characteristics of the online laboratory under development.


I. INTRODUCTION
Practical work is very important in many higher education grades, especially in the majority of the courses of Engineering and applied science careers where students should acquire knowledge over and above the theory [1], [2], [3].Public and Private Institutions promote online training by Information Communication Technologies (ICT) in education [4], [5].These technologies bring significant improvements in educational process [6], [7], including the The associate editor coordinating the review of this manuscript and approving it for publication was Nkaepe Olaniyi .benefits provided by the use of laboratories (Labs) [2], [8], [9], [10], [11].
The use of ICTs and in particular the use of the Internet has changed the way to perform practical work.The steady progress of ICTs promotes the use of online laboratories (OL), laboratories where students could work via Internet [12], which are also known as web labs since in most cases students access them through a web browser.The use of OLs provide several advantages against face-to-face laboratories such as spatial and time slot availability, security to students but also to equipment against certain types of experiments, extension of the use of rare resources, accessibility, share with other institutions, among others [11], [13].Many of these advantages have been more and better valued by the COVID-19 pandemic, which has determined many of the recent scientific works [14].Students via OL could interact with real and/or simulated systems [15].The former is known as Remote Lab (RL) [16], [17], the latter is known as Virtual Lab (VL).When students use both a real system and a simulation at the same time, this laboratory is known as Hybrid Lab (HL) [18].In the literature there are some works that present the use of OLs in Engineering Degrees [14], [19], [20], [24], [25] being present the modalities commented above (VL, RL and HL).They are implemented using a wide range of technological possibilities, but when students use them they must run software on their own device.In fact, this software acts as a Graphical User Interface (GUI) which allows students to interact with the OL to carry out the corresponding experiments.
Prestigious universities and educational institutions around the world have promoted and shared their own OLs, developed in National and International networks and repositories.For example: • iLab Project [21], [26] from MIT (Massachusetts Institute of Technology) • VISIR (Virtual Instrument Systems In Reality) Open Lab Platform [22], [27], which nowadays it is known as the VISIR federation [23], [28] • LiLa (Library of Labs) [24], [29] • UNILabs (University Network of Interactive Labs) [25], [30] • Lab2go [26], [31] • ISILab (Internet Shared Instrumentation Laboratory) [27], [32] • NetLab [28], [33] • DCL (Distributed Control Lab) [29], [34] International consortiums, like GOLC (Global Online Laboratory Consortium) [30], [35], encourage the development, share and integration of laboratories available remotely for education purposes.Additionally, there are organizations that have defined initiatives, networks, platforms, architectures and interfaces and put all of them available to other organisms in order to grant them access to experiments, create laboratory software or make possible connections to physical remote devices [31], [32], [36], [37].In fact, the term RLMS (Remote Laboratory Management System) manages remote laboratories [33], [38].It provides support in user authentication, authorization, management and registration, as well as, APIs to develop new laboratories.ILab, Labshare Sahara and WebLab-Deusto are examples that include RLMSs.There are also companies, such as Landslab, that can help universities to develop online laboratories, share their own online laboratories or use those of other institutions.To this end, they use architectures that allow multi-institutional use that provides great effectiveness [34], [35].
Learning Management Systems (LMSs), also known as Virtual Learning Environment, allow virtual teaching or elearning [36], [37].They offer a website where students and lecturers could communicate, create, share and use learning resources, perform evaluations, find links to other external resources, inveterate external applications, etc. [38].Currently, universities have an institutional LMS to carry out online teaching and to support face-to-face teaching.
LMSs and OLs offer complementary services whose convergence is a tendency for many years [39].This convergence is mainly based on presenting the services provided by LMSs and OLs in a unified way in the same environment; this requires an integration that has been discussed in some works [40], [41].
The wide variety of possibilities to achieve lab-LMS integration gives rise to multiple interpretations of the integration's concept.Someone could consider that integration is achieved just by adding a link to the laboratory in an LMS but others do not.The minimal and simplest integration between an LMS and an online lab is done when students require to access the lab through the LMS.The integration process becomes more complicated when a large number of technological approaches to implement OL are considered [42].The integration of these systems with the institutional LMS is often based on ad-hoc solutions, although standards-based solutions are gradually appearing in the literature [43], [44].
The use of standards is highly valued thanks to the advantages and beneficial properties they provide [45].This is also the objective of the e-learning technological standards, although always referring to systems related to education [46].IEEE is a clear example of an organization that is committed to the use of standards in education, as demonstrated by the existence of the IEEE Education Society Standards Committee.There are multiple organisms that are based on different categories to establish a standardization in the educational process [47], [48]: Accessibility, Architecture, Quality, Competencies, Contents and Evaluation, Digital Rights, Student Information, Interactivity, Metadata, Learning Process, Repositories and Vocabulary and Languages.Standards are very useful elements to create products and processes in technological fields.Standards are implemented through norms and requirements that help products developed in different environments to communicate with each other and be interchangeable.
There is an increase use of OLs but, what are the most developed types of OLs?What is the most used one?Additionally, there is an interest and a need to integrate OLs with LMS.This integration can be done using ad-hoc and/or proprietary solutions, but also using standard based approaches.In this sense, what kind of solutions do often OLs experts prefer?The standards, with which achieve the desired integration, are well known?Which are the most used standards for OL-LMS integration?What standard is the most appropriated for each OL desirable characteristic?
This work tries to answer these questions.To do this, the most used standards up to know in OL-LMS integration have been studied.Besides, authors developed a survey which has been provided to the experts of online laboratories for them to complete.The obtained answers allowed authors to have the enough information for answering each formulated question.
The rest of the work is structured as follows: Section II first presents what is meant in this paper by integration.This section also gives a brief overview or the most commonly used standards in the OLs development process.Section III presents the survey and the procedure followed till its formulation.Section IV illustrates the most relevant results diagnosed from the gathered information.Section V discusses the results obtained in previous section.Finally, Section VI includes the main conclusions and future trends of the work.In addition, 3 appendices have been added that complement this work, including information on the standards considered (A), the questions presented in the survey (B) and some help tables (C).

II. OL-LMS INTEGRATION BASED ON STANDARDS
This section includes two sub-sections.The first sub-section defines what authors understand as OL-LMS integration.Second sub-section lists the he most used standards to date by OL experts to achieve this integration are briefly described.

A. LAB-LMS INTEGRATION
In general, the integration can be understood as make somewhat to be part of a whole.This applied to OLs and LMSs implies that OLs should be part of the global teaching experience of the students that is carried out in the LMS, i.e.OLs become part of the LMS.Thus, both systems are combined in an effective way and work together as one.OL-LMS integration covers the integration between lab software, used by students to practice experience, and the LMS used as learning platform, normally managed by lecturers.
OL-LMS integration can occur in several non-exclusive ways [49], here are the most common ones: • Access: The LMS provides student access to the laboratory software by hosting it in the LMS itself or through an object/link that causes the laboratory software to be launched (located in a location external to the LMS).
• Integration with LMS resources: The LMS configuration allows you to create relationships between the laboratory software with other LMS resources.Based on the use and results obtained in the other LMS resources, restrictions/requirements can be established to access the laboratory software.On the contrary, the use and results obtained in the OL can control the possibility of accessing other LMS resources.This allows the creation of learning paths in which the OL is included.
• LMS-OL Data Exchange: Data transfer between LMS and laboratory software can be performed in one of two directions or simultaneously.This allows obtaining user data provided by the LMS to the OL and/or storing in the LMS from the OL data about usage, preferences, results and ratings obtained by the student when using the OL.
It is important to remark that in all cases the integrations between the lab software, which users run on their computers, and the LMS, occur directly or through interactions with an intermediate system.
When an OL is offered to students through an education platform, independently of the type of lab, it is necessary the use of a software that allows learner interaction with real or/and simulated systems.The software used by the students can be obtained directly or through a resource of the learning platform.In both cases it can be considered as a Learning Object (LO), defined by [50] as ''. . .any entity, digital or non-digital, that may be used for learning, education, or training''.
In this sense, this work considers the common integration problems that are independent of the type of OL.In the case that the OL is a remote laboratory, there are communications and interactions between the laboratory software and the real system with which the experimentation is carried out (directly or through an OL management platform).Since this work is based on the integration of OL with LMS, these interactions are outside of this work.

B. STANDARDS AND OLs
There are many technological standards and specific solutions that have been created with the aim of developing OLs, such as iLabs [21], SDS [51], Remote Interoperability Protocol (RIP) [52] or IEEE1876.However, all these solutions create a problem of lack of agreement on standardization [53].Additionally, there are many e-learning standards developed with a more generic purpose [46], [47].Some of them have been used to obtain OLs, such as LTI, xAPI or SCORM.Many of them can also facilitate the integration of online labs or other learning objects with the LMS [54], [55].These standards help achieve collaboration and understanding between OL and LMS.
Many other standards, to be considered, are related with the Accessibility category to allow the adaptation of the interface to the needs and requirements of users of ICT products and services.Examples are the EN 301 549 V1. 1.2 (2015-04) in Europe [56] and U.S. Section 508 regulations [57] in USA.
Standards related with communication protocols are also indispensable.The IEEE1876 standard itself recommends using standardized protocols to provide for data transfer between clients and laboratory.The most used are based on HTTP and WebSockets.Some of the solutions mentioned above are derived from these protocols, i.e.RIP protocol is based in GET, POST (HTTP methods) and Server Sent Events (SSE).
The most important e-learning standards related with the architecture and interactivity category that can be used to facilitate the integration of online labs in LMS are LTI, SCORM, xAPI, IEEE1876, and, potentially, cmi5 [58].
The most common uses of these standards in the integration of a laboratory with an LMS are the following: • LTI enables the launch of laboratory software (external to LMS) from an LMS for its execution with the possibility of interacting by exchanging user identification data from the LMS to the laboratory and data on the result of the work carried out from the laboratory to the LMS [59], [60].
• SCORM is used to create an LO of e-learning content in an LMS.This LO SCORM includes the laboratory software, when the LMS opens the SCORM it will be able to launch the OL along with other resources, controlling its sequencing.The OL can optionally exchange all data permitted by the SCORM standard data model with the LMS [41], [61].
• xAPI is used in programming the laboratory software so that when a student interacts with the OL and performs experiments, the laboratory software can track and record their interactions and experiences in a Learning Record Store (LRS).The LRS can be an LMS or an intermediate system accessible from the LMS [62], [63].
• IEEE1876 is used to create OL as a smart interactive learning object that can be accessed by methods defined by this standard as well as other method defined to store and retrieve, usually using xAPI [64], [65].
There is another standard, called Cmi5, that can be used to create a e-learning content that contains the laboratory software.This standard can be used from within or external to an LMS, the OL can interact with other systems as LMS to exchange data using xAPI.However, as far as authors know, there are no scientific papers related to the integrations of OL with LMS using the cmi5 standard, for this reason, and despite the potential that this standard presents, it was not considered in the survey.
The use of these standards is not exclusive, as already mentioned, there are works in which 2 of them have been used or cited [66], [67], even 3 [68], [69] and even all 4 [70], [71].
A much more extensive description of these standards can be found in APPENDIX A of this work.

III. SURVEY
An online survey has been designed and formulated in order to collect information about the opinion of world experts about OLs and the standards used to develop them.As commented in previous section, this work considers the most widespread ones for the survey questions formulation, which are: LTI [72], SCORM [73], IEEE P92741.1 (xAPI) [74], and IEEE1876 [75].
This survey was approved by the Ethics Commission of the University of Jaén, which issued a favourable report in March 2022.The survey is intended exclusively for OLs experts.It is not addressed to students because their knowhow and experience relies on a specific use of the OLs, but, they are not implied in the OLs development nor in its integration process.
Participants' email addresses were obtained by searching databases of scientific papers related to OL in STEM fields, some addresses were obtained directly from these papers and others using the names of the authors of the articles to search in the websites of the indicated organizations.The survey can offer up to 23 questions structured in the following 12 sections (S1-S12) shown by Fig. 1: • S1.Presentation and consent (1 question, Q1).A presentation of the survey is made and consent is requested to participate in it.
• S2 Personal data and OLs (7 questions, Q2-Q8).This section deals with collecting personal data from the respondent as well as the use of OLs, their participation in their creation and their preferences regarding the characteristics that the labs can offer.
• S3.Standards for integrating OLs with LMS (2 questions, Q9-10).This section collects information about preferences and knowledge regarding the use of standards to achieve the integration of OLs in LMS.
• S4 (LTI Standard I, 1 question, Q11) briefly describes the LTI standard and obtains information on the respondent's knowledge of LTI.
• S5 (LTI Standard II, 2 questions, Q12-13) obtains information about the use of LTI standard to achieve desirable requirements in the OL-LMS integration.• SCORM Block.
• S6 (SCORM Standard I, 1 question, Q14) briefly describes the SCORM standard and obtains information on the respondent's knowledge of SCORM.
• S7 (SCORM Standard II, 2 questions, Q15-16) obtains information about the use of SCORM standard to achieve desirable requirements in the OL-LMS integration.• xAPI Block.
• S8 (xAPI Standard I, 1 question, Q17) briefly describes the xAPI standard and obtains information on the respondent's knowledge of xAPI.
• S9 (xAPI Standard II, 2 questions, Q18-19) obtains information about the use of xAPI standard to achieve desirable requirements in the OL-LMS integration.• IEEE1876 Block.
• S10 (IEEE1876 Standard I, 1 question, Q20) briefly describes the IEEE1876 standard and obtains information on the respondent's knowledge of IEEE1876.
• S11 (IEEE1876 Standard II, 2 questions, Q21-22) obtains information about the use of IEEE1876 standard to achieve desirable requirements in the OL-LMS integration.• Section 12. Contact (S12, 1 question, Q23).This section requests the respondent's e-mail address.The questions of sections S5, S7, S9 and S11 only appear to be answered if in the immediately preceding section responder indicates that knows the corresponding standard (see Fig. 1).
This has two beneficial effects: on the one hand, it avoids bothering the respondent by not showing him/her questions on topics about which he/she has declared that he/she does not know.On the other hand, it avoids obtaining unreliable information.The answers obtained from the standards is solely from expert people.
A 5-level Likert scale (i.e.1-Nothing, 2-Little, 3-Quiet, 4-Much, 5-Essential) has been used when the questions are used to rate importance or help levels.APPENDIX B include the texts of questions and answers of the survey.

IV. RESULTS
The survey was offered to 375 experts in OLs via email and the 20.3% responded to it (76).The sampling error is acceptable and the sample is reasonably significant since the maximum error for a 95% confidence level is 10.05%.This has been calculated using the sample size and sampling error formulas for a population size of 375 and with a sample size of 76.So, the data from 76 participants carried out in April and May 2022 has been processed.
The origin of the participants in the survey is very varied; Emails and organizations from more than 10 countries on 4 continents have been detected, the vast majority are from universities, although there are also technology companies.To obtain the origin and institution of the participants in the survey, questions 4 and 23 were used, which asked for the name of the institution and its email address.From this information, it has been determined that the countries of origin of the experts are, at least: Argentina, Brazil, Colombia, Ecuador, USA, Taiwan, Austria, Germany, Italy, Portugal, Russia, Spain, Switzerland, and Australia.Many of the institutions of origin of the respondents are known for having developed or participated in many of the most important global initiatives related to online laboratories, such as Polytechnic of Porto, University of Sannio, Universidade Federal de Santa Catarina, Amrita University, University of Rosario, HFT Stuttgart, EPFL (École Polytechnique Fédérale de Lausanne), NTNU, TU Graz, Universidad del Magdalena, Embry Riddle Aeronautical University, UTPL, Bauman Moscow State Technical University (ex-Labicom), UNED, Universidad de Deusto, UPV, UPC, UJA, UPM, UAL, UCM, UM, HP, and LabsLand.
The raw data obtained from the survey has been analysed using Microsoft Excel and, in this way, direct results have been achieved in the form of tables and graphs.When necessary, several Excel functions have been used to process the original data with which indirect data and results have also been obtained in the form of Excel tables and graphs.This section includes the most relevant data obtained from the answers given by the OL experts in the different sections and blocks of the survey, as well as the obtained results.
Result 1.The degree of participation in the creation of OLs and the OLs use depends on the type of laboratory (S2, Q5-Q7).
Fig. 2 includes a bar graphic showing the use of OL in teaching (blue) and the participation on the creation of OL for teaching in the past (red) or currently (gold) and the table that include data used to create the graphic.
Fig. 2 allows deducing several ideas: • The virtual OL is the most used in teaching (78.7%), followed by remote (68%) and at a considerable distance by hybrid (38.7%).
• Creating RL or HL not necessarily has academic purposes.In fact, there are more OL experts who have participated in the creation of remote laboratories (84%) than in their teaching use (68%), The same occurs with hybrid laboratories (49.3% vs 38.7%).
• The number of OL experts who have only dealt with one type of laboratory is a minority, probably, it is due to they are researchers or teachers who are beginning their career in the field of OLs.• 1.3% of participants used OL in teaching but have never participated in the creation of OL.
• 5.3% of participants did not use OL in teaching nor did they participate in the creation of OL • 93.3% of participants have participated in the creation of OL, of which: • 92.9% used OL in teaching • 7.1% did not use OL in teaching Clarification on some of the categories included in Fig. 2: The 'Virtual' category is generic and refers to participants who have tried (used, created) virtual laboratories regardless of whether or not they have tried the other two types of laboratories (remote and hybrid), while the 'Virtual only' category ' is more specific and refers only to participants who have tried (used, created) virtual laboratories, excluding participants who have also tried other types of laboratories.For example, when discussing the use of laboratories, on the one hand, the 'Virtual' category refers to participants who have used virtual laboratories, regardless of whether they have also used remote or hybrid laboratories.On the other hand, the 'Virtual only' category refers to participants who have only used virtual laboratories and have never used remote or hybrid ones.This clarification can be applied similarly to the 'Remote only' and 'Hybrid only' categories.
Result 2. Identification of the most relevant characteristics considered by OL experts to integrate OLs with LMSs (from S2, Q8).
In order to find out what are the most important characteristics that an OL should have when integrating it with an LMS, the survey asked to rate the importance of a series of desirable features obtained from [76].The table in Fig. 3 shows the results obtained, it must be considered that the scale used is 1-5, APPENDIX C has included a table that helps transfer these values to a scale 0-10.These results are also displayed graphically (the limits of the axes used in the graph have been adjusted between the values 2.5 and 4.5, instead of 1-5, so that the differences can be better observed).It has been highlighted in green the characteristics with highest evaluation and in blue those with lowest mark.
In Fig. 3, a considerable distance (28.5% using APPENDIX C) can be observed between the most desired characteristic (4.41) and the least appreciated one (3.27).Considering APPENDIX C, all the desirable characteristics are considered important except two, Identification of students in the laboratory has been considered the most desirable one and Lab integration with other LMS resources has been obtained the less mark.There are 2 other characteristics that obtain high results such as 'Clear lab creation guidelines' and 'Sharing, compatibility and use of laboratories with other organizations.
Result 3.All the characteristics provided by the standards are considered important (from S3, Q9).
The advantages that can be obtained with the use of standards pointed out by [45] can also be obtained when  using standards to integrate online laboratories with LMS.The experts participating in the survey valued the importance of the advantages and beneficial properties provided by the use of standards to integrate OL with LMS.The obtained results are showed by a table and a graphic in Fig. 4.
Based on the observed results, all the evaluations made are quite high (lowest-highest rate difference is 6.75% using APPENDIX C).According to the opinion of OL experts, all the characteristics provided by the standards are important or very important for the development and the integration of OLs into the LMS.They mainly highlight 'Durability', 'Interoperability' and 'Reusability', which are the only ones that achieve a very important rating.
Result 4.There is a preference for solutions based on standards (from S3, Q10).
The vast majority of OL experts have positioned themselves according to the following phrase ''When you compare the use of standards with proprietary solutions, the formers bring benefits to the integration of OLs with LMS'' which states that the integration based on standards is preferable to integration based on proprietary or ad-hoc solutions (Fig. 5).
The average obtained was 4.39 out of 5, with a standard deviation of only 0.82, which can be understood as a broad and consensual agreement.

Result 5. Current situation of the LTI standard, (LTI block).
A) Level of knowledge and use (from S4, Q11).
Table 1 shows the knowledge/use level of LTI standard.The LTI standard is known by more than half of the OL experts (57.89%).Only slightly more than a third of the OL experts have used LTI to create OL (34.21%).However, the majority of OL experts who used LTI did so to create OL.
B) Rating of the desirable characteristics that LTI provides to LOs (from S5, Q12-Q13).All the participants who indicated that they were aware of LTI (57.89%,Table 1) rated the importance of this standard for the integration of an OL in an LMS.Table 2 shows the importance of LTI standard for the integration of an OL in an LMS (classified by LTI use level and total sum).
The mean provided by OL experts who know LTI is 4.32 out of 5, which is a high value.However, the opinion of experts who have used LTI to create OL is even better (4.5), which indicates the great importance given to LTI by the best qualified people to value it to integrate an OL with an LMS.This is a better rating than the rating given by experts who know about LTI but have not used it (4.11) or the rating of experts who have used LTI for anything other than creating OL (4.33).
The same participants (57.89%, people who know LTI) valued the help that the LTI standard can provide to achieve each of the desirable characteristics of an OL included in the table of Fig. 3. Table 3 shows the data obtained and includes the importance given to each desirable characteristic in an OL.
As commented above, there is a great difference between the importance given to the characteristics desired in an OL (column 'Importance').In fact, between the most desired feature 'Identification of students in the laboratory' and the least desired 'Lab integration with other LMS resources' the difference is 28.5%.
In order to correlate the opinion data of the OL experts in Table 3 and the importance indicated by all the respondents, the values have been reformulated with (1) to obtain the table in Fig. 6.

NSCV = New Standard Characteristic Value CI = Characteristic Importance SCV = Standard Characteristic Value IM = Importance Mean
In order to simplify the visualization and for a better understanding, in Fig. 6, data indicating the opinion of OL experts who have not used the LTI standard to create OL have been eliminated, keeping only the final column with the opinion of experts who know LTI and the column indicating the opinion of OL experts who have used LTI to create OL (the most reliable opinion as they are the greatest experts in the integration of OL with LMSs using LTI).For the same reason, the columns showing the standard deviation have also been removed and only the mean values have been kept.Fig. 6 shows that the OL experts who know LTI think that LTI offers a great help to achieve the ''Identification of students in the laboratory'', ''Sharing, compatibility and use of laboratories with other organizations'' and ''Laboratory integration in the LMS'', however, the help provided by LTI is quite minor to achieve ''Similar look and feel in lab and LMS'' and ''Laboratory interface customizable to user needs''.The opinion of OL experts who know LTI and have also used it to create OL, in principle the most reliable experts, agree with these results, although they think that LTI also helps a lot to achieve ''Laboratory access from LMS''.
The same procedure has been carried out in the following sections for the SCORM, xAPI and IEEE1876 standards with the same objective: the simplification of the visualization and a better understanding of the results.

Result 6. Current situation of the SCORM standard (SCORM block). A) Level of knowledge and use, (from S6, Q14).
The knowledge/use level of SCORM standard is shown by Table 4.
The SCORM standard is known by more than half of the OL experts (56.58%), however, most of the experts who have used it (23.68%)have not been to create OLs (only 6.58%), but have used SCORM for other purposes (17.11%).This is because SCORM is a well-known and widely used standard for e-learning content creation in general, and was not specifically conceived for OL.

B) Rating of the desirable characteristics that SCORM provides to LOs (from S7, Q15-Q16).
SCORM knowledgeable OL experts (56.58%, whether they had used it or not) rated the importance of this standard for the OL-LMS integration (Table 5 ).
The mean provided by OL experts who know SCORM is 3.63 out of 5, which is not a high value.However, the opinion of experts who have used SCORM to create OL is quite better (4.40), which indicates the great importance given to SCORM by the best qualified people to value it as a tool to integrate an OL with an LMS.There is a great difference between this rating and the rating given by experts who know about SCORM but have not used it (3.48) or the rating of experts who have used SCORM for anything other than creating OL (3.62).The authors think that this is because experts who have not used SCORM or have used it for purposes other than OL creation do not associate this standard with OLs or see its potential in this regard.
OL experts who know SCORM (56.58%, the same group) valued the help that the SCORM standard can provide to achieve each of the desirable characteristics of an OL included in Fig. 3.The table in Fig. 7 shows the results obtained after applying the formula of equation ( 1) and following the procedure used for LTI in a similar way.Fig. 7 shows that the OL experts who are familiar with SCORM think that SCORM offers a limited help to achieve the ''Identification of students in the laboratory'', ''Laboratory integration in the LMS'', and ''Sharing, compatibility and use of laboratories with other organizations'', however, the help provided by SCORM is quite minor to achieve ''Laboratory interface customizable to user needs'' and ''Student-student and student-tutor communications''.The opinion of OL experts who know SCORM and have also used it to create OL, in principle the most reliable experts, do not fully agree with these results, they believe that SCORM helps a lot to achieve ''Laboratory integration in the LMS'', ''Sharing.compatibility and use of laboratories with other organizations'', and to a lesser extent ''Laboratory access from LMS''.They also believe that SCORM helps less to achieve the ''Student-student and studenttutor communications'' and ''Similar look and feel in lab and LMS''.
Table 6 shows the knowledge/use level of xAPI standard.
The xAPI standard is little known (it is known by only 27.63%), furthermore, the number of LO experts who have used xAPI is even lower (13.16%),although there are many more OL experts have used xAPI to create OLs (11.84% of OL experts, 89.96% of OL experts who used xAPI) than who have used LTI for other purposes (1.32% of OL experts).

B) Rating of the desirable characteristics that xAPI provides to LOs (from S9, Q18-Q19).
Table 7 shows the opinion of xAPI-aware OL experts about the importance of xAPI standard for the integration of an OL with an LMS (classified by xAPI knowledge/use level).
The mean given by OL experts familiar with xAPI is 3.67 out of 5, that is not a very high value.In addition, the opinion of experts who have used xAPI is worse, only slightly worse the opinion given by experts who used xAPI to create OL (3.56), but especially bad is the opinion of experts who used xAPI for other purposes (2.0).This opinion is surprising, especially when compared to the high rating given by OL experts who say they know xAPI but have never used it (3.91).The data means that xAPI is not a highly rated tool to integrate an OL with an LMS.
The same participants (27.63%, those who know xAPI) valued the help that the xAPI standard can provide to achieve each of the desirable characteristics of an OL included in   Table 1 using a Likert scale from 1 to 5. The table in Fig. 8 shows the obtained results.
OL experts who are familiar with xAPI, as well as those who have used xAPI to create OLs, agree that the xAPI standard helps a lot for OLs to achieve the features of 'Access from LMS to performance information obtained by student' and 'Identification of students in the laboratory'.However, neither group believes that the xAPI standard can help much to achieve the rest of the desirable OL features on the list, especially 'Similar look and feel in lab and LMS'.

Result 8. Current situation of the IEEE1876 standard (IEEE1876 block).
A) Level of knowledge and use, (from S10, Q20).Table 8 shows the knowledge level of IEEE1876 standard.IEEE1876 is a standard known to less than a third of OL experts (30.26%), so it is not a well-known standard.
Furthermore, it has only been used by 14.47% of OL experts, but the majority of them with OL development purposes (10.53%).

B) Rating of the desirable characteristics that IEEE1876 provides to LOs (from S11, Q21-Q22).
All the participants who indicated that they were aware of IEEE1876 (30.26%, whether they had used it or not) rated the importance of this standard for the integration of an OL in an LMS.Table 9 shows the importance of IEEE1876 standard for the integration of an OL in an LMS (classified by IEEE1876 knowledge/use level).
Fig. 9 shows the rating of the help that IEEE1876 can offer for a LO to achieve each of the listed desirable characteristics.This rating was done by the OL experts who know IEEE1876 (30.26%).
OL experts who know IEEE1876 and, to a greater extent, OL experts who have used IEEE1876 to create  OLs, agree that the IEEE1876 standard helps a lot for OLs to achieve the characteristics of 'Clear lab creation guidelines', 'Sharing.compatibility and use of laboratories with other organizations' and 'Identification of students in the laboratory'.Both groups also agree that the desirable characteristics of OLs that the IEEE1876 standard can provide the least help in achieving are 'Similar look and feel in lab and LMS' and 'Automatic laboratory evaluation'.

V. DISCUSSIONS
The results obtained in the previous section have shown that the use of solutions based on standards is of great interest to the community OL experts, who prefer the use of standards for the integration of OL with LMS (result 4).It has also been possible to show the knowledge that OL experts have of the standards considered to achieve OL-LMS integration, the use that has been made with them for this purpose and how each of these standards can help to a different extent to achieve desirable characteristics in OLs.In this section the results obtained are explored to compare them and extract new ideas and meanings with the aim of discovering which standard may be more appropriate to achieve desirable characteristics in the OL-LMS integration.Fig. 10 include a bar graphic showing a comparative of the knowledge and use of the standards considered to integrate OL with LMS.Many debates can be made from this information, highlighting that the most unknown standards are xAPI (72.37%) and IEEE1876 (69.74%), while LTI and SCORM are known by more than half of OL experts (57.89% and 56.58% respectively).The order of the usage of the standards to create OLs is: (1) LTI (26.32%), ( 2) xAPI (11.84%), (3) IEEE1876 (10.53%) and ( 4) SCORM (6.58%), being SCORM the most used in other areas (17.11%).
Fig. 11 shows the percentage of experts with respect the total who are aware of the proposed standards and the number of standards they know.It also shows tables including the combinations of standards that are known when OL experts are aware of 2 or 3 of the 4 proposed standards.Almost 30% of experts do not know any of the 4 standards, which is not a positive fact.Almost 75% of experts who know at least one has a multiple standard knowledge.The most common multiple knowledge combinations of standards are LTI+SCORM (when 2 are known) and LTI+SCORM+IEEE1876 (when 3 are known).
Fig. 12 shows the percentage of experts who used at least one of the proposed standards to create OL with respect the total OL experts and the number of standards they used.It also shows tables including the combinations of standards that used when OL experts used 2 or 3 of the 4 proposed standards to create OL.This case is more worrisome since the percentage of OL experts, who have used at least one of the proposed standards, is only a little over a third (34.21%).In addition, most of them (57.69%) have only used one single standard.The most common multi-use combinations of standards are LTI+xAPI (when 2 have been used) and LTI+xAPI+IEEE1876 (when 3 have been used).
Fig. 13 shows two comparisons of the importance that OL experts give to the four standards considered to integrate OLs with LMSs.The upper bar chart shows the importance given by experts who are familiar with each standard, while the lower bar chart shows the importance given by OL experts who have used the standards to create OLs.
The evaluation of these latter improves the marks of all standards except xAPI.Authors consider this evaluation more reliable as these OL experts have been really worked with the standards in OLs.They rated LTI, IEEE1876 and SCORM as very important, while xAPI is rated only as important (APPENDIX C).
In [76] there are listed the desirable characteristics for OL experts when defining a new laboratory.Fig 14 and Fig. 15 show a quantitative score about the facilities provided by each standard to achieve every OL's desirable characteristic for those experts who know the standard and for those experts that also have been used it for creating OLs, respectively.
According to the opinion of the OL experts who know the standards, LTI is the standard that provides the most help to achieve a characteristic.It achieves the highest value in 7 of the 12 characteristics and, in addition, obtains the highest mean value (3.76).The next standard that achieves a higher mean rating value for help is IEEE1876 (3.64), which also achieves the highest rating in 3 characteristics.Finally, xAPI and SCORM only get the maximum rating on one characteristic each.However, OL experts who have used the standards to create OLs believe that IEEE1876 is the standard that provides the most help in obtaining desirable characteristics in OLs.This standard achieves the highest mean value (3.82) and the maximum value in 6 of the 12 characteristics, while the other standards obtain the maximum value in 2 characteristics each of them.
Despite the existence of previous works that analyze the possibility of integrating online laboratories with the LMS [39], [40], some even through the use of standards [76] and as far as authors know, there are no works that carry out a study of the standards used to achieve this integration such as the one shown in this work, or at least, the authors have not found any.For this reason, it has not been possible to carry out a comparative analysis with previous studies.

VI. CONCLUSION
Multiple conclusions can be drawn from this work, although the main one is the high degree of interest of OL experts for the use of standards in OL.In addition, OL experts who used standards to create OLs (most reliable opinions) ranked as very important 3 of the 4 standards considered.
As main conclusions about the usage and participation in the OL creation by OL type: The experts who create OL focus more on the development of remote and hybrid laboratories.However, the most used OL in teaching are the virtual laboratories.Authors hypothesize that this may be due to several non-exclusive factors, mainly three: 1-the lack of knowledge of teachers about the creation of remote or hybrid laboratories for teaching purposes leads them to work more on their creation; 2-the creation of remote and hybrid labs are more attractive to obtain research publications; and 3-the complex management and use of remote and hybrid laboratories imply the use of physical resources that are usually scarce and must be shared by the students.The type of laboratory that OL experts have treated the least (for both creating and teaching use) is undoubtedly the hybrid type.This may be because the hybrid type is the least known, the most recent, and the most complicated to create and use, as it is a combination of virtual and remote, which is not always possible to implement with positive effects that improve the performance of the OL.Although well-designed hybrid labs can greatly increase the learnability of the systems used.
The knowledge of standards is moderate.Despite the general positioning in favour of the use of standards in OLs, the level of knowledge of the 4 standards considered in this work cannot be considered to be very high, since although most OL experts know at least one, approximately 29% of them do not know any.On the other hand, most of the experts who know standards stated that they know more than one (almost 75%).The best-known standards are LTI and SCORM.
The use of standards in OLs is quite low.Almost two thirds of the OL experts (65.79%) have not used standards in their OL, when experts have used standards in their OL, more than half (57.69%) have only used one standard.The most used standard in OL is LTI (26.32%) followed at a considerable distance by xAPI (11.84%) and IEEE1876 (10.53%).
IEEE1876 is the standard that can provide the most help in achieving desirable characteristics in OLs in the reliable opinion of OL experts who used standards to create OLs.However, the other standards can also provide even more help on certain characteristics.Fig. 15 can help future OL developers to decide when choose which standard or standards based on the characteristics they want it to have.
Based on the results obtained, and in the opinion of the authors of this work, the most recommended standard for creating OL is IEEE1876.It suggests the use of other standards such as xAPI to complement it.cmi5 (evolution of xAPI and SCORM) may be an option to be considered to complement IEEE1876.The authors consider that the use of both standards (cmi5 and IEEE1876) could provide more help to create OLs that have the most desirable characteristics.It's an interesting idea to tackle in the future, although it has a big problem due to the low number of LMSs that support cmi5.
Finally, the lack of training and tools/guides that facilitate the use of the standards considered together is a problem.The OL developer has to fight with it, so it depends on his skill whether the solution is ad-hoc or not.Therefore, the need to define a document that guides based on the identified desired LO characteristics has been identified, which could increase if new characteristics are added to the list.This is what will be worked on in the next work.
In order to summarize these conclusions, theoretical, experimental and methodological contributions can be found in this work.The theoretical contribution is constituted by the formulation of the survey itself, as well as the search for experts who have been offered to participate in the survey and the work to obtain the necessary positive report from the ethics committee of the University of Jaén.The practical contribution is based on the analysis of the data obtained in the survey.This analysis has brought as one of the most important results that there is a growing interest in the use of standards, although unfortunately there is a lack of guidance documents that help with their knowledge and practical application in the case of OL.Finally, the methodological contribution of this work has been the identification of the standards that should be used to develop laboratories integrated with LMS based on the characteristics that are desired to be obtained.Objective: Integrate rich learning applications (tools) with learning platforms (LMS or other educational environments).

LTI terms:
• Tool: rich learning application that can be created by any company and method, and provided by a tool provider that can hosted the tool remotely from the tool consumer.
• Tool Provider: system that provide the tool to be used through the tool consumer.
• Tool Consumer: system that consumes the external tool provided by Tool provider, it could be any type of elearning platform or Web portal, although it is usually an LMS.Basic Operation (Fig. 16): A tool consumer (LMS) uses LTI to serve users off-platform content in the form of a learning object (LO) found in the LMS.When the user opens the LO (LTI Compliant Element) in the LMS, the LMS launches (via the LTI standard) a tool located on an external server (Tool Provider) that can be used by the user.The Tool can include an OL software.The LMS and the used tool (OL) can exchange limited information while the user uses the laboratory.
Pros Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
• Record Store (LRS): A web server system responsible for receiving, storing, and allowing access to learning records.
• Learning Record Provider (LRP): An xAPI client system that sends data to an LRS.Optionally, the LRP can create learning records while monitoring a learner's learning experience.
• Learning Record Consumer (LRC): An xAPI client system that accesses the LRS data to process it.
• Statement: A structured object that shows evidence for any sort of experience or event that should be tracked in xAPI as a Learning Record.
• Actor: Statement object property representing a user or group of users that is tracked using statements performing an action within an Activity • xAPI Profile: xAPI particular implementation based in a set of rules and documentation to be applied on a specific context.It uses to include particular vocabulary or vocabularies of terms for the special context.Basic Operation (Fig. 18): A learner has a learning experience of any kind and anywhere that is tracked by a trusted LRP that creates and formats xAPI Learning Records which are sent and stored to one or more LRS.Later, an LRC can access the LRS to obtain data from the learning experience for processing (to interpret, analyse, translate, disseminate and/or aggregate).https://github.com/adlnet/xAPI-Spec/blob/master/xAPI-About.md#partonePros: Popular and widespread [79], can be used at any time from any device.Continuous evolution.Used and/or recommended by other standards.
Cons: Limited to learning data transfer.Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
• Lab as a service (LaaS): Abstraction of the OL to make it available remotely via the Internet as a service, this abstraction is based on the fulfilment of a set of interface requirements to satisfy the first level of standardization of IEEE-1876.OL must provide well-defined services that enable interoperability with external applications and integrate internal functionalities.
• Learning Object (LO): Any entity, digital or non-digital, that can be used for learning, education, or training (IEEE Computer Society -Learning Technology Standards Committee, 2020).Learning Activity: An ordered set of tasks for a learner or a group of learners to acquire some knowledge or skills resulting in learning outcomes.Learning activities are also called experiments when learners work with OL.
Basic Operation: OL developers use IEEE-1876 to create an OL that integrates internal functionalities and provides well-defined services that enable interoperability with external applications to meet with IEEE-1876 layer one requirements (Fig. 19).This standard includes a series of recommendations indicating that the OL should be treated as an LO that should be integrated with a VLE (virtual learning environment).The VLE can be an LMS, an LRS, or an LMS that also performs LRS functions.In these cases, IEEE1876 recommends using xAPI to track and record the student's activities in the OL.
Pros: broad consensus, specific for OL.Cons: Recent, abstract instructions, needing of other standards.

APPENDIX B SURVEY QUESTIONS AND POSSIBLE ANSWERS
The section structure of the survey including the full text of the questions, the response options for each one and the type of question are included in the tables of this appendix.These are the questions' types used in the survey: Most of the questions used in the survey have been made using a Likert 1-5 style, the following tables have been included to make it easier for readers to read and understand 125286 VOLUME 11, 2023 Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.the data: the first Table (Table 10) shows the equivalence between the values of the Likert 1-5 scale, the values of scales 0-10 (to which we are most used), and an importance scale defined by the authors.In addition, the second table (Table 11) shows the equivalence between differences between values of the Likert 1-5 scale, differences between values of a 0-10 scale, and another 0-100% percentage scale.

FIGURE 3 .
FIGURE 3. Importance of the characteristics obtained with the use of standards for the OL-LMS integration (Table and graphic).

FIGURE 4 .
FIGURE 4. Importance of Lab Desirable Characteristics for LMS-integration (Table and graphic).

FIGURE 5 .
FIGURE 5. Expert agreement with the use of standards versus proprietary solutions.

FIGURE 9 .
FIGURE 9. Help that the use of IEEE1876 can provide in order to achieve each characteristic in an OL with adjusted values (Table and graphic).

TABLE 9 .
Importance of IEEE1876 standard for the integration of an OL in an LMS.

FIGURE 10 .
FIGURE 10.Multiple knowledge of standards.

FIGURE 11 .
FIGURE 11.Multiple use of standards to create OL.

FIGURE 12 .
FIGURE 12. Comparative of knowledge and use of considered standards.

FIGURE 13 .
FIGURE 13.Help that standards provide to obtain desirable characteristics in an OL (opinion of OL experts who know standard).

FIGURE 14 .
FIGURE 14. Comparative of Importance of standards to integrate OL with LMS.

FIGURE 15 .
FIGURE 15.Help that standards provide to obtain desirable characteristics in an OL (opinion of OL experts who used standard to create OL).

FIGURE 16 .
FIGURE 16.Basic use scheme of the LTI standard.
• MC-SA: Multiple Choice-Single Answer type question • MC-MA: Multiple Choice-Multiple Answer type question • M-MC-SA: Matrix Multiple Choice-Single Answer type question • M-MC-MA: Matrix Multiple Choice-Multiple Answer type question • SA: Short Answer type question • LS: Linear Scale type question APPENDIX C HELP TABLES

TABLE 1 .
Knowledge/use level of LTI standard.

TABLE 2 .
Importance of LTI standard for the integration of an OL in an LMS.

TABLE 3 .
Help that the use of LTI can provide in order to achieve each characteristic in an OL.
FIGURE 6. Help that the use of LTI can provide in order to achieve each characteristic in an OL with adjusted values (Table and graphic).

TABLE 4 .
Knowledge/use level of SCORM standard.

TABLE 5 .
Importance of SCORM standard for the integration of an OL in an LMS.

FIGURE 7 .
Help that the use of SCORM can provide in order to achieve each characteristic in an OL with adjusted values (Table and graphic).

TABLE 6 .
Knowledge/use level of xAPI standard.

TABLE 7 .
Importance of SCORM standard for the integration of an OL in an LMS.

FIGURE 8 .
Help that the use of xAPI can provide in order to achieve each characteristic in an OL with adjusted values (Table and graphic).

TABLE 8 .
Knowledge/use level of IEEE1876 standard.

:
Popular and widespread.Supported by most LMS.Create and package e-learning content that must be located in learning platforms (LMS) and can communicate with them based on a restricted data model.
To define methods for storing, retrieving, and accessing OLs as smart and interactive learning objects.