Verification and Validation of ITER Interlock System Fast Architecture According to IEC 61508 Standard

The ITER interlock control system (ICS) requires the application of the IEC 61508 standard for all mission-critical (known as investment protection) control functions. Such functions must detect the events of the integrated physical processes and distribute them to the actuators with hard real-time constraints on the order of milliseconds or even microseconds. Systems able to achieve these timing requirements are often bespoke field-programmable gate array (FPGA)-based solutions, which are a well-known challenge to IEC 61508 processes. However, to minimize the variety of components and simplify the procurement process for an international supplier base, ITER decided to standardize the use of commercial off-the-shelf (COTS) devices. The COTS selected for the ICS was the FPGA-based CompactRIO National Instruments (NI) 9159 chassis (and several adapter I/O modules), provided by NI. This COTS requires the use of a high-level language (LabVIEW-FPGA) and the associated integrated development tools to develop the FPGA functionality. Therefore, it is necessary to ensure the required assurance that a COTS device is of sufficient quality, fit for purpose, and can be properly integrated into an investment protection control loop with the necessary level of systematic capability during the development process. This article describes in detail the method ITER uses to perform the verification and validation according to the IEC 61508 standard recommendations, for the logic configuration generated by LabVIEW-FPGA for these COTS, after the compilation of high-level language sources designed during the development.

T HE fusion investment protection environments demand systems that protect the machine against serious damage caused by a failure in the plant systems or dangerous operation of the machine. The ITER system responsible for this function is the interlock control system (ICS) [1]. This The IEC 61508 standard has been chosen as the reference for engineering best practice for ICS [2]. IEC 61508 is the functional safety standard applicable to all industries. The standard defines the methods of designing, deploying, and maintaining electrical/electronic/programmable electronic (E/E/PE) safety-related systems. Its use in assurance for investment protection requires adaptation, but more importantly here, this standard does not define clear guidelines for systems developed using high-level synthesis graphical programming, making it a challenge for ITER.
The following paragraphs describe the process carried out by the interlock ITER team in collaboration with the Instrumentation and Acoustics Research Group team of the Universidad Politécnica de Madrid (UPM) to analyze in-depth the applicability of the IEC 61508 standard on CompactRIO systems and its pluggable adapter modules. Section II presents the ITER fast interlock architecture. Section III justifies the motivation for this research. Section IV describes the technical details of the verification process done. Section V presents the methodology proposed as a result of the research work performed. Section VI contains a summary of the results obtained. Section VII contains the most important conclusions.

II. ITER INTERLOCK FA
The ICS is the system in charge of protecting the tokamak operation from reaching absolute engineering limits. Investment protection hazards are identified transversally across individual plants and reported at a central coordinating body machine protection panel (MPP), where "ITER Interlock Integrity Levels" (3IL) are agreed. These levels are commensurate to IEC 61508 safety integrity levels (SILs). ITER uses the same probability of failure on demand (PFD) targets as the standard. Hence, 3IL is an attempt to capture ITER  LabVIEW FPGA compile system (FPGA module). Reproduced from [4].
deviations from the standard without losing the connection that it nevertheless remains a core input to the design and realization of IO systems. The main distinguishing element being that for investment-protection, functional safety audits are internal to the ITER Organization.
The ICS performs functions with different timing requirements. Some requirements demand controllers capable of delivering an interlock signal with submillisecond response times, requiring hard real-time control cycles in the order of 100 µs. Such requirements are assigned to fast controllers.
In 2016, ITER decided that the fast controllers would use Xilinx FPGA-based COTS CompactRIO NI 9159 provided by NI. A study was conducted on failure, modes, effects, and diagnostics analysis (FMEDA) to analyze the behavior and provide an architecture solution. Specifically, a "doubledecker" solution was proposed [3]. The architecture defined consists of two NI 9159 chassis in 1oo2D configuration. The chassis communicate failure detection with each other and implement diagnostics and voter architectures, thus constituting the ICS-FA.
CompactRIO technology offers cost-saving solutions for hardware development and acquisition. It consists of a controller chassis containing a Xilinx 65-nm Virtex-5 and a series of pluggable and configurable input-output modules. The controller firmware is developed by means of the LabVIEW FPGA graphical programming tool. The LabVIEW FPGA enables developers to create their designs and translate them directly to configuration stream file (bitfile). The steps for the compilation of a LabVIEW FPGA firmware are described as follows [4] (see Fig. 1).
-Generation of Intermediate Files: The FPGA VI from LabVIEW is converted into intermediate files [very high speed integrated circuit hardware description language (VHDL) code] and prepared for sending to the compile worker.
-Queuing: The pending jobs are queued by the compiler server, and later, it sends the intermediate files to the compile worker for the compilation process.
-VHDL Compilation, Analysis, and Synthesis: The digital logic elements are created from the intermediate files by the compile worker.
-Mapping: The compile worker distributes the logic over the physical building blocks of the FPGA.
-Placing and Routing: The logic is assigned to the FPGA's physical building blocks, and the connections between logic blocks are established to meet the space or timing constraints of the compilation.
-Generating Programming File: The compile worker creates the binary data and stores it into a bitfile created by LabVIEW.
-Creating Bitfile: LabVIEW saves the bitfile in a subdirectory of the project directory and can download and/or run the application on the FPGA VI.
ITER developed a methodology based on using preconfigured templates to speed up the PIS-FA development process [5]. As a result, several PIS prototypes were developed. These templates are used as a reference to create systems with specific functionality.

III. CURRENT VERIFICATION AND VALIDATION (V&V) APPROACH
The application of the IEC 61508 standard for this system must be made carefully because it is not completely appropriate for the following reasons. The first reason is that IEC 61508 is oriented to safety environments, and in this article, the concern is investment protection. The second reason comes from the difficulty of applying a rigorous standard to a COTS system. CompactRIO and its pluggable modules were not developed with safety requirements in mind. The last and most important reason is that the standard offers compliance directives for E/E/PE systems developed with hardware description languages (HDLs), not for high-level synthesis languages or graphical languages, such as LabVIEW.
A set of independent reviewers performed a final design review of ICS that intensively reviewed the physical and design requirement. As a result of this process, a concern was raised on insufficient systematic capability that subsequently transitioned into an ITER risk.
LabVIEW generates VHDL code through which the FPGA is programmed, but it is not directly accessible by the user. The user only has access to LabVIEW graphics code, and as explained above, the standard does not offer directives for systems developed with high-level synthesis languages or graphical languages. Furthermore, it is complicated to verify the system in-depth without access to the autogenerated VHDL code.
ITER responded to the chit by declaring that CompactRIObased interlock systems should allow a verification process through the final code configuring the FPGA. The process should cover the following objectives.
-The correct operation of the individual unit function shall be verified.
-The correct integration and communication between the components shall be verified.
-The behavior of the complete system shall be validated.
To perform this task, it is mandatory to access the intermediate VHDL code generated by the LabVIEW tool and apply test techniques to it. Moreover, this process is recommended to be performed using certified third-party VHDL compilers.
As a result, ITER started a project to investigate in-depth the feasibility of applying the IEC 61508 standard to ICS-FA developed by LabVIEW FPGA tool. The final objective was to develop a process of V&V of the firmware developed through the LabVIEW tool based on CompactRIO. This process shall guarantee the high availability and reliability of the developed systems.

IV. RESEARCH DONE
Due to the reasons explained in Section III, an in-depth analysis of the LabVIEW behavior and the HDL code resulting from the compilation process was carried out. The main objective was to analyze the feasibility of the LabVIEW-FPGA workflow to implement a development cycle that meets the criteria and requirements of the IEC 61508-2 specification. For this purpose, a preliminary version of a complete IEC 61508-compliant V&V cycle was developed. Subsequently, this cycle was implemented on a previously developed system, analyzing the possible limitations and issues that might be encountered. Finally, based on the results obtained, a feasible V&V cycle for the target system was presented.
As a use case, a previously developed PIS prototype was chosen. Specifically, a PIS corresponding to the poloidal field and central solenoid coil's power converter protection system [5]. This plant system provides a redundant PIS-FA with a digital TTL input module for receiving action from CIS using a Manchester code link and two digital outputs 24-V actuator interfaces. The PIS includes interchassis communication diagnostics via Manchester code and serial peripheral interface (SPI). Also, it includes output self-diagnostics. Fig. 2 shows the system hardware and its interconnection.
To perform this process, the base element was to access the VHDL code autogenerated by the LabVIEW FPGA tool. For this purpose, a nondisclosure agreement was signed between NI/ITER/UPM to obtain access to the NI proprietary code.

A. V&V Plan
Initially, an analysis was made of the clauses of IEC 61508-2 annex-B and annex-F to determine which are applicable or not to the system in question. In particular, Table I lists the amount of items from table "Techniques and measures to avoid introducing faults during E/E/PE system design and development" (Table B.2) and "Techniques and measures to avoid introducing faults during ASIC design and development: user programmable ICs (FPGA/PLD/CPLD)" (Table F.2) of the IEC 61508-2 standard [1], oriented to a SIL-3 target. For a clearer understanding, the items have been grouped into functionality subgroups. Items marked as "not applicable" are directives that were found not applicable to CompactRIO systems. They were considered oriented to other types of systems. Items marked as "not selected" are the directives that have not been selected because their application is part of a set of options.
Based on the analysis carried out, a preliminary V&V plan, based on the V-model recommended by IEC 61508, was prepared.

B. VHDL Architecture
The first task was focused on the IEC 61508 clauses regarding code analysis and the correct use of the compilation tools. First, a detailed study of the steps run by LabVIEW FPGA to produce the firmware through the automatic compilation process was performed. Then, an attempt was made to replicate the FPGA configuration bitstream autogenerated by LabVIEW compilation using Xilinx tools. Finally, it was possible to create a TCL script that could generate exactly the same result using Xilinx ISE tool as the ones obtained by the compilation from LabVIEW.
Subsequently, a study was carried out to evaluate the appropriateness of the constraints introduced by LabVIEW to direct the synthesis, place-and-route, and timing analysis, such as clock generation, cells location, IO blocks, and timing restrictions. In this first stage, a systematic analysis was performed on the constraints file.
In addition, a study of the VHDL code architecture generated by LabVIEW was performed. A blank project was compared to the targeted system. As a result of this analysis, the interface and functionality of every block comprising the structural design were identified and described. Other fundamental aspects of the design were elicited, such as: -Management of clock and reset domains. -Implementation of synchronization mechanisms.
-The safe operation of finite state machines. -Identification of the design's dependence on encrypted VHDL code and IP cores. -Analysis of "Xilinx advisory" compliance, over and above design rules check (DRC) for Virtex-5 and, in case of noncompliance, it shall be evaluated their importance [6]. -The design's hierarchical structure in VHDL dependence on common resources. The analysis also shed light on the process followed by LabVIEW to generate the VHDL code, explaining in some detail the relation between the graphical code, including stencil effects, and the VHDL code.
Authorized licensed use limited to the terms of the applicable license agreement with IEEE. Restrictions apply.

C. Verification Reports
The next task was to cover the clauses of IEC 61508 regarding verification activities. The IEC 61508 requires the application of coding guidelines to constrain how VHDL language is used to ensure a safe synthesis. These rules aim at avoiding known vulnerabilities and ambiguities that may have unintended consequences on the system operation [7]. To achieve this goal, two courses of action were proposed. First, static analysis was performed on the VHDL code with Mentor Graphics DesignChecker tool. This tool automates the analysis by using a predefined rule set or creating a user-defined rule set. Unfortunately, no rule set related to nuclear investment protection existed, so a customized rule set was designed based on the aeronautical standard DO-254. This analysis covers assignment errors, ensures safe synthesis, and improves the comprehension and portability of the code. It is important to note that a few changes in code are usually needed to allow successful compilation with Mentor Graphics tools due to slight differences between Xilinx and Mentor synthesizers.
Second, the DRC analysis was performed. This analysis is carried out by Xilinx DRC tool and is executed automatically during the different phases of design compilation. The DRC runs mainly two analyses: a logical one immediately after the end of the synthesis phase and a physical one at the end of the place and route (PAR) phase.
The IEC 61508 standard considers performing the static timing analysis (STA) mandatory. The purpose of the STA is to check the compliance of the design with the constraints file and the device-specific constraints, and it must be guaranteed that no timing violations arises after the analysis. The STA is carried out by the TRACE tool available as part of Xilinx ISE design suite. The ISE compilation flow led by the LabVIEW compiler performs an STA automatically at several stages of the compilation, but a verbose analysis was additionally performed to achieve a more detailed verification. This analysis is performed after the mapping and PAR phases. It is important to note that useful and valid results after this analysis rely on the right definition of constraints by the LabVIEW compiler.
Likewise, the standard demands compliance with a set of items based on simulations of the firmware. Simulations were performed to check the correct implementation of the lowlevel functional requirements of the different modules. For designs developed with LabVIEW, modules are understood as the different SubVI and well-differentiated graphic code blocks. Therefore, unit-like tests focused on the set of inputs and outputs contained in the module to be verified.
Implementing this process tested if it is feasible to run functional simulations of the firmware through simulators external to LabVIEW. The third party tool Questa advanced simulator by Mentor Graphics was used to run the simulations. The testbenches were developed using the SystemVerilog language, and advanced testing techniques were incorporated, such as: -Code coverage.
A SystemVerilog simulation framework was created. It is a library of typical interfaces and transfer agent classes implemented to speed up the development of new tests and apply the V&V process to new templates. The structure of the tests can be seen in Fig. 3. A set of n tests were created, one test for each module to be tested. In addition, each test contains an assertion module that implements the functional coverage and verifies compliance with the system requirements intended to verify each test. In order to standardize the connection between the assertions and testbench for a given module and the device under test (DUT), it was necessary to use a SystemVerilog interface.

D. Validation Reports
Another important part of the IEC 61508 is the validation process. The main objective of the validation task was to prove, through the provision of objective evidence, that the application conforms to the specification of the functional properties of the system. Therefore, a set of tests were designed to validate the high-level function specifications using the complete system as DUT.
For this purpose, the first step is to perform validation simulations. These tests validate the behavior by generating stimuli on the lines dedicated to the interface between the systems, reading the system results and checking these results against the expected ones. The tests were developed using Sys-temVerilog, following the structure presented in the previous section, reusing to a large extent the set of libraries developed. SystemVerilog offers to the validation team powerful tools to create automated verification code and able to test complex behavior in an easier way. The standard establishes as mandatory to perform functional, post-synthesis, and post-PAR simulations to verify the fulfillment of the specifications after the subsequent compilation stages. The system validation simulations require generating inside Xilinx ISE tool the various simulation models related to the compilation phases involved in an FPGA development.
As a final stage, the 61 508 standard demand tests performed on the real system with the final configuration bitstream. The validation of the system over the real hardware was conducted using the double-decker connected to an ad hoc test system developed also using CompactRIO technology to assess the functional properties of the system. The validation  report of the real system also includes the system performance, evaluated as the measurement of the system response time. The test architecture developed for the validation of the real system is depicted in Fig. 4. The upper part of the dotted line corresponds to the DUT, and the lower part corresponds to the test system developed. A LabVIEW FPGA-based application has been designed to configure the FPGA logic of the test system, and several host applications have been developed to perform the different tests. These host applications developed for the test system can access the values from controls and indicators in the FPGA logic of the test system (highlighted in light red). Additionally, the test host applications are connected to the PIS host application of the double-decker and can access all its controls, indicators, and DMA channels (highlighted in dark green). The test system is in charge of stimulating and reading the inputs and outputs of the PIS double-decker.
V. METHODOLOGY PROPOSED Clause 7.1.3.1 of the IEC 61508 standard defines the system development lifecycle by the well-known V-model. As a result of the research presented in previous sections, a new system development lifecycle is proposed, which can be seen in Fig. 5. The V-model descending branch represents the system design and development. The system design starts by defining the CompactRIO system requirements and architecture from the complete safety system requirements architecture definition. The CompactRIO architecture definition for a particular application comprises the definition of the hardware required (CompactRIO systems, IO modules, cabling, etc.) and a firstlevel functionality breakdown into modules. The CompactRIO architecture definition must contain the properties expected for the system and the appropriate test plan to validate the system.
Most critical parts of the design, in case of failure of the automatically generated VHDL code, while analyzing the code against coding design rules, can be designed directly in VHDL and injected into the graphical code by means of componentlevel IP (CLIP). This resource allows complete control over the style of the VHDL code to avoid violating any coding rule. The next step is to develop the LabVIEW VIs and VHDL modules to implement the desired architecture. The module design can also be broken down into a hierarchical module design. For each module, documentation must be generated with the formal definition of the modules, the module properties, the hierarchical composition of the module, and the appropriate test plan and testbench to verify the modules implementation. The final step of the descending branch is developing the logic necessary to integrate the system modules.
The ascending branch of the V-model corresponds to the V&V activities. The first step is the verification of the modules. The modules are verified isolated from the rest of the design. Performing simulations for each module can be extremely time-consuming. This section is recommended to be performed using LabVIEW testing and debugging tools for the graphical code of the application. The code developed in VHDL must be verified and validated following the conventional flow, using testbenches in SystemVerilog that test the functional behavior of the design and verifying the formal analysis of the code with design checker. However, to finish this task, the proper integration of the VHDL modules in the LabVIEW project must be verified. For this purpose, a LabVIEW project is created with only the VHDL code developed by the user. Then, the project is compiled, and a simulation of the code corresponding to this application is carried out. In this way, it simulates the VHDL code developed by the user plus the additional logic introduced by LabVIEW for its integration.
The modules verification shall generate the modules testbench and the modules verification report, and all the nonconformities concerning the modules specification must be documented and added to the module documentation. Then, the whole system shall be validated using the simulation models corresponding to the different stages of the FPGA design compilation (functional, post-synthesis, and post-PAR) as explained in Section IV-Validation Results. The validation results and any deviation from the original specification shall be documented as the output of this phase. Finally, the system must be validated through tests on the final hardware. This step is recommended to be performed by implementing a set of ad hoc tests, as explained in Section IV-Validation Results.

VI. IMPLEMENTATION OF SEUS MANAGEMENT AND ISOLATION DESIGN FLOW STRATEGIES USING LABVIEW
Finally, research was carried out in search of tools to increase the safe-failure fraction of NI 9159-based systems. It explores several essential activities in the development of classical redundant safety systems to demonstrate that they can be implemented with the LabVIEW tool.
The first task was the study on the probability that a single event upset (SEU) affects the state of FPGA configuration bit. Specifically, this study was particularized for the device Virtex-5 XC5VLX110, the device mounted in the NI 9159 chassis [8]. Once this calculation was done, common mitigation strategies for Virtex-5 were investigated. The IEC 61508 standard recommends incorporating mechanisms to detect and mitigate possible system errors, including the possible action of SEUs. The "SEU mitigation controller IP" for Virtex-5, developed by Xilinx, is not available for new developments anymore because Xilinx has removed the support for families older than Virtex-6. This drawback can only be sidestepped by designing the SEU mitigation custom hardware. For this purpose, an analysis of the feasibility of designing and introducing an SEU detector and mitigator in applications developed with LabVIEW FPGA was carried out. Specifically, the following statements were considered: -Recommendations for the emergency maintenance strategy. dynamic reconfiguration port (DRP) memory. The following task was to seek tools to analyze and debug the internal operation of the firmware running on the FPGA. The selected way was integrating the ChipScope into a LabVIEW FPGA design and using digital I/O lines (NI 9401 module) to emulate the JTAG interface. It is an embedded logic analyzer to be instantiated inside the FPGA. This logic analyzer allows to probe and trigger the internal signals of the FPGA. This technique was based on the method presented by NIs in [9] and adapted for use in CompactRIO applications. Afterward, it has been possible to adapt the use of ChipScope for CompactRIO system. This demonstrates the feasibility of introducing an internal logic analyzer in an application developed on NI 9159 and its correct operation.
One of the techniques that might be demanded by the design of high reliable systems is to perform physical isolation between the critical parts of the design. It allows isolating the safe operation and the nonsafety parts of the E/E/PE system. However, this process has severe limitations in workflows based on LabVIEW FPGA. Therefore, the possibilities offered by LabVIEW FPGA in the use of user-defined constraints and the implications that would entail were explored.
Finally, it was necessary to find and then use a tool able to ensure the correct isolation of an FPGA region, verifying that the design meets the strict isolation requirements needed for high-reliability applications. The tool must verify that the isolated sections of the FPGA implement correctly the corresponding fences and trusted communications with the rest of the logic [10]. The most challenging part of this process is finding all modules related to the critical part of the design and creating fences and trusted communication with the rest of the system because techniques to implement this communication can require the insertion of code into automatically generated code, which is not possible. The only code that can be modified is the code of the CLIP directly written in VHDL. Constraints necessary to lead the PAR can be inserted into the CLIP code, affecting automatically generated modules. This task only requires knowing the module names, which is possible due to the consistent behavior of LabVIEW when assigning a name to these modules.

VII. RESULTS
Table II summarizes all the accumulated information about all IEC61508-2 items assessed in the project. The items classified as "compliance" are the requirements, in which compliance is considered demonstrated. The items classified as "partial" are the requirements, in which compliance is partial or additional information is required to demonstrate full compliance. The items classified as "noncompliance" are the requirements that the activities performed demonstrated noncompliance. Finally, the "not applicable" items correspond to not applicable requirements found after the research.
It is essential to understand that the items marked as "partial" cannot be fully evaluated, so they cannot be qualified as "compliance" or "noncompliance." It is due to the fact that there is no IEC 61508 compliance information for some of the soft cores used in the design. In addition, another problem encountered has been the lack of information about the requirements and interfaces of some NI internal components and protocols. Noncompliance is not necessarily an issue, for instance, lack of information on a communication protocol can be resolved by ensuring that the design treats the interface as a black channel and that the V&V plan ensures that the error rate is evaluated with statistical methods.
After performing the analysis on SEUs in Virtex-5, the device shows a nominal 165 failures in time (FIT) 1 /Mb for the configuration cells. This device has 22 192 configuration frames, implying 29 115 904 bits (approximately 29.1 Mb). Therefore, the device has a nominal susceptibility of 4801 FIT or an mean time between failures (MTBFs) of 23.8 years. This figure is calibrated for the geographical location and altitude of New York and is not applicable to all environments of the ITER plant, but it is a starting point for demonstrating that it is possible to develop and integrate an application based on NI 9159 an SEU detection and mitigation system. Finally, it has been possible to introduce user-defined constraints that affect the placement of LabVIEW graphic code. Therefore, going further in this process, it has been possible to isolate a VHDL module developed by the user and introduced in LabVIEW from the rest of the logic created by LabVIEW. The process is done by implementing fence regions and trusted communications between the different areas. Additionally, the possibility of verifying the correct isolation of an FPGA region by using the Xilinx Isolation Verification Tool (IVT) has been demonstrated.

VIII. CONCLUSION
The main conclusion of this article is the set of possibilities and limitations of the application of the V&V process according to the IEC 61508 standard on ITER ICS-FA. It ensures the necessary assurance to accept that the COTS devices used are suitable for operation in an investment protection environment. This article describes all the activities carried out to achieve this purpose.
Based on the results obtained, an optimized V&V methodology applicable to ICS-FA has been proposed. Therefore, it can be concluded that the V&V of the system has been possible with some limitations. On the one hand, it has not been possible to cover all the items of the standard, leaving the standard partially applied. On the other hand, it must be considered the great cost, in effort and time, required for the application of this process. In particular, the effort required to understand, manage, and implement this process for the first time is tremendously high. Even so, the subsequent number of times it is performed also requires a significant effort. Several critical systems in the current baseline, such as the disruption mitigation system or the superconducting magnet protection, are likely to undergo the described extremely cautious V&V process. However, the techniques identified can also benefit lower criticality systems if applied in a limited and targeted way.
Additionally, two necessary aspects of developing safety systems have been considered. First, the research demonstrated the necessity and possibility of integrating an SEU detection and mitigation system in NI 9159-based applications. Second, it discovered how to control the placement of the FPGA and isolate a VHDL module introduced in LabVIEW, exposing where single-chip fault tolerance can be relevant in the CIS-FA design.
For future design choices, the industry needs to consider planning such activities in advance, so that design teams can take full benefit of the findings. It is also advisable to pool resources between several organizations so as to stretch this highly valuable investment further, e.g., by creating a fusion catalog where for every item, there is a substantial engineering dossier that covers quality, continuity of supply, techniques, and measures for adequate design and V&V. Such strategic choices have been made by certain fission-plant operators.