Integration of an Arm Kinematics Hot Patch Onboard the Curiosity Rover

NASA's Mars Science Laboratory (MSL) mission has updated the Curiosity rover's flight software multiple times since landing on Mars on August 6, 2012. The most common patching method has been a hot patch, in which running flight software is modified after being copied into RAM from its persistent storage. The latest hot patch to be installed on Curiosity fixed an issue in the robotic arm software that computes generalized inverse kinematics. Additional unit testing performed since the start of the surface mission revealed that this software can sometimes produce erroneous solutions. The cause was identified as numerical instability in a quartic root finder. When the inputs to that solver are not well conditioned, floating-point numerical issues can cause erroneous roots to be reported. In theory, this could result in the robotic arm turret instruments being commanded to unintended positions, for example, below the terrain surface. Out of approximately 3.7 million unit test cases, 97.2 % of the position errors were below 5 mm. However, there were 16 test cases where the position error was greater than 20 cm, and the maximum position error was 1.2 meters. The patch was uploaded to Curiosity on sol 2642 (January 11, 2020) after the solution was developed, re-implemented as a hot patch, and validated and verified using Earth-based Curiosity testbeds. A checkout test of the patch was performed on Curiosity on sol 2657, and nominal use of the patch began on sol 2658. In this paper, we describe the steps that led to integrating the arm kinematic hot patch into Curiosity's flight software, from the discovery of the bug to the nominal use of the patch in flight.