A Physics-Driven Neural Networks-Based Simulation System (PhyNNeSS) for Multimodal Interactive Virtual Environments Involving Nonlinear Deformable Objects

While an update rate of 30 Hz is considered adequate for real-time graphics, a much higher update rate of about 1 kHz is necessary for haptics. Physics-based modeling of deformable objects, especially when large nonlinear deformations and complex nonlinear material properties are involved, at these very high rates is one of the most challenging tasks in the development of real-time simulation systems. While some specialized solutions exist, there is no general solution for arbitrary nonlinearities. In this work we present PhyNNeSS—a Physics-driven Neural Networks-based Simulation System—to address this long-standing technical challenge. The first step is an offline precomputation step in which a database is generated by applying carefully prescribed displacements to each node of the finite element models of the deformable objects. In the next step, the data is condensed into a set of coefficients describing neurons of a Radial Basis Function Network (RBFN). During real-time computation, these neural networks are used to reconstruct the deformation fields as well as the interaction forces. We present realistic simulation examples from interactive surgical simulation with real-time force feedback. As an example, we have developed a deformable human stomach model and a Penrose drain model used in the Fundamentals of Laparoscopic Surgery (FLS) training tool box. A unique computational modeling system has been developed that is capable of simulating the response of nonlinear deformable objects in real time. The method distinguishes itself from previous efforts in that a systematic physics-based precomputational step allows training of neural networks which may be used in real-time simulations. We show, through careful error analysis, that the scheme is scalable, with the accuracy being controlled by the number of neurons used in the simulation. PhyNNeSS has been integrated into SoFMIS (Software Framework for Multimodal Interactive Simulation) for general use.