Reinforcement Learning methods for preventive maintenance and machine replacement problem with second-hand equipment

The amount of second-hand manufacturing equipment available in the market is increasing, driven by two primary factors. First, with the rise of reconfigurable assembly lines, manufacturers frequently modify their assembly processes and sell their surplus equipment. Second, the growing ecological awareness fosters the development of marketplaces for second-hand equipment. While second-hand machines are cheaper, they are also less reliable, and they require more maintenance. This paper studies the impact of buying second-hand equipment on preventive maintenance and machine replacement policies. We model the degradation process of machines as a Gamma process, and we use a Markov decision process (MDP) to capture the relation among different actions and machine states. We consider two approaches to solve the resulting MDP, namely, Q-learning and policy iteration. Our experimental results suggest that the use of second-hand equipment reduces machine replacement and maintenance costs. Our results also show that manufacturers should buy second-hand equipment even when their cost is only slightly lower than the cost of a new machine.