HAM homomorphism for state abstraction

In the HRL field, there are several main methods such as HAMs, options, MAXQ. A main problem that exists in HAMs is its joint state space consisting of the cross-product of the machine states in the HAM and the states in the original MDP, which can not be completely solved by a subroutine-based state abstraction method. This paper analyzes this problem in detail, provides formal definitions of homomorphism in HAMs and proves the invariance of the optimal solution for HAMs. Several typical examples are analyzed and evaluated. The results show that HAM homomorphism can conquer this problem.