In this paper, we present a keyword extraction methodology from handwritten Chinese document image based on matching and voting of the local topological structure. In the process, a handwritten keyword image is used as template, from which the local topological structure features of each character pixel are extracted, including crossing number, direction angle, gravity direction and gravity distance. A reference table is established based on these features. In the matching and voting phase, the features of each character pixel of handwritten document image are compared with the ones in the reference table. A vote is made in the document image when the crossing number and direction angle match each other and the voting location is determined using gravity direction and gravity distance. The handwritten keywords with local shape variation, slight rotation and zooming can be extracted from handwritten Chinese document image effectively. The same keywords from a same document can be used as training samples, which are used to model writers based on the signature verification methods to accomplish the goal of writer identification.