CCTV is one of the most widely used physical security technologies. A surveillance camera is a video collection device installed at a particular location and utilized for a variety of purposes. As CCTV performance has become enhanced recently, technology is being developed that attempts to perform automated processing through facial recognition using the facial information acquired from a CCTV system. However, if these technologies are exploited maliciously, privacy may be seriously violated. In this paper, we propose a security framework that protects personal information obtained from the detected facial area. We pointed out the security threats within CCTV system, and then propose a countermeasures. The key point of this framework is processing the mosaic or scrambling methods to the facial area, so that the facial information cannot be directly obtained without knowledge of the secret key. When the original facial information is needed, such as crime investigation, it can be obtained through reverse scrambling. This framework will contribute to protect privacy and develop the biometric-based physical security technology area.