We implemented real-time image processing program using OpenCV library for Apple's iPhone4 smart mobile phone. Our image processing program can do various operations such as thresholding, adaptive thresholding, edge detection and contour detection. Convenient user interface was developed using Objective-C. Also, we implemented augmented reality program on iPhone4. ARToolKitPlus by Wagner was analyzed for each stage of the library before the implementation. Also, we experimented on augmented reality with smart phone by using VRToolKit which is implemented by Loulier. We confirmed that our created earth model with OpenGL ES is augmented smoothly in real-time. This research will be a basis for developing smart phone programs such as face recognition, marker recognition and augmented reality program.