# I. Introduction uman faces represent detailed sources of information such as identity, age, gender, intention, and reaction (Response of expression). Also, humans have a great inclination towards their face attractiveness [8]. Thus, there exist many cosmetic surgeries to improve face attractiveness by modifying face appearance, face color, and shape. However cosmetic surgery is more expensive and painful, therefore only rich people can afford it [7] [9] [10]. Extensive research is being carried out in this field for improving facial aesthetics. But face aesthetics is subjective. Thus, we need some solution that can generate or predict face appearance before surgery [13] [14]. The computer vision domain provides research for the improvement of facial aesthetics in the digital domain which is virtual improvement in facial aesthetics in images and video. We proposed a real time face feature reshaping technique for eyes, eyebrows, jaw, chin, nose, and lips. So that user can see their manipulated appearance in real-time. # II. Previous Work Face detection is the most important phase in computer vision problems. Michael J. Jones Paul Viola [30] proposed a face detection framework. We categorized the face morphing method into two types: Model-based and non-model-based methods. The model-based method includes various face shape models such as the Active appearance model [24] [29], the Active Shape model [24], the Constrained Local Model [21], the 3D morphable face model [2][3][8] [18] and many more. The base face model [19] and surrey face model [19] are the most widely used 3D face model. The face shape model is constructed from the face, morph the model then fits onto the original face. The model construction and fitting are more difficult tasks that required lots of computation. The modelbased method provides a good result for real-time processing so widely used in image and video morphing. D. Kasat et. al [9] proposed a real-time morphing system that morphs the face feature in realtime using a Kinect sensor to stream input video. The system was able to morph the face features such as the jaw, chin, nose, mouth, and eyes in real-time. The Active appearance model-based method and moving least square method [35] are used for image deformation. The system degraded the performance in some illumination conditions and produce delay. Yuan Lin et. al [3] proposed a face-swapping method that does not require the same pose and appearance of the source and target image. The 3D model-based approach is used therefore allowing any render angle of a pose. The 3D model is constructed from the user's uploaded image and the swapping method is used to replace the character of the face. The result is accurate when the target object has nonfrontal faces. But the method is not able to handle large illumination differences between the source image and target image. A non-model-based method such as face morphing using a critical point filter [6] finds the critical value of the face and modifies that value using a critical point filter (CPF). The filter can filter image properties such as depth value, color, and intensity. The method is developed for two face images of the frontal face. The landmark points used are few to curtail the processing time required for a greater number of landmarks and large image size. # III. System Overview This system is implemented using OpenCV with python 3.7.0. The main objective of this work is to reshape individual features of the face like eyebrows, eyes, nose, lip, chin, jaw, etc. in real-time with minimum delay. An image deformation algorithm is thus applied only on the face area of window size 200 × 200. The window size is fixed to reduce the response delay. The system flowchart is illustrated in figure 1. The working of the system begins with the capture of the real-time video using the device camera. As input to the system, we took the RGB video stream captured by the Sony Laptop (device) front camera at a width of 900 pixels, and convert the RGB video frame into a grayscale image for image deformation operation. We assumed that there is only 1 person (referred to as the actor) in front of the device camera, and take as input a set of face shape modification parameters from the user. Then, we detect the face using a face detection algorithm and identify the face features to localize the certain key feature point. Object detection using Histogram of Oriented Gradients with Linear SVM method [36] is being used to detect facial landmarks. Firstly, OpenCV is used to detect the face from the given input video frame, then pre-processed the image to maintain equal size with a width of 900 pixels. Fig. 2: Visualizing the 68 Facial Landmark Coordinates estimate the location of 68 (x, y)-coordinates that map to facial structures on the face. The indexes of the 68 coordinates can be visualized in figure 2. We can see that the red dots mapped to specific facial features, including the jaw, chin, lips, nose, eyes, and eyebrows. The end result detected facial landmarks in real-time with high accuracy predictions. The Moving Least square (MLS) [35] method is used to reconstruct a surface from a set of landmark points. MLS created deformations using affine similarity, and rigid transformations. These deformations are realistic and give the user the impression of manipulating the facial features in real-time with high speed. The method generates a new image frame by warping the corresponding pixels of the source points to the positions of the destination points. Image deformation performs in each and every video frame and generates an output video in real-time. The output from the system is live and a morphed RGB video stream where the required face shape modifications are performed. Fig. 3: Original image (left) and its deformation using the rigid MLS method (right). After deformation, the face eyebrow is reshape # IV. Result Analysis The result of the proposed work is projected in Figure 4 in which various feature such as eyebrow, eye, nose, lip, chin, and jaw are reshaped as per user requirement. The reshaped feature can be deployed by the common use for various applications such as in plastic beauty surgery in which user can visualize their face before cosmetic surgery. The user can change various parameters in real-time and visualize their face with a modified face feature. The system can also be used for face beautification applications, to have some fun with friends, to share on social media, and capture memorable moments. The best thing is we edit in realtime without using any external device. The system was evaluated for various profile face such as left, frontal, and right profile face. We also analyzed the time required to produce the output video. The result shows that in left and right profile face required more time to produce output video. Table 1 shows the Response time required to generate an output video of ten video frames in a millisecond. Figure 5 shows the time analysis graph of various profile faces. # V. Comparison with Previous Work Work done by D. Kasat et al. [9], in which we can see similar effects (as shown in Figure 8) on face feature are reshaped using Kinect sensor for input video. In our works, (as shown in Figure 4) the system allows users to reshape individual face parameters in real-time without using any external device with less delay. In the existing system, we need to configure and connect the Microsoft Kinect sensor with the device therefore, the system portability is less compared to the proposed system. The quality of output and input images compared with Structural Similarity Measurement Index (SSIM). The SSIM value of an existing and proposed system is shown in table 3. The existing system provides good quality results because they used Microsoft Kinect sensor for input video that provides a 3D view and depth of capture image. In our system, we used a device front camera that is enough to be capable good quality video. The graphical representation of the SSIM value is shown in figure 9. The comparison of the original face and deformed face with SSIM value shows in figure 8 and figure 9. # VI. Conclusion A real-time face feature reshaping for video sequence is introduced which allows reshaping of specific features like eyebrow, eye, nose, lip, jaw, and chin of the face which cannot be achieved by existing morphing technique. We proposed for the first time, a real-time video morphing system without any use of an external device. Our system provides portability that can be in entertainment applications, games, film production, medical, and beautification fields. With full flexibility and good accuracy, the system allows to reshape face features in real-time. The comparison with the existing method using the SSIM index qualifies the quality of generated output. Also, the analysis of the time required for the output of various profile face and illumination condition justify the proposed system. In the future, we will make an expert system that is used in medical applications to visualize modified faces before plastic surgery. 1![Fig. 1: System Diagram of Proposed System Therefore, we converted each video frame to have an equal size. The resolution of the input image allows us to detect more depth of video frames. In our system, we used 68 Primary and Secondary landmark points [33] as shown in figure 2. The landmark points are used to find the exact position of the face feature and used to reshape the features. The system extracts a set of x/y coordinates on the input face. These face landmark points are fed into the image deformation method that reshapes the face parameter as per the user requirement provided as input. The Facial feature is the lip, Right eyebrow, Left eyebrow, Right eye, Left eye, Nose, Jaw, and chin. Kazemi and Sullivan [33]proposed a One Millisecond Face Alignment with an Ensemble of Regression Trees based method used for face landmark detection. We used this method for accurate landmark detection on the user's face. We used a pre-trained facial landmark detector to](image-2.png "Fig. 1 :") 45![Fig. 4: (a) Original image (b) Left eyebrow is reshape (c) Right Eye Reshape (d) Nose Reshape (e) Lip Reshape (f) Chin Reshape (g) jaw Reshape](image-3.png "Fig. 4 :Fig. 5 :") 6![Fig. 6: Time Graph of Various Illumination Conditions](image-4.png "Fig. 6 :") 89![Fig. 8: Existing System (a) Original Image (b) Morphed Image with SSIM=0.73](image-5.png "Fig. 8 :Fig. 9 :") ![](image-6.png "") ![](image-7.png "") ![](image-8.png "") ![](image-9.png "") 1VideoResponse Time in LeftResponseTimeinResponseTimeinFrameProfile FaceFront Profile FaceRight Profile Face13426342332533325242642625245272425625232472624258242524925222510232423 2VideoResponse time in DarkResponse time in Nor-ResponseTimeinFrameIlluminationmal IlluminationLight Illumination14826272232221325232542524245292923627272373025258282429924272410283024 3SSIM value Existing System Proposed SystemResult 10.730.73Result 20.890.77Result 30.800.77Result 40.740.74Fig. 7: Structural Similarity Measurement Index Analysis * Edutainment 2017 a visual and semantic representation of 3D face model for reshaping face in images JDu DSong YTang RTong MTang J. Vis 21 4 2018 * Face swapping under large pose variations: A 3D model based approach YLin SWang QLin FTang 2012. July * Face morphing using 3D-aware appearance optimization FYang EShechtman JWang LBourdev DMetaxas Proc. Graph. Interface 2012 * Image morphing using deformation and patch-based synthesis JChen ZLong BLiu XLi ZYe 2nd International Conference on Image, Vision and Computing(ICIVC) 2017 * Face morphing using critical point filters JAreeyapinan PKanongchaiyos Ninth International Conference on Computer Science and Software Engineering (JCSSE) 2012. 2012 * MovieReshape: Tracking 8. and Reshaping of Humans in Videos AJain TThormahlen H.-PSeidel CTheobalt ACM Transacons Graph 29 5 2010 * 3D aware-automated video morphing RSLiao DLima HNehab PVHoppe Sander Comput. Graph. Forum 33 4 2014 * Real time face morphing DRKasat SJain VMThakare International Conference on Computational Intelligence and Networks (CINE) 2015 * Video-Audio Driven Real-Time Facial Animation YLiu FXu JChai XTong LWang QHuo ACM Transactions on Graphics (TOG) 34 6 2015 * Semi-automated video morphing JLiao RSLima DNehab HHoppe PVSander Comput. Graph. Forum 33 4 2014 * Video face beautification YZhao XHuang JGao ATokuta CZhang RYang 2014 IEEE International Conference on Multimedia and Expo (ICME) 2014 * A novel approach for virtual facial shape cloning and makeup reproducing C.-SChen Y.-JLin T.-AChen Y.-QChen C.-CChiang 2015 International Conference on Informative and Cybernetics for Computational Social Systems (ICCSS) 2015 * Digital face beautification TLeyvand DCohen-Or GDror DLischinski ACM SIGGRAPH 2006 Sketches -SIGGRAPH '06 2006 169 * D. Data-driven enhancement of facial attractiveness TLeyvand DCohen-Or GDror Lischinski ACM SIGGRAPH Papers on -SIGGRAPH 8 2008 * Avatar reshaping and automatic rigging using a deformable model AFeng DCasas AShapiro Proc. 8th ACM SIGGRAPH Conf. Motion Games -SA '15 8th ACM SIGGRAPH Conf. Motion Games -SA '15 2015 * A template-based approach to automatic face enhancement SMelacci LSarti MMaggini MGori Pattern Analysis and Applications 13 3 2009 * Gaussian mixture 3D morphable face model PKoppen Pattern Recognitation 74 2018 * paGAN: Real-time Avatars Using Dynamic Textures KNagano JFursund 2018 37 * R 2 -ResNeXt : A ResNeXt-Based Regression Model with Relative Ranking for Facial Beauty Prediction LLin LLiang LJin Int. Conf. Pattern Recognit 2018. 2018 * Automatic feature localisation with constrained local models DCristinacce TCootes Pattern Recognit 41 10 2008 * Automatic human faces morphing using genetic algorithms based control points selection SKarungaru MFukumi NAkamatsu Int. J. Innov. Comput. Inf. Control 3 2 2007 * Active Appearance Models TCootes IEEE Transactions on Pattern Analysis and Machine Intelligence 23 6 2001 * An accurate active shape model for facial feature extraction KWWan KMLam KCNg Pattern Recognit. Lett 26 15 2005 * Automatic Face Replacement in Video Based on 2D Morphable Model FMin NSang ZWang 2010 * A Survey on Image Matting Techniques JBoda DPandya International Conference on Communication and Signal Processing 2018 * A Survey on Face Beautification Techniques without Cosmetic Surgery JBoda DKasat Research Review International Journal of Multidisciplinary 3085 11 2018 * View morphing SMSeitz CRDyer Proc. 23rd Annu. Conf. Comput. Graph. Interact. Tech. -SIGGRAPH '96 23rd Annu. Conf. Comput. Graph. Interact. Tech. -SIGGRAPH '96 1996 * Fast and Exact Newton and Bidirectional Fitting of Active Appearance Models JKossaifi IEEE Transactions on Image Processing 26 2 2017 * Fast Multi-view Face Detection MJones PViola 2003 Mitsubishi Electric Research Laboratories * Fast Rotation Invariant Multi-View Face Detection Based on Real Adaboost BWu1 SLao Proceedings of the Sixth IEEE International Conference on Automatic Face and Gesture Recognition the Sixth IEEE International Conference on Automatic Face and Gesture Recognition 2004 * One Millisecond Face Alignment with an Ensemble of Regression Trees VKazemi JSullivan IEEE Conference on Computer Vision and Pattern Recognition 2014 * Histograms of Oriented Gradients for Human Detection NDalal BTriggs Computer Society Conference on Computer Vision and Pattern Recognition 2005 * Image deformation using moving least squares SSchaefer TMcphail JWarren ACM Transactions on Graphics 25 3 2006