Classification of Facial Expressions based on Transitions Derived from Third Order Neighborhood LBP

Table of contents

1. Introduction

maging understanding is one of the most important tasks involving a classification system. Its primary purpose is to extract information from the images to allow the discrimination among different objects of interest. The classification process is usually based on grey level intensity, color, shape or texture. Image classification is of great interest in a variety of applications. Most of the image analysis problems are related to the neighborhood properties. Each pixel in a neighborhood or image is considered as a random variable, x r , which can assume values x r ? {0, 1?G-1}, where G is the number of grey levels of the image. The probability P (x r = x r | r), where r is the neighbor set for the element x r . The Fig. 1 illustrates different orders of neighborhood for a central pixel. Most of the research involved in image processing is mostly revolved around second order neighborhood only. This is because all the 8-neighboring pixels are well connected with central central pixels and the methods based on second order neighborhood are given extraordinary results in various issues. The present paper considering the difficulties and complexities involved in the third order neighborhood and derived a new, simple and efficient model for image analysis.

2. Derivations of Transitions on Trapezoids of tn-lbp

The proposed method evaluated transitions on "Trapezoids of Third Order Neighborhood of LBP (T-TN-LBP)" and based on this, derived various algorithms for the recognition of facial expressions. The proposed transition based T-TN-LBP consists of 7 steps as described below.

Step 1: Take facial image as Input Image (Img).

3. II.

Step 3 : Crop the grey scale image.

Step 4: The present research evaluated TN-LBP on each 5 x 5 sub image. The TN contains only 13 pixels of 25 pixels of 5x5 neighborhood as shown in Fig. 1. The TN-LBP grey level sub image is converted into binary sub image by comparing the each pixel of TN grey level sub image with the mean value of TN grey sub image. The following Equation.1 is used for grey level to binary conversion.

TN-Pi= ? 0 if P i < V 0 1 if P i ? V 0 ? for i = 1,2,3(1)

Where V 0 is the mean of the TN sub matrix

Step 5: The present research for classification purpose considered the two reciprocal trapezoids i.e. Top Left (TL) and Bottom Right (BR) trapezoids of TN-LBP. The Fig. 2 shows TL and BR trapezoids of TN-LBP. The each trapezoid pattern consists of 5 pixels. The pixels P 1, P Step 6 : Each trapezoid of TN-LBP consists of five bit patterns. The present research computed the transitions from 0 to 1 and 1 to 0. Generally in 5 bit patterns, 3 types of 0 to 1 and 1 to 0 transitions occur i.e. zero, two and four transitions. The proposed method, considers two and four transitions only, which accounts for 87.5% of patterns.

Step 7 : Based on frequency occurrences of two and four transitions, the facial image is classified as one of the category (Neutral, Happiness, Sadness, Surprise, Anger, Disgust and Fear).

4. Results and Discussions

The proposed transition based T-TN-LBP method is experimented on a database contains 213 images of female facial expressions collected by Kamachi and Gyoba at Kyushu University, Japan In the proposed "Transitions based on T-TN-LBP method", the sample images are grouped into seven categories of expression (neutral, happiness, sadness, surprise, anger, disgust and fear). Each T-TN-LBP consists of 5 bit pattern. It results a total of 32 bit patterns. This forms two-zero transitions i.e. the decimal value 0 and 31. The decimal values 5,9,10,11,13,18,20,21,22,26 results for 0 to 1 or 1 to 0 four transitions.The rest of the binary equivalent decimal values1,2,3,4,6,7,8,12,14,15,16,17,19,23,24,25,27,28,29, 30 results two transitions. The beauty of the proposed transitions on T-TN-LBP method is it evaluated the frequency occurrences of 2 and 4 transitions. This accounts a total of 87.5% of transitions.

The proposed method not considered the zero transitions which accounts for 12.5% of patterns. Further the proposed method evaluated the frequency occurrence of 2 and 4 transitions separately. The proposed method further evaluated sum of frequency occurrences two and four transitions of both TL and BR T-TN-LBP for the different facial expressions separately and listed in tables 1, 2, 3, 4, 5, 6 and 7 respectively. In the tables, STLT denotes sum of transitions (both 2 and 4) of Top Left Trapezoid and SBRT denotes sum of transitions (both 2 and 4) of Bottom Right Trapezoid. Further, the table also gives Total number of ( 2 and 4) transitions of both Trapezoids denoted as TBT in the above tables. Comparison of The Proposed t-tn-lbp with Other Existing Methods

Table 9 shows the classification rate for various groups of facial expression by the proposed T-TN-LBP method with other existing methods like feature-based facial expression recognition within an architecture based on a two-layer perception of Zhengyou Zhang [2], Facial expression analysis by Dela Torre et.al [3] and Facial Expression Recognition Based on Distinct LBP and GLCM by Gorti SatyanarayanaMurthy et.al [4]. These methods are implemented on Kamachi and Gyoba [5] at Kyushu University-data set and compared with the proposed method. From table 9, it is clearly evident that, the proposed method exhibits a high classification rate than the existing methods. The graphical representation of this is also shown in Fig. 5.

Figure 1. Figure 1 :
1Figure 1 : Neighborhood for a central pixel: (a) First Order (b) Second Order (c) Third Order (d) Fourth Order
Figure 2. Step 2 :
2Convert the RGB image into Grey scale Image by using HSV color model.
Figure 3. Figure 2 :
2Figure 2 : The TL and BR trapezoids of TN-LBP
Figure 4.
[1]. A few of them are shown in Fig.3.
Figure 5. Figure 3 :
3Figure 3 : Facial expression database (Kamachi and Gyoba at Kyushu University, Japan).
Figure 6. Figure 4 :
4Figure 4 : Classification Performance of various algorithms.
Figure 7. Figure 5 :
5Figure 5 : Classification chart of proposed T-TN-LBP method with other existing methods.
Figure 8.
P 1
P 4
P 2 P 3
P 5 P 6 P 7 P 8 P 9
P 10 P 11 P 12
P 13
Figure 9. Table 1 :
1
Year
D D D D D D D D ) F
(
Transitions on Top-Left Transitions on Bottom-Right
T-TN-LBP T-TN-LBP
S.No Image Name 2 4 STLT 2 4 SBRT TBT
1 KA.AN1.39 737 137 874 741 152 893 1767
2 KA.AN2.40 723 170 893 708 189 897 1790
3 KA.AN3.41 711 177 888 709 183 892 1780
4 KL.AN1.167 723 170 893 699 179 878 1771
5 KL.AN2.168 729 182 911 726 187 913 1824
6 KL.AN3.169 748 159 907 716 187 903 1810
7 KM.AN1.17 727 152 879 721 153 874 1753
8 KM.AN2.18 696 167 863 698 169 867 1730
9 KM.AN3.19 699 167 866 732 159 891 1757
10 KR.AN1.83 727 158 885 693 193 886 1771
11 KR.AN2.84 759 160 919 723 169 892 1811
12 KR.AN3.85 730 161 891 730 161 891 1782
13 MK.AN1.125 708 173 881 742 162 904 1785
14 MK.AN2.126 678 184 862 733 162 895 1757
15 MK.AN3.127 704 153 857 738 151 889 1746
Figure 10. Table 2 :
2
2014
Year
Transitions on Top-Left Transitions on Bottom-Right
T-TN-LBP T-TN-LBP
S.No Image Name 2 4 STLT 2 4 SBRT TBT
( D D D D ) F 1 2 KA.DI1.42 KA.DI2.43 831 788 158 186 989 974 770 784 163 175 933 959
3 KA.DI3.44 795 150 945 795 175 970
4 KL.DI1.170 820 167 987 749 203 952
5 KL.DI2.171 807 184 991 735 192 927
6 KL.DI3.172 742 178 920 785 173 958
7 KL.DI4.173 758 148 906 775 186 961
8 KM.DI1.20 822 169 991 756 171 927
9 KM.DI3.22 820 150 970 745 184 929
10 KR.DI1.86 819 171 990 763 145 908
11 KR.DI2.87 843 166 1009 726 172 898
12 KR.DI3.88 792 156 948 778 179 957
13 MK.DI1.128 833 144 977 794 151 945
14 MK.DI2.129 837 132 969 789 163 952
15 MK.DI3.130 806 160 966 764 183 947
16 NA.DI1.214 798 182 980 767 186 953
17 NA.DI2.215 834 168 1002 765 160 925
18 NA.DI3.216 834 164 998 773 167 940
19 NM.DI1.107 818 180 998 726 170 896
20 NM.DI3.109 821 177 998 737 189 926
21 TM.DI1.193 754 215 969 753 212 965
Figure 11. Table 3 :
3
Year
Transitions on Top-Left Transitions on Bottom-Right
T-TN-LBP T-TN-LBP
S.No Image Name 2 4 STLT 2 4 SBRT TBT
1 KA.FE1.45 796 195 991 844 194 1038
2 KA.FE2.46 811 178 989 820 183 1003
3 KA.FE3.47 783 192 975 815 189 1004
4 KA.FE4.48 778 206 984 826 210 1036
5 KL.FE1.174 778 197 975 832 192 1024
6 KL.FE2.175 784 205 989 851 173 1024
7 8 KL.FE3.176 KM.FE1.23 796 778 197 198 993 976 843 782 199 201 1042 983 ( D D D D D D D D ) F
9 KM.FE2.24 783 195 978 774 201 975
10 KM.FE3.25 787 181 968 809 185 994
11 KR.FE1.89 769 196 965 832 186 1018
12 KR.FE2.90 792 186 978 818 183 1001
13 KR.FE3.91 801 200 1001 830 197 1027
14 MK.FE2.131 795 184 979 844 165 1009
15 MK.FE3.132 802 180 982 832 174 1006
16 MK.FE4.133 793 165 958 812 193 1005
17 NA.FE1.217 793 188 981 801 190 991
18 NA.FE2.218 783 188 971 824 181 1005
19 NA.FE3.219 797 209 1006 856 173 1029
20 NM.FE1.110 773 200 973 867 162 1029
21 NM.FE2.111 783 186 969 820 177 997
22 NM.FE3.112 798 184 982 825 164 989
23 TM.FE1.196 796 208 1004 833 186 1019
24 TM.FE2.197 814 199 1013 807 208 1015
25 TM.FE3.198 793 189 982 823 200 1023
26 UY.FE1.152 792 199 991 842 172 1014
Figure 12. Table 4 :
4
Transitions on Top-Left Transitions on Bottom-Right
T-TN-LBP T-TN -LBP
S.No Image Name 2 4 STLT 2 4 SBRT TBT
Year 1 KA.HA1.29 847 207 1054 865 220 1085 2139
2 KA.HA2.30 847 193 1040 857 204 1061 2101
3 KA.HA3.31 823 210 1033 887 193 1080 2113
4 KA.HA4.32 832 221 1053 874 211 1085 2138
5 KL.HA1.158 809 251 1060 878 208 1086 2146
6 KL.HA2.159 844 208 1052 864 209 1073 2125
7 KL.HA3.160 839 204 1043 859 209 1068 2111
8 KM.HA1.4 839 217 1056 829 201 1030 2086
9 KM.HA2.5 849 185 1034 865 177 1042 2076
10 KM.HA3.6 782 238 1020 810 232 1042 2062
11 KM.HA4.7 831 215 1046 842 198 1040 2086
D D D D ) F 12 KR.HA1.74 823 217 1040 893 211 1104 2144
( 13 KR.HA2.75 831 204 1035 879 210 1089 2124
14 KR.HA3.76 819 199 1018 864 203 1067 2085
15 MK.HA2.117 827 211 1038 855 200 1055 2093
16 MK.HA3.118 831 185 1016 847 188 1035 2051
17 NA.HA1.202 835 208 1043 835 199 1034 2077
18 NA.HA2.203 833 205 1038 859 208 1067 2105
19 NA.HA3.204 863 196 1059 832 186 1018 2077
20 NM.HA1.95 836 211 1047 851 215 1066 2113
21 NM.HA2.96 842 202 1044 869 197 1066 2110
22 NM.HA3.97 857 186 1043 858 201 1059 2102
23 TM.HA1.180 826 208 1034 852 232 1084 2118
24 TM.HA2.181 817 236 1053 826 262 1088 2141
25 TM.HA3.182 823 223 1046 848 238 1086 2132
26 UY.HA1.137 846 222 1068 860 213 1073 2141
27 UY.HA2.138 861 212 1073 840 228 1068 2141
28 UY.HA3.139 824 213 1037 871 200 1071 2108
29 YM.HA1.52 833 220 1053 864 206 1070 2123
30 YM.HA2.53 826 214 1040 845 216 1061 2101
Figure 13. Table 5 :
5
Transitions on Top-Left Transitions on Bottom-Right
T-TN-LBP T-TN-LBP
S.No Image Name 2 4 STLT 2 4 SBRT TBT
1 KA.NE1.26 871 214 1085 876 227 1103 2188
2 KA.NE2.27 868 195 1063 898 211 1109 2172
3 KA.NE3.28 863 199 1062 892 223 1115 2177
4 KL.NE1.155 861 227 1088 864 222 1086 2174
5 KL.NE2.156 871 220 1091 857 233 1090 2181
6 KL.NE3.157 873 226 1099 887 220 1107 2206
7 KM.NE1.1 844 221 1065 898 195 1093 2158
8 KM.NE2.2 843 242 1085 861 215 1076 2161
9 KM.NE3.3 877 208 1085 866 225 1091 2176
10 KR.NE1.71 858 207 1065 872 223 1095 2160
11 KR.NE2.72 862 224 1086 876 217 1093 2179
12 KR.NE3.73 871 233 1104 878 211 1089 2193
13 MK.NE1.113 894 185 1079 854 219 1073 2152
14 MK.NE2.114 886 203 1089 870 221 1091 2180
15 MK.NE3.115 861 201 1062 926 173 1099 2161
16 NA.NE1.199 888 214 1102 856 202 1058 2160
17 NA.NE2.200 873 237 1110 857 233 1090 2200
18 NA.NE3.201 900 188 1088 886 204 1090 2178
19 NM.NE1.92 860 191 1051 878 230 1108 2159
20 NM.NE2.93 876 202 1078 878 213 1091 2169
21 NM.NE3.94 930 210 1140 856 205 1061 2201
22 TM.NE1.177 855 228 1083 865 237 1102 2185
23 TM.NE2.178 849 245 1094 833 289 1122 2216
24 TM.NE3.179 834 239 1073 882 240 1122 2195
25 UY.NE1.134 873 204 1077 879 213 1092 2169
26 UY.NE2.135 874 214 1088 854 231 1085 2173
27 UY.NE3.136 881 210 1091 873 212 1085 2176
28 YM.NE1.49 851 215 1066 904 194 1098 2164
29 YM.NE2.50 888 186 1074 872 212 1084 2158
30 YM.NE3.51 887 214 1101 863 223 1086 2187
Figure 14. Table 6 :
6
Transitions on Top-Left Transitions on Bottom-Right T-
T-TN-LBP TN-LBP
S.No Image Name 2 4 STLT 2 4 SBRT TBT
Figure 15. Table 7 :
7
Year
D D D D ) F
(
Transitions on Top-Left Transitions on Bottom-Right
T-TN-LBP T-TN-LBP
S.No Image Name 2 4 STLT 2 4 SBRT TBT
1 KA.SU1.36 1005 231 1236 981 235 1216 2452
2 KA.SU2.37 973 234 1207 974 233 1207 2414
3 KA.SU3.38 1006 225 1231 983 237 1220 2451
4 KL.SU1.164 946 265 1211 988 238 1226 2437
5 KL.SU2.165 975 236 1211 991 226 1217 2428
Figure 16. Table 9 :
9
Image Dataset Architecture based on a two-layer perception Facial expression analysis GLCM on DLBP of FCI Method Proposed Method (T-TN-LBP)
Kamachi and Gyoba
at Kyushu University, 80.29 91.79 96.67 100
Japan-data set
1
2

Appendix A

Appendix A.1 Conclusions

The present paper derived new direction for various problems of image processing by deriving LBP on the third order neighborhood. The third order neighborhood consists of 12 pixels excluding centre pixel. This may lead to huge number of patters i.e. 2 12 . The U-LBP on third order neighborhood leads to a negligible percentage of patterns. To overcome this, the present paper proposed transitions on T-TN-LBP. The T-TN-LBP considered 87.5% of transitions thus overcoming the disadvantage of U-LBP of third order neighborhood. The STLT, SBRT and TBT results of Table 8 clearly indicates an average facial expression classification result of 58%, 66% and 100% respectively.

Appendix B

  1. Facial expression analysis. F F Dela Torre , J F Cohn . Guide to Visual Analysis of Humans: Looking at People, . B Th, A Moeslund, V Hilton, L Kruger, Sigal (ed.) p. .
  2. Facial Expression Recognition Based on Features Derived From the Distinct LBP and GLCM. Gorti Satyanarayanamurty , Sasikiran , V Dr , Vijaya , Kumar . International Journal on Image, Graphics and Signal Processing 2014. 2 p. .
  3. Coding facial expressions with gabor wavelets. M Lyons , S Akamatsu , M Kamachi , J Gyoba . Proceedings of the Third IEEE International Conference on Automatic Face and Gesture Recognition, (the Third IEEE International Conference on Automatic Face and Gesture RecognitionNara, Japan
    ) Apr. 1998.
  4. Feature-Based Facial Expression Recognition: Sensitivity Analysis and Experiments with a Multi-Layer Perception. Zhe Zhengyou Zhang . International Journal of pattern recognition and Artificial Intelligence 1999. 13 (6) p. .
Notes
1
© 2014 Global Journals Inc. (US)
2
© 2014 Global Journals Inc. (US) Global Journal of Computer Science and Technology
Date: 2014-01-15