Chip-based adaptive skin color detection using trajectory constraints on hue
© Yu et al.; licensee Springer. 2014
Received: 16 July 2014
Accepted: 24 October 2014
Published: 12 November 2014
In a construction site, the safety of workers can be better secured if vehicle or robot can be properly maneuvered by using image-based gesture guidance. This kind of semi-automatic motion control is based on the features extracted in colors, outlines, textures, and motion intentions. Among these features, detecting worker’s skin color on face and hands could serve as an essential method for enabling robot to determine the region of interest (ROI). However, the performance of skin detection is usually unreliable due to interferences from reflections and shadows. Although many efficient skin color detection methods had been developed in past years, those are mostly based on complicated learning and statistical processes and are unsuitable for embedded systems. The exploration of a concise and efficient skin color detector, therefore, becomes a challenge for applications in mobile robots.
In this paper, we propose a novel adaptive skin detector on face and hands as a fundamental capability of a gesture tracking system. This approach enhances the detection performance of traditional HSV color space but only requires a low computing power. For the design criteria of small size, low-power, low-cost, and minimum computing resource usages on mobile robots, the entire detecting system is built on single field-programmable-gate-array (FPGA) chip. Meanwhile, besides the contributions of adaptive algorithms and FPGA chip designs, the proposed skin detector also employs a touch screen to designate expected skin color of worker as the human-robot interaction (HRI) in real-time.
The chip design of the FPGA is based on hardware circuits in register-transfer-level (RTL) for real-time image processing. A reasonable amount of hardware resource usages of FPGA have consumed 8% of logic elements (LEs) and 1.4% for embedded random access memory (RAM). Demonstrations with various pictures had indicated that sufficient ROI can be efficiently identified by using the proposed adaptive skin color detector.
According to our experimental results, the proposed adaptive skin detection can work well for non-ideal illumination. It has demonstrated the reliability and feasibility on supporting the embedded robotic control in the future.
In past years, skin color detection has been regarded as an essential component in human-robot interaction systems for analyzing human intentions (Kim et al. ; Luo et al. ). This methodology had become popular because it requires a lower computing effort than other image processing approaches even though skin color analysis is a fundamental area of pattern recognition. Recently, we have also developed a chip-based gesture tracking system by cooperating with motion detection to control mobile robots on construction sites (Yu et al. ). Our system simulated a robot to receive worker’s instructions by detecting a traffic light baton on his hand. Once our gesture tracking system confronts the situation of similar lighting sources on the same field, localizing traffic light baton involving regional information of worker’s face and hands will be an efficient and feasible scheme.
In this paper, we propose an adaptive skin color detection strategy by using single FPGA chip. We replaced image collection and training processes with a touch screen as the man–machine interface after considering the characteristics of skin colors of different human races and working environments. The worker can designate his skin color as a target through the interface, then the proposed adaptive skin color detection system, based on HSV color space, can define a preliminary threshold as a skin map. With the input of images, a novel adaptive algorithm named trajectory constraints on hue (TCH) is proposed to dynamically adjust the thresholds of skin color detection.
The contributions of our work rest on the economic circuit resource usages on chip, satisfactory real-time performance, high detection rate under non-ideal illumination, and convenience in installation. Comparing with the state-of-the-art technologies either based on PC simulations and FPGA chip designs, our skin color detection system can achieve superior detecting rate under severe tests. While nowadays efficient skin color detection mostly relies on the complicated algorithms, intelligent learning, skin texture analysis, working on high-end computers or a large amount of FPGA gates count; our concise skin color detection provides the answer to improve detection rates on embedded systems and miniature robots.
In the rest of this paper, the background of skin color detection will be first reviewed. Next, the relationship between HSV and RGB color spaces and the mechanism of the proposed skin color detection by using TCH will be presented. The configuration of chips, hardware resource usages, and the performance of the proposed skin color detection will be described in fourth section. The final sections contain the discussion and the conclusion.
In designing a skin color detection system, there are many color spaces that can be chosen to obtain the components of skin color. The initial idea comes from the use of thresholds to examine human skin color with the red (R), green (G), and blue (B) color space. Human skin color can be determined by the specific magnitudes of R, G, and B. This kind of RGB thresholding method has been recognized as unreliable in that the imaged skin color is highly dependent on the combination of luminance on the skin and results in poor resistance to non-ideal illumination, e.g., shadows and reflections (Yang et al. ). Although the design by using multi-intensity thresholds can mitigate the interferences of shadows and reflections, complicated calibrating processes for different intensity thresholds become another problem for implementation (Yu et al. ). Consequently, modern color detection systems prefer to use color attributes for detection, which are improved from the use of RGB color space for independent color components and discarding intensity, e.g., Normalized RGB, YCbCr, YUV, YIQ, and HSV (Chaves-González et al. ; Liu et al. ). With these kinds of color spaces, it is expected to detect skin without the being affected by intensity. Better detecting performance can be achieved by combining different color spaces in order to support each other (Kumar ; Liu and Shi, ).
However, the hue of skin color may vary in real-life operations. The interferences from shadows and reflections usually shift skin color beyond thresholds and degrade detection performance if intensity-independent color spaces were used (Zhu and Cai, ). This leads to skin color detection methods developed on the basis of adaptive or learning methodologies. For example, Liu employed a variable as the learning rate of histogram to adapt to different environments and then improved skin color detection by using morphological operations (Liu and Peng ). Similar work can be also found on (Li et al. ); where authors simply adopted a ratio to evaluate the deviation distance from centroid value of skin color. This proportion played the role as a threshold to refine their skin color maps in the database. Moreover, Khan et al. () designed a skin detector by using RGB color space and Bayesian-based classifiers training, which contained about 5000 samples of skin samples. Chenaoua and Bouridane () employed the Markov Random Field to estimate the distribution of skin pixels in images. Fotouhi et al. () combined with YCbCr, texture extraction, Gaussian mixture models, multi-layer perceptron, and artificial neural networks to perform a comprehensive skin detector. In summary, adaptive skin color detection usually has higher detection rate than methodologies based on traditional color spaces. Particularly for the system designed with learning capabilities, the regions of skin color can be reliably detected except small areas on chin, neck, and fingers.
Although a sufficient skin detection rate can be achieved by probability and learning, the real-time detection performance while running with limited computing power is also another critical issue in robotic applications. Intensive-computing algorithms may achieve a superior detection rate for a static picture, but it is unsuitable for a high resolution and operating at video frame rate. In the works of Toledo et al. (), Zheng et al. (), and Liu and Shi (), the detection of skin color was obtained from rules table, probability, normalized RGB, YCbCr and I1I2I3 color spaces for low computing effort. However, the shadow regions on skin could not be detected efficiently or they could only work under ideal illumination by lacking of adaptive capability. Zafarifar proposed a skin detector by using YUV color space, texture-adaptive analysis, and DSP slices on chip (Zafarifar et al. ). Their works featured additional capability of background eliminations, which were colored as skin. However, a large amount of hardware circuits on chip were required and detection performance in non-ideal illuminating environments was difficult to verify.
In general, skin color detection based on high-level statistics and machine learning usually requires huge computing resources. In spite of the sufficient performance with static image analysis, these methodologies usually result in low frame rate and therefore are unsuitable for tracking with embedded video systems without any modification (Fotouhi et al. ). On the other hand, traditional skin color detection working on basic color space thresholds cannot provide enough quantity of detection for real-life applications. These design dilemmas had motivated our work both on computer- and embedded-based systems, which provides excellent detection rate to provide ROI of targets for further pattern recognition processes.
Relationship between RGB and HSV
Many intensity independent color spaces can be utilized for skin color detection. Here we have adopted HSV color space after considering the advantage of separating skin color in hue, saturation, and intensity. The distribution of hue and saturation on a circular plan is convenient to develop an algorithm for examining the trajectory variation of skin color.
where MIN and MAX in equations (1) to (3) denote the RGB-intensity bound in [0,1]. The distribution of hue is usually plotted as a circular plan in four quadrants, and the plan radius is equivalent to the saturation value S. Hue is composed of red, green, and blue colors changing by every 120∘. When the HSV color space is plotted as a 3D model, the part right under the H-S circular plan is the intensity V of color space, which is in an inverted conical shape.
Based on above discussion, most colors can be simulated by altering the positions of colors in constraints. For example, the relationship of R > B > G results in a pink color and the B trajectory becomes H instead. If we continually increase the magnitude of B, the output of color will be closer to vivid blue in a clockwise direction for B > R > G towards the plan boundary. Since the behavior of color between upper and lower constraints is similar to hue, here we define the color in the middle as the dominant color. Therefore, it can be seen that human skin color is dominated by green.
The definition of H-S plan
According to equations (7) to (10), various skin color can then be obtained by adjusting the range of H and S. Chen et al. () had defined humans skin colors by using constraints of 0 ≤ H ≤ 50 and 51 ≤ S ≤ 173. Before this, Yang et al. () had also defined the skin color for the east Asian races by using the constraints of 14.4 ≤ H ≤ 46.8 and 69.615 ≤ S ≤ 149.55. Here we define the centroid point T by using 3 ≤ H ≤ 18 and leave S as undefined in that value is varied to different human races and can be dynamically determined by touch screen. Finally, the tolerance of skin hue is set for E H = ± 25 and the range of E S = ± 30 is for saturation testing based on our experiments. This arrangement leads to TCH’s adaptability on H which will mostly stop around yellow and only a small amount of sampled greenish color will be passed at low saturation.
Skin color detection by using TCH algorithm
|H i − H T | ≤ 25 and |S i − S T | ≤ 30:
|H i − H T | ≤ 25 and |S i − S T | > 30:
|H i − H T | > 25 and |S i − S T | ≤ 30:
Record the pixel saturation value S as a constant, which is from the last color T 0 before exceeding the initial skin color map F, as shown in Figure 3(a) and (b). This procedure ignores the effects from saturation besides F. In other words, S will be unvalued if H is too far away from centroid T.
If incoming image pixels colors are all outside the region of F, the proposed TCH will continually replace these image pixels’ saturations from T 0 —otherwise the system resets its operation to the initial condition. Such arrangement leads to varying trajectories of image pixels colors beyond both sides of F and will all follow a concentric circle, as shown in Figure 3(b). This results in a simple and adaptive skin color detection process which only needs to examine the variances on the pixel’s hue.
Examining the slope of line after obtained the coordinates of points T A to T D , as shown in Figure 3(c); T n , and T n+ 1 are derived based on equations (4) to (14), and the corrected hue trajectory of line will run on a circle as tangent. Once the slope m T of tangent exceeds the slopes constraints of four lines , , , and , the pixel color at point T n + 1 is defined as a non-skin color and then the system is reset to the initial condition. The examination mechanism can be represented in a decision tree shown in Figure 4. The m min and m max in the tree denote the upper and lower constraints of four lines’ slopes respectively. The slopes of , , , and on both sides of skin color map F can be summarized for three conditions at H i ≥ 0:
State_1: One positive slope (×1) and three negative slopes (×3).
State_2: Two positive slopes (×2) and two negative slopes (×2).
The detected skin color below line represent lip and blood vessel colors which can be judged as a part of the skin.
FPGA resource usages of proposed skin color detection
Cyclone IV E
Total logic elements
9,216 LEs/114,480 (8%)
De-mosaicking and camera controller
Touch screen control
Embedded multiplier 9-bit elements
Total embedded memory bits
57,400 bits/3,981,312 (1.4%)
The digital camera adopted in our design has a maximum of 5 M pixels resolution, but it was set for 800 × 480 pixels for use with the touch screen at a frame rate of 15fps and 30 MHz pixel clock. Meanwhile, the proposed TCH required 300 MHz clock speed and 37 clocks to run TCH during image pixel scans. Such arrangement leads to about four pixels delay showing on the touch screen.
In the first test scenario of Figure 6, see Figures 6(a) to (d), the interferences of shadows and reflections existed at the same time on tester’s face. It can be seen that the proposed TCH skin color detection system demonstrated excellent adaptive capability to detect skin areas with shadows and reflections. For the test result using HSV threshold, although it has been a popular method on embedded systems, the detection rates at testers’ chin and neck were still deteriorated by shadows. The detection performances between RGB and HSV thresholds had not shown much difference in this test. In the second test scenario, we tested TCH algorithm with extreme interference of shadow by huddling up a palm. The RGB and HSV thresholds both still had low detection rate in test. Comparatively, proposed TCH algorithm in this test could detect the skin color reaching the inner part of palm perfectly. In the third test scenario, we tested our skin detector by using green color to shift skin color moderately. The tester covered his palm with a green paper to obtain some green reflection on palm, as shown in Figure 6(i). It can be seen that the proposed TCH algorithm operating in such environment could still resist to green shadow and reflection. Again, RGB and HSV constant thresholds in this test still had low detection rate at fingers, where was covered by green shadow. After illustrated the success in resistance to shadows and reflections, the fourth test scenario simulated the condition of wearing a cap, as shown in Figure 6(m). In this test, the tester’s face was covered by shadow and some yellow reflections from inner brim of cap. The RGB threshold with this test scenario nearly failed to detect the tester’s face, and even the HSV threshold could only improve little. Nevertheless, our TCH algorithm could still work quite well in this test except a small part right under the brim of cap. In the last test scenario of Figure 6, the capability of skin detection under a dark environment was also tested, as shown in Figure 6(q). The lights in laboratory were nearly turned off in this test scenario. It can be seen the RGB and HSV thresholds could surprisingly detect skin region of the subject, and the silhouette of HSV threshold was sufficient for recognition on the face and palm. The proposed TCH skin color detection system under such severe environment was apparently superior to traditional HSV threshold for full scope detection rate of the subject’s face and palm.
Comparisons of resource usages among different skin detector designs
Chenaoua and Bouridane ()
Fotouhi et al. ()
Khan et al. ()
Liu and Shi ()
Zafarifar et al. ()
Cyclone IV E
Markov random field
Texture analysis & ANN classifier
Cascaded skin training & classifier
YCbCr & I1I2I3 Classifiers
YUV & Adaptive texture analysis
Adaptive HSV constraints
Resource usages (LEs/LUTs)
Embedded memory (bits)
The salient features of the proposed skin color detection system involve simple calibration processes, moderate hardware resource usage on chips, and efficient adaptability without complicated training algorithms. Comparing with the state-of-the-art technologies listed in Table 2, the proposed TCH adaptive algorithm cooperating with a touch screen is comparable to the methodologies using statistical and training classifier for high detection rate and low implementation effort. Besides, the combination of different color thresholds may achieve sufficient detection rate on embedded system by using a small amount of hardware resources (Liu and Shi, ), but such system can only work under adequate illumination and tedious calibration is still unavoidable because of the lack of adaptive mechanisms. After inspecting the skin detector results in Figure 8, the best skin color detection design for embedded system is the texture-based skin color analysis; see the performance in Figure 8(j). This kind of system is not only applicable for skin color detection but also provides additional functionality on background elimination, where it is colored with skin color. However, a large amount of hardware resources usages is required for complicated texture analysis. Consequently, it dramatically increases design difficulty on real-time processing speed and the implementation in a small chip.
Finally, the minor drawback of TCH arises from the delay of image processing and incurs coarse binary picture quality, as shown in Figures 6 to 8. With the pixel-based skin color detection, the image quality is sensitive to random noise in the camera, then the noise is enlarged horizontally due to the late update of pixels information during image acquisition. Although the coarse binary picture quality seems not affecting detection rate, the authors will determine how to improve processing speed with refined hardware circuits in the future, and a real-time and simplified pixel-based noise filter from our previous work may be also implemented (Yu et al. ).
In this paper, we have proposed an efficient adaptive skin color detection system. The proposed algorithm can derive an initial skin color map using traditional HSV color space after captured images from touch screen. Once incoming pixel colors exceed the region of default skin color map, the non-linear adaptive algorithm of trajectory constraints on hue (TCH) can dynamically adjust the range of thresholds for detection. Various severe test scenarios had been arranged to verify the capability of proposed skin detection. Our experimental results indicate that the proposed adaptive skin color detection is efficient and applicable to various effects from illumination conditions. A reasonable amount of hardware resource usages in the FPGA chip can allow the TCH skin color detection to be implemented on single FPGA chip and suitable for miniature electronic devices. The future works involve the refinement of circuits in economical gates count and processing delay improvement.
Supports from Ministry of Science and Technology, funded by the government of Taiwan under Grant NSC 99-2221-E-027-057-MY3, Ministry of Education Taiwan for the Top University Project to the National Cheng Kung University (NCKU), and Mr. En-Shi Lin are gratefully acknowledged.
- 10202blog:Photos collection. 2013. http://swjhgrade10202.blogspot.tw/2013_10_01_archive.html. Accessed 10. July. 2014.Google Scholar
- Chaves-González JM, Vega-Rodríguez MA, Gómez-Pulido JA, Sánchez-Pérez JM: Detecting skin in face recognition systems: a colour spaces study. Digital Image Processing 2010, 20: 806–823. 10.1016/j.dsp.2009.10.008View ArticleGoogle Scholar
- Chen YH, Hu KT, Ruan SJ: Statistical skin color detection method without color transformation for real-time surveillance systems. Engineering Application of Artificial Intelligence 2012, 25(7):1331–1337. 10.1016/j.engappai.2012.02.019View ArticleGoogle Scholar
- Chenaoua K, Bouridane A: Skin Detection Using a Markov Random Field and a new Color Space (IEEE international conference on image processing (ICIP 2006)). Georgia Institute of Technology, Atlanta, USA; 2006.Google Scholar
- Eastday (2013) Photos collection. http://big5.eastday.com:82/gate/big5/msnphoto.eastday.com/hdqxb2013/20130717_9/index.html. Accessed 10. July. 2014.Google Scholar
- Fotouhi M, Rohban MH, Kasaei S: Skin Detection Using Contourlet-Based Texture Analysis (4th international conference on digital telecommunication (ICDT 2009)). University of Haute Alsace, Colmar, France; 2009.Google Scholar
- Gamebase (2008) Photos collection. http://www.gamebase.com.tw/forum/5351/topic/83232878/1. Accessed 10. July. 2014.Google Scholar
- Gigacircle (2014) Photo collection. http://tw.gigacircle.com/1039866-1. Accessed 10. July. 2014.Google Scholar
- Khan S, Gupta GS, Demidenko S: Adaptive Classifier for Robust Detection of Signing Articulators Based on Skin Colour (6th IEEE international symposium on electronic design, test and application (DELTA 2011)). Institute of Electrical and Electronics Engineers Incorporated, Queenstown, New Zealand; 2011.Google Scholar
- Kim KK, Kwak KC, Chi SU: Gesture Analysis for Human-Robot Interaction (8th international conference on advanced communication technology (ICACT 2006)). Institute of Electrical and Electronics Engineers Incorporated, Gangwon-Do, Korea; 2006.Google Scholar
- Kumar A: An Empirical Study of Selection of the Appropriate Color Space for Skin Detection: a Case of Face Detection in Color Images (2nd International conference on issues and challenge in intelligent computing techniques (ICICT 2014)). Krishna Institute of Engineering and Technology, Ghaziabad, India; 2014.Google Scholar
- Li C, Liu YH, Shih HC: Adaptive Skin Color Tone Detection With Morphology-Based Model Refinement (9th international conference on information, communication, and signal processing (ICICS 2013)). National Cheng Kung University, Tainan, Taiwan; 2013.Google Scholar
- Lin C, Su CH, Huang HS: Colour image segmentation using relative values of RGB in various illumination circumstances. Int J Comput 2011, 5(2):252–261.MathSciNetGoogle Scholar
- Liu Q, Peng GZ: A Robust Skin Color Based Face Detection Algorithm (2nd international conference on information in control, automation, and robotics (CAR 2010)). Hubei University of Education, Wuhan, China; 2010.Google Scholar
- Liu G, Shi Z: Embedded Implementation of Real-Time Skin Detection System (International conference on transportation, mechanicl, and electrical engineering (TMEE 2011)). Jilin University, Changchun, China; 2011.Google Scholar
- Liu L, Sang N, Yang S, Hunag R: Real-time skin color detection under rapidly changing illumination conditions. IEEE Trans Consum Electron 2009, 57(3):1295–1302. 10.1109/TCE.2011.6018887View ArticleGoogle Scholar
- Luo RC, Chang SR, Yang YP: Tracking With Pointing Gesture Recognition for Human-Robot Interaction (IEEE/SICE international symposium on system integration (SII-2011)). Kyoto University, Kyoto, Japan; 2011.Google Scholar
- Nextmedia (2013). Photos collection. http://hk.apple.nextmedia.com/realtime/news/20130826/51675805. Accessed 10. July. 2014.Google Scholar
- Sagheer, A, & Aly, S. (2012). An Effective Face Detection Algorithm based on Skin Color Information (8th international conference on signal image technology and internet based systems (SITIS 2012)). Naples, Italy.Google Scholar
- Sina house (2011). Photos collection. http://sx.bbs.house.sina.com.cn/thread-851846640456301-2.html. Accessed 10. July. 2014.Google Scholar
- Sohu sport (2010). Photos collection. http://sports.sohu.com/20100320/n270968863.shtml. Accessed 10. July. 2014.Google Scholar
- Toledo FJ, Martínez JJ, Garrigós J, Ferrández J, Rodellar V: Sink Color Detection for Real Time Mobile Applications (16th international conference on field programmable logic and applications (FPL 2006)). Universidad Autónoma de Madrid, Madrid, Spain; 2006.Google Scholar
- Vanity fair (2014). Photo: The 2014 Vanity Fair Hollywood Portfolio. http://www.vanityfair.com/hollywood/photos/2014/03/chuck-close-hollywoodportfolio#6. Accessed 10. July. 2014.Google Scholar
- Vezhnevets V, Sazonov V, Andreeva A: A Survey on Pixel-Based Skin Color Detection Techniques (13th international conference on computer graphics (GraphiCon 2003)). Moscow State University, Moscow, Russian Federation; 2003.Google Scholar
- Yang G, Ren J, Huang C, Li H: Research on a Skin Color Detection Algorithm Based on Hybrid Color Model (3rd international congress on image signal processing (CISP 2010)). Yantai University, Yantai, China; 2010.Google Scholar
- Yang L, Li H, Wu XY, Zhao D: An Algorithm of Skin Detection Based on Texture (IEEE international congress on image and signal processing (CISP 2011)). Donghua University, Shanghai, China; 2011.Google Scholar
- Yu YH, Kwok NM, Ha QP: Color tracking for multiple robot control using a system-on-programmable-chip. Autom Constr 2011, 20: 669–676. 10.1016/j.autcon.2011.04.013View ArticleGoogle Scholar
- Yu YH, Yeh CH, Lee TT, Chen PY, Shiau YH: Chip-based Real-time Gesture Tracking for Construction Robot’s Guidance (31st international symposium on automation and robotics in construction and mining (ISARC 2014)). The University of Technology Sydney, Sydney, Australia; 2014.Google Scholar
- Zafarifar B, van den Kerkhof T, de With PHN: Texture-adaptive skin detection for TV and its real-time implementation on DSP and FPGA. IEEE Trans Consum Electron 2012, 58(1):161–169. 10.1109/TCE.2012.6170069View ArticleGoogle Scholar
- Zheng Y, Yang G, Wu L: Fast Face Detection in Field Programmable Gate Array (International conference on digital manufacturing and automation (ICDMA 2010)). Institute of Electrical and Electronics Engineers Incorporated, ChangSha, China; 2010.Google Scholar
- Zhu JQ, Cai CH: Region Growing Based High Brightness Skin Detection (10th international symposium on signals, circuits, and systems (ISSCS 2011)). Gheorghe Asachi Technical University, Lasi, Romania; 2011.Google Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited.