Sign language detection and recognition (SLDR) using computer vision is a very challenging task. In respect to Bangladesh, sign language users are around 2.4 million . In this paper, we try to focus for communicating with those users by computer vision. In this respect, an efficient method is propose consists of two basic steps: (a) refinement and (b) recognition. Initially in refinement, a Red-Green-Blue (RGB) color model is adopted to select heuristically threshold value for detecting candidate regions (i.e. hand and wrist band sign regions). After the candidate regions are obtained by applying color segmentation, then procedures for refining the candidate region are followed by using two different color wrist band regions and filtering. Finally, statistically based template matching technique is used for recognition of hand sign regions. Various hand sign images are used to test the proposed method and results are presented to provide its effectiveness.