Gender and face recognition are one of the important challenge in various areas of security and surveillance. Joint Gender and face recognition system is devoted to the development of an automatic system capable to recognizing the faces using the facial and attribute features and to distinguish peoples gender by analyzing their faces on digital images using Discrete Cosine Transform (DCT) and Principal Component Analysis (PCA) features. Such systems can find application in different fields, such as robotics, human computer interaction, demographic data collection, video surveillance, online audience measurement for digital signage networks and many others. RGB-D image is used for face recognition as it contains more information than 2D image thus improves the accuracy. RISE algorithm is used to compute a descriptor from the facial image based on the entropy and saliency features. Geometric facial attributes are extracted from the color image then both the descriptor and attribute match scores are fused for face recognition. In order to increase the security of the system proposed a method to perform gender recognition after performing face recognition. Naive Bayes classifier is used to recognize the HOG features for gender recognition. The experimental results indicate that the gender recognition using DCT achieves higher accuracy on color images when compared with gender recognition using other methods.