In this paper, we proposed a hybrid Bayesian estimation framework to deal with the patch similarity for predicting the gender from the facial images. We used Active Appearance Model (AAM) to align the face image in advance. Images are modeled by the patches around the coordinates of the landmark points. In the training phase, these feature patches are approximated by a pre-trained library. In the inference phase, the choice of feature patch determines the classification decision. We also illustrated a hybrid Bayesian framework to marginalize over the feature patches, and determine the classification decision. A library-image selection manner based on the K-means clustering is introduced.