Human behavior understanding is a fundamental problem in many ubiquitous applications. It aims to automatically uncover and quantify characteristic behavior patterns in users' daily lives as well as disclose behavior clustering structure among multiple users. The key challenge is how to define a naturally interpreted representation for users' daily behavior patterns, which can be easily exploited to not only uncover the behavior similarity among multiple users but also predict users' future activities. In this paper, we define such a representation, and propose a probabilistic framework which can automatically learn it from mass amount of mobile data in unsupervised setting and exploit it to predict user activities. By an appropriate information sharing among multiple users, this framework overcomes single-user data sparsity problem and effectively identifies behavior clustering structures in a set of users. Experiments conducted on a public reality mining data set demonstrate the effectiveness and accuracy of our methods.