Pain assessment is vital for effective pain management in clinical settings. It is generally obtained via patient's self-report or observer's assessment. Both of these approaches suffer from several drawbacks such as unavailability of self-report, idiosyncratic use and observer bias. This work aims at developing automated machine learning based approaches for estimating pain in clinical settings. We propose to use facial expression information to accomplish current goals since previous studies have demonstrated consistency between facial behavior and experienced pain. Moreover, with recent advances in computer vision it is possible to design algorithms for identifying spontaneous expressions such as pain in more naturalistic conditions. Our focus is towards designing robust computer vision models for estimating pain in videos containing patient's facial behavior. In this regard we discuss different research problem, technical approaches and challenges that needs to be addressed. In this work we particularly highlight the problem of predicting self-report measures of pain intensity since this problem is not only more challenging but also received less attention. We also discuss our efforts towards collecting an in-situ pediatric pain dataset for validating these approaches. We conclude the paper by presenting some results on both UNBC Mc-Master Pain dataset and pediatric pain dataset.