• DOI:10.48550/arXiv.2207.12396
• Corpus ID: 251040466

Exploring CLIP for Assessing the Look and Feel of Images

@article{Wang2022ExploringCF,
  title={Exploring CLIP for Assessing the Look and Feel of Images},
  author={Jianyi Wang and Kelvin C. K. Chan and Chen Change Loy},
  journal={ArXiv},
  year={2022},
  volume={abs/2207.12396}
}

• Jianyi Wang, Kelvin C. K. Chan, Chen Change Loy
• Published 25 July 2022
• Computer Science
• ArXiv

. Measuring the perception of visual content is a long-standing problem in computer vision. Many mathematical models have been devel-oped to evaluate the look or quality of an image. Despite the effectiveness of such tools in quantifying degradations such as noise and blurriness lev-els, such quantification is loosely coupled with human language. When it comes to more abstract perception about the feel of visual content, existing methods can only rely on supervised models that are explicitly trained with labeled data collected via laborious user study. In this paper, we go beyond the conventional paradigms by exploring the rich visual language prior encapsulated in Contrastive Language-Image Pre-training (CLIP) models for assessing both the quality perception ( look ) and abstract perception ( feel ) of images in a zero-shot manner. In particular, we discuss effective prompt designs and show an effective prompt pairing strategy to harness the prior. We also provide extensive experiments on controlled datasets and Image Quality Assessment (IQA) benchmarks. Our results show that CLIP captures meaningful priors that general-ize well to different perceptual assessments. Code will be avaliable at https://github.com/IceClear/CLIP-IQA . Abstract perception. Abstract perception is a subjective feeling and is not quantifiable. Existing methods for abstract perception assessment mainly focus on aesthetic image assessment [18,26,35,48,50,63] and image emotion analysis [1,5,13,25,37,61,64]. The majority of these methods require human annotations for training, which are laborious to obtain. Therefore, in this work, we explore the possibility of exploiting vision-language priors captured in CLIP to bypass the quality labeling process. Our results show that our CLIP-IQA is able to perceive abstract aspects of an image, even without training with annotations. approaches. Benefiting from the large-scale visual-language shown impressive capability and generalizability on a wide range of tasks, such as image manipulation image cap-tioning view synthesis 49, and semantic These applications mainly focus on building the semantic relationship between and and hence they suffer less linguistic ambiguity. from focus on the effectiveness understand both quality and abstract perceptions of an image. Specifically, our CLIP-IQA with paired prompts and the removal of positional embedding achieves comparable performance to learning-based methods, which require careful designs in network architecture and extensive task-specific training. Our findings provide a solid ground for CLIP-based quality assessment, and more generally perception assessment. We believe our studies and discussion could motivate future development in various domains, such as sophisticated prompts, better generalizability, and effective adoption of CLIP prior.