Color visual cryptography scheme is useful in systems that need access control management and authentication like digital signature schemes used in banks that demand multi partner shares. In one of the basic visual cryptography schemes for color images, proposed by Hou et. al. it is claimed that an adversary cannot attain any knowledge about the secret image without the black mask. While by security analysis of this scheme we observe that having three color shares and stacking them, an adversary is able to distinguish color boundaries of the image and by considering the available colors of the image he can reach to the main image with a non negligible probability. In this paper, we utilize Arnold’s cat matrix to enhance the security of this scheme by means of pixel scrambling. In the proposed scheme, we mislead the adversary who stacks three color shares of the image to reconstruct the main image. Therefore, without a meaningful loss in the speed of encryption procedure, the security of Hou scheme is increased. The simulation results justify our claim.