In this paper, an optical cryptosystem with adaptive steganography is proposed for color image encryption and decryption. The optical cryptosystem employs a double random phase encoding algorithm to encrypt and decrypt color images. The color image is first separated into three channels: red, green, and blue. Each channel is encrypted by two random phase masks generated from session keys. For higher security, an asymmetric method is applied to cipher these session keys. The ciphered data produced by the asymmetric method is then embedded into the encrypted color image by a content-dependent and low distortion data embedding technique. The key delivery is accomplished by hiding ciphered data into the encrypted color image with a specific hiding sequence generated by the zero-LSB sorting technique. Experimental results show that the proposed adaptive steganography-based cryptosystem has a good performance when it is applied to color images.