The quick deployment of cloud with computing platforms has driven novel tendencies which shifted operations of networks. However, the cloud is facing several security issues and is susceptible because of suspicious tasks and attacks. This paper devises a new method to detect malicious activities in cloud. Here, first step is the simulation of cloud patterns, wherein the data outsourced by the users are utilized for detecting malicious behaviors. The data pre-processing is done to eradicate unnecessary data and noise contained in the data and is performed using a min–max normalization process. The selection of imperative features is done using distance measure, namely Hellinger distance for mining the essential features. The augmentation of data is performed to make the data appropriate for improved processing. The malicious behavior detection is performed by exploiting the Deep Q network wherein training is performed with Autoregressive chimp optimization algorithm (AChOA), which is developed by integrating chimp optimization algorithm (ChOA) and Conditional Autoregressive Value at risk (CAViaR). The proposed AChOA-based Deep Q network outperformed with the highest testing accuracy of 94%, sensitivity of 94.1%, and specificity of 91.4%. © 2022, The Author(s), under exclusive licence to Springer Nature Singapore Pte Ltd.