Enhancing In-loop Filter of HEVC with Integrated Residual Encoder-Decoder Network and Convolutional Neural Network

High Efficiency Video Coding (HEVC) often known as H.265 is a video compression method that outperforms its predecessor H.264. In HEVC, an in-loop filter is an additional processing step that removes compressing artifacts from decoding video frames while improving visual quality. This research article proposes an improved in-loop filter that incorporates a Residual Encoder-Decoder Network based Deblocking Filter (REDNetDF) and a Convolutional Neural Network based Sample Adaptive Offset (CNN-SAO) filter, which together eliminates the smallest range of artifacts in compression video frames. The quantization frame is subjected to REDNetDF, which removes a minute number of blocking artifacts from the compressed frame. To eliminate the ringing artifacts in the compressed frame, CNN-SAO filter is used. The proposed method is used to evaluate the publicly available UVG dataset. To demonstrate efficiency, the new model is evaluated using a variety of metrics. The outcome of this study provides better results like PSNR of 49.7 dB and the SSIM of 0.97 in comparison with other techniques. Besides, the model's outcome indicates an MSE of 1.8 and saves 24.9% more bits on average to provide the same level of quality as previous techniques. The proposed framework also suppresses time complexities regarding encoding and decoding times with the results of 90.5 and 4.5 seconds on average correspondingly.