This study presents an innovative approach that harnesses the power of the Geometry-Oblivious Fast Multipole Method (GOFMM) to compute and approximate kernel matrices derived from Convolutional-Neural-Network-equivalent Gaussian Processes (CNN-GPs). The primary objective is to devise an efficient and scalable linear solver specifically tailored to handle the intricacies associated with kernel matrix approximations within the domain of CNN-GPs. By leveraging hierarchical decomposition techniques, particularly the GOFMM approach, this research aims to significantly enhance both the computational efficiency and the accuracy in approximating kernel matrices in the context of CNN-GPs. The findings from extensive experiments underscore the accuracy of the proposed methodology, showcasing substantial improvements with complex CNN-GP architectures. Additionally, the scalability analysis demonstrates the method's robustness in handling various problem sizes, highlighting its potential versatility and applicability across a myriad of domains. The results of this study offer a promising avenue for enhancing the accuracy and computational efficiency of kernel matrix approximation, particularly in CNN-GP context, thereby facilitating advancements in various real-world applications demanding efficient processing of large-scale data.