Modern astronomy has evolved into a highly computational scenario, as a result of the volume and complexity of the data that must be processed and analyzed. This has motivated the application of advanced computer vision techniques such as Deep Learning, which have shown outstanding results in several applications. However, the implementation of these models usually lack a key factor that impedes the incorporation of such models into complex scientific problems that demand large processing times: consideration of speed of inference as a constraint of the problem. To solve this problem, several authors have presented proposals that incorporate tensor layers into the architecture of CNN networks, showing satisfactory results. Following this rationale, we propose Tensor Mask R-CNN, a combination of a state-of-the-art regional convolutional neural network and tensor methods. This novel approach is based on the incorporation of Tensor Layers (T-L), that is, applying tensor decomposition over fully connected layers in the backbone of a Mask R-CNN network. These tensor layers provide more powerful and cheaper parameterizations for fully connected layers. This results in compressed parameter spaces that lead to a reduction in computational costs and, consequently, a reduction in inference time. As a test of our proposal, we apply Tensor Mask R-CNN to the classification of galaxies based on their morphology, where we validate the increase in inference speed, and reduced costs therefore, against the same architecture without tensor layers.