We construct a family of Neural Networks that approximate matrix multiplication operator for any activation function such that there exists a Neural Network which can approximate the scalar multiplication function. In particular, we use the Strassen algorithm to bound the number of weights and layers needed for such Neural Networks. This allows us to define another Neural Network for approximating the inverse matrix operator. Finally, we discuss how these results can be used for numerically solving parametric elliptic PDEs.