Neural Networks are finding increasing use in many areas of astronomy, but often act as "black boxes". Many techniques exist to probe in the internals of neural networks but not all are relevant to scientists. In this talk I discuss some of the techniques developed in computer vision to investigate what neural networks are learning, and investigate some of their benefits and problems when applied to astronomy. I introduce a simple technique and software package, 'Sensie', to probe what neural networks have learned. I apply it to networks trained to find strong gravitational lenses in the Dark Energy Survey and draw some lessons that may help future searches.