Modern AI systems are often based on powerful machine-learning (ML) techniques; as a result, they can often make highly accurate predictions but are usually hardly intelligible, even to their developers. This opacity presents a major challenge for deployers, regulators and users of alike, specifically with respect to questions of safety, reliability and trustworthiness of AI systems. This paper discusses how AI systems’ opacity might best be addressed. I shall argue that addressing opacity requires the collaboration of different domain experts engaging in coordinated epistemic activities and operations. By applying systematic discovery strategies familiar from the life sciences, domain experts can work towards uncovering the systems’ overall functional architecture and thus render opaque systems mechanistically interpretable. Once a system is mechanistically interpretable, it may be re-engineered to build a transparent system that is arguably safer, does not utilize unfair associations and is less likely to suffer from unexpected failures. Thus, despite all the issues that black box systems raise, they can be a stepping stone for the development of transparent ones.