Aspirin was discovered in 1897, and an explanation of how it works followed in 1995. That, in turn, has encouraged some research leads on making better pain relievers through something other than trial and error. This kind of discovery—answers first, explanations later—is called "intellectual debt ". We gain understanding of what works without knowing why it works. We can put that understanding to use immediately, and then tell ourselves we'll figure out the details later. Sometimes we pay off the debt quickly; sometimes, as with aspirin, it takes a century; and sometimes we never pay it off at all.
In the Age of Intelligence, while machine learning presents lots of problems and gets things wrong, at least we know enough to be wary of the predictions produced by the system and to argue that they shouldn't be blindly followed: but if a system performs perfectly (and we don't know why), then we come to rely on it and forget about it and suffer consequences when it goes wrong.
It's the difference between knowing your car has faulty brakes and not knowing: both are bad, but if you know there is a problem with your brakes, you can increase your following distance, drive slowly and get to a mechanic as soon as possible. If you don't know, you're likely to find out the hard way, at 80mph on the highway when the car in front of you came to a sudden stop and your brakes give out.
We don't have much by way of solutions. Most important, we shouldn't trick ourselves into thinking that machine learning alone is all that matters. Indeed, without life value, machine learning may not be meaningful answers at all.
