Too Big to Know? So What?

The other day someone pointed out an article for me by David Weinberger which appeared in The Atlantic, plugging his new book Too Big to Know. It was a strangely breathless article, but I'm not sure that Weinberger's point is a very interesting one. Essentially, he seems to be saying that we have been developing data sets that are so vast they are beyond what people - with our limited brains and short life-spans - can possibly analyse and understand without the aid of computers. What's more, when we let our computers loose on these huge data sets, the derivation of any new knowledge they come up with (usually by running computer models or simulations on the data) is not accessible to us. To understand why the weather bureau's model predicts a 40% chance of rain in South-East England today would take teams of humans years to calculate by hand using the same rules and the same data. To all intents and purposes, these models might as well be black boxes. Worse than this, there are programs that can derive new mathematical and physical laws and relationships - new knowledge whose creation is forever shrouded in mystery because the cost of understanding how it was discovered is astronomically high for unaided humans. 

While some gasp at the epistemological implications right along with Weinberger, my own reaction is one of puzzlement. After all, isn't this and hasn't it always been how nearly all knowledge comes to us?

At one level, we have been hugely successful at seeing the underlying patterns of the world, without the aid of computers - General and Special Relativity, and the Standard Model reduce everything to a handful of very simple equations and constants. On another level, we have the complexity of chemistry, biology, fluid dynamics, and so on, which, while following the rules of basic physics, have lots of interacting parts that we can only model statistically. That the predictions of some of these statistical models can only be made if they are based on huge data sets and worked out by big, fast computers, doesn't fill me with the same kind of trepidation with which it seems to fill Weinberger. Even the fact that the predictions of many such models are acutely sensitive to their initial conditions is hardly cause for concern (unless you're planning a camping trip and need an accurate weather report).

It's true that the models themselves can be generated or learned by second-order systems and that we do not necessarily have any meaningful way of knowing how they work (something researchers in neural networks have been grappling with for several decades already). But that isn't a particular cause for anxiety either. Such models are in principle analysable, if we should ever want to do that (which, I suggest, removes any hint of scariness). Generally, it is not important to know how a model does its job, as long as we are confident about how it was constructed and have verified its behaviour against the data. If we want to verify their outputs or increase our confidence in what the models are doing, we can always set up a second, third, or nth model to cross-check the first or poll their outputs. (Weather models tend to be run over and over and their results combined, for instance.) The thing is, these models tend to be of systems that do not have the scientific significance of E=mc^2, although they may have ample practical significance for campers and drug companies. 

I used to work in artificial intelligence and I made a particular study of a field called "argumentation". It's all about how and why we find arguments convincing - like a modern Rhetoric. The early pioneers of expert systems understood well the issue this article raises. If you run a system with more than a handful of rules, it quickly gets to the point where you can no longer understand or predict the outputs - not without a disproportionately huge effort to delve into the workings. So they tried to devise schemes for expert system to explain their own reasoning (which boiled down to traces of rule activation - presented with varying degrees of clarity). I worked on some particularly massive rule-based systems and I can attest to the fact that presenting a conclusion is not enough - people need more - but presenting the machine's reasoning is a very difficult task.

However, I believe it is doable. People just haven't put much work into it yet.

There is an example of a massively parallel supercomputer of immense power but which is virtually a black box as far as the question of how it reached its outputs are concerned. In fact there are many such examples. One of the best was Albert Einstein. This processor came up with some astonishing physical laws by a process that nobody understands even a hundred years after they were derived. However, the Einstein processor was able to explain its reasoning and the derivation of its laws in a way that satisfied everyone who was able to understand (and the rest of us have been happy to take their word for it). I've read a few books on relativity now and I must say, the fact that I don't know how Einstein derived it from what was known at the time doesn't bother me at all. The rules are logical, consistent, match the evidence, and (having been shown the way) are derivable by other, similarly-endowed processors.

Certainly machine-derived knowledge raises questions in epistemology and ontology. Does it mean something different to know a physical law derived by a machine, for example, especially where the reasoning processes involved are hidden from us? My argument, already stated, is that it doesn't. In fact, it is similar in all important ways.

The question of how much we can trust machine-derived knowledge is a different kettle of fish. Here, I'm happy to use all the usual methods of improving my confidence - particularly the Gold Standard, empirical testing.

There is a possibility that knowledge could be derived by machine that is so far beyond our human understanding that only other machines of similar capabilities could peer-review it, or understand how to devise and run empirical tests, or interpret the results of such tests. When that day comes, we are put in the position that most of us are in now vis-a-vis the great scientists. Bohr, Pauli, Heisenberg, Einstein and the rest may have been able to understand and devise tests for each others' discoveries in quantum mechanics, but most of humanity could not. To us, it is entirely a matter of trust - even of faith.

It's like the climate change "debate". Here the science is simple enough that any intelligent layman can follow it. You could replicate John Tyndall's experiment that first measured the greenhouse effect in your own kitchen with some pipes and jars and rubber tubing. Yet still, for the majority (especially GOP politicians), it is all incomprehensible scientific mumbo jumbo. They can't grasp the principles. They can't separate out the scientific claims from the denialist obfuscation. They are so far from being able to judge the validity of what the scientific community is telling them, that they can have no confidence in what they are hearing - leaving them free to dismiss it as "scare-mongering" or a left-wing conspiracy to increase government regulation of our lives, or any bizarre rationale they can come up with.

One day, even the brightest of us will probably be in that position, when the machines have taken it all to a new level, way beyond our understanding. Will we look like simple-minded denialists to them when we question what seems to us to be the unfounded gibberish they will be spouting? I think so. Or maybe, being so much cleverer than us, the machines will be rather better than human scientists at explaining what they mean and why they're saying such outrageous things.