Too Big to Know? So What?

The other day someone pointed out an article for me by David Weinberger which appeared in The Atlantic, plugging his new book Too Big to Know. It was a strangely breathless article, but I'm not sure that Weinberger's point is a very interesting one. Essentially, he seems to be saying that we have been developing data sets that are so vast they are beyond what people - with our limited brains and short life-spans - can possibly analyse and understand without the aid of computers. What's more, when we let our computers loose on these huge data sets, the derivation of any new knowledge they come up with (usually by running computer models or simulations on the data) is not accessible to us. To understand why the weather bureau's model predicts a 40% chance of rain in South-East England today would take teams of humans years to calculate by hand using the same rules and the same data. To all intents and purposes, these models might as well be black boxes. Worse than this, there are programs that can derive new mathematical and physical laws and relationships - new knowledge whose creation is forever shrouded in mystery because the cost of understanding how it was discovered is astronomically high for unaided humans. 

While some gasp at the epistemological implications right along with Weinberger, my own reaction is one of puzzlement. After all, isn't this and hasn't it always been how nearly all knowledge comes to us?

At one level, we have been hugely successful at seeing the underlying patterns of the world, without the aid of computers - General and Special Relativity, and the Standard Model reduce everything to a handful of very simple equations and constants. On another level, we have the complexity of chemistry, biology, fluid dynamics, and so on, which, while following the rules of basic physics, have lots of interacting parts that we can only model statistically. That the predictions of some of these statistical models can only be made if they are based on huge data sets and worked out by big, fast computers, doesn't fill me with the same kind of trepidation with which it seems to fill Weinberger. Even the fact that the predictions of many such models are acutely sensitive to their initial conditions is hardly cause for concern (unless you're planning a camping trip and need an accurate weather report).

It's true that the models themselves can be generated or learned by second-order systems and that we do not necessarily have any meaningful way of knowing how they work (something researchers in neural networks have been grappling with for several decades already). But that isn't a particular cause for anxiety either. Such models are in principle analysable, if we should ever want to do that (which, I suggest, removes any hint of scariness). Generally, it is not important to know how a model does its job, as long as we are confident about how it was constructed and have verified its behaviour against the data. If we want to verify their outputs or increase our confidence in what the models are doing, we can always set up a second, third, or nth model to cross-check the first or poll their outputs. (Weather models tend to be run over and over and their results combined, for instance.) The thing is, these models tend to be of systems that do not have the scientific significance of E=mc^2, although they may have ample practical significance for campers and drug companies. 

I used to work in artificial intelligence and I made a particular study of a field called "argumentation". It's all about how and why we find arguments convincing - like a modern Rhetoric. The early pioneers of expert systems understood well the issue this article raises. If you run a system with more than a handful of rules, it quickly gets to the point where you can no longer understand or predict the outputs - not without a disproportionately huge effort to delve into the workings. So they tried to devise schemes for expert system to explain their own reasoning (which boiled down to traces of rule activation - presented with varying degrees of clarity). I worked on some particularly massive rule-based systems and I can attest to the fact that presenting a conclusion is not enough - people need more - but presenting the machine's reasoning is a very difficult task.

However, I believe it is doable. People just haven't put much work into it yet.

There is an example of a massively parallel supercomputer of immense power but which is virtually a black box as far as the question of how it reached its outputs are concerned. In fact there are many such examples. One of the best was Albert Einstein. This processor came up with some astonishing physical laws by a process that nobody understands even a hundred years after they were derived. However, the Einstein processor was able to explain its reasoning and the derivation of its laws in a way that satisfied everyone who was able to understand (and the rest of us have been happy to take their word for it). I've read a few books on relativity now and I must say, the fact that I don't know how Einstein derived it from what was known at the time doesn't bother me at all. The rules are logical, consistent, match the evidence, and (having been shown the way) are derivable by other, similarly-endowed processors.

Certainly machine-derived knowledge raises questions in epistemology and ontology. Does it mean something different to know a physical law derived by a machine, for example, especially where the reasoning processes involved are hidden from us? My argument, already stated, is that it doesn't. In fact, it is similar in all important ways.

The question of how much we can trust machine-derived knowledge is a different kettle of fish. Here, I'm happy to use all the usual methods of improving my confidence - particularly the Gold Standard, empirical testing.

There is a possibility that knowledge could be derived by machine that is so far beyond our human understanding that only other machines of similar capabilities could peer-review it, or understand how to devise and run empirical tests, or interpret the results of such tests. When that day comes, we are put in the position that most of us are in now vis-a-vis the great scientists. Bohr, Pauli, Heisenberg, Einstein and the rest may have been able to understand and devise tests for each others' discoveries in quantum mechanics, but most of humanity could not. To us, it is entirely a matter of trust - even of faith.

It's like the climate change "debate". Here the science is simple enough that any intelligent layman can follow it. You could replicate John Tyndall's experiment that first measured the greenhouse effect in your own kitchen with some pipes and jars and rubber tubing. Yet still, for the majority (especially GOP politicians), it is all incomprehensible scientific mumbo jumbo. They can't grasp the principles. They can't separate out the scientific claims from the denialist obfuscation. They are so far from being able to judge the validity of what the scientific community is telling them, that they can have no confidence in what they are hearing - leaving them free to dismiss it as "scare-mongering" or a left-wing conspiracy to increase government regulation of our lives, or any bizarre rationale they can come up with.

One day, even the brightest of us will probably be in that position, when the machines have taken it all to a new level, way beyond our understanding. Will we look like simple-minded denialists to them when we question what seems to us to be the unfounded gibberish they will be spouting? I think so. Or maybe, being so much cleverer than us, the machines will be rather better than human scientists at explaining what they mean and why they're saying such outrageous things.  

I Know What is Wrong With The World - and there is no way to fix it

I can sum up everything that is wrong with the world in three words:

People Are Stupid.

We like to think we are the pinnacle of evolution (in itself a stupid misconception of how evolution works) and that our vast intelligence separates up from the animals (sorry, stupid mistake, the other animals, I mean), but the fact is that we're not all that bright. We have a few advantages over, say primates, language and better memories for example, but research suggests that when it comes to sheer reasoning ability, we're not all that much brighter than chimps. I don't want to get into an IQ debate here, but let's assume there's some correlation between general intelligence and IQ. The average is around 100 (it varies from group to group and culture to culture - mainly because we're too stupid to devise a sensible test where the average is always 100 for every place and time).

Give or take a couple of standard deviations, most of us - like 96% of us - have that IQ. And it's abysmally low. It's the IQ of the kind of person who reads Murdoch newspapers, the IQ of the kind of person who watches soaps (even if it's the slick US cop show or medical show type and you think it's somehow better than Home and Away), and the kind of person who believes in the supernatural ("well, there has to be something more than this, doesn't there, science can't explain everything").

If you're still reading, it probably means you think you're not one of the stupid people I'm talking about. Well, you're wrong. Here's a little test to show just how stupid you are. 

• Q1 Can you solve world poverty? 
• Q2 Can you stop war? 
• Q3 Can you stop the persecution of minorities?
• Q4 Can you devise an economic system that treats everybody fairly?

The answers to all those questions are "No". I can think of dozens, probably hundreds of other questions that you would have to say no to, too. The fact is, we are all, even the very brightest among us, deeply and unutterably stupid. We can't solve the world's problems because we're too thick. We've been trying throughout recorded history (and presumably long before then) and we have failed. Failed dismally. Failed in a way that should be excruciatingly embarrassing to all of us. Let's face it, we're a bunch of chimps with cars and cell phones and we haven't got a clue.

And that's why there is no way to fix the world; we just haven't got the brains. We might as well give up, go back to the trees and scratch our arses until we're extinct.

Oh, hang on, we can't do that, can we? We stupidly cut down all the trees.

Now how did he get here?

    

Hope For Intelligent Kids Who Are Unhappy

Nearly three years ago, I wrote a post called "Why Ordinary People Make Intelligent Kids Unhappy". It was immediately, and still is, one of my most popular posts. It is clearly an issue that concerns many people. Yet, looking back, I see it is a post in desperate need of a follow-up. The original post merely analyses the problem and offers no solace, and certainly no solutions. Most likely, many of the people who read the post felt worse after reading it rather than better. That's OK, I suppose. The world isn't here to make us feel better. It's just a place we need to cope with, and understanding what is going on in the world can only help us cope better. However, there are some things I could say that might make some people feel better, and it's about time I said them.

So here they are:

1. There is hope. I was a bright child. I was rejected to greater and lesser degrees by my family, my schoolmates and my teachers. Being clever doesn't win you many friends. Sometimes none at all. I grew up in a working-class city in the North of England. The people around me were poor, ill-educated and, almost without exception, ignorant. Most of them were also very stupid. Yet I found a way through. I was certainly luckier than many - my mother was bright and supportive, and I got a free, university-level education. I left the place I grew up in and went in search of better places. Eventually I found them. It's a very big world and there are many, many niches in it. Keep looking and you may find yours. It helps the search if you move to a major city.

2. Other bright people can sustain you, even if you never meet them. I'm not just talking about the Internet, here, although it's an obvious place to look for like-minded people. I didn't have the Internet when I grew up, but I had books. Read widely and read good stuff (on- and off-line). You will find that many of the people who became great writers also went through what we did. One of the best moments in my reading life was when I discovered J.D. Sallinger. In his short stories in particular, I often got that heart-stopping moment of recognition when I realised that this man knew my pain. Maybe Sallinger will do it for you too. Most likely it will be someone else. Just one word of advice - especially about the Internet. While it is easy to find fellow sufferers, and wallowing in misery together can be a relief for a while, in the end, you will get more of a lift out of positive, strong people. However bright you are, you're only human and you have the same psychology as we all do. Don't get locked in a downward spiral of self-pity with someone else. You'd both be better off on your own.

3. Work can help. Clever people tend to be good at certain things. They make good scientists, engineers, writers, and so on. Even in less intellectually challenging jobs - as administrators, planners, managers, etc. - they tend to shine. They might not get the promotions, they might not get the big bucks - for that you also need social skills which cleverness does not guarantee - but they do their jobs so well that they earn the respect of their peers. Respect isn't love, it isn't necessarily acceptance, it isn't even kindness, but it's something and it is not to be sniffed at. Respect from others helps you respect yourself - and self-respect helps in many different ways.

4. Don't worry about the meaning of life. There is none. Bright people are their own worst enemy when it comes to seeing through the crap. Sooner or later, you will conclude that there is no god, there is no deep meaning to the Universe, you have no destiny, and, in fact, there is no point to anything at all. That's fine, but you shouldn't let it worry you.

'Purpose', 'meaning', 'point' and so on are ideas that people come up with , they are not things we find in nature. The 'purpose' of the rain might be to make the crops grow, but we all know that is just a semantic confusion. The physical word doesn't have purposes, only people do. In the long term - the next ten billion years, say - nothing about humanity matters at all, not least your own little wants and needs, your hopes and ambitions, your loves and hates. However, we don't live in the long term. We don't live ten billion years. We live tiny, proscribed little lives. We flicker into self-awareness and are gone in a moment.

Yet, to us, in that moment, our own feelings, desires, and purposes are everything to us. And that is important - by definition. We are the creatures who give meaning to the world. We are the ones who provide purpose to the Universe. We are the ones that imbue existence with value. While we live, while we think and feel, we bring this into reality. You and I create the meaning of the Universe, quite literally. It is ours.

So don't feel shy about the purpose of your life. If you want an iPod, if you love the boy or girl next-door, if you have a craving for a swim, or to work in outer space, each of these is, in a very real sense, the highest purpose in the world - because it's yours, right now, and that, literally, is what matters in this otherwise indifferent Universe.

5. Find out who you are and accept it. The biggest advantage of being clever isn't that you can make money, or design cool stuff, or argue everyone else under the table, it is this: you can understand yourself and the people around you. If you don't understand yourself, you will always be doing stupid things that don't make you happy. If you don't understand other people you cannot love them and you will always be doing stupid things that don't make them happy either. It took me a couple of decades of very hard work to get a deep and thorough understanding of myself and to accept who I am, warts and all. It was the most difficult intellectual challenge I have ever faced - the most difficult emotional challenge too - but it was worth it. Well worth it. It requires strict intellectual rigour. It requires ruthless, painful honesty. It may require you to throw out many myths about yourself and your world that you cherish and hide behind. Don't waste that glorious brain of yours. It's caused you a lot of pain and heartache, set you apart, driven people away. Now, for once, get some good out of it. Use it for something that will really benefit you and everyone around you.

6. Never forget what you are. You are a human being. You evolved from ape-like creatures, who evolved from other creatures. As clever as you are, you are still an animal. You have the physiology of an animal and, importantly, the psychology of an animal. The kind of animal you are has psychological needs for the company and intimacy of its fellow animals. You can't fight your own psychology so try not to. Being cut off from the society of people is what is making you feel bad. Going along with that and cutting yourself off even more will only make you feel worse. The smart thing to do is to understand your animal nature and to start organising your life so that its needs are satisfied. I'm not talking about sex and eating and sleep and all those other 'drives' - although they are important - I'm talking about social interactions, social approval, gossiping, sharing rituals, and finding friends. Right now, those things may seem a million miles away from where you are - but that's what all the points above are about, getting yourself into a societal niche where you fit, finding people who like having you around, ditching false notions that will add to your troubles, and becoming so comfortable inside your own skin that you can face the world on equal terms and get what you need from it.

It will always be the case that you are in a minority. Always. But you don't need six billion people to accept you. You can make a great life with just a handful of close friends and family who see the way you are as a desirable quality, not a freakish aberration.