Sometimes I am asked to review or explain a framework of thinking systems in terms of a very popular book by Nobel laureate and father of behavioral economics, Daniel Kahneman.
…human reason left to its own devices is apt to engage in a number of fallacies and systematic errors, so if we want to make better decisions in our personal lives and as a society, we ought to be aware of these biases and seek workarounds.
I suppose this comes up most when I describe the same things in many of my presentations, such as my last one given at RSAC SF:
RSAC SF 2020 Presentation on AI
My point usually has been that veterinarian science used this duality in thinking to solve Rinderpest (as I wrote here in 2010, a year before Kahneman’s very famous best-selling book was published).
And my point in describing the dual-system that solved Rinderpest, for such a huge accomplishment as ending a disease, has been that our security community maybe could do similar things to solve for integrity attacks on information systems. You say we have a problem with disinformation campaigns and I’ll say I have a possible solution!
Kahneman himself just gave a brief three minute presentation the other day in an “AI Debate”. He quickly starts off by admitting
…they’re not my idea but I wrote a book to describe them.
He then goes on to say his understanding of system one is that “things happen to you, you don’t do them”, calling them automatic and parallelized, whereas system two is “something you do” and serialized… all of which seems very consistent with my slides.
Again for clarity:
System 1) things that happen to you
System 2) things that you do
This is not only consistent with what I studied before his book was published, the split is of course NOT my idea either, as I’ve always said.
I have been writing a book to describe them, but it has been for the purposes of improving safety in engineering practices.
What is most interesting in his presentation is while he tells us that system one is “our world” it’s probably more accurate to say (by his own admission) that in system one we are seeing shadows on Plato’s cave wall, not the strings we pull.
…dropped 600K tons of bombs on DPRK and 2 million civilians perished. It had reverse effect of expected and cauterized resistance.
However, one person who definitely remembered was double-agent for the Soviet Union George Blake, one of the most well-known yet least connected stories to such “cauterized resistance”.
…that he decided to switch sides after seeing civilians massacred by the “American military machine.” “I realized back then that such conflicts are deadly dangerous for the entire humankind and made the most important decision in my life – to cooperate with Soviet intelligence voluntarily and for free to help protect peace in the world”.
…despatched to Seoul in 1950, to set up an anti-Soviet operation on Moscow’s eastern flank…the North Koreans invaded the South and Blake, like many other western diplomats, was interned – and during his three-year period of captivity he changed sides. George Blake was no “Manchurian Candidate”, tortured and brainwashed into working for the communists while a prisoner of war. it was, he insisted, the spectacle of a helpless civilian population being attacked by mighty US bombers that had changed his world-view: “It made me feel ashamed of belonging to these overpowering, technically superior countries fighting against what seemed to me quite defenceless people.” He quietly informed his KGB captors that he was ready to work for them. In 1953, Blake and his fellow detainees were at last released and he returned to London as an SIS hero.
The suspected spy was unmasked by a tip from a defecting Polish intelligence officer who told the CIA that two Soviet agents were operating in Britain, one at a royal navy research centre, the other in SIS. They were codenamed Lambda-1 and Lambda-2. Quickly, Lambda-1 was identified as Harry Houghton, but it was months before Blake, then on temporary assignment in Lebanon to learn Arabic, became the prime suspect for Lambda-2.
He confessed and pleaded guilty, was sentenced to a long jail term but soon escaped (with the help of Irish inmates perhaps enamored with Soviet life) from “maximum security” to the open arms of Russia where he continued to intentionally put hundreds of people in harms way.
Dozens are alleged to have been executed in Russia from his actions, and he denied responsibility for their lives while simultaneously taking credit and awards.
Did you know Nazi minister of propaganda Goebbels, one of Hitlers closest men, said “The truth will always win”?
There’s been a problem on the Internet for a long time, as we all know, that data integrity gets ignored by security professionals. Cliff “Cuckoo Egg” Stoll in 1995 infamously warned us about this in “Why the Web Won’t Be Nirvana“, which everyone has basically ignored.
Sure people work on availability (howabout them nines!) and of course after 2003 the boom of documented huge privacy breaches have been lighting up news headlines and even board-level radar screens.
But — and it’s a very BIG but — integrity largely has been ignored.
People now repeatedly and freely post quotes and attributions that simply were never said, or fake pictures that were never taken (as I made light of several times here).
Yet show me a security team prepared and ready to do a correction on data and deal with sources disputing veracity. It was some kind of major problem to get Facebook to post warnings and moderate speech after how many years of obvious safety harms including atrocity crimes?
So what did Goebbels really say?
This is a natural environment for the historian. Which source to trust, what really happened and was said? That’s the heart of the mission for anyone claiming to understand and be able to explain history.
Now bring the typical security professional into such a fray and it’s like having a deer in headlights.
I’ve given talks about this disconnect in our industry for decades now. In several cases I’ve tried to illuminate how easy it is for security professionals to use low integrity themselves while talking about the importance of privacy.
The over-specialization in security actually has led to an even greater problem (e.g. integrity flaw risk increases dramatically as transparency decreases) few are willing to talk about either.
If you hear a CISO press 100% into encryption and not at all into issues of keeping data safe behind a lock and key, where they throw away the key, hold up one minute and think about what you’re doing.
Anyway, one good example is how Goebbels somehow has been attributed with saying “Truth is the enemy of the state” when in fact he said the opposite. No, seriously, Goebbels was a huge proponent of telling the truth.
Goebbels’ moral position in the diary was straightforward: he told the truth, his enemies told lies. Actually the question for him was one of expediency and not morality. Truth, he thought, should be used as frequently as possible; otherwise the enemy or the facts themselves might expose falsehood, and the credibility of his own output would suffer. Germans, he also stated, had grown more sophisticated since 1914: they could “read between the lines” and hence could not be easily deceived.
Thus we can easily see Goebbels’ actual words in 1941 were that truth wins and the use of lies — such as what he observed the Allies to use — are stupid and will lose:
The astonishing thing is that Mr. Churchill, a genuine John Bull, holds to his lies, and in fact repeats them until he himself believes them.
If you tell a lie big enough and keep repeating it, people will eventually come to believe it. The lie can be maintained only for such time as the State can shield the people from the political, economic and or military consequences of the lie. It thus becomes vitally important for the State to use all of its powers to repress dissent, for the truth is the mortal enemy of the lie, and thus by extension, the truth is the greatest enemy of the State.
Goebbels never said THAT.
What Goebbels believed in, just to be clear, is “the ultimate victory of the truth”, explained by German professor of history Peter Longerich in a 2014 biography.
Source: “Goebbels : a biography” by Peter Longerich, New York: Random House, 2014.
Goebbels said THAT, and good luck getting take downs or corrections filed on all the pages to correct the record. Will the truth really win?
And speaking of Internet activism, guess who has been spreading Goebbels’ saying that truth will always win?
Yup. WikiLeaks has a Nazi propaganda minister reference as their byline. Ok, to be fair, a lot of people say this across the spectrum. Just imagine for a minute that Goebbels’ saying was correctly cited and known.
I mean imagine a future world (it may in fact be coming soon) where security professionals are working on how best to wade into this problem of integrity flaws. Too many have been acting for too long like the risk of Nazis deploying harms on every available platform is some kind of new thing or outside their expertise or domain…
Hitler was photographed with his Minister of Propaganda, Joseph Goebbels, and yet someone painstakingly removed the latter from the image.
If a lie is only printed often enough, it becomes a quasi-truth, and if such a truth is repeated often enough, it becomes an article of belief, a dogma, and men will die for it.
A portmanteau of “binary digit,” a bit could be either a 1 or a 0, and Shannon’s paper is the first to use the word (though he said the mathematician John Tukey used it in a memo first).
Shannon clearly is reporting working around others and sharing attribution. However, the author of the article starts it off by rather ironically making a narrative of wide communication about a single person:
Mathematics searches for new theorems to build upon the old. Engineering builds systems to solve human needs. The three disciplines are interdependent but distinct. Very rarely does one individual simultaneously make central contributions to all three — but Claude Shannon was a rare individual. …more than 70 years ago, in a single groundbreaking paper, he laid the foundation for the entire communication infrastructure underlying the modern information age.
It reads to me as though the person trying to get us to celebrate importance of communication links being simplified and standardized (to bridge any and all individuals together) at the same time is trying to create a super-human myth.
Was Shannon rare, or was he just the natural progression in an old and well-known theory that groups achieve more by working together and being humble about the steps made?
Take for example this analysis:
His theorems led to some counterintuitive conclusions. Suppose you are talking in a very noisy place. What’s the best way of making sure your message gets through? Maybe repeating it many times? That’s certainly anyone’s first instinct in a loud restaurant, but it turns out that’s not very efficient. Sure, the more times you repeat yourself, the more reliable the communication is. But you’ve sacrificed speed for reliability. Shannon showed us we can do far better.
Sorry but I don’t know anyone who thinks repeating the same message in a noisy place is the first instinct, nor that it makes communication more reliable. The opposite, in fact, I know people who hate repeating messages and wisely give up quickly after just one or two attempts fail.
What if his conclusions were more reflections of reality? What if his big contribution was to make acceptable/formal the things already known and practiced, yet codifying it in a way most easily digested by the communities he served?
And most importantly, perhaps, what if he thought the lack of fame and outsized reward for his work isn’t such a bad thing at all? As the founder of the Internet precursor ALOHAnet purportedly once said “I was too busy surfing to worry about that stuff”.
Note how this plays out in a 2013 article about the commonality of humans combining things together, just like Shannon:
Alive and awake to the world, we amass a collection of cross-disciplinary building blocks — knowledge, memories, bits of information, sparks of inspiration, and other existing ideas — that we then combine and recombine, mostly unconsciously, into something “new.” From this vast and cross-disciplinary mental pool of resources beckons the infrastructure of what we call our “own” “original” ideas. The notion, of course, is not new — some of history’s greatest minds across art, science, poetry, and cinema have articulated it, directly or indirectly, in one form or another: Arthur Koestler’s famous theory of “bisociation” explained creativity through the combination of elements that don’t ordinarily belong together; graphic designer Paula Scher likens creativity to a slot machine that aligns the seemingly random jumble of stuff in our heads into a suddenly miraculous combination; T. S. Eliot believed that the poet’s mind incubates fragmentary thoughts into beautiful ideas; the great Stephen Jay Gould maintained that connecting the seemingly unconnected is the secret of genius; Gutenberg’s invention of the printing press embodied this combinatorial creativity; even what we call “intuition” is based on the unconscious application of this very mental faculty.
Of course some cultures still can’t resist trying to focus credit onto one person so that 2013 article also tries to make it seem like Einstein’s version was best:
The concept, in fact, was perhaps best explained by Albert Einstein, who termed it “combinatory play.” (Einstein famously came up with some of his best scientific ideas during his violin breaks.)
To be fair that’s giving credit to Einstein for working so hard at combinatory play that he can explain it well to others.
For a different take on credit and combinatory play as innovation, perhaps take into consideration how an ancient African culture was so successful for hundreds of thousands of years.
When a young man kills much meat, he comes to think of himself as a chief or a big man – and thinks of the rest of us as his servants or inferiors. We can’t accept this … so we always speak of his meat as worthless. This way, we cool his heart and make him gentle.
In other words a young hunter killing big meat would face insults when they presented it to those who would be eating it. Major credit instead went towards the almost random person who delivered the arrow (hunters swap arrows before the hunt), for example.
Leisure and innovation were prized, not infinite aggressive aspiration. Centralized credit was not favored given inter-communication and collaboration.
Some psychologists now call the selfish attributes a function of being disrupted, such that technology may create a domain shift that manifests in “a new selfishness, and ultimately to hierarchical societies, patriarchy and warfare”.
a blog about the poetry of information security, since 1995
