Category Archives: Energy

More people dying in a fire: petroleum-based skin products to blame

An investigation has started to reveal that the practice of putting a distillate of petroleum (parrafin) on your body can lead to a very painful fiery death.

Firefighter Chris Bell, who is a watch commander with West Yorkshire Fire and Rescue Service, says the actual number of deaths linked to the creams is likely to be much higher.

“Hundreds of thousands of people use them, we’re not sure how many fire deaths might have occurred but it could be into the hundreds,” he said.

His concerns were echoed by Mark Hazelton, group manager for community safety at London Fire Brigade.

He said many fire services do not have forensic investigation teams able to properly assess the role of paraffin cream in fires.

In brief, repeated use of a petroleum-based oil in a cream causes soft furniture to become filled with the highly flammable substance. It’s essentially (pun not intended) pouring gasoline on your bed and chair, albeit very very slowly. Then when a fire starts, the outcome of dousing flammable oil is predictable. Product manufacturers haven’t yet been held accountable for this alarming rise in deaths linked to their ingredients.

2018 AppSec California: “Unpoisoned Fruit: Seeding Trust into a Growing World of Algorithmic Warfare”

My latest presentation on securing big data was at the 2018 AppSec California conference:

When: Wednesday, January 31, 3:00pm – 3:50pm
Where: Santa Monica
Event Link: Unpoisoned Fruit: Seeding Trust into a Growing World of Algorithmic Warfare

Artificial Intelligence, or even just Machine Learning for those who prefer organic, is influencing nearly all aspects of modern digital life. Whether it be financial, health, education, energy, transit…emphasis on performance gains and cost reduction has driven the delegation of human tasks to non-human agents. Yet who in infosec today can prove agents worthy of trust? Unbridled technology advances, as we have repeatedly learned in history, bring very serious risks of accelerated and expanded humanitarian disasters. The infosec industry has been slow to address social inequalities and conflict that escalates on the technical platforms under their watch; we must stop those who would ply vulnerabilities in big data systems, those who strive for quick political (arguably non-humanitarian) power wins. It is in this context that algorithm security increasingly becomes synonymous with security professionals working to avert, or as necessary helping win, kinetic conflicts instigated by digital exploits. This presentation therefore takes the audience through technical details of defensive concepts in algorithmic warfare based on an illuminating history of international relations. It aims to show how and why to seed security now into big data technology rather than wait to unpoison its fruit.

Copy of presentation slides: UnpoisonedFruit_Export.pdf

Where is the Revolution in Intelligence? Public, Private or Shared?

Watching Richard Bejtlich’s recent “Revolution in Intelligence” talk about his government training and the ease of attribution is very enjoyable, although at times for me it brought to mind CIA factbook errors in the early 1990s.

Slides that go along with the video are available on Google drive

Let me say, to get this post off the ground, I will be the first one to stand up and defend US government officials as competent and highly skilled professionals. Yet I also will call out an error when I see one. This post is essentially that. Bejtlich is great, yet he often makes some silly errors.

Often I see people characterize a government as made up of inefficient troglodytes falling behind. That’s annoying. Meanwhile often I also see people lionize nation-state capabilities as superior to any other organization. Also annoying. The truth is somewhere in between. Sometimes the government does great work, sometimes it blows compared to private sector.

Take the CIA factbook I mentioned above as an example. It has been unclassified since the 1970s and by the early 1990s it was published on the web. Given wider distribution its “facts” came under closer scrutiny from academics. So non-gov people who long had studied places or lived in them (arguably the world’s true leading experts) read this fact book and wanted to help improve it — outsiders looking in and offering assistance. Perhaps some of you remember the “official” intelligence peddled by the US government at that time?

Bejtlich in his talk gives a nod towards academia being a thorough environment and even offers several criteria for why academic work is superior to some other governments (not realizing he should include his own). Perhaps this is because he is now working on a PhD. I mean it is odd to me he fails to realize this academic community was just as prolific and useful in the 1990s, gathering intelligence and publishing it, giving talks and sending documents to those who were interested. His presentation makes it sound like before search engines appeared it required nation-state sized military departments walking uphill both ways in a blizzard to gather data.

Aside from having this giant blind spot to what he calls the “outsider” community, I also fear I am listening to someone with no field experience gathering intelligence. Sure image analysis is a skill. Sure we can sit in a room and pore over every detail to build up a report on some faraway land. On one of my private sector security teams I had a former US Air Force technician who developed film from surveillance planes. He hated interacting with people, loved being in the darkroom. But what does Bejtlich think of actually walking into an environment as an equal, being on the ground, living among people, as a measure of “insider” intelligence skill?

Almost three decades ago I stepped off a plane into a crowd of unfamiliar faces in a small country in Asia. Over the next five weeks I embedded myself into mountain villages, lived with families on the great plains, wandered with groups through jungles and gathered as much information as I could on the decline of monarchial rule in the face of democratic pressure.

One sunny day on the side of a shoulder-mountain stands out in my memory. As I hiked down a dusty trail a teenage boy dressed all in black walked towards me. He carried a small book under his arm. He didn’t speak English. We communicated in broken phrases and hand gestures. He said he was a member of a new party.

Mao was his leader, he said. The poor villages felt they weren’t treated well, decided to do something about it. I asked about Lenin. The boy had never heard the name. Stalin? Again the boy didn’t know. Mao was the inspiration for his life and he was pleased about this future for his village.

This was before the 1990s. And by most “official” accounts there were no studies or theories about Maoists in this region until at least ten years later. I mention this here not because individual people with a little fieldwork can make a discovery. It should be obvious military schools don’t have a monopoly on intel. The question is what happened to that data. Where did information go and who asked about it? Did others have easy access to data gathered?

Yes, someone from private sector should talk about “The Revolution in Private Sector Intelligence”. Perhaps we can find someone with experience working on intelligence in the private sector for many, many years, to tell us what has changed for them. Maybe there will be stories of pre-ChoicePoint private sector missions to fly in on a moment’s notice into random places to gather intelligence on employees who were stealing money and IP. And maybe non-military experience will unravel why Russian operations in private sector had to be handled uniquely from other countries?

Going by Bejtlich’s talk it would seem that such information gathering simply didn’t exist if the US government wasn’t the one doing it. What I hear from his perspective is you go to a military school that teaches you how to do intelligence. And then you graduate and then you work in a military office. Then you leave that office to teach outsiders because they can learn too.

He sounds genuinely incredulous to discover that someone in the private sector is trainspotting. If you are familiar with the term you know many people enjoy as a hobby building highly detailed and very accurate logs of transportation. Bejtlich apparently is unaware, despite this being a well-known thing for a very long time.

A new record of trainspotting has been discovered from 1861, 80 years earlier than the hobby was first thought to have begun. The National Railway Museum found a reference to a 14 year old girl writing down the numbers of engines heading in and out of Paddington Station.

It reminds me a bit of how things must have moved away from military intelligence for the London School of Oriental and African Studies (now just called SOAS). The British cleverly setup in London a unique training school during the first World War, as explained in the 1917 publication “Nature”:

…war has opened our eyes to the necessity of making an effort to compete vigorously with the activities — political, commercial, and even scientific and linguistic — of the Germans in Asia and Africa. We have discovered that their industry was rarely disinterested, and that political propaganda was too often at the root of “peaceful penetration” in the field of missionary, scientific, and linguistic effort.

In other words, a counter-intelligence school was born. Here the empire could maintain its military grip around the world by developing the skills to better gather intelligence and understand enemy culture (German then, but ultimately native).

By the 1970s SOAS, a function of the rapidly changing British global position, seemed to take on wider purpose. It reached out and looked at new definitions of who might benefit from the study and art of intelligence gathering. By 1992 regulars like you or me could attend and sit within the shell of the former hulk of a global analysis engine. Academics there focused on intelligence gathering related to revolution and independence (e.g. how to maintain profits in trade without being a colonial power).

I was asked by one professor to consider staying on for a PhD to help peel apart Ghana’s 1956 transition away from colonial rule, for only academic purpose of course. Tempted as I was, LSE instead set the next chapters of my study, which itself seems to have become known sometime during the second World War as a public/private shared intelligence analyst training school (Bletchley Park staff tried to convince me Zygalski, inventor of equipment to break the Enigma, lectured at LSE although I could find no records to support that claim).

Fast forward five years to 1997 and the Corner House is a good example of academics in London who formalized public intelligence reports (starting in 1993?) into a commercial portfolio. In their case an “enemy” was more along the lines of companies or even countries harming the environment. This example might seem a bit tangential until you ask someone for expert insights, including field experience, to better understand the infamous pipeline caught in a cyberwar.

Anyway, without me dragging on and on about the richness of an “outside” world, Bejtlich does a fine job describing some of the issues he had adjusting. He just seems to have been blind to communities outside his own and is pleased to now be discovering them. His “inside” perspective on intelligence is really just his view of inside/outside, rather than any absolute one. Despite pointing out how highly he regards academics who source material widely he then unfortunately doesn’t follow his own advice. His talk would have been so much better with a wee bit more depth of field and some history.

Let me drag into this an interesting example that may help make my point, that private analysts not only can be as good or better than government they may even be just as secretive and political.

Eastman Kodak investigated, and found something mighty peculiar: the corn husks from Indiana they were using as packing materials were contaminated with the radioactive isotope iodine-131 (I-131). Eastman Kodak at the time had some of the best researchers in the country on its team (the company even had its own nuclear reactor in the 1970s), and they discovered something that was not public knowledge: those farms in Indiana had been exposed to fallout from the 1945 Trinity Test in New Mexico — the world’s first atmospheric nuclear bomb explosions which ushered in the atomic age. Kodak kept this exposure silent.

The American film industry giant by 1946 realized, from clever digging into the corn husk material used for packaging, that the US government was poisoning its citizens. The company filed a formal complaint and kept quiet. Our government responded by warning Kodak of military research to help them understand how to hide from the public any signs of dangerous nuclear fallout.

Good work by the private sector helping the government more secretly screw the American public without detection, if you see what I mean.

My point is we do not need to say the government gives us the best capability for world-class intelligence skills. Putting pride aside there may be a wider world of training. So we also should not say private-sector makes someone the best in world at uncovering the many and ongoing flaws in government intelligence. Top skills can be achieved in different schools of thought, which serve different purposes. Kodak clearly worried about assets differently than the US government, while they still kind of ended up worrying about the same thing (colluding, if you will). Hard to say who evolved faster.

By the way, speaking of relativity, also I find it amusing Bejtlich’s talk is laced with his political preferences as landmines: Hillary Clinton is setup as so obviously guilty of dumb errors you’d be a fool not to convict her. President Obama is portrayed as maliciously sweeping present and clear danger of terrorism under the carpet, putting us all in grave danger.

And last but not least we’re led to believe if we get a scary black bag indicator we should suspect someone who had something to do with Krav Maga (historians might say an Austro-Hungarian or at least Slovakian man, but I’m sure we are supposed to think Israeli). Is that kind of like saying someone who had something to do with Karate (Bruce Lee!) when hinting at America?

And one last thought. Bejtlich also mentions gathering intelligence on soldiers in the Civil War as if it would be like waiting for letters in the mail. In fact there were many more routes of “real time” information. Soldiers were skilled at sneaking behind lines (pun not intended) tapping copper wires and listening, then riding back with updates. Poetry was a common method of passing time before a battle by creating clever turns of phrase about current events, perhaps a bit like twitter functions today. “Deserters” were a frequent source of updates as well, carrying news across lines.

I get what Bejtlich is trying to say about speed of information today being faster and have to technically agree with that one aspect of a revolution; of course he’s right about raw speed of a photo being posted to the Internet and seen by an analyst. Yet we shouldn’t under-sell what constituted “real-time” 150 years ago, especially if we think about those first trainspotters…

Would removing DMCA reduce pollution?

In response to my earlier posts on VW cheating I have heard several people say “I don’t know engines well so I don’t follow most of what you’re saying”. This is a familiar hurdle, true for most specialized technical fields.

I don’t mind hearing this because I am a believer in bridging. I see no point in shaming people who lack hands-on engine experience or have not thought deeply about the economics of transportation. A technical argument should be able to stand on its own, such that it can be explained to anyone.

So here I will attempt to build a bridge from being a long-time engine tuner to the growing number of very smart IT and infosec people without any real engine experience who suddenly now are looking into smog topics.

More specifically I will answer from experience whether removing engine DMCA immediately would help in the case of VW cheating.

Three Levels of Analytics

On the beginning end of an analytic spectrum, the thought that immediate DMCA removal “probably would help” is a binary form of assessment: see something say something. DMCA is a prior known harm. It has done harm elsewhere. When DMCA is noticed therefore its removal is a simple reaction.

Next on the spectrum is knowing that DMCA can be a harm yet wondering based on ranked data if removal will achieve an objective. Seeing DMCA used by a German car company could mean every German car company is suspect. A ranking system begs the question of how to know when and if safe transition away from DMCA is possible? Is it after German cars no longer are available for sale?

The training examples I suggest to answer this question are from other scandals related to privacy. Lance Armstrong, like VW, was a winner caught cheating. However Lance wasn’t the problem, he was a symptom of demand. He represented a far wider problem.

Using first level analytics (see Lance with privacy and say something) would not be the right approach. Likewise second level analytics are insufficient because Lance was not the only cheater.

Getting beyond level two analytics is very hard. Anyone with audit experience knows it can be a losing battle on the ground unless you have real infrastructure in place to support a search for knowledge. You have to be able to store data, evaluate and adapt. The better your tests the more your cheating adversaries will circumvent them so you need some way to win that race.

A sophisticated level of knowledge is a third level of analytics, which I will call heatmap. As signs of cheating emerge, none very special on their own, the probability is warmer overall. Privacy is not completely lost, but reliable indicators of cheating are developed broadly. This involves sensors so fast, unique and rich in detail that the cheater can not afford to keep ahead of them.

There are two more levels of analytics above heatmap unnecessary to discuss here. Suffice it to say a third level gets us to where we need; it should answer whether and when removing DMCA would be improving air quality.

I use my own experience to work through finding a third level analytics answer. It comes from tuning many engines and even making my own fuel over the last decade. Here are two reasons why I think removing DMCA is a distraction from the main issue: free market risks and the economics of performance tuning.

Free Market Risks

Removing DMCA would be great for innovation and cost improvements from shared knowledge. It would create a more free and unregulated market. That however is not going to magically make pollution stop.

More of something and cheaper doesn’t imply clean. In fact it could be the opposite as the market innovates toward more power for less money. Removing DMCA arguably means the market continues in the worst possible direction and pollution simply increases.

Can we avoid innovation going awry? Yes, with regulation specific to the objective. DMCA is a weak control for issues of competitiveness and innovation, only slightly related to the issue of keeping air clean. Removing it should come when we are able to regulate for clean air.

Removing Lance Armstrong’s privacy could actually make his cheats more pervasive and harder to detect by auditors. So could we improve detection without removing privacy completely? Absolutely yes.

Some suggest the VW cheat was caught using sophisticated testing. I think that’s an exaggeration but we still should look at the tests as an example to model. The auditor success really was in perseverance and perspective more than doing anything clever or novel. Someone kept thinking mpg and power advertised were too good to be clean, so they applied a clean-specific test where VW did not.

Take a moment to think of the VW cheat this way:

  • When you are stationary (garage, warm up in snow) you get cleaner air
  • When you drive, you get more power but it is dirty

This is exactly, and I mean exactly, what typical American customers demand of manufacturers. It is considered acceptable to pollute in the areas least likely to be measured. This is why you can buy “off-road use only” performance parts (meant to be used privately) and then drive them around on roads (publicly) without any real risk of prosecution or fines.

So with pervasive cheating and cheating ingrained in the American engine market why did regulators focus on one company? In brief because it is harder to ban pollution by cheating American consumers than it is to go after a wealthy German company with a minority of vehicles on the road.

To put this in perspective VW already had their cars banned from the California market in 2004. They came back in 2008 with some incredible new numbers and sales took off. All of this has been blogged here extensively before.

A good auditor sees improvement and immediately starts thinking skeptically; how did a small car sales winner get so good so fast (the answer is Bosch, who actually developed “off-road use only” codes). And then the auditor hunts. Sending a car across the country with sensors is not a super special or novel idea, which perhaps you have read in my prior blog posts (e.g. Jaguar boasted 62mpg in cross country test).

Auditors today are closing in on manufacturers because the market functions in a somewhat predictable manner. Changing this abruptly by opening up innovation could lead to many more polluters, groundswell of people acting more like VW (because you’ve removed VW from the equation) and even take us towards weakening of other controls focused on clean air. A focus on a winner with a clear-cut case is a very efficient form of regulation but insufficient, since the problem is widespread.

All of this says to me removing DMCA and opening up a free market without other forms of regulation in place would likely be a clean air setback. It would be like demanding the recipe for cyclist performance enhancing drugs be public in order to reduce their use. Unless cyclists and race organizers are prepared to regulate against use, releasing the recipe can lead to far more cheating and less chance of stopping it.

Performance Tuner Economics

It is well known in the engine market that DMCA does not stop people from completely reverse engineering their cars. Performance tuning firms, not to mention customers themselves, often reverse firmware and/or write their own. In fact you could say there is a symbiotic relationship where the weak enforcement of DMCA allows manufacturers to learn from the after-market crowd what power enhancements to sell next.

Note here there is literally no market for clean enhancements. You simply can not find after-market products designed to get the cleanest possible emissions from your engine.

What VW did was realize that customers wanted more power, more mpg, as they always do. This translates to more convenient “workarounds” and double-speak to avoid regulations of being clean. Thus instead of customers paying $100 and taking 10 minutes to after-market tune their engine, VW essentially modeled customer behavior and provided a solution in software.

VW probably figured why leave the fixes to after-market performance companies. They also likely saw it as a temporary workaround to get back into the market sooner (2008) instead of when they had figured out how to actually comply: both power and clean (2013). Classic product manager risk behavior.

The pervasive cheating that drives VW to do the same is both good and bad. On the one hand it is bad because the market obviously and flagrantly pollutes and no one has budget or tools to stop it at the widespread consumer level. On the other hand it is good because VW took the unrepentant customer bait for better cheats, brought it in-house, and gave regulators a one-stop shop to issue a fine and make an example for everyone to see.

Using our Lance Armstrong example, he cheated more and better than all the other cheaters, which made him the best person to take-down in front of everyone as an example. Some people say VW had 11 million cars affected and this is a lot. Unfortunately this is not a lot in the big picture of cheating.

I mentioned before that California took action in 2005 and knocked VW out of the market. This was because VW was big enough to be a centralized high-profile target but small enough and consumer-centric enough to be made into an easy example. Much more difficult would be for regulators to go after Ford, GM, Kenworth, Caterpillar, John Deere, etc..

Instead of only affecting a few million consumers a regulation at the much larger cheater level could seriously impact business processes and even shut them down. It is common to hear truck drivers complain that if they have to drive a clean engine in order to operate in California they will go out of business; lower mpg or less power to stop polluting is a very hard business decision for hundreds of millions of drivers.

DMCA therefore doesn’t really stop people from innovating (albeit in non-clean direction). So it would have to be enforced far more strictly to help keep air clean. That would be a very bad thing. Harming innovation to reduce pollution sounds backwards because it is. The same resources instead of trying to enforce DMCA could be used directly for enforcing actual clean air controls. The goal being when you finally remove DMCA the resulting innovation would be pointed in a positive direction.

This is why I say stop wasting time talking about DMCA in pollution circles (a mostly non-barrier to reversing and tuning) when you directly could be addressing the actual problems of cheating for actual air quality controls.

Building a Better Solution

In conclusion, I hope I’ve built the argument well enough to stand on its own, no special engine experience necessary. We need to be building a far better surveillance network to monitor for clean air and a far more effective response system for enforcement. This probably sounds shocking so the ethics and norms of behavior have to be ironed out. We should put it in terms of other pollution success stories.

When you see someone smoking a cigarette you say something to them. If that person doesn’t comply you invoke authority. Obviously you can’t tell on the spot you are getting cancer but you have it on good authority that seeing a smoker is reason to act. DMCA of the cigarette industry, such as recipes for mixing and rolling, seem mostly irrelevant because they are.

Thus we really should ask ourselves for engines how do we build a comfortable living environment still capable of finding and stopping engine-smokers?

Imagine every loud pipe you hear is reason enough to say something. Generally loud pipes are after market power improvements that intentionally increase pollution. The ear is no perfect sensor but it’s a start (albeit California regulators have been arguing they can decouple noise from pollution). Imagine neighborhoods using air quality sensors deployed to help build a heatmap; for example monitoring outside popular restaurants collecting data on SUV emissions left behind. You then deliver to the restaurant their pollution results and fine them based on their customer behavior.

There are many possibilities of great impact to consider and plan. Arguments about removing DMCA are mostly irrelevant to clean air economics and technical problems.

Diesel FTW: Throw the Book at Clean Cheaters

Executive summary:

  • The majority of car enthusiasts care more about engine power than pollution. This especially rings true in America where consumers can easily modify hardware and software of their diesel engines. Ten minutes and a couple hundred dollars makes a significant change. Thus it has become common to find consumers seeking personal power gains with little/no concern for environmental impact.
  • Since the late 1940s US federal and state regulatory authorities have set standards and brought action against companies to help the market bear its responsibility for environmental impact. Consumers also increasingly have had to prove ongoing compliance with standards through smog tests linked to vehicle license. The growth of an engine tuning market for power, accelerated by the openness of car software, has forced regulators to crack down on manufacturers as well as move towards greater surveillance of consumers. The latter is less necessary and complex if the former is successful. The gap between demand and responsibility is a key to the issue. People often say “no one has died” regarding engine design despite the fact we know pollution kills and has killed (~58,000 premature American deaths per year).
  • VW was caught giving what most American consumers say they want most, more power. In some sense VW built into their cars before sale what many were doing after sales, which is a common practice. Over 480K cars were illegally fitted with the kind of “clean defeat” practice known to exist at a much larger scale on many more manufacturers led by an emerging “performance” industry. VW happens to have been the largest and most obvious violator caught, which makes it a perfect candidate for heavy regulatory enforcement. Used as a high-profile example, regulators may be able to use this example to shift consumer demand and raise awareness of pollution risk (including fines). US action against a German company also has geopolitical implications.
  • Last but not least, the cheat was unnecessary. VW product managers presumably rushed to market a bolt-on fix rather than a built-in solution. The company could have used a diesel-electric hybrid approach to achieve more power while reducing emissions, as shown with Toyota long-term success in the American market. Worse, VW left the cheats in their newer VW EA288 2L diesel that replaced the “cheater” VW EA188, despite the fact it arguably would be emission compliant anyway using urea injection technology.

Hello diesel fans, welcome back for another post on why diesel is the future of engine technology. Remember when I wrote about NASCAR cheating and included this 1976 quip from Waltrip?

If you don’t cheat, you look like an idiot; if you cheat and don’t get caught, you look like a hero; if you cheat and get caught, you look like a dope. Put me where I belong.

Fast forward almost 50 years and here we are still are talking about cheating to improve engine performance.


Since 2005 you’ve maybe known me to rant about the need for cleaner more-efficient engines and better regulation to make that happen. (e.g. Top Diesel Myths and Why Diesel Hybrids Make Perfect Sense, 2012).

After the VW trivial hack (detect front wheel movement during change in RPM) to cheat regulations I’m even more bullish on diesel and here’s why:

It’s about damn time

First, this government crackdown has been long-overdue and in the works for decades. You know the transportation and automobile lobby finally is losing the dirty fight when the EPA makes this kind of clean success story stick. It seems to me California led since 2000 and took the brunt of counter-attack from those engine enthusiasts who hate being clean.

Anyone who thinks this VW catastrophe is about VW probably does not spend much time tinkering with engines or watch closely all the fighting in the diesel market. Let me be clear here, VW was a business giving the majority consumers exactly what they wanted. And like any very large company it used its size and power to influence governance.

I’ve highlighted some things in an old advertisement here to make it more clear how the spin worked.


Even I have fallen victim to trying to promote power of diesel to make it more appealing (many blog posts in the past about diesel power being a factor).

And that’s a big insight into why this isn’t really about VW. America has a hard time speaking directly to a clean consumer segment; a small, although arguably fast growing, group of people who don’t give a crap about performance when they ask for a clean air car.

Some point to a fact that VW was running ads boasting about achieving the regulatory definitions of clean. That doesn’t mean for a second they cared. It could be they were just following regulators’ lead, talking the talk, playing the game and throwing a few dollars at some words and pictures. The American car companies’ Flex-Fuel campaign is a great example of marketing double-speak that tells insiders at least one car company still doesn’t care about the environment.

Flex fuel: car makers’ way of thumbing their nose at regulations and saying “stop asking, we still don’t care about pollution”. (Sierra Club and Bluewater Network sued to force compliance and reduce gasoline dependence. Detroit smugly responded by delivering much larger engines with higher gasoline consumption)

If you want to get angry about bogus environmental advertising take a swing first at Ford, then BMW, and then…. We have some positive examples too, that suggest clean marketing can be woven into a campaign.

The Prius was introduced by American mavericks in the Japanese Toyota executive office who wanted to test a theory. It was not a customer-driven decision, as I mentioned here in 2006. Most revealing was how clean themes actually took a back-seat to what Toyota really used to push sales:

…the answer lies in Toyota’s clever marketing campaign. To begin with, it wasn’t aimed at the mass market. Instead, Toyota thought that the first hybrid buyers would be “techies” and early adopters (people who are highly likely to buy something just because it’s new).

Americans love early cool tech. They also love luxury. And despite loving power, it was absent from the Prius campaign. You had to look at a Camry for that stuff. The environmental campaign was infused rather than dominant in the carefully targeted Prius themes. Kudos to the late great executive who pushed Japanese sensibility into our thick American tuner heads.

Ok, ok, I’m not being fair to myself or others. Those of us who long pined for environmental improvements in engines just might have grown jaded after seeing twists and turns the product managers used to delay our clean dreams. We found ourselves characterized as a small peanut gallery watching from the outskirts of the big power demonstrations that the “majority” wanted.

Calling for clean diesel regulation has felt a bit like sitting on the sidewalk eating a leafy salad watching the crowds line-up for chemically-enhanced performance-oriented meals in a brightly colored restaurant (i.e.adding hydrogen to vegetable oil to achieve fastest food).

While it is true reading ingredients in a McDonalds Happy Meal might give information to be safer what we really need is regulators or a lawyer in Marin to push for a social norm that even late night talk show hosts can get behind and promote to the majority. Reading ingredients doesn’t do much good if we haven’t fundamentally shifted consumption preferences.

Or let me put it this way: when I was told I could participate in a corporate-sponsored race car event I immediately started asking about how we would measure and explain pollution hitting the crowds. Knowing that cars emitted harmful poisons was insufficient, I needed to get people to question whether we really intended to poison our VIP customers. Unsurprisingly, as those around me sipped their well-labeled alcohol and ate their sugary snacks that clearly listed all the ingredients, they didn’t really see what I was so concerned about.

Later I found myself in an even bigger “our future is data-driven” corporate-marketing event focused on race cars. I asked an Indy car team manager what the brake dust and tire wear meant for people standing near the track. “No idea” was the answer. And years later I asked a F1 team the same. Same answer. Some future. Data data everywhere and not a person who, despite having access to learn about harms, wanted to alter car culture towards being safer.

The point here, after saying this is not really about VW, is that it also is not about openness and transparency of the software. Openness isn’t the fundamental problem in the case of diesel emissions cheating. The real key to driving change is a push from regulators and to create the right pull from consumers; nudge economics is what I’ve heard it called lately.

Being a minority in trying to figure out the push/pull on majority risk issues should surprise no one working in the security industry. It is basically what we’re paid to do. Nonetheless sometimes there are twists we don’t anticipate as these socio political things are hard. The other day I found this curious notice from a security software organization:

A notice by Whisper Systems, considered by some a leader in security software, said majority concerns come first and consumers must swallow their closed sole-source manufacturer distribution channel.

Open WhisperSystems has chosen to focus on serving the millions of users who have GCM capabilities before turning our attention to the small number of users who refuse to install Google Mobile Services. We understand that this is an important issue for some of our users and have our support forum available for discussions.

The arguments used by WhisperSystems to justify this position simply is not true. And they’re telling us being small is why we’re lower priority? The number of privacy-enhancing software use overall is small, so should on that measure alone Google turn their attention elsewhere first? Hey Google, maybe you should start ignoring WhisperSystems because they are only a small number of people who refuse to just be happy with default apps provided by Google.

No I think size is not the right measure to start and end with. Other measures of priority are useful.

Sorry, I digress…let us go back to talking about VW, a software company using false statements to justify their position to appease the majority with a closed sole-source manufacturer distribution channel. Oh, wait a minute.

But seriously, let’s go even further back to regulators stepping in to shape the diesel market and consumer demands. The emissions debacle is really about regulators working over a long time to clean the air. They had to choose targets wisely (deep pockets from large numbers of consumer vehicle sales) and massage timing (emerging shift in public opinion based on solid grains of truth) to move a market after it refused to go cleaner on its own.

The fact that VW didn’t see this coming and thought they could cheat regs, or wait for a GOP victory that would weaken the EPA or worse, is just sad management. Fire that CEO for being out of the loop on political winds that in reality are directed towards everyone but start with the biggest and most useful example. VW deserves the book thrown at them because that’s how this game works. We make an example to educate others and VW had its neck out, way out, the wrong direction.

To really put the game in perspective, don’t forget Ford dumped their CEO after he called for a clean car revolution. Put that in your carburetor and smoke it for a bit. A major car manufacturing board kicked out a CEO who wanted to go clean. Easy to see how VW executives thought cheating with dirty cars would help them fit right into the market, get a nod and a wink rather than a fine.

Except there was a slight problem. They underestimated the importance of a minority voice and opinion.

Here’s the real choice, which apparently they did not see. Either you clean up diesel like we know can be done (gasoline cleaned up and thrived) or you become an example of why actually you have no choice. Too many decades passed when we let the establishment give empty promises and shallow marketing about flex fuel, yada yada. Clean up your engines or we’ll disrupt this market so hard small new-comers can jump in to compete and sell a proper clean product, verstanden sie?

Look closely at 2005 as a huge turning point. California regulators (and NY and some other states of little sales numbers) basically ruled VW out of the market. Cold. No more diesels could be sold by VW. They were nailed, while at the same time the majority of other polluting diesels were given a pass.

I have yet to see any pundits bring this seminal point into focus on today’s news. Watching this fight for decades obviously puts things in a different perspective. Having been a long-time diesel tuner and having made my own diesel fuel I have a few dozens of blog posts related to this topic.

Politics ten years ago proved VW was the easy target to initiate a clean air battle, despite American trucks going on and on spewing poison all around us. That is a key to unlock the context for recent news. VW consumer cars could not emit a “we must pollute to survive” excuse as easily as a Caterpillar, Ford or Kenworth.

2010 was another massive turning point when California applied smog tests to diesels. Even I was shocked when I received my first letter from the state. We all should have seen coming yet I confess, I have to admit, I was amazed the day finally came in 2010 when I had to test my diesel. And I was proud that all my tinkering did not reduce clean.

The regulators slowly were winning these small battles in small markets to test attack methods and gear up for a major war against big air polluters. They were wise. And so at long last, after decades of waiting, here we are…thank you thank you EPA.

Grains of truth

Second, it’s really about the engineering facts. With diesel a smaller engine produces more power, more cleanly, more efficiently from more renewable sources of energy than any of its competitors.

A diesel was not intended to run on petroleum, it was designed to do the exact opposite and free owners from sole-source energy. The petroleum industry bastardized the original diesel design, making it run on their product, which is a disgrace to engineering.

I just have to get this out of the way. Measuring diesels today on petroleum fuel is, albeit necessary because history, technically a petroleum industry’s trick. Don’t fall for it. We really should be testing the latest engines on multiple sources.

Let me present the amazing Subaru STI-D (2008 or even better 2011) as an example of what every American today should be looking for in their next vehicle:


And now let me put this in context. That little tiny light engine is hugely powerful (380 lb-ft torque) while being compliant with the EuroV emissions requirements.

Diesel Emissions Standards

Fantastic progress. As an aside did you know that gasoline engines were not tested at all for particulate matter until EuroV? Shocking. So while lots of writers have jumped on VW to complain about shameful cheating to squeeze under tightening PM filtering rules, they say nothing about gasoline engines not being tested at all. Meh.

Even more to that point the people racing tend to brag about not having to be compliant with any smog requirements at all because they found “exemption” loopholes. Here’s a Subaru diesel racer proudly spewing horrible PM: Jump to 0:53

I see this nearly EVERY DAY from other engine tuners. It’s a hugely widespread problem. Truck drivers might even be the worst and most prevalent. The people gearing and wrenching just don’t talk like they are worried about being clean until regulators clamp down. A big cheater take down is a much easier way to shift majority sentiment than trying to go after every little tuner.

In 2005 I was offered numerous chip options for my engine and remapping software to undermine emission controls and boost performance. It was from a few diesel specialists but things have progressed quickly to many more collaborating on tuning software. Here is a diesel tuner comment from 2011, shortly after the EuroV generation STI-D was announced:


Who in 2011 wanted to be part of open source history? Turns out few signed up and so these guys went proprietary instead. Regulators made an example of VW, the largest car company everyone knows, despite so many lower-profile examples everywhere of the same behavior. In fact VW probably just licensed diesel tuning software from one of the performance shops any customer could buy from.

Today we still have tuners all over SF removing their compliant pipes and putting on “noise and air pollution sticks” given typical motives, which rarely include being kind to their environment. Just last night a Canadian was bragging to me about his Ducati being loud with track pipes and so much fun. I had to cut him off and explain the respiratory damage to our neighborhood.

He had no idea. None. This is the real problem. VW management decisions seem to be more a symptom if you actually get your hands dirty, know engines and talk with people about what is happening. When I meet polluters I often pull out a 2004 report on snowmobiles to try and frame how a feedback loop should work.

In recent years, Yellowstone employees suffered headaches, nausea, sore throats, and watering eyes as they worked in a haze of snowmobile exhaust. The health hazards forced the National Park Service to pump fresh air into entrance booths. When workers continued to get sick, the Park Service issued respirators. So far this winter, the Park Service reports that none of its employees have gotten sick from breathing snowmobile exhaust.

That was five years into the fight. By 2013 the environmentalists had successfully shifted social norms and manufacturers had to admit pollution was an unnecessary loophole.

The rules were 15 years in the making because of intense wrangling between snowmobile operators and environmentalists. But both groups support the plan and give credit to snowmobile makers for designing cleaner machines.

If I remember the Yellowstone ranger studies right, one consumer on a non-compliant or exception engine was the equivalent of nearly 10,000 cars exhaust. 1:10,0000 as a measure of harm. And so many people do it without thinking a second about that kind of damage because it’s all external to them or they leave it behind and go home elsewhere.

If someone in America races, runs off-road or uses engines for special purpose (commerce, showing off to friends how loud and obnoxious you can be) they turn off the environmental concerns; especially if it’s a world they just visit occasionally and don’t have to breathe daily because no feedback loop.

With no feedback Americans will make claims that controls impede an ability to win or impress, or get a job done: make a few extra bucks on a trailer full of unripe bananas they have to deliver before it turns into fruit flies. Here is a classic reaction in 2010 when California announced enforcement of diesel emissions checks would include aftermarket products and tuning:

F.U. SACRAMENTO! I’m just trying to save money by getting better gas mileage and not blow my tranny towing. ARREST THE VIOLENT CRIMINALS AND TAX THE MILLIONAIRES

Don’t get me wrong. Sometimes there are justified reasons to set aside one concern, safety, to focus on another such as performance. The nature of the problem is that a justified delay or postponement of safety concern to allow other values should be revisited quickly.

I used to run into this all the time from cloud vendors, especially Platform as a Service (PaaS) VPs who would claim security means leaving it up to developers to feel and find the right balance. They almost always were trying to escape considering risk, waiting to bolt-on something instead of baking safety into their platform.

Consider how top engineers in the elite tank design unit of the US Army have built a prototype that uses…a Subaru diesel-electric hybrid. The best engineers in the best Army in the world aren’t futzing around and they are pushing the envelope on vehicle design with diesels. Yay.

Their diesel engine can take in fuel from basically anywhere, anything (troops easily can build a quick bio-diesel generation station to use local sources of oil — waste, trees, algae, etc) that will recharge the electric motor. Imagine having no fuel supply issues as you get (or give) orders to advance into the most remote and hostile territory.

My point is after you get to this amazing point on every possible performance level, where diesel-electric hybrid is outshining other power plant designs, you wonder who on that team is really looking at pollution. Why would they? Who measures it as a success?

When there is nothing powerful enough, no external feedback-loop, to push product teams to include safety from the start, they leave it out. That totally safe Army vehicle, where safety is job one, probably has zero pollution assessment in the final tally.

But I could be wrong. To be fair, some regulations have started to show employees around heavy machinery perform better in clean air. There could be someone monitoring soldier health saying air quality must be clean to win wars. Maybe the Army thought about a sick soldier as a problem and wants cleaner vehicles for improved chance of victory.

This kind of economics problem is the problem of security industry in a nutshell; even deeper it is the problem of quality in products. Bolt-on, not built-in is like fingernails scratching the chalkboard to the security professional being dragged into the product management office for an architecture review. We don’t want to have to ask VW “so explain exactly after 30 years of diesel engines you decided to make them clean in 3 years how?”

VW could have done so much more, could have released a far superior product, many many years ago instead of letting down the environmental minority. Instead they gambled and waited for that minority to start to reach greater opinion and political leverage and by then they were caught behaving badly because they listened for too long to the wrong Americans.

It’s economics, stupid: diesel-electric hybrid launch is cheaper than cheating

Ok, but I hear people, especially young people, say they love forward-looking Musk electric cars named after a famous American. That surely is built-in because no pollutants, right? Shouldn’t all companies jump in the race towards electric cars to solve emissions?

The problem is something smells funny in the Musk office. Why is the range of the car so short (under 100 miles) when driven by engineers who build it, but the marketing claims more than double? Cutting the efficiency in half during real-world driving conditions means Musk is sucking serious energy from coal plants, am I right?

And when you look at the refueling model, how do they break away from top-down dictated energy sources if there is a special interface instead of a universal standard? My guess is this is why they released their IP, to encourage other manufacturers to standardize on their interface. Good move yet still begs the question of control.

More to the point why continue any relationship with Musk after you buy the car? Dare I say it should be seen as curiously anti-freedom to build central-control personal cars with top-down tracking of our daily driving experience. I know this is bucking the trend, given Inrix, Google maps, Bluetoad and all the others trying to monitor our every move.

In the long-run however we surely will find drivers wanting to go off-grid and disconnect from mother Musk. Denying a reasonable option by design can lead to some dangerously predictable behavior, such as tuners removing emission controls in a quest for more power. Listen to customers, but listen wisely.

If I buy a $100K Musk-cart I don’t want to be forced to continue my relationship after purchase day. Let me choose the relationship and connection based on my needs. Don’t lock me in with your service-oriented tentacles. Keep the software open and the personal data closed. I certainly don’t want Musk poking around in my internals without my authorization or shutting my car down at his whim.

No thank you. For me, Diesel had the right plan from the start. His genius coupled with Tesla’s would be the ideal car. It’s long past time to throw the book at those cheating on his grave.

So what now should we do about it?

First, further accelerate the clean air standards and regulations and raise mpg requirements now. We are far behind and the manufacturers have abused every bit of leeway allowed. It is time to take up the slack and force innovation through measured feedback (e.g. enforcement). The market is ready to bear many new options and the incumbents are using their cheats for margin to hold back progress.

Second, revisit the 2001 Right to Repair Act as I’ve said before, and ensure customers retain the rights to troubleshoot and understand fully their vehicles. There is no proven risk to opening the information. Actually the opposite tends to be found. Tuners innovate faster and so manufacturers can learn and improve from the collaboration. The catch being tuners also have to be headed towards improvement using social norms. Ask me why bulletin boards are full of how to improve performance of engines, regardless of emissions, yet never seem to talk about pulling seat-belts out.

Third, realize that car companies claim to respond to customer demand. If they don’t sell what people ask for, they lose. That allows us to focus on the problem of defining clean engine demand; changing the voices that manufacturers focus on. We could also cop out and use a Prius “new tech” model with just a hint of clean. But here are two ways we might be able to force direct clean feedback-loops into engineering: monitoring and enforcement.

It is a thorny issue but I believe the answers to monitoring are in randomness and persistence. This is exactly what testing labs did and should continue to do. Testing for environmental pollution during environmental activity is nothing new. After all we have mpg listed on cars for city and highway “conditions”, am I right? Putting sensors on a diesel and measuring it as it drives across the US is a reasonable test, as I’ve written before (#XFCoast2Coast). Even more to the point I believe it was in-field discovery of large trucks in California removing environmental protections in the mid 2000s that helped push towards 2010 enforcement of diesel smog tests.

More research labs, in cooperation with local air quality authorities, should be funded to sample and exhaust the possibilities. The fact that it was a European wing of the US International Council on Clean Transportation (ICCT) that unraveled the VW cheats is a great example to expand from. Resources should be allocated to grow independent and creative ISEA (Identify, Store, Evaluate, Adapt) centers to put manufacturers through rigorous tests, while also scaling up existing ERM (Easy, Routine and Minimal Judgment) smog tests for everyone else — simple scheduled stationary assessments.

Enforcement, given a shift of social norm, becomes easier to solve as this issue drags along. VW has been the whipping standard for over a decade but it makes little sense to pretend that this issue is only about them. Fines for big manufacturers is a start, but let’s also keep an eye on tuners and commercial organizations/fleets as well. Those claiming a test “in the wild” or “during use” must account for the consumers pulling a similar cheat after manufacturers hand over the ECU.

Again I want to reiterate that what VW was caught doing is basically what every diesel tuner forum everywhere talks about. In the older hardware cases I knew big diesel truck drivers who put the original chips back in their engine during a smog test and then swap again when they hit the road. Revising software is clearly easier. Social harms aren’t really part of these folks’ equation. The answer to that is not pervasive surveillance of any potential tuner (testing everyone in the wild) but rather a more systemic approach to encourage behavior change.

While I agree with openness and am a huge proponent of right to repair, the VW situation is a good example of where open software would solve a different problem set than the one directly in front of us. Simply calling for open software, even just escrow, in this case may shift pollution problems worse by expanding cheats undetected, pushing tuners the wrong direction. Enforcement through social pressures and localized testing (ala the seat-belt shift from resistance to desire for self-compliance) must be a consideration.

In conclusion, I’m grateful we finally are seeing California clean air battles with diesel reach the federal level. It has been too long a wait for the book to be thrown.

With any luck the EPA action will be a big help to a certain little American car manufacturer in excellent position to deliver a superior product — clean diesel for freedom and fun to those who have such a desire, even if we’re still a minority. Shame about not being able to crack-down on pollution much sooner, like back in the 1980s…

Subuaru Style

In conclusion, and given the wisdom of NASCAR experts on cheating, put VW where they belong.

The Little Can That Could: History of the Yellow Jerrycan

Part three in a three part series about the history of the Jerry can; this page is a reprint of “The Little Can That Could” a first-person account to support parts one and two.

Written by Richard M. Daniel, retired commander in the U.S. Naval Reserve and a chemical engineer, and published in Fall 1987 Invention and Technology pages 60-64.

During World War II the United States exported more tons of petroleum products than of all other war matériel combined. The mainstay of the enormous oil and gasoline transportation network that fed the war was the oceangoing tanker, supplemented on land by pipelines, railroad tank cars, and trucks. But for combat vehicles on the move, another link was crucial—smaller containers that could be carried and poured by hand and moved around a battle zone by trucks.

Hitler knew this. He perceived early on that the weakest link in his plans for blitzkrieg using his panzer divisions was fuel supply. He ordered his staff to design a fuel container that would minimize gasoline losses under combat conditions. As a result the German army had thousands of jerrycans, as they came to be called, stored and ready when hostilities began in 1939.

The jerrycan had been developed under the strictest secrecy, and its unique features were many. It was flat-sided and rectangular in shape, consisting of two halves welded together as in a typical automobile gasoline tank. It had three handles, enabling one man to carry two cans and pass one to another man in bucket-brigade fashion. Its capacity was approximately five U.S. gallons; its weight filled, forty-five pounds. Thanks to an air chamber at the top, it would float on water if dropped overboard or from a plane. Its short spout was secured with a snap closure that could be propped open for pouring, making unnecessary any funnel or opener. A gasket made the mouth leakproof. An air-breathing tube from the spout to the air space kept the pouring smooth. And most important, the can’s inside was lined with an impervious plastic material developed for the insides of steel beer barrels. This enabled the jerrycan to be used alternately for gasoline and water.

Early in the summer of 1939, this secret weapon began a roundabout odyssey into American hands. An American engineer named Paul Pleiss, finishing up a manufacturing job in Berlin, persuaded a German colleague to join him on a vacation trip overland to India. The two bought an automobile chassis and built a body for it. As they prepared to leave on their journey, they realized that they had no provision for emergency water. The German engineer knew of and had access to thousands of jerrycans stored at Tempelhof Airport. He simply took three and mounted them on the underside of the car.

The two drove across eleven national borders without incident and were halfway across India when Field Marshal Goering sent a plane to take the German engineer back home. Before departing, the engineer compounded his treason by giving Pleiss complete specifications for the jerrycan’s manufacture. Pleiss continued on alone to Calcutta. Then he put the car in storage and returned to Philadelphia.

Back in the United States, Pleiss told military officials about the container, but without a sample can he could stir no interest, even though the war was now well under way. The risk involved in having the cans removed from the car and shipped from Calcutta seemed too great, so he eventually had the complete vehicle sent to him, via Turkey and the Cape of Good Hope. It arrived in New York in the summer of 1940 with the three jerrycans intact. Pleiss immediately sent one of the cans to Washington. The War Department looked at it but unwisely decided that an updated version of their World War I container would be good enough. That was a cylindrical ten-gallon can with two screw closures. It required a wrench and a funnel for pouring.

That one jerrycan in the Army’s possession was later sent to Camp Holabird, in Maryland. There it was poorly redesigned; the only features retained were the size, shape, and handles. The welded circumferential joint was replaced with rolled seams around the bottom and one side. Both a wrench and a funnel were required for its use. And it now had no lining. As any petroleum engineer knows, it is unsafe to store gasoline in a container with rolled seams. This ersatz can did not win wide acceptance.

The British first encountered the jerrycan during the German invasion of Norway, in 1940, and gave it its English name (the Germans were, of course, the “Jerries”). Later that year Pleiss was in London and was asked by British officers if he knew anything about the can’s design and manufacture. He ordered the second of his three jerrycans flown to London. Steps were taken to manufacture exact duplicates of it.

Two years later the United States was still oblivious of the can. Then, in September 1942, two quality-control officers posted to American refineries in the Mideast ran smack into the problems being created by ignoring the jerrycan. I was one of those two. Passing through Cairo two weeks before the start of the Battle of El Alamein, we learned that the British wanted no part of a planned U.S. Navy can; as far as they were concerned, the only container worth having was the Jerrycan, even though their only supply was those captured in battle. The British were bitter; two years after the invasion of Norway there was still no evidence that their government had done anything about the jerrycan.

My colleague and I learned quickly about the jerrycan’s advantages and the Allied can’s costly disadvantages, and we sent a cable to naval officals in Washington stating that 40 percent of all the gasoline sent to Egypt was being lost through spillage and evaporation. We added that a detailed report would follow. The 40 percent figure was actually a guess intended to provoke alarm, but it worked. A cable came back immediately requesting confirmation.

We then arranged a visit to several fuel-handling depots at the rear of Montgomery’s army and found there that conditions were indeed appalling. Fuel arrived by rail from the sea in fifty-five-gallon steel drums with rolled seams and friction-sealed metallic mouths. The drums were handled violently by local laborers. Many leaked. The next link in the chain was the infamous five-gallon “petrol tin.” This was a square can of tin plate that had been used for decades to supply lamp kerosene. It was hardly useful for gasoline. In the hot desert sun, it tended to swell up, burst at the seams, and leak. Since a funnel was needed for pouring, spillage was also a problem.

Allied soldiers in Africa knew that the only gasoline container worth having was German. Similar tins were carried on Liberator bombers in flight. They leaked out perhaps a third of the fuel they carried. Because of this, General Wavell’s defeat of the Italians in North Africa in 1940 had come to naught. His planes and combat vehicles had literally run out of gas. Likewise in 1941, General Auchinleck’s victory over Rommel had withered away. In 1942 General Montgomery saw to it that he had enough supplies, including gasoline, to whip Rommel in spite of terrific wastage. And he was helped by captured jerrycans.

The British historian Desmond Young later confirmed the great importance of oil cans in the early African part of the war. “No one who did not serve in the desert,” he wrote, “can realise to what extent the difference between complete and partial success rested on the simplest item of our equipment—and the worst. Whoever sent our troops into desert warfare with the [five-gallon] petrol tin has much to answer for. General Auchinleck estimates that this ‘flimsy and illconstructed container’ led to the loss of thirty per cent of petrol between base and consumer. … The overall loss was almost incalculable. To calculate the tanks destroyed, the number of men who were killed or went into captivity because of shortage of petrol at some crucial moment, the ships and merchant seamen lost in carrying it, would be quite impossible.”

After my colleague and I made our report, a new five-gallon container under consideration in Washington was canceled. Meanwhile the British were finally gearing up for mass production. Two million British jerrycans were sent to North Africa in early 1943, and by early 1944 they were being manufactured in the Middle East. Since the British had such a head start, the Allies agreed to let them produce all the cans needed for the invasion of Europe. Millions were ready by D-day. By V-E day some twenty-one million Allied jerrycans had been scattered all over Europe. President Roosevelt observed in November 1944, “Without these cans it would have been impossible for our armies to cut their way across France at a lightning pace which exceeded the German Blitz of 1940.”

In Washington little about the jerrycan appears in the official record. A military report says simply, “A sample of the jerry can was brought to the office of the Quartermaster General in the summer of 1940.”

Go back to part one or two in this series.

The story behind the yellow Jerry can

Part two in a three part series. (Part one and part three)

Once upon a time I sailed half-way across the Pacific Ocean with the typical yellow fuel can lashed to the deck.

yellow cans on deck

The yellow Jerry can has specific meaning to me — diesel fuel — which I thought was a standard. Yet recently I found a charity worker showing me yellow cans of… water with smiling children, as they asked me to donate funds.

Stock photos of happy smiling children, poor children, playing with yellow cans; this looked weird to me. I wanted to see charts of health and safety data from operations, not ignorance of toxicity from unsafe oil handling/disposal.

Flashy photos provided questionable value to me, or the opposite…made me curious about what might really be lurking beneath such shallow propaganda.


Is this really any different than children miners (minors) grinning through the toxicity of their forced labor environment?

After 1842, no child under the age of ten was allowed to work underground

Yellow cans in obviously staged photos seemed to be encouraging me to accept that children using them for water is some kind of acceptable normal. In fact the unsettling appearance of a fuel can in the hand of smiling children supposedly can be seen “everywhere”, as they have written without irony:

You’ve seen it everywhere on our site, at our events, on our shirts… tattooed on our arms… and although the Jerry can has become a mainstay for our staff and supporters, we want to let you know what it actually is and why it’s a symbol of the charity: water mission.

The diesel can a symbol of a water mission? “Our site, our events, our shirts, our arms”. Note the emphasis on “our” mainstay, rather than a mainstay of the people being helped. My definition of everywhere is a bit broader. Is this a mission to convince staff and supporters that a yellow can should become a symbol of water or that it already has? Because…why?

Something smelled funny. Globally I had learned in my travels, regardless of continent or sea, yellow cans meant one thing, and it was NOT water. Yellow often is used for warning signs; first-hand experience around the world has associated yellow cans with sickening slicks and fumes of poison.

Red gasoline cans, yellow diesel cans. Those are the ones you DO NOT DRINK from let alone touch and breathe. Often we would end up scrubbing and wiping the nearly permanent mess of petroleum around those cans.

And yet, because standards change, I still am open to be convinced otherwise if someone can show data.

Surely there are cases (no pun intended) where options are limited, and people have to make do with what little they have. Reuse of fuel cans for water? Sounds like an indicator of desperation or lack of regulation. Is this evidence of the need for many more white or blue cans?

Globally white and blue are used to symbolize health and safety (e.g. Blue Cross, Blue Shield, U.N. Department of Peacekeeping Operations blue hats and helmets, as well as the white helmets with blue suits of disaster relief workers)

"clouds in the sky" white helmets and blue suits means safe. yellow means warning or caution
Singapore disaster team prepares for Nepal. White helmets and blue suits (“clouds in the sky”) indicates neutral or safe. Yellow indicates warning or caution.

I mean we are talking about a charity here, where setting a new standard of good is supposed to be the mission, especially where health risks are found. For a charity with wealthy backers and industrial input the choices obviously are many, so the standard should be high. There is great risk in using charity to reinforce harmful behavior.

Confused by charity workers flashing smiling kids in your face to get your money? Me too.

How did someone decide, of all the options, to adopt yellow cans as a sign of health, a symbol for “clean” anything? And why are they just showing stock photos to get donations instead of any real data?

What comes next, bright red oil barrels for charity:meal?


Let’s forget I asked that…although to be fair red in this case could make sense to warn people about heat and to stay away from the barrels.

I searched for answers and some history on can safety. Either I would become convinced that it now is safe for people to drink from yellow cans, and it is safe to give this charity money, or that existing standards need to be defended and propaganda exposed.

My search led to some very interesting surprises.

The charity website reduced my confidence in their ability to collect and analyze data, for example. You might say my opinion worsened as I read through apologetic narratives about Nazi Germany.

Here are four examples, paragraph by paragraph, of what I found and why this charity is so wrong:


To most people, this simple metal or plastic can means ‘gasoline,’ and rightfully so — the first Jerry cans were introduced as gasoline containers by the German military at the start of World War II.

There was some kind of war, a second world war, and this military from Germany that had to go to war also had some need for gasoline, see…


Jerry cans existed during the Spanish Civil War of 1936, years prior to the start of WWII. These cans served both as fuel and water containers, which we know because they were stamped with clear markings for their purpose.

Germany was involved with and supported other fascist militarism. Someone within the growing Nazi war machine was looking at how to improve a fuel can long before Hitler mobilized troops on 15 March 1938 (passive capitulation of Czechoslovakia) or 1 September 1939 (1.5 million marched into Poland, conquering 140 miles in just one week).

I believe the real story goes to lessons in vehicle support and supply containers (e.g. evaporation/expansion) derived from Italian invasion (3 October 1935) of Ethiopia and there is evidence cans were modified and tested during Nazi support for fascists in the Spanish Civil War (17 July 1936).

Handling chemicals in extreme conditions had forced Italy and Spain to innovate their cannister technology. For example the Italians had developed new mustard gas and new bombs to drop on hospitals and ambulances flying the red cross (infamously killing Swedish medical leaders Fride Hylander and Gunnar Lundström).

December 1935 Dolo Ethiopia Italian Bombing Killed Dr Lundstrum in Ambulance

This day is still called “darkest in the history of the International Red Cross“; worth reading if you want to get a sense of how in 1936 a rapidly expanding fascist offensive led to a quickening pace of technology change.

Does the can mean gasoline? The phrase “to most people” used by this charity indicates they have some kind of data or source to check, yet none is provided.

I would say to most people the Jerry can means more than gasoline. It means a variety of fuels and even water. My data on this is based on search engines where the top results are “Jerry Cans – Fuel, Water, Diesel, & Accessories” and “can be used for fuel and drinking water”. The word gasoline does not come up easily.

It is true that 1930s Germany used gasoline for their vehicles. However even they stamped their fuel cans with the generic word Kraftstoff (fuel) or with Wasser (water). The Wasser cans also were painted with broad white lines to ensure it could not be confused with Kraftstoff.

This says to me that today’s use of yellow color on a can would, like the Nazis originally intended, help differentiate unsafe fuel cans. Here is what a Nazi water can, stamped with Wasser and painted with white lines, looks like:


So to most people I think it fair to say the Jerry can means various liquids, not simply gasoline, and most people expect consistent symbols and use to avoid mixing them.

Moving everyone to think of yellow as safe for water seems doable, although expensive and risky, as it really has to be clear where diesel and water are to be found. It seems like a lot of extra work/cost because of confusion, as a friend recently put it:

Whoever made the almond-milk carton the exact same shape as the chicken-broth carton should have to eat this cereal.

Labeling/testing yellow Jerry cans on a massive scale as safe for water seems much, much more complicated and risky than just continuing to use the existing standard of white or blue water cans.


These five-gallon cans, also called ‘Jeep cans’ or ‘blitz cans’ (or, in Germany, ‘Wehrmachtskanisters’) were made of steel and usually sat in the back of vehicles as a reserve tank of gas.

In Germany there were these things with a funny German name in the back of vehicles, kind of like a Jeep, used for an afternoon blitz…


Wehrmachtkanisters means “army can”. Fascists who initiated war without provocation strapped multiples of cans to the side of their vehicles during invasions of foreign countries. In theory the blitzkrieg (German for “lightning war”) was a strategy of very brutal and fast advances to rout an enemy before they could respond.

Obviously there is less surface area in back (width versus length of a vehicle) so lashing cans to the sides has many advantages: leaves space available and makes use of open spaces, balances weight more evenly, while keeping nasty toxic fuel away from doors, passengers and gear. Use of the sides also means the back can be used for less durable/convenient assets and for giant doors and loading (e.g. troop deployment from trucks).

You may notice the white broad lines on some cans, clearly indicating Wasser instead of Kraftstoff.

Bundesarchiv_Bild_101I-022-2926-07,_Russland,_Unternehmen_"Zitadelle",_VW_Kübelwagen Bundesarchiv_Bild

You will find the same behavior on a boat that has to cross an ocean, as you saw at the start of this story. Reserve cans are balanced on either side, not in the back. It would be stupid to weigh down the back of a vehicle/boat with a dozen cans when sides are empty.

Now lets talk about gallons. Jerry cans are 20L capacity and stamped with this unit — about 5.28 US gallons or 4.40 UK gallons. Jerry cans were not “5 gallons” as Charity:water seems to believe. I find it very odd an international organization would use gallons, let alone not specify a system of gallons. Liters are the original and obvious measurement. Someone thinking in gallons has imposed a very narrow and inaccurate perspective over reality.

In terms of material the cans were not only steel; what made Jerry cans most notable in terms of material was a synthetic lining unlike other metal cans. Plastic cans, or even kevlar-lined battle containment for fuels, today could perhaps be linked to the synthetics of the Jerry can.

In terms of brand association, Jerry cans weren’t used by Jeeps until many years later. I am not sure why Jeep gets brought in so subtly next to “blitz cans”. It strangely brands a pre-existing can with a trademark of a specific American vehicle despite the cans not being developed for it originally and being used much more widely. Perhaps Charity:water is thinking ahead about the power of branding and hopes someday we’ll call them Charity:cans?

Speaking of American trademarks, “Blitz” reminds me of a sad and strange twist in history. As I explained above the word means lightning in German; a military campaign tactic attributed to the Nazis. It also refers to a specific 1940 bombing campaign meant to demoralize the British by killing civilians and destroying industry. Not the best connotations. With that in mind an American manufacturing company made the odd decision to adopt it as a name for their “improved” version of Jerry cans.

Originally a US metal container company that made Jerry cans in the 1940s used the words “metal container” in their name. They grew so large and successful that 50 years later the vast majority of American fuel cans were made at this “U.S. Metal Container” (UMC) company. When UMC moved its production away from metal to making only plastic cans in the 1990s they changed their name.

Instead of just switching to the acronym UMC, which would have been clever and celebrating American military history, they adopted the infamous Nazi term “Blitz” as their name because, well, UMC was located in Oklahoma. It should be no secret that neo-nazis and Hitler apologists lived an open life in Oklahoma. But I digress…

Anyhow after changing its name to the Nazi “Blitz” and moving everything to plastic production this venerable Jerry can manufacturer (that perhaps even helped defeat Nazi Germany) soon filed for bankruptcy.

“Blitz” said it could not survive the dozens of lawsuits over its defective cans that were exploding and killing Americans. I told you there was a twist.


It’s said that Adolph Hitler anticipated the biggest challenge to taking over Europe in WWII was fuel supply. So Germany stocked up.

False and super annoying.

Look, this is very wrong for many reasons. I don’t expect to read charitable thoughts on Hitler from a supposed “charity” site. WTF. No really, WTF.

Also I find “it’s said” to be an unacceptable start to a pro-Hitler sentence that lacks any citation. Who said Hitler anticipated…what? Hitler was an insane dictator and deserves no glorifications. I should not need to cover this.

Nonetheless, it is easy to see how badly that fascist leader sucked at planning. The USAF points out he took his country to war with an acute fuel shortage and massive dependence on imports:

At the outbreak of the war, Germany’s stockpiles of fuel consisted of a total of 15 million barrels.

That is basically nothing, given their rate of consumption, and fuel was expected to run out by 1941. Two years after starting the war, stupid Hitler lacked a plan to continue supplying fuel. Cans clearly were not meant to solve the macro challenge. The American pro-fascist company Standard Oil played an essential role in illegally supplying fuel to Hitler’s air-force even as it was bombing London, which arguably had far greater impact than any container holding that fuel.

Actually I’m getting ahead of myself. Assuming a rapid assault that would last only a few weeks or months then yes, perhaps, a large stock of cans would be decisive in lieu of actual fuel supplies. However, anyone anticipating the “biggest challenge” would have probably considered campaigns getting bogged-down or stuck and contemplate future fuel origination options beyond a better container to move it around in.

It makes far more sense to me that some middling Nazi official was eager to solve a small and obvious part of logistics that they were focused on. There was a little fuel distribution problem, they saw it in 1935 or 1936 fascist invasions, and they set about a new can design. Even translating that into a massive pile or distribution of their cans does not equate to truly anticipating the major issues ahead.

I mean of course fuel did not pose the “biggest challenge” to taking over Europe.

This claim is so absurd I don’t even know where to begin. Put it in reverse perspective: having solved fuel supply alone would not have won the war for the Axis. It was not the single deciding factor. It was a factor among many, with the other factors often being far more in focus and difficult.

A Hitler “anticipation” theory simply does not fit with one of the greatest fuel blunders of all time, Operation Barbarossa, to violate borders to the East. Consider that in this operation more than 600,000 Nazi horses were relied upon in 1941.

Vehicle logistics totally failed. That’s right. HORSES.

There were absurd problems from lack of standardization, split and confused leadership and unrealistic (arguably insane) ideas of a “lightning” fast victory that quickly undermined an overstretched and flimsy Nazi supply chain doctrine. And this was after the 1940 “Blitz” against London already had failed its objectives despite America’s Standard Oil constantly re-fueling the bombers.

The simple fact of history is that from June to December 1941 the result of Nazism’s brutal stupidity was “half-starved and half-frozen; out of fuel and ammunition.”

Thus, Nazi leadership represents forever the exact opposite of anticipation and stocking up early. Blitz really translates into blundering into something without a plan and then committing suicide to avoid accountability. (See example two, above)


As Germany moved through Europe and North Africa, so did their thousands of gasoline cans. These cans proved to be dependable and durable; soon, countries all over the world were adapting them to haul and store liquids, coining them ‘Jerry cans’ because of their German origin (‘Jerry’ was a snide name for a German WWII soldier). New water container designs emerged but nothing could top the strength and simplicity of the original rectangular, X-marked Jerry can.


Obviously there were more than thousands of cans. The discovery of the Jerry can did not lead directly to adoption by the Allies. I sense some odd reverence for Nazis, even to the point of trying to apologize for “snide” names. Snide? Is this a concern without context? War against fascism, let alone against genocide, perhaps invites derision?

“Jerry” actually was a term used by Allies during WWI supposedly because the German helmet resembled a British jerry (chamber pot). In that sense a Jerry can is actually still a reference to its contents being toxic or at least unpotable.

As far as “new water container” designs I must again point out the original Jerry can also was used for water, with a designated stamp on the can to differentiate from fuel cans as mentioned above.

So with all that nonsense from Charity:water set aside, let me turn to an actual history of the yellow Jerry can. This is perhaps how I would update their page.


Jerry cans improved greatly upon prior cans, yet are quite simple in retrospect — better durability and portability. This can be explained with a couple short stories from the Allied perspective on winning WWII.


Paul Pleiss was an American engineer in Berlin who in 1936 had discovered a new can while planning to take a huge road trip (see part three of this series). He quickly realized its benefits first-hand. After his road trip, Pleiss spent the summer of 1939 to the summer of 1940 trying to convince the US military to adopt a new can.

American leadership was reluctant, without evidence or proof; they saw no need to alter current production. Only after Pleiss brought a can to show in person and demonstrate, and after the US considered field reports and shortcomings in their North Africa campaign (similar to the experiences of Italy during the 1935 invasion of Ethiopia) did the Jerry can come into better reception.

Things really shifted in 1942 when field qualitative reports backed by quantitative evidence showed US leaders that nearly half of fuel in Egypt was lost due to can failure. Despite sizable impacts while destroying fascists and freeing Africa, as recorded in desert battle outcomes in the preceding years (i.e. Wavell 1940, Auchinleck 1941, Montgomery 1942), measured data is what really hit home for the Americans.

…we sent a cable to naval officials in Washington stating that 40 percent of all the gasoline sent to Egypt was being lost through spillage and evaporation. We added that a detailed report would follow. The 40 percent figure was actually a guess intended to provoke alarm, but it worked. A cable came back immediately requesting confirmation.

So six years after Italy’s campaign in Ethiopia had led to German army equipment design changes, the US reached the same conclusions — fighting in North Africa needs a good fuel can.


The British appear to have ignored can design during the 1936-1939 innovation period. At the start of WWII hostilities a “flimsy” can prone to failure and mess was the UK standard. Still a better Jerry can design only came to light for them in the aftermath of French General Gamelin troops withering in 1940, leaving Britain alone to fight the Germans.

An over-extended and fragile but fast German blitzkreig had led to more careful British study and eventual realization that fuel portability had surely impacted performance. Another example, a similar study of the impact of new technology, was the use of radios by German tanks to update plans with “agile” development (peer communication) instead of waterfall (from the top).

The better containers meant much faster deployments. For example a can with a single handle is inferior to multiple handles when considering a line of soldiers trying to “bucket brigade”. Side handles meant two people could grab a can at the same time, or a single person could grab two with one hand. Faster can opening times mattered, as did less spillage during fuel transfer.

The German designer

Put the British and American realizations together and you get what I believe to have been the same thing that happened to the Germans in November 1936. An Italian invasion into northern Africa sparked the need for improvement, which then was tested during war in Spain.

Someone in Nazi Germany’s military administration invited Vinzenz Grünvogel of Müller to apply for a “Wehrmachtskanister” contract. Given the prior work of Müller with Ambi-Budd Presswerk (German for “pressed metal manufacturing”) the Jerry can method of manufacture probably was a derivative more than a novelty.

So it was with the 1936 Italian vehicles crossing rough African territory in mind that led to these specifications:


  • 465mm tall
  • 340mm wide
  • 20L capacity
  • 4kg dry weight
  • easy to stack
  • easy to manufacture (two plates pressed)
  • easy to carry (one soldier = two full, four empty) +
    (two soldiers = three for bucket brigade speed of transfer)


  • shock (recessed welds)
  • corrosion (synthetic lining)
  • float (air pocket “bump”)
  • pour (short spout)
  • seal (cam with lock)
  • expand (50deg max)

From the list and field experience it should be easy to see why the design has lasted.

Ultimately the cans were manufactured by dozens of companies subjected to Axis rule (Müller, Presswerke, Metalwerk, Nowack, Fischer, Schwelm, etc) and after 1942 by many other companies.

Symbols and markings

Lets go back to the idea of keeping people safe from toxic contents. As I mentioned the Germans stamped cans with “Wasser” (water) or “Kraftstoff” (fuel).

Despite a stamping process there also can be found a white W to indicate “winter” fuel (Winterkraftstoff) on later cans. This reiterates the importance of clear labeling to the original designers. It also points again to a lack of overall planning and preparation mentioned above (Hitler apparently refused to believe war would last into winter).

And that brings us to the creation of the yellow Jerry cans, a warning color for fuel. How should cans with different contents safely be identified? Is there a standard?

The answer is yes and no. Standards tend to evolve. Generally they have run something like this.


  1. Gasoline – Red
  2. Diesel – Yellow
  3. Drinking water (potable) – White
  4. Alt Fuels (Kerosene, JP Jet Fuel, Heli, M1 Meth, etc) – Blue
  5. Non-potable water – Green

Modern (e.g. 2005 California):

  1. Gasoline – red;
  2. Diesel – yellow; and
  3. Kerosene – blue

A typical set of Jerry can color options today:

jerry can colors


Does red look better with your shoes than green? Should we use colors for fashion sense not functional safety because of toxic chemicals?

As far as I can tell standards of color were centered on safety and clarity. Charity:Water uses yellow cans because fashion, and probably convenience, not because of grounded concerns about health and finding the best solutions. I mean has anyone studied the impact of using the correct color cans for water versus reinforcing use of yellow cans? Definitely did not find that on the charity site.

A water charity adopting a yellow can makes about as much sense to me as saying people in need drinking contaminated water should keep doing it because tradition. I’d just drop the color, if I were advising them. It is easy to switch a logo from solid yellow to white, especially since white cans conform to traditional safety standards.

Again, I want to be clear I am not opposed to change or redefinition of standards; here is a clever new white Jerry can:


My concern is with a charity pushing a global campaign that uses a dangerous/toxic liquid indicator as a symbol of clean water. Something seems odd about that decision.

Starting from my basic gut instinct it seems counter-productive to a charity objective to use confusing health/danger symbolism. This especially feels true for a charity that knows how to use imagery for power because they spend money to orchestrate images of smiling children. Moving to deeper analysis I found a very weak grasp of history, a whitewash of Hitler and the Nazis; this group asking for money may be seriously divorced from reality or real facts on the ground about social impact.

More on that…another day.

If you have made it this far (thanks!) you’re ready for a pop-quiz:

Given this typical image showing the various Jerry can colors…

…what word would you put after the word “charity”?

Feel free to put your answer in the comment section below.

Go back to part one or continue to part three in this series…

The story behind the Jerry can

Part one in a three-part series.

You may have heard a story about the Jerry can. Perhaps it goes something like Hitler was such a brilliant strategist that in 1936 he personally called forth an engineer to create a nearly perfect fuel can, which we still use to this day.

As a student of history I find this story nearly impossible to accept, not to mention as a humanist I find it a load of apologist nonsense about a genocidal maniac.

Why 1936, to begin with? Why did other countries take so long to follow? And how could Hitler’s grand supply-chain foresight three years before mobilizing for war with Poland fit into the many infamous Nazi fuel planning disasters that crippled the overall war effort?

No, Hitler wasn’t good at planning. No, Hitler wasn’t good at listening and adapting.

A more plausible story is that someone, probably a German soldier or mercenary assisting with Italian and Spanish fascist war campaigns in 1936, simply grew fed up with gas cans at a micro level. A WWI generation of fuel cans sucked for many reasons (couldn’t be stacked, leaked, couldn’t pour without a mess, couldn’t be carried in bulk).

I believe the archives should show this: from the summer to the fall of 1936, or maybe even earlier, German war management listened to field agents and decided something better was needed. Just like when the Nazis thought about putting radios in tanks for the first time, a decisive advantage in 1940, they also thought about motorized vehicle fuel supply.

It’s very likely some German soldier hated the inefficiency of the prior cans and borrowed or collaborated with Italian and Spanish fascists to find a better one. I see no evidence this can was meant to be a macro strategy for fuel supply management and plenty of evidence that Nazi fuel supply management overall was a disaster. The fact that a better can later was instrumental in battle outcomes was a reflection of grounded principles, not strategic thought.

And so an engineer won a Nazi contract to design a better can on some rather obvious theory of improving durability and portability to increase availability of fuel. Quality of engineering and manufacturing still was high in Germany at that time, despite emigrations and arrests of talent; so the Jerry can was born from a pressed metal factory preparing the Nazi war machine.

Some suggest the can was a military secret. Of course 1936 was full of secrecy to help with propaganda hiding the re-militarization. Hitler was a pathological liar who ran misinformation campaigns, playing a victim card over and over again, making technology secrecy essential. This was a factor.

Even more of a factor was a reluctance of Allies to listen to their field and incorporate feedback. Sending Sherman tanks into battle was an abject lesson in fail-faster because high casualties. Wasted fuel was harder to quantify. Unlike the Japanese however the Americans did adjust if they could see the need or advantage.

It turns out the Jerry can was discovered in 1939 by an American, even before hostilities, due to use at the Berlin airport (a German stole three and shared the technology). The delay in adoption by the Allies is related to their blindness to its value.

It took another four years because leadership of the Allies relied on statistical analysis and probably needed reports formatted with quantitative methods to make a change.

In 1942 an Allied soldier-chemist working on fuel logistics in northern Africa (e.g. facing similar things that soldiers in the Italian campaign of 1936 might have seen) converted qualitative field reports (e.g. old cans suck) into a statistics-based cable to Washington (e.g. we’re losing 40% of fuel before it even gets to the vehicle).

The lesson here is listening to qualitative field reports can inspire innovation in design, and quantitative analysis can show how small and simple changes in efficiency can make a major difference.

I am all ears if someone can find a memo from Hitler calling for a better fuel can and reasons to stockpile. It sounds a lot like revisionist theory to me, as if someone years from now would try to argue that President Obama anticipated social media growth and commissioned Facebook.

My guess based on data is an improved fuel containment design derived as a logical step from soldiers watching their losses during Spanish, Italian and Japanese fascist aggression.

Continue to part two or skip to three in this series…

Cyberwar revisionism: 2008 BTC pipeline explosion

Over on a site called “Genius” I’ve made a few replies to some other peoples’ comments on an old story: “Mysterious ’08 Turkey Pipeline Blast Opened New Cyberwar”

Genius offers the sort of experience where you have to believe a ton of pop-up scripts is a real improvement over plain text threads. Perhaps I don’t understand properly the value of markups and votes. So I’m re-posting my comments here in a more traditional text thread format, rather than using sticky-notes hovering over a story, not least of all because it’s easier for me to read and reference later.

Thinking about the intent of Genius, if there were an interactive interface I would rather see, the power of link-analysis and social data should be put into a 3D rotating text broken into paragraphs connected by lines from sources that you can spin through in 32-bit greyscale…just kidding.

But seriously if I have to click on every paragraph of text just to read it…something more innovative might be in order, more than highlights and replies. Let me know using the “non-genius boxes” (comment section) below if you have any thoughts on a platform you prefer.

Also I suppose I should introduce my bias before I begin this out-take of Genius (my text-based interpretation of their notation system masquerading as a panel discussion):

During the 2008 pipeline explosion I was developing critical infrastructure protection (CIP) tools to help organizations achieve reliability standards of the North American Electric Reliability Council (NERC). I’ve been hands-on since the mid-1990s in pen-tests and security architecture reviews for energy companies. I studied these events extensively at the time and after, and behind the scenes debriefed high-level people who spoke with media. I may not have had a PR campaign for myself or made public appearances on this before now, yet I still think that makes me someone “familiar with events”. Never sure what journalists mean by that phrase.

Bloomberg: Countries have been laying the groundwork for cyberwar operations for years, and companies have been hit recently with digital broadsides bearing hallmarks of government sponsorship.

Thomas Rid: Let’s try to be precise here — and not lump together espionage (exfiltrating data); wiping attacks (damaging data); and physical attacks (damaging hardware, mostly ICS-run). There are very different dynamics at play.

Me: Agree. Would prefer not to treat every disaster as an act of war. In the world of IT the boundary between operational issues and security events (especially because software bugs are so frequent) tends to be very fuzzy. When security want to investigate and treat every event as a possible attack it tends to have the effect of 1) shutting down/slowing commerce instead of helping protect it 2) reducing popularity and trust in security teams. Imagine a city full of roadblocks and checkpoints for traffic instead of streetlights and a police force that responds to accidents. Putting in place the former will have disastrous effects on commerce.
People use terms like sophisticated and advanced to spin up worry about great unknowns in security and a looming cyberwar. Those terms should be justified and defined carefully; otherwise basic operational oversights and lack of quality in engineering will turn into roadblock city.

Bloomberg: Sony Corp.’s network was raided by hackers believed to be aligned with North Korea, and sources have said JPMorgan Chase & Co. blamed an August assault on Russian cyberspies.

Thomas Rid: In mid-February the NYT (Sanger) reported that the JPMorgan investigation has not yielded conclusive evidence.

Mat Brown: Not sure if this is the one but here’s a recent Bits Blog post on the breach

Me: “FBI officially ruled out the Russian government as a culprit” and “The Russian government has been ruled out as sponsor”

Bloomberg: The Refahiye explosion occurred two years before Stuxnet, the computer worm that in 2010 crippled Iran’s nuclear-enrichment program, widely believed to have been deployed by Israel and the U.S.

Robert Lee: Sort of. The explosion of 2008 occurred two years before the world learned about Stuxnet. However, Stuxnet was known to have been in place years before 2010 and likely in development since around 2003-2005. Best estimates/public knowledge place Stuxnet at Natanz in 2006 or 2007.

Me: Robert is exactly right. Idaho National Labs held tests called “Aurora” (over-accelerating destroying a generator) on the morning of March 4, 2007. (
By 2008 it became clear in congressional hearings that NERC had provided false information to a subcommittee in Congress on Aurora mitigation efforts by the electric sector. Tennessee Valley Authority (TVA) in particular was called vulnerable to exploit. Some called for a replacement of NERC. All before Stuxnet was “known”.

Bloomberg: National Security Agency experts had been warning the lines could be blown up from a distance, without the bother of conventional weapons. The attack was evidence other nations had the technology to wage a new kind of war, three current and former U.S. officials said.

Robert Lee: Again, three anonymous officials. Were these senior level officials that would have likely heard this kind of information in the form of PowerPoint briefings? Or were these analysts working this specific area? This report relies entirely on the evidence of “anonymous” officials and personnel. It does not seem like serious journalism.

Me: Agree. Would like to know who the experts were, given we also saw Russia dropping bombs five days later. The bombs after the fire kind of undermines the “without the bother” analysis.

Bloomberg: Stuxnet was discovered in 2010 and this was obviously deployed before that.

Robert Lee: I know and greatly like Chris Blask. But Jordan’s inclusion of his quote here in the story is odd. The timing aspect was brought up earlier and Chris did not have anything to do with this event. It appears to be an attempt to use Chris’ place in the community to add value to the anonymous sources. But Chris is just making an observation here about timing. And again, this was not deployed before Stuxnet — but Chris is right that it was done prior to the discovery of Stuxnet.

Me: Yes although I’ll disagree slightly. As Aurora tests by 2008 were in general news stories, and congress was debating TVA insecurity and NERC ability to honestly report risk, Stuxnet was framed to be more unique and notable that it should have been.

Bloomberg: U.S. intelligence agencies believe the Russian government was behind the Refahiye explosion, according to two of the people briefed on the investigation.

Robert Lee: It’s not accurate to say that “U.S. intelligence agencies” believe something and then source two anonymous individuals. Again, as someone that was in the U.S. Intelligence Community it consistently frustrates me to see people claiming that U.S. intelligence agencies believe things as if they were all tightly interwoven, sharing all intelligence, and believing the same things.

Additionally, these two individuals were “briefed on the investigation” meaning they had no first hand knowledge of it. Making them non-credible sources even if they weren’t anonymous.

Me: Also interesting to note the August 28, 2008 Corner House analysis of the explosion attributed it to Kurdish Rebels (PKK). Yet not even a mention here?

[NOTE: I’m definitely going to leverage Robert’s excellent nuance statements when talking about China. Too often the press will try to paint a unified political picture, despite social scientists working to parse and explain the many different perspectives inside an agency, let alone a government or a whole nation. Understanding facets means creating better controls and more rational policy.]

Bloomberg: Although as many as 60 hours of surveillance video were erased by the hackers

Robert Lee: This is likely the most interesting piece. It is entirely plausible that the cameras were connected to the Internet. This would have been a viable way for the ‘hackers’ to enter the network. Segmentation in industrial control systems (especially older pipelines) is not common — so Internet accessible cameras could have given the intruders all the access they needed.

Me: I’m highly suspect of this fact from experience in the field. Video often is accidentally erased or disabled. Unless there is a verified chain of malicious destruction steps, it almost always is more likely to find surveillance video systems fragile, designed wrong or poorly run.

Bloomberg: …a single infrared camera not connected to the same network captured images of two men with laptop computers walking near the pipeline days before the explosion, according to one of the people, who has reviewed the video. The men wore black military-style uniforms without insignias, similar to the garb worn by special forces troops.

Robert Lee: This is where the story really seems to fall apart. If the hackers had full access to the network and were able to connect up to the alarms, erase the videos, etc. then what was the purpose of the two individuals? For what appears to be a highly covert operation the two individuals add an unnecessary amount of potential error. To be able to disable alerting and manipulate the process in an industrial control system you have to first understand it. This is what makes attacks so hard — you need engineering expertise AND you must understand that specific facility as they are all largely unique. If you already had all the information to do what this story is claiming — you wouldn’t have needed the two individuals to do anything. What’s worse, is that two men walking up in black jumpsuits or related type outfits in the middle of the night sounds more like engineers checking the pipeline than it does special forces. This happened “days before the explosion” which may be interesting but is hardly evidence of anything.

Me: TOTALLY AGREE. I will just add that earlier we were being told “blown up from a distance, without the bother of conventional weapons” and now we’re being told two people on the ground walking next to the pipeline. Not much distance there.

Bloomberg: “Given Russia’s strategic interest, there will always be the question of whether the country had a hand in it,” said Emily Stromquist, an energy analyst for Eurasia Group, a political risk firm based in Washington.

Robert Lee: Absolutely true. “Cyber” events do not happen in a vacuum. There is almost always geopolitical or economical interests at play.

Me: I’m holding off from any conclusion it’s a cyber event. And strategic interest to just Russia? That pipeline ran across how many conflict/war zones? There was much controversy during planning. In 2003 analysts warned that the PKK were highly likely to attack it.

Bloomberg: Eleven companies — including majority-owner BP, a subsidiary of the State Oil Company of Azerbaijan, Chevron Corp. and Norway’s Statoil ASA — built the line, which has carried more than two billion barrels of crude since opening in 2006.

Robert Lee: I have no idea how this is related to the infrared cameras. There is a lot of fluff entered into this article.

Me: This actually supports the argument that the pipeline was complicated both in politics and infrastructure, increasing risks. A better report would run through why BP planning would be less likely to result in disaster in this pipeline compared to their other disasters, especially given the complicated geopolitical risks.

Bloomberg: According to investigators, every mile was monitored by sensors. Pressure, oil flow and other critical indicators were fed to a central control room via a wireless monitoring system. In an extra measure, they were also sent by satellite.

Robert Lee: This would be correct. There is a massive amount of sensor and alert data that goes to any control center — pipelines especially — as safety is of chief importance and interruptions of even a few seconds in data can have horrible consequences.

Me: I believe it is more accurate to say every mile was designed to be monitored by sensors. We see quite clearly from investigations of the San Bruno, California disaster (killing at least 8 people) that documentation and monitoring of lines are imperfect even in the middle of an expensive American suburban neighborhood.

Bloomberg: The Turkish government’s claim of mechanical failure, on the other hand, was widely disputed in media reports.

Thomas Rid: A Wikileaks State Department cable refers to this event — by 20 August 2009, BP CEO Inglis was “absolutely confident” this was a terrorist attack caused by external physical force. I haven’t had the time to dig into this, but here’s the screenshot from the cable:
Thanks to @4Dgifts

Me: It may help to put it into context of regional conflict at that time. Turkey started Operation Sun (Güneş Harekatı) attacking the PKK, lasting into March or April. By May the PKK had claimed retaliation by blowing up a pipeline between Turkey and Iran, which shutdown gas exports for 5 days ( We should at least answer why BTC would not be a follow-up event.
And there have been several explosions since then as well, although I have not seen anyone map all the disasters over time. Figure an energy market analyst must have done one already somewhere.
And then there’s the Turkish news version of events: “Turkish official confirms BTC pipeline blast is a terrorist act”

Thomas Rid: Thanks — Very useful!

Bloomberg: “We have never experienced any kind of signal jamming attack or tampering on the communication lines, or computer systems,” Sagir said in an e-mail.

Robert Lee: This whole section seems to heavily dispute the assumption of this article. There isn’t really anything in the article presented to dispute this statement.

Me: Agree. The entire article goes to lengths to make a case using anonymous sources. Mr. Sagir is the best source so far and says there was no tampering detected. Going back to the surveillance cameras, perhaps they were accidentally erased or non-functioning due to error.

Bloomberg: The investigators — from Turkey, the U.K., Azerbaijan and other countries — went quietly about their business.

Robert Lee: This is extremely odd. There are not many companies who have serious experience with incident response in industrial control system scenarios. Largely because industrial control system digital forensics and incident response is so difficult. Traditional information technology networks have lots of sources of forensic data — operations technology (industrial control systems) generally do not.

The investigators coming from one team that works and has experience together adds value. The investigators coming from multiple countries sounds impressive but on the ground level actually introduces a lot of confusion and conflict as the teams have to learn to work together before they can even really get to work.

Me: Agree. The pipeline would see not only confusion in the aftermath, it also would find confusion in the setup and operation, increasing chance of error or disaster.

Bloomberg: As investigators followed the trail of the failed alarm system, they found the hackers’ point of entry was an unexpected one: the surveillance cameras themselves.

Robert Lee: How? This is a critical detail. As mentioned before, incident response in industrial control systems is extremely difficult. The Industrial Control System — Computer Emergency Response Team (ICS-CERT) has published documents in the past few years talking about the difficulty and basically asking the industry to help out. One chief problem is that control systems usually do not have any ability to perform logging. Even in the rare cases that they do — it is turned off because it uses too much storage. This is extremely common in pipelines. So “investigators” seem to have found something but it is nearly outside the realm of possible that it was out in the field. If they had any chance of finding anything it would have been on the Windows or Linux systems inside the control center itself. The problem here is that wouldn’t have been the data needed to prove a failed alarm system.

It is very likely the investigators found malware. That happens a lot. They likely figured the malware had to be linked to blast. This is a natural assumption but extremely flawed based on the nature of these systems and the likelihood of random malware to be inside of a network.

Me: Agree. Malware noticed after disaster becomes very suspicious. I’m most curious why anyone would setup surveillance cameras for “deep into the internal network” access. Typically cameras are a completely isolated/dedicated stack of equipment with just a browser interface or even dedicated monitors/screens. Strange architecture.

Bloomberg: The presence of the attackers at the site could mean the sabotage was a blended attack, using a combination of physical and digital techniques.

Robert Lee: A blended cyber-physical attack is something that scares a lot of people in the ICS community for good reason. It combines the best of two attack vectors. The problem in this story though is that apparently it was entirely unneeded. When a nation-state wants to spend resources and talents to do an operation — especially when they don’t want to get caught — they don’t say “let’s be fancy.” Operations are run in the “path of least resistance” kind of fashion. It keeps resource expenditures down and keeps the chance of being caught low. With everything discussed as the “central element of the attack” it was entirely unneeded to do a blended attack.

Me: What really chafes my analysis is that the story is trying to build a scary “entirely remote attack” scenario while simultaneously trying to explain why two people are walking around next to the pipeline.
Also agree attackers are like water looking for cracks. Path of least resistance.

Bloomberg: The super-high pressure may have been enough on its own to create the explosion, according to two of the people familiar with the incident.

Robert Lee: Another two anonymous sources.

Me: And “familiar with the incident” is a rather low bar.

Bloomberg: Having performed extensive reconnaissance on the computer network, the infiltrators tampered with the units used to send alerts about malfunctions and leaks back to the control room. The back-up satellite signals failed, which suggested to the investigators that the attackers used sophisticated jamming equipment, according to the people familiar with the probe.

Robert Lee: If the back-up satellite signal failed in addition to alerts not coming from the field (these units are polled every few seconds or minutes depending on the system) there would have been an immediate response from the personnel unless they were entirely incompetent or not present (in that case this story would be even less likely). But jamming satellite links is an even extra level of effort beyond hacking a network and understanding the process. If this was truly the work of Russian hackers they are not impressive for all the things they accomplished — they were embarrassingly bad at how many resources and methods they needed to accomplish this attack when they had multiple ways of accomplishing it with any one of the 3-4 attack vectors.

Me: Agree. The story reads to me like conventional attack, known to be used by PKK, causes fire. Then a series of problems in operations are blamed on super-sophisticated Russians. “All these systems not working are the fault of elite hackers”

Bloomberg: Investigators compared the time-stamp on the infrared image of the two people with laptops to data logs that showed the computer system had been probed by an outsider.

Robert Lee: “Probed by an outsider” reveals the system to be an Internet connected system. “Probes” is a common way to describe scans. Network scans against publicly accessible devices occur every second. There is a vast amount of research and public information on how often Internet scans take place (usually a system begins to be scanned within 3-4 seconds of being placed online). It would have been more difficult to find time-stamps in any image that did not correlate to probing.

Me: Also is there high trust in time-stamps? Accurate time is hard. Looking at the various scenarios (attackers had ability to tamper, operations did a poor job with systems) we should treat a time-stamp-based correlation as how reliable?

Bloomberg: Years later, BP claimed in documents filed in a legal dispute that it wasn’t able to meet shipping contracts after the blast due to “an act of terrorism.”

Robert Lee: Which makes sense due to the attribution the extremists claimed.

Me: I find this sentence mostly meaningless. My guess is BP was using legal or financial language because of the constraints in court. Would have to say terrorism, vandalism, etc. to speak appropriately given precedent. No lawyer wants to use a new term and establish new norms/harm when they can leverage existing work.

Bloomberg: A pipeline bombing may fit the profile of the PKK, which specializes in extortion, drug smuggling and assaults on foreign companies, said Didem Akyel Collinsworth, an Istanbul-based analyst for the International Crisis Group. But she said the PKK doesn’t have advanced hacking capabilities.

Robert Lee: This actually further disproves the article’s theory. If the PKK took credit, the company believed it to be them, the group does not possess hacking skills, and specialists believe this attack was entirely their style — then it was very likely not hacking related.

Me: Agree. Wish this pipeline explosion would be put in context of other similar regional explosions, threats from the PKK that they would attack pipelines and regional analyst warnings of PKK attacks.

Bloomberg: U.S. spy agencies probed the BTC blast independently, gathering information from foreign communications intercepts and other sources, according to one of the people familiar with the inquiry.

Robert Lee: I would hope so. There was a major explosion in a piece of critical infrastructure right before Russia invaded Georgia. If the intelligence agencies didn’t look into it they would be incompetent.

Me: Agree. Not only for defense, also for offense knowledge, right? Would be interesting if someone said they probed it differently than the other blasts, such as the one three months earlier between Turkey and Iran.

Bloomberg: American intelligence officials believe the PKK — which according to leaked State Department cables has received arms and intelligence from Russia — may have arranged in advance with the attackers to take credit, the person said.

Robert Lee: This is all according to one, yet again, anonymous source. It is extremely far fetched. If Russia was going to go through the trouble of doing a very advanced and covert cyber operation (back in 2008 when these types of operations were even less publicly known) it would be very out of character to inform an extremist group ahead of time.

Me: Agree, although also plausible to tell a group a pipeline would be blown up without divulging method. Then the group claims credit without knowing method. The disconnect I see is Russia trying to bomb the same pipeline five days later. Why go all conventional if you’ve owned the systems and can remotely do what you like?

Bloomberg: The U.S. was interested in more than just motive. The Pentagon at the time was assessing the cyber capabilities of potential rivals, as well as weaknesses in its own defenses. Since that attack, both Iran and China have hacked into U.S. pipeline companies and gas utilities, apparently to identify vulnerabilities that could be exploited later.

Robert Lee: The Pentagon is always worried about these types of things. President Clinton published PDD-63 in 1998 talking about these types of vulnerabilities and they have been assessing and researching at least since then. There is also no evidence provided about the Iranian and Chinese hacks claimed here. It’s not that these types of things don’t happen — they most certainly do — it’s that it’s not responsible or good practice to cite events because “we all know it’s happening” instead of actual evidence.

Me: Yes, explaining major disasters already happening and focus of congressional work (2008 TVA) would be a better perspective on this section. August 2003 was a sea change in bulk power risk assessment. Talking about Iran and China seems empty/idle speculation in comparison:

Bloomberg: As tensions over the Ukraine crisis have mounted, Russian cyberspies have been detected planting malware in U.S. systems that deliver critical services like electricity and water, according to John Hultquist, senior manager for cyber espionage threat intelligence at Dallas-based iSight Partners, which first revealed the activity in October.

Robert Lee: It’s not that I doubt this statement, or John, but this is another bad trend in journalism. Using people that have a vested interest in these kind of stories for financial gain is a bad practice in the larger journalism community. iSight Partners offer cybersecurity services and specialize in threat intelligence. So to talk about ‘cyberspies’, ‘cyber espionage’, etc. is something they are financially motivated to hype up. I don’t doubt the credibility or validity of John’s statements but there’s a clear conflict of interest that shouldn’t be used in journalism especially when there are no named sources with first-hand knowledge on the event.

Me: Right! Great point Robert. Reads like free advertising for threat intelligence company X rather than trusted analysis. Would mind a lot less if a non-sales voice was presented with a dissenting view, or the journalist added in caution about the source being of a particular bias.
Also what’s the real value of this statement? As a crisis with Russia unfolds, we see Russia being more active/targeted. Ok, but what does this tell us about August 2008? No connection is made. Reader is left guessing.

Bloomberg: The keyboard was the better weapon.

Robert Lee: The entire article is focused on anonymous sources. In addition, the ‘central element of the attack’ was the computer intrusion which was analyzed by incident responders. Unfortunately, incident response in industrial control systems is at such a juvenile state that even if there were a lot of data, which there never is, it is hard to determine what it means. Attribution is difficult (look at the North Korea and Sony case where much more data was available including government level resources). This story just doesn’t line up.

When journalism reports on something it acknowledges would be history changing better information is needed. When those reports stand to increase hype and tension between nation-states in already politically tense times (Ukraine, Russia, Turkey, and the U.S.). Not including actual evidence is just irresponsible.

Me: Agree. It reads like a revision of history, so perhaps that’s why we’re meant to believe it’s “history changing.” I’m ready to see evidence of a hack yet after six years we have almost nothing to back up these claims. There is better detail about what happened from journalists writing at the time.
Also if we are to believe the conclusion that keyboards are the better weapon, why have two people walking along the pipeline and why bomb infrastructure afterwards? Would Russia send a letter after making a phone call? I mean if you look carefully at what Georgia DID NOT accuse Russia of it was hacking critical infrastructure.
Lack of detailed evidence, anonymous attribution, generic/theoretical vulnerability of infrastructure statements, no contextual explanations…there is little here to believe the risk was more than operational errors coupled with PKK targeted campaign against pipelines.

Was Stuxnet the “First”?

My 2011 presentation on Stuxnet was meant to highlight a few basic concepts. Here are two:

  • Sophisticated attacks are ones we are unable to explain clearly. Spoons are sophisticated to babies. Spoons are not sophisticated to long-time chopstick users. It is a relative measure, not an absolute one. As we increase our ability to explain and use things they become less sophisticated to us. Saying something is sophisticated really is to communicate that we do not understand it, although that may be our own fault.
  • Original attacks are ones we have not seen before. It also is a relative measure, not an absolute one. As we spend more time researching and observing things, fewer things will be seen as original. In fact with just a little bit of digging it becomes hard to find something completely original rather than evolutionary or incremental. Saying something is original therefore is to say we have not seen anything like it before, although that may be our own fault.

Relativity is the key here. Ask yourself if there is someone to easily discuss attacks with to make them less sophisticated and less original. Is there a way to be less in awe and more understanding? It’s easy to say “oooh, spoon” and it should not be that much harder to ask “anyone seen this thing before?”

Here’s a simple thought exercise:

Given that we know critical infrastructure is extremely poorly defended. Given that we know control systems are by design simple. Would an attack designed for simple systems behind simple security therefore be sophisticated? My argument is usually no, that by design the technical aspects of compromise tend to be a low-bar…perhaps especially in Iran.

Since the late 1990s I have been doing assessments inside utilities and I have not yet found one hard to compromise. However, there still is a sophisticated part, where research and skills definitely are required. Knowing exactly how to make an ongoing attack invisible and getting the attack specific to a very intended result, that is a level above getting in and grabbing data or even causing harm.

An even more advanced attack makes trace/tracks of attack invisible. So there definitely are ways to bring sophistication and uniqueness level up substantially from “oooh, spoon” to “I have no idea if that was me that just did that”. I believe this has become known as the Mossad-level attack, at which point defense is not about technology.

I thought with my 2011 presentation I could show how a little analysis makes major portions of Stuxnet less sophisticated and less original; certainly it was not the first of its kind and it is arguable how targeted it was as it spread.

The most sophisticated aspects to me were in that it was moving through many actors across boundaries (e.g. Germany, Iran, Pakistan, Israel, US, Russia) requiring knowledge inside areas not easily accessed or learned. Ok, let’s face it. It turns out that thinking was on the right path, albeit an important role was backwards and I wasn’t sure where it would lead.

A US ex-intel expert mentioned on Twitter during my talk I had “conveniently” ignored motives. This is easy for me to explain: I focus on consequences as motive is basically impossible to know. However, as a clue that comment was helpful. I wasn’t thinking hard enough about the economic-espionage aspect that US intelligence agencies have revealed as a motivator. Recent revelations suggest the US was angry at Germany allowing technology into Iran. I had mistakenly thought Germany would have been working with the US, or Israel would have been able to pressure Germany. Nope.

Alas a simple flip of Germany’s role (critical to good analysis and unfortunately overlooked by me) makes far more sense because they (less often but similar to France) stand accused of illicit sales of dangerous technology to US (and friend of US) enemies. It also fits with accusations I have heard from US ex-intel expert that someone (i.e. Atomstroyexport) tipped-off the Germans, an “unheard of” first responder to research and report Stuxnet. The news cycles actually exposed Germany’s ties to Iran and potentially changed how the public would link similar or follow-up action.

But this post isn’t about the interesting social science aspects driving a geopolitical technology fight (between Germany/Russia and Israel/US over Iran’s nuclear program), it’s about my failure to make an impression enough to add perspective. So I will try again here. I want to address an odd tendency of people to continue to report Stuxnet as the first ever breach of its type. This is what the BSI said in their February 2011 Cyber Security Strategy for Germany (page 3):

Experience with the Stuxnet virus shows that important industrial infrastructures are no longer exempted from targeted IT attacks.

No longer exempted? Targeted attacks go back a long way as anyone familiar with the NIST report on the 2000 Maroochy breach should be aware.

NIST has established an Industrial Control System (ICS) Security Project to improve the security of public and private sector ICS. NIST SP 800-53 revision 2, December 2007, Recommended Security Controls for Federal Information Systems, provides implementing guidance and detail in the context of two mandatory Federal Information Processing Standards (FIPS) that apply to all federal information and information systems, including ICSs.

Note an important caveat in the NIST report:

…”Lessons Learned From the Maroochy Water Breach” refer to a non-public analytic report by the civil engineer in charge of the water supply and sewage systems…during time of the breach…

These non-public analytic reports are where most breach discussions take place. Nonetheless, there never was any exemption and there are public examples of ICS compromise and damage. NIST gives Maroochy from 2000. Here are a few more ICS attacks to consider and research:

  • 1992 Portland/Oroville – Widespread SCADA Compromise, Including BLM Systems Managing Dams for Northern California
  • 1992 Chevron – Refinery Emergency Alert System Disabled
  • 1992 Ignalina, Lithuania – Engineer installs virus on nuclear power plant ICS
  • 1994 Salt River – Water Canal Controls Compromised
  • 1999 Gazprom – Gas Flow Switchboard Compromised
  • 2001 California – Power Distribution Center Compromised
  • 2003 Davis-Besse – Nuclear Safety Parameter Display Systems Offline
  • 2003 Amundsen-Scott – South Pole Station Life Support System Compromised
  • 2003 CSX Corporation – Train Signaling Shutdown
  • 2006 Browns Ferry – Nuclear Reactor Recirculation Pump Failure
  • 2007 Idaho Nuclear Technology & Engineering Complex (INTEC) – Turbine Failure
  • 2008 Hatch – Contractor software update to business system shuts down nuclear power plant ICS
  • 2009 Carrell Clinic – Hospital HVAC Compromised
  • 2013 Austria/Germany – Power Grid Control Network Shutdown

Fast forward to December 2014 and a new breach case inside Germany comes out via the latest BSI report. It involves ICS so the usual industry characters start discussing it.

Immediately I tweet for people to take in the long-view, the grounded-view, on German BSI reports.

Alas, my presentation in 2011 with a history of breaches and my recent tweets clearly failed to sway, so I am here blogging again. I offer as example of my failure the following headlines that really emphasize a “second time ever” event.

That list of four in the last article is interesting. Sets it apart from the other two headlines, yet it also claims “and only the second confirmed digital attack”? That’s clearly a false statement.

Anyway Wired appears to have crafted their story in a strangely similar fashion to another site; perhaps too similar to a Dragos Security blog post a month earlier (same day as the BSI tweets above).

This is only the second time a reliable source has publicly confirmed physical damage to control systems as the result of a cyber-attack. The first instance, the malware Stuxnet, caused damage to nearly 3,000 centrifuges in the Natanz facility in Iran. Stories of damage in other facilities have appeared over the years but mostly based on tightly held rumors in the Industrial Control Systems (ICS) community that have not been made public. Additionally there have been reports of companies operating in ICS being attacked, such as the Shamoon malware which destroyed upwards of 30,000 computers, but these intrusions did not make it into the control system environment or damage actual control systems. The only other two widely reported stories on physical damage were the Trans-Siberian-Pipeline in explosion in 1982 and the BTC Turkey pipeline explosion in 2008. It is worth noting that both stories have come under intense scrutiny and rely on single sources of information without technical analysis or reliable sources. Additionally, both stories have appeared during times where the reporting could have political motive instead of factuality which highlights a growing concern of accurate reporting on ICS attacks. The steelworks attack though is reported from the German government’s BSI who has both been capable and reliable in their reporting of events previously and have the access to technical data and first hand sources to validate the story.

Now here is someone who knows what they are talking about. Note the nuance and details in the Dragos text. So I realize my problem is with a Dragos post regurgitated a month later by Wired without attribution because look at how all the qualifiers disappeared in translation. Wired looks preposterous compared to this more thorough reporting.

The Dragos opening line is a great study in how to setup a series of qualifications before stepping through them with explanations:

This is only the second time a reliable source has publicly confirmed physical damage to control systems as the result of a cyber-attack

The phrase has more qualifications than Lance Armstrong:

  • Has to be a reliable source. Not sure who qualifies that.
  • Has to be publicly confirmed. Does this mean a government agency or the actual victim admitting breach?
  • Has to be physical damage to control systems. Why control systems themselves, not anything controlled by systems? Because ICS security blog writer.
  • Has to result from cyber-attack. They did not say malware so this is very broad.

Ok, Armstrong had more than four… Still, the Wired phrase by comparison uses dangerously loose adaptations and drops half. Wired wrote “This is only the second confirmed case in which a wholly digital attack caused physical destruction of equipment” and that’s it. Two qualifications instead of four.

So we easily can say Maroochy was a wholly digital attack that caused physical destruction of equipment. We reach the Wired bar without a problem. We’d be done already and Stuxnet proved to not be the first.

Dragos is harder. Maroochy also was from a reliable source, publicly confirmed resulting from packet-radio attack (arguably cyber). Only thing left here is physical damage to control systems to qualify. I think the Dragos bar is set oddly high to say the control systems themselves have to be damaged. Granted, ICS management will consider ICS damage differently than external harms; this is true in most industries, although you would expect it to be the opposite in ICS. To the vast majority, news of 800,000 released liters of sewage obviously qualifies as physical damage. So Maroochy would still qualify. Perhaps more to the point, the BSI report says the furnace was set to an unknown state, which caused breakdown. Maroochy had its controls manipulated to an unknown state, albeit not damaging the controls themselves.

If anyone is going to hang their hat on damage to control systems, the perhaps they should refer to it as an Aurora litmus, given the infamous DHS study of substations in 2007 (840pg PDF).


The concern with Aurora, if I understood the test correctly, was not to just manipulate the controls. It was to “exploit the capability of modern protective equipment and cause them to serve as a destructive weapon”. In other words, use the controls that were meant to prevent damage to cause widespread damage instead. Damage to just controls themselves without wider effect would be a premature end to a cyber-physical attack, albeit a warning.

I’d love to dig into that BTC Turkey pipeline explosion in 2008, since I worked on that case at the time. I agree with the Dragos blog it doesn’t qualify, however, so I have to move on. Before I do, there is an important lesson from 2008.

Suffice it to say I was on press calls and I gave clear and documented evidence to those interviewed about cyber attack on critical infrastructure. For example, the Georgia official complaint listed no damage related to cyber attack. The press instead ran a story, without doing any research, using hearsay that Russia knocked the Georgian infrastructure off-line with cyber attack. That often can be a problem with the press and perhaps that is why I am calling Wired out here for their lazy title.

Let’s look at another example, the 2007 TCAA, from a reliable source, publicly confirmed, causing damage to control systems, caused by cyber-attack:

Michael Keehn, 61, former electrical supervisor with Tehama Colusa Canal Authority (TCAA) in Willows, California, faces 10 years in prison on charges that he “intentionally caused damage without authorization to a protected computer,” according to Keehn’s November 15 indictment. He did this by installing unauthorized software on the TCAA’s Supervisory Control and Data Acquisition (SCADA) system, the indictment states.

Perfect example. Meets all four criteria. Sounds bad, right? Aha! Got you.

Unfortunately this incident turns out to be based only an indictment turned into a news story, repeated by others without independent research. Several reporters jumped on the indictment, created a story, and then moved on. Dan Goodin probably had the best perspective, at least introducing skepticism about the indictment. I put the example here not only to trick the reader, but also to highlight how seriously I take the question of “reliable source”.

Journalists often unintentionally muddy waters (pun not intended) and mislead; they can move on as soon as the story goes cold. What stake do they really have when spinning their headline? How much accountability do they hold? Meanwhile, those of us defending infrastructure (should) keep digging for truth in these matters, because we really need it for more than talking point, we need to improve our defenses.

I’ve read the court documents available and they indicate a misunderstanding about software developer copyright, which led to a legal fight, all of which has been dismissed. In fact the accused wrote a book afterwards called “Anatomy of a Criminal Indictment” about how to successfully defend yourself in court.

In 1989 he applied for a job with the Tehama-Colusa Canal Authority, a Joint Powers Authority who operated and maintained two United States Bureau of Reclamation canals. During his tenure there, he volunteered to undertake development of full automated control of the Tehama-Colusa Canal, a 110-mile canal capable of moving 2,000 cfs (cubic feet of water per second). It was out of this development for which he volunteered to undertake, that resulted in a criminal indictment under Title 18, Part I, Chapter 47, Section 1030 (Fraud and related activity in connection with computers). He would be under indictment for three years before the charges were dismissed. During these three years he was very proactive in his own defense and learned a lot that an individual not previously exposed would know about. The defense attorney was functioning as a public defender in this case, and yet, after three years the charges were dismissed under a motion of the prosecution.

One would think reporters would jump on the chance to highlight the dismissal, or promote the book. Sadly the only news I find is about the original indictment. And so we still find the indictment listed by information security references as an example of ICS attack, even though it was not. Again, props to Dragos blog for being skeptical about prior events. I still say, aside from Maroochy, we can prove Stuxnet not the first public case.

The danger in taking the wide-view is that it increases the need to understand far more details and do more deep research to avoid being misled. The benefit, as I pointed out at the start, is we significantly raise the bar for what is considered sophisticated or original attacks.

In my experience Stuxnet is a logical evolution, an application of accumulated methods within a context already well documented and warned about repeatedly. I believe putting it back in that context makes it more accessible to defenders. We need better definitions of physical damage and cyber, let alone reputable sources, before throwing around firsts and seconds.

Yes malware that deviates from normal can be caught, even unfamiliar malware, if we observe and respond quickly to abnormal behavior. Calling Stuxnet the “first” will perhaps garner more attention, which is good for eyeballs on headlines. However it also delays people from realizing how it fits a progression; is the adversary introducing never-seen-before tools and methods or are they just extremely well practiced with what we know?

The latest studies suggest how easy, almost trivial, it would be to detect Stuxnet for security analysts monitoring traffic as well as operations. Regardless of the 0day, the more elements of behavior monitored the higher the attacker has to scale. Companies like ThetaRay have been created on this exact premise, to automate and reduce the cost of the measures a security analyst would use to protect operations. (Already a crowded market)

That’s the way I presented it in 2011 and little has changed since then. Perhaps the most striking attempt to make Stuxnet stand out that I have heard lately was from ex-USAF staff; paraphrasing him, Stuxnet was meant to be to Iran what the atom bomb was to Japan. A weapon of mass-destruction to change the course of war and be apologized for later.

It would be interesting if I could find myself able to agree with that argument. I do not. But if I did agree, then perhaps I could point out in recent research, based on Japanese and Russian first-person reports, the USAF was wrong about Japan. Fear of nuclear assault, let alone mass casualties and destruction from the bombs, did not end the war with Japan; rather leadership gave up hope two days after the Soviets entered the Pacific Theater. And that should make you wonder really about people who say we should be thankful for the consequences of either malware or bombs.

But that is obviously a blog post for another day.

Please find below some references for further reading, which all put Stuxnet in broad context rather than being the “first”:

N. Carr, Development of a Tailored Methodology and Forensic Toolkit for Industrial Control Systems Incident Response, US Naval Postgraduate School 2014

A. Nicholson; S. Webber; S. Dyer; T. Patel; H. Janicke, SCADA security in the light of Cyber-Warfare 2012

C. Wueest, Targeted Attacks Against the Energy Sector, Symantec 2014

B. Miller; D. Rowe, A Survey of SCADA and Critical Infrastructure Incidents, SIGITE/RIIT 2012

C. Baylon; R. Brunt; D. Livingstone, Cyber Security at Civil Nuclear Facilities, Chatham House 2015