You may remember the huge kerfuffle that Rackspace caused among the security community last year. Alison Gianotto, also known for cranky haikus, captured the essence of the problems an open letter to Rackspace Hosting.

And thanks to your logfiles not being able to be viewed in real time (as they are owned by root), this leaves web developers that actually have a clue very few options for forensically backtracking the vector.

I would like to know what Rackspace is doing to help developers isolate these issues? Are logfiles being programmatically reviewed for malicious traffic? Without SSH access and the ability to tail apache logs, we cannot do this ourselves within any kind of timeframe that will be useful in preventing or mitigating an attack. If I am going to continue hosting with Rackspace, I want to be assured that Rackspace is actually doing something to help us protect ourselves other than send emails that overstate the obvious.

Your support staff, at least most of the level 1 techs, are completely and utterly incapable of handling anything relating to hacks. They are slow and under-educated, regardless of how well meaning they might be.

Lack of transparency and lack of talent. Harsh words but it comes straight to the point of trust in a provider will only get you so far before you need to step in and verify that they have the security capabilities you need.

I bring this up as FreshBooks recently spammed me with a “we’re secure” message, which created the following thread with a comical ending. First, here’s the excerpt from their message that caught my attention.

We want you to save time every month by using FreshBooks so you can focus on what you love to do. […] If you…need a nudge, here are some nuggets:

If you’re thinking: “I don’t know if my data is safe on the cloud”

We’d suggest: FreshBooks takes extra steps to ensure your data is kept secure. Having your data in the cloud makes sure it’s always safe and accessible (from anywhere).

Ok, well done. I’m paying attention to a message I would have otherwise tossed into the spam bucket. I wrote a quick reply.

My concern is with security/compliance. What are the extra steps?

I received a response from someone with this signature

xxxxxx from FreshBooks
(very) Small Business Consultant

I suspect the “(very)” is supposed to be humorous. It would be much more humorous if they put “non-VIP”, “n00b” or perhaps even “peasant” in their sig to reinforce a lack of support I should expect. Howdy, I have been assigned to your really tiny and unimportant issues. Now, how may I be of (very little) help? Hilarious.

Here is the actual response they sent me:

I’m not sure I understand. Extra steps to what, exactly? Are you talking about PCI compliance, or the security we have on our servers, or?

Yes, I actually was talking about or. What are the extra steps to or? But that is not what I responded. Instead I simply wrote the following reply to try to get back to their original statement in the email they sent me:

Hi, I was just quoting your email message. I don’t know what steps you meant.

That seemed to help as they then sent back the following response with URLs

Ah, I understand. You can see our security measures here: http://www.freshbooks.com/security-safeguards.php

We also use RackSpace for our server hosting, and you can see their info here: http://www.rackspace.com/

I hope this helps! Let me know if there is anything else I can help with :)

The rackspace URL is the generic front-page. Not a good sign, per the start of this post. I asked about extra steps. So I dig into the Freshbooks security page and it raises far more questions than answers. Here are some examples:

Any unusual behaviour is analyzed by AlertLogic’s CISSP-certified security experts, and responses are coordinated between them, Rackspace, and our system administration team.

Odd. They hold up the CISSP as a qualification for monitoring network traffic? I find that discouraging — indicates a lack of understanding about both the CISSP certification and network monitoring. Responses are coordinated by their system administration team, which suggests no security team. That would explain why they have to delegate. Still looking for the extra steps.

Particularly sensitive information – credit card numbers, bank account information, and your payment gateway account details – are encrypted in our database using AES.

Who gets the keys? How are keys setup and managed? Nothing extra here either. So little information on such a critical issue reads like a Drobbox catastrophe just waiting to happen. Speaking of lessons learned, I then read this section:

FreshBooks has chosen Rackspace for our hosting needs. With clients like General Electric, Hershey, Cisco, Pfizer, EMI Music, Scott’s, Hilton, Sony Music, Columbia House and the US Marines, we know Rackspace provides the hardware, service and expertise you expect.

What are the chances that FreshBooks is going to be able to get good customer support/service while stuck behind a list of giants like Sony who are probably taking up every minute of Rackspace support time during their breaches?

And what are the chances that FreshBooks will be adequately protected from a mess like Sony? Have they verified segmentation? Transparency comes directly to mind. So, of course, I had to ask for clarification again but by this point I confess I was losing patience in finding any extra steps, which their original spam promised me.

your page does not mention compliance standards or third party assessments. are there any? CISSP-certification does not mean anything for analysis of vulnerabilities or threats. it is a general knowledge test, like a bachelor degree does not mean you are qualified to be a doctor.

rackspace disallows physical audits of their datacenter. how do you verify their security? the list of their clients only means you are all going to be competing for lifeboats when that ship sinks, not that it is well run. have you had any audits of your equipment there?

Then came the reply, short and to the point, which confirmed to me that there are no extra steps. I could even make the case that their security page is lacking important details and so they are in fact missing steps. They delegate their security and they simply hope that you will too. Here is their reply:

I spoke to my IT team about your questions, and I’ll quote a response: “If they don’t trust RackSpace, then they probably don’t want to use us”.

Doesn’t look like we’ll be the right fit for you. Better to find out earlier than later :)

Good thing I asked. Thought others might want to know. And with a nod to Alison Gianotto, here is my cranky haiku:

Freshbooks to Davi;
Security extra steps
can’t be verified

Update: An old video has surfaced that shows a trivial exploit of FreshBooks. The attacker logs in as a client who received an invoice and then deletes the invoice simply by changing the SetAction “print” command to “delete” in their browser.