AWS Security Lessons Learned

My Story

Once upon a time not too long ago, I had a project to transport data in and out of S3 using Java. Things started off rather smoothly. Within about 30 minutes I had my Amazon Web Services (AWS) account active, had the Java SDK pulled into Eclipse through Maven and was testing my first AWS S3 API calls. Everything was fine except I embedded my AWS access key in the code! I spent a few hours prototyping, testing API calls and understanding how I wanted to structure the code for this project. Along the way I got side-tracked, switched to working on other things and eventually came back to my AWS S3 project. At this point I had a pretty good prototype for structuring data in S3 as well as getting data in and out. I wanted to save my work and had forgotten all about the fact that my AWS access key was still embedded in the code. I created a public Github repo and checked in the code, it had been a long day and was about to get much longer...

A few hours went by and I got a call from Amazon Web Services, there had been suspicious activity on my account. At this point I started to realise what might have happened and I rushed home. Upon logging in I found 20 XXL EC2 instances running in every AWS region and in just a few hours my bill was already over $700 USD. I couldn't believe it and thought why on Earth would anyone want to startup EC2 instances? I am just going to delete them, what was the purpose of this?

I quickly shutdown all the EC2 instances and expunged them, except one. I wanted to take a closer look and see what was actually running in the instance. Upon gaining access I looked at the process table and saw to my surprise Bitcoin processes running and chewing on the CPU. Bitcoin!!! But of course, it was now so clear, I provided some happy miners free CPU for a few hours and they took full advantage of my stupidity. This was the first time I had ever been hacked or had my security (knowingly) breeched. I felt vulnerable, foolish, and extremely frustrated.

Finally I phoned Amazon Web Services and explained what happened. They confirmed and told me that this happens all too often and this kind of theft is on the rise. Bitcoin miners scrub all code sites like Github and search for patterns like "AWS Secret Key". If they find them, they will try to deploy Bitcoin servers on unsuspecting accounts and why not it is easy money, right? In a quick Github search I found more keys than I knew what to do with, very disturbing. I tried to get Amazon Web Services to track down the culprits but unfortunately they used TOR which masked outbound traffic so there was no way to trace the source. Thankfully Amazon Web Services credited my account so I didn't have to pay the penalty for my carelessness.

What about DevOps?

More and more DevOps teams are using the cloud as it enables getting products to market faster in many cases. Many are startups and were born in the cloud. However a growing number are enterprise IT organisations not used to dealing with security in the cloud. Worse, many of these are shadow IT groups, meaning groups that have abandoned their internal IT and gone to the cloud without an enterprise strategy. This is very similar to my personal experience, they are just jumping into the cloud without a complete security strategy. There have already been numerous companies which have had to close their doors due to cloud security breaches and the threat is real. So what is different in the cloud? The main thing is that we are used to working and trusting our internal IT for protecting us. We as developers have become very security complacent and have not had to be responsible for securing our environments. With the cloud that all changes, everyone must be re-trained on security and become stewards for protecting company assets. The firewall is the people. Don't jump into the cloud until you have your security straight!

Lessons Learned

• Lesson 1: Don't just jump into cloud and start doing stuff, ensure you have appropriate security setup for your environment first
• Lesson 2: Never ever save your AWS access key in code or any file in your code branch. As soon as you do this you have completely compromised your security
• Lesson 3: Always use multi-factor authentication
• Lesson 4: Always create an AWS IAM user and always give that IAM user only access to resources it needs through IAM policies before generating AWS keys
• Lesson 5: Store AWS key in user environment or credentials file outside of code branch
• Lesson 6: Update and change your keys on regular basis
• Lesson 7: Consider using only private repos in Github or ensure code is private and not public

Configure AWS Permissions

• Create IAM user

• Create group for each resource you want to grant for example, one group for S3, another for SNS, etc

• Add permissions to the group

How to properly store AWS credentials

There are two options for storing the AWS access and secret key. You can either save them in a credentials file or the user environment. I would recommend the user environment and this is typically what I use.

Storing AWS key in credentials file

• Create the following file: ~/.aws/credentials
• In the AWS console go to the user and create new access key by selecting the "manage access keys" option.
• Update the credentials file ~/.aws/credentials
  • [default]
  • aws_access_key_id=<access key>
  • aws_secret_access_key=<secret key>
• Get credentials from credentials file. Below is an example in Java:

[code language="java"]
public AWSCredentials getAWSCredentialsFromFile() throws Exception {
AWSCredentials credentials = null;
try {
credentials = new ProfileCredentialsProvider().getCredentials();
} catch (Exception e) {
throw new Exception ("Cannot load the credentials from the credential profiles file. Please make sure that your credentials file is at the correct location (~/.aws/credentials), and is in valid format.", e);
}
return credentials;
}
[/code]

Storing AWS key in environment

• In the AWS console go to the user and create new access key by selecting the "manage access keys" option.
• Add access key and secret key to the user's .bashrc file under ~/.bachrc
  • AWS_ACCESS_KEY_ID=<access key>
  • AWS_SECRET_ACCESS_KEY=<secret key>
• Get the credentials from user environment. Below is an example in Java:

[code language="java"]

public AWSCredentials getAwsCredentialsFromEnvironment() throws IOException {

EnvironmentVariableCredentialsProvider credentials = new EnvironmentVariableCredentialsProvider();
return credentials.getCredentials();
}
[/code]

(c) 2014 Keith Tenzer