There’s an interesting post making the rounds in Twitter called How a bug in Visual Studio 2015 exposed my source code on GitHub and cost me $6,500 in a few hours. The short version is that the developer from Humankode attempted to create private repo on Github via the Visual Studio Git extension. Unfortunately, for this developer, the Visual Studio Git extension made the repository public rather than private. Complicating matters, the developer had committed his AWS access key and AWS secret access key. Thus, the keys were compromised and got into the hands of the wrong party and ran up $6,500 in AWS charges. In the end, the developer had this to say as a lesson learnt:

At face value one might say it’s simple : don’t publish your access keys to a public repository, which is what many before me have done. In my instance, I specifically published to a private repository, but a bug in visual studio meant that the code was published to a public repository. As soon as it was out in the wild, it was too late. Bots scan GitHub repositories and it only takes 2 or 3 minutes for some of them to pick this up.

It’s reasonable advice, but you should do much more.

Don’t ever publish credentials to an SCM, public or private

By credentials, I mean anything that allows someone to gain elevated privileges to your systems, which include:

• Passwords
• SSH Keys
• Private keys or certificates
• OAuth Consumer or Token Secrets
• AWS Access and Secret Keys (also the equivalent of Azure or any other public cloud)

While stating “don’t publish your access keys to a public repository,” is sound advice, I’d expand on that and assert that credentials shouldn’t be published in any repository. Period. Also, don’t put these things into your Wiki, or on a shared file system. Avoid exchanging credentials via email too. They’re credentials. They give anyone the power to do things on your system. You should control who has access to manage your runtime environment, this means locking down the credentials and entitling credentials. If credentials reside in source control, you’re asserting that every developer has full admin rights to your deployment environments as well. That’s probably not what you want.

But why is storing them in an SCM a bad idea? Chances are, you put the credentials in your SCM as a means to simplify deployment. As a result, they’re likely being included in your builds too. If you’re producing a package that gets published to a package repository (NuGet, Maven, RPM, etc.), the credentials are going there too. Is your package repository private too? If you’re using a CI tool like Jenkins, those credentials are also being copied there too. Is that locked down and private too? Can you prevent others from seeing your working directory? Another big gotcha, especially with tools like Git, is the possibility that a developer on your project “backs up” a project to another remote repository. There’s nothing stopping anyone from cloning your private repo on GitHub to a public repo on BitBucket by simply adding a new remote. Your best defense is to not put credentials where your source code lives.

Where should you put credentials?

If not in an SCM, then where? You probably keep your personal credentials private. You may even use a password utility like LastPassword, 1Password, etc. because you can’t remeber all of these passwords in your head and need to retrieve them at some point. Basically, you’re keeping these secrets private, as they should be, as opposed to slapping post-it notes to your desk. Application credentials should be treated no differently. There are tools like Hashi Corp’s Vault, Conjur, or Cyberark to name a few, which are all designed to manage application credentials so that they are protected.

Use a Token Service

If you are using AWS, consider using Amazon’s Secure Token Service. This allows you to use temporary credentials that expire between 15 minutes to 24 hours. You can authenticate users with ADFS or OpenID Connect, and only after they have been authenticated by your systems, will they be able to obtain an STS token. Of course, this all falls apart if you are storing your ADFS or OIDC credentials in your Git repository. Since that’s a crazy idea, you’re not doing that. Right?

Get in the Habit of Rotating your Credentials

It’s usually a good idea to change passwords periodically. And by periodically I mean every few days to weeks, not months. Access keys are no different and you should rotate them on a regular basis. The AWS blog has an informative post on how to rotate access keys for IAM users. If you look closely, it’ll make more sense why IAM users are allowed to have a maximum of 2 access keys. Something like this is pretty easy to automate as well. Keeping an IAM access key valid for more than 2–12 months is not a good idea.

Entitle your Credentials

Proper use of entitlement, roles, or privileges can help minimize the impact that an attacker can have if your credentials are compromised. If an application only needs to have read/write access to DynamoDB, then it should only have read/write access to DynamoDB. It shouldn’t have the ability to spin up new EC2 instances, call CloudFormation, etc.. It’s easy to select “PowerUser” from the IAM console but you shouldn’t. Yeah, an attacker might be able to read your data, but they’re not going to have the ability to spin up 1,000 EC2 instances to mine bitcoins.

Just keep your credentials private, even in private.

Credentials need to be secured, plain and simple. Even if your running you app in-house, with a private wiki and private git repo, you still should not put credentials of any sort into those systems. It’s one of the first places attackers look if your environment is compromised.