By Richard Audette, richard@hotelexistence.ca

Introduction

Credential stuffing is a type of cyberattack where stolen account credentials typically consisting of lists of usernames and/or email addresses and the corresponding passwords (often from a data breach) are used to gain unauthorized access to user accounts through large-scale automated login requests directed against a web application.
from: https://en.wikipedia.org/wiki/Credential_stuffing

These attacks are pretty common. In August, the Canada Revenue Agency was the subject of an attack. Approximately 5500 taxpayers were impacted, including one of my friends:

Friend: “i got a letter in the mail saying my CRA account got hacked (sad) they took out the Canada Emergency Response Benefit in my name and into another bank account, everyone should be care. its 2000$ x2, that i would have to pay back but they flagged it as fraudulant.”
Me: “am curious, in the news, this incident was all credential re-use - were your CRA credentials unique?”
Friend: “unfortunately it was not unique, it was a password i set like 10+ years ago”

The standard practice of enforcing password complexity through minimum lengths, punctuation characters, capital letters and numbers does not stop me from registering on your site with the email address and password that I initially setup with LinkedIn, which were widely circulated following the 2012 LinkedIn breach, leaving me susceptible to a credential stuffing attack on your site.

One approach for protecting your users’ from this type of attack is to ensure they are not using passwords known to be breached. This process is recommended by the US National Institute of Standards and Technology’s (NIST) digital identity guidelines (section 5.1.12, special publication 800-63B). In this article, I’ll outline how to build a service on Amazon Web Services (AWS) which you can use to implement this approach by checking your user’s passwords against Troy Hunt’s ihavebeenpwned database of >600M breached passwords. When your user enters a password, check the password against the service. If the service indicates the password has been breached, ask the user to select a new password. For other approaches, I recommend reviewing the OWASP Credential Stuffing Prevention Cheat Sheet.

How I came to build a Password Breach Check Service on AWS

I have been following Troy Hunt’s site for a while, but was unaware of the Password Check API until reading about his intention to open source his codebase in August. The ihavebeenpwned password check API is really neat, in that you can check to see if a password exists in the database, by providing the first few characters of the hash of a password, which I learned is a mathematical property called k-anonymity - he explains it best. I was impressed with how clever the solution was.

That being said, I could think of environments where you might not want to leverage this API due to:

• Procurement requirements and challenges
• Service level agreements, contracts
• Sharing data (even a few characters of a hash!) with an external party

As Troy Hunt has made his password breach database available, I thought it would be an interesting exercise to build a Docker container which could be easily dropped into existing environments. I hadn’t built a Docker container before, this would be a fun exercise. As is often the case with great ideas, a quick search revealed this had already been done - check out Felix Engelmann’s haveibeenpwned-api Docker container. I tested it, it works great.

So, I took another approach, and decided to build out my service on AWS. The hash lookup seemed like a perfect candidate for AWS Lambda functions, and lookups against a large, single-table password list seemed perfect for DynamoDB. I hadn’t done this before, and I couldn’t find anyone else sharing an AWS-based ihavebeenpwned solution, so this is how I decided to proceed.

I struggled with Amazon’s IAM permissions, following various tutorials, and getting everything in place such that the API gateway could talk to Lambda which could talk to DynamoDB. I ended up using the Serverless framework, and built this service on top of their AWS | REST API With DynamoDB example. This framework had the added benefit of making it easy for others to deploy the service, which I’ll cover in the next section.

Setup Password Breach Check Service on AWS

A Note on AWS Costs

AWS is a great platform for experimentation. I will often set something up on AWS, rather than on a PC at home, because of the ease and speed of spinning something up, then turning it off when I’m done. My projects will often fall into the AWS free tier. I don’t typically incur expenses related to scale. This project was different.

I have configured this project to use Dynamo DB’s on demand billing model, which scales automatically as required. Dynamo DB has two pricing models - on demand, which scales to need, and provisioned capacity. Review both models at https://aws.amazon.com/dynamodb/pricing/. The first 25 GB of storage is free, which I imagine should cover most of the storage costs. The provisioned mode has a free tier of 25 read and write units (4kb/s) - ample for testing service, but at this rate, it would a very long time to upload the password database. In on-demand mode, in the CA-CENTRAL-1 region, writes are priced at CAD$1.375 per million write request units. For this project, loading 1M records costs about $1.50 - at 15M records, I decided I didn’t need to load all 600M records for my purposes - I’m confident DynamoDB would scale! There is likely cost-optimization that I’m missing here - I welcome any recommendations for optimizing this process.

Download and Deploy the Code

To complete this project, you need NodeJS, Git, and AWS CLI installed, and an Amazon AWS account.

1. Download the code:

git clone https://github.com/raudette/lambda-dynamodb-passwordbreachcheck.git

2. You need to configure a AWS user that the Serverless framework can use to deploy the code. Go to the AWS Identity and Access Management console.
3. Select Users
4. Add a user, enable programmatic access
5. In step 2, select “Attach existing policies directly”
6. A new browser tab opened. On the Create Policy screen, select JSON, and paste the contents of the serverless_policy.json included with the code. Save the policy.
7. Return to the browser tab where you created the user, and click refresh. Search for the policy you created, and select it. Continue through the steps without changing anything, until step 5. An access key and secret is presented. Record these values.
8. Return to your terminal window, and go to the folder where you downloaded the code. Install the required node libraries:

npm install

9. Setup serverless by running:

node node_modules/serverless/scripts/serverless.js

10. Follow the prompts until you are asked for your AWS access keys. Use the keys you created above in step 7.
11. You can now deploy the application as follows:

node node_modules/serverless/scripts/serverless.js deploy

12. Once the process has finished deploying, the Serverless tool will present two endpoints - record these endpoints. You can also retrieve these endpoints from the AWS API Gateway console by selecting the region, API, and then clicking on the dashboard.

NOTE: Your endpoints will be different from mine. A custom domain can be assigned, but a random string is used by default.

13. The root endpoint hosts a web page you can use to test the web service - in my installation, this is: https://wwpg6blcog.execute-api.ca-central-1.amazonaws.com/dev/

NOTE: To manage my AWS bill, this URL might not be live. I share for illustrative purposes.

The test page looks like this:

Once deployed, the web service has pre-populated the breach database with 4 records - try them out: Password1! , 123456 , 1q2w3e4r5t and qwer1234

The web service can be integrated with your application. Although it can be secured, it has been configured with no access controls. In my installation this was:
https://wwpg6blcog.execute-api.ca-central-1.amazonaws.com/dev/checkhash/{hash}

Your application can call the API with an HTTP GET, replace the {hash} with a SHA-1 hash of a password as follows: https://{Your Endpoint}/dev/checkhash/7C4A8D09CA3762AF61E59520943DC26494F8941B

The end point should return:

{"passwordhash":"7C4A8D09CA3762AF61E59520943DC26494F8941B","timesseen":24230577}

Now, we have to populate the database with a list of breached passwords.

Download and Setup the ihavebeenpwned Password Breach Database

A Lambda function called importpassword was deployed as a part of the installation to populate the database. This function is triggered by uploading a data file to an Amazon S3 bucket, and loads these records into the database.

We’ll load Troy Hunt’s ihavebeenpwned list, which can be downloaded from https://haveibeenpwned.com/Passwords . At the time of writing, this file contains a list of ~600M passwords. Select the SHA-1, ordered by hash option. As I was working on this project, I couldn’t find samples of this file. It is formatted as follows:

SHA1 Hash of Password:Number of Times Seen (the actual file has no header row)  
000000005AD76BD555C1D6D771DE417A4B87E4B4:4
00000000A8DAE4228F821FB418F59826079BF368:3
00000000DD7F2A1C68A35673713783CA390C9E93:630
00000001E225B908BAC31C56DB04D892E47536E0:5
00000006BAB7FC3113AA73DE3589630FC08218E7:2
00000008CD1806EB7B9B46A8F87690B2AC16F617:4

You will need lots of space to work with the file, as it is big, and we need to split it up to make it useful for our application. Amazon Lambda functions have a 15 minute maximum run time and a starting limit of 500 concurrent calls. In my testing, I could load about 1,000,000 records in 500 seconds - so we’ll split the file into 1M row chunks, and run in separate batches. Note that I personally stopped testing at 15M records - I am extrapolating from there.

1. First, we have to uncompress it with 7-zip.

7z e pwned-passwords-sha1-ordered-by-hash-v6.7z

2. Then, we’ll split the file into 1,000,000 row chunks with split. If you are using Windows, it is included with the Git for Windows package:

split -l 1000000 --numeric-suffixes --suffix-length=3 --additional-suffix=.csv pwned-passwords-sha1-ordered-by-hash-v6.txt pwned-

3. First, send files 0-99:

aws s3 cp ./ s3://passwordbreachcheck-dev/ --recursive --exclude "*" --include "pwned-0*"

4. Now, wait until the files are done processing - probably about 10 minutes after your last file has completed transfering. You can monitor the process with AWS CloudWatch. Click “Log Groups” on the left panel, and select “/aws/lambda/passwordbreachcheck-dev-importpassword”. You should see a stream for each file.

5. Click on the most recent. If the file is the last file of your batch, and you see an “End RequestId”, your batch is complete.

6. Repeat steps 3, 4, and 5 for the remaining batches. Ensure you wait until each batch is complete before starting the next batch!

aws s3 cp ./ s3://passwordbreachcheck-dev/ --recursive --exclude "*" --include "pwned-1*"
aws s3 cp ./ s3://passwordbreachcheck-dev/ --recursive --exclude "*" --include "pwned-2*"
aws s3 cp ./ s3://passwordbreachcheck-dev/ --recursive --exclude "*" --include "pwned-3*"
aws s3 cp ./ s3://passwordbreachcheck-dev/ --recursive --exclude "*" --include "pwned-4*"
aws s3 cp ./ s3://passwordbreachcheck-dev/ --recursive --exclude "*" --include "pwned-5*"

7. You are done! The service is ready for use.

Integrating the service with your application

As secure applications only store hashed passwords, it would not be possible to compare all your users’ current passwords against the breach database. Potential places to check are on password creation, password change, and user login.

Once the user types in their password, create a SHA-1 hash of the password (Javascript example).

Then make an HTTP GET call to: https://{Your Endpoint}/dev/checkhash/{hash} where {Your Endpoint} is the endpoint from Step 12, and hash is the SHA-1 hash of the password.

The end point should return:

{"passwordhash":"{hash}","timesseen":{The number of times this password has been seen}}

If timesseen is greater than 0, you will want to ask your user to select a new password.

Depending on your application, if you have many users who created their accounts long ago, as the Canada Revenue Agency did, you could consider invalidating old passwords, and requiring that your users reset their password using your new policy.

Production Considerations

This project has not been used in production, has not been performance tested, and has not been peer reviewed. That being said, it is pretty simple, and it only accepts password hashes over HTTPS. As it leverages AWS Lambda functions, it should scale to 500 concurrent connections without further optimization. After the first “warm up” call, it responds to calls in 70-100 ms.

If you use this project as a basis for your implementation, in addition to further review and testing, ensure:

• The HTML function is removed
• The checkhash call is limited to your service by implementing API keys
• The passwordbreachcheck-dev S3 bucket is emptied, as it isn’t required after setup

Clean Up

Once you are done with the service, you will want to remove it to stop incurring charges:

1. Remove the Lambda function, API, and code as follows:

node node_modules/serverless/scripts/serverless.js remove

2. There will be an error, as the user created for the Serverless framework does not have the required permissions to delete the passwordbreachcheck-dev AWS S3 bucket that was created during installation. That has to be done through the S3 console.
3. The Serverless remove does not remove the passwordbreachcheck-dev DynamoDB table, this must also be deleted through its console.
4. Finally, delete the passwordbreachcheck-dev Cloudformation template through its console.

Conclusion

To conclude, ensuring your users aren’t using breached credentials is a straightforward approach for protecting them from credential stuffing attacks. Troy Hunt has done a great service for Internet security by building and sharing his breached password list. If you can’t use Troy Hunt’s ihavebeenpwned api with your site, it is fairly straightforward to roll your own.

Please email richard@hotelexistence.ca with any feedback you have regarding this article.