Just a habit

We love keeping configuration in text files so much, that we put there database credentials or another kind of sensitive data without any considerations. Then commit, push and everything goes to the code repo. Why is it good? It’s easy, fast and enables database access for everyone interested in. Why is it not? It’s easy, fast and enables database access for everyone interested in.

Ok, more seriously. What’s wrong with this approach?

  • We use Github, Gitlab or another publicly available code repo which is risky because someone unauthroized can get an access to your code.
  • It’s even more dangerous because it’s enough for an attacker to break just a single account having access to your code.
  • Also, by mistake someone can create a public repo, not a private one and thus anyone can see your credentials.
  • Keep in mind, that anyone having access to the repo, also has an access to the production database. It may result in breaking changes in the database.
  • And changing a password for database isn’t an easy task assuming you want to provide zero-downtime or high availability in your application. Sometimes it’s even hard to identify all the applications using your database.
  • Finally, common! How you feel putting your credentials in plain text into file?!

Fortunately, there are some solutions already available on the market. You can encrypt your data and decrypt it in a CI/CD process. Ansible Vault can be used for that. Alternatively you can retrieve the data from an external service - HashiCorp Vault is an example.

In this post, I will present another solution which is AWS Secrets Manager. It’s not required for your application to run in AWS but if it does, this solution has even more advantages.

This is a next part of our Spring Boot series. Feel free to checkout previous posts!

Meet AWS Secrets Manager (ASM)

Not much to add after reading AWS docs. AWS Secrets Manager provides your with the following functions:

  • Keeping your credentials safely using encryption. Rather obvious.
  • Auditing that helps you to keep track of all the changes made to your secrets.
  • API to access the secrets. The price is quite low.
  • Fine-grained AWS Identity and Access Management (IAM) which helps you to control an access to secrets.
  • Password rotation. Killer feature! Imagine you can change your database password anywhere, anytime, in all of the applications. Actually, ASM will do that for you!

Last but not least, implementation is quite simple and I hope to confirm it in this article.

Plan for today

We will work with the same application that we built in the previous posts.

And this is the agenda for this article:

  • We will create a new database in AWS RDS and will add a user that will be used in the application.
  • User credentials will be put in AWS Secrets Manager.
  • Then we will introduce a simple change in the application allowing to access the database through AWS SDK.
  • We will test it and will make sure that local development is still possible.
  • Corner case: we will test how the application behaves when we change database password.
  • We will enable password rotation in the ASM and will handle it by the application side.
  • We will switch from the raw AWS SDK to the AWS library allowing AWS RDS access.
  • We will take a look on the algorithm that is used to handle password rotation.

The diagram below shows our target architecture.

Spring Boot and AWS Secrets Manager integration.

As usual, I will mention the tag name at the beginning of each section. Let’s start with the authentication-token-revocation.

Step 1: Creating database in AWS RDS

At first, let’s create a database that will be used in our test application. Please open AWS console, go to RDS and create PostgreSQL database in accordance to the instructions here. The smallest one with params below is more than enough.

  • Use case: Dev/Test
  • Class: db.t2.micro
  • Multi-AZ: No
  • Public accessibility: Yes
  • Backup: 0 days (disabled)
  • Enhanced monitoring: disabled
  • Performance insights: disabled
  • Database name: _springbootblog

Using the master account in the app is not recommended so please connect to the database and create another user using the command below. I will call it app user further in the text.

From now on, we won’t use the master account anymore.

Step 2: Storing the password in the ASM

Now let’s store the app user account in the AWS service. Apart from the instructions below, you can checkout the official docs.

Open the AWS console, go to Secrets Manager and add a new secret of Credentials for RDS database type. Copy username and password from the SQL command you ran before and select your database. Then put a name, select Disable automatic rotation and save the changes. Please notice the code snippet at the end. It should be able to retrieve the secret you just added via AWS SDK. We will get back to it later.

At this point, we have everything that is needed by the AWS side. Go ahead and switch to the application.

Step 3: Retrieving the ASM secret by using AWS SDK

Git tag: rds-and-aws-secrets-manager-sdk

Currently, the application keeps the data in memory using H2 database. In this step we will introduce PostgreSQL configuration but also, we will prepare two SpringBoot profiles: production and testing. Nothing will change in the test profile. Also, we will make sure that our changes don’t impact the automated tests. In the production profile, we will use AWS RDS in conjunction with AWS Secrets Manager to initialize database. Let’s do it!

Add gradle dependencies:

We need both H2 and PostgreSQL drivers depending on the profile we use to run the app. AWS SDK is needed to access ASM. Use anything you want to parse a JSON.

Create a class that will initialize database connection.

Please notice the @Profile(“prod”) annotation. That way the bean will be created only in the production profile. Creating DataSource is just a typical spring code and retrieving the password is purely copy-pasted from the ASM page that I mentioned in the previous step. I just added a simple POJO to map the JSON. My choice.

What’s next? Add a configuration file. Mind the prod suffix which is important to run the requested SpringBoot profile.

Worth to notice that the DataSource autoconfiguration is disabled because it’s created manually in the production profile.

That’s it! In the repo you will also find some cosmetic changes to support local development. Now, let’s focus on testing!

Step 4: Testing!

Start with putting your AWS credentials into environment variables. If you don’t access key for now, please generate it in accordance to the doc and then add these three env variables.

To confirm that everything went well, run the command below.

If you get HTTP/1.1 200 in a response, then it works!

So… it’s done, see ya… No! What about the password rotation?!

Step 5: Changing the password and a trap

Before enabling password rotation in the ASM, let’s try a simple test. Leave the application running. Connect to the database and change the password for the app user using the command below. Remember to modify it accordingly.

Run the test HTTP query from the previous step. Does it work? It does!

And this is a trap that may not be visible at first. Let’s review the current solution. Should it work? Database connection is initialized at the app’s startup. DataSource instance is a singleton and won’t change without a manual modifications. On the other hand - we didn’t add any code responsible for handling password change. As a consequence, our application won’t work properly after changing database password, unless…

AWS RDS - current status and number of active connections.

…the app keeps a connection pool! As you can see, there are 11 active connections that remain alive even when database password was changed. Depending on the configuration, number of connections can change over time. Now you know. Current implementation will survive after changing the password as long as there is no need to establish a new database connection.

AWS RDS - active connections over time.

Sooner or later you will find something similar in the app’s log files.

I propose one another challenge. Let’s dive deeper into Spring Framework and look for a confirmation of the experiment we just conducted.

Step 6: Closer look into database connections pool

First, please check which of the javax.sql.DataSource implementations is used in the application. It is com.zaxxer.hikari.HikariDataSource which by default uses connections pool. Here is the confirmation.

Looking into javax.sql.DataSource#getConnection() method, you can see that before creating a new connection there is an attempt to retrieve it from the pool. Moreover, with the default settings, connection is terminated after 30 minutes of inactivity. A complete list of configuration settings is available here.

That’s it! Lessons learned. Let’s do it better!

Step 7: Supporting password rotation in the application

Git tag: rds-password-rotation

Let’s start with a quick change that will provide an access to the RDS database, doesn’t matter if the password rotation is enabled or not. It’s as easy as adding a new AWS library handling the connection.

Please start with adding new dependency.

Then modify the production profile.

Notice that we change the driver - we will analyze its source code soon. For now, database username should contain the name of the secret that you put into ASM.

Remove RdsDataSourceConfig.java. We will use SpringBoot autoconfiguration.

That’s it. Run the application as usual. Again, let’s review the code and find a confirmation that the library supports ASM password rotation.

The most important change that was introduced is a new database driver. As we know, that it has to implement connect() method, let’s take a look on the snippet below.

As you can see, the connectWithSecret(unwrappedUrl, info, credentialsSecretId) is called with the secret name placed in the configuration. Go ahead and see its implementation.

Having the credentialsSecretId it tries to retrieve that password from a cache. If the authentication fails, it retrieves the secret again and tries to open database connection again.

Please notice another important detail. AWS library is just a wrapper responsible for retrieving database credentials. The driver responsible for connection handling is returned by the getWrappedDriver() method.

Step 8: Enabling password rotation and testing

All the changes by the app side are complete. It’s high time to add the last two missing components.

App architecture. Missing password rotation components.

Please log in into AWS console and navigate to the ASM service. Edit your secret by enabling automatic rotation.

ASM secret. Enabling automatic rotation.

Save it and wait until all components are ready. And what’ happening under the hood? AWS creates a new CloudFormation stack which will setup a Lambda Function responsible for the rotation. Also, CloudWatch will be configured providing you with an access to the statistics and logs of the newly created function. Additionally, AWS will configure an access from the function to the RDS database. Of course, you can implement it on your own. AWS implementation is available here.

Quite simple, isn’t it? Just works! Although… I had a chance to spent a little more time here. And want to share it with you.

Step 47: Understanding Lambda

While testing the app I noticed that the rotation is failing for me. Strange, I said to myself. I didn’t touch the code, I did everything using the AWS console. Well… Challenge accepted! I started debugging.

Where to start?

I propose to start with CloudWatch which is configured by default in this case. Please open the AWS console, go to Lambda service, choose your function, then open the Monitoring tab and choose View Logs in CloudWatch.

I found there just a little message.

I need more logs! It shouldn’t be too hard! The rotation function is just a small piece of a python code. We can download this code and modify it.

Again, go to your function but stay in the opened tab. Select Actions, then Export Function and Download deployment package.

Exporting Lambda function.

I wanted to check why the function cannot connect to the database. Thus, I printed the error that the pgdb.connect throws.

To update the Lambda Function, just zip the python script together with the other files and select Upload Function Package in the AWS console.

Another problem is that the new rotation won’t start until the previous one is completed.

No worries! We can create test event and run it through out the AWS console. This is an example of a data structure.

Example event for testing AWS Lambda.

Two first fields are quite obvious. SecretId is just the ARN of the rotation function. The second field is a name of the action we want to run. ClientRequestToken in fact, is an identifier of the secret version, that will be used to open database connection.

It can be found in a metadata being retrieved by this piece of code.

After running the test, it turned out that the root cause of the issue is a timeout while trying to access RDS.

Let’s dig it. After quick investigation, I found that the RDS is not accessible from outside of the VPC. Taking into account that we are running just a test app and the database has public access enabled, the quickest solution will be adjusting the Inbound Rule.

Modifying Inbound Rules for the AWS RDS.

Please notice it’s not a recommended solution for a production system. In that case, it would be better to place the Lambda function inside VPC and configure an access from the RDS.

After applying one of the proposed solution, the app should work.

Summary

Thanks for reading. In this article I presented advantages of the AWS Secrets Manager service, explained how to use it and integrate with the SpringBoot application. Additionally I explained the algorithm of the password rotation and conducted a quick debugging session of a Lambda Function. If you found it interesting, please leave a star in the repo.

Let me mention two additional things related to the topic. If your application runs in AWS, you can benefit from the great integration of the AWS services. In this case, with just a simple configuration, your application doesn’t need to know the AWS credentials to access ASM. It’s enough that you allow an access from the machine hosting your app.

Moreover, for many of you the first steps to jump into a cloud would be to migrate your system and database to AWS. Feel free to contact us with any kind of issues with cloud solutions. Our knowledge is confirmed by many years of hands-on experience.

Efficient Cloud That Suits Your Pocket

We have many years of experience with migrating, designing and optimizing Cloud systems. Let us prepare a solution that suits your needs.

Schedule a call with our expert