Relationships and Connected Threats in the Public Cloud
This blog is a follow up to my blog titled CIS Benchmarks for AWS with CloudHealth and VMware Secure State.
In my previous blog, I highlighted the value and benefits of CIS Benchmark and AWS best practices for Public Cloud workloads. I want to shift gears a bit to look at a new type of threat that we see in the Public Cloud. While benchmarks and best practices are important, there are new challenges that we face with the proliferation of the Public Cloud. Simply put, the public cloud is complex and highly probable to mis-configuration and hence susceptible to threats.
Managing AWS
There are multiple areas of configuration that need to be managed when deploying applications or even simply setting up environments for developers. These areas include:
-
Identity and Access Management (IAM) The ability to manage access to AWS Services and ensure security to them. The number of policies and roles are countless and the inter connectivity between assets including parent and child relations can be a daunting task. There are many solutions and schools of though on how to overcome these difficulties.
-
Virtual Private Clouds (VPCs), Firewall Rules and Security Groups are also a hot spot for configuration issues. AWS has provided Flow Logs to capture IP traffic details but that is only half of the battle. Security Groups and associated Firewall Rules are available to ensure granular access to workloads; if you set them up properly, even across resource types including EC2, RDS, etc.
-
Data Protection is always a topic of discussion. Data at rest, data in transit and ultimately the generic access rules to/from internal and external sources. Data at rest protection is available for EBS, S3, RDS, DynamoDB, etc. The one problem, humans forget to add the necessary parameters in CloudFormation templates, terraform manifest or forget to check a box when using the GUI.
Each of these can and should be highlighted as areas of extreme concern for any organization consuming the Public Cloud. There are plenty of open source and 3rd party solutions available in the market to address each of these areas individually.
In this blog, I will review an example of how a threat can exists due to relationships, connections and simple misconfigurations within the Public Cloud. I will also navigate potential solutions to detect and remediate when necessary.
Relationships and Connections
The example application below highlights a typical AWS environment and how EC2 instances are configured between Public and Private subnets.
This example application includes the core components including a Management instance(jump box), Application Instances, Web Instances and Database Servers. The database servers could be EC2 instance running MySQL or RDS instances running Aurora. The applications was built either using Terraform or CloudFormation templates and like any organization, there has been some modification to the initial build state. IAM permissions have been setup and modified, SSH Keys are the same keys from the initial deployment and who know what else has been modified.
Mistakes can be made during initial deployment or modifications can occur, there are a few things that I want to call out as potential threats.
- EC2 Instances with poorly configured IAM policy
- Publicly Addressable Route for EC2 Instances
- SSH Keys used to access each Instance is shared amongst the team
In many cases, these are common configuration methodologies when looked at from a narrow perspective.
Before we look into the relationships between the three misconfigurations, lets look at them individually with a lense of native AWS solutions including IAM, CloudWatch, etc.
EC2 with Poorly Configured IAM Policy
The practice of providing IAM role for EC2 instances is a common practice. AWS guides can be found here.
Unfortunately, our scenario mistakenly gives EC2 Instances Administrative Access to All AWS Resources and Services instead of limiting to RDS. A simple misconfiguration can lead to a significant threat.
AWS provides a guide on how to configure EC2 access to RDS via Roles and Policies and you can get more information in the article titled How do I allow users to connect to Amazon RDS with IAM credentials?
Identity and Access Management
AWS IAM provides a way to management access to AWS services and resources and is essentially RBAC in AWS.
The following is taken directly from AWS IAM website:
“AWS Identity and Access Management (IAM) enables you to manage access to AWS services and resources securely. Using IAM, you can create and manage AWS users and groups, and use permissions to allow and deny their access to AWS resources.”
AWS also provides best practices for each type of resource that is available, which can be found here.
Default Policies
AWS provides out of the box policies which are really easy to use and highlighted below. Let’s look at a few of these in detail.
There are many out of the box policies but in the scenario, EC2 instances were mistakenly given Administrative Access.
Administrator Access
This next screen highlights the nature of this policy. Allow for all Actions on all Resources. Typically, this is given to an administrator to ensure proper use of the account.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "*",
"Resource": "*"
}
]
}
Amazon RDS Full Access
Many situations call for EC2 instances to have Full Access to RDS and AWS provides the policies out of the box. Below is the default policy from Amazon.
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"rds:*",
"application-autoscaling:DeleteScalingPolicy",
"application-autoscaling:DeregisterScalableTarget",
"application-autoscaling:DescribeScalableTargets",
"application-autoscaling:DescribeScalingActivities",
"application-autoscaling:DescribeScalingPolicies",
"application-autoscaling:PutScalingPolicy",
"application-autoscaling:RegisterScalableTarget",
"cloudwatch:DescribeAlarms",
"cloudwatch:GetMetricStatistics",
"cloudwatch:PutMetricAlarm",
"cloudwatch:DeleteAlarms",
"ec2:DescribeAccountAttributes",
"ec2:DescribeAvailabilityZones",
"ec2:DescribeInternetGateways",
"ec2:DescribeSecurityGroups",
"ec2:DescribeSubnets",
"ec2:DescribeVpcAttribute",
"ec2:DescribeVpcs",
"sns:ListSubscriptions",
"sns:ListTopics",
"sns:Publish",
"logs:DescribeLogStreams",
"logs:GetLogEvents"
],
"Effect": "Allow",
"Resource": "*"
},
{
"Action": "pi:*",
"Effect": "Allow",
"Resource": "arn:aws:pi:*:*:metrics/rds/*"
},
{
"Action": "iam:CreateServiceLinkedRole",
"Effect": "Allow",
"Resource": "*",
"Condition": {
"StringLike": {
"iam:AWSServiceName": [
"rds.amazonaws.com",
"rds.application-autoscaling.amazonaws.com"
]
}
}
}
]
}
Custom Policies
One of my favorite features of AWS IAM is the ability to very granular with permissions. Below is an example from AWS and a link to the direct guide can be found here.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowCreateDBInstanceOnly",
"Effect": "Allow",
"Action": [
"rds:CreateDBInstance"
],
"Resource": [
"arn:aws:rds:*:123456789012:db:test*",
"arn:aws:rds:*:123456789012:og:default*",
"arn:aws:rds:*:123456789012:pg:default*",
"arn:aws:rds:*:123456789012:subgrp:default"
],
"Condition": {
"StringEquals": {
"rds:DatabaseEngine": "mysql",
"rds:DatabaseClass": "db.t2.micro"
}
}
}
]
}
While the above demonstrates the capabilities of IAM, the next section will focus on how to overcome common configuration mistakes with native AWS services and solutions.
How to Detect Common Configuration Mistakes
AWS CloudWatch Logs
If we go back to our original scenarios, the use of IAM policies across the environment is a common practice. AWS CloudWatch Logs provides the ability to log all activities that occur within RDS. Below is a sample screenshot of the output from RDS to CloudWatch Logs from within the AWS Console.
You can take a further look at a blog from AWS that goes into greater detail about this specific subject entitled: Monitor Amazon Aurora MySQL, Amazon RDS for MySQL and MariaDB logs with Amazon CloudWatch.
Due to the initial misconfiguration of the IAM policies for EC2, this policy would log any time the EC2 instance created a Database in RDS. I wont take the time to argue whether or not organizations are being proactive about reviewing logs but my guess is that it really falls back on “we hope so”. While this type of log and subsequent alert allows for notification, this does not solve the remediation issue or the “where did this come from” problem.
AWS CloudTrail
Another option for AWS customers is the use of AWS CloudTrail. Our situation does allow for all recorded events from EC2 instance on the RDS instance to be recorded. Here is a bit about AWS CloudTrail.
“AWS CloudTrail, a service that provides a record of actions taken by an IAM user or role."
This quote is taken directly from AWS CloudTrail website found here.
Whether the application and policies were configured properly, AWS CloudTrail would log any of the events or actions taken by an EC2 instances against an RDS database. For example, what time the call was made and what action was taken. Take a look a CreateDBInstance. policy for AWS CloudTrail that will record an event each time an IAM role creates a new database instance.
Again, this is a nice feature to have but this only tracks what events occur when being driven by EC2 instance in RDS. We will discuss some shortcomings for each of these methods in a bit.
Publicly Addressable Route for EC2 Instances
The backbone of AWS is the Virtual Private Cloud. A VPC consists of many aspects but for this particular scenario I will focus on the Public and Private subnet, along with Security Groups and Firewall Rules.
Going back to my example app. The Management and Web Servers are on the Public Subnet by design while the application and database servers are not. This is only one example of a network topology but misconfigurations do happen.
Security Groups
The screenshot below give us an example of an application that was provisioned with Terraform and the security groups are populated according to type of workload. This example highlights the granularity provided within an AWS VPC.
Firewall Rules
To help isolate communication and traffic to necessary entities, organizations are utilizing Firewall Rules. Below are a few examples to highlight the differences between Management (Jump Box), Web Servers and Applications Servers. Each of this demonstrates a potential best practice to ensure communication is secure and all other traffic is blocked automatically.
Management Instance allows inbound from the internet on port 22 and inbound from the private subnet on port 24224 (fluentd) and port 2878 (telegraf agent for Wavefront). Seems like a pretty strong setup!
Web Instances are accessible via port 80 and port 443 from the internet while port 22 is allowed from the Private subnet. Also a common and secure configuration.
App Instances are not accessible from the internet but accessible from the private subnet via port 22 and port 8000. This is another sound configuration.
Database Instances are only accessible via port 22 and port 3306 from the Private Subnet.
Each of these highlights the value of AWS VPCs, Security Groups and Firewall Rules.
Shared SSH Keys
I am not going to dwell on this section for very long other than to highlight the capabilities found in EC2. Below you will notice a screenshot of the EC2 console where each EC2 instance is using the same SSH key. These objects where provisioned utilizing Terraform and typical best practice is to use a similar setup. At least that is what I have found. That SSH Key can then be shared amongst a team and access via SSH is possible. This alone is not a security issue but when you tie it together with the previous configuration items can provide a very different picture.
Relationship and Risk Based Cloud Security
Most of this blog detailing common configuration scenarios and the associated benefits for each individually, there is still one issue that remains. The depth of relationships and inter connectivity within AWS.
For me to solve this complex puzzle, I would have to navigate through multiple AWS Services, their associated dashboards and even add some intuition to help correlate the threat. All while acknowledging that each configuration might be legitimate and within context of the application.
The following screenshot from VMware Secure State<. details a scenario that based upon individual configuration and best practices seems to be a legitimate configuration but when looked at from a relationship perspective tell a very different story.
You will notice a few things in the screenshot which are common practices:
- Publicly addressable EC2 Instances
- EC2 Instances on the same subnet
- EC2 Instances using shared SSH Keys
- One EC2 Instance with different levels of access based upon IAM policies
Here is the problem with this configuration:
One of the EC2 instance was purposefully given an IAM role for a specific task but due to the other EC2 instance being on the same subnet and with the same Share Key, it also has access to the same IAM policy. While inadvertent and a bit vague, in this case both EC2 instance share a relationship which give each Full Administrative Access within the VPC which was not meant by design.
The above screenshot was take from VMware Secure State. You can find out more information here.
Looking at this a bit further we see that VMware Secure State identifies this violation as “An SSH key is shared by EC2 instances with different levels of access”.
Not only does VMware Security highlights the potential violated object, it will also help you discover the role and associated IAM policy.
One could quickly navigate within the AWS Console to change the IAM Role associated with the EC2 Instance to more restrictive role or remove it all together. Below is an example of a custom policy to allow for granular permission levels.
Custom Policies
One of my favorite features of AWS IAM is the ability to very granular with permissions. Below is an example from AWS and a link to the direct guide can be found here.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowCreateDBInstanceOnly",
"Effect": "Allow",
"Action": [
"rds:CreateDBInstance"
],
"Resource": [
"arn:aws:rds:*:123456789012:db:test*",
"arn:aws:rds:*:123456789012:og:default*",
"arn:aws:rds:*:123456789012:pg:default*",
"arn:aws:rds:*:123456789012:subgrp:default"
],
"Condition": {
"StringEquals": {
"rds:DatabaseEngine": "mysql",
"rds:DatabaseClass": "db.t2.micro"
}
}
}
]
}
Context and Description
While finding these types of threats are difficult and other missed by traditional Best Practices and Benchmarks, they are still real threats
VMware Secure State lists this threat with the following description:
“An EC2 instance that is accessible from the public Internet shares an SSH key with an instance that has administrative privileges. You should create unique SSH keys for each system to ensure that an attacker cannot gain administrative access by compromising the publicly accessible instance.”
The KB article for this particular threat provides not only a Description, but includes Suggested Actions, Compliance and References.
Create unique SSH keys for this scenario or modify the policy, that is up to you.
Context is key in this scenario as it makes sense for SSH Keys to be shared among instances and to even be on the same subnet, but permissions are then “shared” to the other EC2 Instance. Malicious activity like deleting a user, an asset, making changes to policies, etc. The possibilities are endless!
One additional note, CloudTrail would not be able to catch the activity from the non privileged EC2 instance because it looks like the other EC2 instance is making the necessary calls.
Solving Complex Configurations and Subsequent Relationships is Hard
While there are plenty of native, open source and third party solutions in the market today to solve the complexities of the public cloud, they are never perfect. We acknowledge that threats are never ending and new scenarios will arrive each and every day. This is only the beginning of the journey.
I strongly recommend the use of AWS Cloud Watch, CloudTrail and Cloud Watch Logs. VMware Secure State actually utilizes this information for threat modelling. But this only solves the challenges of AWS. What about organizations that are building out strategies in AWS, Azure and Google. These problems only grow and the complexity exceeds our understanding.
You could even utilize “Policy as Code” or place automated remediation actions via Terraform and your favorite CI/CD solution.
Check out a blog from Shri focusing on this exact topic found here.
Be on the lookout for another blog detailing some of the remediation features found in VMware Secure State in the near future.
Until then, let me know your thoughts on Twitter.
