During the first demo I saw of the service that would become VMware Cloud PKS, my first thought was “This is the easiest way I’ve seen to setup a Kubernetes cluster”. My second thought was, “How am I going to connect this cluster to the rest of my cloud services?” Luckily for me (and the rest of the Cloud PKS user base) the engineering team was already working on the feature that we know now as ‘Connections’. In this post, we’re going to look at both some use cases for the connections feature in an Amazon Web Services environment, as well as how the feature works in conjunction with the VMware Smart Cluster™ technology.

To begin, let’s look at why the ability to connect a Kubernetes cluster to another VPC or public cloud service endpoint is important to both developers and administrators working public cloud environments.

For developers, it’s important that their Kubernetes clusters (wherever they exist and however they are created) be able to access the same set of native public cloud services regardless of where their workloads run in order to ensure a consistent set of interfaces for their applications. Since an application running on Kubernetes should be portable between clusters running the same API specifications, having asymmetrical access to services could be gating from a development perspective.

For administrators, the value of the connections feature lies in bringing familiar cloud control points to bear on the managed Kubernetes cluster(s). By provisioning a PKS Cloud connection to a user controlled VPC, an administrator cloud could create a network path to private AWS service endpoints without the need to transit one or more internet gateways. This allows for more secure data transactions by assuring the traffic path is restricted to the cloud provider backbone network. This method can also provide access to existing connectivity to other infrastructure domains via VPNs or other mediums (such as AWS Direct Connect).

VMware Cloud PKS connections via VPC peering in AWS

Now that we’ve looked at why the connections feature is valuable, let’s take a look at how it works. To start, it’s important to understand how Smart Clusters™ are isolated.

A block diagram of VMware Cloud PKS Production Smart Clusters

In Cloud PKS, there are two defined cluster types; development and production. The differences between these two types are the availability and isolation models. A development cluster utilizes a single Kubernetes master node, within a single cloud availability zone. Multiple development clusters may exist within a single VPC.

On the other hand, production clusters begin with three Kuberentes master nodes spread across three cloud availability zones. Productions clusters are strictly isolated to a single cluster per VPC. Because of this strict isolation, a production cluster has a defined and unique networking space. This unique networking space is necessary for the connections feature, as the VPC peering mechanism in AWS is a network connection governed by subnet ranges and routing tables. Without a unique IP space, potential address space overlaps and conflicts would render VPC peering useless for this purpose.

A diagram of the VPC peering connection between a Cloud PKS cluster and a customer VPC

The process of setting up a VPC peering connection in Cloud PKS is simple and automated. Because Cloud PKS is a fully managed cluster-as-a-service offering, any clusters a customer provisions exist in a separate cloud account administered by the Cloud PKS service. As a result, the VPC peering connection must be initiated via the Cloud PKS API, CLI package or web UI. In order to request a VPC peering connection, a few pieces of information are required:

• The account ID of the destination (customer controlled) AWS account
• Peer VPC ID
• Remote VPC CIDR (VPC subnet)
• Destination VPC region
• Connection name

Once these pieces of information have been input, Cloud PKS will provision a VPC peering connection. Once this is done, a VPC Peering Request will arrive in the customer controlled AWS account. This request must be accepted before the connection will become active. Also, the cloud routing table of the customer controlled VPC should be adjusted to include the ‘Cluster CIDR’ of the Cloud PKS environment via the VPC peering link.

In order to successfully use the connections feature in Cloud PKS, there are a few considerations to keep in mind:

• Traffic over the VPC peer is internally routed over the AWS network and may incur charges
• AWS Security Groups can be used to prevent unauthorized access to the Cloud PKS cluster
• Intra-region DNS is supported, cross-region DNS is currently supported
• Up to fifty active connections per production cluster are supported
• IP ranges must not overlap between the cluster network and the remote VPC
• MTU size is limited to a maximum of 1500 bytes

This post provides an overview of the Cloud PKS ‘connections’ feature via AWS VPC peering. We’ve discussed use cases for this feature, as well as the architecture surrounding it and the mechanics involved in provisioning a new connection. More information can be found in the Cloud PKS documentation here: https://docs.vmware.com/en/VMware-Kubernetes-Engine/services/com.vmware.kubernetesengine.using.doc/GUID-8AC7B072-891D-4776-86CC-FDC02C5D3CCF.html

I’ll be speaking live about this topic at AWS re:Invent 2018! If you’re interested in attending the session, it will be at the VMware {code} booth (#Q505 in Aria West, Level 1, Bristlecone Ballroom, The Quad) on Wednesday, November 28 from at 12:30 PM. SPO35 — Connecting native AWS Services with VMware Kubernetes Engine #VMwareCode45. I look forward to seeing everyone then!