The Mechanics of Services in Kubernetes

BY Dan Illson
Jun 28 2019

Before I get started, I’d like to give special thanks to two of my colleagues at VMware, Duffie Cooley and Scott Lowe of the Kubernetes Architecture team for helping me think through and better understand this subject. I’d highly recommend following both for their thoughts on cloud native architectures and Kubernetes.

Services are one of the most commonly configured and used configuration object in Kubernetes. Through I used them frequently as I was learning how to use Kubernetes, I found recently that I didn’t understand how services interacted with the other objects within a Kubernetes cluster. In this post, I’d like to explain what I’ve learned (with help from my colleagues acknowledged above) to help others who may be confused by these objects.

What is a Service in Kubernetes?

The service resource in Kubernetes is an abstract method to expose an application as a network service. The Kubernetes documentation gives this definition:

In Kubernetes, a Service is an abstraction which defines a logical set of Pods and a policy by which to access them (you’ll sometimes see this pattern called a micro-service). The set of Pods targeted by a Service is usually determined by a selector.

For example: consider a stateless image-processing backend which is running with 3 replicas. Those replicas are fungible—frontends do not care which backend they use. While the actual Pods that compose the backend set may change, the frontend clients should not need to be aware of that, nor should they need to keep track of the set of backends themselves.

The Service abstraction enables this decoupling.

There are additional considerations introduced in the section above. The first is that services use a policy to determine how the pods targeted by the service will be accessed. This behavior is governed by the ‘ServiceType’ attribute.

There is an important piece of this puzzle that hasn’t been identified yet, but touches on the last sentence of the quote above. In Kubernetes, behind a service is the concept of ‘endpoints’ in kubernetes. A kubernetes endpoint consists of the IP address and port at which a kubernetes pod can be reached. The endpoints resource for a given service may contain multiple entries for services which represent multiple pods.

The final (and in my mind more complex) item is the concept of ‘targeting’ a set of pods with a selector. This is the concept I had the most trouble fully understanding as I worked on writing the Kubernetes manifests for the ACME Fitness demo application our team created. However, once I better understood the concept of a kubernetes endpoint list, the purpose of selectors and the labels which define them became much clearer to me.

The ‘ServiceType’ Attribute

In order to access a kubernetes service, a ‘ServiceType’ must be selected for it. At present, there are four service types available in kubernetes:

  • ClusterIP
  • NodePort
  • LoadBalancer
  • ExternalName

ClusterIP is the default ServiceType in kubernetes if the field isn’t explicitly given a value. This type exposes a given service on a cluster internal IP address which will leave that service unreachable from outside the kubernetes cluster unless additional components are configured to expose it. This configuration is typically used for services with which direct user interaction is not intended or wanted.

The NodePort type exposes a given service on a specific static port on the IP address of each node in the cluster (hence the term “NodePort”). A ClusterIP Service is automatically created, to which the NodePort Service routes. The NodePort Service is then reachable from outside the cluster by requesting NodeIPAddress:NodePort. This is a useful configuration for testing access to individual services. Our team uses it extensively which developing applications and debugging interactions between the services that makeup those applications.

A LoadBalancer service is fairly simple in concept. This type exposes the specified service externally to the cluster via the cloud provider’s load balancer. NodePort and ClusterIP services, to which the external load balancer routes, will automatically be created. This configuration is recommended for its durability and availability. However, for this may require some additional infrastructure configuration depending on how the kubernetes cluster(s) in use have been deployed.

Finally, an ExternalName service type maps the service (and its endpoints) to a domain name by returning a CNAME record. No proxying is setup in this case. This configuration is intended to make external services available within kubernetes clusters.

NOTE: Kubernetes ingress can also be used to expose services outside of the kubernetes cluster. Ingress is not a service type. However, it does act as a cluster entry point, and can consolidate multiple routing rules into a single resource capable of exposing multiple services behind a single IP address.

Kubernetes Endpoints

Kubernetes endpoints are an odd resource. On one hand, I built several Kubernetes clusters and deployed applications to them before I understood the relationship between kubernetes endpoints and services. Despite this, endpoints can be powerful objects for both traffic control and troubleshooting.

To begin to understand endpoints, let’s take a look at a running application. First, I’ll call up a list of the kubernetes services exposed as part of this application:

$ kubectl get svc
NAME            TYPE           CLUSTER-IP   EXTERNAL-IP   PORT(S)        AGE
cart            ClusterIP      10.0.0.18    <none>        5000/TCP       65d
cart-redis      ClusterIP      None         <none>        6379/TCP       65d
catalog         ClusterIP      10.0.0.67    <none>        8082/TCP       65d
catalog-mongo   ClusterIP      None         <none>        27017/TCP      65d
frontend        LoadBalancer   10.0.0.122   <redacted>    80:30417/TCP   64d
kubernetes      ClusterIP      10.0.0.1     <none>        443/TCP        138d
order           ClusterIP      10.0.0.101   <none>        6000/TCP       65d
order-mongo     ClusterIP      None         <none>        27017/TCP      65d
payment         ClusterIP      10.0.0.111   <none>        9000/TCP       65d
users           ClusterIP      10.0.0.246   <none>        8081/TCP       65d
users-mongo     ClusterIP      None         <none>        27017/TCP      65d

This is the ACME Fitness demo application the cloudjourney.io team has developed. This application consists of ten kubernetes services if we exempt the kubernetes service. As previously discussed in this blog, these services are abstracted from the pods they forward traffic to. In order to look at the number of pods in play here, I’ll run another command:

kubectl get pods
NAME                            READY   STATUS    RESTARTS   AGE
cart-759bbd6c74-2mnvq           1/1     Running   0          65d
cart-redis-758799fd4d-kzfzp     1/1     Running   0          65d
catalog-599fcd856f-f42b7        1/1     Running   0          65d
catalog-mongo-cf5d76b4b-m2522   1/1     Running   0          65d
frontend-c56b64cf7-6lb4f        1/1     Running   0          64d
order-8469f97f54-vjt6v          1/1     Running   0          65d
order-mongo-5748bfc8f7-flxvz    1/1     Running   0          65d
payment-6b9df9cf4c-4rscf        1/1     Running   0          65d
users-64967f5485-rtrqb          1/1     Running   0          65d
users-mongo-677c747869-8hx7n    1/1     Running   0          65d

In this case, each of these services is proxying a single pod. But these commands don’t give a complete picture of the objects involved in passing traffic between kubernetes pods. To really investigate these interactions, let’s dig into one service in particular, the order service. I’ll start with a kubectl describe service command:

$ kubectl describe svc order
Name:              order
Namespace:         default
Labels:            app=acmefit
                   service=order
Annotations:       kubectl.kubernetes.io/last-applied-configuration:
                     {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"acmefit","service":"order"},"name":"order","namespace":"...
Selector:          app=acmefit,service=order
Type:              ClusterIP
IP:                10.0.0.101
Port:              order  6000/TCP
TargetPort:        6000/TCP
Endpoints:         10.2.3.10:6000
Session Affinity:  None
Events:            <none>

In the output of this command, endpoints make their first appearance. Endpoints: 10.2.3.10:6000, indicates that the service proxies a single endpoint. The IP address represents a node on the cluster internal subnet (“order” is a ClusterIP service after all) and the port listed was indicated in the output of the kubectl get services command run previously. Multiple endpoints can exist for a given service. This is easily observable if we scale the number of “order” pods beyond one.

$ kubectl scale deployment order --replicas=2
deployment.extensions/order scaled

$ kubectl describe svc order
Name:              order
Namespace:         default
Labels:            app=acmefit
                   service=order
Annotations:       kubectl.kubernetes.io/last-applied-configuration:
                     {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"acmefit","service":"order"},"name":"order","namespace":"...
Selector:          app=acmefit,service=order
Type:              ClusterIP
IP:                10.0.0.101
Port:              order  6000/TCP
TargetPort:        6000/TCP
Endpoints:         10.2.3.10:6000,10.2.3.14:6000
Session Affinity:  None
Events:            <none>

$ kubectl get pods | grep order
order-8469f97f54-tf4qw          1/1     Running   0          1m
order-8469f97f54-vjt6v          1/1     Running   0          65d
order-mongo-5748bfc8f7-flxvz    1/1     Running   0          65d

Looking at the above output, the order service now proxies two pods. This behavior is based on selectors logic in kubernetes, which will be discussed in the next section of this post. To look at all of the endpoints in a given kubernetes environment, kubectl get endpoints can be used:

$ kubectl get endpoints
NAME            ENDPOINTS          AGE
cart            10.2.3.2:5000      65d
cart-redis      10.2.3.3:6379      65d
catalog         10.2.3.5:8082      65d
catalog-mongo   10.2.3.4:27017     65d
frontend        10.2.3.13:3000     64d
kubernetes      10.1.146.99:6443   138d
order           10.2.3.10:6000     65d
order-mongo     10.2.3.9:27017     65d
payment         10.2.3.6:9000      65d
users           10.2.3.8:8081      65d
users-mongo     10.2.3.7:27017     65d

However, endpoints can also configured manually via manifest. As a means of example, say I’m interested in building an endpoint object and service for an arbitrary web server. If I were, I could deploy a manifest such as the following via ‘kubectl’:

kind: "Service"
apiVersion: "v1"
metadata:
  name: "external-web"
spec:
  ports:
    - protocol: "TCP"
      port: 80
      targetPort: 80
---
kind: "Endpoints"
apiVersion: "v1"
metadata:
  name: "external-web"
subsets:
  - addresses:
    - ip: "10.10.100.100" #The IP Address of the external web server
    ports:
      - port: 80
    - ip: "10.10.100.101" #The IP Address of another external web server
    ports:
      - port: 80

The ability to manually define endpoints is powerful, but for most users it will be a secondary use case to troubleshooting. Most of the kubernetes using population will have an indirect relationship to endpoints via labels and selectors, which are discussed in the next section.

Labels and Selectors

Most pieces on this topic address labels and selectors before endpoints. This is likely because most users will interact with selectors far more often than they manually configure endpoints (if ever). However, I think once endpoints are understood, the mechanics of selectors are fairly straightforward.

Remember the ACME Fitness application from the previous section? The endpoints in that example were not defined manually. Those values were derived by a LabelSelector.

In Kubernetes, labels are key/value pairs which are attached to objects (e.g. pods). Labels can then be used to select subsets of objects, and each object can have a set of labels defined. Once a set of objects have been labeled, a label selector can be used to identify a set of objects.

Once a selector has been defined in a service specification, pod(s) with the correct set of labels will be added as endpoints to a service with the corresponding selector. While selectors are straightforward conceptually, their implementation can be a bit trickier. There are two common issues I’ve encountered while using selectors:

  1. Ensuring that all of the specified MatchLabels are included in the Selector specification
  2. Applying the labels at the correct level of the object hierarchy

With regards to the first item, all of the labels much be matched when more than one are listed within a selector. This can be seen in a section of the manifest for the order service and deployment from the example previously referenced:

apiVersion: v1
kind: Service
metadata:
  name: order
  labels:
    app: acmefit-order-total
    service: order
spec:
  ports:
    - name: http-order
      protocol: TCP
      port: 6000
  selector:
    app: acmefit-order-total
    service: order
---
apiVersion: apps/v1 # for versions before 1.8.0 use apps/v1beta1
kind: Deployment
metadata:
  name: order
  labels:
    app: acmefit-order-total
    service: order
spec:
  selector:
    matchLabels:
      app: acmefit-order-total
      service: order
  strategy:
    type: Recreate
  replicas: 1
  template:
    metadata:
      labels:
        app: acmefit-order-total
        service: order
    spec:

Under the service specification there is a selector defined which consists of two labels. The two keys are app: and service:. If you’ll examine the rest of the manifest section above, those two label keys are always referenced together to avoid the first pitfall above. As for the second issue I mentioned, where the labels get applied is important. In the example above, a kubernetes deployment object is defined. But, the pods which will need to have the correct labels applied to them are not explicitly defined. Instead, the pods will be created by the deployment controller. In order to ensure that the correct information is propagated from the deployment to the pods. To accomplish this, the labels in question are defined in the template: section of the deployment specification. The labels have also been defined on the deployment objects, but this does not effect the pods spawned by the deployment.

Hopefully the examples above clarify some of the mechanics within services in Kubernetes. I’d like to thank Duffie and Scott again for their help in better understanding these concepts. As always, I look forward to discussing any of the topics in my blogs with the greater community either on twitter or in person in the near future.