Stateful Microservices

---

Since our first increment did not persist any data, it wasn’t very useful for our customers. The next high-priority task assigned to our team is to add state to the application. Therefore, in this iteration, our team will build a stateful back end for our application. In this chapter, we will learn several useful concepts that will help you deploy and configure stateful applications on Kubernetes.

To persist temporary data, you can use the storage attached to a container. However, storage in a container is ephemeral, and any data saved in the container will be lost between container restarts. Saving data in a store outside of the container solves many challenges associated with sharing data between containers and maintaining data beyond the lifetime of containers.

Stateful sets

Stateful microservices require one or more of the following characteristics:

• Unique network identifiers that remain the same across pod scheduling operations.
• Persistent storage.
• Ordered deployment and ordered scaling.
• Ordered and automated rolling updates.

The StatefulSet controller manages a group of pods while providing each pod a stable and unique identity, and specific scaling order. Just like the Deployment or ReplicaSet controllers, the StatefulSet controller takes a StatefulSet object as input and makes cluster changes to reach the desired state.

StatefulSet has a dependency on a particular type of service that needs to be provisioned before it, called a headless service. A headless service doesn’t have a cluster IP, but it provides network identity in the form of individual endpoints to all the pods that are associated with it. A client connecting to a pod will need to specify the endpoint of the specific pod that it wants to access.

Let’s create a stateful microservice now to understand it in detail. As we previously discussed, we need to create a headless service before we create a stateful set. The following code listing will create a headless service for us. This looks a lot like the previous service that we created, with the difference being that we ask Kubernetes not to provide a cluster IP to this service (which is what makes this a headless service).

Code Listing 37: Headless service

Next, we will create a StatefulSet for our service. Observe the following spec fragment that will create a StatefulSet for our API. We have deliberately left out some parts of the spec to make it easier to understand.

Code Listing 38: StatefulSet

You might have noticed that we are using a policy for getting images from the container registry whose value we have specified in the imagePullPolicy field. This field affects how kubelet attempts to pull an image from the container registry. The following are the various values that this field can take:

• Never: The image exists locally and kubelet should not attempt to pull the image.
• Always: kubelet should pull the image every time the pod is restarted. This setting helps during development, as you don’t need to provide a new tag for the image every time you fix a bug and build a new image. If you don’t supply the image pull policy and don’t specify a tag except latest for the container image, then the policy Always is applied.
• IfNotPresent: kubelet will attempt to pull the image if it is not locally available. If you omit the policy in the specification and supply a tag for your image, except the latest tag, then the policy IfNotPresent is applied.

To view the API that you just deployed, you can use the proxy verb. proxy is quite helpful in debugging deployed services that don’t have an external IP. In your terminal, execute the following command to start the proxy.

Code Listing 39: kubectl proxy

This command creates a proxy that we can access on localhost:8001.

Tip: By default, kubectl proxy works on the local machine only. If you are running a remote cluster, then you need to enable remote access in the desired service, or have cluster credentials.

To access your service running on the pod remindmeapi-statefulset-0 via the proxy, you can navigate to the following URL on your browser:

http://localhost:8001/api/v1/namespaces/default/pods/remindmeapi-statefulset-0/proxy/index.html

This URL will take you to the Open API specification of the API (v1), which looks like the following.

Figure 29: Remind Me API v1 open API definition

We have accessed this service through a proxy. However, our use case is to enable this service to communicate with the web application, and for that purpose we need to get the Cluster DNS record that points to this service. Let’s quickly discuss how we can interact with the Kubernetes DNS service to find out the DNS records that map to the location where our service is available.

Volumes

The concept of volumes is easy to understand. They are directories with or without data that are accessible to the container. There are different types of volume options available, each with their unique characteristics. Some of the popular volumes available are:

• Downward API: This is used to expose pod configuration to the container.
• emptyDir: This can be used to provide temporary storage to the containers in your pods. It can be backed on any medium available to the local node.
• secret: A secret volume is an in-memory directory that is used to pass sensitive information such as passwords to pods.

To use a volume, you need to specify it under the volumes section in your specification, and reference it under a volumeMounts section of your container to specify where you want to use it. Here is a small YAML snippet to illustrate the specification.

Code Listing 40: Specifying volumes

Volumes persist their data across container restarts and crashes. However, when a pod is deleted from the node, then volumes are also eliminated.

Persistent volume and persistent volume claim

Unlike volumes, persistent volumes can retain your data independently of your pod lifecycle. A persistent volume is backed by persistent storage that may live outside of the cluster. Some of the most popular persistent volumes use cloud storage, for example, awsElasticBlockStore, azureDisk, or gcePersistentDisk, available from public clouds such as Azure, AWS, and GKE.

Kubernetes separates the need for storage by applications from the actual storage implementation by using persistent volume and persistent volume claims. As an application developer, you specify the storage need of your application using a persistent volume claim (PVC). You can think of a PVC as an instruction to the cluster to give the application persistent storage of X capacity with the features that you need.

An administrator, on the other hand, defines one or more storage classes for the cluster. A storage class defines storage policies such as backup policy and quality-of-service for persistent volumes that are available to the cluster. All the volumes available from the cloud have storage classes predefined. For development, Minikube or Kubernetes on Docker for Windows use a particular storage class, which uses the volume of type HostPath.

Adding persistent volume to our service

The stateful back-end service that we are building in this increment uses the LiteDB database, which is a small, embedded NoSQL database. For complex applications, you might want to use SQL Server or Redis back end, but the system requirement would mostly remain the same—use a persistent volume to store data.

Specifying a persistent storage volume for your application requires the developers and operations to provide the following configurations:

Table 3: Persistent volume workflow

To support persistent volumes in our application, we will use the simplest persistent volume types—the HostPath in our application. Since the storage class for this PV is preconfigured in Minikube (or Kubernetes on Docker), we will skip the step of specifying the storage class for our volume.

Create persistent volume

Add the following code to your application spec to create a HostPath persistent volume.

Code Listing 41: HostPath persistent volume

This configuration will use the data directory on the node to emulate network attached storage. It is a great and lightweight option for building and testing your applications.

Create persistent volume claim

A pod requests PersistentVolumeClaim to request persistent storage. Add the following configuration to your spec for creating a PVC.

Code Listing 42: Persistent volume claim

The configuration request simply states that it requires 3GB of storage with read/write access to a single node. Different types of volumes can support different types of access modes, and you can use any of the supported access modes in your claim. For example, NFS (a type of volume) can support multiple read/write clients. However, you can create a read-only persistent volume claim on NFS so that the pod cannot write data to the volume.

The following access mode values are currently supported:

• ReadWriteOnce (RWO): A single pod can mount this volume, and can read and write data to it.
• ReadOnlyMany (ROX): Many pods can mount this volume, but only read data from it.
• ReadWriteMany (RWX): Many pods can mount this volume and read and write data to it.

Be sure to check the documentation of the persistent volume to see what it can support.

Map persistent volume claim to the application

Finally, we will map the persistent volume claim to the application and mount the persistent volume on the container that will host our application.

Code Listing 43: Mapping PVC to application

You can deploy these changes to your cluster and check the status of resources that you configured by executing the following commands.

Code Listing 44: Get PVC

This command will generate an output like the following. You can see that the status of the persistent volume claim is set to bound, which means that the persistent volume claim is now bound or attached to your persistent volume.

Figure 30: kubectl get persistent volume claims

To check the list of persistent volume, use the following command.

Code Listing 45: Get persistent volume

Figure 31: kubectl get persistent volume

Let’s launch an interactive terminal in our container to view the file system. To find out the name of the pod, execute the following command.

Code Listing 46: Get pods

Figure 32: kubectl get pods

This will display all the pods deployed in the default namespace.

Note: If you have used a namespace while creating the service, then you need to specify the namespace tag to the get pods command.

Next, execute the following command to launch an interactive shell on the pod.

Code Listing 47: Launching shell inside container

Note: The syntax presented in the previous code listing is a well-known command for launching a shell with Docker.

As shown in the output, execute the commands inside the terminal to see the contents of the data directory that we mounted previously.

Figure 33: kubectl launch interactive terminal

Adding secrets to an application

In general, applications use configurations to act as knobs for modifying the application behavior. These configurations are visible to team members and systems internally, and they do not generally pose a risk to the security of the system. Kubernetes supports storing configurations in a Kubernetes resource called ConfigMap. ConfigMap is a very simplistic system that saves data in the form of key value pairs. It does not support encryption, so it is not suitable for storing sensitive information. You should store well-known configuration values, such as the API version and the name of environment (development, test, production) in ConfigMap.

Most of the applications rely on security settings, such as database connection strings and token decryption keys, among other things, for functionality. Such configurations require a more controlled mechanism for safety. Kubernetes supports saving application secrets in a resource that is aptly named secrets.

Kubernetes secrets are internally stored in Base64 encoding format, and they are not persisted on disks, but instead present in the memory of the pods. Secrets, just like other resources, are persisted in the etcd cluster, and they are not encrypted by default. Therefore, for production clusters, ensure that you use the encrypting-secret-data-at-rest feature of Kubernetes to safeguard confidential information.

Exposing Secrets and ConfigMaps to a pod is very similar. You can choose to expose secrets as environment variables, or as a file in a volume that can be mounted on the pod. Our application relies on secrets to store a database connection string, for which we use the following configuration.

Code Listing 48: Create secret

Note that we have our secret stored in Base64 encoded format.

Note: ConfigMaps should only be used for nonsensitive configuration, while for sensitive configuration, you should use secrets.

To mount this volume to the container, we will use the following (familiar) configuration.

Code Listing 49: Mount secret

You can check out the complete code listing for this configuration on the companion GitHub repository for this title.

To get all secrets present in the cluster, you can execute the following command.

Code Listing 50: Get secrets

This command will generate output like the following. The secret that we described in our specification is called api-secret. The additional data fields in the output show the type and number of key-value pairs in the secret.

Figure 34: Output get secrets

You can use the following command to see the secret as a Base64-encoded string. You can decode this string to view the secret in plain text.

Figure 35: View secret

We are now ready to deploy the new application on our cluster. This time, we have split the specifications into two separate files, servicepec.yaml and webspec.yaml, for ease of maintenance.­ Execute the following command to apply all the specifications in the kube-manifests folder to your cluster.

Code Listing 51: Apply specification from folder

At this point in time, your application will be backed by a stateful API, and the reminders that you save will be persisted in a database. However, you might have noticed that we are not using StatefulSet and ReplicaSet objects in our specifications, and there is a reason for that: Kubernetes abstractions. Let us discuss how the Kubernetes objects are structured, and which objects you should use for maximum flexibility in building your applications.

Kubernetes abstractions

The Kubernetes API uses abstractions such as services, pods, etc. to help you define the cluster desired state, such as applications that you want to run on the cluster, number of replicas of an application, and so on.

The fine-grained components that define the cluster state are called objects. You can use these objects to define which applications should run on your cluster, the network and disks that should be available to your cluster, and other information about your cluster. Kubernetes objects consist of the following:

• Pod
• Service
• Volume
• Namespace

Kubernetes contains higher-level abstractions as well, called controllers. The controllers internally use the objects and provide additional functionality and convenient features that would otherwise require you to configure multiple objects. For example, in Figure 36 you can see how the controllers build upon other objects as well as other controllers.

Figure 36: Controllers

Controllers in Kubernetes include the following:

• ReplicaSet
• Deployment
• StatefulSet
• DaemonSet
• Job

It is recommended to always use controllers rather than objects, unless you have a very specific use case that the controllers don’t currently support. For example, the StatefulSet controller is an abstraction on a pod object that guarantees ordering and uniqueness of pod objects. Therefore, if you want to build stateful applications, then you should use StatefulSets rather than pod objects. However, no controller extends the service object, and therefore, services should be used to expose a group of pods externally.

A service in Kubernetes is a grouping object that defines a logical set of pods and a policy to access the pods that are a part of the set. The pods themselves are another class of objects in Kubernetes. We use services to logically aggregate pods in a cohesive unit in such a manner that any pod connected to a service will respond to a request in a similar manner. The following diagram shows how services are used to aggregate pods in a Kubernetes cluster.

Figure 37: Pod aggregation

Services themselves differ from each other based on how they allow traffic to reach them. The following diagram illustrates how the different types of services accept traffic originating from outside the cluster.

Figure 38: Service types

The most basic service type is the node port. Traffic arriving on a specific port of any node is forwarded to the service, which in turn directs the traffic to a pod in a node. A NodePort service is generally used with an external LoadBalancer so that the client doesn’t need to know about individual node IPs.

The next type of the service is the load balancer. This service leaves the implementation of LoadBalancer up to the cloud vendor, such as AWS or Azure. Any traffic that arrives on a certain port of the load balancer will be forwarded to the mapped service. Each service that you expose through the load balancer gets its own address, so this option can get quite expensive if you have a high number of services running in your cluster.

The third type of service is the cluster IP, which is also the default service type. The cluster IP service gets a unique cluster IP address that can be used by other components within the cluster to communicate with the service. By default, traffic originating outside the cluster cannot reach this service. Even though this service is visible only inside the cluster, you might find that it is the most-used service in enterprise applications because it uses a router component called Ingress.

Ingress is a smart router that sits inside the cluster and can accept external traffic. There are many Ingress controllers available that you can use in a cluster, such as Nginx, Contour, and Istio, with each differing in terms of features. Fundamentally, an ingress maps a host such as https://a.hostname.com to Service A or https://hotname.com/b to Service B. Thus, a single ingress can be used to expose multiple services on the same IP address. Ingress is implemented using the Ingress Controller in Kubernetes.

Ingress provides additional features, such as TLS termination and Server Name Identification (SNI), so that they need not be implemented by individual services. Because a single IP address is used by the ingress, you pay only for a single IP address when integrating Ingress with Load Balancer on cloud, making it the most cost-effective option.

Another category of abstractions in Kubernetes depends on the lifecycle of pods. StatefulSet, DaemonSet, and job controllers collectively form this category.

Figure 39: Kubernetes abstractions

Jobs are used for applications that require a one-off pod to execute a workload, after which the container can be reused for other applications. We will use Jobs for sending out nightly emails to the users of the Remind Me application.

A DaemonSet controller is spawned on a pod in every node. DaemonSet is used for deploying background tasks that do not require user input. You will find monitoring and logging daemons, such as collectd and fluentd, implemented as DaemonSet.

StatefulSet handles the pod lifecycle events in a different manner to support sequential creation and deletion of pods, amongst other things. They also provide a unique and persistent identity to a set of pods and save pod state data in a persistent storage. Stateful sets are used to build stateful microservices (microservices that persist state data), and they are suitable to deploy applications such as Redis, SQL Server, and Kafka.

Interacting with the Kubernetes DNS service

We discussed in the first chapter that every Kubernetes cluster has a DNS service that is scheduled as a regular pod. Any communication among services happens using the DNS service, which contains records for all services running in the cluster. Often, to connect to a service, you need to know the DNS name of the service. For example, for the application that we are working on, we want to know the DNS name of the remind-me-api service so that we can supply it to our web application, remind-me-web.

For interacting with Kubernetes, a commonly used trick to interact with resources inside the cluster is to deploy a pod in the cluster that supports running a prompt. BusyBox is a commonly used tool for this purpose. It is a set of common Linux programs packaged together. One of its most useful features is getting a prompt running inside the cluster.

To create a pod in the cluster, execute the following command in your terminal.

Code Listing 52: BusyBox pod

This command will launch the BusyBox terminal, in which you can execute any command in the cluster.

Figure 40: Output get BusyBox pod status

Let’s find out what the DNS name for the remind-me-web-svc service is by executing the following command.

Code Listing 53: nslookup

When executed, this command will list the DNS records for our service.

Figure 41: Output nslookup remind-me-web-svc

As you can see, there is a single DNS name available for our stateless application, remind-me-web. Any traffic sent to the address remind-me-web-svc.default.svc.cluster.local will reach one of the randomly selected replicas of the service.

Tip: During development, each developer should provision their unique BusyBox pod in the cluster so that they can work independently of each other. If you want to reuse the terminal after a session, then find out the pod ID of the pod running curl (kubectl get pods), and then execute this command to relaunch the prompt: kubectl attach <POD ID> -c curl -i -t. To clean up the BusyBox deployment, execute this command: kubectl delete deployment curl.

We know that pods in a stateful set have a unique ordinal index. If you execute the following command, it will display the various pods that make up your stateful service.

Code Listing 54: Get pods status

In the following command output, you can see that the pods that make up the stateful set have a unique ordinal index in their names.

Figure 42: Output kubectl get pods for stateful set

Let’s execute the nslookup command for our stateful service to see what we get. This time, execute this command in your BusyBox: nslookup remind-me-api-svc. This will generate an output like the following.

Figure 43: Output BusyBox nslookup

As you can see, unlike the stateless service that we provisioned for the web application, each pod of the stateful service is individually addressable. Therefore, the client of a stateful service should implement a load balancing policy to ensure that the requests to stateful service instances are properly distributed.

Tip: There is a much simpler technique for running nslookup commands. You can execute the nslookpup command after opening an interactive terminal to an existing pod using this command: kubectl exec <running pod name> -it -- /bin/sh.

To keep things simple, we will send all the requests to the first pod (one with ordinal index 0) of the API service. Let’s set the URL of the API endpoint in the environment variable of the web application in the web application specification. You can read the complete code listing in the companion GitHub repository.

Code Listing 55: Front end manifest

This configuration will enable your web application to send requests to the back-end stateful service. This will help preserve the reminders across page refreshes and sessions.

Before concluding this discussion, let’s discuss an important aspect of services. The sequence of creation of resources is very important when it comes to defining services. If pods exist before services are defined, then the environment variables for the service will not exist within the pods that are part of the service. So in the previous specification, if we were to set the value of the GET_HOSTS_FROM environment variable to env, and create the remind-me-web-svc web application first and the remind-me-api-svc second, then the resolution might fail. This is because the existing pods won’t get the environment variables that contain the details of the service remind-me-api-svc. In such cases, you will need to recreate containers by scaling the pod count to 0 and then increasing the count. As a rule, always define and apply service resources before creating other resources.

Summary

In this chapter, we discussed how we can build stateful microservices applications on Kubernetes. Kubernetes abstracts persistent storage volumes from applications and distributes the responsibility to provision and apply persistence capabilities between developers and operations. Abstractions like these make Kubernetes an excellent collaborative platform for both developers and operations.

In the next chapter, we will discuss an important aspect of every enterprise-grade application, which is performing smooth upgrades and scaling applications on demand.