Overview
I have had a question come up a few times with customers and coworkers about how to reduce duplication when creating clusters with Tanzu Mission Control(TMC). The question or issue that is usually brought up is that the platform engineering team wants to be able to create clusters quickly and many of the settings between cluster creation are the exact same, thus having a lot of duplication between clusters. When looking at the TMC UI there's not a way to set custom defaults today to be able to remove the need to fill in every field each time you create a cluster. However, using the UI is probably not the approach a platform team wants to take to scale anyway. It's much more efficient to codify the clusters and automate the creation. In this post, we will walk through creating cluster templates and using the Tanzu CLI to create clusters with minimal inputs. We will focus on TKG Clusters mostly, but I will also provide some commands that work with AKS and EKS clusters are well.
Brief note on the Tanzu CLI
The Tanzu CLI can now be installed through standard package managers or directly via the binary from GitHub, see the install instructions here. The TMC standalone CLI is in the process of being deprecated so you will want to use the new plugins for the Tanzu CLI. They will be installed when you connect the Tanzu CLI to your TMC endpoint.
Templating clusters
Getting the base template
The CLI provides a way to get an existing cluster's spec, I always recommend going with this approach over trying to create one from scratch. Before we start templating the cluster, create a cluster through the UI. Then use the below command to pull the cluster's yaml spec back and save it in a file.
# for TKG
tanzu tmc cluster get <cluster-name> -m <mgmt-cluster> -p <provisioner> > template.yml
## for EKS
tanzu tmc ekscluster get <cluster-name> -c <credential> -r <region> > template.yml
## for AKS
tanzu tmc akscluster get <cluster-name> -c <credential> -r <resource-group> -s <subscription> > template.yml
The contents of the above command will look slightly different depending on the k8s provider you are using, but all of them will be a yaml file that describes all of the settings for the cluster.
Remove any extra fields
Just like when pulling back a resource from a k8s cluster there will be fields that we will want to omit from our template. For example the status
section. This will be different between providers, but because we are focused on TKG in this example I have listed the field that I omitted below. Technically many of these fields do not need to be removed since the API will just ignore them, but since we are creating a template and don't want a bunch of extra stuff that might be confusing we will remove anything that is not needed.
fullname.orgId
meta
labels
tmc.cloud.vmware.com/creator
annotations
creationTime
generation
resourceVersion
uid
updateTime
spec.topology.variables
extensionCert
user
- if you would like to provide your ssh public key, keep this one.clusterEncryptionConfigYaml
TKR_DATA
status
- entire section
All of the fields above are generated by TMC when the cluster is created. Your YAML file should now look similar to the one below.
fullName:
managementClusterName: h2o-4-19340
name: tmc-base-template
provisionerName: lab
meta:
labels:
example-label: example
spec:
clusterGroupName: default
tmcManaged: true
topology:
clusterClass: tanzukubernetescluster
controlPlane:
metadata:
annotations:
example-cp-annotation: example
labels:
example-cp-label: example
osImage:
arch: amd64
name: ubuntu
version: "20.04"
replicas: 3
network:
pods:
cidrBlocks:
- 172.20.0.0/16
serviceDomain: cluster.local
services:
cidrBlocks:
- 10.96.0.0/16
nodePools:
- info:
name: md-0
spec:
class: node-pool
metadata:
labels:
exmaple-np-label: example
osImage:
arch: amd64
name: ubuntu
version: "20.04"
overrides:
- name: vmClass
value: best-effort-large
- name: storageClass
value: vc01cl01-t0compute
replicas: 2
variables:
- name: defaultStorageClass
value: vc01cl01-t0compute
- name: storageClass
value: vc01cl01-t0compute
- name: storageClasses
value:
- vc01cl01-t0compute
- name: vmClass
value: best-effort-large
- name: ntp
value: time1.oc.vmware.com
version: v1.23.8+vmware.2-tkg.2-zshippable
type:
kind: TanzuKubernetesCluster
package: vmware.tanzu.manage.v1alpha1.managementcluster.provisioner.tanzukubernetescluster
version: v1alpha1
Templating with YTT
There are many options for templating files, but since this is a YAML file and we like Carvel YTT that is what is used for this example. I would highly recommend reading up on YTT and testing it out for different use cases, it's a very powerful YAML templating language.
Determine the variable fields
first, we need to determine which fields should be variable. This could be any field, but we want to reuse as much as possible as well. These fields are entirely up to you and your needs.
Template the fields with YTT
the YTT docs explain how to use data values in a YAML file. This is what we will be using to template the file. Using the same file from above the below template is what I have come up with.
#@ load("@ytt:data", "data")
fullName:
managementClusterName: #@ data.values.mgmt_cluster_name
name: #@ data.values.cluster_name
provisionerName: #@ data.values.provisioner
meta:
#@ if/end hasattr( data.values, "cluster_labels"):
labels: #@ data.values.cluster_labels
spec:
clusterGroupName: #@ data.values.cluster_group
tmcManaged: true
topology:
clusterClass: tanzukubernetescluster
controlPlane:
metadata:
#@ if/end hasattr( data.values, "cp_annotations"):
annotations: #@ data.values.cp_annotations
#@ if/end hasattr( data.values, "cp_labels"):
labels: #@ data.values.cluster_labels
osImage:
arch: amd64
name: ubuntu
version: "20.04"
replicas: 3
network:
pods:
cidrBlocks:
- 172.20.0.0/16
serviceDomain: cluster.local
services:
cidrBlocks:
- 10.96.0.0/16
nodePools:
- info:
name: md-0
spec:
class: node-pool
metadata:
#@ if/end hasattr( data.values, "node_labels"):
labels: #@ data.values.node_labels
osImage:
arch: amd64
name: ubuntu
version: "20.04"
overrides:
- name: vmClass
value: #@ data.values.cp_vm_size
- name: storageClass
value: vc01cl01-t0compute
replicas: 2
variables:
- name: defaultStorageClass
value: vc01cl01-t0compute
- name: storageClass
value: vc01cl01-t0compute
- name: storageClasses
value:
- vc01cl01-t0compute
- name: vmClass
value: #@ data.values.worker_vm_size
- name: ntp
value: time1.oc.vmware.com
version: v1.23.8+vmware.2-tkg.2-zshippable
type:
kind: TanzuKubernetesCluster
package: vmware.tanzu.manage.v1alpha1.managementcluster.provisioner.tanzukubernetescluster
version: v1alpha1
You can see that a number of fields now have YTT logic in them. Here is a quick breakdown of what we are doing.
#@ load("@ytt:data", "data")
- tell ytt to load data values into the data object#@ data.values.mgmt_cluster_name
- I won't go through every variable but this syntax is used to pull out a value from the values file that we will create in the next section.#@ if/end hasattr( data.values, "cp_annotations"):
- this syntax is also used a few times, this check to see if our values file has a field and if it does then it adds the field below. This is used becuase certain fields are optional.
There is a lot more that can be done when templating with YTT and this is a fairly basic example. The docs on YTT have a lot of example that can be referenced.
Create a values file
The values file is what we will use to specify the values for all of the fields that we have templated. This is really just a yaml file with a single line of YTT at the top that let's the YTT engine know that the fields are to be used as data values. Since this is just plain yaml, you can also have nested fields etc. Below you can see the values file that was created to work with the above template.
#@data/values
---
mgmt_cluster_name: h2o-4-19340
cluster_name: cluster-from-template
provisioner: lab
cp_vm_size: best-effort-large
worker_vm_size: best-effort-large
cluster_group: default
cluster_labels:
test: test
Create a new cluster
Finally, we can combine these two files into a command that with generate our cluster configuration and then apply it to TMC.
If you want to test the outputs of the templating prior to sending it to TMC you can simply run the below command which will generate the resulting yaml and output to stdout
.
ytt -f values.yml -f template.yml
The next command will generate the resulting yaml and instead of sending it stdout
, it will pass it directly to the Tanzu CLI and start creating the cluster.
tanzu tmc apply -f <(ytt -f values.yml -f template.yml)
If you are using EKS or AKS the command is slightly different since support is not yet added to the apply
command for those clusters. Hopefully it will be added soon. You can still do this with the create
and update
commands though. See the examples below.
# redirecting output does not work currently for the create commands
#EKS
ytt -f values.yml -f template.yml > eks.yml
tanzu tmc ekscluster create -f eks.yml
#AKS
ytt -f values.yml -f template.yml > aks.yml
tanzu tmc akscluster create -f aks.yml
Summary
I summary, this article should give you a good idea of how to make re-usable templates for TMC created clusters. This could even be used to create "plans" for clusters so that it makes self service easier for teams. An example of using this for slef service would be to have a pipeline that executes the apply commands and allow developers, operators, etc. to manage their values file in a git repo. This would provide a nice gitops driven way to create on demand clusters with the ability to apply policy and restrict which fields could be changed.