TKG 2.1 – Initial Impressions

At the time of writing this post, TKG 2.1 has been out for just under a week. TKG 2.1 is a major milestone, showing a truly promising future for TKG as a product. While there are still many rough edges which I will cover some of them in this post as well as some future posts, the path forwards is full of promise. Key Changes In TKG 2.1 Cluster Class Better Air Gapped Support More flexibility OOTB GitOps Friendly Better Windows Support More Powerful TMC integration More Secure Networking Improvements Certificate Management Improvements Clear Feature Support Definitions Lets take a quick look at each of these key topics, what the include today, what will be coming hopefully soon, and what the current limitations are in these areas. Cluster Class The challenges that brought us here TKGm has always been a highly customizable Kubernetes Distribution, however it had some serious drawbacks. In the 1.x series of TKGm, all customizations were performed client side, using YTT which is a great tool, but as it was client side rendering, this meant that we had a serious challenge in managing cluster creation. If we wanted to offer the ability for multiple users to create and manage clusters, but needed to perform any customizations using YTT overlays, we needed to make sure we either had a sort of bastion host, where all cluster creations were performed from, or me had to make sure we synchronized the relevant YTT overlays between all the relevant peoples machines, to make sure clusters were customized and configured as needed. In general the idea of server side rendering is a much better idea, and removes a lot of toil for operations. Why do we need Cluster Class Cluster Class is a feature in the upstream ClusterAPI (CAPI) project which is itself the core of TKG. Cluster Class was inspired in part by the TKGs model of having a single CR for creating a cluster, in the TKGs world called a TanzuKubernetesCluster (TKC), which was an abstraction layer above CAPI, and in the backend when you would create such an object, a Kubernetes controller, would generate all the needed CAPI objects to fulfil the intent described in the TKC YAML. While the TKGs model of applying a single resource with my desired state and having the platform deal with all the backend logic and details is a great idea, there were serious limitations with this approach. The TKGs model severly compromised on customization capabilities in favor of a simple UX. While that model is suitable for some, it is not sustainable at scale. Cluster Class is the upstream solution, which set as its goal to really be the perfect middle ground, supporting customization while not compromising on an easy and simple UX for end users. As it says in the Kubernetes blog that announced Cluster Class: The idea behind ClusterClass is simple: define the shape of your cluster once, and reuse it many times, abstracting the complexities and the internals of a Kubernetes cluster away. The general idea is to have a template of a cluster and its resources in the management cluster and then whenever we create a cluster, we can reference this template and supply the values that are specific to this cluster, this allows for reuse of almost all of the boilerplate, and to allow us to only supply when creating a cluster, the needed values that are specific to this exact cluster we are creating. How does this fit into TKG In TKG 2.1 by default clusters are now created using the Cluster Class mechanism. When we bootstrap a management cluster, A TKG Default Cluster Class is created for us in the default namespace, that we can use to start creating workload clusters. As mentioned above, a Cluster Class has an idea of variables, allowing us to pass in cluster specific details we care about. In TKG, VMware have basically rewritten all of the YTT logic which supported many different customizations and configurations into Cluster Class variables and patches, giving us the same flexibility we had in TKG 1.x, with the benefit of it all being server side. Current Limitations While the idea of cluster class is great, and we even have docs from VMware on how to create our own Cluster Classes to extend what we get OOTB from VMware, the process of creating a cluster class is not as easy as writing some simple YTT templating and placing it in a folder. The templating mechanism and patching is done via JSON Patches and Go Templating, which while powerful are not very easy to grasp at the beginning, then again YTT is also a beast and not very easy for new users to get started with, so its really a matter of preference, but personally I find the new templating mechanism much more clunky and difficult then YTT, however it is a price worth paying in my mind, for all the benefits we gain from the templating happening now server side. There are also some weird decisions that were made in the default Cluster Class, which I know at least some of them are being worked on. While Cluster Class is new for us end users, it is also a new mindset for VMware, and their definitely are some growing pains, but the direction of Cluster Class still seems very promising. Better Air Gapped Support While many have the privilege of having internet access from their environments, many others don’t have this privilege. Till TKG 2.1, the process is TKGm for setting up an air gapped environment was sub optimal to say the least. One of the key annoyances, was the process of relocating images to the air gapped environment. This process required multiple CLI tools, and the download and running of bash scripts, which were meant to be downloaded from a specific commit on github, and overall the UX was not good. Beyond this, we also had a requirement that the customer already have container registry we can use to push the images to, which in some cases simply is not the case. In TKG 2.1, we have answers now to both of these issues. Image Relocation We now have a new CLI command in Tanzu CLI called "isolated-cluster" which helps with this process. To download the needed images, and artifacts one simply needs to run now from a machine with internet access tanzu isolated-cluster download-bundle \ --source-repo projects.registry.vmware.com/tkg \ --tkg-version v2.1.0 This will download all the needed images and artifacts, which you then move into your air gapped environment and then you simply run a command to upload all the configuration to your air gapped registry: tanzu isolated-cluster upload-bundle \ --source-directory <SOURCE-DIRECTORY> \ --destination-repo <DESTINATION-REGISTRY> \ --ca-certificate <SECURITY-CERTIFICATE> Once this is done, you are ready for deploying TKG in your air gapped environment! Image Registry Requirement While making the relocation of images easier is great, we also mentioned the issue of needing a registry in advance to host the TKG images which many organizations may not have. This is solved now also in TKG 2.1, with a new artifact being released with TKG, which is an official OVA for deploying Harbor. This OVA allows us to easily stand up a harbor instance in an air gapped environment, which makes it really easy to get started. Future Improvements While the relocation of images is more streamlined, I wish there was simply a downloadable artifact on the VMware Customer Connect portal, with all the needed binaries for an air gapped installation. requiring the users to run the "isolated-cluster download-bundle" command, while not terrible, is not the ultimate UX I hope we will have in the future. More flexibility OOTB As mentioned before, TKG has always had a large amount of customization capabilities built in to it, however in TKG 2.1 this has been increased now and many important features, which previously required custom YTT overlays, are now possible directly via Cluster Class Variables, making our lives easier. Some of the key features that have been exposed now which previously required custom overlays include: Adding a CA to the kubernetes nodes trust store Configuring Control Plane component arguments Advanced Antrea Configurations and more… This added functionality, makes the number of instances where writing custom YTT overlays or Cluster Classes go down drastically, in the end giving a much better UX for end users and especially for platform operators who need to maintain these customizations and validate them with each and every release of TKG. Current Limitations While there is support for advanced configurations, the documentation on the use of them is limited in the best cases, and non existent in the majority of cases making the task of finding what needs to be set, where to set it, and what it actually does, a not so simple task. Hopefully as time goes on, the documentation will improve, and these features will also improve, and will be less rigid then they are today. GitOps Friendly When dealing with Kubernetes, the industry has standardized on using GitOps as the optimal mechanism for managing kubernetes configuration and deployment. While the industry is split on which GitOps tools to use, be it Flux, ArgoCD, Kapp Controller, Rancher Fleet or any other tool, GitOps is the de-facto standard and for good reason. Pre TKG 2.1 challenges With the 1.x line of TKGm, GitOps for cluster management simply was not a viable solution. Because templating was done client side, managing cluster configurations via a GitOps mechanism was simply not a real option. we could always store our cluster configuration files in git, but the mechanism for applying a cluster manifest, performing an upgrade of a cluster or scaling a cluster, was always a manual step. GitOps Opportunity of TKG 2.1 With the cluster templating now happening server side via Cluster Class, we can easily manage our cluster definitions in a GitOps manner. We already have Kapp Controller installed in our management cluster which provides an easy mechanism for GitOps styled deployments, allowing us to easily stor our cluster configurations in a Git Repo, and having Kapp Controller perform the deployment for us, which is a huge benefit. This makes the self service opportunities of a Cluster-As-A-Service style platform based on TKG, an easy option to implement, and allows us to standardize the management of our clusters in a controlled, and simple manner. While this approach is not documented in the TKG docs, I expect that overtime, as the Cluster Class implementation matures, and the growing pains are behind us, this will end up being the standard approach for managing TKG clusters, which will be really awesome to see. Better Windows Support Windows Support has been around in TKGm officially since TKGm 1.5. While Windows was supported till now, it had some serious limitations that severely impacted its usability. While not all of the limitations have been solved, the improvements in TKG 2.1 are huge, and show a bright light ahead for Windows in TKG. The challenges pre 2.1 Pre TKG 2.1, Windows support was extremely limited. we could not: Upgrade a cluster Use TKG Packages Have Mixed clusters with both linux and windows worker nodes In TKG 2.1, we now have alpha support for cluster upgrades when using windows operating systems, we have the ability to deploy clusters with both windows and linux nodes, allowing for many use cases that require this such as applications where some microservices are linux based and some are windows based. We also do to the fact that we an now have mixed clusters, install TKG packages as they will deploy the needed pods on the linux nodes. Current limitations While the improvements of windows support is great to see, it still is lagging behind quite a lot from linux based nodes. We currently don’t have: A supported monitoring or logging solution for windows nodes (for linux we have Prometheus and Fluentbit packaged as supported addons) Support for Calico as a CNI (only Antrea is supported) Support OOTB for Cluster Autoscaler is not yet available Custom CA trust is not yet possible without writing a custom Cluster Class Static IP support for nodes is not yet possible (requires DHCP) Support for vSphere CSI (its possible today but not supported) While as you can see, there are limitations, seeing the effort that is being made in this area is great, and I am hopeful we will see more advances in this area over the next few releases. More Powerful TMC integration TMC is a great service from VMware, and it has for a long time supported managing the lifecycle of TKG clusters end to end. The challenge that has always existed with this though is that it almost never was an option in reality. As cluster templating was client side, if you had any YTT overlays in your setup, in order to for example trust your internal CA, this could not be applied when creating a cluster via TMC. There was also no support for Cluster Autoscaler, or many other configurations one could make on a cluster via the TKG config file. The configurations exposed were pre-defined by the TMC team, and were a very small subset of what was actually needed in many cases. With the move to Cluster Class in TKG 2.1, we now have support in TMC for Cluster Class, which now allows us to select which class to use (this could be the default one or a custom one you have created), and then also exposes to us the variables available in that cluster class, for us to set in the UI. As the default Cluster Class for example now supports trusting a CA certificate via a simple variable, we can now deploy clusters in environments that require this directly from TMC without needing to use the Tanzu CLI to do it manually. Current limitations While TMC does now support deploying Cluster Class based clusters, the Terraform provider for TMC, does not yet support this functionality. Also there are still some rough edges around the UX of defining Cluster variables, however, the improvements made in this integration are truly awesome to see, and I’m sure we will see it only get better over time. More Secure TKGm was initially released in April 2020, and at that point in time, while not officially announced yet, it was clear that Pod Security Policies (PSPs )were going to be removed from Kubernetes, however it was not clear what the replacement would be at the time. As this was already planned to be removed, and other solutions did exist for example OPA policy based PSPs or Kyverno Policies etc., VMware did not implement PSPs in TKGm. Since then, over 2.5 years have passed, PSPs are officially removed from Kubernetes, and the official replacement is in place called Pod Security Admission (PSA). PSA is a much easier solution to manage then PSPs were, and is the new standard. As such, VMware now in TKG 2.1, have added PSA configuration to the cluster config options. It is important to note, that this is currently in Technical Preview, meaning there is no official support for this feature, however it does not break your support for anything else if you decide to implement it. Current limitations Many of the supported packages in TKG such as prometheus, Fluent Bit, Pinniped, Contour, and Tanzu Auth currently do not comply with the baseline policy of PSA. While this is currently the reality, it does not mean that you can’t deploy such packages, or that any actual functionality is broken, rather it simply means that we will get warnings of some workloads being non compliant in the default config. We can also change the enforcement and warn levels of PSA on the relevant namespaces in order to suppress these issues, and to allow for a less noisy implementation in terms of warnings and audit logs. While PSA is not as flexible or as powerful as PSPs, and in some cases, may simply not be enough, having this as an option is a great thing for many customers. If it doesnt offer you what you need for your environment, OPA policies via TMC are a good solution with full support from VMware for getting the same level of control that we had in PSPs, simply via a different mechanism. Networking Improvements One of the most complex yet critical elements to get right in a Kubernetes environment is the networking. TKG has always offered a good set of tooling and solutions for this space, but much has been left wanted in this area, and TKG 2.1 gets us a step closer in the right direction. Node Networking Cluster API which is the backing technology for cluster lifecycle management in TKG, has at its core, the concept of immutable infrastructure, and as such VMs it deploys are ephemeral in nature. While in theory this is great, this also has some challenges especially when dealing with On-Premises environments, especially in the world of networking. In all 1.x versions of TKGm, node IP address allocation had to be done via DHCP. while DHCP is a viable solution in many cases, it has some serious drawbacks, especially for control plane nodes. DHCP has caused many issues for TKG users over the past 2.5 years. Imagine a scenario where you have a maintenance window in your Datacenter where you power down your vSphere environment which has TKG clusters deployed. When you power back up your environment, depending on how long DHCP leases are configured to last in your environment, when the last lease was going to expire, and other environmental configurations, your nodes may not receive the same IP address as before, which is completely expected behavior of DHCP (the D stands for Dynamic). The issue with this is that it can completely break your clusters. Kubernetes networking relies heavily on certificates for inter component communication, however as these certificates are generated for the IPs of the nodes, if the IP of a control plane node changes, you start having serious issues, as the certificates are not valid anymore. With worker nodes, IP address changes are also problematic, however at least in the case of worker nodes, the CAPI Machine Health Check (MHC) mechanism will kick in and roll out new nodes to replace the old ones automatically solving this issue. Because control plane nodes contain the Kubernetes backing database (ETCD), we can’t treat them as purely ephemeral machines, because they can’t simply be replaced without proper transfer of data, and synchronization of the data to the new node, otherwise we would have data loss. The solution till now has been to create DHCP reservations for control plane nodes, however this is a manual and tedious process which relies on humans doing grunt work to make sure things don’t break. This does not only effect TKG, and has been a longstanding issue many have encountered in CAPI based clusters on vSphere or other on prem platforms. Recently this has been addressed in CAPI by adding a new type of provider called an IPAM provider, which is a generic high level abstraction allowing implementations of Node IPAM controllers to be created that will manage IP Address allocation for CAPI machines. In TKG 2.1, this new mechanism has been implemented and the IPAM controller used is a simple yet powerful in-cluster IPAM controller. In this solution, the controller has a set of CRDs: InClusterIPPool – A definition of a pool of IPs that will be used for allocating IPs to nodes IPAddressClaim – A CR similar to the idea of a PVC. When a machine is created, an IPAddressClaim is created which then gets an ip address allocated back to it from the ipam controller IPAddress- A CR that represents an IP address that has been claimed by an IPAddressClaim for a machine. The newly added support for Static IPs via the IPAM integration is a huge feature, that will simplify deployments, and prevent many unfortunate issues that occur due to DHCP, and will save hours upon hours of manual work repairing clusters when they reach a bad state due to IP changes. Currently IPAM is only available for workload clusters and is not yet possible for the management cluster itself, but I’m hoping we will see this change in the near future. In any event this is a huge move in the right direction. Advanced Antrea Configurations Antrea which is the default CNI in TKG, has a lot of really awesome features, and till now, only a subset of these have been exposed in TKG. We now have access to a much wider set of configurations for Antrea, making it a really great choice for TKG clusters that have interesting and challenging network configuration needs. Some of the new features we can configure now in TKG 2.1 for Antrea include Antrea Multi Cluster, Pod Secondary IPs, Traffic Control, and more. KubeVIP Service Type Load Balancer TKG comes with support for NSX ALB (AVI) as a Load Balancing solution, which while complex to setup, offers a very rich set of capabilities, especially if you have the AVI Enterprise licensing. While it is strongly recommended to use NSX ALB when possible, in environments with limited hardware, sometimes a lighter weight solution is needed. For these cases VMware already support KubeVIP as the control plane VIP provider for TKG since the early days of TKG, however now in TKG 2.1, while currently in tech preview we now have the ability to use KubeVIP to provide Service Type LoadBalncer support as well. Networking Limitations in TKG 2.1 While as mentioned above, we have some great new networking capabilities in TKG 2.1, we also have some limitations that may or may not effect you depending on your specific use case, but that must be called out: You cannot use "CNI: none" like you could in previous releases. this means that currently there is no support for a "Bring Your Own CNI" like we have in previous 1.x versions of TKG NSX ALB currently does not support vSphere 8 IPv6 clusters are not supported on vSphere 8 environments Windows Clusters only support Antrea Static IP support for Node IP Addresses is only supported for workload clusters (Management clusters still require DHCP) Even with these limitations, It is clear that the direction is good for networking in TKG, and I am confident we will see more improvements in this area in the next few releases. Certificate Management Improvements A key challenge with Kubernetes, especially in enterprise environments, is that Kubernetes moves really fast. Currently there are 3 releases of Kubernetes a year, and a Kubernetes version is only supported for a year as well. This means that upgrades of Kubernetes are inevitable and must be taken into consideration. While in the optimal world, companies would perform upgrades of their clusters on a frequent basis, this simply is not the reality in many environments today. As mentioned previously, Kubernetes heavily relies on Certificates for internal communication, and these certificates are created with 1 year expirations. Previously the solution was to remember before the year is up, and to trigger a node rollout which will generate new nodes, and as such new certificates. Now in TKG 2.1, we have the ability to configure Control Plane Node Certificate Auto-Renewal. With this feature we can set how long before a certificate is expired the certificate should be rotated (by default it is set to 60 days). Certificate expiration issues is something I have personally encountered in many environments, and having this mechanism in place, is a huge benefit. While I don’t believe it is a good idea to have clusters not be upgraded for over a year, making sure the clusters don’t simply break after a year is a safeguard I am truly happy that we now have in TKG. Clear Feature Support Definitions TKG is a very complex and fully featured product, with many features at different levels of maturity. As the Kubernetes ecosystem moves so quickly, the balance that is needed in a product like TKG between stability and support for cutting edge features is a complex balance to reach. To reach such a balance, a similar approach like we have in Kubernetes itself with API maturity levels of Alpha, Beta and GA, is needed here as well, with each level having different support guarantees. While TKG since the 1.4 days has had experimental features, and alpha level features included, there was never clear messaging on what features had what level of support, and this was extremely difficult for end users. In TKG 2.1, VMware have now defined 4 levels of features in TKG each with its respective support policy and guarantees: Experimental – Not supported and voids support for the entire environment when used. Technical Preview – Not supported but does not void warranty and support for the environment as a whole. Stable – Tested and Supported for production usage. Deprecated – Tested and supported, but with end of support expected. While this technically is not a feature, it is a huge improvement over what we previously had, as it gives clear messaging on what can be used in what type of environment, and it level sets expectations in regards to support of different features. Summary TKG 2.1 is a huge milestone for the product, and it shows a serious change in the right direction in terms of understanding customer needs, and moving to a more sustainable and upstream aligned architecture. TKG 2.1 definitely has some serious growing pains right now, and the decision to upgrade to it must be taken with great precaution and consideration, however the future of TKG is definitely a bright one, and TKG 2.1 is a huge step in that direction.