With these two features it immediately became clear that there is a good pairing here. I can have ArgoCD OIDC use VCFA as the provider making it so that users can seamlessly login into ArgoCD and as an administrator I can use common roles/groups for access. The rest of this post walks through setting up this integration.

Architecture

Implementation

Setup OIDC for your Org

The first step in this process is making sure you have OIDC setup for your tenant Org. VCFA has the ability to do per tenant(Org) identity provider configuration. You can configure it with OIDC, SAML, or AD, once this is configured then logging into the tenant goes through your provider of choice and we have access to all of the claims, groups etc. that you setup in your upstream IDP. Ultimately we will use these groups in ArgoCD as well. I am not going to go into detail on how to set up the tenant OIDC in this post, but below is a screenshot of my configuration for Okta integration and here are the official docs to configure it.

Setup the OIDC Service

This is the "relying party" I mentioned in the intro. It is basically a way to create OIDC clients that use VCFA as the provider. So in this case because Okta is my backing IDP for VCFA that means the relying party I create will also be able to use Okta for auth, but with far less configuration. Since it all goes through VCFA it will be treated as a single sign on as well so once I log into VCFA then I can seamlessly login to ArgoCD.

1. Login to the the VCFA provider portal as an administrator and go to OIDC services-> Relying Parties

1. Create a new relying party using the DNS name or IP address of the ArgoCD instance that will be deployed. Once you hit save it will generate a client secret, be sure to save that.

   

Deploy ArgoCD

In this step, ArgoCD will be deployed as a service and using the new OIDC client it will be integrated into VCFA authentication.

1. The below yaml can be used to deploy ArgoCD and also integrate OIDC with VCFA. Update the fields that have ##UPDATE THIS next to them. This should use details from the previous steps.

apiVersion: argocd-service.vsphere.vmware.com/v1alpha1
kind: ArgoCD
metadata:
  name: argocd-dev
  namespace: infra-ty3qk ##UPDATE THIS
spec:
  applicationSet:
    enabled: true
  enableLoadBalancer: true
  oidc:
    clientID: 4060b628-c297-49d3-ae0f-31cdcfb9ce86 ##UPDATE THIS
    clientSecret: mmSKfx8UZN6oYa31hGu95N+t+y1rbEih ##UPDATE THIS
    enabled: true
    insecure: true
    issuer: https://vcf-a.vcf.lab/oidc ##UPDATE THIS
    name: vcfa
    requestedIDTokenClaims:
      groups:
        essential: true
      preferred_username:
        essential: true
  rbac:
    policy: |
      g, "Organization Administrator", role:admin 
    policyMatchMode: glob
    scopes: '[groups,roles]'
  serverSideDiff: true
  url: https://argocd-dev.vcf.lab ##UPDATE THIS
  version: 3.0.19+vmware.1-vks.1

1. Apply the yaml into a supervisor namespace. You should see the ArgoCD pods come up and become healthy.

2. Add DNS. In this example I used argocd-dev.vcf.lab as my DNS auth callback in the relying party config, so ArgoCD needs to be available at that address. You can get the IP address for the ArgoCD Server by doing a kubectl get svc -n <supervisor-ns> and getting the external IP for the server.

There are a couple of things to note from the above YAML:

• issuer: https://vcf-a.vcf.lab/oidc - this is your VCFA instance

• g, "Organization Administrator", role:admin - this is what maps the role from VCFA to the ArgoCD admin role. You can make as many policy rules as you would like and also utilize groups from your upstream IDP. For example I also have a group called argocd-admin in my Okta. I could also add the policy g, "argocd-admin", role:admin

• scopes: '[groups,roles]' - this tells ArgoCD to get both the groups and the roles from the token and use them for policy mapping.

• url: https://argocd-dev.vcf.lab - this is a required setting, make sure this matches the callback url without the auth/callback

Validation

Now that the integration is complete we can test the login and make sure that groups and roles are propagating correctly.

1. Go to your ArgoCD server, in my case https://argocd-dev.vcf.lab . You will now see a "login with OIDC" button. If you are already logged into VCFA it will log you in as this user so be sure to logout if you want to use a different user.

2. When it redirects to VCFA choose your org that you have setup the OIDC on

3. Click "Login with OIDC" and it will take you through the normal process and redirect you back to ArgoCD.

4. Validate your groups and username in ArgoCD. Go to User Info on the left side panel. This is what I see when logging in:

   

Building the Image

Before deploying a cluster, we need to build a windows image. There is a process for this using the VKS image builder , this is what we will use in the next few steps to build the image.

Pre-requisites

1. Download the Windows ISO for windows server 2022. You can download the evaluation if needed.

2. Download the Windows VMware tools ISO

3. Upload the ISOs to a directory on a datastore in your vSphere environment that you will be running this build against. The below script can be used with GOVC to upload the ISOs.

#!/bin/bash


set -e # Exit immediately if a command exits with a non-zero status


print_usage() {
    echo "Usage: $0 <LOCAL_ISO_PATH> <DATASTORE_NAME> <REMOTE_FOLDER>"
    echo ""
    echo "Arguments:"
    echo "  LOCAL_ISO_PATH   Path to the ISO file on your local machine."
    echo "  DATASTORE_NAME   Name of the vSphere Datastore (e.g., vsanDatastore)."
    echo "  REMOTE_FOLDER    Folder path inside the datastore (e.g., ISOs/Linux)."
    echo ""
    echo "Example:"
    echo "  $0 ./ubuntu.iso vsanDatastore ISOs/Ubuntu"
}

check_govc() {
    if ! command -v govc &> /dev/null; then
        echo "Error: 'govc' is not installed or not in your PATH."
        echo "Please install it from: https://github.com/vmware/govmomi/tree/master/govc"
        exit 1
    fi
}


if [ "$#" -ne 3 ]; then
    print_usage
    exit 1
fi

LOCAL_ISO="$1"
DATASTORE="$2"
REMOTE_FOLDER="$3"
FILENAME=\((basename "\)LOCAL_ISO")
REMOTE_PATH="\(REMOTE_FOLDER/\)FILENAME"

check_govc

if [ ! -f "$LOCAL_ISO" ]; then
    echo "Error: Local file '$LOCAL_ISO' not found."
    exit 1
fi

if [ -z "$GOVC_URL" ]; then
    echo "Error: GOVC_URL environment variable is not set."
    echo "Please export GOVC_URL, GOVC_USERNAME, and GOVC_PASSWORD."
    exit 1
fi

echo "--- Starting Upload Process ---"
echo "File:      $FILENAME"
echo "Datastore: $DATASTORE"
echo "Folder:    $REMOTE_FOLDER"

if govc datastore.ls -ds="\(DATASTORE" "\)REMOTE_FOLDER" &> /dev/null; then
    echo "[OK] Remote folder '$REMOTE_FOLDER' exists."
else
    echo "[INFO] Remote folder '$REMOTE_FOLDER' does not exist. Creating it..."
    if govc datastore.mkdir -ds="\(DATASTORE" "\)REMOTE_FOLDER"; then
        echo "[OK] Folder created successfully."
    else
        echo "Error: Failed to create folder '\(REMOTE_FOLDER' on datastore '\)DATASTORE'."
        exit 1
    fi
fi

if govc datastore.ls -ds="\(DATASTORE" "\)REMOTE_PATH" &> /dev/null; then
    echo "Warning: File '\(REMOTE_PATH' already exists on datastore '\)DATASTORE'."
    read -p "Do you want to overwrite it? (y/N): " -n 1 -r
    echo
    if [[ ! \(REPLY =~ ^[Yy]\) ]]; then
        echo "[INFO] Upload skipped by user."
        exit 0
    fi
    echo "[INFO] Overwriting existing file..."
fi

echo "[INFO] Uploading '$LOCAL_ISO'... (This may take a while)"
if govc datastore.upload -ds="\(DATASTORE" "\)LOCAL_ISO" "$REMOTE_PATH"; then
    echo ""
    echo "Success: Upload complete!"
    echo "Location: [\(DATASTORE] \)REMOTE_PATH"
else
    echo ""
    echo "Error: Upload failed."
    exit 1
fi

Example usage:

./uploadiso.sh ./VMware-tools-windows-12.5.0-23800621.iso cls-wld9-vsan01 isos

Setting up the repo

1. Clone the vks-image-builder repo

git clone https://github.com/vmware/vks-image-builder.git
cd vks-image-builder

1. Create the windows answers file, the Upstream file can be found here. I have added one below with a few updates. The updates have been marked with comments to highlight them. You will need to also update the areas that have an “Updates this” marker, this is really just updating the Windows password. If you are using the eval version you need to remove the product key from the file.

<unattend xmlns="urn:schemas-microsoft-com:unattend" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State">
    <settings pass="windowsPE">
        <component name="Microsoft-Windows-PnpCustomizationsWinPE" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS">
            <DriverPaths>
                <PathAndCredentials wcm:action="add" wcm:keyValue="A">
                    <Path>a:\</Path>
                </PathAndCredentials>
            </DriverPaths>
        </component>
        <component name="Microsoft-Windows-Setup" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS">
            <DiskConfiguration>
                <Disk wcm:action="add">
                    <CreatePartitions>
                        <CreatePartition wcm:action="add">
                            <Order>1</Order>
                            <Type>EFI</Type>
                            <Size>100</Size>
                        </CreatePartition>
                        <CreatePartition wcm:action="add">
                            <Order>2</Order>
                            <Type>MSR</Type>
                            <Size>16</Size>
                        </CreatePartition>
                        <CreatePartition wcm:action="add">
                            <Order>3</Order>
                            <Type>Primary</Type>
                            <Extend>true</Extend>
                        </CreatePartition>
                    </CreatePartitions>
                    <ModifyPartitions>
                        <ModifyPartition wcm:action="add">
                            <Order>1</Order>
                            <Format>FAT32</Format>
                            <Label>System</Label>
                            <PartitionID>1</PartitionID>
                        </ModifyPartition>
                        <ModifyPartition wcm:action="add">
                            <Order>2</Order>
                            <PartitionID>2</PartitionID>
                        </ModifyPartition>
                        <ModifyPartition wcm:action="add">
                            <Order>3</Order>
                            <Format>NTFS</Format>
                            <Label>Windows</Label>
                            <Letter>C</Letter>
                            <PartitionID>3</PartitionID>
                        </ModifyPartition>
                    </ModifyPartitions>
                    <WillWipeDisk>true</WillWipeDisk>
                    <DiskID>0</DiskID>
                </Disk>
            </DiskConfiguration>
            <ImageInstall>
                <OSImage>
                    <InstallTo>
                        <DiskID>0</DiskID>
                        <PartitionID>3</PartitionID>
                    </InstallTo>
                    <InstallFrom>
                        <MetaData wcm:action="add">
                            <Key>/IMAGE/NAME</Key>
                            <Value>Windows Server 2022 SERVERSTANDARDCORE</Value>
                        </MetaData>
                    </InstallFrom>
                </OSImage>
            </ImageInstall>
            <UserData>
                <AcceptEula>true</AcceptEula>
                <FullName>Administrator</FullName>
                <Organization>Organization</Organization>
                <ProductKey>
                    <Key>VDYBN-27WPP-V4HQT-9VMD4-VMK7H</Key>
                    <WillShowUI>OnError</WillShowUI>
                </ProductKey>
            </UserData>
            <EnableFirewall>true</EnableFirewall>
        </component>
        <component name="Microsoft-Windows-International-Core-WinPE" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS">
            <SetupUILanguage>
                <UILanguage>en-US</UILanguage>
            </SetupUILanguage>
            <InputLocale>0409:00000409</InputLocale>
            <SystemLocale>en-US</SystemLocale>
            <UILanguage>en-US</UILanguage>
            <UILanguageFallback>en-US</UILanguageFallback>
            <UserLocale>en-US</UserLocale>
        </component>
    </settings>
    <settings pass="offlineServicing">
        <component name="Microsoft-Windows-LUA-Settings" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS">
            <EnableLUA>false</EnableLUA>
        </component>
    </settings>
    <settings pass="generalize">
        <component name="Microsoft-Windows-Security-SPP" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS">
            <SkipRearm>1</SkipRearm>
        </component>
    </settings>
    <settings pass="specialize">
        <component name="Microsoft-Windows-Deployment" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS">
            <RunSynchronous>
                <RunSynchronousCommand wcm:action="add">
                    <WillReboot>Always</WillReboot>
                    <Path>%SystemRoot%\System32\reg.exe ADD "HKLM\System\CurrentControlSet\Control\TimeZoneInformation" /v RealTimeIsUniversal /d 1 /t REG_DWORD /f</Path>
                    <Order>1</Order>
                </RunSynchronousCommand>
                <RunSynchronousCommand wcm:action="add">
                    <WillReboot>Always</WillReboot>
                    <Path>e:\setup.exe /s /v "/qb REBOOT=R ADDLOCAL=ALL"</Path>
                    <Order>2</Order>
                </RunSynchronousCommand>
            </RunSynchronous>
        </component>
        <component name="Microsoft-Windows-International-Core" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS">
            <InputLocale>0409:00000409</InputLocale>
            <SystemLocale>en-US</SystemLocale>
            <UILanguage>en-US</UILanguage>
            <UILanguageFallback>en-US</UILanguageFallback>
            <UserLocale>en-US</UserLocale>
        </component>
        <component name="Microsoft-Windows-Security-SPP-UX" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS">
            <SkipAutoActivation>true</SkipAutoActivation>
        </component>
        <component name="Microsoft-Windows-SQMApi" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS">
            <CEIPEnabled>0</CEIPEnabled>
        </component>
        <component name="Microsoft-Windows-Shell-Setup" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS">
            <ComputerName />
            <ProductKey>VDYBN-27WPP-V4HQT-9VMD4-VMK7H</ProductKey>
        </component>
    </settings>
    <settings pass="oobeSystem">
        <component name="Microsoft-Windows-Shell-Setup" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS">
            <AutoLogon>
                <Password>
                    <Value>VMware123!</Value> <!-- Update this -->
                    <PlainText>true</PlainText>
                </Password>
                <Enabled>true</Enabled>
                <Username>Administrator</Username>
            </AutoLogon>
            <FirstLogonCommands>
                <SynchronousCommand wcm:action="add">
                    <Order>1</Order>
                    <Description>Set Execution Policy 64 Bit</Description>
                    <CommandLine>cmd.exe /c powershell -Command "Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Force"</CommandLine>
                    <RequiresUserInput>true</RequiresUserInput>
                </SynchronousCommand>
                <SynchronousCommand wcm:action="add">
                    <Order>2</Order>
                    <Description>Set Execution Policy 32 Bit</Description>
                    <CommandLine>%SystemDrive%\Windows\SysWOW64\cmd.exe /c powershell -Command "Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Force"</CommandLine>
                    <RequiresUserInput>true</RequiresUserInput>
                </SynchronousCommand>
                <SynchronousCommand wcm:action="add">
                    <CommandLine>%SystemRoot%\System32\reg.exe ADD HKCU\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\Advanced\ /v HideFileExt /t REG_DWORD /d 0 /f</CommandLine>
                    <Order>3</Order>
                    <Description>Show file extensions in Explorer</Description>
                </SynchronousCommand>
                <SynchronousCommand wcm:action="add">
                    <CommandLine>%SystemRoot%\System32\reg.exe ADD HKCU\Console /v QuickEdit /t REG_DWORD /d 1 /f</CommandLine>
                    <Order>4</Order>
                    <Description>Enable QuickEdit mode</Description>
                </SynchronousCommand>
                <SynchronousCommand wcm:action="add">
                    <CommandLine>%SystemRoot%\System32\reg.exe ADD HKCU\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\Advanced\ /v Start_ShowRun /t REG_DWORD /d 1 /f</CommandLine>
                    <Order>5</Order>
                    <Description>Show Run command in Start Menu</Description>
                </SynchronousCommand>
                <SynchronousCommand wcm:action="add">
                    <CommandLine>%SystemRoot%\System32\reg.exe ADD HKCU\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\Advanced\ /v StartMenuAdminTools /t REG_DWORD /d 1 /f</CommandLine>
                    <Order>6</Order>
                    <Description>Show Administrative Tools in Start Menu</Description>
                </SynchronousCommand>
                <SynchronousCommand wcm:action="add">
                    <CommandLine>%SystemRoot%\System32\reg.exe ADD HKLM\SYSTEM\CurrentControlSet\Control\Power\ /v HibernateFileSizePercent /t REG_DWORD /d 0 /f</CommandLine>
                    <Order>7</Order>
                    <Description>Zero Hibernation File</Description>
                </SynchronousCommand>
                <SynchronousCommand wcm:action="add">
                    <CommandLine>%SystemRoot%\System32\reg.exe ADD HKLM\SYSTEM\CurrentControlSet\Control\Power\ /v HibernateEnabled /t REG_DWORD /d 0 /f</CommandLine>
                    <Order>8</Order>
                    <Description>Disable Hibernation Mode</Description>
                </SynchronousCommand>
                <SynchronousCommand wcm:action="add">
                    <CommandLine>cmd.exe /c wmic useraccount where "name='Administrator'" set PasswordExpires=FALSE</CommandLine>
                    <Order>9</Order>
                    <Description>Disable password expiration for Administrator user</Description>
                </SynchronousCommand>
                <SynchronousCommand wcm:action="add">
                    <CommandLine>cmd.exe /c %SystemDrive%\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -File a:\enable-winrm.ps1</CommandLine>
                    <Description>Enable WinRM</Description>
                    <Order>10</Order>
                </SynchronousCommand>
                <SynchronousCommand wcm:action="add">
                    <CommandLine>cmd.exe /c a:\disable-network-discovery.cmd</CommandLine>
                    <Description>Disable Network Discovery</Description>
                    <Order>11</Order>
                </SynchronousCommand>
                <SynchronousCommand wcm:action="add"> <!-- Customization for public network issue -->
                    <Order>12</Order>
                    <Description>Configure additional policy for public network</Description>
                    <CommandLine>cmd.exe /c powershell -NoProfile -ExecutionPolicy Bypass -Command "Set-NetFirewallRule -Name 'WINRM-HTTP-In-TCP-PUBLIC' -RemoteAddress Any; Set-Item -Path 'WSMan:\localhost\Service\Auth\Basic' -Value \(true; Set-Item -Path 'WSMan:\localhost\Service\AllowUnencrypted' -Value \)true"</CommandLine>
                    </SynchronousCommand>
            </FirstLogonCommands>
            <OOBE>
                <HideEULAPage>true</HideEULAPage>
                <HideLocalAccountScreen>true</HideLocalAccountScreen>
                <HideOEMRegistrationScreen>true</HideOEMRegistrationScreen>
                <HideOnlineAccountScreens>true</HideOnlineAccountScreens>
                <HideWirelessSetupInOOBE>true</HideWirelessSetupInOOBE>
                <NetworkLocation>Work</NetworkLocation>
                <ProtectYourPC>1</ProtectYourPC>
                <SkipMachineOOBE>true</SkipMachineOOBE>
                <SkipUserOOBE>true</SkipUserOOBE>
            </OOBE>
            <RegisteredOrganization>Organization</RegisteredOrganization>
            <RegisteredOwner>Owner</RegisteredOwner>
            <DisableAutoDaylightTimeSet>false</DisableAutoDaylightTimeSet>
            <TimeZone>Pacific Standard Time</TimeZone>
            <UserAccounts>
                <AdministratorPassword>
                    <Value>VMware123!</Value> <!-- Update this -->
                    <PlainText>true</PlainText>
                </AdministratorPassword>
           <LocalAccounts>
            <LocalAccount wcm:action="add">
                <Description>Administrator</Description>
                <DisplayName>Administrator</DisplayName>
                <Group>Administrators</Group>
                <Name>Administrator</Name>
            </LocalAccount>
            <LocalAccount wcm:action="add">
                <Password>
                    <Value>VMware123!</Value> <!-- Update this -->
                    <PlainText>true</PlainText>
                </Password>
                <Description>For log collection</Description>
                <DisplayName>Admin Account</DisplayName>
                <Name>WindowsAdmin</Name>
                <Group>Administrators</Group>
            </LocalAccount>
        </LocalAccounts>
            </UserAccounts>
        </component>
    </settings>
</unattend>

1. Update the vsphere packer variables in packer-variables/vsphere.j2 here is an example file, the fields that must be updates are marked with a comment {# Update this #}.

{
    {# vCenter server IP or FQDN #}
    "vcenter_server":"vcsa9-wld.vcf.lab",    {# Update this #}
    {# vCenter username #}
    "username":"administrator@vcf9-wld.local", {# Update this #}
    {# vCenter user password #}
    "password":"VMware123!VMware123!", {# Update this #}
    {# Datacenter name where packer creates the VM for customization #}
    "datacenter":"wld9-DC", {# Update this #}
    {# Datastore name for the VM #}
    "datastore":"cls-wld9-vsan01", {# Update this #}
    {# [Optional] Folder name #}
    "folder":"",
    {# Cluster name where packer creates the VM for customization #}
    "cluster": "wls-wld9", {# Update this #}
    {# Packer VM network #}
    "network": "/wld9-DC/network/Virtual Private Clouds/image-builder-3/vm-public/vm-public", {# Update this #}
    {# To use insecure connection with vCenter  #}
    "insecure_connection": "true",
    {# TO create a clone of the Packer VM after customization#}
    "linked_clone": "true",
    {# To create a snapshot of the Packer VM after customization #}
    "create_snapshot": "true",
    {# To destroy Packer VM after Image Build is completed #}
    "destroy": "true"
}

1. Update the windows specific packer vars. There are two files packer-variables/windows/default-args-windows.j2 and packer-variables/windows/vsphere-windows.j2 . Below are sample files with comments on where to update.

vsphere-windows.j2 - update the paths with the output from the iso upload script

{
    {# [Optional] Windows only: Windows OS Image #}
    "os_iso_path": "[cls-wld9-vsan01] isos/en-us_windows_server_2022_x64_dvd_620d7eac.iso", {# Update this #}
    {# [Optional] Windows only: VMware Tools Image #}
    "vmtools_iso_path": "[cls-wld9-vsan01] isos/vmtools-windows.iso" {# Update this #}
}

default-args-windows.j2 - all that needs to be added here is the windows_admin_password

{
  "additional_executables_destination_path": "C:\\ProgramData\\Temp",
  "additional_executables_list": "http://{{ host_ip }}:{{ artifacts_container_port }}/artifacts/{{ kubernetes_version }}/bin/windows/amd64/registry.exe,http://{{ host_ip }}:{{ artifacts_container_port }}/artifacts/{{ kubernetes_version }}/bin/windows/amd64/goss.exe",
  "additional_executables": "true",
  "additional_url_images": "false",
  "additional_url_images_list": "",
  "additional_prepull_images": "",
  "build_version": "{{ os_type }}-kube-{{ kubernetes_series }}-{{ ova_ts_suffix }}",
  "cloudbase_init_url": "http://{{ host_ip }}:{{ artifacts_container_port }}/artifacts/{{ kubernetes_version }}/bin/windows/amd64/CloudbaseInitSetup_x64.msi",
  "cloudbase_real_time_clock_utc": "true",
  "containerd_url": "http://{{ host_ip }}:{{ artifacts_container_port }}/artifacts/{{ kubernetes_version }}/bin/windows/amd64/cri-containerd.tar",
  "containerd_sha256_windows": "{{ containerd_sha256_windows_amd64 }}",
  "containerd_version": "{{ containerd }}",
  "convert_to_template": "true",
  "create_snapshot": "false",
  "disable_hypervisor": "false",
  "disk_size": "40960",
  "kubernetes_base_url": "http://{{ host_ip }}:{{ artifacts_container_port }}/artifacts/{{ kubernetes_version }}/bin/windows/amd64",
  "kubernetes_series": "{{ kubernetes_series }}",
  "kubernetes_semver": "{{ kubernetes_version }}",
  "kubernetes_typed_version": "{{ image_version }}",
  "load_additional_components": "true",
  "netbios_host_name_compatibility": "false",
  "nssm_url": "http://{{ host_ip }}:{{ artifacts_container_port }}/artifacts/{{ kubernetes_version }}/bin/windows/amd64/nssm.exe",
  "prepull": "false",
  "pause_image": "localhost:5000/vmware.io/pause:{{ pause }}",
  "runtime": "containerd",
  "template": "",
  "unattend_timezone": "Pacific Standard Time",
  "windows_updates_categories": "",
  "windows_updates_kbs": "",
  "wins_url": "",
  "custom_role": "true",
  "custom_role_names": "/image-builder/images/capi/image/ansible-windows",
  "ansible_user_vars": "ansible_winrm_read_timeout_sec=600 ansible_winrm_operation_timeout_sec=590 artifacts_container_url=http://{{ host_ip }}:{{ artifacts_container_port }} imageVersion={{ image_version|replace('-', '.') }} registry_store_archive_url=http://{{ host_ip }}:{{ artifacts_container_port }}/artifacts/{{ kubernetes_version }}/registries/{{ registry_store_path }}",
  "vmx_version": "21",
  "debug_tools": "false",
  "enable_auto_kubelet_service_restart": "false",
  "windows_admin_password": "VMware123!" {# Update this #}
}

Running the Build

Now that all of the specific settings are in place we can run the build. This will output an OVA that can then be uploaded to a content lib and used in a VKS cluster.

1. Start the artifacts container and run the build. The TKR_SUFFIX should match the suffix of your current linux VKR versions, this used when resolving the correct ova from the content lib. The HOST_IP should be your workstation IP that this command is running from. The IMAGE_ARTIFACTS_PATH is where you want the OVA to be created. The AUTO_UNATTEND_ANSWER_FILE_PATH is the path to your answers file.

 ##start the artifacts container

make run-artifacts-container ARTIFACTS_CONTAINER_PORT=8081

##run this from the vks-image-builder directory
make build-node-image OS_TARGET=windows-2022-efi TKR_SUFFIX=vkr.4 HOST_IP=10.0.0.180 IMAGE_ARTIFACTS_PATH=/home/will/windows-build-2/image ARTIFACTS_CONTAINER_PORT=8081 PACKER_HTTP_PORT=8082 AUTO_UNATTEND_ANSWER_FILE_PATH=/home/will/windows-build-2/vks-image-builder/windows_autounattend.xml

1. Create a content lib and upload the OVA.

## update the datastore with your datastore as well as the path to the ova
govc library.create -ds "cls-wld9-vsan01" "windows-vkrs"
govc library.import windows-vkrs  ./image/ovas/windows-2022-amd64-v1.34.1---vmware.1-vkr.4.ova

Setting up the content library

Before deploying a cluster, we need to associate our new content library with the supervisor. We can do this by going to the supervisor settings and under general adding another content library.

Verify that the image is showing up in the supervisor. Run this from the supervisor cluster context.

k get osimages -A | grep windows
vmi-4aacfc6e370e28a1e   v1.34.1+vmware.1           windows   2022         amd64   cvmi                13s

k get cclitem -A | grep windows
clitem-4aacfc6e370e28a1e   windows-2022-amd64-v1.34.1---vmware.1-vkr.4                                 cl-c80585866b93825a5       OVF    True    true     20615149892   true                100s

Take note of the content library ID for future use in cluster builds.

Deploy a cluster

Now that we have verified the image is available we can deploy a cluster. There are a number of ways to do this but for the purpose of this post I am just going to share the yaml for the cluster and you can deploy it how you would like. The main thing to note below is the Windows node poo along with the annotation that tells it which content library to use.

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: windows
  namespace: dev-ts6hw
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
        - 192.168.156.0/20
    services:
      cidrBlocks:
        - 10.96.0.0/12
    serviceDomain: cluster.local
  topology:
    class: builtin-generic-v3.5.0
    classNamespace: vmware-system-vks-public
    version: v1.34.1---vmware.1-vkr.4
    variables:
      - name: kubernetes
        value:
          certificateRotation:
            enabled: true
            renewalDaysBeforeExpiry: 90
      - name: vmClass
        value: best-effort-small
      - name: storageClass
        value: vsan-default-storage-policy
    controlPlane:
      replicas: 1
      metadata:
        annotations:
          run.tanzu.vmware.com/resolve-os-image: os-name=photon, content-library=cl-a3b5e06b12f2737ca
    workers:
      machineDeployments:
        - class: node-pool
          name: windows-np-j72b
          replicas: 1
          variables:
            overrides:
              - name: vmClass
                value: best-effort-xsmall
        - class: node-pool
          name: windows-nodepool-l7rs
          replicas: 1
          metadata:
            annotations:
              run.tanzu.vmware.com/resolve-os-image: os-type=windows, content-library=cl-c80585866b93825a5
          variables:
            overrides:
              - name: vmClass
                value: best-effort-large

Customizing images with Ansible

Sometimes it may be necessary to modify the image with custom scripts or binaries. This should be used with caution and also should only be used for things that cannot be done through a native K8s operator during runtime. Also this should not be used for anything that would require embedding passwords into the OVA. With those caveats out of the way let’s add some Ansible to modify the image build.

In this example we will add a simple Ansible task that runs a powershell script. This script does not perform any meaningful actions in this example but provides the details on how to set this up so that you can use a similar process for anything you need to install.

1. Add a new file to the vks-image-builder/ansible-windows/tasks folder. This ansible-windows folder is the Ansible role that executes by default during the build process.

touch vks-image-builder/ansible-windows/tasks/exec-pwsh.yml

1. Add the contents of the Ansible task to the new file .

- name: Execute custom BYOI script
  ansible.builtin.script: scripts/helloworld.ps1

1. Add the powershell script to the files in the Ansible role.

touch vks-image-builder/ansible-windows/files/scripts/helloworld.ps1

1. Add the contents of the script to the file.

Write-Output "Hello, World!"

1. Update the main.yml to include your new tasks

- import_tasks: exec-pwsh.yml

Updating cluster nodes for patching

Since these are images that you maintain, you may need to update an image with a one off patch etc. This may also be the case for images that don’t have a new k8s version to go along with it. In the case of patching a with new K8s version it’s pretty simple, you build a new image with the new K8s version and then update the K8s version on your cluster definition. But what about the case where there are no changes upstream from Broadcom, that’s what we will cover here. In this case we want to patch or change something on the image and want to roll that out to our clusters without changing anything else.

To do this, we need to understand.a bit about how an OVA in a content library is resolved by VKS.

1. The VKR version selected in the cluster definition. This looks something like v1.34.1---vmware.1-vkr.4, we can see that this is version 1.34.1 of K8s, also that it’s a patch release of vmware.1-vkr.4 this suffix is used during the image resolution process.

2. The OS , this is set on the node pools usually Photon, Ubuntu, or Windows.

3. The OS version, for Ubuntu we might see 22.04, Windows we would see 2022

4. The content library setting

When these are combined VKS looks at the available OS images that match the right VKR version and OS settings and then pulls the right ova from the content library to create the node.

Now the challenge we run into is that if we just update our patch suffix to something like vkr.5 and then when we build a cluster the Windows nodes would be resolved but if our Linux nodes don’t also have a vkr.5 patch release then it won’t find the right images. We also need to make sure our content library item is following the standard naming convention so that it is picked up bu VKS, the name geenrated by the image builder creates this in the proper format(windows-2022-amd64-v1.34.1---vmware.1-vkr.4). Since there is already an OVA in the content library named that we can’t upload the new one unless we want to overwrite it. Overwriting the image is a perfectly accetable way to roll out a new patch however I would prefer to have some more control over the process. So we need another way to manage this. This is where the content library setting comes in, we can create a new content library for our patch and upload the new OVA then selectively roll out the new node image.

1. Create a new content library for the patch and upload the new image.

govc library.create -ds "cls-wld9-vsan01" "windows-patch-12626"
govc library.import windows-patch-12626  ./image/ovas/windows-2022-amd64-v1.34.1---vmware.1-vkr.4.ova

1. Update supervisor config to add the new content library

1. Update your cluster yaml to use the new content library ID.

k get cclitem -A | grep -i windows

 run.tanzu.vmware.com/resolve-os-image: os-type=windows, content-library=cl-1c5b6ba4a0e41aa16

One challenge with this from an administrator point of view is getting insight into all of the volumes and what they are being used for and where. vCenter does have a native UI to explore these and filter them and govc can also be used to query them with the CLI, however many times I want to see these volumes from a namespace and PVC based view. It can also be challenging to associate the volumes with the PVC when looking through the raw details from the govc output. That’s why I created this simple python script to help with getting the details for volumes based on a Supervisor namespace.

The script used both kubectl output and govc output and correlates the PVCs to the underlying CNS volumes, it then returns relevant details about the volumes as well as sorts them between node volumes and volumes that are being used in the clusters for workload PVCs.

Example Usage

let’s say I have a supervisor namespace called gitops-932tv and in that namespace I have a VKS cluster deployed. Also in that cluster I have an app deployed that uses a number of persistent disks. As an administrator I may need to determine where these volumes are located on my datastores and also get some details like there volume IDs etc. in case I need to relocate them. To get this info I can simple run the following.

1. Setup the kube context and govc details

    # sets the kube context
    vcf context use <supervisor-conext>
    export GOVC_INSECURE=true
    export GOVC_PASSWORD='password'
    export GOVC_USERNAME=administrator@vsphere.local
    export GOVC_URL=https://vcsa.myorg.com
   

2. clone the repo and run the script

    git clone https://github.com/warroyo/cns-tooling
    cd cns-tooling/pvc-audit
    python3 vks_disk_audit.py gitops-932tv
   

With those two steps the output will look similar to this. You will notice that the we get two separate sections, one for node volumes and one for in cluster volumes. The node volumes section shows which node the volumes are attached to as well as details about the size, cluster, datastore etc. For the in cluster volumes, these are typically PVCs used by workloads in the cluster, these show similar details but also include the referred entity metadata about the PVC name in the cluster and which pod is using it.

--- Starting Audit for Namespace: gitops-932tv ---
Mapping VSphereMachines to Clusters...
Mapped 2 nodes across clusters.
Querying all PVCs in namespace...
Found 7 PVC(s). Resolving PV handles...
Querying vSphere CNS for 7 volumes...


=======================================================================================
                             NODE VOLUMES (Attached)
=======================================================================================
PVC Name                       Node                           Cluster              Volume Name                         Volume ID                                Datastore            Capacity   Referred Entity
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
tfd-1-8q26s-ql69w-vol-b9xl     tfd-1-8q26s-ql69w              tfd-1                pvc-9e7d6921-d573-4203-ad8a-9e420f490446 560fd41e-f243-4c96-997e-8bf7b7996e95     vsan:8740804e6737443c-b388c53757aaaf93/ 20.00 GB   -
tfd-1-tfd-1-jrxdt-pmhmn-cv4hj-vol-b9xl tfd-1-tfd-1-jrxdt-pmhmn-cv4hj  tfd-1                pvc-2417b407-2bed-4cdb-88fb-7b1ebf6653ad ecfffa9d-483c-4fe1-a391-e4fe1986e52d     vsan:8740804e6737443c-b388c53757aaaf93/ 20.00 GB   -


=======================================================================================
                             IN-CLUSTER PVCs
=======================================================================================
PVC Name                       Cluster              Volume Name                         Volume ID                                Datastore            Capacity   Referred Entity
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
f478b761-3832-4c56-8cc7-10b44e5a968b-1fb60ee1-66e8-4ce6-97a6-c7cb12070bdf tfd-1                pvc-6b57785d-828b-4f1d-ab32-7c617e60a908 2b2deeab-53cf-4551-bba4-e787dc1567d8     vsan:8740804e6737443c-b388c53757aaaf93/ 1.00 GB    PVC:music-store/order-pvc, Pod:order-service-69987cbbfd-mlj2c
f478b761-3832-4c56-8cc7-10b44e5a968b-232680b1-d8d7-4eaa-9e50-66a12bb0fb0a tfd-1                pvc-788a9fa5-6db8-4151-ad5b-3f32c5dcda16 b2c6ac20-b002-4f0a-948b-4fccef7d8c3a     vsan:8740804e6737443c-b388c53757aaaf93/ 1.00 GB    PVC:music-store/cart-pvc, Pod:cart-service-7cc8794c86-x2m6v
f478b761-3832-4c56-8cc7-10b44e5a968b-3519cba5-07ba-45a5-b329-e647a3500c34 tfd-1                pvc-b7ec835b-ef13-4b56-9fae-4f6b2c6395ce 5fd5d74c-2b77-4ecb-ae95-c4a22ae99111     vsan:8740804e6737443c-b388c53757aaaf93/ 1.00 GB    Pod:postgres-bcf8997c4-89pkj, PVC:music-store/postgres-pvc
f478b761-3832-4c56-8cc7-10b44e5a968b-7f7b7933-d67a-479c-8197-47c61708f1c6 tfd-1                pvc-92f87c6b-cfdc-4377-8a79-11d6dd6b3b1a 6fe37d12-9c30-4ab5-aaf7-7f9276e3ba49     vsan:8740804e6737443c-b388c53757aaaf93/ 1.00 GB    PVC:music-store/music-store-1-pvc
f478b761-3832-4c56-8cc7-10b44e5a968b-d1d19249-7cb0-4c7d-914e-ec866e998aaa tfd-1                pvc-06cbb085-57c3-4b27-856a-713b45e0c53b 0f28021c-c85c-4f36-b9de-faa26fedb232     vsan:8740804e6737443c-b388c53757aaaf93/ 1.00 GB    PVC:music-store/users-pvc, Pod:users-service-855678b958-9zcdb

--- Audit Complete ---

I have had a question come up a few times with customers and coworkers about how to reduce duplication when creating clusters with Tanzu Mission Control(TMC). The question or issue that is usually brought up is that the platform engineering team wants to be able to create clusters quickly and many of the settings between cluster creation are the exact same, thus having a lot of duplication between clusters. When looking at the TMC UI there's not a way to set custom defaults today to be able to remove the need to fill in every field each time you create a cluster. However, using the UI is probably not the approach a platform team wants to take to scale anyway. It's much more efficient to codify the clusters and automate the creation. In this post, we will walk through creating cluster templates and using the Tanzu CLI to create clusters with minimal inputs. We will focus on TKG Clusters mostly, but I will also provide some commands that work with AKS and EKS clusters are well.

Brief note on the Tanzu CLI

The Tanzu CLI can now be installed through standard package managers or directly via the binary from GitHub, see the install instructions here. The TMC standalone CLI is in the process of being deprecated so you will want to use the new plugins for the Tanzu CLI. They will be installed when you connect the Tanzu CLI to your TMC endpoint.

Templating clusters

Getting the base template

The CLI provides a way to get an existing cluster's spec, I always recommend going with this approach over trying to create one from scratch. Before we start templating the cluster, create a cluster through the UI. Then use the below command to pull the cluster's yaml spec back and save it in a file.

# for TKG 
tanzu tmc cluster get <cluster-name> -m <mgmt-cluster> -p <provisioner> > template.yml
## for EKS
tanzu tmc ekscluster get <cluster-name> -c <credential> -r <region> > template.yml
## for AKS
tanzu tmc akscluster get <cluster-name> -c <credential> -r <resource-group> -s <subscription> > template.yml

The contents of the above command will look slightly different depending on the k8s provider you are using, but all of them will be a yaml file that describes all of the settings for the cluster.

Remove any extra fields

Just like when pulling back a resource from a k8s cluster there will be fields that we will want to omit from our template. For example the status section. This will be different between providers, but because we are focused on TKG in this example I have listed the field that I omitted below. Technically many of these fields do not need to be removed since the API will just ignore them, but since we are creating a template and don't want a bunch of extra stuff that might be confusing we will remove anything that is not needed.

• fullname.orgId

• meta

  • labels

    • tmc.cloud.vmware.com/creator

  • annotations

  • creationTime

  • generation

  • resourceVersion

  • uid

  • updateTime

• spec.topology.variables

  • extensionCert

  • user - if you would like to provide your ssh public key, keep this one.

  • clusterEncryptionConfigYaml

  • TKR_DATA

• status - entire section

All of the fields above are generated by TMC when the cluster is created. Your YAML file should now look similar to the one below.

fullName:
  managementClusterName: h2o-4-19340
  name: tmc-base-template
  provisionerName: lab
meta:
  labels:
    example-label: example
spec:
  clusterGroupName: default
  tmcManaged: true
  topology:
    clusterClass: tanzukubernetescluster
    controlPlane:
      metadata:
        annotations:
          example-cp-annotation: example
        labels:
          example-cp-label: example
      osImage:
        arch: amd64
        name: ubuntu
        version: "20.04"
      replicas: 3
    network:
      pods:
        cidrBlocks:
        - 172.20.0.0/16
      serviceDomain: cluster.local
      services:
        cidrBlocks:
        - 10.96.0.0/16
    nodePools:
    - info:
        name: md-0
      spec:
        class: node-pool
        metadata:
          labels:
            exmaple-np-label: example
        osImage:
          arch: amd64
          name: ubuntu
          version: "20.04"
        overrides:
        - name: vmClass
          value: best-effort-large
        - name: storageClass
          value: vc01cl01-t0compute
        replicas: 2
    variables:
    - name: defaultStorageClass
      value: vc01cl01-t0compute
    - name: storageClass
      value: vc01cl01-t0compute
    - name: storageClasses
      value:
      - vc01cl01-t0compute
    - name: vmClass
      value: best-effort-large
    - name: ntp
      value: time1.oc.vmware.com
    version: v1.23.8+vmware.2-tkg.2-zshippable
type:
  kind: TanzuKubernetesCluster
  package: vmware.tanzu.manage.v1alpha1.managementcluster.provisioner.tanzukubernetescluster
  version: v1alpha1

Templating with YTT

There are many options for templating files, but since this is a YAML file and we like Carvel YTT that is what is used for this example. I would highly recommend reading up on YTT and testing it out for different use cases, it's a very powerful YAML templating language.

Determine the variable fields

first, we need to determine which fields should be variable. This could be any field, but we want to reuse as much as possible as well. These fields are entirely up to you and your needs.

Template the fields with YTT

the YTT docs explain how to use data values in a YAML file. This is what we will be using to template the file. Using the same file from above the below template is what I have come up with.

#@ load("@ytt:data", "data")
fullName:
  managementClusterName: #@ data.values.mgmt_cluster_name
  name: #@ data.values.cluster_name
  provisionerName: #@ data.values.provisioner
meta:
  #@ if/end hasattr( data.values, "cluster_labels"):
  labels: #@ data.values.cluster_labels
spec:
  clusterGroupName: #@ data.values.cluster_group
  tmcManaged: true
  topology:
    clusterClass: tanzukubernetescluster
    controlPlane:
      metadata:
        #@ if/end hasattr( data.values, "cp_annotations"):
        annotations: #@ data.values.cp_annotations
        #@ if/end hasattr( data.values, "cp_labels"):
        labels: #@ data.values.cluster_labels
      osImage:
        arch: amd64
        name: ubuntu
        version: "20.04"
      replicas: 3
    network:
      pods:
        cidrBlocks:
        - 172.20.0.0/16
      serviceDomain: cluster.local
      services:
        cidrBlocks:
        - 10.96.0.0/16
    nodePools:
    - info:
        name: md-0
      spec:
        class: node-pool
        metadata:
          #@ if/end hasattr( data.values, "node_labels"):
          labels: #@ data.values.node_labels
        osImage:
          arch: amd64
          name: ubuntu
          version: "20.04"
        overrides:
        - name: vmClass
          value: #@ data.values.cp_vm_size
        - name: storageClass
          value: vc01cl01-t0compute
        replicas: 2
    variables:
    - name: defaultStorageClass
      value: vc01cl01-t0compute
    - name: storageClass
      value: vc01cl01-t0compute
    - name: storageClasses
      value:
      - vc01cl01-t0compute
    - name: vmClass
      value: #@ data.values.worker_vm_size
    - name: ntp
      value: time1.oc.vmware.com
    version: v1.23.8+vmware.2-tkg.2-zshippable
type:
  kind: TanzuKubernetesCluster
  package: vmware.tanzu.manage.v1alpha1.managementcluster.provisioner.tanzukubernetescluster
  version: v1alpha1

You can see that a number of fields now have YTT logic in them. Here is a quick breakdown of what we are doing.

• #@ load("@ytt:data", "data") - tell ytt to load data values into the data object

• #@ data.values.mgmt_cluster_name - I won't go through every variable but this syntax is used to pull out a value from the values file that we will create in the next section.

• #@ if/end hasattr( data.values, "cp_annotations"): - this syntax is also used a few times, this check to see if our values file has a field and if it does then it adds the field below. This is used becuase certain fields are optional.

There is a lot more that can be done when templating with YTT and this is a fairly basic example. The docs on YTT have a lot of example that can be referenced.

Create a values file

The values file is what we will use to specify the values for all of the fields that we have templated. This is really just a yaml file with a single line of YTT at the top that let's the YTT engine know that the fields are to be used as data values. Since this is just plain yaml, you can also have nested fields etc. Below you can see the values file that was created to work with the above template.

#@data/values
---
mgmt_cluster_name: h2o-4-19340
cluster_name: cluster-from-template
provisioner: lab
cp_vm_size: best-effort-large
worker_vm_size: best-effort-large
cluster_group: default
cluster_labels:
  test: test

Create a new cluster

Finally, we can combine these two files into a command that with generate our cluster configuration and then apply it to TMC.

If you want to test the outputs of the templating prior to sending it to TMC you can simply run the below command which will generate the resulting yaml and output to stdout.

ytt -f values.yml -f template.yml

The next command will generate the resulting yaml and instead of sending it stdout , it will pass it directly to the Tanzu CLI and start creating the cluster.

tanzu tmc apply -f  <(ytt -f values.yml -f template.yml)

If you are using EKS or AKS the command is slightly different since support is not yet added to the apply command for those clusters. Hopefully it will be added soon. You can still do this with the create and update commands though. See the examples below.

# redirecting output does not work currently for the create commands

#EKS
ytt -f values.yml -f template.yml > eks.yml
tanzu tmc ekscluster create -f eks.yml

#AKS
ytt -f values.yml -f template.yml > aks.yml
tanzu tmc akscluster create -f aks.yml

Summary

I summary, this article should give you a good idea of how to make re-usable templates for TMC created clusters. This could even be used to create "plans" for clusters so that it makes self service easier for teams. An example of using this for slef service would be to have a pipeline that executes the apply commands and allow developers, operators, etc. to manage their values file in a git repo. This would provide a nice gitops driven way to create on demand clusters with the ability to apply policy and restrict which fields could be changed.

The overall solution depends on what the full networking stack is that you are using with TKG, but a common challenge is doing this when you are using TKG with supervisor(TKGs) that is backed by NSX-T. The issue is that when using this architecture, all traffic that comes out of a supervisor namespace is routed through a namespace-specific T1 and has a SNAT rule that maps it to a single IP address. This means that all workloads in a supervisor namespace appear as the same IP to the external network.

In this article, we will walk through a solution to the problem mentioned above so that we can associate different external IPs with specific workloads running in TKG clusters.

The solution

To solve this problem we will use a combination of Antrea and NSX-T. Ultimately we need to be able to make sure that when a container in a pod makes a request, eventually it is associated with a specific IP on the physical network. We also want to potentially have pods in the same cluster that have different external-facing IPs on the network. To do this we can use Antrea egress policies and custom NSX-T SNAT rules.

Antrea egress policy: This allows us to specify a static IP or pool of IPs that traffic will be SNAT'd to when leaving the node if it matches a specific set of labels. You can read all the details here.

NSX-T SNAT rules: This allows us to specify a rule that matches an IP or range of IPs and translates it to an IP on the physical network as it exits the T0.

We can now combine the two features above to solve the problem. The way this works is that the Antrea egress policy will handle making sure that the traffic leaving the node for a specific pod(s) will be a specific IP address, rather than the node's IP address. We can then use the NSX-T SNAT rule to ensure that the IP we specified in the Antrea egress rule is translated to the right external facing IP address. This means that we now have a way to label a pod and have it result in a specific IP address on the physical network.

This diagram shows a simplified flow of the IP translations.

Implementation

This will assume that the following prereqs are met before implementing the solution.

• TKG w/supervisor deployed

• TKG w/supervisor backed my NSX-T

• A TKG workload cluster deployed

• Antrea as a CNI

Create the egress policy

The first step is to create the Antrea egress policy and IP pool. This will be where the label matching is set so that we can target specific pods or namespaces in the cluster.

create the following YAML file and apply it to your workload cluster.

apiVersion: crd.antrea.io/v1alpha2
kind: Egress
metadata:
  name: egress-external-db
spec:
  appliedTo:
    podSelector:
      matchLabels:
        external-db-access: 'true'
  externalIPPool: external-ip-pool
---
apiVersion: crd.antrea.io/v1alpha2
kind: ExternalIPPool
metadata:
  name: external-ip-pool
spec:
  ipRanges:
  - start: 10.244.0.70  # IP from segment that nodes are on
    end: 10.244.0.70
  nodeSelector: {}     # All Nodes can be Egress Nodes

After applying the file you should see a status like this:

 k get egresses.crd.antrea.io egress-external-db
NAME                 EGRESSIP      AGE   NODE
egress-external-db   10.244.0.70   18s   tes-snat-1-md-0-bzfm4-7678dc8766-w7rz2

In the above file, you can see that we are creating two resources. the Egress resource defines the match criteria and the IP pool we will use. The ExternalIPPool defines the IP range that we want to use. In this case, it's set to just 1 IP. This IP is pulled from the cluster's segment CIDR in NSX-T, so it is an IP that would typically be used for a node. This IP could be from another range but for simplicity, we are using the segment's CIDR.

Create a workload with matching labels

We now need to create a workload that matches our egress policy label selectors. This will start up a pod that matches the labels, in this example, we are using netshoot which is an open-source tool for network troubleshooting.

Apply the following YAML into the workload cluster.

apiVersion: apps/v1
kind: Deployment
metadata:
    name: netshoot
    labels:
        app: netshoot
spec:
    replicas: 1
    selector:
        matchLabels:
            app: netshoot
    template:
        metadata:
          labels:
            app: netshoot
            external-db-access: 'true'
        spec:
            containers:
            - name: netshoot
              image: nicolaka/netshoot
              command: ["/bin/bash"]
              args: ["-c", "while true; do ping localhost; sleep 60;done"]

At this point, the traffic leaving this pod will be translated to our Antrea egress IP 10.244.0.70 . However we are not finished just yet, the IP at this point will still be translated to the default NSX-T egress IP when leaving the T0 because our default SNAT rule that is created by TKG is capturing all traffic on that segment's subnet.

Add the custom NSX-T SNAT rule

This is where we will create a rule in NSX-T to translate the Antrea egress IP to the final outbound IP address. This can be done either through the UI or automation with the API, etc. For simplicity, this shows how to create the rule in the UI.

1. go into the NSX-T console and navigate to the Networking->NAT section.

2. find the T1 that is associated with your supervisor namespace in the dropdown. the T1 name will contain the ns name.

3. create a new SNAT rule, see the image below for details. This will set the Antrea egress IP as the source and the destination is the outbound IP we want from the NSX-T egress range.

   

   We can now test to see if the IP is being translated properly.

Validate it

To validate this we will exec into the container and ping a Linux box running on a different network while watching traffic with tcpdump. This will make the traffic flow the full course of the network and we should see the outbound translated IP.

Run the ping command to the Linux box on 10.220.13.144

k exec -it netshoot-75dd7f67c9-zck8f -- bash
netshoot-75dd7f67c9-zck8f: ping
netshoot-75dd7f67c9-zck8f: ping 10.220.13.144 
PING 10.220.13.144 (10.220.13.144) 56(84) bytes of data.

Check the tcpdump on the Linux box. Notice the IP is 10.214.185.200 which is our custom SNAT rule IP.

tcpdump -i eth0 icmp
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
17:16:23.009148 IP 10.214.185.200 > photon-machine: ICMP echo request, id 2823, seq 1, length 64

Summary

In summary, using this approach will allow you to be able to conform to existing firewall rules and policies that rely on specifying an IP address for specific workloads. This could be used for many different use cases that require this type of granularity. additionally it lets us use a labeling and code-based mechanism to assign these IPs to workloads and can be very dynamic if needed.

How it works

First, let's break down what is meant by a "custom workload type". In TAP there is a resource called a "workload" which defines some minimal details about an application and abstracts away many of the underlying complexities of the infrastructure and the path that the app will take to get to a running state on that infrastructure i.e. the path to prod. The "workload" definition makes it easy for a developer to supply some basic app configuration and source code and get the app running on K8s with little to no knowledge of the underlying K8s environment. In the workload definition, there is a label that is used to set the type of workload that will be created. Based on that label's value, an instance of the supply chain is created for the workload and will generate a set of K8s resources that are determined by the type. For example the server type will generate a K8s Deployment and Service , while if the web type is specified it will generate a Knative Service. By creating a custom workload type we can define which K8s resources get generated when that type is specified.

TAP provides a way to include new workload types via a setting in the values.yaml called ootb_supply_chain_basic.supported_workloads, more on this in the implementation section below. What's great about this is that this makes it easy to add a custom workload type without having to modify any supply chains or do any real "customization" that strays from the OOTB offerings. In addition to the TAP values file modification, a new ClusterConfigurationTemplate needs to be added. The ClusterConfigurationTemplate is where the K8s resources that will be generated for the type are defined.

Here is a high-level outline of the end-to-end process:

1. A workload is defined with the custom workload type as the value for the label.

2. The workload is applied to the build cluster which instantiates an instance of the supply chain.

3. When the supply chain instance is created there is a selection that happens on the app-config step that looks at the workload type label and chooses the correct ClusterConfigurationTemplate , which in this case would be the custom one.

4. The supply chain eventually reaches the app-config step and the custom template stamps out a ConfigMap with the custom-defined K8s resources that will be eventually deployed on the TAP run clusters.

Diagram of this flow:

The use case

As mentioned in the introduction we will be creating a new workload type for a use case that isn't currently covered by the out-of-the-box types. For this use case, we have an app that needs to mount a volume. Specifically, this app needs to mount a K8s persistent volume to store some data.

Implementation

Create a new Cluster Configuration Template

The ClusterConfigurationTemplate resource is used to define the resulting delivery YAML. This YAML will consist of the Kubernetes manifests that are responsible for running the application.

Apply the below YAML into the TAP build cluster.

apiVersion: carto.run/v1alpha1
kind: ClusterConfigTemplate
metadata:
  annotations:
    doc: |
      This template consumes an input named config which contains a
      PodTemplateSpec and returns a ConfigMap which contains a
      "delivery.yml" which contains a manifests for a Kubernetes
      Deployment which will run the templated pod, and a "service.yml"
      Kubernetes Service to expose the pods on the network. This also supports 
      a workload param that allows for adding volumes
  name: volumes-template
spec:
  configPath: .data
  healthRule:
    alwaysHealthy: {}
  params:
  - default:
    - containerPort: 8080
      name: http
      port: 8080
    name: ports
  ytt: |
    #@ load("@ytt:data", "data")
    #@ load("@ytt:yaml", "yaml")
    #@ load("@ytt:struct","struct")
    #@ load("@ytt:assert", "assert")
    #@ load("@ytt:overlay", "overlay")

    #@ def merge_labels(fixed_values):
    #@   labels = {}
    #@   if hasattr(data.values.workload.metadata, "labels"):
    #@    labels.update(data.values.workload.metadata.labels)
    #@   end
    #@   labels.update(fixed_values)
    #@  return labels
    #@ end

    #@ def intOrString(v):
    #@   return v if type(v) == "int" else int(v.strip()) if v.strip().isdigit() else v
    #@ end

    #@ def merge_ports(ports_spec,containers):
    #@   ports = {}
    #@   for c in containers:
    #@     for p in getattr(c,"ports", []):
    #@       ports[p.containerPort] = {"targetPort": p.containerPort,"port": p.containerPort, "name": getattr(p, "name", str(p.containerPort))}
    #@     end
    #@   end
    #@   for p in ports_spec:
    #@     targetPort = getattr(p,"containerPort", p.port)
    #@     type(targetPort) in ("string", "int") or fail("containerPort must be a string or int")
    #@     targetPort = intOrString(targetPort)
    #@    
    #@     port = p.port
    #@     type(port) in ("string", "int") or fail("port must be a string or int")
    #@     port = int(port)
    #@     ports[p.port] = {"targetPort": targetPort, "port": port, "name": getattr(p, "name", str(p.port))}
    #@   end
    #@  return ports.values()
    #@ end

    #@ def addVolumes():
    spec:
      containers:
      #@overlay/match by="name"
      - name: workload
        #@overlay/match missing_ok=True
        volumeMounts: #@ data.values.params.volumes.volumeMounts
      #@overlay/match missing_ok=True
      volumes: #@ data.values.params.volumes.volumes

    #@ end

    #@ def delivery():
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: #@ data.values.workload.metadata.name
      annotations:
      kapp.k14s.io/update-strategy: "fallback-on-replace"
      ootb.apps.tanzu.vmware.com/servicebinding-workload: "true"
      labels: #@ merge_labels({ "app.kubernetes.io/component": "run","carto.run/workload-name": data.values.workload.metadata.name })
    spec:
      selector:
        matchLabels: #@ data.values.config.metadata.labels
      template: #@ overlay.apply(data.values.config,addVolumes())
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: #@ data.values.workload.metadata.name
      labels: #@ merge_labels({ "app.kubernetes.io/component": "run", "carto.run/workload-name": data.values.workload.metadata.name })
    spec:
      selector: #@ data.values.config.metadata.labels
      ports:
      #@ hasattr(data.values.params, "ports") and len(data.values.params.ports) or assert.fail("one or more ports param must be provided.")
      #@ declared_ports = {}
      #@ if "ports" in data.values.params:
      #@   declared_ports = data.values.params.ports
      #@ else:
      #@   declared_ports = struct.encode([{ "containerPort": 8080, "port": 8080, "name": "http"}])
      #@ end
      #@ for p in merge_ports(declared_ports, data.values.config.spec.containers):
        - #@ p
      #@ end
    #@ end

    ---
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: #@ data.values.workload.metadata.name + "-server"
      labels: #@ merge_labels({ "app.kubernetes.io/component": "config"})
    data:
      delivery.yml: #@ yaml.encode(delivery())

Let's break this template down a bit.

1. The resource takes as input a PodTemplateSpec from the previous step in the supply chain. This is accessible through the data.values.config .

2. The resource defines a ytt template that will be used to generate a ConfigMap that holds the delivery.yaml . The delivery.yaml contains the Kubernetes manifests to run the app.

3. The delivery() function - Defines a templated K8s deployment and service using input from the workload params and the previous step's PodTemplateSpec .

4. The addVolumes() function - this is used within the delivery function to modify the incoming PodTemplateSpec , it will take the volume params provided in the workload spec and add them to the PodTemplateSpec . This is the part that is doing most of the custom work to enable volumes.

5. The merge_ports() function - this takes in any custom ports defined in the params in the workload spec and adds them to the service.

6. The merge_labels() function - this does a similar task to the merge_ports but this time with any labels passed in.

Update the TAP values to include the new workload type

To register this new workload type a new section needs to be added to the supply chain's configuration.

Edit the TAP values and add the below YAML. Notice this is being added to the ootb_supply_chain_basic section so just append this along with any other settings. After updating these settings, update the tap install using the Tanzu cli.

ootb_supply_chain_basic:
  supported_workloads:
  - type: web
    cluster_config_template_name: config-template
  - type: server
     cluster_config_template_name: server-template
  - type: worker
    cluster_config_template_name: worker-template
  - type: server-with-volumes
    cluster_config_template_name: volumes-template

The settings above define four workload types, three of which are provided OOTB. The existing three need to be specified otherwise they will be removed. The fourth is the custom workload type that has been named server-with-volumes and is referencing the new ClusterConfigurationTemplate named volumes-template . These settings are what associate the workload type that will be used in the workload spec with the new template defined in the previous step.

Create a workload using the new type

Keeping with the thread of our new use case, for this workload we need to mount a persistent volume. Creating the volume is out of the scope of this post, but any K8s PVC will work. In this case, the PVC is just using the AWS EBS CSI driver to create a disk.

Apply the below YAML into the TAP build cluster

apiVersion: carto.run/v1alpha1
kind: Workload
metadata:
  labels:
    app.kubernetes.io/part-of: go-sample-pvc
    apps.tanzu.vmware.com/workload-type: server-with-volumes
  name: go-sample-pvc
  namespace: default
spec:
  params:
    - name: volumes
      value:
        volumes:
        - name: data-mount
          persistentVolumeClaim:
            claimName: go-sample-data
        volumeMounts:
         - name: data-mount
           mountPath: /sample-data
  source:
    git:
      ref:
        branch: main
      url: https://github.com/warroyo/tap-go-sample

The main things to point out in the above YAML are the following:

1. apps.tanzu.vmware.com/workload-type: server-with-volumes this label is what is used for selecting the workload type. Notice that the server-with-volumes name matches the one we added to the TAP values.

2. The volumes param - This is where the volumes and volume mounts are defined. To keep it simple and flexible the format of these parameters is just using the same spec from the K8s pod spec for defining volumes. This means anything that can be done through these docs, can be added here.

After deploying this workload the resulting K8s manifests will look like this. Notice that the volume and volume mounts have been added to the pod spec.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-sample-pvc
  annotations: null
  kapp.k14s.io/update-strategy: fallback-on-replace
  ootb.apps.tanzu.vmware.com/servicebinding-workload: "true"
  labels:
    app.kubernetes.io/part-of: go-sample-pvc
    apps.tanzu.vmware.com/has-tests: "true"
    apps.tanzu.vmware.com/workload-type: server-with-volumes
    app.kubernetes.io/component: run
    carto.run/workload-name: go-sample-pvc
spec:
  selector:
    matchLabels:
      app.kubernetes.io/component: run
      app.kubernetes.io/part-of: go-sample-pvc
      apps.tanzu.vmware.com/has-tests: "true"
      apps.tanzu.vmware.com/workload-type: server-with-volumes
      carto.run/workload-name: go-sample-pvc
  template:
    metadata:
      annotations:
        conventions.carto.run/applied-conventions: |-
          spring-boot-convention/auto-configure-actuators-check
          spring-boot-convention/app-live-view-appflavour-check
          appliveview-sample/app-live-view-appflavour-check
        developer.conventions/target-containers: workload
      labels:
        app.kubernetes.io/component: run
        app.kubernetes.io/part-of: go-sample-pvc
        apps.tanzu.vmware.com/has-tests: "true"
        apps.tanzu.vmware.com/workload-type: server-with-volumes
        carto.run/workload-name: go-sample-pvc
    spec:
      containers:
      - image: dev.registry.pivotal.io/warroyo/go-sample-pvc-default@sha256:94eec8cb112d4e860ab6cc9095f40db0779889f7ee0a953f0876d4b4b29ef0ce
        name: workload
        resources: {}
        securityContext:
          runAsUser: 1000
        volumeMounts:
        - mountPath: /sample-data
          name: data-mount
      serviceAccountName: default
      volumes:
      - name: data-mount
        persistentVolumeClaim:
          claimName: go-sample-data
---
apiVersion: v1
kind: Service
metadata:
  name: go-sample-pvc
  labels:
    app.kubernetes.io/part-of: go-sample-pvc
    apps.tanzu.vmware.com/has-tests: "true"
    apps.tanzu.vmware.com/workload-type: server-with-volumes
    app.kubernetes.io/component: run
    carto.run/workload-name: go-sample-pvc
spec:
  selector:
    app.kubernetes.io/component: run
    app.kubernetes.io/part-of: go-sample-pvc
    apps.tanzu.vmware.com/has-tests: "true"
    apps.tanzu.vmware.com/workload-type: server-with-volumes
    carto.run/workload-name: go-sample-pvc
  ports:
  - targetPort: 8080
    port: 8080
    name: http

Summary

In summary, the steps in this post walked through adding a new configuration template that takes parameters from the workload spec and uses those to inject volumes into the resulting pods. We then associated that new configuration template with a new workload type that can be used by a developer when creating a workload. This approach could be used for many other use cases and is not limited to adding volumes. Hopefully, this shows how TAP can be extended to support all kinds of different workloads and application needs while also providing a good user experience for the developer.

How it works

Since TAP already has an integration with Jenkins, we will follow a similar pattern for implementing the ADO integration. Below is a breakdown of how source testing works in TAP with Jenkins. The diagram shows that there is a parameter for testing_pipeline_matching_labels in the workload definition, this is what is used to select the pipeline, via label selectors, that should be used for testing. From there a Runnable is created and uses the label selectors to associate the Jenkins Tekton pipeline. There is also a ClusterTask that the pipeline references; this is where the code exists for communicating with Jenkins. From there, when the supply chain executes it will create a PipelineRun with the parameters from the workload that will spawn a TaskRun.The task run executes the Jenkins job and returns the results. There are a few pieces I left out of the diagram to make it easier to follow but, overall this covers most of what happens.

Based on the above flow, there will be two things needed to implement an ADO equivalent: a custom ClusterTask and a Pipeline. With those defined, we will be able to use native TAP functionality to selectively run tests in ADO. These two resources are core components of Tekton, you can find the docs on them here.

Implementation

Create a simple pipeline in ADO

Login to ADO and create a "new project" or use an existing project that you may have. If you are using the Azure CLI, run the below command.

az devops project create --name tap-ado-blog --org https://dev.azure.com/<your-organization>

After creating the new project there will also be a default repo created by the same name, e.g. tap-ado-blog. This is the repo that will be used for the pipeline in the next step. You can also create a new repo or use an existing one, just be sure to change the names in the next steps accordingly.

The below YAML will be used for the newly created pipeline. This sets up three parameters, two of which are required since the TAP supply chain passes the source_url and source_revision by default. The third is an optional parameter that shows how to pass additional parameters to the ADO pipeline in the workload YAML. The pipeline steps can be customized to handle any testing scenario needed.

trigger:
- none

pr: none 

pool:
  vmImage: ubuntu-latest

parameters:
  - name: source_url
    displayName: source url to clone
    type: string
  - name: source_revision
    displayName: revision to clone
    type: string
  - name: example_param
    displayName: example
    default: ""
    type: string

steps:
- script: echo ${{parameters.source_url}} "succesfully triggered this build from TAP"
- script: echo ${{parameters.source_revision}} "succesfully triggered this build from TAP"

Next, create a new pipeline...

From the UI: make the following selections Pipelines->Create pipeline->Azure Repos Git->tap-ado-blog->Starter Pipeline. Add the above YAML after selecting the "Starter Pipeline" and save it.

Using the CLI: commit the above YAML to the newly created repo as azure-pipelines.yml , run the below commands.

git clone https://<your-org>@dev.azure.com/<your-org>/tap-ado-blog/_git/tap-ado-blog 
cd tap-ado-blog
touch azure-pipelines.yml
#paste the contents from the above yaml into the new file
git add .
git commit -am "adding pipelines"
git push

az pipelines create --name 'tap-ado-blog' --description 'Pipeline for TAP' --repository tap-ado-blog  --branch main --repository-type tfsgit --org https://dev.azure.com/<your-organization> --project tap-ado-blog --yaml-path azure-pipelines.yml

Create a PAT in Azure DevOps

To execute the pipeline from TAP there needs to be a PAT created to auth against the API. If you are running TAP in Azure you could do something like role-based access control to provide auth but that is for another blog post.

From the UI: select Upper right settings->Personal Access Tokens->New Token->Full Access , see here for the full docs.

Using the CLI: as of 12/28/2022, there is no way to get a PAT from the CLI except for going direct to the REST API.

Save the generated PAT for later use.

Relocate the required image to your registry

In the docs for TAP it will typically have you relocate your images to a registry during the installation. The new ClusterTask that will be created for this ADO integration will require a new image due to its Python dependency. Run the following command after exporting the required variables to relocate the image to your registry.

imgpkg copy -i python:3.7-slim --to-repo ${INSTALL_REGISTRY_HOSTNAME}/${INSTALL_REPO}/tap-packages

After running the command you will see some output similar to

will export index.docker.io/library/python@sha256:aa949f5f10e9b28e1f9561fff73d1a359fa8517d4e543451a714d1a4ecc61c56

To get the full path to the copied image, copy everything starting with the @ symbol in your output and append it to the new repo path. The resulting full image path will look something like

${INSTALL_REGISTRY_HOSTNAME}/${INSTALL_REPO}/tap-packages@sha256:aa949f5f10e9b28e1f9561fff73d1a359fa8517d4e543451a714d1a4ecc61c56

# for example: https://dev.registry.pivotal.io/warroyo/tap-packages@sha256:aa949f5f10e9b28e1f9561fff73d1a359fa8517d4e543451a714d1a4ecc61c56

Depending on when you run this the SHA may be different, but the same process applies to get the full path to the image.

Create the TAP resources

NOTE: The next few steps will walk you through creating the required TAP resources. This should be done in the "build" cluster, or if you are using a "full profile", it will be in the single cluster since the build components are colocated. Ensure you are in the correct context when running these commands.

Create a Kubernetes secret to store the PAT from the previous section. This should be done in the "developer namespace".

kubectl -n <developer-ns> create secret generic ado-token --from-literal=pat=<your-pat-here>

Create a new ClusterTask, this will set up the code that will be executed to run the pipeline in Azure. In the Below YAML, replace <your-relocated-image-here> with the image from the above step. After modifying, apply this to the cluster with kubectl.

apiVersion: tekton.dev/v1beta1
kind: ClusterTask
metadata:
  name: ado-task
spec:
  params:
  - name: source-url
    type: string
  - name: source-revision
    type: string
  - name: secret-name
    type: string
  - name: pipeline-id
    type: string
  - name: project-name
    type: string
  - name: org-name
    type: string
  - default: ""
    name: pipeline-params
    type: string
  results:
  - name: ado-pipeline-run-url
    type: string
  steps:
  - name: install-depends
    image:  <your-relocated-image-here>
    script: |
      pip install requests
  - env:
    - name: ADO_API_TOKEN
      valueFrom:
        secretKeyRef:
          key: pat
          name: $(params.secret-name)
    - name: SOURCE_URL
      value: $(params.source-url)
    - name: PIPELINE_PARAMS
      value: $(params.pipeline-params)
    - name: SOURCE_REVISION
      value: $(params.source-revision)
    - name: PIPELINE_ID
      value: $(params.pipeline-id)
    - name: ORG_NAME
      value: $(params.org-name)
    - name: PROJECT_NAME
      value: $(params.project-name)
    image:  <your-relocated-image-here>
    name: trigger-ado-build
    script: |
      #!/usr/bin/env bash
      set -o errexit
      set -o pipefail
      pip install requests

      python3 << END
      import os
      import subprocess
      import logging
      import sys
      import time
      import json
      import requests


      logging.basicConfig(level=logging.DEBUG)

      org = os.getenv('ORG_NAME')
      project = os.getenv('PROJECT_NAME')
      pipeline = os.getenv('PIPELINE_ID')
      token = os.getenv('ADO_API_TOKEN')
      source_url = os.getenv('SOURCE_URL')
      source_revision = os.getenv('SOURCE_REVISION')
      pipeline_params = os.getenv('PIPELINE_PARAMS')

      url = f'https://dev.azure.com/{org}/{project}/_apis/pipelines/{pipeline}/runs?api-version=7.0'
      existing_params = {
          "source_url": f'{source_url}',
          "source_revision": f'{source_revision}'
      }

      input_params = {}
      if pipeline_params != "":
        input_params = json.loads(pipeline_params)

      existing_params.update(input_params)
      payload = json.dumps({
      "templateParameters": existing_params
      })

      headers = {
      'Content-Type': 'application/json'
      }

      pipelineResponse = requests.request("POST", url, headers=headers, data=payload,auth=('',token)) 
      logging.info(pipelineResponse.text)
      #throw error if not 200
      pipelineResponse.raise_for_status()

      #check status of pipeline run and validate it succeeds

      jsonResponse = pipelineResponse.json()

      currentRun = jsonResponse['_links']['self']['href']
      results_url = jsonResponse['_links']['web']['href']
      f = open("$(results.ado-pipeline-run-url.path)", "w")
      f.write(results_url)
      f.close()


      running = True
      while running:
        response = requests.get(currentRun, headers=headers, auth=('',token), timeout=300)
        response.raise_for_status()
        result = response.json()
        if result['state'] != 'completed':
          logging.info(f"pipeline state is {result['state']}, entering sleep for 5 seconds")
          time.sleep(5)
        elif result['result'] == 'succeeded':
          logging.info(f"pipeline was successful, exiting")
          sys.exit(os.EX_OK)
        else:
          logging.info(f"pipeline result is {result['result']}, check ADO")
          sys.exit(os.EX_SOFTWARE)
      END

As you will notice, the main portion of the above file is a script. I chose Python for this since it was much easier to read than the Bash equivalent. Taking a deeper look into what is happening in the script we see the following:

• parameters are being passed in from the supply chain

• a REST call is made to ADO to execute the pipeline

• a while loop is used to continuously execute a REST call to ADO until the status is succeeded or failed

Next create a new Pipeline that will reference the ClusterTask and have the required labels so that it can be selected by the supply chain. This should also be created in the "developer namespace".

---
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: developer-defined-ado-pipeline
  namespace: <developer-ns>
  labels:
    #! This label should be provided to the Workload so that
    #! the supply chain can find this pipeline
    apps.tanzu.vmware.com/pipeline: ado-pipeline
spec:
  results:
  - name: ado-pipeline-run-url   #! To show the job URL on the
    #! Tanzu Application Platform GUI
    value: $(tasks.ado-task.results.ado-pipeline-run-url)
  params:
  - name: source-url        #! Required
  - name: source-revision   #! Required
  - name: secret-name       #! Required
  - name: project-name       #! Required
  - name: pipeline-id       #! Required
  - name: org-name           #! Required
  - name: pipeline-params
    default: ""
  tasks:
  - name: ado-task
    taskRef:
      name: ado-task
      kind: ClusterTask
    params:
      - name: source-url
        value: $(params.source-url)
      - name: source-revision
        value: $(params.source-revision)
      - name: secret-name
        value: $(params.secret-name)
      - name: pipeline-id
        value: $(params.pipeline-id)
      - name: org-name
        value: $(params.org-name)
      - name: project-name
        value: $(params.project-name)
      - name: pipeline-params
        value: $(params.pipeline-params)

Finally, create or modify a workload to use the new labels and parameters that will trigger the ADO testing pipeline. The below workload can be used since the GitHub repo is public and can be cloned without credentials. Just replace the org-name and pipeline-id with the ones from your account. The pipeline-id can be found in the URL when looking at the pipeline in the browser.

---
apiVersion: carto.run/v1alpha1
kind: Workload
metadata:
  labels:
    app.kubernetes.io/part-of: company-api-ado
    apps.tanzu.vmware.com/has-tests: "true"
    apps.tanzu.vmware.com/workload-type: web
  name: company-api-ado
  namespace: default
spec:
  build:
    env:
      - name: BP_KEEP_FILES
        value: "docs/*"
  source:
    git:
      ref:
        branch: main
      url: https://github.com/warroyo/tap-go-sample
  params:
    - name: testing_pipeline_matching_labels
      value:
        apps.tanzu.vmware.com/pipeline: ado-pipeline
    - name: testing_pipeline_params
      value:
        project-name: tap-ado-blog
        secret-name: ado-token
        org-name: <your-org>
        pipeline-id: <pipeline-id> 
        pipeline-params: "{\"newparam\": \"testing\"}"

Once this workload is created the source-tester step should now execute the pipeline in ADO.

Final Result

The final result is illustrated in the diagram below. The workload creates a supply chain and as part of that the source-tester step is executed. The source-tester triggers the ADO pipeline defined in the workload and will either fail and stop the supply chain or succeeded and continue the supply chain.

Conclusion

In summary, the steps in this blog added an option to our TAP supply chain to use ADO pipelines for testing code. One really nice thing about this is that it didn't require any modifications to the supply chains or any core TAP components. Through the native label selection that TAP provides we were able to add another testing option in addition to the OOTB options. This also does not need to be limited to testing code in the ADO pipeline. In a future post, I will cover how the same approach could be used to add steps to the supply chain that will run some arbitrary steps in an ADO Pipeline. This should give an idea of the possibilities that exist for integrating TAP with other toolsets.

I am a Solution Engineer at VMware living in Denver and working on all things Cloud Native with a heavy focus on Kubernetes.

In the past, I have helped build large-scale infrastructure automation systems as well as helped modernize software deployments across enterprises.

I am also passionate about cooking and food in general so you may see some posts about food mixed into this blog.