Skip to main content
Version: Next

Migrating to Gateway API

warning

Gateway API support in metal-stack is still in development. Changes are expected, especially around certificate management.

From version x.y.z metal-stack supports Gateway API as a replacement for Kubernetes Ingress resources.

This guide explains why we are moving to Gateway API, what changes for you as an operator, and how to migrate an existing metal-stack installation.

Motivation​

ingress-nginx has been deprecated and we currently depend on it. metal-stack has control-plane components that are not served via HTTP/GRPC. Those are exposed as TCP Services via ingress-nginx. As a result we are going to move all metal-stack components to Gateway API.

What will change​

Certificate provisioning and TLS termination​

Certificate provisioning and TLS termination will move into the Gateway resource. metal-stack deployed via metal-roles will expect TLS termination to happen on the Gateway.

Exceptions for TLS termination:

  • NSQ will for now still terminate its own certificates
  • The GRPC endpoint of metal-api will stay a TCPRoute and TLS termination will stay in the application. As metal-api will be superseded by metal-apiserver in the near future, we do not see the value in migrating it fully

Continued need for an Ingress Controller​

For some vendor dependencies you will still require an ingress controller. Examples of services still relying on Ingress are Gardener and Thanos. As ingress-nginx is EoL we recommend switching to a different Ingress Controller implementation.

Before you begin​

Familiarize yourself with Gateway API​

Gateway API is more complex than Ingress and consists of multiple different resources and operator personas. The rest of the guide assumes a basic understanding of the HTTPRoute, TCPRoute, Gateway, GatewayClass. An introduction can be found here

Gateway API implementation requirements​

metal-stack does not require one specific Gateway API implementation. You can choose any Gateway API implementation that provides the following features:

  • HTTPRoute
  • TCPRoute

In our mini-lab demo environment we are running Envoy Gateway.

Migration Path decisions​

  • Select a Gateway API implementation, e.g. Envoy Gateway
  • Certificate management: Gateway API now requires the Gateway to know

Migration preparation​

Gateway deployment​

As we require TCPRoutes you will have to make changes to your existing Gateways, if applicable. Still, we suggest you create a dedicated metal-stack gateway, like we do for mini-lab.

We recommend provisioning a metal-stack Gateway resource as metal-stack requires multiple TCP endpoints.

Example​

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: metal-control-plane
  namespace: metal-control-plane
spec:
  gatewayClassName: eg
  listeners:
  - hostname: '*.<metal-stack FQDN>'
    name: http
    port: 80
    protocol: HTTP
  - hostname: '*.<metal-stack FQDN>'
    name: https
    port: 443
    protocol: HTTPS
    tls:
      certificateRefs:
      - group: ""
        kind: Secret
        name: metal-api-tls
      mode: Terminate
  - name: nsq
    port: 4150
    protocol: TCP
  - name: metal-api-grpc
    port: 50051
    protocol: TCP

Certificate management​

The biggest change during the Gateway API migration is certificate management. Using Ingress Controllers the Ingress resource configured if, and using what certificates TLS connections were terminated. This changed in Gateway API, where the Gateway listener you attach the Route on decides if TLS is used and which certificate will be used for termination.

metal-stack.io does not require any specific certificate management process. It is possible to use self-signed/private CA certificates as well as certificates public CAs.

tip

Use automated certificate management. cert-manager can handle your certificate renewal and supports Gateway API. You can use Let's Encrypt for valid public certificates or use a private CA.

You can issue valid public TLS certs using Let's Encrypt against an IP address if no DNS record is available.

TLS termination locations​

Exposed components with TLS terminated at the Gateway:

  • metal-api (excl. GRPC endpoint)
  • metal-apiserver
  • metal-console
  • zitadel
  • headscale

Exposed components terminating TLS in the application Pod:

  • metal-api GRPC endpoint
  • nsq

Migrating Components​

::: warning The old Ingress Controller and new Gateway are going to have different IP addresses. :::

Deploy the HTTPRoutes and TCPRoutes for each component using metal-role. *routes and Ingress resources can be deployed at the same time. To switch over a service to the Gateway, change the DNS record of that service from pointing to the Ingress Controller to the Gateway.

Example for zitadel, nsq and metal_apiserver. For full documentation of all services please consult the metal-roles repository.


zitadel_httproute_enabled: true
zitadel_httproute_parent_refs:
- name: metal-control-plane
  sectionName: https

nsq_tcproute_enabled: true
nsq_tcproute_parent_refs:
- name: metal-control-plane
  sectionName: nsq

metal_apiserver_httproute_enabled: true
metal_apiserver_httproute_parent_refs:
- name: metal-control-plane
  sectionName: http
- name: metal-control-plane
  sectionName: https

Remaining ingress controller dependencies​

For some metal-stack dependencies you will still require an ingress controller:

  • Gardener
  • Thanos

Verification​

Check your endpoints are still reachable on their expected host names/IP addresses. The easiest way to do that is to check with metalctl(metal-api)/metalctlv2(metal-apiserver).

It is also recommended to check if the DNS records really resolve the new Gateway and are not accessed via the old ingress infrastructure. Use nslookup/dig to check your hostname and make sure they resolve to the same IP address as the Gateway

Rollback​

You can roll back to ingress any time if an unresolvable blocker is discovered during the migration. Revert the deployment variables and rerun the playbook.

References​