[DevOps] Operating Multiple k8s GPU Clusters with On-Premises in startup (Part1)

I have many experiences in operating and hosting service with multiple K8s clusters in on-premise (baremetal machines)
I’m going to explain each stack in detail

Main Architecture

Domain Resolving and Setup?

• My company bought domain from “whois.co.kr”
• I setup nameservers as ‘Google Cloud DNS’ at whois.co.kr
• Google Cloud DNS sends traffic to our on-prem edge
• I always use LetsEncrypt to provide https domain, my certificates are automatically updated in my k8s cluster (by using certificate & gcloud credential)

Edge box?

• A small edge box provides port-forwarding by using nginx

• When the primary edge box dies, traffic fails over to the second

• There is nginx forward settings like below

  map $host $ip {
    ~[^\.]prod.kade.com 192.xx.xx.xx;
    ~[^\.]dev.kade.com 192.xx.xx.xx;
  }
  
  server {
    listen <nodeport-range>;
    location / {
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header Host $host;
      proxy_pass http://$ip:$server_port$request_uri;
    }
  }
  

Https (TLS/SSL)

• Nignx forward traffics to each k8s istio-ingressgateway
• After traffic arrived at “istio-ingressgateway”, it handles TLS/SSL network

Required Skills in above architecture

• Network : Domain nameserver, DNS, proxy, ingress, port-forwarding, vpn, DHCP, gateway, static IP… etc
• Network Devices : Switch, Router,