[{"data":1,"prerenderedAt":1691},["ShallowReactive",2],{"page-blog-/blog/homelab-with-proxmox-and-k3s-a-real-ha-cluster-on-mini-pcs":3},{"id":4,"title":5,"body":6,"date_created":1679,"date_modified":1680,"description":1681,"extension":1682,"head":1683,"meta":1684,"navigation":1685,"ogImage":1683,"path":1686,"robots":1683,"schemaOrg":1683,"seo":1687,"sitemap":1688,"stem":1689,"__hash__":1690},"content/blog/homelab-with-proxmox-and-k3s-a-real-ha-cluster-on-mini-pcs.md","Homelab with Proxmox and k3s - A Real HA Cluster on Mini PCs",{"type":7,"value":8,"toc":1653},"minimark",[9,14,18,21,24,28,31,118,121,125,128,145,148,152,155,181,184,188,191,211,214,225,230,233,304,307,310,367,371,386,541,551,555,564,568,571,637,640,686,693,754,765,959,966,970,973,987,1079,1085,1089,1092,1195,1198,1202,1206,1209,1213,1219,1223,1226,1230,1233,1237,1246,1307,1310,1313,1337,1344,1348,1357,1371,1378,1549,1552,1556,1559,1579,1583,1586,1589,1592,1596,1646,1649],[10,11,13],"h2",{"id":12},"introduction","Introduction",[15,16,17],"p",{},"Cloud services are convenient, but they come with a cost — both financial and in terms of privacy. Over time, I found myself paying for Bitwarden, Google Photos, and various other SaaS tools while also handing over my personal data to third parties. I decided it was time to take back control.",[15,19,20],{},"The result is a three-node homelab cluster running Proxmox and k3s, capable of real high availability (HA), hosting everything from password management to photo backups — all on hardware sitting on my desk.",[15,22,23],{},"In this post I'll walk through the hardware, the architecture decisions, and the key components that make it all work.",[10,25,27],{"id":26},"hardware","Hardware",[15,29,30],{},"The cluster is made up of three mini PCs, each running Proxmox as the hypervisor:",[32,33,34,59],"table",{},[35,36,37],"thead",{},[38,39,40,44,47,50,53,56],"tr",{},[41,42,43],"th",{},"Node",[41,45,46],{},"Machine",[41,48,49],{},"CPU",[41,51,52],{},"RAM",[41,54,55],{},"Storage",[41,57,58],{},"Role",[60,61,62,83,103],"tbody",{},[38,63,64,68,71,74,77,80],{},[65,66,67],"td",{},"cp",[65,69,70],{},"Lenovo ThinkCentre M70q Tiny",[65,72,73],{},"Intel Core i7-12700T",[65,75,76],{},"32 GB",[65,78,79],{},"1 TB NVMe",[65,81,82],{},"Control Plane",[38,84,85,88,91,94,97,100],{},[65,86,87],{},"wk1",[65,89,90],{},"Dell OptiPlex 7020 Micro",[65,92,93],{},"Intel Core i5-12500T",[65,95,96],{},"16 GB",[65,98,99],{},"256 GB NVMe",[65,101,102],{},"Worker",[38,104,105,108,110,112,114,116],{},[65,106,107],{},"wk2",[65,109,90],{},[65,111,93],{},[65,113,96],{},[65,115,99],{},[65,117,102],{},[15,119,120],{},"The Lenovo M70q is the beefiest node and acts as the k3s control plane, while the two Dell OptiPlex machines handle the workloads. All three are compact, power-efficient, and surprisingly capable for the price.",[10,122,124],{"id":123},"network-architecture","Network Architecture",[15,126,127],{},"One of the most important decisions in a homelab HA setup is the network layout. I separated traffic into two dedicated networks:",[129,130,131,139],"ul",{},[132,133,134,138],"li",{},[135,136,137],"strong",{},"Management network (1 Gbps)",": Used for Proxmox management, SSH, and general cluster communication.",[132,140,141,144],{},[135,142,143],{},"Corosync/cluster sync network (2.5 Gbps)",": Dedicated to Proxmox cluster heartbeat and live migration traffic. Using a faster, isolated link here prevents cluster split-brain scenarios caused by network congestion on the management interface.",[15,146,147],{},"This separation is critical for a real HA setup. If Corosync traffic shares the same interface as general traffic, a busy network can cause false node fencing events.",[10,149,151],{"id":150},"why-proxmox","Why Proxmox?",[15,153,154],{},"I chose Proxmox VE as the hypervisor for a few reasons:",[129,156,157,163,169,175],{},[132,158,159,162],{},[135,160,161],{},"Free and open source",": No licensing costs, unlike VMware ESXi.",[132,164,165,168],{},[135,166,167],{},"Built-in clustering",": Proxmox Cluster File System (pmxcfs) and Corosync give you HA out of the box.",[132,170,171,174],{},[135,172,173],{},"KVM + LXC",": You can run both full VMs and lightweight containers.",[132,176,177,180],{},[135,178,179],{},"Web UI",": The management interface is excellent for a homelab.",[15,182,183],{},"Each mini PC runs Proxmox, and the three nodes form a Proxmox cluster. This means VMs can be live-migrated between nodes and, in case of a node failure, they can be automatically restarted on a healthy node.",[10,185,187],{"id":186},"k3s-on-top-of-proxmox","k3s on Top of Proxmox",[15,189,190],{},"Rather than running k3s directly on bare metal, I run it inside Proxmox VMs. This gives me:",[129,192,193,199,205],{},[132,194,195,198],{},[135,196,197],{},"Snapshot and backup",": I can snapshot the entire VM state before upgrades.",[132,200,201,204],{},[135,202,203],{},"Resource isolation",": Each k3s node is a VM with defined CPU and memory limits.",[132,206,207,210],{},[135,208,209],{},"Flexibility",": I can easily resize or recreate a node without touching the physical machine.",[15,212,213],{},"The k3s cluster follows the standard single control plane + two worker nodes topology:",[215,216,221],"pre",{"className":217,"code":219,"language":220},[218],"language-text","┌─────────────────────────────────────────────┐\n│              Proxmox Cluster                │\n│                                             │\n│  ┌──────────┐  ┌──────────┐  ┌──────────┐  │\n│  │  M70q    │  │ OPX 7020 │  │ OPX 7020 │  │\n│  │          │  │          │  │          │  │\n│  │  k3s-cp  │  │  k3s-wk1 │  │  k3s-wk2 │  │\n│  └──────────┘  └──────────┘  └──────────┘  │\n└─────────────────────────────────────────────┘\n","text",[222,223,219],"code",{"__ignoreMap":224},"",[226,227,229],"h3",{"id":228},"installing-k3s","Installing k3s",[15,231,232],{},"The control plane node is installed with the embedded etcd datastore for HA readiness:",[215,234,238],{"className":235,"code":236,"language":237,"meta":224,"style":224},"language-bash shiki shiki-themes one-dark-pro","curl -sfL https://get.k3s.io | sh -s - server \\\n  --cluster-init \\\n  --disable traefik \\\n  --disable servicelb\n","bash",[222,239,240,277,285,296],{"__ignoreMap":224},[241,242,245,249,253,257,261,264,267,270,273],"span",{"class":243,"line":244},"line",1,[241,246,248],{"class":247},"sVbv2","curl",[241,250,252],{"class":251},"sVC51"," -sfL",[241,254,256],{"class":255},"subq3"," https://get.k3s.io",[241,258,260],{"class":259},"sn6KH"," | ",[241,262,263],{"class":247},"sh",[241,265,266],{"class":251}," -s",[241,268,269],{"class":255}," -",[241,271,272],{"class":255}," server",[241,274,276],{"class":275},"sjrmR"," \\\n",[241,278,280,283],{"class":243,"line":279},2,[241,281,282],{"class":251},"  --cluster-init",[241,284,276],{"class":275},[241,286,288,291,294],{"class":243,"line":287},3,[241,289,290],{"class":251},"  --disable",[241,292,293],{"class":255}," traefik",[241,295,276],{"class":275},[241,297,299,301],{"class":243,"line":298},4,[241,300,290],{"class":251},[241,302,303],{"class":255}," servicelb\n",[15,305,306],{},"I disable the built-in Traefik and ServiceLB because I use MetalLB and a custom ingress controller instead.",[15,308,309],{},"Worker nodes join using the cluster token:",[215,311,313],{"className":235,"code":312,"language":237,"meta":224,"style":224},"curl -sfL https://get.k3s.io | K3S_URL=https://\u003Ccp-ip>:6443 \\\n  K3S_TOKEN=\u003Cnode-token> sh -\n",[222,314,315,349],{"__ignoreMap":224},[241,316,317,319,321,323,325,329,332,335,338,341,344,347],{"class":243,"line":244},[241,318,248],{"class":247},[241,320,252],{"class":251},[241,322,256],{"class":255},[241,324,260],{"class":259},[241,326,328],{"class":327},"sVyAn","K3S_URL",[241,330,331],{"class":275},"=",[241,333,334],{"class":255},"https://",[241,336,337],{"class":259},"\u003C",[241,339,340],{"class":255},"cp-ip",[241,342,343],{"class":259},">",[241,345,346],{"class":255},":6443",[241,348,276],{"class":247},[241,350,351,354,356,359,362,364],{"class":243,"line":279},[241,352,353],{"class":255},"  K3S_TOKEN=",[241,355,337],{"class":259},[241,357,358],{"class":255},"node-toke",[241,360,361],{"class":259},"n> ",[241,363,263],{"class":255},[241,365,366],{"class":255}," -\n",[10,368,370],{"id":369},"exposing-services-with-metallb","Exposing Services with MetalLB",[15,372,373,374,381,382,385],{},"Since this is a bare-metal cluster (no cloud load balancer), I use ",[375,376,380],"a",{"href":377,"rel":378},"https://metallb.universe.tf/",[379],"nofollow","MetalLB"," in Layer 2 mode to assign real IP addresses to ",[222,383,384],{},"LoadBalancer"," services.",[215,387,391],{"className":388,"code":389,"language":390,"meta":224,"style":224},"language-yaml shiki shiki-themes one-dark-pro","apiVersion: metallb.io/v1beta1\nkind: IPAddressPool\nmetadata:\n  name: homelab-pool\n  namespace: metallb-system\nspec:\n  addresses:\n    - 192.168.1.200-192.168.1.220\n---\napiVersion: metallb.io/v1beta1\nkind: L2Advertisement\nmetadata:\n  name: homelab-l2\n  namespace: metallb-system\nspec:\n  ipAddressPools:\n    - homelab-pool\n","yaml",[222,392,393,404,414,422,432,443,451,459,468,474,483,493,500,510,519,526,534],{"__ignoreMap":224},[241,394,395,398,401],{"class":243,"line":244},[241,396,397],{"class":327},"apiVersion",[241,399,400],{"class":259},": ",[241,402,403],{"class":255},"metallb.io/v1beta1\n",[241,405,406,409,411],{"class":243,"line":279},[241,407,408],{"class":327},"kind",[241,410,400],{"class":259},[241,412,413],{"class":255},"IPAddressPool\n",[241,415,416,419],{"class":243,"line":287},[241,417,418],{"class":327},"metadata",[241,420,421],{"class":259},":\n",[241,423,424,427,429],{"class":243,"line":298},[241,425,426],{"class":327},"  name",[241,428,400],{"class":259},[241,430,431],{"class":255},"homelab-pool\n",[241,433,435,438,440],{"class":243,"line":434},5,[241,436,437],{"class":327},"  namespace",[241,439,400],{"class":259},[241,441,442],{"class":255},"metallb-system\n",[241,444,446,449],{"class":243,"line":445},6,[241,447,448],{"class":327},"spec",[241,450,421],{"class":259},[241,452,454,457],{"class":243,"line":453},7,[241,455,456],{"class":327},"  addresses",[241,458,421],{"class":259},[241,460,462,465],{"class":243,"line":461},8,[241,463,464],{"class":259},"    - ",[241,466,467],{"class":255},"192.168.1.200-192.168.1.220\n",[241,469,471],{"class":243,"line":470},9,[241,472,473],{"class":259},"---\n",[241,475,477,479,481],{"class":243,"line":476},10,[241,478,397],{"class":327},[241,480,400],{"class":259},[241,482,403],{"class":255},[241,484,486,488,490],{"class":243,"line":485},11,[241,487,408],{"class":327},[241,489,400],{"class":259},[241,491,492],{"class":255},"L2Advertisement\n",[241,494,496,498],{"class":243,"line":495},12,[241,497,418],{"class":327},[241,499,421],{"class":259},[241,501,503,505,507],{"class":243,"line":502},13,[241,504,426],{"class":327},[241,506,400],{"class":259},[241,508,509],{"class":255},"homelab-l2\n",[241,511,513,515,517],{"class":243,"line":512},14,[241,514,437],{"class":327},[241,516,400],{"class":259},[241,518,442],{"class":255},[241,520,522,524],{"class":243,"line":521},15,[241,523,448],{"class":327},[241,525,421],{"class":259},[241,527,529,532],{"class":243,"line":528},16,[241,530,531],{"class":327},"  ipAddressPools",[241,533,421],{"class":259},[241,535,537,539],{"class":243,"line":536},17,[241,538,464],{"class":259},[241,540,431],{"class":255},[15,542,543,544,546,547,550],{},"Any service of type ",[222,545,384],{}," now gets an IP from the ",[222,548,549],{},"192.168.1.200–220"," range, making it reachable from anywhere on my local network.",[10,552,554],{"id":553},"tls-with-cert-manager-and-a-local-ca","TLS with cert-manager and a Local CA",[15,556,557,558,563],{},"For internal services I don't want to expose to the internet, I use ",[375,559,562],{"href":560,"rel":561},"https://cert-manager.io/",[379],"cert-manager"," with a self-signed local Certificate Authority. This gives every service a valid TLS certificate without relying on Let's Encrypt or exposing ports 80/443 to the outside world.",[226,565,567],{"id":566},"setting-up-the-local-ca","Setting Up the Local CA",[15,569,570],{},"First, generate the CA key and certificate:",[215,572,574],{"className":235,"code":573,"language":237,"meta":224,"style":224},"openssl genrsa -out ca.key 4096\nopenssl req -new -x509 -days 3650 -key ca.key \\\n  -out ca.crt \\\n  -subj \"/CN=Homelab CA/O=Homelab\"\n",[222,575,576,593,619,629],{"__ignoreMap":224},[241,577,578,581,584,587,590],{"class":243,"line":244},[241,579,580],{"class":247},"openssl",[241,582,583],{"class":255}," genrsa",[241,585,586],{"class":251}," -out",[241,588,589],{"class":255}," ca.key",[241,591,592],{"class":251}," 4096\n",[241,594,595,597,600,603,606,609,612,615,617],{"class":243,"line":279},[241,596,580],{"class":247},[241,598,599],{"class":255}," req",[241,601,602],{"class":251}," -new",[241,604,605],{"class":251}," -x509",[241,607,608],{"class":251}," -days",[241,610,611],{"class":251}," 3650",[241,613,614],{"class":251}," -key",[241,616,589],{"class":255},[241,618,276],{"class":275},[241,620,621,624,627],{"class":243,"line":287},[241,622,623],{"class":251},"  -out",[241,625,626],{"class":255}," ca.crt",[241,628,276],{"class":275},[241,630,631,634],{"class":243,"line":298},[241,632,633],{"class":251},"  -subj",[241,635,636],{"class":255}," \"/CN=Homelab CA/O=Homelab\"\n",[15,638,639],{},"Store it as a Kubernetes secret:",[215,641,643],{"className":235,"code":642,"language":237,"meta":224,"style":224},"kubectl create secret tls homelab-ca \\\n  --cert=ca.crt \\\n  --key=ca.key \\\n  -n cert-manager\n",[222,644,645,664,671,678],{"__ignoreMap":224},[241,646,647,650,653,656,659,662],{"class":243,"line":244},[241,648,649],{"class":247},"kubectl",[241,651,652],{"class":255}," create",[241,654,655],{"class":255}," secret",[241,657,658],{"class":255}," tls",[241,660,661],{"class":255}," homelab-ca",[241,663,276],{"class":275},[241,665,666,669],{"class":243,"line":279},[241,667,668],{"class":251},"  --cert=ca.crt",[241,670,276],{"class":275},[241,672,673,676],{"class":243,"line":287},[241,674,675],{"class":251},"  --key=ca.key",[241,677,276],{"class":275},[241,679,680,683],{"class":243,"line":298},[241,681,682],{"class":251},"  -n",[241,684,685],{"class":255}," cert-manager\n",[15,687,688,689,692],{},"Create a ",[222,690,691],{},"ClusterIssuer"," that uses this CA:",[215,694,696],{"className":388,"code":695,"language":390,"meta":224,"style":224},"apiVersion: cert-manager.io/v1\nkind: ClusterIssuer\nmetadata:\n  name: homelab-ca-issuer\nspec:\n  ca:\n    secretName: homelab-ca\n",[222,697,698,707,716,722,731,737,744],{"__ignoreMap":224},[241,699,700,702,704],{"class":243,"line":244},[241,701,397],{"class":327},[241,703,400],{"class":259},[241,705,706],{"class":255},"cert-manager.io/v1\n",[241,708,709,711,713],{"class":243,"line":279},[241,710,408],{"class":327},[241,712,400],{"class":259},[241,714,715],{"class":255},"ClusterIssuer\n",[241,717,718,720],{"class":243,"line":287},[241,719,418],{"class":327},[241,721,421],{"class":259},[241,723,724,726,728],{"class":243,"line":298},[241,725,426],{"class":327},[241,727,400],{"class":259},[241,729,730],{"class":255},"homelab-ca-issuer\n",[241,732,733,735],{"class":243,"line":434},[241,734,448],{"class":327},[241,736,421],{"class":259},[241,738,739,742],{"class":243,"line":445},[241,740,741],{"class":327},"  ca",[241,743,421],{"class":259},[241,745,746,749,751],{"class":243,"line":453},[241,747,748],{"class":327},"    secretName",[241,750,400],{"class":259},[241,752,753],{"class":255},"homelab-ca\n",[15,755,756,757,760,761,764],{},"Now any ",[222,758,759],{},"Certificate"," or ",[222,762,763],{},"Ingress"," resource can request a TLS certificate signed by your local CA:",[215,766,768],{"className":388,"code":767,"language":390,"meta":224,"style":224},"apiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\n  name: bitwarden\n  annotations:\n    cert-manager.io/cluster-issuer: homelab-ca-issuer\nspec:\n  tls:\n    - hosts:\n        - bitwarden.homelab.local\n      secretName: bitwarden-tls\n  rules:\n    - host: bitwarden.homelab.local\n      http:\n        paths:\n          - path: /\n            pathType: Prefix\n            backend:\n              service:\n                name: bitwarden\n                port:\n                  number: 80\n",[222,769,770,779,788,794,803,810,819,825,832,841,849,859,866,877,884,891,904,914,922,930,940,948],{"__ignoreMap":224},[241,771,772,774,776],{"class":243,"line":244},[241,773,397],{"class":327},[241,775,400],{"class":259},[241,777,778],{"class":255},"networking.k8s.io/v1\n",[241,780,781,783,785],{"class":243,"line":279},[241,782,408],{"class":327},[241,784,400],{"class":259},[241,786,787],{"class":255},"Ingress\n",[241,789,790,792],{"class":243,"line":287},[241,791,418],{"class":327},[241,793,421],{"class":259},[241,795,796,798,800],{"class":243,"line":298},[241,797,426],{"class":327},[241,799,400],{"class":259},[241,801,802],{"class":255},"bitwarden\n",[241,804,805,808],{"class":243,"line":434},[241,806,807],{"class":327},"  annotations",[241,809,421],{"class":259},[241,811,812,815,817],{"class":243,"line":445},[241,813,814],{"class":327},"    cert-manager.io/cluster-issuer",[241,816,400],{"class":259},[241,818,730],{"class":255},[241,820,821,823],{"class":243,"line":453},[241,822,448],{"class":327},[241,824,421],{"class":259},[241,826,827,830],{"class":243,"line":461},[241,828,829],{"class":327},"  tls",[241,831,421],{"class":259},[241,833,834,836,839],{"class":243,"line":470},[241,835,464],{"class":259},[241,837,838],{"class":327},"hosts",[241,840,421],{"class":259},[241,842,843,846],{"class":243,"line":476},[241,844,845],{"class":259},"        - ",[241,847,848],{"class":255},"bitwarden.homelab.local\n",[241,850,851,854,856],{"class":243,"line":485},[241,852,853],{"class":327},"      secretName",[241,855,400],{"class":259},[241,857,858],{"class":255},"bitwarden-tls\n",[241,860,861,864],{"class":243,"line":495},[241,862,863],{"class":327},"  rules",[241,865,421],{"class":259},[241,867,868,870,873,875],{"class":243,"line":502},[241,869,464],{"class":259},[241,871,872],{"class":327},"host",[241,874,400],{"class":259},[241,876,848],{"class":255},[241,878,879,882],{"class":243,"line":512},[241,880,881],{"class":327},"      http",[241,883,421],{"class":259},[241,885,886,889],{"class":243,"line":521},[241,887,888],{"class":327},"        paths",[241,890,421],{"class":259},[241,892,893,896,899,901],{"class":243,"line":528},[241,894,895],{"class":259},"          - ",[241,897,898],{"class":327},"path",[241,900,400],{"class":259},[241,902,903],{"class":255},"/\n",[241,905,906,909,911],{"class":243,"line":536},[241,907,908],{"class":327},"            pathType",[241,910,400],{"class":259},[241,912,913],{"class":255},"Prefix\n",[241,915,917,920],{"class":243,"line":916},18,[241,918,919],{"class":327},"            backend",[241,921,421],{"class":259},[241,923,925,928],{"class":243,"line":924},19,[241,926,927],{"class":327},"              service",[241,929,421],{"class":259},[241,931,933,936,938],{"class":243,"line":932},20,[241,934,935],{"class":327},"                name",[241,937,400],{"class":259},[241,939,802],{"class":255},[241,941,943,946],{"class":243,"line":942},21,[241,944,945],{"class":327},"                port",[241,947,421],{"class":259},[241,949,951,954,956],{"class":243,"line":950},22,[241,952,953],{"class":327},"                  number",[241,955,400],{"class":259},[241,957,958],{"class":251},"80\n",[15,960,961,962,965],{},"The only thing left is to import ",[222,963,964],{},"ca.crt"," into your browser or OS trust store on each device you use. Once done, all your homelab services show a green padlock.",[10,967,969],{"id":968},"storage-nfs-local-path","Storage: NFS + Local Path",[15,971,972],{},"For storage I use a combination of two approaches:",[129,974,975,981],{},[132,976,977,980],{},[135,978,979],{},"local-path",": The default k3s storage class. Fast, simple, uses the node's local NVMe. Good for stateless workloads or anything that doesn't need to survive node failures.",[132,982,983,986],{},[135,984,985],{},"NFS",": A shared NFS export from the M70q (which has the 1 TB NVMe) mounted on all nodes. Used for workloads that need persistent, shared storage — like Nextcloud or Immich.",[215,988,990],{"className":388,"code":989,"language":390,"meta":224,"style":224},"apiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n  name: nextcloud-data\nspec:\n  storageClassName: nfs-client\n  accessModes:\n    - ReadWriteMany\n  resources:\n    requests:\n      storage: 200Gi\n",[222,991,992,1001,1010,1016,1025,1031,1041,1048,1055,1062,1069],{"__ignoreMap":224},[241,993,994,996,998],{"class":243,"line":244},[241,995,397],{"class":327},[241,997,400],{"class":259},[241,999,1000],{"class":255},"v1\n",[241,1002,1003,1005,1007],{"class":243,"line":279},[241,1004,408],{"class":327},[241,1006,400],{"class":259},[241,1008,1009],{"class":255},"PersistentVolumeClaim\n",[241,1011,1012,1014],{"class":243,"line":287},[241,1013,418],{"class":327},[241,1015,421],{"class":259},[241,1017,1018,1020,1022],{"class":243,"line":298},[241,1019,426],{"class":327},[241,1021,400],{"class":259},[241,1023,1024],{"class":255},"nextcloud-data\n",[241,1026,1027,1029],{"class":243,"line":434},[241,1028,448],{"class":327},[241,1030,421],{"class":259},[241,1032,1033,1036,1038],{"class":243,"line":445},[241,1034,1035],{"class":327},"  storageClassName",[241,1037,400],{"class":259},[241,1039,1040],{"class":255},"nfs-client\n",[241,1042,1043,1046],{"class":243,"line":453},[241,1044,1045],{"class":327},"  accessModes",[241,1047,421],{"class":259},[241,1049,1050,1052],{"class":243,"line":461},[241,1051,464],{"class":259},[241,1053,1054],{"class":255},"ReadWriteMany\n",[241,1056,1057,1060],{"class":243,"line":470},[241,1058,1059],{"class":327},"  resources",[241,1061,421],{"class":259},[241,1063,1064,1067],{"class":243,"line":476},[241,1065,1066],{"class":327},"    requests",[241,1068,421],{"class":259},[241,1070,1071,1074,1076],{"class":243,"line":485},[241,1072,1073],{"class":327},"      storage",[241,1075,400],{"class":259},[241,1077,1078],{"class":255},"200Gi\n",[15,1080,1081,1084],{},[222,1082,1083],{},"ReadWriteMany"," is key here — it allows the volume to be mounted by multiple pods simultaneously, which is needed for Nextcloud's data directory.",[10,1086,1088],{"id":1087},"what-i-self-host","What I Self-Host",[15,1090,1091],{},"The whole point of this setup is to replace paid cloud services with self-hosted alternatives. Here's what's running on the cluster:",[32,1093,1094,1107],{},[35,1095,1096],{},[38,1097,1098,1101,1104],{},[41,1099,1100],{},"Service",[41,1102,1103],{},"Replaces",[41,1105,1106],{},"Notes",[60,1108,1109,1124,1139,1154,1169,1184],{},[38,1110,1111,1118,1121],{},[65,1112,1113],{},[375,1114,1117],{"href":1115,"rel":1116},"https://github.com/dani-garcia/vaultwarden",[379],"Vaultwarden",[65,1119,1120],{},"Bitwarden Premium",[65,1122,1123],{},"Lightweight Bitwarden-compatible server",[38,1125,1126,1133,1136],{},[65,1127,1128],{},[375,1129,1132],{"href":1130,"rel":1131},"https://adguard.com/en/adguard-home/overview.html",[379],"AdGuard Home",[65,1134,1135],{},"Pi-hole / DNS-based ad blocking",[65,1137,1138],{},"Network-wide ad and tracker blocking",[38,1140,1141,1148,1151],{},[65,1142,1143],{},[375,1144,1147],{"href":1145,"rel":1146},"https://tailscale.com/",[379],"Tailscale",[65,1149,1150],{},"VPN",[65,1152,1153],{},"Secure remote access to the cluster from anywhere",[38,1155,1156,1163,1166],{},[65,1157,1158],{},[375,1159,1162],{"href":1160,"rel":1161},"https://immich.app/",[379],"Immich",[65,1164,1165],{},"Google Photos",[65,1167,1168],{},"Self-hosted photo and video backup with ML features",[38,1170,1171,1178,1181],{},[65,1172,1173],{},[375,1174,1177],{"href":1175,"rel":1176},"https://nextcloud.com/",[379],"Nextcloud",[65,1179,1180],{},"Google Drive / Dropbox",[65,1182,1183],{},"File sync, calendar, contacts",[38,1185,1186,1189,1192],{},[65,1187,1188],{},"Strava activity sync",[65,1190,1191],{},"Strava API",[65,1193,1194],{},"Powers the sports stats on this website",[15,1196,1197],{},"Running these on my own hardware means my passwords, photos, and files never leave my home network. The only external dependency is Tailscale for remote access, which only acts as a relay — the data itself stays local.",[10,1199,1201],{"id":1200},"lessons-learned","Lessons Learned",[226,1203,1205],{"id":1204},"corosync-needs-a-dedicated-link","Corosync needs a dedicated link",[15,1207,1208],{},"Early on I had Corosync running over the same 1 Gbps interface as everything else. During heavy NFS transfers, the cluster would occasionally report nodes as unreachable and trigger unnecessary fencing. Moving Corosync to the dedicated 2.5 Gbps link solved this completely.",[226,1210,1212],{"id":1211},"local-path-is-not-ha","local-path is not HA",[15,1214,1215,1216,1218],{},"If a pod using ",[222,1217,979],{}," storage is scheduled on a node that goes down, it won't come back up on another node — the data is tied to that specific node's disk. For anything important, use NFS or consider Longhorn for a proper distributed storage layer.",[226,1220,1222],{"id":1221},"resource-requests-matter","Resource requests matter",[15,1224,1225],{},"Without proper CPU and memory requests on pods, the scheduler can overcommit nodes and cause OOM kills. Setting realistic requests and limits on every workload keeps the cluster stable.",[226,1227,1229],{"id":1228},"snapshots-before-upgrades","Snapshots before upgrades",[15,1231,1232],{},"One of the biggest advantages of running k3s inside Proxmox VMs is the ability to snapshot before any major change. Before upgrading k3s or a critical application, I take a VM snapshot. If something breaks, rollback takes about 30 seconds.",[10,1234,1236],{"id":1235},"observability-grafana-prometheus","Observability: Grafana + Prometheus",[15,1238,1239,1240,1245],{},"With multiple services running across three nodes, knowing what's happening inside the cluster is essential. I use the ",[375,1241,1244],{"href":1242,"rel":1243},"https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack",[379],"kube-prometheus-stack"," Helm chart, which bundles Prometheus, Alertmanager, and Grafana in one shot.",[215,1247,1249],{"className":235,"code":1248,"language":237,"meta":224,"style":224},"helm repo add prometheus-community https://prometheus-community.github.io/helm-charts\nhelm repo update\nhelm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \\\n  --namespace monitoring \\\n  --create-namespace\n",[222,1250,1251,1268,1277,1292,1302],{"__ignoreMap":224},[241,1252,1253,1256,1259,1262,1265],{"class":243,"line":244},[241,1254,1255],{"class":247},"helm",[241,1257,1258],{"class":255}," repo",[241,1260,1261],{"class":255}," add",[241,1263,1264],{"class":255}," prometheus-community",[241,1266,1267],{"class":255}," https://prometheus-community.github.io/helm-charts\n",[241,1269,1270,1272,1274],{"class":243,"line":279},[241,1271,1255],{"class":247},[241,1273,1258],{"class":255},[241,1275,1276],{"class":255}," update\n",[241,1278,1279,1281,1284,1287,1290],{"class":243,"line":287},[241,1280,1255],{"class":247},[241,1282,1283],{"class":255}," install",[241,1285,1286],{"class":255}," kube-prometheus-stack",[241,1288,1289],{"class":255}," prometheus-community/kube-prometheus-stack",[241,1291,276],{"class":275},[241,1293,1294,1297,1300],{"class":243,"line":298},[241,1295,1296],{"class":251},"  --namespace",[241,1298,1299],{"class":255}," monitoring",[241,1301,276],{"class":275},[241,1303,1304],{"class":243,"line":434},[241,1305,1306],{"class":251},"  --create-namespace\n",[15,1308,1309],{},"Grafana comes pre-loaded with dashboards for node CPU/memory, pod resource usage, and Kubernetes cluster health. I expose it via an Ingress with a cert-manager TLS certificate, just like every other service on the cluster.",[15,1311,1312],{},"Prometheus scrapes metrics from:",[129,1314,1315,1321,1327],{},[132,1316,1317,1320],{},[135,1318,1319],{},"Node Exporter"," — CPU, RAM, disk, and network stats for each physical node",[132,1322,1323,1326],{},[135,1324,1325],{},"kube-state-metrics"," — Kubernetes object state (deployments, pods, PVCs)",[132,1328,1329,1332,1333,1336],{},[135,1330,1331],{},"Application exporters"," — services like AdGuard Home expose their own ",[222,1334,1335],{},"/metrics"," endpoint",[15,1338,1339,1340,1343],{},"Having this in place means I can spot a node running hot, a pod stuck in ",[222,1341,1342],{},"CrashLoopBackOff",", or a disk filling up before it becomes a problem.",[10,1345,1347],{"id":1346},"dashboard-a-single-pane-of-glass","Dashboard: A Single Pane of Glass",[15,1349,1350,1351,1356],{},"For a quick overview of everything running on the cluster, I use ",[375,1352,1355],{"href":1353,"rel":1354},"https://homarr.dev/",[379],"Homarr"," as a home dashboard. It shows all my self-hosted apps as tiles with live status indicators, so I can see at a glance whether Vaultwarden, Nextcloud, or AdGuard Home is up.",[15,1358,1359,1360,760,1365,1370],{},"Alternatively, tools like ",[375,1361,1364],{"href":1362,"rel":1363},"https://gethomepage.dev/",[379],"Homepage",[375,1366,1369],{"href":1367,"rel":1368},"https://heimdall.site/",[379],"Heimdall"," work equally well — the choice is mostly aesthetic. I went with Homarr because it integrates directly with Docker and Kubernetes service discovery.",[15,1372,1373,1374,1377],{},"The dashboard is deployed as a standard Kubernetes ",[222,1375,1376],{},"Deployment"," and exposed via MetalLB + Ingress, same as everything else:",[215,1379,1381],{"className":388,"code":1380,"language":390,"meta":224,"style":224},"apiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: homarr\n  namespace: default\nspec:\n  replicas: 1\n  selector:\n    matchLabels:\n      app: homarr\n  template:\n    metadata:\n      labels:\n        app: homarr\n    spec:\n      containers:\n        - name: homarr\n          image: ghcr.io/ajnart/homarr:latest\n          ports:\n            - containerPort: 7575\n",[222,1382,1383,1392,1401,1407,1416,1425,1431,1441,1448,1455,1464,1471,1478,1485,1494,1501,1508,1519,1529,1536],{"__ignoreMap":224},[241,1384,1385,1387,1389],{"class":243,"line":244},[241,1386,397],{"class":327},[241,1388,400],{"class":259},[241,1390,1391],{"class":255},"apps/v1\n",[241,1393,1394,1396,1398],{"class":243,"line":279},[241,1395,408],{"class":327},[241,1397,400],{"class":259},[241,1399,1400],{"class":255},"Deployment\n",[241,1402,1403,1405],{"class":243,"line":287},[241,1404,418],{"class":327},[241,1406,421],{"class":259},[241,1408,1409,1411,1413],{"class":243,"line":298},[241,1410,426],{"class":327},[241,1412,400],{"class":259},[241,1414,1415],{"class":255},"homarr\n",[241,1417,1418,1420,1422],{"class":243,"line":434},[241,1419,437],{"class":327},[241,1421,400],{"class":259},[241,1423,1424],{"class":255},"default\n",[241,1426,1427,1429],{"class":243,"line":445},[241,1428,448],{"class":327},[241,1430,421],{"class":259},[241,1432,1433,1436,1438],{"class":243,"line":453},[241,1434,1435],{"class":327},"  replicas",[241,1437,400],{"class":259},[241,1439,1440],{"class":251},"1\n",[241,1442,1443,1446],{"class":243,"line":461},[241,1444,1445],{"class":327},"  selector",[241,1447,421],{"class":259},[241,1449,1450,1453],{"class":243,"line":470},[241,1451,1452],{"class":327},"    matchLabels",[241,1454,421],{"class":259},[241,1456,1457,1460,1462],{"class":243,"line":476},[241,1458,1459],{"class":327},"      app",[241,1461,400],{"class":259},[241,1463,1415],{"class":255},[241,1465,1466,1469],{"class":243,"line":485},[241,1467,1468],{"class":327},"  template",[241,1470,421],{"class":259},[241,1472,1473,1476],{"class":243,"line":495},[241,1474,1475],{"class":327},"    metadata",[241,1477,421],{"class":259},[241,1479,1480,1483],{"class":243,"line":502},[241,1481,1482],{"class":327},"      labels",[241,1484,421],{"class":259},[241,1486,1487,1490,1492],{"class":243,"line":512},[241,1488,1489],{"class":327},"        app",[241,1491,400],{"class":259},[241,1493,1415],{"class":255},[241,1495,1496,1499],{"class":243,"line":521},[241,1497,1498],{"class":327},"    spec",[241,1500,421],{"class":259},[241,1502,1503,1506],{"class":243,"line":528},[241,1504,1505],{"class":327},"      containers",[241,1507,421],{"class":259},[241,1509,1510,1512,1515,1517],{"class":243,"line":536},[241,1511,845],{"class":259},[241,1513,1514],{"class":327},"name",[241,1516,400],{"class":259},[241,1518,1415],{"class":255},[241,1520,1521,1524,1526],{"class":243,"line":916},[241,1522,1523],{"class":327},"          image",[241,1525,400],{"class":259},[241,1527,1528],{"class":255},"ghcr.io/ajnart/homarr:latest\n",[241,1530,1531,1534],{"class":243,"line":924},[241,1532,1533],{"class":327},"          ports",[241,1535,421],{"class":259},[241,1537,1538,1541,1544,1546],{"class":243,"line":932},[241,1539,1540],{"class":259},"            - ",[241,1542,1543],{"class":327},"containerPort",[241,1545,400],{"class":259},[241,1547,1548],{"class":251},"7575\n",[15,1550,1551],{},"Opening the dashboard URL gives me a single page with links to every service, their status, and quick-access widgets for things like AdGuard stats or Proxmox node health.",[10,1553,1555],{"id":1554},"whats-next","What's Next",[15,1557,1558],{},"The cluster is stable and serving all my needs, but there are a few things I want to improve:",[129,1560,1561,1567,1573],{},[132,1562,1563,1566],{},[135,1564,1565],{},"Longhorn",": Replace NFS with a proper distributed block storage solution for better resilience.",[132,1568,1569,1572],{},[135,1570,1571],{},"Flux or ArgoCD",": Move to GitOps for managing all cluster manifests.",[132,1574,1575,1578],{},[135,1576,1577],{},"Second control plane node",": Add a second k3s server node for true control plane HA.",[10,1580,1582],{"id":1581},"conclusion","Conclusion",[15,1584,1585],{},"Building a homelab HA cluster with Proxmox and k3s has been one of the most rewarding infrastructure projects I've worked on. The combination of Proxmox's VM management, k3s's lightweight Kubernetes, MetalLB for load balancing, and cert-manager for TLS creates a surprisingly production-like environment at home.",[15,1587,1588],{},"More importantly, I've taken back control of my data. My passwords, photos, files, and DNS queries no longer pass through third-party servers. The hardware cost was a one-time investment, and the running costs are just electricity — far cheaper than the subscriptions I was paying before.",[15,1590,1591],{},"If you're thinking about building your own homelab, I'd encourage you to start small. Even a single mini PC running Proxmox and k3s is enough to self-host most services. You can always add nodes later.",[10,1593,1595],{"id":1594},"resources","Resources",[129,1597,1598,1605,1612,1618,1625,1630,1636,1641],{},[132,1599,1600],{},[375,1601,1604],{"href":1602,"rel":1603},"https://pve.proxmox.com/pve-docs/",[379],"Proxmox VE Documentation",[132,1606,1607],{},[375,1608,1611],{"href":1609,"rel":1610},"https://docs.k3s.io/",[379],"k3s Documentation",[132,1613,1614],{},[375,1615,1617],{"href":377,"rel":1616},[379],"MetalLB Documentation",[132,1619,1620],{},[375,1621,1624],{"href":1622,"rel":1623},"https://cert-manager.io/docs/",[379],"cert-manager Documentation",[132,1626,1627],{},[375,1628,1244],{"href":1242,"rel":1629},[379],[132,1631,1632],{},[375,1633,1635],{"href":1353,"rel":1634},[379],"Homarr Dashboard",[132,1637,1638],{},[375,1639,1117],{"href":1115,"rel":1640},[379],[132,1642,1643],{},[375,1644,1162],{"href":1160,"rel":1645},[379],[15,1647,1648],{},"Happy homelabbing! 🏠☸️",[1650,1651,1652],"style",{},"html pre.shiki code .sVbv2, html code.shiki .sVbv2{--shiki-default:#61AFEF}html pre.shiki code .sVC51, html code.shiki .sVC51{--shiki-default:#D19A66}html pre.shiki code .subq3, html code.shiki .subq3{--shiki-default:#98C379}html pre.shiki code .sn6KH, html code.shiki .sn6KH{--shiki-default:#ABB2BF}html pre.shiki code .sjrmR, html code.shiki .sjrmR{--shiki-default:#56B6C2}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html pre.shiki code .sVyAn, html code.shiki .sVyAn{--shiki-default:#E06C75}",{"title":224,"searchDepth":279,"depth":279,"links":1654},[1655,1656,1657,1658,1659,1662,1663,1666,1667,1668,1674,1675,1676,1677,1678],{"id":12,"depth":279,"text":13},{"id":26,"depth":279,"text":27},{"id":123,"depth":279,"text":124},{"id":150,"depth":279,"text":151},{"id":186,"depth":279,"text":187,"children":1660},[1661],{"id":228,"depth":287,"text":229},{"id":369,"depth":279,"text":370},{"id":553,"depth":279,"text":554,"children":1664},[1665],{"id":566,"depth":287,"text":567},{"id":968,"depth":279,"text":969},{"id":1087,"depth":279,"text":1088},{"id":1200,"depth":279,"text":1201,"children":1669},[1670,1671,1672,1673],{"id":1204,"depth":287,"text":1205},{"id":1211,"depth":287,"text":1212},{"id":1221,"depth":287,"text":1222},{"id":1228,"depth":287,"text":1229},{"id":1235,"depth":279,"text":1236},{"id":1346,"depth":279,"text":1347},{"id":1554,"depth":279,"text":1555},{"id":1581,"depth":279,"text":1582},{"id":1594,"depth":279,"text":1595},"2026-05-17T21:00:00Z","2026-05-17T21:57:00Z","How I built a real high-availability Kubernetes cluster using Proxmox, k3s, MetalLB, cert-manager, Prometheus, Grafana and a home dashboard on three mini PCs to self-host my own services and take back control of my data","md",null,{},true,"/blog/homelab-with-proxmox-and-k3s-a-real-ha-cluster-on-mini-pcs",{"title":5,"description":1681},{"loc":1686},"blog/homelab-with-proxmox-and-k3s-a-real-ha-cluster-on-mini-pcs","YwxrPPXaJMy5-tVhrWIczodZqGWWpkJpx_bswz5yPlY",1779082031329]