Talos OS: A Hard Earned Understanding Of Storage, Certificates, And Access

Talos OS promises a fully immutable, API driven Kubernetes experience. It removes the idea of logging into nodes, changing files manually, or performing maintenance through traditional means. This design brings a high level of security and predictability. It also brings a set of challenges that many users, including myself, only discover once Talos becomes part of a real cluster.

During my recent effort to run PostgreSQL on a Talos cluster, I experienced several failures, unexpected behaviours, and some difficult recovery situations. This post documents that entire experience. I want this to help anyone who is trying to use Talos in a small cluster or a homelab environment, because the learning curve is very steep.

Understanding Why Storage Becomes Difficult

Talos is an immutable operating system. This sounds ideal until you try to mount storage. Many Kubernetes setups allow you to create directories directly on the node using simple commands. Talos does not allow this approach.

These are the limitations that immediately matter:

• You cannot create directories on the host manually

• You cannot change permissions manually

• You cannot rely on paths that do not already exist

• You cannot SSH into the node to fix things

• You cannot depend on anything that is not part of the machine configuration

My PostgreSQL deployment required a PersistentVolume backed by local storage. I created a PersistentVolume that pointed to a path like /var/lib/postgres or /mnt/postgres. Each attempt failed with the same error.

MountVolume.NewMounter initialization failed for volume "pv-postgres" : path "/var/lib/postgres" does not exist

Talos refuses to mount a path that does not exist. Since I was unable to create that directory manually, I needed a path that already existed.

The solution was surprisingly simple!

I used a directory that Talos already creates by default.

/var/local

The moment I pointed my PersistentVolume to /var/local, PostgreSQL started successfully. The directory existed, the kubelet accepted it, and the pod finally mounted the data volume.

This taught me the most important lesson about Talos and storage. Any stateful workload that needs a hostPath or a local PersistentVolume must rely on a path that exists at boot through the machine configuration. If the path is not created by Talos, then the pod will fail.

I then discovered a default directory that Talos already creates: /var/local. Pointing my PV to this path worked immediately:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-postgres
spec:
  storageClassName: localstorage
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  local:
    path: /var/local
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - talos-1to-jsz

The corresponding PVC was:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-postgresql
  namespace: postgresql
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: localstorage
  resources:
    requests:
      storage: 10Gi

Then I applied my deployment manifest, alongside svc:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
  namespace: postgresql
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:16
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 5432
          env:
            - name: POSTGRES_DB
              value: mydb
            - name: POSTGRES_USER
              value: myuser
            - name: POSTGRES_PASSWORD
              value: mypassword
          volumeMounts:
            - name: postgres-data
              mountPath: /var/lib/postgresql/data
      volumes:
        - name: postgres-data
          persistentVolumeClaim:
            claimName: data-postgresql
---
apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: postgresql
spec:
  selector:
    app: postgres
  ports:
    - name: postgres
      protocol: TCP
      port: 5432
      targetPort: 5432
  type: ClusterIP

After this, the PostgreSQL pod successfully started and mounted the volume. I was able to connect to it using a test pod:

kubectl run psql-test \
  --rm -it \
  --image=postgres:16 \
  --namespace postgresql \
  --env="PGPASSWORD=mypassword" \
  -- psql -h postgres -U myuser -d mydb

This experience taught me a critical lesson about Talos and storage: any stateful workload requiring a hostPath or local PV must rely on directories that exist at boot through the machine configuration. Trying to use arbitrary directories will fail. While /var/local worked, this is not a best practice and should be avoided in production.

Why Using Existing Directories Like /var/local Works But Is Not A Good Practice

When I struggled to mount storage for PostgreSQL on Talos, I eventually discovered that /var/local already existed on every node. Talos creates this directory during early boot. As soon as I pointed my PersistentVolume to /var/local, the database pod started without any issues. The directory already existed, the kubelet was satisfied, and the pod finally mounted the data volume.

This seems like a convenient solution, but it is not a recommended approach. It introduces several long term problems and operational risks.

Here is why it is not a good practice.

1. This directory is not meant for application data

/var/local is an internal Talos directory and is not designed for stateful workloads. Talos can modify or use this directory for its own purposes in future releases. Talos does not document or guarantee that this directory will always exist, or that it will behave the same way across upgrades.

You are depending on behaviour that is a side effect of the operating system rather than a stable feature.

2. All applications will share the same storage location

If you reuse /var/local for every PersistentVolume, then every stateful application on that node will write data into the same directory. This will cause:

• A lack of isolation between applications

• Possible permission conflicts

• A risk of one application filling the entire directory and breaking the others

• Difficulty with debugging and storage visibility

It also becomes impossible to safely delete or migrate individual application data.

3. Talos does not enforce size limits

Kubernetes does not enforce the size declared in a PersistentVolume. Since /var/local is just a directory, any application can exceed the declared ten gibibytes. There is no guarantee that the node will not run out of space.

In a worst case scenario, the node can crash due to disk pressure.

4. This breaks the philosophy of Talos

Talos is meant to be fully declarative. Anything that exists should be defined through the MachineConfig, not through accidental filesystem structure that happens to be present.

If the directory is not created explicitly by configuration, then it is not an intentional part of your infrastructure. It is risky to build application level storage on top of something that was not designed for this purpose.

5. Upgrades and reinstallations can remove the directory

During major upgrades or node reinstalls, Talos can reset or restructure internal directories. If /var/local is removed, renamed, or reformatted, then every application that relies on it will lose its data.

Your data becomes fragile and tied to undocumented filesystem details.

---

The Correct Talos Approved Way

The recommended way to create storage paths in Talos is:

Use a MachineConfig patch to create directories with explicit permissions and ownership.

For example, you can declare this:

machine:
  files:
    - path: /var/data/postgres
      permissions: 0o755
      owner: 0
      group: 0
      directory: true

You can create as many directories as you need. This is the clean, controlled, and safe method.

You then point your PersistentVolumes to these paths with confidence that:

• They will exist on every node

• They will survive upgrades

• They are dedicated to the correct application

• They will not conflict with Talos internals

This is the long term maintainable approach, but I faced the certificate error here, which made life difficult!!

Expanding Persistent Volumes

In the example above, I claimed the full 10Gi of available storage on the node. If I want to expand storage later, I must first ensure more disk space is available at the path and then patch the PVC to request the larger size. Kubernetes will handle resizing if the StorageClass allows it.

For multiple services requiring persistent storage, I can technically reuse /var/local, but this is also not recommended. Each workload should ideally have its own dedicated storage path or volume managed through a proper storage provider.

---

Why Longhorn Does Not Work Easily

Longhorn requires certain kernel modules, directory mounts, and filesystem behaviours that Talos does not support by default. Talos aims for very minimal host configuration. Longhorn expects the opposite. The two conflict in many ways.

As a result, most users who attempt Longhorn on Talos experience failures. This includes random crashes, volume mount issues, replica failures, and inability to start the Longhorn UI.

The safer alternative is to use Talos MachineConfig patches to create custom paths and then rely on local PersistentVolumes. This reduces flexibility but increases stability.

---

The Strange Behaviour Of PersistentVolumes In Talos

When I created a 10 gigabyte PersistentVolume and a matching PersistentVolumeClaim, it worked immediately on /var/local. This made me curious about what was actually happening.

Here is the explanation.

• The size declared in a PersistentVolume does not actually allocate disk space

• The directory simply points to the host filesystem

• Talos does not perform any reservations

• Kubelet does not enforce storage consumption limits

• The declared capacity only informs Kubernetes scheduling

This means that even if you declare ten gibibytes, the actual host directory is unbounded. PostgreSQL can consume much more than the advertised size if it needs to. The responsibility of storage growth sits entirely on you as the administrator.

If you want to expand a PersistentVolume in Talos:

1. You need a larger physical directory available

2. You resize the PersistentVolumeClaim

3. The underlying filesystem must support online expansion

This works with filesystems like ext4 and xfs if configured correctly by the node.

---

Certificates and Node Access

Another issue I faced was related to Talos certificates. After initial node creation, when I tried to apply a new worker configuration (machine files path to create directory), I received errors like:

error applying new configuration: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"

Even though I used the same worker.yaml file, Talos refused the connection. The lesson here is that Talos tightly couples node certificates with the cluster PKI. If a certificate becomes invalid or untrusted, you can lose direct access.

There are several scenarios where this failure can occur:

• You regenerate a machine configuration file

• You recreate a node with a slightly different configuration

• You lose your talosconfig file

• The cluster CA is overridden when bootstrap runs again

• Node IP addresses change

• You accidentally mix configuration files from different clusters

When this happens, you cannot log into the node. You cannot fix files manually. You cannot mount a debug shell. This is both a security feature and a very serious operational risk.

This is the moment when Talos begins to feel unforgiving. A lost certificate means that you lose control of the node unless you have backups of your original configuration, certificates, and bootstrap secrets.

The Difficulty Of Troubleshooting And Recovery

Troubleshooting Talos is not like troubleshooting a normal Linux server. There is no SSH access, no direct shell, and no persistent filesystem to examine. Everything flows through the Talos API. If the Talos API is broken because of mismatched certificates, then you are locked out completely.

The only reliable recovery options are:

• Reboot into maintanence mode

• Reapply a machine configuration

• Restore backed up secrets

• Reinstall the node if nothing else works

This can feel very restrictive. It requires a mindset shift. Talos does not want administrators to fix issues manually. Talos wants everything to be declared in configuration files from the start.

This is powerful, but very easy to break if any detail is forgotten, and if you need to reset the node, consider your previous work gone :(

Recovering Node Access

To regain access to a Talos node when the certificate fails:

1. Use the Talos bootstrap token for insecure communication:

talosctl apply-config --insecure -n <node-ip> --file worker.yaml

2. Alternatively, download a fresh Talos configuration using:

talosctl -n <node-ip> kubeconfig -f kubeconfig.yaml

3. Use talosctl --insecure carefully, as it bypasses certificate validation.

4. Always back up your talosconfig files and certificates. Losing them makes access recovery difficult.

Key Lessons

• Talos does not allow arbitrary hostPath directories. Only paths present at boot or created via machine configuration are valid for PVs.

• Use /var/local as a temporary solution, but do not rely on it for production workloads.

• Always backup Talos certificates and configuration to avoid losing access.

• Stateful workloads require careful planning of persistent storage in Talos.

Talos is secure and minimal by design, but these features make working with storage and configuration more challenging than standard Linux nodes.

---

My Conclusion After Working Through These Issues

Talos OS is impressive. It is secure and consistent. At the same time, it brings operational challenges that are not obvious until you experience them directly.

The three biggest issues I faced were:

1. Storage paths that cannot be created manually

2. Certificates that break node access

3. Recovery procedures that depend entirely on correct configuration files

Talos is ideal for large production environments where every configuration is version controlled, stable, and tested. It can be difficult for homelab environments where experimentation is common and nodes often change.

In the end I created working PersistentVolumes, understood the certificate failures, recovered node access, and built functional PostgreSQL storage. This journey helped me understand Talos in a much deeper way. I now appreciate how strict and predictable it is, even though that strictness caused many of the problems.

If you are planning to learn Talos, try to keep configuration backups from the very beginning. It will save you much more time later!

I am still learning and perhaps there are simpler ways to have Talos in homelab experiments, but for now I will stick to more experiment friendly kubernetes.