Mastering Port Forwarding as a Service: Running Kubernetes Port Forwards with systemd

Kubernetes port forwarding is extremely useful for quick access to internal services. It allows local tools to reach cluster applications without exposing them through Ingress or LoadBalancer services. The problem is that port forwarding breaks easily. It stops when the terminal closes, when the network changes, or when the forwarding process restarts. Many engineers use it only during development because of these limitations.

The good news is that port forwarding can be turned into a persistent background service that runs automatically on boot and restarts on failure. This can be achieved by running kubectl port-forward under systemd. Once configured, the forwarding behaves like any other system-level service.

Understanding the Port-Forward Setup for Postgres

The goal was to expose a PostgreSQL instance running inside your Kubernetes cluster so that it could be accessed securely from another machine. Instead of exposing the database publicly or creating a LoadBalancer, you used a port-forward running as a persistent systemd service. This behaves very much like an SSH tunnel, but managed automatically by Kubernetes.

Accessing a PostgreSQL database that lives inside a Kubernetes cluster often starts with a quick command:

kubectl port-forward pod/postgres 15432:5432 -n postgresql

It works.
It is simple.
And it also breaks the moment:

• your SSH session closes

• the terminal dies

• the pod restarts

• network hiccups occur

For development or for connecting external applications, this becomes frustrating. You want PostgreSQL running inside the cluster to feel like it is running locally. Always reachable. No interruptions.

This is exactly where a persistent port-forwarding setup becomes extremely useful.

In this blog, I explain how I automated PostgreSQL port-forwarding using:

• a simple shell script

• a systemd service

• dynamic pod detection

• persistent reconnection

This ensures that even if the pod restarts or the port-forward crashes, the system automatically brings it back up.

---

Why Manual Port Forwarding Fails Over Time

Port-forwarding is not designed for long-term, production-grade networking. It is a debugging convenience.

When you run:

kubectl port-forward pod/postgres 15432:5432

You are telling Kubernetes:

• Open 15432 on your local machine

• Forward all traffic from this local port

• To port 5432 inside the postgres pod

The moment the terminal stops, or the pod is replaced by a new one during a deployment or restart, the connection is lost.

This leads to:

• connection refused

• ECONNRESET

• your app cannot connect to the database

• your scripts or migrations failing mid-run

The ideal fix is using a LoadBalancer, NodePort, an internal service mesh, or VPN into the cluster.
But when that is not possible (for example in a locked-down internal environment), persistent port-forwarding is a surprisingly useful workaround.

Solution Overview

We will set up:

1. A shell script that:

   • Continuously finds the current PostgreSQL pod

   • Establishes the port-forward

   • Automatically retries if the pod changes or the forward crashes

2. A systemd service that:

   • Runs this script in the background

   • Starts automatically on boot

   • Restarts on failure

This results in a self-healing port-forward that always stays alive.

---

Step 1: The Final Working Script

Save it as:

/home/ubuntu/pg-portforward.sh

#!/bin/bash
set -e

export KUBECONFIG="${HOME}/.k0s/kubeconfig"

NAMESPACE="postgresql"
LOCAL_PORT=15432
REMOTE_PORT=5432

while true; do
  POD=$(kubectl get pod -n $NAMESPACE -l app=postgres \
        -o jsonpath='{.items[0].metadata.name}' 2>/dev/null)

  if [ -z "$POD" ]; then
    echo "[$(date)] Postgres pod not found. Retrying in 5s..."
    sleep 5
    continue
  fi

  echo "[$(date)] Forwarding to pod: $POD"
  kubectl -n $NAMESPACE port-forward pod/$POD ${LOCAL_PORT}:${REMOTE_PORT}

  echo "[$(date)] Port-forward crashed. Restarting in 5s..."
  sleep 5
done

Make it executable:

chmod +x pg-portforward.sh

---

Step 2: The systemd Unit File

Create:

/etc/systemd/system/pg-portforward.service

[Unit]
Description=Persistent port-forward for PostgreSQL pod
After=network.target

[Service]
User=ubuntu
Environment="KUBECONFIG=/home/ubuntu/.k0s/kubeconfig"
Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ExecStart=/home/ubuntu/pg-portforward.sh
Restart=always
RestartSec=5
KillMode=process

[Install]
WantedBy=multi-user.target

Enable and start it:

sudo systemctl daemon-reload
sudo systemctl enable pg-portforward
sudo systemctl start pg-portforward
sudo systemctl status pg-portforward

---

How This Works

1. Dynamic Pod Discovery

PostgreSQL pods often restart during:

• upgrades

• node draining

• scaling events

A static pod name does not work.
The script uses a label selector:

-l app=postgres

You can substitute whatever label your Helm chart applies.

2. Automatic Reconnection

If the port-forward dies (common), the script simply loops and starts again.

3. systemd keeps it alive

If the script itself fails, systemd restarts it.

If the machine reboots, the service auto-starts.

This ensures PostgreSQL inside Kubernetes remains reachable on localhost:5432 at all times.

Why a Systemd Service

Port-forwarding dies when:

• the pod restarts

• the connection breaks

• kubectl crashes

• the terminal closes

To keep the port-forward alive forever, you wrapped it in:

• a small bash script that loops forever, automatically reconnecting

• a systemd service that starts on boot and restarts on failure

This means your Postgres database is always reachable through that forward without babysitting the terminal.

---

Testing the Setup

From your local machine:

psql -h <your-node-ip> -p 15432 -U postgres -d yourdb

If SSH-tunneling:

ssh -L 5432:localhost:15432 ubuntu@<server>
psql -h localhost -p 5432

If port-forward is running correctly, the connection will be instant.

In DBBeaver:

---

Common Issues and Fixes

kubectl not found inside systemd

systemd does not automatically inherit your PATH.
This is why we added:

Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

KUBECONFIG not respected

Your kubeconfig was located here:

/home/ubuntu/.k0s/kubeconfig

So we exported it in both the script and the service.

Pod name changing

The script automatically handles this.

Security Benefits

You avoided exposing Postgres publicly. Only users who can SSH into the server can reach the database. No load balancers. No ingress. No nodePort. Just a controlled, encrypted tunnel.

---

Conclusion

Port forwarding is usually thought of as a temporary debugging feature, but with a small wrapper script and a systemd unit, it becomes a powerful mechanism for stable access to internal services. For PostgreSQL in particular, this method delivers reliable availability without relaxing cluster security.

This setup is easy to maintain, self-healing, and ideal for environments where PostgreSQL must be reachable consistently through a trusted machine.