Kubernetes port forwarding is extremely useful for quick access to internal services. It allows local tools to reach cluster applications without exposing them through Ingress or LoadBalancer services. The problem is that port forwarding breaks easily. It stops when the terminal closes, when the network changes, or when the forwarding process restarts. Many engineers use it only during development because of these limitations.
The good news is that port forwarding can be turned into a persistent background service that runs automatically on boot and restarts on failure. This can be achieved by running kubectl port-forward under systemd. Once configured, the forwarding behaves like any other system-level service.
Understanding the Port-Forward Setup for Postgres
The goal was to expose a PostgreSQL instance running inside your Kubernetes cluster so that it could be accessed securely from another machine. Instead of exposing the database publicly or creating a LoadBalancer, you used a port-forward running as a persistent systemd service. This behaves very much like an SSH tunnel, but managed automatically by Kubernetes.
Accessing a PostgreSQL database that lives inside a Kubernetes cluster often starts with a quick command:
kubectl port-forward pod/postgres 15432:5432 -n postgresql
It works.
It is simple.
And it also breaks the moment:
your SSH session closes
the terminal dies
the pod restarts
network hiccups occur
For development or for connecting external applications, this becomes frustrating. You want PostgreSQL running inside the cluster to feel like it is running locally. Always reachable. No interruptions.
This is exactly where a persistent port-forwarding setup becomes extremely useful.
In this blog, I explain how I automated PostgreSQL port-forwarding using:
a simple shell script
a systemd service
dynamic pod detection
persistent reconnection
This ensures that even if the pod restarts or the port-forward crashes, the system automatically brings it back up.
Why Manual Port Forwarding Fails Over Time
Port-forwarding is not designed for long-term, production-grade networking. It is a debugging convenience.
When you run:
kubectl port-forward pod/postgres 15432:5432
You are telling Kubernetes:
Open 15432 on your local machine
Forward all traffic from this local port
To port 5432 inside the postgres pod
The moment the terminal stops, or the pod is replaced by a new one during a deployment or restart, the connection is lost.
This leads to:
The ideal fix is using a LoadBalancer, NodePort, an internal service mesh, or VPN into the cluster.
But when that is not possible (for example in a locked-down internal environment), persistent port-forwarding is a surprisingly useful workaround.
Solution Overview
We will set up:
A shell script that:
Continuously finds the current PostgreSQL pod
Establishes the port-forward
Automatically retries if the pod changes or the forward crashes
A systemd service that:
This results in a self-healing port-forward that always stays alive.
Step 1: The Final Working Script
Save it as:
/home/ubuntu/pg-portforward.sh
set -e
export KUBECONFIG="${HOME}/.k0s/kubeconfig"
NAMESPACE="postgresql"
LOCAL_PORT=15432
REMOTE_PORT=5432
while true; do
POD=$(kubectl get pod -n $NAMESPACE -l app=postgres \
-o jsonpath='{.items[0].metadata.name}' 2>/dev/null)
if [ -z "$POD" ]; then
echo "[$(date)] Postgres pod not found. Retrying in 5s..."
sleep 5
continue
fi
echo "[$(date)] Forwarding to pod: $POD"
kubectl -n $NAMESPACE port-forward pod/$POD ${LOCAL_PORT}:${REMOTE_PORT}
echo "[$(date)] Port-forward crashed. Restarting in 5s..."
sleep 5
done
Make it executable:
chmod +x pg-portforward.sh
Step 2: The systemd Unit File
Create:
/etc/systemd/system/pg-portforward.service
[Unit]
Description=Persistent port-forward for PostgreSQL pod
After=network.target
[Service]
User=ubuntu
Environment="KUBECONFIG=/home/ubuntu/.k0s/kubeconfig"
Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ExecStart=/home/ubuntu/pg-portforward.sh
Restart=always
RestartSec=5
KillMode=process
[Install]
WantedBy=multi-user.target
Enable and start it:
sudo systemctl daemon-reload
sudo systemctl enable pg-portforward
sudo systemctl start pg-portforward
sudo systemctl status pg-portforward
How This Works
1. Dynamic Pod Discovery
PostgreSQL pods often restart during:
upgrades
node draining
scaling events
A static pod name does not work.
The script uses a label selector:
-l app=postgres
You can substitute whatever label your Helm chart applies.
2. Automatic Reconnection
If the port-forward dies (common), the script simply loops and starts again.
3. systemd keeps it alive
If the script itself fails, systemd restarts it.
If the machine reboots, the service auto-starts.
This ensures PostgreSQL inside Kubernetes remains reachable on localhost:5432 at all times.
Why a Systemd Service
Port-forwarding dies when:
the pod restarts
the connection breaks
kubectl crashes
the terminal closes
To keep the port-forward alive forever, you wrapped it in:
a small bash script that loops forever, automatically reconnecting
a systemd service that starts on boot and restarts on failure
This means your Postgres database is always reachable through that forward without babysitting the terminal.
Testing the Setup
From your local machine:
psql -h <your-node-ip> -p 15432 -U postgres -d yourdb
If SSH-tunneling:
ssh -L 5432:localhost:15432 ubuntu@<server>
psql -h localhost -p 5432
If port-forward is running correctly, the connection will be instant.
In DBBeaver:
Common Issues and Fixes
kubectl not found inside systemd
systemd does not automatically inherit your PATH.
This is why we added:
Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
KUBECONFIG not respected
Your kubeconfig was located here:
/home/ubuntu/.k0s/kubeconfig
So we exported it in both the script and the service.
Pod name changing
The script automatically handles this.
Security Benefits
You avoided exposing Postgres publicly. Only users who can SSH into the server can reach the database. No load balancers. No ingress. No nodePort. Just a controlled, encrypted tunnel.
Conclusion
Port forwarding is usually thought of as a temporary debugging feature, but with a small wrapper script and a systemd unit, it becomes a powerful mechanism for stable access to internal services. For PostgreSQL in particular, this method delivers reliable availability without relaxing cluster security.
This setup is easy to maintain, self-healing, and ideal for environments where PostgreSQL must be reachable consistently through a trusted machine.
