# Mastering Port Forwarding as a Service: Running Kubernetes Port Forwards with systemd

Kubernetes port forwarding is extremely useful for quick access to internal services. It allows local tools to reach cluster applications without exposing them through Ingress or LoadBalancer services. The problem is that port forwarding breaks easily. It stops when the terminal closes, when the network changes, or when the forwarding process restarts. Many engineers use it only during development because of these limitations.

The good news is that port forwarding can be turned into a persistent background service that runs automatically on boot and restarts on failure. This can be achieved by running `kubectl port-forward` under systemd. Once configured, the forwarding behaves like any other system-level service.

## Understanding the Port-Forward Setup for Postgres

The goal was to expose a PostgreSQL instance running **inside your Kubernetes cluster** so that it could be accessed securely from **another machine**. Instead of exposing the database publicly or creating a LoadBalancer, you used a **port-forward** running as a persistent systemd service. This behaves very much like an **SSH tunnel**, but managed automatically by Kubernetes.

Accessing a PostgreSQL database that lives inside a Kubernetes cluster often starts with a quick command:

```python
kubectl port-forward pod/postgres 15432:5432 -n postgresql
```

It works.  
It is simple.  
And it also **breaks** the moment:

* your SSH session closes
    
* the terminal dies
    
* the pod restarts
    
* network hiccups occur
    

For development or for connecting external applications, this becomes frustrating. You want PostgreSQL running inside the cluster to feel like it is running locally. Always reachable. No interruptions.

This is exactly where a **persistent port-forwarding setup** becomes extremely useful.

In this blog, I explain how I automated PostgreSQL port-forwarding using:

* a simple shell script
    
* a systemd service
    
* dynamic pod detection
    
* persistent reconnection
    

This ensures that even if the pod restarts or the port-forward crashes, the system automatically brings it back up.

---

## **Why Manual Port Forwarding Fails Over Time**

Port-forwarding is not designed for long-term, production-grade networking. It is a debugging convenience.

When you run:

```python
kubectl port-forward pod/postgres 15432:5432
```

You are telling Kubernetes:

* Open **15432** on your **local machine**
    
* Forward all traffic from this local port
    
* To **port 5432** inside the **postgres pod**
    

The moment the terminal stops, or the pod is replaced by a new one during a deployment or restart, the connection is lost.

This leads to:

* connection refused
    
* ECONNRESET
    
* your app cannot connect to the database
    
* your scripts or migrations failing mid-run
    

The ideal fix is using a LoadBalancer, NodePort, an internal service mesh, or VPN into the cluster.  
But when that is not possible (for example in a locked-down internal environment), **persistent port-forwarding is a surprisingly useful workaround**.

## **Solution Overview**

We will set up:

1. **A shell script** that:
    
    * Continuously finds the current PostgreSQL pod
        
    * Establishes the port-forward
        
    * Automatically retries if the pod changes or the forward crashes
        
2. **A systemd service** that:
    
    * Runs this script in the background
        
    * Starts automatically on boot
        
    * Restarts on failure
        

This results in a self-healing port-forward that always stays alive.

---

## **Step 1: The Final Working Script**

Save it as:

```python
/home/ubuntu/pg-portforward.sh
```

```python
#!/bin/bash
set -e

export KUBECONFIG="${HOME}/.k0s/kubeconfig"

NAMESPACE="postgresql"
LOCAL_PORT=15432
REMOTE_PORT=5432

while true; do
  POD=$(kubectl get pod -n $NAMESPACE -l app=postgres \
        -o jsonpath='{.items[0].metadata.name}' 2>/dev/null)

  if [ -z "$POD" ]; then
    echo "[$(date)] Postgres pod not found. Retrying in 5s..."
    sleep 5
    continue
  fi

  echo "[$(date)] Forwarding to pod: $POD"
  kubectl -n $NAMESPACE port-forward pod/$POD ${LOCAL_PORT}:${REMOTE_PORT}

  echo "[$(date)] Port-forward crashed. Restarting in 5s..."
  sleep 5
done
```

Make it executable:

```python
chmod +x pg-portforward.sh
```

---

## **Step 2: The systemd Unit File**

Create:

```python
/etc/systemd/system/pg-portforward.service
```

```python
[Unit]
Description=Persistent port-forward for PostgreSQL pod
After=network.target

[Service]
User=ubuntu
Environment="KUBECONFIG=/home/ubuntu/.k0s/kubeconfig"
Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ExecStart=/home/ubuntu/pg-portforward.sh
Restart=always
RestartSec=5
KillMode=process

[Install]
WantedBy=multi-user.target
```

Enable and start it:

```python
sudo systemctl daemon-reload
sudo systemctl enable pg-portforward
sudo systemctl start pg-portforward
sudo systemctl status pg-portforward
```

---

## **How This Works**

### **1\. Dynamic Pod Discovery**

PostgreSQL pods often restart during:

* upgrades
    
* node draining
    
* scaling events
    

A static pod name does not work.  
The script uses a label selector:

```python
-l app=postgres
```

You can substitute whatever label your Helm chart applies.

### **2\. Automatic Reconnection**

If the port-forward dies (common), the script simply loops and starts again.

### **3\. systemd keeps it alive**

If the script itself fails, systemd restarts it.

If the machine reboots, the service auto-starts.

This ensures **PostgreSQL inside Kubernetes remains reachable on** [**localhost:5432**](http://localhost:5432) **at all times**.

### Why a Systemd Service

Port-forwarding dies when:

* the pod restarts
    
* the connection breaks
    
* kubectl crashes
    
* the terminal closes
    

To keep the port-forward **alive forever**, you wrapped it in:

* a small bash script that loops forever, automatically reconnecting
    
* a systemd service that starts on boot and restarts on failure
    

This means your Postgres database is always reachable through that forward without babysitting the terminal.

---

## **Testing the Setup**

From your local machine:

```python
psql -h <your-node-ip> -p 15432 -U postgres -d yourdb
```

If SSH-tunneling:

```python
ssh -L 5432:localhost:15432 ubuntu@<server>
psql -h localhost -p 5432
```

If port-forward is running correctly, the connection will be instant.

In DBBeaver:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1764705001160/cfe7b46c-878b-4c53-9d27-dfe7c2d78590.png align="center")

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1764705003805/0f055eb1-ac85-4ddf-86c7-0260e4fe8256.png align="center")

---

## **Common Issues and Fixes**

### **kubectl not found inside systemd**

systemd does not automatically inherit your PATH.  
This is why we added:

```python
Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
```

### **KUBECONFIG not respected**

Your kubeconfig was located here:

```python
/home/ubuntu/.k0s/kubeconfig
```

So we exported it in both the script *and* the service.

### **Pod name changing**

The script automatically handles this.

### Security Benefits

You avoided exposing Postgres publicly. Only users who can SSH into the server can reach the database. No load balancers. No ingress. No nodePort. Just a controlled, encrypted tunnel.

---

## **Conclusion**

Port forwarding is usually thought of as a temporary debugging feature, but with a small wrapper script and a systemd unit, it becomes a powerful mechanism for stable access to internal services. For PostgreSQL in particular, this method delivers reliable availability without relaxing cluster security.

This setup is easy to maintain, self-healing, and ideal for environments where PostgreSQL must be reachable consistently through a trusted machine.