DIY Multi-Hop Boundary Sessions without HCP

A common theme in my posts, is to treat my homelab as if it has the same security requirements as a production environment. Since, in theory, it is a production environment, and I wouldn't want anything to happen to it. For protected access, I normally use something like Tailscale, since I don't need to expose anything to the internet, but sometimes when I am accessing it via a remote network, that network may be locked down and prevent Tailscale/wireguard connections, so I need a break-glass solution.

It is no secret from other posts on this blog, that I use Hashicorp's suite of tools, and the next one I'd like to introduce is Boundary. Boundary is a tool that allows for identity based access management, meaning that no matter the system I want to connect to in my homelab, I can log into Boundary, and connect to it, without having to worry about SSH tunneling, or exposing it to the internet. I also might not have access to a machine that I can install VPN software on, and in the case of Boundary, it is a single binary that doesn't need administrative access to run, so it is a viable option for me.

The Problem#

Sadly, I am budget constrained for my homelab, and the community version of Boundary does not support ingress/egress workers, which are required for multi-hop sessions. This is a feature that is only available in the enterprise version of Boundary, or HCP (HashiCorp Cloud Platform).

I'd like to have these multi-hop sessions, so I can keep some VLAN isolation, and not have the server in the DMZ be able to connect back directly to every server in my homelab.

Here's a visual of what I wanted to achieve:

My DIY Solution#

The Boundary connection flow is key to my solution. Each session goes through the following steps:

1. The client (my laptop) connects to the controller
2. The controller provides an authorization token
3. The client connects to the worker port using this token
4. The worker establishes the session with the target

With this process in mind, I developed a workaround. First, I had to create an FRP (Fast Reverse Proxy) server on the same server as my DMZ Boundary Controller and Worker. That way the internal workers could reach out to that server, and I wouldn't need to open any firewall rules in the direction of the internal workers. Then, after the workers are registered with Boundary, I also had to create a target for the internal workers which point to a unique port from frps on the Boundary server loopback that aligns with a specific worker.

Here's how the connection flow works with my solution:

Setting it up#

Using the standard Boundary installation process, you'd need to expose the DMZ Boundary Controller (for API access, and usually 9200), and worker (usually port 9202) to the internet. It could be something like boundary.tklk.dev, and based on your preferences you could use Caddy and have it use real TLS certificates for when you are attempting to access the API. The internal workers will also need to connect to the cluster worker coordinator on port 9201, but that doesn't need to be exposed to the internet, and can be done via the same way that the internal workers connect to the FRP server.

Installing FRP Server#

First, you'll need to install FRP on your DMZ server:

1. Download the appropriate FRP release from GitHub
2. Extract the archive and locate the frps binary
3. Create a configuration file as shown below
4. Set up a systemd service (optional) to ensure it runs automatically

A sample frps configuration, based on what I am actually using:

bindPort = 7000
auth.method = "token"
auth.token = "your_secure_token"

# meaning the internal workers can only use this range for listening on
allowPorts = [ {start = 9500, end = 9999}  ]

Internal Worker Setup#

Now that you have the brains of the operation setup, you'll need to move onto the internal workers, and follow these steps:

1. Install and configure a standard Boundary worker

   # minimal worker config
   # you may wish to add more configuration options, such as using a KMS
   listener "tcp" {
     # this is the listener that frpc will connect to over the loop back
     address = "127.0.0.1:9202"
     purpose = "proxy"
   }
   
   worker {
     name = "internal-worker-1"
     # this is the address that the boundary controller will announce to the client
     # so this is what the target session should listen to on your local laptop/client
     public_addr = "127.0.0.1:9500"
     description = "Worker in the internal VLAN"
     controllers = ["boundary.tklk.dev:9201"]
   }
   

2. Install FRP client (frpc)

3. Configure frpc to establish a reverse tunnel back to the primary server on a loopback interface

Here's a sample frpc configuration, based on what I am actually using:

serverAddr = "boundary.tklk.dev"
serverPort = 7000
auth.method = "token"
auth.token = "your_secure_token"

[[proxies]]
name = "boundary_internal_worker"
type = "tcp"
localIP = "127.0.0.1"
# 9202 is the Boundary worker's default port, but since you may have multiple internal workers, you might want to pick
# a unique port for each worker (which you'll also need to set in the worker config)
# It is strongly recommended that you don't use port 9202 as the DMZ worker will already likely be using this port, causing the frps connection to fail
localPort = 9202
remotePort = 9500

Registering the Internal Worker as a Target#

The last part of the setup is that you'll now need to register the internal worker as a target in Boundary. This is done by creating a target that points to the FRP server's loopback interface, and the port that you set in the frpc configuration. This can be done via terraform, the boundary cli, or the Boundary UI.

A target with 127.0.0.1 as the target host, and the port that you set in the frpc configuration, along with the default client port being the same as the worker port is the bare minimum you'll need to set.

You may also want to set an egress worker filter, to ensure that your DMZ worker is the one that connects to the frps port, as the target uses localhost, the routing might be assigned to a different worker that is not yet accessible.

Using the Setup#

Now comes the fun part, actually using the setup. Go through the standard authentication process to get a session token, and then use the boundary connect command to establish a connection to the internal worker target.

# Authenticate with Boundary (if not already authenticated)
boundary authenticate

# List available targets
boundary targets list -recursive

The workflow to access internal resources involves two steps:

1. First establish a connection to the internal worker:

   boundary connect -target-id=ttcp_1234567890  # Your internal worker target ID
   

   If nothing else on your local machine is listening on the worker port as defined above, you'll open a connection to the internal worker. In your session list, this connection may be marked as "pending" since no traffic is being sent yet.

2. Once connected, you can access targets in the internal VLAN:

   boundary connect -target-id=ttcp_0987654321  # Internal resource target ID
   

While this requires an extra step compared to the enterprise solution, it provides the same functionality at no additional cost.

Limitations and Considerations#

• You need to manage FRP alongside Boundary
• The two-step connection process is less streamlined than the enterprise solution
• Nested connections can add a small amount of latency
• Security considerations: Be careful with your FRP token to prevent unauthorized access

Conclusion#

While I probably don't need the multi-hop sessions in my homelab, and likely single hop sessions would be sufficient, I wanted to fully explore the capabilities of Boundary to learn more about it in depth. I came out of this experience with a deeper understanding of the connection flow, and techniques to extend and debug various networking issues of it.

This could be extended to use vault and broker SSH credentials with boundary, but you could leverage my previous post on using vault with SSH to do that instead. For now, I am happy with the setup I have.