Enable session recording with S3-compatible storage
Boundary 0.16 added SSH session recording support using the MinIO storage backend. This support enables users who want to integrate an S3-compatible storage backend for monitoring Boundary session activity. Session recording is available for HCP Boundary Plus and Boundary Enterprise users, and provides insight into user actions over remote SSH sessions to meet regulatory requirements for organizations and prevent malicious behavior. Administrators can enable session recording on SSH targets in their Boundary environment and replay recordings back within the Boundary admin UI.
The MinIO plugin is S3 compatible and enables session recording for other Amazon S3 compatible storage providers, such as Hitachi Object Storage.
This tutorial demonstrates enabling SSH session recording using MinIO as the storage backend and Boundary's native credential management features.
Tutorial overview
- Prerequisites
- Background
- Get setup
- Deploy a target and worker
- Set up MinIO
- Set up Boundary
- Enable session recording
- Verify and play back recordings
Prerequisites
Note
This tutorial was tested on in August 2024 using macOS 14.5, and in the Windows 11 Subsystem for Linux (WSL) with Ubuntu 20.04.
Before moving on, check that the following versions or greater are installed.
This tutorial recommends completing the HCP Boundary administration tutorials first. The learner should have a working Boundary cluster and org running on HCP, or a licensed Boundary Enterprise cluster.
Docker is installed (Note: Learners may also bring their own target, instead of using Docker)
A Boundary binary greater than 0.16.0 in your
PATH
Terraform 0.14.9 or greater provides an optional workflow to complete the lab. The binary must be available in your
PATH
.The
jq
utility is recommended to simplify the CLI workflow. It should be installed and in yourPATH
.Installing the Boundary Desktop App provides an optional workflow for this tutorial. The 2.0.0 version or above is recommended.
Session recording background
A common requirement and challenge in highly regulated environments is having a system of record that archives actions taken on the network so that organizations can improve their security posture and enhance compliance.
Session recording allows administrators to get insight into user actions over remote SSH sessions to meet regulatory requirements for organizations and prevent malicious behavior. Administrators can enable session recording on SSH targets in their Boundary environment, store signed recordings in their storage bucket, and replay recordings back within the Boundary admin UI.
Recorded sessions get converted into a Boundary session recording (BSR) file, a binary file format and specification created to define the structure of Boundary recording files.
BSR files are designed to:
- Support the recording of both multiplexed and non-multiplexed protocols
- Allow recordings of independent byte streams in a session to be written in parallel
- Support an optimal user experience during playback
- Be extensible to support more protocols in the future
BSR contains all the data transmitted between a user and a target during a session and is available within your storage bucket. These files are signed to ensure they are tamper-proof.
SSH session recording is available as a part of the Plus tier in both HCP Boundary and Boundary Enterprise.
Get set up
The following components get configured in the lab environment for this tutorial:
- HCP Boundary Plus or Boundary Enterprise cluster
- MinIO storage bucket
- SSH host
- A worker instance to proxy & record your SSH session
Deploy an HCP Boundary Plus cluster
Session recording, credential injection, and SSH targets are features available in HCP Boundary Plus.
First, deploy an HCP Boundary cluster with the HCP Plus sku selected.
Launch the HCP Portal and login.
Select your organization and project. From within that project, select Boundary from the Services menu in the left navigation.
Click Deploy Boundary.
In the Instance Name text box, provide a name for your Boundary instance.
Under Choose a tier, select the Plus option to enable session recording.
Under the Create an administrator account section, enter the Username and Password for the initial Boundary administrator account.
Click Deploy. Wait for the instance to initialize before proceeding.
Note
The first 5 users for any HCP Boundary cluster are free, after which you are charged per additional user. You can delete the HCP Plus cluster after this tutorial without incurring any costs.
The following values will be used later on. Copy the Boundary Cluster URL from the HCP Boundary portal.
- Boundary address: the
BOUNDARY_ADDR
variable - Boundary Cluster ID: the
BOUNDARY_CLUSTER_ID
variable - Boundary admin username: the
BOUNDARY_USERNAME
variable - Boundary admin password: the
BOUNDARY_PASSWORD
variable
Store these values in a safe location.
Note
The Boundary cluster ID is part of the Boundary address. For example, if your cluster URL is:
https://abcd1234-e567-f890-1ab2-cde345f6g789.boundary.hashicorp.cloud
Your cluster id is abcd1234-e567-f890-1ab2-cde345f6g789
.
Next, click Open Admin UI.
Log in to Boundary using your admin credentials used to launch the cluster.
Navigate to the Auth Methods page using the left navigation panel. Locate the
password
auth method, and copy its ID (such as ampw_AQSr776Hnm
).
If you follow the Terraform workflow, you will use this value later on:
- Boundary auth method ID: the
BOUNDARY_AUTH_METHOD_ID
variable
Deploy a target and worker
Deploy an open-ssh container as a target for Boundary.
Note
You may also bring your own target for this tutorial. You can follow this guide to create a publicly accessible EC2 instance to use for this tutorial. If using a privately accessible target for testing, you must deploy a Boundary worker with network access to the target.
Deploy the target
Ensure Docker is running, and then deploy an openssh container to use as a target.
Note that the username for the openssh server is admin
, the password is password
, and the target is available on your localhost at port 2222
(127.0.0.1:2222
).
Export the openssh container password as an environment variable.
Check that the container is running.
Deploy a worker
Session recording requires HCP Boundary or Boundary Enterprise. This tutorial uses Docker to deploy a Boundary worker to provide access to the openssh target container.
When using this image as a worker with HCP Boundary or Boundary Enterprise, you do not need to provide the license using an environment variable or config, but the worker must connect with a licensed controller.
Create a new file in your working directory called config.hcl
. Supply the hcp_boundary_cluster_id
, which is the UUID in the HCP Boundary cluster URL.
Tip
If your cluster is configured for multi-hop sessions, add an upstream worker using initial_upstreams
to the worker{}
stanza.
Save the config file.
Open a new terminal and switch to the directory with the config.hcl
file, then deploy the boundary-enterprise
Docker container.
The worker will start attempting to connect to the controller.
Scroll to the top of the output and copy the Worker Auth Registration Request value.
You can register the worker using the Admin Console Web UI or the CLI.
Authenticate to HCP Boundary as the admin user.
Log in to the HCP portal.
From the HCP Portal's Boundary page, click Open Admin UI - a new page will open.
Enter the admin username and password you created when you deployed the new instance and click Authenticate.
Once logged in, navigate to the Workers page.
Notice that only HCP workers are listed.
Click New.
The new workers page can be used to construct the contents of the
worker.hcl
file.
Do not fill in any of the worker fields.
Providing the following details will construct the worker config file contents for you:
- Boundary Cluster ID
- Worker Public Address
- Config file path
- Worker Tags
The instructions on this page provide details for installing the Boundary Enterprise binary and deploying the constructed config file.
Because the worker has already been deployed, only the Worker Auth Registration Request key needs to be provided on this page.
Scroll down to the bottom of the New Worker page and paste the Worker Auth Registration Request key you copied earlier.
Click Register Worker.
Click Done and notice the new worker on the Workers page.
Set up MinIO
This tutorial deploys MinIO using Docker. Alternatively, a MinIO server can be deployed locally on Linux, macOS, or Windows.
Deploy MinIO
Open a new terminal session and create a local config directory for MinIO.
Next, deploy a MinIO container using Docker. These instructions come from the MinIO container documentation.
Open a new terminal session and verify that all three containers are running.
Configure a MinIO storage bucket
Open the MinIO Console by visiting the following address in a web browser:
Log in with the following credentials (credentials are case sensitive):
- Username:
ROOTUSER
- Password:
CHANGEME123
Complete the following steps to configure a storage bucket:
Create a bucket.
Under the Administrator panel in the sidebar, click on Buckets.
Click the Create Bucket button.
Give the bucket a name, such as
boundary-recordings
.Click Create Bucket.
If you're following the CLI workflow, export the bucket name as a variable in your terminal session.
Create an access policy.
Under the User panel in the sidebar, click on Access Keys.
Click Create access key.
In the Create Access Key page, toggle the switch for Restrict beyond user policy to On.
Paste in the following IAM policy document:
1 2 3 4 5 6 7 8 9 1011121314151617181920
Ensure that lines 12 and 17 contain the correct name of your bucket, such as
boundary-recordings
.Enter an access key name in the Name field, such as
boundary-admin
.Click Create.
Copy the Access Key and Secret Key. You will not be able to view the Secret Key again, so you should also consider downloading the key.
If you are following the CLI workflow, export the bucket name as a variable in your terminal session.
Set up Boundary
The following resources are used in Boundary to enable session recording for an SSH target:
- A credential store
- A credential library
- A Boundary storage bucket
- An SSH target type with credential injection enabled
These resources can be configured using the Admin Console UI, the CLI, or Terraform. Select a workflow below to continue setting up Boundary.
Tip
Boundary storage bucket lifecycle management is still under development. To prevent unintentional loss of session recordings, orgs that contain storage buckets with recordings cannot currently be deleted. Before continuing, please note that the org created for this tutorial cannot be deleted until the session recordings it contains are deleted.
Start by logging in to the HCP Boundary Admin UI.
Log in to the HCP portal.
From the HCP Portal's Boundary page, click Open Admin UI - a new page will open.
Enter the admin username and password you created when you deployed the new instance and click Authenticate.
Next, set up a new testing org and project scope.
Note
Please use a new test org for this tutorial, because orgs that contain session recordings cannot currently be deleted.Navigate to the Orgs page and click New Org.
Fill out the new org form with a Name of
ssh-recording-org
and Description ofSSH test org
. Click Save.From within the new org, click New Project.
Fill out the new project form with a Name of
ssh-recording-project
and Description ofSecure Socket Handling recordings
. Click Save.
Create a credential store
Credential injection is required to enable session recording. You manage credentials using a Boundary credential store.
Navigate back to the
ssh-recording-org
, and select thessh-recording-project
.Select the Credential Stores page, and click New.
Enter the Name
SSH Credentials
.Select the type Static, and click Save.
Click the Credentials tab in the new credential store. Click New.
Under the New Credential page, enter the following details:
- Name:
openssh-creds
- Type:
Username & Password
- Username & Password:
admin
- Password:
password
Click Save.
- Name:
Enable session recordings
To finish setting up session recordings you need to:
- Set up a Boundary storage bucket
- Create an SSH target
The SSH target requires the injected application credentials (supplied from the static credential store), and the Boundary storage bucket it should associate recordings with.
Create a storage bucket
Within Boundary, a storage bucket resource is used to store the recorded sessions. A storage bucket represents a bucket in an external store, in this case, MinIO. You must create a Boundary storage bucket associated with an external store before enabling session recording.
Navigate to the Global scope, and then the Storage Buckets page.
Click New Storage Bucket. Fill out the following details:
- Name:
ssh-test-bucket
- Scope:
ssh-recording-org
- Provider:
MinIO
- Endpoint URL:
http://127.0.0.1:9000
- Bucket name:
boundary-recordings
Note: this must match the name of the bucket in MinIO. - Access key ID:
YOUR_MINIO_ACCESS_KEY_ID
- Secret access key:
YOUR_MINIO_SECRET_ACCESS_KEY
- Worker filter:
"minio" in "/tags/type"
The
YOUR_MINIO_ACCESS_KEY_ID
andYOUR_MINIO_SECRET_ACCESS_KEY
values were copied from the MinIO console.For the Worker Filter, a worker with access to the MinIO storage bucket is needed. For this tutorial, the minio-worker was deployed locally using Docker to access the MinIO server, which is running in a separate containter.
Select the worker tagged with
"type" = ["minio", "openssh"]
, which provides access to MinIO. The following filter will select this worker:Lastly, check the box next to Disable credential rotation.
- Name:
Click Save.
If there are errors, double-check the Endpoint URL, Bucket name, Access keys, and Worker filter. If there are still problems, check the Access Key Policy for the MinIO bucket, and ensure the bucket name matches the Resource ARN.
Create an SSH target
To finish setting up recordings, create a target for the openssh-server
container.
Navigate to the ssh-recording-org scope, and select the
ssh-recording-project
project.Select the Targets page and click New.
Fill out the New Target form. Select a Type of SSH.
- Name:
openssh-target
- Type:
SSH
- Target Address:
127.0.0.1
- Default Port:
2222
- Maximum Connections:
-1
- Aliases:
openssh
Click Save.
- Name:
Click the Workers tab. Click Edit worker filter for Egress workers.
When you create a target, you can specify an egress worker filter. The filter tells Boundary which worker is deployed in the same network as the target. An ingress worker is not used in this example.
Recall the tags associated with the
minio-worker
, which provides access to theopenssh-server
container:The tags for this worker are:
"type" = ["minio", "openssh"]
An appropriate filter to select this worker is:
Paste in the following worker filter:
"minio" in "/tags/type"
Click Save.
Next, add credential injection for the target.
Select the Injected Application Credentials tab for
openssh-target
.Click +Add Injected Application Credentials.
Check the box next to the credential named openssh-creds, then click Add Injected Application Credentials.
Enable session recording
To enable session recording for the openssh-target
:
Navigate back to the
openssh-target
Details page.Under Session Recording, click Enable recording.
On the Enable Session Recording for Target page, toggle the switch next to Record sessions for this target.
For the Storage buckets, select the
ssh-test-bucket
.Click Save.
Under the openssh-target
Details page, the ssh-test-bucket
should
now be listed under Session Recording.
Record a session
Now you are ready to test session recording for the openssh-target
.
To log into Boundary using the Desktop app, you must gather the BOUNDARY_ADDR
values from the HCP Boundary Admin Console.
Check the value of BOUNDARY_ADDR
in the terminal session where Terraform was
applied.
Open the Boundary Desktop app.
Enter the Boundary cluster URL (for example,
https://d2a6e010-ba05-431a-b7f2-5cbc4e1e9f06.boundary.hashicorp.cloud
) and
click Submit.
Authenticate using your HCP Boundary admin credentials.
On the Targets page, notice the target details for openssh-target
.
Click Connect to initiate a session.
The Successfully Connected page displays the target ID (Target Connection details) and Proxy URL.
To start a session, click on the Shell tab, or open your terminal or SSH client of choice. You can start a session using SSH and the Proxy URL from the Boundary Desktop app.
If you use your own client, connect on 127.0.0.1
and provide the proxy port using the -p
option. Enter yes
when prompted to establish a connection.
Type some simple commands you will recognize when playing back the recording, like pwd
or whoami
. When finished, you can close the connection to the server by entering exit
, or you can cancel the session directly from the Boundary Desktop app under the Sessions view.
View the recording
You can view a list of all recorded sessions, or if you know the ID of a specific recorded session, you can find any channels associated with that recording.
To play back a session, open the Admin Console Web UI, and re-authenticate as the admin user if necessary.
From the global
scope, navigate to the Session Recordings page.
The following details are listed for each recording:
- Time
- Status
- User
- Target
- Duration
Click View next to the openssh-target
recording.
Within the Session Playback page, click Play for Channel 1.
After the recording loads, click the Play button to start playback for Channel 1.
Note the Channel details on the right, which display the duration and bytes up / bytes down.
Validate the recording (optional)
A session recording represents a directory structure of files in an external object store that together are the recording of a single session between a user and a target.
To validate a session recording and download it, you can use the CLI.
First, set the BOUNDARY_ADDR
and BOUNDARY_AUTH_METHOD_ID
environment variables in your shell session. Replace the example address and auth method ID with the values for your cluster.
Log into the CLI as the admin user, providing the admin login name and admin password when prompted.
Verify that the recording exists.
Read the recording's details.
Note the Channel Recordings
, labeled Mime Types: application/x-asciicast
(ID chr_Mgkj7JkB7e
in this example). Downloading this recording would produces a .cast
file, which can be played back locally using asciinema.
If you want to download this file, execute the following command:
BSR files
The Boundary Session Recording (BSR) file defines a hierarchical directory structure of files and a binary file format. It contains all the data transmitted between a user and a target during a single session.
Boundary stores the recordings within the external storage bucket as BSR files.
A BSR connections directory contains a summary of connections, as well as inbound and outbound requests. If you use a multiplexed protocol, there are subdirectories for the channels.
The asciicast format is well suited for the playback of interactive shell activity, but some aspects of the recording cannot be translated into asciicast. For example, if an SSH session uses the RemoteCommand
option, or is used to exec
a command, the command is not displayed in the asciicast. The output of the command may be displayed, though.
If you use SSH for something other than an interactive shell, such as for file transfer, X11 forwarding, or port forwarding, Boundary does not attempt to create an asciicast.
In all cases, the SSH session is still recorded in the BSR file and you can view the BSR file in the external storage bucket.
Cleanup and teardown
Stop the openssh-server, minio, and boundary-enterprise worker containers.
Open the Docker Desktop app and locate the containers used in this tutorial. These include:
- openssh-server
- minio
- hashicorp/boundary-enterprise
Click the trash icon next to each, and confirm to stop and remove them.
Alternatively, open a shell session and destroy each container individually:
Destroy the Boundary resources.
Note
Recall that Boundary storage bucket lifecycle management is still under development. In order to prevent unintentional loss of session recordings, orgs that contain storage buckets with recordings cannot currently be deleted. When destroying your Boundary resources, you will receive an error if you attempt to delete the storage bucket, or the scopes that contain the bucket, without deleting the recordings from the bucket first.
From the Admin Console Web UI, destroy the following resources:
- Session recordings
- Target
- Storage bucket
- Org (also deletes credential store)
- Boundary worker
Orgs cannot be deleted until the session recordings they contain are deleted.
To delete a recording, it must have a storage policy applied that allows deletion. By default, a recording has its retention policy set to
Forever
.Create a new session recording retention policy that allows for deletion.
Select the ssh-recording-org org from the top navigation bar.
Select Storage Policies from the left navigation panel. Click Create a new storage policy.
Fill in the form details:
- Name:
do-not-retain
- Description:
delete after 1 day
- Retention Policy:
Do not protect, allow deletion at any time
- Deletion Policy:
Custom
- Delete after:
1
days
Click Save.
- Name:
Add the storage policy to the org.
From the left navigation panel, select Org Settings.
Click Add Storage Policy.
From the Add Storage Policy page, select the do-not-retain policy. Click Save.
Re-apply the storage policy to the session recording.
Navigate back to the Global scope and select Session Recordings from the left navigation panel.
Open a recording by clicking on View next to the recording. Under the Manage dropdown, select Re-apply storage policy.
Delete the session recording.
Click the Manage dropdown again, then select Delete recording and confirm the operation by clicking OK.
Follow this process for any remaining recordings.
Delete the target resource, or disable session recording for it. A storage bucket cannot be deleted if a target is configured to save sessions to it.
Storage buckets cannot currently be cleaned up within the UI. Switch to the CLI tab to delete the storage bucket, and then use the UI to continue cleanup.
Continue deleting the other resources.
Unset the environment variables used in any active terminal windows for this tutorial.