14 minutes
New all you can eat buffet in town, or how I learned to love Bao with Tofu
In the beginning, there was Hashicorp.
In the early days of DevOps, Hashicorp wrote the tools everyone used to get work done. Vagrant let you have a local, tight loop of building local envs to run tests on. Packer let you build images as code to deploy to VMware, Docker, Hyper-V, and many more. Terraform was Infra-As-Code for all major cloud providers.
Then along came services to deploy onto. Consul, Vault, Nomad. All were, and are still great in their own areas, and solve real world problems.
Sometime back in August 2023 however, something went a bit astray. Hashicorp, kings of open source DevOps tools used by everyone started to move their toolsets to the Business Source License.1
I only bring this up in context, as I wanted to check on the progress of OpenBao, the community’s answer to Hashicorp Vault. It was forked off from the last Mozilla Public License version of Vault and is moving quickly to full release, providing all of the open source Vault functionality once again under an open source license.
I run a pretty complicated configuration within Vault, multiple sources of authentication, group matching the different auth sources, intermediate PKI, ssh certificates, JWT auth for CI/CD. Now that OpenBao was releases alphas, and Helm charts I was keen to try it out and see how it was getting along.
The one caveat with OpenBao is there is currently no UI. So any configuration I wanted to do had to be 100% via the CLI. No cheating and falling back to the GUI to tweak a lease length, or TTL. Some time back I had written some terraform to whip vault into shape, so I figured I would dust that off and try it on OpenBao. Also, why not use OpenTofu as well, which is the open-source splinter off of Terraform.
If you don’t care about the license, and want to use Hashicorp Vault, and Hashicorp Terraform these same instructions will work for you, aside from the specific installation instructions.
This doc assumes familarity with Terraform/OpenTofu, but should be a good source for some in-depth configuration of Vault/OpenBao.
Installing OpenTofu
I use a Mac, so this is as easy as:
brew install opentofu
Installing OpenBao
OpenBao like Vault has many different methods of deployment, from local binary running in dev mode, to highly available auto-unlocking in Kubernetes. For the purposes of this document, a local installation running with server dev is enough to test out.
brew install openbao
bao server -dev
You will see a lot of output, and at the end are the connection, and authorisation string you need to keep going:
WARNING! dev mode is enabled! In this mode, Vault runs entirely in-memory
and starts unsealed with a single unseal key. The root token is already
authenticated to the CLI, so you can immediately begin using Vault.
You may need to set the following environment variables:
$ export BAO_ADDR='http://127.0.0.1:8200'
The unseal key and root token are displayed below in case you want to
seal/unseal the Vault or re-authenticate.
Unseal Key: V4ZGmf60SvRz8Nr/XRfniewW7g4XNE+DgsSnrzJ5yRI=
Root Token: s.bjNM4rpvRjA36o7MgLjBgiCm
Development mode should NOT be used in production installations!
If this is a highly secure prod environment, figure out where to securely store those credentials. For our test/dev setup here, just set them as environment variables in your shell:
export BAO_ADDR='http://127.0.0.1:8200'
export BAO_TOKEN=s.bjNM4rpvRjA36o7MgLjBgiCm
export VAULT_TOKEN=s.bjNM4rpvRjA36o7MgLjBgiCm
Setup terraform
Initialise the providers
Create a file named provider.tf with the following contents:
provider "vault" {
address = "http://127.0.0.1:8200"
}
And run init to fetch the provider files/setup OpenTofu
tofu init
You should see the following:
Initializing the backend...
Initializing provider plugins...
- Finding latest version of hashicorp/vault...
- Installing hashicorp/vault v4.4.0...
- Installed hashicorp/vault v4.4.0 (signed, key ID 0C0AF313E5FD9F80)
Providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://opentofu.org/docs/cli/plugins/signing/
OpenTofu has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that OpenTofu can guarantee to make the same selections by default when
you run "tofu init" in the future.
OpenTofu has been successfully initialized!
You may now begin working with OpenTofu. Try running "tofu plan" to see
any changes that are required for your infrastructure. All OpenTofu commands
should now work.
If you ever set or change modules or backend configuration for OpenTofu,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
We’re now ready to start writing some TF code!
setup variables
Now we need to tell OpenTofu about some variables we need to use later, like our LDAP bind password, and our OIDC secret. Best to leave these as environment vars in your shell, and have OpenTofu reference them rather than committing them into your code.
Create a file named variables.tf with the following content:
variable "ldap_system_bindpass" {
description = "LDAP bind password"
type = string
sensitive = true
}
variable "oidc_secret" {
description = "OIDC client secret"
type = string
sensitive = true
}
variable "ldap_admin_bindpass" {
description = "LDAP admin bind password"
type = string
sensitive = true
}
OpenTofu will look in your env for variables prefixed with TF_VAR_ to find them, so let’s set them in our env:
export TF_VAR_ldap_system_bindpass=MySecretSystemBindPass
export TF_VAR_oidc_secret=MyLongOIDCSecret
export TF_VAR_ldap_admin_bindpass=MyLDAPAdminPass
other files you may need to setup
My LDAP instance uses an internal root certificate, root.pem which I place in a folder, files.
ls files/
root.pem
I also have a lot of vault policy files, which I place in a folder named policies
ls policies/
app-ansible-test.json ca-internal-issuing-client-only.json role-jr-sysadmin.json
app-boundary.json ca-internal-issuing.json role-support.json
app-gitlab-dns-deploy.json default.json role-sysadmin.json
app-telemetry.json role-admin.json
I just find it easier to maintain these outside of the TF code, your mileage may vary.
Lastly, we have some LDIF templates used to manage temporary LDAP users:
cat ldif/create.ldif
dn: UID={{.Username}},OU=users,DC=my,DC=domain
changetype: add
objectClass: hostObject
objectClass: top
objectClass: shadowAccount
objectClass: posixAccount
objectClass: person
objectClass: inetOrgPerson
employeeType: sysadmins
mail: {{.Username}}@my.domain
gidNumber: 4000
cn: Vault
loginShell: /bin/bash
o: My Org.
uid: {{.Username}}
uidNumber: {{.IssueTimeSeconds}}
homeDirectory: /home/{{.Username}}
givenName: Vault
host: *
sn: User
userPassword: {{.Password}}
dn: UID={{.Username}},OU=users,DC=my,DC=domain
changetype: modify
replace: employeeType
employeeType: sysadmins
-
cat ldif/delete.ldif
dn: uid={{.Username}},ou=users,dc=my,dc=domain
changetype: delete
configure authentication
Onto our first bit of TF!
configure LDAP
In my setup, LDAP is considered a secondary access method in case of emergency as there’s no MFA capabilities here. As such, it’s limited to session defaulting to 1h, and a max of 8h.
In the file main.tf write the following:
resource "vault_ldap_auth_backend" "ldap" {
path = "ldap"
url = "ldap://ldap.my.domain"
userdn = "ou=users,dc=my,dc=domain"
userattr = "uid"
discoverdn = false
groupdn = "ou=roles,dc=my,dc=domain"
groupfilter = "(|(memberUid={{.Username}})(member={{.UserDN}})(uniqueMember={{.UserDN}}))"
groupattr = "memberOf"
binddn = "cn=openbao,ou=service_accounts,dc=my,dc=domain"
bindpass = var.bindpass
certificate = file("${path.module}/files/root.pem")
starttls = true
token_ttl = 3600
token_max_ttl = 28800
}
This tells OpenBao to connect to the LDAP server at ldap://ldap.my.domain using StartTLS validated by certificate found a files/root.pem.
It will bind with the DN cn=openbao,ou=service_accounts,dc=my,dc=domain and look for users in the userdn, and groups in the specific groupdn. Nothing too fancy
configure OIDC
This is what most users use, most of the time as it has strong MFA, and logs. This authenticates users to OpenBao for a period of 8 days.
Add the following to main.tf:
resource "vault_jwt_auth_backend" "oidc" {
path = "oidc"
type = "oidc"
oidc_discovery_url = "https://auth.my.domain"
oidc_client_id = "openbao"
oidc_client_secret = var.oidc_secret
default_role = "oidc_user"
tune = [
{
allowed_response_headers = []
audit_non_hmac_request_keys = []
audit_non_hmac_response_keys = []
default_lease_ttl = "192h"
listing_visibility = "unauth"
max_lease_ttl = "720h"
passthrough_request_headers = []
token_type = "default-service"
}]
}
Now create a role within the oidc backend:
resource "vault_jwt_auth_backend_role" "oidc_user" {
backend = vault_jwt_auth_backend.oidc.path
role_name = "oidc_user"
role_type = "oidc"
token_policies = ["default", ]
user_claim = "preferred_username"
oidc_scopes = ["profile", "groups", ]
groups_claim = "groups"
allowed_redirect_uris = ["http://localhost:8500/oidc/callback"]
}
configure group collation
Since our OIDC source points at LDAP, user A in OIDC is equal to user A in LDAP. We need to tell OpenBao this is the case. In this example we’ll use the group “sysadmins”.
Add more to main.tf, starting with:
resource "vault_identity_group" "ldap-sysadmins" {
name = "ldap-sysadmins"
type = "external"
}
This creates a place-holder in OpenBao for the group coming from LDAP. And the next bit creates the place-holder for OIDC groups.
resource "vault_identity_group" "oidc-sysadmins" {
name = "oidc-sysadmins"
type = "external"
}
Now to create the vault specific sysadmins group, and glue them all together as one happy unit.
resource "vault_identity_group" "sysadmins" {
name = "sysadmins"
type = "internal"
policies = ["role/sysadmin", "ca/internal-issuing", "ca/internal-issuing-client-only", "default"]
member_group_ids = [vault_identity_group.ldap-sysadmins.id, vault_identity_group.oidc-sysadmins.id]
}
resource "vault_identity_group_alias" "ldap-sysadmins" {
name = "sysadmins"
mount_accessor = vault_ldap_auth_backend.ldap.accessor
canonical_id = vault_identity_group.ldap-sysadmins.id
}
resource "vault_identity_group_alias" "oidc-sysadmins" {
name = "sysadmins"
mount_accessor = vault_jwt_auth_backend.oidc.accessor
canonical_id = vault_identity_group.oidc-sysadmins.id
}
Now when a member of the group sysadmins logs in to OpenBAO via LDAP, or OIDC they will be granted the same ACLs and policies no matter the method.
add some policies
Now let’s add in the actual policies that get applied to users. Remember our folder of policy files ? Lets use TF to apply them to OpenBao:
resource "vault_policy" "default" {
name = "default"
policy = file("${path.module}/policies/default.json")
}
resource "vault_policy" "role-admin" {
name = "role/admin"
policy = file("${path.module}/policies/role-admin.json")
}
resource "vault_policy" "role-jr-sysadmin" {
name = "role/jr-sysadmin"
policy = file("${path.module}/policies/role-jr-sysadmin.json")
}
resource "vault_policy" "role-support" {
name = "role/support"
policy = file("${path.module}/policies/role-support.json")
}
resource "vault_policy" "role-sysadmin" {
name = "role/sysadmin"
policy = file("${path.module}/policies/role-sysadmin.json")
}
resource "vault_policy" "ca-internal-issuing" {
name = "ca/internal-issuing"
policy = file("${path.module}/policies/ca-internal-issuing.json")
}
resource "vault_policy" "ca-internal-issuing-client-only" {
name = "ca/internal-issuing-client-only"
policy = file("${path.module}/policies/ca-internal-issuing-client-only.json")
}
resource "vault_policy" "app-ansible-test" {
name = "app/ansible-test"
policy = file("${path.module}/policies/app-ansible-test.json")
}
resource "vault_policy" "app-boundary" {
name = "app/boundary"
policy = file("${path.module}/policies/app-boundary.json")
}
resource "vault_policy" "app-gitlab-dns-deploy" {
name = "app/gitlab-dns-deploy"
policy = file("${path.module}/policies/app-gitlab-dns-deploy.json")
}
resource "vault_policy" "app-telemetry" {
name = "app/telemetry"
policy = file("${path.module}/policies/app-telemetry.json")
}
configure JWT authentication
It would be great if our CI/CD pipeline could authenticate to OpenBao as well, which in the case of Gitlab it can!
Lets add in JWT based authentication to main.tf, making sure to specify the URL of your Gitlab instance:
resource "vault_jwt_auth_backend" "jwt" {
path = "jwt"
type = "jwt"
oidc_discovery_url = "https://gl.my.domain"
bound_issuer = "https://gl.my.domain"
tune = [
{
allowed_response_headers = []
listing_visibility = "unauth"
passthrough_request_headers = []
audit_non_hmac_request_keys = []
audit_non_hmac_response_keys = []
default_lease_ttl = "15m"
max_lease_ttl = "60m"
token_type = "default-service"
},
]
}
Now, for each pipeline you’ll want to configure a specific role. In that way you can limit (by OpenBao policy) the access any CI pipeline has. An example looks like:
resource "vault_jwt_auth_backend_role" "gitlab-dns-deploy" {
backend = vault_jwt_auth_backend.jwt.path
role_name = "gitlab-dns-deploy"
role_type = "jwt"
user_claim = "user_email"
bound_claims = {
"project_id" = "19"
"ref" = "master"
"ref_type" = "branch"
}
token_explicit_max_ttl = 60
token_policies = [
"app/gitlab-dns-deploy",
]
}
In this case, project ID 19, in the master branch can get a token and have access under the gitlab-dns-deploy policy.
configure PKI
I use an offline root CA, and set OpenBao up as an online intermediate.
To start off with, enable the PKI engine under the path int_pki:
resource "vault_mount" "int_pki" {
path = "int_pki"
type = "pki"
}
Now, we set OpenBao to create a key, and generate a CSR for our offline root to sign:
resource "vault_pki_secret_backend_intermediate_cert_request" "int_pki" {
depends_on = [vault_mount.int_pki]
backend = vault_mount.int_pki.path
type = "internal"
common_name = "My Org. intermediate"
key_type = "ec"
key_bits = 256
key_name = "myorg"
}
Some slightly bad trickery here, as we don’t have our signed certificate back from the offline root yet, but we still want to continue on configuring OpenBao.
Later we’ll have our signed certificate back, and placed into files/cert.pem and running OpenTofu again will install the signed certificate into OpenBao for us.
locals {
my_file = fileexists("${path.module}/files/cert.pem") ? file("${path.module}/files/cert.pem") : ""
}
output "intermediate_csr" {
value = vault_pki_secret_backend_intermediate_cert_request.intermediate_ca.csr
}
resource "vault_pki_secret_backend_intermediate_set_signed" "intermediate_ca" {
count = local.my_file != "" ? 1 : 0
backend = vault_mount.intermediate_ca.path
certificate = local.my_file
}
Our root ca is the same one that signed our LDAP certificate, so let’s tell OpenBao about it signing the intermediate.
resource "vault_pki_secret_backend_config_ca" "intermediate_ca" {
depends_on = [vault_mount.intermediate_ca]
backend = vault_mount.intermediate_ca.path
pem_bundle = file("${path.module}/files/root.pem")
}
Finally, let’s add a role to actually issue certificates with this intermediate:
resource "vault_pki_secret_backend_role" "my-domain-servers" {
backend = vault_mount.intermediate_ca.path
name = "my-domain-servers"
allow_bare_domains = true
allow_glob_domains = true
allow_subdomains = true
enforce_hostnames = false
key_bits = 256
key_type = "ec"
allowed_domains = [
"my.domain",
]
allowed_other_sans = [
"*",
]
key_usage = [
"DigitalSignature",
"KeyAgreement",
"KeyEncipherment",
]
max_ttl = "315360000"
}
configure SSH certificates
First we initialise the ssh certificate engine:
resource "vault_mount" "ssh-client-signer" {
type = "ssh"
path = "ssh-client-signer"
}
Now we generate the CA key:
resource "vault_ssh_secret_backend_ca" "foo" {
backend = vault_mount.ssh-client-signer.path
generate_signing_key = true
}
And lastly, configure a role used to issue certificates:
resource "vault_ssh_secret_backend_role" "admin" {
name = "admin"
backend = "${vault_mount.ssh-client-signer.path}"
key_type = "ca"
allow_user_certificates = true
allow_user_key_ids = true
default_user = "ubuntu"
allowed_extensions = "permit-pty,permit-port-forwarding,permit-agent-forwarding,permit-x11-forwarding,permit-open"
allowed_users = "*"
default_extensions = {
"permit-agent-forwarding" = ""
"permit-port-forwarding" = ""
"permit-pty" = ""
}
}
configure LDAP secret management
OpenBao can manage LDAP users, and passwords as well, neat!
Instantiate the LDAP secrets engine first:
resource "vault_ldap_secret_backend" "config" {
path = "ldap"
binddn = "cn=admin,dc=my,dc=domain"
bindpass = var.ldap_admin_bindpass
url = "ldaps://ldap.my.domain"
insecure_tls = "true"
}
Now we tell OpenBao about a static user, and it will manage password rotation
resource "vault_ldap_secret_backend_static_role" "role" {
mount = vault_ldap_secret_backend.config.path
username = "bao-user"
dn = "uid=bao-user,ou=users,dc=my,dc=domain"
role_name = "static-role"
rotation_period = 900
}
We can also tell OpenBao how to manage users, and it will create time limited accounts, deleting them after the time is up:
resource "vault_ldap_secret_backend_dynamic_role" "role" {
mount = vault_ldap_secret_backend.config.path
role_name = "dynamic-role"
default_ttl = 900
creation_ldif = file("${path.module}/ldif/create.ldif")
deletion_ldif = file("${path.module}/ldif/delete.ldif")
}
Notice the reference to the LDIF files we created earlier.
Setup done
Now we’ve fully setup OpenBao it can issue credentials, certificates, and much more.
Using OpenBao
Login with LDAP
Let’s say we have a sysadmin user, who wants to authenticate to OpenBao:
bao login -method=ldap username=myuser
You’ll be prompted for your password
Password (will be hidden):
And once successful you will see a message indicating the time your token is valid for, and the policies that apply to your user:
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.
Key Value
--- -----
token s.s4NgYK7TApvqfoWmroxk6Z9U
token_accessor 0nLcpX4ksQGw7DEgm76w3nXr
token_duration 1h
token_renewable true
token_policies ["default"]
identity_policies ["ca/internal-issuing" "ca/internal-issuing-client-only" "default" "role/sysadmin"]
policies ["ca/internal-issuing" "ca/internal-issuing-client-only" "default" "role/sysadmin"]
token_meta_username myuser
Notice the token_duration of 1h here, like we mentioned earlier.
Login with OIDC
To use OIDC instead run:
bao login -method=oidc port=8500
Complete the login via your OIDC provider. Launching browser to:
https://auth.my.domain/api/oidc/authorization?client_id=openbao&code_challenge=..
Waiting for OIDC authentication to complete...
Once you successfully authenticate to your OpenID provider you should see:
WARNING! The BAO_TOKEN environment variable is set! The value of this variable
will take precedence; if this is unwanted please unset BAO_TOKEN or update its
value accordingly.
Key Value
--- -----
token s.Lvunk1sdTRp6asi1GIJEgMc2
token_accessor JRydXDPUqZ4vE5ZjdVqTtNop
token_duration 192h
token_renewable true
token_policies ["default"]
identity_policies ["ca/internal-issuing" "ca/internal-issuing-client-only" "default" "role/sysadmin"]
policies ["ca/internal-issuing" "ca/internal-issuing-client-only" "default" "role/sysadmin"]
token_meta_role oidc_user
Notice the token_duration of 192h here, rather than the 1h we got via LDAP. If you are using the same user for both methods, meake sure you get back the same policy list as well.
Now that we have authenticated to vault, we can use it to issue stuff.
issue a server certificate
Using the server certificate role we had created:
bao write intermediate_ca/issue/my-domain-servers common_name=test.my.domain ttl=1h
You should get a bunch of stuff back, including a certificate, private key, full CA chain and more like so:
Key Value
--- -----
ca_chain [-----BEGIN CERTIFICATE-----
GTdFyk0wHwYDVR0jBBgwFoAUBrWxBJ4UFsvdHjZPQp4fcExMNpkwEgYDVR0TAQH/
...
-----END CERTIFICATE----- -----BEGIN CERTIFICATE-----
MIIB2zCCAYKgAwIBAgIBATAKBggqhkjOPQQDAjBEMRowGAYDVQQDDBFTbHVzaCB5
...
-----END CERTIFICATE-----]
certificate -----BEGIN CERTIFICATE-----
MIICXzCCAgSgAwIBAgIUEvoU7En2wDMZKNakHWtQpCA2eq0wCgYIKoZIzj0EAwIw
...
-----END CERTIFICATE-----
expiration 1724571747
issuing_ca -----BEGIN CERTIFICATE-----
pnRYVdjhn89KE6ABXQE74lgGo2YwZDAdBgNVHQ4EFgQUNy1xLPJwY8+XCXoq7P0Q
...
-----END CERTIFICATE-----
private_key -----BEGIN EC PRIVATE KEY-----
MHcCAQEEICjy4HuGgQbtXF54iYGjSfmekJ25Em2rLBhqB2NmIpP+oAoGCCqGSM49
...
-----END EC PRIVATE KEY-----
private_key_type ec
serial_number 12:fa:14:ec:49:f6:c0:33:19:28:d6:a4:1d:6b:50:a4:20:36:7a:ad
Issue an SSH certificate
Let’s say you wanted to sign your own key, ~/.ssh/id_ed25519.pub:
bao write ssh-client-signer/sign/admin public_key=@/Users/dene/.ssh/id_ed25519.pub
This should return a signed key like so:
Key Value
--- -----
serial_number 757b9d0374be6a9c
signed_key ssh-ed25519-cert-v01@openssh.com AAAAIHNzaC1lZDI1NTE5LWNlcnQtdjAxQG9wZW5zc2guY29tAAAAIB+QXseRopQ0/NO...
Create a temporary user in LDAP
This will create a user in LDAP, with a OpenBao geneated username + password, with a lifetime of 15mins:
bao read ldap/creds/dynamic-role
You will get the username, and password back if successful:
Key Value
--- -----
lease_id ldap/creds/dynamic-role/fLPUTQYawzlDN4RRm5yZw5CE
lease_duration 15m
lease_renewable true
distinguished_names [UID=v_root_dynamic-role_iNnL41SSml_1724568483,OU=users,DC=my,DC=domain UID=v_root_dynamic-role_iNnL41SSml_1724568483,OU=users,DC=my,DC=domain]
password ZtyndY8Mx...
username v_root_dynamic-role_iNnL41SSml_1724568483
Rotate credentials for a static LDAP user
And a similar outcome, there exists a static user already in LDAP, and OpenBao manages the password. Let’s cycle it, and return the newly generated password:
bao read ldap/static-cred/static-role
You will get the new password returned, including some other useful details:
Key Value
--- -----
dn uid=bao-user,ou=users,dc=my,dc=domain
last_password Y98eifOCtHTvMofmhenc3V7uTHYrBiFBLrQqpuppRNB9RVEi2JfMVwH0TmTfSkTh
last_vault_rotation 2024-08-25T18:38:52.295677+12:00
password yUtlximIUdt8IpIgrAw07YVkNMgQo6gmnXP3TT7X3ZhdPhIHXY9Hfm2kn3snL6Lc
rotation_period 15m
ttl 3m8s
username bao-user
Hopefully that gives you a taste of what can be accomplished with OpenBao, and what you need to write in OpenTofu to get there