HashiCorp Vault Deployment

Introduction

This document provides a comprehensive architectural overview of the system, using a number of different architectural views to depict different aspects of the system. It is intended to capture and convey the significant architectural decisions that have been made on the system.

VAULT Deployment Architecture

Deployment

In each DC, Vault will be deployed with the following setup:

1. integrated storage (Raft backend)
2. cluster with 1 active node, 2 standby nodes
3. auto-unseal using transit secrete engine

Deployment

Environment

Host	IP	Node
pln-n1-eso4sap1	204.104.46.242	Cluster Node
pln-n1-eso4sap2	204.104.46.243	Cluster Node
pln-n1-eso4sap3	204.104.46.244	Cluster Node
pln-n1-eso4sap4	204.104.248.31	Auto-unseal Node

Vault Installation and configuration

Run below command to install vault to the four servers detailed above, and enable vault server as a service

1. Run "yum install vault"
2. Run "systemctl enable vault"
3. Put "p4s_vault_cert.pem", "p4s_vault_key.pem" and "vault_ca.pem" to directory /opt/vault/tls
4. edit configure file in /etc/vault.d/vault/hcl in three servers

ui = true
disable_mlock = true
storage "raft" {
path = "/var/vault/data"
node_id = "<%= @node_name%>"
}
listener "tcp" {
address = "0.0.0.0:8200"
cluster_address = "0.0.0.0:8201"
tls_cert_file = "/opt/vault/tls/p4s_vault_cert.pem"
tls_key_file = "/opt/vault/tls/p4s_vault_key.pem"
tls_disable_client_certs = true
}
api_addr = "https://<%= @node_ip%>:8200"
cluster_addr = "https://<%= @node_ip%>:8201"

<%= @node_name%> is node name in cluster, you can pick up any name here, here used host name of server.
<%= @node_ip%> is server IP

Vault server initiallization and cluster configuraion

Go to Server 1: pln-n1-eso4sap1 and run commands

• systemctl start vault
• export VAULT_ADDR="https://204.104.46.242:8200"
• export VAULT_SKIP_VERIFY=true
• vault operator init -key-shares=3 -key-threshold=2
  This command will generate three unseal keys and a root token. Please keep these values in secret place, and will be used later**
• export VAULT_TOKEN=s.vlyrBOoTjpBfU6K1MR2VYeA5(Replace token with gerneated by up step)
• vault secrets enable kv-v2

Go to Server 2: pln-n1-eso4sap2 and run commands

• systemctl start vault
• export VAULT_ADDR="https://204.104.46.243:8200"
• export VAULT_SKIP_VERIFY=true
• vault operator raft join -leader-ca-cert=@/opt/vault/tls/vault_ca.pem https://204.104.46.242:8200
• vault operator unseal zMLoNgWylWjsAWUQa6ZHbb43zvLhs3WBNhvk3Enpp9Q=(Replace unseal key with gerneated by up step)
• vault operator unseal Xlu+UajvWu9RV6sziFh0D6q/xHUeXEYYH0UWdkAtejMI(Replace unseal key with gerneated by up step)

Go to Server 3: pln-n1-eso4sap3 and run commands

• systemctl start vault
• export VAULT_ADDR="https://204.104.46.244:8200"
• export VAULT_SKIP_VERIFY=true
• vault operator raft join -leader-ca-cert=@/opt/vault/tls/vault_ca.pem https://204.104.46.242:8200
• vault operator unseal zMLoNgWylWjsAWUQa6ZHbb43zvLhs3WBNhvk3Enpp9Q=(Replace unseal key with gerneated by up step)
• vault operator unseal Xlu+UajvWu9RV6sziFh0D6q/xHUeXEYYH0UWdkAtejMI(Replace unseal key with gerneated by up step)

Configure "Re-Join" Section

• edit configure file in /etc/vault.d/vault/hcl and update content to "storage" section for each servers
  pln-n1-eso4sap1

storage "raft" {
path = "/var/vault/data"
node_id = "pln-n1-eso4sap1"
retry_join {
leader_api_addr = "https://204.104.46.243:8200"
leader_ca_cert_file = "/opt/vault/tls/vault_ca.pem"
}
retry_join {
leader_api_addr = "https://204.104.46.244:8200"
leader_ca_cert_file = "/opt/vault/tls/vault_ca.pem"
}
}

pln-n1-eso4sap2

storage "raft" {
path = "/var/vault/data"
node_id = "pln-n1-eso4sap2"
retry_join {
leader_api_addr = "https://204.104.46.242:8200"
leader_ca_cert_file = "/opt/vault/tls/vault_ca.pem"
}
retry_join {
leader_api_addr = "https://204.104.46.244:8200"
leader_ca_cert_file = "/opt/vault/tls/vault_ca.pem"
}
}

pln-n1-eso4sap3.env01.mcloud.entsvcs.net

storage "raft" {
path = "/var/vault/data"
node_id = "pln-n1-eso4sap3"
retry_join {
leader_api_addr = "https://204.104.46.242:8200"
leader_ca_cert_file = "/opt/vault/tls/vault_ca.pem"
}
retry_join {
leader_api_addr = "https://204.104.46.243:8200"
leader_ca_cert_file = "/opt/vault/tls/vault_ca.pem"
}
}

Configure Auto-unseal

Run the following commands to initial the unseal server pln-n1-eso4sap4:

• systemctl start vault

• export VAULT_ADDR="https://204.104.248.31:8200"

• export VAULT_SKIP_VERIFY=true

• vault operator init -key-shares=3 -key-threshold=2
  This command will generate three unseal keys and a root token. Please keep these values in secret place, and will be used later

• export VAULT_TOKEN=s.vlyrBOoTjpBfU6K1MR2VYeA5(Replace token with gerneated by up step)

• vault secrets enable kv-v2

Configure Auto-unseal Key Provider

1. Enable audit log

vault audit enable file file_path=/var/log/vault/vault-audit.log

2. Enable transit secrete engine, this can also be configured via UI

vault secrets enable transit

3. Create an encryption key - autounseal, this can also be configured via UI

vault write -f transit/keys/autounseal

4. Create a policy (autounseal) which enable user to update the autounseal key

$ tee autounseal.hcl < path "transit/encrypt/autounseal" {
capabilities = [ "update" ]
}
path "transit/decrypt/autounseal" {
capabilities = [ "update" ]
}
EOF

vault policy write autounseal autounseal.hcl

5. Generate auto-unseal token

vault write auth/token/create policies=autounseal

the current lease ttl can be checked using the following command

vault read sys/auth/token/tune

this vault can be set using

vault write sys/auth/token/tune max_lease_ttl=9000h
vault write sys/auth/token/tune default_lease_ttl=9000h

*9000h for 1 year

token generated here will be passed to the unseal server.

Configure Auto-unseal Server

Auto-unseal can either be configured before server/cluster is initialed or after.

1. Configure auto-unseal in server's configuration file, for example, /etc/vault.d/vault/hcl**

auto-unseal configuration
ui = true
disable_mlock = true
storage "file" {
path = "/var/vault/data"
}
listener "tcp" {
address = "0.0.0.0:8200"
tls_cert_file = "/opt/vault/tls/p4s_vault_cert.pem"
tls_key_file = "/opt/vault/tls/p4s_vault_key.pem"
tls_disable_client_certs = true
}
seal "transit"`{
address = "https://204.104.248.31:8200"
token = "s.lqEgk3aB5rN5xyRmsKm7FnNV"
disable_renewal = "false"
key_name = "autounseal"
mount_path = "transit/"
tls_skip_verify = "true"
}

3. If server is already initialed:

• for each server in the Vault cluster, shut down Vault service

systemctl stop vault

• for each server in the Vault cluster, start Vault service and run unseal with migrate option

systemctl start vault
vault operator unseal -migrate.

This will migrate the previous unseal mechanism to transit key auto-unseal, and the previous unseal keys will be migrated as the recovery keys. The migration process can only be completed when all nodes in the Vault cluster are restarted and unsealed, otherwise all nodes will remain in seal status.