HashiConf 2024 Now streaming live from Boston! Attend for free
Vault
Vault Home

Overview

The Vault 1.17.x upgrade guide contains information on deprecations, important or breaking changes, and remediation recommendations for anyone upgrading from Vault 1.16. Please read carefully.

Important changes

Allowed audit headers now have unremovable defaults

The config auditing API endpoint tells Vault to log incoming request headers (when present) in the audit log.

Previously, Vault only logged headers that were explicitly configured for logging. As of version 1.17, Vault automatically logs a predefined set of default headers. By default, the header values are not HMAC encrypted. You must explicitly configure the HMAC setting for each of the default headers if required.

Refer to the audit request headers documentation for more information.

PKI sign-intermediate now truncates notAfter field to signing issuer

Prior to 1.17.x, Vault allowed the calculated sign-intermediate notAfter field to go beyond the signing issuer notAfter field. The extended value lead to a CA chain that would not validate properly. As of 1.17.x, Vault truncates the intermediary notAfter value to the signing issuer notAfter if the calculated field is greater.

How to opt out

You can use the new enforce_leaf_not_after_behavior flag on the sign-intermediate API along with the leaf_not_after_behavior flag for the signing issuer to opt out of the truncating behavior.

When you set enforce_leaf_not_after_behavior to true, the sign-intermediate API uses the leaf_not_after_behavior value configured for the signing issuer to control truncation the behavior. Setting the issuer leaf_not_after_behavior field to permit and enforce_leaf_not_after_behavior to true restores the legacy behavior.

Request limiter deprecation

Vault 1.16.0 included an experimental request limiter. The limiter was disabled by default. Further testing indicated that an alternative approach improves performance and reduces risk for many workloads. Vault 1.17.0 includes a new adaptive overload protection feature that prevents outages when Vault is overwhelmed by write requests. Adaptive overload protection is a beta feature in 1.17.0 and is disabled by default.

The beta request limiter will be removed from Vault entirely in a later release.

JWT auth login requires bound audiences on the role

The bound_audiences parameter of "jwt" roles is mandatory if the JWT contains an audience (which is more often than not the case), and must match at least one of the JWT's associated aud claims. The aud claim claim can be a single string or a list of strings as per RFC 7519 Section 4.1.3. If the JWT's aud claim is not set, then the role's bound_audiences parameter is not required.

Users may not be able to log into Vault if the JWT role is configured incorrectly. For additional details, refer to the JWT auth method (API) documentation.

Activity Log Changes

Default Activity Log Querying Period

As of 1.17.9 and later, the field default_report_months can no longer be configured or read. Any previously set values will be ignored by the system.

Attempts to modify default_report_months through the /sys/internal/counters/config endpoint, will result in the following warning from Vault:


WARNING! The following warnings were returned from Vault:

* default_report_months is deprecated: defaulting to billing start time

The current_billing_period toggle for /sys/internal/counters/activity is also deprecated, as this will be set true by default.

Attempts to set current_billing_period will result in the following warning from Vault:


WARNING! The following warnings were returned from Vault:

* current_billing_period is deprecated; unless otherwise specified, all requests will default to the current billing period

Auto-rolled billing start date

As of 1.17.3 and later, the billing start date (license start date if not configured) rolls over to the latest billing year at the end of the last cycle.

This means that the billing start dates from earlier years will get updated to current calendar year, unless the date has not yet occured this year, in which case it will remain in the last calendar year. The billing start date will adjust accordingly for leap years.

The default billing start date used in queries and license utilization reporting will be automatically adjusted to the latest billing year. Past year billing information will still be retained.

For example, assume a customer has signed a deal for 3 years with a license start date of Dec 1, 2023. After upgrading to the version with this change, the billing start date auto-rolls to the most recent year.

  • If the upgrade happens on Sept 1, 2024, the billing start date will continue to be Dec 1, 2023. Once we reach Dec 1, 2024, the billing start date will automatically roll to Dec 1, 2024.
  • If the upgrade happens on Dec 15, 2024, the billing start date should automatically roll to Dec 1, 2024 after upgrade.

Docker image no longer contains curl

As of 1.17.3 and later, the curl binary is no longer included in the published Docker container images for Vault and Vault Enterprise. If your workflow depends on curl being available in the container, consider one of the following strategies:

Create a wrapper container image

Use the HashiCorp image as a base image to create a new container image with curl installed.

FROM hashicorp/vault-enterprise
RUN apk add curl

NOTE: While this is the preferred option it will require managing your own registry and rebuilding new images.

Install it at runtime dynamically

When running the image as root (not recommended), you can install it at runtime dynamically by using the apk package manager:

docker exec <CONTAINER-ID> apk add curl

kubectl exec -ti <NAME> -- apk add curl

When running the image as non-root without privilege escalation (recommended) you can use existing tools to install a static binary of curl into the vault users home directory:

docker exec <CONTAINER-ID> wget https://github.com/moparisthebest/static-curl/releases/latest/download/curl-amd64 -O /home/vault/curl && chmod +x /home/vault/curl

kubectl exec -ti <NAME> -- wget https://github.com/moparisthebest/static-curl/releases/latest/download/curl-amd64 -O /home/vault/curl && chmod +x /home/vault/curl

NOTE: When using this option you'll want to verify that the static binary comes from a trusted source.

Product usage reporting

As of 1.17.9, Vault will collect anonymous product usage metrics for HashiCorp. This information will be collected alongside client activity data, and will be sent automatically if automated reporting is configured, or added to manual reports if manual reporting is preferred.

See the main page for Vault product usage metrics reporting for more details, and information about opt-out.

Known issues and workarounds

Client tokens and token accessors audited in plaintext

Affected versions

  • 1.16.7, 1.16.8, 1.17.3, 1.17.4

Issue

In versions 1.16.7, 1.16.8, 1.17.3, and 1.17.4 audit logs may contain non-hmac’d values for client_token and accessor data in the response portion. A fix has been created and is released in 1.16.9 and 1.17.5.

Workaround

It is recommended to avoid affected versions when upgrading. If you are on these versions and using the audit logging feature please upgrade promptly to 1.16.9 or 1.17.5.

PKI OCSP GET requests can return HTTP redirect responses

If a base64 encoded OCSP request contains consecutive '/' characters, the GET request will return a 301 permanent redirect response. If the redirection is followed, the request will not decode as it will not be a properly base64 encoded request.

As a workaround, OCSP POST requests can be used which are unaffected.

Impacted versions

Affects all current versions of 1.12.x, 1.13.x, 1.14.x, 1.15.x, 1.16.x, 1.17.x.

Vault Agent and Vault Proxy consume an excessive amount of CPU

Vault Agent and Vault Proxy consume an exessive amount of CPU during normal operation in 1.17.0.

We recommend waiting until 1.17.1 to upgrade Agent and Proxy.

Impacted versions

Affects 1.17.0.

Potential DoS when using the deny_unauthorized proxy protocol behavior for a TCP listener

Affected versions

Community Edition (CE)
  • 1.10.x - 1.15.x
  • 1.17.1
Enterprise
  • 1.10.x+ent - 1.15.11+ent
  • 1.16.5+ent
  • 1.17.1+ent

Issue

Vault TCP listeners configured with the deny_unauthorized proxy_protocol_behavior close if they receive a request from an untrusted upstream connection. As a result, Vault no longer responds to any request received through that listener.

Being able to force-close listeners with intentionally untrusted connections leaves the Vault API vulnerable to a denial-of-service (DoS) attacks, which could cause the API to become unavailable on that node.

The vulnerability is addressed in Vault 1.15.12+ent, 1.16.6+ent, 1.17.2, 1.17.2+ent and later.

Workaround

Do not configure a proxy_protocol_behavior with the deny_unauthorized value.

Input data on Transit Generate CMAC Response

CMAC support in the Transit engine has a known issue that causes it to return the original input along with the computed CMAC value in the CMAC field of the response.

We recommend waiting until version 1.17.2+ent for using this feature.

Impacted versions

Affects 1.17.0+ent and1.17.1+ent.

Deleting an entity-aliases does not remove it from the in-memory database on standby nodes

Affected versions

Vault Community Edition
  • 1.16.3
  • 1.17.0 - 1.17.2
Enterprise
  • 1.16.3+ent - 1.16.6+ent
  • 1.17.0+ent - 1.17.2+ent

Issue

Entity-aliases deleted from Vault are not removed from the in-memory database on standby nodes within a cluster. As a result, API requests to create a new entity-alias with the same mount accessor and name sent to a standby node will fail.

This bug is fixed in Vault 1.16.7+ent, 1.17.3, 1.17.3+ent and later.

Duplicate identity groups created when concurrent requests sent to the primary and PR secondary cluster

Affected versions

  • All Enterprise versions from 0.7.0 up

Issue

Duplicate identity groups are groups that have the same name, and should not be possible in vault. However, there is a race condition in which duplicate groups are possible. If more than 1 create group request is sent at the same time to a Primary Cluster and the PR secondary cluster, it can result in group duplicates.

Workaround

All create group requests should be sent to the primary cluster.

Manual entity merges sent to a PR secondary cluster are not persisted to storage

Affected versions

  • Vault Enterprise v0.7.0+

Issue

If you send a manual entity merge to a PR secondary cluster, the change is temporarily reflected on the active node of the PR secondary cluster, but Vault does not save the change to persistent storage.

Workaround

Always send manual entity merge requests to the primary cluster.

AWS Auth Role configuration requires an external_id

Affected Versions

  • 1.17.0 - 1.17.3

Issue

You must set the external_id parameter during role configuration, or the Vault AWS authentication plugin returns a validation error.

Workaround

To avoid the error during configuration:

  1. Set the external_id parameter when configuring AWS authentication plugin with a valid ID or any string longer than two characters.
  2. Configure any desired roles.
  3. If you used an arbitrary string, remove the external ID.

Cached activation flags for secrets sync on follower nodes are not updated

Affected versions

  • 1.16.0 - 1.16.2
  • 1.17.0 - 1.17.5

Issue

Vault 1.16 introduced secrets sync with a one-time flag required to activate the feature before use. Writing the activation flag to enable secrets sync is forwarded to leader nodes for storage and distributed to follower nodes, but the in-memory cache for this flag is not updated on the followers.

This prevents any secrets sync endpoints (those starting with sys/sync/) from being usable on follower nodes in a cluster.

Workaround

The cache is force-updated on all nodes when the leader node steps down and the cluster promotes a new leader. First, activate the secrets sync feature as described in the documentation. Then, have the leader node step down.

Duplicate HSM Keys

Affected Versions

  • All versions that support migration from Shamir to HSM-backed unseal in HSM-HA configurations.

Issue

During a migration from Shamir to an HSM-backed unseal configuration with HSM - High Availability (HA), duplicate HSM keys may be created.

  • Unseal failures: Nodes may fail to unseal after a restart, with errors such as CKR_DATA_INVALID.
  • Duplicate HSM keys: These may be created, resulting in intermittent KV read failures with errors such as CKR_SIGNATURE_INVALID and CKR_KEY_HANDLE_INVALID. These issues can occur even after a seal migration to HSM that initially appeared successful. The root cause is under investigation, with potential links to key handling during HA configuration or migration processes.

Workaround

As a workaround, always run Vault with generate_key = false, creating the required keys within the HSM manually during the setup process.

Edit this page on GitHub