Governance & AssuranceExplainer
How do you move HSM keys without trusting the host that moves them?
A design note on migrating keys between HSMs when the host is untrusted: signed bundles, signed receipts, class-aware unwrap and two-phase audit-key destruction
- hsm
- key migration
- pkcs#11
- key management
- audit logging
- threat modelling
- shamir secret sharing
What is the short answer?
A migration between two hardware security modules (HSMs) is safe to the extent that the party moving the keys is treated as the least trusted component in the room. The design described here treats every file on disk as advisory until it has been checked against a key that lives inside an HSM. The source HSM signs a bundle that describes what is leaving. The destination HSM signs a receipt that describes what arrived. Source keys are destroyed only after that receipt verifies, its key set matches the manifest exactly, and the live source fingerprints match the receipt. Each step is recorded in an audit log whose records are authenticated inside the HSM.
This post is the design rationale for a command-line migration tool and its test harness. The interactive recording below is a real run of that harness on two disposable SoftHSM2 tokens, in both RSA signature profiles. In that run the harness passed all 31 steps on SoftHSM2 2.6.1, and 14 of the steps are attacks or injected faults that the tool has to refuse or report. The recording is not an independent audit and it shows nothing about production or vendor HSMs.
Try it out
Right click + Open to see the recording on a new window
Step through it with Next or the arrow keys. The Safeguards view lists each attack with the tool’s own message, and the Report view builds a single HTML file from the run. Nothing in the recording is executed live: the page replays output captured from the harness.
Why is moving a key between HSMs hard?
An HSM exists so that a key never leaves its boundary. A migration is a negotiated exception to that rule, and the exception has to be engineered. Six difficulties recur.
Most private keys are created with CKA_EXTRACTABLE set to false. A key that is extractable can be wrapped, meaning encrypted under a wrapping key that is itself non-exportable, but only if the token advertises the wrap mechanism and the key’s usage flags allow it. A key with CKA_WRAP_WITH_TRUSTED, CKA_ALWAYS_AUTHENTICATE or a non-empty CKA_ALLOWED_MECHANISMS list cannot be unwrapped on the far side with the same semantics.
PKCS #11, the standard interface to HSMs, leaves room for vendor differences. Tokens differ on CKK_SHA256_HMAC versus CKK_GENERIC_SECRET, on CKM_AES_KEY_WRAP versus CKM_AES_KEY_WRAP_PAD, on whether CKM_ECDSA_SHA384 exists or only raw CKM_ECDSA, and on whether an unwrap template may carry CKA_EC_PARAMS. The differences are not always documented, so a migration tool has to probe the token rather than assume.
Attribute templates depend on the object class. C_UnwrapKey accepts different attributes for a private key, a public key and a secret key. Supplying CKA_ENCRYPT on a private-key template, or CKA_EC_PARAMS on any template, does not get ignored. It returns CKR_ATTRIBUTE_TYPE_INVALID. The PKCS #11 specification marks some attributes as ones that must not be specified when an object is unwrapped, and a thread on the OASIS technical committee list describes the same footnote semantics for CKA_VALUE_LEN on an AES key. The stock template in python-pkcs11 applies all seven capability flags to every class, so the migration has to override it.
A wrapped blob proves nothing about where it came from. The wrapped key and its metadata cross a shared filesystem or network segment, and the host running the command line is often the least trusted machine involved. Without an authenticated envelope, “the migration succeeded” is an assertion.
Two administrative domains need evidence. The source operator has to show what left, the destination operator has to show what arrived, and an auditor needs both records to be tamper-evident without trusting either host. A log written by software on a compromised host proves nothing afterwards. It has to be authenticated by something only the HSM can produce, and verified fast enough that a migration of several thousand keys finishes in bounded time.
Finally, a migration is a ceremony, not a script. One script that holds the transport key, moves the keys, destroys the source and destroys the audit key is a single point of total failure. Custody of the transport key has to be split, source destruction has to be authorised by cryptographic proof of a successful import, and audit-key destruction has to be separate from key cleanup and pinned to a chain head archived elsewhere.
What does the design assume about trust?
The design trusts both HSMs’ firmware and their vendor PKCS #11 modules, the custodians collectively (as a Shamir threshold, never individually), and the operator’s terminal session for PIN entry through getpass.
It distrusts the host running the command line (filesystem, process table, shell history, logs), the shared storage holding wrapped_bundle.json and verification_receipt.json, the network path to the destination HSM, and anyone who can modify the audit log, bundle, receipt or manifest after the fact.
Four things are out of scope: compromise of either HSM, collusion of a Shamir quorum, side-channel attacks on an HSM during operation, and a malicious operator who holds the PIN and controls the host. For that last case the tool produces an audit trail for attribution afterwards and requires a quorum for the transport key, so one operator cannot complete the ceremony alone. It does not prevent the misuse.
The rule that follows from this boundary: every byte on disk is advisory until verified against an HSM-resident key.
How does the migration work?
The sequence has seven parts. The table after the list shows which key authenticates which artefact.
- Transport key custody. The transport key is a 256-bit AES key, wrap-enabled on the source and unwrap-enabled on the destination, non-extractable and sensitive on both, and its length is checked against
CKA_VALUE_LENat every use. It is split with Shamir 2-of-3 over GF(2⁸). Each share is a v2 string of the formv2|context|ceremony_id|threshold|total|x|y|commitment. The context binds a share to transport-key or audit-key use, and the ceremony ID binds the quorum to one generation run. The commitment is a SHA-256 over a domain tag, the context and the secret. It catches transcription and mixing errors. It is not a MAC, because anyone who can replace every share can replace the commitment, so it serves operational correctness and not defence against a hostile custodian set. - Wrapping. Keys are wrapped with
CKM_AES_KEY_WRAP_PAD. RFC 3394 rejects lengths that are not a multiple of 8 bytes, and RFC 5649 wraps arbitrary lengths, which a mixed EC, RSA and AES batch needs. - A signed bundle. The full bundle body (migration ID, manifest digest, source token, per-key fingerprints, per-key usage policy) is signed by a key resident on the source HSM. Only the public half exists on the destination. The import command verifies the signature before it reads any key label or unwrap template.
- Import with verification. Keys are unwrapped with class-aware templates, then checked three ways: fingerprints, usage flags read back from the created object, and a pairwise self-test.
- A signed receipt. The destination HSM signs a receipt listing each key with its expected and verified fingerprints, the manifest digest and the signature profile.
- Staged cleanup. Source deletion verifies the receipt signature, the manifest digest, exact equality of the key set, the expected destination token and the live source fingerprints, all before the first
destroy(). The audit log recordsSOURCE_CLEANUP_INTENT, then oneDESTROY_VERIFIED_SOURCE_OBJECTper object, thenSOURCE_KEYS_DESTROYED, so an interrupted run can be read from the log alone. - Audit-key destruction as its own ceremony. Cleanup refuses
--destroy-audit-key. A separateaudit-destroycommand does it in two phases, described below.
| Artefact | Authenticated by | Verified | Before what |
|---|---|---|---|
| Wrapped bundle | Signing key on the source HSM | Public half on the destination, in import | Any key label or unwrap template is read |
| Verification receipt | Signing key on the destination HSM | Public half provisioned on the source side, in cleanup | The first source object is destroyed |
| Audit record | HMAC-SHA256 key inside each HSM, from one Shamir ceremony | C_Verify on the HSM | The next record is appended |
The manifest digest ties these together. It flows from the source manifest into the bundle, the receipt and cleanup, so a receipt for a different manifest cannot authorise cleanup of this one.
How are keys fingerprinted?
Hashes are computed over type-tagged preimages, not over bare bytes.
| Key type | Preimage hashed with SHA-256 |
|---|---|
| EC | "EC-POINT-v2\0", then the curve, then the point |
| RSA | "RSA-N-E-v2\0", then length-prefixed modulus and exponent |
| AES | "AES-ZERO-BLOCK-v2\0", then the AES encryption of a zero block |
A bare hash would allow cross-type substitution: a malicious bundle could supply an EC point whose raw bytes hash to the value expected for an AES check value. The tags remove that collision by construction and tell an auditor what each hash is meant to cover.
What does the import check after unwrap?
The pairwise self-test signs a 32-byte random challenge with the freshly unwrapped private key and verifies it with the freshly created public key, using the curve’s digest for EC and CKM_SHA256_RSA_PKCS for RSA-2048. For AES it re-derives the check value. This catches a blob that decrypts under the transport key but holds a different key from the one that was promised, a substitution a fingerprint check would catch only if the attacker had failed to recompute the fingerprint too.
The template rules are specific to class. Private keys get SIGN, DECRYPT, DERIVE, UNWRAP, SENSITIVE and PRIVATE, and never ENCRYPT, WRAP or VERIFY. Public keys get VERIFY, ENCRYPT and WRAP, and never DECRYPT, UNWRAP, SIGN, DERIVE, PRIVATE or SENSITIVE. Secret keys keep the seven-capability template. This removes the family of CKR_ATTRIBUTE_TYPE_INVALID failures that a naive unwrap produces.
Curve parameters and RSA public material are verified, not supplied. The design reads CKA_EC_PARAMS, CKA_MODULUS and CKA_PUBLIC_EXPONENT back after unwrap and compares them with the signed bundle metadata, because the specification does not allow them in an unwrap template. A different curve or modulus fails the import before any receipt is written.
Usage flags are preserved in both directions. Every capability flag is captured in the bundle, applied to the unwrap template (the false values included, so a key that could not wrap does not gain CKA_WRAP at the destination), and read back after the object exists. A mismatch is a hard failure.
Keys carrying attributes that cannot cross the boundary are refused up front. These are CKA_ALWAYS_AUTHENTICATE, CKA_WRAP_WITH_TRUSTED, a non-empty allowed-mechanisms list and non-empty wrap or unwrap templates. For CKA_ALWAYS_AUTHENTICATE the error tells the operator what to do: exclude the key from this migration, provision it through a separately approved destination ceremony, verify the attribute on the destination, and only then consider deleting the source.
How is the audit trail kept honest?
Every operation appends a JSONL record whose HMAC is computed inside the HSM with C_Sign(CKM_SHA256_HMAC) under a non-extractable audit key. Both HSMs get that key from the same Shamir ceremony. A record carries seq, timestamp_utc, prev_hmac and hmac_sha256. Canonical JSON (sorted keys, no whitespace, duplicate-key detection) fixes the signed preimage.
An earlier prototype re-verified the whole chain on every append, which is quadratic over a ceremony. The current design caches the chain head per process and re-verifies only that head with one C_Verify call, so each append costs a constant amount plus one full verification at process start. Any change to the log file’s device, inode, size, modification time or change time invalidates the cache and forces a full pass. Re-verifying the cached head each time is what catches a replaced audit key under the same label and CKA_ID.
Destroying the audit key is deliberately slow. audit-destroy --prepare --expected-chain-head <archived-head> checks that the current head matches the value the operator recorded independently, then records a preparation ID. --confirm with the same head and the preparation ID checks again and destroys the key. The log is shared by both HSMs, so the harness prepares the source first, uses the source’s head to prepare the destination, confirms the two heads agree, archives the final head and both preparation IDs to a separate file, and only then confirms destruction on each token. A wrong head is rejected and leaves the key intact.
What does a failed import look like?
Take the invalid-ciphertext test in the harness, which you can watch as step 20 of the recording. The manifest lists the EC key and the RSA key before the AES key, on purpose. The AES ciphertext is corrupted, so the import succeeds for two objects and fails on the third.
Every object created during import is tracked. When the third unwrap raises, the tool makes a best-effort attempt to destroy all objects it created, writes an IMPORT_BATCH_ROLLBACK audit event and re-raises. If rollback is itself incomplete, the exception names exactly which objects remain, so the operator does not retry blindly against a dirty destination. Placing the corrupt key last means the test exercises rollback after two successful imports, which is the case where rollback has something to undo.
What does the design not cover?
Several limits are stated in the design note itself.
- The audit cache assumes append-only writing by cooperative writers. It is guarded by a thread lock in-process and
fcntl.flockacross processes, but a writer that edits the log within the same size while preserving modification and change times would not be noticed, and some filesystems do not advance the change time reliably on write. On such storage the cache must be disabled or the log kept on a local filesystem. - Wrap and unwrap templates cannot be inspected on SoftHSM without a test-compatibility flag, because python-pkcs11 has no portable decoder for those attributes. The flag is accepted only when the token identifies itself as SoftHSM. Runs against Utimaco or Crypto4A hardware must leave it off, and the checks then fail closed.
- Keys with
CKA_ALWAYS_AUTHENTICATEstay excluded. The design note says Utimaco PCIe HSMs set this on many payment and signing keys [VERIFY: vendor-specific claim, confirm against Utimaco documentation or soften], so the operator should run the dry-run and read the per-key exclusion list before scheduling anything. - The RSA-PSS salt is fixed at 32 bytes, sized for RSA-2048. For RSA-3072 or RSA-4096 signing keys, stay on PKCS #1 v1.5 or extend the salt selection.
- Truncation of the log’s tail cannot be detected by the chain. The tool prints the final head, and the operator must record it somewhere the host cannot alter. That is an operational requirement the tool cannot enforce.
- A preparation ID binds within one ceremony session. If the ID or the archived head is lost between prepare and confirm, the ceremony restarts from
--prepare.
What does this mean for AI agent workflows?
This section is interpretation. The organising rule, that nothing on the host counts until an HSM-resident key has verified it, applies beyond key migration. In an agent workflow the orchestrating host is also the component most exposed to tampered inputs, and a tool’s output is a claim about what happened. The same questions apply: what authenticates this output, who holds that key, and what stops a step from running on an unverified result? A migration that authorises deletion only on a signed receipt gives a concrete model for a destructive step that depends on proof and not on a status message.
If you are deciding how a workflow should prove its own steps before it is allowed to take an irreversible one, that is the kind of question a Workflow Assessment works through. The reasoning behind this post sits under our evidence over assurance method, and more of our work is in the Insights index.
Glossary
HSM. Hardware security module, a device that holds keys and performs cryptographic operations inside a boundary.
PKCS #11. The standard programming interface for tokens such as HSMs.
Wrapping. Encrypting one key under another so it can be moved.
KEK. Key-encryption key, here the transport key used for wrapping.
Shamir 2-of-3. A scheme that splits a secret into three shares, any two of which recover it.
KCV. Key check value, a short fingerprint used to confirm a symmetric key without exposing it.
Chain head. The latest authenticated record in the audit log; recording it externally pins the log’s length.
Frequently asked questions
Why not export the key in plaintext and import it on the other side?
A plaintext key crosses the host, its filesystem and its logs, which are the components the design distrusts. Private keys are also usually created non-extractable. Where a key is extractable, it is wrapped under a transport key that is itself non-exportable, so the plaintext never exists outside an HSM.
Why does the design use AES Key Wrap with Pad instead of plain AES Key Wrap?
RFC 3394 rejects key material whose length is not a multiple of 8 bytes. RFC 5649 wraps arbitrary lengths. A batch that mixes EC, RSA and AES keys will contain lengths that RFC 3394 refuses, so the design uses the padded mechanism, CKM_AES_KEY_WRAP_PAD.
What stops a tampered bundle from being imported?
The whole bundle body is signed by a key resident on the source HSM, and only the public half is provisioned on the destination. The import command verifies that signature before it reads any key label or unwrap template, so a tampered bundle is rejected without touching the destination keystore.
When are the source keys destroyed?
Only after the destination HSM has signed a receipt that verifies, lists exactly the keys in the source manifest, carries the same manifest digest, names the expected destination token, and whose fingerprints match the live source keys. All of this is checked before the first deletion.
Which keys can this design not migrate?
Keys with CKA_ALWAYS_AUTHENTICATE or CKA_WRAP_WITH_TRUSTED set to true, a non-empty CKA_ALLOWED_MECHANISMS list, or a non-empty wrap or unwrap template are refused, because those semantics cannot be reproduced on the far side. Run the dry-run first and read the exclusion list before scheduling the ceremony.
Does the audit log prove that nothing was deleted from its end?
No. The HMAC chain detects edits and reordering inside the log, but it cannot show that the last records were removed. The tool prints the final chain head so the operator can archive it somewhere independent, and the audit-key destruction path requires that archived value before it proceeds.
Sources
- PKCS #11 Cryptographic Token Interface Base Specification Version 2.40 (OASIS) (primary source), accessed
- OASIS PKCS #11 TC mailing list: CKA_VALUE_LEN in an AES unwrap template (January 2023) (secondary source), accessed
Update log
- : Published.