PQC in VPN/IPsec: IKEv2 Revisited Under PQ Constraints

Monthly research note. Theme: Post-Quantum Cryptography & Migration.

TL;DR

PQC in VPN/IPsec: IKEv2 Revisited Under PQ Constraints as an engineering constraint: write down assumptions, make invariants executable, and design operational recovery as part of correctness.

Key takeaways

• Interop is the migration plan—test matrices are more important than whitepapers.
• PQC changes handshake costs; plan DoS defenses and budgets.
• Migration is mixed-version for years: compatibility and rollback are security features.
• Design rollbacks as part of the happy path.
• Automate guardrails; humans are for judgment, not for consistent enforcement.

Why this matters

• Operationalization (monitoring, rollback) determines success more than crypto choice.
• Constant-time constraints are harder under large primitives.
• PQC changes bandwidth and CPU costs; DoS surfaces move.
• Hybrid designs fail if binding is ambiguous (mix-and-match, downgrade).

Key questions

• What does interoperability testing look like across vendors and stacks?
• Which secrets require long-term confidentiality (HNDL) and where are they today?
• How do you bind hybrid secrets to prevent downgrade and mix-and-match attacks?
• How do you handle failures: decryption failures, invalid ciphertexts, malformed keys?
• How do you rotate algorithms safely (crypto agility without chaos)?
• What are the new DoS surfaces (bigger keys, more CPU, more bandwidth)?

Assumptions

• Deployments are mixed; old clients must interoperate or fail safely.
• Side channels exist: timing and cache behavior leak information.
• Vendors vary: implementations and defaults differ.
• Bandwidth is limited in some environments; larger handshakes matter.

Non-goals

• Ignoring DoS implications of large primitives.
• Relying on silent fallback to weaker modes during interop failures.

Model & invariants

Hybrid composition should be transcript-bound:

Treat algorithm negotiation as adversarial: explicit downgrade resistance.

Make costs explicit: measure CPU and bandwidth, then add protections.

Security properties

• Evidence: critical actions emit verifiable audit events.
• Least authority: privileges are scoped by purpose and time.
• Downgrade resistance: negotiation can’t silently weaken security posture.
• Integrity: invalid transitions are rejected (and detectable).

Failure modes

• Timeout ambiguity causing double-apply or partial state transitions.
• Config drift that weakens security posture over time.
• Mixed-version behavior that violates assumptions silently.
• Recovery paths that only work when nothing is broken.

Design sketch

flowchart TD
  negotiate["Negotiate Algorithms"] --> bind["Bind Transcript"]
  bind --> kdf["KDF (hybrid)"]
  kdf --> keys["Traffic Keys"]
  keys --> monitor["Monitor + Rollback"]

Implementation notes

Explicit binding prevents downgrade and mix-and-match. Don’t leave it implicit.

// Hybrid binding sketch (pseudocode):
// ss = HKDF(ss_classical || ss_pqc, info=transcript_hash)
// Then derive traffic keys from ss.

Verification strategy

• Side-channel tests where tooling exists; constant-time audits.
• Downgrade tests: active attacker manipulates negotiation.
• Interop matrices across vendors/versions and failure modes.
• DoS tests: measure CPU/bandwidth amplification and mitigation impact.
• Chaos deploys: mixed versions + rollback during partial outages.

Operational notes

• Roll out with canaries and explicit rollback triggers.
• Add telemetry for negotiation outcomes, failures, and client cohorts.
• Cap handshake cost per peer/IP; use stateless cookies when needed.
• Inventory long-lived secrets and migrate the highest-risk first.
• Document supported algorithm sets and deprecation timelines.

What to monitor

• Retry/timeout rates by endpoint and client cohort.
• Admission-control / rate-limit rejections (by reason).
• Invariant violation rate (should be ~0).
• Authz failures and policy denials (unexpected spikes).
• Error budget burn + tail latency under load.

Rollback plan

• Define an explicit rollback trigger (metrics + thresholds).
• Prefer backward-compatible changes; avoid “flag day” upgrades.
• Use canaries and staged rollout; stop early when signals degrade.
• Keep dual-write / dual-verify windows where appropriate.
• Preserve evidence (configs, artifacts, audit logs) to reconstruct what changed.

Evidence

• NIST Post-Quantum Cryptography Project (1) — Standardization process and algorithm selections.
  • Evidence: Treat PQ migration as a program (inventory, interop, rollback). Use NIST status to drive prioritization and timelines.
• RFC 5869: HKDF (2) — Useful when discussing hybrid binding and context separation.
  • Evidence: HKDF is the workhorse for domain separation; bind purpose/context to avoid cross-protocol key reuse.

Open questions

• Where would a downgrade be visible today, and how would you detect it?
• What is the worst-case handshake cost under attack?
• How do you rotate algorithms without introducing configuration chaos?
• Which clients will fail first, and what is the safe fallback behavior?

Checklist

• Costs bounded (CPU/memory/bandwidth) under adversarial inputs.
• Assumptions listed and reviewed.
• Failure modes enumerated with mitigations.
• Safety properties stated as invariants.
• Rollback plan rehearsed and automated.
• Telemetry captures correctness signals.

Further reading

• NIST Post-Quantum Cryptography Project — Standardization process and algorithm selections.
• RFC 5869: HKDF — Useful when discussing hybrid binding and context separation.
• CRYSTALS-Dilithium — Signature scheme design and deployment constraints.
• CRYSTALS-Kyber — KEM design and parameters commonly referenced in deployments.
• Learn TLA+ — Practical entry point for specification and model checking.
• Designing Data-Intensive Applications (Kleppmann) — The systems-engineering baseline for correctness, replication, and failure.