Monthly research note. Theme: Blockchain Protocols.

TL;DR

A focused memo on Rust Node Architecture: Storage, Networking, and Deterministic Execution: define the model, state the properties, then design the system so those properties remain true under failure and adversaries.

Key insight

Most failures are boundary failures: parsing, persistence, concurrency, retries, and upgrades.

Key takeaways

  • Upgrades must be compatibility-aware: mixed rulesets are a threat model.
  • Finality guarantees are user security guarantees—document and enforce them.
  • Mempools are adversarial schedulers: admission and fairness are protocol concerns.
  • Make failure modes explicit and observable.
  • Write assumptions down; treat them as interfaces.

Why this matters

  • Consensus safety is meaningless if execution is nondeterministic across nodes.
  • Finality guarantees are user security guarantees; ambiguity is a UX vulnerability.
  • State growth is a security problem: it impacts decentralization and verification.
  • Topology attacks (eclipse, partition) change who sees which transactions.

Key questions

  • Where is the economic/DoS pressure applied (mempool, gossip, execution, storage)?
  • How do upgrades change security assumptions (fork choice, state transition rules)?
  • What is the finality guarantee users can rely on (and when does it break)?
  • What is the reorg budget for applications and how do you communicate it?
  • Where do you enforce resource limits (gas, bandwidth, storage, signature checks)?
  • What is the determinism story (byte-for-byte re-execution across platforms)?

Assumptions

  • Nodes are heterogeneous; determinism must survive platform differences.
  • Users and apps rely on probabilistic finality until proven otherwise.
  • Attackers can buy bandwidth and compute; they can also bribe and censor.
  • Peers are untrusted; gossip can be manipulated for delay or isolation.

Non-goals

  • Relying on client-side heuristics to paper over protocol ambiguity.
  • Treating mempool policy as “local preference” when it affects security.
Attack surface

Parsing is an attacker-controlled interface—validate early and fail fast.

Model & invariants

A ledger is a replicated state machine. Safety is uniqueness of finalized history:

∀h1,h2: Final(h1)∧Final(h2)⇒h1⪯h2 ∨ h2⪯h1.\forall h_1,h_2:\ \mathrm{Final}(h_1)\wedge \mathrm{Final}(h_2)\Rightarrow h_1 \preceq h_2 \ \vee\ h_2 \preceq h_1.

Model the mempool as an adversarial scheduler: it chooses which work gets executed.

Treat reorgs as a user-visible security event; encode reorg-aware semantics.

Invariant

Make the “impossible state” observable: a metric or alert that fires when invariants drift.

Security properties

  • Downgrade resistance: negotiation can’t silently weaken security posture.
  • Replay resistance: duplicated inputs do not change outcomes.
  • Integrity: invalid transitions are rejected (and detectable).
  • Least authority: privileges are scoped by purpose and time.

Failure modes

  • Observability gaps during incidents (missing evidence).
  • Timeout ambiguity causing double-apply or partial state transitions.
  • Resource exhaustion (CPU/bandwidth/storage) turning into correctness failures.
  • Recovery paths that only work when nothing is broken.
Pitfall

Mixed-version deployments create states you never tested—plan for them explicitly.

Design sketch

flowchart TD
  tx["Transaction"] --> mp["Mempool (admission + prioritization)"]
  mp --> prop["Block Proposal"]
  prop --> cons["Consensus / Finality"]
  cons --> exec["Deterministic Execution"]
  exec --> root["State Root Commitment"]

Implementation notes

Encode resource accounting and limits early; retrofits are painful.

Rule of thumb

Make rollbacks boring: if rollback is a hero move, it will fail.

Mempool hardening checklist:
- Per-peer rate limits + global admission budget
- Duplicate detection and eviction policy
- Signature verification batching with caps
- Anti-DoS: bounded decode/parse cost
- Fairness: per-sender quotas (avoid hot-account starvation)

Verification strategy

  • Formal invariants for supply/balance conservation where appropriate.
  • Fuzzing transaction decoding and state transition edge cases.
  • Determinism tests across architectures (x86/ARM) and OSes.
  • Fork/reorg simulations: application-facing invariants under reorgs.
  • Cross-implementation tests when multiple clients exist.

Operational notes

  • Protect peer tables against eclipse attempts (diversity, scoring, rotation).
  • Measure invalid tx rejection reasons and rates (spam signature).
  • Monitor reorg depth and frequency; treat increases as incidents.
  • Keep execution resource limits explicit and enforced.
  • Rehearse upgrades with mixed versions and rollback paths.
Operational note

Attach explicit rollout/rollback triggers to changes that touch security or correctness.

What to monitor

  • Admission-control / rate-limit rejections (by reason).
  • Invariant violation rate (should be ~0).
  • Authz failures and policy denials (unexpected spikes).
  • Rollback events and the conditions that triggered them.
  • Retry/timeout rates by endpoint and client cohort.

Rollback plan

  • Define an explicit rollback trigger (metrics + thresholds).
  • Preserve evidence (configs, artifacts, audit logs) to reconstruct what changed.
  • Use canaries and staged rollout; stop early when signals degrade.
  • Keep dual-write / dual-verify windows where appropriate.
  • Prefer backward-compatible changes; avoid “flag day” upgrades.

Evidence

  • Learn TLA+ (1) — Practical entry point for specification and model checking.
    • Evidence: Model the smallest thing that can break; use model checking to validate invariants before optimizing.
  • Designing Data-Intensive Applications (Kleppmann) (2) — The systems-engineering baseline for correctness, replication, and failure.
    • Evidence: Replication and consistency tradeoffs as engineering constraints; use as reference when naming guarantees.

Open questions

  • What is the worst-case work a single transaction can force?
  • How do you communicate finality uncertainty to users without lying?
  • Where does your implementation accidentally depend on local wall-clock time?
  • Which invariants should be proven vs tested vs monitored?

Checklist

  • Rollback plan rehearsed and automated.
  • Telemetry captures correctness signals.
  • Costs bounded (CPU/memory/bandwidth) under adversarial inputs.
  • Assumptions listed and reviewed.
  • Failure modes enumerated with mitigations.
  • Safety properties stated as invariants.

Further reading

1.
LearnTLA. Learn TLA+ [Internet]. Web; Available from: https://learntla.com/
2.
Kleppmann M. Designing Data-Intensive Applications [Internet]. O’Reilly Media; 2017. Available from: https://dataintensive.net/