Midnight Preprod · Known issues

Publish the node's 9944 as a pure tls service on the host edge and point every consumer at wss://<app>.fly.dev:9944. Then make node-reachable-from-indexer its own explicit preflight step — never trust an indexer answer before that check passes.

# measured facts, not assumptions
# on the node machine
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:9944/health
# on the indexer machine
curl -s -o /dev/null -w '%{http_code}\n' https://<app>.fly.dev:9944

Serve 9944 with a tls handler only — no http handler and no http_options on that service. Keep .internal / private-network names for server-to-server calls that stay inside the stack, and probe reachability from the consuming machine instead of assuming it.

Pin @midnight-ntwrk/compact-js@2.5.1 — it resolves cleanly and has no ledger-v9 edge. Pin every Midnight package to an exact version: never @latest, never a caret. Before retrying, delete the stale lockfile and node_modules.

rm -rf node_modules package-lock.json
npm i @midnight-ntwrk/compact-js@2.5.1 --save-exact

On a persistent build/runner machine, expose a “Clear toolchain” action that resets the install without destroying the volume — the volume holds the chain data and the LevelDB private state.

Give the build/runner machine 4 vCPU / 4 GB RAM and a 10 GB volume. Install in small sequential groups with an npm cache on the volume instead of one giant npm i. Emit a heartbeat every ~30 s so “slow” is distinguishable from “dead”, and surface the host’s machine events (OOM flag, exit code) in the UI — not just the log tail.

For any detached job, print STEP_<name> markers and JOB_FAILED status=<n> during: <phase>, drive a monotonic step timeline off them, and persist the failed job’s log auto-expanded with a copy button. A toast loses the only diagnostic you had.

Two commands — same seed as Lace produces, same address the faucet accepts:

npm i -g midnight-wallet-cli

# --seed is the 64-char hex master seed (32 bytes), NOT the mnemonic
mn address --seed <64-hex-master-seed> --network preprod
# → prints your unshielded mn_addr_preprod1… (paste into the faucet)

# Verify funds landed
mn balance <mn_addr_preprod1…> --network preprod

• --network accepts preprod, preview, undeployed, mainnet.
• The seed MUST be the 64-hex master seed. If you have a BIP-39 mnemonic, derive the master seed first (or use our scripts/derive-unshielded-address.mjs as an offline fallback).
• The shielded address (mn_shield-addr_…) is a different identity — the faucet only accepts the unshielded one.
• Never paste your seed into chat, screenshots, or issue trackers.

• Use Preview for throughput / latency benchmarking.
• Align every dep to the Preprod row on the support matrix — version skew often surfaces as sync decode failures.
• Confirm indexer + node are healthy and at chain tip before sync.
• NODE_OPTIONS="--max-old-space-size=8192" helps OOM but won't fix decode loops.

Two complementary techniques Midnight recommends today:

1. Sync from where you last left off. Serialize the wallet state to disk after the first successful sync, then restore it on the next run so only the delta syncs.

   // After first sync
   const serialized = await wallet.serializeState();
   await fs.writeFile("wallet-state.bin", serialized);
   
   // Next run — restore before wallet.start()
   const restored = await fs.readFile("wallet-state.bin");
   const wallet = await WalletBuilder.restore(
     indexerUrl, indexerWsUrl, proofServerUrl, nodeUrl,
     restored, networkId,
   );
   wallet.start(); // only the delta re-syncs

2. Don't store history you won't read. Pass NoOpTransactionHistoryStorage in your wallet config to cut memory during sync. Only safe if your script never queries transaction history.

   import { NoOpTransactionHistoryStorage } from "@midnight-ntwrk/wallet";
   
   const wallet = await WalletBuilder.buildFromSeed(
     indexerUrl, indexerWsUrl, proofServerUrl, nodeUrl,
     seed, networkId,
     { transactionHistoryStorage: new NoOpTransactionHistoryStorage() },
   );

Combine both for headless deploy / server-append scripts: they never need history, and a serialized wallet-state file keeps CI / Fly.io restarts fast.

1. Pin every package to the current Preprod row on the support matrix.
2. Preprod indexer v4: https://indexer.preprod.midnight.network/api/v4/graphql · wss://indexer.preprod.midnight.network/api/v4/graphql/ws
3. Wallet config workaround that helps some setups: batchUpdates: { size: 5000, timeout: 1, spacing: 4 }
4. NODE_OPTIONS="--max-old-space-size=8192" for OOM during first sync.
5. Confirm the SDK uses the same seed as Lace — different seed = different wallet.
6. Only read DUST after full sync on all legs.
7. Unblock local dev with create-mn-app / local docker network (undeployed) while Preprod sync is rough.

• Re-pin wallet + ledger + midnight-js + proof-server from the Preprod matrix in one pass.
• Resync a fresh wallet dir.
• Wallet config workaround that helps some setups: batchUpdates: { size: 5000, timeout: 1, spacing: 4 }
• Preprod indexer v4: https://indexer.preprod.midnight.network/api/v4/graphql · wss://indexer.preprod.midnight.network/api/v4/graphql/ws

• Serialize submits per wallet (or keep 1–2 in flight).
• Pre-warm DUST: fund NIGHT, wait for cap fill, run a few serial txs before load-testing.
• Use multiple wallets for parallel lanes — each has its own DUST generation.

Decouple with the lower-level pipeline:

createUnprovenCallTx
  → proofProvider.proveTx
  → walletProvider.balanceTx
  → midnightProvider.submitTx

Parallelize proving, serialize balance+submit per wallet. Add app-level timeouts around submit.

1. Confirm WS vs HTTP: if you get `1010 Custom error: N`, WS is fine.
2. Fresh wallet resync (or resync right before submit).
3. Compare indexer block height vs RPC tip.
4. Pin the full stack to the current Preprod matrix.
5. Retry after sync completes — don't submit from a long-idle wallet.

1. cat managed/<circuit>/compiler/contract-info.json — if it's 0.30.x / 0.22.0, confirms the 0.31 ZKIR rework is the trigger.
2. Point httpClientProofProvider at the public prover https://lace-proof-pub.preprod.midnight.network. If register_asset /check passes there, your local image is just behind the deployed prover. (Lace's wallet-delegated proving uses this same public backend, which is why Lace succeeds where local Docker rejects.)
3. Bisect the reworked op: drop decimals Uint<8>, drop the secret-key → owner-id conversion, recompile, retry /check to pin the exact op.
4. Fix is upstream in a later Compact release — see the toolchain 0.31.0 release notes.

Open a Service Desk ticket with two linked issues:

1. /check bad input. Include: deploy / create_market work, register_asset fails, Uint<64> widen didn't fix (→ second reworked op), check() isn't skippable (stub → WASM unreachable), latest stable provider is 4.1.1. Ask which prover parses 0.31 ZKIR on /check, ETA for the ZKIR-format fix from the 0.31.0 notes, and the precise list of reworked ops to avoid on 8.0.3.
2. Matrix conflict. Align ledger-v8, wallet-sdk-dust-wallet, and proof-server to the same support-matrix row. As of the current matrix, that means ledger-v8 8.1.0, proof-server 8.1.0, and Wallet SDK 1.2.0. Ask for the coherent wallet-sdk set if you see Transaction.addIntent mismatches.

Also note in the ticket that lace-proof-pub.preprod.midnight.network from COMPATIBILITY.md doesn't resolve publicly. See the local-repro workaround section for the Lace-vs-httpClientProofProvider bisect that produced this evidence.

• Midnight Node: 1.0.1 (Preview) / 1.0.0 (Preprod) / 1.0.0 (Mainnet)
• Midnight Indexer: 4.3.3
• proof-server: 8.1.0 (public networks; use 8.0.3 for local Undeployed)
• ledger-v8: 8.1.0
• compact: 0.5.1 · toolchain 0.31.1
• compact-runtime: 0.16.0 · compact-js: 2.5.1
• midnight-js-*: 4.1.1
• testkit-js: 4.1.1
• onchain-runtime-v3: 3.0.0
• Wallet SDK: 1.2.0 (facade / dust / shielded must all match this row)
• DApp Connector API: 4.0.1

• HTTP: https://rpc.preprod.midnight.network
• WS: wss://rpc.preprod.midnight.network

Alternatives:

• Blockfrost (needs API key): https://rpc.midnight-preprod.blockfrost.io?project_id=YOUR_KEY
• Self-host midnight-node via Docker (github.com/midnightntwrk/midnight-node).
• Preview WS wss://rpc.preview.midnight.network — only if you can switch networks (not Preprod-compatible).

Smoke check:

curl -X POST https://rpc.preprod.midnight.network \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"chain_getHeader","params":[],"id":1}'

Sign the recipe between balancing and finalising:

const recipe = await wallet.balanceUnboundTransaction(tx, keys, { ttl });
const signed = await wallet.signRecipe(recipe, (p) => keystore.signData(p)); // required
return wallet.finalizeRecipe(signed);

Rule of thumb: the moment a circuit reads or sends NIGHT, signRecipe is mandatory.

• Serialize calls per wallet — an agent loop must queue, not fan out.
• Wrap every submit in an explicit timeout so a rejection surfaces as a failure, not a hang.
• Contract-level write contention cannot be measured from a single-wallet harness: a low "landed" count measures the wallet. Use a second funded wallet.

• Concurrent callers need different midnightDbName values. Different privateStateStoreName values change nothing — they still open the same directory.
• A fresh store is empty: call provider.setContractAddress(contractAddress) first, then await provider.set(id, emptyPrivateState()).
• Passing midnightDbName: undefined is not the same as omitting it — the provider spreads your config over its defaults, so an explicit undefined wipes the default. Spread the key in only when it is set.
• Rule out all four local causes before reading any result as a property of the chain.

• Use a 10-minute sync deadline, overridable via an env var (m402 uses MIDNIGHT_SYNC_TIMEOUT_MS).
• Implement it as an explicit Rx.race against a timer, not Rx.timeout({ each }) placed after a filter — emissions that fail the filter never reach the timeout, so its semantics silently depend on pipe position.
• Restart the command after a deadline hit; a dead fibre never recovers.

Node 22 or 24 only — 22.12.0 and 24.19.0 are both verified against Preview. Check node -v before debugging anything else, and pin it in .nvmrc / package.json engines.

Persist the sub-wallet states and restore them on the next build. Measured on Preview: 687.5s cold → 53.8s warm (12.8x), CPU 644s → 12s — that is trial-decryption replay disappearing.

• Use serializeState() / restore() on all three sub-wallets.
• All three sub-wallets and the facade must share one txHistoryStorage — otherwise shielded and unshielded writes go to a storage the facade never reads.
• Never cache before sync completes. A mid-sync position restores cleanly and resumes from somewhere the wallet never applied.
• Make restore best-effort: any missing/unreadable file falls back to the from-seed build.
• The cache holds the wallet's coins — key it by a hash of master seed + network id and write it 0600. It is a wallet secret.
• Bonus: @midnight-ntwrk/testkit-js costs ~5.2s just to import. Load it lazily so --help, --version and dry-runs don't pay for a wallet builder they never call.

File product feedback via the Midnight Service Desk if settlement flows need it.