For the complete documentation index, see llms.txt

Index contract state with EffectStream

Use this guide to build an indexer for a Midnight contract. An indexer watches the contract's public state, keeps its own records, and answers questions the chain does not.

The procedures target the preview network and a contract you need not deploy, so the whole guide runs with no local stack. Reading one chain is the smallest thing EffectStream does; Additional resources points to the cross-chain path.

Community-maintained, not officially supported

EffectStream maintains the @effectstream/* packages and the templates this guide uses, not the Midnight Foundation. The project changes often, so when its repository and this guide disagree, follow the repository. Report EffectStream problems on the EffectStream tracker, and problems with this page on the Midnight docs repository.

Prerequisites

These apply to every procedure in this guide:

• Bun. EffectStream runs TypeScript directly and uses Bun as its package manager. See Set up Bun for Midnight development.
• Network access to the hosted preview indexer listed in the environment reference.
• curl, or any HTTP client, for the verification steps.

These procedures read from a hosted indexer over HTTPS, so they need no Docker, no Midnight node, no proof server, and no Compact toolchain.

Keep the pinned SDK version

This guide pins @effectstream/node-sdk to 0.104.3, the version npm's midnight-1 tag points to. That release builds on the ledger 8 packages in the support matrix, such as Midnight.js 4.1.1. The newest release on npm, the 0.200 line, depends on ledger 9 prereleases, and Preview, Preprod, and Mainnet run ledger 8.

Why a DApp needs an indexer

A Midnight contract stores its current state: its public ledger, the verifier keys for its circuits, its maintenance authority, and its token balances. Who holds the top score? What changed in the last hour? The chain answers neither. Midnight's own indexer can return a contract's state as of a past block, but it computes no totals and no per-field changes.

An indexer answers them. It watches state changes, applies a function you write to each one, and can store the result in a database you control. It handles changes in chain order, so replaying the same history produces the same result.

Midnight already runs an indexer, and this guide connects to it rather than replacing it. The Midnight indexer reports what the chain holds, and your DApp reaches it through the publicDataProvider in Deploying and operating a contract. EffectStream reads from that indexer and builds your own derived state on top.

Reading the ledger has two consequences. A change reaches your code only after several blocks of confirmation, so a chain reorganization cannot leave your database holding state the chain later drops. And your code sees exactly what the contract discloses, which makes What the ledger schema can read a contract decision.

Run the indexer

Build and run a node that watches a Midnight contract and prints each state change. The contract used here is a counter already running on preview, so you deploy nothing.

Procedure

1. Create an empty project directory, then add a package.json with the single dependency an EffectStream node needs. Pin the version here rather than in the import statement:

   package.json

   {
     "name": "midnight-indexer",
     "private": true,
     "scripts": {
       "start": "bun index.ts"
     },
     "dependencies": {
       "@effectstream/node-sdk": "0.104.3"
     }
   }

2. Install with Bun. Run a real install rather than relying on the auto-install cache: the SDK reaches packages that import themselves by name, and resolving those imports needs a node_modules directory the cache never creates:

   bun install

3. Create index.ts alongside package.json. A single runNode call describes the whole node, taking a name, a database, the sources to watch, and one transition function per source:

   index.ts

   import { midnightContract, pglite, runNode } from "@effectstream/node-sdk";
   
   await runNode({
     appName: "midnight-indexer",
     database: pglite(),
     sources: {
       counter: midnightContract({
         network: "preview",
         address: "c1a9ec7c4d2566f59456fd915a0438bf4dc9b8671d4c2308d30af796c51ad20f",
         startBlockHeight: "latest",
         ledger: { round: "uint128" },
       }),
     },
     transitions: {
       counter: ({ state, blockHeight }) => {
         console.log(`round ${state.round} at block ${blockHeight}`);
       },
     },
   });

   pglite() gives the node an embedded PostgreSQL-compatible database, held in memory unless you pass a dataDir. The sources key, counter here, is yours to choose, and transitions reuses it so the two stay paired. Setting network selects the hosted indexer, and address takes the contract's 64-character hex address with no 0x prefix. startBlockHeight takes "latest" to begin at the block the indexer is on, or a block number to replay from that height. What the ledger schema can read covers the ledger option, and whether it works for the contract you have in mind.

4. Start the node:

   bun start

5. Read the decoded state in the transition, which receives an object with state, blockHeight, blockTimestamp, network, and address. blockHeight is EffectStream's own block number, not the Midnight height. With the in-memory database it starts at 1 on every run. With a dataDir, the node reads the last block it recorded from the database and carries on from there. Unsigned integers arrive as decimal strings, because a uint128 does not fit in a JavaScript number. Byte fields and map keys arrive as 0x-prefixed hex strings. Convert with BigInt(state.round) when you need arithmetic.

Replay history instead of waiting for a call

A transition fires on state changes, not on blocks, so a quiet contract prints nothing. startBlockHeight takes a block number to replay from that height instead of starting at the tip. The node then fetches every block from that height to the tip, in batches of ten with one indexer request per block, before the first transition fires. Replay therefore only suits a call made recently. The hosted indexer answers 403 Forbidden to a client that fetches too much, too fast, which a long replay does. Find the block of a contract's most recent call with the indexer's contractAction query. The counter used in this guide was last called at block 377864, which is too far behind the tip to replay. To watch a transition, point the node at a contract you can call yourself.

Verification

The node connects to preview, starts its HTTP server, and logs each block it fetches, like this:

[MidnightClient] Using indexer https://indexer.preview.midnight.network/api/v4/graphql (network: preview)
Paima Engine HTTP server running on http://127.0.0.1:9999
[Midnight:preview] Fetching blocks from 788014 to 788014.
10:20:19 INFO effectstream-sync-block-merge: finalized block 1 @ undefined... | {"clock":[1,1]}
10:20:21 INFO effectstream-sync-block-merge: finalized block 3 @ undefined... | {"clock":[3,3],"midnight-counter":[788014,788014]}

The finalized block numbers are EffectStream's own clock, which starts at 1 on an empty database, and @ undefined fills the slot where a merged block hash would go. The Midnight height is the midnight-counter pair, which appears on the ticks that carried a Midnight block. Your heights differ.

A round N at block M line appears whenever the contract's state changes. A contract nobody is calling produces no such line, which is the case the tip above covers.

What the ledger schema can read

The ledger option declares how to decode public state, which is why a node needs no compiled contract artifacts. It also decides which contracts a node can read at all, so check it against your contract before you plan around it.

EffectStream matches schema keys to ledger fields by position. The first key reads the first field, the second key reads the second, and so on until your schema runs out. A schema accepts these types:

• "uint8" through "uint128", decoded little-endian into decimal strings
• "bytes", decoded into a 0x-prefixed hex string
• "boolean"
• { type: "map", value: <type> }, nesting arbitrarily, with keys as 0x hex strings
• { type: "option", value: <type> }, decoding to null when absent

A List, MerkleTree, or HistoricMerkleTree fails at parse time. A struct, Vector, or enum reads without error, but a schema cannot split a struct into named fields. Read a struct or Vector as "bytes" to get its raw bytes as one hex string, and an enum as "uint8". Read a struct as a number and you get a wrong value, with no error. Because the match is positional, one unreadable field blocks every field after it. A List sitting third in a ledger costs you the fourth field and the fifth as well.

Some contracts need the compiler-generated decoder instead

A schema cannot read lists or Merkle trees, or split a struct into named fields. When your contract's ledger holds a list or tree near the top, or you need a struct's fields, use EffectStream's full configuration API instead. It takes the decoder the Compact compiler generates, which reads every exported ledger field. See the EffectStream documentation.

When you control the contract, you can design its ledger so a schema reads it. Put the fields an indexer needs at the top, and lists, Merkle trees, structs, and digests at the bottom. Flatten a struct that needs indexing into sibling top-level declarations keyed the same way. A Map<Bytes<32>, GameState> holding four small numbers becomes four Map<Bytes<32>, Uint<8>> declarations. The trade is four ledger containers and four writes per update instead of one, so weigh it against how often the values change.

// An indexer reads these, in this order.
export ledger status: Map<Bytes<32>, Uint<8>>;
export ledger winner: Map<Bytes<32>, Uint<8>>;
export ledger scores: Map<Bytes<32>, Map<Uint<16>, Uint<8>>>;

// Structs and digests go last.
export ledger commitments: Map<Bytes<32>, GameKeys>;

The matching schema stops where the structs begin:

ledger: {
  status: { type: "map", value: "uint8" },
  winner: { type: "map", value: "uint8" },
  scores: { type: "map", value: { type: "map", value: "uint8" } },
}

Ordering is an indexing decision, not a privacy one. Every ledger field is public wherever it sits, which is the point Security and best practices makes about disclose(). Put the fields a schema must read first. Identity commitments and Merkle roots serve in-circuit checks rather than readers, so they belong at the bottom.

Serve indexed state over HTTP

Expose what the node records so the rest of your application can read it. runNode hosts an HTTP server.

Prerequisites

• A running node, from Run the indexer.

Procedure

1. Replace index.ts with this version. It records the value in a variable instead of logging it, and adds an api function to serve it:

   index.ts

   import { midnightContract, pglite, runNode } from "@effectstream/node-sdk";
   
   let round = "waiting for the next contract update";
   
   await runNode({
     appName: "midnight-indexer",
     database: pglite(),
     sources: {
       counter: midnightContract({
         network: "preview",
         address: "c1a9ec7c4d2566f59456fd915a0438bf4dc9b8671d4c2308d30af796c51ad20f",
         startBlockHeight: "latest",
         ledger: { round: "uint128" },
       }),
     },
     transitions: {
       counter: ({ state }) => {
         round = state.round;
       },
     },
     api: async (server) => {
       server.get("/round", async (_request, reply) => reply.send({ round }));
     },
   });

2. Restart the node and request the endpoint. The server listens on port 9999 unless you set apiPort on runNode.

Choose the database mode deliberately

pglite() holds data in memory by default, so every run starts at the chain tip and misses whatever happened while the node was down. Passing pglite({ dataDir: "./data" }) stores the database on disk, so a restart can resume from the block height it recorded. With 0.104.3, the first restart after the node creates the directory still starts at the tip, and later restarts resume. Confirm that your node resumes before you rely on it.

Verification

The endpoint answers with the value the last transition recorded, or the starting value when no transition has fired yet:

curl http://localhost:9999/round

{"round":"waiting for the next contract update"}

Indexer troubleshooting

This table lists the failures you may hit on this guide's path, with their fixes.

Symptom	Cause	Fix
Install fails resolving a package that imports itself by name	Bun's auto-install cache created no node_modules anchor	Run bun install before bun start
The first transition takes far longer than expected	EffectStream holds each change for several blocks of confirmation	Wait. The delay keeps the database consistent with settled state
A comparison against a decoded number never matches	Unsigned integers decode to decimal strings, not numbers	Convert with BigInt(state.field) before comparing
The node stops with Unable to read the Midnight preview tip: 403 Forbidden	The hosted indexer is refusing your client, which happens when a replay fetches too many blocks too fast	Wait before retrying, and replay only from a height close to the tip
Parse error naming an array where a scalar belongs	A List, MerkleTree, or HistoricMerkleTree sits at a schema position	Use the compiler-generated decoder, or reorder the ledger if you control the contract; see What the ledger schema can read

Additional resources

• EffectStream documentation: the project's own documentation, and the cross-chain templates that pair Midnight with an EVM chain, Bitcoin, or Cardano.
• Deploying and operating a contract: the providers a DApp uses to reach the Midnight indexer directly, and how to deploy a contract of your own to index.
• Networks and environments: every endpoint, network ID, and the local stack.
• Security and best practices: what a chain observer sees, and how disclose() decides it.
• Support matrix: the Midnight versions each network runs.