Compact Deep Dive Part 2 - Circuits and Witnesses

This blog post is part of the Compact Deep Dive series, which explores how Compact contracts work on the Midnight network. Each article focuses on a different technical topic and can be read on its own, but together they provide a fuller picture of how Compact functions in practice. Compact Deep Dive - Part 1 looked at the TypeScript API of the Compact compiler-generated contract implementation. If you haven’t read it yet, we encourage you to start there. This article looks at how circuits and witnesses are actually implemented in JavaScript code generated by the Compact compiler.

In part 1, we used the Bulletin Board tutorial contract as an example. We compiled it with the Compact compiler and started looking at the files generated by the compiler. We used Compact toolchain version 0.24.0. Recall the caveat from part 1 that the generated code is an implementation detail of the platform. We freely change it, so it can be different when different versions of the compiler are used.

Circuits

We looked at the bulletin board contract’s post circuit. The compiler generated a TypeScript declaration file that includes a declaration for the circuit:

post(context: __compactRuntime.CircuitContext<T>, newMessage_0: string): __compactRuntime.CircuitResults<T, []>;

Recall that the types CircuitContext and CircuitResults came from the Compact runtime used by the compiler.

In the file index.cjs the compiler has generated a JavaScript implementation of this circuit. You will find this in the contract subdirectory of the compiler output directory you gave on the compact command line when compiling the contract’s Compact source code.

The implementation is installed as a property on the circuits object of the contract in class Contract’s constructor. The entirety of the code is here (we will drill down into this in the rest of this section):

this.circuits = {
  post: (...args_1) => {
    if (args_1.length !== 2)
      throw new __compactRuntime.CompactError(`post: expected 2 arguments (as invoked from Typescript), received ${args_1.length}`);
    const contextOrig_0 = args_1[0];
    const newMessage_0 = args_1[1];
    if (!(typeof(contextOrig_0) === 'object' && contextOrig_0.originalState != undefined && contextOrig_0.transactionContext != undefined))
      __compactRuntime.type_error('post',
                                  'argument 1 (as invoked from Typescript)',
                                  'bboard.compact line 26 char 1',
                                  'CircuitContext',
                                  contextOrig_0)
    const context = { ...contextOrig_0 };
    const partialProofData = {
      input: {
        value: _descriptor_4.toValue(newMessage_0),
        alignment: _descriptor_4.alignment()
      },
      output: undefined,
      publicTranscript: [],
      privateTranscriptOutputs: []
    };
    const result_0 = this.#_post_0(context, partialProofData, newMessage_0);
    partialProofData.output = { value: [], alignment: [] };
    return { result: result_0, context: context, proofData: partialProofData };
  },

Runtime Type Checks

The first part of this implementation is so-called “boilerplate” code that is generated by the compiler for every circuit. Every circuit has essentially the same code, differing only slightly depending on the number and names of the arguments, file names, and source code positions. Let’s focus on that code first:

if (args_1.length !== 2)
  throw new __compactRuntime.CompactError(`post: expected 2 arguments (as invoked from Typescript), received ${args_1.length}`);
const contextOrig_0 = args_1[0];
const newMessage_0 = args_1[1];
if (!(typeof(contextOrig_0) === 'object' && contextOrig_0.originalState != undefined && contextOrig_0.transactionContext != undefined))
  __compactRuntime.type_error('post',
                              'argument 1 (as invoked from Typescript)',
                              'bboard.compact line 26 char 1',
                              'CircuitContext',
                              contextOrig_0)
const context = { ...contextOrig_0 };

There are some runtime type checks here. First, we check that the actual number of arguments passed matches the expected number. In this case, that’s two (the second one is the circuit’s parameter newMessage and the first one is the CircuitContext parameter inserted by the compiler). If not, we’ll throw an exception.

This is one way that Compact differs from TypeScript. The JavaScript code produced by the TypeScript compiler does not check argument counts or types at run time because it assumes that the calling code has also passed the TypeScript type checker. Because the correctness of your contracts will depend on it, we do not assume that. Instead, the generated JavaScript code will perform the appropriate checks at run time.

A const binding is used to give the first argument a name based on contextOrig (always) and the second argument is named based on whatever name we used in the Compact source code. The suffixes like _0 added on variable names is the way that the compiler ensures that it always generates unique names. Then we have some runtime type checks that the first argument actually satisfies the interface for CircuitContext defined in the Compact runtime.

Finally, we copy the original CircuitContext and name it context. We do this so that we can mutate the copy without changing the original one that was passed to us.

Proof Data

The big-picture view of a post transaction is that it runs the JavaScript implementation of the circuit, with full access to the private state provided by its witnesses. Then we ask the proof server to generate a zero-knowledge (ZK) proof that the circuit ran as expected. Specifically, we prove that we know the private data required to produce the observed on-chain behavior—without revealing that private data.

In order to do that, we have to collect some information about running the circuit in the so-called “proof data”. We next initialize that data:

const partialProofData = {
  input: {
    value: _descriptor_4.toValue(newMessage_0),
    alignment: _descriptor_4.alignment()
  },
  output: undefined,
  publicTranscript: [],
  privateTranscriptOutputs: []
};

This is a JavaScript value that satisfies the TypeScript interface ProofData from the Compact runtime. To understand this, we need to look inside of the definition. That definition (from the version of the Compact runtime used by the Compact compiler version 0.24.0) is:

interface ProofData {
  input: AlignedValue;
  output: AlignedValue;
  privateTranscriptOutputs: AlignedValue[];
  publicTranscript: Op<AlignedValue>[];
}

This is called partialProofData because it will not necessarily contain all the proof data that we need. When we run the JavaScript code for the circuit off chain, some conditional branches will be skipped. In these cases, the proof will require 'dummy' data to fill in for branches that were not taken. We’ll explore that in more detail later in the series.

The initial value is kind of like the boilerplate we saw before. The properties input, output, and publicTranscript have default initial values. The initial value of input depends on the number and types of the parameters to the circuit in Compact:

From the TypeScript declaration of ProofData, we can see that input has type AlignedValue. This is the type alias AlignedValue from the Compact runtime. And from the TypeScript declaration for that we can see that it has a pair of properties, alignment and value.

Descriptors

To fully understand this, let’s take a look at the Compact runtime TypeScript declaration of AlignedValue:

type AlignedValue = {
  alignment: Alignment;
  value: Value;
};

We’ll gloss over Alignment, but it’s instructive to see what Value is:

type Value = Uint8Array[];

So an aligned value is an alignment tag of some kind and a value which is an array of Uint8Arrays. This is the ledger’s encoding of Compact values in the on-chain runtime. That’s a different encoding from the JavaScript encoding of the same value.

We have two different representations for the same values. One is native JavaScript objects. The other is a binary encoding, used in the on-chain runtime. To convert between these two representations, we use so-called “descriptors”. They are the JavaScript representation of Compact types. More specifically, they are objects implementing the CompactType interface. They have three methods: toValue to convert from a JavaScript value to an on-chain value, fromValue to convert from an on-chain value to a JavaScript value, and alignment to return the alignment of the on-chain value.

The code for the ProofData’s input (representing the post circuit’s newMessage argument) was:

input: {
  value: _descriptor_4.toValue(newMessage_0),
  alignment: _descriptor_4.alignment()
},

The compiler generates top-level const bindings in JavaScript for a number of descriptors that it has used in the generated JavaScript code. If we look at _descriptor_4 in index.cjs we see:

const _descriptor_4 = new __compactRuntime.CompactTypeOpaqueString();

It’s an instance of a descriptor for a Compact value of type Opaque<"string">. The Compact runtime defines descriptor classes for all the Compact types, such as CompactTypeOpaqueString.

The JavaScript representation of the Compact type Opaque<"string"> is as a JavaScript string and the ledger representation is as a (tagged) single-element array consisting of the JavaScript string’s UTF-8 encoding. The proof data we will build up while executing the circuit contains ledger representations of values, so we use the descriptor’s toValue method to convert the newMessage parameter to that representation.

Wrapping it Up

Finally, we have a last little bit of code:

const result_0 = this.#_post_0(context, partialProofData, newMessage_0);
partialProofData.output = { value: [], alignment: [] };
return { result: result_0, context: context, proofData: partialProofData };

This calls a method on the contract named #_post_0, which contains the actual implementation of the post circuit. What we’ve been looking at (the post method) is a mostly boilerplate wrapper around this implementation. The implementation method takes the context and passes along the arguments, along with the proof data object we’ve constructed for it.

Then after it returns, we will set the output property of the proof data. The way that’s set depends on the return type of the Compact circuit. In this case it was the Compact type [], so that’s relatively uninteresting. If you take a look at the takeDown circuit in the same contract, you will see the slightly more interesting:

partialProofData.output = { value: _descriptor_4.toValue(result_0), alignment: _descriptor_4.alignment() };

Remember that _descriptor_4 was the one for the Compact type Opaque<"string">, and the proof data has ledger values. The circuit implementation in JavaScript will return a JavaScript value, so we need to encode it into the ledger’s representation using toValue and alignment.

And finally, we return the result of the circuit invocation. Recall that this was a CircuitResults<T, []> (where T is the contract’s private state type). That interface is in the Compact runtime and it looks like:

interface CircuitResults<T, U> {
  context: CircuitContext<T>;
  proofData: ProofData;
  result: U;
}

We will look more closely into the actual circuit implementation #_post_0 in the next article in this series.

What are Wrappers For?

Why do we wrap the implementation in this way? There are a couple of reasons.

First, the mostly boilerplate code we have been looking at is used when we call the circuit from JavaScript code in our DApp. So it needs to have the extra runtime checks that we see for safety. But when we call the circuit from another Compact circuit, we do not need these type checks. The Compact type system guarantees that we don’t need extra runtime checks, so we can directly call the implementation function (such as #_post_0).

And second, when we call a Compact circuit from another one, that is considered part of the same transaction that we are constructing. So in that case, we don’t want to construct a fresh ProofData object to pass in, and we don’t want to box up the results in a CircuitResults object. We only need to do that at the outermost circuit call, coming from the DApp’s JavaScript code.

Witnesses

Let’s take a quick look at the implementation of witnesses. Our contract has one, and we expect it to be passed in when we construct the contract. The constructor for class Contract has some runtime type checking code for that:

constructor(...args_0) {
  if (args_0.length !== 1)
    throw new __compactRuntime.CompactError(`Contract constructor: expected 1 argument, received ${args_0.length}`);
  const witnesses_0 = args_0[0];
  if (typeof(witnesses_0) !== 'object')
    throw new __compactRuntime.CompactError('first (witnesses) argument to Contract constructor is not an object');
  if (typeof(witnesses_0.localSecretKey) !== 'function')
    throw new __compactRuntime.CompactError('first (witnesses) argument to Contract constructor does not contain a function-valued field named localSecretKey');
  this.witnesses = witnesses_0;

Then, probably more interesting, the witnesses are also wrapped. The contract has a method for each witness, like:

#_localSecretKey_0(context, partialProofData) {
  const witnessContext_0 = __compactRuntime.witnessContext(ledger(context.transactionContext.state), context.currentPrivateState, context.transactionContext.address);
  const [nextPrivateState_0, result_0] = this.witnesses.localSecretKey(witnessContext_0);
  context.currentPrivateState = nextPrivateState_0;
  if (!(result_0.buffer instanceof ArrayBuffer && result_0.BYTES_PER_ELEMENT === 1 && result_0.length === 32))
    __compactRuntime.type_error('localSecretKey',
                                'return value',
                                'bboard.compact line 24 char 1',
                                'Bytes<32>',
                                result_0)
  partialProofData.privateTranscriptOutputs.push({
    value: _descriptor_2.toValue(result_0),
    alignment: _descriptor_2.alignment()
  });
  return result_0;
}

The circuit wrapper post was used when we called the circuit from JavaScript code, and bypassed (by directly calling the implementation #_post_0) when we called it from Compact code. For witnesses, the situation is reversed. You can call your witnesses (like witnesses.localSecretKey) all you want from JavaScript code and we don’t care and won’t even observe it. But if you call them from a Compact circuit, we will need to know about it, so we’ll call a wrapper (like localSecretKey_0).

Your witness implementation expects to have a WitnessContext argument passed to it, so here we’ll construct one to pass in. It contains the ledger, current private state, and the contract’s address. We get the JavaScript representation of the public ledger state using the compiler-generated ledger function that we looked at in part 1 of this series.

Then we actually invoke your witness implementation, getting the result and a new private state. We mutate the context to update the private state (remember, we copied the original context before invoking a circuit implementation so it was safe to mutate the copied context).

Next, we have runtime type checks after the witness returns. This is for the same reason that we check arguments coming in to circuits from JavaScript: we don’t control the witness implementation and we need to make sure that the return value is as expected for the circuit. Compact’s type safety depends on these runtime checks.

Finally, we record the witness’s return value in the proof data's private transcript that we are building while running the outermost Compact circuit call. This is private data and so for the ZK proof we construct the proof server will need to know it (so the proof server can prove that it knows the private data).

In the next article in this series, "Compact Deep Dive Part 3: The On-Chain Runtime", we will take a closer look at the actual implementation of the post circuit from the bulletin board contract and see how it uses the on-chain runtime.