Benchmarks
Signet memory benchmarks run through memorybench/. The benchmark harness owns
the datasets, checkpointing, answer generation, judging, retrieval metrics, and
reports. Signet is only a MemoryBench provider.
memorybench/ is a root workspace because it is a development harness, not a
Signet runtime package. Runtime code still belongs under platform/, human
surfaces under surfaces/, integrations under integrations/, and reusable
libraries under libs/.
Current LongMemEval score
Section titled “Current LongMemEval score”Signet’s latest tracked MemoryBench LongMemEval runs average 97.6% answer
accuracy under the rules profile. This is an average across the current
tracked local and canary score set, not a claim that every individual run lands
at exactly that value.
For the underlying run ledger and per-run accuracy, Hit@K, F1, MRR, NDCG,
latency, and context-size notes, see
docs/BENCHMARKING-PROGRESS.md.
Default developer run
Section titled “Default developer run”bun run benchThis command:
- Builds the workspace with
bun run build. - Creates a temporary isolated Signet workspace under
/tmp. - Starts a Signet daemon bound to
127.0.0.1on a free port. - Runs MemoryBench against LongMemEval using the
signetprovider. - Shuts the daemon down and removes the temporary workspace.
The default run uses a small LongMemEval sample, one question per question type, so developers can run it while iterating. The command prints the exact MemoryBench command and run id before executing.
MemoryBench reports scores by question type in report.json, so the default
run gives a clean per-type breakdown without extra commands. Benchmark reports
and run artifacts stay under ignored paths and should not be committed until the
team explicitly decides to publish a score.
Prompt-submit gate benchmark
Section titled “Prompt-submit gate benchmark”The prompt-submit hot-path benchmark compares a substantive entity-context recall with a low-signal greeting:
bun run build:corebun scripts/bench-prompt-submit.tsIt prints prompt-submit latency, recall-stage timing, and synthetic embedding call counts for the before-gate substantive path and after-gate low-signal path. The benchmark uses a temporary database and synthetic embedding delay; it never reads the live workspace.
Larger runs
Section titled “Larger runs”Run the full LongMemEval benchmark:
bun run bench -- --fullRun a fixed-size sample:
bun run bench -- --limit 20bun run bench -- --sample 3--limit and --sample select questions, not individual sessions. A small
LongMemEval question set can still ingest many sessions because each question
has its own haystack.
Run one LongMemEval question type:
bun run bench -- --type temporal-reasoning --limit 20bun run bench -- --type knowledge-update --sample 5Valid LongMemEval types include:
single-session-usersingle-session-assistantsingle-session-preferencemulti-sessiontemporal-reasoningknowledge-updateRun an explicit question set:
bun run bench -- --question-id 32260d93 --question-id 54026fcebun run bench:ingest -- --question-ids-file memorybench/config/autoresearch/longmemeval-canary-12.txt--question-id is repeatable and also accepts comma-separated ids.
--question-ids-file reads one id per line and ignores blank lines and
# comments. Use this for fixed canaries so reruns test the same questions
instead of whatever a fresh random sample happens to draw.
Skip the workspace build when you already built locally:
bun run bench -- --no-build --limit 10Keep the isolated workspace for inspection:
bun run bench -- --keep-workspace --limit 5Preview the command without building, starting the daemon, or running the benchmark:
bun run bench -- --dry-runGLM and BEAM runs
Section titled “GLM and BEAM runs”MemoryBench resolves the answering model, judge, and harness-side extraction model through one client, so a run can use Z.ai’s GLM-5.3-Flash for all three:
ZAI_API_KEY=... \SIGNET_BENCH_ANSWERING_MODEL=glm-5.3-flash \SIGNET_BENCH_JUDGE=glm-5.3-flash \MEMORYBENCH_EXTRACTION_MODEL=glm-5.3-flash \bun run bench -- --limit 20glm-5.3-flash sends Chat Completions with thinking disabled;
glm-5.3-flash-thinking enables it. Numbers produced with GLM are not
like-for-like with results published using GPT-4o or GPT-4.1-mini. Compare
Signet against the filesystem and rag baselines run with the same models.
BEAM (beam-1m, beam-10m) needs a prepared, hash-verified snapshot:
cd memorybenchbun run src/index.ts beam prepare --tiers 1Mcd ..bun run bench -- -b beam-1m --evaluation-profile custom-judge \ --data-path ./data/benchmarks/beam --dataset-revision <fingerprint> --sample 1The paper evaluation profile (default) requires gpt-4.1-mini as judge and
reports beamScore only for a complete tier. custom-judge keeps the paper’s
rubric prompts and scoring with any judge and reports beamRubricScore.
BEAM-1M is about 36M tokens across 37,315 sessions; the 20 questions of each
conversation share one ingest.
Reports record API-reported token usage for answering, judging, extraction,
and observed Dreaming passes, plus a list-price cost estimate where model
pricing is known. See memorybench/README.md for the field reference.
Two-stage local model workflow
Section titled “Two-stage local model workflow”For local tuning, keep extraction cheap and reserve the stronger model for the parts that affect reasoning quality:
- Run ingest/indexing with a small fast model.
- Continue the same checkpoint from search with the larger answer/judge model.
This avoids spending hours re-ingesting LongMemEval sessions through a 26B model while still testing answer quality with the model we care about.
Example split run:
export RUN_ID="lme-rules-split-$(date -u +%Y%m%dT%H%M%SZ)"export WORKSPACE=".bench/workspaces/longmemeval-structured"Start a fast OpenAI-compatible extraction server, for example Gemma E4B via vLLM, then ingest. On a single-GPU workstation this can use the same port as the later answer server because the phases run sequentially:
OPENAI_API_KEY=dummy \OPENAI_BASE_URL=http://127.0.0.1:8000/v1 \MEMORYBENCH_EXTRACTION_MODEL=google/gemma-4-E4B-it \MEMORYBENCH_EXTRACTION_MAX_TOKENS=1200 \MEMORYBENCH_STRUCTURED_EXTRACTION_MAX_TOKENS=1800 \SIGNET_BENCH_RUN_ID="$RUN_ID" \bun run bench:ingest -- --profile rules --no-build --workspace "$WORKSPACE" --limit 6 --concurrency-ingest 1For impatient local iteration, ingestion can use OpenRouter while answer/judge
still use a local model later. Store the key in Signet as
OPENROUTER_API_KEY, inject it into the command environment as
OPENROUTER_API_KEY, and pass --ingest-openrouter:
signet secret put OPENROUTER_API_KEYSIGNET_BENCH_RUN_ID="$RUN_ID" \MEMORYBENCH_SESSION_CONCURRENCY=4 \bun run bench:ingest -- --profile rules --ingest-openrouter --no-build --workspace "$WORKSPACE" --limit 6 --concurrency-ingest 2--ingest-openrouter only affects bench:ingest. It maps the injected
OPENROUTER_API_KEY to OPENAI_API_KEY for the MemoryBench extraction client,
sets OPENAI_BASE_URL=https://openrouter.ai/api/v1, and defaults
MEMORYBENCH_EXTRACTION_MODEL=inception/mercury-2. It does not change local
answering or judging in a later bench:evaluate step. For local dev speed, use
modest question concurrency plus MEMORYBENCH_SESSION_CONCURRENCY; this only
changes how many sessions are ingested at once, not what data is written or how
answers are scored.
Then stop the extraction server, start the larger OpenAI-compatible answer/judge server, for example Gemma 26B Q5 via llama.cpp, and continue the same run:
OPENAI_API_KEY=dummy \OPENAI_BASE_URL=http://127.0.0.1:8000/v1 \SIGNET_BENCH_ANSWERING_MODEL=google_gemma-4-26B-A4B-it-Q5_K_M.gguf \SIGNET_BENCH_JUDGE=google_gemma-4-26B-A4B-it-Q5_K_M.gguf \SIGNET_BENCH_RUN_ID="$RUN_ID" \bun run bench:evaluate -- --profile rules --no-build --workspace "$WORKSPACE"The --resume flag is a wrapper guardrail. It prevents bun run bench from
adding --force when continuing a checkpoint. Use it whenever the run already
has ingested data that should be preserved.
Benchmark workspace
Section titled “Benchmark workspace”bun run bench benchmarks a default Signet install. For every run it:
- creates a temporary root with its own
HOMEand workspace - runs
signet setup --non-interactiveinto it, with the embedding provider pinned tonativeand the extraction model set to the benchmark model - adds one inference account so the daemon can authenticate to the model provider
- starts that workspace’s daemon on a private port and runs MemoryBench against it
Pipeline, graph, reranker, autonomous, and Dreaming settings are whatever setup writes. When a benchmark shows a better value, change the product default so the next run measures it. Do not add bench-only configuration.
Setup is passed --remote-url pointing at the benchmark port so it does not
start a daemon on the default port. Only clients read that value.
The credential step changes three things in the setup-written target, because a fresh non-interactive install cannot reach a remote model otherwise:
account: a reference toSIGNET_BENCH_DREAMING_API_KEY, which the wrapper fills fromZAI_API_KEY. Setup writes no credential for a remote endpoint.privacy: restricted_remote: setup’smemory_extractiontask class requires it, and a remote target is otherwise inferred asremote_ok, which the router blocks.executor: set to the catalog provider family (defaultzai-coding-cn) with no endpoint, matching how an installed Z.ai target is configured. The generic OpenAI-compatible executor appends/v1to the Z.ai/v4base URL.
Two concurrency settings are deliberately raised, because a benchmark ingests
its whole corpus at once and each question is its own agent:
memory.dreaming.maxConcurrentPasses is 6 instead of the product default of 2,
and memory.pipelineV2.worker.maxLlmConcurrency is that value plus 2 so the
shared LLM limit does not hold passes back. Dreaming never runs more passes than
the shared LLM limit allows. Set SIGNET_BENCH_DREAMING_CONCURRENCY to measure
another value, including the default. While draining, the harness triggers a new
incremental pass whenever a slot frees up and the backlog is not yet zero, rather
than waiting for every running pass to finish. It stops after three failed
passes per slot in a row, or three passes per slot that apply no mutations.
Set SIGNET_BENCH_DREAMING_PROVIDER_FAMILY=openai-compatible with
SIGNET_BENCH_DREAMING_ENDPOINT to keep a generic endpoint, for example a local
model server.
Benchmark profiles
Section titled “Benchmark profiles”bun run bench # dreaming (default)bun run bench -- --profile rulesbun run bench -- --profile supermemory-paritydreaming is the default and the publishable profile. It uses the
signet-dreaming provider:
- each benchmark session is sent to
/api/hooks/session-endwithreason: "session_shutdown"and an inline transcript, the same path the Pi connectors use. A session-end without a boundary reason is treated as a turn and captures nothing (#2014). - each haystack gets its own agent,
memorybench-<container tag>, so one question’s evidence cannot answer another’s - after capture, the provider triggers one incremental Dreaming pass and waits until every agent’s measured episodic backlog is zero
- recall goes through
/api/memory/recallfor the haystack’s agent
rules uses the signet provider. The harness runs its own LLM extraction and
writes results through /api/memory/remember. That is not the path a user’s
memory takes, so do not publish rules numbers as Signet results.
supermemory-parity uses the signet-supermemory-parity provider. It mirrors
the upstream Supermemory adapter shape (raw session JSON, a hardcoded recall
limit of 30) to diagnose whether a gap comes from Signet or from that
adapter’s more permissive contract. It is not a publishable profile.
Supermemory adapter contract violation
Section titled “Supermemory adapter contract violation”For a fair MemoryBench run, a provider should treat the orchestrator’s
SearchOptions as the required test contract. The common search phase calls
every provider with:
limit: 10threshold: 0.3The imported Supermemory provider currently violates that provider contract. It
does not honor the required limit: 10 shape passed by the harness. Instead, it
hardcodes limit: 30, asks Supermemory to return both summaries and chunks, and
uses a provider-specific answer prompt that tells the model to prioritize those
chunks as raw source material.
This is not just a harmless implementation detail. It means the upstream Supermemory provider is not being tested under the same required adapter shape as strict providers. It can provide more retrieved items and richer raw session context to the answer model than a conforming provider receives from the common harness contract. This can especially affect incidental-fact questions, where a raw chunk may still contain a fact that an extraction-based provider dropped.
For fair reporting, use the rules profile and document the exact provider
contract used. Treat supermemory-parity as a diagnostic profile only. It
exists to measure the advantage created by Supermemory’s current non-conforming
adapter shape, not to produce a publishable score.
Isolation rules
Section titled “Isolation rules”Benchmarks must never read from or write to a user workspace. The wrapper
creates a temporary root with its own HOME and SIGNET_PATH, sets up a fresh
workspace there, and runs that workspace’s daemon on a private port.
The signet-dreaming provider captures each haystack’s sessions under its own
agent, memorybench-<question-id>-<data-source-run-id> (BEAM uses one agent per
conversation). Recall reads the same agent. Dreaming passes cover every
registered agent, and each write must cite evidence from the agent it writes
to.
Persistent tuning workspaces
Section titled “Persistent tuning workspaces”Clean benchmark runs use a fresh temporary Signet database. For development tuning, you can preserve and reuse a benchmark workspace explicitly:
bun run bench -- --workspace .bench/workspaces/longmemeval-structured --sample 1A persistent workspace keeps the Signet database under .bench/, which is
ignored by Git. Use this only for tuning and clearly label any results as
warmed or reused. Public/comparable benchmark numbers should still use a fresh
isolated workspace.
Cached ingestion for local iteration
Section titled “Cached ingestion for local iteration”It is acceptable to do one expensive bulk ingestion into a temporary benchmark workspace, then reuse that workspace while tuning retrieval, ranking, context packing, answering, or judging. Treat this as a development fixture, not a publishable benchmark score.
The safe pattern is:
export RUN_ID="lme-dev-cache-$(date -u +%Y%m%dT%H%M%SZ)"export WORKSPACE=".bench/workspaces/lme-dev-cache"
# One-time extraction + remember + indexing pass.SIGNET_BENCH_RUN_ID="$RUN_ID" \MEMORYBENCH_SESSION_CONCURRENCY=4 \bun run bench:ingest -- --no-build --workspace "$WORKSPACE" --limit 60 --concurrency-ingest 2
# Repeatable search/answer/judge passes against the same isolated DB.SIGNET_BENCH_RUN_ID="$RUN_ID" \bun run bench:evaluate -- --no-build --workspace "$WORKSPACE"Invalidate the cached workspace and ingest again whenever the thing being tuned changes what gets written to Signet:
- extraction prompt or extraction model
- structured entity/aspect/hint schema
/api/memory/rememberrequest shape- daemon graph persistence
- embedding provider, model, dimensions, or indexing behavior
- dataset selection or question sampling
Do not invalidate it for changes that only affect recall behavior after ingest, such as search thresholds, traversal, reranking, context packing, answer model, or judge model. That is the whole point of the cache: isolate ingest cost from the parts we are actively tuning.
When reporting results from a cached workspace, say so plainly. The phrase we use internally is “warmed dev workspace.” A production/comparable score must use a fresh isolated workspace and the intended production model configuration.
Autoresearch ratchet workflow
Section titled “Autoresearch ratchet workflow”The overnight random loop was useful for collecting failures, but it was not a self-improving research loop. It repeatedly sampled new twelve-question runs against the same code, so a good row could just mean the sample was easier. That kind of loop is exploration only. It can feed a failure queue, but it is not a scoreboard.
The local autoresearch workflow uses a fixed LongMemEval canary instead:
memorybench/config/autoresearch/longmemeval-canary-12.txtThe canary mixes known failure targets with stable controls across single-session preference, temporal reasoning, knowledge update, multi-session, single-session assistant, and single-session user questions. Keep that file stable unless we intentionally reset the scoreboard.
The ratchet rule is boring on purpose:
- Pick one hypothesis.
- Make the smallest code or prompt change that tests it.
- Run the fixed canary against the same model split.
- Keep the patch only if the canary improves without known regressions.
- If a random exploration run finds a new skeleton, add it to the failure queue or propose a canary update separately.
Use the helper script for the mechanics:
bun scripts/autoresearch-memorybench.ts statusbun scripts/autoresearch-memorybench.ts idsbun scripts/autoresearch-memorybench.ts triage --run-id <run-id> --write-queuebun scripts/autoresearch-memorybench.ts compare --base <old-run-id> --candidate <new-run-id>bun scripts/autoresearch-memorybench.ts run-canaryrun-canary prints the exact two-stage local commands by default. Add
--execute only when the local OpenAI-compatible model server is already
running. The default local split is Gemma 4 E4B for ingestion through vLLM,
Gemma 4 26B Q5 for answer/judge through llama.cpp, and Ollama
nomic-embed-text for embeddings. Answer and judge phases run with
concurrency 1 by default because the local llama.cpp 26B Q5 server is usually
started with one slot. It does not use OpenRouter.
bun scripts/autoresearch-memorybench.ts run-canary --executeIf vLLM and llama.cpp cannot be resident at the same time, run the phases separately:
bun scripts/autoresearch-memorybench.ts run-canary --ingest-only --execute# swap vLLM for llama.cppbun scripts/autoresearch-memorybench.ts run-canary --skip-ingest --execute --run-id <same-run-id>If they are on separate ports, pass --ingest-base-url and --answer-base-url.
If the answer server has more than one safe slot, pass --answer-concurrency
and --evaluate-concurrency; otherwise leave them at the default.
If a change only affects recall, ranking, context packing, answer prompting, or judging, reuse the warmed canary workspace and skip ingestion:
bun scripts/autoresearch-memorybench.ts run-canary --skip-ingest --executeIf a change affects extraction, the structured remember request shape, graph storage, embeddings, or indexing, invalidate the workspace and ingest again. That is the line between honest fast iteration and fooling ourselves. No need to laminate the pancake.
Progress log
Section titled “Progress log”Development run history and score comparisons live in docs/BENCHMARKING-PROGRESS.md. Keep this file focused on how to run benchmarks and what the harness measures.
What is being measured
Section titled “What is being measured”The default signet provider uses the public Signet daemon HTTP API:
- ingest:
POST /api/memory/remember - search:
POST /api/memory/recallwithexpand: true - health:
GET /health
The default rules profile is retained as a historical structured-ingest
baseline. During ingest it performs MemoryBench-side structured extraction
from each session, then calls the full remember endpoint with:
- extracted memory content
- structured entities
- structured aspects and attributes
- hint questions
- source metadata and per-question scope
- the lossless source transcript
The dreaming profile is the cutover-quality path: it saves only lossless raw
episodic sessions, waits until the full selected corpus is durable, then
triggers exactly one bounded daemon Dreaming pass before retrieval. Its agent
model is explicit (SIGNET_BENCH_DREAMING_MODEL) and travels through the same
daemon router as production. It is the profile used for absolute Dreaming
quality measurements; rules is not a substitute for it.
Retrieval uplift ablation
Section titled “Retrieval uplift ablation”The first pass measures the shipped defaults, with graph boost and traversal
on. For a controlled ablation, keep one workspace and run id, then turn the
graph off in that workspace and restart from the search phase. Both retrieval
passes read the same episodic and semantic state.
export RUN_ID="dreaming-uplift-$(date -u +%Y%m%dT%H%M%SZ)"export WORKSPACE=".bench/workspaces/dreaming-uplift"
# Capture the default (graph-on) report before reusing the run checkpoint.SIGNET_BENCH_RUN_ID="$RUN_ID" \bun scripts/bench-memory.ts --workspace "$WORKSPACE" --fullcp "memorybench/data/runs/$RUN_ID/report.json" "memorybench/data/runs/$RUN_ID/report-graph-on.json"
# Set memory.pipelineV2.graph.enabled and memory.pipelineV2.traversal.enabled# to false in "$WORKSPACE/agents/agent.yaml", then re-run only# search → answer → evaluate → report against the unchanged DB.SIGNET_BENCH_RUN_ID="$RUN_ID" \bun scripts/bench-memory.ts --workspace "$WORKSPACE" --resume -r "$RUN_ID" -f searchcp "memorybench/data/runs/$RUN_ID/report.json" "memorybench/data/runs/$RUN_ID/report-graph-off.json"Compare LongMemEval retrieval aggregates (hit@k, recall, MRR, NDCG), not
answer-model scores alone.
The isolated daemon does not run legacy extraction or structural workers for
benchmark ingestion. Graph and traversal are enabled only so recall can use
the semantic state created by the selected profile. In the baseline that state
comes from the explicit structured payload; in dreaming, it comes only from
the post-ingest Dreaming pass. Recall treats active structured rows as a
first-class candidate source by searching entity names,
aspects, group keys, claim keys, attribute kinds, and attribute content before
SEC reranking. This is deliberately generic: bridges may connect broad query
classes like music, service, brand, or currentness to nearby vocabulary, but the
daemon must not contain LongMemEval answer names or dataset-specific product
terms. Prospective hint recall stays enabled because those hints are part of the
structured remember payload, not a background extraction shortcut. Recall may
attach a bounded transcript excerpt to a retrieved memory, or add a
low-confidence transcript-only supplemental hit, when expand: true is
requested. Those snippets are returned by the recall API and capped so they
cannot outrank real memory evidence; the benchmark harness does not append raw
transcripts on its own.
The isolated daemon also disables structural backfill/classification. No daemon repair job may infer benchmark graph state after the fact.
Structured remembering contract
Section titled “Structured remembering contract”Structured remembering is not “proper noun extraction.” Proper nouns are an easy case, but the graph contract is broader and simpler:
- entity: a durable referent that can be expanded
- aspect: a stable facet of that entity
- group key: a navigable subgroup inside an aspect
- claim key: the specific updateable claim slot within a group
- attribute: a sourced claim value attached to that claim slot
For benchmark memories, generic personal facts should not be dropped just because they are not named products or places. They should be attached to a scoped subject entity. For example:
Entity: MemoryBench User <question-scope>Aspect: music preferencesAttribute: has been listening to Arctic Monkeys and The Neighbourhood on Spotify lately
Aspect: commute routineAttribute: commute to work takes about 30 minutes
Aspect: morning routineAttribute: getting ready takes about one hour
Aspect: dining historyGroup key: restaurantsClaim key: korean_restaurants_tried_countAttribute: tried three Korean restaurants recently on 2023-08-11Attribute: tried four Korean restaurants so far on 2023-09-30The remember endpoint accepts structured data directly. If an aspect references an entity that was not present in the relation list, the daemon creates that entity and attaches the aspect/attributes in the same transaction. This lets the client be explicit about what is being saved and why, without relying on the background pipeline to invent graph structure.
Temporal and update-sensitive attributes should preserve words like
currently, recently, previously, counts, dates, and before/after
relationships. The storage schema has active/superseded status fields, so
structured benchmark ingestion sends each session source timestamp into the
remember endpoint. Supersession requires the same scoped entity, the same
aspect, the same group key, and the same claim key. Attributes without a claim
key are saved as ordinary evidence and do not automatically supersede sibling
attributes. When a newer structured attribute conflicts with an older attribute
on the same grouped claim key, the daemon marks the older attribute as
superseded and records the replacement attribute id. Recall then dampens
memories whose structured facts are only stale and annotates returned context
with a [Signet currentness] note so the answer model can prefer the current
replacement instead of guessing from two conflicting memories.
MemoryBench still performs the answer and judge phases itself. This keeps the benchmark comparable with the other providers and avoids benchmark-specific changes to MemoryBench scoring logic.
Environment knobs
Section titled “Environment knobs”SIGNET_BENCH_FULL=1 Run the full benchmark by default.SIGNET_BENCH_SKIP_BUILD=1 Skip `bun run build`.SIGNET_BENCH_KEEP_WORKSPACE=1 Keep the isolated workspace after the run.SIGNET_BENCH_PROFILE=<profile> dreaming, rules, or supermemory-parity; default dreaming.SIGNET_BENCH_RUN_ID=<id> Override the MemoryBench run id.SIGNET_BENCH_JUDGE=<model> Default judge model, default glm-5.3-flash.SIGNET_BENCH_ANSWERING_MODEL=<m> Default answering model, default glm-5.3-flash.SIGNET_BENCH_SAMPLE_PER_TYPE=<n> Default dev sample size, default 1.SIGNET_BENCH_AGENT_ID=<id> Signet agent scope, default memorybench.SIGNET_BENCH_PROJECT=<name> Signet project scope, default memorybench.SIGNET_BENCH_REQUEST_TIMEOUT_MS=<n> Daemon request timeout, default 60000.SIGNET_BENCH_SESSION_CONCURRENCY=<n> Per-question session ingest concurrency, default 1, max 16.SIGNET_BENCH_INGEST_OPENROUTER=1 Use OpenRouter defaults for bench:ingest.SIGNET_BENCH_OPENROUTER_MODEL=<m> OpenRouter extraction model, default inception/mercury-2.SIGNET_BENCH_OPENROUTER_BASE_URL=<u> OpenRouter-compatible base URL override.SIGNET_BENCH_DREAMING_MODEL=<id> Benchmark model for setup and Dreaming, default glm-5.3-flash.SIGNET_BENCH_DREAMING_ENDPOINT=<u> Model endpoint, default https://open.bigmodel.cn/api/coding/paas/v4.SIGNET_BENCH_DREAMING_API_KEY=<key> Daemon credential; defaults to ZAI_API_KEY from memorybench/.env.SIGNET_BENCH_DREAMING_PROVIDER_FAMILY=<name> Pi catalog provider family, default zai-coding-cn; openai-compatible keeps the endpoint.SIGNET_BENCH_DREAMING_WAIT_SECS=<n> Max time to drain the Dreaming backlog, default 720.SIGNET_BENCH_DREAMING_CONCURRENCY=<n> Concurrent Dreaming passes for bulk ingest, default 6, max 16.SIGNET_BENCH_DREAMING_CODEMODE=1 Run Dreaming with memory.dreaming.codemode on (0 forces it off; unset keeps the product default).SIGNET_BENCH_EMBEDDING_WAIT_SECS=<n> Max time to wait for Dreaming's derived memories to be embedded before retrieval, default 1800.SIGNET_BENCH_CAPTURE_WAIT_SECS=<n> Max time to wait for a question's transcript captures to finish during indexing, default 1800.MEMORYBENCH_EXTRACTION_MODEL=<m> Structured extraction model, default gpt-4o.MEMORYBENCH_EXTRACTION_MAX_TOKENS=<n> Markdown extraction cap, default 1200.MEMORYBENCH_STRUCTURED_EXTRACTION_MAX_TOKENS=<n> Structured JSON extraction cap, default 1800.MEMORYBENCH_SESSION_CONCURRENCY=<n> Per-question session ingest concurrency, default 1, max 16.OPENROUTER_API_KEY Preferred injected env var for OpenRouter ingestion.OPENAI_BASE_URL OpenAI-compatible API base URL.ZAI_API_KEY Z.ai key for glm-* answering, judge, and extraction models.ZAI_BASE_URL Z.ai base URL, default https://open.bigmodel.cn/api/coding/paas/v4.Reports
Section titled “Reports”MemoryBench writes checkpoints and reports under memorybench/data/runs/.
That directory is ignored by Git. Reports should be attached to PRs or release
notes only when benchmark numbers are being used to justify a memory-system
change.

