Glas Intelligence
Multi-agent simulation that turns uploaded documents into stakeholder agents and runs policy scenarios.
Why there is no measurement
No accuracy number yet — and that is the honest answer
No measurement
Why not
There is no accuracy number here that I can stand behind. The repository contains proper scoring code — Brier score, log score, CRPS, Murphy decomposition, calibration curves — and it is unit-tested against textbook identities. But it has never been run against resolved real-world outcomes, so it currently scores nothing. Until a set of these forecasts has actually resolved and been graded, any accuracy claim would be exactly the kind of unbacked number this site exists to avoid.
Problem
Policy and market analysis usually stops at a single narrative: one analyst's read of how stakeholders will react. That read is unfalsifiable and it hides its own assumptions. I wanted a system where the assumptions are explicit objects — who the actors are, what they want, what they know — so a scenario can be re-run with one of them changed and the difference inspected.
Approach
Uploaded documents are chunked and pushed into a Zep Cloud temporal knowledge graph, which extracts entities and relationships. Those nodes and edges are read back to synthesise agent personas, which are written to a config and run as an OASIS (camel-ai) social simulation in an isolated subprocess. The simulated interactions are aggregated into stance distributions and LLM-generated probability triplets, which pass through a calibration guardrail layer before a ReACT-style report agent writes them up. Celery orchestrates the chain; results land in Supabase.
Components
-
Chunks uploaded documents — 500 characters with 50 of overlap — and pushes each chunk into Zep Cloud as an episode, where entity and relationship extraction happens.
-
Reads entities and edges back out of the graph and synthesises the agent personas a run is populated with — a pharmacist, a chain operator, a commissioner, a patient advocate.
-
Launches the OASIS social simulation as a subprocess with its own lifecycle and teardown, so a run that hangs or dies cannot take the Flask process down with it.
-
Aggregates the simulated posts and reactions into stance distributions, then asks an LLM for low, central and high probability triplets on each outcome of interest.
-
Samples the aggregated distributions to propagate uncertainty through to the figures the report quotes.
Wired into the live path, and has no tests
No measurement
Why not: This is the weakest part of the system, and it is not weak by being unfinished. The quantitative analysis service imports it on the running pipeline, so its output reaches the report, and there is no test_monte_carlo_engine.py anywhere in the suite. An untested module nobody calls is dead weight; an untested module on the live path is an unbacked number in every report that quotes it. Saying so here is cheaper than having someone find it in the repository I linked.
-
Rescales and orders the LLM's probability triplets, clamps them off the certainty rails, widens implausibly tight intervals, and bounds the implied likelihood ratios.
36
Tests pinning the guardrail behaviour
Method: grep -c "def test_" backend/tests/test_calibration_guardrails.py against the linked repository. Stated plainly: this is a count of tests, not a measure of correctness. It shows the guardrail behaviour is pinned against regression. It does not show that the forecasts passing through the guardrails are right, and nothing on this page claims it does.
-
Plans an outline, then writes the report section by section in a ReACT loop over graph and statistics tools, with the tool budget capped per section so retrieval cannot run away.
-
Brier score, log score, empirical CRPS, Murphy decomposition and calibration curves, unit-tested against textbook identities.
Scores nothing — never run on real outcomes
No measurement
Why not: The module is finished, and its own docstring says it is not wired into the live report pipeline; the only thing in the repository that imports it is its test file. Until a set of these forecasts has resolved and been graded through it, it produces no accuracy number. That is the same reason the project above carries no headline figure, stated at the level where the absence actually originates.
What it actually does
A user uploads source documents — consultations, financial settlements, trade
press. TextProcessor chunks them and GraphBuilder pushes each chunk into
Zep Cloud as an episode, where Zep performs the entity and relationship
extraction. That graph is the shared world model.
OasisProfileGenerator then reads nodes and edges back out and synthesises
agent personas from them: a community pharmacist, a chain operator, a
commissioner, a patient advocate. The personas are written to a config file and
an OASIS simulation is launched as a subprocess, deliberately isolated from the
Flask process, for a fixed number of rounds.
What comes back is a corpus of simulated posts and reactions.
QuantitativeAnalysisService aggregates those into stance distributions and
asks an LLM for probability triplets — low, central, high — for each outcome of
interest. A report agent then queries the graph and those statistics to produce
the written output.
The part I would defend in a review
The calibration guardrail layer. LLMs emit probability estimates that are individually plausible and collectively incoherent: intervals ordered wrongly, confidence pinned at 0 or 100, ranges narrower than the model could possibly justify, likelihood ratios that imply more evidence than exists.
So the triplets do not reach the report unchecked. They are rescaled to a common basis, sorted, clamped away from the certainty rails, widened when the stated interval is implausibly tight, and their likelihood ratios bounded. Thirty-six tests hold that behaviour in place.
This is a claim about output hygiene, not about correctness. The guardrails stop the system from stating things it has no basis for. They do not make the underlying forecast right.
What is not finished
The forecast scoring module is written but not wired into the live pipeline — its own docstring says so. The Monte Carlo engine has no test file and is imported by the quantitative analysis service on the running path, which is the worse of the two problems and the one I would fix first. Stripe is in test mode. A correlation discount constant in the aggregation is a heuristic with no formal derivation behind it, which is noted in the repository’s own task list rather than quietly left in the code.
I would rather say all of that here than have someone find it by reading the repository I linked.