Vector space, and the three graphs it holds

What a vector actually is, and why anyone bothered

If "vector embedding" means nothing to you yet, this section is the whole idea in about ten minutes. Everything after it is the same idea with the real machinery bolted on.

The problem: a computer matches characters, not meaning

Ordinary search compares letters. You type words, it finds documents containing those words. That works right up until the moment someone words a question differently from the way the answer was written — which is nearly always.

So we need a way to compare meanings rather than characters. Computers only compare numbers well. So: turn meaning into numbers.

Do it by hand first

Forget machine learning for a minute. Suppose you had to describe security topics to a computer using only three dials, each running from −1 to +1:

Now every topic becomes three numbers. Phishing is strongly offensive, strongly about people, moderately hands-on → (−0.8, +0.9, +0.3). Firewall rules is defensive, about machines, hands-on → (+0.9, −0.85, +0.6).

Those three numbers are a vector. That's the entire word. A vector is an ordered list of numbers describing one thing. Each number is a dimension. Three numbers means three dimensions — which happens to be exactly what fits in a room, so we can actually look at it.

Drag it around. Notice what happened without anyone arranging it: phishing and pretexting ended up next to each other, and both ended up far from buffer overflow — not because a human filed them that way, but because the numbers put them there. Meaning became geometry.

Place your own idea in the space

Set the three dials to describe something, and the space will tell you what it sits closest to. This is, in miniature, exactly what happens to every question a student types.

"Close" turns out to mean angle, not distance

Two descriptions of the same idea — one terse, one rambling — point the same direction from the origin, but the long one lands further out. If you measured plain distance, verbosity would look like disagreement. If you measure the angle between the two arrows, length stops mattering and only direction counts.

The cosine of that angle is a single number: 1.0 when two things point identically, 0 when they are unrelated, negative when they point apart. That number is called cosine similarity, and it is the only comparison this entire system makes. Section 2 has a widget you can drag.

Retrieval, in three steps

That is the whole retrieval story. Everything in sections 3 through 5 is refinement on step 3 — which candidates are allowed to compete, how ties break, and what to do when the geometry is wrong.

So why 1,536 dials instead of 3?

Because three dials collide. In the hand-made space above, port scanning and memory forensics both come out as machine-facing and hands-on; the dials have no way to say that one is reconnaissance and the other is evidence recovery. Add a dial for that, and two more things collide somewhere else.

The fix is more dials. text-embedding-3-small uses 1,536 of them — and critically, nobody chose what they mean. The model worked them out from a very large amount of text, and no human can name dial 412. That is the trade: you lose interpretability, and in exchange the space is roomy enough that genuinely different ideas land in genuinely different places.

Vocabulary, once

That's the foundation. From here on the page uses the real corpus and the real code, and gets specific.

A vector is an opinion about meaning, written as coordinates

Embedding is one idea: hand text to a model, get back a fixed-length list of numbers. Same length every time, whatever you put in.

LearningNuggets uses OpenAI text-embedding-3-small. Input is truncated at 2,000 characters; output is 1,536 floating-point numbers. That never varies — a three-word pattern name and a 500-character teaching observation both come back as 1,536 numbers.

# the embedding helper — one function, called from every write and read path
EMBEDDING_MODEL = "text-embedding-3-small"
_MAX_INPUT_CHARS = 2000

async def embed_text(text: str) -> list[float]:
    result = await get_embedder().embed_documents([text[:_MAX_INPUT_CHARS]])
    return list(result.embeddings[0])   # len == 1536, always

The model was trained so that text meaning similar things lands in similar places. That's the whole contract. Nobody assigned axis 412 to "cryptography" — the coordinates are opaque, and the only thing you can do with them is compare them.

# the turn orchestrator — what actually gets embedded at query time
context_text = (f"topic:{state.topic_title} level:{state.proficiency_level} "
                f"focus:{state.session_state.get('current_focus','none')} "
                f"student:{state.student_message[:500]}")
embedding = await embed_text(context_text)   # ONE vector, reused by all four passes

"Close" is an angle, not a distance

pgvector's <=> operator is cosine distance. Similarity is 1 - (a <=> b) — the cosine of the angle between the two vectors, ignoring their lengths entirely.

-- the real ORDER BY inside match_hag_entries
ORDER BY (1 - (e.embedding <=> p_query_embedding))
       * CASE e.maturity WHEN 'proven' THEN 1.1
                       WHEN 'pattern' THEN 1.05
                       ELSE 1.0 END DESC

The index backing this is ivfflat (embedding vector_cosine_ops) WITH (lists = 50) — approximate nearest neighbour, so retrieval stays sub-linear as the corpus grows past a few thousand rows.

The real corpus, flattened onto a screen

56 production entries. Positions come from classical MDS over the true cosine distances — so proximity on screen really is semantic proximity, within the limits of two dimensions.

Even so, the clustering is not an artefact. The nearest neighbour of forensics/encryption-bypass-memory-extraction is forensics/live-vs-dead-acquisition at cosine 0.841. The nearest neighbour of the student pattern visual-learner-mapping is visual-model-builder at 0.731 — two learners, two sessions, two different topics, and the extractor named the trait differently each time. Geometry caught what the string match missed. That is exactly the failure mode the student_pattern_dedup_threshold dial exists to absorb; it has been tuned down from 0.85 to 0.72 in production for this reason.

Retrieval is a filter, then a sort — and the filter isn't on similarity

The single most common misreading of this system: match_hag_entries has no similarity threshold. Weak matches come back. What gets filtered is confidence, per maturity tier.

WHERE e.maturity != 'archived'
  AND e.confidence >= GREATEST(
        p_min_confidence,                       -- caller's global floor (0.4)
        CASE e.maturity WHEN 'proven'    THEN p_floor_proven      -- 0.30
                        WHEN 'pattern'   THEN p_floor_pattern     -- 0.40
                        WHEN 'candidate' THEN p_floor_candidate   -- 0.70
                        ELSE 1.0 END)                    -- unknown maturity locked out
  AND (p_topic_id IS NULL OR e.topic_id = p_topic_id)
  AND e.visibility_scope = ANY(p_visibility_scopes)

An unproven entry has to be much more confident than a proven one to get through — 0.70 against 0.30. That asymmetry is the entire quality gate. A candidate row is something the system believes but hasn't watched work yet, so it must be nearly certain to be allowed near a student.

Note what the boost does and does not do. It sits inside the ORDER BY — it changes which five rows survive the LIMIT, but the score the orchestrator later reads for routing is plain confidence × similarity. Ranking multipliers and decision multipliers are deliberately different quantities.

Bands are labels on the prompt block, not filters. Nugget is told how confident the recall is and left to weigh it. Advisory text is clipped to 400 characters at a sentence boundary before rendering — pre-clip, the topic-memory block averaged 5,108 characters per turn.

When geometry lies, use a name

Similarity search is a heuristic. It has no notion of identity — only of resemblance. So the system runs a second, non-geometric pass in parallel.

match_hag_entries_by_concept never touches an embedding. It joins on concept_slug and returns 1.0::double precision AS similarity — literally, by fiat. If the orchestrator already knows which concepts this turn is about (from the last three turn_memories rows), an exact anchor beats a heuristic every time.

WHERE e.concept_slug = ANY(p_concept_slugs)   -- the join key IS the identity
ORDER BY CASE e.maturity WHEN 'proven' THEN 1.1 WHEN 'pattern' THEN 1.05 ELSE 1.0 END
       * e.confidence DESC, e.last_matched_at DESC NULLS LAST

Both passes fire concurrently, results merge deduped by entry id — semantic wins collisions, because only the semantic pass carries the graph edges. Then everything is re-ranked by a composite score:

score = similarity × confidence × maturity_weight × concept_match_bonus
        #  1.0 for concept-anchored rows        ↑ 1.30 when the slug is in the expected set

Concept identity itself is resolved by a four-step ladder, and the step you land on is recorded on the row as concept_match_quality — which then multiplies the initial confidence. A weakly-anchored observation starts life weak.

You can see the cost of a miss in the corpus above: several high-value entries carry _unanchored_. They still retrieve semantically — one of them has 125 successful outcomes — but they can never be reached by name, never dedupe against a sibling, and never participate in a concept-anchored pass.

One table, three graphs

All of it lives in hag_entries. What separates the graphs is a single text column — visibility_scope — and, more consequentially, three different uniqueness indexes.

The indexes are where the architecture actually lives. Identity for a topic fact is (topic, concept, scope) — the same concept in two topics is two rows. Identity for a student pattern drops topic_id entirely: (learner, pattern), one row per trait per person, reinforced across every topic they touch. Identity for a visual record is (concept) alone — one drawing per concept, platform-wide.

-- migration 081: widened from 2 columns after a production outage
CREATE UNIQUE INDEX hag_entries_topic_concept_slug_uniq
  ON hag_entries (topic_id, concept_slug, visibility_scope);

-- migration 112: topic deliberately absent — a trait is not topic-bound
CREATE UNIQUE INDEX hag_entries_student_pattern_uniq
  ON hag_entries (visibility_scope, concept_slug)
  WHERE visibility_scope LIKE 'student:%' AND concept_slug <> '_unanchored_';

-- migration 184: cross-topic, one curated drawing per concept
CREATE UNIQUE INDEX hag_entries_visual_concept_uniq
  ON hag_entries (concept_slug) WHERE visibility_scope = 'visual';

That difference propagates all the way to retrieval. The topic pass is scoped: p_topic_id = <current>. The tutor and student passes pass p_topic_id = NULL — deliberately cross-topic, because a teaching move that worked in Linux is worth trying in cryptography, and a learner who tests metaphors to breaking does it everywhere. On those rows topic_id survives as provenance only.

Four concurrent round-trips on a cache miss, but only the topic pair competes for the top five. Tutor and student results are appended after ranking, then partitioned back apart at render time by scope prefix:

# the split predicate is literally the scope prefix
topic_advisories   = [a for a in state.hag_advisories
                      if a.visibility_scope != "tutor"
                      and not a.visibility_scope.startswith("student:")]
tutor_advisories   = [a for a in state.hag_advisories if a.visibility_scope == "tutor"]
student_advisories = [a for a in state.hag_advisories if a.visibility_scope.startswith("student:")]

Which become three labelled blocks in the character prompt: ## TOPIC MEMORY, ## TUTOR MEMORY, ## STUDENT PATTERNS. The visual bucket never appears there at all — it serves the artist, through its own RPC, straight to the whiteboard.

Dependency edges: the part geometry can't do

Cosine similarity is symmetric and has no direction. It cannot express "you must understand this before that" — so that lives in a separate table, hag_edges, with five typed relations.

This is the real Cryptography graph: 29 public entries, 97 edges, every edge type present, and all three provenance classes visible at once. Filter by origin below to watch the graph decompose into the part a human authored, the part a model proposed from the content, and the part students taught it.

The traversal is deterministic — no model involved

WHERE e.source_entry_id = p_entry_id
  AND e.confidence >= p_min_confidence              -- 0.3 default
  AND he.maturity <> 'archived'
  AND (p_include_candidates OR he.maturity <> 'candidate')
ORDER BY CASE e.edge_type
           WHEN 'prerequisite' THEN 1  WHEN 'fallback' THEN 2  WHEN 'leads_to' THEN 3
           WHEN 'related' THEN 4      WHEN 'supersedes' THEN 5 END,
         e.confidence DESC

That ordering is a teaching policy encoded as a sort key. Prerequisite first — check the floor before building on it. Fallback second — if the student is struggling, the escape hatch outranks the next step. leads_to third — forward motion only after the first two are satisfied.

Two dials shape what actually gets walked. edge_walk_top_k is 2: the top two retrieved entries get a decision tree, not just the first — one match's graph is a thin basis for a turn. And edge_include_candidate_targets is on, which flips p_include_candidates to true, so an edge may now point at an unproven entry. That is a deliberate loosening: refusing to walk into candidates meant a freshly written entry could never be reached through the graph, only through similarity — and the graph is precisely where a new entry's context lives.

The result lands in the orchestrator prompt as three plain lines:

Graph edges (decision tree from top match):
  prereq:   <target_instruction[:100]>
  fallback: <target_instruction[:100]> (when: <condition>)
  next:     <target_instruction[:100]> (when: <condition>)

How an edge is earned, then judged

At session end the worker runs three steps in order, all writing through the same upsert. Discovery walks the session's orchestrator decisions for sequence and recovery. Prerequisite proposals collects any causal-struggle evidence the per-turn extractor flagged. Reinforcement runs last, so same-session corroboration lands before anything is graded.

# a new edge — probationary but immediately usable
INSERT {confidence: 0.3, hit_count: 1, source_type: "learned",
        source_session_id: session_id}          # provenance, added in migration 237

# re-observed — corroboration raises a FLOOR, it never lowers a reinforced edge
new_confidence = max(current_confidence,
                     0.6 if hit_count >= 3 else 0.3)

0.3 is exactly the traversal floor, so a new edge is visible from the moment it exists. Three independent observations lift it to 0.6. That max() is doing real work — an edge that has been reinforced up to 0.75 does not get knocked back down to 0.6 by a routine corroboration bump.

Then the loop closes. When an edge's rendered line actually reaches the orchestrator prompt, its id is recorded. At session end that edge is graded by the outcome of the turn it informed:

if informing_turn_succeeded: confidence = min(1.0, confidence + 0.05)
else:                        confidence = max(0.05, confidence - 0.05)

Seed edges are reinforced on exactly the same terms as learned ones — an expert's claim about what comes first is still a claim, and it gets evaluated. The floor is 0.05 rather than 0, deliberately: a weakening edge stays visible to the cull sweep instead of pinning at zero and disappearing from view.

A daily sweep then retires the dead wood — learned and enriched edges only, below 0.35 confidence, never once traversed, and at least 90 days old. Edges have no maturity column, so retirement is a hard delete, audited before the fact. Seed edges are never culled, and the protection is a positive allowlist rather than a "not seed" test, so any future provenance class is protected by default until someone explicitly opts it in.

Three ways an edge gets made

Every edge carries a source_type, and it is the most informative column in the table. It records not how confident the system is, but what kind of evidence put the edge there.

The distinction matters more than it looks. An enriched prerequisite is a hypothesis — plausible, cheap, and generated at scale by a small model reading concept descriptions. A learned one is a claim about a person. Collapsing them into "the graph learned it" would be the single most misleading thing this page could say, so they are kept as separate provenance classes and enter at different confidences.

This is the resolution of a question this page asked in an earlier version: which pedagogical relations are learnable from observation, and which are irreducibly editorial? The answer the system now encodes is that sequence is observable, recovery is observable, causation is observable only when someone says it out loud, and kinship is not observable at all. Prerequisite discovery deliberately refuses to infer from co-occurrence — the extractor prompt says so in as many words: "Do NOT propose from co-occurrence, general difficulty, or your own domain knowledge of what usually comes first — only from explicit struggle evidence in THIS conversation."

Worth seeing in the graph above: buffer-overflow, rsa, steganography and hashing-for-integrity are concept-drift residue — entries minted at runtime when the anchoring ladder found no canonical match. They used to sit unconnected. The enrichment pass found them and wired them in, which is exactly the job: a model reading content can rescue an orphan that no author knew existed. _unanchored_ is still isolated, and always will be — it has no identity to reason about.

One more artefact worth seeing in the graph above: node buffer-overflow is sitting in the Cryptography topic, along with a bare cryptography node and an _unanchored_ one. Those are real concept-drift residue from live sessions — the extractor named a concept, the ladder found no canonical match, and step 3 minted a new node. They retrieve; they just don't connect to anything.

Entries earn their place

Every entry enters as a candidate and moves on evidence. Confidence updates are asymmetric and saturating — no entry ever reaches exactly 0 or 1 by reinforcement alone.

learning_rate = 0.10 if maturity == "proven" else 0.20   # proven moves at half speed
if success:  new = old + lr * (1 - old) * strength    # proportional to remaining headroom
else:        new = old - lr * old * strength          # proportional to the current value

There is no time decay anywhere. Confidence moves only when an outcome fires. That leaves a gap — a genuine one-shot insight nobody ever contradicted would sit at candidate forever — so a daily sweep closes it: entries at least 14 days old with confidence ≥ 0.50 and failure_count = 0 are promoted to pattern. Survival by attrition, deliberately.

Entries and edges now run parallel lifecycles, and they are not the same shape. An entry has a maturity ladder — candidate, pattern, proven — and retirement means archiving. An edge has no maturity at all: it has confidence, a corroboration count, and a hard delete at the end. That asymmetry is deliberate. An entry is a claim worth keeping a record of even once it is wrong; an edge is a piece of wiring, and dead wiring is just clutter.

Every threshold on this page is a row in hag_settings — 51 of them now, up from 37, with 13 of the new ones governing the edge lifecycle alone. All are admin-editable at runtime with a 60-second cache. No HAG threshold is hardcoded anywhere in production code, and a CI test enumerates every key and asserts a call site exists for it: a dial without a consumer is documentation theatre and fails the build.