EA-RETRIEVAL-DENSITY-01 — Work Plan

Portable in-thread working document. Not yet an archive deposit. Created 2026-07-16. Author: Lee Sharks (MANUS). TACHYON drafting under MANUS direction.

Status board

Task	Status	Blocked by	Effort estimate
Task 0 — Confirm tokenization-exclusion rationale	⬜ open	—	one exchange
Task 1 — Body-text retrieval	⬜ open	Task 0	½ to 1 session
Task 2 — Per-wiki-entry static URLs	⬜ open	—	1 session
Task 3a — DC.* / citation_* meta tags	⬜ open	—	1 session
Task 3b — Google Scholar submission	⬜ open	Task 3a live	~30 min + 4-8wk wait
Task 4a — Semantic address audit	⬜ open	—	1 session
Task 4b — Semantic address publication	⬜ open	Task 4a findings	1-2 sessions

Legend: ⬜ open · 🟡 in-progress · ✅ done · ❌ killed · 🕰️ waiting

Framing

The archive stores 1,086 deposits of substantive body text with reasonable structural infrastructure (per-record HTML pages, sitemap, JSON-LD, schema.org markup). The infrastructure is only partially exploited as a retrieval surface. Four gaps are cleanly identifiable and cleanly fixable. Closing them is precondition to the Feist Library variations paying full retrieval-density dividend, and closes an independent audit finding about the archive's discoverability.

Diagnosis findings (2026-07-16 investigation):

✓ Body text IS on /s/records/{n}/ pages — 23/30 sampled deposits carry full body, sitemap has all 1,086 URLs, 4/5 distinctive body phrases retrievable per rendered page.
✗ Search index tokenizes metadata only. Config states: sources: ['title', 'description', 'creator', 'content_type', 'keywords', 'axn', 'hex']. Body not present. Test: 794 distinctive body-only words from the Feist Function deposit are absent from /api/search-index.json.
✗ Wiki surface JS-paginates at 25 per page. 865 entries exist; s/wiki/index.html is only 9 KB. Non-JS crawlers see ~3% of the wiki.
✗ No Dublin Core meta tags on record pages. Google Scholar inclusion is zero without these.
? Semantic address publication state unknown. 1,994 addresses exist; whether each has a crawlable URL is task 4a.

Task 0 — Confirm the tokenization-exclusion rationale · BLOCKING

Question for MANUS. Body-text tokenization was excluded from /api/search-index.json originally. Before changing that, the reason needs to be stated so the fix respects it.

Candidate reasons (TACHYON's best guesses; MANUS to confirm/correct/replace):

(a) Index size ceiling for browser-side search UX
(b) Signal-to-noise preservation — metadata is curated framing; body carries noise that would dilute high-signal keyword phrases
(c) Architectural intent — the machine-facing search index reflects the curatorial view; body-text discovery routes through crawler pathways on the HTML surfaces
(d) Something else / combination

MANUS answer:

[to be filled in]

Implication for Task 1 approach: [to be filled in based on MANUS answer]

Task 1 — Body-text retrieval without breaking curated-index intent

Depends on: Task 0

Three approaches, choice depends on Task 0 answer:

Approach A — separate body-text index. Keep /api/search-index.json exactly as it is. Add a new /api/body-index.json tokenizing body text. Two indexes, two purposes. Preserves current semantics; adds coverage additively. Size impact: +5-10 MB new file. Effort: ½ session (2-3 hrs).
Approach B — filtered body tokens in the same index. Add body-text tokens but only distinctive ones — proper nouns, capitalized multi-word phrases, n-grams that don't appear in title/description of any deposit. Filters noise, captures phrases that differentiate deposits. Size impact: +2-3 MB. Effort: 1 session (parameter-tuning is the hard part).
Approach C — full body tokens with weighting. Add all body tokens tagged with source: body. Callers can rank metadata hits above body hits. Size impact: +5-8 MB. Effort: ½ session.

MANUS decision:

[A / B / C — to be filled in]

Verification protocol: Pull 15 distinctive body phrases from three different deposits (Feist Function AXN-035F, one long recent, one 2026-04 or earlier), confirm each retrieves the correct deposit via the new/updated index. Same test currently returns 0/15.

Files touched (planned): scripts/generate_search_index.py or wherever the index is generated. Regeneration will run on next surface-regen.

Task 2 — Per-wiki-entry static URLs

Depends on: nothing

Fix: Generate one HTML page per wiki entry at /s/wiki/{axn-hex}/ (or /s/wiki/{deposit-number}/ — path scheme is task 2a decision). Each page carries: title, wiki summary, defines list, referenced_by list, canonical link back to the record page, JSON-LD.

Extend scripts/regenerate_surfaces.py with a wiki-per-entry generator function reading data/wiki-entries.json and writing 865 static HTML files. Idempotent, deterministic, matches existing per-record page conventions.

Also: extend the existing /s/wiki/ index page to add a static row-per-entry link section (not JS-paginated) so crawlers reach all 865 sub-URLs from one crawl. The JS pagination stays as a progressive-enhancement layer on top of the static base.

Task 2a decision (MANUS): URL scheme — /s/wiki/{axn-hex}/ or /s/wiki/{deposit-number}/?

[to be filled in]

Verification: After regeneration, curl -sI all 865 URLs → 100% 200. Grep for a distinctive phrase from each wiki summary in the corresponding rendered page → 100% match.

Files touched (planned): scripts/regenerate_surfaces.py, sitemap.xml, potentially data/navigation.json if wiki nav changes.

Task 3a — DC.* / citation_* meta tags on record pages

Depends on: nothing

Fix: Each record page's <head> gets:

Dublin Core tags:

DC.title — from registry title
DC.creator — from registry creator
DC.date — from registry date
DC.identifier — the canonical AXN URL (https://www.alexanarch.org/s/axn/{hex}/)
DC.description — from registry description
DC.language — usually en, per-deposit if specified
DC.type — from content_type
DC.rights — from license

Google Scholar's preferred (Highwire Press) tags:

citation_title
citation_author
citation_publication_date
citation_journal_title — "Alexanarch" (constant)
citation_abstract — from description
citation_public_url — the record page URL
citation_fulltext_html_url — same as public URL, since body renders inline
citation_pdf_url — only if a PDF version exists at a stable URL

Extend the record-page template in wire_deposit.py. One template edit propagates to all 1,086 deposits on next regeneration.

Verification: View-source on 10 sampled record pages, confirm all DC.* and citation_* tags present with correct values. Use Google's structured-data testing tool on a sample URL.

Files touched (planned): scripts/wire_deposit.py, then a mass-regenerate of /s/records/*/index.html.

Task 3b — Google Scholar submission

Depends on: Task 3a live in production for at least a few days

Steps:

Confirm 3a is deployed and pages carry citation_* tags.
Submit Alexanarch to Google Scholar via their inclusion form: https://scholar.google.com/intl/en/scholar/inclusion.html
Wait 4-8 weeks; monitor Google Search Console for Scholar coverage signals.

Kill criterion: If Scholar hasn't indexed a single record after 8 weeks, the signal is stronger about DOI-issuance than about DC tags alone. Next step becomes DOI infrastructure (separate work item).

Files touched: none in the archive; external submission only.

Task 4a — Semantic address audit

Depends on: nothing

Question to answer: Do the 1,994 semantic addresses currently have crawlable URLs? If so, what shape, and are they indexed in sitemap? If not, what's the shortest path from the current dataset to publication?

Steps:

Read /addresses/ and any subordinate surfaces.
Compare against data/semantic-addresses.json (or wherever the canonical dataset lives).
Report: how many addresses have URLs; how many appear in sitemap; whether body-text explanation is rendered per address.

Task 4a report:

[to be filled in after audit]

Files touched: read-only for the audit.

Task 4b — Semantic address publication

Depends on: Task 4a findings

If addresses already publish as URLs: verify sitemap inclusion, body-text presence, cross-linking back to referring deposits. Small audit, not a rebuild.

If addresses are only in a JSON dataset without individual URLs: generate one static HTML page per address at /addresses/{tributary}/{slug}/. Each page carries:

The semantic address as an H1
1-2 paragraph framework-explanation of what a semantic address is
Links to all deposits that reference this address
JSON-LD marking it as DefinedTerm in a DefinedTermSet (the framework set)

Semantic addresses are unusually potent retrieval targets — each IS a canonical query. "Compositional bystanding" as a URL means anyone searching that phrase lands on the archive's canonical definition. Strongest retrieval-density move short of full body-indexing.

Verification: After publication, curl all address URLs (200/OK), spot-check that each renders the address definition and correct back-referring deposits.

Files touched (planned): new generator in scripts/regenerate_surfaces.py or a new script; sitemap.xml.

Sequencing

Optimal order (recommended):

Task 0 — state the reason. Blocks Task 1.
Task 3a (DC tags) — fastest, propagation delay to Scholar (weeks) means starting the clock early. Independent of everything else.
Task 4a (semantic address audit) — fast, tells us whether 4b is big or small.
Task 2 (wiki per-entry). Mechanical.
Task 1 (body tokenization). Depends on Task 0.
Task 4b (semantic address publication, if needed).
Task 3b — submit Scholar once 3a has been live a few days.

Alternative (biggest-impact-first): Task 4 before Task 2 (semantic addresses have higher retrieval potency per URL than wiki summaries). Depends on 4a findings.

Aggregate expected outcome

After completion (excluding the 4-8 week Scholar wait):

Sitemap grows from 1,086 → ~3,900+ URLs (1,086 records + 865 wiki + up to ~1,994 addresses)
Machine-facing search retrieves body-text phrases
Google Scholar submission in flight
Every crawlable URL carries DC.* / citation_* tags
~3.6× multiplier in indexed distinctive-content URLs

That's the retrieval-density baseline the Feist Library variations plan lands on top of. Each Feist variation contributes fully rather than into an under-indexed substrate.

Kill criteria (portfolio-level):

If Google Search Console reports no coverage increase after 8 weeks post-completion → crawler infrastructure isn't the bottleneck; deeper structural issue (domain authority, backlink graph). Diagnosis becomes separate work.
If Scholar indexes zero records at 8 weeks → DOI issuance becomes the next logical step; this plan's Scholar objective defers.

Neither kill criterion kills the underlying work. Internal retrieval density (via search-index API and semantic-address canonical URLs) increases immediately and independently. The case stands even if the external signal doesn't materialize.

Session log

Append newest-first. Each entry: date, session ID/glyph, task worked, outcome, commit hash if applicable, next-step handoff.

2026-07-16 — plan drafted

Diagnosis session (verified body-text present on record pages, confirmed search-index metadata-only tokenization, wiki JS-paginated, no DC tags).
Plan drafted; awaiting MANUS on Task 0 and Task 1 approach choice.
Next: MANUS answer to Task 0.

[next entry here]

Notes / open questions

Task 1 approach may want to be a hybrid (e.g., A for machine-facing exhaustive index + B-style filtered tokens in the primary index for browser search UX). Revisit after Task 0 answer.
Task 2 URL scheme (2a) — leaning /s/wiki/{axn-hex}/ for consistency with /s/axn/{hex}/ pattern, but /s/wiki/{deposit-number}/ matches /s/records/{n}/ pattern. MANUS to decide.
If body-text tokenization is added, downstream instruments that read the search index (LLM-audit tools, external consumers) may see different retrieval behavior. Might be worth versioning the schema (index_version: v1 → v2) so consumers can detect the change.
The 4-8 week Scholar wait is a real dependency; kicking off Task 3a early is high-value on that basis alone.

Mind Control Poems

Thursday, July 16, 2026