Upvendo Knowledge Base

Appearance

Knowledge Indexing Process ​

How documentation in this repo becomes Emily's searchable knowledge base.

Overview ​

┌─────────────────┐     ┌─────────────────┐     ┌──────────────────────┐
│   Markdown      │     │   Extraction    │     │  upvendo-backend     │
│   Documentation │ ──▶ │  (extract.js)   │ ──▶ │  KB ingest → Postgres│
└─────────────────┘     └─────────────────┘     └──────────────────────┘
                                                           │
                                                           ▼
                                                  ┌─────────────────┐
                                                  │     Emily       │
                                                  │   Assistant     │
                                                  └─────────────────┘

The backend owns embedding and search (upvendo-backend docs/kb-search.md). This repo only extracts Markdown into chunks and sends them. Azure AI Search was shut down on 2026-09-28; azure-export/ keeps the old index definitions as a historical snapshot.

Step 1: Documentation ​

All knowledge starts as markdown files in this repository:

upvendo-kb/
├── shared/             # → audience `shared` (developer/ excluded)
│   ├── features/       # Feature documentation
│   ├── fields/         # Field reference
│   ├── guides/         # How-to, FAQ, glossary
│   ├── integrations/   # Per-provider docs
│   ├── relations/      # Cross-system relations
│   ├── troubleshooting/# Issue guides
│   └── developer/      # NOT indexed — read directly by Upforge / Claude Code
├── merchant/           # → audience `merchant`
├── reseller/           # → audience `reseller`
└── docs/               # → audience `shared` (this file included)

Documentation Standards ​

  1. Use consistent structure (see templates)
  2. Include all required sections
  3. Write for clarity (a child should understand)
  4. Provide examples
  5. Document customer impact

Step 2: Extraction ​

bash
npm run extract        # node scripts/extract.js

extract.js walks KNOWLEDGE_STRUCTURE (the shared/, merchant/, reseller/, global-admin/ contexts) plus ROOT_DIRS = ['docs'], parses each Markdown file into structured JSON entries, and tags every entry with its context. category is set to the source folder name (e.g. features, onboarding), not a fixed enum. Output: .knowledge/entries.json (plus one entries-{context}.json per context).

There is no content-generation step and no translation step.

Step 3: Ingest to the backend Postgres KB ​

bash
node scripts/ingest-to-backend.js --env test|prod --full                     # full sync
node scripts/ingest-to-backend.js --env prod --files "a.md" --deleted "b.md" # incremental
# add --dry-run to print the batch plan and validate every chunk without calling

Reads .knowledge/entries.json (the output of Step 2) and POSTs it to the ingest URL (/api/internal/kb/ingest on upvendo-backend, a Bearer ingest token). The backend embeds, so this step sends no vectors.

  • Mapping: each extracted entry becomes a chunk (toChunk()), with an audience derived from its context: shared, merchant, reseller, or admin for global-admin; anything else falls back to shared. The backend 422s on any key outside ALLOWED_KEYS.
  • content_hash: sha256 of the JSON of every other chunk field in a fixed order. The backend skips an unchanged hash entirely (no re-embed, no rewrite), so every stored field must be in it.
  • Batches: ≤100 chunks, a file's chunks kept together. Incremental batches name their files in delete_source_files, which drops sections a file no longer has; --deleted files go in extra chunk-less requests. --full ends with a chunk-less replace_all_for_env request whose replace_before is the first response's synced_at, deleting every row the run did not send. A failed batch aborts the run before that final prune.
  • Retries: 429, 5xx, network errors and timeouts, up to 5 times with backoff. Any other 4xx fails the run.
  • Secrets: the ingest URL and the ingest token, set per GitHub environment (testing, production). Without them the script exits non-zero, so a misconfigured environment fails the workflow instead of silently skipping the sync. --dry-run needs neither.

{env} is test for the testing branch and prod for production.

Automation ​

Two workflows run on push to testing or production:

  • .github/workflows/update-knowledge.yml (any *.md except README.md): detect added/modified and deleted files → extract → ingest (--files + --deleted) → invalidate the proxy setup-steps cache → summary → Slack.
  • .github/workflows/index-knowledge.yml (shared/, merchant/, reseller/, global-admin/, docs/): determine changed files → determine env (test / prod) → extract → ingest (--files) → invalidate cache → summary → Slack.

Both skip when the push touches no indexable file (e.g. shared/developer/** only), using extract.js --filter-indexable.

Manual Trigger ​

Run a workflow from the GitHub Actions UI via workflow_dispatch: update-knowledge.yml takes mode (incremental / full) and environment; index-knowledge.yml takes full_reindex. A full sync replaces every row for the environment, which also removes anything that should never have been indexed.

Verification ​

  1. Check the ingest step's log: it prints the request plan and the backend's per-batch upserted / unchanged / embedded / deleted totals.
  2. Ask Emily questions about the indexed content:
    • "How do I set up order capacity?"
    • "What does the time slot duration field do?"
    • "Why are customers not seeing available times?"

Troubleshooting ​

Entries Not Appearing ​

  1. Check the ingest step's log for a 4xx (bad chunk) or exhausted retries
  2. Run node scripts/ingest-to-backend.js --env test --full --dry-run locally after npm run extract -- --full to validate every chunk
  3. Look for ⚠️ No entries extracted for: … — extract.js produced nothing for that file, so its existing rows were left untouched