Knowledge Indexing Process

How to convert documentation into Emily's searchable knowledge base.

Overview

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│   Markdown      │     │   Extraction    │     │  Azure Search   │
│   Documentation │ ──▶ │   & Embeddings  │ ──▶ │   Vector Store  │
└─────────────────┘     └─────────────────┘     └─────────────────┘
                                                        │
                                                        ▼
                                                ┌─────────────────┐
                                                │     Emily       │
                                                │   Assistant     │
                                                └─────────────────┘

Step 1: Documentation

All knowledge starts as markdown files in this repository:

upvendo-kb/
├── shared/             # → emily-kb-shared-{env} (developer/ excluded)
│   ├── features/       # Feature documentation
│   ├── fields/         # Field reference
│   ├── guides/         # How-to, FAQ, glossary
│   ├── integrations/   # Per-provider docs
│   ├── relations/      # Cross-system relations
│   ├── troubleshooting/# Issue guides
│   └── developer/      # NOT indexed — read directly by Upforge / Claude Code
├── merchant/           # → emily-kb-merchant-{env}
├── reseller/           # → emily-kb-reseller-{env}
└── docs/               # → emily-kb-shared-{env} (this file included)

Documentation Standards

1. Use consistent structure (see templates)
2. Include all required sections
3. Write for clarity (a child should understand)
4. Provide examples
5. Document customer impact

Step 2: Extraction

Every pipeline script lives in this repository under scripts/, and each has an npm alias.

npm run extract        # node scripts/extract.js

extract.js walks KNOWLEDGE_STRUCTURE (the shared/, merchant/, reseller/, global-admin/ contexts) plus ROOT_DIRS = ['docs'], parses each Markdown file into structured JSON entries, and tags every entry with the context that decides its target index. category is set to the source folder name (e.g. features, onboarding), not a fixed enum. Output: .knowledge/entries.json.

There is no content-generation step and no translation step. The index schema carries name_nl / content_nl / fr / de fields, but nothing populates them — upload-to-azure.js writes null for each. Any GPT-4 generation or NL/FR/DE/ES/PT/IT translation described in older revisions of this doc is not part of the current pipeline.

Step 3: Embedding Generation

npm run embeddings     # node scripts/generate-embeddings.js

Uses the Azure OpenAI deployment named by AZURE_OPENAI_EMBEDDING_DEPLOYMENT (default text-embedding-ada-002). Output: .knowledge/entries-with-embeddings.json.

Step 4: Upload to Azure Search

npm run upload         # node scripts/upload-to-azure.js

Upload is multi-index. INDEX_NAMES maps each context to an index, and ENV_SUFFIX selects the environment:

Context	Index
shared (incl. docs/)	emily-kb-shared-{env}
merchant	emily-kb-merchant-{env}
reseller	emily-kb-reseller-{env}
global-admin	emily-kb-admin-{env}

{env} is test for the testing branch and prod for production. CI invokes it as upload-to-azure.js --multi --env {suffix}.

Full Pipeline

npm run pipeline       # extract → embeddings → upload → verify
npm run setup          # create-index, then pipeline

There is no --incremental / --full / --feature flag on a local pipeline run; incremental selection is done by CI from the pushed diff (see Automation below).

Azure Search Schema

Index Fields

Field	Type	Purpose
id	String	Unique identifier
type	String	feature/field/workflow/troubleshooting
name	String	English name
name_nl	String	Dutch name
name_fr	String	French name
name_de	String	German name
content	String	English content
content_nl	String	Dutch content
content_fr	String	French content
content_de	String	German content
category	String	Feature category
route	String	Backoffice route
tags	Array	Search keywords
priority	Int	Ranking priority
embedding	Vector	1536-dim embedding

Semantic Configuration

{
  "semantic": {
    "configurations": [{
      "name": "default",
      "prioritizedFields": {
        "titleField": { "fieldName": "name" },
        "contentFields": [{ "fieldName": "content" }]
      }
    }]
  }
}

Cost Estimates

Operation	Cost
Initial full index (~1000 entries)	~$15-25
Incremental update (10 entries)	~$0.10-0.20
Emily chat query	~$0.01-0.03

Automation

GitHub Action

.github/workflows/index-knowledge.yml runs on push to testing or production:

on:
  push:
    branches: [testing, production]
    paths:
      - 'shared/**'
      - 'merchant/**'
      - 'reseller/**'
      - 'global-admin/**'
      - 'docs/**'
  workflow_dispatch:
    inputs:
      full_reindex:

Its steps: determine changed files → determine the environment suffix (test / prod) → create/update the Azure indexes → extract (full or incremental) → generate embeddings → upload multi-index → invalidate the setup-steps cache → post a summary → notify Slack.

There is no main branch in this repo and no full-pipeline.cjs.

Manual Trigger

npm run pipeline

Or run the workflow from the GitHub Actions UI via workflow_dispatch, which takes a full_reindex input for a complete rebuild.

Verification

After indexing, verify:

1. Check Entry Count

npm run verify does this for you. To check one index by hand, substitute the index you actually want — most content lands in emily-kb-shared-{env}, not the merchant index:

curl -X GET "$AZURE_SEARCH_ENDPOINT/indexes/emily-kb-shared-test/docs/\$count?api-version=2023-11-01" \
  -H "api-key: $AZURE_SEARCH_API_KEY"

2. Test Search

curl -X POST "$AZURE_SEARCH_ENDPOINT/indexes/emily-kb-shared-test/docs/search?api-version=2023-11-01" \
  -H "Content-Type: application/json" \
  -H "api-key: $AZURE_SEARCH_API_KEY" \
  -d '{"search": "order capacity", "top": 5}'

3. Test Emily

Ask Emily questions about the indexed content:

• "How do I set up order capacity?"
• "What does the time slot duration field do?"
• "Why are customers not seeing available times?"

Troubleshooting

Entries Not Appearing

1. Check upload logs for errors
2. Verify index exists
3. Check field names match schema
4. Wait for indexing (can take 1-2 minutes)

Search Not Finding Content

1. Check embedding was generated
2. Verify content is not empty
3. Check semantic configuration
4. Try different search terms

Translation Fields Are Empty

Expected. The index schema defines name_nl / content_nl and the fr / de equivalents, but no pipeline step populates them — upload-to-azure.js writes null. There is no translation script in scripts/.