Search Engine Administration — Developer Guide

Overview

The Search subgraph (apps/search) powers unified discovery across all Malets using Meilisearch as the search engine. The Administration layer extends the base search functionality with three operational features that ensure index consistency, data quality, and search relevance:

All admin operations are gated behind Permission.MANAGE_SEARCH — only platform administrators and authorized Malet Owners can trigger these actions.

Architecture

graph TD
    A["SynonymResolver"] --> B["SynonymService"]
    C["ReconciliationResolver"] --> D["ReconciliationService"]
    E["BulkImportResolver"] --> F["BulkImportService"]

    B --> G["Redis (synonym storage)"]
    B --> H["MeilisearchService"]

    D --> I["ProductStub / ServiceStub (MongoDB)"]
    D --> H
    D --> J["Redis (lock + status)"]

    F --> I
    F --> H
    F --> K["Redis (lock + progress)"]

    L["@Cron: Daily 3 AM"] --> D

    style G fill:#dc2626,color:#fff
    style J fill:#dc2626,color:#fff
    style K fill:#dc2626,color:#fff
    style H fill:#3b82f6,color:#fff
    style L fill:#f59e0b,color:#fff

How It Fits Together

The search service already receives real-time updates via TCP events (item_upserted, item_deleted) from source subgraphs like products and services. The admin layer addresses scenarios where real-time sync is insufficient:

• Missed events: If a TCP message is lost during a deployment or network partition, the reconciliation job detects and repairs the drift.
• New verticals: When adding a new entity type (e.g., Malets or Blog Posts), bulk import can backfill the entire index.
• Search quality: Synonyms improve recall — a Visitor searching for "game drive" will also find results tagged "safari".

Per-Vertical Synonyms

Why Verticals?

Different Malet verticals use domain-specific language. A "session" in Photography means a photoshoot booking, while in Wellness it means a spa treatment. Per-vertical synonyms prevent cross-contamination across these domains.

Supported Verticals

enum SynonymVertical {
	GENERAL = 'GENERAL', // Cross-vertical terms (shop ↔ store ↔ malet)
	RESTAURANT = 'RESTAURANT', // Dining-specific (takeout ↔ takeaway ↔ carry out)
	TOUR = 'TOUR', // Safari & excursion terms (safari ↔ game drive)
	PHOTOGRAPHY = 'PHOTOGRAPHY', // Photo session terms (portrait ↔ headshot)
	FASHION = 'FASHION', // Apparel terms (clothing ↔ garments ↔ wear)
	WELLNESS = 'WELLNESS', // Spa & fitness terms (spa ↔ wellness center)
	ENTERTAINMENT = 'ENTERTAINMENT' // Events & tickets (concert ↔ show ↔ gig)
}

Storage Model

Synonyms are stored in Redis with one key per vertical:

search:synonyms:GENERAL     → [{"term":"shop","synonyms":["store","malet","boutique"]}, ...]
search:synonyms:RESTAURANT  → [{"term":"restaurant","synonyms":["dining","eatery","bistro"]}, ...]
search:synonyms:TOUR        → [{"term":"safari","synonyms":["game drive","wildlife tour"]}, ...]

Seeded Defaults

On first startup, the SynonymService seeds default synonym groups for every vertical if the Redis key is empty. This ensures search relevance is improved out of the box without manual configuration.

Bidirectional Push

When applySynonyms is called, all vertical synonym groups are merged into a single bidirectional map and pushed to Meilisearch:

// Input: { term: "safari", synonyms: ["game drive", "wildlife tour"] }
// Output pushed to Meilisearch:
{
  "safari": ["game drive", "wildlife tour"],
  "game drive": ["safari", "wildlife tour"],
  "wildlife tour": ["safari", "game drive"]
}

This means a Visitor searching for any of these terms will see results containing the others.

GraphQL API

# Query synonym groups (optionally filtered by vertical)
query {
	searchSynonyms(vertical: RESTAURANT) {
		term
		synonyms
		vertical
	}
}

# Add or update a synonym group
mutation {
	setSynonymGroup(
		input: { vertical: TOUR, term: "hiking", synonyms: ["trekking", "walking", "trail"] }
	) {
		term
		synonyms
		vertical
	}
}

# Delete a synonym group
mutation {
	deleteSynonymGroup(input: { vertical: TOUR, term: "hiking" })
}

# Push all Redis synonyms to the Meilisearch index
mutation {
	applySynonyms {
		success
		totalGroups
	}
}

Note: Changes via setSynonymGroup and deleteSynonymGroup only update Redis. You must call applySynonyms to push the changes to the live Meilisearch index.

Search Reconciliation

Problem

Real-time TCP event-driven sync handles 99%+ of updates. But edge cases can cause drift:

• A source service emits item_upserted but the search service is restarting → event lost
• A database migration adds items that bypass the event pipeline
• A Meilisearch index corruption or manual deletion

Solution: Daily Reconciliation Cron

The ReconciliationService runs a scheduled job at 3:00 AM daily that compares the MongoDB source-of-truth against the Meilisearch index:

sequenceDiagram
    participant Cron as "@Cron (3 AM)"
    participant Recon as "ReconciliationService"
    participant Redis as "Redis"
    participant DB as "MongoDB (Stubs)"
    participant Meili as "Meilisearch"

    Cron->>Recon: scheduledReconciliation()
    Recon->>Redis: setnx("search:reconciliation:lock", 1h TTL)
    alt Lock acquired
        Recon->>DB: Fetch all Product IDs
        Recon->>DB: Fetch all Service IDs
        Recon->>Meili: getAllDocumentIds()
        Recon->>Recon: Compute drift (missing + orphaned)
        loop Missing items (in DB, not in index)
            Recon->>DB: Fetch full item + Malet + Reviews
            Recon->>Meili: indexItem(enriched document)
        end
        loop Orphaned items (in index, not in DB)
            Recon->>Meili: deleteItems(orphaned IDs)
        end
        Recon->>Redis: Store results + lastRunAt
    else Lock held
        Recon->>Recon: Skip (another run in progress)
    end
    Recon->>Redis: Release lock

Drift Detection

Concurrency Protection

Only one reconciliation can run at a time thanks to a Redis setnx lock with a 1-hour TTL:

search:reconciliation:lock → "running" (TTL: 3600s)

If the service crashes mid-reconciliation, the lock auto-expires after 1 hour, allowing the next scheduled run to proceed.

GraphQL API

# Check reconciliation status
query {
	reconciliationStatus {
		isRunning
		lastRunAt
		lastResult {
			success
			added
			removed
			unchanged
			errors
			durationMs
		}
	}
}

# Manually trigger a reconciliation run
mutation {
	triggerReconciliation {
		success
		added
		removed
		unchanged
		errors
		durationMs
	}
}

Bulk Import

Use Cases

• Initial index population: When deploying a new Meilisearch instance
• Schema migration: After changing index settings that require re-indexing
• Primary key repair: If the Meilisearch index loses its primaryKey: 'id' setting (see Universal Search Index — Gotchas)
• Per-Malet recovery: When a specific Malet's items are corrupted in the index
• Vertical rollout: When a new item type becomes searchable

Import Modes

Batched Processing

Items are processed in batches of 500 to prevent memory exhaustion and Meilisearch queue overflow:

// Internal flow (simplified)
const items = await fetchItems(); // All products + services
for (let i = 0; i < items.length; i += 500) {
	const batch = items.slice(i, i + 500);
	const enriched = await enrichBatch(batch); // Malet + review lookup
	await meilisearchService.addOrUpdate(enriched);
	await redis.set('search:bulk-import:progress', i + 500);
}

Malet Caching

During bulk import, Malet metadata is cached in-memory to avoid repeated database lookups. If 100 Products all belong to the same Malet, the MaletStub is fetched only once.

Progress Tracking

Clients can poll for import progress while a job is running:

query {
	bulkImportStatus {
		isRunning
		currentProgress # e.g. 1500
		currentTotal # e.g. 5000
		lastRunAt
		lastResult {
			success
			totalProcessed
			indexed
			failed
			durationMs
		}
	}
}

GraphQL API

# Re-index everything
mutation {
	bulkImportAll {
		success
		totalProcessed
		indexed
		failed
		durationMs
	}
}

# Re-index only Products
mutation {
	bulkImportByType(input: { type: PRODUCT }) {
		success
		totalProcessed
		indexed
		failed
		durationMs
	}
}

# Re-index all items for a specific Malet
mutation {
	bulkImportByMalet(input: { maletId: "malet-xyz-123" }) {
		success
		totalProcessed
		indexed
		failed
		durationMs
	}
}

Redis Key Reference

All search admin keys use the search: namespace:

Module Structure

apps/search/src/
├── synonyms/
│   ├── dto/
│   │   └── synonym.dto.ts            # SynonymVertical enum, I/O types
│   ├── synonym.service.ts            # Redis CRUD + Meilisearch push
│   ├── synonym.service.spec.ts       # 11 unit tests
│   ├── synonym.resolver.ts           # GraphQL queries + mutations
│   ├── synonym.resolver.spec.ts      # 4 unit tests
│   └── synonym.module.ts             # NestJS module
│
├── reconciliation/
│   ├── dto/
│   │   └── reconciliation.dto.ts     # Result + Status output types
│   ├── reconciliation.service.ts     # Cron job + drift detection
│   ├── reconciliation.service.spec.ts # 7 unit tests
│   ├── reconciliation.resolver.ts    # Admin queries + mutations
│   ├── reconciliation.resolver.spec.ts # 2 unit tests
│   └── reconciliation.module.ts      # NestJS module (ScheduleModule)
│
├── bulk-import/
│   ├── dto/
│   │   └── bulk-import.dto.ts        # Result, Status, Input types
│   ├── bulk-import.service.ts        # Batched import logic
│   ├── bulk-import.service.spec.ts   # 8 unit tests
│   ├── bulk-import.resolver.ts       # Admin queries + mutations
│   ├── bulk-import.resolver.spec.ts  # 4 unit tests
│   └── bulk-import.module.ts         # NestJS module
│
└── meilisearch/
    └── meilisearch.service.ts        # +5 new methods (synonyms, IDs, bulk delete, stats)
                                          # primaryKey: 'id' enforced on createIndex + addDocuments

Environment Variables

Testing

Unit Tests

# Run all search unit tests (105 tests across 13 suites)
npm run test -- apps/search --no-coverage

Key coverage areas:

• SynonymService: CRUD operations, seeding, bidirectional merge, Meilisearch push, graceful init failure
• ReconciliationService: Lock acquisition/skip, missing item detection, orphan cleanup, Redis status storage, lock release on error
• BulkImportService: All 3 import modes, lock handling, empty database, progress tracking, status query
• MeilisearchService: Synonym CRUD, document ID enumeration, bulk delete, stats retrieval

E2E Tests

# Run search E2E tests (20 tests across 3 suites)
npx jest --config apps/search/test/jest-e2e.json --detectOpenHandles

Covers: Full synonym lifecycle (seed → add → update → delete → apply), reconciliation with drift scenarios, and bulk import by type and by Malet.

Cross-Service Integration

Dependency Map

Permission Requirement

All admin mutations require Permission.MANAGE_SEARCH, which is part of the platform's action-based permission system in @app/common. This permission should be assigned to platform administrators or authorized Malet Owners who need index management capabilities.

@Mutation(() => ReconciliationResult)
@RequirePermission(Permission.MANAGE_SEARCH)
async triggerReconciliation(): Promise<ReconciliationResult> {
  return this.reconciliationService.reconcile();
}

Related

• Soft-Delete & Trash System — Soft-deleted items emit TCP events to remove from the search index
• Gateway Rate Limiting — Admin operations benefit from authenticated user rate-limit uplift
• Organizations & Permissions — MANAGE_SEARCH permission used by all admin endpoints
• Tag Registry — Platform-managed curated tags for Malets — future integration point for search facets and real-time tag sync
• Universal Search Index — Blog posts, Malets, and platform guides as searchable entities with ContentSource discrimination
• Entertainment & Experiences — Events, credit wallets, and loyalty from the experiences subgraph — future indexable entity type
• Workspaces & The Tower — Admin dashboard Search Analytics sub-tab consumes faceted search, reconciliation, and synonym data
• Mallnline Algorithm — How the algorithm utilizes Meilisearch custom ranking for trending scores