Ontology in the Loop

A Framework for AI-Assisted Knowledge Graph Evolution

Niran Pravithana

V. Human-in-the-Loop Workflow

This section describes the human review process that ensures quality control over ontology evolution.

5.1 Review Actions

When reviewing a proposal, humans have four options:

  • Approve — Accept proposal as-is for application
  • Reject — Decline with feedback
  • Modify — Edit and then approve
  • Defer — Postpone, await more data

Each action advances the proposal through its lifecycle:

Proposal Lifecycle
Fig. 6. Proposal state transitions.

5.2 Review Interface

The interface presents proposals with all relevant context:

Review Interface
Fig. 7. Proposal review interface mockup.

Information Sections

  • Summary — Plain language explanation
  • Schema changes — Detailed modifications
  • Data enrichment — Instances to insert
  • Source — Document provenance
  • Validation — Duplicate check results

5.3 Feedback Capture

When rejecting, reviewers provide structured feedback:

  • Duplicate — "Same as PARTNERS_WITH"
  • Unnecessary — "Edge case, not worth adding"
  • Incorrect — "OPEC is not a REGION"
  • Incomplete — "Needs inverse relationship"
  • Out of scope — "We don't model individuals"

This feedback can inform future proposal generation.

5.4 Batch Processing

Humans aren't always available. Proposals accumulate between review sessions. The system supports batch workflows:

  1. Human opens review session
  2. System presents priority-sorted queue
  3. Human reviews proposals one by one
  4. Approved proposals batch together
  5. Human confirms batch application
  6. System applies changes atomically

Priority Queue

Proposals are sorted by priority score based on:

  • Confidence level
  • Gap frequency (how often seen)
  • Pending data waiting for this change
  • Time in queue

High-impact, high-confidence proposals surface first.

5.5 Application Protocol

Approved proposals are applied atomically:

  1. Begin transaction
  2. Apply schema changes to meta-schema
  3. Insert enrichment data
  4. Commit transaction
  5. Update proposal status
  6. Update embeddings index
  7. Reprocess pending extractions

If any step fails, the entire transaction rolls back.

Pending Data Reprocessing

Documents waiting for a schema change are automatically reprocessed after approval:

Before PROP-042 approved:
  Doc A: "Qatar joined OPEC" -> PENDING
  Doc B: "UAE is OPEC member" -> PENDING

After PROP-042 approved:
  Reprocess Doc A -> Extract successfully
  Reprocess Doc B -> Extract successfully

5.6 Discussion Support

Complex proposals may require clarification. The system supports discussion threads between humans and AI:

[Human]: Why ORGANIZATION, not extending REGION?

[AI]: ORGANIZATION differs from REGION:
  - Membership is voluntary
  - Not geographically contiguous
  - Has organizational structure
  Different relationship semantics.

[Human]: What about subtypes?

[AI]: Recommend org_type property for now.
  Can evolve to subtypes if needed.

[Human]: Approved with org_type property.

5.7 Cognitive Load

The system minimizes reviewer burden through:

  • Priority ordering — Important proposals first
  • Pre-validation — Obvious duplicates filtered
  • Clear presentation — Structured, scannable format
  • Context provision — Source docs linked
  • AI recommendations — Suggested actions when appropriate

The goal is efficient review sessions where humans focus on judgment calls, not information gathering.