Release Notes: v0.1.2¶

Summary¶

This release introduces Tier 0/Tier 1 indexing foundations, hybrid retrieval, and security hardening for indexing/query workflows.

Highlights¶

Tier 0¶

• Added structured Tier 0 ingestion records per document:
• id, source, timestamp, author_agent, doc_length, metadata
• Added Tier 0 index output command:
• --tier0-index-out (defaults to tier0-index.json)

Tier 1¶

• Added Tier 1 key-entity extraction with pluggable rankers:
• heuristic
• spacy (local subprocess integration)
• Added Tier 1 important-term extraction with pluggable rankers:
• yake, rake, cvalue, textrank

Hybrid In-Memory Indexing and Query¶

• Added in-memory index build and query flow:
• --index
• --query "..."
• Added hybrid scoring and score breakdown in query output.
• Added BM25 lexical component via Tantivy, combined with:
• entity overlap
• term overlap
• topic/doc-type boosts

Security and Safety Hardening¶

• Redacted raw content from serialized index output.
• Added --index-redacted mode for safer index sharing.
• Added spaCy model allowlist in scripts/spacy_ner.py.
• Added spaCy subprocess timeout and kill behavior.
• Added warning logs for lexical index/query fallback conditions.
• Added query length and token caps for defensive resource control.

CLI Additions and Changes¶

• Added:
• --show-tier0
• --show-tier1-entities
• --show-tier1-terms
• --tier1-ner-provider
• --tier1-term-ranker
• --index
• --index-redacted
• --query
• Renamed query/index flow from earlier memory-prefixed naming to:
• --index
• --query

Upgrade Notes¶

• Rust toolchain was upgraded during development to ensure dependency compatibility with current search stack.
• Existing lint commands remain available; new indexing/query commands are additive.