ContextBox

CLI-first personal knowledge assistant. Captures screenshots with OCR, pulls content from the web, embeds everything with sentence-transformers into SQLite, and answers questions over it from the terminal via GitHub Models.

ROLE: Builder
PERIOD: 2024
DOMAIN: AI/ML
STATUS: Published
CODE: GitHub ↗
DEMO: Live ↗

OVERVIEW

ContextBox is a CLI-first personal knowledge assistant in Python. It captures screenshots (with Tesseract OCR) and web, Wikipedia, and YouTube content into a local SQLite store, embeds everything with sentence-transformers (all-MiniLM-L6-v2) for semantic search, and answers questions over your captures via GitHub Models with question classification, source attribution, and confidence levels. It is a 27-module package with a Click + Rich CLI, optional feature groups so heavy dependencies stay optional, a Fernet-based privacy mode, and a React documentation site on GitHub Pages.

ARRIVED AS

The things you reference while working, a screenshot, a web page, a Wikipedia entry, a YouTube transcript, scatter across apps and are gone the moment you close the tab. The goal was a single command-line tool that captures all of it, indexes it for meaning rather than keywords, and lets you ask questions over everything you have seen, without leaving the terminal.

While working you constantly pull in context from different places, a screenshot of an error, a web page, a Wikipedia article, a video transcript, and then lose it. ContextBox is a command-line tool that captures those into one local store, embeds them so they can be searched by meaning, and answers questions over them. It is deliberately CLI-first and local-first: captures live in a SQLite database under your home directory, the embedding model runs locally, and the only external call is the LLM for question-answering.

WHAT I BUILT

01Cross-platform screenshot capture with Tesseract OCR, detecting Wayland, X11, GNOME, or macOS and falling back across gnome-screenshot, scrot, and grim.
02Pluggable content extractors for web pages, Wikipedia, and YouTube transcripts, each behind an availability guard so a missing dependency disables just that extractor.
03Semantic search with sentence-transformers (all-MiniLM-L6-v2) and cosine similarity over captures stored in SQLite, with a keyword fallback when the model is not installed.
04Question-answering over captured content via GitHub Models, with question-type classification, source attribution, and confidence levels, plus a privacy mode that encrypts sensitive captures with Fernet.

WHAT CHANGED

A 27-module Python package (Click + Rich CLI) covering capture, extraction, semantic search, and AI Q&A, installable via pip with optional feature groups ([llm], [ocr], [youtube]).
Graceful degradation throughout: heavy dependencies (sentence-transformers, OCR, YouTube) are optional, and each subsystem disables itself cleanly when its dependency is absent.
Ships with a React documentation site (install, commands, API reference, demo) deployed on GitHub Pages, and is MIT-licensed.

Data flow

click a stage

A screenshot (OCR'd with Tesseract) or an extractor (web page, Wikipedia, YouTube transcript) pulls content in; the right screenshot backend is chosen per platform.

COMPONENT

ScreenshotCapture (capture.py)

Cross-platform screen capture: detects Wayland/GNOME/X11/macOS and falls back across gnome-screenshot, scrot, and grim; stores media under ~/.contextbox/media.

Decisions, with the cost of each.

A decision without its trade-off is marketing. Each row says what was chosen, why, and what it gave up.

DECISIONBECAUSEAT THE COST OF

CLI-first and local-first, not a web app

The tool sits next to where the work happens, the terminal, and the data it captures (screenshots, notes) is personal. Keeping captures in a local SQLite store and running the embedding model locally means nothing leaves the machine except the optional LLM call for Q&A.

A web app or browser extension (more surface, a server to host, and captured personal data leaving the device).

Make every heavy dependency optional behind an availability guard

Sentence-transformers, OCR, and YouTube extraction are large and not everyone needs all of them. Importing each behind a try/except flag and exposing pip extras ([llm], [ocr], [youtube]) keeps a base install small and lets a missing dependency disable just that feature instead of breaking the CLI.

Require the full dependency set (heavy install, fails if any piece is missing); split into several packages (more to maintain and coordinate).

Semantic search with a keyword fallback

Embedding-based search is the point, but it should not be a hard requirement. Cosine similarity over all-MiniLM-L6-v2 embeddings gives meaning-based retrieval, and a keyword fallback keeps search working when the model is not installed.

Embeddings only (search dies without the model); keyword search only (misses semantically related captures).

The part that mattered.

The numbers behind the work, and the code that produced them.

Python package: 27 modules; capture · extractors · search · LLM · privacy
embeddings: all-MiniLM-L6-v2; cosine search, keyword fallback
optional feature groups: 3 extras; [llm] · [ocr] · [youtube]
SQLite store: Local-first; only the Q&A call leaves the machine

Optional heavy deps, guarded at importpython

try:
    from sentence_transformers import SentenceTransformer
    from sklearn.metrics.pairwise import cosine_similarity
    SENTENCE_TRANSFORMERS_AVAILABLE = True
except ImportError:
    SENTENCE_TRANSFORMERS_AVAILABLE = False
    logging.warning("Sentence transformers not available - using basic search")

class EmbeddingManager:
    def __init__(self, model_name: str = 'all-MiniLM-L6-v2', cache_dir=None):
        self.model = None
        if SENTENCE_TRANSFORMERS_AVAILABLE:
            self.model = SentenceTransformer(model_name)

The pattern that runs through the whole package: import the heavy dependency behind a try/except, set an availability flag, and degrade instead of crashing. Semantic search uses all-MiniLM-L6-v2 when present and falls back to basic search when it is not.

Pick a screenshot backend per platformpython

class ScreenshotCapture:
    def __init__(self, media_dir: str = None):
        self.media_dir = Path(media_dir) if media_dir else Path.home() / ".contextbox" / "media"
        self.media_dir.mkdir(parents=True, exist_ok=True)

        available_tools = self.get_available_tools()
        if not available_tools:
            logger.warning(
                "No screenshot tools detected. Install 'gnome-screenshot', "
                "'scrot', or 'grim' to enable screenshot capture."
            )

    def is_wayland(self) -> bool:
        return os.environ.get('WAYLAND_DISPLAY') is not None

Capture detects the display server and desktop and chooses an available backend (gnome-screenshot, scrot, grim) rather than assuming one. Media is written under the user's home directory, keeping captures local.

Classify the question before answering itpython

class QuestionType(Enum):
    FACTUAL = "factual"
    INFERENTIAL = "inferential"
    COMPARATIVE = "comparative"
    PROCEDURAL = "procedural"
    ANALYTICAL = "analytical"
    TEMPORAL = "temporal"
    CAUSAL = "causal"
    DEFINITION = "definition"
    # ... plus a confidence level attached to every answer

Q&A is not a single prompt. The question is classified by type, context is built from the captures retrieved by semantic search, and the answer carries source attribution and a confidence level, so the response is traceable back to what was captured.

✓ LEARNED

Treating every heavy dependency as optional changes the install story: the base tool stays small, and a missing model or OCR binary disables one feature instead of breaking the whole CLI.
A keyword fallback behind semantic search is cheap insurance: retrieval keeps working before the embedding model is installed, which matters for a tool people try once from the terminal.
Question classification plus source attribution makes LLM answers auditable: you can see which captures an answer came from, which is what makes a personal-knowledge tool trustworthy.