ReceiptLens

An offline iPhone prototype that reads receipts, documents, and screenshots by running the MiniCPM-V 4.6 vision-language model fully on-device with llama.cpp, no server, no upload. A SwiftUI app wrapping an on-device multimodal engine.

ROLE: Builder
PERIOD: 2026
DOMAIN: AI/ML
CODE: GitHub ↗

OVERVIEW

A SwiftUI app captures or picks an image, applies one of three mode-specific prompts (receipt, document, screen), and runs inference locally through a Swift engine over llama.cpp's multimodal (mtmd) runtime, with an in-app model downloader and a local scan history.

ARRIVED AS

“Reading a receipt or document with a vision-language model normally means uploading the image to a cloud API. I wanted to keep that entirely on the phone: run a capable multimodal model (MiniCPM-V 4.6) locally so a receipt, document, or screenshot is analyzed offline, with nothing leaving the device.”

ReceiptLens is a prototype exploring on-device multimodal inference on iOS: can a phone read a receipt with a real vision-language model without sending anything to a server? It uses MiniCPM-V 4.6 in GGUF form, run locally through llama.cpp's multimodal (mtmd) support. The original work is the SwiftUI application, the model-download and storage flow, the camera and prompt-mode UX, and the Swift engine that drives inference; the underlying native bridge is adapted from OpenBMB's official iOS demo, and llama.cpp is vendored as a prebuilt xcframework.

WHAT I BUILT

01A SwiftUI iOS app that captures or picks an image, sends it with a mode-specific prompt to an on-device engine, and shows the model's reading, all offline.
02On-device inference of MiniCPM-V 4.6 (GGUF) through llama.cpp and its multimodal mtmd layer, wired into Swift via a native Objective-C++ bridge adapted from OpenBMB's official iOS demo.
03Three analysis modes, receipt, document, and screen, each with its own prompt template, plus an in-app model downloader that fetches the GGUF model files on first run.
04A local history of scans and an image file store, so past readings are kept on-device without any backend.

WHAT CHANGED

Fully offline multimodal reading: the image and the model both stay on the phone, which is the point for receipts and documents.
Shows that a vision-language model (not only a text model) can run on-device through llama.cpp's mtmd path.
Credit where it's due: the SwiftUI app, model management, camera flow, and engine wiring are my own work; the low-level native bridge is adapted from OpenBMB's demo and llama.cpp ships as a vendored xcframework.

ARCHITECTURE

A SwiftUI app shell (RootView, ScannerView, ModelSetupView, HistoryView) holds the UI. A MiniCPMEngine (an ObservableObject) owns an MTMDWrapper, the Swift-facing handle on the native multimodal runtime. The native bridge (MBMtmd, an Objective-C++ layer over llama.cpp's mtmd, adapted from OpenBMB's demo) is exposed to Swift through a bridging header and the MTMDWrapper, with typed MTMDParams, MTMDToken, and MTMDError. On first run, a ModelStore downloads the MiniCPM-V 4.6 GGUF files; analysis takes a captured image plus a mode-specific prompt (from PromptTemplate), loads the model if needed, adds the image and text to the runtime, and streams the generated reading. An ImageFileStore and a scan history keep results locally. llama.cpp itself is a vendored llama.xcframework (device and simulator slices).

Data flow

click a stage

On first run, the model downloader fetches the MiniCPM-V 4.6 GGUF files to the device.

COMPONENT

ModelStore + downloader

Fetches and manages the MiniCPM-V 4.6 GGUF model files on-device, with download progress.

ALSO IN THE BUILD

NativeBridge (MBMtmd + MTMDWrapper)
An Objective-C++ bridge over llama.cpp's mtmd, adapted from OpenBMB's iOS demo, exposed to Swift with typed params, tokens, and errors.
Vendor/llama.xcframework
Prebuilt llama.cpp framework (device and simulator slices), vendored rather than rebuilt.

Decisions, with the cost of each.

A decision without its trade-off is marketing. Each row says what was chosen, why, and what it gave up.

DECISIONBECAUSE

Run the model on-device instead of calling a cloud API

Receipts and documents are exactly the kind of content people do not want to upload. Running MiniCPM-V locally through llama.cpp keeps the image and the inference on the phone, which is the entire reason for the prototype, at the cost of downloading a multi-gigabyte model and slower inference than a server.

INSTEAD OF

A cloud vision API (fast and easy, but the image leaves the device); on-device OCR only (no language understanding of the content).

Adapt OpenBMB's native bridge rather than write the mtmd layer from scratch

The hard, low-level work of bridging llama.cpp's multimodal runtime into iOS already exists in OpenBMB's official demo. Adapting it and focusing original effort on the app, model management, and UX was the sensible split for a prototype, and it is credited as such.

INSTEAD OF

Reimplement the mtmd bridge from scratch (large effort, little new value for a prototype).

The part that mattered.

The facts behind the work, and the code that produced them.

offline VLM: On-device; MiniCPM-V 4.6 (GGUF) via llama.cpp
prompt templates: 3 modes; receipt · document · screen
iOS prototype: SwiftUI; camera · model downloader · history
local model files: GGUF; downloaded in-app, no server

On-device analysis: image plus prompt, generated locallyswift

final class MiniCPMEngine: ObservableObject {
    let wrapper = MTMDWrapper()

    func analyze(imageURL: URL, prompt: String, files: ModelFiles) async throws -> String {
        try await loadIfNeeded(files: files)
        try await wrapper.addUserImageAndText(imagePath: imageURL.path, text: prompt)
        // stream the generated reading from the on-device model
    }

    private func loadIfNeeded(files: ModelFiles) async throws {
        let params = MTMDParams(
            imageMaxSliceNums: EngineConfig.imageMaxSliceNums,
            imageMaxTokens: EngineConfig.imageMaxTokens
        )
        // load the GGUF model + multimodal projector via the native bridge
    }
}

The engine loads the MiniCPM-V model once, then for each scan hands the image path and a mode-specific prompt to the native mtmd wrapper and streams the result. Everything runs against the on-device model; there is no network call in the analysis path.

Three analysis modesswift

enum AnalysisMode: String, CaseIterable, Identifiable, Codable {
    case receipt  = "Receipt"
    case document = "Document"
    case screen   = "Screen"

    var systemImage: String {
        switch self {
        case .receipt:  "receipt"
        case .document: "doc.text.viewfinder"
        case .screen:   "rectangle.and.text.magnifyingglass"
        }
    }
}

Each mode maps to its own prompt template, so the same on-device model is steered differently for a receipt, a document, or a screenshot. The mode is a small enum that also drives the UI's icons.

LEARNED

A multimodal model (not only a text model) can run on a phone through llama.cpp's mtmd path, which makes offline document reading possible.
For a prototype, adapting an existing native bridge (OpenBMB's) and spending the effort on the app and UX was the right trade. Saying so plainly is part of the job.
On-device inference is a UX problem as much as an ML one: model download, storage, and progress are most of what makes a local-model app usable.