ReceiptLens

An offline iPhone prototype that reads receipts, documents, and screenshots by running the MiniCPM-V 4.6 vision-language model fully on-device with llama.cpp, no server, no upload. A SwiftUI app wrapping an on-device multimodal engine.

ROLE
Builder
PERIOD
2026
DOMAIN
On-device AI
STATUS
Published
CODE
GitHub ↗

OVERVIEW

ReceiptLens is an offline iPhone prototype that reads receipts, documents, and screenshots by running the MiniCPM-V 4.6 vision-language model fully on-device with llama.cpp, nothing is uploaded. A SwiftUI app captures or picks an image, applies one of three mode-specific prompts (receipt, document, screen), and runs inference locally through a Swift engine over llama.cpp's multimodal (mtmd) runtime, with an in-app model downloader and a local scan history. It is built honestly as a prototype: the SwiftUI app, model management, camera flow, and engine wiring are the original work, while the low-level native bridge is adapted from OpenBMB's official iOS demo and llama.cpp ships as a vendored xcframework.

ARRIVED AS

Reading a receipt or document with a vision-language model normally means uploading the image to a cloud API. The aim of this prototype was to keep that entirely on the phone: run a capable multimodal model (MiniCPM-V 4.6) locally so a receipt, document, or screenshot is analyzed offline, with nothing leaving the device.

ReceiptLens is a prototype exploring on-device multimodal inference on iOS: can a phone read a receipt with a real vision-language model without sending anything to a server? It uses MiniCPM-V 4.6 in GGUF form, run locally through llama.cpp's multimodal (mtmd) support. The app is built honestly as a prototype, the original work is the SwiftUI application, the model-download and storage flow, the camera and prompt-mode UX, and the Swift engine that drives inference; the underlying native bridge is adapted from OpenBMB's official iOS demo, and llama.cpp is vendored as a prebuilt xcframework.

WHAT I BUILT

  1. 01A SwiftUI iOS app that captures or picks an image, sends it with a mode-specific prompt to an on-device engine, and shows the model's reading, all offline.
  2. 02On-device inference of MiniCPM-V 4.6 (GGUF) through llama.cpp and its multimodal mtmd layer, wired into Swift via a native Objective-C++ bridge adapted from OpenBMB's official iOS demo.
  3. 03Three analysis modes, receipt, document, and screen, each with its own prompt template, plus an in-app model downloader that fetches the GGUF model files on first run.
  4. 04A local history of scans and an image file store, so past readings are kept on-device without any backend.

WHAT CHANGED

  • Fully offline multimodal reading: the image and the model both stay on the phone, which is the point for receipts and documents.
  • Demonstrates that a vision-language model, not just a text model, can run on-device through llama.cpp's mtmd path.
  • An honest prototype: the SwiftUI app, model management, camera flow, and engine wiring are the original work; the low-level native bridge is adapted from OpenBMB's demo and llama.cpp ships as a vendored xcframework.

Data flow

click a stage

On first run, the model downloader fetches the MiniCPM-V 4.6 GGUF files to the device.

COMPONENT

SwiftUI app (RootView / ScannerView / ModelSetupView / HistoryView)

The UI: model setup, capture/scan, results, and a local history of past readings.

Decisions, with the cost of each.

A decision without its trade-off is marketing. Each row says what was chosen, why, and what it gave up.

Run the model on-device instead of calling a cloud API

Receipts and documents are exactly the kind of content people do not want to upload. Running MiniCPM-V locally through llama.cpp keeps the image and the inference on the phone, which is the entire reason for the prototype, at the cost of downloading a multi-gigabyte model and slower inference than a server.

A cloud vision API (fast and easy, but the image leaves the device); on-device OCR only (no language understanding of the content).

Adapt OpenBMB's native bridge rather than write the mtmd layer from scratch

The hard, low-level work of bridging llama.cpp's multimodal runtime into iOS already exists in OpenBMB's official demo. Adapting it and focusing original effort on the app, model management, and UX was the sensible split for a prototype, and it is credited as such.

Reimplement the mtmd bridge from scratch (large effort, little new value for a prototype).

The part that mattered.

The numbers behind the work, and the code that produced them.

offline VLM
On-device
MiniCPM-V 4.6 (GGUF) via llama.cpp
prompt templates
3 modes
receipt · document · screen
iOS prototype
SwiftUI
camera · model downloader · history
local model files
GGUF
downloaded in-app, no server
On-device analysis: image plus prompt, generated locallyswift
final class MiniCPMEngine: ObservableObject {
    let wrapper = MTMDWrapper()

    func analyze(imageURL: URL, prompt: String, files: ModelFiles) async throws -> String {
        try await loadIfNeeded(files: files)
        try await wrapper.addUserImageAndText(imagePath: imageURL.path, text: prompt)
        // stream the generated reading from the on-device model
    }

    private func loadIfNeeded(files: ModelFiles) async throws {
        let params = MTMDParams(
            imageMaxSliceNums: EngineConfig.imageMaxSliceNums,
            imageMaxTokens: EngineConfig.imageMaxTokens
        )
        // load the GGUF model + multimodal projector via the native bridge
    }
}

The engine loads the MiniCPM-V model once, then for each scan hands the image path and a mode-specific prompt to the native mtmd wrapper and streams the result. Everything runs against the on-device model; there is no network call in the analysis path.

Three analysis modesswift
enum AnalysisMode: String, CaseIterable, Identifiable, Codable {
    case receipt  = "Receipt"
    case document = "Document"
    case screen   = "Screen"

    var systemImage: String {
        switch self {
        case .receipt:  "receipt"
        case .document: "doc.text.viewfinder"
        case .screen:   "rectangle.and.text.magnifyingglass"
        }
    }
}

Each mode maps to its own prompt template, so the same on-device model is steered differently for a receipt, a document, or a screenshot. The mode is a small enum that also drives the UI's icons.

✓ LEARNED

  1. A multimodal model, not just a text model, can run on a phone through llama.cpp's mtmd path, which makes offline document reading possible.

  2. For a prototype, adapting an existing native bridge (OpenBMB's) and spending the effort on the app and UX was the right trade, and saying so plainly is part of doing it honestly.

  3. On-device inference is a UX problem as much as an ML one: model download, storage, and progress are most of what makes a local-model app usable.