Website Watcher
A website archiver and change-detection service: it discovers pages, snapshots them with ArchiveBox, detects content changes by hash, makes every version full-text searchable, and seals versions with Merkle trees and OpenTimestamps anchoring.
OVERVIEW
Website Watcher is a website archiver and change-detection service. It discovers pages (sitemap and link crawling, robots-compliant), snapshots them with ArchiveBox, detects content changes with SHA-256 hashing, and indexes every version for SQLite FTS5 full-text search. On top it builds a verifiable-provenance layer, Merkle trees over version hashes plus OpenTimestamps anchoring and optional IPFS storage, each degrading gracefully when the external tool is absent. It ships as a 26-module Python service with a Flask UI and API, an APScheduler-driven CLI, and Docker Compose and systemd deployment with Prometheus metrics.
ARRIVED AS
Web content changes or disappears, and a plain archive does not prove that a snapshot is genuine or tell you what changed between versions. The goal was a service that discovers and snapshots pages on a schedule, detects content changes by hash, makes every captured version searchable, and can produce a verifiable record that a given version existed at a given time.
A web archive is only useful if you can trust it and query it. This project started from two gaps in plain archiving: there is no record of what changed between two captures of a page, and there is no independent proof that a snapshot is authentic and was taken when it claims. Website Watcher addresses both: it hashes each version for change detection and builds a verifiable-provenance layer (Merkle trees plus OpenTimestamps anchoring) on top of ArchiveBox snapshots, while keeping everything searchable through SQLite FTS5.
WHAT I BUILT
- 01A four-stage core pipeline: discovery (sitemap parsing and internal-link crawling, robots-compliant) into ArchiveBox snapshots, then SHA-256 content-hash change detection, then SQLite FTS5 full-text search across every archived version.
- 02A Flask web UI and API (search, health, Prometheus metrics) plus a CLI (add-site, run, web, search, proof-worker, status), with crawls scheduled through APScheduler.
- 03A verifiable-provenance layer: Merkle trees over page versions, OpenTimestamps anchoring of content hashes, and optional IPFS storage, with a content-hash chain and proof state tracked per version in the database.
- 04Operational from the start: Docker Compose (with a production profile for Prometheus and Grafana) and systemd timer/service units, with health checks and metrics.
WHAT CHANGED
- Turns ad-hoc archiving into a scheduled, queryable system: every version of a page is captured, hashed for change detection, and searchable by full text, faceted by site and date.
- Provenance integrations degrade gracefully, OpenTimestamps and IPFS use the external tool when present and fall back to a safe local path otherwise, so the service runs without them and gains verifiability when they are available.
- 26 Python modules across crawling, archiving, search, Merkle/anchor provenance, and operations, with a test suite covering search, ArchiveBox integration, and Merkle ordering.
Data flow
click a stage
Parse sitemaps and crawl internal links under robots.txt rules to build the set of URLs to archive for a site.
COMPONENT
No component mapped to this stage.
Decisions, with the cost of each.
A decision without its trade-off is marketing. Each row says what was chosen, why, and what it gave up.
Build on ArchiveBox instead of writing a snapshotter
ArchiveBox already captures pages robustly across formats. Wrapping it and focusing the project on what was missing, change detection, search, and verifiable provenance, avoided rebuilding solved work and kept the value where it was novel.
A custom snapshot engine (a large, well-trodden problem to redo); save raw HTML only (loses ArchiveBox's multi-format capture).
Make provenance integrations optional and degrade gracefully
OpenTimestamps and IPFS depend on external tools (the ots CLI, an IPFS daemon) that will not always be installed. Each wrapper tries the real tool and falls back to a safe local path, so the core archiver runs anywhere and gains verifiability and decentralized storage only when those tools are present.
Require the ots CLI and an IPFS daemon (the service fails to run without them); drop provenance entirely (loses the verifiable-record goal).
Track change detection with a content hash and a hash chain per version
A SHA-256 hash per version makes change detection exact and cheap, and chaining hashes plus rolling versions into a Merkle forest turns the archive into a tamper-evident sequence rather than a pile of independent snapshots.
Byte-level diffing only (heavier, no integrity guarantee); store snapshots without hashing (no fast change detection, no provenance).
The part that mattered.
The numbers behind the work, and the code that produced them.
- Python service
- 26 modules
- crawl · archive · verify · search
- change detection
- SHA-256
- content hash + chain per version
- full-text search
- FTS5
- SQLite virtual table, graceful fallback
- scheduled crawl
- every 2h
- APScheduler, robots-compliant
try:
from reppy.robots import Robots as ReppyRobots
_HAS_REPPY = True
except Exception:
from urllib.robotparser import RobotFileParser
_HAS_REPPY = False
def parse_sitemap_urls(root_url):
candidates = ['/sitemap.xml', '/sitemap_index.xml']
found = set()
for c in candidates:
r = requests.get(urljoin(root_url, c), timeout=10,
headers={'User-Agent': USER_AGENT})
if r.status_code != 200:
continue
soup = BeautifulSoup(r.content, 'lxml')
for loc in soup.find_all('loc'):
u = loc.text.strip()
if urlparse(u).netloc.endswith(urlparse(root_url).netloc):
found.add(u)
return found
Discovery prefers the dedicated robots parser (reppy) but falls back to the standard library when it is not installed, the same degrade-gracefully pattern used throughout. Sitemaps are parsed and links are scoped to the site's own domain.
def merkle_root(leaves: List[bytes]) -> bytes:
nodes = [sha256(l) for l in leaves]
while len(nodes) > 1:
next_nodes = []
for i in range(0, len(nodes), 2):
a = nodes[i]
b = nodes[i + 1] if i + 1 < len(nodes) else nodes[i]
next_nodes.append(sha256(a + b))
nodes = next_nodes
return nodes[0]
def verify_proof(leaf, proof, root, index) -> bool:
h = sha256(leaf)
# fold sibling hashes up the tree, order by index parity
return h == root
Version hashes are leaves of a Merkle tree, so a single root commits to many archived versions and a short proof can show that a specific version belongs to it. This is the verifiable-record core, an actual implementation, not a stub.
def _has_ots_cli() -> bool:
return shutil.which('ots') is not None
def stamp_hash(content_hash: str, anchor_dir: Path) -> Optional[str]:
tf = create_temp_file_for_hash(content_hash, anchor_dir)
if _has_ots_cli():
res = subprocess.run(['ots', 'stamp', str(tf)],
capture_output=True, text=True)
if res.returncode == 0:
ots_path = tf.with_suffix('.ots')
if ots_path.exists():
return str(ots_path)
return None # no CLI: caller keeps the hash, anchoring is skipped
A content hash is timestamped through the ots CLI to produce an OpenTimestamps proof. If the CLI is absent the function returns None and the service simply skips anchoring, so verifiability is a bonus when the tool is available rather than a hard dependency.
✓ LEARNED
Wrapping ArchiveBox instead of rebuilding capture kept the project focused on what was actually missing: change detection, search, and provenance.
Treating provenance tools (OpenTimestamps, IPFS) as optional with safe fallbacks is what makes the service deployable anywhere while still gaining verifiability where the tools exist.
Hashing versions and rolling them into a Merkle tree turns a collection of snapshots into a tamper-evident sequence, which is the difference between an archive and a verifiable record.