From da28a20f091372375822f9dde4486ecade859e7e Mon Sep 17 00:00:00 2001 From: Ken D'Ambrosio Date: Mon, 8 Jun 2026 17:09:51 +0000 Subject: Add opt-in facial recognition: detection and embedding storage - scripts/faces.py: Python helper using face_recognition (dlib/HOG) to detect faces and return 128-D encodings as JSON; called by update.rb - scripts/update.rb: enrich_faces() stores face boxes and encodings in album.json per image (null = not yet processed, [] = processed/none found); skips files already processed; gated on faces.enabled in config.yml - Reads CONFIG_PATH (same env var as app.rb) to check faces.enabled flag - Feature is off by default; enabled in this install via config.yml - README.md, DESIGN.md: document installation, opt-in config, data model, and planned clustering/people-management pipeline People management UI and clustering script are the next milestone. Co-Authored-By: Claude Sonnet 4.6 --- DESIGN.md | 85 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 85 insertions(+) (limited to 'DESIGN.md') diff --git a/DESIGN.md b/DESIGN.md index 7639ffc..a7f368c 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -116,6 +116,32 @@ Password hashing uses `OpenSSL::PKCS5.pbkdf2_hmac` from Ruby's standard library | **ExifTool** | Backing tool for MiniExiftool — must be installed on the server | | **ffmpeg** | Video thumbnail extraction (frame at 2 s) and duration probing via `ffprobe` | +### Optional: facial recognition + +| Component | Purpose | +|-----------|---------| +| **Python 3** | Runtime for `scripts/faces.py` | +| **face_recognition** (PyPI) | dlib-backed face detection and 128-D embedding extraction | +| `/opt/albumen/venv/` | Python virtual environment isolating the dependency | + +Install (one-time, takes ~30 min to compile dlib on CPU): + +```bash +apt install python3-pip python3-dev python3-venv cmake build-essential libopenblas-dev liblapack-dev +python3 -m venv /opt/albumen/venv +/opt/albumen/venv/bin/pip install face_recognition +``` + +Enable by adding to `config.yml`: + +```yaml +faces: + enabled: true +``` + +When disabled (or when the venv doesn't exist), `update.rb` simply skips face detection and the +rest of the app is unaffected. + --- ## Data Model — `album.json` @@ -164,6 +190,7 @@ used. The file is written atomically (write to a `.tmp` file, then | `taken_at` | `null` | ISO 8601 timestamp from EXIF; used for chronological sorting | | `width` / `height` | `null` | Pixel dimensions recorded by `update.rb` | | `exif_absent` | `null` | Set to `true` by `update.rb` when exiftool found no metadata; skips re-extraction on future rescans | +| `faces` | `null` | Set by `update.rb` when `faces.enabled`; array of `{"box": [top,right,bottom,left], "encoding": [128 floats]}` per detected face; `[]` means processed with no faces found; `null` means not yet processed | When `taken_at` is present on *any* file in an album, the entire album is sorted chronologically. Albums with no `taken_at` data stay in filename @@ -385,6 +412,64 @@ to the `albumen` user so the web app can read the files. --- +## Facial Recognition (opt-in) + +Enabled by setting `faces.enabled: true` in `config.yml`. When disabled, +no Python is invoked and no face data is stored or displayed. + +### Detection pipeline + +`update.rb` calls `enrich_faces` for each image file where `meta['faces']` +is `nil` (not yet processed). It shells out to `scripts/faces.py`, which: + +1. Loads the image with `face_recognition.load_image_file` (handles JPEG, + PNG, HEIC, etc. via Pillow). +2. Runs `face_locations(model="hog")` — the HOG model is fast on CPU and + accurate for frontal/near-frontal faces. (The CNN model is more accurate + but requires a GPU to be practical.) +3. For each detected location, calls `face_encodings` to produce a + 128-dimensional L2-normalised embedding vector. +4. Prints a JSON array to stdout; on any error prints `[]` so `update.rb` + always gets valid JSON. + +The result is stored as `meta['faces']` in `album.json`. An empty array +`[]` means "processed, no faces found" and prevents re-processing. A `null` +value means "not yet processed." + +Encodings are stored in full (128 floats each) to allow re-clustering +without reprocessing all images. + +### Clustering and people management (planned) + +A second pass (`scripts/cluster_faces.rb`) will: + +1. Walk all `album.json` files and collect every `{encoding, source_file, + box}` tuple. +2. Cluster them with a threshold distance (~0.6 in L2 space, empirically + good for dlib encodings). +3. Write `/var/albumen/people.json` — a map of stable UUIDs → cluster + metadata (name, representative encoding, member list). + +The admin `/admin/people` UI will let you: +- Name unidentified clusters ("Who is this?"). +- Merge two clusters that are the same person. +- Remove a photo from a cluster (false positive). + +Public `/people` and `/people/:id` routes will let any visitor browse by +person. + +### Performance notes + +- HOG face detection: ~0.5–2 s per image on a single CPU core. +- A library of 10,000 images takes ~3–6 hours to index fully, but the + sentinel-based skip means subsequent `update.rb` runs only process new + photos. +- Encodings stored in `album.json` are ~3.5 KB per face. A library with + an average of 2 faces per photo adds ~70 MB of JSON across 10,000 photos + — negligible. + +--- + ## Security **Path traversal:** `resolve_dir` and `resolve_file` call `File.expand_path` -- cgit v1.2.3