summaryrefslogtreecommitdiffstats
path: root/DESIGN.md
diff options
context:
space:
mode:
authorKen D'Ambrosio <ken@jots.org>2026-06-08 17:09:51 +0000
committerKen D'Ambrosio <ken@jots.org>2026-06-08 17:09:51 +0000
commitda28a20f091372375822f9dde4486ecade859e7e (patch)
tree80d02f26c1b9d52f1a09e36f5d8946b1e3fedf6a /DESIGN.md
parent4ba9f6451f5ab1e5ae95c0871d6fa594f49372cc (diff)
Add opt-in facial recognition: detection and embedding storage
- scripts/faces.py: Python helper using face_recognition (dlib/HOG) to detect faces and return 128-D encodings as JSON; called by update.rb - scripts/update.rb: enrich_faces() stores face boxes and encodings in album.json per image (null = not yet processed, [] = processed/none found); skips files already processed; gated on faces.enabled in config.yml - Reads CONFIG_PATH (same env var as app.rb) to check faces.enabled flag - Feature is off by default; enabled in this install via config.yml - README.md, DESIGN.md: document installation, opt-in config, data model, and planned clustering/people-management pipeline People management UI and clustering script are the next milestone. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Diffstat (limited to 'DESIGN.md')
-rw-r--r--DESIGN.md85
1 files changed, 85 insertions, 0 deletions
diff --git a/DESIGN.md b/DESIGN.md
index 7639ffc..a7f368c 100644
--- a/DESIGN.md
+++ b/DESIGN.md
@@ -116,6 +116,32 @@ Password hashing uses `OpenSSL::PKCS5.pbkdf2_hmac` from Ruby's standard library
| **ExifTool** | Backing tool for MiniExiftool — must be installed on the server |
| **ffmpeg** | Video thumbnail extraction (frame at 2 s) and duration probing via `ffprobe` |
+### Optional: facial recognition
+
+| Component | Purpose |
+|-----------|---------|
+| **Python 3** | Runtime for `scripts/faces.py` |
+| **face_recognition** (PyPI) | dlib-backed face detection and 128-D embedding extraction |
+| `/opt/albumen/venv/` | Python virtual environment isolating the dependency |
+
+Install (one-time, takes ~30 min to compile dlib on CPU):
+
+```bash
+apt install python3-pip python3-dev python3-venv cmake build-essential libopenblas-dev liblapack-dev
+python3 -m venv /opt/albumen/venv
+/opt/albumen/venv/bin/pip install face_recognition
+```
+
+Enable by adding to `config.yml`:
+
+```yaml
+faces:
+ enabled: true
+```
+
+When disabled (or when the venv doesn't exist), `update.rb` simply skips face detection and the
+rest of the app is unaffected.
+
---
## Data Model — `album.json`
@@ -164,6 +190,7 @@ used. The file is written atomically (write to a `.tmp` file, then
| `taken_at` | `null` | ISO 8601 timestamp from EXIF; used for chronological sorting |
| `width` / `height` | `null` | Pixel dimensions recorded by `update.rb` |
| `exif_absent` | `null` | Set to `true` by `update.rb` when exiftool found no metadata; skips re-extraction on future rescans |
+| `faces` | `null` | Set by `update.rb` when `faces.enabled`; array of `{"box": [top,right,bottom,left], "encoding": [128 floats]}` per detected face; `[]` means processed with no faces found; `null` means not yet processed |
When `taken_at` is present on *any* file in an album, the entire album is
sorted chronologically. Albums with no `taken_at` data stay in filename
@@ -385,6 +412,64 @@ to the `albumen` user so the web app can read the files.
---
+## Facial Recognition (opt-in)
+
+Enabled by setting `faces.enabled: true` in `config.yml`. When disabled,
+no Python is invoked and no face data is stored or displayed.
+
+### Detection pipeline
+
+`update.rb` calls `enrich_faces` for each image file where `meta['faces']`
+is `nil` (not yet processed). It shells out to `scripts/faces.py`, which:
+
+1. Loads the image with `face_recognition.load_image_file` (handles JPEG,
+ PNG, HEIC, etc. via Pillow).
+2. Runs `face_locations(model="hog")` — the HOG model is fast on CPU and
+ accurate for frontal/near-frontal faces. (The CNN model is more accurate
+ but requires a GPU to be practical.)
+3. For each detected location, calls `face_encodings` to produce a
+ 128-dimensional L2-normalised embedding vector.
+4. Prints a JSON array to stdout; on any error prints `[]` so `update.rb`
+ always gets valid JSON.
+
+The result is stored as `meta['faces']` in `album.json`. An empty array
+`[]` means "processed, no faces found" and prevents re-processing. A `null`
+value means "not yet processed."
+
+Encodings are stored in full (128 floats each) to allow re-clustering
+without reprocessing all images.
+
+### Clustering and people management (planned)
+
+A second pass (`scripts/cluster_faces.rb`) will:
+
+1. Walk all `album.json` files and collect every `{encoding, source_file,
+ box}` tuple.
+2. Cluster them with a threshold distance (~0.6 in L2 space, empirically
+ good for dlib encodings).
+3. Write `/var/albumen/people.json` — a map of stable UUIDs → cluster
+ metadata (name, representative encoding, member list).
+
+The admin `/admin/people` UI will let you:
+- Name unidentified clusters ("Who is this?").
+- Merge two clusters that are the same person.
+- Remove a photo from a cluster (false positive).
+
+Public `/people` and `/people/:id` routes will let any visitor browse by
+person.
+
+### Performance notes
+
+- HOG face detection: ~0.5–2 s per image on a single CPU core.
+- A library of 10,000 images takes ~3–6 hours to index fully, but the
+ sentinel-based skip means subsequent `update.rb` runs only process new
+ photos.
+- Encodings stored in `album.json` are ~3.5 KB per face. A library with
+ an average of 2 faces per photo adds ~70 MB of JSON across 10,000 photos
+ — negligible.
+
+---
+
## Security
**Path traversal:** `resolve_dir` and `resolve_file` call `File.expand_path`