diff options
Diffstat (limited to 'DESIGN.md')
| -rw-r--r-- | DESIGN.md | 85 |
1 files changed, 85 insertions, 0 deletions
@@ -116,6 +116,32 @@ Password hashing uses `OpenSSL::PKCS5.pbkdf2_hmac` from Ruby's standard library | **ExifTool** | Backing tool for MiniExiftool — must be installed on the server | | **ffmpeg** | Video thumbnail extraction (frame at 2 s) and duration probing via `ffprobe` | +### Optional: facial recognition + +| Component | Purpose | +|-----------|---------| +| **Python 3** | Runtime for `scripts/faces.py` | +| **face_recognition** (PyPI) | dlib-backed face detection and 128-D embedding extraction | +| `/opt/albumen/venv/` | Python virtual environment isolating the dependency | + +Install (one-time, takes ~30 min to compile dlib on CPU): + +```bash +apt install python3-pip python3-dev python3-venv cmake build-essential libopenblas-dev liblapack-dev +python3 -m venv /opt/albumen/venv +/opt/albumen/venv/bin/pip install face_recognition +``` + +Enable by adding to `config.yml`: + +```yaml +faces: + enabled: true +``` + +When disabled (or when the venv doesn't exist), `update.rb` simply skips face detection and the +rest of the app is unaffected. + --- ## Data Model — `album.json` @@ -164,6 +190,7 @@ used. The file is written atomically (write to a `.tmp` file, then | `taken_at` | `null` | ISO 8601 timestamp from EXIF; used for chronological sorting | | `width` / `height` | `null` | Pixel dimensions recorded by `update.rb` | | `exif_absent` | `null` | Set to `true` by `update.rb` when exiftool found no metadata; skips re-extraction on future rescans | +| `faces` | `null` | Set by `update.rb` when `faces.enabled`; array of `{"box": [top,right,bottom,left], "encoding": [128 floats]}` per detected face; `[]` means processed with no faces found; `null` means not yet processed | When `taken_at` is present on *any* file in an album, the entire album is sorted chronologically. Albums with no `taken_at` data stay in filename @@ -385,6 +412,64 @@ to the `albumen` user so the web app can read the files. --- +## Facial Recognition (opt-in) + +Enabled by setting `faces.enabled: true` in `config.yml`. When disabled, +no Python is invoked and no face data is stored or displayed. + +### Detection pipeline + +`update.rb` calls `enrich_faces` for each image file where `meta['faces']` +is `nil` (not yet processed). It shells out to `scripts/faces.py`, which: + +1. Loads the image with `face_recognition.load_image_file` (handles JPEG, + PNG, HEIC, etc. via Pillow). +2. Runs `face_locations(model="hog")` — the HOG model is fast on CPU and + accurate for frontal/near-frontal faces. (The CNN model is more accurate + but requires a GPU to be practical.) +3. For each detected location, calls `face_encodings` to produce a + 128-dimensional L2-normalised embedding vector. +4. Prints a JSON array to stdout; on any error prints `[]` so `update.rb` + always gets valid JSON. + +The result is stored as `meta['faces']` in `album.json`. An empty array +`[]` means "processed, no faces found" and prevents re-processing. A `null` +value means "not yet processed." + +Encodings are stored in full (128 floats each) to allow re-clustering +without reprocessing all images. + +### Clustering and people management (planned) + +A second pass (`scripts/cluster_faces.rb`) will: + +1. Walk all `album.json` files and collect every `{encoding, source_file, + box}` tuple. +2. Cluster them with a threshold distance (~0.6 in L2 space, empirically + good for dlib encodings). +3. Write `/var/albumen/people.json` — a map of stable UUIDs → cluster + metadata (name, representative encoding, member list). + +The admin `/admin/people` UI will let you: +- Name unidentified clusters ("Who is this?"). +- Merge two clusters that are the same person. +- Remove a photo from a cluster (false positive). + +Public `/people` and `/people/:id` routes will let any visitor browse by +person. + +### Performance notes + +- HOG face detection: ~0.5–2 s per image on a single CPU core. +- A library of 10,000 images takes ~3–6 hours to index fully, but the + sentinel-based skip means subsequent `update.rb` runs only process new + photos. +- Encodings stored in `album.json` are ~3.5 KB per face. A library with + an average of 2 faces per photo adds ~70 MB of JSON across 10,000 photos + — negligible. + +--- + ## Security **Path traversal:** `resolve_dir` and `resolve_file` call `File.expand_path` |