The LLM Proposes, the Code Disposes: Building a Private Search Engine for My Photos
Like most people's, my photo library is a pile of files named IMG_4821.HEIC. The apps that can search photos by what's in them want the photos on their servers first. I wanted that kind of search without the upload. I wanted to type "golden hour on the beach", "street shots at night with the 35mm" or "me and Rohan in Chicago" on my phone and get the photos back from a machine in my own house.
So I built it. It started as a weekend-sized idea (embed every photo with CLIP, embed the query, return the nearest neighbours) and grew into a hybrid retrieval system. It now has a query parser, image captions, OCR, face clustering, duplicate detection, automatic trip albums, an agent that answers questions about the library, a CLIP model fine-tuned on my own photos, and a phone app. Everything runs on one machine and no photo leaves it.
This post covers how it works and what surprised me. Two things surprised me most:
- The most useful thing I built wasn't a model. It was the evaluation set.
- A few hundred lines of regular expressions beat a 14-billion-parameter LLM at understanding search queries, and the numbers made that obvious.
The code is on GitHub: AtharvaPingale/photo-search.
The shape of the system
photo folders ──watchdog──▶ ingest queue (Redis) ──▶ EXIF/GPS · CLIP · captions+OCR · faces
phone backup ──auto import─┘ │
PostgreSQL + pgvector
│
FastAPI: search / agent / labelling / media
│
React PWA ◀── home network / Tailscale ──▶ your phone
A file watcher notices new photos and puts jobs on a Redis queue. Workers extract EXIF and GPS data, reverse-geocode the location offline, make thumbnails, compute a CLIP embedding, write a caption with a local vision-language model (Florence-2), run OCR where there's likely to be text, and detect faces. It all lands in PostgreSQL with pgvector. A FastAPI service handles search, the agent and the media, and a React app installed on my phone talks to it over Tailscale.
The stack, for the curious: Python 3.12, FastAPI, Postgres 17 + pgvector, Redis + RQ, OpenCLIP, Florence-2, PaddleOCR, InsightFace, HDBSCAN, LangGraph, a local qwen2.5:14b-instruct through Ollama, React + TypeScript + Vite, Docker Compose, MLflow and LangSmith.
I built the eval set before the features
Plain CLIP search is shockingly good on the first try. Type "dog on a beach" and you get dogs on beaches. That's also what makes it dangerous: every feature you add after that feels like an improvement, because you try the queries you expect to work and they do.
So before adding anything to the baseline, I built a way to measure it:
- A labelling tool in the app. Pick a query, tap the photos that are relevant (tap twice for "highly relevant"), save.
- Pooled candidates. The photos you're shown to label come from several systems at once: CLIP alone, captions, keyword and OCR matches, the fused ranking. If you only label what the current best system finds, your labels are biased toward it and every later comparison is rigged.
- 150 starter queries in six categories: objects, attributes and mood, metadata ("portraits with the 85mm"), text in the image ("coffee shop menu"), people, and deliberately hard or ambiguous ones.
- A permanent dev/test split. Each query goes to dev (70%) or test (30%) based on a hash of its id. Tuning code only ever reads dev.
- Photos referenced by content hash, so labels survive re-indexing and moved folders.
Each make eval run reports Recall@5, Recall@20, MRR and nDCG@10 per category with 95% bootstrap confidence intervals, plus p50/p95 latency. It diffs against the previous run and logs everything to MLflow with the git commit. The report also lists the ten worst queries, because averages hide the failures you most need to look at.
Every section below ends in a number, and several of those numbers said "no".
How a query becomes results
"sunset in Tokyo last summer with the X100V"
│ query parser (rules + LLM) filters: place=Tokyo, dates=2026-06-01..08-31, camera=X100V
▼ semantic: "sunset"
CLIP text vector ─┐
caption vector ───┼─ each ranks candidates *inside* the SQL filters ─▶ weighted RRF ─▶ results
keyword tsquery ──┤
OCR tsquery ──────┘
The query gets split into two parts. Anything about when, where, what camera or who becomes a structured filter. What's left ("sunset") is the visual part.
Filters come first. They're SQL WHERE clauses with bound parameters, so "in Tokyo last summer" narrows the candidates before any vector math happens. When the filters leave fewer than 20,000 photos, ranking is an exact scan: a few milliseconds, and it never loses results the way approximate HNSW search can when you filter after it. Bigger candidate sets use the HNSW index.
Four signals rank the candidates independently: CLIP text-to-image similarity, similarity between the query and each photo's caption, full-text search over captions and keywords, and full-text search over OCR'd text.
Reciprocal rank fusion combines them. The signals score on completely different scales: CLIP cosine similarities sit around 0.15 to 0.35, while Postgres ts_rank_cd is unbounded. RRF ignores the scores and only looks at ranks, which makes it the easy, robust choice:
def rrf(rankings, weights=None, k=60):
weights = weights or {}
scores, ranks = {}, {}
for signal, ids in rankings.items():
w = weights.get(signal, 1.0)
if w <= 0:
continue
for r, pid in enumerate(ids, start=1):
scores[pid] = scores.get(pid, 0.0) + w / (k + r)
ranks.setdefault(pid, {})[signal] = r
return sorted(
((pid, s, ranks[pid]) for pid, s in scores.items()),
key=lambda t: (-t[1], min(t[2].values()), str(t[0])),
)
A photo gets w / (k + rank) from each signal that found it and nothing from the ones that didn't. Ties break deterministically, so eval runs are reproducible.
A few other things make it feel good to use. What the parser extracted shows up as editable chips, so you can see that "Tokyo" became a place filter and remove it if that's wrong. If the filters match nothing, the search falls back to pure semantic search and says so. And 👍/👎 buttons in the photo viewer nudge the query vector (Rocchio feedback), with every click logged as a future training example.
On my demo library, typical searches take 20 to 40 ms warm, end to end.
Regexes beat the LLM
This was the most instructive part of the project.
The original plan was the obvious one: give an LLM the query and a strict JSON schema (date_from, date_to, place, camera, lens, people, …) and let it fill in the filters. Schema-constrained decoding means the output always parses. I also wrote a rule-based parser to compare against. It's a deterministic set of regexes for dates, seasons, holidays, focal lengths, apertures and ISO, plus lookups against the library's own vocabulary: the cameras, lenses, places and people that actually exist in my database.
Then I labelled 62 queries with the correct filters, each with its own fixed "today" so that relative dates are testable. Later I wrote 25 more that I never looked at while tuning. Here's what happened (field accuracy / exact match / median latency):
| parser | tuning set (62) | held-out set (25) |
|---|---|---|
| rules | 0.976 / 0.968 / 0.1 ms | 0.946 / 0.960 / 0.1 ms |
| LLM, raw (schema-constrained) | 0.364 / 0.081 / 1.8 s | 0.458 / 0.080 / 2.1 s |
| LLM, validated | 0.720 / 0.774 / 2.0 s | 0.829 / 0.800 / 2.0 s |
| hybrid (rules + validated LLM, always) | 0.988 / 0.984 / 1.9 s | 0.921 / 0.920 / 2.0 s |
| auto (LLM only when rules leave cues unexplained) | 0.976 / 0.968 / 0.1 ms | 0.946 / 0.960 / 0.1 ms |
The raw LLM was bad in a very specific way. The JSON was always valid; the content wasn't. It added media: "photo" to nearly every query. It turned "beach" into a place. It answered "Tokyo, Japan" when the library calls it "Tokyo". It invented a camera from a lens mention. It got relative dates wrong: asked in September, "last summer" became the previous year's summer. Schema-constrained decoding guarantees the shape of the answer, not its truth.
The fix was to stop trusting it. Every filter the LLM proposes must be backed by evidence in the query text and, where possible, match something that exists in the library:
def validate_llm_fields(raw, q, vocab):
"""The LLM proposes, the code disposes: every filter needs evidence in the query
text and, where there's a vocabulary, a match in the library."""
f = _coerce(raw, vocab)
low = q.lower()
if f["media"] != "video" or not re.search(r"\b(videos?|clips?|footage|movies?)\b", low):
f["media"] = None
...
f["people"] = [p for p in f["people"] if _mentions(q, p)]
if not _DATE_CUES.search(q):
f["date_from"] = f["date_to"] = f["months"] = None
return f
Validation lifted the LLM from 0.46 to 0.83 on the held-out set. It still lost to the rules. Worse, the hybrid that always asks the LLM and merges its answer was worse than rules alone on held-out queries (0.921 vs 0.946). The LLM added more wrong guesses than it fixed.
The drop from 0.976 to 0.946 for the rules on the held-out set is worth noticing too. I wrote those rules while looking at the tuning set, and that gap is exactly how optimistic the tuning number was. I kept the held-out failures as they are. Rules read "christmas lights 2023" as all of 2023 instead of Christmas 2023. The LLM decided "the drone over the lake" meant South Lake Tahoe. Fixing them now would quietly turn the held-out set into a second tuning set.
In the end, latency settled the architecture. Rules take about 0.1 ms and the LLM about 2 seconds. The default auto mode always runs the rules, then checks whether anything left over in the query still looks like metadata (a date word, a known person, a capitalised word mid-sentence). Only then does it call the LLM, and the rules win any conflict. "cats in Chicago" is fully explained by the rules and never reaches the LLM. Only 7 of the 62 tuning queries do. auto matches the rules' accuracy at a 0.1 ms median, and the LLM is still there for the odd query the rules can't handle.
Captions didn't earn their place (in the smoke test)
Before labelling my own library, I ran a sanity check on 300 COCO val2017 photos with synthetic EXIF. Each query is a human-written COCO caption and the only relevant result is the photo it describes. This test has real limits: it uses web photos instead of a personal library and has exactly one right answer per query. But it runs every real model end to end, and the results were humbling:
| configuration | nDCG@10 (all) | metadata | objects | p50 ms |
|---|---|---|---|---|
| CLIP only | 0.881 | 0.830 | 0.899 | 7 |
| CLIP + query parser | 0.908 | 0.947 | 0.894 | 8 |
| + captions, equal weights | 0.862 | 0.947 | 0.832 | 15 |
| + OCR, equal weights | 0.858 | 0.947 | 0.827 | 16 |
| + tuned fusion | 0.907 | 0.960 | 0.889 | 15 |
The parser does exactly what it's for: metadata queries go from 0.830 to 0.947. But captions hurt at equal weight, and after tuning the fusion weights (5-fold cross-validation on dev) the full system only ties CLIP plus the parser. The tuner settled on CLIP 1, caption 0.5, OCR 0.25 and keyword 0. On this data, keyword search over Florence-2's long, generic captions was pure noise.
That result makes sense. COCO queries are exactly the kind of caption CLIP was trained on, so CLIP has a home-field advantage. Whether captions justify their GPU hours on a personal library, where queries involve counts, actions and text, is a question only my own labelled eval can answer. The honest answer until then is "not proven". I'm glad the eval says so instead of letting me assume.
Bugs the eval caught
That same smoke test found four bugs that no unit test would have:
- The default fusion weights referenced a signal name that didn't exist.
- The tuner's "CLIP only" baseline was quietly giving every other signal weight 1, so the baseline wasn't CLIP only.
- The offline geocoder tagged photos with neighbourhoods ("Chicago Loop", "Hatsudai"), so "in Tokyo" matched nothing.
- The agent sorted results by date before cutting the list down, then confidently answered "you have no giraffe photos". There were ten.
None of these crash anything. They just make the system a bit worse in ways you'd never notice by trying queries by hand.
Faces, opt-in
Face recognition is the feature I was most careful with. It's off unless you list specific folders in PS_FACE_ROOTS. Inside those, InsightFace detects faces and computes embeddings on the GPU, and HDBSCAN clusters them. A People page lets you name clusters (including an alias like "me"), merge them, split faces off, or mark "not this person".
Re-clustering keeps names by letting each new cluster inherit the id of the old cluster it overlaps most, and it never overrides a manual edit. Once people are named, "me and Rohan hiking" filters to photos containing both of us. photo-search faces wipe deletes every detection, embedding, cluster, name and crop in one go.
Duplicates, bursts and trips
The organisation features are simple heuristics that are easy to check:
- Near-duplicates require two independent signals to agree: CLIP cosine similarity ≥ 0.95 and perceptual-hash distance ≤ 10 bits. The suggested keeper is the largest file, then the sharpest.
- Bursts are shots from the same camera ≤ 2 seconds apart that look similar. The best pick is the one with the highest variance of the Laplacian, a cheap sharpness score.
- Trip albums come from splitting the timeline at 8-hour gaps. "Home" is the region with the most photo-days, and consecutive away-from-home stretches merge into a trip. The LLM writes a title and summary from the places, dates and captions.
Duplicate detection has its own labelled eval. The labelling sampler deliberately mixes in near-misses along with the flagged pairs, so that recall is measurable and not just precision.
An agent that can't make up photos
The Ask tab answers questions like "What was my most photographed city in 2025?" or "When did I last shoot the skyline?". It's a LangGraph agent with four tools, and it never writes SQL:
| tool | what it does |
|---|---|
semantic_search | the normal search engine, with an optional "newest first" sort |
metadata_query | one of five named, parameterised SQL templates |
aggregate | counts grouped by a whitelisted field (city, camera, lens, year, person…) |
get_photo_details | metadata, caption, OCR text and people for one photo |
Arguments are validated with pydantic before they get near the database, and all values are bound parameters.
The final answer has to take the form {answer, evidence: [{photo_id, note}]}. The code then checks every cited photo id against the ids the tools actually returned during that run. Any id the agent never saw is dropped. An answer counts as grounded only if it cites at least one real photo and invented none. The UI shows the evidence as thumbnails, so every claim comes with pictures you can check.
For evaluation, a script generates short-answer questions whose ground truth comes from the library through the same safe templates, not from the agent. The agent eval scores correctness (string match plus an LLM judge), groundedness, hallucinated citations and tool calls per answer. It can also collect my own grades, so the report shows how well the LLM judge agrees with a human (Cohen's kappa) instead of assuming it does.
Fine-tuning CLIP without fooling myself
The last model piece fine-tunes CLIP on my own photos. Training pairs come from three places: the first sentences of the VLM captions, Lightroom/XMP keywords turned into text ("a photo of beach, sunset, Portugal"), and 👍 clicks from search.
The training uses rank-8 LoRA adapters on every attention and MLP weight, implemented with torch parametrizations. That trick also works on nn.MultiheadAttention's fused in_proj_weight, which most LoRA helpers skip. The adapters get merged into the weights on save. An optional WiSE-FT step interpolates back toward the original weights so the model keeps CLIP's general knowledge. The new embeddings are stored under a new model name next to the originals, so comparing before and after needs no re-indexing.
The part I spent the most time on was leakage, because fine-tuning on your own eval data is the easiest way to produce an impressive, meaningless number:
- Every photo labelled relevant for any eval query is excluded from training. So is its near-duplicate and burst group, and anything shot within 10 minutes on the same camera. The same scene photographed twice isn't an independent example.
- Feedback queries that match an eval query (same normalised text, or token overlap ≥ 0.6) are dropped.
- The validation split used during training is split by day, not by photo, so near-identical shots can't land on both sides.
- Fusion weights and the WiSE-FT mix are chosen on dev. Test runs once, at the end.
The report is per category and includes the categories that get worse. Fine-tuning on captions tends to help objects and scenes and can hurt text-in-image and hard queries.
On the phone
The web app is a PWA, so "Add to Home Screen" makes it feel like a native app. The grid uses 256 px WebP thumbnails, about 8 KB each. Their URLs contain the content hash, so the service worker can cache them forever. The photo viewer loads a 2048 px version first and the full-resolution original only when you ask for it. Share and Save buttons hand the actual photo to the phone's share sheet. There's also voice search through the browser's speech API. That audio goes to Apple's or Google's recogniser, but photos never do.
Nothing is exposed to the internet. From anywhere, including mobile data, the app is reached through Tailscale, and tailscale serve provides a real HTTPS certificate. At home it can also listen on the local network only.
Access is protected by a token, and getting it onto the phone was a small design problem of its own. photo-search pair prints a QR code that encodes a link with the token in the URL fragment. Fragments are never sent to the server or written to its logs. The app reads the fragment once and exchanges it for an HttpOnly cookie. Rotating the token logs every device out.
The unglamorous parts
Most of the code isn't search. It's the stuff that makes a personal photo library usable:
- RAW files are decoded from their embedded JPEG preview whenever one exists. That's 10 to 50 times faster than a full decode and plenty for a thumbnail and a 224 px CLIP input. HEIC works through
pillow-heif. - Videos are sampled at scene changes, with at least one frame every 20 seconds and at most 60 per video. Each frame is indexed like a photo, and search collapses frames back into their video with the best-matching timestamp.
- OCR is gated. A zero-shot CLIP probe on the embedding that's already stored decides whether a photo probably contains text. In the 300-photo test, only 23 photos went on to the expensive OCR model.
- The watcher polls instead of relying on file events, because WSL doesn't deliver events for Windows drives. In my setup, a new photo became searchable 36 seconds after it landed.
- Auto import watches my phone's backup folder and files new photos into
Imported/YYYY/YYYY-MM/. It de-duplicates by content hash, because backup apps love re-uploadingIMG_0001 (1).HEIC. It keeps a ledger of every file it has ever imported, so a photo I delete from the library doesn't come back on the next sync. - Nightly
pg_dumpbackups with one-command restore. The database is the part worth protecting: the embeddings and captions are hours of GPU time, and the names, merges and albums are my own work. - Dependency wrangling. InsightFace wants
opencv-python, PaddleX pinsopencv-contrib-pythonand checks for it by name, and other packages pull in the headless build. All three install into the samecv2/package and break each other. The fix is to explicitly exclude two of them inpyproject.toml, with a comment explaining why, so future me doesn't "fix" it.
On my machine (Windows 11 + WSL2 + Docker Desktop, RTX 5070 Ti), indexing 300 photos took 53 seconds for scanning, thumbnails, CLIP and Florence-2 captions, then 31 seconds for OCR and 32 seconds for face detection. Those times include loading each model. Captioning dominates the first index of a large library, so every stage is resumable.
Does it scale?
A load test inserts 100,000 synthetic photos (random 512-dimensional embeddings with plausible metadata) and runs queries through the real code path, CLIP text encoder included:
| bulk insert (COPY) | 2.6 s |
| HNSW index build | 40 s |
| storage | 50 MB of photo rows, 566 MB of embeddings (incl. a 273 MB HNSW index) |
| query p50 / p95, no filters | 17 / 19 ms |
| query p50 / p95, place filter (~20k matches) | 90 / 110 ms |
| query p50 / p95, place + year | 19 / 21 ms |
The slowest case is a filter that keeps about 20% of the library. That's small enough to be ranked exactly and big enough that the exact scan costs something. It's still well under my 300 ms target. Postgres with pgvector has been entirely sufficient. I never needed a dedicated vector database.
What I'd tell someone building something similar
Build the eval before the features. Every interesting result in this post, including the ones that said "no", exists because the eval came first. Without it I would have shipped captions as an obvious improvement and the LLM parser as the obvious architecture.
Structured output isn't correct output. A JSON schema guarantees shape, not truth. When an LLM produces something your code will act on, make it show evidence and check that evidence against reality. Let the model propose and the code decide.
Let latency shape the design. A 2-second model call in front of every search is a different product from a 0.1 ms rule. Routing only the hard cases to the model gave me the accuracy of the rules and the flexibility of the LLM.
Protect your held-out set. Fixing held-out failures feels responsible. It actually destroys the only unbiased number you have.
Rank fusion is a great default, and a weight of zero is a valid answer. RRF let me combine signals with incompatible score scales in about fifteen lines. Tuning the weights on dev told me which signals weren't pulling their weight.
The code, with setup instructions for Docker or a native install, is at github.com/AtharvaPingale/photo-search. My next step is labelling my own library properly, to find out whether captions and fine-tuning earn their keep on real personal photos.