Search stack

This page explains how the search works, end to end, for someone who knows search in general but not this build. Read the left side first: it builds the catalogue once from raw data, then shows what it costs to add one more product. The right side lets you run a real query and watch every step, with the time each one took. Every number is read from the files named in the source cells, or measured live against the running service. Where a time was never written down, the cell says so, and where a number is worked out rather than measured, it is marked derived.

What this running process is set to right now (both sides use it):

Building the index from scratch

An index is a set of lookup structures built ahead of time, like a library catalogue, so that a search touches a few thousand rows instead of reading all 1,032,532 every time. This happens once (or on a rebuild), offline. The arrows below feed the live table; dashed boxes were built or tested but are not served.

flowchart LR
  M["Raw catalogue: Marqo-GS-10M (a public shopping dataset)
subset marqo_gs_wfash_1m
1,032,532 products, each with a title and one photo"]
  IMG["The photos, saved locally
images_wfash/, 1,032,532 files, 21 GB"]
  TTL["The product titles (English)"]
  SYN["Made-up price, sizes, in-stock flag
app/loader.py, marked as synthetic on every row"]
  TAX["Category tree, hand-written
86 nodes, 70 leaves, 15 families
named in DE EN FR ES IT"]
  E1["Turn each photo into a 768-number vector
FashionSigLIP image tower, pipeline/embed_images.py
104 batches, 14,502 s in total"]
  E2["Turn each title into a 1024-number vector
bge-m3, pipeline/embed_titles.py
104 batches, 4,938 s in total"]
  LEAF["Pick one category leaf per product
pipeline/product_leaf.py + leaf_reroute.py
from the title: 703,241, from the photo: 329,291
out of scope: 129,873"]
  LOAD["Load into LanceDB, app/loader_full.py
keyword index on titles, filter indexes,
nearest-neighbour index on both vector columns
colours read from titles: 508,890 products"]
  LIVE[("The live table (build9)
1,032,532 rows, 7.3 GB")]
  ATTR["Guessed attributes, pipeline/attrs.py
309,459 products
in build10 only, not live"]
  CARD["Style cards, Qwen3-VL-4B
pilots only, no full run"]
  FP["Street photos, Fashionpedia
used only to test photo search"]
  ESCI["Amazon ESCI dataset
downloaded, never used"]
  M --> IMG
  M --> TTL
  IMG --> E1
  TTL --> E2
  TAX --> LEAF
  TTL --> LEAF
  E1 --> LEAF
  E1 --> LOAD
  E2 --> LOAD
  LEAF --> LOAD
  TTL --> LOAD
  SYN --> LOAD
  LOAD --> LIVE
  E1 -.-> ATTR
  IMG -.-> CARD
  classDef off stroke-dasharray: 5 4,opacity:0.75
  class ATTR,CARD,FP,ESCI off

Roughly 5 to 6 hours of one-time work on a single 16 GB Mac, almost all of it turning a million photos into vectors. That is the sum of the per-stage times recorded below (derived); it only happens on a full build.

The stages, in order

Each row is one stage. "Items" is how much it processed, "time" is how long that stage took, and "source" is the file the number comes from.

Stage	What it does, in plain terms
Raw catalogue	The starting data: a public Google-Shopping dataset (Marqo-GS-10M), filtered to a womens-fashion slice. Each product is a title plus one photo. No price, size or category yet.
Made-up commerce fields	The dataset has no price, sizes or stock, so the loader invents them from the product id (the same id always gives the same values) and labels them synthetic so nothing pretends they are real.
Category tree	A hand-written list of clothing categories in two levels, family then leaf (for example `skirt` then `skirt.midi`), with names in five languages. This is the vocabulary everything else maps onto.
Photo vectors	A vision model (FashionSigLIP) reads each photo and turns it into 768 numbers that place similar-looking clothes near each other. This is what lets "find clothes that look like this" work. Runs on the Mac GPU.
Title vectors	A text model (bge-m3) turns each title into 1024 numbers by meaning. Stored, but the live search does not use it (kept for experiments).
Category per product	Give every product one leaf. First a simple rule reads the main noun in the title ("skort" to `skirt.skort`). If the title is unclear, the vision model guesses from the photo instead. A tuned threshold decides which to trust.
Load into the table	Write every product into LanceDB and build the lookup structures: a keyword index on titles (for word matches), filter indexes (price, size, colour, category), and a nearest-neighbour index on each vector column (for "looks like" search). Colours are read from the title words here.
Nearest-neighbour index	The "looks like" index (type IVF_PQ): it splits the million vectors into 1,024 groups so a search checks a few groups, not all million. Built once per vector column.
Guessed attributes (not live)	Colour, pattern, sleeve and so on, guessed from the photo where the title is silent. Built into build10, not the live table.
Style cards (not live)	A vision-language model writing a short description per item. Only small pilots were run; the live table has none.
Street photos (test only)	Real-world outfit photos (Fashionpedia), used only to measure how well photo search works.
Amazon ESCI (unused)	Another search dataset that was downloaded but never wired into the pipeline.

One product's journey

To make the stages concrete, here is one real row, product 10001248868022977969, as it moves down the solid-arrow path. Each card shows only the fields that stage adds; bold is what that stage wrote. Values are read from the files today.

Start: the raw record

corpus_2.json

All the dataset gives us is a title and a pointer to a photo. Nothing else.

title: Sweaty Betty Power Skort, Black, Women's XXL
photo: /MarqoData/google_shopping/images/10001248868022977969.webp
seen in: the query "Designer brand athletic skorts", ranked 25th there (how the dataset was collected)

Save the photo locally

images_wfash/

The photo is downloaded and named by product id, so every later stage can find it.

file: 10001248868022977969.webp
size: 9,961 bytes

Photo to 768 numbers

FashionSigLIP image tower

The vision model reads the picture and outputs a point in "fashion space" where lookalikes sit together.

image_vec: 768 numbers (first one is 0.0016, read from the live row)

Title to 1024 numbers

bge-m3

The title is also turned into a vector by meaning. Stored, but the live search does not use it.

text_vec: 1024 numbers (first one is -0.0418)

Give it a category

product_leaf.parquet

The title rule finds the word "Skort", which names exactly one leaf, so the category comes from the title and the photo-guess is not needed here.

leaf: skirt.skort
family: skirt
decided by: the title rule
confidence: 0.392
in scope?: yes

Invent price, size, stock

app/loader.py

Computed from the product id so they are stable, and flagged as not real.

price: 106.93
sizes: L, XL
in stock: yes
marked synthetic: price, sizes, in_stock

Write the row into the table

loader_full.py → build9

The loader reads the colour from the title words, builds the text the keyword search matches against, and stores the row with both vectors and all the indexes. This product has no style card.

colour: black (from the word in the title)
keyword text: Sweaty Betty Power Skort, Black, Women's XXL
brand: none (the title rule could not pull one out)

Live

lancedb_live → build9

A pointer aims the service at build9, so this exact row is now what a query can match and return.

findable by: keyword match on the title, "looks like" on the photo vector, and filters on category, colour, price and size

Adding one more item (the marginal cost)

The catalogue above already exists. A new product arrives. What does it cost to make this one item searchable? This is the same code path as a build, run for a single row. Per-item times with derived are worked out from the batch throughput measured above, not timed one row at a time.

Embed its photo

about 13 to 22 ms derived

Run the one new photo through the image tower for its 768 numbers. (The model itself has to be loaded first, which is a one-time cost, not per item.) Derived from the measured 46 to 78 photos per second.

Embed its title

about 5 ms derived

Run the title through bge-m3 for its 1024 numbers. Derived from the measured 204 titles per second.

Assign a category

about 0.2 ms measured

The title rule reads the main noun (the measured parse median is 0.21 ms). If the title is unclear, one extra photo comparison against the 66 leaf prompts settles it, reusing the vector from step 1.

Fill in the rest

microseconds

Price, size and stock from the product id; colour from the title words; the keyword text assembled. All arithmetic and string work, effectively free.

Append the row

one write

LanceDB adds the row to the table as a new fragment. It does not rewrite the existing million rows. Fast, but see the next step for the catch.

Make it findable at full speed

photos 138 s + titles 155 s measured

Here is the real cost. The keyword, filter and "looks like" indexes only cover the rows that existed when they were last built. The new row is not in them until they are rebuilt. This repo rebuilds them over the whole table (replace=True in loader_full.py), so one extra product really means re-running the index build across all 1,032,532 rows.

Why one item is cheap to store but costly to index. The per-item compute is about 40 ms, a rounding error. But the "looks like" index (IVF_PQ) is a single structure built from all the vectors at once, like a sorted phone book: inserting one name means the whole book is out of order until you re-sort it. Until a rebuild, the service still returns correct results by scanning the un-indexed new rows directly, just without the index speed. LanceDB can also fold new rows in incrementally instead of a full rebuild, but this repo always does the full rebuild, so the honest marginal cost here is append in milliseconds, re-index in minutes.

Trace a real query

Type a query and press Trace. It runs the exact code path a normal search uses, through POST /v1/trace, and shows every step with the time it took on this machine. Trace requests are not saved to the query log. The first query after a restart is slow because models are still warming up.

The query path, step by step

The steps one text search runs, in the order the service runs them. "Typical time" is what was recorded earlier; "this process" fills in from your last trace above. A search sends one fast first answer, then may refine it with a slower second pass.

#	Step	What it does	Typical time
1	Clean up the text	Lowercase, strip accents, turn umlauts into ae/oe/ue, pull out numbers. So "Größe 38" and "grosse 38" match.	0.21 ms median, 0.97 ms p95 (status.md:637)
2	Guess the language	Use the language you picked, else the one with the most tell-tale words, else the shop default (German).
3	Read it with rules	Match words to the category tree and build "chips": a hard chip filters (must match), a soft chip only nudges ranking. A chip is allowed to be hard only if the dictionary or your own text backs it up.
4	Pre-filter	Turn the hard chips into a database WHERE clause (category, price, size, colour, "not this"), plus hide out-of-scope rows. Everything below searches inside this filter.	part of retrieval
5	Keyword search (BM25)	Classic word-matching over titles, top 200 inside the filter.	42 ms median, 74 ms p95 (build9.out:24)
6	Combine and tidy (first answer)	Merge the lists, apply soft boosts, demote maternity and workwear, drop duplicate photos, spread out the top, enforce the category. This is the fast answer the shopper sees first.	77 ms median, 89 ms p95 (status.md:814)
7	Ask the small LLM (only if words are left over)	A local model (Qwen3-4B) rewrites the query into clean slots as JSON. Same gate: it may set a hard value only if the dictionary or your text supports it, so it cannot invent filters.	2,139 ms median, 2,867 ms p95 (measure_stack.log:7)
8	Query to a photo vector	Encode the meaning of the words into the same space as the photos, so text can find lookalike images.	17 ms (status.md:467)
9	"Looks like" search	Nearest-neighbour search on the photo index for the 200 products whose picture best matches the query, inside the same filter.	10.3 ms median (loader_full.py:209)
10	Combine again (refined answer)	Merge keyword and "looks like" results; after an LLM wait the first 10 items the shopper already saw are held in place so the page does not jump.	219 ms median, 243 ms p95 (status.md:814)
off	Title-vector search	The bge-m3 vectors are not used by the live search.	34 ms to encode (status.md:467)
off	Cross-encoder re-rank	A heavier re-scorer; switched off here (it barely helped and costs 286 ms).	286 ms for 50 (status.md:467)
photo	Search by photo	Upload a picture: it is encoded and "looks like" search runs on it directly, in one pass, adding a soft category chip if the top results agree.	21 ms to encode (status.md:26)

What was trained versus used as is

No model here was trained from scratch or fine-tuned; the repo has no training code. Everything runs on one 16 GB Mac. "Tuned" means a few numbers were fitted (a threshold, some weights); "as is" means the model is used straight out of the box; "rule code" means no model at all.

Component	Model or method	Kind	Fitted or checked on	Result	Source
Photo and query-text vectors	`Marqo/marqo-fashionSigLIP`	as is	n/a	the single biggest quality gain; photo "looks like" beats keyword search on the test slice	docs/status.md:462-465
Title vectors	`BAAI/bge-m3`	as is	n/a	stored, not used by the live search	docs/status.md:403
Title-vs-photo category switch	which one to trust for the leaf	tuned	299 products hand-labelled from their photos, split 150/150	exact leaf right 83 to 85%, right family 90 to 91%	docs/status.md:477-480, :528-531
Category from the photo	FashionSigLIP against 66 leaf prompts	as is	same 299 labels	photo alone 75% exact, title rule alone 56%	docs/status.md:480-481
How the result lists are merged	weighted RRF, grid-searched, `ranking/tune.py`	tuned	550 queries, 30,000 products, graded by the dataset (not by people)	keyword 1, photo 2, title 1; all three near 0.35 to 0.37 quality	pipeline/out/ranker_weights.json; docs/status.md:462-465
Cross-encoder re-rank	`BAAI/bge-reranker-v2-m3`	as is	test slice	within noise, and slow (286 ms per 50); off in the live config	docs/status.md:464, :467
Attribute thresholds (build10)	FashionSigLIP per-attribute confidence	tuned	300 products judged from the photo	at confidence 0.7: colour 0.99, pattern 0.98, fit 0.97, length 0.78	eval/attr_judge/summary.txt:1-21
Query-rewrite LLM	`Qwen3-4B-Instruct-2507-4bit`, prompt v2, temperature 0	as is (prompted)	63 hard queries; 28 vague ones for the gate	always valid JSON; it never smuggled an invented filter past the gate	bench/bench_llm_parse.log:2-5; bench/gate_proof.log:115-116
Rule parser and gate	dictionary from the tree plus grammar, no model	rule code	200 category queries (5 languages), 63 hard cases	zero wrong hard filters across 2,000 and 630 checks; 0.21 ms median	docs/status.md:636-638, :810-811
Human labels	Helia, 100 blind products	check	n/a	agreed on family 70/100, then 83/96 after re-routing	docs/status.md:280-286
Model-as-judge checks	model subagents	check	299 leaf labels, 300 attribute judgements, result samples	a 300-result sample on an earlier build had 0 clearly wrong	docs/status.md:169, :812