More matching results need a precise score
Amazon S3 Vectors introduced metadata pre-filtering through enhanced index mode on September 30, 2026. The release also added the $startsWith prefix operator and in-place upgrades without re-ingestion or an additional feature charge; ordinary service charges remain. AWS’s walkthrough showed a tenant query returning two results before migration and ten afterward. That example lacks a reproducible benchmark methodology. Sources: AWS News Blog
AWS’s announcement says enhanced indexes can return up to five times as many matching vectors as classic indexes for highly selective filters; its headline gives no figure, and AWS does not report a recall improvement. More matching vectors is not the same as higher recall: more eligible results need not include the nearest eligible neighbors. A useful benchmark separates filter correctness, result completeness, and recall against an exact reference. The following protocol is a proposed experiment, not a report of new AWS measurements.
Establish which filtering modes can be compared
AWS describes enhanced mode as identifying matching vectors before similarity search and classic mode as evaluating filters during candidate search. Calling classic mode client-side post-filtering would misstate that distinction. Keep a separate application baseline that retrieves unfiltered candidates and filters them locally. The table below defines four experimental arms. Sources: AWS
Table
Four retrieval arms and their purposes
| Arm | Operation | Purpose |
|---|---|---|
| Exact filtered reference | Filter locally, exhaustively rank | Establish nearest eligible IDs |
| Enhanced mode | Find matches before similarity search | Measure managed pre-filtering |
| Classic mode | Evaluate filters during candidate search | Measure legacy service behavior |
| Client post-filter | Retrieve unfiltered candidates, filter locally | Measure candidate survival |
Check bucket defaults and index modes before provisioning. Buckets created on or after September 30, 2026, default to enhanced mode and cannot host classic indexes; older buckets default to classic unless changed. A classic index accepts an enhanced query override, allowing paired requests against the same unchanged index. An enhanced index rejects a classic query override. Without an eligible older bucket, omit the measured classic arm. Sources: AWS
Client post-filtering creates a candidate-budget problem. If eligibility is independent of similarity rank, expected survivors equal candidate count multiplied by eligible fraction. That expectation guarantees neither enough survivors nor the nearest eligible vectors. Tenant membership correlated with document topics can change the outcome.
Build a corpus that exposes selective-filter failures
For the proposed experiment, start with 50,000 deterministic vectors, then repeat selected conditions at 500,000 and one million. Preserve uploaded float32 arrays, metadata, query vectors, seed, embedding configuration, and a corpus hash. Use clustered synthetic vectors with independent tenant labels, then repeat with topic-aligned and dispersed assignments. Add real documents with fixed chunks, several chunks per document, and tenant, document, category, active-status, and numeric timestamp fields.
Compute eligible IDs directly from the metadata manifest for every predicate. Do not estimate conjunction selectivity by multiplying individual fields’ match rates. Cross the proposed conditions below, reserving boundary tests for targeted runs. Include fixed-fraction and fixed-count scaling so corpus growth can be distinguished from growth in the eligible search population.
- Requested result count K: 5, 10, 20, and 50.
- Eligible fraction: 50%, 10%, 1%, 0.1%, and 0.01%.
- Eligible-count boundaries: zero, one, three, K minus one, K, K plus one, and twice K.
- Filter shapes: tenant; tenant plus document; tenant plus document list; tenant plus status and time.
- Client candidate budgets: K, 100, 1,000, and 10,000.
- Extensions: document-path prefixes and constraint pressure at 99, 100, and 101 evaluated values.
Keep direct mode comparisons to shared operators because $startsWith requires enhanced behavior. For prefixes, use the guide’s field-first syntax, placing $startsWith inside document_id; the launch article shows a conflicting arrangement. Materialize an equivalent grouping field and verify that equality and prefix predicates select identical IDs. Enhanced indexes allow 100 constraints, with each $in value counting separately, and numeric comparisons require numbers. Keep filterable metadata within the documented 2 KB per-vector allowance. Sources: AWS News Blog, AWS
Score nearest neighbors separately from result counts
Build the reference locally by evaluating the predicate and exhaustively ranking eligible vectors using the uploaded arrays and index distance metric. Define a tie policy and numerical tolerance before scoring. Set the reference size to the smaller of K and the eligible count. Deduplicate returned IDs, retain duplicate counts separately, and truncate the processed result to K. The table below separates measurements that a single recall number can obscure.
Table
Retrieval quality metrics
| Metric | Calculation or check |
|---|---|
| Exact-reference Recall@K | Returned reference neighbors divided by reference size |
| Eligible completeness | Unique eligible results divided by reference size |
| Underfill | Eligible result count below reference size |
| Predicate violations | Returned IDs outside the eligible set |
| Duplicates | Repeated IDs before deduplication |
| Full-answer rate | Share of queries returning every reference neighbor |
| Document diversity | Unique parent documents represented |
For an empty eligible set, mark recall and completeness undefined and check that the response is empty. Returning every eligible vector provides complete eligible coverage when fewer than K qualify. When more than K qualify, a full set of eligible results can still miss the nearest neighbors. Add human-labeled queries for semantic relevance or answer support, keeping those scores separate from agreement with the embedding space.
Ajay Kumar’s October 6, 2026, report used 50,000 synthetic, 384-dimensional vectors, cosine distance, and 50 held-out queries, with exhaustive filtered ground truth. With 50 eligible vectors, enhanced Recall@10 was 0.898 and Recall@50 was 0.610; at K=50, it returned 30.52 results on average. The report included three repetitions and 6,900 logical requests but no measured classic arm. These observations justify reproduction, not a product-wide conclusion. Sources: DEV Community
Measure the complete retrieval operation
Precompute embeddings and run the main experiment from a client in the same region. Fix the SDK version, connection pooling, retry settings, and response fields. Randomize paired classic and enhanced request order wherever classic mode is available. As a starting protocol, use at least 200 distinct queries per principal condition and three repetitions. Bootstrap paired differences over distinct queries rather than treating repetitions as independent observations.
Filtered requests require both s3vectors:QueryVectors and s3vectors:GetVectors permissions. The documented maximum topK is 10,000, and results can span pages. Preserve query parameters while following nextToken until exhaustion; writes between pages are not reflected in that query session. For client post-filtering, include every sequential page and local filtering in the logical query’s elapsed time. Sources: AWS
- Record complete-query latency, page count, API calls, retries, returned bytes, and result count.
- Report median and 95th-percentile latency by filter condition; reserve the 99th percentile for sufficiently large samples.
- Count errors and timeouts separately from successful-request latency.
- Separate warm repeated traffic from traffic after explicitly described idle periods.
- Test concurrency separately from the low-concurrency quality experiment.
AWS says latency increases with index size, matching fraction, and constraint count. Vary those independently. Kumar reported warm client round trips of approximately 329–330 ms. Sequential post-filter pagination took approximately 3.35 seconds for 1,000 candidates and 33.55 seconds for 10,000. Those timings include network overhead, and the report excluded one host-sleep-related failure. Compare fixed budgets and equal-quality outcomes so low recall cannot masquerade as a latency advantage. Sources: AWS, DEV Community
Publish enough evidence to explain underfill
Retain a machine-readable record per query, filter, mode, and repetition, including returned and exact IDs, eligible count, K, scores, latency, pages, retries, and errors. Preserve actual requests and responses alongside the corpus manifest. Kumar’s repository README describes deterministic generation, pinned dependencies, local ground truth, and a manifest for regenerating large files. The research behind this guide inspected its landing page, not every source file or raw response. Sources: GitHub
An October 2, 2026, DevelopersIO report tested 60,785 real article embeddings and retrieved all three matching articles in narrow prefix tests. Its broader claim that enhanced mode returns the smaller of K and matching count conflicts with Kumar’s reported underfill. DevelopersIO supplied neither exhaustive recall validation nor a latency distribution. Before selecting production settings, reproduce the 50-match case across seeds and fresh indexes, compare equivalent prefix and equality predicates, and test realistic concurrency. Publish eligible count, returned count, exact-reference recall, and complete-query latency together, with distributions or confidence intervals. Sources: DEV Community, DevelopersIO