Preventing silent chunk loss in Gemini Embedding 2 pipelines with boundary cardinality assertions

The claim

A retrieval index can degrade without raising a single exception. In distributed systems, Huang et al. (2017) call this a gray failure: every health check stays green while recall drops. Auditing our internal retrieval ingestion pipeline uncovered a gray failure across two widely used open-source libraries. The mechanism affects any stack that pairs gemini-embedding-2 with standard LangChain vector stores.

If you pass a list of bare strings to the Gemini embedding API, you get one vector back for the whole list. Wrap each string in a Content object and you get one vector per string, identical to embedding each string alone. Google documents both behaviours because gemini-embedding-2 is a native multimodal sequence encoder. A bare list lets a caller combine text and images into one joint item. Versions 4.2.1 to 4.2.3 of langchain-google-genai passed bare strings by default, and langchain-postgres truncated the mismatched lists without raising an error.

In a controlled replay on a public corpus, the bare-string path stored 46 of 308 chunks (15%). Search still found the right file 96% of the time. It found the right chunk at rank 1 only 14% of the time, against 73% with a correct index. Widening retrieval to the top 10 chunks recovered almost nothing. Top-10 recall stayed at 15%, against 100% on the full store (99.6% to 14.7%, a loss of 84.9 points). No error appeared at any layer.

I lead Applied AI and Support at Yugabyte. We identified and isolated this mechanism while auditing the internal ingestion pipeline behind our support agent (report 01 and report 02). No customer system is involved, and all benchmark measurements below run on a reproducible public corpus.

What to do this week

1. Test the shape. Send N texts to your embedding client and count the returned vectors. A twenty-line script at the bottom of this report settles the question for your stack.
2. Assert at the pipeline boundary. Fail the job when the number of vectors differs from the number of texts (Breck et al., 2019). That one line protects every model and every client version, including versions that do not exist yet.
3. Check token density on technical text. Hexadecimal dumps and UUID logs pack 3.3 times as many Gemini tokens per byte as prose or Python code. Guard the token cap in your own code before calling the API.

Where the defect sat in the library code

With gemini-embedding-2, a call with 100 bare strings returned 1 vector. The same call with 100 Content objects returned 100 vectors. The check used 21 calls per shape, for 1 to 100 inputs. Every Content vector matched the vector for the same text embedded alone, to a cosine of 1.0.

Google documents this behaviour, so it is not a Google bug. The official guide states that adding multiple inputs to contents "produces one aggregated embedding for all inputs". It also states that wrapping each input in a Content object "returns separate embeddings for each entry". Both sentences were verified on 2026-09-29 against the guide updated on 2026-09-17.

The defect that reached our production system sat across two open-source libraries. Here are the exact lines from their published wheels.

# 1. langchain-google-genai 4.2.1 to 4.2.3 (langchain_google_genai/embeddings.py)
# Passes bare strings -> gemini-embedding-2 returns 1 vector for the whole batch:
res = self.client.models.embed_content(model=self.model, contents=batch, config=config)

# Fixed in langchain-google-genai 4.2.4 (2026-05-28, PR #1766):
contents = [{"parts": [{"text": text}]} for text in batch]

# 2. langchain-postgres 0.0.17 and 0.0.18 (langchain_postgres/vectorstores.py)
# Pairs texts with embeddings without strict=True -> silently drops texts[1:]:
for text, metadata, embedding, id_ in zip(texts, metadatas, embeddings, ids_):

Why exception monitoring misses silent truncation, and the controls that catch it

Standard exception monitoring cannot detect a silent cardinality mismatch across library boundaries.

First, every layer returned a success status. The API returned HTTP 200. The embedding client returned a valid list of length one. The database write committed cleanly and returned every input identifier, keeping all exception-based health checks green.

Second, the signature of batch truncation is a retrieval-distribution shift rather than a runtime error. Auditing our search results showed an elevated share of file headers. That distribution shift triggered our root-cause audit.

Third, deterministic dependency pinning (4.2.1) is standard release hygiene, yet upstream changelogs rarely flag silent data-integrity fixes. Issue 1704 on langchain-ai/langchain-google stayed open after 4.2.4 shipped the fix inside an unlabeled chore(genai) pull request (#1766) on 2026-05-28.

Because dependency tracking cannot catch unlabeled contract changes across 113 days, we pair pinned builds with a runtime boundary assertion (len(embeddings) == len(texts)).

The operational questions

Four questions matter when an ingestion pipeline feeds a vector index.

1. Does the client return N vectors for N texts?
2. If it does not, does anything fail loudly?
3. What does search do over the index that results?
4. What does batching buy, and what other silent losses sit on the same API path?

How it was checked

Three separate runs saw the same behaviour. Each used a different code path, so no single script carries the result alone.

The first ran inside our pipeline through langchain-google-genai 4.2.1. For N of 1, 2, 5, 8, 32 and 100, gemini-embedding-2 returned 1 vector for every N above 1. The older model, gemini-embedding-001, returned 5 vectors for 5 texts. That check sits in a commit record dated 2026-09-18 and re-verified on 2026-09-22.

The second was an ad hoc spike that called google-genai directly with 8 chunks averaging 827 characters. Bare strings returned 1 vector. Content objects returned 8 vectors, each at cosine 1.0 to its solo vector. It also found that the API rejects more than 100 inputs in one call (128 inputs returned HTTP 400).

The third has committed data and scripts. It ran both shapes for N of 1, 2, 4, 8, 16, 32, 64 and 100, giving 21 calls per shape. The texts came from a deterministic public corpus of 308 chunks across 46 sources. Those sources are CPython standard library modules and PyPI package metadata, split the way our pipeline chunks. Nothing in the corpus comes from a customer.

The third run also replayed the incident path. Each source file went through embed_documents of langchain-google-genai 4.2.1, one call per file. Texts were paired with vectors using the zip logic of langchain-postgres. The resulting index was scored with 285 known-item queries, each a 150 to 300 character excerpt copied from one chunk.

The library wheels were inspected directly across every release of langchain-google-genai from 4.2.1 to 4.4.0 and langchain-postgres 0.0.17 and 0.0.18. Every URL and access date is in numbers.json.

Question 1: does the client return N vectors for N texts?

With bare strings, no. With Content objects, yes.

All 18 bare-string calls with two or more inputs returned 1 vector. All 21 Content calls returned N. The lowest cosine between a Content vector and its solo vector was 1.0 to six decimals. Across all 485 embedded inputs, every Content vector was nearest to its own solo vector.

The single bare-string vector is not an average of the inputs. At N of 100, its mean cosine to the individual solo vectors was 0.63. Its cosine to the mean of the solo vectors was 0.86. Twelve multi-input calls with N up to 16 fit inside the token cap. On those calls, the bare-string vector's cosine to the newline-joined text sat at 0.98 to 0.99 (0.984 to 0.994). The model reads the list as one concatenated sequence prior to pooling.

Input order matters too. Sending 8 chunks forward and reversed three times produced vectors with a cosine of 0.97 to 0.98 (0.973 to 0.978). A mean pool would be invariant to order.

Question 2: does anything fail loudly?

Nothing failed in any layer.

The API returned HTTP 200 with one vector. The embedding library returned that vector as a one-element list. langchain-postgres then built its database rows with zip(texts, metadatas, embeddings, ids_). Because Python's zip stops at the shortest iterable without strict=True, it dropped every chunk after the first in each batch. Its add_embeddings method then returned the full list of input IDs.

This zip pattern is still present in langchain-postgres 0.0.18, released on 2026-09-15. Upgrading langchain-google-genai removes the trigger, but it does not remove the missing boundary check.

In the public-corpus replay, 308 chunks went in and 46 rows came out (15%), dropping 262 chunks (85.1%). All 46 source files returned one vector each. The stored row for each file paired chunk 0's text with the sequence vector of the whole file.

Question 3: what does search do afterwards?

Search finds the right file and loses the chunk.

Over the replayed store, the right chunk ranked first for 14% of queries (14.4%), against 73% (72.6%) over the complete store. That is a paired loss of 58.2 points (95% interval 52.3 to 64.2). Fetching the top 10 chunks for a reranker or prompt context did not help. Top-10 chunk recall fell from 100% (99.6%, interval 99% to 100%) to 15% (14.7%, interval 11% to 19%). That is a paired loss of 84.9 points (interval 80.7 to 88.8). Mean reciprocal rank fell from 0.84 to 0.15. Yet right-file retrieval at rank 1 stayed at 96% in both stores.

Splitting the 285 queries by target chunk position shows the mechanism. For the 42 queries targeting chunk 0 of a file, rank-1 recall was 98% in the replay and 88% in the complete store. For the 243 queries targeting a later chunk, recall dropped from 70% to 0%, because those 262 chunks were not in the index. In the replayed store, the top hit was a file header (chunk 0) on 100% of queries, against 16% in the complete store.

A file-level test passes, and an engineer skimming the top hit sees the right module header. Every detail from the body of the file is gone.

Question 4: what does batching buy?

Wrapping inputs in Content objects lets one request carry up to 100 chunks.

Three speedup ratios appeared across our timing runs: 19x in the spike log, 14x in the two-call mean, and 10.7x in the slower draw. Recomputing all three from raw trace timings illustrates why shared-endpoint latency must be reported as a range rather than a single point.

The solo baseline is the median of 308 single-input calls: 462 ms (95% interval 449 to 483 ms). At N of 100, the committed grid holds two Content calls. The first took 4.31 s for 31,384 tokens, or 43 ms per input. The second took 2.36 s for 33,170 tokens, or 24 ms per input.

• First call: 462 ms divided by 43 ms gives 10.7x (11x rounded).
• Second call: 462 ms divided by 24 ms gives 19.6x (20x rounded).
• Mean of the two calls: 3.33 s, or 33 ms per input, gives 13.9x (14x rounded).
• The earlier spike note: 437 ms divided by 23 ms (2.29 s for 100 chunks) gave 19.1x.

Each number is exact arithmetic on one draw. Two calls on a shared API key support a range of 11x to 20x, not a single point estimate. At N of 32 and 64, the same ratio sat between 12x and 16x. By comparison, the two bare-string calls at N of 100 finished in 0.95 s and 1.38 s (mean 1.16 s). Bare strings ran faster because the server truncated the joined sequence at 8,192 tokens and ran the encoder once.

Two more silent traps on the same path: token caps and log density

The same API also truncates over-length inputs without an error. Google's guide documents this: "The system silently truncates inputs that exceed this limit." The published cap is 8,192 tokens for gemini-embedding-2 and 2,048 for gemini-embedding-001. Both models returned identical token counts across 31 of 31 test inputs.

To locate the active window, one sentence was inserted at candidate positions across one code document and one prose document per model. For gemini-embedding-2, the last tested newline position that changed the vector was 8,156 tokens, and the first that produced zero change (cosine 1.0) was 8,191. For gemini-embedding-001, the bounds were 2,038 and 2,042.

Setting the REST flag autoTruncate to false exposed why the effective cliff sits slightly below the published cap. On gemini-embedding-2, sending a document that count_tokens measured at 9,580 tokens returned HTTP 400 with the message "Actual token count: 9579" (and 9,788 vs 9,787 on the second document). The embedder reserves internal framing tokens that count_tokens does not report. Meanwhile, gemini-embedding-001 ignored autoTruncate=false on both documents, returning HTTP 200 and truncating anyway.

When we measured token density across 348 inputs, a second operational risk surfaced. Python code had a median of 3.84 UTF-8 bytes per token (interquartile range 3.46 to 4.14, minimum 1.14, n=156). PyPI prose had a median of 3.70 bytes per token (3.31 to 4.12, n=152). Hexadecimal dumps dropped to 1.15 bytes per token (1.15 to 1.16, n=20) and UUID-heavy logs dropped to 1.10 (1.09 to 1.10, n=20). A character splitter calibrated on prose or code packs 3.3 to 3.5 times as many tokens into a chunk on hex or UUID logs. Across all 348 inputs, token count never exceeded UTF-8 byte length (1.00 ratio ceiling).

On 10 long documents, whole-document search matched the parent file at 96% before the cap (45 queries). Past the cap, it matched at 89% (36 queries, 95% interval 78% to 97%). Chunked search scored 100% and 94% on the same queries, against 10% chance. That high score is a property of the 10-file test pool, not proof that truncation is harmless. Each file in the pool is a single-topic module, so a snippet past token 8,192 still shares identifiers with the file's head. All 4 misses past the cap fell in a single document.

The record upstream

Issue 1704 on langchain-ai/langchain-google reported the batch defect on 2026-04-16 against version 4.2.2, and a commenter noted the zip truncation in langchain-postgres. As of 2026-09-29, Issue 1704 remained open.

The fix arrived in Pull Request 1766, merged on 2026-05-27 and released in 4.2.4 on 2026-05-28. Its title was "chore(genai): Update google-genai dependency to support 2.x", and its description never mentioned embeddings. Two pull requests aimed directly at the bug, 1708 and 1817, stayed open while 4.4.0 shipped with the fix already in place.

Google had seen the same confusion on 2026-03-11 in GoogleCloudPlatform/generative-ai Issue 2721, where a sample notebook passed a bare list and received one vector. A maintainer closed it as "by design" and updated the notebook on 2026-03-19.

Decisions and why

1. Enforce the boundary count check in production. Since 2026-09-18, our embedding wrapper raises an exception whenever len(embeddings) != len(texts). One line of code closes the silent-drop path across every client and store.
2. Isolate calls and move to Vertex AI. Rather than relying on client-library batching alone, our production pipeline keeps batch_size=1 with async document concurrency. On 2026-10-20 we migrated from the Gemini Developer API to Vertex AI, where 26 checked vectors matched the Developer API to a cosine of 1.0.
3. Guard the token cap using the UTF-8 byte bound. Across all 348 tested inputs, len(text.encode("utf-8")) was an upper bound on token count. Any chunk whose UTF-8 byte length sits at or below the empirical ceiling (8,160 tokens for gemini-embedding-2, 2,038 for gemini-embedding-001) passes with zero extra API calls. Only longer chunks call count_tokens.
4. Document exact multimodal API contracts in code. We documented the Content-object batch contract and bare-string sequence behavior directly in our embedding wrapper so maintainers treat input shape as a hard invariant.

Verification gates and methodology checks

• The average hypothesis. At N of 100, the bare-string vector scored 0.86 against the mean of the solo vectors. At N up to 16, it scored 0.98 to 0.99 against the newline-joined text. It is sequence concatenation, not vector averaging.
• The order hypothesis. Reversing 8 chunks moved the vector cosine to 0.97 to 0.98 across 3 runs.
• Count without identity. Checking len(vectors) == N could pass if vectors were scrambled. Every Content vector was checked against its solo vector (min cosine 1.0) and nearest-neighbour argmax (485 of 485).
• Top-k retrieval recovery. Widening search from rank 1 to top 10 left incident recall at 15% (14.7% vs 99.6%). Dropped chunks cannot be recovered by a wider top-k window.
• The single speedup claim. The 19x, 14x and 10.7x numbers came from two draws at N of 100 (2.36 s and 4.31 s). One earlier ad hoc run took 2.29 s.

Hypotheses falsified by the audit

• "The Gemini API silently drops chunks." The API followed its multimodal sequence contract. The library passed the wrong shape, and the vector store used zip without strict=True.
• "Gemini Embedding 2 cannot batch." It batches up to 100 Content objects per call at cosine 1.0.
• "Silent truncation is undocumented." Google documents both the truncation and autoTruncate, though gemini-embedding-001 ignores the opt-out.
• "Character-based chunking has a 3x token margin." Hex dumps (1.15 B/tok) and UUID logs (1.10 B/tok) erase that margin by 3.3x.

Pressure tests

What this does not show

How the rate limiter bills a batched Content call against requests-per-minute quota is untested. Tokens per minute is the binding cap on our workload.

The benchmark corpus is public Python code and package descriptions (46 files, 308 chunks), not support tickets. The 10-document whole-file test used single-topic files where tail vocabulary matches head vocabulary, so its 89% score does not bound truncation loss on multi-issue logs.

Reproduce

Run this self-contained check with your own key (pip install google-genai numpy) to verify both input shapes and your installed langchain-google-genai version.

import os, numpy as np
from google import genai
from google.genai import types

client = genai.Client(api_key=os.environ["GOOGLE_API_KEY"])
model, texts = "gemini-embedding-2-preview", [f"Chunk {i}: distinct text." for i in range(8)]
solo = [np.array(client.models.embed_content(model=model, contents=t).embeddings[0].values) for t in texts]
bare = client.models.embed_content(model=model, contents=texts).embeddings
wrap = client.models.embed_content(model=model, contents=[types.Content(parts=[types.Part(text=t)]) for t in texts]).embeddings
cos_min = min(float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))) for a, b in zip(solo, [ np.array(e.values) for e in wrap ]))
print(f"N={len(texts)} | bare returned: {len(bare)} | Content returned: {len(wrap)} | min cosine vs solo: {cos_min:.6f}")

Every number in this report is in numbers.json. It is generated deterministically by gen_numbers.py from data/results.json and data/replay_per_source.json.

References

• Breck, E., Polyzotis, N., Roy, S., Whang, S., & Zinkevich, M. (2019). Data Validation for Machine Learning. Proceedings of MLSys 2019.
• Huang, P., Guo, C., Zhou, L., Lorch, J. R., Dang, Y., Chintalapati, M., & Yao, R. (2017). Gray Failure: The Achilles' Heel of Cloud-Scale Systems. Proceedings of HotOS '17. https://doi.org/10.1145/3102980.3103005.