Follow-up to my earlier message about RAG context not being used consistently. I’ve isolated the root cause using Chrome DevTools Network tab, and it looks like an internal logic bug rather than a configuration issue on my end.
SETUP
– Knowledge Base: Pinecone, Active/Connected
– Similarity Threshold: 50%
– RAG Sources Limit: 5
– RAG Chunks Limit: 15
– Citation Links: off
– Bot: “testing”
– Model: gpt-5.3-chat-latest
TEST CASE
A single clean Q&A knowledge entry:
Q: “How do I pair Bluetooth on the KD-X280BT?”
A: 6-step pairing instructions (no extra content, no meta-instructions in the entry itself).
Test query: “How do I pair Bluetooth on the KD-X280BT?”
WHAT THE TESTING PANEL SHOWS
– Document match: 94.62% similarity
– Marked “USED FOR AI CONTEXT”
– 1 source used for AI context, 1 chunk sent to AI
– Similarity threshold cleared (1 above threshold, 0 below)
WHAT THE BOT ACTUALLY SAYS
“I don’t have that specific information in my knowledge base, but I can connect you with dealer support or a specialist.”
ROOT CAUSE FOUND VIA NETWORK TAB
I captured the raw admin-ajax.php response for action=mxchat_handle_chat_request (screenshot attached — “Response” tab). The JSON shows an internal contradiction:
Per-match object:
“used_for_context”: true
“has_access”: true
“filtered_out”: false
“content_preview”: “Question: How do I pair Bluetooth on the KD-X280BT?\r\n\r\nAnswer:\r\nTo pair Bluetooth on the KD-X280BT:\r\n…” (full correct content is present and retrieved)
Top-level summary (same response):
“sources_used”: 0
“total_chunks_used”: 1
So the match is found, passes the threshold, is correctly marked used_for_context:true at the per-document level, and the actual chunk text is present in content_preview — but the top-level sources_used counter reports 0. My working theory is that sources_used (or whatever flag/counter actually gates context assembly into the OpenAI prompt) is not being incremented even when used_for_context is true at the per-match level, resulting in the model receiving little or no actual context, despite the Testing panel UI reporting a successful match.
This would also explain the second symptom I originally reported: in other test runs, the bot sometimes answers but pads the response with invented details (SRC button steps, volume knob steps, BT PAIRING menu steps, PIN 0000) that are not in the knowledge entry — consistent with sources_used incrementing correctly in those runs and context reaching the model, but without it being constrained tightly enough to the source text.
WHAT I’VE ALREADY RULED OUT
– AI Instructions (Behavior) field: confirmed strict, explicit “answer ONLY from context, do not add steps/buttons/PIN codes” language is in place and is being correctly obeyed when context IS empty (i.e., the model’s “I don’t have that info” fallback is itself proof the instructions are working — it’s just being triggered with no/empty context).
– Knowledge base content: re-tested after removing an extra in-content instruction line, confirmed clean, re-embedded by Pinecone (similarity score changed from 92.74% to 94.62% between tests, confirming a real re-embed occurred), bug persisted identically.
– Pinecone connection/retrieval: confirmed working correctly via Testing panel debug log and the raw response JSON — correct document is found, correct content is retrieved into content_preview.
This looks like a bug in how MxChat aggregates/counts which retrieved chunks are passed into the final prompt sent to OpenAI, specifically a mismatch between the per-match used_for_context flag and the top-level sources_used/total_chunks_used counters.
Could you confirm whether this is a known issue, and if not, could someone take a look at the context-assembly logic between Pinecone retrieval and the final prompt construction? Happy to provide additional debug captures, my testing bot ID (“testing”), or temporary access if that would help.
I also sent an email on 6-21-26 at 4:26 AM with screen caps to [email protected].
