Same Entity but different Point id in Qdrant - need for entity deduplication?

[jBontinck]

I've encountered a lot of identical entities, while all treated as a different point in Qdrant.
Is entity deduplication on the roadmap?

[JackColquitt]

Hi @jBontinck, because of the way we retrieve context through subgraphs, we haven't found entity duplication to be as big a problem if you're managing data through prescriptive ontologies. Are you running into a specific problem?

[cybermaggedon]

Just been looking...

Firstly, this was a deliberate decision, maybe not well aired or discussed. The thinking is... the vector points represent semantic concepts, so the same concept (e.g. Planet Earth) may have multiple semantic representations (e.g. "Only habitable planet in the Solar System", "Third planet from the Sun"). So, the thought process is that you may want multiple ways to find the same entity.

Can this be done in Qdrant? No... it doesn't seem like it's possible to dedupe in Qdrant at query or load time, I don't see API features to do this. Or maybe it is possible by using the entity identifier as the point ID, so that there can be only one.

Can this be done in TrustGraph? I don't think it makes sense to do this at load time, it will create a performance problem having to compare every point to be loaded with all points already in the system. At query time, yes: the graph-embeddings-query service already does this, and returns a unique set of identifiers in response to a graph embeddings query.

The GraphRAG algorithm uses the graph embeddings query service to do entity lookup so that will be using a deduped list also.

[cybermaggedon]

Would using the graph embeddings API do what you want?

https://github.com/trustgraph-ai/trustgraph/blob/release/v1.1/docs/apis/api-graph-embeddings.md

[jBontinck]

Thanks for the detailed responses — that helps clarify the design decisions behind the current setup.

To provide some context on my end:
I’m working with a large collection of parliamentary documents and using an extraction prompt tailored to identify structured elements like laws, resolutions, key topics, and political parties. During validation, I noticed a significant number of duplicate entities — same entity (e.g., Vlaams Parlement), but represented by different point IDs in Qdrant. So far, I’ve uploaded only 2 documents (~70 pages each), and I’m already seeing 30–40 duplicate entities. I have another 400+ documents lined up, which makes me worried about ending up with a heavily fragmented and inflated knowledge graph — especially since many of these entities will reappear across documents.

Here’s a screenshot showing many near-identical entities:

On this image you'll notice that the vector representations are indeed close, but the separation into different points feels risky, especially in downstream use cases. For example, if entity A has relation A → B and another embedding A′ → C, we miss the opportunity to surface the full context (A → B, A → C) if A and A′ are treated as separate entities.

I understand real-time deduplication at load time isn’t practical. One approach I’ve seen work well in similar contexts is a nightly batch deduplication step — e.g., using clustering or nearest-neighbor merging — to collapse semantically equivalent entities into a canonical form.

Another idea I’ve been exploring is to treat semantic variants (e.g. “Planet Earth”, “Third planet from the Sun”, etc.) as synonym nodes that point back to a central master node. That master node would serve as the anchor for reasoning, versioning, and relationships — while still allowing synonym nodes to exist in their respective semantic neighborhoods for richer retrieval. This could offer a nice balance between semantic flexibility and structural clarity — and even support time-variant or conflicting knowledge over time.

Would something like this fit within TrustGraph’s long-term vision? Or is the expectation that the Graph Embeddings API will abstract this kind of normalization?

[JackColquitt]

Hi @jBontinck, the way the TrustGraph GraphRAG approach works is that, the vector embedding space functions only for understanding how to retrieve information from the knowledge graph. Many GraphRAG approaches use a LLM to generate the Cypher/SPARQL. TrustGraph doesn't do it that way. During the retrieval process, TrustGraph performs a semantic similarity search of the embedding space to find the most relevant starting points for retrieving subgraphs, from the full knowledge graph space.

For the RAG context, we provide the LLM the subgraph structured as knowledge graph triples. There are many, many, many ways to tune this process to improve performance. In fact, you can even specify in TrustGraph how many hops you want the subgraph traversal to make, when generating the subgraphs.

The short is, the data in the knowledge graphs is much more important than the data in the embedding space. Does that make sense, or am I misunderstanding the phenomena you're seeing?

Also, would finer grained access controls for agents (down to the knowledge package, metadata, etc.) also potentially help? This is a feature also on our roadmap.

[jBontinck]

Thanks, @JackColquitt, for the feedback.

To clarify my understanding of the retrieval process in my example:
If my query lands in the middle of all these Vlaams Parlement vectors, will it retrieve triples from each of the surrounding subgraphs that originate from these 'duplicate' Qdrant Points / entities? At that point, do you expect the LLM to treat them as a single, unified entity and consolidate the relations, characteristics, etc., into a sort of “master node” for Vlaams Parlement?

I’m still wondering about one aspect: all these "Vlaams Parlement" nodes share the same label, but their vector representations are different. I assume that’s because they’re embedded based on their surrounding context or chunk? How is it decided to consistently label them as Vlaams Parlement, even though they’re represented differently in the vector space?

If it works like this then it is less of an 'issue', but It does feel error prune when this behavior goes forward to a KG and you have different entities in a knowledge graph, which are actually the same thing. It feels, at some point relations will be mixed up, and it becomes even more difficult when you want to

• version entities (e.g. Football stadium of Madrid has X amount of seats, they are doing construction work, soon it will have X+Y amount of seats)
• entity level security (e.g. if you would add entities/relations from restricted sources that are accessible by a subset of users, if you would implement an 'auto-discovery' enrichment step by an agent - you want to first give 'validation' access to a human-in-the-loop, ....)
• metadata