❗BUG❗ : Duplicate Datasets Appearing in Search Results of agent

[Areeba-Tahir-18]

Problem

While interacting with the KnowledgeSpace UI search from a time , I noticed that same datasets appear multiple times. I started thinking about why this happens why i am seeing same dataset in search result 2 times ??

• Multiple sources: The same dataset may exist in more than one datasource.
• Metadata variations: Enrichment adds metadata or URLs that are slightly different.
• Title differences: The same dataset might have small variations in its title across sources.
• Different IDs: id or dataset identifiers can differ even for the same logical dataset.

Then i decide to solve this problem and here is my approach for solving this .

Proposed Solution :

--> Use a multi-layer deduplication strategy:

1. Canonical Identity: Combine datasource_id + dataset_id to identify duplicates.
2. URL Normalization : Remove query parameters to unify different links.
3. Fuzzy Title Matching: Detect minor variations in titles and merge duplicates.

Expected Impact on Knowledge-space-agent searching

• Each dataset appears only once in the UI search results.
• Improves user experience, making results cleaner and easier to navigate.
• Reduces confusion caused by repeated entries.
• Provides a foundation for future improvements, like semantic deduplication or hybrid search ranking.

[Areeba-Tahir-18]

I'd Like to Work on This

I have started woking on this issue . some of work on this is already done -- will submit my PR soon . Thank you !!

[SandeepChauhan00]

Hey, can you share some screenshots or specific examples of the duplicates you're running into? Like which search query gave you repeated results and what the duplicate entries actually looked like?

It'd help to figure out if this is actually a bug or just expected behavior — for instance, if the same dataset shows up from different datasources, that might be intentional.

Also, I think it makes more sense to first pin down where exactly the duplication is happening — is it during indexing, in the search/retrieval logic, or just on the UI side — before jumping into a fix. Otherwise we might end up solving the wrong problem.

[Areeba-Tahir-18]

Hello Sandeep !! Thank you for sharing your suggestion .

First of all, I want to clarify one thing: the duplicate results are appearing due to not handling of duplicates in the searching logic, as no one previously observed agent search results on queries deeply so no work was done on this case of search result.

This is not on the UI side or during indexing.

Before diving deep into the problem, I first checked where the issue is actually happening and why. I went step by step through different parts of the system to understand the source of duplication. Then i found that problem is occurring in the backend search process.

After identifying this, I started looking into the backend search logic and began working on solving it, as I enjoy exploring and solving such problems.

-> ### For more clarity, I’m going to share some screenshots showing the results where the duplicate entries appear.You can see here Anesthesia EEG Dataset appearing 2 or 3 times .

[Areeba-Tahir-18]

While passing through the entire pipeline step by step to identify the issue i tested whether its UI, indexing or someother .

• Its Not UI , indexing or someother .
• Its Coming from backend .