Add shepherd infores to ARAX messages

[maximusunc]

Please add infores:shepherd-arax as an aggregator knowledge source to the ARAX passthrough implementation. This can just be added to the Shepherd repo when we receive the ARAX response back.

Tagging @dkoslicki for awareness.

[dkoslicki]

Thanks for the tag, @mohsenht is indeed the correct assignee

[maximusunc]

I think the shepherd aggregator knowledge source is already there, but the upstream_resource_ids list is empty? It should include infores:arax.

[dkoslicki]

A couple of questions:

1. Do you want the aggregator_knowledge_source value to be infores:arax or infores:shepherd-arax?
2. And/or do you want the upstream_resources_ids list to include infores:arax

I ask since "shepherd ARAX" is just ARAX and I don't know if we need to update some metadata if we mint a new infores.

[maximusunc]

I'd like the aggregator_knowledge_source to be infores:shepherd-arax and the upstream_resource_ids to be infores:arax. That shows the complete provenance.

Here's what I'm looking at for reference:

[dkoslicki]

@mohsenht Can you give a status update on this issue?

[mohsenht]

I haven't started this one.
Should we add them in the ARAX service or add them manually in the ARAX Shepherd worker after getting result from ARAX?

[dkoslicki]

Do you mean change them in the ARAX/RTX code-base, or change them here? Given that this is shepherd-specific given the aggregator_knowledge_source value of infores:shepherd-arax, I would change them here in the ARAX Shepherd worker after getting results from ARAX. But LMK if you meant something different

[mohsenht]

Yes, I was asking whether we should modify them in the ARAX worker, or determine the changes dynamically in the ARAX instance from the request, which specifies that it originates from a Shepherd instance.

However, since our mission is to migrate all creative query to Shepherd, I agree with you and I will add this in the ARAX worker.

[mohsenht]

I think the shepherd aggregator knowledge source is already there, but the upstream_resource_ids list is empty? It should include infores:arax.

Hi @maximusunc,

I checked the ARAX response in Shepherd, and the upstream_resource_ids field is not empty.

I’m a bit confused about what exactly should be changed for this task.

For example, here is an aggregator_knowledge_source entry. How should this be modified to include infores:shepherd-arax?

Should it be added to the existing upstream_resource_ids list, or should a new resource_id be created, or something else?

{
    "resource_id": "infores:rtx-kg2",
    "resource_role": "aggregator_knowledge_source",
    "source_record_urls": null,
    "upstream_resource_ids": [
        "infores:semmeddb"
    ]
}

[mohsenht]

Hi @maximusunc,

Thanks — I re-read the thread and I think I understand the intended provenance now.

Rather than modifying existing entries like infores:rtx-kg2, I will add an additional knowledge_graph.sources record in Shepherd (in the ARAX worker) after we receive the ARAX response:

resource_id: infores:shepherd-arax

resource_role: aggregator_knowledge_source

upstream_resource_ids: ["infores:arax"]

This should preserve the existing provenance chain returned by ARAX (e.g., rtx-kg2 -> semmeddb) while making Shepherd’s role explicit at the top level.

Could you please confirm this?

[mohsenht]

Here is an example:

"sources": [
  {
    "resource_id": "infores:semmeddb",
    "resource_role": "primary_knowledge_source",
    "source_record_urls": null,
    "upstream_resource_ids": null
  },
  {
    "resource_id": "infores:rtx-kg2",
    "resource_role": "aggregator_knowledge_source",
    "source_record_urls": null,
    "upstream_resource_ids": [
      "infores:semmeddb"
    ]
  },
  {
    "resource_id": "infores:shepherd-arax",
    "resource_role": "aggregator_knowledge_source",
    "source_record_urls": null,
    "upstream_resource_ids": [
      "infores:arax"
    ]
  }
]

[mohsenht]

@maximusunc: Please review this pull request