Tier 0 502 Gateway Error

[maximusunc]

The following query consistently causes a 502 Gateway Error:

curl -X POST https://dev.retriever.biothings.io/query \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "message": {
      "query_graph": {
        "nodes": {
          "SN": { "categories": ["biolink:ChemicalEntity"] },
          "ON": {
            "ids": ["MONDO:0007186"],
            "categories": ["biolink:DiseaseOrPhenotypicFeature"]
          },
          "h": { "categories": ["biolink:Disease"] }
        },
        "edges": {
          "edge_0": {
            "subject": "SN",
            "object": "ON",
            "predicates": ["biolink:treats_or_applied_or_studied_to_treat"]
          },
          "edge_1": {
            "subject": "SN",
            "object": "h",
            "predicates": ["biolink:treats_or_applied_or_studied_to_treat"]
          },
          "edge_2": {
            "subject": "h",
            "object": "ON",
            "predicates": ["biolink:has_phenotype"]
          }
        }
      }
    },
    "parameters": {}
  }' > retriever_response.json

[maximusunc]

The 502 error is returned to a synchronous query after about 1:20. But this also means that for an async query (which Shepherd uses almost exclusively) would then just never get a response back, which causes the query to automatically run until its default lookup timeout of 3.5 minutes.

[tokebe]

Retriever doesn't return 502 as an application, so what this probably means is you've found a query which consistently kills a worker. I'll double check

[tokebe]

Running this query on my local took 15 minutes:

26576 INFO    910af699 Tier 0: Retrieved 1478 results / 13199 nodes / 134483 edges in 931276ms.

Additionally, it peaked at nearly 4gb of memory, which definitely means the worker died going OOM (Retriever's Dev and CI instances have 4gb). We can increase memory, but that won't speed the query up.

I'll do some more looking to make sure the transpiler isn't doing anything incorrectly.

[maximusunc]

I will say that result count is approximately what is expected.

[tokebe]

134k edges is an order of magnitude greater than I'd usually expect for a normal query...

[maximusunc]

It might not be a great query 😆 but it is still a query we send as part of our creative mode queries.

[tokebe]

Ah, looks like the response is actually quite sensible, 1478 results, 1974 edges, 511 nodes. The much higher numbers above are pruned before response due to lack of binding. Which implies something's wrong with the transpiler...

@everaldorodrigo, can you take a look? This is the dgraph payload (with some naïve formatting):

I'd expect that any nodes/edges which were going to get pruned by the kgraph pruner would already have been pruned the cascaded_or filter?

{ 
  q0_node_n0(func: eq(vO_id, "MONDO:0007186")) @cascade(vO_id) { 
    expand(vO_Node) in_edges_e0: ~vO_object @filter(eq(vO_predicate_ancestors, "treats_or_applied_or_studied_to_treat")) @cascade(vO_predicate, vO_subject) { 
      expand(vO_Edge) { vO_sources expand(vO_Source) } 
      node_n1: vO_subject @filter(eq(vO_category, "ChemicalEntity")) @cascade(vO_id, ~vO_subject) {
        expand(vO_Node) out_edges_e1: ~vO_subject @filter(eq(vO_predicate_ancestors, "treats_or_applied_or_studied_to_treat")) @cascade(vO_predicate, vO_object) {
          expand(vO_Edge) { vO_sources expand(vO_Source) } node_n2: vO_object @filter(eq(vO_category, "Disease")) @cascade(vO_id) { expand(vO_Node) } 
        } 
      } 
    } in_edges-subclassObjB_e0: ~vO_object @filter(eq(vO_predicate_ancestors, "subclass_of")) @cascade(vO_predicate, vO_subject) {
      expand(vO_Edge) { vO_sources expand(vO_Source) } node_intermediate_n0: vO_subject @filter(has(vO_id)) @cascade(vO_id, ~vO_object) {
        expand(vO_Node) in_edges-subclassObjB-tail_e0: ~vO_object @filter(eq(vO_predicate_ancestors, "treats_or_applied_or_studied_to_treat")) @cascade(vO_predicate, vO_subject) {
          expand(vO_Edge) { vO_sources expand(vO_Source) } node_n1: vO_subject @filter(eq(vO_category, "ChemicalEntity")) @cascade(vO_id, ~vO_subject) {
            expand(vO_Node) out_edges_e1: ~vO_subject @filter(eq(vO_predicate_ancestors, "treats_or_applied_or_studied_to_treat")) @cascade(vO_predicate, vO_object) {
              expand(vO_Edge) { vO_sources expand(vO_Source) } node_n2: vO_object @filter(eq(vO_category, "Disease")) @cascade(vO_id) { expand(vO_Node) } 
            }
          } 
        } 
      } 
    } in_edges_e2: ~vO_object @filter(eq(vO_predicate_ancestors, "has_phenotype")) @cascade(vO_predicate, vO_subject) {
      expand(vO_Edge) { vO_sources expand(vO_Source) } node_n2: vO_subject @filter(eq(vO_category, "Disease")) @cascade(vO_id, ~vO_object) {
        expand(vO_Node) in_edges_e1: ~vO_object @filter(eq(vO_predicate_ancestors, "treats_or_applied_or_studied_to_treat")) @cascade(vO_predicate, vO_subject) {
          expand(vO_Edge) { vO_sources expand(vO_Source) } node_n1: vO_subject @filter(eq(vO_category, "ChemicalEntity")) @cascade(vO_id) { expand(vO_Node) } 
        } 
      } 
    } in_edges-subclassObjB_e2: ~vO_object @filter(eq(vO_predicate_ancestors, "subclass_of")) @cascade(vO_predicate, vO_subject) {
      expand(vO_Edge) { vO_sources expand(vO_Source) } node_intermediate_n0: vO_subject @filter(has(vO_id)) @cascade(vO_id, ~vO_object) {
        expand(vO_Node) in_edges-subclassObjB-tail_e2: ~vO_object @filter(eq(vO_predicate_ancestors, "has_phenotype")) @cascade(vO_predicate, vO_subject) {
          expand(vO_Edge) { vO_sources expand(vO_Source) } node_n2: vO_subject @filter(eq(vO_category, "Disease")) @cascade(vO_id, ~vO_object) {
            expand(vO_Node) in_edges_e1: ~vO_object @filter(eq(vO_predicate_ancestors, "treats_or_applied_or_studied_to_treat")) @cascade(vO_predicate, vO_subject) {
              expand(vO_Edge) { vO_sources expand(vO_Source) } node_n1: vO_subject @filter(eq(vO_category, "ChemicalEntity")) @cascade(vO_id) { expand(vO_Node) }
            }  
          } 
        } 
      } 
    } 
  }  
}