Discover Searches Not Working On Nested Field Using Plugin

[jkoers29]

Plugin version:
6.3.1
Kibana version:
6.3.1
Plugins installed:
KNQL, kibana-object-format(Installed, but not used)
Description of the problem including expected versus actual behavior:
I am having an issue using your Query syntax where I do not believe it is working in some cases.
I am not sure if it is me, or the language itself.
In The below screenshot you can see that I perform a search on _id and I am able to return the document containing the ID of 1.
In this below Screenshot, you can see that I try to search on a nested field and should only return the 1 document containing that value, but still all 5 records in my Index are returned, signifying to me that the query did no work. LGMC is only present in 1 document in the field message.MSH.MSH-4.
Below is what I am using for mappings to create this index.
From here I put a JSON object into the message field.
I can query message.MSH.MSH-4 before I apply your plugin.
However, after applying the plugin, I don't seem to have the ability to search this field anymore using the KNQL language.
The query completes without error, but returns the same number of documents as the last successful search, which was *.
Any insight on this would be greatly appreciated.
Please let me know if you would rather have this in a Github ticket, or more information is required.

PUT composite_test
{
"mappings": {
"doc": {
"properties": {
"message": {
"type": "object"
}
}
}
}
}
Steps to reproduce:

1. Create an index with one field with type object. Enable Plugin.
2. Index a document putting a JSON object in the one field created in step 1. Json should contain Nested JSON objects.
3. Search in Discover for an element in a nested part of the above JSON.

I should note that we are currently working on our data model, and I strayed away from using NESTED as a type for the field, as Dashboards did not seem to work with Nested, and queries seem to need to have special Nested query syntax. Using Object as a type allowed for quick Dashboard creation and queries that did not rely on Nested query terminology. We do love how this plugin works and makes our Discovery output look readable, but it is the search bar in Discover that is the only thing that does not work, and we would like to still be able to use it. If we disable the plugin with the current data model (Field=Object), then Discover Search bar works just fine. Cheers!

Errors in browser console (if relevant):
No Errors.
Provide logs and/or server output (if relevant):
I tailed both ES and Kibana logs, and there was no output.
If using docker, include your docker file:

Describe the feature:

[jkoers29]

[ppadovani]

I'm in the middle of porting to 6.4. As soon as I have that running, I'll look at this bug in that release, and if it is indeed a bug, I'll fix there and Backport...

[ppadovani]

Quick update... work got in the way of my availability to work on this plugin. I've just released the first release candidate for 6.4. I'll try and take a look at this in the next week or so.

[jkoers29]

Hi Pierre, No worries. I know all about being busy. Lol I am just grateful you are taking the time to look into this for me. Cheers, John

…

On Sat, Oct 20, 2018, 9:01 AM Pierre Padovani ***@***.***> wrote: Quick update... work got in the way of my availability to work on this plugin. I've just released the first release candidate for 6.4. I'll try and take a look at this in the next week or so. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#95 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABhHuXoBtagagyqIfjXD2xFm1_ncHih6ks5um0jsgaJpZM4W0_dV> .

[ppadovani]

I've read this issue a couple times, and believe the main issue is the usage of the type of 'object' in your mapping. This plugin handles nested types, but I didn't account for a mixture of object and nested it seems. Dashboards do work for nested with this plugin as seen in this screenshot of the regression dashboard I use to test this plugin:

I'll do some debugging and see if I can figure out what my code is doing with your type of mapping.

[ppadovani]

Ok... looking at this, it seems to be working in my 6.4.2 support.

There are 300 docs in my test dataset. This is the only one with that particular address. The mapping for this is here:
https://github.com/ppadovani/KibanaNestedSupportPlugin/blob/master/scripts/nested-support/home.json

I don't specifically call out a 'type' for the object based properties, as you don't need to do so when defining the mapping, unless you are using dynamic mappings. If you are indeed using dynamic mappings, I would suggest you create your index as you have above then add some documents, and pull the mapping back. You can then post the mapping here for me to look at and test against.

[jkoers29]

Hi Pierre, Please see below screenshot for the new error we receive in Dashboard when trying to perform a search. I am also sharing my Index creation with you as well as the extracted mappings after I send messages through, and a sample message for you to test with. We are still new to Elasticsearch, but I am wondering if Dynamic Mappings are still in play here even after defining the "message" field as Nested? Could this be why our schema is not playing well with your plugin? I switched from Object to Nested to show you that either way produces the same issue where searching in the Dashboard stops working with the plugin enabled. The command I used to create this index: PUT testing_nested { "settings" : { "index" : { "number_of_shards" : 3, "number_of_replicas" : 0 } }, "mappings": { "doc": { "properties": { "message": { "type": "nested" } } } } } Attached is the output from the below command. GET /testing_nested/_mapping/doc Thanks, John

…

On Sat, Oct 27, 2018 at 8:41 AM Pierre Padovani ***@***.***> wrote: Ok... looking at this, it seems to be working in my 6.4.2 support. [image: image] <https://user-images.githubusercontent.com/11234684/47606058-755b1e00-d9d4-11e8-8a70-37538914796e.png> There are 300 docs in my test dataset. This is the only one with that particular address. The mapping for this is here: https://github.com/ppadovani/KibanaNestedSupportPlugin/blob/master/scripts/nested-support/home.json I don't specifically call out a 'type' for the object based properties, as you don't need to do so when defining the mapping, unless you are using dynamic mappings. If you are indeed using dynamic mappings, I would suggest you create your index as you have above then add some documents, and pull the mapping back. You can then post the mapping here for me to look at and test against. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#95 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABhHuavWXi6p1DUYnLBi4QXWRWfQarnGks5upH6egaJpZM4W0_dV> .

[ppadovani]

Sorry about the delay.. work, life... you know... I'll look at this at bit more this week.

[jkoers29]

Hi Pierre, No worries, I am no rush. I am just grateful you are able to look into this at all for us. Happy Holidays. Thanks, John

…

On Sat, Nov 24, 2018 at 7:19 AM Pierre Padovani ***@***.***> wrote: Sorry about the delay.. work, life... you know... I'll look at this at bit more this week. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#95 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABhHuXpF2fEZYnLv4sdwDlrfoQFlPK84ks5uyWN1gaJpZM4W0_dV> .