How to find out last save position, when do "on _error"?

[XvKuoMing]

Suppose, we have a grammar:

start: value
?value: object
        | array
        | string
        | SIGNED_NUMBER      -> number
        | "true"             -> true
        | "false"            -> false
        | "null"             -> null
array  : "[" [value ("," value)*] "]"
object : "{" [pair ("," pair)*] "}"
pair   : string ":" value
string : ESCAPED_STRING
%import common.ESCAPED_STRING
%import common.SIGNED_NUMBER
%import common.WS
%ignore WS

now, I have a string to validate:

"name": "John",
"age": 30,
"city": "New York
}

I want to write on_error callback function such as it will do:

1. understand that there is broken value after city":
2. give me the position of last accepted token e.g., so I can delete broken "New York and put something else

here is my example:

from lark import Lark
from lark.reconstruct import Reconstructor
from lark import UnexpectedToken, UnexpectedCharacters

grammar = """
start: value
?value: object
        | array
        | string
        | SIGNED_NUMBER      -> number
        | "true"             -> true
        | "false"            -> false
        | "null"             -> null
array  : "[" [value ("," value)*] "]"
object : "{" [pair ("," pair)*] "}"
pair   : string ":" value
string : ESCAPED_STRING
%import common.ESCAPED_STRING
%import common.SIGNED_NUMBER
%import common.WS
%ignore WS
"""


def ignore_errors(e):
    if isinstance(e, UnexpectedCharacters):
        pass # skip
    elif isinstance(e, UnexpectedToken):
        if e.token.type == "$END":
            e.interactive_parser.accepts() # returns a set of accepted tokens after "city":
           # I want to remove everything in a string until string accepted
    else:
        return False
    return True




p = Lark(grammar, parser="lalr", strict=False, maybe_placeholders=False)
r = p.parse("""{
    "name": "John",,
    "age": 30,#
    "city": "New York"
}""", on_error=ignore_errors)


new_json = Reconstructor(p).reconstruct(r)
print (new_json)

# print(r)

I hope I explained detailed enough

[erezsh]

They way I understand it, you need to fix ignore_errors to remove the bad token (I think by returning false) and replacing with something else with the interactive parser provided in the argument

[XvKuoMing]

Yes, I have read your code and did this:

def ignore_errors(e):
return True

so, this function ignores any error at all.
in case of UnexpectedCharacter it works fine,
but when it comes to UnexpectedToken there it does not seem trivial.
Example:

{
"age": 15,
"city": "Toronto
}

Using default json grammar, and my ignore_errors function,
I will recieve the Unexpected $END token.
When I examined what exactly happens it seems to work like this:
last advanced token is ":", we can see it via tokens_history.
since parser fails at creating ESCAPED_STRING token (because we miss closing "), it automatically discards all characters "Toronto
it seems logical, but what if I want to get all this discarded characters from last_accepted : position, till current error pos in a stream?
so ideally I want something like this:

def ignore_errors(e):
    if isinstance(e, UnexpectedCharacter):
        return True
    elif isinstance(e, UnexpectedToken):
        last_accepted_token = e.tokens_history[-1]
        awaited_tokens = e.interactive_parser.accepts()
        discarded_string = "Toronto
        which_terminal_failed_to_lex = ESCAPED_STRING
        return True
    return False

this way I might be able to reconstruct failed token and try again.

Ideally, it would be great if there was a way to find the shortest path to end state by iterative over current accepted tokens.

thank you for your response in advance, sorry for a lot of questions

[erezsh]

it seems logical, but what if I want to get all this discarded characters from last_accepted : position, till current error pos in a stream?

Isn't this information available in the exception / interactive parser?

Ideally, it would be great if there was a way to find the shortest path to end state by iterative over current accepted tokens.

I implemented exactly this is Preql, as a way to do autocomplete. Maybe you can use it:

https://github.com/erezsh/Preql/blob/115a1ce7f3d85331c0f8424fff175d0d39379b87/preql/core/autocomplete.py#L160C8-L160C49

[XvKuoMing]

it seems logical, but what if I want to get all this discarded characters from last_accepted : position, till current error pos in a stream?

Isn't this information available in the exception / interactive parser?

Ideally, it would be great if there was a way to find the shortest path to end state by iterative over current accepted tokens.

I implemented exactly this is Preql, as a way to do autocomplete. Maybe you can use it:

https://github.com/erezsh/Preql/blob/115a1ce7f3d85331c0f8424fff175d0d39379b87/preql/core/autocomplete.py#L160C8-L160C49

I did not found, there is only info about current pos in a stream (position when error raised).

Thank you for sharing code, I will take a look!