How does an LLM compare?

Do an experiment where a version of the code with all test markers, expected diffs, etc., are stripped from the code and an LLM is asked to perform a refactoring.

Is it any good or does it just generate slop because it doesn't understand the AST?