ASSERTION FAILED gc_vlhgc/GlobalMarkingScheme.cpp:922: ((false && (__null != classObject)))

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_sanity.functional_x86-64_mac_OpenJDK11_testList_0/127 - mac12-x86-4
SharedCPEntryInvokerTests_4 -XX:+UseCompressedOops -Xjit -Xgcpolicy:balanced

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk11_j9_sanity.functional_x86-64_mac_OpenJDK11_testList_0/127/functional_test_output.tar.gz

20:25:40  01:25:40.007 0xe8e1c00    j9mm.107    *   ** ASSERTION FAILED ** at /Users/jenkins/workspace/Build_JDK11_x86-64_mac_OpenJDK11/openj9/runtime/gc_vlhgc/GlobalMarkingScheme.cpp:922: ((false && (__null != classObject)))

[dmitripivkine]

Looking to it

[dmitripivkine]

This assertion means GC discovered j9class with class object set to NULL. This is not allowed. I will provide more details later on.

[dmitripivkine]

This assertion occur during scanning object !j9object 0x00000000813f0e60 concurrently. This object happen to be classLoaderObject for !j9classloader 0x00007FAB05140358 org/openj9/test/invoker/util/GeneratedClassLoader. Discovering this object by GC means this class loader is alive, so GC walks this class loader hash table to scan its classes. During this scan one of classes had classObject set to NULL. However in the core this class loader has three RAM classes and each of them has classObject set:

> !allclasses | grep -i 7FAB05140358
0x000000000EAC8D00	0x0000000000000360	0x00007FAB05140358	0x00007FAB07843DF0	0	org/openj9/test/invoker/Dummy
0x000000000EAC8900	0x0000000000000330	0x00007FAB05140358	0x00007FAB07843C20	0x000000000EAC8D00	org/openj9/test/invoker/Dummy
0x000000000EAC8600	0x0000000000000208	0x00007FAB05140358	0x00007FAB03F10E40	0	org/openj9/test/invoker/Intf

I noticed there was hot class replacement and classObject from one class has been reused for another.
Also the tracepoint right before assertion is:

01:25:39.993886038  0xE7EA000 j9codertvm(j9jit).32 Exit       <jitHotswapOccurred
01:25:39.993914699  0xE7EA000 j9jvmti.108          Exit       <RedefineClasses returning 0

My theory is there is a time hole during HCR when new class is discoverable in the class loader hash table but still classObject not to be set.
@JasonFengJ9 Would you please check can it be a case?