100% conformance for a past ECMAScript Test Suite

[alabhyajindal]

Given that ECMAScript is an ever-evolving standard, is it worth it to focus efforts on older language features and achieve a 100% conformance there? Our README file currently states:

...it has support for more than 90% of the latest ECMAScript specification

This is great! But I think mentioning that Boa passes 100% of a previous standard is even more attractive. Something like the following, taken from the QuickJS home page:

Passes nearly 100% of the ECMAScript Test Suite tests when selecting the ES2023 features (see test262.fyi).

Interested to hear what others think about this.

[KieranP]

The Test262.fyi site has some interesting revelations: https://test262.fyi/#|v8,v8_exp,jsc,jsc_exp,sm,sm_exp,hermes,qjs,qjs_ng,libjs,graaljs,boa,kiesel,porffor,nova,babel

Looking at the sidebar, Boa's overall conformance is already doing better (at 93.6%) than JavascriptCore (at 88.8% stable release, 93.2% beta), which is the engine that believe powers Safari and Bun, and V8 (88.6% stable release) which powers Chrome and Node.

And under the "Editions" section, Boa is already beating QuickJS in most areas (though it seems that the ES5 edition must have errored out, I doubt Boa supports 0% of ES5).

[alabhyajindal]

Should we modify the README to better convey Boa's conformance? Something like, Boa fully conforms with the ES2022 specification. I'm suggesting this because I believe the current opening in the README undersells the project:

Boa is an experimental JavaScript lexer, parser and interpreter written in Rust 🦀, it has support for more than 90% of the latest ECMAScript specification. We continuously improve the conformance to keep up with the ever-evolving standard.

Seeing this as a newcomer a couple days ago, my first impression was that this is a toy JavaScript engine, which Boa definitely isn't.

[KieranP]

I understand the enthusiasm. It is nice to see another close contender in the JS engine space. But given my limited testing of Boa, I think the description still accurately coveys its current state.

The engine is still experimental, that is, it can't yet be marked as stable/reliable in the same way that V8/Spidermonkey/JSCore are. I ran a rather simple script earlier today that just does some looping and updating of Arrays and Maps, and the script repeatedly crashes.

And the phrase "support for more than 90% of the latest ECMAScript specification" is accurate, Boa sits at 93.6% overall, quite a feat considering it doesn't have large commercial support like the JS engines from Google, Apple, or Mozilla do. But even then, 100% conformance is very difficult to achieve. If you take a look at the link I posted above, not one other JS engine scores an overall 100% after ES5.

And another aspect to consider: if Boa were to reach and legitimately say "fully conforms to the ES2022 specification" (or some other version), that doesn't mean much if another version (e.g. 2021) is at 50%. It could give a distorted view. Rather than a single % metric, I think the better approach would be provide a table, showing the % of support for each versions additions (see below).

But then there is effort in maintaining/updating that, which is probably why the README just points us to test262.fyi :-) So highlighting the % of overall coverage for the whole current JS spec is a safer metric.

Edition	Conformance
ES5	100%
ES6	97%
ES2016	89%
ES2017	81%
ES2018	98%
ES2019	100%
ES2020	82%
ES2021	72%
ES2022	100%
ES2023	87%
ES2024	90%
ESNext	83%
Total	93.6%

[alabhyajindal]

Thanks! This is really helpful 😁