Add CI functional tests and fix messagesoverlay handler bug by elijahr · Pull Request #4132 · espruino/BangleApps

elijahr · 2026-01-18T22:09:39Z

Summary

This PR adds automated functional testing to CI and fixes a bug in messagesoverlay discovered during test development.

CI Functional Tests

Adds functional-tests job to GitHub Actions workflow
Tests run using the EspruinoWebIDE emulator
On PRs: Only tests apps with changed files that have a test.json
On push to master: Tests all apps with test.json

Test Runner Enhancements (`bin/runapptests.js`)

Add 60-second timeout per test to prevent hangs
Add uncaught error detection in console output
Add --id flag to run tests for specific app(s)
Improve verbose output formatting
Add undefinedOrEmpty assertion for arrays (handles [undefined] edge case)

Bug Fix: messagesoverlay Handler Restoration

Problem: Event handlers were being prematurely restored while a message overlay was still being displayed.

Root cause: The show() function registered a remove:cleanup callback on every call to setLCDOverlay(). During showMessage(), show() is called twice (via drawBorder() and drawMessage()). When the second call replaced the first overlay with the same ID, Espruino triggered the first overlay's remove callback, causing premature handler restoration.

Fix: Track overlay state with overlayShowing flag and only register the remove callback on the first show() call.

Documentation

Add docs/testing.md with comprehensive testing guide
Update apps/messagesoverlay/ChangeLog for version 0.12

Files Changed

.github/workflows/nodejs.yml - Add functional-tests job
bin/runapptests.js - Test runner enhancements
apps/messagesoverlay/lib.js - Bug fix for premature cleanup
apps/messagesoverlay/test.json - Clean up test file
apps/messagesoverlay/ChangeLog - Version 0.12
apps/calendar/calendar.js - Fix non-selectable "No events" menu item
docs/testing.md - Testing documentation

Test plan

All messagesoverlay tests pass locally (3/3)
All android tests pass locally (5/5)
CI workflow runs successfully on this PR
Tests detect actual failures (verified with intentional breakage)

🤖 Generated with Claude Code

- Add functional-tests job to GitHub Actions workflow - Runs app tests for apps with test.json - Smart detection: only tests changed apps on PRs, all on master push - Pins EspruinoWebIDE to specific commit for stability - Enhance bin/runapptests.js test runner - Fix syntax bug on line 479 (exitCode → exit) - Add uncaught error detection (Uncaught, ERROR:, ASSERT, stack traces) - Add 60-second per-test timeout to prevent infinite loop hangs - Add summary table with pass/fail/timeout counts - Add docs/testing.md with test.json format documentation - Fix calendar app bug when no events (menu was malformed array/object hybrid)

- calendar: Make "No events" a non-selectable label (undefined) - runapptests: Remove handleConsoleOutput duplication - CI: Only run all tests on push to master, not feature branches Feature branch pushes now test only changed apps since master

The show() function was registering a remove:cleanup callback on every call to setLCDOverlay(). During showMessage(), show() is called twice (via drawBorder and drawMessage). When the second call replaced the first overlay with the same ID, Espruino triggered the first overlay's remove callback, causing premature handler restoration. Fix: Track overlay state with overlayShowing flag and only register the remove callback on the first show() call. Also enhanced assertArray in test runner to treat arrays containing only undefined/null values as empty.

- Fix messagesoverlay premature cleanup on overlay update - Add regression test for overlay update scenario - Fix error detection false positives in test runner (ASSERT vs assertArray) - Fix triple evaluation in assertArray with IIFE pattern - Add timeout cleanup with emu.stopIdle() - Fix calendar duplicate event labels overwriting menu items - Fix CI workflow error suppression (separate git/grep) - Update docs and ChangeLog

bobrippling

Thanks for the big test improvement and the fix! And I also appreciate you being upfront about using an AI, that explains some of the more verbose changes. I've left a few comments but will let @gfwilliams have a look, with this being both a central app change and change to infrastructure

.github/workflows/nodejs.yml

apps/calendar/calendar.js

apps/messagesoverlay/test.json

bin/runapptests.js

TESTING.md

gfwilliams · 2026-01-19T16:59:41Z

This looks good to me - it's nice to have the test harness start to be used, and right now it doesn't seem to add a bunch of time to the execution of the tests (12s?).

To fix the build you need to update the version in the messagesoverlay metadata.json file - that should be very straightforward.

And please can you move the docs/testing.md into just TESTING.md in the main folder, and link to it from README.md - maybe under https://github.com/espruino/BangleApps/blob/master/README.md#testing ?

But otherwise, if you can take a look at the stuff @bobrippling suggested it'd be great to get this merged

Show message with proper back navigation instead of blocking alert when there are no calendar events.

Distinguishes between truly empty arrays and arrays containing only null/undefined values (e.g., sparse arrays from backgrounded handlers).

- Fix bashism in nodejs.yml (== to =) - Use regex for Uncaught detection in test runner - Combine error detection into single uncaughtError variable - Use isTimeout flag instead of string matching - Remove unicode and header/divider lines from summary output - Move docs/testing.md to TESTING.md, trim API sections - Link TESTING.md from README.md - Bump messagesoverlay version to 0.12

- Remove allNullish assertion (unnecessary, undefinedOrEmpty works) - Replace .filter() calls with truthy checks (Espruino lacks filter)

elijahr · 2026-02-03T06:24:16Z

@bobrippling @gfwilliams this is ready for review again. Thank you for your patience.

gfwilliams · 2026-02-03T10:34:47Z

Looks great to me - thanks for your time working on this!

I'm happy to merge if you are @bobrippling ?

bobrippling · 2026-02-03T18:12:21Z

Certainly am - thanks @elijahr!

bobrippling · 2026-02-03T18:28:47Z

Looks like a problem with the tests - @elijahr do you see the same?

elijahr · 2026-02-03T21:12:34Z

@bobrippling looks like the test checks for memory leaks, and is failing due to memory usage going down on the second run, so I guess a good thing!

Here's a fix: #4152

gfwilliams · 2026-02-04T08:07:43Z

Thanks - just merged that!

bobrippling · 2026-02-04T08:23:40Z

Nice, yes thank you!

thyttan · 2026-02-07T16:37:44Z

What could cause this to fail initially but succeed when re-running?

Happened e.g. with this one: https://github.com/espruino/BangleApps/actions/runs/21783161416

bobrippling · 2026-02-07T21:46:58Z

Test swipe handler backgrounding with fastloading (setUI)' │ 't\x1B[D\x1B[D\x1B[D\x1B[D\x1B[D\r\x1B[JUncaught SyntaxError: Got [ERASED] expected EOF

I wonder if it just ran over the watch's free flash and a compact was triggered or similar

@thyttan

## Problem Tests occasionally fail with: ``` Uncaught SyntaxError: Got [ERASED] expected EOF ``` Reported in espruino#4132 by @thyttan: tests fail initially but succeed on re-run. Example: https://github.com/espruino/BangleApps/actions/runs/21783161416/attempts/1 ## Cause The test runner sends commands without waiting for the emulator to finish processing. When one test ends and the next begins, `factoryReset()` can be called while the previous test's commands are still being processed, corrupting the input stream. ## Solution - Track emulator ready state (has it output a response?) - Add `waitForResponse()` to poll until command completes - Add `safeFactoryReset()` that ensures previous commands finished - Wait after each test step (`cmd`, `emit`, `load`, etc.) - Wait after app upload and reset in test setup ## Testing - All functional tests pass locally and in CI - Full test suite (all 4 apps): https://github.com/elijahr/BangleApps/actions/runs/21789313739 - No significant increase in test duration

elijahr · 2026-02-08T01:11:54Z

@thyttan @bobrippling I think what might be going on is a race condition around factoryReset() and sending commands to the emulator. I have a candidate fix here: #4160

elijahr added 4 commits January 18, 2026 07:01

Update messagesoverlay ChangeLog for 0.12

1a90a36

elijahr marked this pull request as draft January 18, 2026 22:59

bobrippling reviewed Jan 19, 2026

View reviewed changes

elijahr added 5 commits January 20, 2026 02:14

Add back button to agenda "No events" screen

90d54f8

Show message with proper back navigation instead of blocking alert when there are no calendar events.

Add allNullish assertion for sparse array checks

6441811

Distinguishes between truly empty arrays and arrays containing only null/undefined values (e.g., sparse arrays from backgrounded handlers).

Simplify calendar events menu construction

e94b090

Fix test assertions for Espruino compatibility

342bf89

- Remove allNullish assertion (unnecessary, undefinedOrEmpty works) - Replace .filter() calls with truthy checks (Espruino lacks filter)

elijahr marked this pull request as ready for review February 3, 2026 05:25

bobrippling merged commit 01aca84 into espruino:master Feb 3, 2026
2 checks passed

elijahr mentioned this pull request Feb 8, 2026

Fix emulator race condition in tests #4160

Open

Conversation

elijahr commented Jan 18, 2026

Summary

CI Functional Tests

Test Runner Enhancements (bin/runapptests.js)

Bug Fix: messagesoverlay Handler Restoration

Documentation

Files Changed

Test plan

Uh oh!

bobrippling left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gfwilliams commented Jan 19, 2026

Uh oh!

elijahr commented Feb 3, 2026

Uh oh!

gfwilliams commented Feb 3, 2026

Uh oh!

bobrippling commented Feb 3, 2026

Uh oh!

Uh oh!

bobrippling commented Feb 3, 2026

Uh oh!

elijahr commented Feb 3, 2026

Uh oh!

gfwilliams commented Feb 4, 2026

Uh oh!

bobrippling commented Feb 4, 2026

Uh oh!

thyttan commented Feb 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bobrippling commented Feb 7, 2026

Uh oh!

elijahr commented Feb 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Test Runner Enhancements (`bin/runapptests.js`)

thyttan commented Feb 7, 2026 •

edited

Loading