Zigbee sample on bparasite@2.0.0/nrf52840 enters Zephyr fatal handler (blinky/ble OK)

[feliksdd]

Hi,

I’m testing the Zigbee sample on JLCPCB produced bparasite@2.0.0/nrf52840 boards. The same behavior is observed on 10 different boards.

What works

• samples/blinky → OK
• samples/ble → OK
• Another Zigbee firmware (stanvn/zigbee-plant-sensor) → successfully joins network

What does not work

• code/nrf-connect/samples/zigbee builds successfully
• Flash via ST-Link + OpenOCD works (verify OK, mass_erase done before flashing)
• After reset: no LED activity / no pairing indication

Build command

docker run --rm -v ${PWD}:/repo nordicplayground/nrfconnect-sdk:v2.7-branch \
west build --pristine --board bparasite@2.0.0/nrf52840 \
--build-dir /repo/code/nrf-connect/samples/zigbee/build \
/repo/code/nrf-connect/samples/zigbee

Flash command

openocd -f interface/stlink.cfg -f target/nrf52.cfg \
-c "init; reset halt; nrf52 mass_erase; program build/zephyr/zephyr.hex verify; reset run; exit"

Debugging (OpenOCD)

> reset halt
target halted due to debug-request, current mode: Thread
pc: 0x0000aedc

> resume
> sleep 200
> halt
target halted due to debug-request, current mode: Handler SVCall
pc: 0x0004afe6

Registers:

r0 = 0x00000004
lr = 0x00021871
pc = 0x0004afe6

Fault status:

CFSR = 0x00000000
HFSR = 0x00000000

addr2line

arch_system_halt
kernel/fatal.c:30

k_sys_fatal_error_handler
kernel/fatal.c:44

__start
arch/arm/core/cortex_m/reset.S:80

Interpretation

Looks like an early Zephyr fatal error / kernel panic, not a CPU fault and not a flashing issue (image starts and runs briefly).

Additional note

Another project for the same hardware (stanvn/zigbee-plant-sensor) suggests using:

set WORKAREASIZE 0x4000

with OpenOCD. Not sure if relevant here since flashing + verify works.

Questions

• Is this a known issue for the Zigbee sample on bparasite@2.0.0/nrf52840?
• Are there known prerequisites (DT / hardware / config) beyond BLE/Blinky?
• Any recommendation to get more detailed assert/fatal output for this sample?

Thanks!

[rbaron]

Hey @feliksdd, thanks for the nicely explained issue.

This is not to be expected. I think a lot of us (myself included) are running this exact sample for over a year (since 8608cb6 was merged).

I just tried it out myself with the same workflow as you:

$ docker run --rm -v ${PWD}:/repo nordicplayground/nrfconnect-sdk:v2.7-branch \
west build --pristine --board bparasite@2.0.0/nrf52840 \
--build-dir /repo/code/nrf-connect/samples/zigbee/build_docker \
/repo/code/nrf-connect/samples/zigbee

...

[364/365] Linking C executable zephyr/zephyr.elf
Memory region         Used Size  Region Size  %age Used
           FLASH:      356164 B       964 KB     36.08%
             RAM:       68888 B       256 KB     26.28%
        IDT_LIST:          0 GB        32 KB      0.00%
Generating files from /repo/code/nrf-connect/samples/zigbee/build_docker/zephyr/zephyr.elf for board: bparasite
[365/365] Generating zephyr/merged.hex

Then flashing with openocd (I had to tweak your command a bit, nrf52 mass_erase -> nrf5 mass_erase):

$ openocd -f interface/jlink.cfg -c "transport select swd" -f target/nrf52.cfg \
  -c "init; reset halt; nrf5 mass_erase; program build_docker/zephyr/zephyr.hex verify; reset run; exit"

...

Info : nRF52840-xxAA(build code: D0) 1024kB Flash, 256kB RAM
Info : Mass erase completed.
[nrf52.cpu] halted due to debug-request, current mode: Thread
xPSR: 0x01000000 pc: 0xfffffffe msp: 0xfffffffc
** Programming Started **
Warn : Adding extra erase range, 0x00056f44 .. 0x00056fff
** Programming Finished **
** Verify Started **
** Verified OK **

At this point the LED starts to blink, and I can check out the RTT logs:

$ openocd -f interface/jlink.cfg -c 'transport select swd' -f target/nrf52.cfg -c 'init' -c ' rtt setup 0x20000000 80000 "SEGGER RTT"' -c 'rtt start' -c 'rtt server start 9090 0'
...
Info : accepting 'rtt' connection on tcp/9090

Then on a different terminal:

$ nc localhost 9090
[00:00:00.000,457] <inf> ieee802154_nrf5: nRF5 802154 radio initialized
*** Booting nRF Connect SDK v2.7.0-f783fe7fd18a ***
*** Using Zephyr OS v3.6.99-100befc70c74 ***
[00:00:00.003,540] <inf> littlefs: LittleFS version 2.8, disk version 2.1
[00:00:00.003,784] <inf> littlefs: FS at flash-controller@4001e000:0xfa000 is 6 0x1000-byte blocks with 512 cycle
[00:00:00.003,784] <inf> littlefs: sizes: rd 16 ; pr 16 ; ca 64 ; la 32
[00:00:00.075,103] <wrn> debug_counters: fs_read returned 0, expected 4, assuming first access
[00:00:00.076,141] <inf> app: Dumping debug counters:
[00:00:00.080,108] <inf> app: - boot: 2
[00:00:00.083,129] <inf> app: - main_finish: 1
[00:00:00.084,960] <inf> app: - reset_cause_pin: 1
[00:00:00.088,226] <inf> app: - reset_cause_software: 1
[00:00:00.090,484] <inf> app: - sensors_read_after: 4
[00:00:00.092,864] <inf> app: - sensors_read_before: 4
[00:00:00.993,835] <inf> zigbee_app_utils: Enabled sleepy end device behavior.
[00:00:00.996,490] <inf> zigbee_app_utils: Production configuration is not present or invalid (status: -1)
[00:00:00.996,978] <inf> zigbee_app_utils: Zigbee stack initialized
[00:00:01.002,716] <inf> zigbee_app_utils: Device started for the first time
[00:00:01.002,746] <inf> zigbee_app_utils: Start network steering
[00:00:01.002,777] <inf> zigbee_app_utils: Started network rejoin procedure.
[00:00:42.598,205] <inf> zigbee_app_utils: Network steering was not successful (status: -1)
[00:01:25.147,155] <inf> zigbee_app_utils: Network steering was not successful (status: -1)

# My comment: the steering fails as expected, because I'm not pairing the device.

Could you try checking the RTT logs if there's any? Out of excess of caution, can you confirm you're running on a clean main git checkout?

[feliksdd]

Thank you for the detailed response and for trying to reproduce my issue. Unfortunately, the command doesn't work for me because I'm using an ST-Link. I get the following error message:
Error: Debug adapter doesn't support ‘swd’ transport
Do you have any idea how I could adapt the command for ST-Link? Thank you very much!

[rbaron]

I managed to pull RTT logs with an ST-Link v2 using similar commands.

For flashing:

$ openocd -f interface/stlink.cfg -f target/nrf52.cfg \
  -c "init; reset halt; nrf5 mass_erase; program build_docker/zephyr/zephyr.hex verify; reset run; exit"

...

[nrf52.cpu] halted due to debug-request, current mode: Thread
xPSR: 0x01000000 pc: 0xfffffffe msp: 0xfffffffc
Info : nRF52840-xxAA(build code: D0) 1024kB Flash, 256kB RAM
Info : Mass erase completed.
[nrf52.cpu] halted due to debug-request, current mode: Thread
xPSR: 0x01000000 pc: 0xfffffffe msp: 0xfffffffc
** Programming Started **
Warn : Adding extra erase range, 0x00056f44 .. 0x00056fff
** Programming Finished **
** Verify Started **
** Verified OK **

For reading the logs, in one terminal:

$ openocd -f interface/stlink.cfg -f target/nrf52.cfg \
  -c "init; rtt setup 0x20000000 0x40000 \"SEGGER RTT\"; rtt start; rtt server start 9090 0"

In another terminal:

$ nc localhost 9090
[00:00:00.000,427] <inf> ieee802154_nrf5: nRF5 802154 radio initialized
*** Booting nRF Connect SDK v2.7.0-f783fe7fd18a ***
*** Using Zephyr OS v3.6.99-100befc70c74 ***
[00:00:00.003,540] <inf> littlefs: LittleFS version 2.8, disk version 2.1
[00:00:00.003,753] <inf> littlefs: FS at flash-controller@4001e000:0xfa000 is 6 0x1000-byte blocks with 512 cycle
[00:00:00.003,784] <inf> littlefs: sizes: rd 16 ; pr 16 ; ca 64 ; la 32
[00:00:00.079,803] <wrn> debug_counters: fs_read returned 0, expected 4, assuming first access
[00:00:00.080,841] <inf> app: Dumping debug counters:
[00:00:00.088,256] <inf> app: - boot: 4
[00:00:00.092,803] <inf> app: - main_finish: 3
[00:00:00.097,106] <inf> app: - reset_cause_pin: 2
[00:00:00.104,400] <inf> app: - reset_cause_software: 1
[00:00:00.107,635] <inf> app: - reset_cause_unknown: 1
[00:00:00.111,938] <inf> app: - sensors_read_after: 16
[00:00:00.116,394] <inf> app: - sensors_read_before: 16
[00:00:01.017,517] <inf> zigbee_app_utils: Enabled sleepy end device behavior.
[00:00:01.020,294] <inf> zigbee_app_utils: Production configuration is not present or invalid (status: -1)
[00:00:01.020,782] <inf> zigbee_app_utils: Zigbee stack initialized
[00:00:01.026,519] <inf> zigbee_app_utils: Device started for the first time
[00:00:01.026,519] <inf> zigbee_app_utils: Start network steering
[00:00:01.026,580] <inf> zigbee_app_utils: Started network rejoin procedure.

[feliksdd]

Thanks, RTT logging made the issue much clearer.

I tested two hardware variants:

1) 5 Boards without SHTC3 populated

The sample reaches LittleFS mount, then fails in the sensor path with:

<err> shtc3: Error -5 ...
<err> sensors: Error -5 ...
ASSERTION FAIL [!prst_sensors_read_all(&sensors)] @ CMAKE_SOURCE_DIR/src/main.c:304
>>> ZEPHYR FATAL ERROR 4: Kernel panic

So for boards without SHTC3, the behavior is now understood: prst_sensors_read_all(&sensors) returns -5 and the sample asserts in main.c:304.

2) 5 Boards with SHTC3 populated

These fail differently and much earlier:

[00:00:00.000,000] <err> os: ***** BUS FAULT *****
[00:00:00.000,000] <err> os:   Imprecise data bus error
[00:00:00.000,000] <err> os: r0/a1:  0x0004ea44  r1/a2:  0x00050d85  r2/a3:  0x00050d85
[00:00:00.000,000] <err> os: r3/a4:  0x0000ac2b r12/ip:  0xd0000000 r14/lr:  0x2000c9d8
[00:00:00.000,000] <err> os:  xpsr:  0x0000ac00
[00:00:00.000,000] <err> os: s[ 0]:  0x001898e0  s[ 1]:  0x00000000  s[ 2]:  0x0004ea44  s[ 3]:  0x00050d85
[00:00:00.000,000] <err> os: s[ 4]:  0x00050d85  s[ 5]:  0x0000ac2b  s[ 6]:  0xd0000000  s[ 7]:  0x2000c9d8
[00:00:00.000,000] <err> os: s[ 8]:  0x0000001a  s[ 9]:  0x0000ad89  s[10]:  0x001898e0  s[11]:  0x00000000
[00:00:00.000,000] <err> os: s[12]:  0x0004ea44  s[13]:  0x00050d85  s[14]:  0x00050d85  s[15]:  0x0000ac2b
[00:00:00.000,000] <err> os: fpscr:  0xd0000000
[00:00:00.000,000] <err> os: Faulting instruction address (r15/pc): 0x0000001a
[00:00:00.000,000] <err> os: >>> ZEPHYR FATAL ERROR 26: Unknown error on CPU 0
[00:00:00.000,000] <err> os: Current thread: 0x20002c38 (unknown)
[00:00:00.000,000] <err> os: Halting system

This happens essentially immediately at boot and looks like a real bus fault, not the later assertion seen on boards without SHTC3.

So at this point it seems there are two separate behaviors:

• without SHTC3: expected assert due to sensor read failure
• with SHTC3: early bus fault in the SHTC3 / I2C / DT-related path

Does this point to a known issue in the SHTC3 path or DeviceTree configuration for this sample?
Using stanvn/zigbee-plant-sensor all boards are booting and SHTC3 is returning data.

[feliksdd]

It probably really has to do with access to the SHTC3. When I disable it, the program runs as expected:

code/nrf-connect/samples/zigbee/app.overlay:

&shtc3 {
    status = "disabled";
};

[rbaron]

It seems the mystery is slowly unraveling... Let's focus on the second batch of boards, with the temp sensor populated.

I find it odd that it fails earlier. As before, this is basically the config many of us have been running. On this batch, does the ble sample work? Of the read_soil_loop? Now that we can pull RTT logs, this should clarify things.

Other long shots: the SHT3C is mounted wrong (ex. 180 degrees off); or some other systematic hardware assembly that could affect 5 boards. Were these boards hand assembled? Did you edit the BOM? As a last resort, could you link a photo of your board?