Boot process stops on `failed to find dp lane mux`

[jakubgs]

I tried installing 1.1 release on my Rock 5B but it just stops during boot process at:

...
add-symbol-file /home/jakubgs/edk2-rk3588/workspace/Build/ROCK5B/DEBUG_GCC/AARCH64/Silicon/Rockchip/RK3588/Drivers/UsbDpPhyDxe/UsbDpPhyDxe/DEBUG/U0
Loading driver at 0x000E8387000 EntryPoint=0x000E8389F94 UsbDpPhyDxe.efi
failed to find dp lane mux, following dp alt mode

Here is the full log: rock_5b_edk2_boot.log

I've tried several versions like 1.0.1 or 0.12.2 but the result is the same.

[jakubgs]

It appears to be reaching this if:

static int udphy_parse_lane_mux_data(struct rockchip_udphy *udphy, UINT8 *prop, UINTN num_lanes)
{
	int i;

	if (!prop) {
		dev_dbg(udphy->dev,
			"failed to find dp lane mux, following dp alt mode\n");
		udphy->mode = UDPHY_MODE_USB;
		return 0;
	}

https://github.com/edk2-porting/edk2-rk3588/blob/v1.1/edk2-rockchip/Silicon/Rockchip/RK3588/Drivers/UsbDpPhyDxe/UsbDpPhyDxe.c#L539-L548

But not sure why it just stops after that.

[jakubgs]

What's interesting to me is why does this regmap_read_poll_timeout macro not use timeout_ms argument at all?

edk2-rk3588/edk2-rockchip/Silicon/Rockchip/RK3588/Drivers/UsbDpPhyDxe/uboot-env.h

Lines 43 to 57 in 6a682c0

	#define regmap_read_poll_timeout(map, addr, val, cond, sleep_us, \
	timeout_ms) \
	({ \
	int __ret; \
	for (;;) { \
	__ret = regmap_read((map), (addr), &(val)); \
	if (__ret) \
	break; \
	if (cond) \
	break; \
	if ((sleep_us)) \
	udelay((sleep_us)); \
	} \
	__ret ?: ((cond) ? 0 : -ETIMEDOUT); \
	})

It's the last argument but it seems to do nothing at all. Maybe I'm reading it wrong but that seems like a bug.

This function is used inside of rk3588_udphy_status_check several times:

edk2-rk3588/edk2-rockchip/Silicon/Rockchip/RK3588/Drivers/UsbDpPhyDxe/UsbDpPhyDxe.c

Lines 786 to 824 in 6a682c0

	static int rk3588_udphy_status_check(struct rockchip_udphy *udphy)
	{
	unsigned int val;
	int ret;
	if (!(udphy->mode & UDPHY_MODE_USB))
	return 0;
	/* LCPLL check */
	ret = regmap_read_poll_timeout(udphy->pma_regmap,
	CMN_ANA_LCPLL_DONE_OFFSET,
	val, (val & CMN_ANA_LCPLL_AFC_DONE) &&
	(val & CMN_ANA_LCPLL_LOCK_DONE),
	200, 100);
	if (ret) {
	dev_err(udphy->dev, "cmn ana lcpll lock timeout\n");
	return ret;
	}
	if (!udphy->flip) {
	ret = regmap_read_poll_timeout(udphy->pma_regmap,
	TRSV_LN0_MON_RX_CDR_DONE_OFFSET,
	val,
	val & TRSV_LN0_MON_RX_CDR_LOCK_DONE,
	200, 100);
	if (ret)
	dev_err(udphy->dev, "trsv ln0 mon rx cdr lock timeout\n");
	} else {
	ret = regmap_read_poll_timeout(udphy->pma_regmap,
	TRSV_LN2_MON_RX_CDR_DONE_OFFSET,
	val,
	val & TRSV_LN2_MON_RX_CDR_LOCK_DONE,
	200, 100);
	if (ret)
	dev_err(udphy->dev, "trsv ln2 mon rx cdr lock timeout\n");
	}
	return 0;
	}

[jakubgs]

Added some prints to the code:

diff --git a/edk2-rockchip/Silicon/Rockchip/RK3588/Drivers/UsbDpPhyDxe/UsbDpPhyDxe.c b/edk2-rockchip/Silicon/Rockchip/RK3588/Drivers/UsbDpPhyDxe/UsbDpPhyDxe.c
index 8329bf3d..1b5f700c 100644
--- a/edk2-rockchip/Silicon/Rockchip/RK3588/Drivers/UsbDpPhyDxe/UsbDpPhyDxe.c
+++ b/edk2-rockchip/Silicon/Rockchip/RK3588/Drivers/UsbDpPhyDxe/UsbDpPhyDxe.c
@@ -792,30 +792,36 @@ static int rk3588_udphy_status_check(struct rockchip_udphy *udphy)
                return 0;
 
        /* LCPLL check */
+       dev_info(udphy->dev, "BEFORE regmap_read_poll_timeout #1\n");^M
        ret = regmap_read_poll_timeout(udphy->pma_regmap,
                                       CMN_ANA_LCPLL_DONE_OFFSET,
                                       val, (val & CMN_ANA_LCPLL_AFC_DONE) &&
                                       (val & CMN_ANA_LCPLL_LOCK_DONE),
                                       200, 100);
+       dev_info(udphy->dev, "AFTER regmap_read_poll_timeout #1\n");^M
        if (ret) {
                dev_err(udphy->dev, "cmn ana lcpll lock timeout\n");
                return ret;
        }
 
        if (!udphy->flip) {
+               dev_info(udphy->dev, "BEFORE regmap_read_poll_timeout #2\n");^M
                ret = regmap_read_poll_timeout(udphy->pma_regmap,
                                               TRSV_LN0_MON_RX_CDR_DONE_OFFSET,
                                               val,
                                               val & TRSV_LN0_MON_RX_CDR_LOCK_DONE,
                                               200, 100);
+               dev_info(udphy->dev, "AFTER regmap_read_poll_timeout #2\n");^M
                if (ret)
                        dev_err(udphy->dev, "trsv ln0 mon rx cdr lock timeout\n");
        } else {
+               dev_info(udphy->dev, "BEFORE regmap_read_poll_timeout #3\n");^M
                ret = regmap_read_poll_timeout(udphy->pma_regmap,
                                               TRSV_LN2_MON_RX_CDR_DONE_OFFSET,
                                               val,
                                               val & TRSV_LN2_MON_RX_CDR_LOCK_DONE,
                                               200, 100);
+               dev_info(udphy->dev, "AFTER regmap_read_poll_timeout #3\n");^M
                if (ret)
                        dev_err(udphy->dev, "trsv ln2 mon rx cdr lock timeout\n");
        }

What's weird is that it starts printing a BEFORE messages and just stops:

Loading driver at 0x000EEDA0000 EntryPoint=0x000EEDA3074 UsbDpPhyDxe.efi
BEFORE regmap_read_poll_timeout #1
AFTER regmap_read_poll_timeout #1
BEFORE regmap_read_poll_timeout #2
AFTER regmap_read_poll_timeout #2
failed to find dp lane mux, following dp alt mode
BEFORE regmap_read_poll_timeout #1
AFTER regmap_read_poll_timeout #1
BEFORE

Which kinda smells like a hardware issue to me.

[jakubgs]

Another time it passes fine:

add-symbol-file /home/jakubgs/edk2-rk3588/workspace/Build/ROCK5B/DEBUG_GCC/AARCH64/Silicon/Rockchip/RK3588/Drivers/UsbDpPhyDxe/UsbDpPhyDxe/DEBUG/U0
Loading driver at 0x000EEDA0000 EntryPoint=0x000EEDA3074 UsbDpPhyDxe.efi
BEFORE regmap_read_poll_timeout #1
AFTER regmap_read_poll_timeout #1
BEFORE regmap_read_poll_timeout #2
AFTER regmap_read_poll_timeout #2
failed to find dp lane mux, following dp alt mode
BEFORE regmap_read_poll_timeout #1
AFTER regmap_read_poll_timeout #1
BEFORE regmap_read_poll_timeout #2
AFTER regmap_read_poll_timeout #2
add-symbol-file /home/jakubgs/edk2-rk3588/workspace/Build/ROCK5B/DEBUG_GCC/AARCH64/Silicon/Rockchip/RK3588/Drivers/AcpiPlatformDxe/AcpiPlatformDxe0
Loading driver at 0x000EEDDC000 EntryPoint=0x000EEDDF174 AcpiPlatformDxe.efi
add-symbol-file /home/jakubgs/edk2-rk3588/workspace/Build/ROCK5B/DEBUG_GCC/AARCH64/Silicon/Rockchip/RK3588/Drivers/FdtPlatformDxe/FdtPlatformDxe/D0
Loading driver at 0x000EECE1000 EntryPoint=0x000EECE95B0 FdtPlatformDxe.efi

But then stops on:

...
DetectDisplays: HDMI0 status: Not Found
DetectDisplays: HDMI1 status: Not Found
DetectDisplays: DP0 status: Not Found
vp0 have layer nr:0[], primary plane: 2
vp1 have layer nr:4[0 1 2 3 ], primary plane: 2
vp2 have layer nr:4[6 7 8 9 ], primary plane: 8
vp3 have layer nr:0[], primary plane: 2
GetSupportedDisplayModes: Native
GetSupportedDisplayModes: No usable mode found! Falling back to lowest available.
[Variable]END_OF_DXE is signaled
Initialize variable error flag (FF)
Rk860xRegulatorSupported: attached to RK860X_REGULATOR device
Rk860xRegulatorSupported: attached to RK860X_REGULATOR device
UsbPortPowerEnable called
GpioPinSetDirection Group:4 Pin:8 Direction:1
PCI Bus First Scanning

[jakubgs]

Sometimes it stops on 2nd attempt at number one regmap_read_poll_timeout call:

Loading driver at 0x000EEDA0000 EntryPoint=0x000EEDA3074 UsbDpPhyDxe.efi
BEFORE regmap_read_poll_timeout #1
AFTER regmap_read_poll_timeout #1
BEFORE regmap_read_poll_timeout #2
AFTER regmap_read_poll_timeout #2
failed to find dp lane mux, following dp alt mode
BEFORE regmap_read_poll_timeout #1

And sometimes it reaches the AFTER but just stops mid-way, which suggests hardware failure.

Loading driver at 0x000EEDA0000 EntryPoint=0x000EEDA3074 UsbDpPhyDxe.efi
BEFORE regmap_read_poll_timeout #1
AFTER regmap_read_poll_timeout #1
BEFORE regmap_read_poll_timeout #2
AFTER regmap_read_poll_timeout #2
failed to find dp lane mux, following dp alt mode
BEFORE regmap_read_poll_timeout #1
AFTER regmap_

Either clock stopped, or power issue, or some interrupt caused it to hang. But I tried several power supplies.

[jakubgs]

Although it is possible the partial print is due to serial output getting cut mid-string because the CPU is now stuck in a tight loop (or you crashed) before the UART FIFO drains. So it might still be a software issue, or both.

[jakubgs]

I implemented a timeout for regmap_read_poll_timeout, and now it hangs on PCI Bus First Scanning:

PCI Bus First Scanning
CreateRootBridge
PciRootBridgeEnumerator START
PciRootBridgeEnumerator LOOP #1 START
PciRootBridgeEnumerator LOOP #1 ITERATION
PciRootBridgeEnumerator LOOP #1 END
PciRootBridgeEnumerator ResetAllPpbBusNumber BEFORE
ResetAllPpbBusNumber START
ResetAllPpbBusNumber LOOP #1 START
ResetAllPpbBusNumber LOOP #1 ITERATION
ResetAllPpbBusNumber LOOP #1 NESTED ITERATION
ResetAllPpbBusNumber LOOP #1 NESTED PciDevicePresent BEFORE
PciDevicePresent START
PciDevicePresent EFI_PCI_ADDRESS BEFORE
PciDevicePresent EFI_PCI_ADDRESS AFTER
PciDevicePresent PciRootBridgeIo->Pci.Read #1 BEFORE
Pci.Read: PciRootBridgeIo=EEE22270 EfiPciWidthUint32=2 Address=0x0

Specifically on Pci.Read call on the first device. Definitely smells like a hardware issue.

[jakubgs]

What's interesting is that the stock bootloader identifies NVMe just fine without issues:

USB is stopped. Please issue 'usb start' first.
MMC: no card present
mmc_init: -123, time 0
pcie@fe150000: PCIe Linking... LTSSM is 0x1
pcie@fe150000: PCIe Linking... LTSSM is 0x210023
pcie@fe150000: PCIe Link up, LTSSM is 0x230011
pcie@fe150000: PCIE-0: Link up (Gen3-x4, Bus0)
pcie@fe150000: invalid flags type!

Device 0: Vendor: 0x144d Rev: 2B2QEXM7 Prod: S4EUNF0M663546L     
            Type: Hard Disk
            Capacity: 238475.1 MB = 232.8 GB (488397168 x 512)
... is now current device
Scanning nvme 0:1...

Device 1: unknown device
Card did not respond to voltage select!
mmc_init: -95, time 14

Device 2: Vendor: 0x2207 Rev: V1.00 Prod: sfc_nor
            Type: Hard Disk
            Capacity: 16.0 MB = 0.0 GB (32768 x 512)
... is now current device

But it seems to not be able to recognize anything connected via USB:

=> usb start
starting USB...
No working controllers found