Преглед на файлове

Bring up xHCI USB on Braswell and enable the ASIX dongle

The xHCI driver now enumerates every USB2 root port on the HP Chromebook
11 G5 EE.  The real-hardware fixes:

- reset the controller properly: wait for CNR=0, halt and wait for HCH
  before HCRST, leave the required gap after HCRST before touching
  another register (Intel CherryView/Braswell), take ownership from the
  firmware, and program CONFIG/DCBAAP/scratchpads/CRCR/ERST only while
  halted;
- write PORTSC through a neutral mask (read-only + RWS bits only):
  writing the raw read value back re-asserts PR and resets the device
  after SET_ADDRESS, so every later transfer goes to a dead address;
- use the port-speed EP0 max packet size (64 high/full speed, 8 low
  speed) for Address Device instead of a hardcoded 8;
- address the device before the first descriptor read and issue Address
  Device with BSR=0 straight from the Enabled state; the input slot
  context's device address stays 0 and the xHC-assigned address is read
  back from the output context;
- encode EP0 control stages as one TD per stage (IOC on the Setup, Data
  and Status TRB, no CHAIN).  This Braswell xHC never completes a
  chained multi-TRB control TD: the data stage hangs with no transfer
  event at all, while the per-stage encoding completes and returns the
  descriptor.  IN data stages set TRB_ISP.

usb.c assigns the USB address first and reads descriptors from the
addressed slot (the Default-state transfer is unreliable here).  The
ASIX probe now calls netdev_hotplug(), so a dongle plugged in after boot
becomes the active NIC when the boot probe found none.

The UEFI loader allocates the rootfs pack as a loader-data buffer below
the 512 MB identity window instead of copying it to the top of usable
RAM after ExitBootServices.  The old copy walked into OVMF code pages
that stay read-only and faulted once the pack grew, and an address above
512 MB faulted ramfs_init() because the kernel only maps that much.
Johannes Zellner преди 1 седмица
родител
ревизия
0c0b40c1c8
променени са 10 файла, в които са добавени 382 реда и са изтрити 118 реда
  1. 50 9
      AGENTS.md
  2. 2 0
      asix.c
  3. 1 1
      docs/boot.html
  4. 1 1
      docs/drivers.html
  5. 1 1
      docs/memory.html
  6. 28 45
      efi/main.c
  7. 13 0
      netdev.c
  8. 5 0
      netdev.h
  9. 23 18
      usb.c
  10. 258 43
      xhci.c

+ 50 - 9
AGENTS.md

@@ -144,9 +144,11 @@ make clean
    (partition 1 at LBA 63).
 5. **UEFI (`efi/`)** — an x86-64 PE32+ `BOOTX64.EFI` loads from the ESP. It
    picks a GOP mode, converts `GetMemoryMap` into the E820 map at `0x1000`,
-   stages `kernel.bin` below 4 GB, fills `0x500` (boot info) and `0x600`
-   (framebuffer), calls `ExitBootServices`, copies the embedded gz rootfs to
-   the top of RAM below 512 MB, copies a 192-byte trampoline to `0x800`,
+   stages `kernel.bin` below 4 GB, allocates a loader-data buffer below the
+   512 MB identity window (above the low 1 MB) for the embedded gz rootfs
+   and copies it there, fills
+   `0x500` (boot info) and `0x600` (framebuffer), calls `ExitBootServices`,
+   copies a 192-byte trampoline to `0x800`,
    leaves IA-32e mode (`CR0.PG`, `EFER.LME`, `CR4.PAE/PGE/PCIDE/SMEP/SMAP/
    LA57`), copies the kernel to `0x7E00` and jumps to `kernel_entry`.  The
    BIOS boot sector / VESA stage are not executed on this path; the legacy
@@ -251,8 +253,9 @@ it to `0x7E00`.  The kernel region itself only has one wall (`_end` below
   fallback pool (a static BSS array; no fixed address).
 - **UEFI boot**: `bootinfo.h` (the `0x500` block and its checksum/accessors),
   `efi/efi.h` (minimal UEFI types and table/protocol layouts), `efi/main.c`
-  (GOP mode choice, `GetMemoryMap` → E820, staging below 4 GB, rootfs copy,
-  boot-info/framebuffer blocks, `ExitBootServices`), `efi/tramp.S` +
+  (GOP mode choice, `GetMemoryMap` → E820, kernel staging below 4 GB and
+  rootfs pack in loader-data pages below 512 MB, boot-info/framebuffer
+  blocks, `ExitBootServices`), `efi/tramp.S` +
   `efi/tramp.ld` (the `0x800` long-mode exit, kernel copy and jump) and
   `efi/payload.S` (embeds `kernel.bin`, `cmds_pack_gz.bin` and the trampoline
   in `BOOTX64.EFI`).  The legacy `vesa_stage.S` writes the same v2 boot info
@@ -543,8 +546,9 @@ it to `0x7E00`.  The kernel region itself only has one wall (`_end` below
   frame list at `0x3000`, one instance), `ohci.c` (MMIO OHCI, up to 4
   instances, HCCA + ED/TD lists, polling), `ehci.c` (MMIO EHCI, QH/qTD
   asynchronous schedule, up to 4 instances) and `xhci.c` (MMIO xHCI, USB2
-  root ports) implement it.  `lsusb.c` — the `lsusb` command (lists every
-  controller and its tree).  The devices work on every controller type and
+  root ports; halt/HCH reset sequence and BIOS/OS handoff) implement it.
+  `lsusb.c` — the `lsusb` command (lists every controller and its tree).
+  The devices work on every controller type and
   can be plugged in at any time (root-port connect-change watcher); `lsusb`
   re-enumerates the bus, after which `usbhid` re-inits itself via a scan
   generation counter.
@@ -910,8 +914,8 @@ one-sector `NBROOT1` header (magic, version, exact `pack_bytes`, checksum,
 gzip flag).  The build rejects a pack above the `PACK_MAX` cap (1 MB);
 `libdeflate` buys several KB over `gzip -9` and is still preferred.  The boot
 path loads the gz pack into RAM — the legacy stage copies it directly to
-`0x900000` with INT 15h AH=87h, the UEFI loader via an embedded copy at the
-top of RAM below 512 MB, kexec via its staging buffer — and `ramfs_init()`
+`0x900000` with INT 15h AH=87h, the UEFI loader into a loader-data buffer
+below 512 MB, kexec via its staging buffer — and `ramfs_init()`
 inflates it into page-allocated memory and mounts it read-only at `/ramfs`.
 The kernel image no longer contains it, so the pack no longer moves `_end`;
 the high-memory legacy path supports up to 1 MB (known issue 8).  The UI
@@ -1055,6 +1059,43 @@ so addresses do not collide.
   running. The PCI scan checks **all functions** (real PIIX3/PIIX4/VIA
   layouts put the USB function at 0:07.2 / 0:01.2) and ignores controllers
   whose BARs the firmware left unassigned. One instance only.
+- **xHCI**: MMIO BAR0 (PCI class `0c03:30`), one controller, polled.  The
+  Supported Protocol extended capabilities decide which root ports are USB2;
+  USB3 SuperSpeed ports are left alone.  `xhci_init()` waits for `CNR=0`,
+  clears `RS` and waits for `USBSTS.HCH` **before** `HCRST` (the xHC may
+  ignore the reset while running), waits 1-2 ms after the HCRST write before
+  touching any other register (Intel CherryView/Braswell requires this, or
+  the reset can be left partially applied), does the BIOS/OS handoff with a
+  bounded wait for the BIOS-owned semaphore to clear, and only programs
+  `CONFIG`/`DCBAAP`/scratchpads/`CRCR`/`ERST` once the post-reset controller
+  is halted.  Port reset clears stale change latches, waits for `PRC` then
+  `PED`, and adds the 10 ms USB2 reset-recovery delay.  The BSR Address
+  Device uses the port-speed EP0 max packet size (64 for high/full speed,
+  8 for low speed); using 8 for a high-speed slot leaves EP0 unusable on
+  real xHCs.  Enumeration uses the traditional scheme: **SET_ADDRESS before
+  the first descriptor read**.  The xHCI SET_ADDRESS intercept creates the
+  slot in the Enabled state and issues Address Device with **BSR=0**
+  directly (Enabled -> Addressed), never leaving the slot in the Default
+  state; the Default-state EP0 transfer was unreliable on the Braswell.
+  The input slot context's Device Address stays **0** — the xHC assigns
+  the USB device address and reports it in the output context, which the
+  driver stores for later Configure Endpoint commands (writing a target
+  address there makes Address Device fail with Parameter Error).  Control
+  transfers wait 1 s, re-ringing the EP0 doorbell every 100 ms.  EP0 stages
+  are encoded the way EDK2's XhciDxe does it:
+  **each stage is its own TD** (IOC on the Setup, Data and Status TRB, no
+  CHAIN).  This Braswell xHC never completes a chained multi-TRB control TD
+  (CHAIN on setup+data, IOC on status only): the data stage hangs and no
+  transfer event is ever posted, while the per-stage encoding completes and
+  returns the descriptor.  IN data stages also
+  set `TRB_ISP`.  PORTSC writes preserve only the read-only
+  and RWS bits (`portsc_neutral()`, Linux's `xhci_port_state_to_neutral`):
+  writing the raw read value back with just the RW1C bits masked re-asserts
+  PR/WPR and resets the device *after* SET_ADDRESS, so the xHC then talks to
+  an address the device no longer uses.
+  Devices plugged in after boot bind through `asix.c`, which calls
+  `netdev_hotplug()` so the dongle becomes the active NIC when the boot
+  probe found none.
 - **Devices** (`usbhid.c`): every HID (class 3) interface with an
   interrupt-IN endpoint gets its own interrupt slot (USB_HCD_INT_SLOTS = 6).
   Boot interfaces (subclass 1, protocol 2/1) are used as-is in boot

+ 2 - 0
asix.c

@@ -12,6 +12,7 @@
 
 #include "asix.h"
 #include "io.h"
+#include "netdev.h"
 #include "timer.h"
 #include "usb.h"
 
@@ -457,6 +458,7 @@ static int device_probe(unsigned int ctrl, unsigned int addr,
 		return 0;
 	}
 
+	netdev_hotplug();
 	return 1;
 }
 

+ 1 - 1
docs/boot.html

@@ -197,7 +197,7 @@
 			<li>Selects a GOP mode with BGRX or RGBX 32bpp, width at most 1280, <code>pitch × height</code> at most 8 MB, and a framebuffer below 4 GB.</li>
 			<li>Calls <code>GetMemoryMap</code> and converts the firmware map into the kernel's E820 format at <code>0x1000</code>.</li>
 			<li>Stages the kernel image below 4 GB and writes the boot-info and framebuffer blocks.</li>
-			<li>Calls <code>ExitBootServices</code>, copies the embedded rootfs pack to the top of RAM below 512 MB, and copies a 192-byte trampoline to <code>0x800</code>.</li>
+			<li>Allocates the rootfs pack buffer as loader data below the 512 MB identity window while boot services are still live, copies the embedded pack into it, calls <code>ExitBootServices</code>, then copies a 192-byte trampoline to <code>0x800</code>.</li>
 			<li>Leaves IA-32e mode (clears <code>CR0.PG</code>, <code>EFER.LME</code>, <code>CR4.PAE/PGE/PCIDE/SMEP/SMAP/LA57</code>), copies the kernel to <code>0x7E00</code>, and jumps to <code>kernel_entry</code>.</li>
 		</ol>
 		<p>

+ 1 - 1
docs/drivers.html

@@ -258,7 +258,7 @@
 					<tr><td>UHCI</td><td>PCI I/O ports</td><td>1</td><td>Frame list at <code>0x3000</code>; control QH plus one QH per interrupt slot.</td></tr>
 					<tr><td>OHCI</td><td>PCI MMIO</td><td>up to 4</td><td>HCCA from the page allocator, static ED/TD sets, TDs start at <code>CC = NotAccessed</code>.</td></tr>
 					<tr><td>EHCI</td><td>PCI MMIO</td><td>up to 4</td><td>Async QH/qTD schedule, page-allocated descriptors, polling only, no periodic list.</td></tr>
-					<tr><td>xHCI</td><td>PCI MMIO</td><td>—</td><td>USB2 root ports only; USB3 SuperSpeed devices are not handled.</td></tr>
+					<tr><td>xHCI</td><td>PCI MMIO</td><td>—</td><td>USB2 root ports only; USB3 SuperSpeed devices are not handled. Halts and waits for <code>HCH</code> around <code>HCRST</code>; EP0 max packet comes from the port speed; EP0 stages are encoded as one TD per stage (EDK2 style) because chained control TDs stall on Braswell.</td></tr>
 				</tbody>
 			</table>
 		</div>

+ 1 - 1
docs/memory.html

@@ -293,7 +293,7 @@
 	<section id="pack">
 		<h2>Rootfs pack lifetime</h2>
 		<ol>
-			<li>The boot path places the gzip pack in RAM: the legacy stage copies it directly to <code>0x900000</code> with <code>INT 15h AH=87h</code>, UEFI at the top of RAM below 512 MB, kexec in its staging buffer.</li>
+			<li>The boot path places the gzip pack in RAM: the legacy stage copies it directly to <code>0x900000</code> with <code>INT 15h AH=87h</code>, UEFI into a loader-data buffer below 512 MB, kexec in its staging buffer.</li>
 			<li><code>mm_init</code> reserves <code>[pack_addr, pack_addr + pack_size)</code> so the frame allocator cannot reuse it.</li>
 			<li><code>ramfs_init</code> takes a private page-allocated copy, validates the <code>NBROOT1</code> header and inflates the pack into it.</li>
 			<li><code>mm_free_range</code> releases the boot region when it is above the VGA shadow reserve. The private copy lives until <code>ramfs_replace</code> or reboot.</li>

+ 28 - 45
efi/main.c

@@ -437,50 +437,11 @@ static void build_e820(EFI_MEMORY_DESCRIPTOR *map, UINTN map_size,
 
 /* ---- boot structures --------------------------------------------------- */
 
-/* Copy the embedded rootfs pack to the top of RAM (below the 512 MB
- * identity window and above the 1..9 MB VGA shadow reserve), matching the
- * legacy stage's placement.  Returns the address, or 0 when it does not fit.
- * Runs after ExitBootServices, so only plain memory accesses. */
-static UINT32 copy_pack(UINT32 size)
-{
-	volatile UINT32 *map = (volatile UINT32 *)E820_ADDR;
-	volatile struct e820_entry_disk *entries =
-		(volatile struct e820_entry_disk *)(E820_ADDR + 8);
-	UINT32 need, count, i, best = 0;
-	UINT32 p;
-
-	if (size == 0 || size > 0x400000u)
-		return 0;
-	need = (size + PAGE_SIZE_ - 1) & ~(PAGE_SIZE_ - 1);
-	if (map[0] != 0xE820E820u)
-		return 0;
-	count = map[1];
-	if (count > 32)
-		count = 32;
-
-	for (i = 0; i < count; i++) {
-		UINT64 base = entries[i].base;
-		UINT64 end = base + entries[i].length;
-		UINT32 a;
-
-		if (entries[i].type != 1 || base >= MAX_PHYS)
-			continue;
-		if (end > MAX_PHYS)
-			end = MAX_PHYS;
-		if (end < base + need)
-			continue;
-		a = (UINT32)(end - need) & ~(PAGE_SIZE_ - 1);
-		if (a < 0x900000u || a <= best)
-			continue;
-		best = a;
-	}
-	if (!best)
-		return 0;
-
-	for (p = 0; p < size; p++)
-		((UINT8 *)(UINTN)best)[p] = pack_image_start[p];
-	return best;
-}
+/* The rootfs pack used to be copied to the top of usable RAM after
+ * ExitBootServices.  That walks into firmware code pages that OVMF maps
+ * read-only (BootServicesCode is still listed as free memory), so the copy
+ * faulted once the pack grew.  Allocate a loader-data buffer instead:
+ * firmware-owned pages are always writable to the loader. */
 
 static void write_bootinfo(UINT32 rsdp, int have_fb, int rgbx,
 			   UINT32 pack_addr, UINT32 pack_size)
@@ -574,6 +535,7 @@ EFI_STATUS efi_main(EFI_HANDLE image, EFI_SYSTEM_TABLE *st)
 	EFI_GRAPHICS_OUTPUT_MODE_INFORMATION info;
 	struct fb_saved fb;
 	EFI_PHYSICAL_ADDRESS staging = 0xFFFFFFFFull;
+	EFI_PHYSICAL_ADDRESS pack_buf = 0xFFFFFFFFull;
 	UINTN map_size, map_key = 0, desc_size = 0;
 	UINT32 desc_ver = 0;
 	UINT32 rsdp;
@@ -651,6 +613,27 @@ EFI_STATUS efi_main(EFI_HANDLE image, EFI_SYSTEM_TABLE *st)
 	com1_hex(staging, 8);
 	com1_puts("\n");
 
+	/* Rootfs pack: allocate a writable loader-data buffer below the
+	 * kernel's 512 MB identity window (and above the low 1 MB it
+	 * reserves) and copy the embedded image into it while boot services
+	 * are still available.  It must be below MAX_PHYS: the kernel only
+	 * identity-maps that much and ramfs_init() reads it after paging is
+	 * enabled. */
+	if (pack_size == 0 || pack_size > 0x400000u)
+		fail_before(st, "rootfs image size invalid");
+	pages = (pack_size + PAGE_SIZE_ - 1) / PAGE_SIZE_;
+	pack_buf = MAX_PHYS;
+	status = bs->AllocatePages(EFI_ALLOCATE_MAX_ADDRESS, EFI_LOADER_DATA,
+				   pages, &pack_buf);
+	if (status || pack_buf < 0x100000ull ||
+	    pack_buf + pack_size > MAX_PHYS)
+		fail_before(st, "cannot allocate rootfs buffer");
+	for (i = 0; i < pack_size; i++)
+		((UINT8 *)(UINTN)pack_buf)[i] = pack_image_start[i];
+	com1_puts("nebos-efi: rootfs staged at 0x");
+	com1_hex(pack_buf, 8);
+	com1_puts("\n");
+
 	/* GetMemoryMap + ExitBootServices.  A map-key change makes the call
 	 * fail; fetch a fresh map and retry. */
 	for (tries = 0; tries < 4; tries++) {
@@ -681,7 +664,7 @@ EFI_STATUS efi_main(EFI_HANDLE image, EFI_SYSTEM_TABLE *st)
 		fail_after("memory map is empty");
 	build_e820((EFI_MEMORY_DESCRIPTOR *)memmap_buf, map_size, desc_size,
 		   &fb);
-	pack_addr = copy_pack(pack_size);
+	pack_addr = (UINT32)pack_buf;
 	com1_puts("nebos-efi: rootfs at 0x");
 	com1_hex(pack_addr, 8);
 	com1_puts("\n");

+ 13 - 0
netdev.c

@@ -100,6 +100,19 @@ const struct netdev *netdev_mgmt(void)
 	return netdev_active();
 }
 
+/* Called from the ASIX device probe after a successful hotplug
+ * enumeration.  The boot-time probe caches a NULL result, so without this a
+ * dongle plugged in later would only be usable through `netdev asix`. */
+void netdev_hotplug(void)
+{
+	if (active || !probed)
+		return;
+	if (asix_netdev.init() == 0) {
+		active = &asix_netdev;
+		netdev_apply_defaults(active);
+	}
+}
+
 /* Stop the NICs before kexec overwrites the kernel.  A still-running
  * receive ring can DMA into pages the new kernel has already reused. */
 void netdev_shutdown(void)

+ 5 - 0
netdev.h

@@ -35,6 +35,11 @@ struct netdev {
  * NULL when no NIC is present. */
 const struct netdev *netdev_active(void);
 
+/* A USB network device appeared after boot (hotplug probe succeeded).  Makes
+ * it the active NIC when the boot probe found none; never overrides an
+ * already-selected driver. */
+void netdev_hotplug(void);
+
 /* Management NIC for netlog/netkexec.  Prefers the ASIX dongle so the
  * direct debug link survives `netdev e1000e`; falls back to the active
  * driver. */

+ 23 - 18
usb.c

@@ -110,7 +110,7 @@ static int print_devices;
 static int scan_busy;
 
 static unsigned char dev_desc[18];
-static unsigned char desc8[8];
+static unsigned char desc8[64];
 static unsigned char hub_desc[16];
 static unsigned char port_st[4];
 static unsigned char conf_hdr[9];
@@ -564,15 +564,33 @@ static void enumerate_new_device(unsigned int ctrl, unsigned int port, int low,
 		return;
 	}
 
-	/* First 8 bytes at address 0 (learn max packet size). */
-	if (usb_control_in(ctrl, 0, 8, low, RT_DEV2HOST_STD,
+	/* Traditional ("old") enumeration scheme: assign the address before
+	 * reading the descriptor.  The xHCI Default-state EP0 transfer is
+	 * unreliable on this hardware, while a slot in the Addressed state
+	 * works.  The device's real max packet size is not known yet, so the
+	 * high/full-speed guess is 64 (low speed is always 8). */
+	mps = low ? 8 : 64;
+	if (usb_control_out(ctrl, 0, mps, low, RT_HOST2DEV_STD,
+			    REQ_SET_ADDRESS, (unsigned short)addr, 0) < 0) {
+		dbg("usb:   SET_ADDRESS failed\n");
+		if (print_devices) {
+			indent(depth);
+			io_putstr("(SET_ADDRESS failed)\n");
+		}
+		return;
+	}
+	next_addr[ctrl] = addr + 1;
+	msleep(5);
+
+	/* First 8 bytes at the assigned address (learn max packet size). */
+	if (usb_control_in(ctrl, addr, mps, low, RT_DEV2HOST_STD,
 			   REQ_GET_DESCRIPTOR,
 			   (unsigned short)(DESC_DEVICE << 8), 0,
 			   desc8, 8) < 0) {
-		dbg("usb:   no response at address 0\n");
+		dbg("usb:   no response at the assigned address\n");
 		if (print_devices) {
 			indent(depth);
-			io_putstr("(no response at address 0)\n");
+			io_putstr("(device descriptor read failed)\n");
 		}
 		return;
 	}
@@ -581,20 +599,7 @@ static void enumerate_new_device(unsigned int ctrl, unsigned int port, int low,
 	if (mps == 0)
 		mps = 8;
 
-	/* Assign an address. */
-	if (usb_control_out(ctrl, 0, mps, low, RT_HOST2DEV_STD,
-			    REQ_SET_ADDRESS, (unsigned short)addr, 0) < 0) {
-		dbg("usb:   SET_ADDRESS failed\n");
-		if (print_devices) {
-			indent(depth);
-			io_putstr("(SET_ADDRESS failed)\n");
-		}
-		return;
-	}
 	dbg_dec("usb:   enumerated device, addr ", addr);
-	msleep(5);
-
-	next_addr[ctrl] = addr + 1;
 	enumerate_device(ctrl, addr, mps, low, port, depth);
 }
 

+ 258 - 43
xhci.c

@@ -75,6 +75,23 @@
 				 PORTSC_OCC | PORTSC_PRC | PORTSC_PLC | \
 				 PORTSC_CEC | PORTSC_PED)
 
+/* Bits that may be written back while preserving the port state:
+ * read-only status (CCS, OCA, speed, device-removable) and the
+ * read/write-state fields (PLS, PP, PIC, wake enables).  Everything else
+ * -- especially PR/WPR (RW1S) and PED (RW1CS) -- must be written as 0 or
+ * the write changes the port state.  Linux's
+ * xhci_port_state_to_neutral() does the same. */
+#define PORTSC_RO_MASK		(PORTSC_CCS | 0x00000008u | \
+				 (PORTSC_SPEED_MASK << PORTSC_SPEED_SHIFT) | \
+				 0x40000000u)
+#define PORTSC_RWS_MASK		(0x000001E0u | PORTSC_PP | 0x0000C000u | \
+				 0x0E000000u)
+
+static unsigned int portsc_neutral(unsigned int ps)
+{
+	return (ps & PORTSC_RO_MASK) | (ps & PORTSC_RWS_MASK);
+}
+
 /* TRB types */
 #define TRB_NORMAL		1
 #define TRB_SETUP		2
@@ -111,8 +128,6 @@
 #define EP_BULK_IN		6
 #define EP_INT_IN		7
 
-#define XHCI_CTRL_MPS		8
-
 struct xhci_trb {
 	unsigned int p0;
 	unsigned int p1;
@@ -123,7 +138,8 @@ struct xhci_trb {
 struct xhci_slot {
 	int used;
 	unsigned int slot_id;
-	unsigned int addr;
+	unsigned int addr;	/* logical address used by usb.c */
+	unsigned int dev_addr;	/* address assigned by the xHC */
 	unsigned int port;
 	unsigned int speed;
 	unsigned int dc;	/* device context page */
@@ -247,6 +263,14 @@ static void dbg_hex(const char *s, unsigned int v) { (void)s; (void)v; }
 static void dbg_dec(const char *s, unsigned int v) { (void)s; (void)v; }
 #endif
 
+static void delay_ms(unsigned int ms)
+{
+	unsigned int t0 = get_ticks_ms();
+
+	while (get_ticks_ms() - t0 < ms)
+		;
+}
+
 /* ---- MMIO helpers ------------------------------------------------------ */
 static unsigned int mmio_rd(unsigned int off)
 {
@@ -388,10 +412,24 @@ static void xhci_poll_locked(void)
 			unsigned int ep = (ev->control >> 16) & 0x1F;
 
 			if (ctl_busy && slot == ctl_slot && ep == 1) {
-				if (ev->p0 == ctl_data_trb) {
+				if (ev->p0 == ctl_setup_trb) {
+					/* The setup stage is its own TD and
+					 * completes first; only an error stops
+					 * the transfer. */
+					if (code != CC_SUCCESS &&
+					    code != CC_SHORT_PACKET) {
+						ctl_status = code;
+						ctl_done = 1;
+					}
+				} else if (ev->p0 == ctl_data_trb) {
 					ctl_actual = (ctl_req_len >= residual) ?
 						ctl_req_len - residual : 0;
 					ctl_data_seen = 1;
+					if (code != CC_SUCCESS &&
+					    code != CC_SHORT_PACKET) {
+						ctl_status = code;
+						ctl_done = 1;
+					}
 				} else if (ev->p0 == ctl_status_trb) {
 					if (!ctl_data_seen)
 						ctl_actual =
@@ -550,6 +588,19 @@ static unsigned int port_speed(unsigned int port)
 	return sp ? sp : 1;
 }
 
+/* Default control-pipe max packet size for the BSR Address Device command.
+ * High speed must be 64; full speed uses the USB core's 64-byte guess (the
+ * descriptor read then corrects it); low speed is 8.  Using 8 for a
+ * high/full-speed slot leaves EP0 unusable on real xHCs. */
+static unsigned int ep0_mps_for_speed(unsigned int speed)
+{
+	if (speed == 2)		/* low speed */
+		return 8;
+	if (speed >= 4)		/* SuperSpeed / SuperSpeedPlus */
+		return 512;
+	return 64;		/* full speed (guess) and high speed */
+}
+
 static int xhci_config_ep(struct xhci_slot *s, unsigned int ep_index,
 			  unsigned int type, unsigned int mps,
 			  unsigned int interval, unsigned int ring)
@@ -567,7 +618,7 @@ static int xhci_config_ep(struct xhci_slot *s, unsigned int ep_index,
 	zero_page(input_ctx, 4096);
 	*(unsigned int *)input_ctx = 0;
 	*(unsigned int *)(input_ctx + 4) = SLOT_FLAG | ep_flag;
-	fill_slot_ctx(s->addr, s->port, s->speed, last);
+	fill_slot_ctx(s->dev_addr, s->port, s->speed, last);
 	fill_ep_ctx(ep_index, type, mps, interval, ring);
 	spin_unlock_irqrestore(&xhci_lock, flags);
 
@@ -579,7 +630,14 @@ static int xhci_config_ep(struct xhci_slot *s, unsigned int ep_index,
 }
 
 /* ---- slots ------------------------------------------------------------- */
-static struct xhci_slot *new_slot(void)
+/* Create a slot.  bsr=1: Address Device with Block Set Address Request,
+ * leaving the slot in the Default state at device address 0 (new
+ * enumeration scheme).  bsr=0: the command carries target_addr and moves
+ * Enabled -> Addressed directly (traditional scheme), which is what Linux
+ * does when the port is addressed before the first descriptor read.
+ * mps == 0 means "derive it from the port speed". */
+static struct xhci_slot *new_slot_common(unsigned int target_addr,
+					 unsigned int mps, int bsr)
 {
 	struct xhci_slot *s;
 	unsigned int slot_id, dc, ep0, flags;
@@ -620,7 +678,10 @@ static struct xhci_slot *new_slot(void)
 	s->ep0_enq = 0;
 	s->ep0_cycle = 1;
 	s->ep_mask = 0;
+	s->dev_addr = 0;
 	ring_init(ep0, XHCI_EP0_TRBS, &s->ep0_enq, &s->ep0_cycle);
+	if (!mps)
+		mps = ep0_mps_for_speed(s->speed);
 
 	ram_wr64(dcbaa + slot_id * 8, dc);
 
@@ -629,24 +690,39 @@ static struct xhci_slot *new_slot(void)
 	*(unsigned int *)input_ctx = 0;
 	*(unsigned int *)(input_ctx + 4) = SLOT_FLAG | EP0_FLAG;
 	fill_slot_ctx(0, s->port, s->speed, 1);
-	fill_ep0_ctx(XHCI_CTRL_MPS, ep0, s->ep0_enq, s->ep0_cycle);
+	fill_ep0_ctx(mps, ep0, s->ep0_enq, s->ep0_cycle);
 	spin_unlock_irqrestore(&xhci_lock, flags);
 
 	rc = xhci_cmd(input_ctx, 0,
-		      TRB_TYPE(TRB_ADDR_DEV) | (slot_id << 24) | TRB_BSR, 0);
+		      TRB_TYPE(TRB_ADDR_DEV) | (slot_id << 24) |
+		      (bsr ? TRB_BSR : 0), 0);
 	if (rc != 0) {
-		dbg("xhci: address default failed\n");
+		dbg(bsr ? "xhci: address default failed\n" :
+		    "xhci: address device failed\n");
+		ram_wr64(dcbaa + slot_id * 8, 0);
+		xhci_cmd(0, 0,
+			 TRB_TYPE(TRB_DISABLE_SLOT) | (slot_id << 24), 0);
 		mm_free_page(dc);
 		mm_free_page(ep0);
 		s->used = 0;
 		return 0;
 	}
 
-	addr_slot[0] = slot_id;
+	/* The xHC assigns the USB device address (the input slot context
+	 * address is 0); keep it for later Configure Endpoint commands. */
+	s->dev_addr = ((unsigned int *)s->dc)[3] & 0xFF;
+	s->addr = bsr ? 0 : target_addr;
+	addr_slot[s->addr] = slot_id;
 	port_slot[s->port] = slot_id;
 	return s;
 }
 
+/* New-scheme slot: Enabled -> Default, at device address 0. */
+static struct xhci_slot *new_slot(void)
+{
+	return new_slot_common(0, 0, 1);
+}
+
 static void free_slot(struct xhci_slot *s)
 {
 	unsigned int i, flags;
@@ -705,7 +781,7 @@ static int set_address(unsigned int new_addr, unsigned int mps)
 	zero_page(input_ctx, 4096);
 	*(unsigned int *)input_ctx = 0;
 	*(unsigned int *)(input_ctx + 4) = SLOT_FLAG | EP0_FLAG;
-	fill_slot_ctx(new_addr, s->port, s->speed, 1);
+	fill_slot_ctx(0, s->port, s->speed, 1);
 	fill_ep0_ctx(mps, s->ep0, s->ep0_enq, s->ep0_cycle);
 	spin_unlock_irqrestore(&xhci_lock, flags);
 
@@ -717,6 +793,7 @@ static int set_address(unsigned int new_addr, unsigned int mps)
 	}
 
 	spin_lock_irqsave(&xhci_lock, &flags);
+	s->dev_addr = ((unsigned int *)s->dc)[3] & 0xFF;
 	addr_slot[0] = 0;
 	s->addr = new_addr;
 	addr_slot[new_addr] = s->slot_id;
@@ -733,6 +810,7 @@ static int control_transfer(unsigned int ctrl, unsigned int addr,
 	struct xhci_slot *s;
 	unsigned int flags, t0, rc, st;
 	unsigned int setup_c, data_c, status_c;
+	unsigned int last_bell;
 
 	(void)ctrl;
 	(void)low_speed;
@@ -741,12 +819,28 @@ static int control_transfer(unsigned int ctrl, unsigned int addr,
 		return -1;
 
 	/* xHCI owns SET_ADDRESS: it is an Address Device command, not a
-	 * control request on the wire. */
-	if (setup[0] == 0x00 && setup[1] == 0x05 && addr == 0)
-		return set_address((unsigned int)setup[2] |
-				   ((unsigned int)setup[3] << 8), mps);
+	 * control request on the wire.  If a Default-state slot exists (new
+	 * scheme), move it to Addressed.  Otherwise this is the traditional
+	 * scheme: create the slot Enabled and address it in one command,
+	 * without ever using the Default state. */
+	if (setup[0] == 0x00 && setup[1] == 0x05 && addr == 0) {
+		unsigned int new_addr = (unsigned int)setup[2] |
+					((unsigned int)setup[3] << 8);
+
+		s = slot_by_addr(0);
+		if (s)
+			return set_address(new_addr, mps);
+		s = new_slot_common(new_addr, mps, 0);
+		return s ? 0 : -1;
+	}
 
 	s = slot_by_addr(addr);
+	if (s && addr == 0 && (int)s->port != last_port) {
+		/* A failed enumeration on another port left a slot at
+		 * address 0; it must not be reused for this device. */
+		free_slot(s);
+		s = 0;
+	}
 	if (!s) {
 		if (addr != 0)
 			return -1;
@@ -769,7 +863,13 @@ static int control_transfer(unsigned int ctrl, unsigned int addr,
 	ctl_slot = s->slot_id;
 	ctl_req_len = len;
 
-	setup_c = TRB_TYPE(TRB_SETUP) | TRB_IDT | TRB_CHAIN;
+	/* Encode the three stages the way EDK2's XhciDxe does: every stage
+	 * is its own TD (no CHAIN) and carries IOC.  This Braswell xHC never
+	 * completes a chained multi-TRB control TD: with Linux's chaining
+	 * (CHAIN on setup+data, IOC on status) the data stage hangs and no
+	 * transfer event is ever posted, while the per-stage encoding
+	 * completes. */
+	setup_c = TRB_TYPE(TRB_SETUP) | TRB_IDT | TRB_IOC;
 	if (len)
 		setup_c |= (dir_in ? 3u : 2u) << 16;
 	ctl_setup_trb = s->ep0 + s->ep0_enq * 16;
@@ -785,7 +885,7 @@ static int control_transfer(unsigned int ctrl, unsigned int addr,
 	ctl_data_trb = 0;
 	if (len) {
 		ctl_data_trb = s->ep0 + s->ep0_enq * 16;
-		data_c = TRB_TYPE(TRB_DATA) | TRB_CHAIN;
+		data_c = TRB_TYPE(TRB_DATA) | TRB_IOC;
 		if (dir_in)
 			data_c |= TRB_DIR_IN | TRB_ISP;
 		ring_queue(s->ep0, XHCI_EP0_TRBS, &s->ep0_enq,
@@ -804,10 +904,17 @@ static int control_transfer(unsigned int ctrl, unsigned int addr,
 	spin_unlock_irqrestore(&xhci_lock, flags);
 
 	t0 = get_ticks_ms();
+	last_bell = t0;
 	while (get_ticks_ms() - t0 < 1000) {
 		spin_lock_irqsave(&xhci_lock, &flags);
 		xhci_poll_locked();
 		rc = ctl_done;
+		if (!rc && get_ticks_ms() - last_bell >= 100) {
+			/* Re-ring while the endpoint is still busy; a posted
+			 * doorbell can be lost on some real xHCs. */
+			mmio_wr(db_off + 4 * s->slot_id, 1);
+			last_bell = get_ticks_ms();
+		}
 		spin_unlock_irqrestore(&xhci_lock, flags);
 		if (rc)
 			break;
@@ -1146,11 +1253,33 @@ static int port_reset(unsigned int ctrl, unsigned int port)
 	if (!(ps & PORTSC_CCS))
 		return -1;
 
+	/* Force the port into the Disabled state first: some xHCs ignore
+	 * PR while the port is already enabled, leaving the device at its
+	 * firmware-assigned address instead of address 0. */
+	if (ps & PORTSC_PED) {
+		mmio_wr(op(XHCI_OP_PORTSC) + port * 0x10,
+			portsc_neutral(ps) | PORTSC_PED);
+		t0 = get_ticks_ms();
+		while (get_ticks_ms() - t0 < 100) {
+			ps = mmio_rd(op(XHCI_OP_PORTSC) + port * 0x10);
+			if (!(ps & PORTSC_PED))
+				break;
+		}
+	}
+
+	/* Acknowledge stale change latches first; a leftover PRC would
+	 * otherwise let the wait below succeed before this reset has even
+	 * started (the firmware may have left one behind). */
 	mmio_wr(op(XHCI_OP_PORTSC) + port * 0x10,
-		(ps & ~PORTSC_RWC) | PORTSC_PR);
+		portsc_neutral(ps) | PORTSC_CSC | PORTSC_PEC |
+		PORTSC_PRC | PORTSC_PLC);
+	ps = mmio_rd(op(XHCI_OP_PORTSC) + port * 0x10);
+
+	mmio_wr(op(XHCI_OP_PORTSC) + port * 0x10,
+		portsc_neutral(ps) | PORTSC_PR);
 
 	t0 = get_ticks_ms();
-	while (get_ticks_ms() - t0 < 200) {
+	while (get_ticks_ms() - t0 < 500) {
 		ps = mmio_rd(op(XHCI_OP_PORTSC) + port * 0x10);
 		if (ps & PORTSC_PRC)
 			break;
@@ -1160,10 +1289,10 @@ static int port_reset(unsigned int ctrl, unsigned int port)
 		return -1;
 	}
 	mmio_wr(op(XHCI_OP_PORTSC) + port * 0x10,
-		(ps & ~PORTSC_RWC) | PORTSC_PRC);
+		portsc_neutral(ps) | PORTSC_PRC);
 
 	t0 = get_ticks_ms();
-	while (get_ticks_ms() - t0 < 100) {
+	while (get_ticks_ms() - t0 < 500) {
 		ps = mmio_rd(op(XHCI_OP_PORTSC) + port * 0x10);
 		if (ps & PORTSC_PED)
 			break;
@@ -1173,6 +1302,11 @@ static int port_reset(unsigned int ctrl, unsigned int port)
 		return -1;
 	}
 
+	/* USB2 reset recovery: the device needs TRSTRCY (10 ms) before it
+	 * answers the first control request.  QEMU replies immediately,
+	 * real devices do not. */
+	delay_ms(10);
+
 	last_port = (int)port;
 	return 0;
 }
@@ -1211,29 +1345,59 @@ static int port_take_change(unsigned int ctrl, unsigned int port)
 	if (!(ps & PORTSC_CSC))
 		return 0;
 	mmio_wr(op(XHCI_OP_PORTSC) + port * 0x10,
-		(ps & ~PORTSC_RWC) | PORTSC_CSC);
+		portsc_neutral(ps) | PORTSC_CSC);
 	return 1;
 }
 
 /* ---- controller discovery ---------------------------------------------- */
-/* xHCI USB Legacy Support: stop the firmware's SMI from firing on port
- * changes once we own the controller (the same idea as the PCI 0xC0 write
- * in uhci.c/ohci.c, but xHCI keeps it in an extended capability). */
-static void legacy_smi_disable(unsigned int off)
+/* xHCI USB Legacy Support: take ownership from the firmware before the
+ * reset and stop its SMI from firing on port changes once we own the
+ * controller (the same idea as the PCI 0xC0 write in uhci.c/ohci.c, but
+ * xHCI keeps it in an extended capability).
+ *
+ * The BIOS-owned semaphore must actually clear before HCRST: while the
+ * firmware still owns the controller its SMI handler can interfere with
+ * the reset and the command ring.  Linux waits for it too. */
+static void legacy_handoff(unsigned int off)
 {
 	unsigned int sup = mmio_rd(off);
-	unsigned int ctl;
+	unsigned int ctl, t0;
 
 	if ((sup & 0xFF) != 1)
 		return;
-	if (sup & (1u << 16))		/* BIOS still owns it */
+	if (sup & (1u << 16)) {		/* BIOS still owns it */
 		mmio_wr(off, sup | (1u << 24));
+		t0 = get_ticks_ms();
+		while (get_ticks_ms() - t0 < 1000) {
+			if (!(mmio_rd(off) & (1u << 16)))
+				break;
+		}
+	}
 	ctl = mmio_rd(off + 4);
 	ctl &= ~((0x7u << 1) | (0xFFu << 5) | (0x7u << 17));
 	ctl |= 0x7u << 29;
 	mmio_wr(off + 4, ctl);
 }
 
+/* Walk the extended capabilities for the legacy handoff.  Runs before the
+ * controller reset; scan_protocols() parses the port list afterwards. */
+static void ext_caps_handoff(void)
+{
+	unsigned int off, guard = 0;
+
+	off = ((hcc_params >> 16) & 0xFFFF) << 2;
+	while (off && guard++ < 32) {
+		unsigned int cap = mmio_rd(off);
+		unsigned int next = (cap >> 8) & 0xFF;
+
+		if ((cap & 0xFF) == 1)
+			legacy_handoff(off);
+		if (!next)
+			break;
+		off += next << 2;
+	}
+}
+
 static void scan_protocols(void)
 {
 	unsigned int off, guard = 0;
@@ -1246,9 +1410,7 @@ static void scan_protocols(void)
 		unsigned int id = cap & 0xFF;
 		unsigned int next = (cap >> 8) & 0xFF;
 
-		if (id == 1) {
-			legacy_smi_disable(off);
-		} else if (id == 2) {	/* Supported Protocol */
+		if (id == 2) {	/* Supported Protocol */
 			unsigned int major = (cap >> 24) & 0xFF;
 			unsigned int pi = mmio_rd(off + 8);
 			unsigned int po = pi & 0xFF;
@@ -1355,9 +1517,35 @@ found_xhci:
 	zero_page(evt_ring, 4096);
 	zero_page(erst, 4096);
 
-	/* Reset.  HCRST self-clears; wait for CNR to drop. */
+	/* Firmware handoff before the reset: while the BIOS still owns the
+	 * controller its SMI handler can interfere with HCRST and the
+	 * command ring. */
+	ext_caps_handoff();
+
+	/* Wait for the controller to become ready before touching USBCMD. */
+	t0 = get_ticks_ms();
+	while (get_ticks_ms() - t0 < 1000) {
+		if (!(mmio_rd(op(XHCI_OP_USBSTS)) & XHCI_STS_CNR))
+			break;
+	}
+
+	/* Reset.  The xHC may ignore HCRST while it is running, so halt it
+	 * and wait for HCH first (xHCI 4.2; Linux does the same).  QEMU
+	 * halts instantly, real silicon does not. */
 	mmio_wr(op(XHCI_OP_USBCMD), 0);
+	t0 = get_ticks_ms();
+	while (get_ticks_ms() - t0 < 100) {
+		if (mmio_rd(op(XHCI_OP_USBSTS)) & XHCI_STS_HCH)
+			break;
+	}
+	if (!(mmio_rd(op(XHCI_OP_USBSTS)) & XHCI_STS_HCH))
+		dbg("xhci: halt timeout\n");
+
 	mmio_wr(op(XHCI_OP_USBCMD), XHCI_CMD_HCRST);
+	/* Intel xHCI (CherryView/Braswell included) needs 1 ms after the
+	 * HCRST write before any other HC register is accessed; polling
+	 * USBCMD too early can leave the controller partially reset. */
+	delay_ms(2);
 	t0 = get_ticks_ms();
 	while (get_ticks_ms() - t0 < 1000) {
 		if (!(mmio_rd(op(XHCI_OP_USBCMD)) & XHCI_CMD_HCRST))
@@ -1374,6 +1562,29 @@ found_xhci:
 		return 0;
 	}
 
+	/* After HCRST the controller must be halted before the ring
+	 * registers are written; otherwise the writes can be dropped and
+	 * every command times out. */
+	if (!(mmio_rd(op(XHCI_OP_USBSTS)) & XHCI_STS_HCH)) {
+		mmio_wr(op(XHCI_OP_USBCMD), 0);
+		t0 = get_ticks_ms();
+		while (get_ticks_ms() - t0 < 100) {
+			if (mmio_rd(op(XHCI_OP_USBSTS)) & XHCI_STS_HCH)
+				break;
+		}
+	}
+
+	/* A working HCRST clears DCBAAP/CRCR/CONFIG.  If firmware values
+	 * survive, the reset did not take effect: the ports keep their old
+	 * device addresses and every control transfer to address 0 fails. */
+	if (mmio_rd(op(XHCI_OP_DCBAAP)) != 0 ||
+	    (mmio_rd(op(XHCI_OP_CRCR)) & ~0xFu) != 0 ||
+	    (mmio_rd(op(XHCI_OP_CONFIG)) & 0xFFu) != 0)
+		dbg("xhci: hcreset not applied\n");
+
+	/* Clear stale status latches from the firmware's session. */
+	mmio_wr(op(XHCI_OP_USBSTS), (1u << 2) | (1u << 3) | (1u << 4));
+
 	mmio_wr(op(XHCI_OP_PAGESIZE), 1);
 	mmio_wr(op(XHCI_OP_CONFIG), max_slots);
 	mmio_wr64(op(XHCI_OP_DCBAAP), dcbaa);
@@ -1432,21 +1643,25 @@ found_xhci:
 		return 0;
 	}
 
+	/* The first command completion (or timeout) tells whether the ring is
+	 * actually serviced.  Don't gate on CRR: QEMU leaves it clear even
+	 * while commands complete normally. */
+
 	scan_protocols();
 	dbg_hex("xhci: usb2 ports ", usb2_mask);
 
-	/* Port power, where the controller has per-port switching. */
-	if (hcc_params & 0x0008u) {
-		for (i = 0; i < max_ports; i++) {
-			unsigned int ps;
-			if (!(usb2_mask & (1u << i)))
-				continue;
-			ps = mmio_rd(op(XHCI_OP_PORTSC) + i * 0x10);
-			if (!(ps & PORTSC_PP))
-				mmio_wr(op(XHCI_OP_PORTSC) + i * 0x10,
-					(ps & ~PORTSC_RWC) | PORTSC_PP);
-		}
+	/* Power every root port.  VBus is supposed to come up after HCRST,
+	 * but some firmware/PPC combinations leave PP clear; the USB3
+	 * entries may share a switch with a USB2 connector, so write all of
+	 * them, not just the USB2 range. */
+	for (i = 0; i < max_ports; i++) {
+		unsigned int ps = mmio_rd(op(XHCI_OP_PORTSC) + i * 0x10);
+
+		if (!(ps & PORTSC_PP))
+			mmio_wr(op(XHCI_OP_PORTSC) + i * 0x10,
+				portsc_neutral(ps) | PORTSC_PP);
 	}
+	delay_ms(20);
 
 	found = 1;
 	return 1;