development.html 36 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448
  1. <!doctype html>
  2. <html lang="en">
  3. <head>
  4. <meta charset="utf-8">
  5. <meta name="viewport" content="width=device-width, initial-scale=1">
  6. <title>Development | NebOS Architecture</title>
  7. <meta name="description" content="NebOS development: toolchain and build flags, image targets, QEMU runs, serial and network hot reload, runtime command-pack updates, tests and debugging tools.">
  8. <link rel="icon" href="favicon.svg" type="image/svg+xml">
  9. <link rel="stylesheet" href="style.css">
  10. </head>
  11. <body>
  12. <a class="skip" href="#main">Skip to content</a>
  13. <header class="site-header">
  14. <a class="brand" href="index.html">
  15. <span class="brand-mark" aria-hidden="true"></span>
  16. <span>NebOS <span class="brand-sub">Architecture</span></span>
  17. </a>
  18. <nav class="site-nav" aria-label="Architecture sections">
  19. <a href="index.html">Overview</a>
  20. <a href="boot.html">Boot</a>
  21. <a href="memory.html">Memory</a>
  22. <a href="cpu.html">CPU</a>
  23. <a href="drivers.html">Drivers</a>
  24. <a href="filesystems.html">Filesystems</a>
  25. <a href="graphics.html">Graphics</a>
  26. <a href="network.html">Network</a>
  27. <a href="userland.html">Userland</a>
  28. <a href="development.html" aria-current="page">Development</a>
  29. </nav>
  30. </header>
  31. <main id="main" class="page">
  32. <div class="page-head">
  33. <p class="kicker">Development</p>
  34. <h1>Edit, reload, watch the boot line.</h1>
  35. <p class="lede">
  36. A normal iteration does not reflash anything. The board receives a
  37. new kernel and rootfs over serial or UDP, parks its APs, jumps into
  38. the new image and prints the same version line on COM1. A runtime
  39. command-pack update can skip even the kernel reload.
  40. </p>
  41. <div class="meta">
  42. <span class="pill">gcc · as · ld · objcopy</span>
  43. <span class="pill">QEMU + real hardware</span>
  44. <span class="pill">KXE2 · KXN2 · KXNP</span>
  45. <span class="pill">headless tests</span>
  46. </div>
  47. </div>
  48. <nav class="toc" aria-label="On this page">
  49. <h2>On this page</h2>
  50. <ul>
  51. <li><a href="#toolchain">Toolchain</a></li>
  52. <li><a href="#tree">Source tree</a></li>
  53. <li><a href="#pipeline">Build pipeline</a></li>
  54. <li><a href="#targets">Make targets</a></li>
  55. <li><a href="#serial">Serial reload</a></li>
  56. <li><a href="#network">Network reload</a></li>
  57. <li><a href="#pack">Runtime pack</a></li>
  58. <li><a href="#tests">Tests</a></li>
  59. <li><a href="#debug">Debugging</a></li>
  60. <li><a href="#conventions">Conventions</a></li>
  61. <li><a href="#docs">Keeping docs current</a></li>
  62. <li><a href="#sources">Source map</a></li>
  63. </ul>
  64. </nav>
  65. <section id="toolchain">
  66. <h2>Toolchain</h2>
  67. <p>
  68. NebOS builds with a normal GNU toolchain for 32-bit x86, plus
  69. <code>mtools</code> for FAT images and QEMU for testing. The UEFI
  70. loader is the only 64-bit component and needs the mingw cross
  71. compiler.
  72. </p>
  73. <ul>
  74. <li><strong>Compiler.</strong> <code>gcc</code> with 32-bit multilib, <code>as</code>, <code>ld</code>, <code>objcopy</code>.</li>
  75. <li><strong>Kernel flags.</strong> <code>-m32 -march=i586 -ffreestanding -fno-pie -nostdlib -fno-stack-protector -mno-sse -mno-sse2 -mno-mmx -O2</code>.</li>
  76. <li><strong>Warnings.</strong> The build uses <code>-Wall -Wextra</code>; a new warning is a bug, not noise.</li>
  77. <li><strong>User commands.</strong> Linked at <code>0x40000000</code> with <code>user/ucmd.ld</code>, stripped, and packed as <code>bin/&lt;name&gt;</code>.</li>
  78. <li><strong>UEFI.</strong> <code>x86_64-w64-mingw32-gcc</code> builds <code>BOOTX64.EFI</code>; <code>make all</code> does not require it.</li>
  79. <li><strong>Images.</strong> <code>mtools</code> builds <code>root.img</code> and the ESP.</li>
  80. <li><strong>WiFi firmware.</strong> The pack includes <code>firmware/iwlwifi-7265D-29.ucode</code> (Chromebook) and <code>firmware/iwlwifi-6000-4.ucode</code> (X201), decompressed with <code>zstd</code> from the host's linux-firmware (<code>WIFI_FW_7265D=</code> / <code>WIFI_FW_6000=/path/to/file[.zst]</code> override the search). A missing file only prints a note and leaves that card without firmware; the floppy images always use the firmware-less <code>cmds-lite.pack</code>.</li>
  81. </ul>
  82. </section>
  83. <section id="tree">
  84. <h2>Source tree</h2>
  85. <p>
  86. One top-level Makefile builds every directory. Intermediate objects
  87. go to <code>build/</code>, mirroring the source paths, with
  88. compiler-generated header dependencies (<code>-MMD</code>); the images
  89. and the files the dev workflow sends or inspects
  90. (<code>kernel.elf</code>, <code>kernel.bin</code>, <code>floppy.img</code>,
  91. <code>usb.img</code>, <code>usb-efi.img</code>, <code>BOOTX64.EFI</code>,
  92. <code>cmds.pack</code>, <code>cmds_pack_gz.bin</code>,
  93. <code>browser.elf</code>, <code>gallery.elf</code>) stay at the top level.
  94. </p>
  95. <div class="table-wrap">
  96. <table>
  97. <thead>
  98. <tr><th>Directory</th><th>Contents</th><th>Include path</th></tr>
  99. </thead>
  100. <tbody>
  101. <tr><td><code>boot/</code></td><td>Boot sector, MBR, VESA stage; <code>boot/efi/</code> UEFI loader</td><td><code>include/</code>, <code>kernel/</code> (EFI: <code>vga.h</code>)</td></tr>
  102. <tr><td><code>kernel/</code></td><td>Kernel, drivers, services, <code>kernel.ld</code>; host tests for <code>ccmp</code> and the touchpad gestures</td><td><code>include/</code>, <code>common/</code>, <code>kernel/</code></td></tr>
  103. <tr><td><code>common/</code></td><td>Freestanding code linked into both kernel and userland: <code>inflate.c</code>, <code>font_8x16.c</code></td><td>itself</td></tr>
  104. <tr><td><code>include/</code></td><td>Shared ABI headers: <code>syscall.h</code>, <code>wm_abi.h</code>, <code>bootinfo.h</code>, <code>phys.h</code>, <code>rootfs.h</code>, <code>version.h</code>, <code>wifi_cfg.h</code>, <code>rtl8821a_tbl.h</code></td><td>—</td></tr>
  105. <tr><td><code>user/</code></td><td><code>ucmd_lib.h</code>, <code>ucmd.ld</code></td><td><code>include/</code>, <code>common/</code>, <code>user/</code>, every <code>user/lib/*</code>; libc headers by default</td></tr>
  106. <tr><td><code>user/apps/&lt;name&gt;/</code></td><td>One ring-3 program each, entry file <code>ucmd_&lt;name&gt;.c</code> plus its private modules (the browser, chat, gallery)</td><td>as <code>user/</code></td></tr>
  107. <tr><td><code>user/lib/&lt;lib&gt;/</code></td><td><code>libc</code>, <code>ngui</code>, <code>ufont</code>, <code>img</code>, <code>tls</code> (crypto + TLS), <code>uhttp</code>, <code>ujson</code>, <code>uwm</code>, <code>wifi</code> (saved networks, WPA2), each with its host tests</td><td>as <code>user/</code></td></tr>
  108. <tr><td><code>tools/</code></td><td>Host tools (<code>mkpack</code>, <code>mkrootfs</code>, <code>mkrtwtbl</code>, <code>kexec_send</code>, <code>netkexec_send</code>, <code>netmonitor</code>, <code>flash.sh</code>) and the QMP test scripts</td><td><code>include/</code></td></tr>
  109. <tr><td><code>host/</code></td><td>Linux SDL3 browser harness</td><td>as <code>user/</code></td></tr>
  110. </tbody>
  111. </table>
  112. </div>
  113. <p>
  114. The kernel never includes a userland header. The one userland file
  115. that reaches into the kernel is <code>user/apps/term/ucmd_term.c</code>
  116. (<code>IO_KEY_*</code> and <code>VFS_PATH_MAX</code>), which gets
  117. <code>-Ikernel</code> explicitly. A new command is a directory
  118. <code>user/apps/&lt;name&gt;/</code> with <code>ucmd_&lt;name&gt;.c</code>
  119. and an entry in <code>UCMDS</code>; one that links libraries gets an
  120. explicit <code>.elf</code> rule in the Makefile. Per-object flags
  121. (<code>-Os</code>, function sections, no libc headers) are
  122. target-specific variables on the <code>.o</code>, and make does not
  123. track them: after changing one, delete the object or run
  124. <code>make clean</code>.
  125. </p>
  126. </section>
  127. <section id="pipeline">
  128. <h2>Build pipeline</h2>
  129. <figure class="figure">
  130. <svg viewBox="0 0 920 360" role="img" aria-labelledby="build-title build-desc">
  131. <title id="build-title">NebOS build pipeline</title>
  132. <desc id="build-desc">Kernel sources become kernel.bin and boot.bin; ring-3 sources become cmds.pack and a gzip rootfs; the UEFI loader embeds kernel, pack and trampoline. The outputs feed floppy, USB and dual-boot images.</desc>
  133. <rect class="svg-box blue" x="20" y="20" width="250" height="64" rx="9"/>
  134. <text class="svg-text" x="40" y="46">kernel sources</text>
  135. <text class="svg-sub" x="40" y="66">boot/ · kernel/ · common/</text>
  136. <rect class="svg-box blue" x="20" y="116" width="250" height="64" rx="9"/>
  137. <text class="svg-text" x="40" y="142">ring-3 sources</text>
  138. <text class="svg-sub" x="40" y="162">user/apps/* · user/lib/*</text>
  139. <rect class="svg-box blue" x="20" y="212" width="250" height="64" rx="9"/>
  140. <text class="svg-text" x="40" y="238">UEFI loader</text>
  141. <text class="svg-sub" x="40" y="258">boot/efi/*.c · tramp.S</text>
  142. <path class="svg-line blue" d="M270 52h62"/>
  143. <path class="svg-line blue" d="M326 46l7 6-7 6"/>
  144. <path class="svg-line blue" d="M270 148h62"/>
  145. <path class="svg-line blue" d="M326 142l7 6-7 6"/>
  146. <path class="svg-line blue" d="M270 244h62"/>
  147. <path class="svg-line blue" d="M326 238l7 6-7 6"/>
  148. <rect class="svg-box accent" x="338" y="20" width="230" height="64" rx="9"/>
  149. <text class="svg-text" x="358" y="46">gcc · as · ld · objcopy</text>
  150. <text class="svg-sub" x="358" y="66">kernel.elf → kernel.bin</text>
  151. <rect class="svg-box accent" x="338" y="116" width="230" height="64" rx="9"/>
  152. <text class="svg-text" x="358" y="142">mkpack · gzip · mkrootfs</text>
  153. <text class="svg-sub" x="358" y="162">cmds.pack → NBROOT1</text>
  154. <rect class="svg-box accent" x="338" y="212" width="230" height="64" rx="9"/>
  155. <text class="svg-text" x="358" y="238">mingw gcc</text>
  156. <text class="svg-sub" x="358" y="258">BOOTX64.EFI</text>
  157. <path class="svg-line accent" d="M568 52h62"/>
  158. <path class="svg-line accent" d="M624 46l7 6-7 6"/>
  159. <path class="svg-line accent" d="M568 148h62"/>
  160. <path class="svg-line accent" d="M624 142l7 6-7 6"/>
  161. <path class="svg-line accent" d="M568 244h62"/>
  162. <path class="svg-line accent" d="M624 238l7 6-7 6"/>
  163. <rect class="svg-box green" x="636" y="20" width="264" height="64" rx="9"/>
  164. <text class="svg-text" x="656" y="46">floppy.img · final.bin</text>
  165. <text class="svg-sub" x="656" y="66">boot sector + kernel + rootfs</text>
  166. <rect class="svg-box green" x="636" y="116" width="264" height="64" rx="9"/>
  167. <text class="svg-text" x="656" y="142">usb.img</text>
  168. <text class="svg-sub" x="656" y="162">MBR + boot + 8 MB rootfs</text>
  169. <rect class="svg-box green" x="636" y="212" width="264" height="64" rx="9"/>
  170. <text class="svg-text" x="656" y="238">usb-efi.img · 17 MB</text>
  171. <text class="svg-sub" x="656" y="258">usb.img + 8 MB FAT16 ESP</text>
  172. <line class="svg-line" x1="20" y1="310" x2="900" y2="310"/>
  173. <text class="svg-sub" x="20" y="335">Hot reload skips the image step: make dev / net-dev sends kernel.bin + cmds_pack_gz.bin; cmds-dev / net-cmds sends cmds.pack.</text>
  174. </svg>
  175. <figcaption>
  176. <strong>Two independent products.</strong> The kernel image and
  177. the rootfs container have separate lifetimes: a kernel reload can
  178. keep the running rootfs, and a pack update can keep the kernel.
  179. </figcaption>
  180. </figure>
  181. </section>
  182. <section id="targets">
  183. <h2>Make targets</h2>
  184. <div class="table-wrap">
  185. <table>
  186. <thead>
  187. <tr><th>Target</th><th>What it does</th></tr>
  188. </thead>
  189. <tbody>
  190. <tr><td><code>make all</code></td><td>Build <code>floppy.img</code>, <code>usb.img</code>, <code>final.bin</code> and <code>root.img</code>.</td></tr>
  191. <tr><td><code>make run</code> / <code>run-smp</code></td><td>Boot the floppy in QEMU, optionally with four CPUs (Atom 330 topology).</td></tr>
  192. <tr><td><code>make run-usb</code></td><td>Boot <code>usb.img</code> as an IDE disk.</td></tr>
  193. <tr><td><code>make run-uhci</code> / <code>run-ehci</code> / <code>run-xhci</code></td><td>Exercise the USB backends with a mouse/keyboard; the EHCI target also attaches a USB mass-storage disk, and the installer test covers EHCI and xHCI storage.</td></tr>
  194. <tr><td><code>make run-ehci-asix</code> / <code>run-xhci-asix</code></td><td>Pass through a real ASIX AX88772 dongle.</td></tr>
  195. <tr><td><code>make run-net</code></td><td>Add a PCI NE2000 on QEMU's slirp network with host port 8080 forwarded.</td></tr>
  196. <tr><td><code>make run-rndis</code></td><td>Boot with <code>qemu-xhci</code> and QEMU's full-speed <code>usb-net</code> RNDIS device on slirp; the tethering link gets DHCP from QEMU.</td></tr>
  197. <tr><td><code>make run-ahci</code></td><td>Add an ICH9 AHCI controller and a 512 MB SATA disk.</td></tr>
  198. <tr><td><code>make run-amd</code> / <code>test-amd</code></td><td>Boot QEMU with <code>-cpu Opteron_G5 -smp 6</code> (the FX-6350's family 15h model 2), OHCI, six CPUs and an e1000; the headless test also checks <code>ioapic: masked</code>, <code>smp cpus: 6</code> and the PIT.</td></tr>
  199. <tr><td><code>make uefi</code> / <code>run-uefi</code> / <code>test-uefi</code></td><td>Build the dual-boot image, boot it under OVMF, or run the headless serial check.</td></tr>
  200. <tr><td><code>make rootfs</code></td><td>Rebuild <code>cmds.img</code> only.</td></tr>
  201. <tr><td><code>make nowm</code></td><td>Diagnostic kernel with the WM disabled; the boot console stays visible.</td></tr>
  202. <tr><td><code>make dev</code> / <code>kernel-dev</code></td><td>Serial reload of kernel + rootfs, or kernel only (keep the running rootfs).</td></tr>
  203. <tr><td><code>make net-dev</code> / <code>kernel-net-dev</code></td><td>RAM-only network reload of kernel + rootfs, or kernel only.</td></tr>
  204. <tr><td><code>make net-install</code></td><td>Persist kernel + full rootfs to P1/P2 over UDP, keeping <code>/data</code>; <code>NETINSTALL_TARGET=auto</code> selects the internal NebOS disk.</td></tr>
  205. <tr><td><code>make cmds-dev</code> / <code>net-cmds</code></td><td>Replace <code>/ramfs</code> at runtime over serial or UDP.</td></tr>
  206. <tr><td><code>make monitor</code> / <code>netmon</code></td><td>Open the serial console, or listen to the UDP console mirror.</td></tr>
  207. <tr><td><code>make flash</code></td><td>Write the dual-boot image to a USB stick (needs mingw and prompts for the device).</td></tr>
  208. <tr><td><code>make test-usb</code></td><td>Headless QEMU mouse/keyboard regression tests using QMP and screen dumps.</td></tr>
  209. <tr><td><code>make test-install</code></td><td>Boots the dual image with a USB mass-storage source and a blank target, types <code>install ide1</code>, verifies the clone and boots the target in BIOS and UEFI.</td></tr>
  210. <tr><td><code>make test-images</code> / <code>test-images-asan</code></td><td>Host image-decoder and blit tests, including reduced-IDCT JPEG decode against a box-filtered reference; AddressSanitizer variant included.</td></tr>
  211. <tr><td><code>make test-crypto</code></td><td>Host vectors for SHA-256, ChaCha20-Poly1305, X25519, AES-128-GCM, P-256, the TLS 1.2 PRF and TLS helpers.</td></tr>
  212. <tr><td><code>make test-wpa</code></td><td>Host vectors for the WPA2 supplicant: SHA-1, RFC 6070 PBKDF2, the 802.11 PSK and PRF examples, FIPS-197 AES, RFC 3394 unwrap, plus a simulated 4-way handshake, group rekey, replay and wrong-passphrase case.</td></tr>
  213. <tr><td><code>make test-wifi-store</code></td><td>Host test for the saved-network file: newest-first ordering, passphrase update, LRU eviction at the 16-entry cap, per-SSID forget, the old one-record format and malformed or duplicate lists.</td></tr>
  214. <tr><td><code>make test-libc</code></td><td>Host tests for the ring-3 malloc, stdio/float formatting, math, 64-bit division helpers and setjmp/longjmp.</td></tr>
  215. <tr><td><code>make test-js</code></td><td>Host tests (ASan/UBSan) for the vendored QuickJS engine through the <code>js_mu</code> stack API: modern syntax (classes, arrows, templates, destructuring, spread, async/await, BigInt, named regex groups), Promise jobs, C functions and <code>new</code>, thrown TypeErrors, userdata hooks and expandos, registry refs and the wall-clock interrupt.</td></tr>
  216. <tr><td><code>make test-font</code> / <code>test-font-asan</code></td><td>Host specimen test for the fixed-point TrueType parser/rasterizer and glyph cache, with an AddressSanitizer/UBSan variant.</td></tr>
  217. <tr><td><code>make test-svg</code></td><td>Host tests (ASan/UBSan) for the inline-SVG rasterizer: exact rect coverage, antialiased edges, relative commands, nonzero vs even-odd, circle area, curves and arcs, transforms, number grammar and strokes; <code>./svg_test out.ppm</code> writes a specimen.</td></tr>
  218. <tr><td><code>make test-cursors</code></td><td>Host test (ASan/UBSan) for the NBCUR1 cursor bundle built by <code>mkcursors</code>: header and shape count, sizes and hotspots against the Adwaita files, premultiplied pixels, no empty frame, and the 15-frame <code>wait</code>/<code>progress</code> spinners.</td></tr>
  219. <tr><td><code>make test-touchpad</code></td><td>Host tests for the device-independent contact gesture engine: taps, right-click zones, physical-button latch, two-finger scroll, direction and pointer re-anchoring.</td></tr>
  220. <tr><td><code>make test-chat</code></td><td>Host tests (ASan/UBSan) for the chat client's JSON parser, cookie jar and the OIDC login page/reply/config parsers.</td></tr>
  221. <tr><td><code>make test-chat-e2e</code></td><td>Boots QEMU with an NE2000 on slirp, starts <code>tools/mock_chat.py</code> and drives <code>chat</code> through QMP: native login, send, live poll, read marker, older paging, browse/join, new DM, saved-session resume and sign-out. <code>CHAT_E2E=--totp|--consent|--passkey|--wrong</code> runs a login variant.</td></tr>
  222. <tr><td><code>make compat-serve</code></td><td>Serve the browser compatibility ladder from <code>testdata/browser/compat/</code> on port 8000 for the board or QEMU guest.</td></tr>
  223. <tr><td><code>make browser-host</code></td><td>Build the Linux SDL3 development harness (<code>browser-host</code> and its <code>host-curl</code> helper). It implements the ring-3 <code>SYS_*</code> ABI with host files, BSD sockets, <code>fork/exec</code> and an SDL3 window, so pages render and accept input without QEMU. Run <code>./browser-host [--size WxH] [--root DIR] URL|FILE</code>; <code>--headless --wait 3000 --shot out.ppm</code> writes the canvas without a display, and <code>--keys 'clo{down}{enter}'</code> types into it (one key per 100 ms; <code>{enter}</code>, <code>{esc}</code>, <code>{tab}</code>, <code>{bs}</code>, <code>{del}</code>, arrows, <code>{ctrl-l}</code>, <code>{wait}</code>, and <code>{click:X,Y}</code> for a left click at canvas coordinates). Needs the 32-bit SDL3/Wayland libraries (<code>lib32-sdl3</code>, <code>lib32-wayland</code>, <code>lib32-libxkbcommon</code>).</td></tr>
  224. <tr><td><code>make fonts</code></td><td>Regenerate the checked-in Arimo/Liberation Mono subsets with <code>pyftsubset</code>.</td></tr>
  225. </tbody>
  226. </table>
  227. </div>
  228. <p>
  229. QEMU guests default to 1 GB, the lowmem size (<code>MM_MAX_PHYS</code> in
  230. <span class="file">phys.h</span>). Override the size with
  231. <code>QEMU_MEM</code>, for example <code>make run QEMU_MEM=3G</code> to
  232. exercise highmem;
  233. all run and UEFI targets pass it through.
  234. </p>
  235. </section>
  236. <section id="serial">
  237. <h2>Serial reload</h2>
  238. <p>
  239. COM1 runs at 115200 8N1. The WM idle loop watches the receive ring for
  240. the <code>KXE2</code> magic and auto-spawns the reload, so a plain
  241. <code>make dev</code> is enough. The manual <code>kexec</code> command
  242. is a fallback that waits for the same image.
  243. </p>
  244. <pre><code>[magic "KXE2"][entry u32][kernel_size u32][pack_size u32]
  245. [kernel][rootfs gzip][checksum u8]</code></pre>
  246. <ul>
  247. <li><strong>Full reload.</strong> <code>make dev</code> stages kernel and rootfs in one page-allocated buffer, copies the kernel to <code>0x7E00</code> and publishes the pack range in boot info.</li>
  248. <li><strong>Kernel only.</strong> <code>make kernel-dev</code> sends <code>pack_size == 0</code>; the new kernel keeps the running rootfs.</li>
  249. <li><strong>Pack only.</strong> <code>make cmds-dev</code> uses <code>KXNP</code> and <code>ramfs_replace()</code>, without a kernel jump.</li>
  250. <li><strong>Monitor.</strong> Run <code>make monitor</code> in a second terminal; each reload prints <code>NebOS v0.2.0 booted</code> on COM1.</li>
  251. </ul>
  252. <p>
  253. The serial receive drops the BKL while it waits for bytes, so the WM
  254. and the other CPU's syscalls stay responsive during a reload.
  255. </p>
  256. </section>
  257. <section id="network">
  258. <h2>Network reload</h2>
  259. <p>
  260. For boards without a serial port, the same reload runs over UDP port
  261. 19288 with the <code>KXN2</code> magic. The host tool sends the
  262. kernel and rootfs as one stop-and-wait stream; the board acknowledges
  263. every datagram with the next expected offset and verifies a 32-bit
  264. checksum before jumping.
  265. </p>
  266. <p>
  267. <code>make net-install</code> uses <code>KXI2</code> instead. It
  268. sends the kernel plus a full <code>NBROOT1</code> rootfs image
  269. (<code>rootfs-full.bin</code>) and a 16-byte target name; the board
  270. writes P2, the P1 kernel, patches the exact
  271. <code>kernel_sectors</code>, verifies the result, acks
  272. <code>ACK_INSTALLED</code> and jumps. It never touches P4
  273. (<code>/data</code>) or the MBR, so it is the persistent counterpart
  274. to the RAM-only <code>net-dev</code>. <code>NETINSTALL_TARGET=auto</code>
  275. selects the only non-USB NebOS disk.
  276. </p>
  277. <pre><code>make net-dev # RAM-only kernel + rootfs to 192.168.178.240
  278. make net-dev BOARD_IP=192.168.178.42
  279. make kernel-net-dev # kernel only; keep the rootfs
  280. make net-install # persist kernel + full rootfs, keep /data
  281. make net-install NETINSTALL_TARGET=sata0
  282. make net-cmds # replace /ramfs only</code></pre>
  283. <p>
  284. The board must be idle at a shell prompt. The WM loop answers
  285. ARP/ICMP while idle, detects the <code>KXN2</code>/<code>KXNP</code>
  286. header and pauses the auto-detect while a packet-level command owns
  287. the NIC or a transfer is in progress. During the transfer the
  288. receive loop keeps answering ARP/ICMP itself, so the host's
  289. neighbour entry cannot expire mid-upload. Datagrams are at most
  290. 1400 bytes.
  291. </p>
  292. <p>
  293. The board address comes from DHCP when a server is present; use
  294. <code>make net-dev BOARD_IP=&lt;leased-address&gt;</code>, or pin an
  295. address with <code>netip</code> on the board.
  296. </p>
  297. </section>
  298. <section id="pack">
  299. <h2>Runtime command pack</h2>
  300. <p>
  301. <code>make cmds-dev</code> sends the raw <code>cmds.pack</code> with
  302. the <code>KXNP</code> magic. The receiver validates the pack before
  303. the swap, so a truncated transfer leaves the running commands
  304. untouched. A good pack is installed by <code>ramfs_replace()</code>
  305. and the old heap copy is freed.
  306. </p>
  307. <ul>
  308. <li><strong>Existing processes are unaffected.</strong> They already loaded their ELF.</li>
  309. <li><strong>The next spawn sees the new files.</strong></li>
  310. <li><strong>RAM only.</strong> A cold boot uses the flashed rootfs; reflash after changing the bundle.</li>
  311. <li><strong>Size.</strong> The raw pack is about 2 MB with the WiFi firmware, so <code>cmds-dev</code> over 115200 serial takes roughly three minutes; prefer <code>net-cmds</code>.</li>
  312. </ul>
  313. </section>
  314. <section id="tests">
  315. <h2>Tests</h2>
  316. <ul>
  317. <li><strong>USB regression.</strong> <code>make test-usb</code> runs QMP cursor and keyboard tests against UHCI, OHCI, EHCI and xHCI, including hotplug.</li>
  318. <li><strong>Installer.</strong> <code>make test-install</code> boots the dual image with its USB mass-storage source and a blank target, types <code>install ide1</code> through QMP, verifies the clone and boots the target in BIOS and (under OVMF) UEFI.</li>
  319. <li><strong>Image library.</strong> <code>make test-images</code> decodes a corpus of PNG/JPEG/GIF/BMP/TGA/PNM/ICO/PAM files and checks blits; the ASan variant catches memory errors.</li>
  320. <li><strong>Crypto.</strong> <code>make test-crypto</code> runs host vectors for the freestanding crypto code.</li>
  321. <li><strong>WPA2.</strong> <code>make test-wpa</code> runs <span class="file">wpa_test.c</span> against <span class="file">wpa_crypto.c</span> and <span class="file">wpa.c</span>. <code>make test-wifi-store</code> runs the saved-network list against an in-memory FAT file: ordering, update, eviction, forget and the legacy/malformed formats. The WiFi driver itself has no QEMU model: test it on the Chromebook over <code>make net-dev</code> with the RNDIS phone or ASIX dongle as the management link and <code>make netmon</code> for the <code>iwl:</code>/<code>wifi:</code> trail.</li>
  322. <li><strong>Ring-3 libc.</strong> <code>make test-libc</code> runs the malloc stress test (64 MB simulated arena with poisoned pages), exact <code>snprintf</code> float/string checks against the host, x87 math values, the 64-bit division helpers and a 32-bit setjmp/longjmp test.</li>
  323. <li><strong>Fonts.</strong> <code>make test-font</code> parses the checked-in Arimo subset on the host, measures Latin-1 text, renders composites (for example <code>Äé</code>) and writes a specimen PPM. The browser must also be exercised across a navigation to verify detach/reattach around <code>heap_reset()</code>.</li>
  324. <li><strong>Touchpad gestures.</strong> <code>make test-touchpad</code> feeds synthetic contact frames to <span class="file">touchpad_gesture.c</span> and checks one/two-finger taps, the corner and physical-button mapping, scroll start/direction, pointer freeze and re-anchoring, and the option switches.</li>
  325. <li><strong>Chat client.</strong> <code>make test-chat</code> covers the pure modules on the host. <code>tools/mock_chat.py</code> is a two-origin mock of Cloudron Chat (port 18080) and the Cloudron OIDC login (18081) with <code>test</code>, <code>totp</code> (code 123456), <code>passkey</code> and <code>consent</code> accounts (password <code>test</code>) and <code>/debug/state</code>, <code>/debug/inject</code> endpoints; run it alone and enter <code>http://10.0.2.2:18080</code> in a QEMU guest's chat, or let <code>make test-chat-e2e</code> drive it. Screenshots land in <code>chat-test-out/</code>.</li>
  326. <li><strong>Browser compatibility.</strong> <code>make compat-serve</code> serves the numbered A/B ladder in <code>testdata/browser/compat/</code>. Open the index on the board (or at <code>http://10.0.2.2:8000/</code> from a QEMU guest with user networking) and compare each pair; the index table records the current status.</li>
  327. <li><strong>Host browser harness.</strong> <code>make browser-host</code> builds the real browser, ngui, image/TLS code and QuickJS as a 32-bit Linux program against the same <code>SYS_*</code> ABI. <code>./browser-host testdata/browser/compat/010-block-width.html</code> opens an SDL3 window (native Wayland on this desktop); the address bar, wheel, links and keys use the normal browser paths. <code>./browser-host --headless --wait 3000 --shot /tmp/opencode/page.ppm URL</code> renders without a display for scripts and screenshots. The harness uses host sockets rather than the NebOS TCP stack, and an <code>mmap</code> arena with the ring-3 allocator, so it tests browser/HTML/CSS/JS/image/font/TLS logic rather than drivers.</li>
  328. <li><strong>UEFI.</strong> <code>make test-uefi</code> boots the dual image under OVMF headless and greps the serial log for the boot line. <code>make test-amd</code> does the same with QEMU's Piledriver CPU model and checks the IOAPIC mask and six-core ACPI SMP.</li>
  329. <li><strong>Compositor stats.</strong> The ring-3 WM overwrites <code>/share/wmstats</code> every two seconds with <code>frames</code>, total and worst compositing time in milliseconds, and the pixels drawn in the interval. Read it in a terminal or fetch it over WebDAV to measure a target.</li>
  330. </ul>
  331. </section>
  332. <section id="debug">
  333. <h2>Debugging</h2>
  334. <ul>
  335. <li><strong>Serial log.</strong> <code>make monitor</code> shows the boot trace and printf-style driver output. The <code>NebOS v… booted</code> line is written before the framebuffer is ready.</li>
  336. <li><strong>Boot time.</strong> Timestamp COM1 on the host (QEMU <code>-serial stdio</code>): the kernel starts at <code>NebOS v… booted</code>, the desktop at <code>wm: first frame</code>, and the terminal is on screen at <code>wm: first window frame (term) at N ms</code> (kernel uptime). On QEMU the USB image reaches it about 0.4 s after the kernel starts; USB, the NIC and the disks are probed after that (<code>[boot] usb/net/storage</code> lines).</li>
  337. <li><strong>No serial port.</strong> The boot console is the early log. After the desktop, <code>bootlog</code> prints the captured text and <code>bootlog save /usb0p3/BOOTLOG.TXT</code> writes it to the live stick's FAT ESP (mount that partition on the host to read it). Once DHCP works, <code>make netmon</code> mirrors serial and terminal output to UDP 19290, so it becomes the serial console.</li>
  338. <li><strong>Network console.</strong> <code>make netmon</code> listens to UDP 19290 for serial and terminal output; <code>netconsole host &lt;ip&gt;</code> narrows the target.</li>
  339. <li><strong>dbg.</strong> The ring-0 serial monitor accepts <code>r addr len</code>, <code>w addr b0 …</code>, <code>i port</code>, <code>o port val</code>, <code>h</code> and <code>q</code>.</li>
  340. <li><strong>Keys.</strong> <code>keytest</code> drains the raw scancode tap in <span class="file">io.c</span> and prints every byte with its E0-prefix state, make/break kind and decoded <code>IO_KEY_*</code>/ASCII value; Ctrl+C stops it. It sees releases and keys the decoder ignores, so an unknown action-row key is easy to identify.</li>
  341. <li><strong>Panic.</strong> A ring-0 fault dumps vector, error code, registers, CRs, EIP bytes and stack candidates to the LFB and COM1. Resolve addresses with <code>nm kernel.elf</code> / <code>addr2line -e kernel.elf</code>.</li>
  342. <li><strong>Profiling.</strong> <code>gfxbench</code> times fill/copy/cell/canvas/flush; <code>top</code>, <code>sched</code>, <code>sensors</code>, <code>powerinfo</code>, <code>power</code> and <code>lspci</code>/<code>lsusb</code> expose live state.</li>
  343. <li><strong>Memory.</strong> <code>peek addr len</code> hex-dumps physical memory, MMIO registers too (highmem through <code>kmap</code>, MMIO through its <code>ioremap()</code> mapping, created on demand); <code>poke addr val</code> writes one dword and prints the read-back.</li>
  344. </ul>
  345. </section>
  346. <section id="conventions">
  347. <h2>Conventions &amp; gotchas</h2>
  348. <ul>
  349. <li><strong>Freestanding C.</strong> The kernel has no libc, <code>malloc</code> or <code>printf</code>; use <code>io_putstr</code>, <code>io_putchar</code> and <code>io_puthex</code>. Ring-3 commands may link <code>libc.a</code> from <span class="file">user/lib/libc/</span>; ring-3 objects see its headers (<code>-Iuser/lib/libc/include</code>) by default.</li>
  350. <li><strong>Fresh memory is not zeroed.</strong> Clear structures whose fields are read before being written. QEMU starts RAM clear and hides the bug.</li>
  351. <li><strong>Keep the memory map in sync.</strong> Hardcoded addresses appear in assembly, C headers and linker scripts; changing one without the others breaks boot or loading.</li>
  352. <li><strong>Ring-3 data segments.</strong> An IRET to ring 3 nulls DS/ES/FS/GS whose DPL is below CPL 3. Kernel stubs must restore selector <code>0x23</code> with <code>IRQ_RETURN_DS</code>. QEMU does not model this; only real hardware fails.</li>
  353. <li><strong>Paging is flat.</strong> Physical equals virtual everywhere; there is no process isolation.</li>
  354. <li><strong>The BKL is not preemptible.</strong> A BKL holder can be in the middle of kernel structures; block points drop it, and <code>bkl_pause()</code> is for long waits.</li>
  355. <li><strong>Kernel size wall.</strong> <code>_end</code> must stay below <code>0x90000</code>; the legacy staging gap also constrains the loaded image and the compressed pack.</li>
  356. </ul>
  357. </section>
  358. <section id="docs">
  359. <h2>Keeping docs current</h2>
  360. <p>
  361. This site is static HTML with one shared stylesheet, no JavaScript and
  362. no build step. It is part of the tree, not generated output. When a
  363. subsystem changes, update the matching page and the overview card in
  364. the same change:
  365. </p>
  366. <div class="table-wrap">
  367. <table>
  368. <thead>
  369. <tr><th>Change</th><th>Update</th></tr>
  370. </thead>
  371. <tbody>
  372. <tr><td>Boot path, boot info, image layout</td><td><a href="boot.html">boot.html</a></td></tr>
  373. <tr><td>Address map, allocator, paging, user arenas</td><td><a href="memory.html">memory.html</a></td></tr>
  374. <tr><td>Scheduler, interrupts, SMP, BKL, power</td><td><a href="cpu.html">cpu.html</a></td></tr>
  375. <tr><td>PCI, storage, USB, input, sensors</td><td><a href="drivers.html">drivers.html</a></td></tr>
  376. <tr><td>VFS, FAT, ramfs, /share, WebDAV</td><td><a href="filesystems.html">filesystems.html</a></td></tr>
  377. <tr><td>Framebuffer, WM, canvas, ngui, images</td><td><a href="graphics.html">graphics.html</a></td></tr>
  378. <tr><td>NIC, protocols, TLS, netconsole</td><td><a href="network.html">network.html</a></td></tr>
  379. <tr><td>Syscalls, shell, pack, apps</td><td><a href="userland.html">userland.html</a></td></tr>
  380. <tr><td>Build targets, reload, tests, debugging</td><td><a href="development.html">development.html</a></td></tr>
  381. </tbody>
  382. </table>
  383. </div>
  384. <p>
  385. <code>AGENTS.md</code> repeats the rule and lists the same page map.
  386. Numbers in charts must come from a source constant or table; if a
  387. constant moves, update the chart caption and the source file that
  388. defines it.
  389. </p>
  390. </section>
  391. <section id="sources">
  392. <h2>Source map</h2>
  393. <ul class="filelist">
  394. <li><code>Makefile</code> — targets and image rules</li>
  395. <li><code>user/lib/libc/</code>, <code>libc.a</code> — optional ring-3 libc and its host test</li>
  396. <li><code>third_party/quickjs/</code> (see its <code>VENDOR</code>), <code>user/apps/browser/js_mu.c/h</code>, <code>user/apps/browser/js_browser.c/h</code> — vendored JavaScript engine, its MuJS-style stack API and the browser bindings; <code>user/apps/browser/js_test.c</code> host tests</li>
  397. <li><code>host/</code> — Linux SDL3 browser harness: <code>host/syscall.c</code> implements the <code>SYS_*</code> ABI, <code>host/window_sdl3.c</code> the Wayland window/input, <code>host/fs.c</code>/<code>host/net.c</code>/<code>host/proc.c</code> the host file, socket and <code>curl __serve</code> services, and <code>host/window_headless.c</code> the screenshot mode; <span class="file">ucmd_lib.h</span> selects the host prototypes with <code>UCMD_HOST</code></li>
  398. <li><code>user/lib/ufont/ufont.c</code>, <code>user/lib/ufont/ufont_ttf.c</code>, <code>fonts/</code> — UI font engine and subsets; <code>user/lib/ufont/ufont_test.c</code> host test</li>
  399. <li><code>kernel/kexec.c</code>, <code>tools/kexec_send.c</code> — serial reload</li>
  400. <li><code>kernel/netkexec.c</code>, <code>tools/netkexec_send.c</code> — network reload</li>
  401. <li><code>kernel/netlog.c</code>, <code>tools/netmonitor.c</code> — UDP console</li>
  402. <li><code>kernel/dbg.c</code> — serial debug monitor</li>
  403. <li><code>kernel/io.c</code>, <code>kernel/terminal.c</code> — scancode decoder/tap and the kernel command registry</li>
  404. <li><code>kernel/panic.c</code> — fatal exception dump</li>
  405. <li><code>kernel/kernel.ld</code>, <code>user/ucmd.ld</code> — link limits</li>
  406. <li><code>tools/</code> — QMP USB and installer tests</li>
  407. <li><code>user/lib/wifi/wpa_test.c</code> — WPA2 supplicant host test (<code>make test-wpa</code>)</li>
  408. <li><code>user/lib/wifi/wifi_store_test.c</code> — saved WiFi list host test (<code>make test-wifi-store</code>)</li>
  409. <li><code>AGENTS.md</code> — developer notes and maintenance rule</li>
  410. </ul>
  411. </section>
  412. </main>
  413. <footer class="site-footer">
  414. <div class="inner">
  415. <p><strong>NebOS 0.2.0</strong> · Architecture documentation · <a href="index.html">overview</a></p>
  416. <p>Maintain alongside <code>AGENTS.md</code> and the source files listed on each page.</p>
  417. </div>
  418. </footer>
  419. </body>
  420. </html>