ci: optional selfhost-loop workflow + determinism script

Add tools/selfhost_loop.sh with path-normalized C and stripped ELF checks (bootstrap ×2). Wire GitHub Actions (manual/weekly/main paths) outside default make test. Raise selfhost lexer maxTokens for large files.
2026-07-19 23:43:51 +03:00
parent c82757d51c
commit fe1da8b42e
7 changed files with 256 additions and 40 deletions
@@ -0,0 +1,67 @@
+# Optional / not on every PR: bootstrap selfhost determinism (make selfhost-loop).
+# Triggers: manual, weekly, or push to main that touches compiler/stdlib/loop.
+# Fixed-point buxc2→buxc3 is opt-in via BUX_SELFHOST_FIXED_POINT=1 (experimental).
+name: selfhost-loop
+
+on:
+  workflow_dispatch:
+    inputs:
+      fixed_point:
+        description: "Also run experimental buxc2→buxc3 fixed-point"
+        type: boolean
+        default: false
+  schedule:
+    # Sundays 06:00 UTC
+    - cron: "0 6 * * 0"
+  push:
+    branches: [main]
+    paths:
+      - "src/**"
+      - "lib/**"
+      - "rt/**"
+      - "bootstrap/**"
+      - "tools/selfhost_loop.sh"
+      - "Makefile"
+      - ".github/workflows/selfhost-loop.yml"
+
+concurrency:
+  group: selfhost-loop-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  selfhost-loop:
+    name: bootstrap determinism
+    runs-on: ubuntu-latest
+    timeout-minutes: 45
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Install Nim
+        uses: jiro4989/setup-nim-action@v2
+        with:
+          nim-version: "2.0.x"
+          repo-token: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Install build deps
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y --no-install-recommends gcc make binutils libssl-dev
+
+      - name: Selfhost loop
+        run: |
+          if [ "${{ github.event_name }}" = "workflow_dispatch" ] && [ "${{ inputs.fixed_point }}" = "true" ]; then
+            export BUX_SELFHOST_FIXED_POINT=1
+            echo "Fixed-point mode enabled (experimental)"
+          fi
+          make selfhost-loop
+
+      - name: Upload C on failure
+        if: failure()
+        uses: actions/upload-artifact@v4
+        with:
+          name: selfhost-loop-c
+          path: |
+            build/selfhost-loop-a/build/main.c
+            build/selfhost-loop-b/build/main.c
+          if-no-files-found: ignore
@@ -142,40 +142,11 @@ fmt: build
 	@./$(OUT) fmt apps/
 	@echo "Formatted lib/ examples/ src/ tests/ apps/"

+# Fixed-point: bootstrap buxc → buxc2 → buxc3 (path-normalized C + stripped ELF).
+# Slow; not part of default `make test`. Optional CI: .github/workflows/selfhost-loop.yml
 selfhost-loop: build
-	@echo "=== Selfhost loop: bootstrap determinism check ==="
-	@echo "Build A..."
-	@rm -rf build/selfhost-loop-a
-	@mkdir -p build/selfhost-loop-a/src
-	@cp src/*.bux build/selfhost-loop-a/src/
-	@cp src/bux.toml build/selfhost-loop-a/
-	@mv build/selfhost-loop-a/src/main.bux build/selfhost-loop-a/src/Main.bux 2>/dev/null || true
-	@cd build/selfhost-loop-a && ../../$(OUT) build
-	@echo "Build B..."
-	@rm -rf build/selfhost-loop-b
-	@mkdir -p build/selfhost-loop-b/src
-	@cp src/*.bux build/selfhost-loop-b/src/
-	@cp src/bux.toml build/selfhost-loop-b/
-	@mv build/selfhost-loop-b/src/main.bux build/selfhost-loop-b/src/Main.bux 2>/dev/null || true
-	@cd build/selfhost-loop-b && ../../$(OUT) build
-	@echo ""
-	@echo "Comparing C output..."
-	@if diff build/selfhost-loop-a/build/main.c build/selfhost-loop-b/build/main.c > /dev/null 2>&1; then \
-		echo "  C output: IDENTICAL ✓"; \
-	else \
-		echo "  C output: DIFFERENT ✗"; \
-	fi
-	@echo "Comparing ELF binaries (stripped)..."
-	@cp build/selfhost-loop-a/build/buxc2 /tmp/buxc2_a && strip -d /tmp/buxc2_a
-	@cp build/selfhost-loop-b/build/buxc2 /tmp/buxc2_b && strip -d /tmp/buxc2_b
-	@if diff /tmp/buxc2_a /tmp/buxc2_b > /dev/null 2>&1; then \
-		echo "  ELF binary: IDENTICAL ✓"; \
-		echo "=== Selfhost loop PASSED ==="; \
-	else \
-		echo "  ELF binary: DIFFERENT ✗"; \
-		echo "=== Selfhost loop FAILED ==="; \
-		exit 1; \
-	fi
+	@chmod +x tools/selfhost_loop.sh
+	@tools/selfhost_loop.sh

 lsp: tools/bux-lsp
 	@echo "LSP server ready at tools/bux-lsp"
@@ -349,8 +349,10 @@ make selfhost
 ./buxc search greet
 make test-registry          # add greet → install → build temp app

-# Verify selfhost binary parity (buxc2 → buxc3, identical)
+# Selfhost determinism (optional CI job; not in default `make test`)
 make selfhost-loop
+# Experimental fixed-point (buxc2 → buxc3):
+# BUX_SELFHOST_FIXED_POINT=1 make selfhost-loop

 # Clean build artifacts
 make clean
@@ -189,6 +189,17 @@ summary table and exits:

 Use `Std::Test` module for assertions inside test code.

+### Selfhost loop (optional CI)
+```bash
+make selfhost-loop                 # bootstrap builds src/ twice; C+ELF match
+BUX_SELFHOST_FIXED_POINT=1 make selfhost-loop   # also buxc2→buxc3 (experimental)
+```
+Not part of default `make test`. GitHub Actions workflow
+`.github/workflows/selfhost-loop.yml` runs it on:
+- manual `workflow_dispatch`
+- weekly schedule
+- pushes to `main` that touch `src/`, `lib/`, bootstrap, or the loop script
+
 ### Format (`bux fmt`)
 ```bash
 ./buxc fmt examples/hello.bux  # reformat one file
@@ -1,7 +1,7 @@
 # Bux — План към „добър“ език (v0.5 → v1.0)

 > **Дата:** 2026-07-19  
-> **Текущо:** v0.5.x — HirNode sourceFile #line, selfhost CI, **LSP 0.12 path rename**, Nexus KA  
+> **Текущо:** v0.5.x — HirNode sourceFile, **optional selfhost-loop CI**, LSP 0.12, Nexus KA  
 > **Цел:** Език, с който се пишат реални проекти комфортно, безопасно (по избор) и с надежден toolchain.

 ---
@@ -754,9 +754,26 @@ A (stdlib ergonomics)  →  B (compiler holes)  →  C (ownership depth)

 ---

+## Сесия 48 (optional CI: selfhost-loop)
+
+1. **`tools/selfhost_loop.sh`**: bootstrap determinism (buxc × 2)
+   - path-normalized `#line` C compare (abs path / build-dir noise stripped)
+   - stripped ELF compare; **fails** on real C or ELF mismatch
+2. **Lexer:** `maxTokens` 32k → 131k (hir_lower was at the ceiling for buxc2 parse)
+3. **Optional fixed-point** `BUX_SELFHOST_FIXED_POINT=1`: buxc2 → buxc3
+   - experimental; selfhost still diverges on full compiler C emit
+4. **GitHub Actions** `.github/workflows/selfhost-loop.yml`
+   - `workflow_dispatch` (+ optional fixed_point input)
+   - weekly cron (Sunday 06:00 UTC)
+   - push to `main` when `src/` / `lib/` / loop script change
+   - **not** on every PR / not in default `make test`
+5. Verified: `make selfhost-loop` PASS (determinism)
+
+---
+
 ## Следващи стъпки

-1. Optional: selfhost-loop as optional CI job (slow)
-2. LSP find-implementations request (dedicated, beyond call hierarchy)
-3. Workspace-wide import path index without open documents (optional polish)
-4. Expr/Stmt-level sourceFile if macros / cross-file inlining land
+1. LSP find-implementations request (dedicated, beyond call hierarchy)
+2. Workspace-wide import path index without open documents (optional polish)
+3. Expr/Stmt-level sourceFile if macros / cross-file inlining land
+4. Fix selfhost C backend so buxc2→buxc3 fixed-point is green
@@ -42,7 +42,8 @@ module Lexer {
    // Lexer state
    // ---------------------------------------------------------------------------

-    const maxTokens: int = 32768;
+    // hir_lower.bux alone is ~32k+ tokens; keep headroom for larger units / growth
+    const maxTokens: int = 131072;
    const maxDiags: int = 128;

    struct LexerDiag {
@@ -0,0 +1,147 @@
+#!/usr/bin/env bash
+# Selfhost loop (CI-optional / not in default `make test`).
+#
+# Default mode — bootstrap determinism (fast, required):
+#   buxc builds src/ twice → path-normalized main.c + stripped ELF must match.
+#
+# Optional fixed-point (slow, experimental — set BUX_SELFHOST_FIXED_POINT=1):
+#   buxc → buxc2 → buxc3; compare gen1 vs gen2. Currently may fail until
+#   selfhost C backend matches bootstrap on full compiler sources.
+#
+# Usage: tools/selfhost_loop.sh
+#        make selfhost-loop
+set -euo pipefail
+ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+cd "$ROOT"
+
+OUT="${OUT:-buxc}"
+export BUX_STDLIB="${BUX_STDLIB:-$ROOT/lib}"
+unset BUX_DEBUG_FILE || true
+unset BUX_NO_LINE || true
+
+if [[ ! -x "$ROOT/$OUT" ]]; then
+  echo "=== building bootstrap ($OUT) ==="
+  make -C "$ROOT" build
+fi
+
+A="$ROOT/build/selfhost-loop-a"
+B="$ROOT/build/selfhost-loop-b"
+TMP=$(mktemp -d)
+trap 'rm -rf "$TMP"' EXIT
+
+prepare_tree() {
+  local dest="$1"
+  rm -rf "$dest"
+  mkdir -p "$dest/src"
+  cp src/*.bux "$dest/src/"
+  cp src/bux.toml "$dest/"
+  if [[ -f "$dest/src/main.bux" ]]; then
+    mv "$dest/src/main.bux" "$dest/src/Main.bux"
+  fi
+}
+
+# Normalize #line paths so different build dirs / abs roots don't fail the diff.
+normalize_c() {
+  local src="$1" dst="$2"
+  sed -E \
+    -e 's|#line ([0-9]+) "[^"]*/(src/[^"]+\.bux)"|#line \1 "\2"|g' \
+    -e 's|#line ([0-9]+) "[^"]*/(lib/[^"]+\.bux)"|#line \1 "\2"|g' \
+    -e 's|#line ([0-9]+) "[^"]+/([^"/]+\.bux)"|#line \1 "\2"|g' \
+    "$src" > "$dst"
+}
+
+compare_c_and_elf() {
+  local label_a="$1" bin_a="$2" c_a="$3"
+  local label_b="$4" bin_b="$5" c_b="$6"
+  local ok=0
+
+  echo "--- Compare path-normalized C ($label_a vs $label_b) ---"
+  normalize_c "$c_a" "$TMP/main_a.c"
+  normalize_c "$c_b" "$TMP/main_b.c"
+  if diff -q "$TMP/main_a.c" "$TMP/main_b.c" >/dev/null 2>&1; then
+    echo "  C output: IDENTICAL ✓ (paths normalized)"
+  else
+    echo "  C output: DIFFERENT ✗"
+    diff -u "$TMP/main_a.c" "$TMP/main_b.c" | head -60 || true
+    ok=1
+  fi
+
+  echo "--- Compare stripped ELF ($label_a vs $label_b) ---"
+  cp "$bin_a" "$TMP/elf_a"
+  cp "$bin_b" "$TMP/elf_b"
+  strip -d "$TMP/elf_a" 2>/dev/null || strip "$TMP/elf_a"
+  strip -d "$TMP/elf_b" 2>/dev/null || strip "$TMP/elf_b"
+  if diff -q "$TMP/elf_a" "$TMP/elf_b" >/dev/null 2>&1; then
+    echo "  ELF binary: IDENTICAL ✓"
+  else
+    echo "  ELF binary: DIFFERENT ✗"
+    ls -la "$TMP/elf_a" "$TMP/elf_b"
+    ok=1
+  fi
+  return $ok
+}
+
+# ---------------------------------------------------------------------------
+# Mode 1: bootstrap determinism (always)
+# ---------------------------------------------------------------------------
+echo "=== Selfhost loop: bootstrap determinism (buxc × 2) ==="
+echo "--- Build A (bootstrap) ---"
+prepare_tree "$A"
+(cd "$A" && "$ROOT/$OUT" build)
+echo "--- Build B (bootstrap) ---"
+prepare_tree "$B"
+(cd "$B" && "$ROOT/$OUT" build)
+
+if ! compare_c_and_elf \
+  "A" "$A/build/buxc2" "$A/build/main.c" \
+  "B" "$B/build/buxc2" "$B/build/main.c"
+then
+  echo "=== Selfhost loop FAILED (bootstrap determinism) ==="
+  exit 1
+fi
+echo "=== Bootstrap determinism PASSED ==="
+
+# ---------------------------------------------------------------------------
+# Mode 2: fixed-point buxc2 → buxc3 (optional)
+# ---------------------------------------------------------------------------
+if [[ "${BUX_SELFHOST_FIXED_POINT:-0}" != "1" ]]; then
+  echo ""
+  echo "Note: fixed-point buxc2→buxc3 skipped (set BUX_SELFHOST_FIXED_POINT=1 to run)."
+  echo "=== Selfhost loop PASSED ==="
+  exit 0
+fi
+
+echo ""
+echo "=== Fixed-point: buxc2 → buxc3 (experimental) ==="
+BUXC2="$A/build/buxc2"
+if [[ ! -x "$BUXC2" ]]; then
+  echo "error: buxc2 missing at $BUXC2" >&2
+  exit 1
+fi
+
+# Rebuild B with buxc2
+prepare_tree "$B"
+set +e
+(cd "$B" && "$BUXC2" build)
+fp_status=$?
+set -e
+if [[ $fp_status -ne 0 ]]; then
+  echo "=== Fixed-point FAILED (buxc2 could not build gen2) ==="
+  exit 1
+fi
+
+BUXC3="$B/build/buxc2"
+if [[ ! -x "$BUXC3" ]]; then
+  echo "error: gen2 binary missing" >&2
+  exit 1
+fi
+
+if ! compare_c_and_elf \
+  "buxc2" "$BUXC2" "$A/build/main.c" \
+  "buxc3" "$BUXC3" "$B/build/main.c"
+then
+  echo "=== Fixed-point FAILED (gen1 vs gen2 mismatch) ==="
+  exit 1
+fi
+
+echo "=== Selfhost loop PASSED (determinism + fixed-point) ==="