feat: bootstrap skeleton + lexer (Phase 0)

.gitignore

@@ -0,0 +1,13 @@
+# Compiled binary
+buxc
+
+# Nim cache
+nimcache/
+
+# Test binaries
+tests/lexer_test
+tests/*.exe
+
+# Build artifacts
+build/
+*.o

Makefile

@@ -0,0 +1,26 @@
+NIM := nim
+SRC := src/main.nim
+OUT := buxc
+BUILD_DIR := build
+
+.PHONY: all build test clean
+
+all: build
+
+build:
+	$(NIM) c -o:$(OUT) -d:release $(SRC)
+
+dev:
+	$(NIM) c -o:$(OUT) $(SRC)
+
+test: build
+	@echo "Running lexer tests..."
+	$(NIM) c -r tests/lexer_test.nim
+	@echo "Running integration tests..."
+	./$(OUT) new _test_tmp_pkg
+	./$(OUT) --version
+
+clean:
+	rm -f $(OUT)
+	rm -rf $(BUILD_DIR)
+	rm -rf nimcache

PLAN.md

@@ -0,0 +1,273 @@
+# Bux Programming Language — Roadmap to Self-Hosting
+
+> **Reference:** [Rux Language](https://rux-lang.dev/) | [Rux Source](../_rux/)  
+> **Bootstrap Implementation:** Nim  
+> **Target:** Bux compiler written in Bux (self-hosting)
+
+---
+
+## Overview
+
+Bux is a fast, compiled, strongly-typed, multi-paradigm systems programming language inspired by Rux. The strategy is **bootstrap via Nim** — we build the first Bux compiler in Nim, then progressively rewrite it in Bux until it compiles itself.
+
+---
+
+## Phase 0 — Bootstrap Foundation (Week 1-2)
+
+**Goal:** Working Nim project that can lex, parse, and dump a Bux AST.
+
+| Task | Details |
+|------|---------|
+| `0.1` Project skeleton | `buxc` CLI in Nim, `bux.toml` manifest parser |
+| `0.2` Token model | All Rux tokens (`TokenKind`, `SourceLocation`, literal suffixes) |
+| `0.3` Lexer | UTF-8 source, identifiers, numbers (dec/hex/bin/oct), strings (`c8""`, `c16""`, `c32""`), chars, operators, nested `/* */`, `//` comments, intrinsics (`#line`, `#file`, etc.) |
+| `0.4` CLI commands | `bux new`, `bux init`, `bux build`, `bux run`, `bux check` |
+| `0.5` Test harness | Golden-file tests for lexer output (`.tokens`) |
+
+**Deliverable:** `echo 'let x = 42' | bux check` prints token stream.
+
+---
+
+## Phase 1 — Frontend: Parser & AST (Week 3-4)
+
+**Goal:** Parse every construct present in Rux v0.2.0 into a Nim AST.
+
+| Task | Details |
+|------|---------|
+| `1.1` AST nodes | All `Expr`, `Stmt`, `Decl`, `Pattern`, `TypeExpr`, `Block` variants (see `_rux/Include/Rux/Ast.h`) |
+| `1.2` Pratt parser | Full precedence climbing for all binary/unary/postfix operators including `**` (right-assoc) and range `..` / `..=` |
+| `1.3` Declarations | `func`, `struct`, `enum`, `union`, `interface`, `extend`/`impl`, `module`, `const`, `type`, `extern`, `import`/`use` |
+| `1.4` Statements | `let`/`var`, `if`/`else if`/`else`, `while`, `do while`, `loop`, `for in`, `match`, `return`, `break`/`continue` (with labels) |
+| `1.5` Expressions | Literals, identifiers, paths (`a::b`), calls, index, field access, struct init, slice init `[a,b]`, tuple `(a,b)`, cast `as`, test `is`, ternary `? :`, block-expr `{ ... }` |
+| `1.6` Patterns | Wildcard `_`, literal, ident, range, enum destructuring, struct destructuring, tuple, guarded `if` |
+| `1.7` Attributes | `@[Import(lib: "...")]`, calling-convention, platform-conditional imports |
+| `1.8` Error recovery | Synchronize on declaration/statement boundaries; emit multiple diagnostics |
+
+**Deliverable:** All `_rux/Tests/**/*.rux` files parse without error and produce `.ast` dumps.
+
+---
+
+## Phase 2 — Semantic Analysis (Week 5-7)
+
+**Goal:** Type-check the AST and produce a typed symbol table.
+
+| Task | Details |
+|------|---------|
+| `2.1` Type model | `TypeRef` with primitives, pointers, slices, tuples, named types, type parameters, functions (see `_rux/Include/Rux/Type.h`) |
+| `2.2` Scopes | Module scope, block scope, namespace resolution for `Std::Io::PrintLine` |
+| `2.3` First pass | Collect global symbols (functions, structs, enums, unions, interfaces, consts, type aliases, imports) |
+| `2.4` Type checking | Expression typing, operator overload resolution per Rux rules, assignment compatibility |
+| `2.5` Name resolution | Resolve identifiers, paths, `self`, `super`; report undeclared / ambiguous names |
+| `2.6` Interface conformance | Check that `extend T for I` provides all required methods; build vtable map |
+| `2.7` Generics (basic) | Monomorphization of generic structs and functions at call sites |
+| `2.8` Diagnostics | Multi-file error messages with source locations |
+
+**Deliverable:** `bux check` rejects ill-typed programs and passes `Tests/Echo`, `Tests/Io`, `Tests/Pow` type-checking.
+
+---
+
+## Phase 3 — High-Level IR (HIR) (Week 8)
+
+**Goal:** Lower AST to a simplified, fully-typed HIR.
+
+| Task | Details |
+|------|---------|
+| `3.1` HIR nodes | Desugared equivalents of AST nodes (see `_rux/Include/Rux/Hir.h`) |
+| `3.2` Lowering | Desugar `for` → `while`+iterator, `match` → decision tree, method calls to explicit receiver calls |
+| `3.3` Constant folding | Evaluate `const` and simple compile-time expressions |
+| `3.4` Interface lowering | Convert interface values to fat pointers `{data_ptr, vtable_ptr}`; generate vtable labels |
+
+**Deliverable:** HIR dump matches Rux HIR semantics for sample programs.
+
+---
+
+## Phase 4 — Low-Level IR (LIR) (Week 9-10)
+
+**Goal:** Generate SSA-like LIR with virtual registers and basic blocks.
+
+| Task | Details |
+|------|---------|
+| `4.1` LIR model | `LirInstr`, `LirBlock`, `LirTerminator`, `LirFunc`, `LirReg`, opcodes (`Const`, `Alloca`, `Load`, `Store`, arithmetic, `Call`, `Phi`, `GlobalAddr`, etc.) |
+| `4.2` Control flow | Lower `if`, `while`, `loop`, `match` to blocks with `Jump` / `Branch` / `Switch` terminators |
+| `4.3` Memory | Stack allocation (`alloca`), pointer arithmetic, field/index pointer computation |
+| `4.4` Calls | Direct calls, indirect calls, extern calls with correct ABI marking (System V / Win64) |
+
+**Deliverable:** `bux build --emit-lir` produces readable LIR for all test programs.
+
+---
+
+## Phase 5 — Backend & Code Generation (Week 11-14)
+
+**Strategy:** Two backends in parallel — a **C transpiler** for instant portability and a **native x86-64** backend for performance.
+
+### 5A — C Transpiler (Primary bootstrap path)
+
+| Task | Details |
+|------|---------|
+| `5A.1` C emitter | Walk LIR and emit C11 code |
+| `5A.2` Types to C | Bux primitives → C primitives; structs → C structs; enums → C enums + tagged unions; slices → `{T* data; size_t len;}` |
+| `5A.3` Functions to C | Bux functions → C functions with `static` / `extern`; name mangling for overloads/generics |
+| `5A.4` FFI | `extern` / `@[Import]` → `#include` + function declarations; link with system `cc` |
+| `5A.5` Runtime shim | Small C runtime providing `bux_alloc`, `bux_print`, panic/abort for div-by-zero, etc. |
+| `5A.6` Build integration | `bux build` invokes `cc` / `clang` / `gcc` automatically |
+
+**Deliverable:** `bux run` on `Tests/Io/Main.bux` prints "Hello from a Bux binary!".
+
+### 5B — Native x86-64 Backend (Secondary, for self-hosting speed)
+
+| Task | Details |
+|------|---------|
+| `5B.1` Assembly emitter | NASM-syntax text output (like Rux `Asm`) |
+| `5B.2` Register allocation | Naive stack-spill allocator first; later linear-scan |
+| `5B.3` ABI lowering | System V AMD64 ABI (Linux/macOS) and Win64 ABI (Windows) |
+| `5B.4` Object format | Emit ELF64 (Linux), Mach-O (macOS), PE/COFF (Windows) — or use `nasm` + system linker |
+| `5B.5` Custom linker (optional) | `.bcu` (Bux Compiled Unit) format + bespoke linker à la Rux `.rcu` |
+
+**Deliverable:** `bux build --backend=native` produces working Linux x86-64 binary.
+
+---
+
+## Phase 6 — Standard Library (Week 15-18)
+
+**Goal:** Enough stdlib to write the compiler in Bux.
+
+| Module | Requirements |
+|--------|-------------|
+| `Std::Io` | `Print`, `PrintLine`, `ReadLine`, file read/write (wrap C stdio initially) |
+| `Std::Memory` | `Alloc`, `Free`, `Realloc` (wrap `malloc`/`free`) |
+| `Std::String` | Basic string builder, concatenation, slicing |
+| `Std::Array` | Dynamic array (`Vec<T>` equivalent): `push`, `pop`, `get`, `len`, `capacity` |
+| `Std::Map` | Hash map with string keys (needed for symbol tables) |
+| `Std::Math` | `Sqrt`, `Pow`, `Min`, `Max`, `Abs` |
+| `Std::Os` | `Args`, `Env`, `Exit`, `Cwd` |
+| `Std::Path` | Path joining, extension splitting |
+| `Std::Process` | Spawn subprocess, read stdout/stderr |
+
+**Deliverable:** Can write a non-trivial CLI tool (e.g., a file copier or a basic grep) entirely in Bux.
+
+---
+
+## Phase 7 — Self-Hosting: The Great Rewrite (Week 19-26)
+
+**Goal:** Bux compiler compiles itself. This is the **main milestone**.
+
+| Task | Details |
+|------|---------|
+| `7.1` Port lexer | Rewrite `lexer.nim` → `Lexer.bux` |
+| `7.2` Port parser | Rewrite `parser.nim` → `Parser.bux` |
+| `7.3` Port sema | Rewrite `sema.nim` → `Sema.bux` |
+| `7.4` Port HIR | Rewrite `hir.nim` → `Hir.bux` |
+| `7.5` Port LIR | Rewrite `lir.nim` → `Lir.bux` |
+| `7.6` Port C backend | Rewrite `c_backend.nim` → `CBackend.bux` |
+| `7.7` Port CLI | Rewrite `main.nim` → `Main.bux` |
+| `7.8` Dogfooding | Use `buxc` (Nim) to build `buxc2` (Bux). Then use `buxc2` to build `buxc3`. Compare bit-for-bit. |
+| `7.9` Fix bootstrap loop | Once `buxc2 == buxc3`, we are self-hosted. Freeze Nim version as reference. |
+
+**Deliverable:** `make selfhost` succeeds; Bux compiler is written entirely in Bux.
+
+---
+
+## Phase 8 — Ecosystem & Tooling (Week 27+)
+
+| Task | Details |
+|------|---------|
+| `8.1` Package manager | `bux add`, `bux remove`, `bux update`, `bux install` with lockfile |
+| `8.2` Registry protocol | Simple HTTP git-based registry (like Go modules or Cargo) |
+| `8.3` Formatter | `bux fmt` — auto-format Bux source |
+| `8.4` LSP | Language Server Protocol for autocomplete, hover, go-to-definition |
+| `8.5` Tests | `bux test` runner with assertions and golden tests |
+| `8.6` Documentation | `bux doc` — generate HTML from `///` doc comments |
+| `8.7` Cross-compilation | `--target` flag leveraging C backend portability |
+
+---
+
+## File Structure (Target)
+
+```
+bux/
+├── bux.toml                  # Compiler package manifest
+├── README.md
+├── PLAN.md
+├── Makefile                  # build, test, selfhost
+├── src/
+│   ├── Main.bux              # CLI entry point
+│   ├── Lexer.bux
+│   ├── Parser.bux
+│   ├── Ast.bux
+│   ├── Sema.bux
+│   ├── Type.bux
+│   ├── Hir.bux
+│   ├── Lir.bux
+│   ├── CBackend.bux          # C transpiler (primary backend)
+│   ├── X64Backend.bux        # Native x86-64 backend (optional)
+│   ├── Linker.bux            # Custom linker / build driver
+│   ├── Manifest.bux          # bux.toml parser
+│   └── Package.bux           # Package resolution
+├── stdlib/
+│   ├── Std/
+│   │   ├── Io.bux
+│   │   ├── Memory.bux
+│   │   ├── String.bux
+│   │   ├── Array.bux
+│   │   ├── Map.bux
+│   │   ├── Math.bux
+│   │   ├── Os.bux
+│   │   ├── Path.bux
+│   │   └── Process.bux
+│   └── Runtime.c             # C runtime shim
+├── tests/
+│   ├── Lexer/
+│   ├── Parser/
+│   ├── Sema/
+│   ├── Codegen/
+│   └── Integration/
+└── docs/
+    └── LanguageRef.md
+```
+
+---
+
+## Language Design Decisions (Bux vs Rux)
+
+| Feature | Bux Decision | Rationale |
+|---------|-------------|-----------|
+| **Backend (bootstrap)** | C transpiler first | Fastest path to working compiler; leverages existing C toolchains |
+| **Backend (final)** | Native x86-64 + optional LLVM | Match Rux ambition; self-hosting needs speed |
+| **Memory safety** | Raw pointers + optional borrow checker (Phase 9) | Match Rux current model; gradual safety |
+| **Generics** | Monomorphization only | Simpler than Rust-style trait objects; enough for self-hosting |
+| **Error handling** | Explicit `Result<T, E>` + `?` operator (later) | Start with C-style returns; add sugar after self-hosting |
+| **String literals** | `c8""`, `c16""`, `c32""` + `""` defaulting to `c8` | Same as Rux |
+| **Build system** | `bux.toml` (same as `Rux.toml`) | Compatible manifest format |
+| **Module system** | `import Std::Io::PrintLine` | Same as Rux path syntax |
+
+---
+
+## Milestones Summary
+
+| Milestone | Phase | Success Criteria |
+|-----------|-------|------------------|
+| **M0** | 0 | `bux check` lexes source |
+| **M1** | 1 | All Rux test files parse |
+| **M2** | 2 | Type-checker rejects invalid programs |
+| **M3** | 3+4 | LIR emits for all constructs |
+| **M4** | 5A | `bux run` produces working binary via C |
+| **M5** | 6 | Can write compiler-adjacent tools in Bux |
+| **M6** | 7 | **Self-hosted**: Bux compiler builds itself |
+| **M7** | 8 | Package manager + LSP + formatter shipped |
+
+---
+
+## Risk Mitigation
+
+| Risk | Mitigation |
+|------|------------|
+| Nim bootstrap too slow | Keep Nim code simple; aim for rewrite in ~3 months |
+| C backend limits performance | Maintain parallel native backend; C is only bootstrap |
+| Generics get complex | Restrict to monomorphization; no higher-kinded types |
+| Self-hosting too hard | Ensure stdlib has `Array`, `Map`, `String` before starting rewrite |
+
+---
+
+## Next Immediate Step
+
+Create the Nim bootstrap skeleton and implement the **Lexer** (`0.1`–`0.3`).

README.md

@@ -0,0 +1,38 @@
+# Bux Programming Language
+
+> **Status:** Bootstrap phase — compiler written in Nim, targeting self-hosting.
+
+Bux is a fast, compiled, strongly-typed systems programming language inspired by [Rux](https://rux-lang.dev/). The long-term goal is a self-hosted compiler with a minimal runtime, native x86-64 backend, and modern tooling.
+
+## Quick Start
+
+```bash
+# Build the bootstrap compiler (Nim)
+make build
+
+# Create a new project
+bux new hello
+
+# Build and run
+bux run
+```
+
+## Syntax Preview
+
+```bux
+import Std::Io::PrintLine;
+
+func Main() -> int {
+    let message: *char8 = c8"Hello, Bux!";
+    PrintLine(message);
+    return 0;
+}
+```
+
+## Roadmap
+
+See [`PLAN.md`](PLAN.md) for the full roadmap to self-hosting.
+
+## License
+
+MIT

bux.toml

@@ -0,0 +1,7 @@
+[Package]
+Name    = "buxc"
+Version = "0.1.0"
+Type    = "bin"
+
+[Build]
+Output = "Bin"

src/cli.nim

@@ -0,0 +1,234 @@
+import std/[os, strutils, terminal, strformat]
+import lexer, manifest
+
+type
+  ColorMode* = enum
+    cmAuto
+    cmOn
+    cmOff
+
+  GlobalOptions* = object
+    color*: ColorMode
+    quiet*: bool
+    verbose*: bool
+
+proc printUsage*() =
+  echo """Bux Programming Language (bootstrap compiler)
+
+Usage: bux [options] <command> [command-options]
+
+Commands:
+  new <name>          Create a new Bux package
+  init                Initialize a Bux package in the current directory
+  build               Build the current package
+  run                 Build and run the current package
+  check               Type-check the current package
+  clean               Remove build artifacts
+  help                Show this help message
+  version             Show version
+
+Global options:
+  --color <auto|on|off>   Control colored output (default: auto)
+  -q, --quiet             Suppress non-error output
+  -v, --verbose           Verbose output
+"""
+
+proc parseGlobalOptions(args: seq[string]): tuple[opts: GlobalOptions, rest: seq[string], ok: bool] =
+  result.opts = GlobalOptions(color: cmAuto, quiet: false, verbose: false)
+  result.rest = @[]
+  result.ok = true
+  var i = 0
+  while i < args.len:
+    let arg = args[i]
+    if arg == "--color":
+      if i + 1 >= args.len:
+        stderr.writeLine("error: --color requires an argument")
+        result.ok = false
+        return
+      inc i
+      case args[i].toLowerAscii()
+      of "auto": result.opts.color = cmAuto
+      of "on": result.opts.color = cmOn
+      of "off": result.opts.color = cmOff
+      else:
+        stderr.writeLine(&"error: unknown --color value '{args[i]}'")
+        result.ok = false
+        return
+    elif arg == "-q" or arg == "--quiet":
+      result.opts.quiet = true
+    elif arg == "-v" or arg == "--verbose":
+      result.opts.verbose = true
+    else:
+      result.rest.add(arg)
+    inc i
+
+proc shouldUseColor(opts: GlobalOptions): bool =
+  case opts.color
+  of cmOn: true
+  of cmOff: false
+  of cmAuto: terminal.isatty(stdout)
+
+proc printError(msg: string, useColor: bool) =
+  if useColor:
+    stdout.setForegroundColor(fgRed)
+    stdout.write("error: ")
+    stdout.resetAttributes()
+    stdout.writeLine(msg)
+  else:
+    stderr.writeLine("error: " & msg)
+
+proc printInfo(msg: string, useColor: bool) =
+  if useColor:
+    stdout.setForegroundColor(fgCyan)
+    stdout.write("info: ")
+    stdout.resetAttributes()
+    stdout.writeLine(msg)
+  else:
+    echo("info: " & msg)
+
+# ---------------------------------------------------------------------------
+# Commands
+# ---------------------------------------------------------------------------
+
+proc cmdNew*(args: seq[string], opts: GlobalOptions): int =
+  let useColor = shouldUseColor(opts)
+  if args.len < 1:
+    printError("'new' requires a package name", useColor)
+    return 1
+  let name = args[0]
+  let root = getCurrentDir() / name
+  if dirExists(root):
+    printError(&"directory '{name}' already exists", useColor)
+    return 1
+  createDir(root / "src")
+  writeFile(root / "bux.toml", &"""[Package]
+Name    = "{name}"
+Version = "0.1.0"
+Type    = "bin"
+
+[Build]
+Output = "Bin"
+""")
+  writeFile(root / "src" / "Main.bux", """import Std::Io::PrintLine;
+
+func Main() -> int {
+    PrintLine(c8"Hello, Bux!");
+    return 0;
+}
+""")
+  if not opts.quiet:
+    printInfo(&"Created Bux package '{name}'", useColor)
+  return 0
+
+proc cmdInit*(args: seq[string], opts: GlobalOptions): int =
+  let useColor = shouldUseColor(opts)
+  let root = getCurrentDir()
+  if fileExists(root / "bux.toml"):
+    printError("bux.toml already exists", useColor)
+    return 1
+  let name = splitPath(root).tail
+  writeFile(root / "bux.toml", &"""[Package]
+Name    = "{name}"
+Version = "0.1.0"
+Type    = "bin"
+
+[Build]
+Output = "Bin"
+""")
+  if not dirExists(root / "src"):
+    createDir(root / "src")
+  if not opts.quiet:
+    printInfo(&"Initialized Bux package '{name}'", useColor)
+  return 0
+
+proc cmdCheck*(args: seq[string], opts: GlobalOptions): int =
+  let useColor = shouldUseColor(opts)
+  let root = getCurrentDir()
+  let manifestPath = root / "bux.toml"
+  if not fileExists(manifestPath):
+    printError("no bux.toml found", useColor)
+    return 1
+  let man = loadManifest(manifestPath)
+  let srcDir = root / "src"
+  if not dirExists(srcDir):
+    printError("no src/ directory found", useColor)
+    return 1
+
+  var foundMain = false
+  for kind, path in walkDir(srcDir):
+    if kind == pcFile and path.endsWith(".bux"):
+      let source = readFile(path)
+      let res = tokenize(source, path)
+      if res.hasErrors:
+        printError(&"lex errors in {path}", useColor)
+        for d in res.diagnostics:
+          echo $d
+        return 1
+      if opts.verbose:
+        printInfo(&"lexed {path} ({res.tokens.len} tokens)", useColor)
+      if splitFile(path).name == "Main":
+        foundMain = true
+
+  if not foundMain:
+    printError("no Main.bux found in src/", useColor)
+    return 1
+
+  if not opts.quiet:
+    printInfo("check passed (lexer only)", useColor)
+  return 0
+
+proc cmdBuild*(args: seq[string], opts: GlobalOptions): int =
+  let useColor = shouldUseColor(opts)
+  # For now, just type-check
+  let checkRes = cmdCheck(args, opts)
+  if checkRes != 0:
+    return checkRes
+  if not opts.quiet:
+    printInfo("build: nothing to do yet (no codegen)", useColor)
+  return 0
+
+proc cmdRun*(args: seq[string], opts: GlobalOptions): int =
+  let useColor = shouldUseColor(opts)
+  let buildRes = cmdBuild(args, opts)
+  if buildRes != 0:
+    return buildRes
+  printError("run: no executable produced yet (no codegen)", useColor)
+  return 1
+
+proc cmdClean*(args: seq[string], opts: GlobalOptions): int =
+  let useColor = shouldUseColor(opts)
+  # Placeholder
+  if not opts.quiet:
+    printInfo("clean: nothing to do yet", useColor)
+  return 0
+
+proc cmdVersion*(args: seq[string], opts: GlobalOptions): int =
+  echo "bux 0.1.0 (bootstrap)"
+  return 0
+
+proc runCli*(args: seq[string]): int =
+  let (opts, rest, ok) = parseGlobalOptions(args)
+  if not ok:
+    return 1
+  if rest.len == 0:
+    printUsage()
+    return 0
+
+  let cmd = rest[0]
+  let cmdArgs = if rest.len > 1: rest[1..^1] else: @[]
+
+  case cmd
+  of "new": return cmdNew(cmdArgs, opts)
+  of "init": return cmdInit(cmdArgs, opts)
+  of "build": return cmdBuild(cmdArgs, opts)
+  of "run": return cmdRun(cmdArgs, opts)
+  of "check": return cmdCheck(cmdArgs, opts)
+  of "clean": return cmdClean(cmdArgs, opts)
+  of "version", "--version", "-v": return cmdVersion(cmdArgs, opts)
+  of "help", "--help", "-h":
+    printUsage()
+    return 0
+  else:
+    let useColor = shouldUseColor(opts)
+    printError(&"unknown command '{cmd}'", useColor)
+    return 1

src/lexer.nim

@@ -0,0 +1,565 @@
+import std/[strutils, strformat]
+import token, source_location
+
+type
+  LexerDiagnosticSeverity* = enum
+    ldsWarning
+    ldsError
+
+  LexerDiagnostic* = object
+    severity*: LexerDiagnosticSeverity
+    loc*: SourceLocation
+    message*: string
+
+  LexerResult* = object
+    tokens*: seq[Token]
+    diagnostics*: seq[LexerDiagnostic]
+
+proc hasErrors*(res: LexerResult): bool =
+  for d in res.diagnostics:
+    if d.severity == ldsError:
+      return true
+  return false
+
+proc `$`*(d: LexerDiagnostic): string =
+  let sev = if d.severity == ldsError: "error" else: "warning"
+  result = &"{sev}: {d.message} at {d.loc}"
+
+type
+  Lexer* = object
+    source: string
+    sourceName: string
+    pos: int
+    line: uint32
+    col: uint32
+    tokens: seq[Token]
+    diagnostics: seq[LexerDiagnostic]
+
+proc initLexer*(source, sourceName: string): Lexer =
+  result.source = source
+  result.sourceName = sourceName
+  result.pos = 0
+  result.line = 1
+  result.col = 1
+
+proc isAtEnd(lex: Lexer): bool =
+  lex.pos >= lex.source.len
+
+proc peek(lex: Lexer, ahead: int = 0): char =
+  let i = lex.pos + ahead
+  if i < lex.source.len:
+    return lex.source[i]
+  return '\0'
+
+proc advance(lex: var Lexer): char =
+  result = lex.peek()
+  if not lex.isAtEnd():
+    inc lex.pos
+    if result == '\n':
+      inc lex.line
+      lex.col = 1
+    else:
+      inc lex.col
+
+proc match(lex: var Lexer, expected: char): bool =
+  if lex.isAtEnd(): return false
+  if lex.peek() != expected: return false
+  discard lex.advance()
+  return true
+
+proc matchStr(lex: var Lexer, s: string): bool =
+  for i, c in s:
+    if lex.peek(i) != c:
+      return false
+  for _ in s:
+    discard lex.advance()
+  return true
+
+proc currentLocation(lex: Lexer): SourceLocation =
+  result = SourceLocation(line: lex.line, column: lex.col, offset: uint32(lex.pos))
+
+proc emitError(lex: var Lexer, loc: SourceLocation, message: string) =
+  lex.diagnostics.add(LexerDiagnostic(severity: ldsError, loc: loc, message: message))
+
+proc emitWarning(lex: var Lexer, loc: SourceLocation, message: string) =
+  lex.diagnostics.add(LexerDiagnostic(severity: ldsWarning, loc: loc, message: message))
+
+proc makeToken(lex: Lexer, kind: TokenKind, startLoc: SourceLocation, startPos: int): Token =
+  let text = lex.source[startPos ..< lex.pos]
+  result = Token(kind: kind, text: text, loc: startLoc)
+
+# ---------------------------------------------------------------------------
+# Whitespace / comments
+# ---------------------------------------------------------------------------
+
+proc skipLineComment(lex: var Lexer) =
+  while not lex.isAtEnd() and lex.peek() != '\n':
+    discard lex.advance()
+
+proc skipBlockComment(lex: var Lexer) =
+  let startLoc = lex.currentLocation()
+  var depth = 1
+  while not lex.isAtEnd() and depth > 0:
+    if lex.peek() == '/' and lex.peek(1) == '*':
+      discard lex.advance()
+      discard lex.advance()
+      inc depth
+    elif lex.peek() == '*' and lex.peek(1) == '/':
+      discard lex.advance()
+      discard lex.advance()
+      dec depth
+    else:
+      discard lex.advance()
+  if depth > 0:
+    lex.emitError(startLoc, "unterminated block comment")
+
+proc skipWhitespace(lex: var Lexer) =
+  while not lex.isAtEnd():
+    let c = lex.peek()
+    if c in {' ', '\t', '\r'}:
+      discard lex.advance()
+    elif c == '/' and lex.peek(1) == '/':
+      lex.skipLineComment()
+    elif c == '/' and lex.peek(1) == '*':
+      discard lex.advance()
+      discard lex.advance()
+      lex.skipBlockComment()
+    else:
+      break
+
+# ---------------------------------------------------------------------------
+# Identifiers
+# ---------------------------------------------------------------------------
+
+proc isIdentStart(c: char): bool =
+  c in {'a'..'z', 'A'..'Z', '_'}
+
+proc isIdentChar(c: char): bool =
+  c in {'a'..'z', 'A'..'Z', '0'..'9', '_'}
+
+proc scanIdent(lex: var Lexer, startLoc: SourceLocation): Token =
+  let startPos = lex.pos
+  while not lex.isAtEnd() and isIdentChar(lex.peek()):
+    discard lex.advance()
+  let text = lex.source[startPos ..< lex.pos]
+  let kind = keywordKind(text)
+  result = Token(kind: kind, text: text, loc: startLoc)
+
+# ---------------------------------------------------------------------------
+# Numbers
+# ---------------------------------------------------------------------------
+
+proc scanIntSuffix(lex: var Lexer) =
+  # i8, i16, i32, i64, u8, u16, u32, u64, f32, f64
+  if lex.isAtEnd(): return
+  let c = lex.peek()
+  if c in {'i', 'u', 'f'}:
+    discard lex.advance()
+    while not lex.isAtEnd() and lex.peek() in {'0'..'9'}:
+      discard lex.advance()
+
+proc scanHexDigits(lex: var Lexer) =
+  while not lex.isAtEnd() and lex.peek() in {'0'..'9', 'a'..'f', 'A'..'F'}:
+    discard lex.advance()
+
+proc scanBinDigits(lex: var Lexer) =
+  while not lex.isAtEnd() and lex.peek() in {'0', '1'}:
+    discard lex.advance()
+
+proc scanOctDigits(lex: var Lexer) =
+  while not lex.isAtEnd() and lex.peek() in {'0'..'7'}:
+    discard lex.advance()
+
+proc scanDecDigits(lex: var Lexer) =
+  while not lex.isAtEnd() and lex.peek() in {'0'..'9'}:
+    discard lex.advance()
+
+proc scanNumber(lex: var Lexer, startLoc: SourceLocation): Token =
+  let startPos = lex.pos
+  var isFloat = false
+
+  if lex.peek() == '0' and lex.peek(1) in {'x', 'X', 'b', 'B', 'o', 'O'}:
+    discard lex.advance()  # '0'
+    let prefix = lex.advance()
+    case prefix
+    of 'x', 'X': lex.scanHexDigits()
+    of 'b', 'B': lex.scanBinDigits()
+    of 'o', 'O': lex.scanOctDigits()
+    else: discard
+    lex.scanIntSuffix()
+    return lex.makeToken(tkIntLiteral, startLoc, startPos)
+
+  lex.scanDecDigits()
+
+  # Fractional part
+  if lex.peek() == '.' and lex.peek(1) in {'0'..'9'}:
+    isFloat = true
+    discard lex.advance()  # '.'
+    lex.scanDecDigits()
+
+  # Exponent
+  if lex.peek() in {'e', 'E'}:
+    isFloat = true
+    discard lex.advance()
+    if lex.peek() in {'+', '-'}:
+      discard lex.advance()
+    if lex.peek() notin {'0'..'9'}:
+      lex.emitError(lex.currentLocation(), "expected digits in exponent")
+    else:
+      lex.scanDecDigits()
+
+  if isFloat:
+    # Optional f32/f64 suffix
+    if lex.peek() == 'f' and lex.peek(1) in {'3', '6'}:
+      discard lex.advance()
+      discard lex.advance()
+    result = lex.makeToken(tkFloatLiteral, startLoc, startPos)
+  else:
+    lex.scanIntSuffix()
+    result = lex.makeToken(tkIntLiteral, startLoc, startPos)
+
+# ---------------------------------------------------------------------------
+# Strings and chars
+# ---------------------------------------------------------------------------
+
+proc scanEscapeSequence(lex: var Lexer): string =
+  if lex.isAtEnd():
+    return ""
+  let c = lex.advance()
+  case c
+  of '\\': result = "\\"
+  of '"': result = "\""
+  of '\'': result = "\'"
+  of 'n': result = "\n"
+  of 'r': result = "\r"
+  of 't': result = "\t"
+  of '0': result = "\0"
+  of 'x':
+    var hexVal = ""
+    for _ in 0..<2:
+      if lex.isAtEnd() or lex.peek() notin {'0'..'9', 'a'..'f', 'A'..'F'}:
+        lex.emitError(lex.currentLocation(), "expected two hex digits after \\x")
+        break
+      hexVal.add(lex.advance())
+    if hexVal.len == 2:
+      try:
+        let code = parseHexInt(hexVal)
+        result = $chr(code)
+      except ValueError:
+        lex.emitError(lex.currentLocation(), &"invalid hex escape \\x{hexVal}")
+        result = ""
+    else:
+      result = ""
+  else:
+    lex.emitError(lex.currentLocation(), &"unknown escape sequence \\\\{c}")
+    result = $c
+
+proc scanString(lex: var Lexer, startLoc: SourceLocation, prefixLen: int): Token =
+  let startPos = lex.pos - prefixLen
+  # prefixLen characters before the opening quote were already consumed by caller
+  # but we need to handle the quote itself
+  if lex.peek() == '"':
+    discard lex.advance()
+  while not lex.isAtEnd() and lex.peek() != '"':
+    if lex.peek() == '\n':
+      lex.emitError(lex.currentLocation(), "unterminated string literal")
+      break
+    if lex.peek() == '\\':
+      discard lex.advance()
+      discard lex.scanEscapeSequence()
+    else:
+      discard lex.advance()
+  if lex.isAtEnd():
+    lex.emitError(startLoc, "unterminated string literal")
+  else:
+    discard lex.advance()  # closing "
+  result = lex.makeToken(tkStringLiteral, startLoc, startPos)
+
+proc scanChar(lex: var Lexer, startLoc: SourceLocation, prefixLen: int): Token =
+  let startPos = lex.pos - prefixLen
+  if lex.peek() == '\'':
+    discard lex.advance()
+  if lex.isAtEnd():
+    lex.emitError(startLoc, "unterminated char literal")
+    return lex.makeToken(tkCharLiteral, startLoc, startPos)
+  if lex.peek() == '\n':
+    lex.emitError(lex.currentLocation(), "newline in char literal")
+  elif lex.peek() == '\\':
+    discard lex.advance()
+    discard lex.scanEscapeSequence()
+  else:
+    discard lex.advance()
+  if lex.isAtEnd() or lex.peek() != '\'':
+    lex.emitError(lex.currentLocation(), "expected closing ' for char literal")
+  else:
+    discard lex.advance()
+  result = lex.makeToken(tkCharLiteral, startLoc, startPos)
+
+# ---------------------------------------------------------------------------
+# Symbols / operators
+# ---------------------------------------------------------------------------
+
+proc scanSymbol(lex: var Lexer, startLoc: SourceLocation): Token =
+  let startPos = lex.pos
+  let c1 = lex.advance()
+
+  template check2(c2: char, kind2: TokenKind, kind1: TokenKind) =
+    if lex.peek() == c2:
+      discard lex.advance()
+      return lex.makeToken(kind2, startLoc, startPos)
+    else:
+      return lex.makeToken(kind1, startLoc, startPos)
+
+  template check3(c2: char, kind2: TokenKind, c3: char, kind3: TokenKind, kind1: TokenKind) =
+    if lex.peek() == c2:
+      discard lex.advance()
+      if lex.peek() == c3:
+        discard lex.advance()
+        return lex.makeToken(kind3, startLoc, startPos)
+      return lex.makeToken(kind2, startLoc, startPos)
+    else:
+      return lex.makeToken(kind1, startLoc, startPos)
+
+  template checkEq(c2: char, kind2: TokenKind, kind1: TokenKind) =
+    check2(c2, kind2, kind1)
+
+  case c1
+  of '(': return lex.makeToken(tkLParen, startLoc, startPos)
+  of ')': return lex.makeToken(tkRParen, startLoc, startPos)
+  of '{': return lex.makeToken(tkLBrace, startLoc, startPos)
+  of '}': return lex.makeToken(tkRBrace, startLoc, startPos)
+  of '[': return lex.makeToken(tkLBracket, startLoc, startPos)
+  of ']': return lex.makeToken(tkRBracket, startLoc, startPos)
+  of ',': return lex.makeToken(tkComma, startLoc, startPos)
+  of ';': return lex.makeToken(tkSemicolon, startLoc, startPos)
+  of '@': return lex.makeToken(tkAt, startLoc, startPos)
+  of '?': return lex.makeToken(tkQuestion, startLoc, startPos)
+  of '~': return lex.makeToken(tkTilde, startLoc, startPos)
+  of ':': check2(':', tkColonColon, tkColon)
+  of '.':
+    if lex.peek() == '.' and lex.peek(1) == '.':
+      discard lex.advance()
+      discard lex.advance()
+      return lex.makeToken(tkDotDotDot, startLoc, startPos)
+    elif lex.peek() == '.' and lex.peek(1) == '=':
+      discard lex.advance()
+      discard lex.advance()
+      return lex.makeToken(tkDotDotEqual, startLoc, startPos)
+    elif lex.peek() == '.':
+      discard lex.advance()
+      return lex.makeToken(tkDotDot, startLoc, startPos)
+    else:
+      return lex.makeToken(tkDot, startLoc, startPos)
+  of '-':
+    if lex.peek() == '>':
+      discard lex.advance()
+      return lex.makeToken(tkArrow, startLoc, startPos)
+    elif lex.peek() == '-':
+      discard lex.advance()
+      return lex.makeToken(tkMinusMinus, startLoc, startPos)
+    elif lex.peek() == '=':
+      discard lex.advance()
+      return lex.makeToken(tkMinusAssign, startLoc, startPos)
+    else:
+      return lex.makeToken(tkMinus, startLoc, startPos)
+  of '+':
+    if lex.peek() == '+':
+      discard lex.advance()
+      return lex.makeToken(tkPlusPlus, startLoc, startPos)
+    elif lex.peek() == '=':
+      discard lex.advance()
+      return lex.makeToken(tkPlusAssign, startLoc, startPos)
+    else:
+      return lex.makeToken(tkPlus, startLoc, startPos)
+  of '*':
+    if lex.peek() == '*':
+      discard lex.advance()
+      return lex.makeToken(tkStarStar, startLoc, startPos)
+    elif lex.peek() == '=':
+      discard lex.advance()
+      return lex.makeToken(tkStarAssign, startLoc, startPos)
+    else:
+      return lex.makeToken(tkStar, startLoc, startPos)
+  of '/':
+    if lex.peek() == '=':
+      discard lex.advance()
+      return lex.makeToken(tkSlashAssign, startLoc, startPos)
+    else:
+      return lex.makeToken(tkSlash, startLoc, startPos)
+  of '%':
+    if lex.peek() == '=':
+      discard lex.advance()
+      return lex.makeToken(tkPercentAssign, startLoc, startPos)
+    else:
+      return lex.makeToken(tkPercent, startLoc, startPos)
+  of '=':
+    if lex.peek() == '=':
+      discard lex.advance()
+      return lex.makeToken(tkEq, startLoc, startPos)
+    elif lex.peek() == '>':
+      discard lex.advance()
+      return lex.makeToken(tkFatArrow, startLoc, startPos)
+    else:
+      return lex.makeToken(tkAssign, startLoc, startPos)
+  of '!':
+    if lex.peek() == '=':
+      discard lex.advance()
+      return lex.makeToken(tkNe, startLoc, startPos)
+    else:
+      return lex.makeToken(tkBang, startLoc, startPos)
+  of '<':
+    if lex.peek() == '=':
+      discard lex.advance()
+      return lex.makeToken(tkLe, startLoc, startPos)
+    elif lex.peek() == '<':
+      discard lex.advance()
+      if lex.peek() == '=':
+        discard lex.advance()
+        return lex.makeToken(tkShlAssign, startLoc, startPos)
+      return lex.makeToken(tkShl, startLoc, startPos)
+    else:
+      return lex.makeToken(tkLt, startLoc, startPos)
+  of '>':
+    if lex.peek() == '=':
+      discard lex.advance()
+      return lex.makeToken(tkGe, startLoc, startPos)
+    elif lex.peek() == '>':
+      discard lex.advance()
+      if lex.peek() == '=':
+        discard lex.advance()
+        return lex.makeToken(tkShrAssign, startLoc, startPos)
+      return lex.makeToken(tkShr, startLoc, startPos)
+    else:
+      return lex.makeToken(tkGt, startLoc, startPos)
+  of '&':
+    if lex.peek() == '&':
+      discard lex.advance()
+      return lex.makeToken(tkAmpAmp, startLoc, startPos)
+    elif lex.peek() == '=':
+      discard lex.advance()
+      return lex.makeToken(tkAmpAssign, startLoc, startPos)
+    else:
+      return lex.makeToken(tkAmp, startLoc, startPos)
+  of '|':
+    if lex.peek() == '|':
+      discard lex.advance()
+      return lex.makeToken(tkPipePipe, startLoc, startPos)
+    elif lex.peek() == '=':
+      discard lex.advance()
+      return lex.makeToken(tkPipeAssign, startLoc, startPos)
+    else:
+      return lex.makeToken(tkPipe, startLoc, startPos)
+  of '^':
+    if lex.peek() == '=':
+      discard lex.advance()
+      return lex.makeToken(tkCaretAssign, startLoc, startPos)
+    else:
+      return lex.makeToken(tkCaret, startLoc, startPos)
+  of '#':
+    # Check for intrinsics: #line, #column, #file, #function, #date, #time, #module
+    let afterHash = lex.peek()
+    if afterHash == 'l' and lex.matchStr("line"):
+      return lex.makeToken(tkHashLine, startLoc, startPos)
+    elif afterHash == 'c' and lex.matchStr("column"):
+      return lex.makeToken(tkHashColumn, startLoc, startPos)
+    elif afterHash == 'f' and lex.peek(1) == 'i' and lex.peek(2) == 'l' and lex.peek(3) == 'e':
+      discard lex.matchStr("file")
+      return lex.makeToken(tkHashFile, startLoc, startPos)
+    elif afterHash == 'f' and lex.peek(1) == 'u' and lex.peek(2) == 'n':
+      discard lex.matchStr("function")
+      return lex.makeToken(tkHashFunction, startLoc, startPos)
+    elif afterHash == 'd' and lex.matchStr("date"):
+      return lex.makeToken(tkHashDate, startLoc, startPos)
+    elif afterHash == 't' and lex.matchStr("time"):
+      return lex.makeToken(tkHashTime, startLoc, startPos)
+    elif afterHash == 'm' and lex.matchStr("module"):
+      return lex.makeToken(tkHashModule, startLoc, startPos)
+    else:
+      return lex.makeToken(tkHash, startLoc, startPos)
+  else:
+    lex.emitError(startLoc, &"unexpected character '{c1}'")
+    return lex.makeToken(tkUnknown, startLoc, startPos)
+
+# ---------------------------------------------------------------------------
+# Main scanning loop
+# ---------------------------------------------------------------------------
+
+proc nextToken(lex: var Lexer): Token =
+  lex.skipWhitespace()
+  let startLoc = lex.currentLocation()
+  let startPos = lex.pos
+
+  if lex.isAtEnd():
+    return Token(kind: tkEndOfFile, text: "", loc: startLoc)
+
+  let c = lex.peek()
+
+  if c == '\n':
+    discard lex.advance()
+    return Token(kind: tkNewLine, text: "\n", loc: startLoc)
+
+  # String prefixes: c8" c16" c32" — must come before ident check
+  if c == 'c' and lex.peek(1) in {'8', '1', '3'}:
+    let d = lex.peek(1)
+    if d == '8' and lex.peek(2) == '"':
+      discard lex.advance()  # c
+      discard lex.advance()  # 8
+      return lex.scanString(startLoc, 2)
+    elif d == '1' and lex.peek(2) == '6' and lex.peek(3) == '"':
+      discard lex.advance()
+      discard lex.advance()
+      discard lex.advance()
+      return lex.scanString(startLoc, 3)
+    elif d == '3' and lex.peek(2) == '2' and lex.peek(3) == '"':
+      discard lex.advance()
+      discard lex.advance()
+      discard lex.advance()
+      return lex.scanString(startLoc, 3)
+
+  if c == '"':
+    return lex.scanString(startLoc, 0)
+
+  # Char prefixes: c8' c16' c32' — must come before ident check
+  if c == 'c' and lex.peek(1) in {'8', '1', '3'}:
+    let d = lex.peek(1)
+    if d == '8' and lex.peek(2) == '\'':
+      discard lex.advance()
+      discard lex.advance()
+      return lex.scanChar(startLoc, 2)
+    elif d == '1' and lex.peek(2) == '6' and lex.peek(3) == '\'':
+      discard lex.advance()
+      discard lex.advance()
+      discard lex.advance()
+      return lex.scanChar(startLoc, 3)
+    elif d == '3' and lex.peek(2) == '2' and lex.peek(3) == '\'':
+      discard lex.advance()
+      discard lex.advance()
+      discard lex.advance()
+      return lex.scanChar(startLoc, 3)
+
+  if c == '\'':
+    return lex.scanChar(startLoc, 0)
+
+  if isIdentStart(c):
+    return lex.scanIdent(startLoc)
+
+  if c in {'0'..'9'}:
+    return lex.scanNumber(startLoc)
+
+  return lex.scanSymbol(startLoc)
+
+proc tokenize*(lex: var Lexer): LexerResult =
+  while true:
+    let tok = lex.nextToken()
+    lex.tokens.add(tok)
+    if tok.kind == tkEndOfFile:
+      break
+  result = LexerResult(tokens: lex.tokens, diagnostics: lex.diagnostics)
+
+proc tokenize*(source, sourceName: string): LexerResult =
+  var lex = initLexer(source, sourceName)
+  result = lex.tokenize()
+
+proc dumpTokens*(res: LexerResult): string =
+  for tok in res.tokens:
+    result.add($tok & "\n")

src/main.nim

@@ -0,0 +1,6 @@
+import std/os
+import cli
+
+when isMainModule:
+  let args = commandLineParams()
+  quit(runCli(args))

src/manifest.nim

@@ -0,0 +1,79 @@
+import std/[strutils, os, tables]
+
+type
+  PackageType* = enum
+    ptExecutable
+    ptSharedLibrary
+    ptStaticLibrary
+    ptSource
+
+  Manifest* = object
+    name*: string
+    version*: string
+    pkgType*: PackageType
+    output*: string          ## from [Build] Output
+    dependencies*: OrderedTableRef[string, string]
+
+type
+  TomlValueKind = enum
+    tvkString, tvkTable
+  TomlValue = object
+    case kind*: TomlValueKind
+    of tvkString:
+      strVal*: string
+    of tvkTable:
+      tableVal*: OrderedTableRef[string, TomlValue]
+
+proc parseSimpleToml(path: string): OrderedTableRef[string, TomlValue] =
+  result = newOrderedTable[string, TomlValue]()
+  if not fileExists(path):
+    return result
+  let content = readFile(path)
+  var currentTable = ""
+  for rawLine in content.splitLines():
+    let line = rawLine.strip()
+    if line.len == 0 or line.startsWith("#"):
+      continue
+    if line.startsWith("[") and line.endsWith("]"):
+      currentTable = line[1 ..< ^1]
+      if not result.hasKey(currentTable):
+        result[currentTable] = TomlValue(kind: tvkTable, tableVal: newOrderedTable[string, TomlValue]())
+    else:
+      let eqIdx = line.find('=')
+      if eqIdx >= 0:
+        let key = line[0 ..< eqIdx].strip()
+        var val = line[eqIdx + 1 .. ^1].strip()
+        # Remove surrounding quotes
+        if val.len >= 2 and val[0] == '"' and val[^1] == '"':
+          val = val[1 ..< ^1]
+        if currentTable != "" and result.hasKey(currentTable):
+          result[currentTable].tableVal[key] = TomlValue(kind: tvkString, strVal: val)
+        else:
+          result[key] = TomlValue(kind: tvkString, strVal: val)
+
+proc loadManifest*(path: string): Manifest =
+  let data = parseSimpleToml(path)
+  result.name = ""
+  result.version = "0.1.0"
+  result.pkgType = ptExecutable
+  result.output = "Bin"
+  result.dependencies = newOrderedTable[string, string]()
+
+  if data.hasKey("Package"):
+    let pkg = data["Package"].tableVal
+    if pkg.hasKey("Name"):
+      result.name = pkg["Name"].strVal
+    if pkg.hasKey("Version"):
+      result.version = pkg["Version"].strVal
+    if pkg.hasKey("Type"):
+      case pkg["Type"].strVal.toLowerAscii()
+      of "bin", "executable": result.pkgType = ptExecutable
+      of "shared", "sharedlibrary", "dll", "so", "dylib": result.pkgType = ptSharedLibrary
+      of "static", "staticlibrary", "lib", "a": result.pkgType = ptStaticLibrary
+      of "source": result.pkgType = ptSource
+      else: result.pkgType = ptExecutable
+
+  if data.hasKey("Build"):
+    let bld = data["Build"].tableVal
+    if bld.hasKey("Output"):
+      result.output = bld["Output"].strVal

src/source_location.nim

@@ -0,0 +1,8 @@
+type
+  SourceLocation* = object
+    line*: uint32      ## 1-based
+    column*: uint32    ## 1-based (UTF-8 byte offset in line)
+    offset*: uint32    ## byte offset from start of file
+
+proc `$`*(loc: SourceLocation): string =
+  $loc.line & ":" & $loc.column

src/token.nim

@@ -0,0 +1,300 @@
+import source_location
+
+type
+  TokenKind* = enum
+    ##Literals
+    tkIntLiteral      # 42  0xFF  0b1010  0o77
+    tkFloatLiteral    # 3.14  1.0e-9
+    tkStringLiteral   # "hello"  c8"hello"  c16"hello"  c32"hello"
+    tkCharLiteral     # 'A'  c8'A'  c16'A'  c32'A'
+    tkBoolLiteral     # true  false
+
+    ##Identifiers
+    tkIdent           # foo  Bar  _x
+
+    ##Control flow keywords
+    tkIf              # if
+    tkElse            # else
+    tkWhile           # while
+    tkDo              # do
+    tkLoop            # loop
+    tkFor             # for
+    tkIn              # in
+    tkBreak           # break
+    tkContinue        # continue
+    tkReturn          # return
+    tkMatch           # match
+
+    ##Declaration keywords
+    tkFunc            # func
+    tkLet             # let
+    tkVar             # var
+    tkConst           # const
+    tkType            # type
+    tkStruct          # struct
+    tkEnum            # enum
+    tkUnion           # union
+    tkInterface       # interface
+    tkExtend          # extend
+    tkModule          # module
+    tkImport          # import
+    tkPub             # pub
+    tkExtern          # extern
+
+    ##Other keywords
+    tkAs              # as
+    tkIs              # is
+    tkNull            # null
+    tkSelf            # self
+    tkSuper           # super
+
+    ##Punctuation
+    tkLParen          # (
+    tkRParen          # )
+    tkLBrace          # {
+    tkRBrace          # }
+    tkLBracket        # [
+    tkRBracket        # ]
+    tkComma           # ,
+    tkSemicolon       # ;
+    tkColon           # :
+    tkColonColon      # ::
+    tkDot             # .
+    tkDotDot          # ..
+    tkDotDotDot       # ...
+    tkDotDotEqual     # ..=
+    tkArrow           # ->
+    tkFatArrow        # =>
+    tkAt              # @
+    tkHash            # #
+    tkQuestion        # ?
+
+    ##Arithmetic operators
+    tkPlus            # +
+    tkMinus           # -
+    tkStar            # *
+    tkSlash           # /
+    tkPercent         # %
+    tkStarStar        # **
+    tkPlusPlus        # ++
+    tkMinusMinus      # --
+
+    ##Bitwise operators
+    tkAmp             # &
+    tkPipe            # |
+    tkCaret           # ^
+    tkTilde           # ~
+    tkShl             # <<
+    tkShr             # >>
+
+    ##Logical operators
+    tkAmpAmp          # &&
+    tkPipePipe        # ||
+    tkBang            # !
+
+    ##Comparison operators
+    tkEq              # ==
+    tkNe              # !=
+    tkLt              # <
+    tkLe              # <=
+    tkGt              # >
+    tkGe              # >=
+
+    ##Assignment operators
+    tkAssign          # =
+    tkPlusAssign      # +=
+    tkMinusAssign     # -=
+    tkStarAssign      # *=
+    tkSlashAssign     # /=
+    tkPercentAssign   # %=
+    tkAmpAssign       # &=
+    tkPipeAssign      # |=
+    tkCaretAssign     # ^=
+    tkShlAssign       # <<=
+    tkShrAssign       # >>=
+
+    ##Compile-time intrinsics
+    tkHashLine        # #line
+    tkHashColumn      # #column
+    tkHashFile        # #file
+    tkHashFunction    # #function
+    tkHashDate        # #date
+    tkHashTime        # #time
+    tkHashModule      # #module
+
+    ##Special
+    tkNewLine         # significant newline (if grammar uses them)
+    tkEndOfFile       # end of file
+    tkUnknown         # unrecognized character
+
+  Token* = object
+    kind*: TokenKind
+    text*: string       # original source spelling
+    loc*: SourceLocation
+
+proc isKeyword*(kind: TokenKind): bool =
+  case kind
+  of tkIf, tkElse, tkWhile, tkDo, tkLoop, tkFor, tkIn,
+     tkBreak, tkContinue, tkReturn, tkMatch,
+     tkFunc, tkLet, tkVar, tkConst, tkType, tkStruct, tkEnum,
+     tkUnion, tkInterface, tkExtend, tkModule, tkImport,
+     tkPub, tkExtern, tkAs, tkIs, tkNull, tkSelf, tkSuper:
+    true
+  else:
+    false
+
+proc isLiteral*(kind: TokenKind): bool =
+  case kind
+  of tkIntLiteral, tkFloatLiteral, tkStringLiteral, tkCharLiteral, tkBoolLiteral:
+    true
+  else:
+    false
+
+proc isOperator*(kind: TokenKind): bool =
+  case kind
+  of tkPlus..tkShrAssign:
+    true
+  else:
+    false
+
+proc isEof*(kind: TokenKind): bool = kind == tkEndOfFile
+
+proc keywordKind*(text: string): TokenKind =
+  case text
+  of "func": tkFunc
+  of "let": tkLet
+  of "var": tkVar
+  of "const": tkConst
+  of "type": tkType
+  of "struct": tkStruct
+  of "enum": tkEnum
+  of "union": tkUnion
+  of "interface": tkInterface
+  of "extend": tkExtend
+  of "module": tkModule
+  of "import": tkImport
+  of "pub": tkPub
+  of "extern": tkExtern
+  of "if": tkIf
+  of "else": tkElse
+  of "while": tkWhile
+  of "do": tkDo
+  of "loop": tkLoop
+  of "for": tkFor
+  of "in": tkIn
+  of "break": tkBreak
+  of "continue": tkContinue
+  of "return": tkReturn
+  of "match": tkMatch
+  of "as": tkAs
+  of "is": tkIs
+  of "null": tkNull
+  of "self": tkSelf
+  of "super": tkSuper
+  of "true", "false": tkBoolLiteral
+  else: tkIdent
+
+proc tokenKindName*(kind: TokenKind): string =
+  case kind
+  of tkIntLiteral: "integer literal"
+  of tkFloatLiteral: "float literal"
+  of tkStringLiteral: "string literal"
+  of tkCharLiteral: "char literal"
+  of tkBoolLiteral: "boolean literal"
+  of tkIdent: "identifier"
+  of tkIf: "'if'"
+  of tkElse: "'else'"
+  of tkWhile: "'while'"
+  of tkDo: "'do'"
+  of tkLoop: "'loop'"
+  of tkFor: "'for'"
+  of tkIn: "'in'"
+  of tkBreak: "'break'"
+  of tkContinue: "'continue'"
+  of tkReturn: "'return'"
+  of tkMatch: "'match'"
+  of tkFunc: "'func'"
+  of tkLet: "'let'"
+  of tkVar: "'var'"
+  of tkConst: "'const'"
+  of tkType: "'type'"
+  of tkStruct: "'struct'"
+  of tkEnum: "'enum'"
+  of tkUnion: "'union'"
+  of tkInterface: "'interface'"
+  of tkExtend: "'extend'"
+  of tkModule: "'module'"
+  of tkImport: "'import'"
+  of tkPub: "'pub'"
+  of tkExtern: "'extern'"
+  of tkAs: "'as'"
+  of tkIs: "'is'"
+  of tkNull: "'null'"
+  of tkSelf: "'self'"
+  of tkSuper: "'super'"
+  of tkLParen: "'('"
+  of tkRParen: "')'"
+  of tkLBrace: "'{'"
+  of tkRBrace: "'}'"
+  of tkLBracket: "'['"
+  of tkRBracket: "']'"
+  of tkComma: "','"
+  of tkSemicolon: "';'"
+  of tkColon: "':'"
+  of tkColonColon: "'::'"
+  of tkDot: "'.'"
+  of tkDotDot: "'..'"
+  of tkDotDotDot: "'...'"
+  of tkDotDotEqual: "'..='"
+  of tkArrow: "'->'"
+  of tkFatArrow: "'=>'"
+  of tkAt: "'@'"
+  of tkHash: "'#'"
+  of tkQuestion: "'?'"
+  of tkPlus: "'+'"
+  of tkMinus: "'-'"
+  of tkStar: "'*'"
+  of tkSlash: "'/'"
+  of tkPercent: "'%'"
+  of tkStarStar: "'**'"
+  of tkPlusPlus: "'++'"
+  of tkMinusMinus: "'--'"
+  of tkAmp: "'&'"
+  of tkPipe: "'|'"
+  of tkCaret: "'^'"
+  of tkTilde: "'~'"
+  of tkShl: "'<<'"
+  of tkShr: "'>>'"
+  of tkAmpAmp: "'&&'"
+  of tkPipePipe: "'||'"
+  of tkBang: "'!'"
+  of tkEq: "'=='"
+  of tkNe: "'!='"
+  of tkLt: "'<'"
+  of tkLe: "'<='"
+  of tkGt: "'>'"
+  of tkGe: "'>='"
+  of tkAssign: "'='"
+  of tkPlusAssign: "'+='"
+  of tkMinusAssign: "'-='"
+  of tkStarAssign: "'*='"
+  of tkSlashAssign: "'/='"
+  of tkPercentAssign: "'%='"
+  of tkAmpAssign: "'&='"
+  of tkPipeAssign: "'|='"
+  of tkCaretAssign: "'^='"
+  of tkShlAssign: "'<<='"
+  of tkShrAssign: "'>>='"
+  of tkHashLine: "'#line'"
+  of tkHashColumn: "'#column'"
+  of tkHashFile: "'#file'"
+  of tkHashFunction: "'#function'"
+  of tkHashDate: "'#date'"
+  of tkHashTime: "'#time'"
+  of tkHashModule: "'#module'"
+  of tkNewLine: "newline"
+  of tkEndOfFile: "end of file"
+  of tkUnknown: "unknown token"
+
+proc `$`*(tok: Token): string =
+  result = tokenKindName(tok.kind) & " '" & tok.text & "' @ " & $tok.loc

tests/lexer_test.nim

@@ -0,0 +1,124 @@
+import std/[unittest, os, strutils]
+import ../src/[lexer, token, source_location]
+
+proc tokenKinds(source: string): seq[TokenKind] =
+  let res = tokenize(source, "<test>")
+  for t in res.tokens:
+    result.add(t.kind)
+
+proc tokenTexts(source: string): seq[string] =
+  let res = tokenize(source, "<test>")
+  for t in res.tokens:
+    result.add(t.text)
+
+suite "Lexer":
+  test "empty source":
+    let res = tokenize("", "<test>")
+    check res.tokens.len == 1
+    check res.tokens[0].kind == tkEndOfFile
+    check not res.hasErrors
+
+  test "simple identifiers":
+    let kinds = tokenKinds("foo bar _x Baz123")
+    check kinds == @[tkIdent, tkIdent, tkIdent, tkIdent, tkEndOfFile]
+
+  test "keywords":
+    let kinds = tokenKinds("func let var if else while for return match struct enum")
+    check kinds == @[tkFunc, tkLet, tkVar, tkIf, tkElse, tkWhile, tkFor, tkReturn, tkMatch, tkStruct, tkEnum, tkEndOfFile]
+
+  test "bool literals":
+    let kinds = tokenKinds("true false")
+    check kinds == @[tkBoolLiteral, tkBoolLiteral, tkEndOfFile]
+    let texts = tokenTexts("true false")
+    check texts == @["true", "false", ""]
+
+  test "integers":
+    let kinds = tokenKinds("42 0xFF 0b1010 0o77")
+    check kinds == @[tkIntLiteral, tkIntLiteral, tkIntLiteral, tkIntLiteral, tkEndOfFile]
+
+  test "floats":
+    let kinds = tokenKinds("3.14 1.0e-9 2.5E+3")
+    check kinds == @[tkFloatLiteral, tkFloatLiteral, tkFloatLiteral, tkEndOfFile]
+
+  test "operators":
+    let kinds = tokenKinds("+ - * / % ** ++ --")
+    check kinds == @[tkPlus, tkMinus, tkStar, tkSlash, tkPercent, tkStarStar, tkPlusPlus, tkMinusMinus, tkEndOfFile]
+
+  test "comparison operators":
+    let kinds = tokenKinds("== != < <= > >=")
+    check kinds == @[tkEq, tkNe, tkLt, tkLe, tkGt, tkGe, tkEndOfFile]
+
+  test "assignment operators":
+    let kinds = tokenKinds("= += -= *= /= %= &= |= ^= <<= >>=")
+    check kinds == @[tkAssign, tkPlusAssign, tkMinusAssign, tkStarAssign, tkSlashAssign, tkPercentAssign, tkAmpAssign, tkPipeAssign, tkCaretAssign, tkShlAssign, tkShrAssign, tkEndOfFile]
+
+  test "punctuation":
+    let kinds = tokenKinds("( ) { } [ ] , ; : :: . .. ... ..= -> => @ # ?")
+    check kinds == @[tkLParen, tkRParen, tkLBrace, tkRBrace, tkLBracket, tkRBracket, tkComma, tkSemicolon, tkColon, tkColonColon, tkDot, tkDotDot, tkDotDotDot, tkDotDotEqual, tkArrow, tkFatArrow, tkAt, tkHash, tkQuestion, tkEndOfFile]
+
+  test "logical operators":
+    let kinds = tokenKinds("&& || !")
+    check kinds == @[tkAmpAmp, tkPipePipe, tkBang, tkEndOfFile]
+
+  test "bitwise operators":
+    let kinds = tokenKinds("& | ^ ~ << >>")
+    check kinds == @[tkAmp, tkPipe, tkCaret, tkTilde, tkShl, tkShr, tkEndOfFile]
+
+  test "string literals":
+    let kinds = tokenKinds("\"hello\" c8\"world\"")
+    check kinds == @[tkStringLiteral, tkStringLiteral, tkEndOfFile]
+
+  test "char literals":
+    let kinds = tokenKinds("'A' c8'B'")
+    check kinds == @[tkCharLiteral, tkCharLiteral, tkEndOfFile]
+
+  test "line comments are skipped":
+    let kinds = tokenKinds("let x = 42 // this is a comment")
+    check kinds == @[tkLet, tkIdent, tkAssign, tkIntLiteral, tkEndOfFile]
+
+  test "block comments are skipped":
+    let kinds = tokenKinds("let /* inline comment */ x = 42")
+    check kinds == @[tkLet, tkIdent, tkAssign, tkIntLiteral, tkEndOfFile]
+
+  test "nested block comments":
+    let kinds = tokenKinds("let /* outer /* inner */ still outer */ x = 42")
+    check kinds == @[tkLet, tkIdent, tkAssign, tkIntLiteral, tkEndOfFile]
+
+  test "intrinsics":
+    let kinds = tokenKinds("#line #column #file #function #date #time #module")
+    check kinds == @[tkHashLine, tkHashColumn, tkHashFile, tkHashFunction, tkHashDate, tkHashTime, tkHashModule, tkEndOfFile]
+
+  test "newline tokens":
+    let kinds = tokenKinds("let x = 42\nlet y = 10")
+    check kinds == @[tkLet, tkIdent, tkAssign, tkIntLiteral, tkNewLine, tkLet, tkIdent, tkAssign, tkIntLiteral, tkEndOfFile]
+
+  test "paths":
+    let kinds = tokenKinds("Std::Io::PrintLine")
+    check kinds == @[tkIdent, tkColonColon, tkIdent, tkColonColon, tkIdent, tkEndOfFile]
+
+  test "unterminated string error":
+    let res = tokenize("\"hello", "<test>")
+    check res.hasErrors
+
+  test "unterminated char error":
+    let res = tokenize("'a", "<test>")
+    check res.hasErrors
+
+  test "escape sequences in string":
+    let res = tokenize("\"hello\\nworld\\t!\"", "<test>")
+    check not res.hasErrors
+    check res.tokens[0].kind == tkStringLiteral
+
+  test "full function":
+    let src = "func Main() -> int {\n    let x: int32 = 42;\n    return x;\n}\n"
+    let res = tokenize(src, "<test>")
+    check not res.hasErrors
+    check res.tokens[0].kind == tkFunc
+    check res.tokens[1].kind == tkIdent
+    check res.tokens[1].text == "Main"
+
+  test "dumpTokens produces output":
+    let res = tokenize("let x = 42", "<test>")
+    let dump = dumpTokens(res)
+    check "let" in dump
+    check "42" in dump

tests/testdata/sample.bux

@@ -0,0 +1,11 @@
+import Std::Io::PrintLine;
+
+func Add(a: int32, b: int32) -> int32 {
+    return a + b;
+}
+
+func Main() -> int {
+    let result: int32 = Add(10, 20);
+    PrintLine(c8"Hello from Bux!");
+    return 0;
+}