commit 1b708ec75552e62cbf4e7aa724d6f005922f54c8 Author: dimgigov Date: Sat May 30 21:01:26 2026 +0300 feat: bootstrap skeleton + lexer (Phase 0) diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..fdb156f --- /dev/null +++ b/.gitignore @@ -0,0 +1,13 @@ +# Compiled binary +buxc + +# Nim cache +nimcache/ + +# Test binaries +tests/lexer_test +tests/*.exe + +# Build artifacts +build/ +*.o diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..de398b7 --- /dev/null +++ b/Makefile @@ -0,0 +1,26 @@ +NIM := nim +SRC := src/main.nim +OUT := buxc +BUILD_DIR := build + +.PHONY: all build test clean + +all: build + +build: + $(NIM) c -o:$(OUT) -d:release $(SRC) + +dev: + $(NIM) c -o:$(OUT) $(SRC) + +test: build + @echo "Running lexer tests..." + $(NIM) c -r tests/lexer_test.nim + @echo "Running integration tests..." + ./$(OUT) new _test_tmp_pkg + ./$(OUT) --version + +clean: + rm -f $(OUT) + rm -rf $(BUILD_DIR) + rm -rf nimcache diff --git a/PLAN.md b/PLAN.md new file mode 100644 index 0000000..c5d1156 --- /dev/null +++ b/PLAN.md @@ -0,0 +1,273 @@ +# Bux Programming Language — Roadmap to Self-Hosting + +> **Reference:** [Rux Language](https://rux-lang.dev/) | [Rux Source](../_rux/) +> **Bootstrap Implementation:** Nim +> **Target:** Bux compiler written in Bux (self-hosting) + +--- + +## Overview + +Bux is a fast, compiled, strongly-typed, multi-paradigm systems programming language inspired by Rux. The strategy is **bootstrap via Nim** — we build the first Bux compiler in Nim, then progressively rewrite it in Bux until it compiles itself. + +--- + +## Phase 0 — Bootstrap Foundation (Week 1-2) + +**Goal:** Working Nim project that can lex, parse, and dump a Bux AST. + +| Task | Details | +|------|---------| +| `0.1` Project skeleton | `buxc` CLI in Nim, `bux.toml` manifest parser | +| `0.2` Token model | All Rux tokens (`TokenKind`, `SourceLocation`, literal suffixes) | +| `0.3` Lexer | UTF-8 source, identifiers, numbers (dec/hex/bin/oct), strings (`c8""`, `c16""`, `c32""`), chars, operators, nested `/* */`, `//` comments, intrinsics (`#line`, `#file`, etc.) | +| `0.4` CLI commands | `bux new`, `bux init`, `bux build`, `bux run`, `bux check` | +| `0.5` Test harness | Golden-file tests for lexer output (`.tokens`) | + +**Deliverable:** `echo 'let x = 42' | bux check` prints token stream. + +--- + +## Phase 1 — Frontend: Parser & AST (Week 3-4) + +**Goal:** Parse every construct present in Rux v0.2.0 into a Nim AST. + +| Task | Details | +|------|---------| +| `1.1` AST nodes | All `Expr`, `Stmt`, `Decl`, `Pattern`, `TypeExpr`, `Block` variants (see `_rux/Include/Rux/Ast.h`) | +| `1.2` Pratt parser | Full precedence climbing for all binary/unary/postfix operators including `**` (right-assoc) and range `..` / `..=` | +| `1.3` Declarations | `func`, `struct`, `enum`, `union`, `interface`, `extend`/`impl`, `module`, `const`, `type`, `extern`, `import`/`use` | +| `1.4` Statements | `let`/`var`, `if`/`else if`/`else`, `while`, `do while`, `loop`, `for in`, `match`, `return`, `break`/`continue` (with labels) | +| `1.5` Expressions | Literals, identifiers, paths (`a::b`), calls, index, field access, struct init, slice init `[a,b]`, tuple `(a,b)`, cast `as`, test `is`, ternary `? :`, block-expr `{ ... }` | +| `1.6` Patterns | Wildcard `_`, literal, ident, range, enum destructuring, struct destructuring, tuple, guarded `if` | +| `1.7` Attributes | `@[Import(lib: "...")]`, calling-convention, platform-conditional imports | +| `1.8` Error recovery | Synchronize on declaration/statement boundaries; emit multiple diagnostics | + +**Deliverable:** All `_rux/Tests/**/*.rux` files parse without error and produce `.ast` dumps. + +--- + +## Phase 2 — Semantic Analysis (Week 5-7) + +**Goal:** Type-check the AST and produce a typed symbol table. + +| Task | Details | +|------|---------| +| `2.1` Type model | `TypeRef` with primitives, pointers, slices, tuples, named types, type parameters, functions (see `_rux/Include/Rux/Type.h`) | +| `2.2` Scopes | Module scope, block scope, namespace resolution for `Std::Io::PrintLine` | +| `2.3` First pass | Collect global symbols (functions, structs, enums, unions, interfaces, consts, type aliases, imports) | +| `2.4` Type checking | Expression typing, operator overload resolution per Rux rules, assignment compatibility | +| `2.5` Name resolution | Resolve identifiers, paths, `self`, `super`; report undeclared / ambiguous names | +| `2.6` Interface conformance | Check that `extend T for I` provides all required methods; build vtable map | +| `2.7` Generics (basic) | Monomorphization of generic structs and functions at call sites | +| `2.8` Diagnostics | Multi-file error messages with source locations | + +**Deliverable:** `bux check` rejects ill-typed programs and passes `Tests/Echo`, `Tests/Io`, `Tests/Pow` type-checking. + +--- + +## Phase 3 — High-Level IR (HIR) (Week 8) + +**Goal:** Lower AST to a simplified, fully-typed HIR. + +| Task | Details | +|------|---------| +| `3.1` HIR nodes | Desugared equivalents of AST nodes (see `_rux/Include/Rux/Hir.h`) | +| `3.2` Lowering | Desugar `for` → `while`+iterator, `match` → decision tree, method calls to explicit receiver calls | +| `3.3` Constant folding | Evaluate `const` and simple compile-time expressions | +| `3.4` Interface lowering | Convert interface values to fat pointers `{data_ptr, vtable_ptr}`; generate vtable labels | + +**Deliverable:** HIR dump matches Rux HIR semantics for sample programs. + +--- + +## Phase 4 — Low-Level IR (LIR) (Week 9-10) + +**Goal:** Generate SSA-like LIR with virtual registers and basic blocks. + +| Task | Details | +|------|---------| +| `4.1` LIR model | `LirInstr`, `LirBlock`, `LirTerminator`, `LirFunc`, `LirReg`, opcodes (`Const`, `Alloca`, `Load`, `Store`, arithmetic, `Call`, `Phi`, `GlobalAddr`, etc.) | +| `4.2` Control flow | Lower `if`, `while`, `loop`, `match` to blocks with `Jump` / `Branch` / `Switch` terminators | +| `4.3` Memory | Stack allocation (`alloca`), pointer arithmetic, field/index pointer computation | +| `4.4` Calls | Direct calls, indirect calls, extern calls with correct ABI marking (System V / Win64) | + +**Deliverable:** `bux build --emit-lir` produces readable LIR for all test programs. + +--- + +## Phase 5 — Backend & Code Generation (Week 11-14) + +**Strategy:** Two backends in parallel — a **C transpiler** for instant portability and a **native x86-64** backend for performance. + +### 5A — C Transpiler (Primary bootstrap path) + +| Task | Details | +|------|---------| +| `5A.1` C emitter | Walk LIR and emit C11 code | +| `5A.2` Types to C | Bux primitives → C primitives; structs → C structs; enums → C enums + tagged unions; slices → `{T* data; size_t len;}` | +| `5A.3` Functions to C | Bux functions → C functions with `static` / `extern`; name mangling for overloads/generics | +| `5A.4` FFI | `extern` / `@[Import]` → `#include` + function declarations; link with system `cc` | +| `5A.5` Runtime shim | Small C runtime providing `bux_alloc`, `bux_print`, panic/abort for div-by-zero, etc. | +| `5A.6` Build integration | `bux build` invokes `cc` / `clang` / `gcc` automatically | + +**Deliverable:** `bux run` on `Tests/Io/Main.bux` prints "Hello from a Bux binary!". + +### 5B — Native x86-64 Backend (Secondary, for self-hosting speed) + +| Task | Details | +|------|---------| +| `5B.1` Assembly emitter | NASM-syntax text output (like Rux `Asm`) | +| `5B.2` Register allocation | Naive stack-spill allocator first; later linear-scan | +| `5B.3` ABI lowering | System V AMD64 ABI (Linux/macOS) and Win64 ABI (Windows) | +| `5B.4` Object format | Emit ELF64 (Linux), Mach-O (macOS), PE/COFF (Windows) — or use `nasm` + system linker | +| `5B.5` Custom linker (optional) | `.bcu` (Bux Compiled Unit) format + bespoke linker à la Rux `.rcu` | + +**Deliverable:** `bux build --backend=native` produces working Linux x86-64 binary. + +--- + +## Phase 6 — Standard Library (Week 15-18) + +**Goal:** Enough stdlib to write the compiler in Bux. + +| Module | Requirements | +|--------|-------------| +| `Std::Io` | `Print`, `PrintLine`, `ReadLine`, file read/write (wrap C stdio initially) | +| `Std::Memory` | `Alloc`, `Free`, `Realloc` (wrap `malloc`/`free`) | +| `Std::String` | Basic string builder, concatenation, slicing | +| `Std::Array` | Dynamic array (`Vec` equivalent): `push`, `pop`, `get`, `len`, `capacity` | +| `Std::Map` | Hash map with string keys (needed for symbol tables) | +| `Std::Math` | `Sqrt`, `Pow`, `Min`, `Max`, `Abs` | +| `Std::Os` | `Args`, `Env`, `Exit`, `Cwd` | +| `Std::Path` | Path joining, extension splitting | +| `Std::Process` | Spawn subprocess, read stdout/stderr | + +**Deliverable:** Can write a non-trivial CLI tool (e.g., a file copier or a basic grep) entirely in Bux. + +--- + +## Phase 7 — Self-Hosting: The Great Rewrite (Week 19-26) + +**Goal:** Bux compiler compiles itself. This is the **main milestone**. + +| Task | Details | +|------|---------| +| `7.1` Port lexer | Rewrite `lexer.nim` → `Lexer.bux` | +| `7.2` Port parser | Rewrite `parser.nim` → `Parser.bux` | +| `7.3` Port sema | Rewrite `sema.nim` → `Sema.bux` | +| `7.4` Port HIR | Rewrite `hir.nim` → `Hir.bux` | +| `7.5` Port LIR | Rewrite `lir.nim` → `Lir.bux` | +| `7.6` Port C backend | Rewrite `c_backend.nim` → `CBackend.bux` | +| `7.7` Port CLI | Rewrite `main.nim` → `Main.bux` | +| `7.8` Dogfooding | Use `buxc` (Nim) to build `buxc2` (Bux). Then use `buxc2` to build `buxc3`. Compare bit-for-bit. | +| `7.9` Fix bootstrap loop | Once `buxc2 == buxc3`, we are self-hosted. Freeze Nim version as reference. | + +**Deliverable:** `make selfhost` succeeds; Bux compiler is written entirely in Bux. + +--- + +## Phase 8 — Ecosystem & Tooling (Week 27+) + +| Task | Details | +|------|---------| +| `8.1` Package manager | `bux add`, `bux remove`, `bux update`, `bux install` with lockfile | +| `8.2` Registry protocol | Simple HTTP git-based registry (like Go modules or Cargo) | +| `8.3` Formatter | `bux fmt` — auto-format Bux source | +| `8.4` LSP | Language Server Protocol for autocomplete, hover, go-to-definition | +| `8.5` Tests | `bux test` runner with assertions and golden tests | +| `8.6` Documentation | `bux doc` — generate HTML from `///` doc comments | +| `8.7` Cross-compilation | `--target` flag leveraging C backend portability | + +--- + +## File Structure (Target) + +``` +bux/ +├── bux.toml # Compiler package manifest +├── README.md +├── PLAN.md +├── Makefile # build, test, selfhost +├── src/ +│ ├── Main.bux # CLI entry point +│ ├── Lexer.bux +│ ├── Parser.bux +│ ├── Ast.bux +│ ├── Sema.bux +│ ├── Type.bux +│ ├── Hir.bux +│ ├── Lir.bux +│ ├── CBackend.bux # C transpiler (primary backend) +│ ├── X64Backend.bux # Native x86-64 backend (optional) +│ ├── Linker.bux # Custom linker / build driver +│ ├── Manifest.bux # bux.toml parser +│ └── Package.bux # Package resolution +├── stdlib/ +│ ├── Std/ +│ │ ├── Io.bux +│ │ ├── Memory.bux +│ │ ├── String.bux +│ │ ├── Array.bux +│ │ ├── Map.bux +│ │ ├── Math.bux +│ │ ├── Os.bux +│ │ ├── Path.bux +│ │ └── Process.bux +│ └── Runtime.c # C runtime shim +├── tests/ +│ ├── Lexer/ +│ ├── Parser/ +│ ├── Sema/ +│ ├── Codegen/ +│ └── Integration/ +└── docs/ + └── LanguageRef.md +``` + +--- + +## Language Design Decisions (Bux vs Rux) + +| Feature | Bux Decision | Rationale | +|---------|-------------|-----------| +| **Backend (bootstrap)** | C transpiler first | Fastest path to working compiler; leverages existing C toolchains | +| **Backend (final)** | Native x86-64 + optional LLVM | Match Rux ambition; self-hosting needs speed | +| **Memory safety** | Raw pointers + optional borrow checker (Phase 9) | Match Rux current model; gradual safety | +| **Generics** | Monomorphization only | Simpler than Rust-style trait objects; enough for self-hosting | +| **Error handling** | Explicit `Result` + `?` operator (later) | Start with C-style returns; add sugar after self-hosting | +| **String literals** | `c8""`, `c16""`, `c32""` + `""` defaulting to `c8` | Same as Rux | +| **Build system** | `bux.toml` (same as `Rux.toml`) | Compatible manifest format | +| **Module system** | `import Std::Io::PrintLine` | Same as Rux path syntax | + +--- + +## Milestones Summary + +| Milestone | Phase | Success Criteria | +|-----------|-------|------------------| +| **M0** | 0 | `bux check` lexes source | +| **M1** | 1 | All Rux test files parse | +| **M2** | 2 | Type-checker rejects invalid programs | +| **M3** | 3+4 | LIR emits for all constructs | +| **M4** | 5A | `bux run` produces working binary via C | +| **M5** | 6 | Can write compiler-adjacent tools in Bux | +| **M6** | 7 | **Self-hosted**: Bux compiler builds itself | +| **M7** | 8 | Package manager + LSP + formatter shipped | + +--- + +## Risk Mitigation + +| Risk | Mitigation | +|------|------------| +| Nim bootstrap too slow | Keep Nim code simple; aim for rewrite in ~3 months | +| C backend limits performance | Maintain parallel native backend; C is only bootstrap | +| Generics get complex | Restrict to monomorphization; no higher-kinded types | +| Self-hosting too hard | Ensure stdlib has `Array`, `Map`, `String` before starting rewrite | + +--- + +## Next Immediate Step + +Create the Nim bootstrap skeleton and implement the **Lexer** (`0.1`–`0.3`). diff --git a/README.md b/README.md new file mode 100644 index 0000000..8702791 --- /dev/null +++ b/README.md @@ -0,0 +1,38 @@ +# Bux Programming Language + +> **Status:** Bootstrap phase — compiler written in Nim, targeting self-hosting. + +Bux is a fast, compiled, strongly-typed systems programming language inspired by [Rux](https://rux-lang.dev/). The long-term goal is a self-hosted compiler with a minimal runtime, native x86-64 backend, and modern tooling. + +## Quick Start + +```bash +# Build the bootstrap compiler (Nim) +make build + +# Create a new project +bux new hello + +# Build and run +bux run +``` + +## Syntax Preview + +```bux +import Std::Io::PrintLine; + +func Main() -> int { + let message: *char8 = c8"Hello, Bux!"; + PrintLine(message); + return 0; +} +``` + +## Roadmap + +See [`PLAN.md`](PLAN.md) for the full roadmap to self-hosting. + +## License + +MIT diff --git a/bux.toml b/bux.toml new file mode 100644 index 0000000..8eee9db --- /dev/null +++ b/bux.toml @@ -0,0 +1,7 @@ +[Package] +Name = "buxc" +Version = "0.1.0" +Type = "bin" + +[Build] +Output = "Bin" diff --git a/src/cli.nim b/src/cli.nim new file mode 100644 index 0000000..486491c --- /dev/null +++ b/src/cli.nim @@ -0,0 +1,234 @@ +import std/[os, strutils, terminal, strformat] +import lexer, manifest + +type + ColorMode* = enum + cmAuto + cmOn + cmOff + + GlobalOptions* = object + color*: ColorMode + quiet*: bool + verbose*: bool + +proc printUsage*() = + echo """Bux Programming Language (bootstrap compiler) + +Usage: bux [options] [command-options] + +Commands: + new Create a new Bux package + init Initialize a Bux package in the current directory + build Build the current package + run Build and run the current package + check Type-check the current package + clean Remove build artifacts + help Show this help message + version Show version + +Global options: + --color Control colored output (default: auto) + -q, --quiet Suppress non-error output + -v, --verbose Verbose output +""" + +proc parseGlobalOptions(args: seq[string]): tuple[opts: GlobalOptions, rest: seq[string], ok: bool] = + result.opts = GlobalOptions(color: cmAuto, quiet: false, verbose: false) + result.rest = @[] + result.ok = true + var i = 0 + while i < args.len: + let arg = args[i] + if arg == "--color": + if i + 1 >= args.len: + stderr.writeLine("error: --color requires an argument") + result.ok = false + return + inc i + case args[i].toLowerAscii() + of "auto": result.opts.color = cmAuto + of "on": result.opts.color = cmOn + of "off": result.opts.color = cmOff + else: + stderr.writeLine(&"error: unknown --color value '{args[i]}'") + result.ok = false + return + elif arg == "-q" or arg == "--quiet": + result.opts.quiet = true + elif arg == "-v" or arg == "--verbose": + result.opts.verbose = true + else: + result.rest.add(arg) + inc i + +proc shouldUseColor(opts: GlobalOptions): bool = + case opts.color + of cmOn: true + of cmOff: false + of cmAuto: terminal.isatty(stdout) + +proc printError(msg: string, useColor: bool) = + if useColor: + stdout.setForegroundColor(fgRed) + stdout.write("error: ") + stdout.resetAttributes() + stdout.writeLine(msg) + else: + stderr.writeLine("error: " & msg) + +proc printInfo(msg: string, useColor: bool) = + if useColor: + stdout.setForegroundColor(fgCyan) + stdout.write("info: ") + stdout.resetAttributes() + stdout.writeLine(msg) + else: + echo("info: " & msg) + +# --------------------------------------------------------------------------- +# Commands +# --------------------------------------------------------------------------- + +proc cmdNew*(args: seq[string], opts: GlobalOptions): int = + let useColor = shouldUseColor(opts) + if args.len < 1: + printError("'new' requires a package name", useColor) + return 1 + let name = args[0] + let root = getCurrentDir() / name + if dirExists(root): + printError(&"directory '{name}' already exists", useColor) + return 1 + createDir(root / "src") + writeFile(root / "bux.toml", &"""[Package] +Name = "{name}" +Version = "0.1.0" +Type = "bin" + +[Build] +Output = "Bin" +""") + writeFile(root / "src" / "Main.bux", """import Std::Io::PrintLine; + +func Main() -> int { + PrintLine(c8"Hello, Bux!"); + return 0; +} +""") + if not opts.quiet: + printInfo(&"Created Bux package '{name}'", useColor) + return 0 + +proc cmdInit*(args: seq[string], opts: GlobalOptions): int = + let useColor = shouldUseColor(opts) + let root = getCurrentDir() + if fileExists(root / "bux.toml"): + printError("bux.toml already exists", useColor) + return 1 + let name = splitPath(root).tail + writeFile(root / "bux.toml", &"""[Package] +Name = "{name}" +Version = "0.1.0" +Type = "bin" + +[Build] +Output = "Bin" +""") + if not dirExists(root / "src"): + createDir(root / "src") + if not opts.quiet: + printInfo(&"Initialized Bux package '{name}'", useColor) + return 0 + +proc cmdCheck*(args: seq[string], opts: GlobalOptions): int = + let useColor = shouldUseColor(opts) + let root = getCurrentDir() + let manifestPath = root / "bux.toml" + if not fileExists(manifestPath): + printError("no bux.toml found", useColor) + return 1 + let man = loadManifest(manifestPath) + let srcDir = root / "src" + if not dirExists(srcDir): + printError("no src/ directory found", useColor) + return 1 + + var foundMain = false + for kind, path in walkDir(srcDir): + if kind == pcFile and path.endsWith(".bux"): + let source = readFile(path) + let res = tokenize(source, path) + if res.hasErrors: + printError(&"lex errors in {path}", useColor) + for d in res.diagnostics: + echo $d + return 1 + if opts.verbose: + printInfo(&"lexed {path} ({res.tokens.len} tokens)", useColor) + if splitFile(path).name == "Main": + foundMain = true + + if not foundMain: + printError("no Main.bux found in src/", useColor) + return 1 + + if not opts.quiet: + printInfo("check passed (lexer only)", useColor) + return 0 + +proc cmdBuild*(args: seq[string], opts: GlobalOptions): int = + let useColor = shouldUseColor(opts) + # For now, just type-check + let checkRes = cmdCheck(args, opts) + if checkRes != 0: + return checkRes + if not opts.quiet: + printInfo("build: nothing to do yet (no codegen)", useColor) + return 0 + +proc cmdRun*(args: seq[string], opts: GlobalOptions): int = + let useColor = shouldUseColor(opts) + let buildRes = cmdBuild(args, opts) + if buildRes != 0: + return buildRes + printError("run: no executable produced yet (no codegen)", useColor) + return 1 + +proc cmdClean*(args: seq[string], opts: GlobalOptions): int = + let useColor = shouldUseColor(opts) + # Placeholder + if not opts.quiet: + printInfo("clean: nothing to do yet", useColor) + return 0 + +proc cmdVersion*(args: seq[string], opts: GlobalOptions): int = + echo "bux 0.1.0 (bootstrap)" + return 0 + +proc runCli*(args: seq[string]): int = + let (opts, rest, ok) = parseGlobalOptions(args) + if not ok: + return 1 + if rest.len == 0: + printUsage() + return 0 + + let cmd = rest[0] + let cmdArgs = if rest.len > 1: rest[1..^1] else: @[] + + case cmd + of "new": return cmdNew(cmdArgs, opts) + of "init": return cmdInit(cmdArgs, opts) + of "build": return cmdBuild(cmdArgs, opts) + of "run": return cmdRun(cmdArgs, opts) + of "check": return cmdCheck(cmdArgs, opts) + of "clean": return cmdClean(cmdArgs, opts) + of "version", "--version", "-v": return cmdVersion(cmdArgs, opts) + of "help", "--help", "-h": + printUsage() + return 0 + else: + let useColor = shouldUseColor(opts) + printError(&"unknown command '{cmd}'", useColor) + return 1 diff --git a/src/lexer.nim b/src/lexer.nim new file mode 100644 index 0000000..0033e46 --- /dev/null +++ b/src/lexer.nim @@ -0,0 +1,565 @@ +import std/[strutils, strformat] +import token, source_location + +type + LexerDiagnosticSeverity* = enum + ldsWarning + ldsError + + LexerDiagnostic* = object + severity*: LexerDiagnosticSeverity + loc*: SourceLocation + message*: string + + LexerResult* = object + tokens*: seq[Token] + diagnostics*: seq[LexerDiagnostic] + +proc hasErrors*(res: LexerResult): bool = + for d in res.diagnostics: + if d.severity == ldsError: + return true + return false + +proc `$`*(d: LexerDiagnostic): string = + let sev = if d.severity == ldsError: "error" else: "warning" + result = &"{sev}: {d.message} at {d.loc}" + +type + Lexer* = object + source: string + sourceName: string + pos: int + line: uint32 + col: uint32 + tokens: seq[Token] + diagnostics: seq[LexerDiagnostic] + +proc initLexer*(source, sourceName: string): Lexer = + result.source = source + result.sourceName = sourceName + result.pos = 0 + result.line = 1 + result.col = 1 + +proc isAtEnd(lex: Lexer): bool = + lex.pos >= lex.source.len + +proc peek(lex: Lexer, ahead: int = 0): char = + let i = lex.pos + ahead + if i < lex.source.len: + return lex.source[i] + return '\0' + +proc advance(lex: var Lexer): char = + result = lex.peek() + if not lex.isAtEnd(): + inc lex.pos + if result == '\n': + inc lex.line + lex.col = 1 + else: + inc lex.col + +proc match(lex: var Lexer, expected: char): bool = + if lex.isAtEnd(): return false + if lex.peek() != expected: return false + discard lex.advance() + return true + +proc matchStr(lex: var Lexer, s: string): bool = + for i, c in s: + if lex.peek(i) != c: + return false + for _ in s: + discard lex.advance() + return true + +proc currentLocation(lex: Lexer): SourceLocation = + result = SourceLocation(line: lex.line, column: lex.col, offset: uint32(lex.pos)) + +proc emitError(lex: var Lexer, loc: SourceLocation, message: string) = + lex.diagnostics.add(LexerDiagnostic(severity: ldsError, loc: loc, message: message)) + +proc emitWarning(lex: var Lexer, loc: SourceLocation, message: string) = + lex.diagnostics.add(LexerDiagnostic(severity: ldsWarning, loc: loc, message: message)) + +proc makeToken(lex: Lexer, kind: TokenKind, startLoc: SourceLocation, startPos: int): Token = + let text = lex.source[startPos ..< lex.pos] + result = Token(kind: kind, text: text, loc: startLoc) + +# --------------------------------------------------------------------------- +# Whitespace / comments +# --------------------------------------------------------------------------- + +proc skipLineComment(lex: var Lexer) = + while not lex.isAtEnd() and lex.peek() != '\n': + discard lex.advance() + +proc skipBlockComment(lex: var Lexer) = + let startLoc = lex.currentLocation() + var depth = 1 + while not lex.isAtEnd() and depth > 0: + if lex.peek() == '/' and lex.peek(1) == '*': + discard lex.advance() + discard lex.advance() + inc depth + elif lex.peek() == '*' and lex.peek(1) == '/': + discard lex.advance() + discard lex.advance() + dec depth + else: + discard lex.advance() + if depth > 0: + lex.emitError(startLoc, "unterminated block comment") + +proc skipWhitespace(lex: var Lexer) = + while not lex.isAtEnd(): + let c = lex.peek() + if c in {' ', '\t', '\r'}: + discard lex.advance() + elif c == '/' and lex.peek(1) == '/': + lex.skipLineComment() + elif c == '/' and lex.peek(1) == '*': + discard lex.advance() + discard lex.advance() + lex.skipBlockComment() + else: + break + +# --------------------------------------------------------------------------- +# Identifiers +# --------------------------------------------------------------------------- + +proc isIdentStart(c: char): bool = + c in {'a'..'z', 'A'..'Z', '_'} + +proc isIdentChar(c: char): bool = + c in {'a'..'z', 'A'..'Z', '0'..'9', '_'} + +proc scanIdent(lex: var Lexer, startLoc: SourceLocation): Token = + let startPos = lex.pos + while not lex.isAtEnd() and isIdentChar(lex.peek()): + discard lex.advance() + let text = lex.source[startPos ..< lex.pos] + let kind = keywordKind(text) + result = Token(kind: kind, text: text, loc: startLoc) + +# --------------------------------------------------------------------------- +# Numbers +# --------------------------------------------------------------------------- + +proc scanIntSuffix(lex: var Lexer) = + # i8, i16, i32, i64, u8, u16, u32, u64, f32, f64 + if lex.isAtEnd(): return + let c = lex.peek() + if c in {'i', 'u', 'f'}: + discard lex.advance() + while not lex.isAtEnd() and lex.peek() in {'0'..'9'}: + discard lex.advance() + +proc scanHexDigits(lex: var Lexer) = + while not lex.isAtEnd() and lex.peek() in {'0'..'9', 'a'..'f', 'A'..'F'}: + discard lex.advance() + +proc scanBinDigits(lex: var Lexer) = + while not lex.isAtEnd() and lex.peek() in {'0', '1'}: + discard lex.advance() + +proc scanOctDigits(lex: var Lexer) = + while not lex.isAtEnd() and lex.peek() in {'0'..'7'}: + discard lex.advance() + +proc scanDecDigits(lex: var Lexer) = + while not lex.isAtEnd() and lex.peek() in {'0'..'9'}: + discard lex.advance() + +proc scanNumber(lex: var Lexer, startLoc: SourceLocation): Token = + let startPos = lex.pos + var isFloat = false + + if lex.peek() == '0' and lex.peek(1) in {'x', 'X', 'b', 'B', 'o', 'O'}: + discard lex.advance() # '0' + let prefix = lex.advance() + case prefix + of 'x', 'X': lex.scanHexDigits() + of 'b', 'B': lex.scanBinDigits() + of 'o', 'O': lex.scanOctDigits() + else: discard + lex.scanIntSuffix() + return lex.makeToken(tkIntLiteral, startLoc, startPos) + + lex.scanDecDigits() + + # Fractional part + if lex.peek() == '.' and lex.peek(1) in {'0'..'9'}: + isFloat = true + discard lex.advance() # '.' + lex.scanDecDigits() + + # Exponent + if lex.peek() in {'e', 'E'}: + isFloat = true + discard lex.advance() + if lex.peek() in {'+', '-'}: + discard lex.advance() + if lex.peek() notin {'0'..'9'}: + lex.emitError(lex.currentLocation(), "expected digits in exponent") + else: + lex.scanDecDigits() + + if isFloat: + # Optional f32/f64 suffix + if lex.peek() == 'f' and lex.peek(1) in {'3', '6'}: + discard lex.advance() + discard lex.advance() + result = lex.makeToken(tkFloatLiteral, startLoc, startPos) + else: + lex.scanIntSuffix() + result = lex.makeToken(tkIntLiteral, startLoc, startPos) + +# --------------------------------------------------------------------------- +# Strings and chars +# --------------------------------------------------------------------------- + +proc scanEscapeSequence(lex: var Lexer): string = + if lex.isAtEnd(): + return "" + let c = lex.advance() + case c + of '\\': result = "\\" + of '"': result = "\"" + of '\'': result = "\'" + of 'n': result = "\n" + of 'r': result = "\r" + of 't': result = "\t" + of '0': result = "\0" + of 'x': + var hexVal = "" + for _ in 0..<2: + if lex.isAtEnd() or lex.peek() notin {'0'..'9', 'a'..'f', 'A'..'F'}: + lex.emitError(lex.currentLocation(), "expected two hex digits after \\x") + break + hexVal.add(lex.advance()) + if hexVal.len == 2: + try: + let code = parseHexInt(hexVal) + result = $chr(code) + except ValueError: + lex.emitError(lex.currentLocation(), &"invalid hex escape \\x{hexVal}") + result = "" + else: + result = "" + else: + lex.emitError(lex.currentLocation(), &"unknown escape sequence \\\\{c}") + result = $c + +proc scanString(lex: var Lexer, startLoc: SourceLocation, prefixLen: int): Token = + let startPos = lex.pos - prefixLen + # prefixLen characters before the opening quote were already consumed by caller + # but we need to handle the quote itself + if lex.peek() == '"': + discard lex.advance() + while not lex.isAtEnd() and lex.peek() != '"': + if lex.peek() == '\n': + lex.emitError(lex.currentLocation(), "unterminated string literal") + break + if lex.peek() == '\\': + discard lex.advance() + discard lex.scanEscapeSequence() + else: + discard lex.advance() + if lex.isAtEnd(): + lex.emitError(startLoc, "unterminated string literal") + else: + discard lex.advance() # closing " + result = lex.makeToken(tkStringLiteral, startLoc, startPos) + +proc scanChar(lex: var Lexer, startLoc: SourceLocation, prefixLen: int): Token = + let startPos = lex.pos - prefixLen + if lex.peek() == '\'': + discard lex.advance() + if lex.isAtEnd(): + lex.emitError(startLoc, "unterminated char literal") + return lex.makeToken(tkCharLiteral, startLoc, startPos) + if lex.peek() == '\n': + lex.emitError(lex.currentLocation(), "newline in char literal") + elif lex.peek() == '\\': + discard lex.advance() + discard lex.scanEscapeSequence() + else: + discard lex.advance() + if lex.isAtEnd() or lex.peek() != '\'': + lex.emitError(lex.currentLocation(), "expected closing ' for char literal") + else: + discard lex.advance() + result = lex.makeToken(tkCharLiteral, startLoc, startPos) + +# --------------------------------------------------------------------------- +# Symbols / operators +# --------------------------------------------------------------------------- + +proc scanSymbol(lex: var Lexer, startLoc: SourceLocation): Token = + let startPos = lex.pos + let c1 = lex.advance() + + template check2(c2: char, kind2: TokenKind, kind1: TokenKind) = + if lex.peek() == c2: + discard lex.advance() + return lex.makeToken(kind2, startLoc, startPos) + else: + return lex.makeToken(kind1, startLoc, startPos) + + template check3(c2: char, kind2: TokenKind, c3: char, kind3: TokenKind, kind1: TokenKind) = + if lex.peek() == c2: + discard lex.advance() + if lex.peek() == c3: + discard lex.advance() + return lex.makeToken(kind3, startLoc, startPos) + return lex.makeToken(kind2, startLoc, startPos) + else: + return lex.makeToken(kind1, startLoc, startPos) + + template checkEq(c2: char, kind2: TokenKind, kind1: TokenKind) = + check2(c2, kind2, kind1) + + case c1 + of '(': return lex.makeToken(tkLParen, startLoc, startPos) + of ')': return lex.makeToken(tkRParen, startLoc, startPos) + of '{': return lex.makeToken(tkLBrace, startLoc, startPos) + of '}': return lex.makeToken(tkRBrace, startLoc, startPos) + of '[': return lex.makeToken(tkLBracket, startLoc, startPos) + of ']': return lex.makeToken(tkRBracket, startLoc, startPos) + of ',': return lex.makeToken(tkComma, startLoc, startPos) + of ';': return lex.makeToken(tkSemicolon, startLoc, startPos) + of '@': return lex.makeToken(tkAt, startLoc, startPos) + of '?': return lex.makeToken(tkQuestion, startLoc, startPos) + of '~': return lex.makeToken(tkTilde, startLoc, startPos) + of ':': check2(':', tkColonColon, tkColon) + of '.': + if lex.peek() == '.' and lex.peek(1) == '.': + discard lex.advance() + discard lex.advance() + return lex.makeToken(tkDotDotDot, startLoc, startPos) + elif lex.peek() == '.' and lex.peek(1) == '=': + discard lex.advance() + discard lex.advance() + return lex.makeToken(tkDotDotEqual, startLoc, startPos) + elif lex.peek() == '.': + discard lex.advance() + return lex.makeToken(tkDotDot, startLoc, startPos) + else: + return lex.makeToken(tkDot, startLoc, startPos) + of '-': + if lex.peek() == '>': + discard lex.advance() + return lex.makeToken(tkArrow, startLoc, startPos) + elif lex.peek() == '-': + discard lex.advance() + return lex.makeToken(tkMinusMinus, startLoc, startPos) + elif lex.peek() == '=': + discard lex.advance() + return lex.makeToken(tkMinusAssign, startLoc, startPos) + else: + return lex.makeToken(tkMinus, startLoc, startPos) + of '+': + if lex.peek() == '+': + discard lex.advance() + return lex.makeToken(tkPlusPlus, startLoc, startPos) + elif lex.peek() == '=': + discard lex.advance() + return lex.makeToken(tkPlusAssign, startLoc, startPos) + else: + return lex.makeToken(tkPlus, startLoc, startPos) + of '*': + if lex.peek() == '*': + discard lex.advance() + return lex.makeToken(tkStarStar, startLoc, startPos) + elif lex.peek() == '=': + discard lex.advance() + return lex.makeToken(tkStarAssign, startLoc, startPos) + else: + return lex.makeToken(tkStar, startLoc, startPos) + of '/': + if lex.peek() == '=': + discard lex.advance() + return lex.makeToken(tkSlashAssign, startLoc, startPos) + else: + return lex.makeToken(tkSlash, startLoc, startPos) + of '%': + if lex.peek() == '=': + discard lex.advance() + return lex.makeToken(tkPercentAssign, startLoc, startPos) + else: + return lex.makeToken(tkPercent, startLoc, startPos) + of '=': + if lex.peek() == '=': + discard lex.advance() + return lex.makeToken(tkEq, startLoc, startPos) + elif lex.peek() == '>': + discard lex.advance() + return lex.makeToken(tkFatArrow, startLoc, startPos) + else: + return lex.makeToken(tkAssign, startLoc, startPos) + of '!': + if lex.peek() == '=': + discard lex.advance() + return lex.makeToken(tkNe, startLoc, startPos) + else: + return lex.makeToken(tkBang, startLoc, startPos) + of '<': + if lex.peek() == '=': + discard lex.advance() + return lex.makeToken(tkLe, startLoc, startPos) + elif lex.peek() == '<': + discard lex.advance() + if lex.peek() == '=': + discard lex.advance() + return lex.makeToken(tkShlAssign, startLoc, startPos) + return lex.makeToken(tkShl, startLoc, startPos) + else: + return lex.makeToken(tkLt, startLoc, startPos) + of '>': + if lex.peek() == '=': + discard lex.advance() + return lex.makeToken(tkGe, startLoc, startPos) + elif lex.peek() == '>': + discard lex.advance() + if lex.peek() == '=': + discard lex.advance() + return lex.makeToken(tkShrAssign, startLoc, startPos) + return lex.makeToken(tkShr, startLoc, startPos) + else: + return lex.makeToken(tkGt, startLoc, startPos) + of '&': + if lex.peek() == '&': + discard lex.advance() + return lex.makeToken(tkAmpAmp, startLoc, startPos) + elif lex.peek() == '=': + discard lex.advance() + return lex.makeToken(tkAmpAssign, startLoc, startPos) + else: + return lex.makeToken(tkAmp, startLoc, startPos) + of '|': + if lex.peek() == '|': + discard lex.advance() + return lex.makeToken(tkPipePipe, startLoc, startPos) + elif lex.peek() == '=': + discard lex.advance() + return lex.makeToken(tkPipeAssign, startLoc, startPos) + else: + return lex.makeToken(tkPipe, startLoc, startPos) + of '^': + if lex.peek() == '=': + discard lex.advance() + return lex.makeToken(tkCaretAssign, startLoc, startPos) + else: + return lex.makeToken(tkCaret, startLoc, startPos) + of '#': + # Check for intrinsics: #line, #column, #file, #function, #date, #time, #module + let afterHash = lex.peek() + if afterHash == 'l' and lex.matchStr("line"): + return lex.makeToken(tkHashLine, startLoc, startPos) + elif afterHash == 'c' and lex.matchStr("column"): + return lex.makeToken(tkHashColumn, startLoc, startPos) + elif afterHash == 'f' and lex.peek(1) == 'i' and lex.peek(2) == 'l' and lex.peek(3) == 'e': + discard lex.matchStr("file") + return lex.makeToken(tkHashFile, startLoc, startPos) + elif afterHash == 'f' and lex.peek(1) == 'u' and lex.peek(2) == 'n': + discard lex.matchStr("function") + return lex.makeToken(tkHashFunction, startLoc, startPos) + elif afterHash == 'd' and lex.matchStr("date"): + return lex.makeToken(tkHashDate, startLoc, startPos) + elif afterHash == 't' and lex.matchStr("time"): + return lex.makeToken(tkHashTime, startLoc, startPos) + elif afterHash == 'm' and lex.matchStr("module"): + return lex.makeToken(tkHashModule, startLoc, startPos) + else: + return lex.makeToken(tkHash, startLoc, startPos) + else: + lex.emitError(startLoc, &"unexpected character '{c1}'") + return lex.makeToken(tkUnknown, startLoc, startPos) + +# --------------------------------------------------------------------------- +# Main scanning loop +# --------------------------------------------------------------------------- + +proc nextToken(lex: var Lexer): Token = + lex.skipWhitespace() + let startLoc = lex.currentLocation() + let startPos = lex.pos + + if lex.isAtEnd(): + return Token(kind: tkEndOfFile, text: "", loc: startLoc) + + let c = lex.peek() + + if c == '\n': + discard lex.advance() + return Token(kind: tkNewLine, text: "\n", loc: startLoc) + + # String prefixes: c8" c16" c32" — must come before ident check + if c == 'c' and lex.peek(1) in {'8', '1', '3'}: + let d = lex.peek(1) + if d == '8' and lex.peek(2) == '"': + discard lex.advance() # c + discard lex.advance() # 8 + return lex.scanString(startLoc, 2) + elif d == '1' and lex.peek(2) == '6' and lex.peek(3) == '"': + discard lex.advance() + discard lex.advance() + discard lex.advance() + return lex.scanString(startLoc, 3) + elif d == '3' and lex.peek(2) == '2' and lex.peek(3) == '"': + discard lex.advance() + discard lex.advance() + discard lex.advance() + return lex.scanString(startLoc, 3) + + if c == '"': + return lex.scanString(startLoc, 0) + + # Char prefixes: c8' c16' c32' — must come before ident check + if c == 'c' and lex.peek(1) in {'8', '1', '3'}: + let d = lex.peek(1) + if d == '8' and lex.peek(2) == '\'': + discard lex.advance() + discard lex.advance() + return lex.scanChar(startLoc, 2) + elif d == '1' and lex.peek(2) == '6' and lex.peek(3) == '\'': + discard lex.advance() + discard lex.advance() + discard lex.advance() + return lex.scanChar(startLoc, 3) + elif d == '3' and lex.peek(2) == '2' and lex.peek(3) == '\'': + discard lex.advance() + discard lex.advance() + discard lex.advance() + return lex.scanChar(startLoc, 3) + + if c == '\'': + return lex.scanChar(startLoc, 0) + + if isIdentStart(c): + return lex.scanIdent(startLoc) + + if c in {'0'..'9'}: + return lex.scanNumber(startLoc) + + return lex.scanSymbol(startLoc) + +proc tokenize*(lex: var Lexer): LexerResult = + while true: + let tok = lex.nextToken() + lex.tokens.add(tok) + if tok.kind == tkEndOfFile: + break + result = LexerResult(tokens: lex.tokens, diagnostics: lex.diagnostics) + +proc tokenize*(source, sourceName: string): LexerResult = + var lex = initLexer(source, sourceName) + result = lex.tokenize() + +proc dumpTokens*(res: LexerResult): string = + for tok in res.tokens: + result.add($tok & "\n") diff --git a/src/main.nim b/src/main.nim new file mode 100644 index 0000000..dfb06ab --- /dev/null +++ b/src/main.nim @@ -0,0 +1,6 @@ +import std/os +import cli + +when isMainModule: + let args = commandLineParams() + quit(runCli(args)) diff --git a/src/manifest.nim b/src/manifest.nim new file mode 100644 index 0000000..9a56b58 --- /dev/null +++ b/src/manifest.nim @@ -0,0 +1,79 @@ +import std/[strutils, os, tables] + +type + PackageType* = enum + ptExecutable + ptSharedLibrary + ptStaticLibrary + ptSource + + Manifest* = object + name*: string + version*: string + pkgType*: PackageType + output*: string ## from [Build] Output + dependencies*: OrderedTableRef[string, string] + +type + TomlValueKind = enum + tvkString, tvkTable + TomlValue = object + case kind*: TomlValueKind + of tvkString: + strVal*: string + of tvkTable: + tableVal*: OrderedTableRef[string, TomlValue] + +proc parseSimpleToml(path: string): OrderedTableRef[string, TomlValue] = + result = newOrderedTable[string, TomlValue]() + if not fileExists(path): + return result + let content = readFile(path) + var currentTable = "" + for rawLine in content.splitLines(): + let line = rawLine.strip() + if line.len == 0 or line.startsWith("#"): + continue + if line.startsWith("[") and line.endsWith("]"): + currentTable = line[1 ..< ^1] + if not result.hasKey(currentTable): + result[currentTable] = TomlValue(kind: tvkTable, tableVal: newOrderedTable[string, TomlValue]()) + else: + let eqIdx = line.find('=') + if eqIdx >= 0: + let key = line[0 ..< eqIdx].strip() + var val = line[eqIdx + 1 .. ^1].strip() + # Remove surrounding quotes + if val.len >= 2 and val[0] == '"' and val[^1] == '"': + val = val[1 ..< ^1] + if currentTable != "" and result.hasKey(currentTable): + result[currentTable].tableVal[key] = TomlValue(kind: tvkString, strVal: val) + else: + result[key] = TomlValue(kind: tvkString, strVal: val) + +proc loadManifest*(path: string): Manifest = + let data = parseSimpleToml(path) + result.name = "" + result.version = "0.1.0" + result.pkgType = ptExecutable + result.output = "Bin" + result.dependencies = newOrderedTable[string, string]() + + if data.hasKey("Package"): + let pkg = data["Package"].tableVal + if pkg.hasKey("Name"): + result.name = pkg["Name"].strVal + if pkg.hasKey("Version"): + result.version = pkg["Version"].strVal + if pkg.hasKey("Type"): + case pkg["Type"].strVal.toLowerAscii() + of "bin", "executable": result.pkgType = ptExecutable + of "shared", "sharedlibrary", "dll", "so", "dylib": result.pkgType = ptSharedLibrary + of "static", "staticlibrary", "lib", "a": result.pkgType = ptStaticLibrary + of "source": result.pkgType = ptSource + else: result.pkgType = ptExecutable + + if data.hasKey("Build"): + let bld = data["Build"].tableVal + if bld.hasKey("Output"): + result.output = bld["Output"].strVal diff --git a/src/source_location.nim b/src/source_location.nim new file mode 100644 index 0000000..a31ea96 --- /dev/null +++ b/src/source_location.nim @@ -0,0 +1,8 @@ +type + SourceLocation* = object + line*: uint32 ## 1-based + column*: uint32 ## 1-based (UTF-8 byte offset in line) + offset*: uint32 ## byte offset from start of file + +proc `$`*(loc: SourceLocation): string = + $loc.line & ":" & $loc.column diff --git a/src/token.nim b/src/token.nim new file mode 100644 index 0000000..f569bc0 --- /dev/null +++ b/src/token.nim @@ -0,0 +1,300 @@ +import source_location + +type + TokenKind* = enum + ##Literals + tkIntLiteral # 42 0xFF 0b1010 0o77 + tkFloatLiteral # 3.14 1.0e-9 + tkStringLiteral # "hello" c8"hello" c16"hello" c32"hello" + tkCharLiteral # 'A' c8'A' c16'A' c32'A' + tkBoolLiteral # true false + + ##Identifiers + tkIdent # foo Bar _x + + ##Control flow keywords + tkIf # if + tkElse # else + tkWhile # while + tkDo # do + tkLoop # loop + tkFor # for + tkIn # in + tkBreak # break + tkContinue # continue + tkReturn # return + tkMatch # match + + ##Declaration keywords + tkFunc # func + tkLet # let + tkVar # var + tkConst # const + tkType # type + tkStruct # struct + tkEnum # enum + tkUnion # union + tkInterface # interface + tkExtend # extend + tkModule # module + tkImport # import + tkPub # pub + tkExtern # extern + + ##Other keywords + tkAs # as + tkIs # is + tkNull # null + tkSelf # self + tkSuper # super + + ##Punctuation + tkLParen # ( + tkRParen # ) + tkLBrace # { + tkRBrace # } + tkLBracket # [ + tkRBracket # ] + tkComma # , + tkSemicolon # ; + tkColon # : + tkColonColon # :: + tkDot # . + tkDotDot # .. + tkDotDotDot # ... + tkDotDotEqual # ..= + tkArrow # -> + tkFatArrow # => + tkAt # @ + tkHash # # + tkQuestion # ? + + ##Arithmetic operators + tkPlus # + + tkMinus # - + tkStar # * + tkSlash # / + tkPercent # % + tkStarStar # ** + tkPlusPlus # ++ + tkMinusMinus # -- + + ##Bitwise operators + tkAmp # & + tkPipe # | + tkCaret # ^ + tkTilde # ~ + tkShl # << + tkShr # >> + + ##Logical operators + tkAmpAmp # && + tkPipePipe # || + tkBang # ! + + ##Comparison operators + tkEq # == + tkNe # != + tkLt # < + tkLe # <= + tkGt # > + tkGe # >= + + ##Assignment operators + tkAssign # = + tkPlusAssign # += + tkMinusAssign # -= + tkStarAssign # *= + tkSlashAssign # /= + tkPercentAssign # %= + tkAmpAssign # &= + tkPipeAssign # |= + tkCaretAssign # ^= + tkShlAssign # <<= + tkShrAssign # >>= + + ##Compile-time intrinsics + tkHashLine # #line + tkHashColumn # #column + tkHashFile # #file + tkHashFunction # #function + tkHashDate # #date + tkHashTime # #time + tkHashModule # #module + + ##Special + tkNewLine # significant newline (if grammar uses them) + tkEndOfFile # end of file + tkUnknown # unrecognized character + + Token* = object + kind*: TokenKind + text*: string # original source spelling + loc*: SourceLocation + +proc isKeyword*(kind: TokenKind): bool = + case kind + of tkIf, tkElse, tkWhile, tkDo, tkLoop, tkFor, tkIn, + tkBreak, tkContinue, tkReturn, tkMatch, + tkFunc, tkLet, tkVar, tkConst, tkType, tkStruct, tkEnum, + tkUnion, tkInterface, tkExtend, tkModule, tkImport, + tkPub, tkExtern, tkAs, tkIs, tkNull, tkSelf, tkSuper: + true + else: + false + +proc isLiteral*(kind: TokenKind): bool = + case kind + of tkIntLiteral, tkFloatLiteral, tkStringLiteral, tkCharLiteral, tkBoolLiteral: + true + else: + false + +proc isOperator*(kind: TokenKind): bool = + case kind + of tkPlus..tkShrAssign: + true + else: + false + +proc isEof*(kind: TokenKind): bool = kind == tkEndOfFile + +proc keywordKind*(text: string): TokenKind = + case text + of "func": tkFunc + of "let": tkLet + of "var": tkVar + of "const": tkConst + of "type": tkType + of "struct": tkStruct + of "enum": tkEnum + of "union": tkUnion + of "interface": tkInterface + of "extend": tkExtend + of "module": tkModule + of "import": tkImport + of "pub": tkPub + of "extern": tkExtern + of "if": tkIf + of "else": tkElse + of "while": tkWhile + of "do": tkDo + of "loop": tkLoop + of "for": tkFor + of "in": tkIn + of "break": tkBreak + of "continue": tkContinue + of "return": tkReturn + of "match": tkMatch + of "as": tkAs + of "is": tkIs + of "null": tkNull + of "self": tkSelf + of "super": tkSuper + of "true", "false": tkBoolLiteral + else: tkIdent + +proc tokenKindName*(kind: TokenKind): string = + case kind + of tkIntLiteral: "integer literal" + of tkFloatLiteral: "float literal" + of tkStringLiteral: "string literal" + of tkCharLiteral: "char literal" + of tkBoolLiteral: "boolean literal" + of tkIdent: "identifier" + of tkIf: "'if'" + of tkElse: "'else'" + of tkWhile: "'while'" + of tkDo: "'do'" + of tkLoop: "'loop'" + of tkFor: "'for'" + of tkIn: "'in'" + of tkBreak: "'break'" + of tkContinue: "'continue'" + of tkReturn: "'return'" + of tkMatch: "'match'" + of tkFunc: "'func'" + of tkLet: "'let'" + of tkVar: "'var'" + of tkConst: "'const'" + of tkType: "'type'" + of tkStruct: "'struct'" + of tkEnum: "'enum'" + of tkUnion: "'union'" + of tkInterface: "'interface'" + of tkExtend: "'extend'" + of tkModule: "'module'" + of tkImport: "'import'" + of tkPub: "'pub'" + of tkExtern: "'extern'" + of tkAs: "'as'" + of tkIs: "'is'" + of tkNull: "'null'" + of tkSelf: "'self'" + of tkSuper: "'super'" + of tkLParen: "'('" + of tkRParen: "')'" + of tkLBrace: "'{'" + of tkRBrace: "'}'" + of tkLBracket: "'['" + of tkRBracket: "']'" + of tkComma: "','" + of tkSemicolon: "';'" + of tkColon: "':'" + of tkColonColon: "'::'" + of tkDot: "'.'" + of tkDotDot: "'..'" + of tkDotDotDot: "'...'" + of tkDotDotEqual: "'..='" + of tkArrow: "'->'" + of tkFatArrow: "'=>'" + of tkAt: "'@'" + of tkHash: "'#'" + of tkQuestion: "'?'" + of tkPlus: "'+'" + of tkMinus: "'-'" + of tkStar: "'*'" + of tkSlash: "'/'" + of tkPercent: "'%'" + of tkStarStar: "'**'" + of tkPlusPlus: "'++'" + of tkMinusMinus: "'--'" + of tkAmp: "'&'" + of tkPipe: "'|'" + of tkCaret: "'^'" + of tkTilde: "'~'" + of tkShl: "'<<'" + of tkShr: "'>>'" + of tkAmpAmp: "'&&'" + of tkPipePipe: "'||'" + of tkBang: "'!'" + of tkEq: "'=='" + of tkNe: "'!='" + of tkLt: "'<'" + of tkLe: "'<='" + of tkGt: "'>'" + of tkGe: "'>='" + of tkAssign: "'='" + of tkPlusAssign: "'+='" + of tkMinusAssign: "'-='" + of tkStarAssign: "'*='" + of tkSlashAssign: "'/='" + of tkPercentAssign: "'%='" + of tkAmpAssign: "'&='" + of tkPipeAssign: "'|='" + of tkCaretAssign: "'^='" + of tkShlAssign: "'<<='" + of tkShrAssign: "'>>='" + of tkHashLine: "'#line'" + of tkHashColumn: "'#column'" + of tkHashFile: "'#file'" + of tkHashFunction: "'#function'" + of tkHashDate: "'#date'" + of tkHashTime: "'#time'" + of tkHashModule: "'#module'" + of tkNewLine: "newline" + of tkEndOfFile: "end of file" + of tkUnknown: "unknown token" + +proc `$`*(tok: Token): string = + result = tokenKindName(tok.kind) & " '" & tok.text & "' @ " & $tok.loc diff --git a/tests/lexer_test.nim b/tests/lexer_test.nim new file mode 100644 index 0000000..beb6c5e --- /dev/null +++ b/tests/lexer_test.nim @@ -0,0 +1,124 @@ +import std/[unittest, os, strutils] +import ../src/[lexer, token, source_location] + +proc tokenKinds(source: string): seq[TokenKind] = + let res = tokenize(source, "") + for t in res.tokens: + result.add(t.kind) + +proc tokenTexts(source: string): seq[string] = + let res = tokenize(source, "") + for t in res.tokens: + result.add(t.text) + +suite "Lexer": + test "empty source": + let res = tokenize("", "") + check res.tokens.len == 1 + check res.tokens[0].kind == tkEndOfFile + check not res.hasErrors + + test "simple identifiers": + let kinds = tokenKinds("foo bar _x Baz123") + check kinds == @[tkIdent, tkIdent, tkIdent, tkIdent, tkEndOfFile] + + test "keywords": + let kinds = tokenKinds("func let var if else while for return match struct enum") + check kinds == @[tkFunc, tkLet, tkVar, tkIf, tkElse, tkWhile, tkFor, tkReturn, tkMatch, tkStruct, tkEnum, tkEndOfFile] + + test "bool literals": + let kinds = tokenKinds("true false") + check kinds == @[tkBoolLiteral, tkBoolLiteral, tkEndOfFile] + let texts = tokenTexts("true false") + check texts == @["true", "false", ""] + + test "integers": + let kinds = tokenKinds("42 0xFF 0b1010 0o77") + check kinds == @[tkIntLiteral, tkIntLiteral, tkIntLiteral, tkIntLiteral, tkEndOfFile] + + test "floats": + let kinds = tokenKinds("3.14 1.0e-9 2.5E+3") + check kinds == @[tkFloatLiteral, tkFloatLiteral, tkFloatLiteral, tkEndOfFile] + + test "operators": + let kinds = tokenKinds("+ - * / % ** ++ --") + check kinds == @[tkPlus, tkMinus, tkStar, tkSlash, tkPercent, tkStarStar, tkPlusPlus, tkMinusMinus, tkEndOfFile] + + test "comparison operators": + let kinds = tokenKinds("== != < <= > >=") + check kinds == @[tkEq, tkNe, tkLt, tkLe, tkGt, tkGe, tkEndOfFile] + + test "assignment operators": + let kinds = tokenKinds("= += -= *= /= %= &= |= ^= <<= >>=") + check kinds == @[tkAssign, tkPlusAssign, tkMinusAssign, tkStarAssign, tkSlashAssign, tkPercentAssign, tkAmpAssign, tkPipeAssign, tkCaretAssign, tkShlAssign, tkShrAssign, tkEndOfFile] + + test "punctuation": + let kinds = tokenKinds("( ) { } [ ] , ; : :: . .. ... ..= -> => @ # ?") + check kinds == @[tkLParen, tkRParen, tkLBrace, tkRBrace, tkLBracket, tkRBracket, tkComma, tkSemicolon, tkColon, tkColonColon, tkDot, tkDotDot, tkDotDotDot, tkDotDotEqual, tkArrow, tkFatArrow, tkAt, tkHash, tkQuestion, tkEndOfFile] + + test "logical operators": + let kinds = tokenKinds("&& || !") + check kinds == @[tkAmpAmp, tkPipePipe, tkBang, tkEndOfFile] + + test "bitwise operators": + let kinds = tokenKinds("& | ^ ~ << >>") + check kinds == @[tkAmp, tkPipe, tkCaret, tkTilde, tkShl, tkShr, tkEndOfFile] + + test "string literals": + let kinds = tokenKinds("\"hello\" c8\"world\"") + check kinds == @[tkStringLiteral, tkStringLiteral, tkEndOfFile] + + test "char literals": + let kinds = tokenKinds("'A' c8'B'") + check kinds == @[tkCharLiteral, tkCharLiteral, tkEndOfFile] + + test "line comments are skipped": + let kinds = tokenKinds("let x = 42 // this is a comment") + check kinds == @[tkLet, tkIdent, tkAssign, tkIntLiteral, tkEndOfFile] + + test "block comments are skipped": + let kinds = tokenKinds("let /* inline comment */ x = 42") + check kinds == @[tkLet, tkIdent, tkAssign, tkIntLiteral, tkEndOfFile] + + test "nested block comments": + let kinds = tokenKinds("let /* outer /* inner */ still outer */ x = 42") + check kinds == @[tkLet, tkIdent, tkAssign, tkIntLiteral, tkEndOfFile] + + test "intrinsics": + let kinds = tokenKinds("#line #column #file #function #date #time #module") + check kinds == @[tkHashLine, tkHashColumn, tkHashFile, tkHashFunction, tkHashDate, tkHashTime, tkHashModule, tkEndOfFile] + + test "newline tokens": + let kinds = tokenKinds("let x = 42\nlet y = 10") + check kinds == @[tkLet, tkIdent, tkAssign, tkIntLiteral, tkNewLine, tkLet, tkIdent, tkAssign, tkIntLiteral, tkEndOfFile] + + test "paths": + let kinds = tokenKinds("Std::Io::PrintLine") + check kinds == @[tkIdent, tkColonColon, tkIdent, tkColonColon, tkIdent, tkEndOfFile] + + test "unterminated string error": + let res = tokenize("\"hello", "") + check res.hasErrors + + test "unterminated char error": + let res = tokenize("'a", "") + check res.hasErrors + + test "escape sequences in string": + let res = tokenize("\"hello\\nworld\\t!\"", "") + check not res.hasErrors + check res.tokens[0].kind == tkStringLiteral + + test "full function": + let src = "func Main() -> int {\n let x: int32 = 42;\n return x;\n}\n" + let res = tokenize(src, "") + check not res.hasErrors + check res.tokens[0].kind == tkFunc + check res.tokens[1].kind == tkIdent + check res.tokens[1].text == "Main" + + test "dumpTokens produces output": + let res = tokenize("let x = 42", "") + let dump = dumpTokens(res) + check "let" in dump + check "42" in dump diff --git a/tests/testdata/sample.bux b/tests/testdata/sample.bux new file mode 100644 index 0000000..b9fc2ac --- /dev/null +++ b/tests/testdata/sample.bux @@ -0,0 +1,11 @@ +import Std::Io::PrintLine; + +func Add(a: int32, b: int32) -> int32 { + return a + b; +} + +func Main() -> int { + let result: int32 = Add(10, 20); + PrintLine(c8"Hello from Bux!"); + return 0; +}