# Mist Language Documentation ## Introduction Mist is a pragmatic systems programming language that compiles directly to Rust. It provides C/C++-style syntax with zero-cost abstractions, full interoperability with the Rust ecosystem, and additional language features like classes with inheritance. The compiler is written in Rust and Mist itself, and is organized into four main crates: - **mist-parser** — Parses Mist source code into an AST using pest - **mist-codegen** — Generates Rust code from the Mist AST - **mist-analyzer** — Language server built on top of rust-analyzer - **mist-api** — Mist crate that orchestrates transpilation and building --- ## Installation ```bash cargo install mist-lang ``` This installs the `mist` CLI binary and the `mist-analyzer` LSP binary. --- ## Project Structure A Mist project requires the following structure: ``` my-project/ ├── Mist.toml # Project configuration ├── Cargo.toml # Standard Rust Cargo.toml (managed by mist) ├── src/ │ ├── main.mist # Entry point (or lib.mist for libraries) │ └── ... # Other .mist source files └── .mist/ └── src/ # Generated Rust output (do not edit) ``` ### Mist.toml ```toml package = "main.mist" # Entry point file (relative to src/) packages = ["utils"] # Subdirectories to treat as sub-modules include = ["crates/*"] # Additional Mist crates to transpile ``` The `packages` field lists subdirectories under `src/` that should be treated as child modules. Each such directory should contain a `package.mist` file acting as the module root. The `include` field supports glob patterns for including external Mist crates. A trailing `*` treats all immediate subdirectories that contain a `Mist.toml` as independent crates. --- ## CLI Commands | Command | Short | Description | |------------------|-------|---------------------------------| | `mist run` | `r` | Run the project | | `mist build` | `b` | Build the project | | `mist check` | `c` | Check without building | | `mist test` | | Run tests | | `mist publish` | | Publish the project | | `mist bench` | | Benchmark the project | | `mist doc` | | Build documentation | | `mist fix` | | Auto-fix warnings | | `mist clippy` | | Lint the project | | `mist clean` | | Clean build artifacts | | `mist transpile` | `t` | Transpile Mist to Rust only | | `mist init` | | Init a project in current dir | | `mist new` | | Create a new project | | `mist version` | `-v` | Print compiler version | | `mist help` | `-h` | Print usage information | All Cargo subcommands delegate to `cargo` with Mist error remapping. --- ## Language Syntax ### Comments ```mist // Line comments only ``` ### Literals ```mist 42 // Integer 3.14 // Float true // Boolean false // Boolean "hello" // String (1, true, "x") // Tuple ``` ### Identifiers & Keywords Keywords are reserved and cannot be used as identifiers: `if`, `else`, `fn`, `for`, `while`, `match`, `return`, `break`, `continue`, `struct`, `enum`, `class`, `trait`, `impl`, `use`, `pub`, `mut`, `let`, `true`, `false`, `dyn`, `loop`, `unsafe`, `override`, `const`, `type` Identifiers follow the pattern `[a-zA-Z_][a-zA-Z0-9_]*`. --- ### Variables ```mist let x = 42; // Type-inferred immutable let mut y = 10; // Mutable variable i32 z = 100; // Explicit type annotation str& s = "hello"; // Typed string reference bool b = true; // Typed boolean f64 f = 3.14; // Typed float let (a, b) = (1, "two"); // Destructuring let (x, (y, z)) = (1, (2, 3)); // Nested destructuring ``` Variable declarations follow either of two forms: ```mist let [= ]; [= ]; ``` --- ### Functions ```mist // Return type is `void` (unit) void greet(str& name) { println!("Hello, {name}"); } // With return type i32 add(i32 a, i32 b) { return a + b; } // Expression body (last expression is the return value) i32 square(i32 x) { x * x } // Public function pub i32 multiply(i32 a, i32 b) { a * b } // Generic function T identity(T value) { value } // Unsafe function unsafe i32 dangerous() { 42 } ``` Function syntax: `[pub] [void | ] []() [override] { }` Functions use **Allman brace style** (opening brace on the next line). ### Self Parameter Methods can take `self`, `&self`, `&mut self`, or `self` with lifetimes: ```mist pub void set_value(&mut self, i32 v) { self.value = v; } ``` --- ### Control Flow #### If/Else ```mist if condition { // body } else if other_condition { // body } else { // body } ``` Conditions must be parenthesized: `if (expr) { }` or `if expr_no_struct { }` (struct literals require parentheses). Used as an expression: ```mist let x = if true { 1 } else { 2 }; ``` #### While ```mist while condition { // body } ``` #### For ```mist for pattern in iterator { // body } for i in 0 .. 10 { // body } ``` #### C-Style For ```mist for (let mut i = 0; i < 10; i++) { // body } ``` #### Loop ```mist loop { // infinite loop } ``` #### Match ```mist match value { pattern1 => expr, pattern2 => { // block body } pattern3 | pattern4 => expr, } ``` #### Break/Continue ```mist break; continue; ``` #### Return ```mist return; return value; ``` --- ### Types #### Type System ```mist i32 // Path type str& // Reference type &mut i32 // Mutable reference &'a i32 // Reference with lifetime *const i32 // Const pointer *mut i32 // Mutable pointer (i32, bool) // Tuple type fn(i32) -> bool // Function pointer type Fn(i32) -> bool // Closure trait (Fn) FnMut(i32) -> bool // Closure trait (FnMut) FnOnce(i32) -> bool // Closure trait (FnOnce) dyn Trait // Trait object void // Unit type (maps to Rust's ()) 'i32 // Type ascription not to be confused with lifetime ``` Type expressions can be composed: ```mist type_expr = { (void | path_type | tuple_type | dyn_type) ~ (unsafe_ref_type | ref_type | fn_type)* } ``` This means types are written left-to-right naturally: ```mist i32& // &i32 i32&mut // &mut i32 i32&'a // &'a i32 i32 fn() -> bool // fn(i32) -> bool ``` #### Type Aliases ```mist type MyInt = i32; type Result = std::result::Result; ``` #### Const and Static ```mist const MAX: i32 = 100; static NAME: str& = "hello"; ``` --- ### Structs ```mist pub struct Point { i32 x, i32 y, } pub struct Generic { T value, } ``` Fields can be public or private: ```mist struct User { pub str& name, i32 age, // private } ``` --- ### Enums ```mist pub enum Option { Some(T), None, } pub enum Message { Quit, Move(i32, i32), Write { str& content, i32 length }, } ``` Enum variants can be: - **Named** — `Variant` - **Tuple** — `Variant(T1, T2)` - **Struct** — `Variant { T1 field1, T2 field2 }` --- ### Traits ```mist pub trait Drawable { void draw(&self); } pub trait Comparable : Eq { i32 cmp(&self, T other); } ``` Trait requirements are specified after `:`: ```mist trait MyTrait : SuperTrait + OtherTrait { void required_method(&self); void another(&self); } ``` --- ### Impl Blocks ```mist impl MyType { void method(&self) { } } impl Trait for MyType { void method(&self) { } } impl GenericTrait for MyType { void method(&self, T value) { } } ``` --- ### Classes Mist introduces `class` as syntactic sugar for a Rust struct with a virtual method table (vtable). ```mist pub class Animal { str& name, pub constructor(str& name) { self.name = name; } pub void speak(&self) { println!("..."); } } ``` Class fields can have default initializers: ```mist class Player { i32 health = 100, str& name, } ``` #### Inheritance ```mist class Dog : Animal { pub constructor(str& name) { super(name); } override void speak(&self) { println!("Woof!"); } } ``` The `override` keyword supports explicit base class targeting: ```mist override(Animal) void speak(&self) { println!("Woof!"); } ``` #### Under the Hood A class `Dog : Animal` generates: 1. A Rust struct with a `_super: Animal` field (or `_vptr: &'static [*const c_void]` for root classes) 2. A vtable constant with function pointers for each public method 3. An `impl` block with `Deref` and `DerefMut` 4. Method trampolines (`__m_`) that are dispatched through the vtable 5. A `new()` constructor that initializes via `MaybeUninit` and calls the user's `constructor(&mut self)` The vtable is unified: parent entries are copied, overridden entries replace parent slots, and new methods are appended. --- ### Generics ```mist // Generic function T id(T x) { x } // Generic struct struct Pair { A first, B second, } // Generic enum enum Result { Ok(T), Err(E), } // Generic with trait bounds T max(T a, T b) { if a > b { a } else { b } } // Lifetime generics void process<'a>(&'a str& data) { // ... } ``` Generic syntax uses `<` `>` delimiters. Lifetimes are prefixed with `'`. --- ### Visibility ```mist // Private (default) void internal() { } // Public pub void external() { } // Public to specific path pub(crate) void crate_only() { } pub(super) void parent_only() { } pub(in my::module) void module_only() { } ``` --- ### Modules ```mist // Declare a submodule pub module foo; // Import use std::collections::HashMap; // Re-exporting import pub use my_module::MyType; ``` #### Module Resolution Mist maps the module tree to Rust's module system: | Mist Path | Rust Output | |------------------------------|------------------------------| | `src/main.mist` | `.mist/src/main.rs` | | `src/utils/package.mist` | `.mist/src/utils/mod.rs` | | `src/foo.mist` | `.mist/src/foo.rs` | | `src/utils/helper.mist` | `.mist/src/utils/helper.rs` | A `package.mist` file acts as a directory's module root, analogous to `mod.rs`. --- ### Attributes Inner attributes apply to the containing module: ```mist #![allow(unused_variables)] ``` Outer attributes apply to the next item: ```mist #[derive(Debug, Clone)] struct Point { i32 x, i32 y, } #[test] void my_test() { assert_eq!(1, 1); } ``` Attribute syntax: - `#[path]` - `#[path = literal]` - `#[path(item1, item2, ...)]` --- ### Patterns ```mist // Literal patterns match x { 1 => "one", 2 => "two", _ => "other", } // Tuple patterns let (a, b) = (1, 2); // Struct patterns match value { Point { x, y } => x + y, Point { x: 0, y } => y, _ => 0, } // Named tuple patterns (newtype) let MyType(value) = my_var; // Wildcard / etc let _ = get_side_effect(); match x { 1 => ..., .. => ..., // rest / etc } // Mutable binding in pattern let mut x = 42; match ref_to_option { Some(mut value) => value += 1, None => {}, } ``` --- ### Operators #### Binary Operators | Operator | Description | |----------|----------------------| | `+` | Addition | | `-` | Subtraction | | `*` | Multiplication | | `/` | Division | | `%` | Modulus | | `==` | Equality | | `!=` | Inequality | | `<` | Less than | | `>` | Greater than | | `<=` | Less or equal | | `>=` | Greater or equal | | `&&` | Logical AND | | `\|\|` | Logical OR | | `&` | Bitwise AND | | `\|` | Bitwise OR | | `^` | Bitwise XOR | | `<<` | Left shift | | `>>` | Right shift | | `=` | Assignment | | `+=` | Add assign | | `-=` | Subtract assign | | `*=` | Multiply assign | | `/=` | Divide assign | | `%=` | Modulus assign | | `&=` | Bitwise AND assign | | `\|=` | Bitwise OR assign | | `^=` | Bitwise XOR assign | | `<<=` | Left shift assign | | `>>=` | Right shift assign | | `..` | Range (exclusive) | | `..=` | Range (inclusive) | | `->` | Pointer write | #### Prefix Operators | Operator | Description | |----------|-------------------| | `*` | Dereference | | `&` | Reference | | `&mut` | Mutable reference | | `!` | Logical NOT | | `-` | Numeric negation | #### Postfix Operators | Operator | Description | |------------|----------------------| | `.field` | Field access | | `.0` | Tuple field access | | `()` | Function call | | `[]` | Index | | `{ f: v }` | Struct literal | | `as Type` | Type cast | | `?` | Try (error prop) | | `++` | Increment | | `--` | Decrement | | `!()` | Macro call (paren) | | `![]` | Macro call (bracket) | | `!{}` | Macro call (brace) | --- ### Macros Mist reuses Rust's macro system directly: ```mist println!("hello"); assert_eq!(a, b); vec![1, 2, 3]; ``` Macro calls use `!` followed by parentheses, brackets, or braces. --- ### Closures ```mist let add = (a, b) => a + b; let result = add(2, 3); // 5 let square = f64 x => x * x; ``` Closures can have explicit return types: ```mist let transform = i32 x => { x * 2 }; ``` --- ### Attributes Mist supports Rust-style attributes at module and item level: ```mist #![crate_type = "lib"] #[derive(Clone)] #[repr(C)] pub struct External { } ``` --- ## Compiler Architecture ### Pipeline Overview The Mist compiler operates in distinct phases: ``` Source (.mist) │ ▼ ┌─────────────┐ │ Lexer │ (pest PEG grammar) │ & Parser │ └─────────────┘ │ AST ▼ ┌─────────────┐ │ Semantics │ (field init checking) └─────────────┘ │ ▼ ┌─────────────┐ │ Codegen │ (AST → Rust source) └─────────────┘ │ .rs + .map.json ▼ ┌─────────────┐ │ cargo │ (Rust compilation) └─────────────┘ │ ▼ Binary / Library ``` ### 1. Parsing (`mist-parser`) The parser is built with [pest](https://pest.rs), a PEG parser generator. Grammar rules are defined in `grammar.pest` (693 rules). **Key files**: - `grammar.pest` — PEG grammar defining the entire language syntax - `src/parser/common/` — Parse rule → AST conversions for expressions, statements, types, declarations - `src/parser/items/` — Parse rule → AST conversions for top-level items (structs, enums, classes, functions, traits, impls, attributes) - `src/ast/` — AST node types (expr.rs, statement.rs, top_level.rs) - `src/semantics.rs` — Semantic checks (e.g., ensuring all class fields are initialized in constructors) - `src/error.rs` — Error types (PreAst parsing errors, Ast generation errors) - `src/rev_mapper.rs` — Position mapping between Mist and Rust source **Parsing entry points**: ```rust // Parse a complete program pub fn parse<'a>(source: &'a str) -> Result> // Parse only the module declaration pub fn parse_module<'a>(source: &'a str) -> Result, ParseError<'a>> ``` #### Grammar Structure The grammar follows a layered approach: 1. **Lexical rules** (silent) — `WHITESPACE`, `COMMENT`, `identifier`, `integer`, `float`, `string_lit` 2. **Primary expressions** — literals, paths, tuples, arrays, closures, control flow 3. **Term** — prefix operators + primary + postfix operators 4. **Expression** — terms joined by binary operators (via Pratt parser) 5. **Statements** — variable declarations, control flow (`if`, `while`, `for`, `match`, `loop`), blocks 6. **Top-level items** — functions, structs, enums, classes, traits, impls, type aliases, imports, module declarations, constants #### AST Structure The AST preserves source positions via `Spanned`: ```rust pub struct Spanned { pub line: usize, pub column: usize, pub item: T, } ``` Top-level items are wrapped as: ```rust pub struct TopLevel(pub Spanned, pub Vec); ``` Expressions use a fix-point representation for prefix/postfix operators: ```rust Expression::Fix { initial: Box, prefixes: Vec, postfixes: Vec, } ``` Binary expressions use Pratt parsing for correct precedence: ```rust Expression::Binary { lhs: Box, op: String, rhs: Box, } ``` All operators are left-associative with a single precedence level. ### 2. Semantic Analysis (`mist-parser`) The semantic checker (`semantics.rs`) performs class field initialization analysis. When a class has a constructor, it verifies that every declared field is mutated (directly or indirectly via method calls) within the constructor body. **Key check**: `check_class_semantics()` — Collects all field names, then walks the constructor body tracking which identifiers are assigned. If any field is uninitialized, an error is reported. The mutability analysis (`GetMutability` trait) works by: 1. Collecting all identifiers that get `&mut` references 2. Tracking `self.field = value` patterns 3. Following method calls that might initialize fields 4. Ensuring all branches initialize the same fields (intersection semantics) ### 3. Code Generation (`mist-codegen`) The code generator converts the Mist AST into Rust source code. It does not generate an intermediate representation — it produces Rust source text directly. **Key files**: - `src/lib.rs` — `RustCodegen` struct with output buffer, indentation tracking, `Mapping` for position translation - `src/top_level.rs` — Generates Rust for top-level items (structs, enums, traits, functions, impls, imports, type aliases, const/static) - `src/statement.rs` — Generates Rust for statements and blocks - `src/expr.rs` — Generates Rust for expressions (literals, paths, binary ops, closures, arrays, prefix/postfix) - `src/class_decl.rs` — Class-specific code generation (struct + vtable + impl blocks + Deref) #### Codegen Design The `RustCodegen` maintains: - A string `output` buffer - An `indent` level (4 spaces per level) - A `Mapping` that records `(RustMap, MistMap)` pairs for source position remapping - A current `position` tracker (`RustMap(line, column)`) Generation follows the `GenRust` trait: ```rust pub trait GenRust { fn gen_rust(&self, ctx: &mut Context, cg: &mut RustCodegen); } ``` And `GetRust` for simple string-returning types: ```rust pub trait GetRust { fn get_rust(&self) -> String; } ``` The `Context` carries optional expression path information for class `super` / `Super` resolution. #### Class Codegen (detailed) Classes are the most complex codegen path. `ClassProcessedData` analyzes a class declaration and then emits: 1. **Struct declaration** — `struct ClassName { pub _super: Parent, pub field1: T1, ... }` (or `_vptr` for root classes) 2. **Vtable constants** — Index constants `__FN_METHOD` and a `__V_TABLE` static array of function pointers. For inherited classes, parent vtable entries are copied and overridden entries replaced. 3. **Constructor** — `pub fn new(...) -> Self` that: - Creates an uninitialized instance via `MaybeUninit::zeroed().assume_init()` - Sets the vtable pointer - Writes field defaults - Calls `self.constructor(...)` - Returns `this` 4. **Method trampolines** — Public methods with `self` get wrapper functions `__m_method` that are stored in the vtable, plus virtual dispatch methods that look up the function pointer at runtime. 5. **Override support** — Methods marked `override` are validated at compile time by generating test code that dereferences `&Self` to `&Target`, confirming the Deref chain works. 6. **Deref impls** — `impl Deref for Child` and `DerefMut` for inherited classes. 7. **Impl declarations** — Inner `impl` blocks are rewritten to use the self type. #### Mapping System The `rev_mapper` module provides bidirectional position mapping: ```rust pub struct Mapping { pub mist_path: PathBuf, pub map: HashSet<(RustMap, MistMap)>, } ``` - `RustMap(usize, usize)` — line/column in generated Rust - `MistMap(usize, usize)` — line/column in original Mist The mapping is populated during codegen via `GenSpanTranslation`: ```rust impl GenSpanTranslation for Spanned { fn gen_span(&self, cg: &mut RustCodegen) { cg.mapping.map.insert((cg.position, MistMap(self.line, self.column))); } } ``` This allows the builder to remap Rust compiler errors back to the original Mist source positions. ### 4. Builder & Error Remapping (`mist-api`) The builder module (`builder.mist` in the `mist_api` crate) wraps `cargo` to: 1. Spawn `cargo` with `--message-format=json` 2. Parse JSON compiler messages 3. For each diagnostic span, look up the corresponding `.map.json` file 4. Remap Rust line/column to Mist line/column using the mapping 5. Display errors/warnings with Mist source locations and context lines ```rust pub fn build(args: Vec, root: PathBuf) -> bool { // Spawn cargo with JSON output // Parse CompilerMessage for each span // Look up rev_mapper::Mapping from .map.json // Remap to Mist positions // Print diagnostics } ``` ### 5. Transpilation Pipeline The transpiler (`transpiler.mist` in `mist_api`) orchestrates the full pipeline: 1. Read `Mist.toml` configuration 2. Build a `Module` tree from the filesystem (discovering `package.mist` files) 3. For each module: a. Parse Mist source b. Run semantic checks c. Generate Rust code and position mapping d. Write `.rs` file and `.map.json` file 4. Recursively transpile included crates The `Module` struct represents the file-to-module mapping: ```rust pub class Module { pub String name; pub PathBuf path; pub Vec children; pub constructor(PathBuf mist_path, MistConfig& config) { ... } pub bool is_package(&self) { ... } pub PathBuf output_dir(&self, PathBuf& parent_dir) { ... } pub PathBuf output_path(&self, PathBuf& parent_dir, ...) { ... } } ``` #### Transpilation Mapping Rules | Mist Source | Rust Output | |-----------------|--------------------------------| | `src/main.mist` | `.mist/src/main.rs` | | `src/foo.mist` | `.mist/src/foo.rs` | | `src/bar/package.mist` | `.mist/src/bar/mod.rs` | | `src/bar/baz.mist` | `.mist/src/bar/baz.rs` | A `package.mist` file: - Generates `mod.rs` in the output directory - Contains `pub mod ;` declarations for its children - Its own items are prepended to the output ### 6. Caching The transpiler implements basic caching via file modification times: ```rust fn is_source_newer(source: &Path, output: &Path) -> io::Result { if !output.exists() { return Ok(true); } let source_time = fs::metadata(source)?.modified()?; let output_time = fs::metadata(output)?.modified()?; Ok(source_time > output_time) } ``` If a source file has not changed since the last transpilation, the `.rs` file is not regenerated. --- ## LSP Support (`mist-analyzer`) The Mist Language Server Protocol implementation runs a headless `rust-analyzer` instance and bridges Mist editor requests to Rust positions. ### Architecture ``` Editor (LSP client) │ ▼ ┌─────────────────────┐ │ mist-analyzer │ │ (Rust + Mist) │ └─────────────────────┘ │ ▲ │ JSON-RPC │ ▼ │ ┌─────────────────────┐ │ rust-analyzer │ │ (headless child) │ └─────────────────────┘ ``` ### Flow 1. Editor sends Mist file edits to `mist-analyzer` via LSP 2. `mist-analyzer` transpiles Mist to Rust 3. The transpiled Rust is forwarded to the headless `rust-analyzer` process 4. For goto-definition, hover, and completion, a unique marker token is injected at the cursor position 5. The transpiled output with the marker is sent to `rust-analyzer` 6. The marker position is located in the response 7. Results are mapped back to Mist positions using the `rev_mapper` ### Key Features - **Full document sync** — Open, change, save, close - **Go to definition** — Maps through transpiled Rust positions - **Completions** — Supports trigger characters `:`, `.`, `'`, `(` - **Hover** — Type information and docs - **Diagnostics** — Real-time errors from transpilation failures and Rust compilation - **Formatting** — Document formatting support (currently passthrough) - **Auto-import** — Automatically inserts `pub module ;` when new `.mist` files are created - **File watching** — Monitors `**/*.mist` for new files ### Diagnostic Remapping When a transpilation error occurs in the `mist-analyzer`: 1. Parse errors are converted to `Diagnostic` with Mist source positions 2. Semantic errors (uninitialized class fields) use the stored line/column 3. Rust compiler errors are remapped via the `Mapping` system back to Mist positions --- ## Semantic Checks ### Field Initialization The primary semantic check ensures all class fields are initialized in the constructor: ```mist class Player { i32 health, str& name, pub constructor(str& name) { self.name = name; // Error: field 'health' is uninitialized } } ``` The analysis tracks: - Direct field assignment: `self.field = value` - `&mut self.field` patterns - Method calls that might initialize fields (transitively) - All conditional branches must initialize the same fields - `super` assignments count for `_super` field initialization --- ## Position Mapping Mist maintains a bidirectional mapping between Mist source positions and Rust output positions. This is essential for: 1. **Error remapping** — Rust compiler errors point to the correct Mist source location 2. **LSP features** — Go-to-definition, hover, and completion work from Mist source The mapping is stored as `HashSet<(RustMap, MistMap)>` pairs and serialized to `.map.json` files alongside the transpiled `.rs` output. ### Mapping Lifecycle 1. **Codegen** — Each `Spanned` AST node records its Mist position and the current Rust position in the codegen output buffer 2. **Persistence** — The mapping is written to `.map.json` 3. **Build-time remapping** — The builder reads `.map.json` to remap `cargo` diagnostics back to Mist positions 4. **LSP remapping** — Both Mist→Rust and Rust→Mist direction queries are supported via `find()` and `find_by_mist()`