Iota Syntaxis Preprocessor Specification

Version 1.0 alpha1 (2026-05-09)

This document defines a preprocessor layer to Iota Syntaxis: includes, execution, macros, and pipelines.

The preprocessor runs before the Iota Syntaxis core scanner. It expands execution constructs (##command >>) and inline include directives (--<id, //<file, ;;<resource) so the parser sees only core constructs. Implementations MAY support the preprocessor independently of the parser (SPEC.md) and renderer (RENDERER.md).

The key words MUST, SHOULD, and MAY are used as defined in RFC 2119.

Design Principles

The Iota Syntaxis Preprocessor Specification follows the same design principles as the Iota Syntaxis Specification (Iota Syntaxis core), with one addition:

7. Active content through macros. texts can be executable. A text marked as a macro can generate content dynamically, transforming input text into output that may itself contain Iota Syntaxis constructs for further processing.

Terminology: Macros

We use the term macro rather than “function” for several reasons:

• Historical precedent. Like C preprocessor macros, these constructs expand before main processing.

• User-friendly. “Macro” doesn’t carry programming-language baggage for non-technical users.

• Semantically accurate. A macro expands input into output, just as word processor macros record and replay actions.

A macro is invoked via the \#\# sigil, which distinguishes it from notes (--), resources (;;), and other constructs. Macro names are always lowercase to visually distinguish them from Markdown headings (which conventionally use title case) and to follow the Unix tradition of lowercase commands.

Execution Constructs

All execution constructs follow this pattern:

## COMMAND [args] [-- SOURCE] [>>]

Where:

• \#\# — macro sigil.

• COMMAND [args] — the command and its arguments (standard Unix syntax).

• -- SOURCE — optional input source marker (POSIX-style -- followed by a source token).

• >> — output insertion (double arrow for non-consuming, triple for consuming).

No-Argument Execution (\#\#command >>)

The simplest form executes a macro with no input. The macro’s output is inserted at the >>.

Examples.

Common no-input macros:

Today's date: ## date >>
Random quote: ## quote >>
System status: ## health-check >>

Input Sources

When input is needed, the -- SOURCE marker specifies where it comes from. The source token determines the range of text passed to the macro.

Key behavior.

When -- appears with no following token, it means “the entire current line” (preceding and following content). This is a change from earlier drafts — it’s simpler and more intuitive.

Line Input (--)

The most common case: operate on the current line.

The quick brown fox ## rev -- >>

The macro rev receives the entire line “The quick brown fox” (excluding the ## rev -- >> construct) and outputs “xof nworb kciuq ehT”. The result is appended after the construct.

For partial lines, use -- 0 (start to construct) or -- 9 (construct to end):

The quick brown fox ## wc -w -- 0 >>    (counts words before)
## wc -w -- 9 >> fox                     (counts words after)

Document Ranges (-- A, -- Z, -- \%)

These markers work with the entire document:

[full document content...]
## summarize -- A >>    (summarize from start to here)

Introduction...
## keywords -- Z >>     (extract keywords from here to end)

## wordcount -- 

The |in vi/ex commands to mean “entire file.”

Bookmark Ranges (-- name)

A bookmark name as the source token defines an input range from the named bookmark to the current position:

!!data
10
20
30
40

## average -- data >>    (receives "10\n20\n30\n40")

Consuming Execution (>>>)

By default, macros append their output (non-consuming). Use triple arrow >>> to replace the input with the output.

Examples.

The forms below contrast non-consuming and consuming insertion:

Raw data: 10,20,30 ## normalize -- >>>

The line is replaced with the normalized result. The original “Raw data: 10,20,30” is gone.

!!draft
[rough notes...]
## format -- draft >>>

The bookmark range is replaced by the formatted version.

## restructure -- A >>>    (from start to here is replaced)

Pipelines

Pipelines connect multiple commands, with the output of each feeding into the next. Only the first command may have an input source.

Syntax.

## COMMAND1 [args] [-- SOURCE] COMMAND2 [args] | ... | COMMANDn [args] >>|

Examples.

Three typical pipelines:

## cut -d, -f2 -- | sort -n | uniq -c >>

This extracts the second field from the current line, sorts numerically, counts unique values, and appends the result.

!!data
10,20,30,10,20,30
## cut -d, -f1 -- data | sort | uniq -c >>

Pipeline from bookmark data.

## extract -- A | analyze | format >>>

Consuming pipeline: from document start to here is replaced by final formatted output.

Pipeline rules.

• Only the first command may have an input source (-- SOURCE).

• All subsequent commands receive stdin from the previous command’s stdout.

• The final output is inserted/replaced according to the final >> or >>>.

• Spaces around +|+ are optional.

Macro texts

A macro is any text that can be executed. Macros are identified at the filesystem level — how a tool determines that a text is executable is implementation-defined. Common approaches:

• File extension (.sh.jot, .py.jot, .js.jot).

• Shebang-like first line (#!jot python).

• Metadata (tag ..executable, category ::macro).

• Configuration (.jot.toml mapping).

Declaring a Macro

A macro text contains:

1. The macro code in whatever language the implementation supports.

2. Optional metadata (tags, categories, bookmarks) that are parsed normally.

Example macro text (ID 42 or file summarize.jot):

..executable ::python
#!jot python
import sys
text = sys.stdin.read()
# ... summarization logic ...
print(summary)

Input and Output

Input.

The macro receives its input via stdin. The input is plain text (UTF-8). No additional metadata is passed.

Output.

The macro writes its output to stdout. The output may contain Iota Syntaxis constructs, which will be processed by subsequent stages (parser and renderer). This allows macros to generate dynamic tags, references, bookmarks, or even further execution constructs (though nested execution is not recommended — see §4.4).

Exit code.

A zero exit code indicates success. Non-zero exit codes trigger error handling (see §4.3).

Error Handling

If a macro execution fails (non-zero exit code, timeout, or other error):

1. The preprocessor replaces the execution construct (or pipeline) with an error message.

2. The error message SHOULD include: the macro name(s), the error, and optionally the location.

3. Processing continues — one failing macro does not halt the entire document.

Example error output.

A failed single-macro execution might render as:

[!-- ERROR in macro 'analyze': division by zero --]

For pipelines, the error identifies which stage failed:

[!-- ERROR in pipeline stage 'average': invalid input --]

The exact format is implementation-defined but SHOULD be visually distinct and clearly indicate an error.

Important Restrictions

No nested execution.

Macros MUST NOT contain execution constructs (##name, pipelines, etc.) that are intended to be expanded during the same preprocessor pass. Implementations SHOULD detect and reject nested execution to avoid recursion and complexity. (Macros may generate text that looks like execution constructs, but those will be treated as plain text in the current pass.)

No side effects.

Macros SHOULD be idempotent and free of side effects. The preprocessor MAY cache results for identical inputs within a single run.

Sandboxing.

Implementations SHOULD run macros in a sandboxed environment with restricted filesystem and network access. Security is implementation-defined but strongly encouraged.

Architecture: The Preprocessor Stage

The Iota Syntaxis Preprocessor Specification introduces a new preprocessor stage that runs before the Iota Syntaxis core scanner. This stage handles macro expansion and inline includes, transforming execution constructs and include directives into their output.

Preprocessor

The preprocessor:

1. Scans the input text for execution constructs (single macros and pipelines) and inline include directives.

2. For each execution construct, identifies the input source (if any) and gathers the appropriate text.

3. For pipelines, executes each macro in sequence, piping output to input.

4. Replaces the construct (and possibly input) with the final macro’s output based on >> vs >>>.

5. For each inline include directive, resolves the target text or resource and substitutes its content in place.

6. Produces expanded text that contains only Iota Syntaxis core constructs (no execution syntax, no inline include directives).

The expanded text is then fed to the standard Iota Syntaxis core scanner, parser, and renderer.

Updated Pipeline

┌──────────────┐
source text ->│ Preprocessor │ (preprocessor: macro expansion, inline includes)
              └──────┬───────┘
                     │ expanded text (Jot core constructs only)
                     ▼
              ┌──────────────┐
              │   Scanner    │ (core)
              └──────┬───────┘
                     │ scan results
              ┌──────┴──────┐
              ▼             ▼
         ┌────────┐   ┌──────────────┐
         │ Parser │   │   Renderer   │ (core)
         └───┬────┘   └──────┬───────┘
             │               │
       structured        transformed
        metadata           text

Key insight.

The preprocessor outputs only Iota Syntaxis core syntax. This means:

• Macros can generate tags, categories, references, bookmarks, etc.

• Inline includes are resolved and substituted before the scanner ever sees them.

• The rest of the toolchain doesn’t need to know about execution or includes.

• Iota Syntaxis core tools that don’t implement the preprocessor will see the raw execution constructs and inline include directives (which is fine — they remain valid text).

Inline Includes

Syntax.

--<ID, --<SPACE--ID, //<FILEPATH, ///<SPACE/FILEPATH, ;;<FILENAME.

Inline includes are a preprocessor-level feature. They do not affect the parser’s metadata extraction.

Syntax and Examples

Each form may optionally carry a single specifier — a line fragment specifier (defined in SPEC.md §1.4.11) or a bookmark range specifier (defined in SPEC.md §1.4.10) — placed immediately after the ID, filepath, or filename to inline a subset of the target’s content.

The --< directive instructs the preprocessor to inline the content of the named text directly into the rendered output. The //< include directive works analogously with file-referenced texts. The ///< directive inlines a file-text from a named space (cross-space file include). The ;;< include directive inlines the content of a resource.

Format Compatibility

Inlining only makes sense if the source and the target texts or resources are of the same format. Otherwise, inlining can only produce valid output if enclosed in a preformatted block (e.g. <pre> in HTML or a \begin{verbatim}...\end{verbatim} block in LaTeX). By default, the preprocessor SHOULD reject inlining a different format.

While the format of a text is usually given by the file extension, in a heterogeneous collection it may not be obvious. Therefore, when processing an inlined reference --<42 in a heterogeneous collection, the preprocessor MUST look up the format of the source text --42 in order to validate format compatibility.

Error Handling

When the target of an inline include cannot be resolved — for example, because the referenced text or resource has been deleted or is otherwise unavailable — the preprocessor SHOULD emit a warning and leave the include construct as-is in the output. The unavailability of an included target must not prevent the current text from being processed. Implementations MAY offer a strict mode that treats unresolvable includes as errors.

For a full classification of inline include error conditions, see §8.1.

Formal Grammar

Add the following productions to the Iota Syntaxis core grammar. Note that these constructs are matched only by the preprocessor, not by the Iota Syntaxis core scanner.

<lowercase-letter>  ::= "a" | "b" | "c" | "d" | "e" | "f" | "g" | "h" | "i"
                      | "j" | "k" | "l" | "m" | "n" | "o" | "p" | "q" | "r"
                      | "s" | "t" | "u" | "v" | "w" | "x" | "y" | "z"

<lowercase-word>    ::= <lowercase-letter> <word-char>*

<exec-construct>    ::= <construct-boundary> "##" <ws>* <pipeline> <ws>* <output-arrow>

<pipeline>          ::= <command> ( <ws>* "|" <ws>* <command> )*
<command>           ::= <lowercase-word> <macro-args>*

<macro-args>        ::= ( <ws>+ <arg> )*
<arg>               ::= <unquoted-arg> | <quoted-arg>
<unquoted-arg>      ::= <arg-char>+
<arg-char>          ::= any character except whitespace, "|", ">", and control chars
<quoted-arg>        ::= "'" <sq-char>* "'" | '"' <dq-char>* '"'
<sq-char>           ::= any character except "'"
<dq-char>           ::= any character except '"'

<input-source>      ::= "--" ( <ws>+ <source-token> )?
<source-token>      ::= "0" | "^" | "9" | "$"
                      | "A" | "a" | "gg"
                      | "Z" | "z" | "G"
                      | "                      | <word>   (* bookmark names *)

<output-arrow>      ::= ">>" | ">>>"

Note: the <input-source> production appears only on the first command of a pipeline, but this semantic rule is enforced by the preprocessor, not the grammar.

The following productions define inline includes. <line-fragment-specifier> is defined in SPEC.md §1.7.13; <bookmark-range-specifier> is defined in SPEC.md §1.7.14; <word>, <filepath>, <filename>, and <construct-boundary> are defined in SPEC.md §1.7.1.

<inline-include>        ::= <id-include> | <cross-id-include> | <file-include> | <cross-file-include> | <resource-include>
<id-include>            ::= <construct-boundary> "--<" <word>                  ( <line-fragment-specifier> | <bookmark-range-specifier> )?
<cross-id-include>      ::= <construct-boundary> "--<" <word> "--" <word>      ( <line-fragment-specifier> | <bookmark-range-specifier> )?
<file-include>          ::= <construct-boundary> "//<" <filepath>              ( <line-fragment-specifier> | <bookmark-range-specifier> )?
<cross-file-include>    ::= <construct-boundary> "///<" <word> "/" <filepath>  ( <line-fragment-specifier> | <bookmark-range-specifier> )?
<resource-include>      ::= <construct-boundary> ";;<" <filename>              ( <line-fragment-specifier> | <bookmark-range-specifier> )?

The <cross-id-include> form (--<SPACE--ID) inlines a text from a named space, mirroring the cross-space ID reference syntax. The <cross-file-include> form (///<SPACE/FILEPATH) inlines a file-text from a named space, mirroring the cross-space file reference syntax. When a specifier is present, the preprocessor includes only the selected portion of the target: a line range for line fragment specifiers, or the content between two bookmarks for bookmark range specifiers.

Inline includes are matched and expanded by the preprocessor only. The Iota Syntaxis core scanner, parser, and renderer do not process them.

Conformance

A conforming tool or library that implements the Iota Syntaxis Preprocessor Specification also implements all of Iota Syntaxis core conformance plus:

1. Provides a preprocessor that correctly expands all execution constructs (##command >>, ##command -- SOURCE >>, pipelines, and consuming >>> variants) as specified in §3.

2. Enforces the lowercase macro name rule: macro names MUST begin with a lowercase letter. Implementations SHOULD report a parse error for macro invocations with uppercase names.

3. Correctly identifies input sources and gathers the appropriate text:

   • -- : entire current line (excluding the construct).

   • -- 0 / -- ^ : from line start to construct.

   • -- 9 / -- $ : from construct to line end.

   • -- A / -- a / -- gg : from document start to current position.

   • -- Z / -- z / -- G : from current position to document end.

   • |– the construct).

   • -- NAME : from bookmark NAME to current position.

   • No -- : empty input.

4. For pipelines, correctly connects the output of each macro to the input of the next, and ensures only the first command may have an input source.

5. Respects the distinction between non-consuming (>>) and consuming (>>>) output insertion.

6. Handles macro execution in a sandboxed environment appropriate to the implementation.

7. Reports macro execution errors (including pipeline stage failures) by replacing the construct with a visible error message, without halting processing.

8. Detects and rejects nested execution (macros containing execution constructs) to prevent recursion.

9. Ensures the preprocessor output contains only Iota Syntaxis core constructs (no execution syntax remains).

10. Correctly resolves all five forms of inline include directives (--<ID, --<SPACE--ID, //<FILEPATH, ///<SPACE/FILEPATH, ;;<FILENAME) as specified in §6, substituting the content of the target text or resource in place of the directive.

11. Correctly parses line fragment specifiers and bookmark range specifiers on inline include directives, and includes only the selected portion of the target text or resource.

12. Validates format compatibility before inlining (see §6.2) and rejects format-mismatched includes by default.

13. Reports inline include errors (unresolvable targets, format mismatches, circular includes, fragment-out-of-range) as specified in §8.1, without halting processing.

A tool MAY implement a subset of macro languages (e.g., only Python, only shell) and still conform, provided it clearly documents which languages it supports.

Error Conditions

Inline Include Errors

Codes are assigned per the canonical-code policy in SPEC.md §3.1.1.