Once you start thinking about it, it's mind boggling that we have thousands of languages and yet most of them don't have built-in facilities to construct and parse grammars (at least context-free ones). Every single designer seems to think that their language is finally good enough and will not be used as a starting point for another one.
Perhaps Raku would be of interest to you. It was specifically designed to make it a good starting point for its ongoing replacement, Ship of Theseus style.[1]
----
The first half of this comment paints the big picture of Raku's approach to enabling this.
The second shows actual code demonstrating the approach.
----
Raku has a built in grammar + actions formalism, and corresponding built in parsing/compiling machinery.
Raku's grammars/actions are used to declare, parse, and compile Raku -- it is entirely defined in terms of itself.
----
Raku isn't just one language, but a collection of sub-languages, aka slangs. Each of these is comprised of a grammar/action pair. These are mixed together to form a composed result: "Raku".
----
Any slang can include grammar rules that invoke rules in other slangs with which its mixed. The slangs in standard Raku take advantage of this. Thus a function written in the Main (GPL) slang will accept regexes in some spots in the Main slang, with those regexes being parsed/compiled according to the Regex slang.
This ability for slangs to use each other can be (and is, for the standard slangs) mutually recursive. Thus, just as an ordinary function written in the Main slang can include a regex in its code, so too a regex can include an ordinary function.
This overall approach of weaving slangs together is called a "language braid"[2]. Multiple slangs are seamlessly woven together as if they were just one language -- because in fact they are, despite its declaration being broken out into slang modules.
----
User defined slangs can replace or modify the "official" standard slangs that ship together to comprise "Raku".
Thus, at the most trivial level comes slang modules like Slang::Tuxic.[3] This was one of the first slangs. It is just a few lines of code written by a core Raku dev; it overrides a few existing minor rules to tweak the parsing of the Main slang to make another dev called Tux stop complaining about some syntax decisions he didn't like.
More advanced slangs are internal DSLs such as Slang::SQL.[4] This weaves SQL and standard Raku code so that each can enrich the other.
----
Use of slangs is lexically scoped. That is to say, a slang can be invoked within a particular module, or function, or even just an `if` statement's True block, and at the end of that scope the slang's alteration of Raku vanishes.
And if that was done in an inner scope rather than at the top level, then at the end of that scope, pop, the old Raku would return. (Thus lending a new twist to the Ship of Theseus[1] thought experiment.)
----
This last section includes code showing the approach in action, after a bit of introduction of what I will and won't include.
Raku's grammars declare arbitrary parsers. One can arbitrarily override any part of an existing grammar by composing it into a new grammar with selective replacement of particular parts. But "them's big guns". Instead we'll stick with a simpler mechanism. It's still powerful, but it's particularly easy to use and demonstrate.
A key element underlying what I show below is an alternate rule construct that's available in any Raku grammar. This makes it trivial and practical to add any number of new alternatives for particular grammatical "slots".
The grammars used to declare Raku itself make use of this construct to declare much of its grammar. In particular, the Main slang does and then builds on that in a principled manner to exposes itself to user extension of those parts of its grammar. This is what we're going to see in action.
There are over a dozen of these grammatical "slots". For our example here we'll extend the `postfix` slot:
sub postfix:<!> ( \N ) { # Declare factorial op
N > 2
?? ( N * ( N - 1 )! )
!! N
}
say 5! # 120
The syntax aspect of this extension happens as soon as the opening brace of the operator's definition block is reached. This is why the use of `!` that's inside the block is correctly recognized, despite appearing before the block defining its semantics is even closed.
While the syntax is utterly trivial (literally just the token `!` where the Main slang accepts postfix ops), the semantics are defined by the arbitrary body of the declaration. This body compiles into corresponding AST that is added to the MAIN slang's semantic actions class.
To be crystal clear, this all happens at compile time, and this new op would be part of a module's compiled result if it were in that module, despite the code that declares the op being ordinary Raku code.
