Abstract

Extensible languages allow a programmer to import into his or her "host" language a collection of domain-specific language features that raise the level of abstraction of the language. Such features typically add new syntactic constructs and semantic analyses to the host language. Among the challenges to implementing such extensible languages is the problem of scanning and parsing. In this talk, I describe parsing and context-aware scanning algorithms in which the scanner uses contextual information from the parser to disambiguate lexical syntax. The parser utilizes a slightly modified LR-style algorithm that passes to the scanner the set of valid symbols that the scanner may return at that point in parsing. I will also describe an analysis that checks that a deterministic scanner and parser can be generated from the declarative specifications in which regular expressions specify the scanner and a context free grammar specifies the parser.