Background
Occasionally, you may need to process a small, special-purpose language.
Essentially, you have only a few choices, One choice is to roll your own parser (and lexical analyzer). If you are not an expert, this is hard, if you are an expert, it is still time consuming
An alternative choice is to use a parser generator, there exist quite a few of these generators. some of the better known are Yacc and Bison for parser written in C and ANTLR for parer written in Java.
You will probably also need a scanner generator such as Lex, Flex, or JFlexto go with it. However, you need to learn new tools, including their sometimes obscure - error messages.
This chapter presents a third alternative. Instead of using the standalong domain specific language of a parser generator, you will learn an internal domain specific language, or internal DSL for short. The internal DSL will consist of a library of parser combinator - functions and operatos defined in Scala that will server as building block for parsers. These building block will map one to one to the constructions of a "context-free" grammer, to make it easy to understand
We will show by an example
Example : Arithmetic expressions
suppose that the grammer for an arithmetic expression is as follow.
/*
expr ::= term { "+" term | "-" term }.
term ::= factor { "*" factor | "/" factor }.
factor ::= floatingPointNumber | "(" expr ")"
| : denote alternative production
{...}: denotes repetition (zero or more times)
[...]: means an optional occurrence
*/
import scala.util.parsing.combinator._
class Arith extends JavaTokenParsers {
def expr: Parser[Any] = term ~ rep ("+" ~ term | "-" ~ term)
def term: Parser[Any] = factor ~ rep("*" ~ factor | "/" ~ factor)
def factor: Parser[Any] = floatingPointNumber | "(" ~ expr ~ ")" // what is the definition of the floatingPointNumber, is that an internal one?
// actually, it is defined in trait JavaTokenParser, the definition being: def floatingPointNumber: Parser[String] = """-?(\d+(\.\d*)?|\d*\.\d+)([eE][+-]?\d+)?[fFdD]?""".r
}- every production becomes a method
- results of each method is Parser[Any], so you need to change ::= to ": Parser[Any]" , you can make it more precise, we will show later
- in the grammer, sequential composition was implicit, in program, it is expressed by an explicit operator: ~.
- Repetition is expressed rep(...) instead of {...}. Analogously, option is expressed opt(...) instead of [...]
- The period (.) at the end of each production is omitted - you can, however, write a semicolon (;) if you prefer
Running the Parser
Nowe we have the grammer and we have formalize that into combinator syntax.
// running your parser
import scala.util.parsing.combinator._
object ParseExpr extends Arith {
def main(args : Array[String]) {
println("input : " + args(0))
println(parseAll(expr, args(0)))
}
}to run the main method.
ParseExpr.main(Array[String]("2 * ( 3 * 7)"))
//
// input : 2 * (3 + 7)
// [1.12] parsed: ((2~List((*~(((~((3~List())~List((+~(7~List())))))~)))))~List())
ParseExpr.main(Array[String]("2 * (3 + 7))")) // deliberately something wrong to the syntax
//
//[1.12] failure: string matching regex `\z' expected but `)' found
//
//2 * (3 + 7))
// ^
Basic regular expressions parsers
we have seen from above combinator, which has floatingPointNumber, this is inherited from Arith's super trait, which is called JavaTokenParsers. That uses a regular expresssion parser. The idea is that y ou can use any regular expression as a parser, the regular expression parses all strings it can match. the result is the parsed string.
import scala.util.parsing.combinator._
object MyParsers extends RegexParsers {
val ident : Parser[String] = """[a-zA-Z_]\w*""".r
// to test
def main(args : Array[String] ) {
println("input " + args(0))
println(parseAll(ident, args(0)))
}
}
MyParsers.main(Array[String]("joe"))
MyParsers.main(Array[String]("-joe")) // fail
MyParsers.main(Array[String]("joe.wang")) // fail
MyParsers.main(Array[String]("123")) // failAnother example : JSON
/*
value ::= obj | arr | stringLiteral |
floatingPointerNumber |
"nul" | "true" | "false"
obj ::= "{" [members] "}"
arr ::= "{" [values] "}"
members ::= member {"," member}
member ::= stringLiteral ":" value
values ::= value {",", value}
*/{
"address book": {
"name": "John Smith",
"address" : {
"street": "10 Market Street",
"city" : "San Francisco, CA",
"zip" : 94111
},
"phone numbers": [
"408 338-4238",
"408 111-6892"
]
}
}import java.io.FileReader
import scala.util.parsing.combinator._
class JSON extends JavaTokenParsers {
def value : Parser[Any] = obj | arr | stringLiteral | floatingPointNumber | "null" | "true" | "false"
def obj : Parser[Any] = "{" ~ repsep(member, ",") ~ "}"
def arr : Parser[Any] = "["~ repsep(value, ",") ~ "]"
def member : Parser[Any] = stringLiteral ~ ":" ~ value
}object ParseJSON extends JSON {
def main(args : Array[String]) {
val reader = new FileReader(args(0))
println(parseAll(value, reader))
}
}// scala ParseJSON address-book.json
ParseJSON.main(Array[String]("address-book.json")) // you might need to given the right path for the *.json fileParser output
If you run the command, you might get results such as
[14.2] parsed: (({~List((("address book"~:)~(({~List((("name"~:)~"John Smith"),
(("address"~:)~(({~List((("street"~:)~"10 Market Street"), (("city"~:)~"San Fran
cisco, CA"), (("zip"~:)~94111)))~})), (("phone numbers"~:)~(([~List("408 338-423
8", "408 111-6892"))~]))))~}))))~})the output of the previous JSON program is that it does not have intuitive meanings. It seems to be a sequence composed of bits and pieces of the input glued together with lists and ~ combinations.
and check on the code, each and every combinator factor returns just a Parser[Any], it does not reflect the business model, the JSON object a more likely result type should be Map[String, Any].. and JSON array shuld be List[Any]., values "true", "false", should just be true, false.
there is an operator called "^^" which can help you do the result transformation.
Also, from the output, something like ~(~("{", ms, "}"), this is illegible. why we keep the "{" or "}" in the result? the combinator has introduced the fllowign operator, which allow you to just keep the left or right match . they are ~> and <~
now, equipped with the new operator, we have this:
import scala.util.parsing.combinator._
class JSON1 extends JavaTokenParsers {
def obj : Parser[Map[String, Any]] =
"{" ~> repsep (member, ",") <~ "}" ^^ (Map() ++ _)
def arr : Parser[List[Any]] =
"[" ~> repsep (value, ",") <~ "]"
def member : Parser[(String, Any)] =
stringLiteral ~ ":" ~ value ^^
{ case name ~ ":" ~value => (name, value) }
def value : Parser[Any] = (
obj
| arr
| stringLiteral
| floatingPointNumber ^^ (_.toDouble)
| "null" ^^ (x => null)
| "true" ^^ (x => true)
| "fale" ^^ (x => false)
)
}
import java.io.FileReader
object JSON1Test extends JSON1 {
def main(args : Array[String]) {
val reader = new FileReader(args(0))
println(parseAll(value, reader))
}
}
[14.2] parsed: Map("address book" -> Map("name" -> "John Smith", "address" -> Ma
p("street" -> "10 Market Street", "city" -> "San Francisco, CA", "zip" -> 94111.
0), "phone numbers" -> List("408 338-4238", "408 111-6892")))As a side note, since there is no need to explicitly insert some semi-colon, so you write
def value : Parser[Any] =
obj |
arr |
stringLiteral |
...obj; // semicolon implicitly inserted
| arrand the summary of the Parser combinators
| "..." |
literal |
| "...".r |
regular expressions |
| P~Q |
sequential composition |
| P<~Q, P~>Q | sequential composition; keep left/right only |
| P | Q |
alternative |
| opt(P) |
option |
| rep(P) |
repetition |
| repsep(P, Q) |
interleaved repetition |
| p ^^ f |
result conversion |
Why we hvae those symbolic operators, what if we have alphabetic ones, it has too much visual estate. and the symbolic operators are specially chosen so that it has decreasing order of precedence.
suppose that we have the following alphabetic operators.
// suppose for a moment that we have only those alphabetic ones instead of the symbolic ones.
class ArithHypothetical extends JavaTokenParsers {
def expr : Parser[Any] =
term andThen rep(("+" andThen term) orElse
("-" andThen term))
def term: Parser[Any] =
factor andThen rep(("*" andThen factor) orElse
("/" andThen factor))
def factor : Parser[Any] = floatingPointNumber orElse ("(" andThen expr andThen ")")
}Implementing combinator parsers
The previous sections have shown that Scala's combinator provides a convenient means for constructing your own parsers.
the core of Scala's combinator parsing framework is contained in the trait scala.util.parsing.combinator.Parsers
package scala.util.parsing.combinator
trait Parser {
... // code goes here unless otherwise stated.
}
// a parser is in essence just a function from some input type to a parser result
type Parser[T] = Input = ParseResult[T]Parser input
// reader is from scala.util.parsing.input, it is similar to Stream, but it also keeps track of the positions of all the elements it reads.
type Input = Reader[Elem]Parser Results
sealed abstract class ParseResult[+T]
case class Success[T] (result : T, in :Input) extends ParseResult[T]
case class Failure (msg: String, in :Input) extends ParseResult[Nothing] T stands for the type of results that returned by a success match, while the second parameter, in, is used for chaining.
for failure, the None denote nothing result returned, but the second in is used for not chaining, but positioning.
The parser class
abstract class Parser[+T] extends (Input => ParserResult[T]) {
p =>
// an unspecified method that denote
// the behavior of this parser
def appply(in : INput) : ParserResult[T]
def ~ ...
def ! ...
}any case inherited from the parent type Input => ParserResult[T]. (Input => ParserResult[T]) is an abbreviation for scala.Function1[Input, ParserResult[T]).
Alias this
abstract class Parser[+T] extends (Input => ParserResult[T]) { p =>val id = this
use of this alias "is":
class Outer { outer =>
class Inner {
println(Outer.this eq outer) // prints : trueSingle-token parsers
def elem(kind : String, p : Elem => Boolean) =
new Parser[Elem] {
def apply(in : Input) =
if (p(in.first)) Succes(in.first, in.rest)
else Failure(Kind + " expected ", in)
}Sequential composition
// as we can know that ~ is case class, and with element of type T and U
abstract class Parser[+T] extends (Input => ParserResult[T]) { p =>
def ~[U] (q : => Parser[U]) = new Parser[T ~U] {
case Success(x, in1) =>
q(in1) match {
case Success(y, in2) => Success(new ~(x, y), in2)
case failure => failure
}
case failure => failure
}
}def <~ [U](q : => Parser[U]) : Parser [T] = (p ~q) ^^ { case x ~ y => x }
def ~> [U](q : => Parser[U]) : Parser [U] = (p ~q) ^^ { case x ~ y => y }alternative compositions
def | (q : => Input) => new Parser[T] {
def apply(in : Input) = p(in) match {
case s1 @ Success(_, _) => s1
case failure => q(in)
}
}Dealing with recursion
Note that the q parameter is method ~ and | is by-name - its type is precede by =>, this means that the actual parser argumetn will be evaluated only when q is needed. which should only be the case after p has run. this make sit possible to write recursive parser like the following.
def parens = floatingPointNumber | "(" ~ parens ~ ")"Result conversion
def ^^ [U](f : T => U) : Parser[U] = new Parser[U] {
def apply(in : Input) = p(in) match {
case Success(x, in1) => Success(f(x), in1)
case failure => failure
}
} // end ParserParsers that don't read any input
success and failure does not consume any input.
def success[T] (v : T) = new Parser[T] {
def apply(in :Input) = Success(v, in)
}
def failure(msg : String) = new Parser[Nothing] {
def apply(in : Input) = Failure(msg, in)
}Option and Repetition
def opt[T](p : => Parser[T]): Parser[Option[T]] = (
p ^^ Some(_)
| success(None)
)
def rep[T](p : => Parser[T]): Parser[List[T]] = (
p ~ rep(p) ^^ { case x~xs => x :: xs }
| success(List())
)
def repsep[T](p : => Parser[T], q : => Parser[Any]): Parser[List[T]] = (
p ~ rep(q ~> p) ^^ { case r~rs => r :: rs }
| success(List())
) // end ParsersString Literals and regular expressions
the definition of the RegexParsers is
trait RegexParsers extends Parsers {
type Elem = Charimplicit def literal(s : String) : Parser[String] = ...
implicit def regex(r : Regex) : Parser[String] = ...RegexParsers trait also takes care of handling white space, it uses the regular expression
protected val whiteSpace = """\s+""".rbut you can choose to override it
// you can choose to override it
object MyParsers extends RegexParsers {
override val whiteSpace = "".r
...
}lexing and Parsing
the task of syntax analysis is often split into two phase,
the lexer phaser: recognize individual words into the input and classifies them into some token classes. the phase is also called lexical analysis.
the syntactical analysis: sometime is called parsing,
The parsers described in the previous section can be used for either phase, because its input element are of the abstract type Elem.
Scala's parsing combinations provides several utilities classes for lexical and syntactic analysis. they are comtained in two sub-packages, one for each kind of analysis
scala.util.parsing.combinator.lexical
scala.util.parsing.combinator.syntacticalError reporting
Scala's parsing library implements a simple heuristic: among all failures, the one that occurred at the latest position in the input is chosen.
supose for the json examples:
{ "name" : Johnyou will have the following
[1.13] failure : "false" expected but identifier John found
{ "name" : John
^def value : Parser[Any] =
obj | arr | stringLit | floatingPointNumber | "null" | "true" | "false" | failure("illegal start of value")[1.13] failure : illegal start of value
{ "name" : John
^var lastFailure : Option[Failure] = Nonethe field is initialized to None, it is updated in the constructor of the failure class:
case class Failure(msg : String, in : input) {
if (lastFailure.isDefined && lastFailure.get.in.pos <= in.pos)
lastFailure = Some(this)
}
def phrase[T] (p : Parser[T]) = new Parser[T] {
lastFailure = None
def apply(in : Input) = p(in) match {
case s @ Successs(out, in1) =>
if (in1.atEnd) s
else Failure("end of inpput expected", in1)
case f : Failure =>
lastfailure
}
}Backtracking vesus LL(1)
The mainstream compiler do not use backtracking, why?
However, backtracking has imposed restriction
e.g.
expr ::= expr + "+" term | termprogress never further and it is potentially costly because the same input can be parsed serveral times.
2. Maybe too costy
expr ::= term "+" expr | termfor input "(1 + 2) * 3", because first try will fail and second try succeed, we waste one try here
So, it is common to modify the gramer, so that backtracking can be avoided. e..g , either one of the following works.
expr ::= term ["+" expr]
expr ::= term {"+" expr}that have already been parsed. using this, we can reformat the arithmetic expression alternatively as follow
def expr : Parser[Any] =
term ~! rep("+" ~! term | "-" ~! term)
def term : Parser[Any] =
term ~! rep("*" ~! factor | "/" ~! factor)
def factor : Parser[Any] =
"(" ~! expr ~! ")" | floatingPointNumberConclusion
downside of Combinator in Scala: not very effecient
- comparing to tools generator ones, e.g. Yacc, Bison, , backtracking method not effecient
- they mix parser construction and input alaysis in the same set of operation, in effect, parser is generated anew for each input that is parsed.
- you factor out parser construction from input analysis, you can construct parser once and parse all inputs
- the parser generation can use more efficient algorithm such as LALR(1)..