Writing a simple Lexer in PHP/C++/Java

catalog

0. Comparison of parser generators
1. Writing a simple lexer in PHP
2. phc
3. JLexPHP: A PHP Lexer(xx.lex.php) Created By Java By x.lex File Input
4. JFlex
5. JLex: A Lexical Analyzer Generator for Java
6. PhpParser

 

0. Comparison of parser generators

Relevant Link:

https://en.wikipedia.org/wiki/Comparison_of_parser_generators
http://www2.cs.tum.edu/projekte/cup/examples.php
http://pygments.org/docs/lexers/
http://www.phpdeveloper.org/news/5678

 

1. Writing a simple lexer in PHP

0x1: Introduction

Router::connect('/login', array('Sessions::add'));

The map /login -> Sessions::add may be translated into the following token stream by the lexer

"map">
"/login">
"->">
"Sessions::add">

he parser can parse the following notation

     := [\s]

very rule identified by a := is called a production rule. I leave the parsing part for a later post, because that would be too much for a single posting

0x2: Implementation

The basic idea is that we match a list of regexes against the current line. If one of them matches, we store that token and advance our offset to the first character after the match. If no token is found and we are not yet at the end of the line, raise an exception (because something invalid is in front of our offset).
Let's stick with the example already mentioned above. We want to create tokens for an input file like map /login -> Sessions::add or root -> Pages::home.
The first thing we need is an array of terminal symbols that map to token identifiers.

php
class Lexer {
    protected static $_terminals = array(
        "/^(root)/" => "T_ROOT",
        "/^(map)/" => "T_MAP",
        "/^(\s+)/" => "T_WHITESPACE",
        "/^(\/[A-Za-z0-9\/:]+[^\s])/" => "T_URL",
        "/^(->)/" => "T_BLOCKSTART",
        "/^(::)/" => "T_DOUBLESEPARATOR",
        "/^(\w+)/" => "T_IDENTIFIER",
    );
}
?>

Now, let's implement a run method that accepts an array of source lines and returns an array of tokens. It calls the helper _match method that performs the actual matching and raises an exception if no token was found.

public static function run($source) {
    $tokens = array();
 
    foreach($source as $number => $line) {            
        $offset = 0;
        while($offset < strlen($line)) {
            $result = static::_match($line, $number, $offset);
            if($result === false) {
                throw new Exception("Unable to parse line " . ($line+1) . ".");
            }
            $tokens[] = $result;
            $offset += strlen($result['match']);
        }
    }
 
    return $tokens;
}

ote that how we advance the offset further every iteration and work towards the end of the string. To get the full picture, here's the _match helper method.

protected static function _match($line, $number, $offset) {
    $string = substr($line, $offset);
 
    foreach(static::$_terminals as $pattern => $name) {
        if(preg_match($pattern, $string, $matches)) {
            return array(
                'match' => $matches[1],
                'token' => $name,
                'line' => $number+1
            );
        }
    }
 
    return false;
}

We use the preg_match method to check if one of our pattern matches the current string. If you look closely, you can see that all of our regexes start at the beginning of the line (^) and are enclosed (()) so we can find them exactly at the beginning and get the inner content. We also store the current line, because we need it in our parser and to display helpful error messages.
Let's run our lexer with some example input:

$input = array('root -> Foo::bar');
$result = Lexer::run($input);
var_dump($result);

0x3: Wrapping Up

One thing you need to be aware of is that you need to place your token regex in the correct order - namely from special to general. If we put the T_IDENTIFIER before T_ROOT, the root keyword would always be matched as an identifier. While I haven't tried it out yet
在一个完备的词法解析器中，匹配状态机应该是采取"贪婪策略"，即尽可能地匹配最长的"词法模式"

Relevant Link:

http://nitschinger.at/Writing-a-simple-lexer-in-PHP
http://www.codediesel.com/php/building-a-simple-parser-and-lexer-in-php/

 

2. phc

phc is an open source compiler for PHP with support for plugins. In addition, it can be used to pretty-print or obfuscate PHP code, as a framework for developing applications that process PHP scripts, or to convert PHP into XML and back, enabling processing of PHP scripts using XML tools.

1. phc for PHP programmers  
    1) Compile PHP source into an (optimized) executable (supports entire PHP standard library).
    2) Compile a web application into an (optimized) extension (supports entire PHP standard library).
    3) Pretty-print PHP code.
    4) Obfuscate PHP code (--obfuscate flag - experimental).
    5) Combine many php scripts into a single file (--include flag - experimental).
    6) Optimize PHP code using classical compiler optimizations (in the dataflow branch - very experimental).

2. phc for tools developers 
    1) Analyse, modify or refactor PHP scripts using C++ plugins.
    2) Convert PHP into a well-defined XML format, process it with your own tools, and convert it back to PHP.
    3) Operate on ASTs, simplified ASTs, or 3-address code.
    4) Analyse or optimize PHP code using an SSA-based IR (in the dataflow branch - very experimental).

0x1: Installation Instructions

1. g++ version 3.4.0 or higher
2. make
3. Boost version 1.34 or higher
4. PHP5 embed SAPI (version 5.2.x recommended; refer to PHP embed SAPI installation instructions for more details). This is required to compile PHP code with phc.
5. Xerces-C++ if you want support for XML parsing (you don’t need Xerces for XML unparsing).
6. Boehm garbage collector is used in phc, but not in code compiled by phc. If unavailable, it can be disabled with --disable-gc, but phc will leak all memory it uses.
//The following dependencies are optional:
7. a DOT viewer such as graphviz if you want to be able to view the graphical output generated by phc (for example, syntax trees)
/*
Under Debian/Ubuntu, the following command will install nearly all dependencies:
apt-get install build-essential libboost-all-dev libxerces27-dev graphviz libgc-dev
*/

0x2: Running phc

php
   echo "Hello world!";
?>

phc -c helloworld.php -o helloworld
//This creates an executable helloworld, which can then be run
./helloworld

0x3: Traversing the Tree

php
   $x = 5;
   if($x == 5)
      echo "yes";
   else
      echo "no";
?>

phc解析得到的AST(抽象语法树)如下

Relevant Link:

http://www.phpcompiler.org/downloads.html
http://www.phpcompiler.org/ 
http://www.phpcompiler.org/doc/latest/install.html 
http://www.phpcompiler.org/doc/latest/runningphc.html
http://www.phpcompiler.org/doc/latest/treetutorial1.html#treetutorial1
http://www.phpcompiler.org/doc/latest/manual.html

 

3. JLexPHP: A PHP Lexer(xx.lex.php) Created By Java By x.lex File Input

A lexer generator for PHP. It is based on JLex and requires Java to generate the lexer. Once generated, the lexer only requires PHP to run

0x1: x.lex(词法规则文件)

php
include 'jlex.php';

%%

%{
// L? \" (\\.|[^\\\"])* \"    { $this->createToken(CParser::TK_STRING_LITERAL); }
    /* blah */
%}

%function nextToken
%line
%char
%state COMMENTS

ALPHA=[A-Za-z_]
DIGIT=[0-9]
ALPHA_NUMERIC={ALPHA}|{DIGIT}
IDENT={ALPHA}({ALPHA_NUMERIC})*
NUMBER=({DIGIT})+
WHITE_SPACE=([\ \n\r\t\f])+

%%

 {NUMBER} { 
      return $this->createToken();
}
 {WHITE_SPACE} { }

 "+" { 
      return $this->createToken();
} 
 "-" { 
      return $this->createToken();
} 
 "*" { 
      return $this->createToken();
} 
 "/" { 
      return $this->createToken();
} 
 ";" { 
      return $this->createToken();
} 
 "//" {
      $this->yybegin(self::COMMENTS);
}
 [^\n] {
}
 [\n] {
      $this->yybegin(self::YYINITIAL);
}
 . {
      throw new Exception("bah!");
}

0x2: Lexer Generator(By Java Language)

//create the jar file
javac -Xlint:unchecked JLexPHP/Main.java
jar cvf JLexPHP.jar JLexPHP/*.class
//负责读取词法规则文件，并生成Lexer 
java -cp JLexPHP.jar JLexPHP.Main simple.lex 

编译得到simple.lex.php，这个.php文件中包含了PHP Lexer的代码逻辑

0x3: 调用simple.lex.php、解析PHP文件词法

php
    $scanner = new Yylex(fopen("file", "r"));
    while ($scanner->yylex())
        ;

?>

Relevant Link:

https://github.com/wez/JLexPHP/blob/master/JLexPHP/Main.java
https://github.com/wez/JLexPHP
http://wezfurlong.org/blog/2006/nov/parser-and-lexer-generators-for-php/

 

4. JFlex

1. JFlex is a lexical analyzer generator (also known as scanner generator) for Java, written in Java.
2. A lexical analyzer generator takes as input a specification with a set of regular expressions and corresponding actions. It generates a program (a lexer) that reads input, matches the input against the regular expressions in the spec file, and runs the corresponding action if a regular expression matched. 
3. Lexers usually are the first front-end step in compilers, matching keywords, comments, operators, etc, and generating an input token stream for parsers. Lexers can also be used for many other purposes.
4. JFlex lexers are based on deterministic finite automata (DFAs). They are fast, without expensive backtracking.
5. JFlex is designed to work together with the LALR parser generator CUP by Scott Hudson, and the Java modification of Berkeley Yacc BYacc/J by Bob Jamison. It can also be used together with other parser generators like ANTLR or as a standalone tool.

Relevant Link:

http://jflex.de/

 

5. JLex: A Lexical Analyzer Generator for Java

JLex is a lexical analyzer generator, written for Java, in Java

Relevant Link:

http://www.cs.princeton.edu/~appel/modern/java/JLex/
http://www.cs.princeton.edu/~appel/modern/java/JLex/current/manual.html
http://www.cs.princeton.edu/~appel/modern/java/JLex/current/manual.html#SECTION1

 

6. PhpParser

PhpParser generates a pure Java parser for PHP programs. Invoking this parser yields an explicit parse tree suitable for further analysis. This package is based upon

1. JFlex 1.4.1
2. Cup 0.10k
3. Grammar and lexer specifications of PHP 4.3.10. 

0x1: Project settings for IntelliJ IDEA

1. Project > Language Level: 1.7
2. Modules > Sources: only src/java_cup, src/project, src/jFlex

0x2: Building and cleaning the project with Ant from within Eclipse

1. Project > Properties > Builders
2. Deactivate the Java Builder.
3. New ...
4. Select "Ant builder"
5. Name it "Ant build" or "PhpParser build" (or any other suitable name).
6. In the Main tab, select the build.xml in the project directory as Buildfile and the project directory as Base directory.
7. In the Targets tab for "Manual build", select "build".
8. In the Targets tab for "During a clean", select "clean all".
9. OK the changes for both dialogs.
10. Project > Build Project and Clean the project using Project > Clean ...
//Or you can build the project using the command line from within the project main directory
ant build

build.xml

"PhpParser" basedir="." default="build">
    

    
    "src.dir" value="src"/>
    "src.project.dir" value="${src.dir}/project"/>
    "src.spec.dir" value="${src.dir}/spec"/>
    "src.jflex.dir" value="${src.dir}/JFlex"/>
    "src.cup.dir" value="${src.dir}/java_cup"/>

    "build.dir" value="build"/>
    "build.java.dir" value="${build.dir}/java"/>
    "build.class.dir" value="${build.dir}/class"/>

    "lexparse.package" value="at.ac.tuwien.infosys.www.phpparser"/>
    "lexparse.dir" value="${build.java.dir}/at/ac/tuwien/infosys/www/phpparser"/>

    "javadoc.dir" value="doc/html"/>
    "javadoc.lexparse.dir" value="${javadoc.dir}/phpparser"/>

    
    "lexgen.main" value="JFlex.Main"/>
    "lexgen.input" value="${src.spec.dir}/php.jflex"/>
    
    "lexer.name" value="PhpLexer"/>
    "lexer.source" value="${lexer.name}.java"/>
    "lexer.class" value="${lexer.name}.class"/>

    
    "parsegen.main" value="java_cup.Main"/>
    "parsegen.input" value="${src.spec.dir}/php.cup"/>
    
    "parser.name" value="PhpParser"/>
    "parser.source" value="${parser.name}.java"/>
    "parser.sym.name" value="PhpSymbols"/>
    "parser.sym.source" value="${parser.sym.name}.java"/>

    
    "classpath">
        "${build.class.dir}"/>
        
        "${src.jflex.dir}"/>
    


    

    "cup" description="Compiles the modified Cup.">
        "${build.class.dir}"/>
        "${src.cup.dir}" destdir="${build.class.dir}" debug="on"  includeantruntime="false">
            "-encoding GBK"/>     
            "classpath"/>
        
    

    "jflex" description="Compiles the modified JFlex.">
        "${src.jflex.dir}" destdir="${build.class.dir}" debug="on" includeantruntime="false">
            "-encoding GBK"/>
            "classpath"/>
        
    

    "lexer.source" depends="cup,jflex"
        description="Uses the lexer generator to create a Java lexer from the input file.">
        "${lexparse.dir}"/>
        "${lexgen.main}" fork="yes">
            "${lexgen.input}"/>
            "-d"/>
            "${lexparse.dir}"/>
            "classpath"/>
        
    

    "parser.source" depends="cup"
        description="Uses the parser generator to create a Java parser from the input file.">
        "${lexparse.dir}"/>
        "${parsegen.main}" fork="yes">
            "-parser"/>
            "${parser.name}"/>
            "-symbols"/>
            "${parser.sym.name}"/>
            "-nonterms"/>
            "-expect"/>
            "1"/>
            "${parsegen.input}"/>
            "classpath"/>
        
        "${basedir}/${parser.source}" todir="${lexparse.dir}"/>
        "${basedir}/${parser.sym.source}" todir="${lexparse.dir}"/>
    

    "javac"
        description="Internal target for Java development. Doesn't try to generate lexer and parser.">
        "${build.class.dir}"/>
        "${build.class.dir}" debug="on" includeantruntime="false">
            "-encoding GBK"/>
            
                "${src.project.dir}"/>
                "${build.java.dir}"/>
            
            "classpath"/>
        
    

    "javadoc" depends="javac" description="Generates JavaDoc.">
        "${javadoc.lexparse.dir}" packagenames="${lexparse.package}" Windowtitle="PhpParser 1.0">
            
                "${src.project.dir}"/>
                "${build.java.dir}"/>
            
            "classpath"/>
        
    

    "build" depends="lexer.source,parser.source,javac,javadoc"
        description="Builds the whole project together with the generated lexer and parser."/>

    "clean" description="Cleans up.">
        "${build.java.dir}"/>
        "${build.class.dir}"/>
        "${graphs.dir}"/>
        "${jar.file}"/>
    

    "cleanall" depends="clean" description="Cleans up JFlex, Cup and JavaDoc as well.">
        "${lib.dir}/JFlex"/>
        "${lib.dir}/java_cup"/>
        "${javadoc.dir}"/>
    

    "dist">
        "dist"/>
    

    "help">
        "You probably want to do 'ant build'. Otherwise, type 'ant -projecthelp' for help."/>
    

编译得到的Java版本的Lexer解析引擎，我们可以直接在代码中实例化并调用其中的函数

0x3: Usage

Example.java

import at.ac.tuwien.infosys.www.phpparser.*;
import java.io.*;
import java.util.*;

class Example {

    public static void main(String[] args) {

        if (args.length == 0) {
            System.out.println("Please specify one or more PHP files to be parsed.");
            System.exit(1);
        }

        for (int i = 0; i < args.length; i++) {

            String fileName = args[i];
            
            ParseTree parseTree = null;
            try {
                PhpParser parser = new PhpParser(new PhpLexer(new FileReader(fileName)));
                ParseNode rootNode = (ParseNode) parser.parse().value;
                parseTree = new ParseTree(rootNode);
            } catch (FileNotFoundException e) {
                System.err.println("File not found: " + fileName);
                System.exit(1);
            } catch (Exception e) {
                System.err.println("Error parsing " + fileName);
                System.err.println(e.getMessage());
                e.printStackTrace();
                System.exit(1);
            }

            System.out.println("*** Printing tokens for file " + fileName + "...");
            for (Iterator iter = parseTree.leafIterator(); iter.hasNext(); ) {
                ParseNode leaf = (ParseNode) iter.next();
                System.out.println(leaf.getLexeme());
            }
        }
    }

}

编译

pushd D:\eclipse-javaEE\workspace\phpparser\doc\example
javac -classpath ../../build/class Example.java

运行

java -classpath ../../build/class:. Example test1.php test2.php

0x4: Directory layout

build.xml
README
build
    class: generated java class files
    java:generated java source files (PHP Lexer and Parser)
doc
    various documentation files
src
    java_cup: modified version of the Cup parser generator
    jflex: modified version of the JFlex scanner generator
    project: parse tree data structures
    spec: specification (input) files for Cup and JFlex

Relevant Link:

https://github.com/oliverklee/phpparser/blob/master/src/spec/php.cup
https://github.com/oliverklee/phpparser/blob/master/src/spec/php.jflex
https://github.com/oliverklee/phpparser

 

Copyright (c) 2015 LittleHann All rights reserved