Extending the lexer

Convert code or other text into a structured tree (multi-dimensional array).

In This File:

• create a grammar with directives & handler functions
• test a grammar
• a complex directive that builds an AST without php
• How to write an extension

Extending

To extend the lexer, we will do four things, iteratively.

1. Create a Grammar
2. Create an array of Directives
3. Create a trait for callbacks Directives use
4. Write Directive tests

To run it, I suggest using your preferred test suite. Though, the built-in directive tests (composer) require taeluf/tester

1. Grammar

We'll be recreating part of the Bash Grammar.

<?php

class MyBashGrammar extends \Tlf\Lexer\Grammar {

    use MyBashGrammarCallbacks;

    public function getNamespace(){return 'mybash';}

    protected $directives = [];

    public function __construct(){
        $file = file_get_contents(__DIR__.'/directives.php');
        $directives = require($file);
        $this->directives = $directives;
        // @tip use `array_merge()` to load directives from multiple files
    }

    public function onLexerStart($lexer,$file,$token){
        // if you have any additional setup to do
    }
}

2. Directives

directives.php

return [
        'root'=>[
            'is'=>[
                // ':comment',
                ':docblock',
                ':function',
            ],
        ],

        'docblock'=>[
            'start'=>[
                'match'=>'##',
            ],
            'stop'=>[
                'match'=>'/(^\s*[^\#])/m',
                'rewind 2',
                'this:handleDocblockEnd',
                'buffer.clear',
                // 'forward 2'
            ]
        ],

        'function'=>[
            'start'=>[
                'match'=>'/(?:function\s+)?([a-zA-Z\_0-9]*)(?:(?:\s*\(\))|\s+)\{/',
                'this:handleFunction',
                'stop',
                'buffer.clear',
            ]
        ],
        // an additional 'comment' directive is below
    ];

3. Callbacks

We'll write a trait with function to handle directive calls. You'll notice this:handleFunction in the directives above. That will call handleFunction(...) on your trait below.

<?php
trait MyBashGrammarCallbacks {

    public function handleDocblockEnd($lexer, $ast, $token, $directive){
        $block = $token->buffer();
        $clean_input = preg_replace('/^\s*#+/m','',$block);
        $db_grammar = new \Tlf\Lexer\DocblockGrammar();
        $ast = $db_grammar->buildAstWithAttributes(explode("\n",$clean_input));
        $lexer->setPrevious('docblock', $ast);
    }

    public function handleFunction($lexer, $ast, $token, $directive){
        // $func_name = $token->match(1);
        $func = new \Tlf\Lexer\Ast('function');
        $func->name = $token->match(1);
        $func->docblock = $lexer->previous('docblock');
        $lexer->getHead()->add('function', $func);
    }

4. Directive Tests

You need to be using taeluf/tester for built-in tests to work

<?php

class MyBashGrammarTest extends extends \Tlf\Lexer\Test\Tester {

    protected $my_tests = [
        'Comments'=>[
            // the 'comment' directive is below and can be added to the `MyGrammar` that is above
            'start'=>'comment', // t
            'input'=>"var=\"abc\"\n#I am a comment\nvarb=\"def\"",
            'expect'=>[
                "comments"=>[
                    0=>[
                        'type'=>'comment',
                        'src'=>'#I am a comment',
                        'description'=> "I am a comment",
                    ]
                ],
            ],
        ],
    ];

    public function testBashDirectives(){
        $myGrammar = new \MyGrammar();
        $grammars = [
            $myGrammar
        ];
        // $docGram->buildDirectives();

        $this->runDirectiveTests($grammars, $this->my_tests);
    }

}

A more complex directive

<?php
// you would put this in your directives class
$directives = [
    'comment'=>[
        'start'=>[
            'match'=>'/#[^\#]/',
            'rewind 2',
            'buffer.clear',
            'forward 1',
            // you can create & modify ASTs all in the directive code, without php
            'ast.new'=>[
                '_addto'=>'comments',
                '_type'=>'comment',
                'src'=>'_token:buffer',
            ],
            'buffer.clear //again',
        ],

        // `match` gets called for each char after `start`
        'match'=>[
            'match'=>'/@[a-zA-Z0-9]/', // match an @attribute
            'rewind 1',
            'ast.append src',
            'rewind 1 // again',
            'ast.append description',
            'forward 2',
            'buffer.clear',
            'then :+'=>[ // the :+ means that we're defining a new directive rather than referencing an existing one
                'start'=>[
                    //just immediately start
                    'match'=>'',
                    'rewind 1',
                ],
                'stop'=>[
                    // i honestly don't know why I have this here.
                    'match'=>'/(\\r|\\n)/',
                    'rewind 1',
                    'ast.append src',
                    'buffer.clear',
                ]
            ],

        ],
        'stop'=>[
            'match'=>'/(\\r|\\n)/',
            'rewind'=>1,
            'ast.append src',
            'ast.append description',
            'forward'=>1,
            'buffer.clear',
        ],
    ]
];