Lexer Examples

Lex a file

$lexer = new \Tlf\Lexer();  
$lexer->useCache = false; // cache is disabled only for testing  
$lexer->addGrammar($phpGrammar = new \Tlf\Lexer\PhpGrammar());  
  
$ast = $lexer->lexFile(dirname(__DIR__).'/php/SampleClass.php');  
  
// An array detailing the file   
$tree = $ast->getTree();   

See test/php/SampleClass.php for the input file and test/php/SampleClass.tree.php for the output $tree.

Lex a string

This example is a bit more involved. Docblock is generally added by a programming language's grammar, so Docblock does not automatically start being listened for, the way <?php would be with the PhpGrammar.

$lexer = new \Tlf\Lexer();  
$docGrammar = new \Tlf\Lexer\DocblockGrammar();  
$lexer->addGrammar($docGrammar);  
$lexer->addDirective($docGrammar->getDirectives(':/*')['/*']);  
$str = "/** I am docblock */";  
$ast = $lexer->lex($str);  
  
$tree = $ast->getTree();  
$actual = $tree['docblock'][0];  
$expect = [  
    'type'=>'docblock',  
    'description'=>'I am docblock',  
];  

The root ast contains the string, which we're not really interested in.

Lex with your own root ast

This is basically what happens when you lex a file, EXCEPT lexFile() automatically handles caching, so subsequent runs on the same unchanged file will be loaded from cache. This approach ignores the cache completely.

$lexer = new \Tlf\Lexer();  
$phpGrammar = new \Tlf\Lexer\PhpGrammar();  
$lexer->addGrammar($phpGrammar);  
  
// set up the ast  
$ast = new \Tlf\Lexer\Ast('code');  
$ast->set ('language', 'php');  
$code = '<?php class Abc extends Alphabet {}';  
$ast->set('src', $code);  
  
  
$ast = $lexer->lex($code, $ast);  
$actual = $ast->getTree();  
$expect = [  
    'type'=>'code',  
    'language'=>'php',  
    'src'=>$code,  
    'class'=>[  
        0=>[  
            'type'=>'class',  
            'docblock'=>'',  
            'namespace'=>'',  
            'name'=>'Abc',  
            'declaration'=>'class Abc extends Alphabet ',  
        ],  
    ],  
];