Taeluf.com

Devlog

This contains old notes i no longer need

March 10, 2022

I didn't work on re-adding empty bodies to method asts, so a lot of the directive tests are still failing.

I want to capture the entire method body as a string. I also want to capture every expression in a list of expressions (whether in a method or just in a script). I'm concerned that manually catching each expression will be overly complex & patchwork, having a handler on semicolons, another one on parenthesis close (for if statements, foreach, etc), another on block open & block close & probably others i'm not thinking of (comments, docblocks) ...

I'm concerned that adding the body by hooking on each of the individual things will add unnecessary complication. I tried adding a hook for everytime $token->next() is called, then ended up with a LOT of repeated characters, bc many of the directives rewind.

I might be able to find a generic way to address this. By somehow manually keeping track of the position in the string & probably some custom code in the rewind section. I'm not exactly sure. I might be able to dump all that code into token, so the lexer doesn't have to "think" about it. Then have a hook for "new_char_added" or something that ONLY gets called the first time a character is added to the token's buffer.

Something like this would allow me to just capture every character while the method_body ast is the head & then have a string body. I do like this approach, to some extent.

However, I also want to (eventually) have a list (array) of expressions. So if I had something like:

function abc(){
    $var = true;
    $abc = new \Whatever();
    if ($var){
        echo "cats";
    }
    return false;
}

Then I would have 4 expressions as direct children of the method body. The if ($var){echo "cats";} block would be one expression that also contains expressions. (well it contains one expression, 'echo cats', but still)

I'm not sure how to manage all that. And then I think a comment would also be in this expressions list. Doing this would require me to hook on each individual thing ... but in these cases, the specific functionality is actually needed. There will need to be custom logic to handle blocks & whatnot, i guess ...

And if I'm going to implement that, why not use it for the method body string as well? Maybe because they have different purposes. I have a tendency to ignore whitespace. I don't think I have any special handling for whitespace. I suppose I could add it. But I definitely don't want whitespace in the expressions. And docblocks aren't expressions. the docblocks would be attached to the expression, not BE an expression in the array ...

So there is a distinct need for both approaches, i guess? I could merge them, but the two approaches solve two different (though related) feature requests.

Okay. So, this is going to require some though & some experimentation. And definitely a lot of patience. It'll be hard not to break things in this process.

Okay, thanks Reed! Hope you come back to this feeling good :)

Updating Directive tests to be passing

Method.ReturnReference Method.WithNestedBlock.2 Method.WithNestedBlock Method.WithReturnType Method.TwoArgs Method.OneArg Method.Simple.OneArg Method.Static Method.Simple //Class.Implements //Class.Extends //Class.Abstract //Class.OpenInFile //Class.OpenInNamespace Integration.Class.Methods.WithNestedBlock Integration.Class.Final.Method Integration.Class.Bug.ModifiersMethods Integration.Class.Method.2 Integration.Trait.Method Integration.Class.Method

March 5, 2022

Majorly cleaned up Test\PhpGrammar. Big issue: Comment counts are all wrong because they're not being extracted from the body. docblock has the same issue. These are also awful unit tests. These are fine for debugging & then making sure i don't later screw things up, but they're awful unit tests & directive tests are GOOD unit tests ... i need to get the directive tests passing. I think they're all failing bc of the body issue, but it might be other stuff too. idk

March 3, 2022

I haven't resolved the feb 11 issue I'm working on cleaning up tests

TODO

  • fix the method body issue (see Grammars/PhpGrammar test, even though that test should be in debug)
    • maybe add unit tests for the ast classes??
  • separate the lex dir into ones just for counts, then another for debugging, then probably another for validating full trees
  • move input/php/tree/ to the output dir (in the code)
  • clean up the php grammar parse_file function
  • work through the failing php directive tests & get them passing ()
  • create a passing bash grammar test
  • create a wrapper class or convenience functions that make it easier to just run the lexer on a file or string for a given language
  • delete the old BashGrammar, JsonGrammar, defunct Docblock grammar,
  • document directives
  • idk what else! :)

Feb 11, 2022 ISSUE

  • phptest -test MethodBodyIsString ... the method body is showin gas an array when it should be a string ... i messed around with it a bit & at times it was returning an ast object instead of an array ... idk what the problem is ... ALSO, the third method has a body nested in the body ... which makes no sense ... so this is a whole heck of a thing, it seems.
    • i was accidentally running the test with cache enabled so ... whoops
    • i decided to just remove body because it wasn't working anyway ...
    • so i commented out a line in Operations that set 'body' on the ast
    • this needs to be investigated

Jan 26, 2022

  • the PhpGrammar tests have a bunch of commented out lines ... this needs refactored so I can properly run these tests with confidence rather than require manual review
  • I need to write unit tests for the string_backslash issue (and maybe others that I've solved, with operations & such)

Jan 25, 2022

  • work on the php grammar ... handles functions (not just methods), and fixes bugs with catching 'class' and 'function' keywords in the wrong context

Next

  • the string_backslash directive is supposed to handle backslash escaped things inside a string ... well it catches the first one, so a string like "iam \" a string \" with two quote marks" will fail on the second \". I tried to do a work-around in php, but that was a mess ... I don't think it's worth refactoring the internals ... I need to make a unit test just to test strings that contain backslash-escaped quotation marks. i THINK it's because of the order ... after the first time string_backslash is hit, the list of unstarted directives is changed (it getting stopped appends it, rather than putting it back in its original spot) ... that's 100% it

... i found a workaround for the string backslash ... added its own handler ...

... phtml classes are all passing the methdos count ...

I need to improve the generic test & its output ... its all just takes up so much space for no good reason

Known Issues (PhpGrammar)

  • when there IS a namespace in a file, class is nested under neath it. When there is NO namespace in the file, namespace='' and class=[] is adjacent, ... for now, i just have a workaround in the test code
  • the string backslash issue ... i'd rather not have it be a workaround in php ... idk how to fix it and its probably not a priority

Welcome back (Jan 24, 2022)

  • I can't parse a method with a body
  • See run/Documentation.php ... There is a test for PhpGrammarNew that shows very nicely how to run the lexer.
  • test/output/PhpFeatures.md contains a list of directive unit tests (their description, input code, and whether they passed their tests). Looking at this is a good way to see what language features have been implemented
  • PhpGrammarNew's test PhpGrammarNew_SampleClass is failing and i don't know why (it was failing before i moved files around, but i need to fix that too)

Things I want to do:

  • Restructure the test directory
  • delete old code that I don't need
  • disable tests that I don't care about (old grammars and such)
  • turn PhpGrammarNew into PhpGrammar & either delete the old one or rename the old one to PhpGrammarOld
  • structure the code/PhpNew directory ... the code/Lexer dir ... maybe just review all the files & try to structure things better ... code/ should have code/Grammars/...
  • genericize the running & testing of lexing full php files ... the idea is ... i WILL run into bugs while I'm working on other software. WHEN i run into a bug, i could just copy+paste the entire source code of the file into this genericized test suite & make it easy to view the ast output & make it easy to change what's expected
  • clean up PhpGrammarNew ?? I think delete the old code I don't need ... maybe write other tests? idk

PhpGrammarNew status

Current

  • Working on phptest -test PhpGrammarNew_SampleClass. Manually comparing test/php-new/SampleClass.tree2.php to test/php-new/SampleClass.php.
    • The class is not getting a docblock. Maybe I need to write a little test for this.

Next Major steps

  • Test that docblocks work on: (I just kind of assume they do...)
    • consts
    • properties
    • methods
    • classes
    • traits
    • DONE use TraitName;
    • namespace NS;

TODO (low priority)

  • Consider always setting namespace & (or at least) fqn on class...

Latest

  • Handling nowdoc & heredoc <<<STR && <<<'STR' (somewhat lazily)
  • Rewrote strings directives (phpgrammar)
  • Fix bug (docblockgrammar): Empty docblocks led to infinite loop
  • Fixed Bug (phpgrammar): Nested blocks inside methods were not being caught, so methods were ending when nested blocks closed
  • Fixed Bug (phpgrammar): <?php class Abc extends Alphabet{} Fails to get Alphabet. Write a test for & fix it.
  • Move cmdMethods and the method map into their own trait
  • add :+directive_name as alternate to :_blank-directive_name
  • fix stop instruction, so it now ONLY stops directives if they're started in the top stack
  • add inherit/directive.inherit instruction
  • add then.pop instruction
  • Write Documentation tests & an Examples doc file & cleanup README
  • add namespace.docblock to file ast & update tests
  • Added modifier to const ast
  • Docblock grammar integrated into phpgrammar
  • Docblock Grammar passing
  • Fixed: Some lines would show * after docblock grammar was done parsing.
  • then instruction now allows you to specify grammar like then docblock:/*
  • Clean up DocBlockGrammar stuff
  • Wrote a docblock grammar in almost pure php
  • Add Tester class to simplify testing of many directives
  • Minor improvements to intuitiveness of lex api
  • remove inspect-loop param from lexer
  • Refactor Command Processing
  • Refactor instruction execution
    • Move the switch into its entire own function & provide more clarity about passed in args
    • Improve naming scheme by namespacing all commands. There are also namespace free defaults
    • Create separate methods to replace complex cases (for readability & maintainability)
  • Create command parsing function (executeMethodString())that handles things like _token:buffer or _lexer:unsetPrevious docblock
  • Disable bash grammar tests
  • Reorganize lexer's internals into traits & remove old functions
  • cleaned up PhpGrammar test.
  • Refactored names & namespaces of php grammar & directives
  • Deleted old, useless php grammars
  • Sorted PhpGrammar directives into traits
  • removed unneeded functions for phpgrammarnew & separated directives into a trait
  • Got PhpGrammarNew working for SampleClass.php
  • Got v0.6 working