This is a tokenizer for the Lua language. I put it up as a library so others may benefit from my weird love of creating low-level code. It is written to be run in standard (CLI) Lua 5.1 as well as WoW Lua.
The Library is not yet fully featured, but I consider the "API" stable and the tokenizer itself as well. There is a sample parser that outputs HTML in the
test folder and some sample WoW Lua parser will follow or will be accessible in the Addon GreaseGoblin.
`array = LuaTokenizer:Tokenize(string[, transform_callback])`
Tokenizes a string
transform_callback for each token and returns an array containing all return values of the callback. If no callback is given, a default callback is used, that returns all arguments as an array.
mixed = transform_callback(token, value, line-start, line-end, char-start, char-end, ...)
The callback is called for every token generated by
Tokenize, and is expected to return a value that represents the token for your application. This can be any value, the tokenizer does not constrain this in any way. Note that returning
nil will not create an entry in the array returned by
valuecontains the source string that represents the token. If
tokenis not a symbolic name, this value is the same as
tokenthe token name. This can be either a symbolic name for
"STRING". In this case
valuewill contain the actual source string of that token.
line-startis the line where the token starts [NYI]
line-endis the line where the token ends
character-startis the character (counting from the beginning of the string) where the token starts [NYI]
character-endis the character (counting from the beginning of the string) where the token ends [NYT]
...Additional parameters: For "STRING" this will contain the string delimiter (
'), and for "MLSTRING" this will contain the
=signs, if the opening sequence is only
[[this is an empty string (not nil!).