LuaTokenizer

This project is experimental.

This project is only experimental and may change drastically between now and when it comes out of experimental status. It may even be disapproved once that happens, worst-case.

Be forewarned before downloading anything from this project.

This is a tokenizer for the Lua language. I put it up as a library so others may benefit from my weird love of creating low-level code. It is written to be run in standard (CLI) Lua 5.1 as well as WoW Lua.

The Library is not yet fully featured, but I consider the "API" stable and the tokenizer itself as well. There is a sample parser that outputs HTML in the test folder and some sample WoW Lua parser will follow or will be accessible in the Addon GreaseGoblin.

API

LuaTokenizer:Tokenize(...)

`array = LuaTokenizer:Tokenize(string[, transform_callback])`

Tokenizes a string string, calls transform_callback for each token and returns an array containing all return values of the callback. If no callback is given, a default callback is used, that returns all arguments as an array.

transform_callback(...)

mixed = transform_callback(token, value, line-start, line-end, char-start, char-end, ...)

The callback is called for every token generated by Tokenize, and is expected to return a value that represents the token for your application. This can be any value, the tokenizer does not constrain this in any way. Note that returning nil will not create an entry in the array returned by Tokenize.

Arguments:

  • value contains the source string that represents the token. If token is not a symbolic name, this value is the same as token
  • token the token name. This can be either a symbolic name for "ERROR", "NEWLINE", "MLSTRING", "COMMENT", "KEYWORD", "ID", "HEXNUM", "NUMBER" or "STRING". In this case value will contain the actual source string of that token.
  • line-start is the line where the token starts [NYI]
  • line-end is the line where the token ends
  • character-start is the character (counting from the beginning of the string) where the token starts [NYI]
  • character-end is the character (counting from the beginning of the string) where the token ends [NYT]
  • ... Additional parameters: For "STRING" this will contain the string delimiter (" or '), and for "MLSTRING" this will contain the = signs, if the opening sequence is only [[ this is an empty string (not nil!).

You must login to post a comment. Don't have an account? Register to get one!

Facts

Date created
Jun 19, 2013
Category
Last update
Jul 03, 2013
Development stage
Alpha
License
Apache License version 2.0
Downloads
10
Recent files
  • A: r29 for 5.3.0 Jul 03, 2013
  • A: r27 for 5.3.0 Jun 26, 2013
  • A: r24 for 5.3.0 Jun 25, 2013
  • A: r13 for 5.3.0 Jun 19, 2013

Authors