Package provided.token.tokenizer

Interface Summary
IStringTokenizer Tokenizer that only returns string tokens.
ITokenizer Extract and return an appropriate Token from some given source.
 

Class Summary
ABCTokenizer Tokenizer for the abc music language.
StringTokenizer Special tokenizer that converts a stream of various types of tokens from StreamTokenizer and converts them into a stream of strings.