provided.token.tokenizer
Class ABCTokenizer

java.lang.Object
  provided.token.tokenizer.StringTokenizer
      provided.token.tokenizer.ABCTokenizer

All Implemented Interfaces:

IStringTokenizer, ITokenizer

public class ABCTokenizer
extends StringTokenizer
implements ITokenizer

Tokenizer for the abc music language. getNextToken() returns next Token in stream. Does not recognize all possible language constructs and symbols (returned as "?" token). Tokens are the atomic psemantic pieces of a language, such as Note, repeat start, measure, chord, etc. The token ids (names) returned are:

EOF - end of file
"|:" - repeat start
":|" - repeat end
"|1" - first repeat start
"[1" - alternative first repeat start
"|2" - second repeat start
"[2" - alternative second repeat start
"[|" - thick-thin double bar line
"|]" - thin-thick double bar line
":|2" - alternative second repeat start
"||" - double measure bar
"::" - short hand for repeat end-start ":||:"
"|" - Measure bar
"[") - Chord start
"]" - Chord end
"-" - tie
">" - the previous note is dotted, the next note halved
"<" - the previous note is halved, the next dotted
"{" - Grace begin
"}" - Grace end
"%" - Comment
"[Header ID]" - where [Header ID] is a single letter from {ABCDEFGHIKLMNOPQRSTWXZ}
"Note" - a single note whose lexeme contains the name, accidentals, octave and duration
"Tuplet" - the lexeme is the number of subsequent Notes comprise this tuplet.
"Quote" - lexeme is the quoted text
"?" - unknown language construct.

References:
java.util.regex.Pattern for regex guide
ASCII table for codes
abc music standard 1.6

Field Summary

static java.lang.String	ACCIDENTALS Allowable note accidentals
private java.util.Map<java.lang.String,java.lang.String>	keywords1 maps regex to their ASCII string counterparts.
private java.util.Map<java.lang.String,java.lang.String>	keywords2 keywords2 is searched second
private java.util.Vector<java.util.Map<java.lang.String,java.lang.String>>	keywordsVector Ordered sets of keywords to check.
private java.util.Stack<Token>	stack Push-back stack for tokens.
private ITokenFactory	tokFac Factory for the tokens

Fields inherited from class provided.token.tokenizer.StringTokenizer

_fileReader, _st

Constructor Summary

ABCTokenizer(java.lang.String inputFileName)
Constructor for the class

Method Summary

Token	getNextToken() Return the next token, either from the stream or the last token if it was pushed back
private Token	makeNextToken() Parse the strings down from the getNextStringToken further, separating out the abc keywords into tokens.
private java.lang.String	parseNote(java.lang.String sTok) Returns the first note string in the given string.
private java.lang.String	parseQuote(java.lang.String acc, java.lang.String sTok) Returns the rest of a string, incl.
void	putBack(Token t) Push token back into the stream

Methods inherited from class provided.token.tokenizer.StringTokenizer

getNextStringToken, putStringBack

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

keywords1

private java.util.Map<java.lang.String,java.lang.String> keywords1

maps regex to their ASCII string counterparts. The key is the regex and the value is the keyword. Used for splitting strings up on the keywords. keywords1 is searched first

keywords2

private java.util.Map<java.lang.String,java.lang.String> keywords2

keywords2 is searched second

keywordsVector

private java.util.Vector<java.util.Map<java.lang.String,java.lang.String>> keywordsVector

Ordered sets of keywords to check.

tokFac

private ITokenFactory tokFac

Factory for the tokens

stack

private java.util.Stack<Token> stack

Push-back stack for tokens.

ACCIDENTALS

public static final java.lang.String ACCIDENTALS

Allowable note accidentals

See Also:

Constant Field Values

Constructor Detail

ABCTokenizer

public ABCTokenizer(java.lang.String inputFileName)
             throws java.io.FileNotFoundException

Constructor for the class

Parameters:

inputFileName - File to tokenize

Throws:

java.io.FileNotFoundException

Method Detail

getNextToken

public Token getNextToken()

Return the next token, either from the stream or the last token if it was pushed back

Specified by:

getNextToken in interface ITokenizer

putBack

public void putBack(Token t)

Push token back into the stream

Specified by:

putBack in interface ITokenizer

Parameters:

t - the Token you need to put back in the stream.

makeNextToken

private Token makeNextToken()

Parse the strings down from the getNextStringToken further, separating out the abc keywords into tokens.

Returns:

The next Token in the stream

parseNote

private java.lang.String parseNote(java.lang.String sTok)

Returns the first note string in the given string.

Parameters:

sTok - The string to parse

Returns:

A String that is the first note in the given string

parseQuote

private java.lang.String parseQuote(java.lang.String acc,
                                    java.lang.String sTok)

Returns the rest of a string, incl. spaces until the next double quote

Parameters:

acc - Accumulated result string

sTok - the current string token

Returns:

the full quoted string