Package gate.creole.tokeniser

Class Summary
DefaultTokeniser Implementation of a Unicode rule based tokeniser.
DFSMState Implements a state of the deterministic finite state machine of the tokeniser.
FSMState A state of the finite state machine that is the kernel tokeniser
TestTokeniser  
UnicodeType Used as an object wrapper that holds an Unicode type (the byte value of the static member of java.lang.Character).
 

Exception Summary
InvalidRuleException Fired when an invalid tokeniser rule is found
TokeniserException The top level exception for all the exceptions fired by the tokeniser