## Package gate.creole.tokeniser

Class Summary
DefaultTokeniser: A composed tokeniser containing a `SimpleTokeniser` and a `Transducer`.
DFSMState: Implements a state of the deterministic finite state machine of the tokeniser.
FSMState: A state of the finite state machine that is the kernel tokeniser
SimpleTokeniser: Implementation of a Unicode rule based tokeniser.
TestTokeniser
UnicodeType: Used as an object wrapper that holds an Unicode type (the byte value of the static member of java.lang.Character).

Exception Summary
InvalidRuleException: Fired when an invalid tokeniser rule is found
TokeniserException: The top level exception for all the exceptions fired by the tokeniser