flambe.tokenizer.tokenizer
¶
Module Contents¶
-
class
flambe.tokenizer.tokenizer.
Tokenizer
[source]¶ Bases:
flambe.Component
Base interface to a Tokenizer object.
Tokenizers implement the tokenize method, which takes a string as input and produces a list of strings as output.