Definition of "tokenize"

tokenize

/ˈtoʊ.kən.aɪz/

verb
third-person singular simple present tokenizes, present participle tokenizing, simple past and past participle tokenized
1. (transitive, computing) To reduce to a set of tokens by lexical analysis.
2. (transitive, computing) To substitute sensitive data with meaningless placeholders.
3. (transitive) To treat as a token minority.