Token types
token(values = values_token) values_token
values | A character string of possible values. See |
---|
An object of class character
of length 12.
This parameter is used in textrecipes::step_tokenize()
.
values_token#> [1] "words" "characters" "character_shingle" #> [4] "lines" "ngrams" "paragraphs" #> [7] "ptb" "regex" "sentences" #> [10] "skip_ngrams" "tweets" "word_stems"token()#> Token Unit (qualitative) #> 12 possible value include: #> 'words', 'characters', 'character_shingle', 'lines', 'ngrams', 'paragraphs', ...