Token types

token(values = values_token)

values_token

Arguments

values

A character string of possible values. See values_token in examples below.

Format

An object of class character of length 12.

Details

This parameter is used in textrecipes::step_tokenize().

Examples

values_token
#> [1] "words" "characters" "character_shingle" #> [4] "lines" "ngrams" "paragraphs" #> [7] "ptb" "regex" "sentences" #> [10] "skip_ngrams" "tweets" "word_stems"
token()
#> Token Unit (qualitative) #> 12 possible value include: #> 'words', 'characters', 'character_shingle', 'lines', 'ngrams', 'paragraphs', ...