An R wrapper for Jagger's tokenizer

An R wrapper for Jagger's tokenizer

tokenize(input, model_path = NULL, keep = NULL, concat = TRUE)

Arguments

input

an input.

model_path

a path to the model.

keep

a vector of POS(s) to keep. Default is NULL.

concat

logical. If TRUE, the function returns a concatenated string. Default is TRUE.

Value

a vector (if concat = TRUE) or a list (if concat = FALSE).

Examples

 data(sentence_example)
 res_tokenize <- tokenize(sentence_example$text)