
Sentencepiece Tokenization of Character Variables
Source:R/tokenize_sentencepiece.R
step_tokenize_sentencepiece.Rdstep_tokenize_sentencepiece() creates a specification of a recipe step
that will convert a character predictor into a token
variable using SentencePiece tokenization.
Arguments
- recipe
A
recipes::recipe()object. The step will be added to the sequence of operations for this recipe.- ...
One or more selector functions to choose which variables are affected by the step. See
recipes::selections()for more details.- role
Not used by this step since no new variables are created.
- trained
A logical to indicate if the quantities for preprocessing have been estimated.
- columns
A character vector of the selected variable names. This is
NULLuntil the step is trained byrecipes::prep.recipe(), and should not be set by hand.- vocabulary_size
An integer, indicating the number of tokens in the final vocabulary. Defaults to
1000. Highly encouraged to be tuned. Ignored whenmodelis supplied.- model
A string giving the path to a pretrained sentencepiece model file, such as one downloaded with
sentencepiece::sentencepiece_download_model()or trained withsentencepiece::sentencepiece(). When supplied, no model is trained atprep()time and the same model is used for all selected columns. The model is read when the step is prepped, so the file is not needed afterwards. Defaults toNULL, which trains a new model.- options
A list of options passed to
sentencepiece::sentencepiece()when the tokenizer is trained atprep()time. These options have no effect on already-trained models atbake()time. Note that the argumentsx,model_dir, andvocab_sizeshould not be passed here, usevocabulary_sizeinstead. Ignored whenmodelis supplied. Defaults tolist().- res
The fitted
sentencepiece::sentencepiece()model tokenizer. This isNULLuntil the step is trained byrecipes::prep.recipe()and should not be set by hand.- skip
A logical. Should the step be skipped when the recipe is baked by
recipes::bake()? While all operations are baked whenrecipes::prep()is run, some operations may not be able to be conducted on new data (e.g. processing the outcome variable(s)). Care should be taken when usingskip = TRUE, as it may affect the computations for subsequent operations.- id
A character string that is unique to this step to identify it.
Value
An updated version of recipe with the new step added
to the sequence of existing steps (if any).
Details
If you are running into errors, you can investigate the progress of the
compiled code by setting options = list(verbose = TRUE). This can reveal if
sentencepiece ran correctly or not.
Tidying
When you tidy() this step, a tibble is returned with
columns terms and id:
- terms
character, the selectors or variables selected
- id
character, id of this step
See also
step_untokenize() to untokenize.
Other Steps for Tokenization:
step_tokenize(),
step_tokenize_bpe(),
step_tokenize_wordpiece()
Examples
library(recipes)
library(modeldata)
data(tate_text)
tate_rec <- recipe(~., data = tate_text) |>
step_tokenize_sentencepiece(medium)
tate_obj <- tate_rec |>
prep()
bake(tate_obj, new_data = NULL, medium) |>
slice(1:2)
#> # A tibble: 2 × 1
#> medium
#> <tknlist>
#> 1 [12 tokens]
#> 2 [3 tokens]
bake(tate_obj, new_data = NULL) |>
slice(2) |>
pull(medium)
#> <textrecipes_tokenlist[1]>
#> [1] [3 tokens]
#> # Unique Tokens: 3
tidy(tate_rec, number = 1)
#> # A tibble: 1 × 2
#> terms id
#> <chr> <chr>
#> 1 medium tokenize_sentencepiece_nVEI6
tidy(tate_obj, number = 1)
#> # A tibble: 1 × 2
#> terms id
#> <chr> <chr>
#> 1 medium tokenize_sentencepiece_nVEI6
# Using a pretrained model
model_dir <- tempfile()
dir.create(model_dir)
file <- tempfile(tmpdir = model_dir)
writeLines(as.character(tate_text$medium), file)
trained <- sentencepiece::sentencepiece(
file,
vocab_size = 100,
model_dir = model_dir
)
recipe(~., data = tate_text) |>
step_tokenize_sentencepiece(medium, model = trained$model_path) |>
prep() |>
bake(new_data = NULL, medium) |>
slice(1:2)
#> # A tibble: 2 × 1
#> medium
#> <tknlist>
#> 1 [43 tokens]
#> 2 [3 tokens]