Combine Multiple Token Variables Into One

step_tokenmerge() creates a specification of a recipe step that will take multiple token variables and combine them into one token variable.

Usage

step_tokenmerge(
  recipe,
  ...,
  role = "predictor",
  trained = FALSE,
  columns = NULL,
  prefix = "tokenmerge",
  keep_original_cols = FALSE,
  skip = FALSE,
  id = rand_id("tokenmerge")
)

Arguments

recipe

A recipes::recipe() object. The step will be added to the sequence of operations for this recipe.

...

One or more selector functions to choose which variables are affected by the step. See recipes::selections() for more details.

role

For model terms created by this step, what analysis role should they be assigned? By default, the new columns created by this step from the original variables will be used as predictors in a model.

trained

A logical to indicate if the quantities for preprocessing have been estimated.

columns

A character vector of the selected variable names. This is NULL until the step is trained by recipes::prep.recipe(), and should not be set by hand.

prefix

A character string that will be the prefix to the resulting new variables. See notes below.

keep_original_cols

A logical to keep the original variables in the output. Defaults to FALSE.

skip

A logical. Should the step be skipped when the recipe is baked by recipes::bake()? While all operations are baked when recipes::prep() is run, some operations may not be able to be conducted on new data (e.g. processing the outcome variable(s)). Care should be taken when using skip = TRUE, as it may affect the computations for subsequent operations.

id

A character string that is unique to this step to identify it.

Value

An updated version of recipe with the new step added to the sequence of existing steps (if any).

Details

All the selected variables are merged into a single new variable named prefix, which is "tokenmerge" by default. The name of the selected variables is not part of the new name.

Tidying

When you tidy() this step, a tibble is returned with columns terms and id:

terms

character, the selectors or variables selected

id

character, id of this step

Case weights

The underlying operation does not allow for case weights.

See also

step_tokenize() to turn characters into tokens

Other Steps for Token Modification: step_lemma(), step_ngram(), step_pos_filter(), step_stem(), step_stopwords(), step_token_replace(), step_tokenfilter()

Examples

library(recipes)
library(modeldata)
data(tate_text)

tate_rec <- recipe(~., data = tate_text) |>
  step_tokenize(medium, artist) |>
  step_tokenmerge(medium, artist)

tate_obj <- tate_rec |>
  prep()

bake(tate_obj, new_data = NULL)
#> # A tibble: 4,284 × 4
#>        id title                                         year tokenmerge
#>     <dbl> <fct>                                        <dbl>  <tknlist>
#>  1  21926 Proposals for a Habitat                       1990 [9 tokens]
#>  2  20472 Michael                                       1990 [5 tokens]
#>  3  20474 Geoffrey                                      1990 [5 tokens]
#>  4  20473 Jake                                          1990 [5 tokens]
#>  5  20513 To the Studios                                1990 [6 tokens]
#>  6  21389 Phaëthon                                      1990 [7 tokens]
#>  7 121187 Untitled                                      1990 [6 tokens]
#>  8  19455 Green VIII                                    1990 [5 tokens]
#>  9  20938 Present Bound                                 1990 [8 tokens]
#> 10 105941 Joseph Beuys: A Private Collection. A11 Art…  1990 [5 tokens]
#> # ℹ 4,274 more rows

tidy(tate_rec, number = 2)
#> # A tibble: 2 × 2
#>   terms  id              
#>   <chr>  <chr>           
#> 1 medium tokenmerge_WS9uJ
#> 2 artist tokenmerge_WS9uJ
tidy(tate_obj, number = 2)
#> # A tibble: 2 × 2
#>   terms  id              
#>   <chr>  <chr>           
#> 1 medium tokenmerge_WS9uJ
#> 2 artist tokenmerge_WS9uJ