Manipulating penguin data à la dplyr with H2O

1 What this package does

h2oplyr provides familiar transformation verbs while keeping the observations inside an H2O cluster. The only external runtime dependency is h2o. The tidytable API inspires parts of the interface, but neither tidytable nor dplyr participates in a calculation. knitr and rmarkdown are optional documentation-building dependencies.

An h2o_tbl is a small R object containing a remote frame handle and grouping column names. R assembles expressions and inspects metadata; H2O processes the rows. Scalar reductions and category domains can reach R, but no transformation uses a local data frame, matrix or vector as its computational backend. For display only, the previews below request a few scalar values and categorical labels. Displaying a value necessarily brings that value to the reader; it does not change where selection, aggregation, joins and reshaping run.

This tutorial is for R users familiar with selecting columns, filtering rows and grouping data. You will learn to inspect remote results, handle missing values, check joins and verify a reshape round trip. Java and a matching H2O client/server are the only external runtime requirements.

This tutorial uses measurements of 344 penguins from three species in the Palmer Archipelago, collected during 2007–2009. The CSV is included in the package, so examples do not depend on a future download or a palmerpenguins installation.

2 Connect once, then import

Install the package archive using install.packages(path, repos = NULL, type = "source") after installing h2o. Start a matching H2O server explicitly:

# Run this yourself, once per session. Loading h2oplyr never starts a server.
h2o::h2o.init(nthreads = 2, max_mem_size = "2G")
library(h2oplyr)

# Escape labels before placing them inside HTML.
html_text <- function(x) {
  x <- gsub("&", "&amp;", x, fixed = TRUE)
  x <- gsub("<", "&lt;", x, fixed = TRUE)
  gsub(">", "&gt;", x, fixed = TRUE)
}

# Show bounded remote values without creating a local data.frame.
preview_h2o <- function(x, title, columns = names(as_h2o(x)), rows = 4L) {
  frame <- as_h2o(x)
  stopifnot(inherits(frame, "H2OFrame"), rows >= 1L, rows <= 6L,
            all(columns %in% names(frame)))
  types <- unlist(h2o::h2o.getTypes(frame[, columns]), use.names = FALSE)
  shown <- min(rows, nrow(frame))
  cat('\n<div class="result"><h3>', html_text(title), '</h3>', sep = "")
  cat('<p class="result-meta">H2OFrame · ', nrow(frame), ' rows × ',
      ncol(frame), ' columns · showing ', shown, ' rows and ',
      length(columns), ' columns</p><div class="table-scroll"><table><thead><tr>',
      sep = "")
  for (j in seq_along(columns)) {
    alignment <- if (types[j] %in% c("int", "real")) ' class="num"' else ""
    cat('<th scope="col"', alignment, '>', html_text(columns[j]), '</th>', sep = "")
  }
  cat('</tr></thead><tbody>')
  for (i in seq_len(shown)) {
    cat('<tr>')
    for (j in seq_along(columns)) {
      # Two-step indexing keeps the cell as a remote frame in H2O.
      cell <- frame[i, ][, columns[j]]
      stopifnot(inherits(cell, "H2OFrame"))
      numeric_column <- types[j] %in% c("int", "real")
      alignment <- if (numeric_column) ' class="num"' else ""
      if (h2o::h2o.sum(is.na(cell)) == 1) {
        value <- '<span class="missing">NA</span>'
      } else if (numeric_column) {
        number <- h2o::h2o.sum(cell)
        value <- if (number == floor(number)) sprintf("%.0f", number) else sprintf("%.2f", number)
      } else {
        # Re-encode in H2O so the domain contains only this cell's label.
        label <- h2o::h2o.levels(h2o::h2o.asfactor(h2o::h2o.ascharacter(cell)))
        stopifnot(length(label) == 1L)
        value <- html_text(as.character(label[[1L]]))
      }
      cat('<td', alignment, '>', value, '</td>', sep = "")
    }
    cat('</tr>')
  }
  cat('</tbody></table></div></div>\n\n')
  invisible(NULL)
}

# Only a successful, evaluated assertion receives a green check.
verify_output <- function(condition, label) {
  stopifnot(isTRUE(condition))
  cat('\n<p class="check"><strong>Verified.</strong> ', html_text(label), '</p>\n\n', sep = "")
}

# Import the bundled CSV and confirm its dimensions.
penguins <- h_penguins()
stopifnot(nrow(as_h2o(penguins)) == 344)

Verified. 344 observations and 8 variables imported into H2O.

h_penguins() uploads the local CSV through h2o.uploadFile() and declares "NA" as missing in every column. Species, island and sex are categorical; measurements are numeric. For your own files, upload directly with h2o::h2o.uploadFile() and wrap the returned frame with h2o_tbl().

The default table printer displays metadata only. This vignette adds bounded value previews so you can inspect the results directly. h_collect(penguins) and as_h2o(penguins) both return the remote H2OFrame, preserving the backend contract. This is an intentional change from the original standalone prototype.

The eight variables are species, island, bill length and depth (mm), flipper length (mm), body mass (g), sex and year. A missing measurement appears as NA, not as zero. The same penguin can have one measurement missing and others present.

3 Select only the columns needed

# Keep relevant columns, then rename and order the bill measurements.
bills <- h_select(penguins, species, starts_with("bill_"))
measurements <- h_select(penguins, where(is.numeric))
renamed <- h_rename(bills, bill_length = bill_length_mm)
relocated <- h_relocate(renamed, species, .before = bill_length)

Verified. Selection, renaming and final column order match the requested schema.

Selection operates on column names and H2O schema types. where() supports the base predicates is.numeric, is.character and is.factor; arbitrary predicates are deliberately rejected. all_of() requires every requested column, while any_of() allows absent names. Ranges such as species:island and exclusions such as -year are supported. A grouped selection retains its grouping columns.

4 Missing values and filtering

# Compare complete-case filtering with two measurement thresholds.
complete_penguins <- h_drop_na(penguins)
stopifnot(nrow(as_h2o(complete_penguins)) == 333)
cat("Complete observations:", nrow(as_h2o(complete_penguins)), "\n")
#> Complete observations: 333
large <- h_filter(penguins, body_mass_g >= 5000)
cat("Penguins weighing at least 5000 g:", nrow(as_h2o(large)), "\n")
#> Penguins weighing at least 5000 g: 67
bill_range <- h_filter(penguins, h_between(bill_length_mm, 40, 50))
cat("Penguins with bill length from 40 to 50 mm:", nrow(as_h2o(bill_range)), "\n")
#> Penguins with bill length from 40 to 50 mm: 190

Verified. Complete-case count and both filter boundaries checked over the full remote results.

A missing filter condition is dropped. The expression interpreter propagates missing comparisons explicitly, because H2O’s raw comparison behavior can differ from R’s. h_filter_out() negates the combined predicate and also drops missing conditions. Use is.na() when you intend to select missing measurements.

# Select the missing body masses explicitly.
missing_mass <- h_filter(penguins, is.na(body_mass_g))
stopifnot(nrow(as_h2o(missing_mass)) == 2)

Verified. Both retained rows have a missing body mass.

Dropping all incomplete rows yields 333 penguins, whereas a mass-only analysis can retain 342. Those are different analysis populations. Replacing missing body mass with zero would invent a biologically impossible measurement, so this tutorial removes missing mass only where it is needed.

5 Derive columns without moving observations

# Drop only missing mass, then derive units and a mass category.
prepared <- penguins |>
  h_drop_na(body_mass_g) |>
  h_mutate(
    mass_kg = body_mass_g / 1000,
    mass_band = h_case_when(
      body_mass_g >= 5000 ~ "large",
      body_mass_g >= 3500 ~ "medium",
      TRUE ~ "small"
    )
  )
stopifnot(nrow(as_h2o(prepared)) == 342)
cat("Total measured mass (kg):",
    h2o::h2o.sum(as_h2o(prepared)[, "mass_kg"]), "\n")
#> Total measured mass (kg): 1437

Verified. 342 measured masses retained; the kg-to-g conversion agrees for every row.

Named mutations run sequentially, so a later expression can use a newly created column. Arithmetic, comparisons, selected H2O mathematical and text functions, and the package’s vector helpers are supported. Calls to arbitrary R functions or other package namespaces raise an error. h_transmute() retains only the new named columns. Grouped mutation supports the five window helpers introduced below. Other grouped expressions are rejected: derive ordinary columns before grouping, or ungroup explicitly when a global calculation is intended.

The native |> pipe and the package’s %.% pipe can both chain these verbs.

6 Summarize by species

# Compute counts and means inside H2O, grouped by species.
by_species <- h_summarize(
  penguins,
  observations = n(),
  mean_mass_g = mean(body_mass_g, na.rm = TRUE),
  mean_bill_mm = mean(bill_length_mm, na.rm = TRUE),
  .by = species
)

Verified. Three groups account for all 344 observations.

The output has one row per species. n() counts all observations; each mean removes missing values independently. Gentoo penguins have the highest mean body mass in this sample. This is a descriptive comparison, not a causal effect of species or island.

.by applies grouping to this call only. The equivalent persistent grouping form is h_group_by(penguins, species) |> h_summarize(...); summaries themselves return ungrouped results. .add = TRUE adds grouping columns, and h_group_vars() reports the current grouping metadata.

Supported reductions are n(), n_distinct(column), sum, mean, min, max, sd, var and median. Each reduction takes a column, optionally with na.rm. Create transformed columns with h_mutate() before aggregating. The adapter rejects empty-input summaries explicitly.

7 Transform several columns consistently

# Apply unit conversion and grouped means to multiple columns.
scaled <- h_mutate(
  bills,
  h_across(starts_with("bill_"), list(cm = function(z) z / 10))
)
average_bills <- bills |>
  h_drop_na() |>
  h_group_by(species) |>
  h_summarize_across(starts_with("bill_"), list(average = mean))

Verified. Across conversion and the three grouped summaries checked.

h_across() builds column expressions with names such as bill_length_mm_cm. The one-argument function body is interpreted through the same remote expression rules as h_mutate(). Summary function identity is used, so an alias such as average = mean still computes the mean. Across summaries default to na.rm = FALSE; this example deliberately removes missing bill measurements first. The standalone h_mutate_across() wrapper is retained for compatibility.

8 Join summaries back to observations

# Attach a unique species summary to each observed mass.
enriched <- h_left_join(
  prepared, by_species,
  by = "species", relationship = "many-to-one"
)
stopifnot(nrow(as_h2o(enriched)) == nrow(as_h2o(prepared)))
enriched <- h_mutate(enriched, mass_minus_species_mean = body_mass_g - mean_mass_g)

Verified. The join preserves all 342 measured penguins and matches a species mean to each.

The relationship check ensures the right table has at most one record per species, preventing accidental multiplication of penguin rows. Inner, left, right and full equijoins are available. h_semi_join() selects left rows with a match; h_anti_join() selects those without one. Named keys map different column names, for example by = c(species = "species_code").

A full join is implemented using a left join plus unmatched right rows, all inside H2O. Missing-key matching follows H2O, and join row order is unspecified. Always use h_arrange() when subsequent calculations depend on ordering.

9 Reshape measurements from wide to long and back

# Use species-level summaries so the ID is naturally unique.
wide <- h_summarize(
  penguins,
  bill_length_mm = mean(bill_length_mm, na.rm = TRUE),
  bill_depth_mm = mean(bill_depth_mm, na.rm = TRUE),
  .by = species
)
long <- h_pivot_longer(
  wide, starts_with("bill_"),
  names_to = "measurement", values_to = "mean_mm"
)
stopifnot(nrow(as_h2o(long)) == 6)
restored <- h_pivot_wider(long, measurement, mean_mm)
stopifnot(nrow(as_h2o(restored)) == 3)

Verified. Wide → long → wide preserves both mean measurements for every species.

The long result contains six species/measurement combinations. Widening gets the output names from the remote categorical domain and combines values with remote joins. Repeated ID/name combinations are rejected: decide on an aggregation before reshaping. This avoids silently choosing one measurement or creating a local list-column. values_fill replaces missing result cells, including explicitly missing input values.

For text transformations, h_separate() uses H2O regular-expression splitting. h_unite() uses H2O categorical interaction with encoded original underscores; its current separator cannot contain dollar signs or backslashes. High-cardinality interaction domains can consume substantial cluster memory.

10 Sampling, ties and order-dependent calculations

# Sample reproducibly, retain tied maxima and compute a sorted lag.
sample_penguins <- h_slice_sample(prepared, n = 12, seed = 2026)
stopifnot(nrow(as_h2o(sample_penguins)) == 12)
heaviest <- h_slice_max(prepared, body_mass_g, n = 3, with_ties = TRUE)
cat("Rows among the three heaviest positions, including ties:",
    nrow(as_h2o(heaviest)), "\n")
#> Rows among the three heaviest positions, including ties: 4
ordered <- h_arrange(prepared, body_mass_g)
ordered <- h_mutate(ordered, previous_mass = h_lag(body_mass_g))

Verified. Sample size is 12; the first lag is missing and every later lag matches the preceding mass.

Sampling uses H2O random columns. Sampling with replacement creates draw indices in H2O and joins them to rows. With seed = NULL (the default), no seed is passed to H2O; supply a seed explicitly when reproducibility is needed. Identical seeds are reproducible on the same frame and cluster layout; they are not a promise of identical samples across versions or distributed layouts. with_ties = TRUE can return more than n rows.

Lag, lead, cumulative operations and h_fill() use current row order. Penguins are not a longitudinal time series: the lag example illustrates an ordered comparison, not a temporal interpretation. Grouped windows are covered next. Grouped slicing, filling and cumulative helpers remain unsupported.

11 Windows within each group

Version 0.3.0 adds grouped h_mutate() for five direct window calls: h_lag(), h_lead(), h_roll_mean(), h_min_rank() and h_row_number(). Use .by for one call, or h_group_by() for persistent grouping. Each window follows the existing row order within its group, even when groups are interleaved. The output preserves the input row order. Missing keys are valid grouping values; multiple grouping columns are supported.

# Number the input rows before sorting, then use that position to break ties.
species_order <- penguins |>
  h_filter(!is.na(body_mass_g)) |>
  h_mutate(input_row = h_row_number()) |>
  h_arrange(species, body_mass_g, input_row)
species_windows <- species_order |>
  h_group_by(species) |>
  h_mutate(previous_mass = h_lag(body_mass_g),
           mean_last_three = h_roll_mean(body_mass_g, k = 3),
           mass_rank = h_min_rank(desc(body_mass_g)),
           position = h_row_number())
# Grouping persists until removed explicitly.
stopifnot(identical(h_group_vars(species_windows), "species"))
first_in_species <- species_windows |> h_ungroup() |> h_filter(position == 1)

reference <- h2o::h2o.uploadFile(
  system.file("extdata", "windows.csv", package = "h2oplyr"),
  na.strings = "NA",
  col.types = c("numeric", "enum", "enum", rep("numeric", 15)))
observations <- h2o_tbl(reference[, c("id", "group", "subgroup", "amount")])
windows <- h_mutate(observations,
  prev = h_lag(amount), next_amount = h_lead(amount),
  roll = h_roll_mean(amount, k = 3),
  roll_rm = h_roll_mean(amount, k = 3, na.rm = TRUE),
  full = h_roll_mean(amount, k = 3, partial = FALSE),
  rank_desc = h_min_rank(desc(amount)), row = h_row_number(),
  .by = c(group, subgroup))
# .by does not attach grouping to the result.
stopifnot(length(h_group_vars(windows)) == 0)
example_group <- h_filter(windows, group == "A", subgroup == "x")

transactions <- observations |>
  h_select(id, group, amount) |>
  h_rename(txn_id = id, customer_id = group) |>
  h_mutate(txn_time = txn_id) |>
  h_filter(!is.na(amount)) |>
  h_arrange(customer_id, txn_time, txn_id)
transaction_windows <- h_mutate(transactions,
  prev_amount = h_lag(amount),
  roll_mean = h_roll_mean(amount, k = 3, partial = TRUE),
  rank_amount = h_min_rank(desc(amount)),
  .by = customer_id)
customer_summary <- h_summarize(transaction_windows,
  observations = n(), average_window = mean(roll_mean), .by = customer_id)

12 Validation and current boundaries

Every exported function has an executable example with an assertion. The package also includes regression tests for missing values, aggregation, ties, empty slices, join multiplicity, invalid expressions and input preservation. Integration tests compare H2O function bindings before and after execution without modifying the H2O namespace. A static audit rejects local-table calls in package functions. Additional maintainer-only download instrumentation is kept outside the package.

Run all examples and regressions against a cluster you started explicitly:

h2o::h2o.init(nthreads = 2, max_mem_size = "2G")
library(h2oplyr)
source(system.file("examples", "examples.R", package = "h2oplyr"))
for (nm in names(examples)) eval(examples[[nm]])
source(system.file("examples", "regressions.R", package = "h2oplyr"))

This package is an explicit subset of tidy-style manipulation, not a complete drop-in replacement for dplyr or tidytable. It has no local rowwise/map backend, non-equality joins, automatic list-column nesting, arbitrary R callbacks, general grouped mutation beyond the five supported window helpers, or implicit duplicate-cell pivot aggregation. h_bind_rows() requires matching column-name sets. h_nest_by() returns a list of remote group handles and is intended for a modest number of groups.

When no cluster is connected, rebuilding this vignette retains the code but skips H2O execution. The distributed HTML was built against an explicitly connected server. The package never starts or shuts down a user’s cluster while loading, printing, or running its ordinary examples.

13 Reproduce and interpret the checks

The green checks accompany the tutorial’s calculations. The installed package also contains 76 asserted API examples and semantic regression tests; those are a separate suite, not 76 examples all reproduced on this page. The commands in “Connect once” and “Validation” are instructions to run yourself, rather than claims that this HTML launched a cluster or reran the complete package test suite.

Executed with H2O. R 4.6.1; h2oplyr 0.3.0; h2o 3.44.0.3. All tutorial assertions completed during rendering.

Try changing the mass threshold or selecting another species, then rerun the section and inspect both the values and the assertion. Do not infer a successful transformation merely from seeing an object name or from a silent stopifnot().

14 Short glossary

• H2OFrame: data stored in an H2O cluster, accessed through an R handle.
• h2o_tbl: the package wrapper around that handle and grouping metadata.
• Preview: a small display of remote values; its caption reports the full dimensions.
• NA: a missing value, distinct from a zero measurement.
• Assertion: a condition that must hold; a failure stops the document build.
• Many-to-one join: each left key can match at most one right-hand record.

15 Data provenance

Data: Horst AM, Hill AP, Gorman KB (2020), palmerpenguins: Palmer Archipelago (Antarctica) penguin data, doi:10.5281/zenodo.3960218. The data are available under CC0. Measurements were collected by Kristen Gorman and Palmer Station LTER. The research description is Gorman KB, Williams TD, Fraser WR (2014), doi:10.1371/journal.pone.0090081. The bundled CSV’s source, retrieval date and SHA-256 are recorded in system.file("extdata", "PROVENANCE.md", package = "h2oplyr").

The synthetic windows.csv fixture was authored for h2oplyr and is covered by the package MIT license. It contains no personal or transactional records.