Contributing to PRIOGRID

This tutorial walks through the process of adding a new data source and variable to PRIOGRID. Contributions are welcome — report issues or suggest new sources via the Issue Tracker.

Package Layout for Contributors

priogrid/
├── R/
│   ├── data_{source}.R        # read_*() and gen_*() functions for each source
│   ├── config.R               # pg_config(), pg_set_config(), pg_set_rawfolder(), .pg_release_specs
│   ├── build_priogrid.R       # calc_pg(), load_pgvariable(), read_pg_static(), etc.
│   ├── utility.R              # robust_transformation(), rast_to_df(), prio_blank_grid()
│   ├── references.R           # pgcitations(), get_bibliography()
│   ├── source_class.R         # Source R6 class (dev-only: used in devtools::load_all())
│   └── download_data.R        # pgsearch(), pg_rawfiles(), check_pgsourcefiles()
├── data_raw/
│   ├── sources.csv            # Add new source here (tab-separated)
│   ├── variables.csv          # Add new variable here (tab-separated)
│   ├── pgsources.R            # Regenerates data/pgsources.rda from sources.csv
│   └── pgvariables.R          # Regenerates data/pgvariables.rda from variables.csv
├── data/
│   ├── pgsources.rda          # Compiled metadata (auto-generated, do not edit)
│   ├── pgvariables.rda        # Compiled metadata (auto-generated, do not edit)
│   └── pgchecksum.rda         # MD5 reference checksums (auto-generated)
├── inst/
│   ├── REFERENCES.bib         # Full bibliography — add new BibTeX entries here
│   └── extdata/urls/          # Multi-file URL lists (one .txt per source)
└── tests/testthat/
    └── test-build_priogrid.R  # Integration tests for gen_*() functions

Step 1: Register the Data Source

Add to `data_raw/sources.csv`

sources.csv is tab-separated with 18 columns. The intended path is not hand-editing but the Source class + add_source() dev workflow (see below), which fills id, created_at, and the three *_exists booleans automatically.

Required fields:

Column	Description	Example
`source_name`	Full name of the dataset	`"My New Dataset"`
`source_version`	Version string	`"1.0"`
`license`	SPDX or common license name	`"CC BY 4.0"`
`website_url`	Landing page	`"https://example.com/data"`
`spatial_extent`	One of: `"World"`, `"Multiple continents"`, `"Single continent"`, `"Several countries"`	`"World"`
`temporal_resolution`	One of: `"Static"`, `"Higher than monthly"`, `"Monthly"`, `"Quarterly"`, `"Yearly"`, `"Less than yearly"`	`"Yearly"`
`citation_keys`	Semicolon-separated BibTeX keys	`"doeNewDataset2025"`
`download_url`	Direct download URL, or `"urls/{uuid}.txt"` for multi-file sources	`"https://example.com/data.zip"`

Optional fields: aws_bucket, aws_region, prio_mirror, tags, reference_keys.

The following columns are auto-generated by Source$to_tibble() and should not be hand-written: id, download_url_exists, website_url_exists, prio_mirror_exists, created_at. If you are writing a row by hand (not via add_source()), generate a UUID first:

uuid::UUIDgenerate()
# "a1b2c3d4-e5f6-7890-abcd-ef1234567890"

Use the `Source` Class (dev mode)

In development mode (devtools::load_all()), use the Source R6 class to validate a new source before adding it to the CSV:

new_source <- Source$new(
  source_name          = "My New Dataset",
  source_version       = "1.0",
  license              = "CC BY 4.0",
  website_url          = "https://example.com/data",
  spatial_extent       = "World",
  temporal_resolution  = "Yearly",
  citation_keys        = "doeNewDataset2025",
  download_url         = "https://example.com/data/dataset_v1.zip",
  tags                 = "climate, land cover"
)
new_source  # prints validation report; warns if citation key not in REFERENCES.bib

The Source class validates required fields, checks that spatial_extent and temporal_resolution use the allowed vocabulary, and verifies that citation keys exist in inst/REFERENCES.bib. Use new_source$get_existing_tags() and new_source$get_existing_licenses() to inspect vocabulary already in use and keep tags and licenses consistent across sources.

Once validated, write the row to sources.csv with add_source():

add_source(new_source)
# Appends a tab-separated row to data_raw/sources.csv and writes
# inst/extdata/urls/{id}.txt if download_url or prio_mirror points at a URL list.

add_source(source, csv_file = "data_raw/sources.csv") is dev-only (requires devtools::load_all()). It appends the row and calls new_source$save_url_files() automatically. After running it, regenerate pgsources.rda as below.

Add the Citation to `inst/REFERENCES.bib`

Add a BibTeX entry to inst/REFERENCES.bib for each citation_keys value:

@article{doeNewDataset2025,
  author  = {Doe, Jane},
  title   = {My New Dataset: A Global Resource},
  journal = {Scientific Data},
  year    = {2025},
  volume  = {12},
  pages   = {1--10},
  doi     = {10.1234/sdata.2025.001}
}

Regenerate `pgsources.rda`

After adding a source, regenerate the .rda file:

source("data_raw/pgsources.R")

Verify the result:

pgsources[pgsources$source_name == "My New Dataset", ]

Download the Raw Files

Before writing read_*()/gen_*(), fetch the source’s files. Set the raw-data folder first if you haven’t already:

pg_set_rawfolder("/path/to/rawdata")

Then download and verify:

files <- pg_rawfiles() |> dplyr::filter(id == "a1b2c3d4-...")
download_pg_rawdata(file_info = files)
pg_data_availability()   # confirm n_present == n_files

download_pg_rawdata() is resumable (.part files), batched (batch_size = 20, max_concurrent = 4), retries with backoff (max_retry = 10), and verifies MD5 against pgchecksum advisorily. See the metadata vignette for the discovery helpers (pgsearch, pg_rawfiles, pg_data_availability).

Step 2: Write `read_()` and `gen_()` Functions

Create a new file R/data_mynewsource.R.

`read_*()` — Load Raw Data

The read function downloads (via get_pgfile()) and returns the raw data as an sf or SpatRaster object:

#' Read My New Dataset
#'
#' Downloads and imports My New Dataset.
#'
#' @return An \code{sf} object
#' @export
#' @references
#' \insertRef{doeNewDataset2025}{priogrid}
read_mynewsource <- function() {
  f <- get_pgfile(
    source_name    = "My New Dataset",
    source_version = "1.0",
    id             = "a1b2c3d4-e5f6-7890-abcd-ef1234567890"
  )
  sf::st_read(f)
}

get_pgfile() automatically downloads the file if it is not present locally (when automatic_download = TRUE in the config).

`gen_*()` — Generate the PRIOGRID Variable

The generate function transforms raw data to the PRIOGRID grid and returns a SpatRaster. The signature must accept config:

#' My New Variable
#'
#' Computes my_new_variable for each PRIOGRID cell.
#'
#' @param config A \code{pg_config} object. Defaults to \code{\link{pg_current_config}()}.
#' @return A \code{SpatRaster} object
#' @export
#' @references
#' \insertRef{doeNewDataset2025}{priogrid}
gen_my_new_variable <- function(config = pg_current_config()) {
  raw <- read_mynewsource()

  # Reproject if needed
  pg_crs <- sf::st_crs(config$crs)
  if (sf::st_crs(raw) != pg_crs) {
    raw <- sf::st_transform(raw, pg_crs)
  }

  # Convert to raster and re-grid to PRIOGRID resolution
  raw_rast <- terra::rasterize(terra::vect(raw), prio_blank_grid(config),
                               field = "my_value_column", fun = "mean")

  r <- robust_transformation(raw_rast, agg_fun = "mean", config = config)

  names(r) <- "my_new_variable"
  return(r)
}

Using `robust_transformation()`

robust_transformation() handles reprojection, cropping, aggregation, disaggregation, and final resampling in a single call. It is the standard way to re-grid any raster:

robust_transformation(
  r             = raw_rast,      # Any SpatRaster (any resolution / CRS / extent)
  agg_fun       = "mean",        # Aggregation function for higher-res inputs
  disagg_method = "near",        # Disaggregation for lower-res inputs
  config        = config         # Target config
)

Common agg_fun values: "mean", "sum", "max", "min", "modal" (for categorical).

Temporal Variables

For time-varying variables, iterate over pg_dates(config) and combine layers:

gen_my_yearly_variable <- function(config = pg_current_config()) {
  dates <- pg_dates(config)
  layers <- lapply(dates, function(d) {
    raw <- read_mynewsource_for_year(lubridate::year(d))  # your reader
    r   <- robust_transformation(raw, agg_fun = "mean", config = config)
    names(r) <- as.character(d)
    r
  })
  do.call(c, layers)  # stack into a multi-layer SpatRaster
}

Step 3: Register the Variable

Add to `data_raw/variables.csv`

variables.csv is tab-separated with 7 columns. Add one row per variable:

Column	Description / Allowed values
`name`	Must match the string set by `names(r)` in `gen_*()` and equal `gen_{name}` without the prefix
`static`	`TRUE` for variables without a time dimension, `FALSE` for time-varying
`source_ids`	Comma-separated source UUIDs from `sources.csv`
`label`	Human-readable display title for plots/legends
`unit`	Unit string (e.g. `°C`, `mm`); leave empty if dimensionless
`transform`	One of: `identity`, `log1p`, `log10`, `sqrt`
`plot_type`	One of: `continuous`, `positive_real`, `count`, `share`, `discrete`

A hand-written row must include all 7 tab-separated fields; empty fields still need their tab delimiter. Example (tabs shown as →):

name→static→source_ids→label→unit→transform→plot_type
my_new_variable→FALSE→a1b2c3d4-e5f6-7890-abcd-ef1234567890→My new variable→→identity→continuous

Regenerate pgvariables.rda:

source("data_raw/pgvariables.R")

Verify:

pgvariables[pgvariables$name == "my_new_variable", ]

Display Metadata and COG Metatags

The four authored display columns (label, unit, transform, plot_type) are stamped into each built COG as pg_* GDAL metatags by save_pgvariable() via .pg_build_metatags(), making the .tif self-describing. Built COGs additionally carry the derived pg_colormap and computed pg_value_min/pg_value_max/pg_value_mean/pg_value_std/pg_nunique/pg_class_values metatags — these are not authored columns.

After editing display columns for an already-built variable, re-stamp without recomputing:

priogrid:::.pg_restamp_metatags("my_new_variable")

Step 4: Document with roxygen2

PRIOGRID uses roxygen2 with Markdown support and Rdpack for citation references.

Key tags: - @param config — always document the config parameter for gen_*() functions - @return — describe what the function returns - @export — all user-facing functions must be exported - @references \insertRef{key}{priogrid} — links to inst/REFERENCES.bib

Regenerate documentation:

devtools::document()

Step 5: Add Tests

Add tests to tests/testthat/. For a gen_* function, use the shared test helpers from tests/testthat/helper-priogrid.R: test_config() (nrow=5, ncol=10, 2010–2012 yearly) and skip_if_no_rawdata(). A gen_* test needs downloaded raw data, so it must call skip_if_no_rawdata():

# tests/testthat/test-data_mynewsource.R
test_that("gen_my_new_variable returns a SpatRaster", {
  skip_if_not_installed("terra")
  skip_if_no_rawdata()
  r <- gen_my_new_variable(config = test_config())
  expect_s4_class(r, "SpatRaster")
  expect_equal(names(r), "my_new_variable")
})

Internal accessors (hashes, resolve_pg_mode) are tested via priogrid::: — see tests/testthat/test-build_priogrid.R for examples.

Run tests:

devtools::test()

Step 6: Verify End-to-End

Test the full pipeline:

cfg <- pg_config()

# 1. Calculate
calc_pg("my_new_variable", config = cfg)

# 2. Load as raster
r <- load_pgvariable("my_new_variable", config = cfg)
terra::plot(r)

# 3. Load as table
pg_tv <- read_pg_timevarying(config = cfg)
"my_new_variable" %in% names(pg_tv)

# 4. Check citations resolve
pgcitations("my_new_variable")

Multi-file Sources and URL Lists

When a source serves multiple files — or when URLs do not carry a usable filename (e.g. FigShare download endpoints) — place a plain-text URL list at inst/extdata/urls/{id}.txt. Each line is one URL; an optional tab-separated second column names the local file:

https://example.com/files/data_part1.nc
https://ndownloader.figshare.com/files/17626052 my_local_name.nc

The second column is only needed when the server does not expose the filename via Content-Disposition or the post-redirect URL. Helpers: pg_read_url_list(), pg_format_url_list(), pg_default_filename(). add_source() calls new_source$save_url_files() automatically and writes the file when download_url or prio_mirror is a local list path.

Resolving opaque filenames. priogrid:::pg_resolve_filenames(id, write = FALSE) queries each server (Content-Disposition header, then post-redirect URL) and prints a proposed filename column, flagging suspicious names (no extension, login endpoints). Re-run with write = TRUE to commit the names, then rebuild and run pg_update_checksums().

priogrid:::pg_resolve_filenames("a1b2c3d4-e5f6-7890-abcd-ef1234567890", write = FALSE)
# Review proposed names, then commit:
priogrid:::pg_resolve_filenames("a1b2c3d4-e5f6-7890-abcd-ef1234567890", write = TRUE)

Renaming already-downloaded files. When a source gains an explicit filename column, use priogrid:::pg_migrate_rawfiles() to rename already-downloaded files in place so they are not re-downloaded:

priogrid:::pg_migrate_rawfiles(dry_run = TRUE)   # preview renames
priogrid:::pg_migrate_rawfiles(dry_run = FALSE)  # apply

The Build Pipeline: From gen_*() to a Dataset

`calc_pg()` and output layout

calc_pg(varnames = NULL, overwrite = FALSE, config) calls each gen_<name>(config=) and passes the result to save_pgvariable(). Outputs land under:

<rawfolder>/priogrid/custom/<pkg_version>/<spatial_hash>/<temporal_hash>/

The folder is created on first build and includes a _config.R script (a reproducible pg_config() call). Skips already-built variables unless overwrite = TRUE; per-variable failures are collected and reported, never fatal.

After calc_pg(), the folder contains:

cog/<varname>.tif — Cloud Optimised GeoTIFF (COG, DEFLATE) with time()/units() fields and pg_* GDAL metatags
_checksums.csv — per-file MD5s
_config.R — reproducible config script

Spatial/temporal hashing. pgout_path() keys output folders on 6-character MD5 hashes derived from grid dimensions/CRS/extent (get_spatial_hash()) and from the date range (get_temporal_hash()). The temporal hash floors end_date so small date changes do not fork folders into new locations.

`load_pgvariable()`

r <- load_pgvariable(
  varname = "my_new_variable",
  config  = cfg,                       # NULL => release mode (auto-download_priogrid())
  extent  = c(-20, 50, -10, 40),       # optional: windowed spatial read
  layers  = 1:5                        # optional: subset time layers
)

Returns a lazy, file-backed SpatRaster; extent/layers enable windowed larger-than-memory reads.

Table readers

# Static variables → pg_static.parquet + pg_static.csv.gz
static <- read_pg_static(config = cfg)

# Time-varying → hive-partitioned parquet + pg_timevarying.csv.gz + pg_config.json manifest
# Supports lazy subsetting:
tv <- read_pg_timevarying(
  config    = cfg,
  years     = 2000:2010,
  variables = c("my_new_variable", "another_var"),
  extent    = c(-20, 50, -10, 40)
)

build_pg_dataset() writes the hive dataset memory-safely without collecting the full table into RAM.

Listing custom builds

customs <- pg_list_custom()
# Prints a summary of each custom folder; returns an indexed list of pg_config objects.
read_pg_static(config = customs[[1]])
calc_pg("new_var", config = customs[[2]])

Memory management for large rasters

pg_configure_terra_memory() (called inside robust_transformation()) auto-enables terra on-disk processing when an estimated raster exceeds ~4 GB. This is relevant when developing high-resolution gen_*() functions.

File Integrity and Checksums

pgchecksum is a bundled data object (data/pgchecksum.rda) with reference MD5s for the exact raw files used to build the official release. Columns: source_name, source_version, id, filename, md5.

Runtime verification. check_pgsourcefiles() compares local raw files to pgchecksum and warns for files with no reference entry (sources added since the last tested build). Enable per-read verification with:

cfg <- pg_config(verify_checksums = TRUE)
# or per file:
get_pgfile(source_name = "My New Dataset", source_version = "1.0",
           id = "a1b2c3d4-...", verify_checksums = TRUE)

Regenerating checksums (maintainer). After a fully verified, clean download set, recompute and rewrite pgchecksum.rda:

priogrid:::pg_update_checksums(only_present = TRUE)

New sources have no checksum entry until PRIO-GRID has been built from them once and this is run.

Cutting an Official Release

A release is registered in two places, both keyed "<version>_<type>":

1. Grid spec — add an entry to .pg_release_specs in R/config.R:

# Inside .pg_release_specs in R/config.R:
"3.0.2_05deg_yearly" = list(
  nrow               = 360L,
  ncol               = 720L,
  crs                = "epsg:4326",
  extent             = c(xmin = -180, xmax = 180, ymin = -90, ymax = 90),
  temporal_resolution = "1 year",
  start_date         = as.Date("1850-12-31"),
  end_date           = as.Date("2026-08-26")
)

2. CDN URL — add the corresponding entry to the releases list inside download_priogrid() in R/build_priogrid.R:

# Inside the releases list in download_priogrid() in R/build_priogrid.R:
"3.0.2_05deg_yearly" = "https://cdn.cloud.prio.org/files/<uuid>"

Then build the release:

build_release(
  version             = "3.0.2",
  type                = "05deg_yearly",
  nrow                = 360,
  ncol                = 720,
  crs                 = "epsg:4326",
  extent              = c(xmin = -180, xmax = 180, ymin = -90, ymax = 90),
  temporal_resolution = "1 year",
  start_date          = as.Date("1850-12-31"),
  end_date            = as.Date("2026-08-26")
)

build_release() calculates all variables to the custom location, builds static and hive time-varying tables, copies the output to releases/<version>/<type>/, and writes two zip archives: priogrid_<ver>_<type>.zip and priogrid_<ver>_<type>_csv.zip.

Retrieve a release config programmatically, or list all published releases:

cfg <- pg_release_config("3.0.2")
download_priogrid(list_releases = TRUE)

Developer Utilities Reference

Function	Purpose	Location
`Source`	R6 class: validate and serialise a new source	`R/source_class.R`
`add_source()`	Append validated source to `sources.csv`; write URL list file	`R/utility.R`
`priogrid:::pg_resolve_filenames()`	Query servers for filenames of opaque URLs	`R/download_data.R`
`priogrid:::pg_migrate_rawfiles()`	Rename downloaded files after explicit filename column is added	`R/download_data.R`
`priogrid:::pg_update_checksums()`	Recompute MD5s and rewrite `pgchecksum.rda`	`R/download_data.R`
`priogrid:::.pg_restamp_metatags()`	Re-stamp GDAL `pg_*` metatags without recomputing a variable	`R/build_priogrid.R`
`priogrid:::.pg_build_metatags()`	Build the metatag list from `pgvariables` for `save_pgvariable()`	`R/build_priogrid.R`
`get_spatial_hash()` / `get_temporal_hash()`	6-char MD5 hashes that key output folders	`R/build_priogrid.R`
`resolve_pg_mode()`	Determine release vs. custom output path	`R/build_priogrid.R`
`pg_configure_terra_memory()`	Auto-enable on-disk processing for large rasters	`R/config.R`
`create_pg_indices()`	Create PRIOGRID cell indices	`R/utility.R`
`rast_to_df()`	Convert a SpatRaster to a data frame	`R/utility.R`
`pg_dates()`	Sequence of dates for a config	`R/utility.R`
`pg_date_intervals()`	Date intervals for a config	`R/utility.R`

Functions marked priogrid::: lack @export; call them with the triple-colon operator in dev mode (devtools::load_all()).

Package Layout for Contributors

Step 1: Register the Data Source

Add to `data_raw/sources.csv`

Use the `Source` Class (dev mode)

Add the Citation to `inst/REFERENCES.bib`

Regenerate `pgsources.rda`

Download the Raw Files

Step 2: Write `read_()` and `gen_()` Functions

`read_*()` — Load Raw Data

`gen_*()` — Generate the PRIOGRID Variable

Using `robust_transformation()`

Temporal Variables

Step 3: Register the Variable

Add to `data_raw/variables.csv`

Display Metadata and COG Metatags

Step 4: Document with roxygen2

Step 5: Add Tests

Step 6: Verify End-to-End

Multi-file Sources and URL Lists

The Build Pipeline: From gen_*() to a Dataset

`calc_pg()` and output layout

`load_pgvariable()`

Table readers

Listing custom builds

Memory management for large rasters

File Integrity and Checksums

Cutting an Official Release

Developer Utilities Reference

Summary Checklist

Contributor Checklist

Maintainer Checklist

Contributing to PRIOGRID

Package Layout for Contributors

Step 1: Register the Data Source

Add to data_raw/sources.csv

Use the Source Class (dev mode)

Add the Citation to inst/REFERENCES.bib

Regenerate pgsources.rda

Download the Raw Files

Step 2: Write read_*() and gen_*() Functions

read_*() — Load Raw Data

gen_*() — Generate the PRIOGRID Variable

Using robust_transformation()

Temporal Variables

Step 3: Register the Variable

Add to data_raw/variables.csv

Display Metadata and COG Metatags

Step 4: Document with roxygen2

Step 5: Add Tests

Step 6: Verify End-to-End

Multi-file Sources and URL Lists

The Build Pipeline: From gen_*() to a Dataset

calc_pg() and output layout

load_pgvariable()

Table readers

Listing custom builds

Memory management for large rasters

File Integrity and Checksums

Cutting an Official Release

Developer Utilities Reference

Summary Checklist

Contributor Checklist

Maintainer Checklist

Add to `data_raw/sources.csv`

Use the `Source` Class (dev mode)

Add the Citation to `inst/REFERENCES.bib`

Regenerate `pgsources.rda`

Step 2: Write `read_()` and `gen_()` Functions

`read_*()` — Load Raw Data

`gen_*()` — Generate the PRIOGRID Variable

Using `robust_transformation()`

Add to `data_raw/variables.csv`

`calc_pg()` and output layout

`load_pgvariable()`