diff --git a/.Rbuildignore b/.Rbuildignore index bd6b4b0..71b58ad 100644 --- a/.Rbuildignore +++ b/.Rbuildignore @@ -1,11 +1,17 @@ ^scythe\.Rproj$ ^\.Rproj\.user$ ^LICENSE\.md$ -^\.travis\.yml$ ^codecov\.yml$ ^CONTRIBUTING\.md$ ^api-scopus-search.sh$ ^\.github$ ^logo$ ^data$ -^results$ \ No newline at end of file +^results$ +^tools$ +^helm$ +^TODO\.md$ +^openalex-duckdb$ +^dev$ +^doc$ +^Meta$ \ No newline at end of file diff --git a/.github/workflows/R-CMD-check.yaml b/.github/workflows/R-CMD-check.yaml index 733853a..74a829b 100644 --- a/.github/workflows/R-CMD-check.yaml +++ b/.github/workflows/R-CMD-check.yaml @@ -20,6 +20,8 @@ jobs: steps: - uses: actions/checkout@v2 - uses: r-lib/actions/setup-r@v2 + # Pandoc is needed to build the R Markdown vignettes. + - uses: r-lib/actions/setup-pandoc@v2 - name: Install dependencies run: | install.packages(c("remotes", "rcmdcheck")) diff --git a/.gitignore b/.gitignore index 6551700..e24fd84 100644 --- a/.gitignore +++ b/.gitignore @@ -13,3 +13,8 @@ key.txt data-raw # Ignore Vim's swap files .*.swp + +# Built vignettes +inst/doc +/doc/ +/Meta/ diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 398f795..8de0875 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -71,7 +71,8 @@ particular release. The release process starts with integration testing in the `develop` branch. Once all changes that are desired in a release are merged into the `develop` branch, we run -the full set of tests on a clean checkout, and conduct any code reviews needed. A pull request +the full set of tests on a clean checkout, including the live network tests that CI +skips (see [Testing](#testing)), and conduct any code reviews needed. A pull request can be used for that review, or it can be done directly for simpler sets of changes. Once review is completed, make sure all version numbers are set correctly and documentation is complete on `develop`, and then merge to `main`, and tag that merge @@ -90,7 +91,54 @@ as well as various classes of corrupt or bad data. All tests should pass before the `develop` branch is merged to main, and all tests should pass before the `main` branch is tagged as a release. -**Continuous integration**. We can use Travis for some of our repositories. +**Continuous integration**. GitHub Actions runs `R CMD check`, including the +tests, on every push and pull request to `main` and `develop`. CI deliberately +skips the live tests of the sources that are metered or rate limited (Springer, +Scopus, OpenAlex and DataCite). Those limits apply to a shared key or IP +address, so these tests would fail for reasons that have nothing to do with the +code, such as a Springer request rejected for quota, or an exhausted OpenAlex +daily budget. They are also skipped on CRAN. So **a passing CI run does not +mean the live searches work**; run the live tests below before each release. + +**Live network tests**. These run against the real services. They are +skipped on CRAN and on CI (any system that sets `CI=true`), and otherwise run +only when their keys or switches are set: + +| Tests | Runs locally when | +|---|---| +| PLOS, xDD | always (they need no key; they also run on CI) | +| Springer, Scopus | the `springer` or `scopus` key is set | +| OpenAlex, and the searches in the OpenAlex vignette | `SCYTHE_TEST_OPENALEX=true` is set; uses up to 100 OpenAlex credits per test, plus more when building the vignette | +| DataCite relation classification | always (a few free requests) | + +To run the full live suite before a release, from a clean checkout of +`develop`, with a good network connection and no `CI` variable set: + +1. Set the API keys. `scythe_set_key()` stores them in the `scythe` keyring; + environment variables with the same names (`springer`, `scopus`, + `openalex`, `openalex_mailto`) also work and take precedence: + + ```r + scythe_set_key(source = "springer", secret = "...") + scythe_set_key(source = "scopus", secret = "...") + scythe_set_key(source = "openalex", secret = "...") + scythe_set_key(source = "openalex_mailto", secret = "you@example.org") + ``` + +2. Opt in to the tests that spend OpenAlex credits, and run every test: + + ```r + Sys.setenv(SCYTHE_TEST_OPENALEX = "true") + devtools::test() + ``` + +3. Check the test summary for skips. Any live test reported as skipped did not + run, usually because a key is missing or OpenAlex is rate limited; fix + that and rerun it before releasing. (An OpenAlex test that skips because + of a rate limit can be rerun after the time it reports.) + +4. Optionally, build the OpenAlex vignette with its searches, as described in + its "Running this vignette" section; this spends more OpenAlex credits. ## Code style diff --git a/DESCRIPTION b/DESCRIPTION index 61e8fc7..118f87a 100644 --- a/DESCRIPTION +++ b/DESCRIPTION @@ -1,5 +1,5 @@ Package: scythe -Title: Harvest and register data package citations +Title: Harvest and Register Data Package Citations Version: 1.1.0 Authors@R: c( person("Jeanette", "Clark", role = c("aut", "cre"), email = "jclark@nceas.ucsb.edu", comment=c(ORCID = "0000-0003-4703-1974")), @@ -7,7 +7,7 @@ Authors@R: c( person("Maya", "Samet", role = "aut", email = "samet@nceas.ucsb.edu", comment=c(ORCID = "0000-0002-5248-9712")), person("Althea", "Marks", role = "aut", email = "marks@nceas.ucsb.edu", comment=c(ORCID = "0000-0002-9370-9128")) ) -Description: Harvests data package citations from several API sources, including PLOS, Scopus, and Springer. This package uses modified functions from `rplos`, which is no longer maintained. +Description: Harvests data package citations from several API sources, including 'PLOS', 'Scopus', 'Springer', 'xDD', and 'OpenAlex'. This package uses modified functions from 'rplos', which is no longer maintained. License: Apache License (>= 2.0) Encoding: UTF-8 LazyData: true @@ -23,7 +23,11 @@ Remotes: ropensci/bib2df@a8e96e13f5 Suggests: bib2df, covr, + knitr, purrr, - testthat (>= 3.0.0) -RoxygenNote: 7.3.1 + rmarkdown, + testthat (>= 3.0.0), + withr +VignetteBuilder: knitr Config/testthat/edition: 3 +Config/roxygen2/version: 8.1.0 diff --git a/NAMESPACE b/NAMESPACE index 535786a..e557ae4 100644 --- a/NAMESPACE +++ b/NAMESPACE @@ -1,10 +1,12 @@ # Generated by roxygen2: do not edit by hand export(citation_search) +export(citation_search_openalex) export(citation_search_plos) export(citation_search_scopus) export(citation_search_springer) export(citation_search_xdd) +export(classify_data_relations) export(scythe_get_key) export(scythe_set_key) export(write_citation_pairs) diff --git a/R/citation_search.R b/R/citation_search.R index 707af5e..b6cf755 100644 --- a/R/citation_search.R +++ b/R/citation_search.R @@ -2,7 +2,10 @@ #' #' @param identifiers a vector of identifiers to be searched for #' @param sources a vector indicating which sources to query (one or -#' more of plos, scopus, springer) +#' more of plos, scopus, springer, xdd, openalex). OpenAlex spends credits +#' from a daily budget, so it is only searched when named here; call +#' [citation_search_openalex()] directly to control its cost and to resume +#' an interrupted search. #' @return tibble of matching dataset and publication identifiers #' #' @export @@ -15,7 +18,7 @@ citation_search <- function(identifiers, sources = c("plos", "scopus", "springer", "xdd")) { stopifnot(is.character(identifiers)) - unid <- sources[!(sources %in% c("plos", "scopus", "springer", "xdd"))] + unid <- sources[!(sources %in% scythe_sources)] if (length(unid) > 0) { stop(paste("Source", unid, "is not recognized.", collapse = ". ")) } @@ -43,10 +46,15 @@ citation_search <- function(identifiers, return(result) } +# The sources that citation_search() can query; each one has a matching +# citation_search_() adapter. +scythe_sources <- c("plos", "scopus", "springer", "xdd", "openalex") + # Check identifiers to remove characters that interfere with query strings check_identifiers <- function(identifiers) { - if (any(!grepl("10\\.|urn:uuid", identifiers))) { + is_uuid <- grepl("^urn:uuid:", identifiers, ignore.case = TRUE) + if (any(is.na(normalize_doi(identifiers)) & !is_uuid)) { warning( call. = FALSE, "One or more identifiers does not appear to be a DOI or uuid", @@ -54,9 +62,7 @@ check_identifiers <- function(identifiers) { ) } - if (any(grepl("doi:|urn:uuid", identifiers))) { - identifiers <- gsub("(doi:)|(urn:uuid:)", "", identifiers) - } + identifiers <- gsub("(doi:\\s*)|(urn:uuid:)", "", identifiers, ignore.case = TRUE) return(identifiers) diff --git a/R/citation_search_openalex.R b/R/citation_search_openalex.R new file mode 100644 index 0000000..6017ac7 --- /dev/null +++ b/R/citation_search_openalex.R @@ -0,0 +1,219 @@ +#' Search for citations in OpenAlex +#' +#' This function searches [OpenAlex](https://openalex.org) for works that cite +#' or mention each dataset DOI, in two ways: +#' +#' - `"text"`: an exact full-text search for the DOI, which finds DOIs +#' mentioned anywhere in a work's text (methods, data availability statements, +#' acknowledgements, reference lists), whether or not OpenAlex indexes the +#' dataset. DOIs are searched in batches, and batches with matches are split +#' until each match is attributed to a single DOI. +#' - `"graph"`: OpenAlex's citation graph, which finds works whose reference +#' lists resolve to the dataset, for datasets that OpenAlex indexes. +#' +#' The result only includes citing works that have a DOI and are not datasets +#' themselves. Everything else is written as JSON Lines files to `log_dir` for +#' review: `openalex-matches.jsonl` (every article match and which method found +#' it), `openalex-data-relations.jsonl` (datasets citing or mentioning other +#' datasets, which [classify_data_relations()] can label as version, +#' derivation, composition or citation links afterwards; that isn't needed for +#' the results), `openalex-no-doi-citers.jsonl` (citing works +#' without a DOI), `openalex-unresolved.jsonl` (identifiers that are not DOIs, +#' or that OpenAlex does not index) and `openalex-failed-batches.jsonl`. +#' The logs are appended to, and a run that resumes from `checkpoint_dir` only +#' logs what it found itself, so that a search run in installments logs each +#' match once. +#' +#' OpenAlex charges credits for each request against a daily budget: 1 for a +#' list query and 10 for a full-text search. Set an API key with +#' `scythe_set_key("openalex", ...)` for a larger budget than the 1,000 daily +#' credits available without one, and optionally a contact email as +#' `openalex_mailto`. Before sending any request, the function estimates the +#' credits the search will use and stops if that exceeds `max_credits`; set +#' `max_credits = Inf` to approve a large search. While running, it never +#' spends more than `max_credits`, and stops sending requests as soon as +#' OpenAlex reports a rate limit, returning the results found so far (see the +#' `openalex_status` attribute of the result). +#' +#' @param identifiers a vector of identifiers to be searched for +#' @param methods one or both of "text" and "graph" +#' @param since only find works published on or after this date +#' (a Date or "YYYY-MM-DD"), or NULL to search all works +#' @param log_dir directory for the review logs; defaults to the +#' `scythe.log_dir` option, or a temporary directory +#' @param checkpoint_dir directory where completed batches are recorded, so +#' that a rerun with the same directory continues where an interrupted run +#' stopped; NULL to disable checkpointing +#' @param state_file file recording when OpenAlex rate limits end, shared by +#' every process using the same API key so none of them send requests before +#' then; defaults to `openalex-state.json` in `checkpoint_dir` +#' @param max_credits the most OpenAlex credits to spend; defaults to the +#' `scythe.openalex.max_credits` option, or 500 +#' @param preflight if TRUE, stop before sending any request when the +#' estimated cost exceeds `max_credits`; if FALSE, run until `max_credits` +#' have been spent +#' @param batch_size the number of DOIs in each OpenAlex query (at most 100) +#' +#' @return data.frame of matching dataset and publication identifiers, with an +#' `openalex_status` attribute recording whether the search completed, why +#' it stopped if not, and the credits used +#' @export +#' @examples +#' \dontrun{ +#' identifiers <- c("10.18739/A22274", "10.5063/F1T151VR") +#' result <- citation_search_openalex(identifiers) +#' attr(result, "openalex_status") +#' } +citation_search_openalex <- function(identifiers, + methods = c("text", "graph"), + since = NULL, + log_dir = NULL, + checkpoint_dir = NULL, + state_file = NULL, + max_credits = getOption("scythe.openalex.max_credits", 500), + preflight = TRUE, + batch_size = 100) { + methods <- match.arg(methods, several.ok = TRUE) + batch_size <- min(batch_size, 100) + if (!is.null(since)) since <- format(as.Date(since)) + if (is.null(state_file) && !is.null(checkpoint_dir)) { + state_file <- file.path(checkpoint_dir, "openalex-state.json") + } + log_dir <- openalex_log_dir(log_dir) + retrieved_at <- format_utc(openalex_now()) + + openalex_results <- data.frame( + article_id = character(), + article_title = character(), + dataset_id = character(), + source = character() + ) + status <- list(complete = FALSE, stopped = NULL, paused_until = NULL, + credits_used = 0, requests = 0, estimate = 0, log_dir = log_dir) + + identifiers <- check_identifiers(identifiers) + dois_in <- normalize_doi(identifiers) + write_jsonl(log_rows(identifier = identifiers[is.na(dois_in)], reason = "not a DOI", + retrieved_at = retrieved_at), file.path(log_dir, "openalex-unresolved.jsonl")) + dois <- unique(dois_in[!is.na(dois_in)]) + if (length(dois) == 0) { + status$complete <- TRUE + attr(openalex_results, "openalex_status") <- status + return(openalex_results) + } + + status$estimate <- estimate_openalex_cost(length(dois), methods, batch_size) + message("OpenAlex: searching ", length(dois), " DOIs (", paste(methods, collapse = " and "), + "); estimated cost about ", status$estimate, " credits") + if (preflight && status$estimate > max_credits) { + stop(call. = FALSE, + "The estimated cost of ", status$estimate, " OpenAlex credits exceeds max_credits = ", + max_credits, ". To approve this search, rerun with a larger max_credits ", + "(or max_credits = Inf).") + } + + client <- openalex_client(max_credits = max_credits, state_file = state_file) + if (is.na(client$key)) { + message( + "No OpenAlex API key is set, so the daily budget is 1,000 credits. ", + "Set one with scythe_set_key(\"openalex\", ...).") + } + + checkpoint <- read_checkpoint(checkpoint_dir) + # Pairs from batches completed by an earlier run are returned again, but + # were already logged and classified by that run. + found <- list() + found_now <- list() + incomplete <- FALSE + for (method in methods) { + if (method == "text") { + unsearchable <- openalex_unsearchable(dois) + write_jsonl(log_rows(identifier = dois[unsearchable], + reason = "characters not supported by OpenAlex search", + retrieved_at = retrieved_at), + file.path(log_dir, "openalex-unresolved.jsonl")) + batches <- chunk_for_search(dois[!unsearchable], batch_size) + } else { + batches <- chunk(dois, batch_size) + } + keys <- vapply(batches, function(b) { + paste(method, since %||% "all", paste(b, collapse = "|")) + }, "") + cached <- keys %in% names(checkpoint) + found <- c(found, unname(checkpoint[keys[cached]])) + if (all(cached)) next + + run <- if (method == "text") { + openalex_text_search(client, batches[!cached], since) + } else { + openalex_graph_search(client, batches[!cached], since) + } + found <- c(found, list(run$pairs)) + found_now <- c(found_now, list(run$pairs)) + + for (b in which(run$completed)) { + batch_pairs <- run$pairs[run$pairs$dataset_doi %in% batches[!cached][[b]], , drop = FALSE] + write_checkpoint(checkpoint_dir, keys[!cached][b], batch_pairs) + } + if (any(run$failed)) { + write_jsonl(log_rows(dois = vapply(batches[!cached][run$failed], paste, "", collapse = "|"), + method = method, since = since %||% NA_character_, + retrieved_at = retrieved_at), + file.path(log_dir, "openalex-failed-batches.jsonl")) + } + if (length(run$unresolved)) { + write_jsonl(log_rows(identifier = run$unresolved, reason = "not indexed in OpenAlex", + retrieved_at = retrieved_at), + file.path(log_dir, "openalex-unresolved.jsonl")) + } + if (!all(run$completed)) incomplete <- TRUE + } + + pairs <- do.call(rbind, c(list(empty_pairs()), found)) + pairs <- pairs[is.na(pairs$work_doi) | pairs$work_doi != pairs$dataset_doi, , drop = FALSE] + + # One row per dataset and citing work, recording which methods found it. + key <- paste(pairs$dataset_doi, pairs$work_id) + methods_found <- tapply(pairs$method, key, function(m) { + if (length(unique(m)) > 1) "both" else m[1] + }) + pairs <- pairs[!duplicated(key), , drop = FALSE] + pairs$match_method <- as.character(methods_found[paste(pairs$dataset_doi, pairs$work_id)]) + pairs$method <- NULL + pairs$retrieved_at <- rep(retrieved_at, nrow(pairs)) + bin <- citer_bin(pairs) + + # Log only the pairs this run found. A pair found by one method in an + # earlier run and by the other now is logged again, as "both". + pairs_now <- do.call(rbind, c(list(empty_pairs()), found_now)) + new <- paste(pairs$dataset_doi, pairs$work_id) %in% + paste(pairs_now$dataset_doi, pairs_now$work_id) + articles <- pairs[bin == "article", , drop = FALSE] + write_jsonl(pairs[new & bin == "article", , drop = FALSE], + file.path(log_dir, "openalex-matches.jsonl")) + write_jsonl(pairs[new & bin == "data_relation", , drop = FALSE], + file.path(log_dir, "openalex-data-relations.jsonl")) + write_jsonl(pairs[new & bin == "no_doi", , drop = FALSE], + file.path(log_dir, "openalex-no-doi-citers.jsonl")) + + # Report each match under every form of the dataset identifier the caller + # passed in that normalizes to the same DOI. + rows <- lapply(seq_len(nrow(articles)), function(i) { + ids <- unique(identifiers[!is.na(dois_in) & dois_in == articles$dataset_doi[i]]) + data.frame(article_id = articles$work_doi[i], article_title = articles$title[i], + dataset_id = ids, source = "openalex", stringsAsFactors = FALSE) + }) + openalex_results <- do.call(rbind, c(list(openalex_results), rows)) + + status$complete <- !incomplete && is.null(client$stopped) + status$stopped <- client$stopped + status$paused_until <- if (!is.null(client$paused_until)) format_utc(client$paused_until) + status$credits_used <- client$credits_used + status$requests <- client$requests + message("OpenAlex: found ", nrow(articles), " article citations, ", + sum(bin == "data_relation"), " data relations and ", sum(bin == "no_doi"), + " citing works without a DOI, using ", client$credits_used, " credits in ", + client$requests, " requests. Review logs are in ", log_dir) + attr(openalex_results, "openalex_status") <- status + openalex_results +} diff --git a/R/helpers.R b/R/helpers.R index df37e54..5d75d6b 100644 --- a/R/helpers.R +++ b/R/helpers.R @@ -3,11 +3,12 @@ #' @param n_queries number of queries #' @param wait_seconds wait time in seconds between queries #' -#' @return -#' +#' @return `NULL`, invisibly. Called for its side effect of printing the +#' rate limit and estimated run time with [message()]. +#' @noRd report_est_wait <- function(n_queries, wait_seconds) { if (n_queries <= 1) { - return() + return(invisible()) } rate_info <- if (wait_seconds == 0) { "None" @@ -27,3 +28,54 @@ report_est_wait <- function(n_queries, wait_seconds) { message("Rate limit: ", rate_info) message("Estimated time to run ", n_queries, " searches: ", wait_info) } + +# Prefixes that can precede a DOI name: resolver URLs (http or https, with or +# without dx./www.), handle URLs, and the doi:, info:doi/ and urn:doi: schemes. +doi_prefix_pattern <- paste0( + "^(?:(?:https?://)?(?:dx\\.|www\\.)?doi\\.org/", + "|(?:https?://)?hdl\\.handle\\.net/", + "|info:doi/|urn:doi:|doi:\\s*|doi\\s+)" +) + +#' Normalize DOIs to bare, lowercased DOI names +#' +#' Strips any mix of resolver URLs and DOI schemes (for example +#' `https://dx.doi.org/`, `doi:`, `DOI: `, `info:doi/`, or stacked forms such +#' as `doi:https://doi.org/`), URL-decodes the identifier, and lowercases it. +#' DOI names are case-insensitive, so the lowercased form can be compared +#' directly with the DOIs that OpenAlex returns. +#' +#' @param x a character vector of identifiers +#' @return a character vector the same length as `x`, with `NA` for any +#' identifier that is not a DOI after normalization +#' @noRd +normalize_doi <- function(x) { + x <- as.character(x) + x[is.na(x)] <- "" + x <- vapply(x, function(s) { + tryCatch(utils::URLdecode(s), error = function(e) s) + }, "", USE.NAMES = FALSE) + x <- trimws(x) + x <- gsub("^[<\"'(\\[]+|[>\"'.,;\\]]+$", "", x, perl = TRUE) + repeat { + y <- trimws(sub(doi_prefix_pattern, "", x, perl = TRUE, ignore.case = TRUE)) + if (identical(y, x)) break + x <- y + } + # Drop a trailing ")" only when it is unbalanced, since DOIs such as + # 10.1002/(SICI)1097-... legitimately contain parentheses. + unbalanced <- grepl("\\)$", x) & + lengths(regmatches(x, gregexpr("\\)", x))) > lengths(regmatches(x, gregexpr("\\(", x))) + x[unbalanced] <- sub("\\)$", "", x[unbalanced]) + x <- tolower(x) + x[!grepl("^10\\.\\d{4,9}/\\S+$", x, perl = TRUE)] <- NA_character_ + x +} + +# Split a vector into consecutive chunks of at most `size` elements. +chunk <- function(x, size) { + if (length(x) == 0) return(list()) + unname(split(x, ceiling(seq_along(x) / size))) +} + +`%||%` <- function(x, y) if (is.null(x) || length(x) == 0) y else x diff --git a/R/openalex_client.R b/R/openalex_client.R new file mode 100644 index 0000000..e4fcc6d --- /dev/null +++ b/R/openalex_client.R @@ -0,0 +1,337 @@ +# HTTP client for the OpenAlex API (https://openalex.org). +# +# OpenAlex charges credits per request (1 for a list or filter query, 10 for a +# search query) against a daily budget, and returns the remaining budget in +# X-RateLimit-* headers. The client sends requests concurrently, stops before +# it would spend more than `max_credits`, and keeps a circuit breaker so that +# once OpenAlex rate-limits it, no further requests are sent until the limit +# resets. Pauses longer than `max_wait` are written to a shared state file so +# that other processes using the same key (later cron runs, other shards) also +# hold off until the reset. + +openalex_api <- "https://api.openalex.org" + +# Wrappers around the clock, sleeping and the network so tests can replace them. +openalex_now <- function() Sys.time() +openalex_sleep <- function(seconds) Sys.sleep(seconds) + +# Send GET requests for `urls` concurrently and return, for each, a list with +# the HTTP `status` (NA on a network error), the response `headers` as a named +# list with lowercase names, and the `body` as text. +openalex_http <- function(urls) { + pool <- curl::new_pool(total_con = length(urls), host_con = length(urls)) + out <- vector("list", length(urls)) + for (i in seq_along(urls)) { + local({ + j <- i + handle <- curl::new_handle( + url = urls[j], + useragent = "scythe (https://github.com/DataONEorg/scythe)", + timeout = 120 + ) + curl::multi_add( + handle, + pool = pool, + done = function(res) { + out[[j]] <<- list( + status = res$status_code, + headers = curl::parse_headers_list(res$headers), + body = rawToChar(res$content) + ) + }, + fail = function(msg) { + out[[j]] <<- list(status = NA_integer_, headers = list(), body = NULL, error = msg) + } + ) + }) + } + curl::multi_run(pool = pool) + out +} + +# Credits OpenAlex charges for one request: 10 for any search, 1 otherwise. +openalex_request_cost <- function(query) { + if ("search" %in% names(query) || grepl("\\.search", query$filter %||% "")) 10 else 1 +} + +openalex_url <- function(client, path, query) { + query$include_xpac <- "true" + if (!is.na(client$key)) query$api_key <- client$key + if (!is.na(client$mailto)) query$mailto <- client$mailto + values <- vapply(query, function(v) utils::URLencode(as.character(v), reserved = TRUE), "") + paste0(openalex_api, path, "?", paste(names(query), values, sep = "=", collapse = "&")) +} + +# Read and write the JSON state file shared by every process that uses the +# same OpenAlex key. Writes go through a temporary file and a rename so that a +# reader never sees a partly written file. Updates aren't locked, so a writer +# never shortens a pause another process has recorded. +read_openalex_state <- function(path) { + if (is.null(path) || !file.exists(path)) return(list()) + tryCatch(jsonlite::read_json(path, simplifyVector = TRUE), error = function(e) list()) +} + +update_openalex_state <- function(path, ...) { + if (is.null(path)) return(invisible()) + state <- read_openalex_state(path) + fields <- list(...) + if (!is.null(fields$paused_until) && !is.null(state$paused_until) && + isTRUE(parse_utc(state$paused_until) > parse_utc(fields$paused_until))) { + fields$paused_until <- state$paused_until + } + state[names(fields)] <- fields + state <- Filter(Negate(is.null), state) + dir.create(dirname(path), showWarnings = FALSE, recursive = TRUE) + tmp <- paste0(path, ".tmp", Sys.getpid()) + jsonlite::write_json(state, tmp, auto_unbox = TRUE, pretty = TRUE) + file.rename(tmp, path) + invisible(state) +} + +format_utc <- function(time) format(time, "%Y-%m-%dT%H:%M:%SZ", tz = "UTC") +parse_utc <- function(text) as.POSIXct(text, format = "%Y-%m-%dT%H:%M:%SZ", tz = "UTC") + +# The end of a pause recorded in the shared state file, or NULL if none is +# still in force. +shared_pause <- function(path) { + paused_until <- read_openalex_state(path)$paused_until + if (is.null(paused_until)) return(NULL) + paused_until <- parse_utc(paused_until) + if (is.na(paused_until) || paused_until <= openalex_now()) return(NULL) + paused_until +} + +#' Create an OpenAlex client +#' +#' @param max_credits the most credits this client may spend +#' @param state_file path of the shared state file recording rate-limit +#' pauses, or NULL to keep pauses within this process only +#' @return an environment holding the client's settings and running totals +#' @noRd +openalex_client <- function(max_credits = Inf, + state_file = NULL, + concurrency = getOption("scythe.openalex.concurrency", 4), + max_wait = getOption("scythe.openalex.max_wait", 300), + reserve = getOption("scythe.openalex.reserve", 0.05)) { + client <- new.env(parent = emptyenv()) + client$key <- scythe_get_key("openalex") + client$mailto <- scythe_get_key("openalex_mailto") + client$max_credits <- max_credits + client$state_file <- state_file + client$concurrency <- concurrency + client$active_concurrency <- concurrency + client$max_wait <- max_wait + client$reserve <- reserve + client$credits_used <- 0 + client$requests <- 0 + client$remaining <- NA_real_ + client$limit <- NA_real_ + client$reset <- NA_real_ + client$paused_until <- NULL + client$consecutive_429 <- 0 + client$last_pause <- 0 + client$successes <- 0 + # Why the client stopped sending requests, or NULL while it is still active. + client$stopped <- NULL + + # Honor a pause recorded by an earlier run or by another process. + paused_until <- shared_pause(state_file) + if (!is.null(paused_until)) { + client$paused_until <- paused_until + client$stopped <- "rate_limited" + message("OpenAlex is rate limited until ", format_utc(paused_until), + " (recorded in ", state_file, "); not sending any requests.") + } + client +} + +# Record the rate-limit headers from a response, and open the breaker when the +# daily budget has run out. +openalex_note_headers <- function(client, headers) { + num <- function(name) suppressWarnings(as.numeric(headers[[name]] %||% NA)) + if (!is.na(num("x-ratelimit-limit"))) client$limit <- num("x-ratelimit-limit") + if (!is.na(num("x-ratelimit-reset"))) client$reset <- num("x-ratelimit-reset") + if (!is.na(num("x-ratelimit-remaining"))) { + client$remaining <- num("x-ratelimit-remaining") + if (client$remaining <= 0) { + openalex_pause(client, client$reset %||% NA, "daily budget used up") + } + } +} + +# Open the circuit breaker for `seconds`. A pause longer than `max_wait` is +# written to the shared state file straight away so other processes see it. +openalex_pause <- function(client, seconds, reason) { + if (is.na(seconds)) seconds <- 60 * 2^max(0, client$consecutive_429 - 1) + if (client$consecutive_429 >= 3) seconds <- max(seconds, 2 * client$last_pause) + client$last_pause <- seconds + until <- openalex_now() + seconds + if (is.null(client$paused_until) || until > client$paused_until) { + client$paused_until <- until + } + message("OpenAlex: ", reason, "; pausing requests until ", format_utc(client$paused_until)) + if (seconds > client$max_wait) { + update_openalex_state(client$state_file, paused_until = format_utc(client$paused_until)) + } +} + +# Return TRUE when the client may send requests. A short pause is waited out +# here, after which requests resume one at a time; a long pause stops the +# client for the rest of this run. Pauses that other processes record in the +# shared state file while this client runs are honoured too. +openalex_ready <- function(client) { + if (!is.null(client$stopped)) return(FALSE) + shared <- shared_pause(client$state_file) + if (!is.null(shared) && (is.null(client$paused_until) || shared > client$paused_until)) { + client$paused_until <- shared + } + if (is.null(client$paused_until)) return(TRUE) + wait <- as.numeric(difftime(client$paused_until, openalex_now(), units = "secs")) + if (wait > client$max_wait) { + client$stopped <- "rate_limited" + update_openalex_state(client$state_file, paused_until = format_utc(client$paused_until)) + warning(call. = FALSE, + "OpenAlex is rate limited until ", format_utc(client$paused_until), + "; stopping with partial results. Run again after that time to continue.") + return(FALSE) + } + if (wait > 0) openalex_sleep(wait) + client$paused_until <- NULL + client$active_concurrency <- 1 + client$successes <- 0 + TRUE +} + +#' Run OpenAlex queries +#' +#' Sends each query in `queries` (a list of named lists of query parameters +#' for `path`), following cursor pagination for the queries where `paginate` +#' is TRUE, and collects the results. +#' +#' @return a list with `results` (the collected works for each query), `count` +#' (`meta$count` for each query) and `status` ("done", "failed", or +#' "pending" when the client stopped before the query finished) +#' @noRd +openalex_run <- function(client, queries, paginate = TRUE, path = "/works") { + n <- length(queries) + paginate <- rep_len(paginate, n) + results <- vector("list", n) + count <- rep(NA_real_, n) + status <- rep("pending", n) + cursor <- ifelse(paginate, "*", NA_character_) + attempts <- rep(0L, n) + errors <- rep(NA_character_, n) + + repeat { + todo <- which(status == "pending") + if (length(todo) == 0 || !openalex_ready(client)) break + + wave <- utils::head(todo, client$active_concurrency) + cost <- vapply(queries[wave], openalex_request_cost, 1) + within_cap <- client$credits_used + cumsum(cost) <= client$max_credits + if (!within_cap[1]) { + client$stopped <- "max_credits" + warning(call. = FALSE, "Stopped after spending ", client$credits_used, + " OpenAlex credits (max_credits = ", client$max_credits, ").") + break + } + wave <- wave[within_cap] + cost <- cost[within_cap] + if (!is.na(client$remaining)) { + reserve <- if (is.na(client$limit)) 0 else client$reserve * client$limit + affordable <- client$remaining - cumsum(cost) >= reserve + if (!affordable[1]) { + openalex_pause(client, client$reset, "daily budget is down to its reserve") + next + } + wave <- wave[affordable] + } + + urls <- vapply(wave, function(i) { + q <- queries[[i]] + if (paginate[i]) q$cursor <- cursor[i] + openalex_url(client, path, q) + }, "") + responses <- openalex_http(urls) + client$requests <- client$requests + length(wave) + + for (k in seq_along(wave)) { + i <- wave[k] + res <- responses[[k]] + openalex_note_headers(client, res$headers) + code <- res$status + if (!is.na(code) && code == 200) { + # OpenAlex charges for the request whether or not its body parses. + used <- suppressWarnings(as.numeric(res$headers[["x-ratelimit-credits-used"]] %||% NA)) + client$credits_used <- client$credits_used + if (is.na(used)) cost[k] else used + body <- tryCatch(jsonlite::fromJSON(res$body, simplifyVector = FALSE), + error = function(e) NULL) + if (is.null(body)) { + status[i] <- "failed" + errors[i] <- "unparseable response" + next + } + client$consecutive_429 <- 0 + client$successes <- client$successes + 1 + results[[i]] <- c(results[[i]], body$results) + count[i] <- body$meta$count %||% length(body$results) + next_cursor <- body$meta$next_cursor + if (paginate[i] && !is.null(next_cursor) && length(body$results) > 0) { + cursor[i] <- next_cursor + } else { + status[i] <- "done" + } + } else if (!is.na(code) && code == 429) { + # Not retried directly: the breaker decides when requests resume. + client$consecutive_429 <- client$consecutive_429 + 1 + wait <- suppressWarnings(as.numeric( + res$headers[["retry-after"]] %||% res$headers[["x-ratelimit-reset"]] %||% NA + )) + openalex_pause(client, wait, "rate limited (HTTP 429)") + } else if (is.na(code) || code >= 500) { + attempts[i] <- attempts[i] + 1L + errors[i] <- res$error %||% paste("HTTP", code) + if (attempts[i] >= 4) { + status[i] <- "failed" + } else { + wait <- suppressWarnings(as.numeric(res$headers[["retry-after"]] %||% NA)) + openalex_sleep(if (is.na(wait)) min(60, 2^attempts[i]) else wait) + } + } else { + status[i] <- "failed" + msg <- tryCatch(jsonlite::fromJSON(res$body)$message, error = function(e) NULL) + errors[i] <- paste0("HTTP ", code, if (!is.null(msg)) paste0(": ", msg)) + } + } + + if (client$active_concurrency < client$concurrency && client$successes >= 10) { + client$active_concurrency <- client$concurrency + } + } + + for (i in which(status == "failed")) { + message("OpenAlex query failed: ", errors[i]) + } + list(results = results, count = count, status = status, error = errors) +} + +#' Estimate the credits a search will use +#' +#' Text screening costs 10 credits per batch; attributing hits to single DOIs +#' costs about 2 searches per level of bisection for each cited DOI (assuming +#' `hit_rate` of the DOIs are cited); the citation graph costs about 3 list +#' requests per batch (resolving DOIs, then one or more pages of citing works). +#' @noRd +estimate_openalex_cost <- function(n, methods = c("text", "graph"), batch_size = 100, + hit_rate = 0.05) { + if (n == 0) return(0) + batches <- ceiling(n / batch_size) + cost <- 0 + if ("text" %in% methods) { + hits <- max(1, ceiling(hit_rate * n)) + depth <- ceiling(log2(min(batch_size, n))) + cost <- cost + 10 * batches + 10 * 2 * hits * depth + } + if ("graph" %in% methods) cost <- cost + 3 * batches + ceiling(cost) +} diff --git a/R/openalex_search.R b/R/openalex_search.R new file mode 100644 index 0000000..81ee159 --- /dev/null +++ b/R/openalex_search.R @@ -0,0 +1,181 @@ +# The two ways the OpenAlex adapter finds works that cite a dataset DOI: +# +# - text: an exact full-text search for the DOI string, which finds DOIs +# mentioned anywhere in a work (methods, data availability statements, +# acknowledgements) whether or not OpenAlex indexes the dataset itself. +# - graph: OpenAlex's citation graph, which finds works whose parsed reference +# lists resolve to the dataset, for datasets that OpenAlex indexes. +# +# Both return a list with `pairs` (a data.frame with one row per dataset DOI +# and citing work) and, per input batch, whether it `completed` or `failed`. + +openalex_work_fields <- "id,doi,title,type,publication_year,primary_location" + +# Characters that would break OpenAlex's filter syntax; DOIs containing them +# are left out of text searches. +openalex_unsearchable <- function(dois) grepl("[\",|[:space:]]", dois) + +# Split DOIs into batches of at most `size`, also keeping each batch's search +# filter short enough for a URL. +chunk_for_search <- function(dois, size, max_chars = 3000) { + batches <- list() + current <- character() + chars <- 0 + for (d in dois) { + len <- nchar(utils::URLencode(d, reserved = TRUE)) + 9 + if (length(current) >= size || (length(current) > 0 && chars + len > max_chars)) { + batches[[length(batches) + 1]] <- current + current <- character() + chars <- 0 + } + current <- c(current, d) + chars <- chars + len + } + if (length(current)) batches[[length(batches) + 1]] <- current + batches +} + +openalex_since_filter <- function(since) { + if (is.null(since)) "" else paste0(",from_publication_date:", since) +} + +openalex_id <- function(x) sub("^https://openalex.org/", "", x %||% NA_character_) +openalex_doi <- function(x) tolower(sub("^https://doi.org/", "", x %||% NA_character_)) + +# Flatten OpenAlex work records into a data.frame of the fields the adapter and +# its review logs use, one row per work. +works_to_df <- function(works) { + if (length(works) == 0) { + return(data.frame(work_id = character(), work_doi = character(), title = character(), + type = character(), year = integer(), landing_page = character(), + venue = character(), stringsAsFactors = FALSE)) + } + data.frame( + work_id = vapply(works, function(w) openalex_id(w$id), ""), + work_doi = vapply(works, function(w) openalex_doi(w$doi), ""), + title = vapply(works, function(w) w$title %||% NA_character_, ""), + type = vapply(works, function(w) w$type %||% NA_character_, ""), + year = vapply(works, function(w) as.integer(w$publication_year %||% NA), 1L), + landing_page = vapply(works, function(w) w$primary_location$landing_page_url %||% NA_character_, ""), + venue = vapply(works, function(w) w$primary_location$source$display_name %||% NA_character_, ""), + stringsAsFactors = FALSE + ) +} + +empty_pairs <- function() cbind(dataset_doi = character(), works_to_df(list()), method = character()) + +#' Find works that mention DOIs in their full text +#' +#' Each batch of DOIs is first screened with one search that ORs all of them +#' together; most batches have no matches. A batch with matches is split in +#' half and each half searched again, down to single DOIs, so the works found +#' can be attributed to the DOI they mention (adaptive group testing). Only the +#' single-DOI searches fetch the works themselves; the others just count them. +#' @noRd +openalex_text_search <- function(client, batches, since = NULL) { + # "running" until every search for the batch has finished, then "done"; + # "incomplete" if the client stopped first, "failed" if a search failed. + state <- rep("running", length(batches)) + pairs <- list() + groups <- lapply(seq_along(batches), function(b) list(batch = b, dois = batches[[b]])) + + while (length(groups) > 0) { + leaf <- lengths(lapply(groups, `[[`, "dois")) == 1 + queries <- lapply(seq_along(groups), function(g) { + dois <- groups[[g]]$dois + list( + filter = paste0("fulltext.search.exact:", paste0("\"", dois, "\"", collapse = "|"), + openalex_since_filter(since)), + select = if (leaf[g]) openalex_work_fields else "id", + `per-page` = if (leaf[g]) "200" else "1" + ) + }) + run <- openalex_run(client, queries, paginate = leaf) + + next_groups <- list() + for (g in seq_along(groups)) { + batch <- groups[[g]]$batch + dois <- groups[[g]]$dois + if (run$status[g] == "failed") { + state[batch] <- "failed" + } else if (run$status[g] == "pending") { + if (state[batch] == "running") state[batch] <- "incomplete" + } else if (run$count[g] > 0 && leaf[g]) { + works <- works_to_df(run$results[[g]]) + if (nrow(works)) { + pairs[[length(pairs) + 1]] <- cbind(dataset_doi = dois, works, method = "text", + stringsAsFactors = FALSE) + } + } else if (run$count[g] > 0) { + half <- ceiling(length(dois) / 2) + next_groups <- c(next_groups, list( + list(batch = batch, dois = dois[seq_len(half)]), + list(batch = batch, dois = dois[-seq_len(half)]) + )) + } + } + groups <- Filter(function(g) state[g$batch] == "running", next_groups) + if (!is.null(client$stopped)) { + state[unique(vapply(groups, `[[`, 1, "batch"))] <- "incomplete" + groups <- list() + } + } + state[state == "running"] <- "done" + + list(pairs = if (length(pairs)) do.call(rbind, pairs) else empty_pairs(), + completed = state == "done", failed = state == "failed") +} + +#' Find works that cite DOIs through OpenAlex's citation graph +#' +#' Resolves each batch of DOIs to OpenAlex work IDs (DOIs that OpenAlex does +#' not index are returned in `unresolved`), then fetches the works citing any +#' of them. Each citing work's `referenced_works` shows which of the batch's +#' datasets it cites. +#' @noRd +openalex_graph_search <- function(client, batches, since = NULL) { + completed <- rep(FALSE, length(batches)) + failed <- rep(FALSE, length(batches)) + pairs <- list() + unresolved <- character() + + resolve <- openalex_run(client, lapply(batches, function(dois) { + list(filter = paste0("doi:", paste(dois, collapse = "|")), + select = "id,doi", `per-page` = "200") + }), paginate = FALSE) + failed[resolve$status == "failed"] <- TRUE + + ids <- vector("list", length(batches)) + for (b in which(resolve$status == "done")) { + found <- works_to_df(resolve$results[[b]]) + ids[[b]] <- stats::setNames(found$work_doi, found$work_id) + unresolved <- c(unresolved, setdiff(batches[[b]], found$work_doi)) + if (nrow(found) == 0) completed[b] <- TRUE + } + + todo <- which(resolve$status == "done" & lengths(ids) > 0) + if (length(todo) && is.null(client$stopped)) { + citing <- openalex_run(client, lapply(todo, function(b) { + list(filter = paste0("cites:", paste(names(ids[[b]]), collapse = "|"), + openalex_since_filter(since)), + select = paste0(openalex_work_fields, ",referenced_works"), `per-page` = "200") + }), paginate = TRUE) + for (k in seq_along(todo)) { + b <- todo[k] + if (citing$status[k] == "failed") failed[b] <- TRUE + if (citing$status[k] != "done") next + completed[b] <- TRUE + for (w in citing$results[[k]]) { + cited <- intersect(openalex_id(unlist(w$referenced_works)), names(ids[[b]])) + if (length(cited)) { + pairs[[length(pairs) + 1]] <- cbind(dataset_doi = unname(ids[[b]][cited]), + works_to_df(list(w)), method = "graph", + stringsAsFactors = FALSE) + } + } + } + } + + list(pairs = if (length(pairs)) do.call(rbind, pairs) else empty_pairs(), + completed = completed & !failed, failed = failed, unresolved = unresolved) +} diff --git a/R/relations.R b/R/relations.R new file mode 100644 index 0000000..2ec5fc2 --- /dev/null +++ b/R/relations.R @@ -0,0 +1,268 @@ +# Routing of citing works into bins, classification of data-to-data links, +# and the JSONL review logs written alongside the OpenAlex adapter's results. + +# DOI prefixes of DataONE member repositories. A work under one of these +# prefixes that cites a dataset is treated as a data-to-data link, since +# OpenAlex's `type` field is unreliable for repository records (EDI and +# ESS-DIVE datasets are often typed "article"). Override with +# options(scythe.repository_prefixes = ...). +repository_prefixes <- function() { + getOption("scythe.repository_prefixes", c( + "10.18739", # Arctic Data Center + "10.5063", # KNB + "10.6073", # Environmental Data Initiative (PASTA) + "10.15485", # ESS-DIVE + "10.5061" # Dryad + )) +} + +doi_prefix <- function(doi) sub("/.*$", "", doi) + +#' Sort citing works into bins +#' +#' Returns "no_doi" for works without a DOI (which `write_citation_pairs()` +#' cannot look up), "data_relation" for works that are datasets or come from +#' a data repository, and "article" for everything else. +#' @noRd +citer_bin <- function(pairs) { + no_doi <- is.na(pairs$work_doi) | pairs$work_doi == "" + prefix <- doi_prefix(pairs$work_doi) + is_data <- pairs$type %in% "dataset" | + prefix %in% repository_prefixes() | + prefix == doi_prefix(pairs$dataset_doi) + ifelse(no_doi, "no_doi", ifelse(is_data, "data_relation", "article")) +} + +# How DataCite relationTypes map onto the kinds of data-to-data relationship +# that the review log distinguishes. +relation_classes <- c( + IsVersionOf = "version", HasVersion = "version", + IsNewVersionOf = "version", IsPreviousVersionOf = "version", + Obsoletes = "version", IsObsoletedBy = "version", + IsDerivedFrom = "derivation", IsSourceOf = "derivation", + IsVariantFormOf = "derivation", IsOriginalFormOf = "derivation", + HasPart = "composition", IsPartOf = "composition", + Compiles = "composition", IsCompiledBy = "composition", + Cites = "citation", IsCitedBy = "citation", + References = "citation", IsReferencedBy = "citation", + IsSupplementTo = "citation", IsSupplementedBy = "citation" +) + +# When a pair has several relations, the first of these classes wins. +relation_class_order <- c("version", "derivation", "composition", "citation") + +# Fetch a URL from DataCite, returning curl's response (status_code NA on a +# network error). A wrapper so that tests can replace the network. +datacite_fetch <- function(url) { + handle <- curl::new_handle(useragent = "scythe (https://github.com/DataONEorg/scythe)", + timeout = 60) + tryCatch(curl::curl_fetch_memory(url, handle = handle), + error = function(e) list(status_code = NA_integer_, headers = raw(0), + content = raw(0))) +} + +# Fetch a DOI's related identifiers from DataCite, as a data.frame with the +# related DOI (normalized) and the relationType. Returns NULL for DOIs that +# DataCite does not register (such as Crossref DOIs) or that have no related +# identifiers, and NA when the lookup failed. Lookups are sent one at a time, +# well under DataCite's rate limit; a 429, a server error or a network error +# is retried with backoff before giving up. +datacite_related <- function(doi, attempts = 4) { + url <- paste0("https://api.datacite.org/dois/", utils::URLencode(doi, reserved = TRUE)) + for (attempt in seq_len(attempts)) { + res <- datacite_fetch(url) + code <- res$status_code + if (!is.na(code) && code == 404) return(NULL) + if (!is.na(code) && code == 200) { + record <- tryCatch(jsonlite::fromJSON(rawToChar(res$content), simplifyVector = FALSE), + error = function(e) NULL) + if (is.null(record)) return(NA) + related <- record$data$attributes$relatedIdentifiers + if (is.null(related) || length(related) == 0) return(NULL) + return(data.frame( + doi = normalize_doi(vapply(related, function(r) r$relatedIdentifier %||% NA_character_, "")), + relation_type = vapply(related, function(r) r$relationType %||% NA_character_, ""), + stringsAsFactors = FALSE + )) + } + # Other client errors won't succeed on a retry. + if (!is.na(code) && code != 429 && code < 500) return(NA) + if (attempt < attempts) { + headers <- if (length(res$headers)) curl::parse_headers_list(res$headers) else list() + wait <- suppressWarnings(as.numeric(headers[["retry-after"]] %||% NA)) + openalex_sleep(if (is.na(wait)) min(60, 2^attempt) else wait) + } + } + NA +} + +#' Classify the data-to-data links found by an OpenAlex search +#' +#' [citation_search_openalex()] returns only articles that cite datasets. Works +#' that are datasets themselves, or that come from a data repository, are +#' logged separately to `openalex-data-relations.jsonl`, since OpenAlex only +#' records that one cites or mentions the other. Many of these links are +#' housekeeping rather than reuse: a new version of a dataset that mentions +#' its predecessor, or a data package that lists its members. +#' +#' This function looks up each link in DataCite, which often records how two +#' datasets are related, and labels it with a `relation_class` of "version", +#' "derivation", "composition" or "citation", so that links that reflect +#' reuse of data ("derivation" and "citation") can be told apart from the +#' rest. It is a separate step, run after a search is complete, because it +#' sends up to two DataCite requests for each dataset in the log. +#' +#' **It is not needed to get citation counts.** It changes neither the results +#' of [citation_search_openalex()] nor its other logs, and nothing in scythe +#' uses its labels yet. It is here for future versions that count or report +#' data-to-data citations, and for reviewing what the data relations log holds. +#' +#' Links with no DataCite relation between the two DOIs, in either direction, +#' are labelled "unknown". If that is because a DataCite lookup failed (after +#' retrying rate limits and server errors), `evidence` also records +#' `datacite:lookup_failed:`, so that the link can be checked again later. +#' +#' @param log_dir the directory holding the review logs of +#' [citation_search_openalex()]; defaults to the `scythe.log_dir` option, or +#' the temporary directory a search in this session used +#' @return data.frame of the data-to-data links, one per pair of DOIs, with +#' `relation_type` (the DataCite relationTypes found, or "openalex:cites" or +#' "openalex:mentions" if none), `relation_class` and `evidence`. It is also +#' written to `openalex-data-relations-classified.jsonl` in `log_dir`, +#' replacing any earlier copy. +#' @export +#' @examples +#' \dontrun{ +#' citation_search_openalex(identifiers, log_dir = "logs") +#' relations <- classify_data_relations("logs") +#' table(relations$relation_class) +#' } +classify_data_relations <- function(log_dir = NULL) { + log_dir <- openalex_log_dir(log_dir) + path <- file.path(log_dir, "openalex-data-relations.jsonl") + relations <- if (file.exists(path)) { + jsonlite::stream_in(file(path), verbose = FALSE) + } else { + data.frame(dataset_doi = character(), work_doi = character(), match_method = character()) + } + # The log is appended to by every run; a pair logged again later (found by + # a second method, say) replaces the earlier record. + relations <- relations[!duplicated(paste(relations$dataset_doi, relations$work_doi), + fromLast = TRUE), , drop = FALSE] + relations <- classify_relations(relations) + out <- file.path(log_dir, "openalex-data-relations-classified.jsonl") + unlink(out) + write_jsonl(relations, out) + relations +} + +# Label each data-to-data link in `relations` (with dataset_doi, work_doi and +# match_method columns) from DataCite's relatedIdentifiers. +classify_relations <- function(relations) { + if (nrow(relations) == 0) { + return(cbind(relations, relation_type = character(), relation_class = character(), + evidence = character())) + } + relations$relation_type <- ifelse(relations$match_method == "text", + "openalex:mentions", "openalex:cites") + relations$relation_class <- rep("unknown", nrow(relations)) + relations$evidence <- paste0("openalex:", relations$match_method) + + cache <- new.env(parent = emptyenv()) + lookup <- function(doi) { + if (!exists(doi, envir = cache, inherits = FALSE)) { + assign(doi, datacite_related(doi), envir = cache) + } + get(doi, envir = cache, inherits = FALSE) + } + for (i in seq_len(nrow(relations))) { + from <- relations$dataset_doi[i] + to <- relations$work_doi[i] + types <- character() + evidence <- character() + failed <- character() + for (side in list(c(from, to), c(to, from))) { + related <- lookup(side[1]) + if (identical(related, NA)) { + failed <- c(failed, side[1]) + next + } + hit <- unique(related$relation_type[!is.na(related$doi) & related$doi == side[2]]) + if (length(hit)) { + types <- c(types, hit) + evidence <- c(evidence, paste0("datacite:", side[1])) + } + } + if (length(types)) { + # A pair can have several relations (a new version that also references + # its predecessor); keep them all and classify by the most specific. + classes <- unname(relation_classes[types]) + classes <- classes[!is.na(classes)] + relations$relation_type[i] <- paste(unique(types), collapse = ";") + relations$relation_class[i] <- if (length(classes)) { + relation_class_order[min(match(classes, relation_class_order))] + } else { + "unknown" + } + relations$evidence[i] <- paste(evidence, collapse = ";") + } else if (length(failed)) { + relations$evidence[i] <- paste(c(relations$evidence[i], + paste0("datacite:lookup_failed:", failed)), collapse = ";") + } + } + n_failed <- sum(vapply(ls(cache), function(d) identical(get(d, envir = cache), NA), TRUE)) + if (n_failed) { + message("DataCite lookups failed for ", n_failed, " DOIs; their relations are ", + "logged as unknown with datacite:lookup_failed evidence.") + } + relations +} + +# Directory for the review logs: `log_dir`, else the scythe.log_dir option, +# else a temporary directory. +openalex_log_dir <- function(log_dir = NULL) { + log_dir <- log_dir %||% getOption("scythe.log_dir") %||% file.path(tempdir(), "scythe-logs") + dir.create(log_dir, showWarnings = FALSE, recursive = TRUE) + log_dir +} + +# Build a data.frame of log records from columns whose first argument may be +# empty, recycling the remaining (scalar) columns; NULL when there is nothing +# to log. +log_rows <- function(...) { + cols <- list(...) + n <- length(cols[[1]]) + if (n == 0) return(NULL) + as.data.frame(lapply(cols, rep_len, n), stringsAsFactors = FALSE) +} + +# Append the rows of a data.frame to a JSON Lines file, one object per row. +write_jsonl <- function(df, path) { + if (is.null(df) || nrow(df) == 0) return(invisible()) + con <- file(path, open = "a", encoding = "UTF-8") + on.exit(close(con)) + jsonlite::stream_out(df, con, verbose = FALSE, na = "null", auto_unbox = TRUE) +} + +# Completed batches are recorded in a JSON Lines checkpoint so that a rerun +# with the same checkpoint directory skips them. Each line holds the batch key +# (method, date filter and DOIs) and the pairs that batch found. +read_checkpoint <- function(checkpoint_dir) { + path <- file.path(checkpoint_dir %||% "", "openalex-checkpoint.jsonl") + if (is.null(checkpoint_dir) || !file.exists(path)) return(list()) + lines <- readLines(path, encoding = "UTF-8", warn = FALSE) + records <- lapply(lines, function(l) tryCatch(jsonlite::fromJSON(l), error = function(e) NULL)) + records <- Filter(Negate(is.null), records) + stats::setNames(lapply(records, function(r) { + if (length(r$pairs) == 0) empty_pairs() else as.data.frame(r$pairs, stringsAsFactors = FALSE) + }), vapply(records, `[[`, "", "key")) +} + +write_checkpoint <- function(checkpoint_dir, key, pairs) { + if (is.null(checkpoint_dir)) return(invisible()) + dir.create(checkpoint_dir, showWarnings = FALSE, recursive = TRUE) + line <- jsonlite::toJSON(list(key = key, pairs = pairs), auto_unbox = TRUE, + na = "null", dataframe = "rows") + cat(line, "\n", file = file.path(checkpoint_dir, "openalex-checkpoint.jsonl"), + sep = "", append = TRUE) +} diff --git a/R/scythe_set_key.R b/R/scythe_set_key.R index cb327f5..30a6afd 100644 --- a/R/scythe_set_key.R +++ b/R/scythe_set_key.R @@ -6,14 +6,18 @@ #' prompt you to set a password for your keyring, should you need to lock #' or unlock it. See `keyring::keyring_unlock` for more details. #' -#' @param source (char) Key source, one of "scopus" or "springer" +#' @param source (char) Key source, one of "scopus", "springer", "openalex" +#' (an OpenAlex API key) or "openalex_mailto" (the email address sent to +#' OpenAlex with each request) #' @param secret (char) API key value #' #' @export scythe_set_key <- function(source, secret) { - if (!(source %in% c("springer", "scopus"))) { - stop("Please set the key source to be one of 'springer' or 'scopus.'", + sources <- c("springer", "scopus", "openalex", "openalex_mailto") + if (!(source %in% sources)) { + stop("Please set the key source to be one of ", + paste0("'", sources, "'", collapse = ", "), ".", call. = FALSE) } diff --git a/README.md b/README.md index de0a761..708273c 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ - [**Submit Bugs and feature requests**](https://github.com/DataONEorg/scythe/issues) Automates the full text harvesting of dataset citations from various full text article databases, -including Scopus, PLOS, Springer, and XDD. +including Scopus, PLOS, Springer, XDD, and OpenAlex. ## Installation @@ -63,6 +63,46 @@ scythe_set_key(source = "springer", secret = "YOUR_KEY") keyring::key_get("springer", keyring = "scythe") ``` +#### OpenAlex + +[OpenAlex](https://openalex.org) works without a key, but its daily budget without one is only 1,000 credits. A free API key, from an [OpenAlex account](https://openalex.org/settings/api), raises it to $1 of usage a day (10,000 credits). A full-text search costs 10 credits and any other query costs 1. Optionally, also set a contact email: + +``` +scythe_set_key(source = "openalex", secret = "YOUR_KEY") +scythe_set_key(source = "openalex_mailto", secret = "you@example.org") +``` + +`citation_search_openalex()` finds works that mention a dataset DOI anywhere in their full text, and works whose reference lists cite the dataset in OpenAlex's citation graph. It returns only citing works that have a DOI and are not datasets. Everything else is written as JSON Lines files to a review log directory, set with `log_dir` or `options(scythe.log_dir = ...)`: + +- datasets that cite or mention other datasets; +- citing works without a DOI; +- identifiers that OpenAlex doesn't index; +- batches that failed. + +Many dataset-to-dataset links are versions of a dataset or members of a data package, rather than reuse of data. After a search, `classify_data_relations(log_dir)` looks up each link in DataCite and labels it as a version, derivation, composition or citation link. It isn't needed to get citation counts, and it doesn't change the search results; it's there for reviewing these links, and for future versions that count data-to-data citations. It sends up to two DataCite requests for each dataset, but uses no OpenAlex credits. + +Before sending any request, the function estimates the credits a search will use. It stops if the estimate exceeds `max_credits`, which defaults to 500. Pass `max_credits = Inf` to approve a larger search: + +``` +results <- citation_search_openalex(identifiers, max_credits = Inf, + checkpoint_dir = "openalex-checkpoints") +``` + +If OpenAlex rate-limits a search, or the daily budget runs out, the search stops sending requests and returns what it has found so far. It records when the limit resets, so later runs that use the same `checkpoint_dir` wait until then. A rerun with the same `checkpoint_dir` skips the batches that already finished. + +For the scheduled helm job, approve OpenAlex spending with the `cronjob.openalex.approve` value: a number of credits, or `unlimited`. See `tools/README.md` for measuring OpenAlex's precision and recall. + +The package tests never spend OpenAlex credits by default. The live OpenAlex tests are skipped on CRAN and on CI, and run locally only when you opt in. Each one is capped at 100 credits: + +``` +Sys.setenv(SCYTHE_TEST_OPENALEX = "true") +devtools::test(filter = "openalex") +``` + +The same switch runs the searches in the OpenAlex vignette when it's built, which spends more credits; without it, the vignette only shows their code. + +See the Testing section of [CONTRIBUTING.md](CONTRIBUTING.md) for running all the live tests, including Springer and Scopus, before a release. + ## Acknowledgments Work on this package was supported by: diff --git a/helm/scripts/search.R b/helm/scripts/search.R index 24a6378..2766e19 100644 --- a/helm/scripts/search.R +++ b/helm/scripts/search.R @@ -13,7 +13,19 @@ main <- function(){ make_option(c("-r", "--rows"), type="integer", default=100000, help="Number of rows to return from query [default %default]"), make_option(c("-n", "--nodes"), type="character", help="Comma separated list of nodes to query"), - make_option(c("-s", "--sources"), type="character", help="Comma separated list of sources to use") + make_option(c("-s", "--sources"), type="character", help="Comma separated list of sources to use"), + make_option("--since", type="character", default=NULL, + help="OpenAlex: only find works published on or after this date (YYYY-MM-DD) [default: the last complete run minus --lookback days, or all works if there is none]"), + make_option("--lookback", type="integer", default=365, + help="OpenAlex: days before the last complete run to search from [default %default]"), + make_option("--shard", type="character", default=NULL, + help="OpenAlex: search only shard i of n of the DOIs, given as i/n"), + make_option("--log-dir", type="character", default="openalex-logs", + help="OpenAlex: directory for the review logs [default %default]"), + make_option("--checkpoint-dir", type="character", default="openalex-checkpoints", + help="OpenAlex: directory for checkpoints and the shared rate-limit state [default %default]"), + make_option("--approve", type="character", default=Sys.getenv("SCYTHE_OPENALEX_APPROVE"), + help="OpenAlex: approve spending up to this many credits, or 'unlimited'; also read from SCYTHE_OPENALEX_APPROVE") ) # parse command-line arguments @@ -61,6 +73,12 @@ main <- function(){ found_citations_xdd <- citation_search(dois_unique, "xdd") found_citations <- dplyr::bind_rows(found_citations, found_citations_xdd) } + + if ("openalex" %in% sources){ + message("Searching OpenAlex.") + found_citations_openalex <- search_openalex(dois_unique, opts) + found_citations <- dplyr::bind_rows(found_citations, found_citations_openalex) + } if (is.null(found_citations) || nrow(found_citations) == 0){ writeLines("No citations found.", fp) @@ -74,11 +92,86 @@ main <- function(){ # if (nrow(new_citations) > 0) { write.csv(found_citations, fp, row.names = FALSE) # } else { - writeLines("No new citations found.", fp) + # writeLines("No new citations found.", fp) # } } } +# Search OpenAlex, resuming an interrupted run from its checkpoints and +# recording when the last complete run happened so that the next one only +# searches recent works. +search_openalex <- function(dois, opts) { + if (!is.null(opts$shard)) { + shard <- as.integer(strsplit(opts$shard, "/", fixed = TRUE)[[1]]) + # Assign DOIs to shards by a hash of the DOI, so a DOI stays in the + # same shard as the set of DOIs changes between runs. + hash <- vapply(dois, function(d) sum(utf8ToInt(d) * seq_len(nchar(d))), 1) + dois <- dois[hash %% shard[2] == shard[1] - 1] + message("OpenAlex shard ", opts$shard, ": ", length(dois), " DOIs") + } + shard_label <- if (is.null(opts$shard)) "all" else sub("/", "-of-", opts$shard, fixed = TRUE) + checkpoint_dir <- file.path(opts$`checkpoint-dir`, shard_label) + # Rate-limit pauses are shared by every shard using the same API key. + state_file <- file.path(opts$`checkpoint-dir`, "openalex-state.json") + run_file <- file.path(checkpoint_dir, "openalex-run.json") + run <- if (file.exists(run_file)) jsonlite::read_json(run_file, simplifyVector = TRUE) else list() + + # Resume an interrupted run with the date filter it started with. + since <- opts$since + if (is.null(since) && !is.null(run$pending_since)) { + since <- if (run$pending_since == "all") NULL else run$pending_since + } else if (is.null(since) && !is.null(run$last_run)) { + since <- format(as.Date(run$last_run) - opts$lookback) + } + if (is.null(since)) message("OpenAlex: no previous complete run, so searching all works (a backfill).") + + approve <- opts$approve + max_credits <- if (identical(approve, "unlimited")) { + Inf + } else if (nzchar(approve)) { + as.numeric(approve) + } else { + getOption("scythe.openalex.max_credits", 500) + } + preflight <- !nzchar(approve) + if (preflight && interactive()) { + estimate <- scythe:::estimate_openalex_cost(length(dois)) + if (estimate > max_credits && + tolower(readline(paste0("Spend about ", estimate, " OpenAlex credits? [y/N] "))) == "y") { + max_credits <- Inf + } + } + + dir.create(checkpoint_dir, showWarnings = FALSE, recursive = TRUE) + started <- format(Sys.Date()) + found <- tryCatch( + citation_search_openalex(dois, since = since, log_dir = opts$`log-dir`, + checkpoint_dir = checkpoint_dir, state_file = state_file, + max_credits = max_credits, preflight = preflight), + error = function(e) { + message("OpenAlex search not run: ", conditionMessage(e), + " Pass --approve (or set SCYTHE_OPENALEX_APPROVE) to approve it.") + NULL + } + ) + if (is.null(found)) return(data.frame()) + + status <- attr(found, "openalex_status") + if (isTRUE(status$complete)) { + run <- list(last_run = run$pending_started %||% started) + unlink(file.path(checkpoint_dir, "openalex-checkpoint.jsonl")) + } else { + run$pending_since <- since %||% "all" + run$pending_started <- run$pending_started %||% started + message("OpenAlex search is incomplete (", status$stopped %||% "some batches failed", + "); the next run will continue from the checkpoints in ", checkpoint_dir) + } + jsonlite::write_json(run, run_file, auto_unbox = TRUE, pretty = TRUE) + found +} + +`%||%` <- function(x, y) if (is.null(x)) y else x + get_node_dois <- function(node_id, num_rows) { mn <- getMNode(CNode("PROD"), node_id) diff --git a/helm/templates/cron-job.yaml b/helm/templates/cron-job.yaml index 3046a72..a10360f 100644 --- a/helm/templates/cron-job.yaml +++ b/helm/templates/cron-job.yaml @@ -4,6 +4,9 @@ metadata: name: {{ .Values.cronjob.name }} spec: schedule: {{ .Values.cronjob.schedule | quote }} + # Never start a run while the previous one is still going, so that runs do + # not compete for the same OpenAlex budget. + concurrencyPolicy: Forbid successfulJobsHistoryLimit: 5 failedJobsHistoryLimit: 5 jobTemplate: @@ -26,6 +29,20 @@ spec: secretKeyRef: name: api-keys key: scopus + - name: openalex + valueFrom: + secretKeyRef: + name: api-keys + key: openalex + optional: true + - name: openalex_mailto + valueFrom: + secretKeyRef: + name: api-keys + key: openalex_mailto + optional: true + - name: SCYTHE_OPENALEX_APPROVE + value: {{ .Values.cronjob.openalex.approve | quote }} command: ["Rscript"] args: - "--vanilla" @@ -36,6 +53,12 @@ spec: - "{{ .Values.cronjob.nodes }}" - "-s" - "{{ .Values.cronjob.sources }}" + - "--lookback" + - "{{ .Values.cronjob.openalex.lookback }}" + - "--log-dir" + - "{{ .Values.persistence.mountPath }}/openalex-logs" + - "--checkpoint-dir" + - "{{ .Values.persistence.mountPath }}/openalex-checkpoints" volumeMounts: - name: {{ .Values.persistence.claimName }} diff --git a/helm/values.yaml b/helm/values.yaml index 579449b..e17fea9 100644 --- a/helm/values.yaml +++ b/helm/values.yaml @@ -22,8 +22,16 @@ cronjob: schedule: "0 12 1 * *" script: /apps/scythe/search.R nodes: 'urn:node:ARCTIC,urn:node:ESS_DIVE' - sources: 'springer,scopus,xdd' + sources: 'springer,scopus,xdd,openalex' rows: 10 + openalex: + # Approval for OpenAlex searches: a number of credits the job may spend, + # or "unlimited" to run without a limit (until the search completes or the + # daily budget runs out, resuming on the next run). Leave empty to only run + # searches whose estimated cost is under 500 credits. + approve: "" + # Days before the last complete run that each search looks back over. + lookback: 365 serviceAccount: # Specifies whether a service account should be created diff --git a/inst/testdata/doi-variants.csv b/inst/testdata/doi-variants.csv new file mode 100644 index 0000000..04b375a --- /dev/null +++ b/inst/testdata/doi-variants.csv @@ -0,0 +1,29 @@ +input,expected +10.18739/A22274,10.18739/a22274 +doi:10.18739/A22274,10.18739/a22274 +DOI:10.18739/A22274,10.18739/a22274 +doi: 10.18739/A22274,10.18739/a22274 +DOI 10.18739/A22274,10.18739/a22274 +https://doi.org/10.18739/A22274,10.18739/a22274 +http://doi.org/10.18739/A22274,10.18739/a22274 +https://dx.doi.org/10.18739/A22274,10.18739/a22274 +http://dx.doi.org/10.18739/A22274,10.18739/a22274 +https://www.doi.org/10.18739/A22274,10.18739/a22274 +doi.org/10.18739/A22274,10.18739/a22274 +dx.doi.org/10.18739/A22274,10.18739/a22274 +HTTPS://DOI.ORG/10.18739/A22274,10.18739/a22274 +https://hdl.handle.net/10.18739/A22274,10.18739/a22274 +info:doi/10.18739/A22274,10.18739/a22274 +urn:doi:10.18739/A22274,10.18739/a22274 +doi:https://doi.org/10.18739/A22274,10.18739/a22274 +https://doi.org/doi:10.18739/A22274,10.18739/a22274 +https://doi.org/10.18739%2FA22274,10.18739/a22274 + doi:10.18739/A22274. ,10.18739/a22274 +,10.18739/a22274 +"10.18739/A22274",10.18739/a22274 +(https://doi.org/10.18739/A22274),10.18739/a22274 +10.1002/(SICI)1097-4679(199711)53:7<657::AID-JCLP5>3.0.CO;2-F,10.1002/(sici)1097-4679(199711)53:7<657::aid-jclp5>3.0.co;2-f +urn:uuid:3249ada0-afe3-4dd6-875e-0f7928a4c171, +https://example.org/10.18739/A22274, +not a doi, +, diff --git a/inst/testdata/openalex-negatives.csv b/inst/testdata/openalex-negatives.csv new file mode 100644 index 0000000..a1cae74 --- /dev/null +++ b/inst/testdata/openalex-negatives.csv @@ -0,0 +1,5 @@ +article_id,dataset_id,reason +10.1371/journal.pone.0213037,10.18739/A2227,DOI that is a prefix of the cited 10.18739/A22274 +10.1371/journal.pone.0213037,10.18739/A222745,DOI that extends the cited 10.18739/A22274 +10.18739/a22274,10.18739/A22274,the dataset itself +10.15485/1378108,10.17190/AMF/1245971,newer version of the dataset (belongs in the data relations log) diff --git a/inst/testdata/test-citations-mbj.csv b/inst/testdata/test-citations-mbj.csv new file mode 100644 index 0000000..90924a0 --- /dev/null +++ b/inst/testdata/test-citations-mbj.csv @@ -0,0 +1,5 @@ +pub_id, lead_author_surname, year, title, dataset_id_as_cited, dataone_pid +10.1371/journal.pone.0213037,Beard,2019,"Migratory goose arrival time plays a larger role in influencing forage quality than advancing springs in an Arctic coastal wetland",https://doi.org/10.18739/A22274,10.18739/A22274 +10.1371/journal.pone.0199789,Couture,2018,"A funder-imposed data publication requirement seldom inspired data sharing",https://doi.org/10.5063/F1T151VR,10.5063/F1T151VR +10.1371/journal.pone.0162907,Harris,2016,"A Complete Fossil-Calibrated Phylogeny of Seed Plant Families as a Tool for Comparative Analyses: Testing the ‘Time for Speciation’ Hypothesis",http://dx.doi.org/10.5063/F13T9F5P,10.5063/F13T9F5P +10.1371/journal.pone.0090826,Olson,2014,Sampling Strategies and Biodiversity of Influenza A Subtypes in Wild Birds,http://doi.org/10.5063/F1HT2M7Q,10.5063/F1HT2M7Q diff --git a/inst/testdata/test-citations.csv b/inst/testdata/test-citations.csv index de971b1..f2c2f65 100644 --- a/inst/testdata/test-citations.csv +++ b/inst/testdata/test-citations.csv @@ -7,3 +7,5 @@ pub_id,lead_author_surname,year,title,dataset_id_as_cited,dataone_pid,source 10.5194/essd-11-1957-2019,Broadman,2019,"An Arctic watershed observatory at Lake Peters, Alaska: Weather-glacier-river-lake system data for 2015-2018",https://doi.org/10.18739/a2zs2kc8c,10.18739/A2ZS2KC8C,scopus 10.1029/2018JG004528,Koch,2018,Ice Wedge Degradation and Stabilization Impact Water Budgets and Nutrient Cycling in Arctic Trough Ponds,https://arcticdata.io/catalog/#view/doi:10.18739/A29K97,10.18739/A29K97,xdd 10.1007/s13280-019-01295-7,Schwoerer,2020,Hitchhikers on floats to Arctic freshwater: Private aviation and recreation loss from aquatic invasion,https://doi.org/10.18739/A2M32N95V,10.18739/A2M32N95V,springer +10.1371/journal.pone.0213037,Beard,2019,Migratory goose arrival time plays a larger role in influencing forage quality than advancing springs in an Arctic coastal wetland,https://doi.org/10.18739/A22274,10.18739/A22274,openalex +10.1111/1365-2745.13191,Choi,2019,Phenological mismatch between season advancement and migration timing alters Arctic plant traits,https://doi.org/10.18739/A22274,10.18739/A22274,openalex diff --git a/man/citation_search.Rd b/man/citation_search.Rd index e3bae35..8afa49a 100644 --- a/man/citation_search.Rd +++ b/man/citation_search.Rd @@ -10,7 +10,10 @@ citation_search(identifiers, sources = c("plos", "scopus", "springer", "xdd")) \item{identifiers}{a vector of identifiers to be searched for} \item{sources}{a vector indicating which sources to query (one or -more of plos, scopus, springer)} +more of plos, scopus, springer, xdd, openalex). OpenAlex spends credits +from a daily budget, so it is only searched when named here; call +[citation_search_openalex()] directly to control its cost and to resume +an interrupted search.} } \value{ tibble of matching dataset and publication identifiers diff --git a/man/citation_search_openalex.Rd b/man/citation_search_openalex.Rd new file mode 100644 index 0000000..0882bfe --- /dev/null +++ b/man/citation_search_openalex.Rd @@ -0,0 +1,95 @@ +% Generated by roxygen2: do not edit by hand +% Please edit documentation in R/citation_search_openalex.R +\name{citation_search_openalex} +\alias{citation_search_openalex} +\title{Search for citations in OpenAlex} +\usage{ +citation_search_openalex( + identifiers, + methods = c("text", "graph"), + since = NULL, + log_dir = NULL, + checkpoint_dir = NULL, + state_file = NULL, + max_credits = getOption("scythe.openalex.max_credits", 500), + preflight = TRUE, + batch_size = 100 +) +} +\arguments{ +\item{identifiers}{a vector of identifiers to be searched for} + +\item{methods}{one or both of "text" and "graph"} + +\item{since}{only find works published on or after this date +(a Date or "YYYY-MM-DD"), or NULL to search all works} + +\item{log_dir}{directory for the review logs; defaults to the +`scythe.log_dir` option, or a temporary directory} + +\item{checkpoint_dir}{directory where completed batches are recorded, so +that a rerun with the same directory continues where an interrupted run +stopped; NULL to disable checkpointing} + +\item{state_file}{file recording when OpenAlex rate limits end, shared by +every process using the same API key so none of them send requests before +then; defaults to `openalex-state.json` in `checkpoint_dir`} + +\item{max_credits}{the most OpenAlex credits to spend; defaults to the +`scythe.openalex.max_credits` option, or 500} + +\item{preflight}{if TRUE, stop before sending any request when the +estimated cost exceeds `max_credits`; if FALSE, run until `max_credits` +have been spent} + +\item{batch_size}{the number of DOIs in each OpenAlex query (at most 100)} +} +\value{ +data.frame of matching dataset and publication identifiers, with an + `openalex_status` attribute recording whether the search completed, why + it stopped if not, and the credits used +} +\description{ +This function searches [OpenAlex](https://openalex.org) for works that cite +or mention each dataset DOI, in two ways: +} +\details{ +- `"text"`: an exact full-text search for the DOI, which finds DOIs + mentioned anywhere in a work's text (methods, data availability statements, + acknowledgements, reference lists), whether or not OpenAlex indexes the + dataset. DOIs are searched in batches, and batches with matches are split + until each match is attributed to a single DOI. +- `"graph"`: OpenAlex's citation graph, which finds works whose reference + lists resolve to the dataset, for datasets that OpenAlex indexes. + +The result only includes citing works that have a DOI and are not datasets +themselves. Everything else is written as JSON Lines files to `log_dir` for +review: `openalex-matches.jsonl` (every article match and which method found +it), `openalex-data-relations.jsonl` (datasets citing or mentioning other +datasets, which [classify_data_relations()] can label as version, +derivation, composition or citation links afterwards; that isn't needed for +the results), `openalex-no-doi-citers.jsonl` (citing works +without a DOI), `openalex-unresolved.jsonl` (identifiers that are not DOIs, +or that OpenAlex does not index) and `openalex-failed-batches.jsonl`. +The logs are appended to, and a run that resumes from `checkpoint_dir` only +logs what it found itself, so that a search run in installments logs each +match once. + +OpenAlex charges credits for each request against a daily budget: 1 for a +list query and 10 for a full-text search. Set an API key with +`scythe_set_key("openalex", ...)` for a larger budget than the 1,000 daily +credits available without one, and optionally a contact email as +`openalex_mailto`. Before sending any request, the function estimates the +credits the search will use and stops if that exceeds `max_credits`; set +`max_credits = Inf` to approve a large search. While running, it never +spends more than `max_credits`, and stops sending requests as soon as +OpenAlex reports a rate limit, returning the results found so far (see the +`openalex_status` attribute of the result). +} +\examples{ +\dontrun{ +identifiers <- c("10.18739/A22274", "10.5063/F1T151VR") +result <- citation_search_openalex(identifiers) +attr(result, "openalex_status") +} +} diff --git a/man/classify_data_relations.Rd b/man/classify_data_relations.Rd new file mode 100644 index 0000000..af75f25 --- /dev/null +++ b/man/classify_data_relations.Rd @@ -0,0 +1,53 @@ +% Generated by roxygen2: do not edit by hand +% Please edit documentation in R/relations.R +\name{classify_data_relations} +\alias{classify_data_relations} +\title{Classify the data-to-data links found by an OpenAlex search} +\usage{ +classify_data_relations(log_dir = NULL) +} +\arguments{ +\item{log_dir}{the directory holding the review logs of +[citation_search_openalex()]; defaults to the `scythe.log_dir` option, or +the temporary directory a search in this session used} +} +\value{ +data.frame of the data-to-data links, one per pair of DOIs, with + `relation_type` (the DataCite relationTypes found, or "openalex:cites" or + "openalex:mentions" if none), `relation_class` and `evidence`. It is also + written to `openalex-data-relations-classified.jsonl` in `log_dir`, + replacing any earlier copy. +} +\description{ +[citation_search_openalex()] returns only articles that cite datasets. Works +that are datasets themselves, or that come from a data repository, are +logged separately to `openalex-data-relations.jsonl`, since OpenAlex only +records that one cites or mentions the other. Many of these links are +housekeeping rather than reuse: a new version of a dataset that mentions +its predecessor, or a data package that lists its members. +} +\details{ +This function looks up each link in DataCite, which often records how two +datasets are related, and labels it with a `relation_class` of "version", +"derivation", "composition" or "citation", so that links that reflect +reuse of data ("derivation" and "citation") can be told apart from the +rest. It is a separate step, run after a search is complete, because it +sends up to two DataCite requests for each dataset in the log. + +**It is not needed to get citation counts.** It changes neither the results +of [citation_search_openalex()] nor its other logs, and nothing in scythe +uses its labels yet. It is here for future versions that count or report +data-to-data citations, and for reviewing what the data relations log holds. + +Links with no DataCite relation between the two DOIs, in either direction, +are labelled "unknown". If that is because a DataCite lookup failed (after +retrying rate limits and server errors), `evidence` also records +`datacite:lookup_failed:`, so that the link can be checked again later. +} +\examples{ +\dontrun{ +citation_search_openalex(identifiers, log_dir = "logs") +relations <- classify_data_relations("logs") +table(relations$relation_class) +} +} diff --git a/man/report_est_wait.Rd b/man/report_est_wait.Rd deleted file mode 100644 index 7384b06..0000000 --- a/man/report_est_wait.Rd +++ /dev/null @@ -1,16 +0,0 @@ -% Generated by roxygen2: do not edit by hand -% Please edit documentation in R/helpers.R -\name{report_est_wait} -\alias{report_est_wait} -\title{Report estimated wait for rate limited queries} -\usage{ -report_est_wait(n_queries, wait_seconds) -} -\arguments{ -\item{n_queries}{number of queries} - -\item{wait_seconds}{wait time in seconds between queries} -} -\description{ -Report estimated wait for rate limited queries -} diff --git a/man/scythe_set_key.Rd b/man/scythe_set_key.Rd index f468b24..1d80d42 100644 --- a/man/scythe_set_key.Rd +++ b/man/scythe_set_key.Rd @@ -7,7 +7,9 @@ scythe_set_key(source, secret) } \arguments{ -\item{source}{(char) Key source, one of "scopus" or "springer"} +\item{source}{(char) Key source, one of "scopus", "springer", "openalex" +(an OpenAlex API key) or "openalex_mailto" (the email address sent to +OpenAlex with each request)} \item{secret}{(char) API key value} } diff --git a/tests/testthat/helper-openalex.R b/tests/testthat/helper-openalex.R new file mode 100644 index 0000000..efaf0af --- /dev/null +++ b/tests/testthat/helper-openalex.R @@ -0,0 +1,132 @@ +# Live OpenAlex tests spend credits, so they never run on CRAN or CI, and run +# locally only when a developer opts in with SCYTHE_TEST_OPENALEX=true. An API +# key alone isn't enough: without one, OpenAlex charges the per-IP budget. +skip_unless_openalex_live <- function() { + testthat::skip_on_cran() + testthat::skip_on_ci() + testthat::skip_if_not( + isTRUE(as.logical(Sys.getenv("SCYTHE_TEST_OPENALEX", "false"))), + "live OpenAlex tests spend credits; set SCYTHE_TEST_OPENALEX=true to run them" + ) + testthat::skip_if_offline("api.openalex.org") +} + +# Skip a live test whose search stopped early because the OpenAlex budget is +# rate limited, rather than failing on its partial results. +skip_if_openalex_rate_limited <- function(res) { + status <- attr(res, "openalex_status") + testthat::skip_if(identical(status$stopped, "rate_limited"), + paste("OpenAlex is rate limited until", status$paused_until %||% "later")) +} + +# Read a JSON Lines review log, or an empty data.frame if nothing was logged. +read_log <- function(res, name) { + path <- file.path(attr(res, "openalex_status")$log_dir, name) + if (!file.exists(path)) return(data.frame()) + jsonlite::stream_in(file(path), verbose = FALSE) +} + +# A fake OpenAlex API for testing the OpenAlex adapter offline. +# +# `works` is a list of work records; each may have a `text` field (the DOIs +# its full text mentions) and `referenced_works`. The fake answers the +# fulltext.search.exact, doi and cites filters the adapter uses, records every +# URL it is sent, and returns the responses queued in `responses` (one list of +# status/headers, and optionally a body, per request) before falling back to +# answering normally. +fake_openalex <- function(works = list(), responses = list(), headers = list()) { + server <- new.env() + server$works <- works + server$responses <- responses + server$headers <- headers + server$urls <- character() + server$http <- function(urls) { + lapply(urls, function(url) { + server$urls <- c(server$urls, url) + if (length(server$responses)) { + res <- server$responses[[1]] + server$responses <- server$responses[-1] + if (!is.null(res$status) && res$status != 200) { + return(list(status = res$status, headers = res$headers %||% list(), body = "{}")) + } + if (!is.null(res$body)) { + return(list(status = 200, headers = c(res$headers, server$headers), body = res$body)) + } + extra <- res$headers %||% list() + } else { + extra <- list() + } + body <- fake_openalex_answer(server$works, url) + list(status = 200, headers = c(extra, server$headers), body = body) + }) + } + server +} + +fake_openalex_answer <- function(works, url) { + query <- strsplit(sub("^[^?]*\\?", "", url), "&")[[1]] + params <- stats::setNames( + lapply(query, function(q) utils::URLdecode(sub("^[^=]*=", "", q))), + sub("=.*$", "", query) + ) + filter <- params$filter + value <- sub("^[^:]*:", "", sub(",from_publication_date:.*$", "", filter)) + terms <- strsplit(value, "|", fixed = TRUE)[[1]] + if (startsWith(filter, "fulltext.search.exact:")) { + terms <- gsub("\"", "", terms) + hits <- Filter(function(w) any(terms %in% w$text), works) + } else if (startsWith(filter, "doi:")) { + hits <- Filter(function(w) sub("^https://doi.org/", "", w$doi %||% "") %in% terms, works) + } else if (startsWith(filter, "cites:")) { + hits <- Filter(function(w) { + any(paste0("https://openalex.org/", terms) %in% unlist(w$referenced_works)) + }, works) + } else { + stop("fake OpenAlex does not support filter ", filter) + } + per_page <- as.integer(params[["per-page"]] %||% 25) + start <- if (identical(params$cursor, "*") || is.null(params$cursor)) 0 else as.integer(params$cursor) + page <- utils::head(utils::tail(hits, max(0, length(hits) - start)), per_page) + next_cursor <- if (!is.null(params$cursor) && start + per_page < length(hits)) { + as.character(start + per_page) + } + page <- lapply(page, function(w) { w$text <- NULL; w }) + jsonlite::toJSON(list(meta = list(count = length(hits), next_cursor = next_cursor), + results = page), + auto_unbox = TRUE, null = "null") +} + +fake_work <- function(id, doi = NULL, type = "article", text = character(), + cites = character(), title = paste("Work", id)) { + list(id = paste0("https://openalex.org/", id), + doi = if (!is.null(doi)) paste0("https://doi.org/", doi), + title = title, type = type, publication_year = 2020, + primary_location = list(landing_page_url = paste0("https://example.org/", id), + source = list(display_name = "Journal")), + referenced_works = as.list(paste0("https://openalex.org/", cites)), + text = as.list(text)) +} + +# Run `code` against a fake OpenAlex server with a fake clock, without any +# API key and with logs in a temporary directory. +with_fake_openalex <- function(server, code, env = parent.frame()) { + clock <- new.env() + clock$now <- as.POSIXct("2026-10-01 12:00:00", tz = "UTC") + clock$sleeps <- numeric() + testthat::local_mocked_bindings( + openalex_http = server$http, + openalex_now = function() clock$now, + openalex_sleep = function(s) { + clock$sleeps <- c(clock$sleeps, s) + clock$now <- clock$now + s + }, + scythe_get_key = function(source) NA, + # The search itself never classifies relations, so never calls DataCite. + datacite_fetch = function(url) stop("the OpenAlex search must not call DataCite"), + .package = "scythe", + .env = env + ) + withr::local_options(scythe.log_dir = withr::local_tempdir(.local_envir = env), .local_envir = env) + server$clock <- clock + suppressMessages(force(code)) +} diff --git a/tests/testthat/helper-test-functions.R b/tests/testthat/helper-test-functions.R index 4fb4b71..4ddc996 100644 --- a/tests/testthat/helper-test-functions.R +++ b/tests/testthat/helper-test-functions.R @@ -71,3 +71,22 @@ citation_test_doi <- function(source) { result } } + + +# Sources that need an API key (Springer, Scopus) are rate limited and +# metered, so their live tests never run on CRAN or CI, even where a key is +# available there. They run locally when the key is set. +on_ci <- function() isTRUE(as.logical(Sys.getenv("CI", "false"))) + +skip_unless_keyed_live <- function(source) { + testthat::skip_on_cran() + testthat::skip_on_ci() + testthat::skip_if(is.na(scythe_get_key(source)), + paste("no", source, "API key set")) +} + +# The keyed sources whose live tests may run here. +live_keyed_sources <- function() { + if (on_ci()) return(character()) + Filter(function(source) !is.na(scythe_get_key(source)), c("scopus", "springer")) +} diff --git a/tests/testthat/test-openalex.R b/tests/testthat/test-openalex.R new file mode 100644 index 0000000..f9f5f8d --- /dev/null +++ b/tests/testthat/test-openalex.R @@ -0,0 +1,358 @@ +test_that("normalize_doi handles DOI prefixes, schemes and encodings", { + variants_file <- system.file("testdata", "doi-variants.csv", package = "scythe") + variants <- read.csv(variants_file, stringsAsFactors = FALSE, na.strings = "") + expect_equal(normalize_doi(variants$input), variants$expected) +}) + +test_that("check_identifiers warns about identifiers that are not DOIs or uuids", { + expect_warning(check_identifiers("not-a-doi"), "does not appear") + expect_no_warning(check_identifiers(c("DOI:10.18739/A22274", "urn:uuid:1234"))) + expect_equal(check_identifiers(c("DOI: 10.18739/A22274", "urn:uuid:1234")), + c("10.18739/A22274", "1234")) +}) + +test_that("text search attributes each matching work to the DOI it mentions", { + dois <- sprintf("10.9999/d%03d", 1:40) + server <- fake_openalex(list( + fake_work("W1", "10.1/a", text = dois[3]), + fake_work("W2", "10.1/b", text = c(dois[3], dois[29])), + fake_work("W3", "10.1/c", text = "10.9999/unrelated") + )) + with_fake_openalex(server, { + res <- citation_search_openalex(dois, methods = "text") + pairs <- res[order(res$dataset_id, res$article_id), c("article_id", "dataset_id")] + expect_equal(pairs$article_id, c("10.1/a", "10.1/b", "10.1/b")) + expect_equal(pairs$dataset_id, dois[c(3, 3, 29)]) + expect_true(attr(res, "openalex_status")$complete) + # One screening search, then two searches per level of bisection for + # each of the two branches with matches (40 -> 20 -> 10 -> 5 -> 3 -> 2 -> 1). + expect_lte(length(server$urls), 1 + 2 * 2 * 6) + }) +}) + +test_that("text search sends a single search when no DOI is mentioned", { + server <- fake_openalex(list(fake_work("W1", "10.1/a", text = "10.9999/other"))) + with_fake_openalex(server, { + res <- citation_search_openalex(sprintf("10.9999/d%03d", 1:50), methods = "text") + expect_equal(nrow(res), 0) + expect_named(res, c("article_id", "article_title", "dataset_id", "source")) + expect_length(server$urls, 1) + expect_match(server$urls, "fulltext.search.exact", fixed = TRUE) + expect_match(server$urls, "include_xpac=true", fixed = TRUE) + }) +}) + +test_that("graph search joins citing works to the datasets they reference", { + server <- fake_openalex(list( + fake_work("D1", "10.18739/ds1", type = "dataset"), + fake_work("D2", "10.18739/ds2", type = "dataset"), + fake_work("A1", "10.1/a1", cites = c("D1", "X9")), + fake_work("A2", "10.1/a2", cites = c("D1", "D2")) + )) + with_fake_openalex(server, { + res <- citation_search_openalex(c("10.18739/DS1", "10.18739/DS2", "10.18739/missing"), + methods = "graph") + pairs <- res[order(res$dataset_id, res$article_id), c("article_id", "dataset_id")] + expect_equal(pairs$article_id, c("10.1/a1", "10.1/a2", "10.1/a2")) + expect_equal(pairs$dataset_id, c("10.18739/DS1", "10.18739/DS1", "10.18739/DS2")) + unresolved <- jsonlite::stream_in(file(file.path(attr(res, "openalex_status")$log_dir, + "openalex-unresolved.jsonl")), verbose = FALSE) + expect_equal(unresolved$identifier, "10.18739/missing") + }) +}) + +test_that("graph search follows pagination of citing works", { + citers <- lapply(1:450, function(i) fake_work(paste0("A", i), paste0("10.1/a", i), cites = "D1")) + server <- fake_openalex(c(list(fake_work("D1", "10.18739/ds1", type = "dataset")), citers)) + with_fake_openalex(server, { + res <- citation_search_openalex("10.18739/ds1", methods = "graph") + expect_equal(nrow(res), 450) + expect_length(server$urls, 1 + 3) + }) +}) + +test_that("matches are reported once, under each input form of the dataset DOI", { + server <- fake_openalex(list( + fake_work("D1", "10.18739/ds1", type = "dataset"), + fake_work("A1", "10.1/a1", cites = "D1", text = "10.18739/ds1") + )) + with_fake_openalex(server, { + res <- citation_search_openalex(c("doi:10.18739/DS1", "https://doi.org/10.18739/ds1")) + expect_equal(sort(res$dataset_id), sort(c("10.18739/DS1", "https://doi.org/10.18739/ds1"))) + matches <- jsonlite::stream_in(file(file.path(attr(res, "openalex_status")$log_dir, + "openalex-matches.jsonl")), verbose = FALSE) + expect_equal(matches$match_method, "both") + }) +}) + +test_that("datasets and works without DOIs go to review logs, not results", { + server <- fake_openalex(list( + fake_work("A1", "10.1/a1", text = "10.18739/ds1"), + fake_work("V2", "10.18739/ds1.v2", type = "dataset", text = "10.18739/ds1"), + fake_work("E1", "10.6073/pasta/abc", type = "article", text = "10.18739/ds1"), + fake_work("N1", NULL, type = "article", text = "10.18739/ds1"), + fake_work("S1", "10.18739/ds1", type = "dataset", text = "10.18739/ds1") + )) + with_fake_openalex(server, { + res <- citation_search_openalex("10.18739/ds1", methods = "text") + expect_equal(res$article_id, "10.1/a1") + + # Data relations are logged as OpenAlex found them, without classifying + # them (with_fake_openalex fails any call to DataCite). + relations <- read_log(res, "openalex-data-relations.jsonl") + relations <- relations[order(relations$work_doi), ] + expect_equal(relations$work_doi, c("10.18739/ds1.v2", "10.6073/pasta/abc")) + expect_equal(relations$match_method, c("text", "text")) + expect_false("relation_class" %in% names(relations)) + + no_doi <- read_log(res, "openalex-no-doi-citers.jsonl") + expect_equal(no_doi$work_id, "N1") + expect_equal(no_doi$dataset_doi, "10.18739/ds1") + }) +}) + +test_that("the cost check stops before any request is sent", { + server <- fake_openalex() + with_fake_openalex(server, { + expect_error(citation_search_openalex(sprintf("10.9999/d%04d", 1:5000), max_credits = 100), + "exceeds max_credits") + expect_length(server$urls, 0) + + res <- citation_search_openalex(sprintf("10.9999/d%04d", 1:5000), max_credits = Inf) + expect_true(attr(res, "openalex_status")$complete) + }) +}) + +test_that("without the preflight check, a search stops once max_credits is spent", { + server <- fake_openalex() + with_fake_openalex(server, { + expect_warning( + res <- citation_search_openalex(sprintf("10.9999/d%04d", 1:5000), methods = "text", + max_credits = 50, preflight = FALSE), + "Stopped after spending 50" + ) + expect_length(server$urls, 5) + status <- attr(res, "openalex_status") + expect_false(status$complete) + expect_equal(status$stopped, "max_credits") + }) +}) + +test_that("a response that can't be parsed still counts against max_credits", { + server <- fake_openalex(responses = rep(list(list(body = "{\"meta\": {")), 3)) + with_fake_openalex(server, { + res <- citation_search_openalex(sprintf("10.9999/d%04d", 1:300), methods = "text", + max_credits = 30, preflight = FALSE) + status <- attr(res, "openalex_status") + expect_length(server$urls, 3) + expect_equal(status$credits_used, 30) + expect_false(status$complete) + }) +}) + +test_that("a short rate limit is waited out, then requests resume one at a time", { + server <- fake_openalex( + works = list(fake_work("A1", "10.1/a1", text = "10.9999/d0001")), + responses = list(list(status = 429, headers = list(`retry-after` = "2"))) + ) + with_fake_openalex(server, { + res <- citation_search_openalex(sprintf("10.9999/d%04d", 1:300), methods = "text", + max_credits = Inf) + expect_equal(res$article_id, "10.1/a1") + expect_equal(server$clock$sleeps, 2) + expect_true(attr(res, "openalex_status")$complete) + }) +}) + +test_that("a long rate limit stops all requests and is shared through the state file", { + dir <- withr::local_tempdir() + server <- fake_openalex( + responses = list(list(status = 429, headers = list(`x-ratelimit-reset` = "5545"))) + ) + with_fake_openalex(server, { + withr::local_options(scythe.openalex.concurrency = 1) + expect_warning( + res <- citation_search_openalex(sprintf("10.9999/d%04d", 1:300), methods = "text", + checkpoint_dir = dir, max_credits = Inf), + "rate limited until" + ) + expect_length(server$urls, 1) + expect_length(server$clock$sleeps, 0) + status <- attr(res, "openalex_status") + expect_false(status$complete) + expect_equal(status$stopped, "rate_limited") + state <- jsonlite::read_json(file.path(dir, "openalex-state.json")) + expect_equal(state$paused_until, "2026-10-01T13:32:25Z") + + # A later run (or another process) sharing the state file sends nothing. + res <- citation_search_openalex("10.9999/d0001", methods = "text", checkpoint_dir = dir) + expect_length(server$urls, 1) + expect_equal(attr(res, "openalex_status")$stopped, "rate_limited") + + # Once the pause has passed, searches resume. + server$clock$now <- server$clock$now + 6000 + res <- citation_search_openalex("10.9999/d0001", methods = "text", checkpoint_dir = dir) + expect_length(server$urls, 2) + }) +}) + +test_that("updating the state file never shortens a recorded pause", { + path <- file.path(withr::local_tempdir(), "openalex-state.json") + update_openalex_state(path, paused_until = "2026-10-01T15:00:00Z") + update_openalex_state(path, paused_until = "2026-10-01T13:00:00Z") + expect_equal(read_openalex_state(path)$paused_until, "2026-10-01T15:00:00Z") + update_openalex_state(path, paused_until = "2026-10-01T16:00:00Z") + expect_equal(read_openalex_state(path)$paused_until, "2026-10-01T16:00:00Z") +}) + +test_that("a pause recorded by another process stops a running search", { + dir <- withr::local_tempdir() + state_file <- file.path(dir, "openalex-state.json") + server <- fake_openalex() + http <- server$http + # Another shard is rate limited while this search's first request is out. + server$http <- function(urls) { + update_openalex_state(state_file, paused_until = format_utc(server$clock$now + 3600)) + http(urls) + } + with_fake_openalex(server, { + withr::local_options(scythe.openalex.concurrency = 1) + expect_warning( + res <- citation_search_openalex(sprintf("10.9999/d%04d", 1:300), methods = "text", + checkpoint_dir = dir, max_credits = Inf), + "rate limited until" + ) + expect_length(server$urls, 1) + expect_equal(attr(res, "openalex_status")$stopped, "rate_limited") + }) +}) + +test_that("running out of daily budget stops requests before a 429", { + server <- fake_openalex(headers = list(`x-ratelimit-limit` = "1000", + `x-ratelimit-remaining` = "40", + `x-ratelimit-reset` = "3600")) + with_fake_openalex(server, { + withr::local_options(scythe.openalex.concurrency = 1) + expect_warning( + res <- citation_search_openalex(sprintf("10.9999/d%04d", 1:500), methods = "text", + max_credits = Inf), + "rate limited until" + ) + # The first response shows 40 credits left, below the 5% reserve of 50. + expect_length(server$urls, 1) + expect_false(attr(res, "openalex_status")$complete) + }) +}) + +test_that("repeated 429s double the pause", { + server <- fake_openalex(responses = rep(list(list(status = 429, headers = list(`retry-after` = "10"))), 4)) + with_fake_openalex(server, { + withr::local_options(scythe.openalex.concurrency = 1) + res <- citation_search_openalex("10.9999/d0001", methods = "text") + expect_equal(server$clock$sleeps, c(10, 10, 20, 40)) + expect_length(server$urls, 5) + expect_true(attr(res, "openalex_status")$complete) + }) +}) + +test_that("server errors are retried, then the batch is logged as failed", { + server <- fake_openalex(responses = rep(list(list(status = 500)), 4)) + with_fake_openalex(server, { + res <- citation_search_openalex("10.9999/d0001", methods = "text") + expect_length(server$urls, 4) + expect_false(attr(res, "openalex_status")$complete) + failed <- jsonlite::stream_in(file(file.path(attr(res, "openalex_status")$log_dir, + "openalex-failed-batches.jsonl")), verbose = FALSE) + expect_equal(failed$dois, "10.9999/d0001") + }) +}) + +test_that("a rerun with the same checkpoint directory skips completed batches", { + dir <- withr::local_tempdir() + server <- fake_openalex(list(fake_work("A1", "10.1/a1", text = "10.9999/d0001"), + fake_work("D1", "10.9999/d0099", type = "dataset", + text = "10.9999/d0002"))) + with_fake_openalex(server, { + dois <- sprintf("10.9999/d%04d", 1:150) + first <- citation_search_openalex(dois, checkpoint_dir = dir, max_credits = Inf) + sent <- length(server$urls) + second <- citation_search_openalex(dois, checkpoint_dir = dir, max_credits = Inf) + expect_length(server$urls, sent) + expect_equal(second, first, ignore_attr = TRUE) + # The rerun doesn't log the matches from the batches it skipped again. + expect_equal(nrow(read_log(second, "openalex-matches.jsonl")), 1) + expect_equal(nrow(read_log(second, "openalex-data-relations.jsonl")), 1) + }) +}) + +test_that("a resumed search logs only what it found itself", { + dir <- withr::local_tempdir() + dois <- sprintf("10.9999/d%04d", 1:150) + server <- fake_openalex(list(fake_work("A1", "10.1/a1", text = dois[1]), + fake_work("A2", "10.1/a2", text = dois[150]))) + with_fake_openalex(server, { + # Both batches are bisected level by level; the 50-DOI batch finishes a + # level before the 100-DOI one, at 240 credits. + first <- suppressWarnings(citation_search_openalex( + dois, methods = "text", checkpoint_dir = dir, max_credits = 240, preflight = FALSE + )) + expect_false(attr(first, "openalex_status")$complete) + expect_equal(first$article_id, "10.1/a2") + second <- citation_search_openalex(dois, methods = "text", checkpoint_dir = dir, + max_credits = Inf) + expect_true(attr(second, "openalex_status")$complete) + expect_setequal(second$article_id, c("10.1/a1", "10.1/a2")) + expect_equal(read_log(second, "openalex-matches.jsonl")$work_doi, c("10.1/a2", "10.1/a1")) + }) +}) + +test_that("since adds a publication date filter", { + server <- fake_openalex() + with_fake_openalex(server, { + citation_search_openalex("10.9999/d0001", since = as.Date("2025-10-01")) + text_search <- grepl("fulltext.search.exact", server$urls, fixed = TRUE) + expect_match(server$urls[text_search], "from_publication_date%3A2025-10-01", fixed = TRUE) + # Resolving dataset DOIs is not limited by date; only their citing works are. + expect_no_match(server$urls[!text_search], "from_publication_date", fixed = TRUE) + }) +}) + +test_that("openalex results combine with other sources in citation_search", { + server <- fake_openalex(list(fake_work("A1", "10.1/a1", text = "10.9999/d0001"))) + with_fake_openalex(server, { + res <- citation_search("10.9999/d0001", sources = "openalex") + expect_equal(res$article_id, "10.1/a1") + expect_true(all(vapply(res, is.character, TRUE))) + }) +}) + +# Live tests against the OpenAlex API. They spend credits, so they run only +# when a developer sets SCYTHE_TEST_OPENALEX=true, never on CRAN or CI, and +# each is capped at 100 credits. + +test_that("openalex finds known citations by text and by citation graph", { + skip_unless_openalex_live() + citations <- read.csv(system.file("testdata", "test-citations.csv", package = "scythe"), + stringsAsFactors = FALSE) + citations <- citations[citations$source == "openalex", ] + res <- citation_search_openalex(unique(citations$dataone_pid), log_dir = withr::local_tempdir(), + max_credits = 100) + skip_if_openalex_rate_limited(res) + expect_true(all(tolower(citations$pub_id) %in% res$article_id)) + + matches <- read_log(res, "openalex-matches.jsonl") + method <- matches$match_method[match(tolower(citations$pub_id), matches$work_doi)] + expect_true("text" %in% method) + expect_true("graph" %in% method) +}) + +test_that("openalex does not return known non-citations", { + skip_unless_openalex_live() + negatives <- read.csv(system.file("testdata", "openalex-negatives.csv", package = "scythe"), + stringsAsFactors = FALSE) + res <- citation_search_openalex(unique(negatives$dataset_id), log_dir = withr::local_tempdir(), + max_credits = 100) + skip_if_openalex_rate_limited(res) + found <- paste(res$article_id, normalize_doi(res$dataset_id)) + expect_false(any(paste(negatives$article_id, normalize_doi(negatives$dataset_id)) %in% found)) +}) diff --git a/tests/testthat/test-relations.R b/tests/testthat/test-relations.R new file mode 100644 index 0000000..385ed6d --- /dev/null +++ b/tests/testthat/test-relations.R @@ -0,0 +1,140 @@ +# Tests of classify_data_relations(). The offline tests replace DataCite with +# fakes; the live test, which sends a few DataCite requests, never runs on CRAN +# or CI. + +# Write `relations` as the data relations log of a search, in a temporary +# log directory, and return the directory. +local_relations_log <- function(relations, env = parent.frame()) { + log_dir <- withr::local_tempdir(.local_envir = env) + write_jsonl(relations, file.path(log_dir, "openalex-data-relations.jsonl")) + log_dir +} + +relation_pairs <- function(dataset_doi, work_doi, match_method = "graph") { + data.frame(dataset_doi = dataset_doi, work_doi = work_doi, match_method = match_method, + stringsAsFactors = FALSE) +} + +test_that("data relations are classified from DataCite relations in either direction", { + local_mocked_bindings(datacite_related = function(doi) { + switch(doi, + "10.18739/ds1.v2" = data.frame(doi = "10.18739/ds1", + relation_type = c("References", "IsNewVersionOf")), + "10.18739/ds3" = data.frame(doi = "10.18739/derived", relation_type = "IsSourceOf"), + NULL + ) + }) + log_dir <- local_relations_log(relation_pairs( + dataset_doi = c("10.18739/ds1", "10.18739/ds3", "10.18739/ds4"), + work_doi = c("10.18739/ds1.v2", "10.18739/derived", "10.6073/pasta/abc"), + match_method = c("text", "graph", "both") + )) + out <- classify_data_relations(log_dir) + expect_equal(out$relation_class, c("version", "derivation", "unknown")) + # A pair with several relations keeps them all and takes the most specific. + expect_equal(out$relation_type, c("References;IsNewVersionOf", "IsSourceOf", "openalex:cites")) + expect_equal(out$evidence, c("datacite:10.18739/ds1.v2", "datacite:10.18739/ds3", + "openalex:both")) + + classified <- jsonlite::stream_in( + file(file.path(log_dir, "openalex-data-relations-classified.jsonl")), verbose = FALSE + ) + expect_equal(classified$relation_class, out$relation_class) +}) + +test_that("each pair is classified once, from its latest log record", { + lookups <- character() + local_mocked_bindings(datacite_related = function(doi) { + lookups <<- c(lookups, doi) + NULL + }) + log_dir <- local_relations_log(relation_pairs( + dataset_doi = c("10.18739/ds1", "10.18739/ds1"), + work_doi = c("10.18739/ds2", "10.18739/ds2"), + match_method = c("text", "both") + )) + out <- classify_data_relations(log_dir) + expect_equal(nrow(out), 1) + expect_equal(out$match_method, "both") + expect_setequal(lookups, c("10.18739/ds1", "10.18739/ds2")) + expect_length(lookups, 2) + + # Running it again replaces the classified log rather than appending to it. + classify_data_relations(log_dir) + classified <- jsonlite::stream_in( + file(file.path(log_dir, "openalex-data-relations-classified.jsonl")), verbose = FALSE + ) + expect_equal(nrow(classified), 1) +}) + +test_that("without a data relations log, there is nothing to classify", { + local_mocked_bindings(datacite_related = function(doi) stop("no lookups expected")) + out <- classify_data_relations(withr::local_tempdir()) + expect_equal(nrow(out), 0) + expect_true(all(c("relation_class", "evidence") %in% names(out))) +}) + +test_that("DataCite lookups retry rate limits and report failures", { + record <- charToRaw(jsonlite::toJSON(list(data = list(attributes = list( + relatedIdentifiers = list(list(relatedIdentifier = "10.18739/DS1", + relationType = "IsNewVersionOf")) + ))), auto_unbox = TRUE)) + respond <- function(codes) { + calls <- 0 + function(url) { + calls <<- calls + 1 + code <- codes[min(calls, length(codes))] + list(status_code = code, headers = raw(0), + content = if (code %in% 200) record else raw(0)) + } + } + sleeps <- numeric() + local_mocked_bindings(openalex_sleep = function(s) sleeps <<- c(sleeps, s)) + + local_mocked_bindings(datacite_fetch = respond(c(429, 503, 200))) + related <- datacite_related("10.18739/ds1.v2") + expect_equal(related$doi, "10.18739/ds1") + expect_equal(related$relation_type, "IsNewVersionOf") + expect_equal(sleeps, c(2, 4)) + + local_mocked_bindings(datacite_fetch = respond(404)) + expect_null(datacite_related("10.1/crossref")) + + local_mocked_bindings(datacite_fetch = respond(NA_integer_)) + expect_identical(datacite_related("10.18739/ds1.v2"), NA) +}) + +test_that("relations whose DataCite lookup failed are marked for review", { + local_mocked_bindings(datacite_related = function(doi) { + if (doi == "10.18739/ds2") NA + }) + relations <- data.frame(dataset_doi = c("10.18739/ds1", "10.18739/ds2"), + work_doi = c("10.18739/other", "10.18739/other"), + match_method = "graph", stringsAsFactors = FALSE) + out <- suppressMessages(classify_relations(relations)) + expect_equal(out$relation_class, c("unknown", "unknown")) + expect_equal(out$evidence, c("openalex:graph", + "openalex:graph;datacite:lookup_failed:10.18739/ds2")) + expect_message(classify_relations(relations), "DataCite lookups failed for 1 DOIs") +}) + +test_that("relation types map to relation classes", { + expect_equal(unname(relation_classes[c("IsVersionOf", "IsDerivedFrom", "HasPart", "Cites")]), + c("version", "derivation", "composition", "citation")) +}) + +test_that("a known DataCite version relation is classified (live)", { + skip_on_cran() + skip_on_ci() + skip_if_offline("api.datacite.org") + # In DataCite, 10.18739/A2P26Q584 IsNewVersionOf 10.18739/A2GF0MZ7C. + log_dir <- local_relations_log(relation_pairs( + dataset_doi = c("10.18739/a2gf0mz7c", "10.18739/a2d50g069"), + work_doi = c("10.18739/a2p26q584", "10.18739/a2p26q584"), + match_method = "text" + )) + out <- classify_data_relations(log_dir) + expect_equal(out$relation_class, c("version", "unknown")) + expect_match(out$relation_type[1], "IsNewVersionOf") + expect_equal(out$evidence[1], "datacite:10.18739/a2p26q584") +}) diff --git a/tests/testthat/test-scopus.R b/tests/testthat/test-scopus.R index 385d562..6704974 100644 --- a/tests/testthat/test-scopus.R +++ b/tests/testthat/test-scopus.R @@ -1,14 +1,6 @@ test_that("scopus finds single known doi", { - skip_on_cran() - - key <- scythe_get_key("scopus") - - if (is.na(key)){ - skip() - message("No Scopus key set. Skipping.") - } - - + # Runs only locally with a Scopus API key; never on CRAN or CI. + skip_unless_keyed_live("scopus") # Pull single test doi citation from test-citations.csv test_cit <- get_test_doi("scopus") diff --git a/tests/testthat/test-scythe.R b/tests/testthat/test-scythe.R index 53f824d..01c45c6 100644 --- a/tests/testthat/test-scythe.R +++ b/tests/testthat/test-scythe.R @@ -3,15 +3,11 @@ test_that("scythe finds multiple dois", { # skip on CRAN skip_on_cran() - # Determine which keys have been set - keyed_sources <- - unlist(lapply(c("scopus", "springer"), function(x) { - if (!is.na(scythe_get_key(x))) { - return(x) - } - else - return(NULL) - })) + # Springer and Scopus are only included locally, when their keys are set; + # on CI they are rate limited, so this test uses only the keyless sources. + keyed_sources <- live_keyed_sources() + # OpenAlex is left out because it spends credits; test-openalex.R has its + # own opt-in live tests. keyed_sources <- c(keyed_sources, "plos", "xdd") # Load a set of example citations that should be found by the API diff --git a/tests/testthat/test-springer.R b/tests/testthat/test-springer.R index 67fcab7..33ea331 100644 --- a/tests/testthat/test-springer.R +++ b/tests/testthat/test-springer.R @@ -1,13 +1,6 @@ test_that("springer finds single known doi", { - skip_on_cran() - - key <- scythe_get_key("springer") - - if (is.na(key)){ - skip() - message("No Springer key set. Skipping.") - } - + # Runs only locally with a Springer API key; never on CRAN or CI. + skip_unless_keyed_live("springer") # Pull single test doi citation from test-citations.csv test_cit <- get_test_doi("springer") diff --git a/tools/README.md b/tools/README.md new file mode 100644 index 0000000..140463e --- /dev/null +++ b/tools/README.md @@ -0,0 +1,36 @@ +# Developer tools + +The scripts here aren't part of the package and aren't run by the tests or by CI. + +## Benchmarking the OpenAlex adapter + +`benchmark_openalex.R` measures how well `citation_search_openalex()` finds a gold set of known dataset citations: + +- **Recall.** Which gold citations it found. For each one it missed, the reason, found by looking the article up in OpenAlex: + - the article isn't in OpenAlex; + - the article has no full text, and its references don't resolve the dataset; + - the article has full text, but the DOI search didn't match it; + - the identifier can't be normalized. +- **Precision.** Each match that isn't in the gold set is checked against the citing work's Crossref reference list. Matches that Crossref doesn't corroborate go to `needs-review.csv`, for someone to check against the article's full text. +- **Coverage by method and source.** Recall broken down by repository DOI prefix, matches broken down by method (`text`, `graph`, `both`), and overlap with the sources in the gold set. + +### Before you run it + +A benchmark spends real OpenAlex credits. Full-text searches cost 10 credits each, and a gold set of a few thousand datasets can need tens of thousands of credits. So: + +1. Run it only after the package tests pass (`devtools::test()`, including the live OpenAlex tests, which run only with `SCYTHE_TEST_OPENALEX=true`). +2. Set an OpenAlex API key with `scythe_set_key("openalex", ...)`. The script won't run without one. +3. Run it from a machine with a good, stable connection, such as a cluster node, in a session that can run for hours (`screen` or `tmux`). +4. Approve the cost explicitly. The script prints its estimate and requires either `--approve `, `--approve unlimited`, or a "y" at the prompt in an interactive R session. It never starts on its own. + +If OpenAlex rate-limits the run, or the daily budget runs out, the script stops and keeps its checkpoints. Rerun the same command after the reset time it prints, and it continues where it stopped. + +### Gold set + +Use a CSV with `article_id` and `dataset_id` columns (DOIs, in any form), and optionally a `source` column. For example, export the confirmed citations from the DataONE metrics service with `get_metrics_citations()` in `helm/scripts/search.R`, renaming `source_id` to `article_id` and `target_id` to `dataset_id`. + +``` +Rscript tools/benchmark_openalex.R --gold gold.csv --out openalex-benchmark --approve 20000 +``` + +The results go in the output directory: `summary.txt`, `gold-recall.csv`, `precision-by-method.csv`, `needs-review.csv`, `overlap-by-source.csv`, and the adapter's review logs in `logs/`. diff --git a/tools/benchmark_openalex.R b/tools/benchmark_openalex.R new file mode 100644 index 0000000..1154267 --- /dev/null +++ b/tools/benchmark_openalex.R @@ -0,0 +1,170 @@ +#!/usr/bin/env Rscript +# Measure the recall and precision of citation_search_openalex() against a gold +# set of known dataset citations. +# +# This is a developer procedure, not a test: a gold set covers thousands of +# DOIs, and searching them spends real OpenAlex credits. Run it only after the +# package tests pass, with an OpenAlex API key, from a host with a good +# connection, and only with explicit approval of the cost (see tools/README.md). +# +# Rscript tools/benchmark_openalex.R --gold gold.csv --out benchmark \ +# --approve 20000 +# +# The gold CSV needs `article_id` and `dataset_id` columns (DOIs, in any +# form), and may have a `source` column naming the source that found each +# citation. + +suppressPackageStartupMessages({ + library(optparse) + library(scythe) +}) + +opts <- parse_args(OptionParser(option_list = list( + make_option("--gold", type = "character", help = "CSV of known citations (article_id, dataset_id[, source])"), + make_option("--out", type = "character", default = "openalex-benchmark", + help = "Output directory [default %default]"), + make_option("--approve", type = "character", default = "", + help = "Credits approved for this run, or 'unlimited' (required unless interactive)"), + make_option("--methods", type = "character", default = "text,graph", + help = "OpenAlex methods to benchmark [default %default]") +))) +if (is.null(opts$gold)) stop("--gold is required") + +normalize_doi <- scythe:::normalize_doi +`%||%` <- function(x, y) if (is.null(x)) y else x +gold <- read.csv(opts$gold, stringsAsFactors = FALSE) +gold$article_doi <- normalize_doi(gold$article_id) +gold$dataset_doi <- normalize_doi(gold$dataset_id) +dataset_dois <- unique(stats::na.omit(gold$dataset_doi)) +methods <- strsplit(opts$methods, ",", fixed = TRUE)[[1]] + +# The cost must be approved explicitly: on the command line, or at a prompt. +estimate <- scythe:::estimate_openalex_cost(length(dataset_dois), methods) +message("Benchmark: ", nrow(gold), " gold citations of ", length(dataset_dois), + " datasets; estimated cost about ", estimate, " OpenAlex credits.") +if (is.na(scythe_get_key("openalex"))) stop("Set an OpenAlex API key before running the benchmark.") +max_credits <- if (identical(opts$approve, "unlimited")) { + Inf +} else if (nzchar(opts$approve)) { + as.numeric(opts$approve) +} else if (interactive() && + tolower(readline(paste0("Spend up to ", estimate, " credits? [y/N] "))) == "y") { + estimate +} else { + stop("Not approved. Pass --approve with a number of credits, or 'unlimited'.") +} + +dir.create(opts$out, showWarnings = FALSE, recursive = TRUE) +checkpoint_dir <- file.path(opts$out, "checkpoints") +state_file <- file.path(checkpoint_dir, "openalex-state.json") +found <- citation_search_openalex( + dataset_dois, methods = methods, + log_dir = file.path(opts$out, "logs"), + checkpoint_dir = checkpoint_dir, state_file = state_file, + max_credits = max_credits, preflight = FALSE +) +status <- attr(found, "openalex_status") +# The diagnostics below need more OpenAlex requests, so don't send them while +# the budget is rate limited. +if (identical(status$stopped, "rate_limited")) { + stop("OpenAlex is rate limited until ", status$paused_until %||% "its reset time", + ". Rerun the same command after then to continue from the checkpoints.", call. = FALSE) +} +if (!isTRUE(status$complete)) { + warning("The OpenAlex search did not complete (", status$stopped %||% "failed batches", + "); rerun the same command to continue from the checkpoints. ", + "Recall below only covers the batches finished so far.") +} +# A search that found no articles writes no matches log. +matches_file <- file.path(opts$out, "logs", "openalex-matches.jsonl") +matches <- if (file.exists(matches_file)) { + jsonlite::stream_in(file(matches_file), verbose = FALSE) +} else { + data.frame(dataset_doi = character(), work_doi = character(), match_method = character()) +} +matches <- matches[!duplicated(paste(matches$dataset_doi, matches$work_doi)), ] +found_key <- paste(matches$work_doi, matches$dataset_doi) +gold_key <- paste(gold$article_doi, gold$dataset_doi) + +# Recall: which gold citations were found, and why the others were missed. +client <- scythe:::openalex_client(max_credits = max_credits - status$credits_used, + state_file = state_file) +lookup <- function(dois, select) { + dois <- unique(stats::na.omit(dois)) + batches <- scythe:::chunk(dois, 100) + run <- scythe:::openalex_run(client, lapply(batches, function(b) { + list(filter = paste0("doi:", paste(b, collapse = "|")), select = select, `per-page` = "200") + }), paginate = FALSE) + works <- unlist(run$results, recursive = FALSE) + stats::setNames(works, vapply(works, function(w) scythe:::openalex_doi(w$doi), "")) +} +gold$found <- gold_key %in% found_key +missed <- gold[!gold$found, ] +articles <- lookup(missed$article_doi, "id,doi,has_fulltext,referenced_works") +datasets <- lookup(missed$dataset_doi, "id,doi") +missed$reason <- vapply(seq_len(nrow(missed)), function(i) { + if (is.na(missed$article_doi[i]) || is.na(missed$dataset_doi[i])) { + return("identifier is not a DOI that scythe can normalize") + } + a <- articles[[missed$article_doi[i]]] + d <- datasets[[missed$dataset_doi[i]]] + if (is.null(a)) { + "article not in OpenAlex" + } else if (!isTRUE(a$has_fulltext)) { + "article has no full text in OpenAlex, and its references do not resolve the dataset" + } else if (!is.null(d) && d$id %in% unlist(a$referenced_works)) { + "article cites the dataset in OpenAlex but the search did not return it" + } else { + "article has full text but the exact DOI search did not match it" + } +}, "") +gold$reason <- NA_character_ +gold$reason[!gold$found] <- missed$reason +gold$prefix <- sub("/.*$", "", gold$dataset_doi) +recall <- aggregate(found ~ prefix, gold, function(x) c(n = length(x), recall = mean(x))) +utils::write.csv(gold, file.path(opts$out, "gold-recall.csv"), row.names = FALSE) + +# Precision: matches that are not in the gold set are checked against the +# citing work's Crossref reference list; the rest need a person to check them +# against the full text. +new <- matches[!found_key %in% gold_key, ] +corroborated <- vapply(seq_len(nrow(new)), function(i) { + # DataCite DOIs are not in Crossref, so a 404 here just means "unknown". + refs <- tryCatch(suppressWarnings(rcrossref::cr_works(dois = new$work_doi[i])$data$reference[[1]]), + error = function(e) NULL) + if (is.null(refs)) return(NA) + text <- tolower(paste(unlist(refs), collapse = " ")) + grepl(new$dataset_doi[i], text, fixed = TRUE) +}, NA) +new$crossref_corroborated <- corroborated +utils::write.csv(new[!(corroborated %in% TRUE), ], + file.path(opts$out, "needs-review.csv"), row.names = FALSE) +matches$status <- ifelse(found_key %in% gold_key, "in gold set", + ifelse(found_key %in% paste(new$work_doi, new$dataset_doi)[new$crossref_corroborated %in% TRUE], + "corroborated by Crossref", "unverified")) +precision <- as.data.frame.matrix(table(matches$match_method, matches$status)) +utils::write.csv(precision, file.path(opts$out, "precision-by-method.csv")) + +# Overlap with the other sources in the gold set. +if ("source" %in% names(gold)) { + overlap <- as.data.frame.matrix(table(gold$source, ifelse(gold$found, "found by OpenAlex", "missed"))) + utils::write.csv(overlap, file.path(opts$out, "overlap-by-source.csv")) +} + +summary_lines <- c( + paste0("OpenAlex benchmark, ", format(Sys.time(), "%Y-%m-%d %H:%M %Z")), + paste0("Search complete: ", isTRUE(status$complete), "; credits used: ", + status$credits_used + client$credits_used), + paste0("Recall: ", sum(gold$found), " of ", nrow(gold), " gold citations (", + round(100 * mean(gold$found), 1), "%)"), + "Misses by reason:", + paste0(" ", names(table(missed$reason)), ": ", table(missed$reason)), + paste0("Matches not in the gold set: ", nrow(new), " (", sum(corroborated %in% TRUE), + " corroborated by Crossref references; the rest are in needs-review.csv)"), + "Matches by method and status:", + utils::capture.output(print(precision)), + "Recall by repository prefix:", + utils::capture.output(print(recall)) +) +writeLines(summary_lines, file.path(opts$out, "summary.txt")) +writeLines(summary_lines) diff --git a/vignettes/openalex-arctic-data-center.Rmd b/vignettes/openalex-arctic-data-center.Rmd new file mode 100644 index 0000000..7779ec9 --- /dev/null +++ b/vignettes/openalex-arctic-data-center.Rmd @@ -0,0 +1,278 @@ +--- +title: "Searching OpenAlex for Arctic Data Center Citations" +output: rmarkdown::html_vignette +vignette: > + %\VignetteIndexEntry{Searching OpenAlex for Arctic Data Center Citations} + %\VignetteEngine{knitr::rmarkdown} + %\VignetteEncoding{UTF-8} +--- + +```{r setup, include = FALSE} +# The searches below call OpenAlex and spend credits, so like the live tests +# they only run when SCYTHE_TEST_OPENALEX=true, and never on CI. Otherwise the +# vignette shows the code without running it, which keeps R CMD check offline +# and free. +live <- isTRUE(as.logical(Sys.getenv("SCYTHE_TEST_OPENALEX", "false"))) && + !isTRUE(as.logical(Sys.getenv("CI", "false"))) +knitr::opts_chunk$set(collapse = TRUE, comment = "#>", eval = live) +``` + +This vignette uses `citation_search_openalex()` to find articles that cite +datasets in the [Arctic Data Center](https://arcticdata.io) (ADC), whose DOIs +all start with `10.18739`. It works through the search in stages, from +cheapest to most expensive, so that each stage checks the one before it +before you spend more of your OpenAlex budget. + +## How OpenAlex charges for searches + +OpenAlex charges credits for each request against a daily budget: 1,000 +credits a day without an API key, and more with one (typically 10,000 credits a day). +`citation_search_openalex()` has two search methods, and their costs are very different: + +- **`"graph"`** follows OpenAlex's citation graph, finding works whose + reference lists resolve to the dataset. It costs about 3 credits per batch + of 100 DOIs, so even the whole ADC costs a few hundred credits. The graph + search examines the citation relationships that OpenAlex has already resolved, + so it only finds citations that are already in the citation graph, but it is inexpensive. +- **`"text"`** runs an exact full-text search for each DOI, which also finds + DOIs mentioned outside reference lists, for example in data availability + statements. Screening costs 10 credits per batch of 100 DOIs, but each DOI + that matches costs about `20 * log2(batch size)` more credits to work out + which DOI in the batch matched. The cost is driven by how many datasets are + cited, not how many are searched. + +Before sending any request, `citation_search_openalex()` estimates the cost +and stops if it exceeds `max_credits`. Setting `max_credits = 0` therefore +gives a free estimate: the function prints the estimate and stops without +sending anything. Each stage below starts with one. + +## Setup + +Every search writes its review logs and checkpoints to `work_dir`. A +checkpoint records each completed batch, so rerunning a stage, on the same +day or later, skips the batches already done. Keep `batch_size` and `since` +the same between runs, or the checkpoints won't match. + +```{r work-dir, eval = TRUE} +library(scythe) + +work_dir <- if (live) path.expand("~/scythe-adc") else tempfile("scythe-adc") +dir.create(work_dir, showWarnings = FALSE, recursive = TRUE) +options(scythe.log_dir = work_dir) + +show_status <- function(res) str(attr(res, "openalex_status")[ + c("complete", "stopped", "paused_until", "estimate", "credits_used", "requests")]) +``` + +Set an OpenAlex API key for a larger daily budget, and a contact email to send +with each request. This only needs doing once, since the keys are kept in your +operating system's credential store. + +```{r keys, eval = FALSE} +scythe_set_key("openalex", "") +# Or use `scythe_set_key("openalex_mailto", "")` +``` + +You can check whether a key is set with: +```{r key-check, eval = FALSE} +!is.na(scythe_get_key("openalex")) # TRUE means a key is set +``` + +## Stage 1: Known citations + +The first stage searches for a few ADC citations already recorded in the +package's test data, to check that the search works end to end before +spending more. The test data lists citations found by several services, so +the `sources` column records which ones previously found each citation. Where a dataset +is cited in a form that isn't a DOI, such as a catalog URL, its `dataone_pid` +is used instead. + +```{r known, eval = TRUE} +known <- read.csv(system.file("testdata", "test-citations.csv", package = "scythe")) +known$dataset_doi <- scythe:::normalize_doi(known$dataset_id_as_cited) +known$dataset_doi <- ifelse(is.na(known$dataset_doi), + scythe:::normalize_doi(known$dataone_pid), known$dataset_doi) +known <- known[grepl("^10\\.18739/", known$dataset_doi), ] +known <- aggregate(source ~ pub_id + dataset_doi, known, + function(x) paste(sort(unique(x)), collapse = ", ")) +names(known)[names(known) == "source"] <- "sources" +known +``` + +Estimate the cost, then run both methods on these datasets: + +```{r known-search} +try(citation_search_openalex(unique(known$dataset_doi), max_credits = 0)) + +positives <- citation_search_openalex(unique(known$dataset_doi), + checkpoint_dir = work_dir, max_credits = 200) +show_status(positives) +``` + +Then check which known citations were found: + +```{r known-check} +pair <- function(article, dataset) { + paste(scythe:::normalize_doi(article), scythe:::normalize_doi(dataset)) +} +known$found <- pair(known$pub_id, known$dataset_doi) %in% + pair(positives$article_id, positives$dataset_id) +known +``` + +A citation that OpenAlex found before (`sources` includes `openalex`) should +be found again, so `FALSE` on one of those rows is a bug. `FALSE` on a row +found only by other services is a gap in OpenAlex's coverage instead. For +example, Radville et al. (2016) cites `10.18739/A23H1N` in its text, and +Springer's API finds it there. But that text is missing from OpenAlex's +full-text index, and the paper's reference list doesn't include the dataset, +so neither method can find it. These rows show what OpenAlex misses compared +with the other services. + +Any matches that aren't in the test data are new, and worth checking by hand: + +```{r known-new} +positives[!pair(positives$article_id, positives$dataset_id) %in% + pair(known$pub_id, known$dataset_doi), ] +``` + +## Stage 2: All current ADC DOIs + +The second stage collects the DOIs to search: every current ADC metadata +record in the [DataONE](https://www.dataone.org) search index, leaving out +versions that have been replaced by newer ones. There were about 6,000 in +October 2026. Saving the list keeps later runs searching the same DOIs, in +the same order, so that their checkpoints match. + +```{r adc-dois} +solr <- paste0("https://cn.dataone.org/cn/v2/query/solr/", + "?q=id:doi%5C:10.18739*+AND+formatType:METADATA+AND+-obsoletedBy:*", + "&fl=id&rows=20000&wt=csv") +adc_file <- file.path(work_dir, "adc-dois.rds") +if (file.exists(adc_file)) { + adc <- readRDS(adc_file) +} else { + adc <- sub("^doi:", "", read.csv(solr)$id) + saveRDS(adc, adc_file) +} +length(adc) +``` + +Delete `adc-dois.rds` to fetch a fresh list. + +## Stage 3: The citation graph for all of ADC + +Because graph searches are cheap, the third stage runs one over every ADC +DOI. It finds the citations that OpenAlex has linked from articles' reference +lists, and gives a baseline to compare full-text search with in stage 4. + +```{r graph} +try(citation_search_openalex(adc, methods = "graph", max_credits = 0)) + +graph <- citation_search_openalex(adc, methods = "graph", + checkpoint_dir = work_dir, max_credits = 500) +show_status(graph) +nrow(graph) # citing article and dataset pairs +length(unique(graph$dataset_id)) # ADC datasets with at least one citing article +``` + +## Stage 4: Full text on a random sample + +Full-text search can be expensive, so the fourth stage runs it on a random +sample of 200 DOIs first. The sample shows what a full run would cost, and +whether full-text search finds enough citations that the graph misses to be +worth that cost. + +```{r text} +set.seed(1) +sample_dois <- sample(adc, 200) + +try(citation_search_openalex(sample_dois, methods = "text", max_credits = 0)) + +text <- citation_search_openalex(sample_dois, methods = "text", + checkpoint_dir = work_dir, max_credits = 1500) +show_status(text) +``` + +The estimate assumes that 5% of the DOIs are cited. If more ADC datasets are +cited than that, full-text search costs more than estimated. Compare the +estimate with what was spent, and with the hit rate in the sample: + +```{r text-calibrate} +s <- attr(text, "openalex_status") +c(estimate = s$estimate, spent = s$credits_used, + hit_rate = length(unique(text$dataset_id)) / length(sample_dois)) +``` + +Then compare the full-text matches with the graph matches for the same +sample: + +```{r text-vs-graph} +graph_sample <- graph[scythe:::normalize_doi(graph$dataset_id) %in% + scythe:::normalize_doi(sample_dois), ] +text_pairs <- pair(text$article_id, text$dataset_id) +graph_pairs <- pair(graph_sample$article_id, graph_sample$dataset_id) +c(text_only = sum(!text_pairs %in% graph_pairs), + graph_only = sum(!graph_pairs %in% text_pairs), + both = sum(text_pairs %in% graph_pairs)) +``` + +`text_only` counts the citations that only full-text search found. Check a +few of them by hand, since full-text matches can include false positives: + +```{r text-only} +text[!text_pairs %in% graph_pairs, ] +``` + +## Stage 5: The full search, in daily installments + +The last stage runs both methods over every ADC DOI. Its estimated cost is far +more than a single day's budget, so it runs in installments: it spends credits +until the day's budget is down to its reserve (5% by default; see the +`scythe.openalex.reserve` option) or OpenAlex returns a rate limit error, and +then returns the results found so far. + +Run this chunk again each day, with the same `work_dir`, until `complete` is +`TRUE`. The checkpoints skip the batches already done. The file +`openalex-state.json` in `work_dir` records when the rate limit ends, so that +no R session using the same directory sends requests before then. + +```{r full, eval = FALSE} +full <- citation_search_openalex(adc, checkpoint_dir = work_dir, + max_credits = Inf, preflight = FALSE) +show_status(full) +saveRDS(full, file.path(work_dir, paste0("full-", Sys.Date(), ".rds"))) +``` + +## Reviewing the results + +Besides the citations it returns, each search writes JSON Lines files to +`work_dir` for review. `openalex-matches.jsonl` lists every article match and +which method found it. `openalex-data-relations.jsonl` lists datasets that +cite or mention other datasets. Spot-check both for false positives. +`classify_data_relations(work_dir)` can also label each data relation, using +DataCite, as a version, derivation, composition or citation link; that isn't +needed for the citation counts above. + +```{r review} +list.files(work_dir, pattern = "\\.jsonl$") +matches <- jsonlite::stream_in(file(file.path(work_dir, "openalex-matches.jsonl")), + verbose = FALSE) +head(matches) +``` + +## Running this vignette + +Building the package doesn't run the searches in this vignette, which only +shows their code. To run stages 1 to 4 and see their output, set the +`SCYTHE_TEST_OPENALEX` environment variable and build the vignette from the +package source directory: + +```{r render, eval = FALSE} +Sys.setenv(SCYTHE_TEST_OPENALEX = "true") +rmarkdown::render("vignettes/openalex-arctic-data-center.Rmd") +``` + +When it runs live, the vignette keeps its logs and checkpoints in +`~/scythe-adc`. Stage 5 never runs when the vignette is built. Run it from the +console.