Add OpenAlex as a citation source

.Rbuildignore

@@ -1,11 +1,17 @@
 ^scythe\.Rproj$
 ^\.Rproj\.user$
 ^LICENSE\.md$
-^\.travis\.yml$
 ^codecov\.yml$
 ^CONTRIBUTING\.md$
 ^api-scopus-search.sh$
 ^\.github$
 ^logo$
 ^data$
-^results$
+^results$
+^tools$
+^helm$
+^TODO\.md$
+^openalex-duckdb$
+^dev$
+^doc$
+^Meta$

.github/workflows/R-CMD-check.yaml

@@ -20,6 +20,8 @@ jobs:
     steps:
       - uses: actions/checkout@v2
       - uses: r-lib/actions/setup-r@v2
+      # Pandoc is needed to build the R Markdown vignettes.
+      - uses: r-lib/actions/setup-pandoc@v2
       - name: Install dependencies
         run: |
           install.packages(c("remotes", "rcmdcheck"))

.gitignore

@@ -13,3 +13,8 @@ key.txt
 data-raw
 # Ignore Vim's swap files
 .*.swp
+
+# Built vignettes
+inst/doc
+/doc/
+/Meta/

CONTRIBUTING.md

@@ -71,7 +71,8 @@ particular release.
 
 The release process starts with integration testing in the `develop` branch. Once all
 changes that are desired in a release are merged into the `develop` branch, we run
-the full set of tests on a clean checkout, and conduct any code reviews needed. A pull request
+the full set of tests on a clean checkout, including the live network tests that CI
+skips (see [Testing](#testing)), and conduct any code reviews needed. A pull request
 can be used for that review, or it can be done directly for simpler sets of changes.
 Once review is completed, make sure all version numbers are set correctly and 
 documentation is complete on `develop`, and then merge to `main`, and tag that merge 
@@ -90,7 +91,54 @@ as well as various classes of corrupt or bad data.  All tests should pass before
 the `develop` branch is merged to main, and all tests should pass before the `main`
 branch is tagged as a release.
 
-**Continuous integration**. We can use Travis for some of our repositories.
+**Continuous integration**. GitHub Actions runs `R CMD check`, including the
+tests, on every push and pull request to `main` and `develop`. CI deliberately
+skips the live tests of the sources that are metered or rate limited (Springer,
+Scopus, OpenAlex and DataCite). Those limits apply to a shared key or IP
+address, so these tests would fail for reasons that have nothing to do with the
+code, such as a Springer request rejected for quota, or an exhausted OpenAlex
+daily budget. They are also skipped on CRAN. So **a passing CI run does not
+mean the live searches work**; run the live tests below before each release.
+
+**Live network tests**. These run against the real services. They are
+skipped on CRAN and on CI (any system that sets `CI=true`), and otherwise run
+only when their keys or switches are set:
+
+| Tests | Runs locally when |
+|---|---|
+| PLOS, xDD | always (they need no key; they also run on CI) |
+| Springer, Scopus | the `springer` or `scopus` key is set |
+| OpenAlex, and the searches in the OpenAlex vignette | `SCYTHE_TEST_OPENALEX=true` is set; uses up to 100 OpenAlex credits per test, plus more when building the vignette |
+| DataCite relation classification | always (a few free requests) |
+
+To run the full live suite before a release, from a clean checkout of
+`develop`, with a good network connection and no `CI` variable set:
+
+1. Set the API keys. `scythe_set_key()` stores them in the `scythe` keyring;
+   environment variables with the same names (`springer`, `scopus`,
+   `openalex`, `openalex_mailto`) also work and take precedence:
+
+   ```r
+   scythe_set_key(source = "springer", secret = "...")
+   scythe_set_key(source = "scopus", secret = "...")
+   scythe_set_key(source = "openalex", secret = "...")
+   scythe_set_key(source = "openalex_mailto", secret = "you@example.org")
+   ```
+
+2. Opt in to the tests that spend OpenAlex credits, and run every test:
+
+   ```r
+   Sys.setenv(SCYTHE_TEST_OPENALEX = "true")
+   devtools::test()
+   ```
+
+3. Check the test summary for skips. Any live test reported as skipped did not
+   run, usually because a key is missing or OpenAlex is rate limited; fix
+   that and rerun it before releasing. (An OpenAlex test that skips because
+   of a rate limit can be rerun after the time it reports.)
+
+4. Optionally, build the OpenAlex vignette with its searches, as described in
+   its "Running this vignette" section; this spends more OpenAlex credits.
 
 ## Code style
 

DESCRIPTION

@@ -1,13 +1,13 @@
 Package: scythe
-Title: Harvest and register data package citations 
+Title: Harvest and Register Data Package Citations 
 Version: 1.1.0
 Authors@R: c(
     person("Jeanette", "Clark", role = c("aut", "cre"), email = "jclark@nceas.ucsb.edu", comment=c(ORCID = "0000-0003-4703-1974")),
     person("Matthew B.", "Jones", role = "aut", email = "jones@nceas.ucsb.edu", comment=c(ORCID = "0000-0003-0077-4738")),
     person("Maya", "Samet", role = "aut", email = "samet@nceas.ucsb.edu", comment=c(ORCID = "0000-0002-5248-9712")),
     person("Althea", "Marks", role = "aut", email = "marks@nceas.ucsb.edu", comment=c(ORCID = "0000-0002-9370-9128"))
     )
-Description: Harvests data package citations from several API sources, including PLOS, Scopus, and Springer. This package uses modified functions from `rplos`, which is no longer maintained.
+Description: Harvests data package citations from several API sources, including 'PLOS', 'Scopus', 'Springer', 'xDD', and 'OpenAlex'. This package uses modified functions from 'rplos', which is no longer maintained.
 License: Apache License (>= 2.0)
 Encoding: UTF-8
 LazyData: true
@@ -23,7 +23,11 @@ Remotes: ropensci/bib2df@a8e96e13f5
 Suggests:
     bib2df,
     covr,
+    knitr,
     purrr,
-    testthat (>= 3.0.0)
-RoxygenNote: 7.3.1
+    rmarkdown,
+    testthat (>= 3.0.0),
+    withr
+VignetteBuilder: knitr
 Config/testthat/edition: 3
+Config/roxygen2/version: 8.1.0

NAMESPACE

@@ -1,10 +1,12 @@
 # Generated by roxygen2: do not edit by hand
 
 export(citation_search)
+export(citation_search_openalex)
 export(citation_search_plos)
 export(citation_search_scopus)
 export(citation_search_springer)
 export(citation_search_xdd)
+export(classify_data_relations)
 export(scythe_get_key)
 export(scythe_set_key)
 export(write_citation_pairs)

R/citation_search.R

@@ -2,7 +2,10 @@
 #'
 #' @param identifiers a vector of identifiers to be searched for
 #' @param sources a vector indicating which sources to query (one or
-#' more of plos, scopus, springer)
+#' more of plos, scopus, springer, xdd, openalex). OpenAlex spends credits
+#' from a daily budget, so it is only searched when named here; call
+#' [citation_search_openalex()] directly to control its cost and to resume
+#' an interrupted search.
 #' @return tibble of matching dataset and publication identifiers
 #'
 #' @export
@@ -15,7 +18,7 @@ citation_search <- function(identifiers,
                             sources = c("plos", "scopus", "springer", "xdd")) {
   stopifnot(is.character(identifiers))
 
-  unid <- sources[!(sources %in% c("plos", "scopus", "springer", "xdd"))]
+  unid <- sources[!(sources %in% scythe_sources)]
   if (length(unid) > 0) {
     stop(paste("Source", unid, "is not recognized.", collapse = ". "))
   }
@@ -43,20 +46,23 @@ citation_search <- function(identifiers,
   return(result)
 }
 
+# The sources that citation_search() can query; each one has a matching
+# citation_search_<source>() adapter.
+scythe_sources <- c("plos", "scopus", "springer", "xdd", "openalex")
+
 # Check identifiers to remove characters that interfere with query strings
 
 check_identifiers <- function(identifiers) {
-  if (any(!grepl("10\\.|urn:uuid", identifiers))) {
+  is_uuid <- grepl("^urn:uuid:", identifiers, ignore.case = TRUE)
+  if (any(is.na(normalize_doi(identifiers)) & !is_uuid)) {
     warning(
       call. = FALSE,
       "One or more identifiers does not appear to be a DOI or uuid",
       immediate. = TRUE
     )
   }
 
-  if (any(grepl("doi:|urn:uuid", identifiers))) {
-    identifiers <- gsub("(doi:)|(urn:uuid:)", "", identifiers)
-  }
+  identifiers <- gsub("(doi:\\s*)|(urn:uuid:)", "", identifiers, ignore.case = TRUE)
 
 
   return(identifiers)