[mbjones]

Closes #44.

Adds OpenAlex as a citation source, through a new exported citation_search_openalex() and as "openalex" in citation_search() and the helm cron job. OpenAlex doesn't fit the existing adapters. Rather than one keyword query per DOI, it has two search modes with very different costs, and it charges credits against a daily budget. So most of this PR deals with spending those credits deliberately and resuming work across days.

Approach

Two search methods (R/openalex_search.R)

• graph: resolves each batch of up to 100 DOIs to OpenAlex work IDs, then fetches the works that cite any of them (cites: filter) and uses each citing work's referenced_works to tell which datasets it cites. This costs about 3 credits per 100 DOIs, but only finds datasets that OpenAlex indexes and that are linked from parsed reference lists.
• text: an exact full-text search (fulltext.search.exact) for the DOI string, which also finds DOIs in data availability statements, methods and acknowledgements. A search costs 10 credits, so it uses adaptive group testing: each batch is screened with one OR'd query that only counts matches. A batch with matches is split in half and searched again, down to single DOIs, and only the single-DOI searches fetch the works themselves. Most batches have no matches, so the cost grows with the number of datasets that are cited, roughly 20 × log2(batch size) per cited DOI, not with the number searched.

The methods find largely different citations. On a random sample of 200 Arctic Data Center (ADC) DOIs, 11 citations were found only by text, 30 only by graph, and 7 by both. Matches found by both methods are merged and labelled match_method = "both".

Results versus review logs (R/relations.R)

The returned data.frame keeps the existing article_id / article_title / dataset_id / source shape, so it combines with the other sources in citation_search(). It only includes citing works that have a DOI and aren't datasets. Everything else is appended as JSON Lines to log_dir, for review rather than dropped:

• openalex-matches.jsonl: every article match, with its method and the work's metadata.
• openalex-data-relations.jsonl: data-to-data links. A citing work counts as data if its type is dataset, or if its DOI prefix belongs to a DataONE member repository (scythe.repository_prefixes), since OpenAlex often types repository records as article. These links are classified as version, derivation, composition or citation from DataCite relatedIdentifiers, checked from both sides; links with no DataCite relation stay unknown.
• openalex-no-doi-citers.jsonl, openalex-unresolved.jsonl, openalex-failed-batches.jsonl.

Spending credits safely (R/openalex_client.R)

• Cost check before any request. estimate_openalex_cost() estimates the cost of a search, and the search stops before sending anything if the estimate exceeds max_credits, which defaults to 500. max_credits = 0 therefore gives a free estimate. With preflight = FALSE it runs until max_credits is spent instead.
• A hard cap while running. Each group of concurrent requests (4 by default) is cut so that it never goes past max_credits, and stops early when OpenAlex's reported remaining budget is down to a reserve (scythe.openalex.reserve, 5% by default).
• Rate limits. A short 429 is waited out, then requests resume one at a time. Repeated 429s double the pause. A pause longer than max_wait, or an exhausted daily budget, stops all requests and returns partial results. The time the pause ends is written to a shared state_file, so other processes using the same key don't send requests before then. Server errors are retried with backoff, then the batch is logged as failed.
• Status. The result carries an openalex_status attribute: complete, stopped, paused_until, estimate, credits_used and requests.

Resuming across runs

Completed batches are appended to openalex-checkpoint.jsonl in checkpoint_dir, keyed by method, since and the batch's DOIs. A rerun with the same directory skips them, so a large search can be run in daily installments until complete is TRUE.

Helm job (helm/scripts/search.R, chart)

• --since and --lookback: after the first complete run, which is a full backfill, each run only searches works published since the last complete run minus lookback days. An interrupted run resumes with the date filter it started with.
• --shard i/n: splits the DOIs between jobs by a stable hash of each DOI.
• Spending is approved with cronjob.openalex.approve (a number of credits, or unlimited). The cron job is concurrencyPolicy: Forbid, so runs don't compete for the budget, and the OpenAlex secrets are optional.

Supporting changes

• normalize_doi() (R/helpers.R) handles resolver URLs, doi:, info:doi/ and urn:doi: schemes, stacked prefixes, URL encoding, surrounding punctuation, and parentheses inside DOIs. check_identifiers() now uses it.
• scythe_set_key() accepts openalex and openalex_mailto, and builds its error message from the list of valid sources.
• Docs: README section, a vignette (vignettes/openalex-arctic-data-center.Rmd) that walks through a staged search of ADC DOIs, and tools/benchmark_openalex.R for measuring precision and recall against a gold set. The vignette's searches only run with SCYTHE_OPENALEX_LIVE=true, so R CMD check and CI stay offline and free.

Testing

• tests/testthat/helper-openalex.R is a fake OpenAlex API that answers the fulltext.search.exact, doi and cites filters, and can return queued 429s, server errors and rate-limit headers. The offline tests cover attributing text matches to single DOIs, graph joins and pagination, routing results to the logs, the cost check and max_credits cap, rate limits shared through the state file, retries, checkpoint resume, and since.
• Two live tests check known citations and known non-citations from inst/testdata.
• devtools::check() passes locally. The ADC vignette stages were also run by hand against the live API.

Worth a close look

• Live tests fail when the budget is spent. The live OpenAlex tests fail, rather than skip, once the daily budget is down to its reserve, because they then get partial results. That's likely in CI on a shared key. They should probably skip when openalex_status$stopped is "rate_limited".
• OpenAlex coverage gaps are expected. For example, Radville et al. 2016 cites 10.18739/A23H1N, which Springer's API finds, but the DOI isn't in OpenAlex's full-text index and the paper's reference list doesn't link it. OpenAlex complements the other sources rather than replacing them.
• Some identifiers aren't recognized. normalize_doi() doesn't extract DOIs from catalog URLs such as https://arcticdata.io/catalog/#view/doi:10.18739/A29K97; they're logged as "not a DOI".
• DataCite lookups aren't throttled. Data-relation classification makes one DataCite request per distinct DOI in a data relation. These are cached within a run but not throttled. It can be turned off with options(scythe.classify_relations = FALSE).
• Estimates depend on the hit rate. Text-search estimates assume 5% of the DOIs are cited, and real costs scale with the actual hit rate. The vignette shows how to calibrate on a sample first.
• Changing batch_size or since invalidates checkpoints. Batch keys are built from the DOIs in each batch, so changing either one between runs means none of the earlier checkpoints are reused.

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

Credit accounting, concurrent state handling, resumable logging, and rate-limit-aware tests contain correctness and operational issues.

9 open findings

• Lock shard state updates and preserve the longest pause · New
• Count OpenAlex requests before parsing responses · New
• Keep OpenAlex opt-in to avoid unexpected credit spending · New
• Exclude cached pairs from repeat logging and classification · New
• Throttle and distinguish DataCite relation lookup failures · New
• Skip live benchmark assertions when OpenAlex is rate-limited · New
• Keep the broad live test free of OpenAlex budget usage · New
• Handle missing match logs as empty results · New
• Honor the OpenAlex circuit breaker before diagnostics · New

What changed in this PR

Adds OpenAlex citation discovery with resumable, credit-aware text and graph searches, review logging, Helm scheduling, documentation, and tests.

Changes:

• Implements OpenAlex searching, rate limiting, checkpointing, and relation classification.
• Integrates OpenAlex with package APIs and the Helm cron job.
• Adds offline/live tests, fixtures, benchmarking tools, and documentation.

File	Description
.Rbuildignore	Excludes development artifacts.
.gitignore	Ignores generated vignette files.
DESCRIPTION	Adds vignette dependencies and metadata.
NAMESPACE	Exports the OpenAlex API.
README.md	Documents OpenAlex usage and costs.
R/​citation_search.R	Registers OpenAlex as a source.
R/​citation_search_openalex.R	Orchestrates searches, checkpoints, logs, and results.
R/​helpers.R	Adds DOI normalization and chunking helpers.
R/​openalex_client.R	Implements HTTP, budgeting, retries, and rate limiting.
R/​openalex_search.R	Implements text and graph searches.
R/​relations.R	Routes and classifies citation relations.
R/​scythe_set_key.R	Supports OpenAlex credentials.
helm/​scripts/​search.R	Adds resumable scheduled searches.
helm/​templates/​cron-job.yaml	Configures OpenAlex secrets and persistence.
helm/​values.yaml	Enables OpenAlex cron settings.
inst/​testdata/​doi-variants.csv	Adds DOI normalization fixtures.
inst/​testdata/​openalex-negatives.csv	Adds negative citation fixtures.
inst/​testdata/​test-citations-mbj.csv	Adds benchmark citation data.
inst/​testdata/​test-citations.csv	Adds known OpenAlex citations.
man/​citation_search.Rd	Updates source documentation.
man/​citation_search_openalex.Rd	Documents the new API.
man/​report_est_wait.Rd	Removes obsolete generated documentation.
man/​scythe_set_key.Rd	Documents OpenAlex credential sources.
tests/​testthat/​helper-openalex.R	Adds a fake OpenAlex service.
tests/​testthat/​test-openalex.R	Tests searching, limits, retries, and checkpoints.
tests/​testthat/​test-scythe.R	Extends aggregate source testing.
tools/​README.md	Documents benchmarking workflow.
tools/​benchmark_openalex.R	Adds recall and precision benchmarking.
vignettes/​openalex-arctic-data-center.Rmd	Demonstrates staged ADC searches.

Files not reviewed (3)

• man/citation_search.Rd: Generated file
• man/citation_search_openalex.Rd: Generated file
• man/scythe_set_key.Rd: Generated file

🧠 Review effort: Balanced

---

Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

[mbjones]

Responses to the Copilot review, addressed in 8e04871 and 1a96352:

Finding	Response
OpenAlex in the default citation_search() sources	Fixed. OpenAlex is opt-in (sources = "openalex"). Before this, 100 DOIs would hit the 500-credit preflight check and error after the other four sources had already run.
Cached pairs re-logged and re-classified on resume	Fixed. Each run logs only the pairs it found itself. The returned results still include cached pairs.
Credits not counted for unparseable 200 responses	Fixed. Credits are counted before parsing.
Live test asserts on partial results when rate limited	Fixed. Live OpenAlex tests now skip on CRAN and CI, and run locally only with SCYTHE_TEST_OPENALEX=true. They're capped at 100 credits each and skip if rate limited.
Broad live test uses OpenAlex credits	Fixed. OpenAlex was removed from test-scythe.R.
Benchmark errors with no matches log	Fixed.
Benchmark diagnostics bypass the circuit breaker	Fixed. The benchmark stops if the search was rate limited, and the diagnostic client shares the search's state file.
Unlocked shard state updates	Partly fixed. Writes never shorten a recorded pause, and clients re-read the state file before each wave, so a running shard honours another's pause. No file locking: shards are opt-in, and the state file holds only paused_until.
Unthrottled DataCite lookups	Fixed differently. The search no longer calls DataCite. Classification is now a separate classify_data_relations(log_dir), run after a search. It isn't needed for citation counts. Lookups retry 429s and 5xx errors, and failed ones are marked datacite:lookup_failed. Its live test skips on CRAN and CI.

🤖 Generated with Claude Code

[mbjones]

Closing this PR, as its functionality was fully merged into PR #47 and so it will be reviewed as a whole there.