Skip to content

Fix University of Twente scraper (0 events ingested) - #1379

Open
ioan-alexandra wants to merge 1 commit into
ElixirTeSS:masterfrom
ioan-alexandra:fix/utwente-scraper-neopage-api
Open

ioan-alexandra wants to merge 1 commit into
ElixirTeSS:masterfrom
ioan-alexandra:fix/utwente-scraper-neopage-api

Conversation

@ioan-alexandra

@ioan-alexandra ioan-alexandra commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Summary

  • UT rebuilt their events page as a fully client-rendered app at a new URL (/en/about-us/news-events-ceremonies/events/, redirected from the old /en/events/).
  • By inspecting the site's JS bundle, found the actual WebHare JSON-RPC endpoint the page itself calls to load events: POST /wh_services/utwente_base/rpc/GetNEOItems, paginated via a skip parameter.
  • Rewrote UtwenteIngestor to call this endpoint directly, with proper pagination (capped at MAX_PAGES as a safety net) instead of a single hardcoded request assuming one page.

Verification

  • Called the discovered endpoint directly against the live site - confirmed it returns real, current UT events with pagination (moreitems/skip).
  • Rewrote the VCR cassette and test to cover a 2-page pagination scenario.

@ioan-alexandra
ioan-alexandra force-pushed the fix/utwente-scraper-neopage-api branch from 2816809 to 24cba8e Compare September 16, 2026 12:05
@ioan-alexandra
ioan-alexandra force-pushed the fix/utwente-scraper-neopage-api branch from 24cba8e to 23ae89c Compare September 16, 2026 12:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant