Update Deno documentation scraper by srpatcha · Pull Request #2681 · freeCodeCamp/devdocs

srpatcha · 2026-04-25T02:42:40Z

No description provided.

Ensure consistent line endings and proper diff handling for text and binary files.

Add a new UrlScraper for Deno standard library and runtime documentation (lib/docs/scrapers/deno.rb) with: - Deno docs site scraping (docs.deno.com) - Page parsing with Nokogiri (main/article content extraction) - Link resolution (relative to absolute URL conversion) - Version handling with semver normalization (v2 and v1 support) - Module categorization (Web APIs, I/O, File System, Network, etc.) - Code example extraction with language detection - HTML filter pipeline (clean_html and entries filters) Also includes: - Minitest test class for scraper configuration validation - Bug fix: replace File.open(path).read with File.read(path) in sprites.thor to prevent unclosed file handle leak Signed-off-by: Srikanth Patchava <spatchava@meta.com>

simon04 · 2026-07-19T11:19:06Z

The scraper needs fine tuning:

srpatcha · 2026-07-20T03:58:29Z

Thanks for the review and the screenshots @simon04! You're right that the output needs refinement.

From the screenshots I can see a few issues to address:

Residual navigation/UI chrome that isn't being stripped by the current clean_html selectors
Code block language detection could be more accurate

I'll iterate on the clean_html.rb and entries.rb filters to clean up the rendering. Could you confirm which Deno docs section/URL the screenshots were captured from so I can target the exact page structure when tuning the selectors? That will help me verify the fixes against the same pages you're seeing.

simon04

.

Remove source-site navigation and controls that leaked into generated API and runtime pages, while preserving the source language metadata DevDocs uses for syntax highlighting. Add representative regression fixtures for the reviewer-reported page shapes. Test Plan: - Ran the focused Deno suite: 17 runs, 73 assertions, 0 failures. - Ran the remaining Ruby suite excluding two independently failing Windows-only baseline files: 665 runs, 927 assertions, 0 failures. - Compared old and new filters on current Network and @std/fmt source pages; all measured chrome counts dropped to zero while TypeScript, JavaScript, and shell language metadata was preserved. - Ran Ruby syntax checks and git diff --check.

srpatcha · 2026-09-08T22:33:46Z

@simon04 I pushed d17aafb to address the requested scraper fine-tuning without waiting for the missing source URL. I matched the screenshots to the current pages and validated the previous vs updated filter on both:

https://docs.deno.com/api/deno/network/: breadcrumbs 1 -> 0, symbol-kind badges 64 -> 0, code-copy buttons 23 -> 0, feedback sections 1 -> 0; all 23 TypeScript blocks still emit pre[data-language="ts"].
https://docs.deno.com/runtime/reference/std/fmt/: copy-page controls 1 -> 0, code-copy buttons 2 -> 0, mobile “On this page” blocks 1 -> 0, heading anchors 1 -> 0, feedback sections 1 -> 0; JavaScript and shell blocks remain js and sh.

The filter now scopes output to main#content article, removes the confirmed source-site chrome, and transfers each live language-* class to the enclosing pre[data-language] instead of defaulting every block to TypeScript. I also added reduced API/runtime fixtures and filter/entry regressions.

Validation:

Focused Deno tests: 17 runs, 73 assertions, 0 failures/errors.
Remaining Ruby suite excluding two independently reproduced Windows-only baseline files: 665 runs, 927 assertions, 0 failures/errors.
Full suite reached 761 runs; only the existing Windows case-sensitivity URL assertion and FileStore test/tmp path errors failed when run independently.
Ruby syntax checks, editor diagnostics, change validation, and git diff --check passed.
The live Thor scraper reached both exact URLs; local Windows curl then stopped at its CA-path configuration, so the before/after output comparison used the already-downloaded current HTML through the real DevDocs parser/filter stack.

Re-review is requested because the two issues identified from the screenshots are now covered directly.

Copilot

🟡 Changes recommended

The v1 URL configuration is invalid, API symbols are omitted from the index, and version and styling metadata regress.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Updates the Deno scraper for the current documentation layout, adds fixtures and tests, and configures line-ending normalization.

Changes:

Expands Deno crawling, filtering, and version configuration.
Adds scraper and HTML-cleaning tests with fixtures.
Adds .gitattributes and simplifies file reading in the sprite task.

File summaries

File	Description
`.gitattributes`	Defines text normalization and binary-file handling.
`lib/docs/scrapers/deno.rb`	Reconfigures Deno scraping and versions.
`lib/docs/filters/deno/entries.rb`	Updates entry names and categories.
`lib/docs/filters/deno/clean_html.rb`	Handles the current page structure.
`lib/tasks/sprites.thor`	Uses `File.read` for ERB input.
`test/lib/docs/scrapers/deno_test.rb`	Adds Deno scraper and filter tests.
`test/files/deno_api.html`	Adds an API-page fixture.
`test/files/deno_runtime.html`	Adds a runtime-page fixture.

Review details

Files reviewed: 8/8 changed files
Comments generated: 6
Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

    version '1' do
-      self.release = '1.27.0'
+      self.release = '1.46.3'
+      self.base_url = 'https://docs.deno.com/api/'
    end


      def get_name
-        if result[:path].start_with?('api/deno/')
-          at_css('main[id!="content"]')['id'][/\Asymbol_([.\w]+)/, 1]
-        else
-          at_css('main article h1').content
-        end
+        name = at_css('h1')
+        name ? name.content.strip : slug.split('/').last


  class Deno < UrlScraper
    self.name = 'Deno'
-    self.type = 'simple'
+    self.type = 'deno'


-        /\Aapi\/deno\/~\/Deno\.OpMetrics/, # deprecated in deno 2
-      ]
-      options[:trailing_slash] = false
+      self.release = '2.3.1'



-    # https://github.com/denoland/manual/blob/main/LICENSE
-    # https://github.com/denoland/deno/blob/main/LICENSE.md
+    html_filters.push 'deno/clean_html', 'deno/entries'


+    def parse_page(response)
+      doc = Nokogiri::HTML.parse(response.body)
+      return nil if doc.at_css('meta[http-equiv="refresh"]')
+
+      content = doc.at_css('main, article, [role="main"], .markdown-body')


simon04

Thank you!

chore: add .gitattributes for line ending normalization

1b24573

Ensure consistent line endings and proper diff handling for text and binary files.

srpatcha requested a review from a team as a code owner April 25, 2026 02:42

simon04 changed the title ~~chore: add .gitattributes for line ending normalization~~ Update Deno documentation scraper May 26, 2026

simon04 added the docs/update label May 26, 2026

simon04 requested changes Jul 22, 2026

View reviewed changes

simon04 requested a balanced review from Copilot September 13, 2026 10:04

Copilot started reviewing on behalf of simon04 September 13, 2026 10:05 View session

Copilot AI reviewed Sep 13, 2026

View reviewed changes

simon04 added 2 commits September 13, 2026 13:24

Merge branch 'main' into chore/add-gitattributes

a40be57

Update Deno documentation (2.9.6)

9999535

simon04 approved these changes Sep 13, 2026

View reviewed changes

simon04 merged commit ee39489 into freeCodeCamp:main Sep 13, 2026
1 check passed

Navigation Menu

Uh oh!

Uh oh!

Update Deno documentation scraper - #2681

Update Deno documentation scraper#2681
simon04 merged 5 commits into
freeCodeCamp:mainfrom
srpatcha:chore/add-gitattributes

srpatcha commented Apr 25, 2026 •

edited by simon04

Loading

Uh oh!

simon04 commented Jul 19, 2026

Uh oh!

srpatcha commented Jul 20, 2026

Uh oh!

simon04 left a comment

Uh oh!

srpatcha commented Sep 8, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

simon04 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Uh oh!

Conversation

srpatcha commented Apr 25, 2026 • edited by simon04 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

simon04 commented Jul 19, 2026

Uh oh!

srpatcha commented Jul 20, 2026

Uh oh!

simon04 left a comment

Choose a reason for hiding this comment

Uh oh!

srpatcha commented Sep 8, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

simon04 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

srpatcha commented Apr 25, 2026 •

edited by simon04

Loading