Skip to content

bug: query parameters and fragments in download URLs corrupt filenames, crash on Windows, and break format detection #95

Description

@yush-1018

Bug Report

Describe the bug
In databusclient/api/download.py (line 452), target filenames are derived by naive slash-splitting of the raw download URL:

file = url.split("/")[-1]
filename = os.path.join(localDir, file)

When downloading files from URLs that contain query parameters or fragments (such as presigned S3/GCS links, Nextcloud/WebDAV shared links, or token-based downloads like https://example.org/data.ttl.bz2?token=abc123&expires=1700000), the entire query string remains attached to file (data.ttl.bz2?token=abc123&expires=1700000).

This causes two critical issues:

  1. Windows crash: On Windows, ? is a reserved illegal character in file paths. Opening the file with open(filename, "wb") crashes immediately with OSError: [Errno 22] Invalid argument.
  2. Format & compression detection failure: On Linux/macOS, the file is saved with the query string in its name. Subsequent checks like _detect_compression_format(file) (which checks .endswith(".bz2") or .endswith(".gz")) fail because the filename ends with the query parameters instead of the file extension. Consequently, on-the-fly decompression (--compression none) and format conversions fail.

To Reproduce
Attempt to download any file using a URL that contains query parameters:

databusclient download "https://raw.githubusercontent.com/dbpedia/databus-python-client/main/tests/resources/test.ttl?raw=true"
  • On Windows: immediate crash with OSError: [Errno 22] Invalid argument.
  • On Linux/macOS: saved as test.ttl?raw=true and format detection fails.

Expected behavior
The client should parse the URL and extract the clean path component without query strings or fragments, ensuring safe file paths across all operating systems and proper format detection.

Proposed Solution
Extract the clean basename from the URL's path component using standard library tools:

import posixpath
from urllib.parse import unquote, urlparse

url_path = urlparse(url).path
file = posixpath.basename(unquote(url_path)) or "downloaded_file"
filename = os.path.join(localDir, file)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions