Skip to content

Add an opt-in async client to the Python SDK - #2167

Open
Xharry01X wants to merge 5 commits into
appwrite:mainfrom
Xharry01X:feat/python-async-sdk
Open

Xharry01X wants to merge 5 commits into
appwrite:mainfrom
Xharry01X:feat/python-async-sdk

Conversation

@Xharry01X

@Xharry01X Xharry01X commented Oct 28, 2026 •

Copy link
Copy Markdown

What does this PR do?

Adds an asynchronous client to the generated Python SDK, next to the existing synchronous one.

The synchronous Client and the services stay the default and are generated exactly as before, so nothing changes for existing users. The asynchronous client is opt-in and ships with the async extra:

# before — unchanged, still the default
user = users.get(user_id='<USER_ID>')

# after — opt-in
user = await async_users.get(user_id='<USER_ID>')

Changes

Sync stays the default

  • templates/python/package/client.py.twig and the service template are unchanged, and the generated sync SDK is byte-identical to main
  • The only difference in the generated sync tree is appwrite/utils/deprecated.py, whose @deprecated decorator now keeps coroutine functions coroutines

Async is added alongside

  • templates/python/base/service_class.twig holds the shared service body. One flag drives the class name (Async prefix), async def and the awaited client call, so the two trees cannot drift apart
  • templates/python/package/async_client.py.twig generates appwrite/async_client.py with AsyncClient
  • templates/python/package/async_services/service.py.twig generates appwrite/async_services/<service>.py with Async<Service>
  • The async client owns a pooled httpx2.AsyncClient instead of using requests per call, supports async with and aclose(), and swaps the pool out when set_self_signed() changes the TLS settings
  • A redirect is still reported as an error rather than followed, so the API key is never carried to another host

Chunked upload

  • The ThreadPoolExecutor is replaced by an AnyIO task group with a semaphore
  • Every sibling chunk settles before a failure surfaces. The task group cancels its siblings on the first raise, so failures are captured in the worker and re-raised after the group closes
  • A progress callback that raises is captured with the other failures, so it cannot cancel the uploads that are still running
  • The progress callback may be a coroutine function; it is awaited when it returns an awaitable

Dependencies and tests

  • httpx2 and anyio ship in the new async extra. The base install keeps requests and urllib3, so Python 3.10 is still supported and the sync suite is unchanged
  • test_async holds the asynchronous suite, which mocks the transport with respx and runs on unittest.IsolatedAsyncioTestCase. test is unchanged
  • The e2e job runs the async suite on the Python versions that can install httpx2

Testing

  • php example.php python server, then the generated suites: 884 sync tests and 781 async tests, all passing
  • black --check on the generated output: 553 files unchanged
  • composer lint-twig: 641 templates, 0 errors
  • composer refactor:check: clean
  • The generated sync output was diffed against main and is byte-identical apart from the additions listed above
  • The Python fragments in tests/generation/GenerationTest.php were checked against the generated fixture

The e2e job needs Docker, so it was not run locally.

Related

The same work is open against the generated SDK for review: appwrite/sdk-for-python#177

The Python SDK is the last server SDK that blocks on a synchronous
transport. Move it to `httpx2` and `anyio` so every call is a coroutine
and the SDK runs on both asyncio and trio.

- The client owns a pooled `httpx2.AsyncClient`, supports `async with`
  and `aclose()`, and swaps the pool out when `set_self_signed()` changes
  the TLS settings
- Chunked uploads run in an AnyIO task group with a semaphore. Every
  sibling chunk settles before a failure surfaces, so an error in one
  chunk never cancels the uploads that are still running, and a progress
  callback that raises is captured with the other failures
- Service methods become `async def` and await the client
- The `@deprecated` decorator keeps coroutine functions coroutines and
  guards the temporary warning filter with a lock
- Docs examples await the call and run inside an async main
- Dependencies move from `requests` to `httpx2` and `anyio`, and the
  generated tests mock the transport with `respx`
- The minimum Python version moves to 3.11, which `httpx2` requires, so
  the 3.10 e2e job is dropped
@hansi-codes

hansi-codes Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

🟢 Tier S · Ready to merge

This PR adds an opt-in asynchronous Python SDK alongside the existing synchronous client and services, with generated async tests, optional dependencies, and Python runtime coverage. The latest updates make async redirect handling return location responses correctly, serialize chunk-upload progress callbacks, preserve cancellation during upload resumption, and show a runnable async README example.

Latest changes: The newest commits distinguish redirect responses from HTTP errors, serialize progress updates during concurrent chunk uploads, allow cancellation to propagate during upload resumption, and wrap the README async usage in an entry point.

Verdict New comments Fixed Still open
💬 Commented 0 2 0

Note

@Xharry01X does not have write access to this repository, so Hansi does not approve automatically. A maintainer can review and approve.

📂 Walkthrough · 16
File Change
src/SDK/Language/Python.php Registers generated async client, service, and test files.
templates/python/README.md.twig Documents sync-default and opt-in async usage; the async example now runs through asyncio.run.
templates/python/base/requests/api.twig; templates/python/base/requests/file.twig; templates/python/base/service_class.twig Shares service generation between sync and async clients and awaits async API and upload calls.
templates/python/package/client.py.twig; templates/python/package/async_client.py.twig; templates/python/package/async_services/__init__.py.twig Adds the pooled async HTTP client and generated async service exports; the latest updates handle redirects and serialize upload progress.
templates/python/package/services/service.py.twig; templates/python/package/async_services/service.py.twig Generates synchronous and asynchronous service methods from the shared service body.
templates/python/package/utils/deprecated.py.twig Keeps coroutine functions coroutine-compatible when decorated as deprecated.
templates/python/pyproject.toml.twig; templates/python/requirements.txt.twig; templates/python/setup.py.twig Defines package extras and dependency and Python-version requirements for sync and async installations.
templates/python/test_async/__init__.py.twig; templates/python/test_async/services/__init__.py.twig; templates/python/test_async/services/test_service.py.twig Adds generated asynchronous service tests using an async test case and mocked transport.
tests/e2e/Runtime/Python.php Adds Python runtime coverage for the generated SDK.
tests/generation/GenerationTest.php Updates generated-tree expectations for the async Python SDK.
templates/python/base/requests/api.twig Awaits API calls made through the client.
templates/python/base/requests/file.twig Awaits chunked file uploads.
templates/python/package/services/service.py.twig Generates async service methods that await client calls.
templates/python/pyproject.toml.twig Updates supported Python versions and async runtime/test dependencies.
templates/python/requirements.txt.twig Replaces synchronous HTTP dependencies and adds async test dependencies.
templates/python/setup.py.twig Updates package dependencies and the minimum Python version.
✅ Fixed since the last review · 2
  • Return locations before raising on 3xx responses · templates/python/package/async_client.py.twig:164
  • Wrap the async example in an entry point · templates/python/README.md.twig:63

Reviewed the commits since 2eb3419 · Details · Comment @hansi-codes review to re-run, or mention @hansi-codes with a question.

hansi-codes[bot]
hansi-codes Bot previously requested changes Oct 28, 2026

@hansi-codes hansi-codes Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Tier B · 1 blocking finding to address. Summary

Comment thread templates/python/package/client.py.twig Resolved
Comment thread templates/python/package/client.py.twig Resolved
Comment thread templates/python/package/client.py.twig Resolved
httpx2 raises for 3xx responses, so the redirect error and the OAuth
Location were never reached. Let a cancellation escape the upload
resume probe, and serialize the async progress callbacks so they never
overlap and always report the latest state.

@hansi-codes hansi-codes Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Tier S · Looks good to merge. Summary

@hansi-codes
hansi-codes Bot dismissed their stale review October 28, 2026 15:25

The findings that requested changes are resolved. See the summary comment for what is still open.

@Xharry01X

Copy link
Copy Markdown
Author

Addressed the three review findings in 1dfb764.

Finding Fix
3xx raised before the redirect handling The redirect is reported first and only error statuses are raised for
Cancellation swallowed by the resume probe The bare except became except Exception
Overlapping progress callbacks The workers take an anyio.Lock around the counters and the callback

Validation of the regenerated SDK:

  • 884 generated tests pass
  • black --check is clean
  • djlint lints 635 templates with 0 errors
  • The generation suite reports the same failures as the base branch (a local Windows path separator artifact)

The Validation and CI workflows are waiting for a maintainer to approve the run.

@ChiragAgg5k

Copy link
Copy Markdown
Member

Thanks for the thorough work here. The async client itself looks solid, and the review fixes in 1dfb764 are right.

We'd like async to be opt-in rather than replacing the sync client, though. Turning every service method into a coroutine breaks every existing integration (scripts, Django views, Celery tasks), and the AsyncClient is tied to one event loop, so wrapping calls in asyncio.run isn't a clean workaround. Could you keep the current sync Client and services as the default, and add the async versions alongside them (e.g. AsyncClient plus async services), similar to how httpx ships both? Users who want asyncio or trio can switch over, and nobody else has to change anything.

@XChikuX

XChikuX commented Oct 30, 2026 •

Copy link
Copy Markdown

@ChiragAgg5k

Is that the only thing you'll need? Should be doable with unasync, But I'd like async version to be default (future tests and the like will test against async first), but the imports and working will remain the same for the sync version. No python backend should be sync default in 2026.

Anyone who needs the sync version will be able to use it as before.

We'll update to the latest SDK as well since we already have some conflicts.

# Conflicts:
#	tests/e2e/languages/python/tests.py
The async client replaces the sync one instead of sitting next to it,
which breaks every existing integration. Ship both instead: the sync
`Client` and its services stay the default and are generated exactly as
before, and `AsyncClient` plus the `Async*` services are added next to
them as an opt-in.

- The service template is shared by both, with the class name, the
  `async def` and the awaited client call driven by one flag
- The async client keeps the pooled `httpx2` transport, the AnyIO task
  group for chunked uploads and the `async with`/`aclose()` lifecycle
- `httpx2` and `anyio` ship in the `async` extra, so the base install
  and the 3.10 runtime are unchanged
- The async suite mocks the transport with `respx` and runs in the e2e
  job on the Python versions that can install `httpx2`
hansi-codes[bot]
hansi-codes Bot previously requested changes Oct 31, 2026

@hansi-codes hansi-codes Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Tier B · 1 blocking finding to address. Summary

Comment thread templates/python/README.md.twig Resolved
@Xharry01X Xharry01X changed the title Make the Python SDK async on httpx and anyio Add an opt-in async client to the Python SDK Oct 31, 2026
The async client template was taken from the revision before the review
fixes landed on the client.

- A 3xx response is reported before the status is raised, so a
  web-auth call with response_type='location' returns the Location
  header again
- The upload resume probe no longer swallows a cancellation, which is a
  BaseException and would restart the upload from chunk zero
- The chunk workers take an anyio.Lock around the counters and the
  progress callback, the way the threaded client took its threading.Lock
- The README example awaits inside an async main, so it is valid Python

@hansi-codes hansi-codes Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Tier S · Looks good to merge. Summary

@hansi-codes
hansi-codes Bot dismissed their stale review October 31, 2026 13:11

The findings that requested changes are resolved. See the summary comment for what is still open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants