Use httpx instead of requests

[lemon24]

Requests is great and battle-tested, but suffers from a number of issues, which may never get fixed due to feature freeze:

• no default timeouts (work-around: TimeoutHTTPAdapter)
• underpowered response hooks (can't change and retry a request) and no request hooks (details) (work-around: SessionWrapper)
  • to be fair, request hooks are currently unused by reader itself (but there may be other users, How to add custom headers to the reader? #337)
• Requests does not support HTTP2
  • but urllib3 is working on it: https://github.com/urllib3/urllib3/milestone/10

Bad reasons to move away from Requests:

• lack of async support – I have no plan for reader async support at this point
• lack of support for Brotli/Zstandard encodings – urllib3 does support them

Reasons to move to httpx:

• largely Requests-compatible API
• while the ecosystem is probably not comparable, the basics (mocking, auth) are there
• default timeouts (and more kinds than Requests has)
• requests/response hooks (the writable response.next_request attribute should allow for hooks to trigger retries, which should allow getting rid of SessionWrapper entirely)
• URL normalization
• HTTP2 support (but I don't think we're in any hurry for this)

Reasons to not move to httpx:

• not 1.0 yet (but coming soon)
• not as battle-tested as Requests (but can support urllib3 if needed)

Important: Under no circumstances will reader support both Requests and httpx – the entire point of using a high level library like this is to have a single abstraction.

[Ousret]

Probably unexpected,
but I would present another alternative. Namely Niquests.

It's a drop-in replacement for Requests. We took what was established and proven; only to extend its capabilities.
Almost every plugin (aka. extension like requests_mock) work the same.

You mentioned some pros/cons, I would present them as follow for Niquests:

• Completely Requests-compatible API
• Almost every plugins/extensions from Requests era are usable
• default timeouts
• HTTP/2 and HTTP/3 done effortlessly
• stable API (not 0.x ver)
• extended hooks available, like pre_request
• fix a ton of issues from Requests era
• Kept every bit of tests from Requests x urllib3 to ensure highest compat and reliability

This will produce a patch that will (i) simplify your code (ii) have a true upgrade from Requests.

Regards,

[lemon24]

Hi @Ousret, Niquests looks very interesting, I will consider it when I start working on this issue; thank you!

[Ousret]

Don't hesitate to ping us if you need anything. We can assist.