Engineer's Guide to the Requests Library

Module Map

Design Patterns in Use

1. Strategy Pattern — Authentication Handlers

AuthBase defines an abstract __call__ interface. Concrete strategies (HTTPBasicAuth, HTTPDigestAuth, HTTPProxyAuth) implement it. The PreparedRequest.prepare_auth method (models.py:443) applies whichever strategy is passed, supporting both tuple-based basic auth and callable auth handlers. This is the extension point for custom auth (e.g., OAuth).

2. Template Method — Request Preparation

PreparedRequest.prepare() (models.py:422-449) calls sub-methods in a fixed order:

prepare_method → prepare_url → prepare_headers → prepare_cookies → prepare_body → prepare_auth → prepare_hooks

The comment at line 445-449 is critical: prepare_auth must be last because authenticators (like OAuth) may need to modify the fully-prepared request, and prepare_hooks must go after prepare_auth because authenticators can add hooks.

3. Adapter Pattern — Transport Abstraction

BaseAdapter (adapters.py:201-221) defines the interface (send, close). HTTPAdapter is the concrete implementation wrapping urllib3. The Session maintains an OrderedDict of adapters keyed by URL prefix (sessions.py:501-503), and get_adapter (sessions.py:655-671) iterates in descending prefix length order. This allows custom adapters for specific schemes or hosts.

4. Mixin Pattern — Redirect Handling

SessionRedirectMixin (sessions.py:127-392) is a standalone mixin providing all redirect logic (get_redirect_target, should_strip_auth, resolve_redirects, rebuild_auth, rebuild_proxies, rebuild_method). The Session class inherits from it. The mixin is designed to be composable — the docstring at line 136-141 explicitly suggests caching the redirect location on the response object in custom mixins.

5. Null Object Pattern — add_headers

HTTPAdapter.add_headers (adapters.py:599-611) is a no-op (pass) intended for subclasses to override. This avoids conditional checks throughout the send method.

6. MutableMapping + CookieJar — Dual Inheritance

RequestsCookieJar (cookies.py:191-476) inherits from both http.cookiejar.CookieJar and MutableMapping, providing both dict-like access (jar['key']) and standard CookieJar semantics. The __getitem__ method raises CookieConflictError if multiple cookies match the same name (cookies.py:211-227).

7. Context Manager — Session Lifecycle

Session.__enter__/__exit__ (sessions.py:505-509) enables with requests.Session() as s:. The __exit__ calls self.close(), which closes all adapters. Response also supports context manager protocol (models.py:810-811).

Critical Implementation Details

1. The head() function defaults allow_redirects=False

Both api.head() (api.py:102-114) and Session.head() (sessions.py:684-693) use kwargs.setdefault('allow_redirects', False). This is the only method that defaults to not following redirects — all others default to True. This is correct per HTTP spec (HEAD should not follow redirects), but it's a common source of confusion.

2. prepare_auth must be called last — and the comment is the only documentation

In PreparedRequest.prepare() (models.py:445-449):

# Note that prepare_auth must be last to enable authentication schemes
# such as OAuth to work on a fully prepared request.
# This MUST go after prepare_auth. Authenticators could add a hook
self.prepare_hooks(hooks)

If you add a new prepare_* method, it must go before prepare_auth unless you understand the auth implications.

3. The max_retries default is deceptive

In HTTPAdapter.__init__ (adapters.py:207-210):

if max_retries == DEFAULT_RETRIES:  # DEFAULT_RETRIES = 0
    self.max_retries = Retry(0, read=False)
else:
    self.max_retries = Retry.from_int(max_retries)

The default Retry(0, read=False) means zero retries and no read retries. This is not "retry until success" — it's "don't retry at all." The Retry.from_int() factory handles None (disables retries entirely) and integers (creates total retries with read=False).

4. Redirect body rewind is fragile

When following redirects, resolve_redirects (sessions.py:212-214) consumes the response body:

try:
    resp.content  # Consume socket so it can be released
except (ChunkedEncodingError, ContentDecodingError, RuntimeError):
    resp.raw.read(decode_content=False)

For the request body, rewind_body (utils.py:1139-1155) seeks back to _body_position. If the body is a stream that doesn't support seek(), this raises UnrewindableBodyError. The _body_position is recorded in prepare_body (models.py:422-449) only for streamed bodies.

5. Latin1 encoding workaround for redirect locations

In get_redirect_target (sessions.py:142-151):

location = resp.headers["location"]
location = location.encode("latin1")
return to_native_string(location, "utf8")

HTTP headers are decoded as latin1 by the underlying http module. Non-ASCII characters in Location headers are more likely UTF-8 than latin1. This re-encodes to latin1 bytes then decodes as UTF-8, which works for both ASCII and UTF-8 content but will produce garbage for actual latin1 content.

6. should_strip_auth has a backwards-compatibility special case

In should_strip_auth (sessions.py:154-184), there's a special case (lines 164-170) that allows http→https redirects on standard ports to keep the Authorization header. This is not RFC 7235 compliant but is preserved for backwards compatibility.

7. CaseInsensitiveDict preserves original key case

In CaseInsensitiveDict (structures.py:20-93), keys are stored as (lowercase_key, original_key) tuples. __iter__ yields original-case keys (line 70-71), while lower_items() yields lowercased keys (line 76-78). __eq__ converts the other operand to CaseInsensitiveDict if it's a Mapping (line 80-86).

8. JSONDecodeError uses dual inheritance for compatibility

In exceptions.py (lines 38-63), JSONDecodeError inherits from both InvalidJSONError (Requests-specific) and CompatJSONDecodeError (stdlib's json.JSONDecodeError). The __init__ calls both parent constructors, and __reduce__ delegates to the stdlib version for correct pickling.

9. MockResponse.getheaders may have a bug

In cookies.py (lines 131-132):

def getheaders(self, name):
    self._headers.getheaders(name)

The return value is not returned. This appears to be a bug — the method should return self._headers.getheaders(name). However, this method is only used by http.cookiejar internals, and the specific code path may not rely on the return value.

10. SNI support injection at import time

In __init__.py (lines 126-141), if HAS_SNI is False, the module attempts to inject pyopenssl into urllib3. This happens at import time and can fail silently. The _check_cryptography function (lines 99-108) warns if cryptography < 1.3.4.

Entry Points — Start Here

1. src/requests/api.py — The public API surface

Start here to understand what users actually call. All convenience functions (get, post, etc.) delegate to request(), which creates a Session and calls session.request(). This is the thinnest possible layer — everything interesting happens downstream.

2. src/requests/sessions.py — Session class (line 395)

The central orchestrator. Session.request() (line 557) creates a Request, calls prepare_request() to merge session defaults, then calls send(). send() (line 752) resolves adapters, handles redirects, extracts cookies, and dispatches hooks. Understanding this flow is understanding Requests.

3. src/requests/models.py — PreparedRequest.prepare() (line 422)

The request preparation pipeline. This is where URL encoding, header validation, body encoding, cookie handling, and auth all happen in a specific order. The Response class (line 730) handles content consumption, encoding detection, JSON parsing, and link header parsing.

4. src/requests/adapters.py — HTTPAdapter.send() (line 634)

The transport layer. This is where urllib3 is called, exceptions are translated, and the Response is built. The exception translation logic (lines 710-746) is the most complex part — it maps ~10 urllib3 exception types to Requests exceptions.

5. src/requests/__init__.py — The compatibility gate

Read this to understand the dependency requirements and the SNI injection logic. The check_compatibility function (line 60) enforces version constraints at import time.

Known Gotchas

1. head() doesn't follow redirects by default

api.head() and Session.head() both set allow_redirects=False. All other methods default to True. This is correct per HTTP spec but frequently surprises users who expect head() to follow redirects like get() does.

2. max_retries=0 means "no retries" — not "infinite retries"

The default Retry(0, read=False) means zero total retries. Retry.from_int(None) disables retries entirely. Retry.from_int(5) allows 5 retries but with read=False (no retries on read timeouts). File: adapters.py:207-210.

3. prepare_auth must be last in the prepare pipeline

If you add a new prepare_* method to PreparedRequest, it must go before prepare_auth. Authenticators (especially OAuth) depend on the request being fully prepared. File: models.py:445-449.

4. CaseInsensitiveDict.__eq__ converts non-dict Mappings

When comparing a CaseInsensitiveDict to any Mapping, it converts the other operand to CaseInsensitiveDict first. This means CaseInsensitiveDict({'Foo': 1}) == {'foo': 1} is True, but the reverse ({'foo': 1} == CaseInsensitiveDict({'Foo': 1})) is False because regular dict doesn't do this conversion. File: structures.py:80-86.

5. Response.ok catches and suppresses exceptions

Response.ok (models.py:860-872) calls raise_for_status() and catches HTTPError. This means ok returns False for any 4xx/5xx, but it also means any other exception (e.g., from a bug in raise_for_status) would propagate and crash. This is a fragile pattern.

6. Response.content is a property that consumes the stream

Accessing response.content (models.py:1031-1047) reads the entire raw response into memory. After this, response.raw is consumed. If you need streaming, use response.iter_content() or response.raw directly before accessing .content.

7. extract_cookies_to_jar returns early if _original_response is missing

In cookies.py (lines 135-150), if the response doesn't have an _original_response attribute (or it's falsy), cookies are not extracted. This can happen with mock responses or certain proxy setups.

8. MockResponse.getheaders doesn't return a value

In cookies.py:131-132, getheaders calls self._headers.getheaders(name) but doesn't return the result. This is likely a bug, though it may not manifest because http.cookiejar may use other methods.

9. Redirect body consumption catches three exception types

In resolve_redirects (sessions.py:212-214), the response body is consumed with a try/except that catches ChunkedEncodingError, ContentDecodingError, and RuntimeError. Any other exception during body consumption propagates and breaks redirect handling.

10. trust_env=True reads environment variables at request time

When trust_env is True (the default), merge_environment_settings (sessions.py:831-868) reads HTTP_PROXY, HTTPS_PROXY, NO_PROXY, REQUESTS_CA_BUNDLE, and CURL_CA_BUNDLE from the environment on every request. This means changing environment variables between requests affects behavior without any explicit API call.