diff --git a/connect_attempts_report.md b/connect_attempts_report.md new file mode 100644 index 00000000000..311ba741ef6 --- /dev/null +++ b/connect_attempts_report.md @@ -0,0 +1,305 @@ +# Report of connect attempts bugs + +- affected versions: 10.1.x and smaller + +Relevant configs: + +- `proxy.config.http.connect_attempts_max_retries` +- `proxy.config.http.connect_attempts_max_retries_down_server` +- `proxy.config.http.connect_attempts_rr_retries` +- `proxy.config.http.down_server.cache_time` (the `fail_window`) + +PRs: + +- https://github.com/apache/trafficserver/pull/12846 +- https://github.com/apache/trafficserver/pull/12880 +- https://github.com/apache/trafficserver/pull/13092 +- https://github.com/apache/trafficserver/pull/13102 + +## Net effect between 10.1.x and later + +On 10.1.x and earlier, the three `connect_attempts_*` configs behaved +erratically: + +- `connect_attempts_max_retries_down_server` was effectively unreachable + because the `is_down()` check was inverted (#12846). +- Single-address origins ignored down-state entirely — `select_best_http` + unconditionally returned the first (and only) entry (#12880). +- The "mark host DOWN" threshold was gated by `connect_attempts_rr_retries` + instead of `connect_attempts_max_retries` (#13102). +- The fail-window clock used `client_request_time` instead of the actual + failure time, so the window could start in the past or be pre-expired + (#13102). +- After `select_next_rr()`, `dst_addr` was not updated, so RR "switches" + silently kept hitting the same target (#13102). +- `mark_host_failure()` ran at end-of-txn against the post-switch + `dns_info.active`, so failures on A could be charged to B (#13102). +- RR exhaustion terminated retries early even when the current target + still had per-host budget (#13102). + +The combined effect of PRs #12846, #12880, #13092, and #13102 is a +coherent model in which each config has a well-defined role tied to an +explicit `UP / DOWN / SUSPECT` state for the active target. + +### Worked examples + +All examples use the ATS defaults: + +| Config | Default | +|----------------------------------------------|---------| +| `connect_attempts_max_retries` | 3 | +| `connect_attempts_max_retries_down_server` | 1 | +| `connect_attempts_rr_retries` | 3 | +| `down_server.cache_time` (`fail_window`) | 60s | + +#### Example 1 — Single-address origin goes offline + +Client sends repeated requests to an origin whose only A record points to +a down IP. How many connect attempts does ATS make before the IP is +marked DOWN in HostDB? + +**10.1.x behavior — IP marked DOWN after 12 connect attempts (3 failed +transactions).** + +Why: `mark_host_failure()` is only called from +`do_hostdb_update_if_necessary()`, which is only reached at the end of a +transaction (via `handle_server_connection_not_open`). So `fail_count` +increments **once per failed transaction**, not once per attempt. The +threshold is `connect_attempts_rr_retries` (default `3`) — the wrong +config, but that's what 10.1.x uses. Per-transaction there are `1 + +connect_attempts_max_retries = 4` attempts. + +| Attempt | Txn | `fail_count` after | marked DOWN? | +|---------|-----|--------------------|--------------| +| 1 | 1 | 0 | no | +| 2 | 1 | 0 | no | +| 3 | 1 | 0 | no | +| 4 | 1 | 1 (end of txn) | no | +| 5 | 2 | 1 | no | +| 6 | 2 | 1 | no | +| 7 | 2 | 1 | no | +| 8 | 2 | 2 (end of txn) | no | +| 9 | 3 | 2 | no | +| 10 | 3 | 2 | no | +| 11 | 3 | 2 | no | +| **12** | 3 | 3 ≥ 3 | **yes** | + +And because `select_best_http()` returns `info[0]` unconditionally for +single-address records (#12880), every one of those 12 attempts actually +hits the network. Even after the IP is marked DOWN, subsequent requests +keep trying it because the single-address code path ignores down-state. + +**Later behavior — IP marked DOWN after 4 connect attempts (1 failed +transaction).** + +Why: `do_hostdb_update_if_necessary()` is now called after **each** +failure in `handle_response_from_server()`, so `fail_count` increments +**once per attempt**. The threshold is `max_retries + 1 = 4` (for an UP +target). Per-transaction there are still `1 + max_retries = 4` attempts, +so the threshold is reached on the last attempt of the first failed +transaction. + +| Attempt | Txn | `fail_count` after | marked DOWN? | +|---------|-----|--------------------|--------------| +| 1 | 1 | 1 | no | +| 2 | 1 | 2 | no | +| 3 | 1 | 3 | no | +| **4** | 1 | 4 ≥ 4 | **yes** | + +Subsequent requests within `fail_window` (60s): +`select_best_http()` now honors down-state for single-address records, +so HostDB lookup returns `nullptr` and the SM reports `OriginDown` +immediately — **zero connect attempts**. + +After `fail_window` elapses, the entry is SUSPECT. The next request gets +`connect_attempts_max_retries_down_server + 1 = 2` attempts (1 probe + +1 retry). Success → UP; failure → back to DOWN for another 60s. + +#### Example 2 — Round-robin with one bad backend (A fails, B/C healthy) + +Defaults apply: `rr_retries = 3`, `max_retries = 3`. A is unresponsive, +B and C are fine. Initial active = A. + +**10.1.x behavior — client gets 502, and the wrong host is blamed.** + +Two compounding bugs: (a) after `select_next_rr()` is called, neither +`dns_info.addr` nor `server_info.dst_addr` is updated, so the next +connect still goes to A's address. (b) `mark_host_failure()` is called +once at end-of-txn against `dns_info.active`, which by that point is +the *post-switch* host (B), so B's `fail_count` is incremented for A's +failures. + +| Attempt | dst_addr (target) | dns_info.active | retry_attempts after | `(retry+1) % 3 == 0`? | Result | +|---------|-------------------|-----------------|----------------------|----------------------|--------| +| 1 | A | A | 1 | no | fail | +| 2 | A | A | 2 | no | fail | +| 3 | A | A → switched to B (`select_next_rr`); dst_addr untouched | 3 | **yes** | fail | +| 4 | A (still!) | B | 3 ≥ 3, give up | n/a | fail → 502 to client | + +End of txn: `mark_host_failure()` increments `B.fail_count` even +though B was never tried. Repeated requests do this 3 times → **B +gets marked DOWN** while A (the actually-broken host) keeps getting +hit on every transaction. + +**Later behavior — client gets 200; A's failures are correctly +recorded against A.** + +`do_hostdb_update_if_necessary()` runs after each failure (so the +correct `dns_info.active` is in scope), and after `select_next_rr()` +the SM explicitly assigns `dns_info.addr` and `server_info.dst_addr` +to the new target. + +| Attempt | dst_addr (target) | dns_info.active | A.fail_count after | retry_attempts after | RR switch? | Result | +|---------|-------------------|-----------------|--------------------|----------------------|------------|--------| +| 1 | A | A | 1 | 1 | no | fail | +| 2 | A | A | 2 | 2 | no | fail | +| 3 | A | A | 3 | 3 | **yes** → active=B, dst_addr=B | fail | +| **4** | **B** | B | 3 (unchanged) | n/a | n/a | **success → 200** | + +A finishes this transaction with `fail_count = 3` (not yet at the +threshold of `max_retries + 1 = 4`). The next request that hashes to +A bumps it to 4 and marks A DOWN; subsequent requests within +`fail_window` skip A entirely until the window expires. + +If B and C had also been DOWN (no selectable alternate), +`select_next_rr()` returns false and the SM stays on A using its +remaining per-host retry budget instead of giving up — new in +#13102. + +**Later behavior** + +- Failure timestamp is `ts_clock::now()` at the moment `mark_down` + is called, so the full `fail_window` is honored regardless of how + long the transaction took to decide the host was bad. + + +## Summary of bug-fix PRs + +### PR #12846 — Fix `HostDBInfo::is_down` condition + +**Bug.** `HostDBInfo::is_down()` had an inverted comparison: + +```cpp +// Before +return (last_fail != TS_TIME_ZERO) && (last_fail + fail_window < now); +// After +return (last_fail != TS_TIME_ZERO) && (now <= last_fail + fail_window); +``` + +The pre-fix code reported a host as DOWN only *after* `fail_window` had elapsed — +the opposite of intent. During the fail window (when the host should be +considered blocked) it returned `false`. + +**Behavior change.** A failed host is now correctly treated as DOWN for the +entire `fail_window` following `last_failure`, and returns to UP only after +the window has elapsed. Every downstream path that calls `is_down()` (retry +budget, RR selection, negative-cached check) is impacted. + +### PR #12880 — Check state of HostDBInfo in `select_best_http` + +**Bug.** For single-address (non round-robin) records, +`HostDBRecord::select_best_http()` bypassed the state check: + +```cpp +// Before +best_alive = &info[0]; // unconditional +// After +if (info[0].select(now, fail_window)) { + best_alive = &info[0]; +} +``` + +A single-address host that was marked down would still be returned from +HostDB and reused for the next request. + +**Behavior change.** Single-address origins now honor `down_server` state the +same way RR origins do: if the only target is DOWN and still within the fail +window, HostDB lookup fails (`nullptr`), the SM reports `OriginDown`, and no +connection attempt is made until the window expires. The gold autest +`dns_host_down` was updated accordingly (second request returns 500 instead +of 502 once the first failure marked the IP down). + +### PR #13092 — Clarify `HostDBInfo` state + +**Not a bug fix — a model/refactor.** Introduces an explicit tri-state for +upstream health and reshapes the API around it: + +```cpp +enum class HostDBInfo::State { UP, DOWN, SUSPECT }; +``` + +- **UP** — no known failure; normal selection. +- **DOWN** — `_last_failure` set, `now` is within `fail_window`; not eligible. +- **SUSPECT** — `fail_window` has elapsed; a probe is permitted. A successful + response transitions it to UP via `mark_up()`; another failure returns it + to DOWN. + +API renames: `is_alive` → `is_up`, `mark_active_server_alive` → +`mark_active_server_up`. `mark_down` now takes `fail_window`. `select()` is +replaced by callers using `is_down()` directly. `last_failure` / `fail_count` +become private atomics (`_last_failure`, `_fail_count`). + +**Behavior change.** The previous two-state model collapsed SUSPECT into UP +once `fail_window` expired, which meant a recovering host was treated +identically to a never-failed host for retry sizing. With SUSPECT explicit, +callers (notably the retry machinery in #13102) can apply +`connect_attempts_max_retries_down_server` to probing traffic and +`connect_attempts_max_retries` to UP traffic. + +### PR #13102 — Fix connect attempt retries + +**Bugs.** The retry machinery in +`HttpTransact::handle_response_from_server()` and +`HttpSM::mark_host_failure()` had several interlocking issues: + +1. **Wrong threshold used to mark host down.** `mark_host_failure()` called + `increment_fail_count(..., connect_attempts_rr_retries, ...)`. The RR + switch-over config was reused as the "mark down" threshold, so a host was + marked DOWN after `connect_attempts_rr_retries` failures instead of + `connect_attempts_max_retries + 1` total attempts. +2. **Wrong clock for failure timestamp.** + `do_hostdb_update_if_necessary()` recorded the failure at + `t_state.client_request_time` (request-receipt time), not + `ts_clock::now()`. For long requests the fail window started in the past, + sometimes expiring before the failure was even recorded. +3. **RR exhaustion short-circuited all retries.** When the + `connect_attempts_rr_retries` boundary was hit and no other RR member was + selectable, the old code gave up, even though the current target still + had per-host retry budget. +4. **`connect_attempts_max_retries_down_server` had no effect in practice.** + It was only consulted through `is_server_negative_cached()`, which + depended on the inverted `is_down()` from #12846. In many cases it was + either never applied or applied when it shouldn't have been. +5. **`connect_attempts_max_retries_down_server == 0` blocked the probe.** + A separate code path in `do_http_server_open()` refused to connect at all + when the target was negative-cached and the config was zero — preventing + the SUSPECT-state probe that the fail window was designed to allow. +6. **Retry config range allowed overflow.** `[0-255]` with `uint8_t` + threshold = `max_retries + 1` wraps to 0. +7. Rename `proxy.config.http.connect_attempts_max_retries_down_server` with `proxy.config.http.connect_attempts_max_retries_suspicious_server` to clarify behavior. + The `proxy.config.http.connect_attempts_max_retries_down_server` config is deprecated. + + + +**Behavior change.** + +- Retry budget is now driven by HostDB state (via the new helper + `HttpTransact::origin_server_connect_attempts_max_retries`): + - `UP` → `connect_attempts_max_retries` + - `DOWN` → 0 (bail out immediately) + - `SUSPECT` → `connect_attempts_max_retries_down_server` +- `mark_host_failure` uses `max_retries + 1` as the attempt budget and + `ts_clock::now()` as the failure time. +- RR switch and per-host retry are separated: if RR has no selectable + alternate, the SM stays on the current target and keeps retrying up to + the per-host budget rather than giving up. +- The `do_http_server_open()` early-bail on "negative cached + + down_server=0" is removed — DOWN vs SUSPECT is now the single source of + truth. +- `proxy.config.http.connect_attempts_max_retries`, + `…_max_retries_down_server`, and `…_rr_retries` clamped to `[0-254]` to + avoid the `uint8_t` overflow when adding 1 for the total-attempt count. +- A new warning is logged at config reconfigure when + `connect_attempts_max_retries_down_server == 0` and + `connect_attempts_rr_retries > 0`, because that combination prevents any + probe of a recovering (SUSPECT) origin. diff --git a/doc/admin-guide/files/records.yaml.en.rst b/doc/admin-guide/files/records.yaml.en.rst index 6fdc5242750..ee739acccfd 100644 --- a/doc/admin-guide/files/records.yaml.en.rst +++ b/doc/admin-guide/files/records.yaml.en.rst @@ -1812,15 +1812,28 @@ Origin Server Connect Attempts The maximum number of connection retries |TS| can make when the origin server is not responding. Each retry attempt lasts for `proxy.config.http.connect_attempts_timeout`_ seconds. Once the maximum number of retries is reached, the origin is marked down (as controlled by `proxy.config.http.connect.down.policy`_. After this, the setting - `proxy.config.http.connect_attempts_max_retries_down_server`_ is used to limit the number of retry attempts to the known down origin. + `proxy.config.http.connect_attempts_max_retries_suspect_server`_ is used to limit the number of retry attempts when the origin is + in the SUSPECT state (recovering after `proxy.config.http.down_server.cache_time`_ has elapsed). + +.. ts:cv:: CONFIG proxy.config.http.connect_attempts_max_retries_suspect_server INT 1 + :reloadable: + :overridable: + + Maximum number of connection retries |TS| can make while an origin is in the SUSPECT state (the first request after + `proxy.config.http.down_server.cache_time`_ has elapsed on a previously-down origin). The total attempt budget for a SUSPECT + origin is therefore ``connect_attempts_max_retries_suspect_server + 1`` (the initial probe plus each retry). If any attempt + succeeds, the origin transitions back to UP; if all attempts fail, the origin returns to DOWN for another + `proxy.config.http.down_server.cache_time`_ seconds. Typically smaller than `proxy.config.http.connect_attempts_max_retries`_ + so the recovering origin is not flooded. .. ts:cv:: CONFIG proxy.config.http.connect_attempts_max_retries_down_server INT 1 :reloadable: :overridable: + :deprecated: - Maximum number of connection attempts |TS| can make while an origin is marked down per request. Typically this value is smaller than - `proxy.config.http.connect_attempts_max_retries`_ so an error is returned to the client faster and also to reduce the load on the down origin. - The timeout interval `proxy.config.http.connect_attempts_timeout`_ in seconds is used with this setting. + This setting is deprecated in favor of :ts:cv:`proxy.config.http.connect_attempts_max_retries_suspect_server`. If the + deprecated setting is set explicitly and the replacement is not, the deprecated value is mirrored forward and a warning is + logged. If both are set explicitly, the new setting wins and the deprecated value is ignored. .. ts:cv:: CONFIG proxy.config.http.connect_attempts_retry_backoff_base INT 0 :reloadable: diff --git a/doc/developer-guide/api/functions/TSHttpOverridableConfig.en.rst b/doc/developer-guide/api/functions/TSHttpOverridableConfig.en.rst index 9a175f4e15f..1e22d33fd61 100644 --- a/doc/developer-guide/api/functions/TSHttpOverridableConfig.en.rst +++ b/doc/developer-guide/api/functions/TSHttpOverridableConfig.en.rst @@ -65,139 +65,140 @@ Testing :enumerator:`TS_CONFIG_BODY_FACTORY_TEMPLATE_BASE`. The following configurations (from ``records.yaml``) are overridable: -====================================================================== ==================================================================== -TSOverridableConfigKey Value Configuration Value -====================================================================== ==================================================================== -:enumerator:`TS_CONFIG_BODY_FACTORY_TEMPLATE_BASE` :ts:cv:`proxy.config.body_factory.template_base` -:enumerator:`TS_CONFIG_HTTP_ALLOW_HALF_OPEN` :ts:cv:`proxy.config.http.allow_half_open` -:enumerator:`TS_CONFIG_HTTP_ALLOW_MULTI_RANGE` :ts:cv:`proxy.config.http.allow_multi_range` -:enumerator:`TS_CONFIG_HTTP_ANONYMIZE_INSERT_CLIENT_IP` :ts:cv:`proxy.config.http.insert_client_ip` -:enumerator:`TS_CONFIG_HTTP_ANONYMIZE_REMOVE_CLIENT_IP` :ts:cv:`proxy.config.http.anonymize_remove_client_ip` -:enumerator:`TS_CONFIG_HTTP_ANONYMIZE_REMOVE_COOKIE` :ts:cv:`proxy.config.http.anonymize_remove_cookie` -:enumerator:`TS_CONFIG_HTTP_ANONYMIZE_REMOVE_FROM` :ts:cv:`proxy.config.http.anonymize_remove_from` -:enumerator:`TS_CONFIG_HTTP_ANONYMIZE_REMOVE_REFERER` :ts:cv:`proxy.config.http.anonymize_remove_referer` -:enumerator:`TS_CONFIG_HTTP_ANONYMIZE_REMOVE_USER_AGENT` :ts:cv:`proxy.config.http.anonymize_remove_user_agent` -:enumerator:`TS_CONFIG_HTTP_ATTACH_SERVER_SESSION_TO_CLIENT` :ts:cv:`proxy.config.http.attach_server_session_to_client` -:enumerator:`TS_CONFIG_HTTP_MAX_PROXY_CYCLES` :ts:cv:`proxy.config.http.max_proxy_cycles` -:enumerator:`TS_CONFIG_HTTP_AUTH_SERVER_SESSION_PRIVATE` :ts:cv:`proxy.config.http.auth_server_session_private` -:enumerator:`TS_CONFIG_HTTP_BACKGROUND_FILL_ACTIVE_TIMEOUT` :ts:cv:`proxy.config.http.background_fill_active_timeout` -:enumerator:`TS_CONFIG_HTTP_BACKGROUND_FILL_COMPLETED_THRESHOLD` :ts:cv:`proxy.config.http.background_fill_completed_threshold` -:enumerator:`TS_CONFIG_HTTP_CACHE_CACHE_RESPONSES_TO_COOKIES` :ts:cv:`proxy.config.http.cache.cache_responses_to_cookies` -:enumerator:`TS_CONFIG_HTTP_CACHE_CACHE_URLS_THAT_LOOK_DYNAMIC` :ts:cv:`proxy.config.http.cache.cache_urls_that_look_dynamic` -:enumerator:`TS_CONFIG_HTTP_CACHE_IGNORE_QUERY` :ts:cv:`proxy.config.http.cache.ignore_query` -:enumerator:`TS_CONFIG_HTTP_CACHE_GENERATION` :ts:cv:`proxy.config.http.cache.generation` -:enumerator:`TS_CONFIG_HTTP_CACHE_GUARANTEED_MAX_LIFETIME` :ts:cv:`proxy.config.http.cache.guaranteed_max_lifetime` -:enumerator:`TS_CONFIG_HTTP_CACHE_GUARANTEED_MIN_LIFETIME` :ts:cv:`proxy.config.http.cache.guaranteed_min_lifetime` -:enumerator:`TS_CONFIG_HTTP_CACHE_HEURISTIC_LM_FACTOR` :ts:cv:`proxy.config.http.cache.heuristic_lm_factor` -:enumerator:`TS_CONFIG_HTTP_CACHE_HEURISTIC_MAX_LIFETIME` :ts:cv:`proxy.config.http.cache.heuristic_max_lifetime` -:enumerator:`TS_CONFIG_HTTP_CACHE_HEURISTIC_MIN_LIFETIME` :ts:cv:`proxy.config.http.cache.heuristic_min_lifetime` -:enumerator:`TS_CONFIG_HTTP_CACHE_HTTP` :ts:cv:`proxy.config.http.cache.http` -:enumerator:`TS_CONFIG_HTTP_CACHE_IGNORE_ACCEPT_CHARSET_MISMATCH` :ts:cv:`proxy.config.http.cache.ignore_accept_charset_mismatch` -:enumerator:`TS_CONFIG_HTTP_CACHE_IGNORE_ACCEPT_ENCODING_MISMATCH` :ts:cv:`proxy.config.http.cache.ignore_accept_encoding_mismatch` -:enumerator:`TS_CONFIG_HTTP_CACHE_IGNORE_ACCEPT_LANGUAGE_MISMATCH` :ts:cv:`proxy.config.http.cache.ignore_accept_language_mismatch` -:enumerator:`TS_CONFIG_HTTP_CACHE_IGNORE_ACCEPT_MISMATCH` :ts:cv:`proxy.config.http.cache.ignore_accept_mismatch` -:enumerator:`TS_CONFIG_HTTP_CACHE_IGNORE_AUTHENTICATION` :ts:cv:`proxy.config.http.cache.ignore_authentication` -:enumerator:`TS_CONFIG_HTTP_CACHE_IGNORE_CLIENT_CC_MAX_AGE` :ts:cv:`proxy.config.http.cache.ignore_client_cc_max_age` -:enumerator:`TS_CONFIG_HTTP_CACHE_IGNORE_CLIENT_NO_CACHE` :ts:cv:`proxy.config.http.cache.ignore_client_no_cache` -:enumerator:`TS_CONFIG_HTTP_CACHE_IGNORE_SERVER_NO_CACHE` :ts:cv:`proxy.config.http.cache.ignore_server_no_cache` -:enumerator:`TS_CONFIG_HTTP_CACHE_IMS_ON_CLIENT_NO_CACHE` :ts:cv:`proxy.config.http.cache.ims_on_client_no_cache` -:enumerator:`TS_CONFIG_HTTP_CACHE_MAX_OPEN_READ_RETRIES` :ts:cv:`proxy.config.http.cache.max_open_read_retries` -:enumerator:`TS_CONFIG_HTTP_CACHE_MAX_OPEN_WRITE_RETRIES` :ts:cv:`proxy.config.http.cache.max_open_write_retries` -:enumerator:`TS_CONFIG_HTTP_CACHE_MAX_STALE_AGE` :ts:cv:`proxy.config.http.cache.max_stale_age` -:enumerator:`TS_CONFIG_HTTP_CACHE_OPEN_READ_RETRY_TIME` :ts:cv:`proxy.config.http.cache.open_read_retry_time` -:enumerator:`TS_CONFIG_HTTP_CACHE_OPEN_WRITE_FAIL_ACTION` :ts:cv:`proxy.config.http.cache.open_write_fail_action` -:enumerator:`TS_CONFIG_HTTP_CACHE_RANGE_LOOKUP` :ts:cv:`proxy.config.http.cache.range.lookup` -:enumerator:`TS_CONFIG_HTTP_CACHE_RANGE_WRITE` :ts:cv:`proxy.config.http.cache.range.write` -:enumerator:`TS_CONFIG_HTTP_CACHE_REQUIRED_HEADERS` :ts:cv:`proxy.config.http.cache.required_headers` -:enumerator:`TS_CONFIG_HTTP_CACHE_WHEN_TO_REVALIDATE` :ts:cv:`proxy.config.http.cache.when_to_revalidate` -:enumerator:`TS_CONFIG_HTTP_CHUNKING_ENABLED` :ts:cv:`proxy.config.http.chunking_enabled` -:enumerator:`TS_CONFIG_HTTP_CHUNKING_SIZE` :ts:cv:`proxy.config.http.chunking.size` -:enumerator:`TS_CONFIG_HTTP_STRICT_CHUNK_PARSING` :ts:cv:`proxy.config.http.strict_chunk_parsing` -:enumerator:`TS_CONFIG_HTTP_DROP_CHUNKED_TRAILERS` :ts:cv:`proxy.config.http.drop_chunked_trailers` -:enumerator:`TS_CONFIG_HTTP_CONNECT_ATTEMPTS_MAX_RETRIES_DOWN_SERVER` :ts:cv:`proxy.config.http.connect_attempts_max_retries_down_server` -:enumerator:`TS_CONFIG_HTTP_CONNECT_ATTEMPTS_MAX_RETRIES` :ts:cv:`proxy.config.http.connect_attempts_max_retries` -:enumerator:`TS_CONFIG_HTTP_CONNECT_ATTEMPTS_RR_RETRIES` :ts:cv:`proxy.config.http.connect_attempts_rr_retries` -:enumerator:`TS_CONFIG_HTTP_CONNECT_ATTEMPTS_TIMEOUT` :ts:cv:`proxy.config.http.connect_attempts_timeout` -:enumerator:`TS_CONFIG_HTTP_CONNECT_ATTEMPTS_RETRY_BACKOFF_BASE` :ts:cv:`proxy.config.http.connect_attempts_retry_backoff_base` -:enumerator:`TS_CONFIG_HTTP_DEFAULT_BUFFER_SIZE` :ts:cv:`proxy.config.http.default_buffer_size` -:enumerator:`TS_CONFIG_HTTP_DEFAULT_BUFFER_WATER_MARK` :ts:cv:`proxy.config.http.default_buffer_water_mark` -:enumerator:`TS_CONFIG_HTTP_DOC_IN_CACHE_SKIP_DNS` :ts:cv:`proxy.config.http.doc_in_cache_skip_dns` -:enumerator:`TS_CONFIG_HTTP_DOWN_SERVER_CACHE_TIME` :ts:cv:`proxy.config.http.down_server.cache_time` -:enumerator:`TS_CONFIG_HTTP_FLOW_CONTROL_ENABLED` :ts:cv:`proxy.config.http.flow_control.enabled` -:enumerator:`TS_CONFIG_HTTP_FLOW_CONTROL_HIGH_WATER_MARK` :ts:cv:`proxy.config.http.flow_control.high_water` -:enumerator:`TS_CONFIG_HTTP_FLOW_CONTROL_LOW_WATER_MARK` :ts:cv:`proxy.config.http.flow_control.low_water` -:enumerator:`TS_CONFIG_HTTP_FORWARD_CONNECT_METHOD` :ts:cv:`proxy.config.http.forward_connect_method` -:enumerator:`TS_CONFIG_HTTP_FORWARD_PROXY_AUTH_TO_PARENT` :ts:cv:`proxy.config.http.forward.proxy_auth_to_parent` -:enumerator:`TS_CONFIG_HTTP_GLOBAL_USER_AGENT_HEADER` :ts:cv:`proxy.config.http.global_user_agent_header` -:enumerator:`TS_CONFIG_HTTP_INSERT_AGE_IN_RESPONSE` :ts:cv:`proxy.config.http.insert_age_in_response` -:enumerator:`TS_CONFIG_HTTP_INSERT_FORWARDED` :ts:cv:`proxy.config.http.insert_forwarded` -:enumerator:`TS_CONFIG_HTTP_INSERT_REQUEST_VIA_STR` :ts:cv:`proxy.config.http.insert_request_via_str` -:enumerator:`TS_CONFIG_HTTP_INSERT_RESPONSE_VIA_STR` :ts:cv:`proxy.config.http.insert_response_via_str` -:enumerator:`TS_CONFIG_HTTP_INSERT_SQUID_X_FORWARDED_FOR` :ts:cv:`proxy.config.http.insert_squid_x_forwarded_for` -:enumerator:`TS_CONFIG_HTTP_KEEP_ALIVE_ENABLED_IN` :ts:cv:`proxy.config.http.keep_alive_enabled_in` -:enumerator:`TS_CONFIG_HTTP_KEEP_ALIVE_ENABLED_OUT` :ts:cv:`proxy.config.http.keep_alive_enabled_out` -:enumerator:`TS_CONFIG_HTTP_KEEP_ALIVE_NO_ACTIVITY_TIMEOUT_IN` :ts:cv:`proxy.config.http.keep_alive_no_activity_timeout_in` -:enumerator:`TS_CONFIG_HTTP_KEEP_ALIVE_NO_ACTIVITY_TIMEOUT_OUT` :ts:cv:`proxy.config.http.keep_alive_no_activity_timeout_out` -:enumerator:`TS_CONFIG_HTTP_KEEP_ALIVE_POST_OUT` :ts:cv:`proxy.config.http.keep_alive_post_out` -:enumerator:`TS_CONFIG_HTTP_NEGATIVE_CACHING_ENABLED` :ts:cv:`proxy.config.http.negative_caching_enabled` -:enumerator:`TS_CONFIG_HTTP_NEGATIVE_CACHING_LIFETIME` :ts:cv:`proxy.config.http.negative_caching_lifetime` -:enumerator:`TS_CONFIG_HTTP_NEGATIVE_CACHING_LIST` :ts:cv:`proxy.config.http.negative_caching_list` -:enumerator:`TS_CONFIG_HTTP_NEGATIVE_REVALIDATING_ENABLED` :ts:cv:`proxy.config.http.negative_revalidating_enabled` -:enumerator:`TS_CONFIG_HTTP_NEGATIVE_REVALIDATING_LIFETIME` :ts:cv:`proxy.config.http.negative_revalidating_lifetime` -:enumerator:`TS_CONFIG_HTTP_NEGATIVE_REVALIDATING_LIST` :ts:cv:`proxy.config.http.negative_revalidating_list` -:enumerator:`TS_CONFIG_HTTP_NO_DNS_JUST_FORWARD_TO_PARENT` :ts:cv:`proxy.config.http.no_dns_just_forward_to_parent` -:enumerator:`TS_CONFIG_HTTP_NORMALIZE_AE` :ts:cv:`proxy.config.http.normalize_ae` -:enumerator:`TS_CONFIG_HTTP_NUMBER_OF_REDIRECTIONS` :ts:cv:`proxy.config.http.number_of_redirections` -:enumerator:`TS_CONFIG_HTTP_PARENT_PROXY_FAIL_THRESHOLD` :ts:cv:`proxy.config.http.parent_proxy.fail_threshold` -:enumerator:`TS_CONFIG_HTTP_PARENT_PROXY_RETRY_TIME` :ts:cv:`proxy.config.http.parent_proxy.retry_time` -:enumerator:`TS_CONFIG_HTTP_PARENT_PROXY_TOTAL_CONNECT_ATTEMPTS` :ts:cv:`proxy.config.http.parent_proxy.total_connect_attempts` -:enumerator:`TS_CONFIG_HTTP_PER_PARENT_CONNECT_ATTEMPTS` :ts:cv:`proxy.config.http.parent_proxy.per_parent_connect_attempts` -:enumerator:`TS_CONFIG_HTTP_PER_SERVER_CONNECTION_MATCH` :ts:cv:`proxy.config.http.per_server.connection.match` -:enumerator:`TS_CONFIG_HTTP_PER_SERVER_CONNECTION_MAX` :ts:cv:`proxy.config.http.per_server.connection.max` -:enumerator:`TS_CONFIG_HTTP_POST_CHECK_CONTENT_LENGTH_ENABLED` :ts:cv:`proxy.config.http.post.check.content_length.enabled` -:enumerator:`TS_CONFIG_HTTP_REDIRECT_USE_ORIG_CACHE_KEY` :ts:cv:`proxy.config.http.redirect_use_orig_cache_key` -:enumerator:`TS_CONFIG_HTTP_REQUEST_BUFFER_ENABLED` :ts:cv:`proxy.config.http.request_buffer_enabled` -:enumerator:`TS_CONFIG_HTTP_REQUEST_HEADER_MAX_SIZE` :ts:cv:`proxy.config.http.request_header_max_size` -:enumerator:`TS_CONFIG_HTTP_RESPONSE_HEADER_MAX_SIZE` :ts:cv:`proxy.config.http.response_header_max_size` -:enumerator:`TS_CONFIG_HTTP_RESPONSE_SERVER_ENABLED` :ts:cv:`proxy.config.http.response_server_enabled` -:enumerator:`TS_CONFIG_HTTP_RESPONSE_SERVER_STR` :ts:cv:`proxy.config.http.response_server_str` -:enumerator:`TS_CONFIG_HTTP_SEND_HTTP11_REQUESTS` :ts:cv:`proxy.config.http.send_http11_requests` -:enumerator:`TS_CONFIG_HTTP_SERVER_SESSION_SHARING_MATCH` :ts:cv:`proxy.config.http.server_session_sharing.match` -:enumerator:`TS_CONFIG_HTTP_SLOW_LOG_THRESHOLD` :ts:cv:`proxy.config.http.slow.log.threshold` -:enumerator:`TS_CONFIG_HTTP_TRANSACTION_ACTIVE_TIMEOUT_IN` :ts:cv:`proxy.config.http.transaction_active_timeout_in` -:enumerator:`TS_CONFIG_HTTP_TRANSACTION_ACTIVE_TIMEOUT_OUT` :ts:cv:`proxy.config.http.transaction_active_timeout_out` -:enumerator:`TS_CONFIG_HTTP_TRANSACTION_NO_ACTIVITY_TIMEOUT_IN` :ts:cv:`proxy.config.http.transaction_no_activity_timeout_in` -:enumerator:`TS_CONFIG_HTTP_TRANSACTION_NO_ACTIVITY_TIMEOUT_OUT` :ts:cv:`proxy.config.http.transaction_no_activity_timeout_out` -:enumerator:`TS_CONFIG_HTTP_UNCACHEABLE_REQUESTS_BYPASS_PARENT` :ts:cv:`proxy.config.http.uncacheable_requests_bypass_parent` -:enumerator:`TS_CONFIG_NET_SOCK_OPTION_FLAG_OUT` :ts:cv:`proxy.config.net.sock_option_flag_out` -:enumerator:`TS_CONFIG_NET_SOCK_PACKET_MARK_OUT` :ts:cv:`proxy.config.net.sock_packet_mark_out` -:enumerator:`TS_CONFIG_NET_SOCK_PACKET_TOS_OUT` :ts:cv:`proxy.config.net.sock_packet_tos_out` -:enumerator:`TS_CONFIG_NET_SOCK_RECV_BUFFER_SIZE_OUT` :ts:cv:`proxy.config.net.sock_recv_buffer_size_out` -:enumerator:`TS_CONFIG_NET_DEFAULT_INACTIVITY_TIMEOUT` :ts:cv:`proxy.config.net.default_inactivity_timeout` -:enumerator:`TS_CONFIG_NET_SOCK_SEND_BUFFER_SIZE_OUT` :ts:cv:`proxy.config.net.sock_send_buffer_size_out` -:enumerator:`TS_CONFIG_PARENT_FAILURES_UPDATE_HOSTDB` :ts:cv:`proxy.config.http.parent_proxy.mark_down_hostdb` -:enumerator:`TS_CONFIG_SRV_ENABLED` :ts:cv:`proxy.config.srv_enabled` -:enumerator:`TS_CONFIG_SSL_CLIENT_CERT_FILENAME` :ts:cv:`proxy.config.ssl.client.cert.filename` -:enumerator:`TS_CONFIG_SSL_CERT_FILEPATH` :ts:cv:`proxy.config.ssl.client.cert.path` -:enumerator:`TS_CONFIG_SSL_CLIENT_VERIFY_SERVER_PROPERTIES` :ts:cv:`proxy.config.ssl.client.verify.server.properties` -:enumerator:`TS_CONFIG_SSL_CLIENT_VERIFY_SERVER_POLICY` :ts:cv:`proxy.config.ssl.client.verify.server.policy` -:enumerator:`TS_CONFIG_SSL_CLIENT_SNI_POLICY` :ts:cv:`proxy.config.ssl.client.sni_policy` -:enumerator:`TS_CONFIG_SSL_HSTS_INCLUDE_SUBDOMAINS` :ts:cv:`proxy.config.ssl.hsts_include_subdomains` -:enumerator:`TS_CONFIG_SSL_HSTS_MAX_AGE` :ts:cv:`proxy.config.ssl.hsts_max_age` -:enumerator:`TS_CONFIG_URL_REMAP_PRISTINE_HOST_HDR` :ts:cv:`proxy.config.url_remap.pristine_host_hdr` -:enumerator:`TS_CONFIG_WEBSOCKET_ACTIVE_TIMEOUT` :ts:cv:`proxy.config.websocket.active_timeout` -:enumerator:`TS_CONFIG_WEBSOCKET_NO_ACTIVITY_TIMEOUT` :ts:cv:`proxy.config.websocket.no_activity_timeout` -:enumerator:`TS_CONFIG_SSL_CLIENT_CERT_FILENAME` :ts:cv:`proxy.config.ssl.client.cert.filename` -:enumerator:`TS_CONFIG_SSL_CLIENT_PRIVATE_KEY_FILENAME` :ts:cv:`proxy.config.ssl.client.private_key.filename` -:enumerator:`TS_CONFIG_SSL_CLIENT_CA_CERT_FILENAME` :ts:cv:`proxy.config.ssl.client.CA.cert.filename` -:enumerator:`TS_CONFIG_HTTP_HOST_RESOLUTION_PREFERENCE` :ts:cv:`proxy.config.hostdb.ip_resolve` -:enumerator:`TS_CONFIG_PLUGIN_VC_DEFAULT_BUFFER_INDEX` :ts:cv:`proxy.config.plugin.vc.default_buffer_index` -:enumerator:`TS_CONFIG_PLUGIN_VC_DEFAULT_BUFFER_WATER_MARK` :ts:cv:`proxy.config.plugin.vc.default_buffer_water_mark` -:enumerator:`TS_CONFIG_NET_SOCK_NOTSENT_LOWAT` :ts:cv:`proxy.config.net.sock_notsent_lowat` -:enumerator:`TS_CONFIG_BODY_FACTORY_RESPONSE_SUPPRESSION_MODE` :ts:cv:`proxy.config.body_factory.response_suppression_mode` -:enumerator:`TS_CONFIG_HTTP_CACHE_POST_METHOD` :ts:cv:`proxy.config.http.cache.post_method` -:enumerator:`TS_CONFIG_HTTP_CACHE_TARGETED_CACHE_CONTROL_HEADERS` :ts:cv:`proxy.config.http.cache.targeted_cache_control_headers` -====================================================================== ==================================================================== +======================================================================== ==================================================================== +TSOverridableConfigKey Value Configuration Value +======================================================================== ==================================================================== +:enumerator:`TS_CONFIG_BODY_FACTORY_TEMPLATE_BASE` :ts:cv:`proxy.config.body_factory.template_base` +:enumerator:`TS_CONFIG_HTTP_ALLOW_HALF_OPEN` :ts:cv:`proxy.config.http.allow_half_open` +:enumerator:`TS_CONFIG_HTTP_ALLOW_MULTI_RANGE` :ts:cv:`proxy.config.http.allow_multi_range` +:enumerator:`TS_CONFIG_HTTP_ANONYMIZE_INSERT_CLIENT_IP` :ts:cv:`proxy.config.http.insert_client_ip` +:enumerator:`TS_CONFIG_HTTP_ANONYMIZE_REMOVE_CLIENT_IP` :ts:cv:`proxy.config.http.anonymize_remove_client_ip` +:enumerator:`TS_CONFIG_HTTP_ANONYMIZE_REMOVE_COOKIE` :ts:cv:`proxy.config.http.anonymize_remove_cookie` +:enumerator:`TS_CONFIG_HTTP_ANONYMIZE_REMOVE_FROM` :ts:cv:`proxy.config.http.anonymize_remove_from` +:enumerator:`TS_CONFIG_HTTP_ANONYMIZE_REMOVE_REFERER` :ts:cv:`proxy.config.http.anonymize_remove_referer` +:enumerator:`TS_CONFIG_HTTP_ANONYMIZE_REMOVE_USER_AGENT` :ts:cv:`proxy.config.http.anonymize_remove_user_agent` +:enumerator:`TS_CONFIG_HTTP_ATTACH_SERVER_SESSION_TO_CLIENT` :ts:cv:`proxy.config.http.attach_server_session_to_client` +:enumerator:`TS_CONFIG_HTTP_MAX_PROXY_CYCLES` :ts:cv:`proxy.config.http.max_proxy_cycles` +:enumerator:`TS_CONFIG_HTTP_AUTH_SERVER_SESSION_PRIVATE` :ts:cv:`proxy.config.http.auth_server_session_private` +:enumerator:`TS_CONFIG_HTTP_BACKGROUND_FILL_ACTIVE_TIMEOUT` :ts:cv:`proxy.config.http.background_fill_active_timeout` +:enumerator:`TS_CONFIG_HTTP_BACKGROUND_FILL_COMPLETED_THRESHOLD` :ts:cv:`proxy.config.http.background_fill_completed_threshold` +:enumerator:`TS_CONFIG_HTTP_CACHE_CACHE_RESPONSES_TO_COOKIES` :ts:cv:`proxy.config.http.cache.cache_responses_to_cookies` +:enumerator:`TS_CONFIG_HTTP_CACHE_CACHE_URLS_THAT_LOOK_DYNAMIC` :ts:cv:`proxy.config.http.cache.cache_urls_that_look_dynamic` +:enumerator:`TS_CONFIG_HTTP_CACHE_IGNORE_QUERY` :ts:cv:`proxy.config.http.cache.ignore_query` +:enumerator:`TS_CONFIG_HTTP_CACHE_GENERATION` :ts:cv:`proxy.config.http.cache.generation` +:enumerator:`TS_CONFIG_HTTP_CACHE_GUARANTEED_MAX_LIFETIME` :ts:cv:`proxy.config.http.cache.guaranteed_max_lifetime` +:enumerator:`TS_CONFIG_HTTP_CACHE_GUARANTEED_MIN_LIFETIME` :ts:cv:`proxy.config.http.cache.guaranteed_min_lifetime` +:enumerator:`TS_CONFIG_HTTP_CACHE_HEURISTIC_LM_FACTOR` :ts:cv:`proxy.config.http.cache.heuristic_lm_factor` +:enumerator:`TS_CONFIG_HTTP_CACHE_HEURISTIC_MAX_LIFETIME` :ts:cv:`proxy.config.http.cache.heuristic_max_lifetime` +:enumerator:`TS_CONFIG_HTTP_CACHE_HEURISTIC_MIN_LIFETIME` :ts:cv:`proxy.config.http.cache.heuristic_min_lifetime` +:enumerator:`TS_CONFIG_HTTP_CACHE_HTTP` :ts:cv:`proxy.config.http.cache.http` +:enumerator:`TS_CONFIG_HTTP_CACHE_IGNORE_ACCEPT_CHARSET_MISMATCH` :ts:cv:`proxy.config.http.cache.ignore_accept_charset_mismatch` +:enumerator:`TS_CONFIG_HTTP_CACHE_IGNORE_ACCEPT_ENCODING_MISMATCH` :ts:cv:`proxy.config.http.cache.ignore_accept_encoding_mismatch` +:enumerator:`TS_CONFIG_HTTP_CACHE_IGNORE_ACCEPT_LANGUAGE_MISMATCH` :ts:cv:`proxy.config.http.cache.ignore_accept_language_mismatch` +:enumerator:`TS_CONFIG_HTTP_CACHE_IGNORE_ACCEPT_MISMATCH` :ts:cv:`proxy.config.http.cache.ignore_accept_mismatch` +:enumerator:`TS_CONFIG_HTTP_CACHE_IGNORE_AUTHENTICATION` :ts:cv:`proxy.config.http.cache.ignore_authentication` +:enumerator:`TS_CONFIG_HTTP_CACHE_IGNORE_CLIENT_CC_MAX_AGE` :ts:cv:`proxy.config.http.cache.ignore_client_cc_max_age` +:enumerator:`TS_CONFIG_HTTP_CACHE_IGNORE_CLIENT_NO_CACHE` :ts:cv:`proxy.config.http.cache.ignore_client_no_cache` +:enumerator:`TS_CONFIG_HTTP_CACHE_IGNORE_SERVER_NO_CACHE` :ts:cv:`proxy.config.http.cache.ignore_server_no_cache` +:enumerator:`TS_CONFIG_HTTP_CACHE_IMS_ON_CLIENT_NO_CACHE` :ts:cv:`proxy.config.http.cache.ims_on_client_no_cache` +:enumerator:`TS_CONFIG_HTTP_CACHE_MAX_OPEN_READ_RETRIES` :ts:cv:`proxy.config.http.cache.max_open_read_retries` +:enumerator:`TS_CONFIG_HTTP_CACHE_MAX_OPEN_WRITE_RETRIES` :ts:cv:`proxy.config.http.cache.max_open_write_retries` +:enumerator:`TS_CONFIG_HTTP_CACHE_MAX_STALE_AGE` :ts:cv:`proxy.config.http.cache.max_stale_age` +:enumerator:`TS_CONFIG_HTTP_CACHE_OPEN_READ_RETRY_TIME` :ts:cv:`proxy.config.http.cache.open_read_retry_time` +:enumerator:`TS_CONFIG_HTTP_CACHE_OPEN_WRITE_FAIL_ACTION` :ts:cv:`proxy.config.http.cache.open_write_fail_action` +:enumerator:`TS_CONFIG_HTTP_CACHE_RANGE_LOOKUP` :ts:cv:`proxy.config.http.cache.range.lookup` +:enumerator:`TS_CONFIG_HTTP_CACHE_RANGE_WRITE` :ts:cv:`proxy.config.http.cache.range.write` +:enumerator:`TS_CONFIG_HTTP_CACHE_REQUIRED_HEADERS` :ts:cv:`proxy.config.http.cache.required_headers` +:enumerator:`TS_CONFIG_HTTP_CACHE_WHEN_TO_REVALIDATE` :ts:cv:`proxy.config.http.cache.when_to_revalidate` +:enumerator:`TS_CONFIG_HTTP_CHUNKING_ENABLED` :ts:cv:`proxy.config.http.chunking_enabled` +:enumerator:`TS_CONFIG_HTTP_CHUNKING_SIZE` :ts:cv:`proxy.config.http.chunking.size` +:enumerator:`TS_CONFIG_HTTP_STRICT_CHUNK_PARSING` :ts:cv:`proxy.config.http.strict_chunk_parsing` +:enumerator:`TS_CONFIG_HTTP_DROP_CHUNKED_TRAILERS` :ts:cv:`proxy.config.http.drop_chunked_trailers` +:enumerator:`TS_CONFIG_HTTP_CONNECT_ATTEMPTS_MAX_RETRIES_DOWN_SERVER` :ts:cv:`proxy.config.http.connect_attempts_max_retries_down_server` +:enumerator:`TS_CONFIG_HTTP_CONNECT_ATTEMPTS_MAX_RETRIES_SUSPECT_SERVER` :ts:cv:`proxy.config.http.connect_attempts_max_retries_suspect_server` +:enumerator:`TS_CONFIG_HTTP_CONNECT_ATTEMPTS_MAX_RETRIES` :ts:cv:`proxy.config.http.connect_attempts_max_retries` +:enumerator:`TS_CONFIG_HTTP_CONNECT_ATTEMPTS_RR_RETRIES` :ts:cv:`proxy.config.http.connect_attempts_rr_retries` +:enumerator:`TS_CONFIG_HTTP_CONNECT_ATTEMPTS_TIMEOUT` :ts:cv:`proxy.config.http.connect_attempts_timeout` +:enumerator:`TS_CONFIG_HTTP_CONNECT_ATTEMPTS_RETRY_BACKOFF_BASE` :ts:cv:`proxy.config.http.connect_attempts_retry_backoff_base` +:enumerator:`TS_CONFIG_HTTP_DEFAULT_BUFFER_SIZE` :ts:cv:`proxy.config.http.default_buffer_size` +:enumerator:`TS_CONFIG_HTTP_DEFAULT_BUFFER_WATER_MARK` :ts:cv:`proxy.config.http.default_buffer_water_mark` +:enumerator:`TS_CONFIG_HTTP_DOC_IN_CACHE_SKIP_DNS` :ts:cv:`proxy.config.http.doc_in_cache_skip_dns` +:enumerator:`TS_CONFIG_HTTP_DOWN_SERVER_CACHE_TIME` :ts:cv:`proxy.config.http.down_server.cache_time` +:enumerator:`TS_CONFIG_HTTP_FLOW_CONTROL_ENABLED` :ts:cv:`proxy.config.http.flow_control.enabled` +:enumerator:`TS_CONFIG_HTTP_FLOW_CONTROL_HIGH_WATER_MARK` :ts:cv:`proxy.config.http.flow_control.high_water` +:enumerator:`TS_CONFIG_HTTP_FLOW_CONTROL_LOW_WATER_MARK` :ts:cv:`proxy.config.http.flow_control.low_water` +:enumerator:`TS_CONFIG_HTTP_FORWARD_CONNECT_METHOD` :ts:cv:`proxy.config.http.forward_connect_method` +:enumerator:`TS_CONFIG_HTTP_FORWARD_PROXY_AUTH_TO_PARENT` :ts:cv:`proxy.config.http.forward.proxy_auth_to_parent` +:enumerator:`TS_CONFIG_HTTP_GLOBAL_USER_AGENT_HEADER` :ts:cv:`proxy.config.http.global_user_agent_header` +:enumerator:`TS_CONFIG_HTTP_INSERT_AGE_IN_RESPONSE` :ts:cv:`proxy.config.http.insert_age_in_response` +:enumerator:`TS_CONFIG_HTTP_INSERT_FORWARDED` :ts:cv:`proxy.config.http.insert_forwarded` +:enumerator:`TS_CONFIG_HTTP_INSERT_REQUEST_VIA_STR` :ts:cv:`proxy.config.http.insert_request_via_str` +:enumerator:`TS_CONFIG_HTTP_INSERT_RESPONSE_VIA_STR` :ts:cv:`proxy.config.http.insert_response_via_str` +:enumerator:`TS_CONFIG_HTTP_INSERT_SQUID_X_FORWARDED_FOR` :ts:cv:`proxy.config.http.insert_squid_x_forwarded_for` +:enumerator:`TS_CONFIG_HTTP_KEEP_ALIVE_ENABLED_IN` :ts:cv:`proxy.config.http.keep_alive_enabled_in` +:enumerator:`TS_CONFIG_HTTP_KEEP_ALIVE_ENABLED_OUT` :ts:cv:`proxy.config.http.keep_alive_enabled_out` +:enumerator:`TS_CONFIG_HTTP_KEEP_ALIVE_NO_ACTIVITY_TIMEOUT_IN` :ts:cv:`proxy.config.http.keep_alive_no_activity_timeout_in` +:enumerator:`TS_CONFIG_HTTP_KEEP_ALIVE_NO_ACTIVITY_TIMEOUT_OUT` :ts:cv:`proxy.config.http.keep_alive_no_activity_timeout_out` +:enumerator:`TS_CONFIG_HTTP_KEEP_ALIVE_POST_OUT` :ts:cv:`proxy.config.http.keep_alive_post_out` +:enumerator:`TS_CONFIG_HTTP_NEGATIVE_CACHING_ENABLED` :ts:cv:`proxy.config.http.negative_caching_enabled` +:enumerator:`TS_CONFIG_HTTP_NEGATIVE_CACHING_LIFETIME` :ts:cv:`proxy.config.http.negative_caching_lifetime` +:enumerator:`TS_CONFIG_HTTP_NEGATIVE_CACHING_LIST` :ts:cv:`proxy.config.http.negative_caching_list` +:enumerator:`TS_CONFIG_HTTP_NEGATIVE_REVALIDATING_ENABLED` :ts:cv:`proxy.config.http.negative_revalidating_enabled` +:enumerator:`TS_CONFIG_HTTP_NEGATIVE_REVALIDATING_LIFETIME` :ts:cv:`proxy.config.http.negative_revalidating_lifetime` +:enumerator:`TS_CONFIG_HTTP_NEGATIVE_REVALIDATING_LIST` :ts:cv:`proxy.config.http.negative_revalidating_list` +:enumerator:`TS_CONFIG_HTTP_NO_DNS_JUST_FORWARD_TO_PARENT` :ts:cv:`proxy.config.http.no_dns_just_forward_to_parent` +:enumerator:`TS_CONFIG_HTTP_NORMALIZE_AE` :ts:cv:`proxy.config.http.normalize_ae` +:enumerator:`TS_CONFIG_HTTP_NUMBER_OF_REDIRECTIONS` :ts:cv:`proxy.config.http.number_of_redirections` +:enumerator:`TS_CONFIG_HTTP_PARENT_PROXY_FAIL_THRESHOLD` :ts:cv:`proxy.config.http.parent_proxy.fail_threshold` +:enumerator:`TS_CONFIG_HTTP_PARENT_PROXY_RETRY_TIME` :ts:cv:`proxy.config.http.parent_proxy.retry_time` +:enumerator:`TS_CONFIG_HTTP_PARENT_PROXY_TOTAL_CONNECT_ATTEMPTS` :ts:cv:`proxy.config.http.parent_proxy.total_connect_attempts` +:enumerator:`TS_CONFIG_HTTP_PER_PARENT_CONNECT_ATTEMPTS` :ts:cv:`proxy.config.http.parent_proxy.per_parent_connect_attempts` +:enumerator:`TS_CONFIG_HTTP_PER_SERVER_CONNECTION_MATCH` :ts:cv:`proxy.config.http.per_server.connection.match` +:enumerator:`TS_CONFIG_HTTP_PER_SERVER_CONNECTION_MAX` :ts:cv:`proxy.config.http.per_server.connection.max` +:enumerator:`TS_CONFIG_HTTP_POST_CHECK_CONTENT_LENGTH_ENABLED` :ts:cv:`proxy.config.http.post.check.content_length.enabled` +:enumerator:`TS_CONFIG_HTTP_REDIRECT_USE_ORIG_CACHE_KEY` :ts:cv:`proxy.config.http.redirect_use_orig_cache_key` +:enumerator:`TS_CONFIG_HTTP_REQUEST_BUFFER_ENABLED` :ts:cv:`proxy.config.http.request_buffer_enabled` +:enumerator:`TS_CONFIG_HTTP_REQUEST_HEADER_MAX_SIZE` :ts:cv:`proxy.config.http.request_header_max_size` +:enumerator:`TS_CONFIG_HTTP_RESPONSE_HEADER_MAX_SIZE` :ts:cv:`proxy.config.http.response_header_max_size` +:enumerator:`TS_CONFIG_HTTP_RESPONSE_SERVER_ENABLED` :ts:cv:`proxy.config.http.response_server_enabled` +:enumerator:`TS_CONFIG_HTTP_RESPONSE_SERVER_STR` :ts:cv:`proxy.config.http.response_server_str` +:enumerator:`TS_CONFIG_HTTP_SEND_HTTP11_REQUESTS` :ts:cv:`proxy.config.http.send_http11_requests` +:enumerator:`TS_CONFIG_HTTP_SERVER_SESSION_SHARING_MATCH` :ts:cv:`proxy.config.http.server_session_sharing.match` +:enumerator:`TS_CONFIG_HTTP_SLOW_LOG_THRESHOLD` :ts:cv:`proxy.config.http.slow.log.threshold` +:enumerator:`TS_CONFIG_HTTP_TRANSACTION_ACTIVE_TIMEOUT_IN` :ts:cv:`proxy.config.http.transaction_active_timeout_in` +:enumerator:`TS_CONFIG_HTTP_TRANSACTION_ACTIVE_TIMEOUT_OUT` :ts:cv:`proxy.config.http.transaction_active_timeout_out` +:enumerator:`TS_CONFIG_HTTP_TRANSACTION_NO_ACTIVITY_TIMEOUT_IN` :ts:cv:`proxy.config.http.transaction_no_activity_timeout_in` +:enumerator:`TS_CONFIG_HTTP_TRANSACTION_NO_ACTIVITY_TIMEOUT_OUT` :ts:cv:`proxy.config.http.transaction_no_activity_timeout_out` +:enumerator:`TS_CONFIG_HTTP_UNCACHEABLE_REQUESTS_BYPASS_PARENT` :ts:cv:`proxy.config.http.uncacheable_requests_bypass_parent` +:enumerator:`TS_CONFIG_NET_SOCK_OPTION_FLAG_OUT` :ts:cv:`proxy.config.net.sock_option_flag_out` +:enumerator:`TS_CONFIG_NET_SOCK_PACKET_MARK_OUT` :ts:cv:`proxy.config.net.sock_packet_mark_out` +:enumerator:`TS_CONFIG_NET_SOCK_PACKET_TOS_OUT` :ts:cv:`proxy.config.net.sock_packet_tos_out` +:enumerator:`TS_CONFIG_NET_SOCK_RECV_BUFFER_SIZE_OUT` :ts:cv:`proxy.config.net.sock_recv_buffer_size_out` +:enumerator:`TS_CONFIG_NET_DEFAULT_INACTIVITY_TIMEOUT` :ts:cv:`proxy.config.net.default_inactivity_timeout` +:enumerator:`TS_CONFIG_NET_SOCK_SEND_BUFFER_SIZE_OUT` :ts:cv:`proxy.config.net.sock_send_buffer_size_out` +:enumerator:`TS_CONFIG_PARENT_FAILURES_UPDATE_HOSTDB` :ts:cv:`proxy.config.http.parent_proxy.mark_down_hostdb` +:enumerator:`TS_CONFIG_SRV_ENABLED` :ts:cv:`proxy.config.srv_enabled` +:enumerator:`TS_CONFIG_SSL_CLIENT_CERT_FILENAME` :ts:cv:`proxy.config.ssl.client.cert.filename` +:enumerator:`TS_CONFIG_SSL_CERT_FILEPATH` :ts:cv:`proxy.config.ssl.client.cert.path` +:enumerator:`TS_CONFIG_SSL_CLIENT_VERIFY_SERVER_PROPERTIES` :ts:cv:`proxy.config.ssl.client.verify.server.properties` +:enumerator:`TS_CONFIG_SSL_CLIENT_VERIFY_SERVER_POLICY` :ts:cv:`proxy.config.ssl.client.verify.server.policy` +:enumerator:`TS_CONFIG_SSL_CLIENT_SNI_POLICY` :ts:cv:`proxy.config.ssl.client.sni_policy` +:enumerator:`TS_CONFIG_SSL_HSTS_INCLUDE_SUBDOMAINS` :ts:cv:`proxy.config.ssl.hsts_include_subdomains` +:enumerator:`TS_CONFIG_SSL_HSTS_MAX_AGE` :ts:cv:`proxy.config.ssl.hsts_max_age` +:enumerator:`TS_CONFIG_URL_REMAP_PRISTINE_HOST_HDR` :ts:cv:`proxy.config.url_remap.pristine_host_hdr` +:enumerator:`TS_CONFIG_WEBSOCKET_ACTIVE_TIMEOUT` :ts:cv:`proxy.config.websocket.active_timeout` +:enumerator:`TS_CONFIG_WEBSOCKET_NO_ACTIVITY_TIMEOUT` :ts:cv:`proxy.config.websocket.no_activity_timeout` +:enumerator:`TS_CONFIG_SSL_CLIENT_CERT_FILENAME` :ts:cv:`proxy.config.ssl.client.cert.filename` +:enumerator:`TS_CONFIG_SSL_CLIENT_PRIVATE_KEY_FILENAME` :ts:cv:`proxy.config.ssl.client.private_key.filename` +:enumerator:`TS_CONFIG_SSL_CLIENT_CA_CERT_FILENAME` :ts:cv:`proxy.config.ssl.client.CA.cert.filename` +:enumerator:`TS_CONFIG_HTTP_HOST_RESOLUTION_PREFERENCE` :ts:cv:`proxy.config.hostdb.ip_resolve` +:enumerator:`TS_CONFIG_PLUGIN_VC_DEFAULT_BUFFER_INDEX` :ts:cv:`proxy.config.plugin.vc.default_buffer_index` +:enumerator:`TS_CONFIG_PLUGIN_VC_DEFAULT_BUFFER_WATER_MARK` :ts:cv:`proxy.config.plugin.vc.default_buffer_water_mark` +:enumerator:`TS_CONFIG_NET_SOCK_NOTSENT_LOWAT` :ts:cv:`proxy.config.net.sock_notsent_lowat` +:enumerator:`TS_CONFIG_BODY_FACTORY_RESPONSE_SUPPRESSION_MODE` :ts:cv:`proxy.config.body_factory.response_suppression_mode` +:enumerator:`TS_CONFIG_HTTP_CACHE_POST_METHOD` :ts:cv:`proxy.config.http.cache.post_method` +:enumerator:`TS_CONFIG_HTTP_CACHE_TARGETED_CACHE_CONTROL_HEADERS` :ts:cv:`proxy.config.http.cache.targeted_cache_control_headers` +======================================================================== ==================================================================== Examples ======== diff --git a/doc/developer-guide/api/types/TSOverridableConfigKey.en.rst b/doc/developer-guide/api/types/TSOverridableConfigKey.en.rst index 56d325e6192..298d48f45e3 100644 --- a/doc/developer-guide/api/types/TSOverridableConfigKey.en.rst +++ b/doc/developer-guide/api/types/TSOverridableConfigKey.en.rst @@ -79,6 +79,7 @@ Enumeration Members .. enumerator:: TS_CONFIG_HTTP_ORIGIN_MAX_CONNECTIONS .. enumerator:: TS_CONFIG_HTTP_CONNECT_ATTEMPTS_MAX_RETRIES .. enumerator:: TS_CONFIG_HTTP_CONNECT_ATTEMPTS_MAX_RETRIES_DOWN_SERVER +.. enumerator:: TS_CONFIG_HTTP_CONNECT_ATTEMPTS_MAX_RETRIES_SUSPECT_SERVER .. enumerator:: TS_CONFIG_HTTP_CONNECT_ATTEMPTS_RR_RETRIES .. enumerator:: TS_CONFIG_HTTP_CONNECT_ATTEMPTS_TIMEOUT .. enumerator:: TS_CONFIG_HTTP_CONNECT_ATTEMPTS_RETRY_BACKOFF_BASE diff --git a/doc/release-notes/upgrading.en.rst b/doc/release-notes/upgrading.en.rst index 4fde3d3c7fa..ed7c7ddee3b 100644 --- a/doc/release-notes/upgrading.en.rst +++ b/doc/release-notes/upgrading.en.rst @@ -146,6 +146,11 @@ The following :file:`records.yaml` changes have been made: - The records.yaml entry ``proxy.config.http.down_server.abort_threshold`` has been removed. - The records.yaml entry ``proxy.config.http.connect_attempts_max_retries_dead_server`` has been renamed to :ts:cv:`proxy.config.http.connect_attempts_max_retries_down_server`. +- The records.yaml entry ``proxy.config.http.connect_attempts_max_retries_down_server`` is now deprecated in favor of + :ts:cv:`proxy.config.http.connect_attempts_max_retries_suspect_server`. The new name aligns with the + ``HostDBInfo::State::SUSPECT`` state it actually applies to (a recovering origin allowed a limited probe budget after + :ts:cv:`proxy.config.http.down_server.cache_time` elapses). When only the deprecated record is set, its value is mirrored + forward to the new record and a warning is logged. When both are set, the new record wins. - The entry ``proxy.config.http.connect.dead.policy`` has been renamed to :ts:cv:`proxy.config.http.connect.down.policy`. - The records.yaml entry ``proxy.config.http.parent_proxy.connect_attempts_timeout`` and ``proxy.config.http.post_connect_attempts_timeout`` have been removed. Instead use diff --git a/include/cripts/Configs.hpp b/include/cripts/Configs.hpp index 6826f64247f..de2a613b1e8 100644 --- a/include/cripts/Configs.hpp +++ b/include/cripts/Configs.hpp @@ -154,6 +154,8 @@ class Proxy cripts::IntConfig connect_attempts_max_retries{"proxy.config.http.connect_attempts_max_retries"}; cripts::IntConfig connect_attempts_max_retries_down_server{"proxy.config.http.connect_attempts_max_retries_down_server"}; + cripts::IntConfig connect_attempts_max_retries_suspect_server{ + "proxy.config.http.connect_attempts_max_retries_suspect_server"}; cripts::IntConfig connect_attempts_rr_retries{"proxy.config.http.connect_attempts_rr_retries"}; cripts::IntConfig connect_attempts_timeout{"proxy.config.http.connect_attempts_timeout"}; cripts::IntConfig default_buffer_size{"proxy.config.http.default_buffer_size"}; diff --git a/include/iocore/hostdb/HostDBProcessor.h b/include/iocore/hostdb/HostDBProcessor.h index c369aace4fa..1a274a10e02 100644 --- a/include/iocore/hostdb/HostDBProcessor.h +++ b/include/iocore/hostdb/HostDBProcessor.h @@ -582,7 +582,7 @@ struct ResolveInfo { bool mark_active_server_up(); /// Select / resolve to the next RR entry for the record. - bool select_next_rr(); + bool select_next_rr(ts_time now, ts_seconds fail_window); bool is_srv() const; }; diff --git a/include/proxy/http/HttpConfig.h b/include/proxy/http/HttpConfig.h index 05a7cc511f3..f357309c9dd 100644 --- a/include/proxy/http/HttpConfig.h +++ b/include/proxy/http/HttpConfig.h @@ -706,11 +706,11 @@ struct OverridableHttpConfigParams { //////////////////////////////////// // origin server connect attempts // //////////////////////////////////// - MgmtInt connect_attempts_max_retries = 0; - MgmtInt connect_attempts_max_retries_down_server = 3; - MgmtInt connect_attempts_rr_retries = 3; - MgmtInt connect_attempts_timeout = 30; - MgmtInt connect_attempts_retry_backoff_base = 0; + MgmtInt connect_attempts_max_retries = 0; + MgmtInt connect_attempts_max_retries_suspect_server = 1; + MgmtInt connect_attempts_rr_retries = 3; + MgmtInt connect_attempts_timeout = 30; + MgmtInt connect_attempts_retry_backoff_base = 0; MgmtInt connect_down_policy = 2; diff --git a/include/proxy/http/HttpTransact.h b/include/proxy/http/HttpTransact.h index a6d48a12f7f..9215f5508e9 100644 --- a/include/proxy/http/HttpTransact.h +++ b/include/proxy/http/HttpTransact.h @@ -1003,7 +1003,6 @@ class HttpTransact static void Forbidden(State *s); static void SelfLoop(State *s); static void TooEarly(State *s); - static void OriginDown(State *s); static void PostActiveTimeoutResponse(State *s); static void PostInactiveTimeoutResponse(State *s); static void DecideCacheLookup(State *s); @@ -1034,7 +1033,7 @@ class HttpTransact static void handle_response_from_parent_plugin(State *s); static void handle_response_from_server(State *s); static void delete_server_rr_entry(State *s, int max_retries); - static void retry_server_connection_not_open(State *s, ServerState_t conn_state, unsigned max_retries); + static void retry_server_connection_not_open(State *s, unsigned max_retries); static void error_log_connection_failure(State *s, ServerState_t conn_state); static void handle_server_connection_not_open(State *s); static void handle_forward_server_connection_open(State *s); @@ -1078,6 +1077,8 @@ class HttpTransact static bool handle_trace_and_options_requests(State *s, HTTPHdr *incoming_hdr); static void bootstrap_state_variables_from_request(State *s, HTTPHdr *incoming_request); + static uint8_t origin_server_connect_attempts_max_retries(State *s); + // WARNING: this function may be called multiple times for the same transaction. // static void initialize_state_variables_from_request(State *s, HTTPHdr *obsolete_incoming_request); diff --git a/include/proxy/http/OverridableConfigDefs.h b/include/proxy/http/OverridableConfigDefs.h index d70c4c54caa..58b3b10a035 100644 --- a/include/proxy/http/OverridableConfigDefs.h +++ b/include/proxy/http/OverridableConfigDefs.h @@ -160,7 +160,8 @@ X(HTTP_TRANSACTION_NO_ACTIVITY_TIMEOUT_OUT, transaction_no_activity_timeout_out, "proxy.config.http.transaction_no_activity_timeout_out", INT, GENERIC) \ X(HTTP_TRANSACTION_ACTIVE_TIMEOUT_OUT, transaction_active_timeout_out, "proxy.config.http.transaction_active_timeout_out", INT, GENERIC) \ X(HTTP_CONNECT_ATTEMPTS_MAX_RETRIES, connect_attempts_max_retries, "proxy.config.http.connect_attempts_max_retries", INT, GENERIC) \ - X(HTTP_CONNECT_ATTEMPTS_MAX_RETRIES_DOWN_SERVER, connect_attempts_max_retries_down_server, "proxy.config.http.connect_attempts_max_retries_down_server", INT, GENERIC) \ + X(HTTP_CONNECT_ATTEMPTS_MAX_RETRIES_DOWN_SERVER, connect_attempts_max_retries_suspect_server, "proxy.config.http.connect_attempts_max_retries_down_server", INT, GENERIC) \ + X(HTTP_CONNECT_ATTEMPTS_MAX_RETRIES_SUSPECT_SERVER, connect_attempts_max_retries_suspect_server, "proxy.config.http.connect_attempts_max_retries_suspect_server", INT, GENERIC) \ X(HTTP_CONNECT_ATTEMPTS_RR_RETRIES, connect_attempts_rr_retries, "proxy.config.http.connect_attempts_rr_retries", INT, GENERIC) \ X(HTTP_CONNECT_ATTEMPTS_TIMEOUT, connect_attempts_timeout, "proxy.config.http.connect_attempts_timeout", INT, GENERIC) \ X(HTTP_DOWN_SERVER_CACHE_TIME, down_server_timeout, "proxy.config.http.down_server.cache_time", INT, HttpDownServerCacheTimeConv) \ diff --git a/include/ts/apidefs.h.in b/include/ts/apidefs.h.in index f458884779e..284e1c94785 100644 --- a/include/ts/apidefs.h.in +++ b/include/ts/apidefs.h.in @@ -825,6 +825,7 @@ enum TSOverridableConfigKey { TS_CONFIG_HTTP_TRANSACTION_ACTIVE_TIMEOUT_OUT, TS_CONFIG_HTTP_CONNECT_ATTEMPTS_MAX_RETRIES, TS_CONFIG_HTTP_CONNECT_ATTEMPTS_MAX_RETRIES_DOWN_SERVER, + TS_CONFIG_HTTP_CONNECT_ATTEMPTS_MAX_RETRIES_SUSPECT_SERVER, TS_CONFIG_HTTP_CONNECT_ATTEMPTS_RR_RETRIES, TS_CONFIG_HTTP_CONNECT_ATTEMPTS_TIMEOUT, TS_CONFIG_HTTP_DOWN_SERVER_CACHE_TIME, diff --git a/src/iocore/hostdb/HostDB.cc b/src/iocore/hostdb/HostDB.cc index ec7473e2929..0eadbf8cee6 100644 --- a/src/iocore/hostdb/HostDB.cc +++ b/src/iocore/hostdb/HostDB.cc @@ -1706,13 +1706,19 @@ ResolveInfo::set_active(HostDBInfo *info) } bool -ResolveInfo::select_next_rr() +ResolveInfo::select_next_rr(ts_time now, ts_seconds fail_window) { if (active) { if (auto rr_info{this->record->rr_info()}; rr_info.count() > 1) { - unsigned limit = active - rr_info.data(), idx = (limit + 1) % rr_info.count(); - while ((idx = (idx + 1) % rr_info.count()) != limit && !rr_info[idx].is_up()) {} - active = &rr_info[idx]; + const unsigned limit = active - rr_info.data(); + size_t idx = (limit + 1) % rr_info.count(); + for (; idx != limit; idx = (idx + 1) % rr_info.count()) { + if (!rr_info[idx].is_down(now, fail_window)) { + active = &rr_info[idx]; + break; + } + } + return idx != limit; // if the active record was actually changed. } } diff --git a/src/proxy/http/HttpConfig.cc b/src/proxy/http/HttpConfig.cc index aa9775eda13..2bfb487b439 100644 --- a/src/proxy/http/HttpConfig.cc +++ b/src/proxy/http/HttpConfig.cc @@ -1043,8 +1043,8 @@ HttpConfig::startup() HttpEstablishStaticConfigFloat(c.oride.background_fill_threshold, "proxy.config.http.background_fill_completed_threshold"); HttpEstablishStaticConfigLongLong(c.oride.connect_attempts_max_retries, "proxy.config.http.connect_attempts_max_retries"); - HttpEstablishStaticConfigLongLong(c.oride.connect_attempts_max_retries_down_server, - "proxy.config.http.connect_attempts_max_retries_down_server"); + HttpEstablishStaticConfigLongLong(c.oride.connect_attempts_max_retries_suspect_server, + "proxy.config.http.connect_attempts_max_retries_suspect_server"); HttpEstablishStaticConfigLongLong(c.oride.connect_attempts_retry_backoff_base, "proxy.config.http.connect_attempts_retry_backoff_base"); @@ -1340,14 +1340,46 @@ HttpConfig::reconfigure() params->oride.background_fill_active_timeout = m_master.oride.background_fill_active_timeout; params->oride.background_fill_threshold = m_master.oride.background_fill_threshold; - params->oride.connect_attempts_max_retries = m_master.oride.connect_attempts_max_retries; - params->oride.connect_attempts_max_retries_down_server = m_master.oride.connect_attempts_max_retries_down_server; + params->oride.connect_attempts_max_retries = m_master.oride.connect_attempts_max_retries; + params->oride.connect_attempts_max_retries_suspect_server = m_master.oride.connect_attempts_max_retries_suspect_server; + + // Deprecation handling for connect_attempts_max_retries_down_server: if the operator explicitly set the deprecated record + // but did not set the replacement, mirror the value forward so existing configs keep working. If both are set, the new + // record wins and the deprecated one is ignored. Always warn when the deprecated record is explicitly set. The deprecated + // record has no bound struct field; fetch its value on demand from RecCore. + { + RecSourceT old_src = REC_SOURCE_NULL; + RecSourceT new_src = REC_SOURCE_NULL; + RecGetRecordSource("proxy.config.http.connect_attempts_max_retries_down_server", &old_src); + RecGetRecordSource("proxy.config.http.connect_attempts_max_retries_suspect_server", &new_src); + const bool old_explicit = (old_src == REC_SOURCE_EXPLICIT || old_src == REC_SOURCE_ENV); + const bool new_explicit = (new_src == REC_SOURCE_EXPLICIT || new_src == REC_SOURCE_ENV); + if (old_explicit && !new_explicit) { + RecInt deprecated_val = RecGetRecordInt("proxy.config.http.connect_attempts_max_retries_down_server") + .value_or(params->oride.connect_attempts_max_retries_suspect_server); + Warning("proxy.config.http.connect_attempts_max_retries_down_server is deprecated; " + "use proxy.config.http.connect_attempts_max_retries_suspect_server instead. " + "Using deprecated value %" PRIu64 " for now.", + deprecated_val); + params->oride.connect_attempts_max_retries_suspect_server = deprecated_val; + } else if (old_explicit && new_explicit) { + Warning("proxy.config.http.connect_attempts_max_retries_down_server is deprecated and is being ignored " + "in favor of proxy.config.http.connect_attempts_max_retries_suspect_server (%" PRIu64 ").", + m_master.oride.connect_attempts_max_retries_suspect_server); + } + } + if (m_master.oride.connect_attempts_rr_retries > params->oride.connect_attempts_max_retries) { Warning("connect_attempts_rr_retries (%" PRIu64 ") is greater than " "connect_attempts_max_retries (%" PRIu64 "), this means requests " "will never redispatch to another server", m_master.oride.connect_attempts_rr_retries, params->oride.connect_attempts_max_retries); } + if (m_master.oride.connect_attempts_rr_retries > 0 && params->oride.connect_attempts_max_retries_suspect_server == 0) { + Warning("connect_attempts_max_retries_suspect_server=0 with round-robin enabled leaves no retry budget for recovering " + "(SUSPECT) origins beyond the initial attempt; " + "setting proxy.config.http.connect_attempts_max_retries_suspect_server >= 1 is recommended"); + } params->oride.connect_attempts_retry_backoff_base = m_master.oride.connect_attempts_retry_backoff_base; params->oride.connect_attempts_rr_retries = m_master.oride.connect_attempts_rr_retries; diff --git a/src/proxy/http/HttpSM.cc b/src/proxy/http/HttpSM.cc index c8a6b128dc8..4012b704f4a 100644 --- a/src/proxy/http/HttpSM.cc +++ b/src/proxy/http/HttpSM.cc @@ -24,6 +24,7 @@ #include "proxy/http/HttpConfig.h" #include "tscore/ink_hrtime.h" +#include "tscore/ink_time.h" #include "tsutil/Metrics.h" #include "tsutil/ts_bw_format.h" #include "proxy/ProxyTransaction.h" @@ -4731,9 +4732,12 @@ HttpSM::do_hostdb_update_if_necessary() t_state.dns_info.active->http_version = t_state.updated_server_version; } + char addrbuf[INET6_ADDRPORTSTRLEN]; + SMDbg(dbg_ctl_http, "update hostdb info: %s", ats_ip_nptop(&t_state.current.server->dst_addr.sa, addrbuf, sizeof(addrbuf))); + // Check to see if we need to report or clear a connection failure if (track_connect_fail()) { - this->mark_host_failure(&t_state.dns_info, ts_clock::from_time_t(t_state.client_request_time)); + this->mark_host_failure(&t_state.dns_info, ts_clock::now()); } else { if (t_state.dns_info.mark_active_server_up()) { char addrbuf[INET6_ADDRPORTSTRLEN]; @@ -4748,8 +4752,6 @@ HttpSM::do_hostdb_update_if_necessary() } } - char addrbuf[INET6_ADDRPORTSTRLEN]; - SMDbg(dbg_ctl_http, "server info = %s", ats_ip_nptop(&t_state.current.server->dst_addr.sa, addrbuf, sizeof(addrbuf))); return; } @@ -5521,12 +5523,6 @@ HttpSM::do_http_server_open(bool raw, bool only_direct) return; } } - if (HttpTransact::is_server_negative_cached(&t_state) == true && - t_state.txn_conf->connect_attempts_max_retries_down_server <= 0) { - SMDbg(dbg_ctl_http_seq, "Not connecting to the server because it is marked down."); - call_transact_and_set_next_state(HttpTransact::OriginDown); - return; - } // Check for self loop. if (!_ua.get_txn()->is_outbound_transparent() && HttpTransact::will_this_request_self_loop(&t_state)) { @@ -5972,34 +5968,36 @@ HttpSM::do_transform_open() void HttpSM::mark_host_failure(ResolveInfo *info, ts_time time_down) { - char addrbuf[INET6_ADDRPORTSTRLEN]; + ink_assert(time_down != TS_TIME_ZERO); - if (info->active) { - if (time_down != TS_TIME_ZERO) { - ats_ip_nptop(&t_state.current.server->dst_addr.sa, addrbuf, sizeof(addrbuf)); - // Increment the fail_count - if (auto [down, fail_count] = info->active->increment_fail_count(time_down, t_state.txn_conf->connect_attempts_rr_retries, - t_state.txn_conf->down_server_timeout); - down) { - char *url_str = t_state.hdr_info.client_request.url_string_get_ref(nullptr); - std::string_view host_name{t_state.unmapped_url.host_get()}; - swoc::bwprint(error_bw_buffer, "CONNECT : {::s} connecting to {} for host='{}' url='{}' fail_count='{}' marking down", - swoc::bwf::Errno(t_state.current.server->connect_result), t_state.current.server->dst_addr, host_name, - swoc::bwf::FirstOf(url_str, ""), fail_count); - Log::error("%s", error_bw_buffer.c_str()); - SMDbg(dbg_ctl_http, "hostdb update marking IP: %s as down", addrbuf); - ATS_PROBE2(hostdb_mark_ip_as_down, sm_id, addrbuf); - } else { - ATS_PROBE3(hostdb_inc_ip_failcount, sm_id, addrbuf, fail_count); - SMDbg(dbg_ctl_http, "hostdb increment IP failcount %s to %d", addrbuf, fail_count); - } - } else { // Clear the failure - info->active->mark_up(); - } + if (info->active == nullptr) { + return; + } + + char addrbuf[INET6_ADDRPORTSTRLEN]; + ats_ip_nptop(&t_state.current.server->dst_addr.sa, addrbuf, sizeof(addrbuf)); + + const uint8_t max_connect_retries = HttpTransact::origin_server_connect_attempts_max_retries(&t_state); + const ts_seconds fail_window = t_state.txn_conf->down_server_timeout; + + // Mark the host DOWN only after every attempt has failed. `max_connect_retries` counts only "retries", so the total attempt + // budget is `max_connect_retries + 1` (the initial connect plus each retry). + auto [down, fail_count] = info->active->increment_fail_count(time_down, max_connect_retries + 1, fail_window); + + if (down) { + Metrics::Counter::increment(http_rsb.down_server_no_requests); + char *url_str = t_state.hdr_info.client_request.url_string_get_ref(nullptr); + std::string_view host_name{t_state.unmapped_url.host_get()}; + swoc::bwprint(error_bw_buffer, "CONNECT : {::s} connecting to {} for host='{}' url='{}' fail_count='{}' marking down", + swoc::bwf::Errno(t_state.current.server->connect_result), t_state.current.server->dst_addr, host_name, + swoc::bwf::FirstOf(url_str, ""), fail_count); + Log::error("%s", error_bw_buffer.c_str()); + SMDbg(dbg_ctl_http, "hostdb update marking IP: %s as down", addrbuf); + ATS_PROBE2(hostdb_mark_ip_as_down, sm_id, addrbuf); + } else { + ATS_PROBE3(hostdb_inc_ip_failcount, sm_id, addrbuf, fail_count); + SMDbg(dbg_ctl_http, "hostdb increment IP failcount %s to %d", addrbuf, fail_count); } -#ifdef DEBUG - ink_assert(std::chrono::system_clock::now() + t_state.txn_conf->down_server_timeout > time_down); -#endif } void diff --git a/src/proxy/http/HttpTransact.cc b/src/proxy/http/HttpTransact.cc index dba7105f3c8..39841fb7f7c 100644 --- a/src/proxy/http/HttpTransact.cc +++ b/src/proxy/http/HttpTransact.cc @@ -562,6 +562,41 @@ HttpTransact::is_server_negative_cached(State *s) } } +/** + ATS has two configuration options controlling how many times it retries a connection attempt against origin servers. + + - proxy.config.http.connect_attempts_max_retries + - proxy.config.http.connect_attempts_max_retries_suspect_server + + The choice is based on the state of the active HostDBInfo. + + - HostDBInfo::State::UP: use proxy.config.http.connect_attempts_max_retries + - HostDBInfo::State::DOWN: no retry + - HostDBInfo::State::SUSPECT: use proxy.config.http.connect_attempts_max_retries_suspect_server + +*/ +uint8_t +HttpTransact::origin_server_connect_attempts_max_retries(State *s) +{ + HostDBInfo *active = s->dns_info.active; + if (active == nullptr) { + return 0; + } + + switch (active->state(ts_clock::now(), s->txn_conf->down_server_timeout)) { + case HostDBInfo::State::UP: + return s->txn_conf->connect_attempts_max_retries; + case HostDBInfo::State::DOWN: + return 0; + case HostDBInfo::State::SUSPECT: + return s->txn_conf->connect_attempts_max_retries_suspect_server; + default: + break; + } + + return 0; +} + inline static void update_current_info(HttpTransact::CurrentInfo *into, HttpTransact::ConnectionAttributes *from, ResolveInfo::UpstreamResolveStyle who, bool clear_retry_attempts) @@ -974,22 +1009,6 @@ HttpTransact::TooEarly(State *s) TRANSACT_RETURN(StateMachineAction_t::SEND_ERROR_CACHE_NOOP, nullptr); } -void -HttpTransact::OriginDown(State *s) -{ - TxnDbg(dbg_ctl_http_trans, "origin server is marked down"); - bootstrap_state_variables_from_request(s, &s->hdr_info.client_request); - build_error_response(s, HTTPStatus::BAD_GATEWAY, "Origin Server Marked Down", "connect#failed_connect"); - Metrics::Counter::increment(http_rsb.down_server_no_requests); - char *url_str = s->hdr_info.client_request.url_string_get_ref(nullptr); - std::string_view host_name{s->unmapped_url.host_get()}; - swoc::bwprint(error_bw_buffer, "CONNECT: down server no request to {} for host='{}' url='{}'", s->current.server->dst_addr, - host_name, swoc::bwf::FirstOf(url_str, "")); - Log::error("%s", error_bw_buffer.c_str()); - - TRANSACT_RETURN(StateMachineAction_t::SEND_ERROR_CACHE_NOOP, nullptr); -} - void HttpTransact::HandleBlindTunnel(State *s) { @@ -1969,6 +1988,9 @@ HttpTransact::OSDNSLookup(State *s) build_error_response(s, HTTPStatus::INTERNAL_SERVER_ERROR, "Cannot find server.", Dns_error_body); log_msg = "looking up"; } else { + // HostDB has the record but every address is DOWN within `down_server.cache_time`. This is a refusal-to-attempt + // rather than a true DNS failure; track it under down_server_no_requests. + Metrics::Counter::increment(http_rsb.down_server_no_requests); build_error_response(s, HTTPStatus::INTERNAL_SERVER_ERROR, "No valid server.", "connect#all_down"); log_msg = "no valid server"; } @@ -1998,7 +2020,8 @@ HttpTransact::OSDNSLookup(State *s) // We've backed off from a client supplied address and found some // HostDB addresses. We use those if they're different from the CTA. // In all cases we now commit to client or HostDB for our source. - if (s->dns_info.set_active(&s->current.server->dst_addr.sa) && s->dns_info.select_next_rr()) { + if (s->dns_info.set_active(&s->current.server->dst_addr.sa) && + s->dns_info.select_next_rr(ts_clock::now(), s->txn_conf->down_server_timeout)) { s->dns_info.os_addr_style = ResolveInfo::OS_Addr::USE_HOSTDB; } else { // nothing else there, continue with CTA. @@ -2687,6 +2710,7 @@ HttpTransact::CallOSDNSLookup(State *s) } else { s->cache_info.action = CacheAction_t::NO_ACTION; } + error_log_connection_failure(s, s->current.state); handle_server_connection_not_open(s); } else { TRANSACT_RETURN(StateMachineAction_t::DNS_LOOKUP, OSDNSLookup); @@ -3895,6 +3919,7 @@ HttpTransact::handle_response_from_parent(State *s) case ResolveInfo::HOST_NONE: // Check if content can be served from cache s->current.request_to = ResolveInfo::PARENT_PROXY; + error_log_connection_failure(s, s->current.state); handle_server_connection_not_open(s); break; default: @@ -3927,7 +3952,6 @@ HttpTransact::handle_response_from_server(State *s) { TxnDbg(dbg_ctl_http_trans, "(hrfs)"); HTTP_RELEASE_ASSERT(s->current.server == &s->server_info); - unsigned max_connect_retries = 0; // plugin call s->server_info.state = s->current.state; @@ -3946,6 +3970,7 @@ HttpTransact::handle_response_from_server(State *s) TxnDbg(dbg_ctl_http_trans, "Error. congestion control -- congested."); SET_VIA_STRING(VIA_DETAIL_SERVER_CONNECT, VIA_DETAIL_SERVER_FAILURE); s->set_connect_fail(EUSERS); // too many users + error_log_connection_failure(s, s->current.state); handle_server_connection_not_open(s); break; case OPEN_RAW_ERROR: @@ -3954,8 +3979,7 @@ HttpTransact::handle_response_from_server(State *s) case INACTIVE_TIMEOUT: case PARSE_ERROR: case CONNECTION_CLOSED: - case BAD_INCOMING_RESPONSE: - + case BAD_INCOMING_RESPONSE: { // Ensure cause_of_death_errno is set for all error states if not already set. // This prevents the assertion failure in retry_server_connection_not_open. if (s->cause_of_death_errno == -UNKNOWN_INTERNAL_ERROR) { @@ -3970,63 +3994,92 @@ HttpTransact::handle_response_from_server(State *s) } } - if (is_server_negative_cached(s)) { - max_connect_retries = s->txn_conf->connect_attempts_max_retries_down_server - 1; - } else { - // server not yet negative cached - use default number of retries - max_connect_retries = s->txn_conf->connect_attempts_max_retries; - } - + unsigned max_connect_retries = s->txn_conf->connect_attempts_max_retries; TxnDbg(dbg_ctl_http_trans, "max_connect_retries: %d s->current.retry_attempts: %d", max_connect_retries, s->current.retry_attempts.get()); - if (is_request_retryable(s) && s->current.retry_attempts.get() < max_connect_retries && - !HttpTransact::is_response_valid(s, &s->hdr_info.server_response)) { - // If this is a round robin DNS entry & we're tried configured - // number of times, we should try another node - if (ResolveInfo::OS_Addr::TRY_CLIENT == s->dns_info.os_addr_style) { - // attempt was based on client supplied server address. Try again using HostDB. - // Allow DNS attempt - s->dns_info.resolved_p = false; - // See if we can get data from HostDB for this. - s->dns_info.os_addr_style = ResolveInfo::OS_Addr::TRY_HOSTDB; - // Force host resolution to have the same family as the client. - // Because this is a transparent connection, we can't switch address - // families - that is locked in by the client source address. - ats_force_order_by_family(s->current.server->dst_addr.family(), s->my_txn_conf().host_res_data.order); - return CallOSDNSLookup(s); - } else if (ResolveInfo::OS_Addr::USE_API == s->dns_info.os_addr_style && !s->api_server_addr_set_retried) { - // Plugin set the server address via TSHttpTxnServerAddrSet(). Clear resolution - // state to allow the OS_DNS hook to be called again, giving the plugin a chance - // to set a different server address for retry (issue #12611). - // Only retry once to avoid infinite loops if the plugin keeps setting failing addresses. - s->api_server_addr_set_retried = true; - s->dns_info.resolved_p = false; - s->dns_info.os_addr_style = ResolveInfo::OS_Addr::TRY_DEFAULT; - // Clear the server request so it can be rebuilt for the new destination - s->hdr_info.server_request.destroy(); - TxnDbg(dbg_ctl_http_trans, "Retrying with plugin-set address, returning to OS_DNS hook"); - return CallOSDNSLookup(s); - } else { - if ((s->txn_conf->connect_attempts_rr_retries > 0) && - ((s->current.retry_attempts.get() + 1) % s->txn_conf->connect_attempts_rr_retries == 0)) { - s->dns_info.select_next_rr(); + // Bail out if the request is not retryable, the global retry cap is reached, or we already have a usable response. + if (!is_request_retryable(s) || s->current.retry_attempts.get() >= max_connect_retries || + HttpTransact::is_response_valid(s, &s->hdr_info.server_response)) { + TxnDbg(dbg_ctl_http_trans, "Error. No more retries. %d/%d", s->current.retry_attempts.get(), max_connect_retries); + SET_VIA_STRING(VIA_DETAIL_SERVER_CONNECT, VIA_DETAIL_SERVER_FAILURE); + error_log_connection_failure(s, s->current.state); + s->state_machine->do_hostdb_update_if_necessary(); + handle_server_connection_not_open(s); + break; + } + + // Attempt was based on a client-supplied address. Re-resolve via HostDB. + if (ResolveInfo::OS_Addr::TRY_CLIENT == s->dns_info.os_addr_style) { + // Allow DNS attempt + s->dns_info.resolved_p = false; + // See if we can get data from HostDB for this. + s->dns_info.os_addr_style = ResolveInfo::OS_Addr::TRY_HOSTDB; + // Force host resolution to have the same family as the client. + // Because this is a transparent connection, we can't switch address + // families - that is locked in by the client source address. + ats_force_order_by_family(s->current.server->dst_addr.family(), s->my_txn_conf().host_res_data.order); + return CallOSDNSLookup(s); + } + + // Plugin set the server address via TSHttpTxnServerAddrSet(). Clear resolution + // state to allow the OS_DNS hook to be called again, giving the plugin a chance + // to set a different server address for retry (issue #12611). + // Only retry once to avoid infinite loops if the plugin keeps setting failing addresses. + if (ResolveInfo::OS_Addr::USE_API == s->dns_info.os_addr_style && !s->api_server_addr_set_retried) { + s->api_server_addr_set_retried = true; + s->dns_info.resolved_p = false; + s->dns_info.os_addr_style = ResolveInfo::OS_Addr::TRY_DEFAULT; + // Clear the server request so it can be rebuilt for the new destination + s->hdr_info.server_request.destroy(); + TxnDbg(dbg_ctl_http_trans, "Retrying with plugin-set address, returning to OS_DNS hook"); + return CallOSDNSLookup(s); + } + + // Record the failure on the current active target. + error_log_connection_failure(s, s->current.state); + s->state_machine->do_hostdb_update_if_necessary(); + + // Decide between switching to the next round-robin member or staying on the same target. + if ((s->txn_conf->connect_attempts_rr_retries > 0) && + ((s->current.retry_attempts.get() + 1) % s->txn_conf->connect_attempts_rr_retries == 0)) { + if (s->dns_info.select_next_rr(ts_clock::now(), s->txn_conf->down_server_timeout)) { + // select_next_rr() only updates dns_info.active; change the dst_addr too. + s->dns_info.addr.assign(s->dns_info.active->data.ip); + s->server_info.dst_addr.assign(s->dns_info.active->data.ip, s->server_info.dst_addr.network_order_port()); + if (dbg_ctl_http_trans.on()) { + ip_port_text_buffer addrbuf; + TxnDbg(dbg_ctl_http_trans, "switched to next round-robin upstream addr=%s", + ats_ip_nptop(&s->server_info.dst_addr.sa, addrbuf, sizeof(addrbuf))); } - retry_server_connection_not_open(s, s->current.state, max_connect_retries); - TxnDbg(dbg_ctl_http_trans, "Error. Retrying..."); - s->next_action = how_to_open_connection(s); + } else { + TxnDbg(dbg_ctl_http_trans, "No round-robin targets available, retrying current upstream if possible"); } - } else { - error_log_connection_failure(s, s->current.state); - TxnDbg(dbg_ctl_http_trans, "Error. No more retries."); + } + + // The active target (HostDB) may be SUSPECT state, so re-evaluate the retry limit. + // Skip when there is no HostDBInfo (e.g. USE_CLIENT / USE_API) and keep the configured baseline. + if (s->dns_info.active != nullptr) { + max_connect_retries = origin_server_connect_attempts_max_retries(s); + } + if (max_connect_retries <= s->current.retry_attempts.get()) { + TxnDbg(dbg_ctl_http_trans, "Per-host retries exhausted. Giving up. %d/%d", s->current.retry_attempts.get(), + max_connect_retries); SET_VIA_STRING(VIA_DETAIL_SERVER_CONNECT, VIA_DETAIL_SERVER_FAILURE); handle_server_connection_not_open(s); + break; } + + TxnDbg(dbg_ctl_http_trans, "Error. Retrying..."); + retry_server_connection_not_open(s, max_connect_retries); + s->next_action = how_to_open_connection(s); break; + } case ACTIVE_TIMEOUT: TxnDbg(dbg_ctl_http_trans, "[hrfs] connection not alive"); SET_VIA_STRING(VIA_DETAIL_SERVER_CONNECT, VIA_DETAIL_SERVER_FAILURE); s->set_connect_fail(ETIMEDOUT); + s->state_machine->do_hostdb_update_if_necessary(); handle_server_connection_not_open(s); break; default: @@ -4075,15 +4128,13 @@ HttpTransact::error_log_connection_failure(State *s, ServerState_t conn_state) // /////////////////////////////////////////////////////////////////////////////// void -HttpTransact::retry_server_connection_not_open(State *s, ServerState_t conn_state, unsigned max_retries) +HttpTransact::retry_server_connection_not_open(State *s, unsigned max_retries) { ink_assert(s->current.state != CONNECTION_ALIVE); ink_assert(s->current.state != ACTIVE_TIMEOUT); ink_assert(s->current.retry_attempts.get() < max_retries); ink_assert(s->cause_of_death_errno != -UNKNOWN_INTERNAL_ERROR); - error_log_connection_failure(s, conn_state); - ////////////////////////////////////////////// // disable keep-alive for request and retry // ////////////////////////////////////////////// @@ -4117,9 +4168,6 @@ HttpTransact::handle_server_connection_not_open(State *s) SET_VIA_STRING(VIA_SERVER_RESULT, VIA_SERVER_ERROR); Metrics::Counter::increment(http_rsb.broken_server_connections); - // Fire off a hostdb update to mark the server as down - s->state_machine->do_hostdb_update_if_necessary(); - switch (s->cache_info.action) { case CacheAction_t::UPDATE: case CacheAction_t::SERVE: diff --git a/src/records/RecordsConfig.cc b/src/records/RecordsConfig.cc index 8c887a2e0f3..743eece21d6 100644 --- a/src/records/RecordsConfig.cc +++ b/src/records/RecordsConfig.cc @@ -496,11 +496,13 @@ static constexpr RecordElement RecordsConfig[] = // ################################## // # origin server connect attempts # // ################################## - {RECT_CONFIG, "proxy.config.http.connect_attempts_max_retries", RECD_INT, "3", RECU_DYNAMIC, RR_NULL, RECC_NULL, nullptr, RECA_NULL} + {RECT_CONFIG, "proxy.config.http.connect_attempts_max_retries", RECD_INT, "3", RECU_DYNAMIC, RR_NULL, RECC_INT, "[0-254]", RECA_NULL} , - {RECT_CONFIG, "proxy.config.http.connect_attempts_max_retries_down_server", RECD_INT, "1", RECU_DYNAMIC, RR_NULL, RECC_NULL, nullptr, RECA_NULL} + {RECT_CONFIG, "proxy.config.http.connect_attempts_max_retries_down_server", RECD_INT, "1", RECU_DYNAMIC, RR_NULL, RECC_INT, "[0-254]", RECA_NULL} , - {RECT_CONFIG, "proxy.config.http.connect_attempts_rr_retries", RECD_INT, "3", RECU_DYNAMIC, RR_NULL, RECC_NULL, nullptr, RECA_NULL} + {RECT_CONFIG, "proxy.config.http.connect_attempts_max_retries_suspect_server", RECD_INT, "1", RECU_DYNAMIC, RR_NULL, RECC_INT, "[0-254]", RECA_NULL} + , + {RECT_CONFIG, "proxy.config.http.connect_attempts_rr_retries", RECD_INT, "3", RECU_DYNAMIC, RR_NULL, RECC_INT, "[0-254]", RECA_NULL} , {RECT_CONFIG, "proxy.config.http.connect_attempts_timeout", RECD_INT, "30", RECU_DYNAMIC, RR_NULL, RECC_NULL, nullptr, RECA_NULL} , diff --git a/tests/gold_tests/autest-site/verifier_client.test.ext b/tests/gold_tests/autest-site/verifier_client.test.ext index 53f0f5f2691..60ca3942272 100755 --- a/tests/gold_tests/autest-site/verifier_client.test.ext +++ b/tests/gold_tests/autest-site/verifier_client.test.ext @@ -35,7 +35,8 @@ def _configure_client( verbose=True, other_args='', run_parallel=False, - context=None): + context=None, + poll_timeout=None): """ Configure the process for running the verifier-client. @@ -136,6 +137,9 @@ def _configure_client( if keys is not None: command += f" --keys {keys}" + if poll_timeout is not None: + command += f" --poll-timeout {poll_timeout}" + # Generally we prefer the deterministic behavior of a single thread where # each connection is processed sequentially. if not run_parallel and 'thread-limit' not in command: @@ -161,7 +165,8 @@ def AddVerifierClientProcess( verbose=True, other_args='', run_parallel=False, - context=None): + context=None, + poll_timeout=None): """ Set the Default process of the test run to a verifier-client Process. @@ -198,6 +203,10 @@ def AddVerifierClientProcess( Template strings support $-based substitutions in the replay file. You can refer to https://docs.python.org/3/library/string.html#template-strings for more information how to add template strings to the replay file. + + poll_timeout: (int) The poll timeout in milliseconds passed to + verifier-client via --poll-timeout. Defaults to None which leaves the + verifier-client default (3000 ms) in place. Returns: The newly constructed verifier-client for the test run, which is also the Default Process of the test run. @@ -213,7 +222,7 @@ def AddVerifierClientProcess( p = run.Processes.Default _configure_client( run, p, name, replay_path, http_ports, https_ports, http3_ports, keys, ssl_cert, ca_cert, verbose, other_args, run_parallel, - context) + context, poll_timeout) return p diff --git a/tests/gold_tests/connect_down_policy/replay/connect_down_policy_4.replay.yaml b/tests/gold_tests/connect_down_policy/replay/connect_down_policy_4.replay.yaml index 554e90d138e..45fe1d8e5d3 100644 --- a/tests/gold_tests/connect_down_policy/replay/connect_down_policy_4.replay.yaml +++ b/tests/gold_tests/connect_down_policy/replay/connect_down_policy_4.replay.yaml @@ -48,7 +48,7 @@ autest: proxy.config.http.connect.down.policy: 4 proxy.config.http.connect_attempts_rr_retries: 0 proxy.config.http.connect_attempts_max_retries: 0 - proxy.config.http.connect_attempts_max_retries_down_server: 0 + proxy.config.http.connect_attempts_max_retries_suspect_server: 0 proxy.config.http.connect_attempts_timeout: 1 proxy.config.http.down_server.cache_time: 5 diff --git a/tests/gold_tests/dns/connect_attempts.test.py b/tests/gold_tests/dns/connect_attempts.test.py index d46cb32d3d5..e5b572d366b 100644 --- a/tests/gold_tests/dns/connect_attempts.test.py +++ b/tests/gold_tests/dns/connect_attempts.test.py @@ -21,6 +21,11 @@ Verify Origin Server Connect Attempts Behavior ''' +# Single DNS Record +# max_retries & down_server +Test.ATSReplayTest(replay_file="replay/connect_attempts_single_max_retries.replay.yaml") + +# Multiple DNS Records # No retry Test.ATSReplayTest(replay_file="replay/connect_attempts_rr_no_retry.replay.yaml") diff --git a/tests/gold_tests/dns/dns_host_down.test.py b/tests/gold_tests/dns/dns_host_down.test.py index 80b59e0fb83..295100c2fd2 100644 --- a/tests/gold_tests/dns/dns_host_down.test.py +++ b/tests/gold_tests/dns/dns_host_down.test.py @@ -73,9 +73,23 @@ def _test_error_log(self): self._ts.Disk.error_log.Content = Testers.ContainsExpression( "/dns/mark/down' fail_count='1' marking down", "host should be marked down") + # Verify down_server_no_requests metric is incremented: + # - once when the first request marks the origin DOWN (502) + # - once when the second request is rejected because HostDB has no live address (500) + def _test_down_server_no_requests_metric(self): + tr = Test.AddTestRun('Check proxy.process.http.down_server.no_requests metric') + tr.Processes.Default.Command = 'traffic_ctl metric get proxy.process.http.down_server.no_requests' + tr.Processes.Default.Env = self._ts.Env + tr.Processes.Default.ReturnCode = 0 + tr.Processes.Default.Streams.All = Testers.ContainsExpression( + 'proxy.process.http.down_server.no_requests 2', + 'down_server.no_requests should be 2 (mark-down on first txn, no-live-address on second)') + tr.StillRunningAfter = self._ts + def run(self): self._test_host_mark_down() self._test_error_log() + self._test_down_server_no_requests_metric() DownCachedOriginServerTest().run() diff --git a/tests/gold_tests/dns/gold/connect_attempts_rr_max_retries_error_log.gold b/tests/gold_tests/dns/gold/connect_attempts_rr_max_retries_error_log.gold index 2172f69d461..bc2cbc1c802 100644 --- a/tests/gold_tests/dns/gold/connect_attempts_rr_max_retries_error_log.gold +++ b/tests/gold_tests/dns/gold/connect_attempts_rr_max_retries_error_log.gold @@ -1,8 +1,8 @@ `` CONNECT: attempt fail [CONNECTION_ERROR] to 0.0.0.1:`` for host='example.com' sm_id=0 `` retry_attempts=0 url='http://backend.example.com:``/path/' `` CONNECT: attempt fail [CONNECTION_ERROR] to 0.0.0.1:`` for host='example.com' sm_id=0 `` retry_attempts=1 url='http://backend.example.com:``/path/' -`` CONNECT : `` connecting to 0.0.0.1:`` for host='example.com' url='http://backend.example.com:``/path/' fail_count='1' marking down +`` CONNECT : `` connecting to 0.0.0.1:`` for host='example.com' url='http://backend.example.com:``/path/' fail_count='2' marking down `` CONNECT: attempt fail [CONNECTION_ERROR] to 0.0.0.2:`` for host='example.com' sm_id=1 `` retry_attempts=0 url='http://backend.example.com:``/path/' `` CONNECT: attempt fail [CONNECTION_ERROR] to 0.0.0.2:`` for host='example.com' sm_id=1 `` retry_attempts=1 url='http://backend.example.com:``/path/' -`` CONNECT : `` connecting to 0.0.0.2:`` for host='example.com' url='http://backend.example.com:``/path/' fail_count='1' marking down +`` CONNECT : `` connecting to 0.0.0.2:`` for host='example.com' url='http://backend.example.com:``/path/' fail_count='2' marking down `` DNS Error: no valid server http://backend.example.com:``/path/ `` diff --git a/tests/gold_tests/dns/gold/connect_attempts_rr_retries_error_log.gold b/tests/gold_tests/dns/gold/connect_attempts_rr_retries_error_log.gold index c835cd3fa5d..a4e061a67d4 100644 --- a/tests/gold_tests/dns/gold/connect_attempts_rr_retries_error_log.gold +++ b/tests/gold_tests/dns/gold/connect_attempts_rr_retries_error_log.gold @@ -1,8 +1,9 @@ `` CONNECT: attempt fail [CONNECTION_ERROR] to 0.0.0.1:`` for host='example.com' sm_id=0 connection_result=`` error=`` retry_attempts=0 url='http://backend.example.com:``/path/' +`` CONNECT: attempt fail [CONNECTION_ERROR] to 0.0.0.2:`` for host='example.com' sm_id=0 connection_result=`` error=`` retry_attempts=1 url='http://backend.example.com:``/path/' `` CONNECT: attempt fail [CONNECTION_ERROR] to 0.0.0.1:`` for host='example.com' sm_id=1 connection_result=`` error=`` retry_attempts=0 url='http://backend.example.com:``/path/' `` CONNECT : `` connecting to 0.0.0.1:`` for host='example.com' url='http://backend.example.com:``/path/' fail_count='2' marking down -`` CONNECT: attempt fail [CONNECTION_ERROR] to 0.0.0.2:`` for host='example.com' sm_id=2 connection_result=`` error=`` retry_attempts=0 url='http://backend.example.com:``/path/' -`` CONNECT: attempt fail [CONNECTION_ERROR] to 0.0.0.2:`` for host='example.com' sm_id=3 connection_result=`` error=`` retry_attempts=0 url='http://backend.example.com:``/path/' +`` CONNECT: attempt fail [CONNECTION_ERROR] to 0.0.0.2:`` for host='example.com' sm_id=1 connection_result=`` error=`` retry_attempts=1 url='http://backend.example.com:``/path/' `` CONNECT : `` connecting to 0.0.0.2:`` for host='example.com' url='http://backend.example.com:``/path/' fail_count='2' marking down `` DNS Error: no valid server http://backend.example.com:``/path/ -`` +`` CONNECT: attempt fail [CONNECTION_ERROR] to 0.0.0.1:`` for host='example.com' sm_id=3 connection_result=`` error=`` retry_attempts=0 url='http://backend.example.com:``/path/' +`` CONNECT: attempt fail [CONNECTION_ERROR] to 0.0.0.2:`` for host='example.com' sm_id=3 connection_result=`` error=`` retry_attempts=1 url='http://backend.example.com:``/path/' diff --git a/tests/gold_tests/dns/gold/connect_attempts_single_max_retries_error_log.gold b/tests/gold_tests/dns/gold/connect_attempts_single_max_retries_error_log.gold new file mode 100644 index 00000000000..5aef6b9655a --- /dev/null +++ b/tests/gold_tests/dns/gold/connect_attempts_single_max_retries_error_log.gold @@ -0,0 +1,9 @@ +`` CONNECT: attempt fail [CONNECTION_ERROR] to 0.0.0.1:`` for host='example.com' sm_id=0 `` retry_attempts=0 url='http://backend.example.com:``/path/' +`` CONNECT: attempt fail [CONNECTION_ERROR] to 0.0.0.1:`` for host='example.com' sm_id=0 `` retry_attempts=1 url='http://backend.example.com:``/path/' +`` CONNECT: attempt fail [CONNECTION_ERROR] to 0.0.0.1:`` for host='example.com' sm_id=0 `` retry_attempts=2 url='http://backend.example.com:``/path/' +`` CONNECT : `` connecting to 0.0.0.1:`` for host='example.com' url='http://backend.example.com:``/path/' fail_count='3' marking down +`` DNS Error: no valid server http://backend.example.com:``/path/ +`` CONNECT: attempt fail [CONNECTION_ERROR] to 0.0.0.1:`` for host='example.com' sm_id=2 `` retry_attempts=0 url='http://backend.example.com:``/path/' +`` CONNECT: attempt fail [CONNECTION_ERROR] to 0.0.0.1:`` for host='example.com' sm_id=2 `` retry_attempts=1 url='http://backend.example.com:``/path/' +`` CONNECT : `` connecting to 0.0.0.1:`` for host='example.com' url='http://backend.example.com:``/path/' fail_count='2' marking down +`` DNS Error: no valid server http://backend.example.com:``/path/ diff --git a/tests/gold_tests/dns/replay/connect_attempts_rr_max_retries.replay.yaml b/tests/gold_tests/dns/replay/connect_attempts_rr_max_retries.replay.yaml index b1c918ab2ce..834c60fb912 100644 --- a/tests/gold_tests/dns/replay/connect_attempts_rr_max_retries.replay.yaml +++ b/tests/gold_tests/dns/replay/connect_attempts_rr_max_retries.replay.yaml @@ -41,7 +41,7 @@ autest: proxy.config.diags.debug.tags: 'http|hostdb|dns' proxy.config.http.connect_attempts_rr_retries: 0 proxy.config.http.connect_attempts_max_retries: 1 - proxy.config.http.connect_attempts_max_retries_down_server: 0 + proxy.config.http.connect_attempts_max_retries_suspect_server: 0 proxy.config.http.connect_attempts_timeout: 1 proxy.config.http.down_server.cache_time: 10 diff --git a/tests/gold_tests/dns/replay/connect_attempts_rr_no_retry.replay.yaml b/tests/gold_tests/dns/replay/connect_attempts_rr_no_retry.replay.yaml index b0ec1819eb9..258653fbb8c 100644 --- a/tests/gold_tests/dns/replay/connect_attempts_rr_no_retry.replay.yaml +++ b/tests/gold_tests/dns/replay/connect_attempts_rr_no_retry.replay.yaml @@ -41,7 +41,7 @@ autest: proxy.config.diags.debug.tags: 'http|hostdb|dns' proxy.config.http.connect_attempts_rr_retries: 0 proxy.config.http.connect_attempts_max_retries: 0 - proxy.config.http.connect_attempts_max_retries_down_server: 0 + proxy.config.http.connect_attempts_max_retries_suspect_server: 0 proxy.config.http.connect_attempts_timeout: 1 proxy.config.http.down_server.cache_time: 5 diff --git a/tests/gold_tests/dns/replay/connect_attempts_rr_retries.replay.yaml b/tests/gold_tests/dns/replay/connect_attempts_rr_retries.replay.yaml index 0d040345ffa..971e623ff33 100644 --- a/tests/gold_tests/dns/replay/connect_attempts_rr_retries.replay.yaml +++ b/tests/gold_tests/dns/replay/connect_attempts_rr_retries.replay.yaml @@ -39,11 +39,11 @@ autest: records_config: proxy.config.diags.debug.enabled: 1 proxy.config.diags.debug.tags: 'http|hostdb|dns' - proxy.config.http.connect_attempts_rr_retries: 2 - proxy.config.http.connect_attempts_max_retries: 0 - proxy.config.http.connect_attempts_max_retries_down_server: 0 + proxy.config.http.connect_attempts_rr_retries: 1 + proxy.config.http.connect_attempts_max_retries: 1 + proxy.config.http.connect_attempts_max_retries_suspect_server: 1 proxy.config.http.connect_attempts_timeout: 1 - proxy.config.http.down_server.cache_time: 10 + proxy.config.http.down_server.cache_time: 3 remap_config: - from: "http://example.com/" @@ -55,7 +55,9 @@ autest: sessions: - transactions: - # try 0.0.0.1 + # Both 0.0.0.1 and 0.0.0.2 are UP. ATS tries one host, then rr_retries to the + # other (connect_attempts_rr_retries=1). Both attempts fail, so the client + # gets 502. Each host's fail_count is now 1. - client-request: method: GET url: /path/ @@ -73,7 +75,8 @@ sessions: proxy-response: status: 502 - # try 0.0.0.1 again + # Same round-robin path as above. After this attempt each host's fail_count + # reaches connect_attempts_max_retries + 1 (=2), so both are marked DOWN. - client-request: method: GET url: /path/ @@ -91,7 +94,8 @@ sessions: proxy-response: status: 502 - # try 0.0.0.2 + # Within down_server.cache_time (3s): both hosts are still cached as DOWN, so + # ATS skips the connect attempts entirely and returns 500. - client-request: method: GET url: /path/ @@ -106,46 +110,11 @@ sessions: status: 200 reason: OK - proxy-response: - status: 502 - - # try 0.0.0.2 again - - client-request: - method: GET - url: /path/ - version: '1.1' - headers: - fields: - - [Host, example.com] - - [uuid, 4] - - # should not hit - server-response: - status: 200 - reason: OK - - proxy-response: - status: 502 - - # The request is expected to hit the down_server cache and immediately receive a 500 response. - - client-request: - method: GET - url: /path/ - version: '1.1' - headers: - fields: - - [Host, example.com] - - [uuid, 10] - - # should not hit - server-response: - status: 200 - reason: OK - proxy-response: status: 500 - # when down_server.cache_time is expired, try connect attempts + # After the 5s delay, down_server.cache_time (3s) has expired. ATS tries both + # hosts again via round-robin; both still fail, so the client gets 502. - client-request: method: GET url: /path/ @@ -154,7 +123,7 @@ sessions: fields: - [Host, example.com] - [uuid, 20] - delay: 10s + delay: 5s # should not hit server-response: diff --git a/tests/gold_tests/dns/replay/connect_attempts_single_max_retries.replay.yaml b/tests/gold_tests/dns/replay/connect_attempts_single_max_retries.replay.yaml new file mode 100644 index 00000000000..85285c551a1 --- /dev/null +++ b/tests/gold_tests/dns/replay/connect_attempts_single_max_retries.replay.yaml @@ -0,0 +1,134 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +meta: + version: "1.0" + +# Configuration section for autest integration +autest: + description: 'Verify connect attempts behavior - no round-robin (single host)' + + dns: + name: 'dns-single-retries' + records: {"backend.example.com": ["0.0.0.1"]} + + server: + name: 'server-single-retries' + + client: + name: 'client-single-retries' + process_config: + poll_timeout: 10000 # ms + + ats: + name: 'ts-single-retries' + process_config: + enable_cache: false + + records_config: + proxy.config.diags.debug.enabled: 1 + proxy.config.diags.debug.tags: 'http|hostdb|dns' + proxy.config.http.connect_attempts_rr_retries: 99 # make sure this doesn't affect this case + proxy.config.http.connect_attempts_max_retries: 2 + proxy.config.http.connect_attempts_max_retries_suspect_server: 1 + proxy.config.http.connect_attempts_timeout: 1 + proxy.config.http.down_server.cache_time: 3 + + remap_config: + - from: "http://example.com/" + to: "http://backend.example.com:{SERVER_HTTP_PORT}/" + + log_validation: + error_log: + gold_file: "gold/connect_attempts_single_max_retries_error_log.gold" + +sessions: +- transactions: + # First request: ATS attempts 3 times (1 initial + 2 retries) to connect to 0.0.0.1. + # The server is marked as down and ATS returns 502. + - client-request: + method: GET + url: /path/ + version: '1.1' + headers: + fields: + - [Host, example.com] + - [uuid, 1] + + # should not hit + server-response: + status: 200 + reason: OK + + proxy-response: + status: 502 + + # Second request: within down_server.cache_time=3s so the server is still marked down. + # ATS skips the connection attempt entirely and returns 500. + - client-request: + method: GET + url: /path/ + version: '1.1' + headers: + fields: + - [Host, example.com] + - [uuid, 2] + + # should not hit + server-response: + status: 200 + reason: OK + + proxy-response: + status: 500 + + # Third request: delayed 5s so the down_server.cache_time=3s has expired. + # ATS attempts 2 times (1 initial + 1 retries) to connect to 0.0.0.1. + # The server is marked as down again and ATS returns 502. + - client-request: + method: GET + url: /path/ + version: '1.1' + headers: + fields: + - [Host, example.com] + - [uuid, 3] + delay: 5s + + # should not hit + server-response: + status: 200 + reason: OK + + proxy-response: + status: 502 + + # Fourth request: within down_server.cache_time=3s so the server is still marked down. + # ATS skips the connection attempt entirely and returns 500. + - client-request: + method: GET + url: /path/ + version: '1.1' + headers: + fields: + - [Host, example.com] + - [uuid, 4] + + server-response: + status: 200 + + proxy-response: + status: 500