AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Rust agent-memoryai-agentsbm25claude-codeclicode-searchcursordatabaseelasticsearchfull-text-searchhybrid-searchjsonllocal-firstmcpmcp-servermodel-context-protocolragrustsearch-enginevector-search
feat(bench): neural-bulk A/B harness and frozen Okta corpus for ingest changes (#2448)
Christian Todiecb990c63de7e
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 04026c4ca..883132225 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -49,6 +49,13 @@ continues in #2427 and #1122, which stay open.### Added+- `scripts/bench/neural-bulk-ab.sh CONTROL_SHA CANDIDATE_SHA [N]` is an+ interleaved A/B harness for neural-embedding `_bulk` ingest. It builds each+ SHA in a worktree, runs the real `xerj autoindex` client against a frozen+ Okta corpus on a CPU-capped throwaway node, and reports median, p95 and the+ control arm's spread per metric. It measures the harness only; no+ performance claim is made. See `benchmarks/ingest-neural-bulk/README.md`.+- `xerj sessions import --format claude-code --input FILE --output NEW.jsonl`prepares an explicit Claude Code JSONL transcript for ordinary `autoindex`.It emits text-bearing user/assistant records through the existing nativediff --git a/benchmarks/ingest-neural-bulk/README.md b/benchmarks/ingest-neural-bulk/README.mdnew file mode 100644index 000000000..de92b5e14--- /dev/null+++ b/benchmarks/ingest-neural-bulk/README.md@@ -0,0 +1,84 @@+# ingest-neural-bulk++A/B harness for neural-embedding `_bulk` ingest. It answers one question: does+a change to the server make the same `xerj autoindex` run faster, slower, or+noisier, with neural embeddings on and the CPU held fixed?++## Run++```+scripts/bench/neural-bulk-ab.sh [--dry-run] CONTROL_SHA CANDIDATE_SHA [N=5]+```++`--dry-run` prints every command and writes nothing. Without it the script:++1. Verifies `corpus/okta-bulk.tar.zst` against `corpus/SHA256SUMS`, extracts+ it, and verifies each extracted file against `corpus/MANIFEST.tsv`.+2. Builds each SHA once, in its own `git worktree`, with+ `cargo build --release -j 2 -p xerj-server`. The same SHA twice builds once.+ `CONTROL_BIN` / `CANDIDATE_BIN` skip a build.+3. Runs control, candidate, control, candidate, ... N times each. Every run+ gets a fresh data dir and a fresh client state dir.+4. Starts the server on the first free port from 9560 (never 9200) under+ `systemd-run --user --scope -p CPUQuota=400% -p MemoryMax=16G`, falling+ back to `taskset -c 0-3` with a warning. Env: `XERJ_NUM_CPUS=4`,+ `XERJ_PROF=1`, `XERJ_EMBED_MODE=neural`, `HF_HOME` (default+ `~/.cache/huggingface`; warm it first or run 1 includes the model download).+5. Drives it with+ `xerj autoindex <corpus> --prefix bench --bulk-mb 8 --workers 4 --progress json --approve proceed`.+6. Stops the server on exit, including on Ctrl-C, and removes the arm worktrees.++Output goes to `benchmarks/ingest-neural-bulk/results/<date>/`: `meta.json`,+`report.md`, and per run `run-<arm>-<i>/{server.log,client.log,summary.json}`.+Set `RESULTS_DATE` to avoid clobbering an earlier run on the same day.++## Control-vs-control precondition++Before any candidate number is published, run the harness with the same SHA+for both arms. The two arms' median `bulk total_ms p95` must differ by less+than 10%. `report.md` prints PASS or FAIL for this. If it fails, the host is+too noisy for the effect being measured.++## Corpus++`corpus/okta-bulk.tar.zst` is 203 files, 11,257,358 bytes raw, 461 KB+compressed, stored directly in git (the repo has no `.gitattributes` and no+binary-size policy). It holds the three Okta Management OpenAPI YAML copies+(golang, nodejs, python SDKs) and the first 200 `okta-sdk-golang/okta/docs`+files by sorted name. `MANIFEST.tsv` gives upstream repo, commit, source path,+in-tarball path, bytes and sha256 for each file. The golang spec lives under+`.generator/` upstream; `autoindex` skips hidden directories, so it is stored+as `generator/` and `source_path` records the original.++## Columns in `summary.json` and `report.md`++| field | source |+|---|---|+| `wall_s` | `wall_s` in the client's `xerj-done` JSON object (plain `wall=` as fallback) |+| `wall_wrapper_s` | shell-measured client wall time, for cross-checking |+| `records` | `records` in the same `xerj-done` object |+| `records_per_s` | `records / wall_s` |+| `bulk_total_ms_p50`, `_p95` | nearest-rank percentiles of `total_ms=` on server `XERJ_PROF bulk` lines (`engine/crates/xerj-engine/src/bulk.rs`) |+| `embed_ms_p50`, `_p95` | `embed_ms=` anywhere on a server `XERJ_PROF` line. Null on builds that do not emit it. |+| `retries_window_failed` | server log count of `semantic embedding window failed; retrying per item` |+| `retries_embed` | server log count of `embed retry` |+| `client_bulk_retries` | `bulk_retries` in the `xerj-done` object (items re-sent after a 429); absent means 0 |+| `retries` | sum of the three above |+| `http_429` | log lines in server and client logs containing the whole word `429`. The server does not log requests, so this is indicative; `client_bulk_retries` is the authoritative backpressure count. |+| `peak_rss_kb` | `VmHWM` from `/proc/<server pid>/status`, read after the client exits and before the server stops |+| `cpu_s` | `utime + stime` from `/proc/<server pid>/stat`, divided by `CLK_TCK` |+| `ok` | client exit 0 and `xerj-done` reported `ok=true`. Failed runs are listed in the report and excluded from statistics. |++`report.md` shows, per metric, each arm's median and p95 across ok runs (with+N below 20 the nearest-rank p95 is the max), the control arm's max-min spread,+and the candidate-minus-control median delta. A delta is marked `EXCEEDS+SPREAD` only when its magnitude is larger than the control spread. A spread of+zero (for example N=1) flags any difference, so read it with N of at least 5.++## Not covered++- The client is not confined to the CPU quota; only the server is.+- `passages/s` is not computed: the server does not report a passage count+ that the harness can read.+- Throughput is end-to-end with a cold index, per `docs/CONTRIBUTION_REVIEW.md`+ matched-build guidance. Do not compare it with warm or partial-corpus numbers.diff --git a/benchmarks/ingest-neural-bulk/corpus/MANIFEST.tsv b/benchmarks/ingest-neural-bulk/corpus/MANIFEST.tsvnew file mode 100644index 000000000..d1682c134--- /dev/null+++ b/benchmarks/ingest-neural-bulk/corpus/MANIFEST.tsv@@ -0,0 +1,204 @@+upstream_repo commit source_path path bytes sha256+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db .generator/okta-management-APIs-oasv3-noEnums-inheritance.yaml okta-bulk/okta-sdk-golang/generator/okta-management-APIs-oasv3-noEnums-inheritance.yaml 3556393 0423a06b46f37ec8eda61fd0fd6934d2cba321946b68f0e5850f70a600306048+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AAGUIDAuthenticatorCharacteristics.md okta-bulk/okta-sdk-golang/okta/docs/AAGUIDAuthenticatorCharacteristics.md 4036 c6bacf320cc19c7bc41b17dd2fc69bcf9c38d0a8d83c0e55120253a4883e7d2f+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AAGUIDGroupObject.md okta-bulk/okta-sdk-golang/okta/docs/AAGUIDGroupObject.md 2519 7eb23678f829d777a6769b2ab94e8f87688153d7d8374c1172815a8f05ec9733+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/APIServiceIntegrationInstance.md okta-bulk/okta-sdk-golang/okta/docs/APIServiceIntegrationInstance.md 8571 530c6e9f3bfa70a674b206b511748e6abbc523128dbd67e5f0a6a34e72c18e0f+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/APIServiceIntegrationInstanceSecret.md okta-bulk/okta-sdk-golang/okta/docs/APIServiceIntegrationInstanceSecret.md 6041 502f4a11a66c0abce7fc7d55495b46da03c22f1adc6894acdda1f349f8b1e258+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/APIServiceIntegrationLinks.md okta-bulk/okta-sdk-golang/okta/docs/APIServiceIntegrationLinks.md 3262 85becc1500678e45d3998ae9b717f4d71da1899520c89ee0829398cecb52dc36+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/APIServiceIntegrationSecretLinks.md okta-bulk/okta-sdk-golang/okta/docs/APIServiceIntegrationSecretLinks.md 3610 62d7620f2ce6e4f57b2e4afc1809ff6b2627eedffa8b9661fe51d7a55ecb9a32+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/APNSConfiguration.md okta-bulk/okta-sdk-golang/okta/docs/APNSConfiguration.md 3882 7633252a54edbeeecf2a84c658743b5f49d919ba1c9b39c808b4824f5950c167+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/APNSPushProvider.md okta-bulk/okta-sdk-golang/okta/docs/APNSPushProvider.md 1791 9dace36c18f619b36ffc62c68cf8f5faa004ad43586db05e28a11f909b0f7c0f+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AccessPolicy.md okta-bulk/okta-sdk-golang/okta/docs/AccessPolicy.md 2775 f5773ca9a7dbe0e9bdf2d61cffa41a5d14bb1a4eaa6f003544227dcc0acc6ea1+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AccessPolicyAllOfEmbedded.md okta-bulk/okta-sdk-golang/okta/docs/AccessPolicyAllOfEmbedded.md 2119 616f50f1a8dc41f18b5630a36b344901e2f349ff7928053271acce5d76214111+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AccessPolicyConstraint.md okta-bulk/okta-sdk-golang/okta/docs/AccessPolicyConstraint.md 7141 8e963f24c5bfcec2245fdc2f2ad487b85fec2191c7219f0b5273e882d5cd79a7+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AccessPolicyConstraints.md okta-bulk/okta-sdk-golang/okta/docs/AccessPolicyConstraints.md 2645 311415bd0f63e1b8493306f699370234052382ad0b675141f5ee373af3a096dc+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AccessPolicyLink.md okta-bulk/okta-sdk-golang/okta/docs/AccessPolicyLink.md 4181 fe1c923f520f4cd7e710c43d6661104db1fbcaf6a9b9b6e711a5871806d9d7a3+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AccessPolicyRule.md okta-bulk/okta-sdk-golang/okta/docs/AccessPolicyRule.md 2530 0528c4dfb52d73d28774d9d1a1ec9d92e2e48f8cfe1d4f51c7bab17bdc4234df+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AccessPolicyRuleActions.md okta-bulk/okta-sdk-golang/okta/docs/AccessPolicyRuleActions.md 1912 1e6861c1ab25672af01c3613b46cdabb59ec0888bdd6249722da65bd37418ba6+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AccessPolicyRuleApplicationSignOn.md okta-bulk/okta-sdk-golang/okta/docs/AccessPolicyRuleApplicationSignOn.md 3704 e9a9eccaabc865ec4dbd4b9b9fc44a66bab809f018c6d91ec739195b3f665ad5+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AccessPolicyRuleConditions.md okta-bulk/okta-sdk-golang/okta/docs/AccessPolicyRuleConditions.md 7658 ffd657d4b9a76c6c9a074a0e11d8aabf77da820a63125a34c45c59291fb1e5a5+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AccessPolicyRuleCustomCondition.md okta-bulk/okta-sdk-golang/okta/docs/AccessPolicyRuleCustomCondition.md 2214 ce7053ae6d28bedcc0d006c01d1586062ded352f8e7688c14424e1d81f00f091+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AccountLinkedEnrollment.md okta-bulk/okta-sdk-golang/okta/docs/AccountLinkedEnrollment.md 6414 41ba6321f4449bb06434814b747619e0235959309dd126cd767e18c9fa6d1a39+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AccountLinkedEnrollmentEmbedded.md okta-bulk/okta-sdk-golang/okta/docs/AccountLinkedEnrollmentEmbedded.md 2103 f3e14910518d7c32e58bcbcf127b13f6ba65660f4b5ee47a012e4688a5e2d4f9+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AccountLinkedEnrollmentProfile.md okta-bulk/okta-sdk-golang/okta/docs/AccountLinkedEnrollmentProfile.md 2090 354544829098d08d878b1669e3c09b31f60b66ef9583bf167ed586557adaed6d+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AcsEndpoint.md okta-bulk/okta-sdk-golang/okta/docs/AcsEndpoint.md 1909 6d451ea531b6ab0c0c4892f771b13d9b39e344126548645a9f1ab338a5c3e38a+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/Actions.md okta-bulk/okta-sdk-golang/okta/docs/Actions.md 1720 45b586f31d3deacbfcf2a255e6729c3ff2c13857eb4737f306d592bbe55383b9+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ActivateFactor200Response.md okta-bulk/okta-sdk-golang/okta/docs/ActivateFactor200Response.md 8894 7254bdb64130eb35a7990f7093929f371541840dc21884973fdde86c598caa51+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AddGroupRequest.md okta-bulk/okta-sdk-golang/okta/docs/AddGroupRequest.md 1695 2527be18faaf5c7fd7a0f8128622771295a9bc1ba4766b6dbae49c83f64fc33d+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AddJwkRequest.md okta-bulk/okta-sdk-golang/okta/docs/AddJwkRequest.md 5501 43d1325fa5b7988fce0dc86e57cac8baf5d2d7f72005add54c56f85595c6fdc1+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AdminConsoleSettings.md okta-bulk/okta-sdk-golang/okta/docs/AdminConsoleSettings.md 3138 e1ea6ca01ca9a9c3aa31db87d2c4b1eb5090fec2938afdc2701517d7c73d247c+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/Agent.md okta-bulk/okta-sdk-golang/okta/docs/Agent.md 8978 f7c63b0c1031ec1c670907b8523a839da27b6b26195874162ef423eacffdfc85+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AgentAction.md okta-bulk/okta-sdk-golang/okta/docs/AgentAction.md 1997 7ef79c74007504f3977f784cd75a70f94cfa8d35333cddee70be2c1f47277b44+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AgentPool.md okta-bulk/okta-sdk-golang/okta/docs/AgentPool.md 6272 c0da4167c9dd9758a853e28fd85855ac3dd0db9478fef9ac873ed668282a2a53+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AgentPoolUpdate.md okta-bulk/okta-sdk-golang/okta/docs/AgentPoolUpdate.md 9128 a72178c1a12f61abfcd8fb359877c05d65d3cc309b479dbc3d4a8ff5b97fa181+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AgentPoolUpdateSettingRequest.md okta-bulk/okta-sdk-golang/okta/docs/AgentPoolUpdateSettingRequest.md 6717 e3307be6abcb800c8b84679833128c6299164ecbbd96591bfc1ede69f279c907+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AgentPoolUpdateSettingResponse.md okta-bulk/okta-sdk-golang/okta/docs/AgentPoolUpdateSettingResponse.md 6897 48e00c92aef166d1766c6fb91c71d485a59c4a0263301ec752ecfb1bf146c810+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AgentPoolsAPI.md okta-bulk/okta-sdk-golang/okta/docs/AgentPoolsAPI.md 31542 f02a196154ddc46add8e3ad82f73435f7ecfc8a1c236df595ddd3beb5574a01c+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AndroidDeviceTrust.md okta-bulk/okta-sdk-golang/okta/docs/AndroidDeviceTrust.md 7716 da37a14a379b0c671890b6c50b729446225335f9fe2edec0a6517f9faf3612a6+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApiServiceIntegrationsAPI.md okta-bulk/okta-sdk-golang/okta/docs/ApiServiceIntegrationsAPI.md 22782 3e3b0f984762cbb957059e130eaa9702d111d018d684d3972337f787c4e890b5+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApiToken.md okta-bulk/okta-sdk-golang/okta/docs/ApiToken.md 7200 07aa79c01678a080e8a4592fee2bc9529bcfac9fe7091d2fa780782c2ecbd5c1+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApiTokenAPI.md okta-bulk/okta-sdk-golang/okta/docs/ApiTokenAPI.md 8494 66659169d309253ca9e98f62aea736effac97e8202c1ad59265df0d848799865+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApiTokenNetwork.md okta-bulk/okta-sdk-golang/okta/docs/ApiTokenNetwork.md 3066 ad11339c1b5b04f5d746d5a65747830ca7f06b00376e116aca0584c547be0da4+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApiTokenUpdate.md okta-bulk/okta-sdk-golang/okta/docs/ApiTokenUpdate.md 4382 2fbf1f7e85053e710f2b696bcd704a037c71ea09f5029c3f58f7f48500bb1585+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AppAccountContainerDetails.md okta-bulk/okta-sdk-golang/okta/docs/AppAccountContainerDetails.md 8534 8e47f82c6e608d8764e21e06ec54e2d6d224f7929b3e113fc9adb895bdd8c6e7+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AppAccountContainerLink.md okta-bulk/okta-sdk-golang/okta/docs/AppAccountContainerLink.md 2465 465c9e8eee28eaf32b3c3679a79e26fddbcc3754eabe083a0b06d7de317b8541+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AppAndInstanceConditionEvaluatorAppOrInstance.md okta-bulk/okta-sdk-golang/okta/docs/AppAndInstanceConditionEvaluatorAppOrInstance.md 3450 49f474af04b653f87a8c3583d1ea196b52e68321e23e5040d283ad4302a62f14+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AppAndInstancePolicyRuleCondition.md okta-bulk/okta-sdk-golang/okta/docs/AppAndInstancePolicyRuleCondition.md 3115 92ceac063b9e1231e34822efc017426efab1e6955f557160336798c58780be25+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AppConfig.md okta-bulk/okta-sdk-golang/okta/docs/AppConfig.md 1397 959adcbc5a7c95facce7e924cb800c9b06a02ec94ef8b7fc45ad155fbadacb97+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AppConfigActiveDirectory.md okta-bulk/okta-sdk-golang/okta/docs/AppConfigActiveDirectory.md 3746 4ed0efa60fc3bdc8452bf7b97ddbe07b07a5e7d55b2d01d4b24bc811013d147f+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AppConfigOffice365.md okta-bulk/okta-sdk-golang/okta/docs/AppConfigOffice365.md 1723 f5195133a68e1098c417f7360987fa0c160d6d69af98e72a2a43347a947f800d+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AppConnectionUserProvisionJWKList.md okta-bulk/okta-sdk-golang/okta/docs/AppConnectionUserProvisionJWKList.md 1746 aebe7268ed0f1b299660ae50b0bacb47c7479e12f4cac793ffe9a274fe387dcf+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AppConnectionUserProvisionJWKResponse.md okta-bulk/okta-sdk-golang/okta/docs/AppConnectionUserProvisionJWKResponse.md 1930 7fee8df5723c2170f1ad2ed3799969e7ffe969b346c18868bfb0158a682377a0+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AppCustomHrefObject.md okta-bulk/okta-sdk-golang/okta/docs/AppCustomHrefObject.md 3565 f81061ff0080e230430fe2357003b2a4dd0e870bd10fe3ef274d2fdb59b31d54+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AppCustomHrefObjectHints.md okta-bulk/okta-sdk-golang/okta/docs/AppCustomHrefObjectHints.md 1723 25567a8151b634129a223dde077f08a369f640aa48b6438fe04bb244cf6d8676+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AppGroup.md okta-bulk/okta-sdk-golang/okta/docs/AppGroup.md 2032 1e3c86652fe3c1a82d2a82ff46e39755c5fabe0cf409c517720c83b5ab08d52b+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AppInstancePolicyRuleCondition.md okta-bulk/okta-sdk-golang/okta/docs/AppInstancePolicyRuleCondition.md 2553 784ee34b3a239d65f6f019e485806393807d31d454b001597494e9ca62292793+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AppInstanceSearchType.md okta-bulk/okta-sdk-golang/okta/docs/AppInstanceSearchType.md 1597 f27b34aeafa1c274b4e23a6e309a0f62eada972493d0ef9995c45cbba88e3947+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AppPropertiesValue.md okta-bulk/okta-sdk-golang/okta/docs/AppPropertiesValue.md 2270 21e34106df54edb50646cb940823a62b3770f997fac9cb0c8b108b6f61ac40c9+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AppResourceHrefObject.md okta-bulk/okta-sdk-golang/okta/docs/AppResourceHrefObject.md 2303 bc0148c512269a1217d03ab085c135d3bfe42322b00f497cc66250a3ebb35f41+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AppServiceAccount.md okta-bulk/okta-sdk-golang/okta/docs/AppServiceAccount.md 11416 efd571e92e708df9286863a1376569449adb86f0ffa609e2fdacc71dfd833b00+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AppServiceAccountCredentials.md okta-bulk/okta-sdk-golang/okta/docs/AppServiceAccountCredentials.md 2489 4fb611baa904fdc9a2e67ba9676172fc06ec33c6cd69cd3e3ec8cbe8e173341a+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AppServiceAccountForUpdate.md okta-bulk/okta-sdk-golang/okta/docs/AppServiceAccountForUpdate.md 4255 18512b93e9f6ba9807dfeba4e93d849a80e80fb3c9cb1585311e8681ef35aff7+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AppUser.md okta-bulk/okta-sdk-golang/okta/docs/AppUser.md 12451 04d8a1accc380c765704f2f1c381f4001897fc1a6ce25b5d95bf340cfae1032c+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AppUserAssignRequest.md okta-bulk/okta-sdk-golang/okta/docs/AppUserAssignRequest.md 13348 e152eb7fbd2bd162aa3b38a3b8e0a9628bece445f7c5f9a6d9d5579c43c87226+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AppUserCredentials.md okta-bulk/okta-sdk-golang/okta/docs/AppUserCredentials.md 3235 395dad452a1fbb1c3947afe08a7807add8bff8fcd1e4a2d7371be6ee42a75057+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AppUserCredentialsRequestPayload.md okta-bulk/okta-sdk-golang/okta/docs/AppUserCredentialsRequestPayload.md 2004 fb9b2473ab99122c12d49d309b964498c4008b4ff36d0afe7e309e3a141bb4d3+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AppUserPasswordCredential.md okta-bulk/okta-sdk-golang/okta/docs/AppUserPasswordCredential.md 1744 2d48ecae3657aef99e22c24ba89b2bb3ed34994b0f86fa34afdca360c70e3a20+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AppUserProfileRequestPayload.md okta-bulk/okta-sdk-golang/okta/docs/AppUserProfileRequestPayload.md 2283 bf9d80366bf2309dbd994a5942a3e1eae43c164e2c0348b3cb11c8c603d583d5+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AppUserUpdateRequest.md okta-bulk/okta-sdk-golang/okta/docs/AppUserUpdateRequest.md 2960 8969b7bc60fd3f11e3508544fbd63bca0ca7649058707e1faa412c61e6e00ffb+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AppleClientSigning.md okta-bulk/okta-sdk-golang/okta/docs/AppleClientSigning.md 3163 84ccc831f1234f1af6e1d7c2b13bdf4cf29572856a3b79ce8058e3fd977ba194+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/Application.md okta-bulk/okta-sdk-golang/okta/docs/Application.md 13774 44cff976f73fa7cf3ad691a18004676605830c5658bb45cbdf0244f56b68ad68+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApplicationAPI.md okta-bulk/okta-sdk-golang/okta/docs/ApplicationAPI.md 18086 53f7bbc36634110eee771fae8025aea15a164f7a8d78403ad2a6ca58daaf9570+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApplicationAccessibility.md okta-bulk/okta-sdk-golang/okta/docs/ApplicationAccessibility.md 4116 62d180b669691c4aa3aafb692ca854a2e1a468ec4698a1c131954fd8d807a321+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApplicationConnectionsAPI.md okta-bulk/okta-sdk-golang/okta/docs/ApplicationConnectionsAPI.md 14361 916e6c5e2df0f9165a70c1d7b7b1f79afcc8a5025611a1314e31626604971e75+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApplicationCredentials.md okta-bulk/okta-sdk-golang/okta/docs/ApplicationCredentials.md 2830 cb2cd0e34fda866e6d150817729e3bfbb6a6d3f210aea33609d06c047ba044b7+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApplicationCredentialsOAuthClient.md okta-bulk/okta-sdk-golang/okta/docs/ApplicationCredentialsOAuthClient.md 6700 020ba72186f793fbf0635ec126ad9d3581d9dd3f21e7326b664efe33806ebf39+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApplicationCredentialsSigning.md okta-bulk/okta-sdk-golang/okta/docs/ApplicationCredentialsSigning.md 5062 f5e3d82789e0dc842be18e865be340ab965f255e8eebd37b39bde1047fe2d388+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApplicationCredentialsUsernameTemplate.md okta-bulk/okta-sdk-golang/okta/docs/ApplicationCredentialsUsernameTemplate.md 5899 5ac056e90ff5eb284f21b578f05da72a4fcf1f618933d49ef846af47ad642f02+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApplicationEmbedded.md okta-bulk/okta-sdk-golang/okta/docs/ApplicationEmbedded.md 1843 944b30ba693db22e5499c7291ef66700b44dedbe9e264bd81e0d6381dd288d40+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApplicationExpressConfiguration.md okta-bulk/okta-sdk-golang/okta/docs/ApplicationExpressConfiguration.md 3063 c27aeed42eae5d8d16cd3cb6cedce2ee511a6214768c37fb3ff38489d88bfe0a+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApplicationFeature.md okta-bulk/okta-sdk-golang/okta/docs/ApplicationFeature.md 4156 b22bb5d352e0151ab081b813a914d76919904f9092d9618c05e568b37b3dd764+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApplicationFeaturesAPI.md okta-bulk/okta-sdk-golang/okta/docs/ApplicationFeaturesAPI.md 7353 976415ed53c8c7e8bc8fb2360fc91be967895aa2c637a925daf7f3fd89350225+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApplicationGrantsAPI.md okta-bulk/okta-sdk-golang/okta/docs/ApplicationGrantsAPI.md 9019 ce7081b09161c4bd904e26dfc96278cb8c430001fc11ae192a1d6d7aa07dc54b+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApplicationGroupAssignment.md okta-bulk/okta-sdk-golang/okta/docs/ApplicationGroupAssignment.md 7170 06dc077d878a5c26ae6acc00ed0629f5e0e78e319ee7ea1a3e142d17de628e7a+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApplicationGroupAssignmentLinks.md okta-bulk/okta-sdk-golang/okta/docs/ApplicationGroupAssignmentLinks.md 3335 77c3ea4e2021ef0962053ad79a1287ec0c59e9cbec512dc6248a3c004bf83e86+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApplicationGroupsAPI.md okta-bulk/okta-sdk-golang/okta/docs/ApplicationGroupsAPI.md 14470 b15ff0e0920f257aa59f7289ae269d61813c98c8365b27b59801fcd1255fea6e+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApplicationInterclientTrustMappingsAPI.md okta-bulk/okta-sdk-golang/okta/docs/ApplicationInterclientTrustMappingsAPI.md 10999 1f60b12829555e219f357ff8c66c28cfd89f5a4e798b5736da719d8f46580b63+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApplicationLayout.md okta-bulk/okta-sdk-golang/okta/docs/ApplicationLayout.md 4962 2648895a1b884da9118d6fdbf4174fe35dc73bf604d6013c78774db024361554+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApplicationLayoutRule.md okta-bulk/okta-sdk-golang/okta/docs/ApplicationLayoutRule.md 2515 ea0d6ea31500327a9eab43a9d965b2d70fc68127da92125c48350f4143a9a6d5+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApplicationLayoutRuleCondition.md okta-bulk/okta-sdk-golang/okta/docs/ApplicationLayoutRuleCondition.md 2553 7f458fb78713b1d39cd2edac4b6ffc89cf17dfac3cd99d6ed8071f240f576c4d+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApplicationLayouts.md okta-bulk/okta-sdk-golang/okta/docs/ApplicationLayouts.md 1723 cd950c715b238847d10798bc27a1a71da9978af7051e05849644093b7f3c487f+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApplicationLayoutsLinks.md okta-bulk/okta-sdk-golang/okta/docs/ApplicationLayoutsLinks.md 3263 f2c9ba04ca3620121eb579b17f729cca77755b3b1a2d929462d56e4cebf2cf35+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApplicationLicensing.md okta-bulk/okta-sdk-golang/okta/docs/ApplicationLicensing.md 1755 81158cf89e2b1da4469b27caff5a985654e5e76dea32343e0d7c97935dd9e4d4+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApplicationLinks.md okta-bulk/okta-sdk-golang/okta/docs/ApplicationLinks.md 8000 9f9f680550384c0a26ea30702ae1078c988dac4e603264e6c49a9c04d12966d2+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApplicationLogosAPI.md okta-bulk/okta-sdk-golang/okta/docs/ApplicationLogosAPI.md 3255 3d6f8f7e228088dd4d25cd9b85b71e4dd6dc1207fdbce7f56c3f4833d781936d+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApplicationPoliciesAPI.md okta-bulk/okta-sdk-golang/okta/docs/ApplicationPoliciesAPI.md 2093 1d9531362c534ef76e8c90bfd236089d921bb6b8b6a492e08ae64d2c3d2b9222+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApplicationSSOAPI.md okta-bulk/okta-sdk-golang/okta/docs/ApplicationSSOAPI.md 2304 13eef0d39bac9c1c93104c5714aaaddc3eabf714e0fc9602b704fa4ad1f86768+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApplicationSSOCredentialKeyAPI.md okta-bulk/okta-sdk-golang/okta/docs/ApplicationSSOCredentialKeyAPI.md 19374 88c862f27126078ab769041a638eef176d887a948f5d371d3af7ac1449dcef37+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApplicationSSOFederatedClaimsAPI.md okta-bulk/okta-sdk-golang/okta/docs/ApplicationSSOFederatedClaimsAPI.md 11057 b41dcafb4270ddf8ed21b38b152294580b9d262f73a61ac665fad9b0bd927a97+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApplicationSSOPublicKeysAPI.md okta-bulk/okta-sdk-golang/okta/docs/ApplicationSSOPublicKeysAPI.md 26312 765fba4236a09e0961f738cb40d9ac7ab0fc3661f2cd800ed2a9fd59ec18c812+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApplicationSettings.md okta-bulk/okta-sdk-golang/okta/docs/ApplicationSettings.md 6411 e1fa64a4a2ee4d57e02108699742ebd4c6c712581d85d142fcf643a12737f1d2+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApplicationSettingsNotes.md okta-bulk/okta-sdk-golang/okta/docs/ApplicationSettingsNotes.md 2488 f06fca3e953d1d6f52269387a072a7351f58154f2545c896df0b6fe9486c4c4b+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApplicationSettingsNotifications.md okta-bulk/okta-sdk-golang/okta/docs/ApplicationSettingsNotifications.md 1961 d703feb1e6bc16c4c5241fca7fcc034a9a982bc9c732ac1322589a0066182766+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApplicationSettingsNotificationsVpn.md okta-bulk/okta-sdk-golang/okta/docs/ApplicationSettingsNotificationsVpn.md 3582 be8cb516a659de0ab90efebf7ff0ee68c56b071d472ef27d4eb2b91e777b8a50+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApplicationSettingsNotificationsVpnNetwork.md okta-bulk/okta-sdk-golang/okta/docs/ApplicationSettingsNotificationsVpnNetwork.md 3800 4a20cdcdaad103551eacd07bd9e09d291502c961b644c6bbfcf13e258fffa669+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApplicationTokensAPI.md okta-bulk/okta-sdk-golang/okta/docs/ApplicationTokensAPI.md 9459 ff66399cfc71db97cfdb6d63c147672be9c810ed53e29b1b2b913418a0d7fbf1+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApplicationUniversalLogout.md okta-bulk/okta-sdk-golang/okta/docs/ApplicationUniversalLogout.md 4268 2b5aebb3d8bd32e3c7aa28ac1123a951208cf2d92f417868c347a69a8d0c6239+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApplicationUsersAPI.md okta-bulk/okta-sdk-golang/okta/docs/ApplicationUsersAPI.md 13991 bd4289614b9b6407fc6021954f0342a4106bfd6937515b4ab3898afa271f23ad+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApplicationVisibility.md okta-bulk/okta-sdk-golang/okta/docs/ApplicationVisibility.md 4206 c5a3247e1aa627fc4acef170a4185707b3186b0543451bc0ef72997ca8ae1462+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/ApplicationVisibilityHide.md okta-bulk/okta-sdk-golang/okta/docs/ApplicationVisibilityHide.md 2424 8103edae79c2451a7a8b7cf38542a918753609535a4c5d0ca26fc3d891460a34+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AssignGroupOwnerRequestBody.md okta-bulk/okta-sdk-golang/okta/docs/AssignGroupOwnerRequestBody.md 2417 081d0578533f3d33b7b0fa1daeebb0a9f8d2dcbd5e8dbf101010f45de1051e38+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AssignRoleRequest.md okta-bulk/okta-sdk-golang/okta/docs/AssignRoleRequest.md 3018 7582381cdb957a7a0ad9fa96272fc0643b759bfd0dce6d3775d87695c4272307+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AssignRoleToClientRequest.md okta-bulk/okta-sdk-golang/okta/docs/AssignRoleToClientRequest.md 2953 8e9c652b322ba1811db99a7370117a7d793e6bf178c2e14def83d6e132c0706d+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AssignRoleToGroup200Response.md okta-bulk/okta-sdk-golang/okta/docs/AssignRoleToGroup200Response.md 10489 d5bc6cc45e3a2986cadeee28d01a280af2f809f6bde43b4850d6abaaf2faf666+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AssignRoleToGroupRequest.md okta-bulk/okta-sdk-golang/okta/docs/AssignRoleToGroupRequest.md 2933 d85afede5c8f6572b10e487e6c099755a1b7b0366ab3af96f078e859e69990db+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AssignRoleToUser201Response.md okta-bulk/okta-sdk-golang/okta/docs/AssignRoleToUser201Response.md 10435 6342d66fcca6b88e7fe0d4502f39d06a7a3023b50f48e67161f2281db5117ef9+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AssignRoleToUserRequest.md okta-bulk/okta-sdk-golang/okta/docs/AssignRoleToUserRequest.md 2913 53b2af47e481c42b5eb434a34c8363d0584d30c127bdc9af1675684218d98134+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AssignUserToRealm.md okta-bulk/okta-sdk-golang/okta/docs/AssignUserToRealm.md 1657 9a6d082c5e18767b27a71d10b0274f65ce950ce759434f27fd620122be16dcf2+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AssignedAppLink.md okta-bulk/okta-sdk-golang/okta/docs/AssignedAppLink.md 7734 99a4618acd6233f2af20882f95fceb735ff698b6a5a81a4187bbbbfa8e09ba0a+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AssociatedDomainCustomizationsAPI.md okta-bulk/okta-sdk-golang/okta/docs/AssociatedDomainCustomizationsAPI.md 14938 7684436535b51b90c93fcbba7b0c7128ec1e2378892029088ad64f4ad7d4c351+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AssociatedServerMediated.md okta-bulk/okta-sdk-golang/okta/docs/AssociatedServerMediated.md 1793 f2375df24aefc07f2bec5d8b31d06e3000d02f1237b8dc9a7ba7a71c4592233c+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AssuranceMethod.md okta-bulk/okta-sdk-golang/okta/docs/AssuranceMethod.md 4663 eaa1fb7359d63fbeb393689fc542389930366b93ac3bffb3a5afba6908b8e5fc+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AttackProtectionAPI.md okta-bulk/okta-sdk-golang/okta/docs/AttackProtectionAPI.md 7581 daa06319e12b6976b4ed1a4b0d45a5441a59070032b4f497c7020b073de7556e+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AttackProtectionAuthenticatorSettings.md okta-bulk/okta-sdk-golang/okta/docs/AttackProtectionAuthenticatorSettings.md 2566 e85d6a467fb03efb5321ec415785e2dbfb10ccc6644424824d2f8baa55bbb939+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AttestationRootCertificatesRequestInner.md okta-bulk/okta-sdk-golang/okta/docs/AttestationRootCertificatesRequestInner.md 1931 83a37cbc458f0aa7ac395537a18af0be1c293b3f4119975b9d42f7cbeba6214b+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AttestationRootCertificatesResponseInner.md okta-bulk/okta-sdk-golang/okta/docs/AttestationRootCertificatesResponseInner.md 4135 e578b339c8edc009141c9e2a5018937aa12c477c223cd2e6b70cdcdae0d8bb7a+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthServerLinks.md okta-bulk/okta-sdk-golang/okta/docs/AuthServerLinks.md 6038 7ab70bb87ed130539be89d71fb1b2c622c2df8f62befb38da6cfc8ea1d5868f3+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticationMethod.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticationMethod.md 7106 15e3c78a34102261c53cf59e3cf246aaa39f7e2aed73616c3a1b21738969bd97+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticationMethodChain.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticationMethodChain.md 4045 dbe5ba524341a55398bcfc6584465e52b95c0f0a77fa85938196acbdee07499a+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticationMethodChainMethod.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticationMethodChainMethod.md 3183 ec4e9973bbb5946e7b67dda53c5fe42e697a5631424e0193c98267555934af24+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticationMethodObject.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticationMethodObject.md 3410 a853cf8fcf7fdba982c6aa57ff4a8dfc12fb7cc5e914dd425fa5b42c32e0c169+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticationProvider.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticationProvider.md 2385 6765890a295172165d998a93db325e6fc05a2d4fe5b90b85c25e366ea07681a2+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticationProviderWritable.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticationProviderWritable.md 2515 4665a464afb4b319404c015533ac8d815d2fac13fc6238c901a25c0597f28b02+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorAPI.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorAPI.md 42518 7d824c84b5a53f730493bc1e495bd7510a20ece9474848a327de42e205f71edc+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorBase.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorBase.md 7201 29b21aaa79f38b5439d0cfd29cf589e7d2a6c33dc365781ee774d0bde02ed002+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorEnrollment.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorEnrollment.md 7418 d1809258947cc7c7be767650a5604473a8a1fa0395efe394cac5c5ef237f236e+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorEnrollmentCreateRequest.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorEnrollmentCreateRequest.md 2628 254d5f131a65f20bf5f955883b3c0d2d352b594919fdc2d5a821a2eef7c1a051+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorEnrollmentCreateRequestTac.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorEnrollmentCreateRequestTac.md 2848 022c50c28d9c01d42fde8212df8e9684c401f44bdf36fba03a94d999542e73f4+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorEnrollmentLinks.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorEnrollmentLinks.md 3526 8b7e3a147fe12823292fa2bf7facb05da04e1f8d1fdeea4041b35c60451d1b62+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorEnrollmentPolicy.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorEnrollmentPolicy.md 2928 f43107664703381b41211c7c4f038afc079d065adf37b23030f5de90bcd887aa+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorEnrollmentPolicyAuthenticatorSettings.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorEnrollmentPolicyAuthenticatorSettings.md 5594 f1692fa7ec67d430cb96e08531da31215e9807414df06a0b77ec530657f54e6a+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorEnrollmentPolicyAuthenticatorSettingsConstraints.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorEnrollmentPolicyAuthenticatorSettingsConstraints.md 2978 09e03c3aa68a1aae5d32f5a3aedcf4aac9d764b185bcc7acd4ea4f34acbda773+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorEnrollmentPolicyAuthenticatorSettingsEnroll.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorEnrollmentPolicyAuthenticatorSettingsEnroll.md 6636 2297547b9334d6b26cd29543c9e2e995b381ae512f5e9e3e82121fac1efbebfe+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorEnrollmentPolicyConditions.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorEnrollmentPolicyConditions.md 2189 f8f87df456f9025e44ea936fb4b19583782bfff8d899cd1896851a1a69607c6c+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorEnrollmentPolicyConditionsAllOfPeople.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorEnrollmentPolicyConditionsAllOfPeople.md 2384 b1e9b376fd88aecb903448d2e9b4c487a2486a741005b9a928a12a63ab67f057+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorEnrollmentPolicyConditionsAllOfPeopleGroups.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorEnrollmentPolicyConditionsAllOfPeopleGroups.md 2256 ca07fce7bbf3e35dc6f5d954525a542b0a9b393af4d02d706e58a212b260f832+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorEnrollmentPolicyRule.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorEnrollmentPolicyRule.md 3023 6ce0f573ab91332b6543cf3dc4d501d824bcfd8a600036beff40cf06e9a2f97b+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorEnrollmentPolicyRuleActionEnroll.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorEnrollmentPolicyRuleActionEnroll.md 2014 018e50a52f45ec9aeacd408c594e6cbd096b14d518362f3a754cd69fcb93554e+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorEnrollmentPolicyRuleActions.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorEnrollmentPolicyRuleActions.md 2179 35aa31003e4f554375bd6c5889b85c67128e4e7040622a9f419a8e096d985639+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorEnrollmentPolicyRuleConditions.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorEnrollmentPolicyRuleConditions.md 3925 43e84785e78057bc570369ed94e97f25fa2210d0936d428c62cf2bf50d832f5e+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorEnrollmentPolicyRuleConditionsPeople.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorEnrollmentPolicyRuleConditionsPeople.md 2343 d512704d759556dea5b795deac57374c5b27598fdf4aa5b6d6edd9fb1a2205e1+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorEnrollmentPolicyRuleConditionsPeopleUsers.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorEnrollmentPolicyRuleConditionsPeopleUsers.md 2225 ae69a9811ded73a7f485954438ac5ff1b034f31de3d5e3c71f83b974f87b5f68+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorEnrollmentPolicySettings.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorEnrollmentPolicySettings.md 3972 420c8c5e1927aa5a94bc6536f2717188dde21020399565fa0094bab28f8d1f7a+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorIdentity.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorIdentity.md 1638 38d89f953eba5d757ff26e612c14b8005d29f0050e87b02d57b55793cb287166+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorKeyCustomApp.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorKeyCustomApp.md 3873 e3e9bdfb93ce8aacdd5bc77035ea4fb68f7f58dbe25944c633f4557f74f4a76c+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorKeyCustomAppAllOfProvider.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorKeyCustomAppAllOfProvider.md 2988 87f4d59be65000a5f3335aaa2ae4819615759878c22049881c8ef523033e2120+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorKeyCustomAppAllOfProviderConfiguration.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorKeyCustomAppAllOfProviderConfiguration.md 3335 729607147ae878eba4f81b01beaba1c85e9031d852bae16f61c44771d5e02c3c+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorKeyCustomAppAllOfProviderConfigurationApns.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorKeyCustomAppAllOfProviderConfigurationApns.md 4347 aea8f9cd18ca289c06c60765e4b61156c8b5ca799b38a3705b1f36d9a711f567+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorKeyCustomAppAllOfProviderConfigurationFcm.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorKeyCustomAppAllOfProviderConfigurationFcm.md 2245 c3639b111385517e4e96753462a9b3f469b5f131cf4310de65d217369c279d68+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorKeyCustomAppAllOfSettings.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorKeyCustomAppAllOfSettings.md 3304 2b7ae8ddbbd4ab2a3d598858a285b2a7e239d5f12759f1d6c1687d59658d2fca+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorKeyCustomOtp.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorKeyCustomOtp.md 1082 a658f60d437c953b0f9fe67511bc3b34acf79e0b8fd4b16adc485c71209f4a92+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorKeyDuo.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorKeyDuo.md 1831 249dbb2fa26905c122bc398beb7e3f63a2b195cbb41e46a68841e2c0f9ac09aa+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorKeyDuoAllOfProvider.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorKeyDuoAllOfProvider.md 2844 07a68d7277c22f2a2c0f7b84bd1c73812ad62803dd861688b5df1ca9b7b7b3d1+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorKeyDuoAllOfProviderConfiguration.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorKeyDuoAllOfProviderConfiguration.md 4958 df4cef950e8e8f583ae28b32af540c7e1ff5f9f90b4e392af3780a09606cad55+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorKeyDuoAllOfProviderConfigurationUserNameTemplate.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorKeyDuoAllOfProviderConfigurationUserNameTemplate.md 2353 997e809f5b25d7bebb7b003457c49c899187a0a6e02d2f56aea764e0240d696f+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorKeyEmail.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorKeyEmail.md 1871 6379d6fb6d4ca3210853106345ad195e8c155c7c36567ef307b61019c915b00f+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorKeyEmailAllOfSettings.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorKeyEmailAllOfSettings.md 3035 ff0459ee97fe2ededbcb5a84390966541d726ebbb81b02c700c3d16ed01a83b5+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorKeyExternalIdp.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorKeyExternalIdp.md 1104 36725528b5f1d6863f37a1fd9605d00492cfd7f77f12ef9fd37ac4a223dad27e+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorKeyGoogleOtp.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorKeyGoogleOtp.md 1082 750dfe77b4a96b72f48dfbc3cf7ec3ed5eb979861ad332191c969a1b5ec1b7c2+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorKeyOktaVerify.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorKeyOktaVerify.md 1971 79e17fb786b93e5f635aad74814226ab895027060c8eda7d6231e046f5fe356b+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorKeyOktaVerifyAllOfSettings.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorKeyOktaVerifyAllOfSettings.md 7613 ed82d3f9898ba282d4fb403a1855244163fdcd8eda8fba7ee7e3bd7561a2c70b+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorKeyOnprem.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorKeyOnprem.md 1049 36fbbd72a7b84be8222f7e9e9e8a2d9e26cc69e305506f5840f20e54c3ae586a+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorKeyPassword.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorKeyPassword.md 1071 3fb4991095939c2ef31cb3259e4293e8eedda27735bbc42c6588849b79dee9b7+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorKeyPhone.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorKeyPhone.md 1871 c66f059d9c9beaef64a85464a5feb787ee3cc062d06d6c903de496e439a16ba6+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorKeyPhoneAllOfSettings.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorKeyPhoneAllOfSettings.md 1992 459affba41d264a855dfb0d88232060f4333411840946325ee63828eef38dbcd+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorKeySecurityKey.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorKeySecurityKey.md 1104 ae41c578d0405d279c2e1b7cdb3b2fc1303464eb4e23b6db68eec63ada114467+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorKeySecurityQuestion.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorKeySecurityQuestion.md 2036 a2007a9eb9d13e6203b7f21d48ef7018365107159c4ae9cd1325af81657e1ca5+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorKeySmartCard.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorKeySmartCard.md 1082 18711ccf2d57c41c3bf9df4f842a93515d0a1e2906086342c14bf60e28fdc6cf+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorKeySymantecVip.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorKeySymantecVip.md 1104 b383639475b98d0c1f0a84dafdfdb3313c0009493722c5a24679cf0643079ec4+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorKeyTac.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorKeyTac.md 1831 334a6ca10e0a9080dbdd067700d276e962f9bd900628a3705e84c698380b0d6f+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorKeyTacAllOfProvider.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorKeyTacAllOfProvider.md 2844 77c2a58097ddb400f0e94d7b43af9ee1f75084966dc1b0fbcdce940f6161e4e0+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorKeyTacAllOfProviderConfiguration.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorKeyTacAllOfProviderConfiguration.md 6503 deb8ec9b896bc1ea6dcff2087fa35a3a18fe530661174bc9dd3f9a8bd1315579+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorKeyTacAllOfProviderConfigurationComplexity.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorKeyTacAllOfProviderConfigurationComplexity.md 4100 44d15005b3b3020efc447be5900781aa2990894653d4dbd3a50f5db79b52f90e+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorKeyWebauthn.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorKeyWebauthn.md 1071 3dcddd1a26d91dd527ecb5a6d6fb0f910cb416b34282aed8e6100a675eee1495+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorKeyYubikey.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorKeyYubikey.md 1060 24aa780103bf61bf112135de6cc0723748bfdbcf30b3024dea08ac21cc983787+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorLinks.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorLinks.md 3960 ee0207a40059c68d581c241f280c249073cbdc789e0ecf1a0ed867c4b4142344+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorMethodBase.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorMethodBase.md 3138 9489a9c2d81371c9aaf7e2d558477bea640935d73e829cd8bc8f330d48a671d9+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorMethodConstraint.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorMethodConstraint.md 2822 71cdda0db725bf8d04254f2d340c3502fa77fad2b3594e94f8402d96a4eec15d+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorMethodOtp.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorMethodOtp.md 7143 d09bde618651aa29a469c9656e8f5c338a0af1b8d6d20385f9d436bb2cbae8c2+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorMethodPush.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorMethodPush.md 1911 5ff27d9aae0cd08c302abe178f2bce70ca26fb6a471787916dcc0d8420aac664+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorMethodPushAllOfSettings.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorMethodPushAllOfSettings.md 3730 6d2d4b2bf294a47eab422d6354819e01abf5288848d2b74f495d6324bd75e885+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorMethodSignedNonce.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorMethodSignedNonce.md 2051 2d04461da6fc31c32db236df277369957d46245b6061ff8a3aa649d0868df4c5+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorMethodSignedNonceAllOfSettings.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorMethodSignedNonceAllOfSettings.md 4225 0a7ba442063ef1b3dace04cde1899aa21c2f52fa14803665fed6da6e886a4d99+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorMethodSimple.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorMethodSimple.md 1082 fb7302a53b421796fb250ef68e8d0bb6d958fa17f7452d446c296d18a35eebe5+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorMethodTac.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorMethodTac.md 3115 12ca657bd676a72abd38234a32df8236254d684886bba95219d9e1dda929df6d+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorMethodTotp.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorMethodTotp.md 1911 50b3e670ef931c3d39188f622229769e0f459b75cd0c3ff9b05b6b49d94d534f+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorMethodTotpAllOfSettings.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorMethodTotpAllOfSettings.md 4580 16f327821aa80b459e4868a2e43d4a3463cd439e3d4e57d4ccbac493de86aa08+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorMethodWebAuthn.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorMethodWebAuthn.md 1991 5584f447f3537a03243230de9b12f526ba3862b27cc925900e1f4ec7f0a66ad2+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorMethodWebAuthnAllOfSettings.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorMethodWebAuthnAllOfSettings.md 13687 5bc20214ee88f60468185a3e2118b2d9eccea2f5389c6af1c0aa29bb3ad508d2+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorMethodWithVerifiableProperties.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorMethodWithVerifiableProperties.md 2248 008782835b7323c8bd4e76464e36cbdfd4336f44ba58a7730d97bfb3ea0e19ec+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorProfile.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorProfile.md 1699 250f585be56ce57298b926e403ab25a3c92b96641460dc6af05e953307cd9acd+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorProfileTacRequest.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorProfileTacRequest.md 2935 aa621c3bee839b039a4707bae58ef86241d5df172b6ad2857ede02f5a6e44b0a+https://github.com/okta/okta-sdk-golang.git 9b9ccab705d864095ce554834e2959e93efb82db okta/docs/AuthenticatorProfileTacResponsePost.md okta-bulk/okta-sdk-golang/okta/docs/AuthenticatorProfileTacResponsePost.md 3812 8cf04dd4dca64cbe9981e99ad8213f6117e6e5d45d07237afa58560a9eec0f0c+https://github.com/okta/okta-sdk-nodejs.git f4794e3e2d61da08c37c8c348bdb2fc698e90e4a oas-spec/management.yaml okta-bulk/okta-sdk-nodejs/oas-spec/management.yaml 3438586 1bba3fa7ca5dd8dcff930e6775aeb033e4d6f0349dda0583ef8eee58d3f8d637+https://github.com/okta/okta-sdk-python.git 95f2e0fddc23f29a814e85b9e969e85386bf5b03 openapi/api.yaml okta-bulk/okta-sdk-python/openapi/api.yaml 3287809 1b87cadc20cc39756c9ac4a865a690a9b6d57c5c1a34b8b4543a12468610e152diff --git a/benchmarks/ingest-neural-bulk/corpus/SHA256SUMS b/benchmarks/ingest-neural-bulk/corpus/SHA256SUMSnew file mode 100644index 000000000..1d6108196--- /dev/null+++ b/benchmarks/ingest-neural-bulk/corpus/SHA256SUMS@@ -0,0 +1 @@+289afe6af7d2d3a86b98a97895357543d6a45dc360e64accb743bee59a313699 okta-bulk.tar.zstdiff --git a/benchmarks/ingest-neural-bulk/corpus/okta-bulk.tar.zst b/benchmarks/ingest-neural-bulk/corpus/okta-bulk.tar.zstnew file mode 100644index 000000000..d50162fbeBinary files /dev/null and b/benchmarks/ingest-neural-bulk/corpus/okta-bulk.tar.zst differdiff --git a/scripts/bench/neural-bulk-ab.sh b/scripts/bench/neural-bulk-ab.shnew file mode 100755index 000000000..577d449ef--- /dev/null+++ b/scripts/bench/neural-bulk-ab.sh@@ -0,0 +1,467 @@+#!/usr/bin/env bash+# ─────────────────────────────────────────────────────────────────────────────+# neural-bulk-ab.sh — A/B harness for neural-embedding `_bulk` ingest.+#+# scripts/bench/neural-bulk-ab.sh [--dry-run] CONTROL_SHA CANDIDATE_SHA [N=5]+#+# Builds the server at each SHA in its own git worktree, then runs+# control, candidate, control, candidate, ... (N each, interleaved) against a+# cold throwaway data dir per run, driving the real client+# (`xerj autoindex`) at a frozen corpus. Writes, per run,+# benchmarks/ingest-neural-bulk/results/<date>/run-<arm>-<i>/+# server.log client.log summary.json+# and, per invocation, meta.json + report.md in the same <date> directory.+# Column definitions and the control-vs-control precondition are in+# benchmarks/ingest-neural-bulk/README.md.+#+# Pass the same SHA twice for the control-vs-control precondition run.+#+# Environment (all optional):+# SCRATCH work dir (default: mktemp under /tmp)+# RESULTS_DATE results subdirectory name (default: today, UTC)+# CONTROL_BIN / CANDIDATE_BIN prebuilt `xerj` binaries; skips that build+# HF_HOME model cache (default: $HOME/.cache/huggingface)+# RUN_TIMEOUT seconds allowed per client run (default 3600)+# PORT_MIN first port tried (default 9560; 9200 is never used)+# KEEP=1 keep each run's data dir (repro.sh knob)+# ─────────────────────────────────────────────────────────────────────────────+set -euo pipefail++HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"+# shellcheck source=lib/repro.sh+source "$HERE/lib/repro.sh"++DRY=0+POS=()+for a in "$@"; do+ case "$a" in+ --dry-run) DRY=1 ;;+ -h | --help) sed -n '2,29p' "${BASH_SOURCE[0]}"; exit 0 ;;+ *) POS+=("$a") ;;+ esac+done+if [ "${#POS[@]}" -lt 2 ] || [ "${#POS[@]}" -gt 3 ]; then+ echo "usage: $0 [--dry-run] CONTROL_SHA CANDIDATE_SHA [N=5]" >&2+ exit 2+fi+CONTROL_REF="${POS[0]}"+CANDIDATE_REF="${POS[1]}"+N="${POS[2]:-5}"+case "$N" in '' | *[!0-9]* | 0) echo "N must be a positive integer, got '$N'" >&2; exit 2 ;; esac++BENCH_DIR="$REPO_ROOT/benchmarks/ingest-neural-bulk"+CORPUS_TAR="$BENCH_DIR/corpus/okta-bulk.tar.zst"+DATE="${RESULTS_DATE:-$(date -u +%F)}"+RESULTS="$BENCH_DIR/results/$DATE"+SCRATCH="${SCRATCH:-$(mktemp -u /tmp/neural-bulk-ab.XXXXXX)}"+HF_HOME="${HF_HOME:-$HOME/.cache/huggingface}"+RUN_TIMEOUT="${RUN_TIMEOUT:-3600}"+PORT_MIN="${PORT_MIN:-9560}"+CLK_TCK="$(getconf CLK_TCK)"++# run CMD... executes, or prints the command in dry-run mode.+run() {+ if [ "$DRY" = 1 ]; then+ printf '+ '+ printf '%q ' "$@"+ printf '\n'+ else+ "$@"+ fi+}+say() { printf '%s\n' "$*"; }++CONTROL_SHA="$(git -C "$REPO_ROOT" rev-parse --verify "$CONTROL_REF^{commit}")"+CANDIDATE_SHA="$(git -C "$REPO_ROOT" rev-parse --verify "$CANDIDATE_REF^{commit}")"++# Isolation: user scope with a CPU quota and memory cap, else taskset.+ISOLATION=""+if systemd-run --user --scope -q -p CPUQuota=400% -- true >/dev/null 2>&1; then+ ISOLATION="systemd-scope"+elif command -v taskset >/dev/null 2>&1; then+ ISOLATION="taskset"+ echo "WARNING: systemd-run --user --scope unavailable; falling back to taskset -c 0-3" \+ "(no memory cap, no CPU quota)." >&2+else+ echo "ERROR: neither systemd-run --user --scope nor taskset is available." >&2+ exit 1+fi+if [ ! -d "$HF_HOME" ]; then+ echo "WARNING: HF_HOME=$HF_HOME does not exist; run 1 would include the model download." >&2+fi++say "control $CONTROL_SHA"+say "candidate $CANDIDATE_SHA"+say "runs $N per arm, interleaved; isolation=$ISOLATION; results=$RESULTS"+[ "$DRY" = 1 ] && say "(dry run: commands are printed, nothing is built, started or written)"++# ── cleanup: stop the server, drop arm worktrees ────────────────────────────+cleanup() {+ set +e+ repro_node_stop+ local w+ for w in "$SCRATCH"/wt-*; do+ [ -e "$w/.git" ] && git -C "$REPO_ROOT" worktree remove --force "$w" >/dev/null 2>&1+ done+ git -C "$REPO_ROOT" worktree prune+ return 0+}+trap cleanup EXIT+trap 'exit 130' INT+trap 'exit 143' TERM++# ── corpus ──────────────────────────────────────────────────────────────────+CORPUS_DIR="$SCRATCH/corpus"+run mkdir -p "$SCRATCH" "$CORPUS_DIR"+if [ "$DRY" = 1 ]; then+ say "+ (cd $BENCH_DIR/corpus && sha256sum -c SHA256SUMS)"+ say "+ zstd -dc $CORPUS_TAR | tar -x -C $CORPUS_DIR"+ say "+ verify extracted files against corpus/MANIFEST.tsv (sha256 column)"+else+ (cd "$BENCH_DIR/corpus" && sha256sum -c --quiet SHA256SUMS)+ zstd -dc "$CORPUS_TAR" | tar -x -C "$CORPUS_DIR"+ (cd "$CORPUS_DIR" && tail -n +2 "$BENCH_DIR/corpus/MANIFEST.tsv" \+ | awk -F'\t' '{print $6 " " $4}' | sha256sum -c --quiet)+fi+CORPUS="$CORPUS_DIR/okta-bulk"++# ── build each distinct SHA once ────────────────────────────────────────────+# build_arm SHA PREBUILT -> prints path of the binary to use+build_arm() {+ local sha="$1" prebuilt="$2" short wt out+ short="${sha:0:9}"+ out="$SCRATCH/bin/xerj-$short"+ if [ -n "$prebuilt" ]; then+ echo "$prebuilt"+ return+ fi+ if [ ! -x "$out" ] || [ "$DRY" = 1 ]; then+ wt="$SCRATCH/wt-$short"+ run git -C "$REPO_ROOT" worktree add --detach "$wt" "$sha" >&2+ run bash -c "cd '$wt/engine' && cargo build --release -j 2 -p xerj-server" >&2+ run mkdir -p "$SCRATCH/bin" >&2+ run cp "$wt/engine/target/release/xerj" "$out" >&2+ fi+ echo "$out"+}+# BIN_<arm> are read indirectly in run_one.+# shellcheck disable=SC2034+BIN_control="$(build_arm "$CONTROL_SHA" "${CONTROL_BIN:-}")"+if [ "$CANDIDATE_SHA" = "$CONTROL_SHA" ] && [ -z "${CANDIDATE_BIN:-}" ]; then+ # shellcheck disable=SC2034+ BIN_candidate="$BIN_control"+else+ # shellcheck disable=SC2034+ BIN_candidate="$(build_arm "$CANDIDATE_SHA" "${CANDIDATE_BIN:-}")"+fi++# ── server launcher (isolation wrapper; XERJ_BIN points at this) ───────────+LAUNCH="$SCRATCH/launch.sh"+if [ "$DRY" != 1 ]; then+ if [ "$ISOLATION" = systemd-scope ]; then+ cat >"$LAUNCH" <<'EOF'+#!/usr/bin/env bash+exec systemd-run --user --scope -q -p CPUQuota=400% -p MemoryMax=16G -- "$REAL_BIN" "$@"+EOF+ else+ cat >"$LAUNCH" <<'EOF'+#!/usr/bin/env bash+exec taskset -c 0-3 "$REAL_BIN" "$@"+EOF+ fi+ chmod +x "$LAUNCH"+fi++# proc_stat PID -> "utime_ticks stime_ticks" (comm may contain spaces)+proc_cpu_ticks() {+ local s+ s="$(cat "/proc/$1/stat")"+ s="${s##*) }"+ # shellcheck disable=SC2086+ set -- $s+ echo "${12} ${13}"+}++# resolve_server_pid: the pid whose exe is the server binary (the launcher+# execs, so this is normally REPRO_PID itself).+resolve_server_pid() {+ local real="$1" p c+ local -a kids+ p="$REPRO_PID"+ if [ "$(readlink -f "/proc/$p/exe" 2>/dev/null)" = "$(readlink -f "$real")" ]; then+ echo "$p"+ return+ fi+ for c in /proc/"$p"/task/*/children; do+ read -r -a kids <"$c" 2>/dev/null || continue+ for p in "${kids[@]}"; do+ if [ "$(readlink -f "/proc/$p/exe" 2>/dev/null)" = "$(readlink -f "$real")" ]; then+ echo "$p"+ return+ fi+ done+ done+ echo "$REPRO_PID"+}++# ── one run ─────────────────────────────────────────────────────────────────+run_one() {+ local arm="$1" i="$2" bin rundir state port_msg+ local binvar="BIN_$arm"+ bin="${!binvar}"+ rundir="$RESULTS/run-$arm-$i"+ state="$SCRATCH/state-$arm-$i"+ say ""+ say "== run $arm #$i (binary $bin) =="+ if [ "$DRY" = 1 ]; then+ say "+ mkdir -p $rundir $state"+ say "+ port=\$(repro_free_port $PORT_MIN) # never 9200"+ say "+ env REAL_BIN=$bin XERJ_NUM_CPUS=4 XERJ_PROF=1 XERJ_EMBED_MODE=neural HF_HOME=$HF_HOME \\"+ if [ "$ISOLATION" = systemd-scope ]; then+ say " systemd-run --user --scope -q -p CPUQuota=400% -p MemoryMax=16G -- $bin --insecure --port <port> --data-dir <fresh mktemp dir> &"+ else+ say " taskset -c 0-3 $bin --insecure --port <port> --data-dir <fresh mktemp dir> &"+ fi+ say "+ wait for GET / on <port>"+ say "+ timeout $RUN_TIMEOUT $bin autoindex $CORPUS --url http://127.0.0.1:<port> --prefix bench --bulk-mb 8 --workers 4 --progress json --state-dir $state --approve proceed > $rundir/client.log 2>&1"+ say "+ read VmHWM from /proc/<pid>/status and utime+stime from /proc/<pid>/stat"+ say "+ copy server log to $rundir/server.log; stop server; write $rundir/summary.json"+ return+ fi+ if [ -e "$rundir" ]; then+ echo "ERROR: $rundir already exists; set RESULTS_DATE to a new name." >&2+ return 1+ fi+ mkdir -p "$rundir" "$state"++ repro_free_port "$PORT_MIN"+ if [ "$REPRO_PORT" = 9200 ]; then echo "refusing port 9200" >&2; return 1; fi+ XERJ_BIN="$LAUNCH"+ export XERJ_BIN+ repro_node_start - "REAL_BIN=$bin" XERJ_NUM_CPUS=4 XERJ_PROF=1 \+ XERJ_EMBED_MODE=neural "HF_HOME=$HF_HOME"+ repro_node_wait++ local spid t0 t1 rc=0 hwm cpu_ticks+ spid="$(resolve_server_pid "$bin")"+ t0="$(date +%s.%N)"+ timeout "$RUN_TIMEOUT" "$bin" autoindex "$CORPUS" \+ --url "http://127.0.0.1:$REPRO_PORT" --prefix bench \+ --bulk-mb 8 --workers 4 --progress json \+ --state-dir "$state" --approve proceed >"$rundir/client.log" 2>&1 || rc=$?+ t1="$(date +%s.%N)"++ hwm="$(awk '/^VmHWM:/ {print $2}' "/proc/$spid/status" 2>/dev/null || true)"+ cpu_ticks="$(proc_cpu_ticks "$spid" 2>/dev/null || true)"+ cp "$REPRO_LOG" "$rundir/server.log"+ repro_node_stop+ port_msg="$REPRO_PORT"+ REPRO_PID=""++ python3 -I - "$rundir" "$rc" "$(awk -v a="$t0" -v b="$t1" 'BEGIN{printf "%.3f", b-a}')" \+ "${hwm:-}" "${cpu_ticks:-}" "$CLK_TCK" "$arm" "$i" "$port_msg" <<'PY'+import json, math, re, sys++(rundir, rc, wrapper_s, hwm, ticks, clk, arm, idx, port) = sys.argv[1:10]+++def pct(vals, p):+ if not vals:+ return None+ s = sorted(vals)+ return s[max(0, math.ceil(p * len(s)) - 1)]+++def read(path):+ try:+ with open(path, errors="replace") as f:+ return f.read().splitlines()+ except OSError:+ return []+++client = read(rundir + "/client.log")+server = read(rundir + "/server.log")++done = None+for line in client:+ line = line.strip()+ if line.startswith("{"):+ try:+ o = json.loads(line)+ except ValueError:+ continue+ if o.get("event") == "done":+ done = o+if done is None:+ for line in client:+ if line.startswith("xerj-done "):+ done = {}+ for tok in line.split()[1:]:+ if "=" in tok:+ k, v = tok.split("=", 1)+ done[k] = v+ if "wall" in done:+ done["wall_s"] = done["wall"].rstrip("s")+++def num(v):+ try:+ return float(v)+ except (TypeError, ValueError):+ return None+++bulk, embed = [], []+for line in server:+ if "XERJ_PROF" not in line:+ continue+ m = re.search(r"XERJ_PROF bulk .*?\btotal_ms=(\d+)", line)+ if m:+ bulk.append(float(m.group(1)))+ m = re.search(r"\bembed_ms=(\d+)", line)+ if m:+ embed.append(float(m.group(1)))++retry_item = sum("semantic embedding window failed; retrying per item" in l for l in server)+retry_embed = sum("embed retry" in l for l in server)+r429 = sum(bool(re.search(r"\b429\b", l)) for l in server + client)++wall = num(done.get("wall_s")) if done else None+records = num(done.get("records")) if done else None+cpu_s = None+if ticks.strip():+ a, b = ticks.split()+ cpu_s = round((int(a) + int(b)) / int(clk), 2)++summary = {+ "arm": arm,+ "run": int(idx),+ "port": int(port),+ "ok": bool(done and done.get("ok") in (True, "true") and rc == "0"),+ "client_exit": int(rc),+ "wall_s": wall,+ "wall_wrapper_s": num(wrapper_s),+ "records": int(records) if records is not None else None,+ "records_per_s": round(records / wall, 2) if records and wall else None,+ "bulk_total_ms_n": len(bulk),+ "bulk_total_ms_p50": pct(bulk, 0.50),+ "bulk_total_ms_p95": pct(bulk, 0.95),+ "embed_ms_n": len(embed),+ "embed_ms_p50": pct(embed, 0.50),+ "embed_ms_p95": pct(embed, 0.95),+ "retries_window_failed": retry_item,+ "retries_embed": retry_embed,+ "client_bulk_retries": int(num(done.get("bulk_retries")) or 0) if done else None,+ "http_429": r429,+ "peak_rss_kb": int(hwm) if hwm.strip() else None,+ "cpu_s": cpu_s,+}+summary["retries"] = retry_item + retry_embed + (summary["client_bulk_retries"] or 0)+with open(rundir + "/summary.json", "w") as f:+ json.dump(summary, f, indent=2)+ f.write("\n")+print(json.dumps(summary))+PY+}++# ── interleave ──────────────────────────────────────────────────────────────+run mkdir -p "$RESULTS"+if [ "$DRY" != 1 ]; then+ python3 -I - "$RESULTS/meta.json" "$CONTROL_SHA" "$CANDIDATE_SHA" "$N" "$ISOLATION" <<'PY'+import json, platform, os, sys, time+path, c, k, n, iso = sys.argv[1:6]+json.dump({+ "control_sha": c, "candidate_sha": k, "n": int(n), "isolation": iso,+ "started_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),+ "host": platform.platform(), "nproc": os.cpu_count(),+}, open(path, "w"), indent=2)+PY+fi+for i in $(seq 1 "$N"); do+ run_one control "$i"+ run_one candidate "$i"+done++# ── report ──────────────────────────────────────────────────────────────────+if [ "$DRY" = 1 ]; then+ say ""+ say "+ write $RESULTS/report.md (median and p95 per arm; control max-min spread; delta flagged only if > spread)"+ exit 0+fi+python3 -I - "$RESULTS" <<'PY'+import glob, json, math, statistics, sys++res = sys.argv[1]+meta = json.load(open(res + "/meta.json"))+runs = {"control": [], "candidate": []}+for p in sorted(glob.glob(res + "/run-*/summary.json")):+ s = json.load(open(p))+ runs[s["arm"]].append(s)++METRICS = [+ ("wall_s", "wall_s"),+ ("records_per_s", "records/s"),+ ("bulk_total_ms_p50", "bulk total_ms p50"),+ ("bulk_total_ms_p95", "bulk total_ms p95"),+ ("embed_ms_p50", "embed_ms p50"),+ ("embed_ms_p95", "embed_ms p95"),+ ("retries", "retries"),+ ("http_429", "http_429"),+ ("peak_rss_kb", "peak_rss_kb"),+ ("cpu_s", "cpu_s"),+]+++def vals(arm, key):+ return [r[key] for r in runs[arm] if r.get("ok") and r.get(key) is not None]+++def p95(v):+ s = sorted(v)+ return s[max(0, math.ceil(0.95 * len(s)) - 1)]+++def f(x):+ return "-" if x is None else ((f"{x:.2f}" if abs(x) < 10 else f"{x:.1f}") if isinstance(x, float) else str(x))+++out = []+out.append(f"# neural-bulk A/B report {meta['started_utc']}\n")+out.append(f"- control `{meta['control_sha']}`")+out.append(f"- candidate `{meta['candidate_sha']}`")+out.append(f"- runs per arm requested {meta['n']}; isolation {meta['isolation']}; host {meta['host']}; nproc {meta['nproc']}")+for arm in runs:+ good = sum(1 for r in runs[arm] if r.get("ok"))+ bad = [f"#{r['run']}(exit {r['client_exit']})" for r in runs[arm] if not r.get("ok")]+ out.append(f"- {arm}: {good} ok runs" + (f", EXCLUDED failed runs {', '.join(bad)}" if bad else ""))+out.append("")+out.append("Spread is the control arm's max-min over its ok runs. A delta is flagged only if |delta| exceeds it.")+out.append("p95 over N runs is the nearest-rank value (the max when N <= 20).\n")+out.append("| metric | control median | control p95 | control spread | candidate median | candidate p95 | delta (cand - ctrl median) | verdict |")+out.append("|---|---|---|---|---|---|---|---|")+for key, label in METRICS:+ c, k = vals("control", key), vals("candidate", key)+ if not c or not k:+ out.append(f"| {label} | {f(statistics.median(c)) if c else '-'} | - | - | {f(statistics.median(k)) if k else '-'} | - | - | no data |")+ continue+ cm, km = statistics.median(c), statistics.median(k)+ spread = max(c) - min(c)+ d = km - cm+ verdict = "EXCEEDS SPREAD" if abs(d) > spread else "within noise"+ pctd = f" ({100 * d / cm:+.1f}%)" if cm else ""+ out.append(f"| {label} | {f(cm)} | {f(p95(c))} | {f(spread)} | {f(km)} | {f(p95(k))} | {f(d)}{pctd} | {verdict} |")++if meta["control_sha"] == meta["candidate_sha"]:+ c, k = vals("control", "bulk_total_ms_p95"), vals("candidate", "bulk_total_ms_p95")+ out.append("")+ if c and k:+ cm, km = statistics.median(c), statistics.median(k)+ diff = abs(km - cm) / cm * 100 if cm else float("inf")+ out.append(f"Control-vs-control precondition: bulk total_ms p95 medians {f(cm)} vs {f(km)}, "+ f"difference {diff:.1f}% -> {'PASS (<10%)' if diff < 10 else 'FAIL (>=10%): no candidate number may be published from this host/setup'}")+ else:+ out.append("Control-vs-control precondition: no data (runs failed or XERJ_PROF lines missing).")+open(res + "/report.md", "w").write("\n".join(out) + "\n")+print("\n".join(out))+PY