best way to save what you love
Svelte collaborationdownloaderinstagramjavascriptmusicredditsocial-mediasoundcloudsveltetiktoktwittertypescriptvideovimeovkwebappyoutubeyoutube-downloader
docs: create a dedicated doc for api instance env variables
wukko545971186af0
diff --git a/docs/api-env-variables.md b/docs/api-env-variables.mdnew file mode 100644index 00000000..34de4b0a--- /dev/null+++ b/docs/api-env-variables.md@@ -0,0 +1,228 @@+# cobalt api instance environment variables+you can customize your processing instance's behavior using these environment variables. all of them but `API_URL` are optional.+this document is not final and will expand over time. feel free to improve it!++### general vars+| name | default | value example |+|:--------------------|:----------|:--------------------------------------|+| API_URL | | `https://api.url.example/` |+| API_PORT | `9000` | `1337` |+| COOKIE_PATH | | `/cookies.json` |+| PROCESSING_PRIORITY | | `10` |+| API_INSTANCE_COUNT | | `6` |+| API_REDIS_URL | | `redis://localhost:6379` |+| DISABLED_SERVICES | | `bilibili,youtube` |++[*view details*](#general)++### networking vars+| name | default | value example |+|:--------------------|:----------|:--------------------------------------|+| API_LISTEN_ADDRESS | `0.0.0.0` | `127.0.0.1` |+| API_EXTERNAL_PROXY | | `http://user:[email protected]:8080` |+| FREEBIND_CIDR | | `2001:db8::/32` |++[*view details*](#networking)++### limit vars+| name | default | value example |+|:-------------------------|:--------|:--------------|+| DURATION_LIMIT | `10800` | `18000` |+| TUNNEL_LIFESPAN | `90` | `120` |+| RATELIMIT_WINDOW | `60` | `120` |+| RATELIMIT_MAX | `20` | `30` |+| SESSION_RATELIMIT_WINDOW | `60` | `60` |+| SESSION_RATELIMIT | `10` | `10` |++[*view details*](#limits)++### security vars+| name | default | value example |+|:------------------|:--------|:--------------------------------------|+| CORS_WILDCARD | `1` | `0` |+| CORS_URL | | `https://web.url.example` |+| TURNSTILE_SITEKEY | | `1x00000000000000000000BB` |+| TURNSTILE_SECRET | | `1x0000000000000000000000000000000AA` |+| JWT_SECRET | | see [details](#security) |+| JWT_EXPIRY | `120` | `240` |+| API_KEY_URL | | `file://keys.json` |+| API_AUTH_REQUIRED | | `1` |++[*view details*](#security)++### service-specific vars+| name | value example |+|:---------------------------------|:-------------------------|+| CUSTOM_INNERTUBE_CLIENT | `IOS` |+| YOUTUBE_SESSION_SERVER | `http://localhost:8080/` |+| YOUTUBE_SESSION_INNERTUBE_CLIENT | `WEB_EMBEDDED` |++[*view details*](#service-specific)++## general+[*jump to the table*](#general-vars)++### API_URL+> [!NOTE]+> API_URL is required to run the API instance.++the URL from which your instance will be accessible. can be external or internal, but it must be a valid URL or else tunnels will not work.++the value is a URL.++### API_PORT+port from which the API server will be accessible.++the value is a number from 1024 to 65535.++### COOKIE_PATH+path to the `cookies.json` file relative to the current working directory of your cobalt instance (usually the main (src/api) folder).++### PROCESSING_PRIORITY+`nice` value for ffmpeg subprocesses. available only on unix systems.++note: the higher the nice value, the lower the priority. you can [read more about nice here](https://en.wikipedia.org/wiki/Nice_(Unix)).++the value is a number.++### API_INSTANCE_COUNT+supported only on linux and node.js `>=23.1.0`. when configured, cobalt will spawn multiple sub-instances amongst which requests will be balanced. `API_REDIS_URL` is required to use this option.++the value is a number.++### API_REDIS_URL+when configured, cobalt will use this redis instance for tunnel cache. required when `API_INSTANCE_COUNT` is more than 1, because else sub-instance wouldn't be able to share cache.++the value is a URL.++### DISABLED_SERVICES+comma-separated list which disables certain services from being used.++the value is a string of cobalt-supported services.++## networking+[*jump to the table*](#networking-vars)++### API_LISTEN_ADDRESS+defines the local address for the api instance. if you are using a docker container, you usually don't need to configure this.++the value is a local IP address.++### API_EXTERNAL_PROXY+URL of the proxy that will be passed to [`ProxyAgent`](https://undici.nodejs.org/#/docs/api/ProxyAgent) and used for all external requests. HTTP(S) only.++if some feature breaks when using a proxy, please make a new issue about it!++the value is a URL.++### FREEBIND_CIDR+IPv6 prefix used for randomly assigning addresses to cobalt requests. available only on linux systems.++setting a `FREEBIND_CIDR` allows cobalt to pick a random IP for every download and use it for all requests it makes for that particular download.++to use freebind in cobalt, you need to follow its [setup instructions](https://github.com/imputnet/freebind.js?tab=readme-ov-file#setup) first.++if you want to use this option and run cobalt in a docker container, you also need to set the `API_LISTEN_ADDRESS` env variable to `127.0.0.1` and set `network_mode` for the container to `host`.++the value is an IPv6 range.++## limits+[*jump to the table*](#limit-vars)++### DURATION_LIMIT+media duration limit, in **seconds**++the value is a number.++### TUNNEL_LIFESPAN+the duration for which tunnel info is stored in ram, **in seconds**.++it's recommended to keep this value either default or as low as possible to preserve efficiency and user privacy.++the value is a number.++### RATELIMIT_WINDOW+rate limit time window for api requests, but not session requests, in **seconds**.++the value is a number.++### RATELIMIT_MAX+amount of api requests to be allowed within the time window of `RATELIMIT_WINDOW`.++the value is a number.++### SESSION_RATELIMIT_WINDOW+rate limit time window for session creation requests, in **seconds**.++the value is a number.++### SESSION_RATELIMIT+amount of session requests to be allowed within the time window of `SESSION_RATELIMIT_WINDOW`.++the value is a number.++## security+[*jump to the table*](#security-vars)++> [!NOTE]+> in order to enable turnstile bot protection, `TURNSTILE_SITEKEY`, `TURNSTILE_SECRET`, and `JWT_SECRET` must be set. all three at once.++### CORS_WILDCARD+defines whether cross-origin resource sharing is enabled. when enabled, your instance will be accessible from foreign web pages.++the value is a number. 0: disabled. 1: enabled.++### CORS_URL+configures the [cross-origin resource sharing origin](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Access-Control-Allow-Origin). your instance will be available only from this URL if `CORS_WILDCARD` is set to `0`.++the value is a URL.++### TURNSTILE_SITEKEY+[cloudflare turnstile](https://www.cloudflare.com/products/turnstile/) sitekey used by the web client to request & solve a challenge to prove that the user is not a bot.++the value is a specific key.++### TURNSTILE_SECRET+[cloudflare turnstile](https://www.cloudflare.com/products/turnstile/) secret used by the processing instance to verify that the client solved the challenge successfully.++the value is a specific key.++### JWT_SECRET+the secret used for issuing JWT tokens for request authentication. the value must be a random, secure, and long string (over 16 characters).++the value is a specific key.++### JWT_EXPIRY+the duration of how long a cobalt-issued JWT token will remain valid, in seconds.++the value is a number.++### API_KEY_URL+the URL to the the external or local key database. for local files you have to specify a local path using the `file://` protocol.++see [the api key section](/docs/protect-an-instance.md#api-key-file-format) in the "how to protect your cobalt instance" document for more details.++the value is a URL.++### API_AUTH_REQUIRED+when set to `1`, the user always needs to be authenticated in some way before they can access the API (either via an api key or via turnstile, if enabled).++the value is a number.++## service-specific+[*jump to the table*](#service-specific-vars)++### CUSTOM_INNERTUBE_CLIENT+innertube client that will be used instead of the default one.++the value is a string.++### YOUTUBE_SESSION_SERVER+URL to an instance of [yt-session-generator](https://github.com/imputnet/yt-session-generator). used for automatically pulling `poToken` & `visitor_data` for youtube. can be local or remote.++the value is a URL.++### YOUTUBE_SESSION_INNERTUBE_CLIENT+innertube client that's compatible with botguard's (web) `poToken` and `visitor_data`.++the value is a string.diff --git a/docs/protect-an-instance.md b/docs/protect-an-instance.mdindex 9b4131c1..30584102 100644--- a/docs/protect-an-instance.md+++ b/docs/protect-an-instance.md@@ -114,7 +114,7 @@ if you want to use your instance outside of web interface, you'll need an api ke>> if api keys leak, you'll have to update/remove all UUIDs to revoke them.-1. create a `keys.json` file following [the schema and example here](/docs//run-an-instance.md#api-key-file-format).+1. create a `keys.json` file following [the schema and example down below](#api-key-file-format).2. expose the `keys.json` to the docker container:```yml@@ -148,3 +148,55 @@ environment:### why not make keys exclusive by default?keys may be useful for going around rate limiting,while keeping the rest of api rate limited, with no turnstile in place.++## api key file format+the file is a JSON-serialized object with the following structure:+```typescript++type KeyFileContents = Record<+ UUIDv4String,+ {+ name?: string,+ limit?: number | "unlimited",+ ips?: (CIDRString | IPString)[],+ userAgents?: string[]+ }+>;+```++where *`UUIDv4String`* is a stringified version of a UUIDv4 identifier.+- **name** is a field for your own reference, it is not used by cobalt anywhere.++- **`limit`** specifies how many requests the API key can make during the window specified in the `RATELIMIT_WINDOW` env.+ - when omitted, the limit specified in `RATELIMIT_MAX` will be used.+ - it can be also set to `"unlimited"`, in which case the API key bypasses all rate limits.++- **`ips`** contains an array of allowlisted IP ranges, which can be specified both as individual ips or CIDR ranges (e.g. *`["192.168.42.69", "2001:db8::48", "10.0.0.0/8", "fe80::/10"]`*).+ - when specified, only requests from these ip ranges can use the specified api key.+ - when omitted, any IP can be used to make requests with that API key.++- **`userAgents`** contains an array of allowed user agents, with support for wildcards (e.g. *`["cobaltbot/1.0", "Mozilla/5.0 * Chrome/*"]`*).+ - when specified, requests with a `user-agent` that does not appear in this array will be rejected.+ - when omitted, any user agent can be specified to make requests with that API key.++- if both `ips` and `userAgents` are set, the tokens will be limited by both parameters.+- if cobalt detects any problem with your key file, it will be ignored and a warning will be printed to the console.++an example key file could look like this:+```json+{+ "b5c7160a-b655-4c7a-b500-de839f094550": {+ "limit": 10,+ "ips": ["10.0.0.0/8", "192.168.42.42"],+ "userAgents": ["*Chrome*"]+ },+ "b00b1234-a3e5-99b1-c6d1-dba4512ae190": {+ "limit": "unlimited",+ "ips": ["192.168.1.2"],+ "userAgents": ["cobaltbot/1.0"]+ }+}+```++if you are configuring a key file, **do not use the UUID from the example** but instead generate your own. you can do this by running the following command if you have node.js installed:+`node -e "console.log(crypto.randomUUID())"`diff --git a/docs/run-an-instance.md b/docs/run-an-instance.mdindex b79e3fe8..daedac09 100644--- a/docs/run-an-instance.md+++ b/docs/run-an-instance.md@@ -54,93 +54,5 @@ sudo apt install nscdsudo service nscd start```-## list of environment variables for api-| variable name | default | example | description |-|:----------------------|:----------|:------------------------|:------------|-| `API_PORT` | `9000` | `9000` | changes port from which api server is accessible. |-| `API_LISTEN_ADDRESS` | `0.0.0.0` | `127.0.0.1` | changes address from which api server is accessible. **if you are using docker, you usually don't need to configure this.** |-| `API_URL` | ➖ | `https://api.cobalt.tools/` | changes url from which api server is accessible. <br> ***REQUIRED TO RUN THE API***. |-| `API_NAME` | `unknown` | `ams-1` | api server name that is shown in `/api/serverInfo`. |-| `API_EXTERNAL_PROXY` | ➖ | `http://user:[email protected]:8080`| url of the proxy that will be passed to [`ProxyAgent`](https://undici.nodejs.org/#/docs/api/ProxyAgent) and used for all external requests. HTTP(S) only. |-| `CORS_WILDCARD` | `1` | `0` | toggles cross-origin resource sharing. <br> `0`: disabled. `1`: enabled. |-| `CORS_URL` | not used | `https://cobalt.tools` | cross-origin resource sharing url. api will be available only from this url if `CORS_WILDCARD` is set to `0`. |-| `COOKIE_PATH` | not used | `/cookies.json` | path for cookie file relative to main folder. |-| `PROCESSING_PRIORITY` | not used | `10` | changes `nice` value* for ffmpeg subprocess. available only on unix systems. |-| `FREEBIND_CIDR` | ➖ | `2001:db8::/32` | IPv6 prefix used for randomly assigning addresses to cobalt requests. only supported on linux systems. see below for more info. |-| `RATELIMIT_WINDOW` | `60` | `120` | rate limit time window in **seconds**. |-| `RATELIMIT_MAX` | `20` | `30` | max requests per time window. requests above this amount will be blocked for the rate limit window duration. |-| `DURATION_LIMIT` | `10800` | `18000` | max allowed video duration in **seconds**. |-| `TUNNEL_LIFESPAN` | `90` | `120` | the duration for which tunnel info is stored in ram, **in seconds**. |-| `TURNSTILE_SITEKEY` | ➖ | `1x00000000000000000000BB` | [cloudflare turnstile](https://www.cloudflare.com/products/turnstile/) sitekey used by browser clients to request a challenge.\*\* |-| `TURNSTILE_SECRET` | ➖ | `1x0000000000000000000000000000000AA` | [cloudflare turnstile](https://www.cloudflare.com/products/turnstile/) secret used by cobalt to verify the client successfully solved the challenge.\*\* |-| `JWT_SECRET` | ➖ | ➖ | the secret used for issuing JWT tokens for request authentication. to choose a value, generate a random, secure, long string (ideally >=16 characters).\*\* |-| `JWT_EXPIRY` | `120` | `240` | the duration of how long a cobalt-issued JWT token will remain valid, in seconds. |-| `API_KEY_URL` | ➖ | `file://keys.json` | the location of the api key database. for loading API keys, cobalt supports HTTP(S) urls, or local files by specifying a local path using the `file://` protocol. see the "api key file format" below for more details. |-| `API_AUTH_REQUIRED` | ➖ | `1` | when set to `1`, the user always needs to be authenticated in some way before they can access the API (either via an api key or via turnstile, if enabled). |-| `API_REDIS_URL` | ➖ | `redis://localhost:6379` | when set, cobalt uses redis instead of internal memory for the tunnel cache. |-| `API_INSTANCE_COUNT` | ➖ | `2` | supported only on Linux and node.js `>=23.1.0`. when configured, cobalt will spawn multiple sub-instances amongst which requests will be balanced. |-| `DISABLED_SERVICES` | ➖ | `bilibili,youtube` | comma-separated list which disables certain services from being used. |-| `CUSTOM_INNERTUBE_CLIENT` | ➖ | `IOS` | innertube client that will be used instead of the default one. |-| `YOUTUBE_SESSION_SERVER` | ➖ | `http://localhost:8080/` | url to an instance of [invidious' youtube-trusted-session-generator](https://github.com/iv-org/youtube-trusted-session-generator) or its fork/counterpart. used for automatically pulling poToken & visitor_data for youtube. can be local or remote. |--\* the higher the nice value, the lower the priority. [read more here](https://en.wikipedia.org/wiki/Nice_(Unix)).--\*\* in order to enable turnstile bot protection, all three **`TURNSTILE_SITEKEY`, `TURNSTILE_SECRET` and `JWT_SECRET`** need to be set.--#### FREEBIND_CIDR-setting a `FREEBIND_CIDR` allows cobalt to pick a random IP for every download and use it for all-requests it makes for that particular download. to use freebind in cobalt, you need to follow its [setup instructions](https://github.com/imputnet/freebind.js?tab=readme-ov-file#setup) first. if you configure this option while running cobalt-in a docker container, you also need to set the `API_LISTEN_ADDRESS` env to `127.0.0.1`, and set-`network_mode` for the container to `host`.--## api key file format-the file is a JSON-serialized object with the following structure:-```typescript--type KeyFileContents = Record<- UUIDv4String,- {- name?: string,- limit?: number | "unlimited",- ips?: (CIDRString | IPString)[],- userAgents?: string[]- }->;-```--where *`UUIDv4String`* is a stringified version of a UUIDv4 identifier.-- **name** is a field for your own reference, it is not used by cobalt anywhere.--- **`limit`** specifies how many requests the API key can make during the window specified in the `RATELIMIT_WINDOW` env.- - when omitted, the limit specified in `RATELIMIT_MAX` will be used.- - it can be also set to `"unlimited"`, in which case the API key bypasses all rate limits.--- **`ips`** contains an array of allowlisted IP ranges, which can be specified both as individual ips or CIDR ranges (e.g. *`["192.168.42.69", "2001:db8::48", "10.0.0.0/8", "fe80::/10"]`*).- - when specified, only requests from these ip ranges can use the specified api key.- - when omitted, any IP can be used to make requests with that API key.--- **`userAgents`** contains an array of allowed user agents, with support for wildcards (e.g. *`["cobaltbot/1.0", "Mozilla/5.0 * Chrome/*"]`*).- - when specified, requests with a `user-agent` that does not appear in this array will be rejected.- - when omitted, any user agent can be specified to make requests with that API key.--- if both `ips` and `userAgents` are set, the tokens will be limited by both parameters.-- if cobalt detects any problem with your key file, it will be ignored and a warning will be printed to the console.--an example key file could look like this:-```json-{- "b5c7160a-b655-4c7a-b500-de839f094550": {- "limit": 10,- "ips": ["10.0.0.0/8", "192.168.42.42"],- "userAgents": ["*Chrome*"]- },- "b00b1234-a3e5-99b1-c6d1-dba4512ae190": {- "limit": "unlimited",- "ips": ["192.168.1.2"],- "userAgents": ["cobaltbot/1.0"]- }-}-```--if you are configuring a key file, **do not use the UUID from the example** but instead generate your own. you can do this by running the following command if you have node.js installed:-`node -e "console.log(crypto.randomUUID())"`+## list of environment variables+[this section has moved](/docs/api-env-variables.md) to a dedicated document that is way easier to understand and maintain. go check it out!