The Postgres development platform. Supabase gives you a dedicated Postgres database to build your web, mobile, and AI applications.
TypeScript aialternativeauthdatabasedenoembeddingsexamplefirebasenextjsoauth2pgvectorpostgispostgrespostgresqlpostgrestrealtimesupabasevectorswebsockets
docs: Combine log querying and filtering sources (#39528)
Chris Chinchillaa7489eb69100
diff --git a/apps/docs/components/Navigation/NavigationMenu/NavigationMenu.constants.ts b/apps/docs/components/Navigation/NavigationMenu/NavigationMenu.constants.tsindex 4ac385cff16..77a5a6ef81d 100644--- a/apps/docs/components/Navigation/NavigationMenu/NavigationMenu.constants.ts+++ b/apps/docs/components/Navigation/NavigationMenu/NavigationMenu.constants.ts@@ -2855,6 +2855,10 @@ export const telemetry: NavMenuConstant = {name: 'Advanced log filtering',url: '/guides/telemetry/advanced-log-filtering' as `/${string}`,},+ {+ name: 'Logs field reference',+ url: '/guides/telemetry/log-field-reference' as `/${string}`,+ },{name: 'Log drains',url: '/guides/telemetry/log-drains' as `/${string}`,diff --git a/apps/docs/content/guides/telemetry/advanced-log-filtering.mdx b/apps/docs/content/guides/telemetry/advanced-log-filtering.mdxindex 84890cc2d4b..36dbb69b60c 100644--- a/apps/docs/content/guides/telemetry/advanced-log-filtering.mdx+++ b/apps/docs/content/guides/telemetry/advanced-log-filtering.mdx@@ -1,19 +1,135 @@----title: 'Advanced Log Filtering'-description: 'Filter logs with regular expressions'+title: 'Advanced Log Querying and Filtering'+description: 'Query and filter logs with regular expressions'----# Querying the logs+The [Logs Explorer](/dashboard/project/_/logs-explorer) exposes logs from each part of the Supabase stack as a separate table that can be queried and joined using SQL.++++You can access the following logs from the **Sources** drop-down:++- `auth_logs`: GoTrue server logs, containing authentication/authorization activity.+- `edge_logs`: Edge network logs, containing request and response metadata retrieved from Cloudflare.+- `function_edge_logs`: Edge network logs for only edge functions, containing network requests and response metadata for each execution.+- `function_logs`: Function internal logs, containing any `console` logging from within the edge function.+- `postgres_logs`: Postgres database logs, containing statements executed by connected applications.+- `realtime_logs`: Realtime server logs, containing client connection information.+- `storage_logs`: Storage server logs, containing object upload and retrieval information.++The Logs Explorer uses BigQuery and supports all [available SQL functions and operators](https://cloud.google.com/bigquery/docs/reference/standard-sql/functions-and-operators).++## Timestamp display and behavior++BigQuery stores each log entry with a `timestamp` as a `TIMESTAMP` data type. Use the appropriate [timestamp function](https://cloud.google.com/bigquery/docs/reference/standard-sql/timestamp_functions#timestamp) to utilize the `timestamp` field in a query.++BigQuery renders raw top-level timestamp values as Unix microseconds. To render the timestamps in a human-readable format, use the `DATETIME()` function to convert the Unix timestamp display into an ISO-8601 timestamp.++```sql+-- timestamp column without datetime()+select timestamp from ....+-- 1664270180000++-- timestamp column with datetime()+select datetime(timestamp) from ....+-- 2022-09-27T09:17:10.439Z+```++## Unnesting arrays++Each log event stores metadata as an array of objects with multiple levels, as you can see by selecting single log events in the Logs Explorer. To query arrays, use `unnest()` on each array field and add it to the query as a join. This allows you to reference the nested objects with an alias and select their individual fields.++For example, to query the edge logs without any joins:++```sql+select timestamp, metadata from edge_logs as t;+```++The Logs Explorer renders the resulting `metadata` key as an array of objects in the Logs Explorer. In the following diagram, each box represents a nested array of objects:++++Perform a `cross join unnest()` to work with the keys nested in the `metadata` key.++To query for a nested value, add a join for each array level:++```sql+select timestamp, request.method, header.cf_ipcountry+from+ edge_logs as t+ cross join unnest(t.metadata) as metadata+ cross join unnest(metadata.request) as request+ cross join unnest(request.headers) as header;+```++This surfaces the following columns available for selection:+++This allows you to select the `method` and `cf_ipcountry` columns. In JS dot notation, the full paths for each selected column are:++- `metadata[].request[].method`+- `metadata[].request[].headers[].cf_ipcountry`++## LIMIT and result row limitations++The Logs Explorer has a maximum of 1000 rows per run. Use `LIMIT` to optimize your queries by reducing the number of rows returned further.++## Best practices++1. Include a filter over **timestamp**++Querying your entire log history might seem appealing. For **Enterprise** customers that have a large retention range, you run the risk of timeouts due to additional time required to scan the larger dataset.++2. Avoid selecting large nested objects. Select individual values instead.++When querying large objects, the columnar storage engine selects each column associated with each nested key, resulting in a large number of columns being selected. This inadvertently impacts the query speed and may result in timeouts or memory errors, especially for projects with a lot of logs.++Instead, select only the values required.++```sql+-- ❌ Avoid doing this+select+ datetime(timestamp),+ m as metadata -- <- metadata contains many nested keys+from+ edge_logs as t+ cross join unnest(t.metadata) as m;++-- ✅ Do this+select+ datetime(timestamp),+ r.method -- <- select only the required values+from+ edge_logs as t+ cross join unnest(t.metadata) as m+ cross join unnest(m.request) as r;+```++## Examples and templates++The Logs Explorer includes **Templates** (available in the Templates tab or the dropdown in the Query tab) to help you get started.++For example, you can enter the following query in the SQL Editor to retrieve each user's IP address:++```sql+select datetime(timestamp), h.x_real_ip+from+ edge_logs+ cross join unnest(metadata) as m+ cross join unnest(m.request) as r+ cross join unnest(r.headers) as h+where h.x_real_ip is not null and r.method = "GET";+```## Understanding field references-The log tables are queried with a subset of BigQuery SQL syntax. They all have three columns: `event_message`, `timestamp`, and `metadata`.+You query log tables with a subset of BigQuery SQL syntax. They all have three columns: `event_message`, `timestamp`, and `metadata`.-| column | description |-| ------------- | --------------------------- |-| timestamp | time event was recorded |-| event_message | the log's message |-| metadata | information about the event |+| column | description |+| --------------- | --------------------------- |+| `timestamp` | time event was recorded |+| `event_message` | the log's message |+| `metadata` | information about the event |The `metadata` column is an array of JSON objects that stores important details about each recorded event. For example, in the Postgres table, the `metadata.parsed.error_severity` field indicates the error level of an event. To work with its values, you need to `unnest` them using a `cross join`.@@ -40,7 +156,7 @@ Logs returned by queries may be difficult to read in table format. A row can be## Filtering with [regular expressions](https://en.wikipedia.org/wiki/Regular_expression)-The Logs use BigQuery Style regular expressions with the [regexp_contains function](https://cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#regexp_contains). In its most basic form, it will check if a string is present in a specified column.+The Logs use BigQuery-style regular expressions with the [regexp_contains function](https://cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#regexp_contains). In its most basic form, it checks if a string is present in a specified column.```sqlselect@@ -146,7 +262,7 @@ where### Filtering and unnesting example-**Filter for Postgres**+Filter for Postgres:```sqlselect@@ -175,7 +291,7 @@ The parser does not yet support `with` and subquery statements.### The `ilike` and `similar to` keywords are not supported-Although `like` and other comparison operators can be used, `ilike` and `similar to` are incompatible with BigQuery's variant of SQL. `regexp_contains` can be used as an alternative.+Although you can use `like` and other comparison operators, `ilike` and `similar to` are incompatible with BigQuery's variant of SQL. Use `regexp_contains` as an alternative.### The wildcard operator `*` to select columns is not supporteddiff --git a/apps/docs/content/guides/telemetry/log-field-reference.mdx b/apps/docs/content/guides/telemetry/log-field-reference.mdxnew file mode 100644index 00000000000..c7eb0fa1bfc--- /dev/null+++ b/apps/docs/content/guides/telemetry/log-field-reference.mdx@@ -0,0 +1,36 @@+---+id: 'logs-field-reference'+title: 'Logs field reference'+description: 'Supabase Logs field reference'+---++Refer to the full field reference for each available source below. To access each nested key, you need to perform the [necessary unnesting joins](/docs/guides/telemetry/advanced-log-filtering#unnesting-arrays)++<SharedData data="logConstants">+ {(logConstants) => (+ <Tabs scrollable size="small" type="underlined" defaultActiveId="edge_logs" queryGroup="source">+ {logConstants.schemas.map((schema) => (+ <TabPanel id={schema.reference} key={schema.reference} label={schema.name}>+ <table>+ <thead>+ <tr>+ <th className="font-bold">Path</th>+ <th className="font-bold">Type</th>+ </tr>+ </thead>+ <tbody>+ {schema.fields+ .sort((a, b) => a.path - b.path)+ .map((field) => (+ <tr key={field.path}>+ <td className="font-mono">{field.path}</td>+ <td className="font-mono">{field.type}</td>+ </tr>+ ))}+ </tbody>+ </table>+ </TabPanel>+ ))}+ </Tabs>+ )}+</SharedData>