Skip to content

The Postgres development platform. Supabase gives you a dedicated Postgres database to build your web, mobile, and AI applications.

docs: Combine log querying and filtering sources (#39528)

Chris Chinchillaa7489eb69100
diff --git a/apps/docs/components/Navigation/NavigationMenu/NavigationMenu.constants.ts b/apps/docs/components/Navigation/NavigationMenu/NavigationMenu.constants.ts
index 4ac385cff16..77a5a6ef81d 100644
--- a/apps/docs/components/Navigation/NavigationMenu/NavigationMenu.constants.ts
+++ b/apps/docs/components/Navigation/NavigationMenu/NavigationMenu.constants.ts
@@ -2855,6 +2855,10 @@ export const telemetry: NavMenuConstant = {
name: 'Advanced log filtering',
url: '/guides/telemetry/advanced-log-filtering' as `/${string}`,
},
+ {
+ name: 'Logs field reference',
+ url: '/guides/telemetry/log-field-reference' as `/${string}`,
+ },
{
name: 'Log drains',
url: '/guides/telemetry/log-drains' as `/${string}`,
diff --git a/apps/docs/content/guides/telemetry/advanced-log-filtering.mdx b/apps/docs/content/guides/telemetry/advanced-log-filtering.mdx
index 84890cc2d4b..36dbb69b60c 100644
--- a/apps/docs/content/guides/telemetry/advanced-log-filtering.mdx
+++ b/apps/docs/content/guides/telemetry/advanced-log-filtering.mdx
@@ -1,19 +1,135 @@
---
-title: 'Advanced Log Filtering'
-description: 'Filter logs with regular expressions'
+title: 'Advanced Log Querying and Filtering'
+description: 'Query and filter logs with regular expressions'
---
-# Querying the logs
+The [Logs Explorer](/dashboard/project/_/logs-explorer) exposes logs from each part of the Supabase stack as a separate table that can be queried and joined using SQL.
+
+![Logs Explorer](/docs/img/guides/platform/logs/logs-explorer.png)
+
+You can access the following logs from the **Sources** drop-down:
+
+- `auth_logs`: GoTrue server logs, containing authentication/authorization activity.
+- `edge_logs`: Edge network logs, containing request and response metadata retrieved from Cloudflare.
+- `function_edge_logs`: Edge network logs for only edge functions, containing network requests and response metadata for each execution.
+- `function_logs`: Function internal logs, containing any `console` logging from within the edge function.
+- `postgres_logs`: Postgres database logs, containing statements executed by connected applications.
+- `realtime_logs`: Realtime server logs, containing client connection information.
+- `storage_logs`: Storage server logs, containing object upload and retrieval information.
+
+The Logs Explorer uses BigQuery and supports all [available SQL functions and operators](https://cloud.google.com/bigquery/docs/reference/standard-sql/functions-and-operators).
+
+## Timestamp display and behavior
+
+BigQuery stores each log entry with a `timestamp` as a `TIMESTAMP` data type. Use the appropriate [timestamp function](https://cloud.google.com/bigquery/docs/reference/standard-sql/timestamp_functions#timestamp) to utilize the `timestamp` field in a query.
+
+BigQuery renders raw top-level timestamp values as Unix microseconds. To render the timestamps in a human-readable format, use the `DATETIME()` function to convert the Unix timestamp display into an ISO-8601 timestamp.
+
+```sql
+-- timestamp column without datetime()
+select timestamp from ....
+-- 1664270180000
+
+-- timestamp column with datetime()
+select datetime(timestamp) from ....
+-- 2022-09-27T09:17:10.439Z
+```
+
+## Unnesting arrays
+
+Each log event stores metadata as an array of objects with multiple levels, as you can see by selecting single log events in the Logs Explorer. To query arrays, use `unnest()` on each array field and add it to the query as a join. This allows you to reference the nested objects with an alias and select their individual fields.
+
+For example, to query the edge logs without any joins:
+
+```sql
+select timestamp, metadata from edge_logs as t;
+```
+
+The Logs Explorer renders the resulting `metadata` key as an array of objects in the Logs Explorer. In the following diagram, each box represents a nested array of objects:
+
+![Without Unnesting](/docs/img/unnesting-none.png)
+
+Perform a `cross join unnest()` to work with the keys nested in the `metadata` key.
+
+To query for a nested value, add a join for each array level:
+
+```sql
+select timestamp, request.method, header.cf_ipcountry
+from
+ edge_logs as t
+ cross join unnest(t.metadata) as metadata
+ cross join unnest(metadata.request) as request
+ cross join unnest(request.headers) as header;
+```
+
+This surfaces the following columns available for selection:
+![With Two Level Unnesting](/docs/img/unnesting-2.png)
+
+This allows you to select the `method` and `cf_ipcountry` columns. In JS dot notation, the full paths for each selected column are:
+
+- `metadata[].request[].method`
+- `metadata[].request[].headers[].cf_ipcountry`
+
+## LIMIT and result row limitations
+
+The Logs Explorer has a maximum of 1000 rows per run. Use `LIMIT` to optimize your queries by reducing the number of rows returned further.
+
+## Best practices
+
+1. Include a filter over **timestamp**
+
+Querying your entire log history might seem appealing. For **Enterprise** customers that have a large retention range, you run the risk of timeouts due to additional time required to scan the larger dataset.
+
+2. Avoid selecting large nested objects. Select individual values instead.
+
+When querying large objects, the columnar storage engine selects each column associated with each nested key, resulting in a large number of columns being selected. This inadvertently impacts the query speed and may result in timeouts or memory errors, especially for projects with a lot of logs.
+
+Instead, select only the values required.
+
+```sql
+-- ❌ Avoid doing this
+select
+ datetime(timestamp),
+ m as metadata -- <- metadata contains many nested keys
+from
+ edge_logs as t
+ cross join unnest(t.metadata) as m;
+
+-- ✅ Do this
+select
+ datetime(timestamp),
+ r.method -- <- select only the required values
+from
+ edge_logs as t
+ cross join unnest(t.metadata) as m
+ cross join unnest(m.request) as r;
+```
+
+## Examples and templates
+
+The Logs Explorer includes **Templates** (available in the Templates tab or the dropdown in the Query tab) to help you get started.
+
+For example, you can enter the following query in the SQL Editor to retrieve each user's IP address:
+
+```sql
+select datetime(timestamp), h.x_real_ip
+from
+ edge_logs
+ cross join unnest(metadata) as m
+ cross join unnest(m.request) as r
+ cross join unnest(r.headers) as h
+where h.x_real_ip is not null and r.method = "GET";
+```
## Understanding field references
-The log tables are queried with a subset of BigQuery SQL syntax. They all have three columns: `event_message`, `timestamp`, and `metadata`.
+You query log tables with a subset of BigQuery SQL syntax. They all have three columns: `event_message`, `timestamp`, and `metadata`.
-| column | description |
-| ------------- | --------------------------- |
-| timestamp | time event was recorded |
-| event_message | the log's message |
-| metadata | information about the event |
+| column | description |
+| --------------- | --------------------------- |
+| `timestamp` | time event was recorded |
+| `event_message` | the log's message |
+| `metadata` | information about the event |
The `metadata` column is an array of JSON objects that stores important details about each recorded event. For example, in the Postgres table, the `metadata.parsed.error_severity` field indicates the error level of an event. To work with its values, you need to `unnest` them using a `cross join`.
@@ -40,7 +156,7 @@ Logs returned by queries may be difficult to read in table format. A row can be
## Filtering with [regular expressions](https://en.wikipedia.org/wiki/Regular_expression)
-The Logs use BigQuery Style regular expressions with the [regexp_contains function](https://cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#regexp_contains). In its most basic form, it will check if a string is present in a specified column.
+The Logs use BigQuery-style regular expressions with the [regexp_contains function](https://cloud.google.com/bigquery/docs/reference/standard-sql/string_functions#regexp_contains). In its most basic form, it checks if a string is present in a specified column.
```sql
select
@@ -146,7 +262,7 @@ where
### Filtering and unnesting example
-**Filter for Postgres**
+Filter for Postgres:
```sql
select
@@ -175,7 +291,7 @@ The parser does not yet support `with` and subquery statements.
### The `ilike` and `similar to` keywords are not supported
-Although `like` and other comparison operators can be used, `ilike` and `similar to` are incompatible with BigQuery's variant of SQL. `regexp_contains` can be used as an alternative.
+Although you can use `like` and other comparison operators, `ilike` and `similar to` are incompatible with BigQuery's variant of SQL. Use `regexp_contains` as an alternative.
### The wildcard operator `*` to select columns is not supported
diff --git a/apps/docs/content/guides/telemetry/log-field-reference.mdx b/apps/docs/content/guides/telemetry/log-field-reference.mdx
new file mode 100644
index 00000000000..c7eb0fa1bfc
--- /dev/null
+++ b/apps/docs/content/guides/telemetry/log-field-reference.mdx
@@ -0,0 +1,36 @@
+---
+id: 'logs-field-reference'
+title: 'Logs field reference'
+description: 'Supabase Logs field reference'
+---
+
+Refer to the full field reference for each available source below. To access each nested key, you need to perform the [necessary unnesting joins](/docs/guides/telemetry/advanced-log-filtering#unnesting-arrays)
+
+<SharedData data="logConstants">
+ {(logConstants) => (
+ <Tabs scrollable size="small" type="underlined" defaultActiveId="edge_logs" queryGroup="source">
+ {logConstants.schemas.map((schema) => (
+ <TabPanel id={schema.reference} key={schema.reference} label={schema.name}>
+ <table>
+ <thead>
+ <tr>
+ <th className="font-bold">Path</th>
+ <th className="font-bold">Type</th>
+ </tr>
+ </thead>
+ <tbody>
+ {schema.fields
+ .sort((a, b) => a.path - b.path)
+ .map((field) => (
+ <tr key={field.path}>
+ <td className="font-mono">{field.path}</td>
+ <td className="font-mono">{field.type}</td>
+ </tr>
+ ))}
+ </tbody>
+ </table>
+ </TabPanel>
+ ))}
+ </Tabs>
+ )}
+</SharedData>