Skip to content

The Postgres development platform. Supabase gives you a dedicated Postgres database to build your web, mobile, and AI applications.

Add readme for SQL Editor overview (#42167)

Joshen Lim85aa76ba0a8f
diff --git a/apps/studio/components/interfaces/SQLEditor/README.md b/apps/studio/components/interfaces/SQLEditor/README.md
new file mode 100644
index 00000000000..1370eb5885f
--- /dev/null
+++ b/apps/studio/components/interfaces/SQLEditor/README.md
@@ -0,0 +1,83 @@
+# SQL Editor Overview
+
+Quick run-through of the different building blocks that make up the SQL Editor, to help navigating the codebase a little easier 🙂🙏
+
+## UI structure
+
+- Folders and snippets (in the product menu) are rendered via `SQLEditorMenu` and `SQLEditorNav`, in which the data are all loaded from the API via React Query directly. (Refer to point 3 under "Data Structure" below)
+
+ - [`SQLEditorMenu`](https://github.com/supabase/supabase/blob/master/apps/studio/components/layouts/SQLEditorLayout/SQLEditorMenu.tsx): Wraps around `SQLEditorNav` + renders search input + View running queries button
+ - [`SQLEditorNav`](https://github.com/supabase/supabase/blob/master/apps/studio/components/layouts/SQLEditorLayout/SQLEditorNavV2/SQLEditorNav.tsx): Renders the 3 collapsible snippet sections
+
+- The Tabs interface is powered by a separate state [`tabs.ts`](https://github.com/supabase/supabase/blob/master/apps/studio/state/tabs.ts). (Also used by the Table Editor)
+
+- Route validation to check for snippet validity + last visited snippet lies in the page level on [`[id].tsx`](https://github.com/supabase/supabase/blob/master/apps/studio/pages/project/%5Bref%5D/sql/%5Bid%5D.tsx)
+
+- When searching for snippets, we're deliberately opting to render the results as a flat list in [`SearchList`](https://github.com/supabase/supabase/blob/master/apps/studio/components/layouts/SQLEditorLayout/SQLEditorNavV2/SearchList.tsx) for ease of finding (rather than keeping the 3 separate sections / having folders)
+
+## Data structure
+
+- SQL Editor is mainly powered by a Valtio store in [`sql-editor-v2.ts`](https://github.com/supabase/supabase/blob/master/apps/studio/state/sql-editor-v2.ts), in which most of the data is being managed on the client side for optimistic rendering to keep the editor feeling snappy. (unlike other parts of the dashboard where the data is always invalidated whenever a mutation happens).
+
+- The Valtio store here stores snippets across multiple projects as we aren't using a context provider, though this was a legacy decision (refer to [`ProjectContext`](https://github.com/supabase/supabase/blob/master/apps/studio/components/layouts/ProjectLayout/ProjectContext.tsx) for more context on using providers with stores)
+
+- While [`SQLEditorNav`](https://github.com/supabase/supabase/blob/master/apps/studio/components/layouts/SQLEditorLayout/SQLEditorNavV2/SQLEditorNav.tsx) renders the folders + snippets directly from the API endpoints via React Query, we still store them in the Valtio store to store some properties used on the client side like `splitSizes` and `projectRef` for snippets, and `status` for folders (although it's possible that we can simplify the Valtio store).
+
+## Data fetching
+
+- The endpoint to fetch snippets and folders are via [`useSQLSnippetFoldersQuery`](https://github.com/supabase/supabase/blob/master/apps/studio/data/content/sql-folders-query.ts) and [`useSqlSnippetsQuery`](https://github.com/supabase/supabase/blob/master/apps/studio/data/content/sql-snippets-query.ts), both of which are paginated (limit set at 100)
+
+ - [`useSQLSnippetFoldersQuery`](https://github.com/supabase/supabase/blob/master/apps/studio/data/content/sql-folders-query.ts): Specifically for fetching private snippets and folders
+ - [`useSqlSnippetsQuery`](https://github.com/supabase/supabase/blob/master/apps/studio/data/content/sql-snippets-query.ts): For fetching shared and favorite snippets
+
+- Page fetching is done on demand for the snippets via a "Load more" button due to the complexity of a tree view (we've deliberate avoided an infinite loading UX which we commonly do across other parts of the dashboard)
+
+## Data flow
+
+### Landing on the SQL Editor
+
+- Snippets and folders are all initially loaded via React Query in [`SQLEditorNav`](https://github.com/supabase/supabase/blob/master/apps/studio/components/layouts/SQLEditorLayout/SQLEditorNavV2/SQLEditorNav.tsx), which are then initialized into the Valtio store via [`snapV2.addSnippet`](https://github.com/supabase/supabase/blob/master/apps/studio/components/layouts/SQLEditorLayout/SQLEditorNavV2/SQLEditorNav.tsx#L415) calls in the `useEffects`
+
+- On `/editor/sql`
+
+ - We'll redirect users to the last visited snippet if there's one (`/editor/sql/[id]`), otherwise will redirect to `/editor/sql/new` (within [`[id].tsx`](https://github.com/supabase/supabase/blob/master/apps/studio/pages/project/%5Bref%5D/sql/%5Bid%5D.tsx) )
+
+- On `/editor/sql/[id]`
+ - We'll load the content of the snippet via `useContentQuery` and update the Valtio store via [`snapV2.setSnippet`](https://github.com/supabase/supabase/blob/master/apps/studio/pages/project/%5Bref%5D/sql/%5Bid%5D.tsx#L69)
+
+### Writing a snippet
+
+- On `/editor/sql/new`:
+
+ - The first character input will [update the Valtio store](https://github.com/supabase/supabase/blob/master/apps/studio/components/interfaces/SQLEditor/MonacoEditor.tsx#L192) with a snippet skeleton via `snapV2.addSnippet` and user will be redirected to `/editor/sql/[id]`, using the `id` from the skeleton
+ - Note that `snapV2.addSnippet` only handles adding snippets to the store and does not queue the snippet for saving
+ - Subsequent character inputs will follow below as per `/editor/sql/[id]`
+
+- On `/editor/sql/[id]`:
+ - [`snapV2.setSql`](https://github.com/supabase/supabase/blob/master/apps/studio/components/interfaces/SQLEditor/MonacoEditor.tsx#L207) will be called based on the debounced value of the code editor, in which we'll then queue the snippet for [saving](https://github.com/supabase/supabase/blob/master/apps/studio/state/sql-editor-v2.ts#L477) via `upsertSnippet`.
+ - Note that we do invalidate some React Queries (snippet count, snippets, and folders) after saving via [`upsertSnippet`](https://github.com/supabase/supabase/blob/master/apps/studio/state/sql-editor-v2.ts#L412), but the invalidation is only [triggered](https://github.com/supabase/supabase/blob/master/apps/studio/components/interfaces/SQLEditor/MonacoEditor.tsx#L209) if it's a new snippet that's not saved in the DB yet
+
+### Running a snippet
+
+- There's several safeguards in place before we run the query
+
+ - [Check for destructive operations](https://github.com/supabase/supabase/blob/master/apps/studio/components/interfaces/SQLEditor/SQLEditor.tsx#L293): Will open a confirmation modal before running the query
+ - [Check for update statements without where clause](https://github.com/supabase/supabase/blob/master/apps/studio/components/interfaces/SQLEditor/SQLEditor.tsx#L300): Will open a confirmation modal before running the query
+ - [Append a preset limit to the query if it's a select](https://github.com/supabase/supabase/blob/master/apps/studio/components/interfaces/SQLEditor/SQLEditor.tsx#L333): To prevent accidentally running an expensive query on the database. Users can explicitly opt out of this to set "No limit"
+
+- If the snippet's name is "Untitled query", we'll also trigger a non-blocking request to [rename the snippet via AI](https://github.com/supabase/supabase/blob/master/apps/studio/components/interfaces/SQLEditor/SQLEditor.tsx#L317). Snippet name will be updated whenever the request completes.
+
+### Renaming, Moving, Deleting, Sharing snippets
+
+- These functionalities all call `upsertContent` via React Query directly without going through the Valtio store
+- We thereafter update the Valtio stores for the SQL Editor and Tabs as required if the upsert is successful
+
+## Possible areas to simplify, refactor, or improve
+
+- `updateSnippet` and `setSql` could be consolidated in `sql-editor-v2.ts`
+
+- Refactor renaming a query to have optimistic rendering for consistency on how we update snippets in the `sql-editor-v2.ts`
+
+- Implement drag and drop functionality for snippets into folders
+
+- `RenameQueryModal` and `MoveQueryModal`, could call `updateSnippet` instead of `removeSnippet` + `addSnippet`?
diff --git a/apps/studio/data/sql/execute-sql-mutation.ts b/apps/studio/data/sql/execute-sql-mutation.ts
index e55d95782b0..763e4dadcaa 100644
--- a/apps/studio/data/sql/execute-sql-mutation.ts
+++ b/apps/studio/data/sql/execute-sql-mutation.ts
@@ -4,8 +4,8 @@ import { toast } from 'sonner'
import { useSendEventMutation } from 'data/telemetry/send-event-mutation'
import { useSelectedOrganizationQuery } from 'hooks/misc/useSelectedOrganization'
import { sqlEventParser } from 'lib/sql-event-parser'
-import { executeSql, ExecuteSqlData, ExecuteSqlVariables } from './execute-sql-query'
import { UseCustomMutationOptions } from 'types'
+import { executeSql, ExecuteSqlData, ExecuteSqlVariables } from './execute-sql-query'
// [Joshen] Intention is that we invalidate all database related keys whenever running a mutation related query
// So we attempt to ignore all the non-related query keys. We could probably look into grouping our query keys better
diff --git a/apps/studio/state/sql-editor-v2.ts b/apps/studio/state/sql-editor-v2.ts
index fc6e4c45f96..77e43106e2b 100644
--- a/apps/studio/state/sql-editor-v2.ts
+++ b/apps/studio/state/sql-editor-v2.ts
@@ -1,52 +1,63 @@
-import { debounce, memoize } from 'lodash'
-import { useMemo } from 'react'
-import { toast } from 'sonner'
-import { proxy, ref, snapshot, subscribe, useSnapshot } from 'valtio'
-import { devtools, proxyMap } from 'valtio/utils'
-
-import { DiffType } from 'components/interfaces/SQLEditor/SQLEditor.types'
import type { QueryPlanRow } from 'components/interfaces/ExplainVisualizer/ExplainVisualizer.types'
-import { upsertContent, UpsertContentPayload } from 'data/content/content-upsert-mutation'
+import { DiffType } from 'components/interfaces/SQLEditor/SQLEditor.types'
+import { UpsertContentPayload, upsertContent } from 'data/content/content-upsert-mutation'
import { contentKeys } from 'data/content/keys'
import { createSQLSnippetFolder } from 'data/content/sql-folder-create-mutation'
import { updateSQLSnippetFolder } from 'data/content/sql-folder-update-mutation'
import { Snippet, SnippetFolder } from 'data/content/sql-folders-query'
import { getQueryClient } from 'data/query-client'
+import { debounce, memoize } from 'lodash'
+import { useMemo } from 'react'
+import { toast } from 'sonner'
import type { SqlSnippets } from 'types'
+import { proxy, ref, snapshot, subscribe, useSnapshot } from 'valtio'
+import { devtools, proxyMap } from 'valtio/utils'
-export type StateSnippetFolder = {
+type StateSnippetFolder = {
projectRef: string
folder: SnippetFolder
status?: 'editing' | 'saving' | 'idle'
}
+type StateSnippet = {
+ projectRef: string
+ splitSizes: number[]
+ snippet: SnippetWithContent
+}
+
// [Joshen] API codegen is somehow missing the content property
export interface SnippetWithContent extends Snippet {
content?: SqlSnippets.Content
isNotSavedInDatabaseYet?: boolean
}
-export type StateSnippet = {
- projectRef: string
- splitSizes: number[]
- snippet: SnippetWithContent
-}
-
const NEW_FOLDER_ID = 'new-folder'
export const sqlEditorState = proxy({
// ========================================================================
// ## Data properties within the store
// ========================================================================
+
+ /**
+ * Currently limitations include supporting up to one level of folders from root, and only private snippets
+ */
folders: {} as {
[folderId: string]: StateSnippetFolder
},
- // Private and Shared snippets only
+
+ /**
+ * Private and shared snippets only, favorite snippets are derivatives of them by the `favorite` property
+ */
snippets: {} as {
[snippetId: string]: StateSnippet
},
- // Query results, if any, for a snippet
+ /**
+ * Query results, if any, for a snippet. Set as an array per snippetId as we were previously experimenting
+ * with having a Jupyter notebook like UI but it never took off. Nonetheless kept this data structure as
+ * we'd also want to support returning multiple results from a single query (e.g From a query that contains
+ * multiple select statements), and this will allow us to do quite easily.
+ */
results: {} as {
[snippetId: string]: {
rows: any[]
@@ -54,25 +65,41 @@ export const sqlEditorState = proxy({
autoLimit?: number
}[]
},
- // Explain results, if any, for a snippet
+
+ /**
+ * Explain results, if any, for a snippet
+ */
explainResults: {} as {
[snippetId: string]: {
rows: QueryPlanRow[]
error?: { message: string; formattedError?: string }
}
},
- // Synchronous saving of folders and snippets (debounce behavior)
- // key is the snippet id, value is shouldInvalidate
+ /**
+ * Synchronous saving of folders and snippets (debounce behavior). Key is the snippet id, value is shouldInvalidate
+ */
needsSaving: proxyMap<string, boolean>([]),
- // Stores the state of each snippet
+ /**
+ * Stores the state of each snippet
+ */
savingStates: {} as {
[snippetId: string]: 'IDLE' | 'UPDATING' | 'UPDATING_FAILED'
},
+ /**
+ * UI-imposed limit for the number of results a query can return (applied to the SQL query being run if applicable).
+ * Acts as a safeguard to prevent accidentally taking down the database from a really large SELECT query.
+ * Related to `autoLimit` in `results`. Refer to `checkIfAppendLimitRequired` and `suffixWithLimit` for usage.
+ */
limit: 100,
- // For handling renaming folder failed
+
+ /**
+ * Used for error handling after optimistical rendering from renaming a folder
+ */
lastUpdatedFolderName: '',
- // For Assistant to render diffing into the editor
+ /**
+ * For Assistant to render diffing into the editor
+ */
diffContent: undefined as undefined | { sql: string; diffType: DiffType },
get allFolderNames() {
@@ -86,6 +113,9 @@ export const sqlEditorState = proxy({
setDiffContent: (sql: string, diffType: DiffType) =>
(sqlEditorState.diffContent = { sql, diffType }),
+ /**
+ * Load snippet into SQL Editor Valtio store
+ */
addSnippet: ({ projectRef, snippet }: { projectRef: string; snippet: SnippetWithContent }) => {
if (sqlEditorState.snippets[snippet.id]) return
@@ -95,6 +125,9 @@ export const sqlEditorState = proxy({
sqlEditorState.savingStates[snippet.id] = 'IDLE'
},
+ /**
+ * Update snippet data (e.g name, visibility, chart) and queue for sync saving
+ */
updateSnippet: ({
id,
snippet,
@@ -113,6 +146,11 @@ export const sqlEditorState = proxy({
}
},
+ /**
+ * Load snippet content into the snippet within the Valtio store.
+ * Snippets fetched from the GET /content or /folders endpoints do not have the content loaded initially
+ * to reduce the response size from the API. Hence content for each snippet has to be loaded on demand
+ */
setSnippet: (projectRef: string, snippet: SnippetWithContent) => {
let storedSnippet = sqlEditorState.snippets[snippet.id]
if (storedSnippet) {
@@ -124,6 +162,10 @@ export const sqlEditorState = proxy({
}
},
+ /**
+ * Update the snippet content of a snippet and queue for sync saving
+ * Possibly can consolidate with `updateSnippet` to simplify
+ */
setSql: ({
id,
sql,
@@ -140,6 +182,11 @@ export const sqlEditorState = proxy({
}
},
+ /**
+ * Update snippet in Valtio store after renaming
+ * Renaming a snippet follows an async saving and hence doesnt require queuing for sync saving here
+ * Refer to `RenameQueryModal.tsx` for more details
+ */
renameSnippet: ({
id,
name,
@@ -156,6 +203,9 @@ export const sqlEditorState = proxy({
}
},
+ /**
+ * Remove snippet from the Valtio store, and optionally remove snippet from the sync saving queue
+ */
removeSnippet: (id: string, skipSave: boolean = false) => {
const { [id]: snippet, ...otherSnippets } = sqlEditorState.snippets
sqlEditorState.snippets = otherSnippets
@@ -169,12 +219,17 @@ export const sqlEditorState = proxy({
if (!skipSave) sqlEditorState.needsSaving.delete(id)
},
+ /**
+ * Load folder into SQL Editor Valtio store
+ */
addFolder: ({ projectRef, folder }: { projectRef: string; folder: SnippetFolder }) => {
if (sqlEditorState.folders[folder.id]) return
-
sqlEditorState.folders[folder.id] = { projectRef, folder }
},
+ /**
+ * Adds a new folder placeholder for the UI to render
+ */
addNewFolder: ({ projectRef }: { projectRef: string }) => {
// [Joshen] Use this to identify new folders that have yet to be saved
const id = NEW_FOLDER_ID
@@ -195,6 +250,9 @@ export const sqlEditorState = proxy({
sqlEditorState.folders[id].status = 'editing'
},
+ /**
+ * For renaming a folder, queue for sync saving if pass all validations
+ */
saveFolder: ({ id, name }: { id: string; name: string }) => {
let storeFolder = sqlEditorState.folders[id]
const isNewFolder = id === 'new-folder'
@@ -220,11 +278,19 @@ export const sqlEditorState = proxy({
}
},
+ /**
+ * Remove folder from the Valtio store
+ * Deleting a folder follows an async saving and hence doesnt require queuing for sync saving here
+ * Refer to `SQLEditorNav` for more details (ConfirmationModal for deleting a folder)
+ */
removeFolder: (id: string) => {
const { [id]: folder, ...otherFolders } = sqlEditorState.folders
sqlEditorState.folders = otherFolders
},
+ /**
+ * Set the value for the auto limit for SELECT based SQL queries
+ */
setLimit: (value: number) => (sqlEditorState.limit = value),
addNeedsSaving: (id: string) => sqlEditorState.needsSaving.set(id, true),
@@ -333,7 +399,7 @@ async function upsertSnippet(
id: string,
projectRef: string,
payload: UpsertContentPayload,
- shouldInvalidate = true
+ shouldInvalidate = false
) {
try {
sqlEditorState.savingStates[id] = 'UPDATING'
@@ -358,14 +424,14 @@ async function upsertSnippet(
}
}
-const memoizedUpdateSnippet = memoize((_id: string) => debounce(upsertSnippet, 1000))
+const memoizedUpsertSnippet = memoize((_id: string) => debounce(upsertSnippet, 1000))
const debouncedUpdateSnippet = (
id: string,
projectRef: string,
payload: UpsertContentPayload,
shouldInvalidate = false
-) => memoizedUpdateSnippet(id)(id, projectRef, payload, shouldInvalidate)
+) => memoizedUpsertSnippet(id)(id, projectRef, payload, shouldInvalidate)
async function upsertFolder(id: string, projectRef: string, name: string) {
try {