Skip to content

Kepler.gl is a powerful open source geospatial analysis tool for large-scale data sets.

feat: addToDataset / removeFromDataset for in-place row edits (#176) (#3684)

Igor Dykhtabcf2be722e9b
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 4095bc454..8887faa66 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -15,6 +15,10 @@ Each version should:
Ref: http://keepachangelog.com/en/0.3.0/
-->
+## Unreleased
+
+- feat: `addToDataset` / `removeFromDataset` to append, upsert, and delete rows in place without breaking layers (#176). Row tables only; Arrow/DuckDB warn and no-op until INSERT/concat exist.
+
## [3.3.0-alpha.8] - Aug 26 2026
- af5d8173 chore: pin patched transitive deps via Yarn resolutions (#3654)
diff --git a/docs/api-reference/actions/actions.md b/docs/api-reference/actions/actions.md
index 629ac9170..117b1cb1f 100644
--- a/docs/api-reference/actions/actions.md
+++ b/docs/api-reference/actions/actions.md
@@ -26,6 +26,7 @@
- [visStateActions][42]
- [addFilter][43]
- [addLayer][45]
+ - [addToDataset](#addtodataset)
- [applyCPUFilter][47]
- [enlargeFilter][49]
- [interactionConfigChange][51]
@@ -41,6 +42,7 @@
- [onMapClick][71]
- [onMouseMove][72]
- [removeDataset][74]
+ - [removeFromDataset](#removefromdataset)
- [removeFilter][76]
- [removeLayer][78]
- [reorderLayer][80]
@@ -524,6 +526,23 @@ Add a new layer
Returns **{type: ActionTypes.ADD_LAYER, props: props}**
+### addToDataset
+
+Append or upsert rows on an existing in-memory row dataset without `addDataToMap`. Keeps layer identity and style. Pass `options.upsertBy` to replace rows that share that key and append the rest.
+
+Not implemented for Arrow or DuckDB tables (no INSERT / concat yet); those calls warn and leave the table unchanged. Use `addDataToMap` with `keepExistingConfig` for a full replace.
+
+- **ActionTypes**: `ActionTypes.ADD_TO_DATASET`
+- **Updaters**: `visStateUpdaters.addToDatasetUpdater`
+
+**Parameters**
+
+- `dataId` **[string][162]** dataset id
+- `rows` one row or an array of rows (column-ordered arrays, or objects keyed by field name)
+- `options` **[Object][164]** (optional) `{upsertBy: string}` unique key field
+
+Returns **{type: ActionTypes.ADD_TO_DATASET, dataId, rows, options}**
+
### applyCPUFilter
Trigger CPU filter of selected dataset
@@ -723,6 +742,22 @@ Remove a dataset and all layers, filters, tooltip configs that based on it
Returns **{type: ActionTypes.REMOVE_DATASET, key: key}**
+### removeFromDataset
+
+Delete rows from an existing in-memory row dataset without restyling layers. The second argument is either row indexes or `{field, values}` to match a column (for example an id).
+
+Not implemented for Arrow or DuckDB tables; those calls warn and leave the table unchanged.
+
+- **ActionTypes**: `ActionTypes.REMOVE_FROM_DATASET`
+- **Updaters**: `visStateUpdaters.removeFromDatasetUpdater`
+
+**Parameters**
+
+- `dataId` **[string][162]** dataset id
+- `rowIndexesOrMatcher` one index, an array of indexes, or `{field, values}`
+
+Returns **{type: ActionTypes.REMOVE_FROM_DATASET, dataId, rowIndexes?, byField?}**
+
### removeFilter
Remove a filter from `visState.filters`, once a filter is removed, data will be re-filtered and layer will be updated
diff --git a/docs/api-reference/reducers/vis-state.md b/docs/api-reference/reducers/vis-state.md
index 1996af724..365c0cd48 100644
--- a/docs/api-reference/reducers/vis-state.md
+++ b/docs/api-reference/reducers/vis-state.md
@@ -5,6 +5,7 @@
- [visStateUpdaters](#visstateupdaters)
- [addFilterUpdater](#addfilterupdater)
- [addLayerUpdater](#addlayerupdater)
+ - [addToDatasetUpdater](#addtodatasetupdater)
- [applyCPUFilterUpdater](#applycpufilterupdater)
- [enlargeFilterUpdater](#enlargefilterupdater)
- [INITIAL\_VIS\_STATE](#initial_vis_state)
@@ -20,6 +21,7 @@
- [mapClickUpdater](#mapclickupdater)
- [receiveMapConfigUpdater](#receivemapconfigupdater)
- [removeDatasetUpdater](#removedatasetupdater)
+ - [removeFromDatasetUpdater](#removefromdatasetupdater)
- [removeFilterUpdater](#removefilterupdater)
- [removeLayerUpdater](#removelayerupdater)
- [reorderLayerUpdater](#reorderlayerupdater)
@@ -103,6 +105,22 @@ Add a new layer
Returns **[Object][69]** nextState
+### addToDatasetUpdater
+
+Append or upsert rows on an existing in-memory row dataset. Keeps layers and re-runs filters. Arrow/DuckDB tables warn and are left unchanged (INSERT / concat not implemented).
+
+- **Action**: [`addToDataset`](../actions/actions.md#addtodataset)
+
+**Parameters**
+
+- `state` **[Object][69]** `visState`
+- `action` **[Object][69]** action
+ - `action.dataId` **[string][70]** dataset id
+ - `action.rows` rows to append or upsert
+ - `action.options.upsertBy` **[string][70]** (optional) unique key field
+
+Returns **[Object][69]** nextState
+
### applyCPUFilterUpdater
When select dataset for export, apply cpu filter to selected dataset
@@ -318,6 +336,22 @@ Remove a dataset and all layers, filters, tooltip configs that based on it
Returns **[Object][69]** nextState
+### removeFromDatasetUpdater
+
+Delete rows by index or by field values from an existing in-memory row dataset. Keeps layers. Arrow/DuckDB tables warn and are left unchanged.
+
+- **Action**: [`removeFromDataset`](../actions/actions.md#removefromdataset)
+
+**Parameters**
+
+- `state` **[Object][69]** `visState`
+- `action` **[Object][69]** action
+ - `action.dataId` **[string][70]** dataset id
+ - `action.rowIndexes` row indexes (optional)
+ - `action.byField` `{field, values}` matcher (optional)
+
+Returns **[Object][69]** nextState
+
### removeFilterUpdater
Remove a filter
diff --git a/examples/README.md b/examples/README.md
index f6c4cee83..08f970a23 100644
--- a/examples/README.md
+++ b/examples/README.md
@@ -52,7 +52,7 @@ yarn start
Read `visState.clicked` from Redux and show a GeoJSON feature's `shapeName` in a host-app sidebar outside the map.
- ### [Live Data][live-data]
- Remotely hosted CSV of points orbiting San Francisco on a 2-minute loop. Kepler.gl hydrates the URL on load and polls it every 300 ms so points move on the map.
+ Remotely hosted CSV of points orbiting San Francisco on a 2-minute loop. Toggle **Poll URL** (HTTP snapshot replace every 300 ms) and **Host rows** (`addToDataset` / `removeFromDataset`, including upsert and delete by `id`, without rebuilding the point layer).
- ### [Custom Layer][custom-layer]
Add a custom deck.gl layer (`ContourLayer`) to kepler.gl's layer type selector, so it can be picked from the dropdown, configured with dataset columns, and rendered on the map.
diff --git a/examples/live-data/README.md b/examples/live-data/README.md
index 0223a174e..309f6f9e8 100644
--- a/examples/live-data/README.md
+++ b/examples/live-data/README.md
@@ -1,13 +1,23 @@
# Live remotely hosted data
Loads a CORS CSV of points orbiting San Francisco as a remotely hosted dataset
-and **polls it every 1 second**. Positions are computed from wall-clock time:
-each point completes one full loop in **2 minutes**. A fetch returns the
-positions at that moment (there is no server-side snapshot timer).
-
-This is a manual test harness for snapshot refresh (`refreshDataset` /
-`metadata.refreshIntervalMs`) on `DatasetType.EXTERNALLY_HOSTED`. It bundles
-local monorepo `src/` (not a published npm package).
+and **polls it every 300 ms**. The right sidebar switches between two host-app
+update styles:
+
+- **Poll URL** — HTTP snapshot replace (`refreshDataset` /
+ `metadata.refreshIntervalMs`) on `DatasetType.EXTERNALLY_HOSTED`. Positions
+ come from wall-clock time; one full loop takes **2 minutes**.
+- **Host rows** — pauses the poll, then the example dispatches `addToDataset` /
+ `removeFromDataset` so rows change in place without tearing down the point
+ layer. Switching back to Poll URL resumes fetching and **replaces** the table
+ (injected rows disappear).
+
+This bundles local monorepo `src/` (not a published npm package). Host-row
+edits only work for in-memory CSV (`RowDataContainer`), which this example
+uses. Arrow and DuckDB tables log a warning and leave the data unchanged
+(INSERT / concat are not implemented yet). The **Keyed by id** controls
+exercise `addToDataset(id, rows, {upsertBy: 'id'})` and
+`removeFromDataset(id, {field: 'id', values})`.
## Pre-requirements
@@ -28,14 +38,27 @@ run next to `yarn start` on 8080). The CSV server listens at
## What to look for
+### Poll URL
+
- Three points on three rings around San Francisco. Adjacent rings move in
- opposite directions. One full orbit takes 2 minutes.
-- The right sidebar **orbit** percent advances with each poll (`progress` /
+ opposite directions. Color is `ring`. One full orbit takes 2 minutes.
+- The sidebar **orbit** percent advances with each poll (`progress` /
`orbit_s` in the table).
-- Layers panel → one **Point** layer (trip is not auto-created). Refresh is
- set on the dataset row; the reload icon samples the current time immediately.
-- Field names stay the same, so the point layer is kept (not rebuilt from
- scratch).
+- Layers panel → one **Point** layer. Refresh is set on the dataset row; the
+ reload icon samples the current time immediately.
+
+### Host rows
+
+- Orbit **freezes** (interval set to 0).
+- **Add 1 / Add 5** drop points on outer rings (4–6), same layer id.
+- **Remove last / Remove random** delete by row index.
+- **Keyed by id** — **Insert/Move tracker** upserts `track-01` (`upsertBy: 'id'`).
+ First click adds a ring-7 point; later clicks move that same row (ids list
+ and row count stay put). **Remove tracker** / **Remove veh-01** /
+ **Remove last host** delete with `{field: 'id', values}`.
+- **Auto trail** appends a point every 800ms and drops the oldest host row
+ after 20 so the cloud stays bounded.
+- Switch back to Poll URL: next CSV fetch restores the 3 orbiting vehicles.
## CSV server only (Add Data URL)
@@ -52,6 +75,6 @@ URL**
http://localhost:4010/vehicles.csv
```
-Fetch, then set **Refresh** to 1s (or Custom) on the dataset in the Layers panel.
+Fetch, then set **Refresh** to Custom (0.3s) on the dataset in the Layers panel.
[yarn-install]: https://yarnpkg.com/getting-started/install
diff --git a/examples/live-data/src/app.tsx b/examples/live-data/src/app.tsx
index 80d23e2b7..dd7eaa6f4 100644
--- a/examples/live-data/src/app.tsx
+++ b/examples/live-data/src/app.tsx
@@ -2,24 +2,42 @@
// Copyright contributors to the kepler.gl project
import * as React from 'react';
-import {useState, useEffect} from 'react';
+import {useCallback, useEffect, useRef, useState} from 'react';
import ReactDOM from 'react-dom/client';
import {Provider, useDispatch, useSelector} from 'react-redux';
import {applyMiddleware, combineReducers, compose, createStore} from 'redux';
import keplerGlReducer, {enhanceReduxMiddleware} from '@kepler.gl/reducers';
import KeplerGl from '@kepler.gl/components';
-import {addDataToMap, wrapTo} from '@kepler.gl/actions';
+import {
+ addDataToMap,
+ addToDataset,
+ refreshDataset,
+ removeFromDataset,
+ updateDatasetProps,
+ wrapTo
+} from '@kepler.gl/actions';
import {DatasetType} from '@kepler.gl/constants';
import {initApplicationConfig} from '@kepler.gl/utils';
initApplicationConfig({enableAnnotations: false});
const MAP_ID = 'map';
-const SIDEBAR_WIDTH = 280;
+const SIDEBAR_WIDTH = 300;
const DATASET_ID = 'live-vehicles';
const LIVE_DATA_URL = process.env.LIVE_DATA_URL || 'http://localhost:4010/vehicles.csv';
const REFRESH_INTERVAL_MS = 300;
+const AUTO_INTERVAL_MS = 800;
+const MAX_HOST_ROWS = 20;
+const TRACKER_ID = 'track-01';
+const POLL_VEHICLE_ID = 'veh-01';
+
+const SF = {lat: 37.7749, lng: -122.4194};
+const COS_LAT = Math.cos((SF.lat * Math.PI) / 180);
+const KM_PER_DEG_LAT = 111.32;
+const GOLDEN_ANGLE = Math.PI * (3 - Math.sqrt(5));
+
+type LiveMode = 'poll' | 'host';
type KeplerRootState = {
keplerGl: {
@@ -47,6 +65,8 @@ type KeplerRootState = {
};
};
+type LiveDataset = KeplerRootState['keplerGl'][string]['visState']['datasets'][string];
+
const reducers = combineReducers({
keplerGl: keplerGlReducer.initialState({
mapStyle: {
@@ -64,10 +84,7 @@ const enhancers = applyMiddleware(...middlewares);
const composeEnhancers = (window as any).__REDUX_DEVTOOLS_EXTENSION_COMPOSE__ || compose;
const store = createStore(reducers, {}, composeEnhancers(enhancers));
-function fieldValue(
- dataset: KeplerRootState['keplerGl'][string]['visState']['datasets'][string] | undefined,
- name: string
-) {
+function fieldValue(dataset: LiveDataset | undefined, name: string) {
const col = dataset?.fields?.findIndex(field => field.name === name) ?? -1;
if (col < 0 || !dataset?.dataContainer?.numRows()) {
return undefined;
@@ -75,6 +92,50 @@ function fieldValue(
return dataset.dataContainer.valueAt(0, col);
}
+function columnIndex(dataset: LiveDataset | undefined, name: string) {
+ return dataset?.fields?.findIndex(field => field.name === name) ?? -1;
+}
+
+function rowIds(dataset: LiveDataset | undefined): string[] {
+ const idCol = columnIndex(dataset, 'id');
+ const rows = dataset?.dataContainer?.numRows?.() ?? 0;
+ if (idCol < 0 || !rows) {
+ return [];
+ }
+ const ids: string[] = [];
+ for (let i = 0; i < rows; i++) {
+ ids.push(String(dataset?.dataContainer?.valueAt(i, idCol) ?? ''));
+ }
+ return ids;
+}
+
+function hostRowIndexes(dataset: LiveDataset | undefined): number[] {
+ return rowIds(dataset).reduce<number[]>((indexes, id, index) => {
+ if (id.startsWith('host-')) {
+ indexes.push(index);
+ }
+ return indexes;
+ }, []);
+}
+
+function makeHostVehicle(seq: number, options?: {id?: string; ring?: number}) {
+ const angle = seq * GOLDEN_ANGLE;
+ const radiusKm = 5.2 + (seq % 8) * 0.35;
+ const lat = SF.lat + (radiusKm / KM_PER_DEG_LAT) * Math.sin(angle);
+ const lng = SF.lng + (radiusKm / (KM_PER_DEG_LAT * COS_LAT)) * Math.cos(angle);
+ return {
+ id: options?.id ?? `host-${String(seq).padStart(3, '0')}`,
+ lat,
+ lng,
+ heading: ((angle * 180) / Math.PI + 360) % 360,
+ speed: 0,
+ ring: options?.ring ?? 4 + (seq % 3),
+ progress: 0,
+ orbit_s: 0,
+ updated_at: new Date().toISOString()
+ };
+}
+
function useWindowSize() {
const [size, setSize] = useState({width: window.innerWidth, height: window.innerHeight});
useEffect(() => {
@@ -98,10 +159,65 @@ const sidebarStyle: React.CSSProperties = {
flexShrink: 0
};
-function LiveDataSidebar() {
- const dataset = useSelector(
- (state: KeplerRootState) => state.keplerGl[MAP_ID]?.visState?.datasets?.[DATASET_ID]
+const buttonStyle: React.CSSProperties = {
+ padding: '7px 10px',
+ border: '1px solid #4a5160',
+ borderRadius: 4,
+ background: '#3a3f4b',
+ color: '#f7f7f7',
+ cursor: 'pointer',
+ fontSize: 13
+};
+
+function ModeToggle({
+ mode,
+ hostEnabled,
+ onChange
+}: {
+ mode: LiveMode;
+ hostEnabled: boolean;
+ onChange: (next: LiveMode) => void;
+}) {
+ return (
+ <div
+ style={{
+ display: 'flex',
+ marginBottom: 14,
+ border: '1px solid #4a5160',
+ borderRadius: 6,
+ overflow: 'hidden'
+ }}
+ >
+ {(
+ [
+ ['poll', 'Poll URL', true],
+ ['host', 'Host rows', hostEnabled]
+ ] as const
+ ).map(([value, label, enabled]) => (
+ <button
+ key={value}
+ type="button"
+ disabled={!enabled}
+ onClick={() => onChange(value)}
+ style={{
+ flex: 1,
+ padding: '8px 10px',
+ border: 0,
+ cursor: enabled ? 'pointer' : 'default',
+ background: mode === value ? '#6a7485' : 'transparent',
+ color: enabled ? '#f7f7f7' : '#6a7485',
+ fontWeight: mode === value ? 600 : 400,
+ fontSize: 13
+ }}
+ >
+ {label}
+ </button>
+ ))}
+ </div>
);
+}
+
+function PollSidebar({dataset}: {dataset: LiveDataset | undefined}) {
const progressAlongOrbit = fieldValue(dataset, 'progress');
const orbitS = fieldValue(dataset, 'orbit_s');
const updatedAt = fieldValue(dataset, 'updated_at');
@@ -110,12 +226,11 @@ function LiveDataSidebar() {
const progress = dataset?.metadata?.refreshProgress;
return (
- <aside style={sidebarStyle}>
- <h2 style={{margin: '0 0 8px', fontSize: 16}}>Live orbit</h2>
+ <>
<p style={{margin: '0 0 12px', color: '#c3c9d5'}}>
- CSV at <code>{LIVE_DATA_URL}</code> returns the current positions of 3 points circling San
- Francisco (one full loop every 2 min). Kepler polls that URL once a second and replaces the
- table without tearing down the point layer.
+ CSV at <code>{LIVE_DATA_URL}</code> returns 3 points circling San Francisco (one loop every
+ 2 min). Kepler polls that URL and <strong>replaces the whole table</strong> without tearing
+ down the point layer.
</p>
<div style={{display: 'grid', gap: 8}}>
<div style={{display: 'flex', justifyContent: 'space-between', gap: 12}}>
@@ -152,9 +267,280 @@ function LiveDataSidebar() {
<p style={{margin: '12px 0 0', color: '#f19c99'}}>{dataset.metadata.refreshError}</p>
) : null}
<p style={{margin: '16px 0 0', color: '#c3c9d5'}}>
- In the Layers panel, dataset refresh is set to 1s. Click the reload icon to sample the orbit
- immediately.
+ Layers panel: dataset refresh is 300ms. Click reload to sample the orbit immediately.
+ Switching to <strong>Host rows</strong> pauses the poll so you can append or delete without
+ the next snapshot wiping your edits.
</p>
+ </>
+ );
+}
+
+function HostSidebar({
+ dataset,
+ autoTrail,
+ keyedNote,
+ onAutoTrail,
+ onAdd,
+ onRemoveLast,
+ onRemoveRandom,
+ onUpsertTracker,
+ onRemoveTracker,
+ onRemovePollVehicle,
+ onRemoveLastHostById
+}: {
+ dataset: LiveDataset | undefined;
+ autoTrail: boolean;
+ keyedNote: string;
+ onAutoTrail: (next: boolean) => void;
+ onAdd: (count: number) => void;
+ onRemoveLast: () => void;
+ onRemoveRandom: () => void;
+ onUpsertTracker: () => void;
+ onRemoveTracker: () => void;
+ onRemovePollVehicle: () => void;
+ onRemoveLastHostById: () => void;
+}) {
+ const rows = dataset?.dataContainer?.numRows?.() ?? 0;
+ const ids = rowIds(dataset);
+ const hostCount = hostRowIndexes(dataset).length;
+ const hasTracker = ids.includes(TRACKER_ID);
+ const hasPollVehicle = ids.includes(POLL_VEHICLE_ID);
+ const hostIds = ids.filter(id => id.startsWith('host-'));
+ const lastHostId = hostIds[hostIds.length - 1];
+
+ return (
+ <>
+ <p style={{margin: '0 0 12px', color: '#c3c9d5'}}>
+ Polling is off. The host app dispatches <code>addToDataset</code> /{' '}
+ <code>removeFromDataset</code> so rows change in place — same layer id and style. Host
+ points land on outer rings (4–6) and color by <code>ring</code>.
+ </p>
+ <div style={{display: 'grid', gap: 8, marginBottom: 12}}>
+ <div style={{display: 'flex', justifyContent: 'space-between', gap: 12}}>
+ <span style={{color: '#c3c9d5'}}>rows</span>
+ <span>
+ {rows} <span style={{color: '#c3c9d5'}}>({hostCount} host)</span>
+ </span>
+ </div>
+ <div
+ style={{
+ display: 'flex',
+ justifyContent: 'space-between',
+ gap: 12,
+ alignItems: 'flex-start'
+ }}
+ >
+ <span style={{color: '#c3c9d5', flexShrink: 0}}>ids</span>
+ <span style={{textAlign: 'right', wordBreak: 'break-word'}}>
+ {ids.length ? ids.join(', ') : '—'}
+ </span>
+ </div>
+ </div>
+ <div style={{display: 'grid', gridTemplateColumns: '1fr 1fr', gap: 8}}>
+ <button type="button" style={buttonStyle} onClick={() => onAdd(1)} disabled={!dataset}>
+ Add 1
+ </button>
+ <button type="button" style={buttonStyle} onClick={() => onAdd(5)} disabled={!dataset}>
+ Add 5
+ </button>
+ <button type="button" style={buttonStyle} onClick={onRemoveLast} disabled={rows < 1}>
+ Remove last
+ </button>
+ <button type="button" style={buttonStyle} onClick={onRemoveRandom} disabled={rows < 1}>
+ Remove random
+ </button>
+ </div>
+ <p style={{margin: '16px 0 8px', fontWeight: 600}}>Keyed by id</p>
+ <p style={{margin: '0 0 8px', color: '#c3c9d5'}}>
+ <code>upsertBy: &apos;id&apos;</code> keeps the same row. First click inserts{' '}
+ <code>{TRACKER_ID}</code> (ring 7); later clicks move it and the row count stays put.
+ Removes use <code>{'{field, values}'}</code>, not a row index.
+ </p>
+ <div style={{display: 'grid', gridTemplateColumns: '1fr 1fr', gap: 8}}>
+ <button type="button" style={buttonStyle} onClick={onUpsertTracker} disabled={!dataset}>
+ {hasTracker ? 'Move tracker' : 'Insert tracker'}
+ </button>
+ <button type="button" style={buttonStyle} onClick={onRemoveTracker} disabled={!hasTracker}>
+ Remove tracker
+ </button>
+ <button
+ type="button"
+ style={buttonStyle}
+ onClick={onRemovePollVehicle}
+ disabled={!hasPollVehicle}
+ >
+ Remove {POLL_VEHICLE_ID}
+ </button>
+ <button
+ type="button"
+ style={buttonStyle}
+ onClick={onRemoveLastHostById}
+ disabled={!lastHostId}
+ >
+ Remove last host
+ </button>
+ </div>
+ <p style={{margin: '8px 0 0', color: hasTracker ? '#9ad0a8' : '#c3c9d5'}}>
+ {TRACKER_ID}: {hasTracker ? 'on map' : 'missing'}
+ {lastHostId ? ` · last host ${lastHostId}` : ''}
+ </p>
+ {keyedNote ? <p style={{margin: '6px 0 0', color: '#9ad0a8'}}>{keyedNote}</p> : null}
+ <label
+ style={{
+ display: 'flex',
+ alignItems: 'center',
+ gap: 8,
+ marginTop: 14,
+ cursor: 'pointer',
+ color: '#c3c9d5'
+ }}
+ >
+ <input
+ type="checkbox"
+ checked={autoTrail}
+ onChange={event => onAutoTrail(event.target.checked)}
+ disabled={!dataset}
+ />
+ Auto trail (add every 800ms, keep {MAX_HOST_ROWS} host rows)
+ </label>
+ <p style={{margin: '16px 0 0', color: '#c3c9d5'}}>
+ Switch back to <strong>Poll URL</strong> to resume HTTP snapshot replace — injected rows
+ disappear on the next fetch.
+ </p>
+ </>
+ );
+}
+
+function LiveDataSidebar() {
+ const dispatch = useDispatch();
+ const dataset = useSelector(
+ (state: KeplerRootState) => state.keplerGl[MAP_ID]?.visState?.datasets?.[DATASET_ID]
+ );
+ const [mode, setMode] = useState<LiveMode>('poll');
+ const [autoTrail, setAutoTrail] = useState(false);
+ const [keyedNote, setKeyedNote] = useState('');
+ const hostSeq = useRef(1);
+ const trackSeq = useRef(1);
+ const datasetRef = useRef(dataset);
+ datasetRef.current = dataset;
+
+ const toKepler = useCallback(
+ (action: Parameters<typeof wrapTo>[1]) => dispatch(wrapTo(MAP_ID, action)),
+ [dispatch]
+ );
+
+ const setLiveMode = useCallback(
+ (next: LiveMode) => {
+ setMode(next);
+ if (next === 'poll') {
+ setAutoTrail(false);
+ setKeyedNote('');
+ toKepler(
+ updateDatasetProps(DATASET_ID, {metadata: {refreshIntervalMs: REFRESH_INTERVAL_MS}})
+ );
+ toKepler(refreshDataset(DATASET_ID));
+ return;
+ }
+ toKepler(updateDatasetProps(DATASET_ID, {metadata: {refreshIntervalMs: 0}}));
+ },
+ [toKepler]
+ );
+
+ const addHostRows = useCallback(
+ (count: number) => {
+ const rows = Array.from({length: count}, () => makeHostVehicle(hostSeq.current++));
+ toKepler(addToDataset(DATASET_ID, rows));
+ },
+ [toKepler]
+ );
+
+ const removeLast = useCallback(() => {
+ const rows = dataset?.dataContainer?.numRows?.() ?? 0;
+ if (rows < 1) {
+ return;
+ }
+ toKepler(removeFromDataset(DATASET_ID, rows - 1));
+ }, [dataset, toKepler]);
+
+ const removeRandom = useCallback(() => {
+ const rows = dataset?.dataContainer?.numRows?.() ?? 0;
+ if (rows < 1) {
+ return;
+ }
+ toKepler(removeFromDataset(DATASET_ID, Math.floor(Math.random() * rows)));
+ }, [dataset, toKepler]);
+
+ const upsertTracker = useCallback(() => {
+ const ids = rowIds(datasetRef.current);
+ const existed = ids.includes(TRACKER_ID);
+ const row = makeHostVehicle(trackSeq.current++, {id: TRACKER_ID, ring: 7});
+ toKepler(addToDataset(DATASET_ID, row, {upsertBy: 'id'}));
+ setKeyedNote(
+ existed
+ ? `upsert ${TRACKER_ID} (replaced — row count should stay ${ids.length})`
+ : `upsert ${TRACKER_ID} (inserted — row count ${ids.length + 1})`
+ );
+ }, [toKepler]);
+
+ const removeByFieldId = useCallback(
+ (id: string) => {
+ if (!rowIds(datasetRef.current).includes(id)) {
+ return;
+ }
+ toKepler(removeFromDataset(DATASET_ID, {field: 'id', values: id}));
+ setKeyedNote(`removeFromDataset({field: 'id', values: '${id}'})`);
+ },
+ [toKepler]
+ );
+
+ const removeLastHostById = useCallback(() => {
+ const hostIds = rowIds(datasetRef.current).filter(id => id.startsWith('host-'));
+ const lastHostId = hostIds[hostIds.length - 1];
+ if (!lastHostId) {
+ return;
+ }
+ removeByFieldId(lastHostId);
+ }, [removeByFieldId]);
+
+ useEffect(() => {
+ if (mode !== 'host' || !autoTrail) {
+ return undefined;
+ }
+ const timer = window.setInterval(() => {
+ const extras = hostRowIndexes(datasetRef.current);
+ if (extras.length >= MAX_HOST_ROWS) {
+ toKepler(removeFromDataset(DATASET_ID, extras[0]));
+ }
+ toKepler(addToDataset(DATASET_ID, makeHostVehicle(hostSeq.current++)));
+ }, AUTO_INTERVAL_MS);
+ return () => window.clearInterval(timer);
+ }, [autoTrail, mode, toKepler]);
+
+ return (
+ <aside style={sidebarStyle}>
+ <h2 style={{margin: '0 0 8px', fontSize: 16}}>Live data</h2>
+ <ModeToggle
+ mode={mode}
+ hostEnabled={(dataset?.dataContainer?.numRows?.() ?? 0) > 0}
+ onChange={setLiveMode}
+ />
+ {mode === 'poll' ? (
+ <PollSidebar dataset={dataset} />
+ ) : (
+ <HostSidebar
+ dataset={dataset}
+ autoTrail={autoTrail}
+ keyedNote={keyedNote}
+ onAutoTrail={setAutoTrail}
+ onAdd={addHostRows}
+ onRemoveLast={removeLast}
+ onRemoveRandom={removeRandom}
+ onUpsertTracker={upsertTracker}
+ onRemoveTracker={() => removeByFieldId(TRACKER_ID)}
+ onRemovePollVehicle={() => removeByFieldId(POLL_VEHICLE_ID)}
+ onRemoveLastHostById={removeLastHostById}
+ />
+ )}
</aside>
);
}
@@ -200,6 +586,10 @@ const App = () => {
columns: {lat: 'lat', lng: 'lng', altitude: null},
isVisible: true,
visConfig: {radius: 24, filled: true, opacity: 0.85}
+ },
+ visualChannels: {
+ colorField: {name: 'ring', type: 'integer'},
+ colorScale: 'quantize'
}
}
]
diff --git a/src/actions/src/action-types.ts b/src/actions/src/action-types.ts
index 2e5999fb5..7285edbfa 100644
--- a/src/actions/src/action-types.ts
+++ b/src/actions/src/action-types.ts
@@ -120,6 +120,8 @@ export const ActionTypes = {
REFRESH_DATASET_SUCCESS: `${ACTION_PREFIX}REFRESH_DATASET_SUCCESS`,
REFRESH_DATASET_ERROR: `${ACTION_PREFIX}REFRESH_DATASET_ERROR`,
REFRESH_DATASET_PROGRESS: `${ACTION_PREFIX}REFRESH_DATASET_PROGRESS`,
+ ADD_TO_DATASET: `${ACTION_PREFIX}ADD_TO_DATASET`,
+ REMOVE_FROM_DATASET: `${ACTION_PREFIX}REMOVE_FROM_DATASET`,
ADD_EFFECT: `${ACTION_PREFIX}ADD_EFFECT`,
REORDER_EFFECT: `${ACTION_PREFIX}REORDER_EFFECT`,
REMOVE_EFFECT: `${ACTION_PREFIX}REMOVE_EFFECT`,
diff --git a/src/actions/src/actions.ts b/src/actions/src/actions.ts
index 7695fc8b2..743cc586d 100644
--- a/src/actions/src/actions.ts
+++ b/src/actions/src/actions.ts
@@ -51,6 +51,7 @@ export type ActionHandlers<T extends {[k: string]: Handler}> = {
* @param {boolean} data.options.readOnly `default: false` if `readOnly` is set to `true`
* the left setting panel will be hidden
* @param {boolean} data.options.keepExistingConfig whether to keep exiting map data and associated layer filter interaction config `default: false`.
+ * To append, upsert, or delete rows on an existing in-memory row dataset without replacing it or creating layers, use `addToDataset` / `removeFromDataset`. Arrow and DuckDB tables do not support those actions yet.
* @param {Object} data.config this object will contain the full kepler.gl instance configuration {mapState, mapStyle, visState}
* @public
* @example
diff --git a/src/actions/src/vis-state-actions.ts b/src/actions/src/vis-state-actions.ts
index 9266587e0..80710fbfd 100644
--- a/src/actions/src/vis-state-actions.ts
+++ b/src/actions/src/vis-state-actions.ts
@@ -1031,6 +1031,101 @@ export function refreshDatasetProgress(
};
}
+/**
+ * One dataset row: a column-ordered array (same as `ProtoDataset.data.rows`)
+ * or a field-name record. Missing keys become `null`.
+ */
+export type DatasetRow = any[] | Record<string, unknown>;
+
+export type AddToDatasetOptions = {
+ /**
+ * Field name used as a unique key. Matching rows are replaced in place;
+ * keys that are not in the table are appended. Last incoming row wins when
+ * the same key appears twice. Row tables only — Arrow/DuckDB no-op with a warning.
+ */
+ upsertBy?: string;
+};
+
+export type AddToDatasetUpdaterAction = {
+ dataId: string;
+ rows: DatasetRow | DatasetRow[];
+ options?: AddToDatasetOptions;
+};
+/**
+ * Append rows to an existing in-memory row dataset without `addDataToMap`.
+ * Keeps layer identity and style. Pass `options.upsertBy` to replace rows that
+ * share that key and append the rest.
+ *
+ * Not implemented for Arrow or DuckDB tables (no INSERT / concat yet); those
+ * calls warn and leave the table unchanged. Use `addDataToMap` with
+ * `keepExistingConfig` for a full replace.
+ * @memberof visStateActions
+ * @param dataId dataset id
+ * @param rows one row or an array of rows (arrays in field order, or objects keyed by field name)
+ * @param options optional `{upsertBy}` field name
+ * @returns action
+ * @public
+ */
+export function addToDataset(
+ dataId: string,
+ rows: DatasetRow | DatasetRow[],
+ options?: AddToDatasetOptions
+): Merge<AddToDatasetUpdaterAction, {type: typeof ActionTypes.ADD_TO_DATASET}> {
+ return {
+ type: ActionTypes.ADD_TO_DATASET,
+ dataId,
+ rows,
+ options
+ };
+}
+
+export type RemoveFromDatasetByField = {
+ field: string;
+ values: unknown | unknown[];
+};
+
+export type RemoveFromDatasetUpdaterAction = {
+ dataId: string;
+ rowIndexes?: number | number[];
+ byField?: RemoveFromDatasetByField;
+};
+/**
+ * Delete rows from an existing in-memory row dataset without restyling layers.
+ * The second argument is either row indexes or `{field, values}` to match a
+ * column (e.g. an id). Out-of-range indexes are a no-op for the whole action.
+ *
+ * Not implemented for Arrow or DuckDB tables; those calls warn and leave the
+ * table unchanged. For a full table replace use `addDataToMap` with
+ * `keepExistingConfig`.
+ * @memberof visStateActions
+ * @param dataId dataset id
+ * @param rowIndexesOrMatcher one index, an array of indexes, or `{field, values}`
+ * @returns action
+ * @public
+ */
+export function removeFromDataset(
+ dataId: string,
+ rowIndexesOrMatcher: number | number[] | RemoveFromDatasetByField
+): Merge<RemoveFromDatasetUpdaterAction, {type: typeof ActionTypes.REMOVE_FROM_DATASET}> {
+ if (
+ rowIndexesOrMatcher &&
+ typeof rowIndexesOrMatcher === 'object' &&
+ !Array.isArray(rowIndexesOrMatcher) &&
+ 'field' in rowIndexesOrMatcher
+ ) {
+ return {
+ type: ActionTypes.REMOVE_FROM_DATASET,
+ dataId,
+ byField: rowIndexesOrMatcher
+ };
+ }
+ return {
+ type: ActionTypes.REMOVE_FROM_DATASET,
+ dataId,
+ rowIndexes: rowIndexesOrMatcher
+ };
+}
+
export type SortTableColumnUpdaterAction = {
dataId: string;
column: string;
diff --git a/src/reducers/src/vis-state-updaters.ts b/src/reducers/src/vis-state-updaters.ts
index 31531714a..dc79fa0a0 100644
--- a/src/reducers/src/vis-state-updaters.ts
+++ b/src/reducers/src/vis-state-updaters.ts
@@ -3213,8 +3213,16 @@ export function refreshDatasetSuccessUpdater(
return rebindLayersAndFiltersAfterSchemaChange(state, dataId, datasets);
}
- const filteredDatasets = applyFiltersToDatasets([dataId], datasets, state.filters, state.layers);
+ return reapplyFiltersAndLayerData({...state, datasets}, dataId);
+}
+function reapplyFiltersAndLayerData(state: VisState, dataId: string): VisState {
+ const filteredDatasets = applyFiltersToDatasets(
+ [dataId],
+ state.datasets,
+ state.filters,
+ state.layers
+ );
let nextState: VisState = {
...state,
datasets: filteredDatasets
@@ -3223,6 +3231,118 @@ export function refreshDatasetSuccessUpdater(
return updateAnimationDomain(nextState);
}
+function isDatasetRowList(
+ rows: VisStateActions.DatasetRow | VisStateActions.DatasetRow[]
+): rows is VisStateActions.DatasetRow[] {
+ if (!Array.isArray(rows) || !rows.length) {
+ return Array.isArray(rows);
+ }
+ // A single column-ordered row is an array of primitives; a list of rows is
+ // an array of arrays or an array of field-name records.
+ return Array.isArray(rows[0]) || isPlainObject(rows[0]);
+}
+
+function toArrayRow(row: VisStateActions.DatasetRow, fields: Field[]): any[] | null {
+ if (Array.isArray(row)) {
+ return row.length === fields.length ? row : null;
+ }
+ if (isPlainObject(row)) {
+ const record = row as Record<string, unknown>;
+ return fields.map(field =>
+ Object.prototype.hasOwnProperty.call(record, field.name) ? record[field.name] : null
+ );
+ }
+ return null;
+}
+
+function normalizeDatasetRows(
+ rows: VisStateActions.DatasetRow | VisStateActions.DatasetRow[],
+ fields: Field[]
+): any[][] | null {
+ if (!fields.length) {
+ return null;
+ }
+ const list = isDatasetRowList(rows) ? rows : [rows];
+ if (!list.length) {
+ return null;
+ }
+ const normalized: any[][] = [];
+ for (let i = 0; i < list.length; i++) {
+ const arrayRow = toArrayRow(list[i], fields);
+ if (!arrayRow) {
+ return null;
+ }
+ normalized.push(arrayRow);
+ }
+ return normalized;
+}
+
+/**
+ * Append or upsert rows on an existing in-memory row dataset. Keeps layers and re-runs filters.
+ * Arrow/DuckDB tables warn and are left unchanged (INSERT / concat not implemented).
+ * @memberof visStateUpdaters
+ * @public
+ */
+export function addToDatasetUpdater(
+ state: VisState,
+ {dataId, rows, options}: VisStateActions.AddToDatasetUpdaterAction
+): VisState {
+ const dataset = state.datasets[dataId];
+ if (!dataset) {
+ return state;
+ }
+
+ const normalized = normalizeDatasetRows(rows, dataset.fields);
+ if (!normalized) {
+ return state;
+ }
+
+ const applied = options?.upsertBy
+ ? dataset.upsertRows(normalized, options.upsertBy)
+ : dataset.appendRows(normalized);
+ if (!applied) {
+ return state;
+ }
+
+ const datasets = {
+ ...state.datasets,
+ [dataId]: copyTableAndUpdate(dataset, {})
+ };
+ return reapplyFiltersAndLayerData({...state, datasets}, dataId);
+}
+
+/**
+ * Delete rows by index or by field values from an existing in-memory row dataset.
+ * Keeps layers. Arrow/DuckDB tables warn and are left unchanged.
+ * @memberof visStateUpdaters
+ * @public
+ */
+export function removeFromDatasetUpdater(
+ state: VisState,
+ {dataId, rowIndexes, byField}: VisStateActions.RemoveFromDatasetUpdaterAction
+): VisState {
+ const dataset = state.datasets[dataId];
+ if (!dataset) {
+ return state;
+ }
+
+ const indexes = byField
+ ? dataset.findRowIndexesByFieldValues(byField.field, toArray(byField.values))
+ : toArray(rowIndexes ?? []);
+ if (!indexes.length) {
+ return state;
+ }
+ if (!dataset.removeRows(indexes)) {
+ return state;
+ }
+
+ const datasets = {
+ ...state.datasets,
+ [dataId]: copyTableAndUpdate(dataset, {})
+ };
+ return reapplyFiltersAndLayerData({...state, datasets}, dataId);
+}
+
/**
* Record a failed remote dataset refresh without dropping existing rows.
* @memberof visStateUpdaters
diff --git a/src/reducers/src/vis-state.ts b/src/reducers/src/vis-state.ts
index 790d2bfe2..9cc07b9f8 100644
--- a/src/reducers/src/vis-state.ts
+++ b/src/reducers/src/vis-state.ts
@@ -111,6 +111,10 @@ const actionHandler = {
[ActionTypes.REFRESH_DATASET_PROGRESS]: visStateUpdaters.refreshDatasetProgressUpdater,
+ [ActionTypes.ADD_TO_DATASET]: visStateUpdaters.addToDatasetUpdater,
+
+ [ActionTypes.REMOVE_FROM_DATASET]: visStateUpdaters.removeFromDatasetUpdater,
+
[ActionTypes.TOGGLE_FILTER_ANIMATION]: visStateUpdaters.toggleFilterAnimationUpdater,
[ActionTypes.UPDATE_FILTER_ANIMATION_SPEED]: visStateUpdaters.updateFilterAnimationSpeedUpdater,
diff --git a/src/table/src/kepler-table.ts b/src/table/src/kepler-table.ts
index b0653c242..7216428b1 100644
--- a/src/table/src/kepler-table.ts
+++ b/src/table/src/kepler-table.ts
@@ -107,6 +107,13 @@ export type TimeFieldFilterProps = TimeRangeFieldDomain & {
// Unique identifier of each field
const FID_KEY = 'name';
+const ROW_EDIT_NOT_IMPLEMENTED =
+ 'In-place row edits (append, upsert, delete) are not implemented for Arrow/DuckDB tables. DuckDB INSERT and Arrow concat are not supported yet. Use addDataToMap with keepExistingConfig to replace the table.';
+
+function warnRowEditsUnavailable(method: string): void {
+ Console.warn(`KeplerTable.${method}: ${ROW_EDIT_NOT_IMPLEMENTED}`);
+}
+
function readFieldValue(
fieldIdx: number,
dc: DataContainerInterface,
@@ -385,6 +392,170 @@ class KeplerTable<F extends Field = Field> {
return this;
}
+ /**
+ * Append column-ordered rows in place. No-op for Arrow/DuckDB (no `append`).
+ * Accessors stay bound: the container instance is not replaced.
+ * @returns false when the payload is rejected and the table is unchanged.
+ */
+ appendRows(rows: any[][]): boolean {
+ if (typeof this.dataContainer.append !== 'function') {
+ warnRowEditsUnavailable('appendRows');
+ return false;
+ }
+ if (!rows.length) {
+ return false;
+ }
+ const columnCount = this.fields.length;
+ if (!columnCount || rows.some(row => !Array.isArray(row) || row.length !== columnCount)) {
+ return false;
+ }
+
+ const start = this.dataContainer.numRows();
+ if (!this.dataContainer.append(rows)) {
+ return false;
+ }
+
+ const nextIndexes =
+ this.allIndexes.length === start
+ ? this.allIndexes.concat(rows.map((_, i) => start + i))
+ : this.dataContainer.getPlainIndex();
+ this.afterRowMutation(nextIndexes);
+ return true;
+ }
+
+ /**
+ * Replace rows that share `keyField` and append the rest. Last incoming row
+ * wins for a repeated key. No-op for Arrow/DuckDB.
+ * @returns false when the payload is rejected and the table is unchanged.
+ */
+ upsertRows(rows: any[][], keyField: string): boolean {
+ if (
+ typeof this.dataContainer.append !== 'function' ||
+ typeof this.dataContainer.replace !== 'function'
+ ) {
+ warnRowEditsUnavailable('upsertRows');
+ return false;
+ }
+ const keyIdx = this.fields.findIndex(field => field.name === keyField);
+ if (keyIdx < 0 || !rows.length) {
+ return false;
+ }
+ const columnCount = this.fields.length;
+ if (rows.some(row => !Array.isArray(row) || row.length !== columnCount)) {
+ return false;
+ }
+
+ const start = this.dataContainer.numRows();
+ const keyToIndex = new Map<unknown, number>();
+ for (let i = 0; i < start; i++) {
+ const key = this.dataContainer.valueAt(i, keyIdx);
+ if (!keyToIndex.has(key)) {
+ keyToIndex.set(key, i);
+ }
+ }
+
+ const lastByKey = new Map<unknown, any[]>();
+ const keyOrder: unknown[] = [];
+ for (const row of rows) {
+ const key = row[keyIdx];
+ if (!lastByKey.has(key)) {
+ keyOrder.push(key);
+ }
+ lastByKey.set(key, row);
+ }
+
+ const toAppend: any[][] = [];
+ for (const key of keyOrder) {
+ const row = lastByKey.get(key);
+ if (!row) {
+ continue;
+ }
+ const existing = keyToIndex.get(key);
+ if (existing !== undefined) {
+ this.dataContainer.replace(existing, row);
+ } else {
+ toAppend.push(row);
+ }
+ }
+
+ if (toAppend.length && !this.dataContainer.append(toAppend)) {
+ return false;
+ }
+
+ const nextIndexes =
+ toAppend.length && this.allIndexes.length === start
+ ? this.allIndexes.concat(toAppend.map((_, i) => start + i))
+ : toAppend.length
+ ? this.dataContainer.getPlainIndex()
+ : this.allIndexes;
+ this.afterRowMutation(nextIndexes);
+ return true;
+ }
+
+ /**
+ * Remove rows by index in place. No-op for Arrow/DuckDB (no `remove`).
+ * @returns false when any index is invalid and the table is unchanged.
+ */
+ removeRows(indexes: number[]): boolean {
+ if (typeof this.dataContainer.remove !== 'function') {
+ warnRowEditsUnavailable('removeRows');
+ return false;
+ }
+ if (!indexes.length) {
+ return false;
+ }
+ if (!this.dataContainer.remove(indexes)) {
+ return false;
+ }
+ this.afterRowMutation(this.dataContainer.getPlainIndex());
+ return true;
+ }
+
+ /**
+ * Indexes of rows whose `fieldName` value is in `values` (first-column-wins
+ * equality via Set). Empty when the field is missing.
+ */
+ findRowIndexesByFieldValues(fieldName: string, values: unknown[]): number[] {
+ const fieldIdx = this.fields.findIndex(field => field.name === fieldName);
+ if (fieldIdx < 0 || !values.length) {
+ return [];
+ }
+ const match = new Set(values);
+ const indexes: number[] = [];
+ const numRows = this.dataContainer.numRows();
+ for (let i = 0; i < numRows; i++) {
+ if (match.has(this.dataContainer.valueAt(i, fieldIdx))) {
+ indexes.push(i);
+ }
+ }
+ return indexes;
+ }
+
+ private afterRowMutation(allIndexes: number[]): void {
+ const fieldsToRebuild = this.fields.filter(f => f.filterProps).map(f => f.name);
+
+ // Drop cached per-row props (mappedValue is indexed by row). Same as a
+ // snapshot update(); getColumnFilterProps rebuilds after indexes are current.
+ this.fields = this.fields.map(f => {
+ const {filterProps: _filterProps, ...rest} = f;
+ return rest;
+ }) as F[];
+
+ this.allIndexes = allIndexes;
+ this.filteredIndex = allIndexes;
+ this.filteredIndexForDomain = allIndexes;
+ this.filteredIndexByLayer = {};
+ this.filteredIdxCPU = undefined;
+ this.filterRecord = undefined;
+ this.filterRecordCPU = undefined;
+ this.changedFilters = undefined;
+ this.sortColumn = undefined;
+ this.sortOrder = null;
+ this.dataRevision += 1;
+
+ fieldsToRebuild.forEach(name => this.getColumnFilterProps(name));
+ }
+
get length() {
return this.dataContainer.numRows();
}
diff --git a/src/utils/src/data-container-interface.ts b/src/utils/src/data-container-interface.ts
index dd7b06bd1..31b254f82 100644
--- a/src/utils/src/data-container-interface.ts
+++ b/src/utils/src/data-container-interface.ts
@@ -19,6 +19,24 @@ export interface DataContainerInterface {
*/
update?(updateData: any[] | arrow.Table): void;
+ /**
+ * Append rows in place. Implemented by row containers; Arrow/DuckDB omit this.
+ * @returns false when the rows are rejected (wrong shape) and the container is unchanged.
+ */
+ append?(rows: any[][]): boolean;
+
+ /**
+ * Replace one row in place. Implemented by row containers; Arrow/DuckDB omit this.
+ * @returns false when the index or row is rejected and the container is unchanged.
+ */
+ replace?(index: number, row: any[]): boolean;
+
+ /**
+ * Remove rows by index in place. Implemented by row containers; Arrow/DuckDB omit this.
+ * @returns false when any index is invalid and the container is unchanged.
+ */
+ remove?(indexes: number[]): boolean;
+
/**
* Returns the number of rows in the data container.
* @returns Number of rows in the data container.
diff --git a/src/utils/src/row-data-container.ts b/src/utils/src/row-data-container.ts
index 5a2805404..f595780f6 100644
--- a/src/utils/src/row-data-container.ts
+++ b/src/utils/src/row-data-container.ts
@@ -63,6 +63,68 @@ export class RowDataContainer implements DataContainerInterface {
this._numColumns = updateData[0]?.length || 0;
}
+ /**
+ * Append rows without copying the existing table. Rejects the whole batch when
+ * any row is not an array of the current column count (empty tables take the
+ * width from the first row).
+ */
+ append(rows: any[][]): boolean {
+ if (!Array.isArray(rows) || !rows.length) {
+ return false;
+ }
+
+ const expected = this._numColumns || (Array.isArray(rows[0]) ? rows[0].length : 0);
+ if (!expected) {
+ return false;
+ }
+ for (let i = 0; i < rows.length; i++) {
+ if (!Array.isArray(rows[i]) || rows[i].length !== expected) {
+ return false;
+ }
+ }
+
+ this._rows.push(...rows);
+ this._numColumns = expected;
+ return true;
+ }
+
+ /**
+ * Overwrite one row. Rejects an out-of-range index or a row of the wrong width.
+ */
+ replace(index: number, row: any[]): boolean {
+ if (!Number.isInteger(index) || index < 0 || index >= this._rows.length) {
+ return false;
+ }
+ if (!Array.isArray(row) || row.length !== this._numColumns) {
+ return false;
+ }
+ this._rows[index] = row;
+ return true;
+ }
+
+ /**
+ * Drop unique valid indexes in one rebuild. Any out-of-range or non-integer
+ * index rejects the whole call so callers can no-op instead of shifting twice.
+ */
+ remove(indexes: number[]): boolean {
+ if (!Array.isArray(indexes) || !indexes.length) {
+ return false;
+ }
+
+ const numRows = this._rows.length;
+ const toRemove = new Set<number>();
+ for (let i = 0; i < indexes.length; i++) {
+ const index = indexes[i];
+ if (!Number.isInteger(index) || index < 0 || index >= numRows) {
+ return false;
+ }
+ toRemove.add(index);
+ }
+
+ this._rows = this._rows.filter((_, rowIndex) => !toRemove.has(rowIndex));
+ return true;
+ }
+
numRows(): number {
return this._rows.length;
}
diff --git a/test/node/reducers/vis-state-test.js b/test/node/reducers/vis-state-test.js
index 0ac57c25d..f3da417ac 100644
--- a/test/node/reducers/vis-state-test.js
+++ b/test/node/reducers/vis-state-test.js
@@ -8228,3 +8228,371 @@ test('VisStateUpdater -> refreshDataset 304 and error', async t => {
t.end();
});
+
+test('VisStateUpdater -> addToDataset appends rows and keeps layers', async t => {
+ const initialData = processCsvData('lat,lng\n37.77,-122.42');
+ const datasets = await createNewDataEntryMock({
+ info: {id: 'live', label: 'live.csv'},
+ data: initialData
+ });
+ const {props} = PointLayer.findDefaultLayerProps(datasets['live']);
+ const pointLayer = new PointLayer({
+ id: 'p1',
+ dataId: 'live',
+ isVisible: true,
+ ...props[0]
+ });
+ const {layerData, layer} = calculateLayerData(
+ pointLayer,
+ {...INITIAL_VIS_STATE, datasets, layers: [pointLayer], layerData: [{}]},
+ undefined
+ );
+ const state = {
+ ...INITIAL_VIS_STATE,
+ datasets,
+ layers: [layer],
+ layerData: [layerData],
+ layerOrder: [layer.id]
+ };
+
+ const revisionBefore = state.datasets.live.dataRevision;
+ const appended = reducer(
+ state,
+ VisStateActions.addToDataset('live', [
+ [10, 20],
+ [30, 40]
+ ])
+ );
+
+ t.equal(appended.datasets.live.dataContainer.numRows(), 3, 'should append both rows');
+ t.deepEqual(appended.datasets.live.allIndexes, [0, 1, 2], 'should extend allIndexes');
+ t.notEqual(appended.datasets.live.dataRevision, revisionBefore, 'should bump dataRevision');
+ t.notEqual(appended.datasets.live, state.datasets.live, 'should copy the table for Redux');
+ t.equal(appended.layers.length, 1, 'should not create extra layers');
+ t.equal(appended.layers[0].id, 'p1', 'should keep the same layer id');
+ t.equal(appended.layerOrder[0], 'p1', 'should keep layerOrder');
+ t.equal(appended.layerData[0].data.length, 3, 'should rebuild layer data for the new rows');
+ t.deepEqual(
+ appended.layerData[0].data[1].position.slice(0, 2),
+ [20, 10],
+ 'should plot the first appended point'
+ );
+
+ const asObjects = reducer(appended, VisStateActions.addToDataset('live', {lat: 11, lng: 22}));
+ t.equal(asObjects.datasets.live.dataContainer.numRows(), 4, 'should append a field-name record');
+ t.equal(asObjects.datasets.live.dataContainer.valueAt(3, 0), 11, 'should map lat by field name');
+ t.equal(asObjects.layers[0].id, 'p1', 'object rows should not restyle layers');
+
+ const missingKeys = reducer(asObjects, VisStateActions.addToDataset('live', {lat: 99}));
+ t.equal(
+ missingKeys.datasets.live.dataContainer.numRows(),
+ 5,
+ 'should accept a partial object row'
+ );
+ t.equal(
+ missingKeys.datasets.live.dataContainer.valueAt(4, 1),
+ null,
+ 'missing object keys should become null'
+ );
+
+ t.end();
+});
+
+test('VisStateUpdater -> addToDataset object rows ignore prototype keys', async t => {
+ const initialData = processCsvData('lat,lng\n1,2');
+ const datasets = await createNewDataEntryMock({
+ info: {id: 'live', label: 'live.csv'},
+ data: initialData
+ });
+ const state = {...INITIAL_VIS_STATE, datasets};
+
+ const inheritedLng = Object.assign(Object.create({lng: 999}), {lat: 50});
+ const missing = reducer(state, VisStateActions.addToDataset('live', inheritedLng));
+ t.equal(missing.datasets.live.dataContainer.numRows(), 2, 'should append the object row');
+ t.equal(missing.datasets.live.dataContainer.valueAt(1, 0), 50, 'own lat should be copied');
+ t.equal(
+ missing.datasets.live.dataContainer.valueAt(1, 1),
+ null,
+ 'inherited lng should become null'
+ );
+
+ const own = reducer(missing, VisStateActions.addToDataset('live', {lat: 7, lng: 8}));
+ t.equal(own.datasets.live.dataContainer.valueAt(2, 1), 8, 'own lng should still be copied');
+
+ t.end();
+});
+
+test('VisStateUpdater -> addToDataset no-ops', async t => {
+ const initialData = processCsvData('lat,lng\n1,2');
+ const datasets = await createNewDataEntryMock({
+ info: {id: 'live', label: 'live.csv'},
+ data: initialData
+ });
+ const state = {...INITIAL_VIS_STATE, datasets};
+
+ t.equal(
+ reducer(state, VisStateActions.addToDataset('missing', [1, 2])),
+ state,
+ 'unknown dataset is a no-op'
+ );
+ t.equal(
+ reducer(state, VisStateActions.addToDataset('live', [1])),
+ state,
+ 'wrong column count is a no-op'
+ );
+ t.equal(
+ reducer(state, VisStateActions.addToDataset('live', [])),
+ state,
+ 'empty row list is a no-op'
+ );
+
+ datasets.live.dataContainer.append = undefined;
+ const warn = sinon.stub(console, 'warn');
+ t.equal(
+ reducer(state, VisStateActions.addToDataset('live', [3, 4])),
+ state,
+ 'tables without append (Arrow/DuckDB) are a no-op'
+ );
+ t.ok(warn.called, 'should warn that in-place row edits are not implemented for Arrow/DuckDB');
+ warn.restore();
+ t.equal(state.datasets.live.dataContainer.numRows(), 1, 'no-op must not mutate rows');
+
+ t.end();
+});
+
+test('VisStateUpdater -> addToDataset reapplies filters', async t => {
+ const initialData = processCsvData('lat,lng,value\n37.77,-122.42,10\n37.78,-122.43,20');
+ const datasets = await createNewDataEntryMock({
+ info: {id: 'live', label: 'live.csv'},
+ data: initialData
+ });
+ const withFilter = reducer({...INITIAL_VIS_STATE, datasets}, VisStateActions.addFilter('live'));
+ const named = reducer(withFilter, VisStateActions.setFilter(0, 'name', 'value'));
+ const filtered = reducer(named, VisStateActions.setFilter(0, 'value', [0, 15]));
+
+ t.deepEqual(
+ filtered.datasets.live.filteredIndexForDomain,
+ [0],
+ 'range filter should hide the out-of-range seed row'
+ );
+
+ const appended = reducer(
+ filtered,
+ VisStateActions.addToDataset('live', [
+ [37.79, -122.44, 12],
+ [37.8, -122.45, 99]
+ ])
+ );
+
+ t.equal(
+ appended.datasets.live.dataContainer.numRows(),
+ 4,
+ 'should still store both appended rows'
+ );
+ t.deepEqual(
+ appended.datasets.live.filteredIndexForDomain,
+ [0, 2],
+ 'should keep in-range rows and hide the out-of-range append'
+ );
+ t.equal(appended.filters.length, 1, 'should keep the existing filter');
+
+ t.end();
+});
+
+test('VisStateUpdater -> removeFromDataset deletes rows and keeps layers', async t => {
+ const initialData = processCsvData('lat,lng\n1,2\n3,4\n5,6\n7,8');
+ const datasets = await createNewDataEntryMock({
+ info: {id: 'live', label: 'live.csv'},
+ data: initialData
+ });
+ const {props} = PointLayer.findDefaultLayerProps(datasets['live']);
+ const pointLayer = new PointLayer({
+ id: 'p1',
+ dataId: 'live',
+ isVisible: true,
+ ...props[0]
+ });
+ const {layerData, layer} = calculateLayerData(
+ pointLayer,
+ {...INITIAL_VIS_STATE, datasets, layers: [pointLayer], layerData: [{}]},
+ undefined
+ );
+ const state = {
+ ...INITIAL_VIS_STATE,
+ datasets,
+ layers: [layer],
+ layerData: [layerData],
+ layerOrder: [layer.id]
+ };
+
+ const removed = reducer(state, VisStateActions.removeFromDataset('live', [0, 2]));
+ t.equal(removed.datasets.live.dataContainer.numRows(), 2, 'should drop both indexes');
+ t.deepEqual(removed.datasets.live.allIndexes, [0, 1], 'should rebuild contiguous indexes');
+ t.equal(removed.datasets.live.dataContainer.valueAt(0, 0), 3, 'should keep the former row 1');
+ t.equal(removed.datasets.live.dataContainer.valueAt(1, 0), 7, 'should keep the former row 3');
+ t.equal(removed.layers.length, 1, 'should not create extra layers');
+ t.equal(removed.layers[0].id, 'p1', 'should keep the same layer id');
+ t.equal(removed.layerData[0].data.length, 2, 'should rebuild layer data');
+
+ const oneIndex = reducer(removed, VisStateActions.removeFromDataset('live', 0));
+ t.equal(oneIndex.datasets.live.dataContainer.numRows(), 1, 'should accept a single index');
+ t.equal(
+ oneIndex.datasets.live.dataContainer.valueAt(0, 0),
+ 7,
+ 'should keep the last remaining row'
+ );
+
+ t.equal(
+ reducer(oneIndex, VisStateActions.removeFromDataset('missing', 0)),
+ oneIndex,
+ 'unknown dataset is a no-op'
+ );
+ t.equal(
+ reducer(oneIndex, VisStateActions.removeFromDataset('live', 99)),
+ oneIndex,
+ 'out-of-range index is a no-op'
+ );
+
+ oneIndex.datasets.live.dataContainer.remove = undefined;
+ const warn = sinon.stub(console, 'warn');
+ t.equal(
+ reducer(oneIndex, VisStateActions.removeFromDataset('live', 0)),
+ oneIndex,
+ 'tables without remove (Arrow/DuckDB) are a no-op'
+ );
+ t.ok(warn.called, 'should warn that in-place deletes are not implemented for Arrow/DuckDB');
+ warn.restore();
+
+ t.end();
+});
+
+test('VisStateUpdater -> addToDataset rebuilds timestamp mappedValue', async t => {
+ const initialData = processCsvData(
+ 'lat,lng,ts\n37.77,-122.42,2016-09-17 00:09:55\n37.78,-122.43,2016-09-17 00:10:55'
+ );
+ const datasets = await createNewDataEntryMock({
+ info: {id: 'live', label: 'live.csv'},
+ data: initialData
+ });
+ const withFilter = reducer({...INITIAL_VIS_STATE, datasets}, VisStateActions.addFilter('live'));
+ const filtered = reducer(withFilter, VisStateActions.setFilter(0, 'name', 'ts'));
+ const tsField = filtered.datasets.live.fields.find(f => f.name === 'ts');
+ t.equal(
+ tsField.filterProps.mappedValue.length,
+ 2,
+ 'time filter should cache mappedValue per row'
+ );
+
+ const appended = reducer(
+ filtered,
+ VisStateActions.addToDataset('live', [37.79, -122.44, '2016-09-17 00:11:55'])
+ );
+ const nextTs = appended.datasets.live.fields.find(f => f.name === 'ts');
+ t.equal(
+ nextTs.filterProps.mappedValue.length,
+ 3,
+ 'mappedValue should cover the appended timestamp row'
+ );
+ t.ok(
+ nextTs.filterProps.mappedValue[2] > nextTs.filterProps.mappedValue[1],
+ 'appended timestamp should parse into mappedValue'
+ );
+
+ const removed = reducer(appended, VisStateActions.removeFromDataset('live', 0));
+ const afterRemove = removed.datasets.live.fields.find(f => f.name === 'ts');
+ t.equal(
+ afterRemove.filterProps.mappedValue.length,
+ 2,
+ 'mappedValue should shrink after a row is removed'
+ );
+
+ t.end();
+});
+
+test('VisStateUpdater -> addToDataset upserts by key', async t => {
+ const initialData = processCsvData('id,lat,lng\na,1,2\nb,3,4');
+ const datasets = await createNewDataEntryMock({
+ info: {id: 'live', label: 'live.csv'},
+ data: initialData
+ });
+ const state = {...INITIAL_VIS_STATE, datasets};
+
+ const upserted = reducer(
+ state,
+ VisStateActions.addToDataset(
+ 'live',
+ [
+ {id: 'b', lat: 30, lng: 40},
+ {id: 'c', lat: 5, lng: 6},
+ {id: 'b', lat: 33, lng: 44}
+ ],
+ {upsertBy: 'id'}
+ )
+ );
+
+ t.equal(upserted.datasets.live.dataContainer.numRows(), 3, 'should append only the new key');
+ t.equal(upserted.datasets.live.dataContainer.valueAt(0, 0), 'a', 'should keep unmatched rows');
+ t.equal(
+ upserted.datasets.live.dataContainer.valueAt(1, 1),
+ 33,
+ 'should replace the matching key with the last incoming row'
+ );
+ t.equal(upserted.datasets.live.dataContainer.valueAt(2, 0), 'c', 'should append the new key');
+ t.equal(upserted.layers.length, 0, 'upsert should not create layers');
+
+ t.end();
+});
+
+test('VisStateUpdater -> removeFromDataset by field values', async t => {
+ const initialData = processCsvData('id,lat,lng\na,1,2\nb,3,4\nc,5,6\nb,7,8');
+ const datasets = await createNewDataEntryMock({
+ info: {id: 'live', label: 'live.csv'},
+ data: initialData
+ });
+ const state = {...INITIAL_VIS_STATE, datasets};
+
+ const removed = reducer(
+ state,
+ VisStateActions.removeFromDataset('live', {field: 'id', values: ['b', 'missing']})
+ );
+ t.equal(removed.datasets.live.dataContainer.numRows(), 2, 'should drop every row with that id');
+ t.equal(removed.datasets.live.dataContainer.valueAt(0, 0), 'a', 'should keep unmatched ids');
+ t.equal(
+ removed.datasets.live.dataContainer.valueAt(1, 0),
+ 'c',
+ 'should keep later unmatched ids'
+ );
+
+ const none = reducer(
+ removed,
+ VisStateActions.removeFromDataset('live', {field: 'id', values: 'zzz'})
+ );
+ t.equal(none, removed, 'unknown values are a no-op');
+
+ t.end();
+});
+
+test('VisStateUpdater -> addToDataset and removeFromDataset clear table sort', async t => {
+ const initialData = processCsvData('lat,lng\n1,2\n3,4\n5,6');
+ const datasets = await createNewDataEntryMock({
+ info: {id: 'live', label: 'live.csv'},
+ data: initialData
+ });
+ const unsorted = {...INITIAL_VIS_STATE, datasets};
+ const sorted = reducer(unsorted, VisStateActions.sortTableColumn('live', 'lat'));
+ t.ok(sorted.datasets.live.sortOrder, 'should have a sort permutation before mutate');
+ t.ok(sorted.datasets.live.sortColumn, 'should have sortColumn before mutate');
+
+ const appended = reducer(sorted, VisStateActions.addToDataset('live', [7, 8]));
+ t.equal(appended.datasets.live.sortOrder, null, 'append should drop sortOrder');
+ t.equal(appended.datasets.live.sortColumn, undefined, 'append should drop sortColumn');
+
+ const sortedAgain = reducer(appended, VisStateActions.sortTableColumn('live', 'lat'));
+ t.ok(sortedAgain.datasets.live.sortOrder, 'should be able to sort again after append');
+
+ const removed = reducer(sortedAgain, VisStateActions.removeFromDataset('live', 0));
+ t.equal(removed.datasets.live.sortOrder, null, 'remove should drop sortOrder');
+ t.equal(removed.datasets.live.sortColumn, undefined, 'remove should drop sortColumn');
+
+ t.end();
+});
diff --git a/test/node/utils/data-container-test.js b/test/node/utils/data-container-test.js
index f50c8a454..78ccca7f6 100644
--- a/test/node/utils/data-container-test.js
+++ b/test/node/utils/data-container-test.js
@@ -86,6 +86,71 @@ test('RowDataContainer.update replaces rows', t => {
t.end();
});
+test('RowDataContainer.append and remove', t => {
+ const dc = createDataContainer([
+ [10, 20],
+ [30, 40],
+ [50, 60]
+ ]);
+
+ t.equal(
+ dc.append([
+ [70, 80],
+ [90, 100]
+ ]),
+ true,
+ 'should append matching rows'
+ );
+ t.equal(dc.numRows(), 5, 'should grow without replacing existing rows');
+ t.equal(dc.valueAt(0, 0), 10, 'should keep the original first row');
+ t.equal(dc.valueAt(4, 1), 100, 'should read the last appended row');
+ t.deepEqual(dc.getPlainIndex(), [0, 1, 2, 3, 4], 'should expose contiguous indexes');
+
+ t.equal(dc.append([[1]]), false, 'should reject a wrong column count');
+ t.equal(dc.append([]), false, 'should reject an empty batch');
+ t.equal(dc.numRows(), 5, 'should leave rows unchanged after a rejected append');
+
+ t.equal(dc.replace(0, [11, 22]), true, 'should overwrite a row in place');
+ t.deepEqual(dc.rowAsArray(0), [11, 22], 'should read the replaced row');
+ t.equal(dc.replace(99, [1, 2]), false, 'should reject a replace with an out-of-range index');
+ t.equal(dc.replace(0, [1]), false, 'should reject a replace with the wrong column count');
+ t.deepEqual(
+ dc.rowAsArray(0),
+ [11, 22],
+ 'should leave the row unchanged after a rejected replace'
+ );
+
+ t.equal(dc.remove([1, 1, 3]), true, 'should drop unique valid indexes in one rebuild');
+ t.deepEqual(
+ dc.flattenData(),
+ [
+ [11, 22],
+ [50, 60],
+ [90, 100]
+ ],
+ 'should keep the remaining rows in order'
+ );
+ t.deepEqual(dc.getPlainIndex(), [0, 1, 2], 'should rebuild indexes after remove');
+
+ t.equal(dc.remove([99]), false, 'should reject an out-of-range index');
+ t.equal(dc.remove([-1]), false, 'should reject a negative index');
+ t.equal(dc.numRows(), 3, 'should leave rows unchanged after a rejected remove');
+
+ dc.update([[1, 2]]);
+ t.equal(dc.numRows(), 1, 'update should still replace after append/remove');
+ t.equal(dc.valueAt(0, 0), 1, 'update should write the new snapshot');
+
+ const empty = createDataContainer([]);
+ t.equal(
+ empty.append([[3, 4]]),
+ true,
+ 'should set column count from the first row on an empty table'
+ );
+ t.equal(empty.numRows(), 1, 'empty table should accept the first append');
+ t.equal(empty.numColumns(), 2, 'empty table should take width from the first row');
+ t.end();
+});
+
test('IndexedDataContainer', t => {
const dc = createIndexedDataContainer(createDataContainer(data), indices);