Kepler.gl is a powerful open source geospatial analysis tool for large-scale data sets.
feat: addToDataset / removeFromDataset for in-place row edits (#176) (#3684)
Igor Dykhtabcf2be722e9b
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 4095bc454..8887faa66 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -15,6 +15,10 @@ Each version should:Ref: http://keepachangelog.com/en/0.3.0/-->+## Unreleased++- feat: `addToDataset` / `removeFromDataset` to append, upsert, and delete rows in place without breaking layers (#176). Row tables only; Arrow/DuckDB warn and no-op until INSERT/concat exist.+## [3.3.0-alpha.8] - Aug 26 2026- af5d8173 chore: pin patched transitive deps via Yarn resolutions (#3654)diff --git a/docs/api-reference/actions/actions.md b/docs/api-reference/actions/actions.mdindex 629ac9170..117b1cb1f 100644--- a/docs/api-reference/actions/actions.md+++ b/docs/api-reference/actions/actions.md@@ -26,6 +26,7 @@- [visStateActions][42]- [addFilter][43]- [addLayer][45]+ - [addToDataset](#addtodataset)- [applyCPUFilter][47]- [enlargeFilter][49]- [interactionConfigChange][51]@@ -41,6 +42,7 @@- [onMapClick][71]- [onMouseMove][72]- [removeDataset][74]+ - [removeFromDataset](#removefromdataset)- [removeFilter][76]- [removeLayer][78]- [reorderLayer][80]@@ -524,6 +526,23 @@ Add a new layerReturns **{type: ActionTypes.ADD_LAYER, props: props}**+### addToDataset++Append or upsert rows on an existing in-memory row dataset without `addDataToMap`. Keeps layer identity and style. Pass `options.upsertBy` to replace rows that share that key and append the rest.++Not implemented for Arrow or DuckDB tables (no INSERT / concat yet); those calls warn and leave the table unchanged. Use `addDataToMap` with `keepExistingConfig` for a full replace.++- **ActionTypes**: `ActionTypes.ADD_TO_DATASET`+- **Updaters**: `visStateUpdaters.addToDatasetUpdater`++**Parameters**++- `dataId` **[string][162]** dataset id+- `rows` one row or an array of rows (column-ordered arrays, or objects keyed by field name)+- `options` **[Object][164]** (optional) `{upsertBy: string}` unique key field++Returns **{type: ActionTypes.ADD_TO_DATASET, dataId, rows, options}**+### applyCPUFilterTrigger CPU filter of selected dataset@@ -723,6 +742,22 @@ Remove a dataset and all layers, filters, tooltip configs that based on itReturns **{type: ActionTypes.REMOVE_DATASET, key: key}**+### removeFromDataset++Delete rows from an existing in-memory row dataset without restyling layers. The second argument is either row indexes or `{field, values}` to match a column (for example an id).++Not implemented for Arrow or DuckDB tables; those calls warn and leave the table unchanged.++- **ActionTypes**: `ActionTypes.REMOVE_FROM_DATASET`+- **Updaters**: `visStateUpdaters.removeFromDatasetUpdater`++**Parameters**++- `dataId` **[string][162]** dataset id+- `rowIndexesOrMatcher` one index, an array of indexes, or `{field, values}`++Returns **{type: ActionTypes.REMOVE_FROM_DATASET, dataId, rowIndexes?, byField?}**+### removeFilterRemove a filter from `visState.filters`, once a filter is removed, data will be re-filtered and layer will be updateddiff --git a/docs/api-reference/reducers/vis-state.md b/docs/api-reference/reducers/vis-state.mdindex 1996af724..365c0cd48 100644--- a/docs/api-reference/reducers/vis-state.md+++ b/docs/api-reference/reducers/vis-state.md@@ -5,6 +5,7 @@- [visStateUpdaters](#visstateupdaters)- [addFilterUpdater](#addfilterupdater)- [addLayerUpdater](#addlayerupdater)+ - [addToDatasetUpdater](#addtodatasetupdater)- [applyCPUFilterUpdater](#applycpufilterupdater)- [enlargeFilterUpdater](#enlargefilterupdater)- [INITIAL\_VIS\_STATE](#initial_vis_state)@@ -20,6 +21,7 @@- [mapClickUpdater](#mapclickupdater)- [receiveMapConfigUpdater](#receivemapconfigupdater)- [removeDatasetUpdater](#removedatasetupdater)+ - [removeFromDatasetUpdater](#removefromdatasetupdater)- [removeFilterUpdater](#removefilterupdater)- [removeLayerUpdater](#removelayerupdater)- [reorderLayerUpdater](#reorderlayerupdater)@@ -103,6 +105,22 @@ Add a new layerReturns **[Object][69]** nextState+### addToDatasetUpdater++Append or upsert rows on an existing in-memory row dataset. Keeps layers and re-runs filters. Arrow/DuckDB tables warn and are left unchanged (INSERT / concat not implemented).++- **Action**: [`addToDataset`](../actions/actions.md#addtodataset)++**Parameters**++- `state` **[Object][69]** `visState`+- `action` **[Object][69]** action+ - `action.dataId` **[string][70]** dataset id+ - `action.rows` rows to append or upsert+ - `action.options.upsertBy` **[string][70]** (optional) unique key field++Returns **[Object][69]** nextState+### applyCPUFilterUpdaterWhen select dataset for export, apply cpu filter to selected dataset@@ -318,6 +336,22 @@ Remove a dataset and all layers, filters, tooltip configs that based on itReturns **[Object][69]** nextState+### removeFromDatasetUpdater++Delete rows by index or by field values from an existing in-memory row dataset. Keeps layers. Arrow/DuckDB tables warn and are left unchanged.++- **Action**: [`removeFromDataset`](../actions/actions.md#removefromdataset)++**Parameters**++- `state` **[Object][69]** `visState`+- `action` **[Object][69]** action+ - `action.dataId` **[string][70]** dataset id+ - `action.rowIndexes` row indexes (optional)+ - `action.byField` `{field, values}` matcher (optional)++Returns **[Object][69]** nextState+### removeFilterUpdaterRemove a filterdiff --git a/examples/README.md b/examples/README.mdindex f6c4cee83..08f970a23 100644--- a/examples/README.md+++ b/examples/README.md@@ -52,7 +52,7 @@ yarn startRead `visState.clicked` from Redux and show a GeoJSON feature's `shapeName` in a host-app sidebar outside the map.- ### [Live Data][live-data]- Remotely hosted CSV of points orbiting San Francisco on a 2-minute loop. Kepler.gl hydrates the URL on load and polls it every 300 ms so points move on the map.+ Remotely hosted CSV of points orbiting San Francisco on a 2-minute loop. Toggle **Poll URL** (HTTP snapshot replace every 300 ms) and **Host rows** (`addToDataset` / `removeFromDataset`, including upsert and delete by `id`, without rebuilding the point layer).- ### [Custom Layer][custom-layer]Add a custom deck.gl layer (`ContourLayer`) to kepler.gl's layer type selector, so it can be picked from the dropdown, configured with dataset columns, and rendered on the map.diff --git a/examples/live-data/README.md b/examples/live-data/README.mdindex 0223a174e..309f6f9e8 100644--- a/examples/live-data/README.md+++ b/examples/live-data/README.md@@ -1,13 +1,23 @@# Live remotely hosted dataLoads a CORS CSV of points orbiting San Francisco as a remotely hosted dataset-and **polls it every 1 second**. Positions are computed from wall-clock time:-each point completes one full loop in **2 minutes**. A fetch returns the-positions at that moment (there is no server-side snapshot timer).--This is a manual test harness for snapshot refresh (`refreshDataset` /-`metadata.refreshIntervalMs`) on `DatasetType.EXTERNALLY_HOSTED`. It bundles-local monorepo `src/` (not a published npm package).+and **polls it every 300 ms**. The right sidebar switches between two host-app+update styles:++- **Poll URL** — HTTP snapshot replace (`refreshDataset` /+ `metadata.refreshIntervalMs`) on `DatasetType.EXTERNALLY_HOSTED`. Positions+ come from wall-clock time; one full loop takes **2 minutes**.+- **Host rows** — pauses the poll, then the example dispatches `addToDataset` /+ `removeFromDataset` so rows change in place without tearing down the point+ layer. Switching back to Poll URL resumes fetching and **replaces** the table+ (injected rows disappear).++This bundles local monorepo `src/` (not a published npm package). Host-row+edits only work for in-memory CSV (`RowDataContainer`), which this example+uses. Arrow and DuckDB tables log a warning and leave the data unchanged+(INSERT / concat are not implemented yet). The **Keyed by id** controls+exercise `addToDataset(id, rows, {upsertBy: 'id'})` and+`removeFromDataset(id, {field: 'id', values})`.## Pre-requirements@@ -28,14 +38,27 @@ run next to `yarn start` on 8080). The CSV server listens at## What to look for+### Poll URL+- Three points on three rings around San Francisco. Adjacent rings move in- opposite directions. One full orbit takes 2 minutes.-- The right sidebar **orbit** percent advances with each poll (`progress` /+ opposite directions. Color is `ring`. One full orbit takes 2 minutes.+- The sidebar **orbit** percent advances with each poll (`progress` /`orbit_s` in the table).-- Layers panel → one **Point** layer (trip is not auto-created). Refresh is- set on the dataset row; the reload icon samples the current time immediately.-- Field names stay the same, so the point layer is kept (not rebuilt from- scratch).+- Layers panel → one **Point** layer. Refresh is set on the dataset row; the+ reload icon samples the current time immediately.++### Host rows++- Orbit **freezes** (interval set to 0).+- **Add 1 / Add 5** drop points on outer rings (4–6), same layer id.+- **Remove last / Remove random** delete by row index.+- **Keyed by id** — **Insert/Move tracker** upserts `track-01` (`upsertBy: 'id'`).+ First click adds a ring-7 point; later clicks move that same row (ids list+ and row count stay put). **Remove tracker** / **Remove veh-01** /+ **Remove last host** delete with `{field: 'id', values}`.+- **Auto trail** appends a point every 800ms and drops the oldest host row+ after 20 so the cloud stays bounded.+- Switch back to Poll URL: next CSV fetch restores the 3 orbiting vehicles.## CSV server only (Add Data URL)@@ -52,6 +75,6 @@ URL**http://localhost:4010/vehicles.csv```-Fetch, then set **Refresh** to 1s (or Custom) on the dataset in the Layers panel.+Fetch, then set **Refresh** to Custom (0.3s) on the dataset in the Layers panel.[yarn-install]: https://yarnpkg.com/getting-started/installdiff --git a/examples/live-data/src/app.tsx b/examples/live-data/src/app.tsxindex 80d23e2b7..dd7eaa6f4 100644--- a/examples/live-data/src/app.tsx+++ b/examples/live-data/src/app.tsx@@ -2,24 +2,42 @@// Copyright contributors to the kepler.gl projectimport * as React from 'react';-import {useState, useEffect} from 'react';+import {useCallback, useEffect, useRef, useState} from 'react';import ReactDOM from 'react-dom/client';import {Provider, useDispatch, useSelector} from 'react-redux';import {applyMiddleware, combineReducers, compose, createStore} from 'redux';import keplerGlReducer, {enhanceReduxMiddleware} from '@kepler.gl/reducers';import KeplerGl from '@kepler.gl/components';-import {addDataToMap, wrapTo} from '@kepler.gl/actions';+import {+ addDataToMap,+ addToDataset,+ refreshDataset,+ removeFromDataset,+ updateDatasetProps,+ wrapTo+} from '@kepler.gl/actions';import {DatasetType} from '@kepler.gl/constants';import {initApplicationConfig} from '@kepler.gl/utils';initApplicationConfig({enableAnnotations: false});const MAP_ID = 'map';-const SIDEBAR_WIDTH = 280;+const SIDEBAR_WIDTH = 300;const DATASET_ID = 'live-vehicles';const LIVE_DATA_URL = process.env.LIVE_DATA_URL || 'http://localhost:4010/vehicles.csv';const REFRESH_INTERVAL_MS = 300;+const AUTO_INTERVAL_MS = 800;+const MAX_HOST_ROWS = 20;+const TRACKER_ID = 'track-01';+const POLL_VEHICLE_ID = 'veh-01';++const SF = {lat: 37.7749, lng: -122.4194};+const COS_LAT = Math.cos((SF.lat * Math.PI) / 180);+const KM_PER_DEG_LAT = 111.32;+const GOLDEN_ANGLE = Math.PI * (3 - Math.sqrt(5));++type LiveMode = 'poll' | 'host';type KeplerRootState = {keplerGl: {@@ -47,6 +65,8 @@ type KeplerRootState = {};};+type LiveDataset = KeplerRootState['keplerGl'][string]['visState']['datasets'][string];+const reducers = combineReducers({keplerGl: keplerGlReducer.initialState({mapStyle: {@@ -64,10 +84,7 @@ const enhancers = applyMiddleware(...middlewares);const composeEnhancers = (window as any).__REDUX_DEVTOOLS_EXTENSION_COMPOSE__ || compose;const store = createStore(reducers, {}, composeEnhancers(enhancers));-function fieldValue(- dataset: KeplerRootState['keplerGl'][string]['visState']['datasets'][string] | undefined,- name: string-) {+function fieldValue(dataset: LiveDataset | undefined, name: string) {const col = dataset?.fields?.findIndex(field => field.name === name) ?? -1;if (col < 0 || !dataset?.dataContainer?.numRows()) {return undefined;@@ -75,6 +92,50 @@ function fieldValue(return dataset.dataContainer.valueAt(0, col);}+function columnIndex(dataset: LiveDataset | undefined, name: string) {+ return dataset?.fields?.findIndex(field => field.name === name) ?? -1;+}++function rowIds(dataset: LiveDataset | undefined): string[] {+ const idCol = columnIndex(dataset, 'id');+ const rows = dataset?.dataContainer?.numRows?.() ?? 0;+ if (idCol < 0 || !rows) {+ return [];+ }+ const ids: string[] = [];+ for (let i = 0; i < rows; i++) {+ ids.push(String(dataset?.dataContainer?.valueAt(i, idCol) ?? ''));+ }+ return ids;+}++function hostRowIndexes(dataset: LiveDataset | undefined): number[] {+ return rowIds(dataset).reduce<number[]>((indexes, id, index) => {+ if (id.startsWith('host-')) {+ indexes.push(index);+ }+ return indexes;+ }, []);+}++function makeHostVehicle(seq: number, options?: {id?: string; ring?: number}) {+ const angle = seq * GOLDEN_ANGLE;+ const radiusKm = 5.2 + (seq % 8) * 0.35;+ const lat = SF.lat + (radiusKm / KM_PER_DEG_LAT) * Math.sin(angle);+ const lng = SF.lng + (radiusKm / (KM_PER_DEG_LAT * COS_LAT)) * Math.cos(angle);+ return {+ id: options?.id ?? `host-${String(seq).padStart(3, '0')}`,+ lat,+ lng,+ heading: ((angle * 180) / Math.PI + 360) % 360,+ speed: 0,+ ring: options?.ring ?? 4 + (seq % 3),+ progress: 0,+ orbit_s: 0,+ updated_at: new Date().toISOString()+ };+}+function useWindowSize() {const [size, setSize] = useState({width: window.innerWidth, height: window.innerHeight});useEffect(() => {@@ -98,10 +159,65 @@ const sidebarStyle: React.CSSProperties = {flexShrink: 0};-function LiveDataSidebar() {- const dataset = useSelector(- (state: KeplerRootState) => state.keplerGl[MAP_ID]?.visState?.datasets?.[DATASET_ID]+const buttonStyle: React.CSSProperties = {+ padding: '7px 10px',+ border: '1px solid #4a5160',+ borderRadius: 4,+ background: '#3a3f4b',+ color: '#f7f7f7',+ cursor: 'pointer',+ fontSize: 13+};++function ModeToggle({+ mode,+ hostEnabled,+ onChange+}: {+ mode: LiveMode;+ hostEnabled: boolean;+ onChange: (next: LiveMode) => void;+}) {+ return (+ <div+ style={{+ display: 'flex',+ marginBottom: 14,+ border: '1px solid #4a5160',+ borderRadius: 6,+ overflow: 'hidden'+ }}+ >+ {(+ [+ ['poll', 'Poll URL', true],+ ['host', 'Host rows', hostEnabled]+ ] as const+ ).map(([value, label, enabled]) => (+ <button+ key={value}+ type="button"+ disabled={!enabled}+ onClick={() => onChange(value)}+ style={{+ flex: 1,+ padding: '8px 10px',+ border: 0,+ cursor: enabled ? 'pointer' : 'default',+ background: mode === value ? '#6a7485' : 'transparent',+ color: enabled ? '#f7f7f7' : '#6a7485',+ fontWeight: mode === value ? 600 : 400,+ fontSize: 13+ }}+ >+ {label}+ </button>+ ))}+ </div>);+}++function PollSidebar({dataset}: {dataset: LiveDataset | undefined}) {const progressAlongOrbit = fieldValue(dataset, 'progress');const orbitS = fieldValue(dataset, 'orbit_s');const updatedAt = fieldValue(dataset, 'updated_at');@@ -110,12 +226,11 @@ function LiveDataSidebar() {const progress = dataset?.metadata?.refreshProgress;return (- <aside style={sidebarStyle}>- <h2 style={{margin: '0 0 8px', fontSize: 16}}>Live orbit</h2>+ <><p style={{margin: '0 0 12px', color: '#c3c9d5'}}>- CSV at <code>{LIVE_DATA_URL}</code> returns the current positions of 3 points circling San- Francisco (one full loop every 2 min). Kepler polls that URL once a second and replaces the- table without tearing down the point layer.+ CSV at <code>{LIVE_DATA_URL}</code> returns 3 points circling San Francisco (one loop every+ 2 min). Kepler polls that URL and <strong>replaces the whole table</strong> without tearing+ down the point layer.</p><div style={{display: 'grid', gap: 8}}><div style={{display: 'flex', justifyContent: 'space-between', gap: 12}}>@@ -152,9 +267,280 @@ function LiveDataSidebar() {<p style={{margin: '12px 0 0', color: '#f19c99'}}>{dataset.metadata.refreshError}</p>) : null}<p style={{margin: '16px 0 0', color: '#c3c9d5'}}>- In the Layers panel, dataset refresh is set to 1s. Click the reload icon to sample the orbit- immediately.+ Layers panel: dataset refresh is 300ms. Click reload to sample the orbit immediately.+ Switching to <strong>Host rows</strong> pauses the poll so you can append or delete without+ the next snapshot wiping your edits.</p>+ </>+ );+}++function HostSidebar({+ dataset,+ autoTrail,+ keyedNote,+ onAutoTrail,+ onAdd,+ onRemoveLast,+ onRemoveRandom,+ onUpsertTracker,+ onRemoveTracker,+ onRemovePollVehicle,+ onRemoveLastHostById+}: {+ dataset: LiveDataset | undefined;+ autoTrail: boolean;+ keyedNote: string;+ onAutoTrail: (next: boolean) => void;+ onAdd: (count: number) => void;+ onRemoveLast: () => void;+ onRemoveRandom: () => void;+ onUpsertTracker: () => void;+ onRemoveTracker: () => void;+ onRemovePollVehicle: () => void;+ onRemoveLastHostById: () => void;+}) {+ const rows = dataset?.dataContainer?.numRows?.() ?? 0;+ const ids = rowIds(dataset);+ const hostCount = hostRowIndexes(dataset).length;+ const hasTracker = ids.includes(TRACKER_ID);+ const hasPollVehicle = ids.includes(POLL_VEHICLE_ID);+ const hostIds = ids.filter(id => id.startsWith('host-'));+ const lastHostId = hostIds[hostIds.length - 1];++ return (+ <>+ <p style={{margin: '0 0 12px', color: '#c3c9d5'}}>+ Polling is off. The host app dispatches <code>addToDataset</code> /{' '}+ <code>removeFromDataset</code> so rows change in place — same layer id and style. Host+ points land on outer rings (4–6) and color by <code>ring</code>.+ </p>+ <div style={{display: 'grid', gap: 8, marginBottom: 12}}>+ <div style={{display: 'flex', justifyContent: 'space-between', gap: 12}}>+ <span style={{color: '#c3c9d5'}}>rows</span>+ <span>+ {rows} <span style={{color: '#c3c9d5'}}>({hostCount} host)</span>+ </span>+ </div>+ <div+ style={{+ display: 'flex',+ justifyContent: 'space-between',+ gap: 12,+ alignItems: 'flex-start'+ }}+ >+ <span style={{color: '#c3c9d5', flexShrink: 0}}>ids</span>+ <span style={{textAlign: 'right', wordBreak: 'break-word'}}>+ {ids.length ? ids.join(', ') : '—'}+ </span>+ </div>+ </div>+ <div style={{display: 'grid', gridTemplateColumns: '1fr 1fr', gap: 8}}>+ <button type="button" style={buttonStyle} onClick={() => onAdd(1)} disabled={!dataset}>+ Add 1+ </button>+ <button type="button" style={buttonStyle} onClick={() => onAdd(5)} disabled={!dataset}>+ Add 5+ </button>+ <button type="button" style={buttonStyle} onClick={onRemoveLast} disabled={rows < 1}>+ Remove last+ </button>+ <button type="button" style={buttonStyle} onClick={onRemoveRandom} disabled={rows < 1}>+ Remove random+ </button>+ </div>+ <p style={{margin: '16px 0 8px', fontWeight: 600}}>Keyed by id</p>+ <p style={{margin: '0 0 8px', color: '#c3c9d5'}}>+ <code>upsertBy: 'id'</code> keeps the same row. First click inserts{' '}+ <code>{TRACKER_ID}</code> (ring 7); later clicks move it and the row count stays put.+ Removes use <code>{'{field, values}'}</code>, not a row index.+ </p>+ <div style={{display: 'grid', gridTemplateColumns: '1fr 1fr', gap: 8}}>+ <button type="button" style={buttonStyle} onClick={onUpsertTracker} disabled={!dataset}>+ {hasTracker ? 'Move tracker' : 'Insert tracker'}+ </button>+ <button type="button" style={buttonStyle} onClick={onRemoveTracker} disabled={!hasTracker}>+ Remove tracker+ </button>+ <button+ type="button"+ style={buttonStyle}+ onClick={onRemovePollVehicle}+ disabled={!hasPollVehicle}+ >+ Remove {POLL_VEHICLE_ID}+ </button>+ <button+ type="button"+ style={buttonStyle}+ onClick={onRemoveLastHostById}+ disabled={!lastHostId}+ >+ Remove last host+ </button>+ </div>+ <p style={{margin: '8px 0 0', color: hasTracker ? '#9ad0a8' : '#c3c9d5'}}>+ {TRACKER_ID}: {hasTracker ? 'on map' : 'missing'}+ {lastHostId ? ` · last host ${lastHostId}` : ''}+ </p>+ {keyedNote ? <p style={{margin: '6px 0 0', color: '#9ad0a8'}}>{keyedNote}</p> : null}+ <label+ style={{+ display: 'flex',+ alignItems: 'center',+ gap: 8,+ marginTop: 14,+ cursor: 'pointer',+ color: '#c3c9d5'+ }}+ >+ <input+ type="checkbox"+ checked={autoTrail}+ onChange={event => onAutoTrail(event.target.checked)}+ disabled={!dataset}+ />+ Auto trail (add every 800ms, keep {MAX_HOST_ROWS} host rows)+ </label>+ <p style={{margin: '16px 0 0', color: '#c3c9d5'}}>+ Switch back to <strong>Poll URL</strong> to resume HTTP snapshot replace — injected rows+ disappear on the next fetch.+ </p>+ </>+ );+}++function LiveDataSidebar() {+ const dispatch = useDispatch();+ const dataset = useSelector(+ (state: KeplerRootState) => state.keplerGl[MAP_ID]?.visState?.datasets?.[DATASET_ID]+ );+ const [mode, setMode] = useState<LiveMode>('poll');+ const [autoTrail, setAutoTrail] = useState(false);+ const [keyedNote, setKeyedNote] = useState('');+ const hostSeq = useRef(1);+ const trackSeq = useRef(1);+ const datasetRef = useRef(dataset);+ datasetRef.current = dataset;++ const toKepler = useCallback(+ (action: Parameters<typeof wrapTo>[1]) => dispatch(wrapTo(MAP_ID, action)),+ [dispatch]+ );++ const setLiveMode = useCallback(+ (next: LiveMode) => {+ setMode(next);+ if (next === 'poll') {+ setAutoTrail(false);+ setKeyedNote('');+ toKepler(+ updateDatasetProps(DATASET_ID, {metadata: {refreshIntervalMs: REFRESH_INTERVAL_MS}})+ );+ toKepler(refreshDataset(DATASET_ID));+ return;+ }+ toKepler(updateDatasetProps(DATASET_ID, {metadata: {refreshIntervalMs: 0}}));+ },+ [toKepler]+ );++ const addHostRows = useCallback(+ (count: number) => {+ const rows = Array.from({length: count}, () => makeHostVehicle(hostSeq.current++));+ toKepler(addToDataset(DATASET_ID, rows));+ },+ [toKepler]+ );++ const removeLast = useCallback(() => {+ const rows = dataset?.dataContainer?.numRows?.() ?? 0;+ if (rows < 1) {+ return;+ }+ toKepler(removeFromDataset(DATASET_ID, rows - 1));+ }, [dataset, toKepler]);++ const removeRandom = useCallback(() => {+ const rows = dataset?.dataContainer?.numRows?.() ?? 0;+ if (rows < 1) {+ return;+ }+ toKepler(removeFromDataset(DATASET_ID, Math.floor(Math.random() * rows)));+ }, [dataset, toKepler]);++ const upsertTracker = useCallback(() => {+ const ids = rowIds(datasetRef.current);+ const existed = ids.includes(TRACKER_ID);+ const row = makeHostVehicle(trackSeq.current++, {id: TRACKER_ID, ring: 7});+ toKepler(addToDataset(DATASET_ID, row, {upsertBy: 'id'}));+ setKeyedNote(+ existed+ ? `upsert ${TRACKER_ID} (replaced — row count should stay ${ids.length})`+ : `upsert ${TRACKER_ID} (inserted — row count ${ids.length + 1})`+ );+ }, [toKepler]);++ const removeByFieldId = useCallback(+ (id: string) => {+ if (!rowIds(datasetRef.current).includes(id)) {+ return;+ }+ toKepler(removeFromDataset(DATASET_ID, {field: 'id', values: id}));+ setKeyedNote(`removeFromDataset({field: 'id', values: '${id}'})`);+ },+ [toKepler]+ );++ const removeLastHostById = useCallback(() => {+ const hostIds = rowIds(datasetRef.current).filter(id => id.startsWith('host-'));+ const lastHostId = hostIds[hostIds.length - 1];+ if (!lastHostId) {+ return;+ }+ removeByFieldId(lastHostId);+ }, [removeByFieldId]);++ useEffect(() => {+ if (mode !== 'host' || !autoTrail) {+ return undefined;+ }+ const timer = window.setInterval(() => {+ const extras = hostRowIndexes(datasetRef.current);+ if (extras.length >= MAX_HOST_ROWS) {+ toKepler(removeFromDataset(DATASET_ID, extras[0]));+ }+ toKepler(addToDataset(DATASET_ID, makeHostVehicle(hostSeq.current++)));+ }, AUTO_INTERVAL_MS);+ return () => window.clearInterval(timer);+ }, [autoTrail, mode, toKepler]);++ return (+ <aside style={sidebarStyle}>+ <h2 style={{margin: '0 0 8px', fontSize: 16}}>Live data</h2>+ <ModeToggle+ mode={mode}+ hostEnabled={(dataset?.dataContainer?.numRows?.() ?? 0) > 0}+ onChange={setLiveMode}+ />+ {mode === 'poll' ? (+ <PollSidebar dataset={dataset} />+ ) : (+ <HostSidebar+ dataset={dataset}+ autoTrail={autoTrail}+ keyedNote={keyedNote}+ onAutoTrail={setAutoTrail}+ onAdd={addHostRows}+ onRemoveLast={removeLast}+ onRemoveRandom={removeRandom}+ onUpsertTracker={upsertTracker}+ onRemoveTracker={() => removeByFieldId(TRACKER_ID)}+ onRemovePollVehicle={() => removeByFieldId(POLL_VEHICLE_ID)}+ onRemoveLastHostById={removeLastHostById}+ />+ )}</aside>);}@@ -200,6 +586,10 @@ const App = () => {columns: {lat: 'lat', lng: 'lng', altitude: null},isVisible: true,visConfig: {radius: 24, filled: true, opacity: 0.85}+ },+ visualChannels: {+ colorField: {name: 'ring', type: 'integer'},+ colorScale: 'quantize'}}]diff --git a/src/actions/src/action-types.ts b/src/actions/src/action-types.tsindex 2e5999fb5..7285edbfa 100644--- a/src/actions/src/action-types.ts+++ b/src/actions/src/action-types.ts@@ -120,6 +120,8 @@ export const ActionTypes = {REFRESH_DATASET_SUCCESS: `${ACTION_PREFIX}REFRESH_DATASET_SUCCESS`,REFRESH_DATASET_ERROR: `${ACTION_PREFIX}REFRESH_DATASET_ERROR`,REFRESH_DATASET_PROGRESS: `${ACTION_PREFIX}REFRESH_DATASET_PROGRESS`,+ ADD_TO_DATASET: `${ACTION_PREFIX}ADD_TO_DATASET`,+ REMOVE_FROM_DATASET: `${ACTION_PREFIX}REMOVE_FROM_DATASET`,ADD_EFFECT: `${ACTION_PREFIX}ADD_EFFECT`,REORDER_EFFECT: `${ACTION_PREFIX}REORDER_EFFECT`,REMOVE_EFFECT: `${ACTION_PREFIX}REMOVE_EFFECT`,diff --git a/src/actions/src/actions.ts b/src/actions/src/actions.tsindex 7695fc8b2..743cc586d 100644--- a/src/actions/src/actions.ts+++ b/src/actions/src/actions.ts@@ -51,6 +51,7 @@ export type ActionHandlers<T extends {[k: string]: Handler}> = {* @param {boolean} data.options.readOnly `default: false` if `readOnly` is set to `true`* the left setting panel will be hidden* @param {boolean} data.options.keepExistingConfig whether to keep exiting map data and associated layer filter interaction config `default: false`.+ * To append, upsert, or delete rows on an existing in-memory row dataset without replacing it or creating layers, use `addToDataset` / `removeFromDataset`. Arrow and DuckDB tables do not support those actions yet.* @param {Object} data.config this object will contain the full kepler.gl instance configuration {mapState, mapStyle, visState}* @public* @examplediff --git a/src/actions/src/vis-state-actions.ts b/src/actions/src/vis-state-actions.tsindex 9266587e0..80710fbfd 100644--- a/src/actions/src/vis-state-actions.ts+++ b/src/actions/src/vis-state-actions.ts@@ -1031,6 +1031,101 @@ export function refreshDatasetProgress(};}+/**+ * One dataset row: a column-ordered array (same as `ProtoDataset.data.rows`)+ * or a field-name record. Missing keys become `null`.+ */+export type DatasetRow = any[] | Record<string, unknown>;++export type AddToDatasetOptions = {+ /**+ * Field name used as a unique key. Matching rows are replaced in place;+ * keys that are not in the table are appended. Last incoming row wins when+ * the same key appears twice. Row tables only — Arrow/DuckDB no-op with a warning.+ */+ upsertBy?: string;+};++export type AddToDatasetUpdaterAction = {+ dataId: string;+ rows: DatasetRow | DatasetRow[];+ options?: AddToDatasetOptions;+};+/**+ * Append rows to an existing in-memory row dataset without `addDataToMap`.+ * Keeps layer identity and style. Pass `options.upsertBy` to replace rows that+ * share that key and append the rest.+ *+ * Not implemented for Arrow or DuckDB tables (no INSERT / concat yet); those+ * calls warn and leave the table unchanged. Use `addDataToMap` with+ * `keepExistingConfig` for a full replace.+ * @memberof visStateActions+ * @param dataId dataset id+ * @param rows one row or an array of rows (arrays in field order, or objects keyed by field name)+ * @param options optional `{upsertBy}` field name+ * @returns action+ * @public+ */+export function addToDataset(+ dataId: string,+ rows: DatasetRow | DatasetRow[],+ options?: AddToDatasetOptions+): Merge<AddToDatasetUpdaterAction, {type: typeof ActionTypes.ADD_TO_DATASET}> {+ return {+ type: ActionTypes.ADD_TO_DATASET,+ dataId,+ rows,+ options+ };+}++export type RemoveFromDatasetByField = {+ field: string;+ values: unknown | unknown[];+};++export type RemoveFromDatasetUpdaterAction = {+ dataId: string;+ rowIndexes?: number | number[];+ byField?: RemoveFromDatasetByField;+};+/**+ * Delete rows from an existing in-memory row dataset without restyling layers.+ * The second argument is either row indexes or `{field, values}` to match a+ * column (e.g. an id). Out-of-range indexes are a no-op for the whole action.+ *+ * Not implemented for Arrow or DuckDB tables; those calls warn and leave the+ * table unchanged. For a full table replace use `addDataToMap` with+ * `keepExistingConfig`.+ * @memberof visStateActions+ * @param dataId dataset id+ * @param rowIndexesOrMatcher one index, an array of indexes, or `{field, values}`+ * @returns action+ * @public+ */+export function removeFromDataset(+ dataId: string,+ rowIndexesOrMatcher: number | number[] | RemoveFromDatasetByField+): Merge<RemoveFromDatasetUpdaterAction, {type: typeof ActionTypes.REMOVE_FROM_DATASET}> {+ if (+ rowIndexesOrMatcher &&+ typeof rowIndexesOrMatcher === 'object' &&+ !Array.isArray(rowIndexesOrMatcher) &&+ 'field' in rowIndexesOrMatcher+ ) {+ return {+ type: ActionTypes.REMOVE_FROM_DATASET,+ dataId,+ byField: rowIndexesOrMatcher+ };+ }+ return {+ type: ActionTypes.REMOVE_FROM_DATASET,+ dataId,+ rowIndexes: rowIndexesOrMatcher+ };+}+export type SortTableColumnUpdaterAction = {dataId: string;column: string;diff --git a/src/reducers/src/vis-state-updaters.ts b/src/reducers/src/vis-state-updaters.tsindex 31531714a..dc79fa0a0 100644--- a/src/reducers/src/vis-state-updaters.ts+++ b/src/reducers/src/vis-state-updaters.ts@@ -3213,8 +3213,16 @@ export function refreshDatasetSuccessUpdater(return rebindLayersAndFiltersAfterSchemaChange(state, dataId, datasets);}- const filteredDatasets = applyFiltersToDatasets([dataId], datasets, state.filters, state.layers);+ return reapplyFiltersAndLayerData({...state, datasets}, dataId);+}+function reapplyFiltersAndLayerData(state: VisState, dataId: string): VisState {+ const filteredDatasets = applyFiltersToDatasets(+ [dataId],+ state.datasets,+ state.filters,+ state.layers+ );let nextState: VisState = {...state,datasets: filteredDatasets@@ -3223,6 +3231,118 @@ export function refreshDatasetSuccessUpdater(return updateAnimationDomain(nextState);}+function isDatasetRowList(+ rows: VisStateActions.DatasetRow | VisStateActions.DatasetRow[]+): rows is VisStateActions.DatasetRow[] {+ if (!Array.isArray(rows) || !rows.length) {+ return Array.isArray(rows);+ }+ // A single column-ordered row is an array of primitives; a list of rows is+ // an array of arrays or an array of field-name records.+ return Array.isArray(rows[0]) || isPlainObject(rows[0]);+}++function toArrayRow(row: VisStateActions.DatasetRow, fields: Field[]): any[] | null {+ if (Array.isArray(row)) {+ return row.length === fields.length ? row : null;+ }+ if (isPlainObject(row)) {+ const record = row as Record<string, unknown>;+ return fields.map(field =>+ Object.prototype.hasOwnProperty.call(record, field.name) ? record[field.name] : null+ );+ }+ return null;+}++function normalizeDatasetRows(+ rows: VisStateActions.DatasetRow | VisStateActions.DatasetRow[],+ fields: Field[]+): any[][] | null {+ if (!fields.length) {+ return null;+ }+ const list = isDatasetRowList(rows) ? rows : [rows];+ if (!list.length) {+ return null;+ }+ const normalized: any[][] = [];+ for (let i = 0; i < list.length; i++) {+ const arrayRow = toArrayRow(list[i], fields);+ if (!arrayRow) {+ return null;+ }+ normalized.push(arrayRow);+ }+ return normalized;+}++/**+ * Append or upsert rows on an existing in-memory row dataset. Keeps layers and re-runs filters.+ * Arrow/DuckDB tables warn and are left unchanged (INSERT / concat not implemented).+ * @memberof visStateUpdaters+ * @public+ */+export function addToDatasetUpdater(+ state: VisState,+ {dataId, rows, options}: VisStateActions.AddToDatasetUpdaterAction+): VisState {+ const dataset = state.datasets[dataId];+ if (!dataset) {+ return state;+ }++ const normalized = normalizeDatasetRows(rows, dataset.fields);+ if (!normalized) {+ return state;+ }++ const applied = options?.upsertBy+ ? dataset.upsertRows(normalized, options.upsertBy)+ : dataset.appendRows(normalized);+ if (!applied) {+ return state;+ }++ const datasets = {+ ...state.datasets,+ [dataId]: copyTableAndUpdate(dataset, {})+ };+ return reapplyFiltersAndLayerData({...state, datasets}, dataId);+}++/**+ * Delete rows by index or by field values from an existing in-memory row dataset.+ * Keeps layers. Arrow/DuckDB tables warn and are left unchanged.+ * @memberof visStateUpdaters+ * @public+ */+export function removeFromDatasetUpdater(+ state: VisState,+ {dataId, rowIndexes, byField}: VisStateActions.RemoveFromDatasetUpdaterAction+): VisState {+ const dataset = state.datasets[dataId];+ if (!dataset) {+ return state;+ }++ const indexes = byField+ ? dataset.findRowIndexesByFieldValues(byField.field, toArray(byField.values))+ : toArray(rowIndexes ?? []);+ if (!indexes.length) {+ return state;+ }+ if (!dataset.removeRows(indexes)) {+ return state;+ }++ const datasets = {+ ...state.datasets,+ [dataId]: copyTableAndUpdate(dataset, {})+ };+ return reapplyFiltersAndLayerData({...state, datasets}, dataId);+}+/*** Record a failed remote dataset refresh without dropping existing rows.* @memberof visStateUpdatersdiff --git a/src/reducers/src/vis-state.ts b/src/reducers/src/vis-state.tsindex 790d2bfe2..9cc07b9f8 100644--- a/src/reducers/src/vis-state.ts+++ b/src/reducers/src/vis-state.ts@@ -111,6 +111,10 @@ const actionHandler = {[ActionTypes.REFRESH_DATASET_PROGRESS]: visStateUpdaters.refreshDatasetProgressUpdater,+ [ActionTypes.ADD_TO_DATASET]: visStateUpdaters.addToDatasetUpdater,++ [ActionTypes.REMOVE_FROM_DATASET]: visStateUpdaters.removeFromDatasetUpdater,+[ActionTypes.TOGGLE_FILTER_ANIMATION]: visStateUpdaters.toggleFilterAnimationUpdater,[ActionTypes.UPDATE_FILTER_ANIMATION_SPEED]: visStateUpdaters.updateFilterAnimationSpeedUpdater,diff --git a/src/table/src/kepler-table.ts b/src/table/src/kepler-table.tsindex b0653c242..7216428b1 100644--- a/src/table/src/kepler-table.ts+++ b/src/table/src/kepler-table.ts@@ -107,6 +107,13 @@ export type TimeFieldFilterProps = TimeRangeFieldDomain & {// Unique identifier of each fieldconst FID_KEY = 'name';+const ROW_EDIT_NOT_IMPLEMENTED =+ 'In-place row edits (append, upsert, delete) are not implemented for Arrow/DuckDB tables. DuckDB INSERT and Arrow concat are not supported yet. Use addDataToMap with keepExistingConfig to replace the table.';++function warnRowEditsUnavailable(method: string): void {+ Console.warn(`KeplerTable.${method}: ${ROW_EDIT_NOT_IMPLEMENTED}`);+}+function readFieldValue(fieldIdx: number,dc: DataContainerInterface,@@ -385,6 +392,170 @@ class KeplerTable<F extends Field = Field> {return this;}+ /**+ * Append column-ordered rows in place. No-op for Arrow/DuckDB (no `append`).+ * Accessors stay bound: the container instance is not replaced.+ * @returns false when the payload is rejected and the table is unchanged.+ */+ appendRows(rows: any[][]): boolean {+ if (typeof this.dataContainer.append !== 'function') {+ warnRowEditsUnavailable('appendRows');+ return false;+ }+ if (!rows.length) {+ return false;+ }+ const columnCount = this.fields.length;+ if (!columnCount || rows.some(row => !Array.isArray(row) || row.length !== columnCount)) {+ return false;+ }++ const start = this.dataContainer.numRows();+ if (!this.dataContainer.append(rows)) {+ return false;+ }++ const nextIndexes =+ this.allIndexes.length === start+ ? this.allIndexes.concat(rows.map((_, i) => start + i))+ : this.dataContainer.getPlainIndex();+ this.afterRowMutation(nextIndexes);+ return true;+ }++ /**+ * Replace rows that share `keyField` and append the rest. Last incoming row+ * wins for a repeated key. No-op for Arrow/DuckDB.+ * @returns false when the payload is rejected and the table is unchanged.+ */+ upsertRows(rows: any[][], keyField: string): boolean {+ if (+ typeof this.dataContainer.append !== 'function' ||+ typeof this.dataContainer.replace !== 'function'+ ) {+ warnRowEditsUnavailable('upsertRows');+ return false;+ }+ const keyIdx = this.fields.findIndex(field => field.name === keyField);+ if (keyIdx < 0 || !rows.length) {+ return false;+ }+ const columnCount = this.fields.length;+ if (rows.some(row => !Array.isArray(row) || row.length !== columnCount)) {+ return false;+ }++ const start = this.dataContainer.numRows();+ const keyToIndex = new Map<unknown, number>();+ for (let i = 0; i < start; i++) {+ const key = this.dataContainer.valueAt(i, keyIdx);+ if (!keyToIndex.has(key)) {+ keyToIndex.set(key, i);+ }+ }++ const lastByKey = new Map<unknown, any[]>();+ const keyOrder: unknown[] = [];+ for (const row of rows) {+ const key = row[keyIdx];+ if (!lastByKey.has(key)) {+ keyOrder.push(key);+ }+ lastByKey.set(key, row);+ }++ const toAppend: any[][] = [];+ for (const key of keyOrder) {+ const row = lastByKey.get(key);+ if (!row) {+ continue;+ }+ const existing = keyToIndex.get(key);+ if (existing !== undefined) {+ this.dataContainer.replace(existing, row);+ } else {+ toAppend.push(row);+ }+ }++ if (toAppend.length && !this.dataContainer.append(toAppend)) {+ return false;+ }++ const nextIndexes =+ toAppend.length && this.allIndexes.length === start+ ? this.allIndexes.concat(toAppend.map((_, i) => start + i))+ : toAppend.length+ ? this.dataContainer.getPlainIndex()+ : this.allIndexes;+ this.afterRowMutation(nextIndexes);+ return true;+ }++ /**+ * Remove rows by index in place. No-op for Arrow/DuckDB (no `remove`).+ * @returns false when any index is invalid and the table is unchanged.+ */+ removeRows(indexes: number[]): boolean {+ if (typeof this.dataContainer.remove !== 'function') {+ warnRowEditsUnavailable('removeRows');+ return false;+ }+ if (!indexes.length) {+ return false;+ }+ if (!this.dataContainer.remove(indexes)) {+ return false;+ }+ this.afterRowMutation(this.dataContainer.getPlainIndex());+ return true;+ }++ /**+ * Indexes of rows whose `fieldName` value is in `values` (first-column-wins+ * equality via Set). Empty when the field is missing.+ */+ findRowIndexesByFieldValues(fieldName: string, values: unknown[]): number[] {+ const fieldIdx = this.fields.findIndex(field => field.name === fieldName);+ if (fieldIdx < 0 || !values.length) {+ return [];+ }+ const match = new Set(values);+ const indexes: number[] = [];+ const numRows = this.dataContainer.numRows();+ for (let i = 0; i < numRows; i++) {+ if (match.has(this.dataContainer.valueAt(i, fieldIdx))) {+ indexes.push(i);+ }+ }+ return indexes;+ }++ private afterRowMutation(allIndexes: number[]): void {+ const fieldsToRebuild = this.fields.filter(f => f.filterProps).map(f => f.name);++ // Drop cached per-row props (mappedValue is indexed by row). Same as a+ // snapshot update(); getColumnFilterProps rebuilds after indexes are current.+ this.fields = this.fields.map(f => {+ const {filterProps: _filterProps, ...rest} = f;+ return rest;+ }) as F[];++ this.allIndexes = allIndexes;+ this.filteredIndex = allIndexes;+ this.filteredIndexForDomain = allIndexes;+ this.filteredIndexByLayer = {};+ this.filteredIdxCPU = undefined;+ this.filterRecord = undefined;+ this.filterRecordCPU = undefined;+ this.changedFilters = undefined;+ this.sortColumn = undefined;+ this.sortOrder = null;+ this.dataRevision += 1;++ fieldsToRebuild.forEach(name => this.getColumnFilterProps(name));+ }+get length() {return this.dataContainer.numRows();}diff --git a/src/utils/src/data-container-interface.ts b/src/utils/src/data-container-interface.tsindex dd7b06bd1..31b254f82 100644--- a/src/utils/src/data-container-interface.ts+++ b/src/utils/src/data-container-interface.ts@@ -19,6 +19,24 @@ export interface DataContainerInterface {*/update?(updateData: any[] | arrow.Table): void;+ /**+ * Append rows in place. Implemented by row containers; Arrow/DuckDB omit this.+ * @returns false when the rows are rejected (wrong shape) and the container is unchanged.+ */+ append?(rows: any[][]): boolean;++ /**+ * Replace one row in place. Implemented by row containers; Arrow/DuckDB omit this.+ * @returns false when the index or row is rejected and the container is unchanged.+ */+ replace?(index: number, row: any[]): boolean;++ /**+ * Remove rows by index in place. Implemented by row containers; Arrow/DuckDB omit this.+ * @returns false when any index is invalid and the container is unchanged.+ */+ remove?(indexes: number[]): boolean;+/*** Returns the number of rows in the data container.* @returns Number of rows in the data container.diff --git a/src/utils/src/row-data-container.ts b/src/utils/src/row-data-container.tsindex 5a2805404..f595780f6 100644--- a/src/utils/src/row-data-container.ts+++ b/src/utils/src/row-data-container.ts@@ -63,6 +63,68 @@ export class RowDataContainer implements DataContainerInterface {this._numColumns = updateData[0]?.length || 0;}+ /**+ * Append rows without copying the existing table. Rejects the whole batch when+ * any row is not an array of the current column count (empty tables take the+ * width from the first row).+ */+ append(rows: any[][]): boolean {+ if (!Array.isArray(rows) || !rows.length) {+ return false;+ }++ const expected = this._numColumns || (Array.isArray(rows[0]) ? rows[0].length : 0);+ if (!expected) {+ return false;+ }+ for (let i = 0; i < rows.length; i++) {+ if (!Array.isArray(rows[i]) || rows[i].length !== expected) {+ return false;+ }+ }++ this._rows.push(...rows);+ this._numColumns = expected;+ return true;+ }++ /**+ * Overwrite one row. Rejects an out-of-range index or a row of the wrong width.+ */+ replace(index: number, row: any[]): boolean {+ if (!Number.isInteger(index) || index < 0 || index >= this._rows.length) {+ return false;+ }+ if (!Array.isArray(row) || row.length !== this._numColumns) {+ return false;+ }+ this._rows[index] = row;+ return true;+ }++ /**+ * Drop unique valid indexes in one rebuild. Any out-of-range or non-integer+ * index rejects the whole call so callers can no-op instead of shifting twice.+ */+ remove(indexes: number[]): boolean {+ if (!Array.isArray(indexes) || !indexes.length) {+ return false;+ }++ const numRows = this._rows.length;+ const toRemove = new Set<number>();+ for (let i = 0; i < indexes.length; i++) {+ const index = indexes[i];+ if (!Number.isInteger(index) || index < 0 || index >= numRows) {+ return false;+ }+ toRemove.add(index);+ }++ this._rows = this._rows.filter((_, rowIndex) => !toRemove.has(rowIndex));+ return true;+ }+numRows(): number {return this._rows.length;}diff --git a/test/node/reducers/vis-state-test.js b/test/node/reducers/vis-state-test.jsindex 0ac57c25d..f3da417ac 100644--- a/test/node/reducers/vis-state-test.js+++ b/test/node/reducers/vis-state-test.js@@ -8228,3 +8228,371 @@ test('VisStateUpdater -> refreshDataset 304 and error', async t => {t.end();});++test('VisStateUpdater -> addToDataset appends rows and keeps layers', async t => {+ const initialData = processCsvData('lat,lng\n37.77,-122.42');+ const datasets = await createNewDataEntryMock({+ info: {id: 'live', label: 'live.csv'},+ data: initialData+ });+ const {props} = PointLayer.findDefaultLayerProps(datasets['live']);+ const pointLayer = new PointLayer({+ id: 'p1',+ dataId: 'live',+ isVisible: true,+ ...props[0]+ });+ const {layerData, layer} = calculateLayerData(+ pointLayer,+ {...INITIAL_VIS_STATE, datasets, layers: [pointLayer], layerData: [{}]},+ undefined+ );+ const state = {+ ...INITIAL_VIS_STATE,+ datasets,+ layers: [layer],+ layerData: [layerData],+ layerOrder: [layer.id]+ };++ const revisionBefore = state.datasets.live.dataRevision;+ const appended = reducer(+ state,+ VisStateActions.addToDataset('live', [+ [10, 20],+ [30, 40]+ ])+ );++ t.equal(appended.datasets.live.dataContainer.numRows(), 3, 'should append both rows');+ t.deepEqual(appended.datasets.live.allIndexes, [0, 1, 2], 'should extend allIndexes');+ t.notEqual(appended.datasets.live.dataRevision, revisionBefore, 'should bump dataRevision');+ t.notEqual(appended.datasets.live, state.datasets.live, 'should copy the table for Redux');+ t.equal(appended.layers.length, 1, 'should not create extra layers');+ t.equal(appended.layers[0].id, 'p1', 'should keep the same layer id');+ t.equal(appended.layerOrder[0], 'p1', 'should keep layerOrder');+ t.equal(appended.layerData[0].data.length, 3, 'should rebuild layer data for the new rows');+ t.deepEqual(+ appended.layerData[0].data[1].position.slice(0, 2),+ [20, 10],+ 'should plot the first appended point'+ );++ const asObjects = reducer(appended, VisStateActions.addToDataset('live', {lat: 11, lng: 22}));+ t.equal(asObjects.datasets.live.dataContainer.numRows(), 4, 'should append a field-name record');+ t.equal(asObjects.datasets.live.dataContainer.valueAt(3, 0), 11, 'should map lat by field name');+ t.equal(asObjects.layers[0].id, 'p1', 'object rows should not restyle layers');++ const missingKeys = reducer(asObjects, VisStateActions.addToDataset('live', {lat: 99}));+ t.equal(+ missingKeys.datasets.live.dataContainer.numRows(),+ 5,+ 'should accept a partial object row'+ );+ t.equal(+ missingKeys.datasets.live.dataContainer.valueAt(4, 1),+ null,+ 'missing object keys should become null'+ );++ t.end();+});++test('VisStateUpdater -> addToDataset object rows ignore prototype keys', async t => {+ const initialData = processCsvData('lat,lng\n1,2');+ const datasets = await createNewDataEntryMock({+ info: {id: 'live', label: 'live.csv'},+ data: initialData+ });+ const state = {...INITIAL_VIS_STATE, datasets};++ const inheritedLng = Object.assign(Object.create({lng: 999}), {lat: 50});+ const missing = reducer(state, VisStateActions.addToDataset('live', inheritedLng));+ t.equal(missing.datasets.live.dataContainer.numRows(), 2, 'should append the object row');+ t.equal(missing.datasets.live.dataContainer.valueAt(1, 0), 50, 'own lat should be copied');+ t.equal(+ missing.datasets.live.dataContainer.valueAt(1, 1),+ null,+ 'inherited lng should become null'+ );++ const own = reducer(missing, VisStateActions.addToDataset('live', {lat: 7, lng: 8}));+ t.equal(own.datasets.live.dataContainer.valueAt(2, 1), 8, 'own lng should still be copied');++ t.end();+});++test('VisStateUpdater -> addToDataset no-ops', async t => {+ const initialData = processCsvData('lat,lng\n1,2');+ const datasets = await createNewDataEntryMock({+ info: {id: 'live', label: 'live.csv'},+ data: initialData+ });+ const state = {...INITIAL_VIS_STATE, datasets};++ t.equal(+ reducer(state, VisStateActions.addToDataset('missing', [1, 2])),+ state,+ 'unknown dataset is a no-op'+ );+ t.equal(+ reducer(state, VisStateActions.addToDataset('live', [1])),+ state,+ 'wrong column count is a no-op'+ );+ t.equal(+ reducer(state, VisStateActions.addToDataset('live', [])),+ state,+ 'empty row list is a no-op'+ );++ datasets.live.dataContainer.append = undefined;+ const warn = sinon.stub(console, 'warn');+ t.equal(+ reducer(state, VisStateActions.addToDataset('live', [3, 4])),+ state,+ 'tables without append (Arrow/DuckDB) are a no-op'+ );+ t.ok(warn.called, 'should warn that in-place row edits are not implemented for Arrow/DuckDB');+ warn.restore();+ t.equal(state.datasets.live.dataContainer.numRows(), 1, 'no-op must not mutate rows');++ t.end();+});++test('VisStateUpdater -> addToDataset reapplies filters', async t => {+ const initialData = processCsvData('lat,lng,value\n37.77,-122.42,10\n37.78,-122.43,20');+ const datasets = await createNewDataEntryMock({+ info: {id: 'live', label: 'live.csv'},+ data: initialData+ });+ const withFilter = reducer({...INITIAL_VIS_STATE, datasets}, VisStateActions.addFilter('live'));+ const named = reducer(withFilter, VisStateActions.setFilter(0, 'name', 'value'));+ const filtered = reducer(named, VisStateActions.setFilter(0, 'value', [0, 15]));++ t.deepEqual(+ filtered.datasets.live.filteredIndexForDomain,+ [0],+ 'range filter should hide the out-of-range seed row'+ );++ const appended = reducer(+ filtered,+ VisStateActions.addToDataset('live', [+ [37.79, -122.44, 12],+ [37.8, -122.45, 99]+ ])+ );++ t.equal(+ appended.datasets.live.dataContainer.numRows(),+ 4,+ 'should still store both appended rows'+ );+ t.deepEqual(+ appended.datasets.live.filteredIndexForDomain,+ [0, 2],+ 'should keep in-range rows and hide the out-of-range append'+ );+ t.equal(appended.filters.length, 1, 'should keep the existing filter');++ t.end();+});++test('VisStateUpdater -> removeFromDataset deletes rows and keeps layers', async t => {+ const initialData = processCsvData('lat,lng\n1,2\n3,4\n5,6\n7,8');+ const datasets = await createNewDataEntryMock({+ info: {id: 'live', label: 'live.csv'},+ data: initialData+ });+ const {props} = PointLayer.findDefaultLayerProps(datasets['live']);+ const pointLayer = new PointLayer({+ id: 'p1',+ dataId: 'live',+ isVisible: true,+ ...props[0]+ });+ const {layerData, layer} = calculateLayerData(+ pointLayer,+ {...INITIAL_VIS_STATE, datasets, layers: [pointLayer], layerData: [{}]},+ undefined+ );+ const state = {+ ...INITIAL_VIS_STATE,+ datasets,+ layers: [layer],+ layerData: [layerData],+ layerOrder: [layer.id]+ };++ const removed = reducer(state, VisStateActions.removeFromDataset('live', [0, 2]));+ t.equal(removed.datasets.live.dataContainer.numRows(), 2, 'should drop both indexes');+ t.deepEqual(removed.datasets.live.allIndexes, [0, 1], 'should rebuild contiguous indexes');+ t.equal(removed.datasets.live.dataContainer.valueAt(0, 0), 3, 'should keep the former row 1');+ t.equal(removed.datasets.live.dataContainer.valueAt(1, 0), 7, 'should keep the former row 3');+ t.equal(removed.layers.length, 1, 'should not create extra layers');+ t.equal(removed.layers[0].id, 'p1', 'should keep the same layer id');+ t.equal(removed.layerData[0].data.length, 2, 'should rebuild layer data');++ const oneIndex = reducer(removed, VisStateActions.removeFromDataset('live', 0));+ t.equal(oneIndex.datasets.live.dataContainer.numRows(), 1, 'should accept a single index');+ t.equal(+ oneIndex.datasets.live.dataContainer.valueAt(0, 0),+ 7,+ 'should keep the last remaining row'+ );++ t.equal(+ reducer(oneIndex, VisStateActions.removeFromDataset('missing', 0)),+ oneIndex,+ 'unknown dataset is a no-op'+ );+ t.equal(+ reducer(oneIndex, VisStateActions.removeFromDataset('live', 99)),+ oneIndex,+ 'out-of-range index is a no-op'+ );++ oneIndex.datasets.live.dataContainer.remove = undefined;+ const warn = sinon.stub(console, 'warn');+ t.equal(+ reducer(oneIndex, VisStateActions.removeFromDataset('live', 0)),+ oneIndex,+ 'tables without remove (Arrow/DuckDB) are a no-op'+ );+ t.ok(warn.called, 'should warn that in-place deletes are not implemented for Arrow/DuckDB');+ warn.restore();++ t.end();+});++test('VisStateUpdater -> addToDataset rebuilds timestamp mappedValue', async t => {+ const initialData = processCsvData(+ 'lat,lng,ts\n37.77,-122.42,2016-09-17 00:09:55\n37.78,-122.43,2016-09-17 00:10:55'+ );+ const datasets = await createNewDataEntryMock({+ info: {id: 'live', label: 'live.csv'},+ data: initialData+ });+ const withFilter = reducer({...INITIAL_VIS_STATE, datasets}, VisStateActions.addFilter('live'));+ const filtered = reducer(withFilter, VisStateActions.setFilter(0, 'name', 'ts'));+ const tsField = filtered.datasets.live.fields.find(f => f.name === 'ts');+ t.equal(+ tsField.filterProps.mappedValue.length,+ 2,+ 'time filter should cache mappedValue per row'+ );++ const appended = reducer(+ filtered,+ VisStateActions.addToDataset('live', [37.79, -122.44, '2016-09-17 00:11:55'])+ );+ const nextTs = appended.datasets.live.fields.find(f => f.name === 'ts');+ t.equal(+ nextTs.filterProps.mappedValue.length,+ 3,+ 'mappedValue should cover the appended timestamp row'+ );+ t.ok(+ nextTs.filterProps.mappedValue[2] > nextTs.filterProps.mappedValue[1],+ 'appended timestamp should parse into mappedValue'+ );++ const removed = reducer(appended, VisStateActions.removeFromDataset('live', 0));+ const afterRemove = removed.datasets.live.fields.find(f => f.name === 'ts');+ t.equal(+ afterRemove.filterProps.mappedValue.length,+ 2,+ 'mappedValue should shrink after a row is removed'+ );++ t.end();+});++test('VisStateUpdater -> addToDataset upserts by key', async t => {+ const initialData = processCsvData('id,lat,lng\na,1,2\nb,3,4');+ const datasets = await createNewDataEntryMock({+ info: {id: 'live', label: 'live.csv'},+ data: initialData+ });+ const state = {...INITIAL_VIS_STATE, datasets};++ const upserted = reducer(+ state,+ VisStateActions.addToDataset(+ 'live',+ [+ {id: 'b', lat: 30, lng: 40},+ {id: 'c', lat: 5, lng: 6},+ {id: 'b', lat: 33, lng: 44}+ ],+ {upsertBy: 'id'}+ )+ );++ t.equal(upserted.datasets.live.dataContainer.numRows(), 3, 'should append only the new key');+ t.equal(upserted.datasets.live.dataContainer.valueAt(0, 0), 'a', 'should keep unmatched rows');+ t.equal(+ upserted.datasets.live.dataContainer.valueAt(1, 1),+ 33,+ 'should replace the matching key with the last incoming row'+ );+ t.equal(upserted.datasets.live.dataContainer.valueAt(2, 0), 'c', 'should append the new key');+ t.equal(upserted.layers.length, 0, 'upsert should not create layers');++ t.end();+});++test('VisStateUpdater -> removeFromDataset by field values', async t => {+ const initialData = processCsvData('id,lat,lng\na,1,2\nb,3,4\nc,5,6\nb,7,8');+ const datasets = await createNewDataEntryMock({+ info: {id: 'live', label: 'live.csv'},+ data: initialData+ });+ const state = {...INITIAL_VIS_STATE, datasets};++ const removed = reducer(+ state,+ VisStateActions.removeFromDataset('live', {field: 'id', values: ['b', 'missing']})+ );+ t.equal(removed.datasets.live.dataContainer.numRows(), 2, 'should drop every row with that id');+ t.equal(removed.datasets.live.dataContainer.valueAt(0, 0), 'a', 'should keep unmatched ids');+ t.equal(+ removed.datasets.live.dataContainer.valueAt(1, 0),+ 'c',+ 'should keep later unmatched ids'+ );++ const none = reducer(+ removed,+ VisStateActions.removeFromDataset('live', {field: 'id', values: 'zzz'})+ );+ t.equal(none, removed, 'unknown values are a no-op');++ t.end();+});++test('VisStateUpdater -> addToDataset and removeFromDataset clear table sort', async t => {+ const initialData = processCsvData('lat,lng\n1,2\n3,4\n5,6');+ const datasets = await createNewDataEntryMock({+ info: {id: 'live', label: 'live.csv'},+ data: initialData+ });+ const unsorted = {...INITIAL_VIS_STATE, datasets};+ const sorted = reducer(unsorted, VisStateActions.sortTableColumn('live', 'lat'));+ t.ok(sorted.datasets.live.sortOrder, 'should have a sort permutation before mutate');+ t.ok(sorted.datasets.live.sortColumn, 'should have sortColumn before mutate');++ const appended = reducer(sorted, VisStateActions.addToDataset('live', [7, 8]));+ t.equal(appended.datasets.live.sortOrder, null, 'append should drop sortOrder');+ t.equal(appended.datasets.live.sortColumn, undefined, 'append should drop sortColumn');++ const sortedAgain = reducer(appended, VisStateActions.sortTableColumn('live', 'lat'));+ t.ok(sortedAgain.datasets.live.sortOrder, 'should be able to sort again after append');++ const removed = reducer(sortedAgain, VisStateActions.removeFromDataset('live', 0));+ t.equal(removed.datasets.live.sortOrder, null, 'remove should drop sortOrder');+ t.equal(removed.datasets.live.sortColumn, undefined, 'remove should drop sortColumn');++ t.end();+});diff --git a/test/node/utils/data-container-test.js b/test/node/utils/data-container-test.jsindex f50c8a454..78ccca7f6 100644--- a/test/node/utils/data-container-test.js+++ b/test/node/utils/data-container-test.js@@ -86,6 +86,71 @@ test('RowDataContainer.update replaces rows', t => {t.end();});+test('RowDataContainer.append and remove', t => {+ const dc = createDataContainer([+ [10, 20],+ [30, 40],+ [50, 60]+ ]);++ t.equal(+ dc.append([+ [70, 80],+ [90, 100]+ ]),+ true,+ 'should append matching rows'+ );+ t.equal(dc.numRows(), 5, 'should grow without replacing existing rows');+ t.equal(dc.valueAt(0, 0), 10, 'should keep the original first row');+ t.equal(dc.valueAt(4, 1), 100, 'should read the last appended row');+ t.deepEqual(dc.getPlainIndex(), [0, 1, 2, 3, 4], 'should expose contiguous indexes');++ t.equal(dc.append([[1]]), false, 'should reject a wrong column count');+ t.equal(dc.append([]), false, 'should reject an empty batch');+ t.equal(dc.numRows(), 5, 'should leave rows unchanged after a rejected append');++ t.equal(dc.replace(0, [11, 22]), true, 'should overwrite a row in place');+ t.deepEqual(dc.rowAsArray(0), [11, 22], 'should read the replaced row');+ t.equal(dc.replace(99, [1, 2]), false, 'should reject a replace with an out-of-range index');+ t.equal(dc.replace(0, [1]), false, 'should reject a replace with the wrong column count');+ t.deepEqual(+ dc.rowAsArray(0),+ [11, 22],+ 'should leave the row unchanged after a rejected replace'+ );++ t.equal(dc.remove([1, 1, 3]), true, 'should drop unique valid indexes in one rebuild');+ t.deepEqual(+ dc.flattenData(),+ [+ [11, 22],+ [50, 60],+ [90, 100]+ ],+ 'should keep the remaining rows in order'+ );+ t.deepEqual(dc.getPlainIndex(), [0, 1, 2], 'should rebuild indexes after remove');++ t.equal(dc.remove([99]), false, 'should reject an out-of-range index');+ t.equal(dc.remove([-1]), false, 'should reject a negative index');+ t.equal(dc.numRows(), 3, 'should leave rows unchanged after a rejected remove');++ dc.update([[1, 2]]);+ t.equal(dc.numRows(), 1, 'update should still replace after append/remove');+ t.equal(dc.valueAt(0, 0), 1, 'update should write the new snapshot');++ const empty = createDataContainer([]);+ t.equal(+ empty.append([[3, 4]]),+ true,+ 'should set column count from the first row on an empty table'+ );+ t.equal(empty.numRows(), 1, 'empty table should accept the first append');+ t.equal(empty.numColumns(), 2, 'empty table should take width from the first row');+ t.end();+});+test('IndexedDataContainer', t => {const dc = createIndexedDataContainer(createDataContainer(data), indices);