Kepler.gl is a powerful open source geospatial analysis tool for large-scale data sets.
feat: in-place row edits for Arrow tables (#3688)
Igor Dykhtaab6cbe3f6eb4
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex e8f38d489..9d84b0079 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -15,6 +15,10 @@ Each version should:Ref: http://keepachangelog.com/en/0.3.0/-->+## [Unreleased]++- feat: support in-place append, upsert, and delete for primitive Arrow tables (#3688). DuckDB row edits remain a warned no-op.+## [3.3.0-alpha.18] - Oct 19 2026- c7a6399f fix(export): restore annotation and chart panels (#3853)diff --git a/docs/api-reference/actions/actions.md b/docs/api-reference/actions/actions.mdindex 1f00bcc3c..0e53cd654 100644--- a/docs/api-reference/actions/actions.md+++ b/docs/api-reference/actions/actions.md@@ -528,9 +528,9 @@ Returns **{type: ActionTypes.ADD_LAYER, props: props}**### addToDataset-Append or upsert rows on an existing in-memory row dataset without `addDataToMap`. Keeps layer identity and style. Pass `options.upsertBy` to replace rows that share that key and append the rest.+Append or upsert rows on an existing in-memory row or Arrow dataset without `addDataToMap`. Keeps layer identity and style. Pass `options.upsertBy` to replace rows that share that key and append the rest.-Not implemented for Arrow or DuckDB tables (no INSERT / concat yet); those calls warn and leave the table unchanged. Use `addDataToMap` with `keepExistingConfig` for a full replace.+DuckDB tables (no INSERT yet) warn and leave the table unchanged. Arrow tables support primitive columns; nested, binary, and geoarrow columns are rejected. Encoding failures and dictionary index overflow reject the whole edit batch. Use `addDataToMap` with `keepExistingConfig` for a full replace.- **ActionTypes**: `ActionTypes.ADD_TO_DATASET`- **Updaters**: `visStateUpdaters.addToDatasetUpdater`@@ -744,9 +744,9 @@ Returns **{type: ActionTypes.REMOVE_DATASET, key: key}**### removeFromDataset-Delete rows from an existing in-memory row dataset without restyling layers. The second argument is either row indexes or `{field, values}` to match a column (for example an id).+Delete rows from an existing in-memory row or Arrow dataset without restyling layers. The second argument is either row indexes or `{field, values}` to match a column (for example an id).-Not implemented for Arrow or DuckDB tables; those calls warn and leave the table unchanged.+DuckDB tables warn and leave the table unchanged. Arrow tables compact the retained rows while preserving nested values, exact timestamp storage, and shared dictionaries.- **ActionTypes**: `ActionTypes.REMOVE_FROM_DATASET`- **Updaters**: `visStateUpdaters.removeFromDatasetUpdater`diff --git a/docs/api-reference/reducers/vis-state.md b/docs/api-reference/reducers/vis-state.mdindex 365c0cd48..80446747d 100644--- a/docs/api-reference/reducers/vis-state.md+++ b/docs/api-reference/reducers/vis-state.md@@ -107,7 +107,7 @@ Returns **[Object][69]** nextState### addToDatasetUpdater-Append or upsert rows on an existing in-memory row dataset. Keeps layers and re-runs filters. Arrow/DuckDB tables warn and are left unchanged (INSERT / concat not implemented).+Append or upsert rows on an existing in-memory row or Arrow dataset. Keeps layers and re-runs filters. DuckDB tables warn and are left unchanged (INSERT not implemented). Nested, binary, and geoarrow Arrow columns are rejected.- **Action**: [`addToDataset`](../actions/actions.md#addtodataset)@@ -338,7 +338,7 @@ Returns **[Object][69]** nextState### removeFromDatasetUpdater-Delete rows by index or by field values from an existing in-memory row dataset. Keeps layers. Arrow/DuckDB tables warn and are left unchanged.+Delete rows by index or by field values from an existing in-memory row or Arrow dataset. Keeps layers. DuckDB tables warn and are left unchanged.- **Action**: [`removeFromDataset`](../actions/actions.md#removefromdataset)diff --git a/examples/live-data/README.md b/examples/live-data/README.mdindex d84373aea..9e789a142 100644--- a/examples/live-data/README.md+++ b/examples/live-data/README.md@@ -1,7 +1,7 @@# Live remotely hosted dataLoads a CORS CSV of points orbiting San Francisco as a remotely hosted dataset-and **polls it every 300 ms**. The right sidebar switches between three host-app+and **polls it every 300 ms**. The right sidebar switches between four host-appupdate styles:- **Poll URL** — HTTP snapshot replace (`refreshDataset` /@@ -15,12 +15,17 @@ update styles:`ws://localhost:4010/vehicles.ws` and maps each JSON upsert to`addToDataset(..., {upsertBy: 'id'})`. Kepler has no websocket client andstill rebuilds layer data on every message.+- **Arrow rows** — pauses polling and opens a separate local Arrow dataset.+ The same host controls append, upsert and delete through the public actions.+ Mode changes hide the inactive dataset's layers; Arrow rows and layer/filter+ settings remain available when you return.This bundles local monorepo `src/` (not a published npm package). Host-row-and websocket edits only work for in-memory CSV (`RowDataContainer`), which this-example uses. Arrow and DuckDB tables log a warning and leave the data unchanged-(INSERT / concat are not implemented yet). The **Keyed by id** controls-exercise `addToDataset(id, rows, {upsertBy: 'id'})` and+and websocket edits use in-memory CSV (`RowDataContainer`); **Arrow rows**+uses an explicitly typed Arrow table (`ArrowDataContainer`).+The same actions support primitive Arrow columns; nested, binary, and GeoArrow+columns cannot be appended or upserted. DuckDB tables remain a warned no-op.+The **Keyed by id** controls exercise `addToDataset(id, rows, {upsertBy: 'id'})` and`removeFromDataset(id, {field: 'id', values})`.## Pre-requirements@@ -77,6 +82,25 @@ run next to `yarn start` on 8080). The CSV server listens atpattern: `ws → addToDataset`. It is **not** a Kepler websocket layer and does**not** skip a full layer-data rebuild.+### Arrow rows++- Select **Arrow rows**. It starts with 3 local points and requires no CSV or+ WebSocket connection. Storage should read **ArrowDataContainer · 1 batch(es)**.+ The first visit fits the map to include the outer rings used by new points.+- **Add 1 / Add 5** append rows. **Insert tracker**, then **Move tracker**+ repeatedly: its coordinates and speed change while the row count stays put.+ **Remove tracker / Remove veh-01 / Remove last host** delete by id;+ **Remove last / Remove random** delete by index.+- Change the **Live Arrow** layer radius/color or add a numeric **speed** filter.+ Repeat the edits and check that the layer id, styling and filter selection+ remain. The revision counter advances after each successful edit.+- **Auto trail** adds every 800ms and keeps at most 20 host points. Switching+ modes turns it off; returning to Arrow keeps the edited rows without restarting+ the timer. The original CSV dataset keeps its existing mode behavior.+- The Arrow table is initialized once. Edit controls never call `addDataToMap`+ or replace its full snapshot. Sample columns are Utf8, Float64 and Int32;+ nested/binary/GeoArrow append/upsert is deliberately outside this example.+## CSV server only (Add Data URL)From this folder, or the repo root:diff --git a/examples/live-data/esbuild.config.mjs b/examples/live-data/esbuild.config.mjsindex 75097ea45..05bcb6d92 100644--- a/examples/live-data/esbuild.config.mjs+++ b/examples/live-data/esbuild.config.mjs@@ -57,7 +57,7 @@ const config = {},absWorkingDir: __dirname,loader: {'.js': 'jsx', '.css': 'css'},- entryPoints: [path.join(__dirname, 'src/app.tsx')],+ entryPoints: [path.join(__dirname, 'src/index.tsx')],outfile: path.join(DIST_DIR, 'bundle.js'),bundle: true,alias: localKeplerAliases(),diff --git a/examples/live-data/src/app.tsx b/examples/live-data/src/app.tsxindex 5bfad6be0..140e96ea7 100644--- a/examples/live-data/src/app.tsx+++ b/examples/live-data/src/app.tsx@@ -3,8 +3,7 @@import * as React from 'react';import {useCallback, useEffect, useRef, useState} from 'react';-import ReactDOM from 'react-dom/client';-import {Provider, useDispatch, useSelector} from 'react-redux';+import {Provider, useDispatch, useSelector, useStore} from 'react-redux';import {applyMiddleware, combineReducers, compose, createStore} from 'redux';import 'maplibre-gl/dist/maplibre-gl.css';@@ -13,13 +12,16 @@ import KeplerGl from '@kepler.gl/components';import {addDataToMap,addToDataset,+ fitBounds,+ layerConfigChange,refreshDataset,removeFromDataset,updateDatasetProps,wrapTo} from '@kepler.gl/actions';import {DatasetType} from '@kepler.gl/constants';-import {initApplicationConfig} from '@kepler.gl/utils';+import {ArrowDataContainer, initApplicationConfig} from '@kepler.gl/utils';+import {ARROW_DATASET_ID, createArrowDataset} from './arrow-data';initApplicationConfig({enableAnnotations: false});@@ -39,17 +41,19 @@ const COS_LAT = Math.cos((SF.lat * Math.PI) / 180);const KM_PER_DEG_LAT = 111.32;const GOLDEN_ANGLE = Math.PI * (3 - Math.sqrt(5));-type LiveMode = 'poll' | 'host' | 'ws';+type LiveMode = 'poll' | 'host' | 'ws' | 'arrow';type WsStatus = 'idle' | 'connecting' | 'open' | 'closed' | 'error';type KeplerRootState = {keplerGl: {[id: string]: {visState: {+ layers: Parameters<typeof layerConfigChange>[0][];datasets: Record<string,{fields?: {name: string}[];+ dataRevision?: number;dataContainer?: {numRows: () => number;valueAt: (row: number, col: number) => unknown;@@ -87,12 +91,12 @@ const enhancers = applyMiddleware(...middlewares);const composeEnhancers = (window as any).__REDUX_DEVTOOLS_EXTENSION_COMPOSE__ || compose;const store = createStore(reducers, {}, composeEnhancers(enhancers));-function fieldValue(dataset: LiveDataset | undefined, name: string) {+function fieldValue(dataset: LiveDataset | undefined, name: string, row = 0) {const col = dataset?.fields?.findIndex(field => field.name === name) ?? -1;if (col < 0 || !dataset?.dataContainer?.numRows()) {return undefined;}- return dataset.dataContainer.valueAt(0, col);+ return dataset.dataContainer.valueAt(row, col);}function columnIndex(dataset: LiveDataset | undefined, name: string) {@@ -121,7 +125,7 @@ function hostRowIndexes(dataset: LiveDataset | undefined): number[] {}, []);}-function makeHostVehicle(seq: number, options?: {id?: string; ring?: number}) {+function makeHostVehicle(seq: number, options?: {id?: string; ring?: number; speed?: number}) {const angle = seq * GOLDEN_ANGLE;const radiusKm = 5.2 + (seq % 8) * 0.35;const lat = SF.lat + (radiusKm / KM_PER_DEG_LAT) * Math.sin(angle);@@ -131,7 +135,7 @@ function makeHostVehicle(seq: number, options?: {id?: string; ring?: number}) {lat,lng,heading: ((angle * 180) / Math.PI + 360) % 360,- speed: 0,+ speed: options?.speed ?? 0,ring: options?.ring ?? 4 + (seq % 3),progress: 0,orbit_s: 0,@@ -184,7 +188,8 @@ function ModeToggle({return (<divstyle={{- display: 'flex',+ display: 'grid',+ gridTemplateColumns: '1fr 1fr',marginBottom: 14,border: '1px solid #4a5160',borderRadius: 6,@@ -195,13 +200,15 @@ function ModeToggle({[['poll', 'Poll URL', true],['host', 'Host rows', hostEnabled],- ['ws', 'WebSocket', hostEnabled]+ ['ws', 'WebSocket', hostEnabled],+ ['arrow', 'Arrow rows', true]] as const).map(([value, label, enabled]) => (<buttonkey={value}type="button"disabled={!enabled}+ aria-pressed={mode === value}onClick={() => onChange(value)}style={{flex: 1,@@ -281,6 +288,7 @@ function PollSidebar({dataset}: {dataset: LiveDataset | undefined}) {function HostSidebar({dataset,+ arrowMode,autoTrail,keyedNote,onAutoTrail,@@ -293,6 +301,7 @@ function HostSidebar({onRemoveLastHostById}: {dataset: LiveDataset | undefined;+ arrowMode: boolean;autoTrail: boolean;keyedNote: string;onAutoTrail: (next: boolean) => void;@@ -311,6 +320,8 @@ function HostSidebar({const hasPollVehicle = ids.includes(POLL_VEHICLE_ID);const hostIds = ids.filter(id => id.startsWith('host-'));const lastHostId = hostIds[hostIds.length - 1];+ const trackerIndex = ids.indexOf(TRACKER_ID);+ const container = dataset?.dataContainer;return (<>@@ -319,6 +330,18 @@ function HostSidebar({<code>removeFromDataset</code> so rows change in place — same layer id and style. Hostpoints land on outer rings (4–6) and color by <code>ring</code>.</p>+ <p style={{margin: '0 0 12px', color: '#c3c9d5'}}>+ Storage:{' '}+ <output aria-label="Data storage">+ {container instanceof ArrowDataContainer+ ? `ArrowDataContainer · ${container.numChunks()} batch(es)`+ : container+ ? 'RowDataContainer'+ : 'loading'}+ </output>+ {' · Revision '}+ <output aria-label="Dataset revision">{dataset?.dataRevision ?? 0}</output>+ </p><div style={{display: 'grid', gap: 8, marginBottom: 12}}><div style={{display: 'flex', justifyContent: 'space-between', gap: 12}}><span style={{color: '#c3c9d5'}}>rows</span>@@ -389,6 +412,16 @@ function HostSidebar({{lastHostId ? ` · last host ${lastHostId}` : ''}</p>{keyedNote ? <p style={{margin: '6px 0 0', color: '#9ad0a8'}}>{keyedNote}</p> : null}+ {hasTracker ? (+ <p style={{margin: '6px 0 0', color: '#c3c9d5'}}>+ <output aria-label="Tracker position and speed">+ {Number(fieldValue(dataset, 'lat', trackerIndex)).toFixed(5)},{' '}+ {Number(fieldValue(dataset, 'lng', trackerIndex)).toFixed(5)}+ {' · speed '}+ {String(fieldValue(dataset, 'speed', trackerIndex))}+ </output>+ </p>+ ) : null}<labelstyle={{display: 'flex',@@ -408,8 +441,17 @@ function HostSidebar({Auto trail (add every 800ms, keep {MAX_HOST_ROWS} host rows)</label><p style={{margin: '16px 0 0', color: '#c3c9d5'}}>- Switch back to <strong>Poll URL</strong> to resume HTTP snapshot replace — injected rows- disappear on the next fetch.+ {arrowMode ? (+ <>+ This local Arrow dataset keeps its edits when you switch modes. Change its point radius+ or add a speed filter, then repeat edits to check the settings stay put.+ </>+ ) : (+ <>+ Switch back to <strong>Poll URL</strong> to resume HTTP snapshot replace — injected rows+ disappear on the next fetch.+ </>+ )}</p></>);@@ -481,12 +523,18 @@ function WsSidebar({);}-function LiveDataSidebar() {+export function LiveDataSidebar() {const dispatch = useDispatch();- const dataset = useSelector(- (state: KeplerRootState) => state.keplerGl[MAP_ID]?.visState?.datasets?.[DATASET_ID]+ const reduxStore = useStore<KeplerRootState>();+ const datasets = useSelector(+ (state: KeplerRootState) => state.keplerGl[MAP_ID]?.visState?.datasets);const [mode, setMode] = useState<LiveMode>('poll');+ const datasetId = mode === 'arrow' ? ARROW_DATASET_ID : DATASET_ID;+ const dataset = datasets?.[datasetId];+ const datasetReady = Boolean(dataset);+ const hasRowDataset = Boolean(datasets?.[DATASET_ID]);+ const hasArrowDataset = Boolean(datasets?.[ARROW_DATASET_ID]);const [autoTrail, setAutoTrail] = useState(false);const [keyedNote, setKeyedNote] = useState('');const [wsStatus, setWsStatus] = useState<WsStatus>('idle');@@ -495,17 +543,30 @@ function LiveDataSidebar() {const [wsError, setWsError] = useState<string | null>(null);const hostSeq = useRef(1);const trackSeq = useRef(1);+ const arrowViewInitialized = useRef(false);const datasetRef = useRef(dataset);datasetRef.current = dataset;const toKepler = useCallback(- (action: Parameters<typeof wrapTo>[1]) => dispatch(wrapTo(MAP_ID, action)),+ (action: Parameters<ReturnType<typeof wrapTo>>[0]) => dispatch(wrapTo(MAP_ID, action)),[dispatch]);const setLiveMode = useCallback((next: LiveMode) => {setMode(next);+ setKeyedNote('');+ if (next === 'arrow' && !arrowViewInitialized.current) {+ toKepler(+ fitBounds([SF.lng - 0.1, SF.lat - 0.08, SF.lng + 0.1, SF.lat + 0.08], {+ left: 320,+ right: 24,+ top: 24,+ bottom: 24+ })+ );+ arrowViewInitialized.current = true;+ }if (next === 'poll') {setAutoTrail(false);setKeyedNote('');@@ -528,12 +589,29 @@ function LiveDataSidebar() {[toKepler]);+ useEffect(() => {+ // Only switch visibility on mode/load transitions, so user styling and+ // visibility choices stay untouched during edits.+ const layers = reduxStore.getState().keplerGl[MAP_ID]?.visState.layers || [];+ layers.forEach(layer => {+ if (layer.config.dataId !== DATASET_ID && layer.config.dataId !== ARROW_DATASET_ID) return;+ const isVisible = layer.config.dataId === datasetId;+ if (layer.config.isVisible !== isVisible) {+ toKepler(layerConfigChange(layer, {isVisible}));+ }+ });+ }, [datasetId, hasRowDataset, hasArrowDataset, reduxStore, toKepler]);+const addHostRows = useCallback((count: number) => {- const rows = Array.from({length: count}, () => makeHostVehicle(hostSeq.current++));- toKepler(addToDataset(DATASET_ID, rows));+ const rows = Array.from({length: count}, () =>+ makeHostVehicle(hostSeq.current++, {+ speed: mode === 'arrow' ? 20 + (hostSeq.current % 8) * 5 : undefined+ })+ );+ toKepler(addToDataset(datasetId, rows));},- [toKepler]+ [datasetId, mode, toKepler]);const removeLast = useCallback(() => {@@ -541,38 +619,42 @@ function LiveDataSidebar() {if (rows < 1) {return;}- toKepler(removeFromDataset(DATASET_ID, rows - 1));- }, [dataset, toKepler]);+ toKepler(removeFromDataset(datasetId, rows - 1));+ }, [dataset, datasetId, toKepler]);const removeRandom = useCallback(() => {const rows = dataset?.dataContainer?.numRows?.() ?? 0;if (rows < 1) {return;}- toKepler(removeFromDataset(DATASET_ID, Math.floor(Math.random() * rows)));- }, [dataset, toKepler]);+ toKepler(removeFromDataset(datasetId, Math.floor(Math.random() * rows)));+ }, [dataset, datasetId, toKepler]);const upsertTracker = useCallback(() => {const ids = rowIds(datasetRef.current);const existed = ids.includes(TRACKER_ID);- const row = makeHostVehicle(trackSeq.current++, {id: TRACKER_ID, ring: 7});- toKepler(addToDataset(DATASET_ID, row, {upsertBy: 'id'}));+ const row = makeHostVehicle(trackSeq.current++, {+ id: TRACKER_ID,+ ring: 7,+ speed: mode === 'arrow' ? 20 + ((trackSeq.current * 7) % 60) : undefined+ });+ toKepler(addToDataset(datasetId, row, {upsertBy: 'id'}));setKeyedNote(existed? `upsert ${TRACKER_ID} (replaced — row count should stay ${ids.length})`: `upsert ${TRACKER_ID} (inserted — row count ${ids.length + 1})`);- }, [toKepler]);+ }, [datasetId, mode, toKepler]);const removeByFieldId = useCallback((id: string) => {if (!rowIds(datasetRef.current).includes(id)) {return;}- toKepler(removeFromDataset(DATASET_ID, {field: 'id', values: id}));+ toKepler(removeFromDataset(datasetId, {field: 'id', values: id}));setKeyedNote(`removeFromDataset({field: 'id', values: '${id}'})`);},- [toKepler]+ [datasetId, toKepler]);const removeLastHostById = useCallback(() => {@@ -585,18 +667,18 @@ function LiveDataSidebar() {}, [removeByFieldId]);useEffect(() => {- if (mode !== 'host' || !autoTrail) {+ if ((mode !== 'host' && mode !== 'arrow') || !autoTrail || !datasetReady) {return undefined;}const timer = window.setInterval(() => {const extras = hostRowIndexes(datasetRef.current);if (extras.length >= MAX_HOST_ROWS) {- toKepler(removeFromDataset(DATASET_ID, extras[0]));+ toKepler(removeFromDataset(datasetId, extras[0]));}- toKepler(addToDataset(DATASET_ID, makeHostVehicle(hostSeq.current++)));+ addHostRows(1);}, AUTO_INTERVAL_MS);return () => window.clearInterval(timer);- }, [autoTrail, mode, toKepler]);+ }, [autoTrail, mode, datasetId, datasetReady, addHostRows, toKepler]);useEffect(() => {if (mode !== 'ws') {@@ -621,13 +703,13 @@ function LiveDataSidebar() {}};socket.onmessage = event => {- let message: {op?: string; rows?: unknown; upsertBy?: string};+ let message: {op?: string; rows?: Parameters<typeof addToDataset>[1]; upsertBy?: string};try {message = JSON.parse(String(event.data));} catch {return;}- if (message.op !== 'upsert' || !message.rows) {+ if (cancelled || message.op !== 'upsert' || !message.rows) {return;}toKepler(addToDataset(DATASET_ID, message.rows, {upsertBy: message.upsertBy || 'id'}));@@ -664,7 +746,7 @@ function LiveDataSidebar() {<h2 style={{margin: '0 0 8px', fontSize: 16}}>Live data</h2><ModeTogglemode={mode}- hostEnabled={(dataset?.dataContainer?.numRows?.() ?? 0) > 0}+ hostEnabled={(datasets?.[DATASET_ID]?.dataContainer?.numRows?.() ?? 0) > 0}onChange={setLiveMode}/>{mode === 'poll' ? (@@ -680,6 +762,7 @@ function LiveDataSidebar() {) : (<HostSidebardataset={dataset}+ arrowMode={mode === 'arrow'}autoTrail={autoTrail}keyedNote={keyedNote}onAutoTrail={setAutoTrail}@@ -705,19 +788,22 @@ const App = () => {wrapTo(MAP_ID,addDataToMap({- datasets: {- info: {- id: DATASET_ID,- label: 'Live orbit',- type: DatasetType.EXTERNALLY_HOSTED+ datasets: [+ {+ info: {+ id: DATASET_ID,+ label: 'Live orbit',+ type: DatasetType.EXTERNALLY_HOSTED+ },+ data: {fields: [], rows: []},+ metadata: {+ source: LIVE_DATA_URL,+ sourceFormat: 'csv',+ refreshIntervalMs: REFRESH_INTERVAL_MS+ }},- data: {fields: [], rows: []},- metadata: {- source: LIVE_DATA_URL,- sourceFormat: 'csv',- refreshIntervalMs: REFRESH_INTERVAL_MS- }- },+ createArrowDataset()+ ],options: {centerMap: true,autoCreateLayers: false,@@ -734,7 +820,7 @@ const App = () => {config: {dataId: DATASET_ID,label: 'Live orbit',- columns: {lat: 'lat', lng: 'lng', altitude: null},+ columns: {lat: 'lat', lng: 'lng'},isVisible: true,visConfig: {radius: 24, filled: true, opacity: 0.85}},@@ -742,6 +828,21 @@ const App = () => {colorField: {name: 'ring', type: 'integer'},colorScale: 'quantize'}+ },+ {+ id: 'live-arrow-point',+ type: 'point',+ config: {+ dataId: ARROW_DATASET_ID,+ label: 'Live Arrow',+ columns: {lat: 'lat', lng: 'lng'},+ isVisible: false,+ visConfig: {radius: 24, filled: true, opacity: 0.85}+ },+ visualChannels: {+ colorField: {name: 'ring', type: 'integer'},+ colorScale: 'quantize'+ }}]},@@ -783,6 +884,4 @@ const Root = () => (</Provider>);-const container = document.getElementById('root');-const root = ReactDOM.createRoot(container!);-root.render(<Root />);+export default Root;diff --git a/examples/live-data/src/arrow-data.ts b/examples/live-data/src/arrow-data.tsnew file mode 100644index 000000000..62e956061--- /dev/null+++ b/examples/live-data/src/arrow-data.ts@@ -0,0 +1,31 @@+// SPDX-License-Identifier: MIT+// Copyright contributors to the kepler.gl project++import {Table, Vector, Utf8, Float64, Int32, vectorFromArray} from 'apache-arrow';+import {arrowSchemaToFields} from '@kepler.gl/processors';++export const ARROW_DATASET_ID = 'live-arrow-vehicles';++/** Build once; subsequent changes use addToDataset/removeFromDataset. */+export function createArrowDataset() {+ const table = new Table({+ id: vectorFromArray(['veh-01', 'veh-02', 'veh-03'], new Utf8()),+ lat: vectorFromArray([37.79, 37.76, 37.78], new Float64()),+ lng: vectorFromArray([-122.45, -122.42, -122.39], new Float64()),+ heading: vectorFromArray([0, 120, 240], new Float64()),+ speed: vectorFromArray([25, 40, 55], new Float64()),+ ring: vectorFromArray([1, 2, 3], new Int32()),+ progress: vectorFromArray([0, 0, 0], new Float64()),+ orbit_s: vectorFromArray([0, 0, 0], new Float64()),+ updated_at: vectorFromArray(Array(3).fill(new Date().toISOString()), new Utf8())+ });+ return {+ info: {id: ARROW_DATASET_ID, label: 'Live Arrow'},+ data: {+ fields: arrowSchemaToFields(table),+ cols: table.schema.fields.map((_, index) => table.getChildAt(index) as Vector),+ rows: [],+ arrowTable: table+ }+ };+}diff --git a/examples/live-data/src/index.tsx b/examples/live-data/src/index.tsxnew file mode 100644index 000000000..b61605842--- /dev/null+++ b/examples/live-data/src/index.tsx@@ -0,0 +1,10 @@+// SPDX-License-Identifier: MIT+// Copyright contributors to the kepler.gl project++import * as React from 'react';+import ReactDOM from 'react-dom/client';+import Root from './app';++const container = document.getElementById('root') as HTMLElement;+const root = ReactDOM.createRoot(container);+root.render(<Root />);diff --git a/src/actions/src/actions.ts b/src/actions/src/actions.tsindex aca490910..8f74debd3 100644--- a/src/actions/src/actions.ts+++ b/src/actions/src/actions.ts@@ -51,7 +51,7 @@ export type ActionHandlers<T extends {[k: string]: Handler}> = {* @param {boolean} data.options.readOnly `default: false` if `readOnly` is set to `true`* the left setting panel will be hidden* @param {boolean} data.options.keepExistingConfig whether to keep exiting map data and associated layer filter interaction config `default: false`.- * To append, upsert, or delete rows on an existing in-memory row dataset without replacing it or creating layers, use `addToDataset` / `removeFromDataset`. Arrow and DuckDB tables do not support those actions yet.+ * To append, upsert, or delete rows on an existing in-memory row or Arrow dataset without replacing it or creating layers, use `addToDataset` / `removeFromDataset`. DuckDB tables do not support those actions yet.* @param {Object} data.config this object will contain the full kepler.gl instance configuration {mapState, mapStyle, visState}* @public* @examplediff --git a/src/actions/src/vis-state-actions.ts b/src/actions/src/vis-state-actions.tsindex ec41f54e4..94b5a36f6 100644--- a/src/actions/src/vis-state-actions.ts+++ b/src/actions/src/vis-state-actions.ts@@ -1125,7 +1125,8 @@ export type AddToDatasetOptions = {/*** Field name used as a unique key. Matching rows are replaced in place;* keys that are not in the table are appended. Last incoming row wins when- * the same key appears twice. Row tables only — Arrow/DuckDB no-op with a warning.+ * the same key appears twice. DuckDB tables no-op with a warning. Nested, binary, and+ * geoarrow Arrow columns also no-op (primitive Arrow columns concat in place).*/upsertBy?: string;};@@ -1136,13 +1137,14 @@ export type AddToDatasetUpdaterAction = {options?: AddToDatasetOptions;};/**- * Append rows to an existing in-memory row dataset without `addDataToMap`.+ * Append rows to an existing in-memory row or Arrow dataset without `addDataToMap`.* Keeps layer identity and style. Pass `options.upsertBy` to replace rows that* share that key and append the rest.*- * Not implemented for Arrow or DuckDB tables (no INSERT / concat yet); those- * calls warn and leave the table unchanged. Use `addDataToMap` with- * `keepExistingConfig` for a full replace.+ * DuckDB tables (no INSERT yet) warn and leave the table unchanged. Arrow+ * tables concat primitive columns in place; nested, binary, and geoarrow+ * columns are rejected. Use `addDataToMap` with `keepExistingConfig` for a+ * full replace.* @memberof visStateActions* @param dataId dataset id* @param rows one row or an array of rows (arrays in field order, or objects keyed by field name)@@ -1174,12 +1176,12 @@ export type RemoveFromDatasetUpdaterAction = {byField?: RemoveFromDatasetByField;};/**- * Delete rows from an existing in-memory row dataset without restyling layers.+ * Delete rows from an existing in-memory row or Arrow dataset without restyling layers.* The second argument is either row indexes or `{field, values}` to match a* column (e.g. an id). Out-of-range indexes are a no-op for the whole action.*- * Not implemented for Arrow or DuckDB tables; those calls warn and leave the- * table unchanged. For a full table replace use `addDataToMap` with+ * DuckDB tables warn and leave the table unchanged. Arrow tables concat the+ * kept slices in place. For a full table replace use `addDataToMap` with* `keepExistingConfig`.* @memberof visStateActions* @param dataId dataset iddiff --git a/src/duckdb/src/table/duckdb-table.ts b/src/duckdb/src/table/duckdb-table.tsindex b2086bbb9..220dd550e 100644--- a/src/duckdb/src/table/duckdb-table.ts+++ b/src/duckdb/src/table/duckdb-table.ts@@ -2,6 +2,7 @@// Copyright contributors to the kepler.gl projectimport * as arrow from 'apache-arrow';+import Console from 'global/console';import {ALL_FIELD_TYPES,@@ -75,6 +76,13 @@ const DUCKDB_WKB_COLUMN = 'wkb_geometry';*/const KEPLER_GEOM_FROM_GEOJSON_COLUMN = '_geojson';+const DUCKDB_ROW_EDIT_NOT_IMPLEMENTED =+ 'In-place row edits (append, upsert, delete) are not implemented for DuckDB tables. DuckDB INSERT is not supported yet. Use addDataToMap with keepExistingConfig to replace the table.';++function warnDuckDbRowEdits(method: string): void {+ Console.warn(`KeplerGlDuckDbTable.${method}: ${DUCKDB_ROW_EDIT_NOT_IMPLEMENTED}`);+}+/*** Names of columns that most likely contain binary wkb geometry*/@@ -341,6 +349,25 @@ export class KeplerGlDuckDbTable extends KeplerTable {return super.update({cols, rows: [], fields: []});}+ /**+ * DuckDB's Arrow view is a query snapshot. Mutating it without SQL INSERT+ * would desync the WASM table, so row edits stay a warned no-op.+ */+ appendRows(_rows: any[][]): boolean {+ warnDuckDbRowEdits('appendRows');+ return false;+ }++ upsertRows(_rows: any[][], _keyField: string): boolean {+ warnDuckDbRowEdits('upsertRows');+ return false;+ }++ removeRows(_indexes: number[]): boolean {+ warnDuckDbRowEdits('removeRows');+ return false;+ }+static getFileProcessor = function (data: any, inputFormat?: string) {let processor;let format;diff --git a/src/reducers/src/vis-state-updaters.ts b/src/reducers/src/vis-state-updaters.tsindex 18ab26b3e..1ef6f66b3 100644--- a/src/reducers/src/vis-state-updaters.ts+++ b/src/reducers/src/vis-state-updaters.ts@@ -4097,8 +4097,8 @@ function normalizeDatasetRows(}/**- * Append or upsert rows on an existing in-memory row dataset. Keeps layers and re-runs filters.- * Arrow/DuckDB tables warn and are left unchanged (INSERT / concat not implemented).+ * Append or upsert rows on an existing in-memory row or Arrow dataset. Keeps layers and re-runs filters.+ * DuckDB tables warn and are left unchanged (INSERT not implemented). Nested, binary, and geoarrow Arrow columns are rejected.* @memberof visStateUpdaters* @public*/@@ -4131,8 +4131,8 @@ export function addToDatasetUpdater(}/**- * Delete rows by index or by field values from an existing in-memory row dataset.- * Keeps layers. Arrow/DuckDB tables warn and are left unchanged.+ * Delete rows by index or by field values from an existing in-memory row or Arrow dataset.+ * Keeps layers. DuckDB tables warn and are left unchanged.* @memberof visStateUpdaters* @public*/diff --git a/src/table/src/kepler-table.ts b/src/table/src/kepler-table.tsindex bb4a9847f..e06fa5232 100644--- a/src/table/src/kepler-table.ts+++ b/src/table/src/kepler-table.ts@@ -34,6 +34,7 @@ import {getSortingFunction,timeToUnixMilli,createDataContainer,+ ArrowDataContainer,DataForm,diffFilters,filterDataByFilterTypes,@@ -109,7 +110,7 @@ export type TimeFieldFilterProps = TimeRangeFieldDomain & {const FID_KEY = 'name';const ROW_EDIT_NOT_IMPLEMENTED =- 'In-place row edits (append, upsert, delete) are not implemented for Arrow/DuckDB tables. DuckDB INSERT and Arrow concat are not supported yet. Use addDataToMap with keepExistingConfig to replace the table.';+ 'In-place row edits (append, upsert, delete) are not implemented for this table. DuckDB tables omit row edits (no INSERT yet). Use addDataToMap with keepExistingConfig to replace the table.';function warnRowEditsUnavailable(method: string): void {Console.warn(`KeplerTable.${method}: ${ROW_EDIT_NOT_IMPLEMENTED}`);@@ -366,8 +367,8 @@ class KeplerTable<F extends Field = Field> {}/**- * Append column-ordered rows in place. No-op for Arrow/DuckDB (no `append`).- * Accessors stay bound: the container instance is not replaced.+ * Append column-ordered rows in place. Accessors stay bound: the container+ * instance is not replaced. No-op when the container has no `append`.* @returns false when the payload is rejected and the table is unchanged.*/appendRows(rows: any[][]): boolean {@@ -398,7 +399,7 @@ class KeplerTable<F extends Field = Field> {/*** Replace rows that share `keyField` and append the rest. Last incoming row- * wins for a repeated key. No-op for Arrow/DuckDB.+ * wins for a repeated key. No-op when the container has no `append`/`replace`.* @returns false when the payload is rejected and the table is unchanged.*/upsertRows(rows: any[][], keyField: string): boolean {@@ -418,6 +419,16 @@ class KeplerTable<F extends Field = Field> {return false;}+ // Match Arrow keys after encoding so accepted representations such as an+ // Int64 decimal string or a Date compare equal to the stored column value.+ const keys =+ this.dataContainer instanceof ArrowDataContainer+ ? this.dataContainer.getRowEditKeys(rows, keyIdx)+ : rows.map(row => row[keyIdx]);+ if (!keys) {+ return false;+ }+const start = this.dataContainer.numRows();const keyToIndex = new Map<unknown, number>();for (let i = 0; i < start; i++) {@@ -428,31 +439,30 @@ class KeplerTable<F extends Field = Field> {}const lastByKey = new Map<unknown, any[]>();- const keyOrder: unknown[] = [];- for (const row of rows) {- const key = row[keyIdx];- if (!lastByKey.has(key)) {- keyOrder.push(key);- }- lastByKey.set(key, row);- }+ rows.forEach((row, index) => lastByKey.set(keys[index], row));+ const replacements = new Map<number, any[]>();const toAppend: any[][] = [];- for (const key of keyOrder) {- const row = lastByKey.get(key);- if (!row) {- continue;- }+ for (const [key, row] of lastByKey) {const existing = keyToIndex.get(key);if (existing !== undefined) {- this.dataContainer.replace(existing, row);+ replacements.set(existing, row);} else {toAppend.push(row);}}- if (toAppend.length && !this.dataContainer.append(toAppend)) {- return false;+ if (this.dataContainer instanceof ArrowDataContainer) {+ if (!this.dataContainer.replaceRows(replacements, toAppend)) {+ return false;+ }+ } else {+ for (const [index, row] of replacements) {+ this.dataContainer.replace(index, row);+ }+ if (toAppend.length && !this.dataContainer.append(toAppend)) {+ return false;+ }}const nextIndexes =@@ -466,7 +476,7 @@ class KeplerTable<F extends Field = Field> {}/**- * Remove rows by index in place. No-op for Arrow/DuckDB (no `remove`).+ * Remove rows by index in place. No-op when the container has no `remove`.* @returns false when any index is invalid and the table is unchanged.*/removeRows(indexes: number[]): boolean {diff --git a/src/utils/src/arrow-data-container.ts b/src/utils/src/arrow-data-container.tsindex 46ae8e6f3..e74a235af 100644--- a/src/utils/src/arrow-data-container.ts+++ b/src/utils/src/arrow-data-container.ts@@ -1,7 +1,7 @@// SPDX-License-Identifier: MIT// Copyright contributors to the kepler.gl project-import {ALL_FIELD_TYPES} from '@kepler.gl/constants';+import {ALL_FIELD_TYPES, GEOARROW_METADATA_KEY} from '@kepler.gl/constants';import {GeoArrowBinaryData, ProtoDatasetField} from '@kepler.gl/types';import * as arrow from 'apache-arrow';import {console as globalConsole} from 'global/window';@@ -251,13 +251,18 @@ function tableFromCompactedColumns(* Convert an Arrow cell from Vector.get() into a value makeBuilder.append accepts.* Nested GeoArrow (FixedSizeList / List) cells are Vectors, not arrays.*/-function arrowCellToBuilderValue(value: unknown): unknown {+function arrowCellToBuilderValue(value: unknown, logicalChildren = false): unknown {if (value == null) {return null;}if (ArrayBuffer.isView(value)) {return Array.from(value as any);}+ if (logicalChildren && (value instanceof arrow.StructRow || value instanceof arrow.MapRow)) {+ return new Map(+ Array.from(value, ([key, child]) => [key, arrowCellToBuilderValue(child, true)])+ );+ }const nested = value as {toArray?: () => ArrayLike<unknown>;@@ -267,8 +272,9 @@ function arrowCellToBuilderValue(value: unknown): unknown {isValid?: (index: number) => boolean;};- // Primitive numeric child of a FixedSizeList (lng/lat).- if (typeof nested.toArray === 'function' && nested.numChildren === 0) {+ // Keep ordinary import compaction unchanged. Edited snapshots reinterpret+ // timestamp storage first, so their children can safely use logical getters.+ if (!logicalChildren && typeof nested.toArray === 'function' && nested.numChildren === 0) {return Array.from(nested.toArray());}@@ -276,7 +282,7 @@ function arrowCellToBuilderValue(value: unknown): unknown {const out: unknown[] = [];for (let i = 0; i < nested.length; i++) {const valid = typeof nested.isValid === 'function' ? nested.isValid(i) : true;- out.push(valid ? arrowCellToBuilderValue(nested.get(i)) : null);+ out.push(valid ? arrowCellToBuilderValue(nested.get(i), logicalChildren) : null);}return out;}@@ -286,9 +292,11 @@ function arrowCellToBuilderValue(value: unknown): unknown {function compactArrowVector(column: arrow.Vector,- type: arrow.DataType = column.type+ type: arrow.DataType = column.type,+ logicalChildren = false,+ force = false): arrow.Vector {- if (!column || !column.data || column.data.length <= 1) {+ if (!column || !column.data || (!force && column.data.length <= 1)) {return column;}@@ -304,7 +312,25 @@ function compactArrowVector(}}- const builder = arrow.makeBuilder({type, nullValues: [null]});+ const builder = arrow.makeBuilder({+ type,+ nullValues: [null],+ ...(logicalChildren && arrow.DataType.isDictionary(type)+ ? {+ // Struct/Map cells become Maps for builders; their default string key+ // is always "[object Map]". Hash their typed contents instead.+ dictionaryHashFunction: value =>+ JSON.stringify(value, (_, child) =>+ child instanceof Map+ ? Array.from(child)+ : typeof child === 'bigint' ||+ (typeof child === 'number' && !Number.isFinite(child))+ ? String(child)+ : child+ )+ }+ : {})+ });const length = column.length;for (let i = 0; i < length; i++) {const isValid =@@ -314,7 +340,7 @@ function compactArrowVector(// FixedSizeList/List .get() returns nested Vectors. Appending those// silently writes NaN (points) or throws (lines/polygons). Plain JS// arrays are what makeBuilder expects.- builder.append(isValid ? arrowCellToBuilderValue(column.get(i)) : null);+ builder.append(isValid ? arrowCellToBuilderValue(column.get(i), logicalChildren) : null);}const finished = builder.finish() as arrow.Vector | {toVector: () => arrow.Vector};if (finished instanceof arrow.Vector) {@@ -326,6 +352,215 @@ function compactArrowVector(return column;}+/**+ * Primitive (and dictionary-of-primitive) columns can be rebuilt from JS row+ * arrays via vectorFromArray. Nested, binary, and geoarrow columns cannot.+ */+function isSupportedArrowRowEditType(type: arrow.DataType): boolean {+ if (+ arrow.DataType.isUtf8(type) ||+ arrow.DataType.isNull(type) ||+ arrow.DataType.isInt(type) ||+ arrow.DataType.isFloat(type) ||+ arrow.DataType.isBool(type) ||+ arrow.DataType.isDate(type) ||+ arrow.DataType.isTimestamp(type) ||+ arrow.DataType.isTime(type)+ ) {+ return true;+ }+ if (arrow.DataType.isDictionary(type)) {+ const dictionary = (type as arrow.Dictionary).dictionary;+ return Boolean(dictionary && isSupportedArrowRowEditType(dictionary));+ }+ return false;+}++/** Reinterpret timestamp storage for builders without rounding through JS milliseconds. */+function rowEditStorageType(type: arrow.DataType): arrow.DataType {+ if (arrow.DataType.isTimestamp(type)) return new arrow.Int64();+ if (arrow.DataType.isDictionary(type)) return type.indices;+ if (type.children?.length) {+ // Retain the Arrow type's prototype and layout (list size, union mode, etc.).+ // Field.clone preserves names, nullability and extension metadata.+ return Object.assign(Object.create(Object.getPrototypeOf(type)), type, {+ children: type.children.map(field => field.clone({type: rowEditStorageType(field.type)}))+ });+ }+ return type;+}++function rowEditStorageVector(+ column: arrow.Vector,+ type: arrow.DataType,+ remapped = new Map<arrow.Data, arrow.Data>(),+ dictionaries = new Map<number, arrow.Vector>()+): arrow.Vector {+ const convert = (original: arrow.Data, nextType: arrow.DataType): arrow.Data => {+ const data = remapped.get(original) || original;+ const copy = data.clone(+ nextType,+ data.offset,+ data.length,+ data.nullCount,+ data.buffers,+ data.children.map((child, i) => convert(child, nextType.children[i].type))+ );+ if (arrow.DataType.isDictionary(nextType)) {+ const canonical = dictionaries.get(nextType.id);+ if (canonical) {+ // Equivalent field types from different batches need the same object,+ // even when their DataType instances differ.+ copy.dictionary = canonical;+ } else {+ if (!data.dictionary) throw new Error('Missing Arrow dictionary');+ copy.dictionary = rowEditStorageVector(+ data.dictionary,+ nextType.dictionary,+ remapped,+ dictionaries+ );+ }+ }+ return copy;+ };+ return new arrow.Vector(column.data.map(data => convert(data, type)));+}++function rowEditDictionaryCapacity(type: arrow.Dictionary): number {+ return 2 ** (type.indices.bitWidth - (type.indices.isSigned ? 1 : 0));+}++function collectRowEditDictionaries(data: arrow.Data, groups: Map<number, Set<arrow.Data>>): void {+ data.children.forEach(child => collectRowEditDictionaries(child, groups));+ if (!arrow.DataType.isDictionary(data.type) || !data.dictionary) return;+ // Inner dictionaries must be reconciled before their enclosing dictionary.+ data.dictionary.data.forEach(child => collectRowEditDictionaries(child, groups));+ const group = groups.get(data.type.id) || new Set<arrow.Data>();+ group.add(data);+ groups.set(data.type.id, group);+}++function checkRowEditDictionaryCapacity(column: arrow.Vector): void {+ if (!arrow.DataType.isDictionary(column.type)) return;+ const capacity = rowEditDictionaryCapacity(column.type);+ if (column.data.some(data => (data.dictionary?.length || 0) > capacity)) {+ throw new Error('Row edit exceeds the Arrow dictionary index capacity');+ }+}++/** Compact only edited snapshots; progressive loading and other rebuilds keep their behavior. */+function compactRowEditColumns(columns: arrow.Vector[]): arrow.Vector[] {+ columns = columns.map(+ column =>+ new arrow.Vector(+ column.data.map(data => {+ if (!data.children.length) return data;+ // List/Map slices retain their entire child buffer. Normalize each+ // retained chunk before merging dictionaries so deleted/null child+ // entries cannot cause a false index-capacity overflow.+ const sourceGroups = new Map<number, Set<arrow.Data>>();+ collectRowEditDictionaries(data, sourceGroups);+ const sourceDictionaries = new Map<number, arrow.Vector>();+ sourceGroups.forEach((entries, id) => {+ const dictionary = entries.values().next().value?.dictionary;+ if (dictionary) sourceDictionaries.set(id, dictionary);+ });+ const type = rowEditStorageType(data.type);+ const compacted = compactArrowVector(+ rowEditStorageVector(new arrow.Vector([data]), type),+ type,+ true,+ true+ );+ return rowEditStorageVector(compacted, data.type, new Map(), sourceDictionaries).data[0];+ })+ )+ );+ const groups = new Map<number, Set<arrow.Data>>();+ columns.forEach(column => column.data.forEach(data => collectRowEditDictionaries(data, groups)));+ const dictionaries = new Map<number, arrow.Vector>();+ const remapped = new Map<arrow.Data, arrow.Data>();+ for (const entries of groups.values()) {+ // Narrow indexes get the first dictionary entries before wider columns add values.+ const group = [...entries].sort(+ (a, b) =>+ rowEditDictionaryCapacity(a.type as arrow.Dictionary) -+ rowEditDictionaryCapacity(b.type as arrow.Dictionary)+ );+ const type = group[group.length - 1].type as arrow.Dictionary;+ if (+ group.some(+ data => String((data.type as arrow.Dictionary).dictionary) !== String(type.dictionary)+ )+ ) {+ throw new Error('Shared Arrow dictionary value types must match');+ }+ const original = group[0].dictionary;+ if (!original) throw new Error('Missing Arrow dictionary');+ if (group.every(data => data.dictionary === original)) {+ // Deletion usually keeps one original mapping. Preserve its exact payload,+ // including nested timestamps, and only compact the retained indices.+ dictionaries.set(+ type.id,+ rowEditStorageVector(original, original.type, remapped, dictionaries)+ );+ continue;+ }+ const storageType = new arrow.Dictionary(+ rowEditStorageType(type.dictionary),+ type.indices,+ type.id,+ type.isOrdered+ );+ const combined = new arrow.Vector(+ group.flatMap(+ data =>+ rowEditStorageVector(new arrow.Vector([data]), storageType, remapped, dictionaries).data+ )+ );+ const compacted = compactArrowVector(combined, storageType, true);+ checkRowEditDictionaryCapacity(compacted);+ const compactedDictionary = compacted.data[0].dictionary;+ if (!compactedDictionary) throw new Error('Missing compacted Arrow dictionary');+ const dictionary = rowEditStorageVector(+ compactedDictionary,+ type.dictionary,+ new Map(),+ dictionaries+ );+ dictionaries.set(type.id, dictionary);+ let offset = 0;+ for (const originalData of group) {+ const columnType = originalData.type as arrow.Dictionary;+ const chunk = compacted.slice(offset, offset + originalData.length).data[0];+ const capacity = rowEditDictionaryCapacity(columnType);+ for (let index = 0; index < chunk.length; index++) {+ if (chunk.getValid(index) && Number(chunk.values[index]) >= capacity) {+ throw new Error('Row edit exceeds the Arrow dictionary index capacity');+ }+ }+ const buffers = chunk.buffers;+ buffers[arrow.BufferType.DATA] = new columnType.indices.ArrayType(chunk.values);+ const data = chunk.clone(columnType, chunk.offset, chunk.length, chunk.nullCount, buffers);+ data.dictionary = dictionary;+ remapped.set(originalData, data);+ offset += originalData.length;+ }+ }+ return columns.map(column => {+ // Builders see raw timestamp units and canonical dictionary indices, even+ // inside lists/structs. Restore the original types after one-batch compaction.+ const type = rowEditStorageType(column.type);+ return rowEditStorageVector(+ compactArrowVector(rowEditStorageVector(column, type, remapped, dictionaries), type, true),+ column.type,+ new Map(),+ dictionaries+ );+ });+}+/*** A data container where all data is stored in raw Arrow table*/@@ -433,6 +668,160 @@ export class ArrowDataContainer implements DataContainerInterface {// this._colData = this._cols.map(c => c.toArray());}+ /** Append typed rows atomically while retaining the container and prior snapshots. */+ append(rows: any[][]): boolean {+ if (!Array.isArray(rows)) return false;+ return this.replaceRows(new Map(), rows);+ }++ replace(index: number, row: any[]): boolean {+ return this.replaceRows(new Map([[index, row]]), []);+ }++ /** @internal Match upsert keys in the same representation returned by valueAt. */+ getRowEditKeys(rows: any[][], keyIndex: number): unknown[] | null {+ const type = this._arrowTable.schema.fields[keyIndex]?.type;+ if (!type) return null;+ let valueType = type;+ while (arrow.DataType.isDictionary(valueType)) valueType = valueType.dictionary;+ try {+ if (arrow.DataType.isInt(valueType)) {+ const bits = valueType.bitWidth;+ const limit = 1n << BigInt(bits - (valueType.isSigned ? 1 : 0));+ const min = valueType.isSigned ? -limit : 0n;+ const max = limit - 1n;+ for (const row of rows) {+ const value = row[keyIndex];+ if (value === null) continue;+ // Arrow's typed array assignment wraps out-of-range integers. A key+ // must not silently turn into a different existing row's identity.+ if (typeof value === 'string' && !value.trim()) return null;+ const integer = BigInt(bits === 64 ? value : Number(value));+ if (integer < min || integer > max) return null;+ }+ }+ const keys = arrow.vectorFromArray(+ rows.map(row => row[keyIndex]),+ type+ );+ checkRowEditDictionaryCapacity(keys);+ return Array.from(keys);+ } catch {+ return null;+ }+ }++ /** @internal Encode a complete edit batch before replacing any live columns. */+ replaceRows(replacements: Map<number, any[]>, appended: any[][]): boolean {+ for (const index of replacements.keys()) {+ if (!Number.isInteger(index) || index < 0 || index >= this._numRows) return false;+ }+ const incoming = this._columnsFromRows([...replacements.values(), ...appended]);+ if (!incoming) return false;+ const ordered = [...replacements.keys()]+ .map((index, incomingIndex) => ({index, incomingIndex}))+ .sort((a, b) => a.index - b.index);+ try {+ const columns = this._cols.map((column, i) => {+ const chunks: arrow.Data[] = [];+ let start = 0;+ for (const {index, incomingIndex} of ordered) {+ chunks.push(+ ...column.slice(start, index).data,+ ...incoming[i].slice(incomingIndex, incomingIndex + 1).data+ );+ start = index + 1;+ }+ chunks.push(...column.slice(start).data, ...incoming[i].slice(replacements.size).data);+ return new arrow.Vector(chunks);+ });+ this._commitColumns(columns);+ return true;+ } catch {+ return false;+ }+ }++ /** Delete unique valid indexes without decoding retained values. */+ remove(indexes: number[]): boolean {+ if (!Array.isArray(indexes) || !indexes.length) return false;+ if (indexes.some(index => !Number.isInteger(index) || index < 0 || index >= this._numRows))+ return false;+ const sorted = [...new Set(indexes)].sort((a, b) => a - b);+ try {+ const columns = this._cols.map(column => {+ const chunks: arrow.Data[] = [];+ let start = 0;+ for (const index of sorted) {+ if (start < index) chunks.push(...column.slice(start, index).data);+ start = index + 1;+ }+ if (start < this._numRows) chunks.push(...column.slice(start).data);+ return chunks.length ? new arrow.Vector(chunks) : column.slice(0, 0);+ });+ this._commitColumns(columns);+ return true;+ } catch {+ return false;+ }+ }++ private _commitColumns(columns: arrow.Vector[]): void {+ const {schema} = this._arrowTable;+ const compacted = compactRowEditColumns(columns);+ // Ordinal keys let Arrow align chunks without interpreting real column names.+ const ordered = new arrow.Table(Object.fromEntries(compacted.map((column, i) => [i, column])));+ const batches = ordered.batches.map(batch => {+ const result = new arrow.RecordBatch(schema, batch.data);+ // Arrow reconciles fields by name, even for positional input. Retain the+ // source fields when an external schema contains duplicate names.+ schema.fields.forEach((field, i) => {+ result.schema.fields[i] = field;+ });+ // Validate shared dictionary identities before publishing the snapshot.+ result.dictionaries;+ return result;+ });+ this._assignTable(new arrow.Table(schema, batches));+ this._geoArrowBinaryData = {};+ }++ private _columnsFromRows(rows: any[][]): arrow.Vector[] | null {+ const {fields} = this._arrowTable.schema;+ if (+ !rows.length ||+ !fields.length ||+ rows.some(row => !Array.isArray(row) || row.length !== fields.length)+ )+ return null;+ for (const field of fields) {+ if (+ field.metadata.get(GEOARROW_METADATA_KEY)?.startsWith('geoarrow') ||+ !isSupportedArrowRowEditType(field.type)+ ) {+ globalConsole.warn(+ `ArrowDataContainer: row edits do not support column "${field.name}" (${field.type})`+ );+ return null;+ }+ }+ try {+ return fields.map((field, i) => {+ const column = arrow.vectorFromArray(+ rows.map(row => row[i]),+ field.type+ );+ checkRowEditDictionaryCapacity(column);+ return column;+ });+ } catch {+ globalConsole.warn(+ 'ArrowDataContainer: could not encode row edits using the existing column types'+ );+ return null;+ }+ }+numChunks(): number {return this._numChunks;}diff --git a/src/utils/src/data-container-interface.ts b/src/utils/src/data-container-interface.tsindex 61b437ef9..1d2a48b1a 100644--- a/src/utils/src/data-container-interface.ts+++ b/src/utils/src/data-container-interface.ts@@ -20,19 +20,19 @@ export interface DataContainerInterface {update?(updateData: any[] | arrow.Table, options?: Record<string, unknown>): void;/**- * Append rows in place. Implemented by row containers; Arrow/DuckDB omit this.- * @returns false when the rows are rejected (wrong shape) and the container is unchanged.+ * Append rows in place. Implemented by row and Arrow containers.+ * @returns false when the rows are rejected (wrong shape or unsupported Arrow type) and the container is unchanged.*/append?(rows: any[][]): boolean;/**- * Replace one row in place. Implemented by row containers; Arrow/DuckDB omit this.+ * Replace one row in place. Implemented by row and Arrow containers.* @returns false when the index or row is rejected and the container is unchanged.*/replace?(index: number, row: any[]): boolean;/**- * Remove rows by index in place. Implemented by row containers; Arrow/DuckDB omit this.+ * Remove rows by index in place. Implemented by row and Arrow containers.* @returns false when any index is invalid and the container is unchanged.*/remove?(indexes: number[]): boolean;diff --git a/test/browser/live-data.spec.js b/test/browser/live-data.spec.jsnew file mode 100644index 000000000..f760c7d15--- /dev/null+++ b/test/browser/live-data.spec.js@@ -0,0 +1,117 @@+// SPDX-License-Identifier: MIT+// Copyright contributors to the kepler.gl project++import React from 'react';+import {act, fireEvent, render, screen} from '@testing-library/react';+import {Provider} from 'react-redux';+import {combineReducers, createStore} from 'redux';+import keplerGlReducer from '@kepler.gl/reducers';+import {addFilter, registerEntry, setFilter, wrapTo} from '@kepler.gl/actions';+import {KeplerGlLayers} from '@kepler.gl/layers';+import {KeplerTable} from '@kepler.gl/table';+import {ArrowDataContainer} from '@kepler.gl/utils';+import {RowDataContainer} from '../../src/utils/src/row-data-container';+import {LiveDataSidebar} from '../../examples/live-data/src/app';+import {ARROW_DATASET_ID, createArrowDataset} from '../../examples/live-data/src/arrow-data';++jest.mock('@kepler.gl/components', () => ({__esModule: true, default: () => null}));+jest.mock('maplibre-gl/dist/maplibre-gl.css', () => ({}));++const {PointLayer} = KeplerGlLayers;+afterEach(() => jest.useRealTimers());++function setup() {+ const sample = createArrowDataset();+ const arrow = new KeplerTable({info: sample.info, color: [0, 0, 0]}).updateSchema(sample.data);+ const rows = new KeplerTable({info: {id: 'live-vehicles'}, color: [0, 0, 0]}).updateSchema({+ fields: sample.data.fields,+ rows: arrow.dataContainer.flattenData()+ });+ const layers = [rows, arrow].map(table => {+ const {props} = PointLayer.findDefaultLayerProps(table);+ const layer = new PointLayer({+ ...props[0],+ id: `${table.id}-point`,+ dataId: table.id,+ isVisible: table === rows+ });+ layer.updateLayerVisConfig({radius: 17});+ return layer;+ });+ const reducer = combineReducers({keplerGl: keplerGlReducer});+ const initial = reducer(undefined, registerEntry({id: 'map'}));+ initial.keplerGl.map.visState = {+ ...initial.keplerGl.map.visState,+ datasets: {[rows.id]: rows, [arrow.id]: arrow},+ layers,+ layerData: [{}, {}],+ layerOrder: [0, 1]+ };+ const store = createStore(reducer, initial);+ const view = render(+ <Provider store={store}>+ <LiveDataSidebar />+ </Provider>+ );+ const state = () => store.getState().keplerGl.map.visState;+ const dataset = id => state().datasets[id];+ return {...view, store, state, dataset};+}++test('Arrow controls edit the real container and retain settings across mode switches', () => {+ const {store, state, dataset} = setup();+ const original = dataset(ARROW_DATASET_ID).dataContainer;+ expect(original).toBeInstanceOf(ArrowDataContainer);+ expect(dataset('live-vehicles').dataContainer).toBeInstanceOf(RowDataContainer);+ fireEvent.click(screen.getByRole('button', {name: 'Arrow rows'}));+ act(() => {+ store.dispatch(wrapTo('map', addFilter(ARROW_DATASET_ID)));+ store.dispatch(wrapTo('map', setFilter(0, 'name', 'speed')));+ store.dispatch(wrapTo('map', setFilter(0, 'value', [30, 50])));+ });+ const filterId = state().filters[0].id;+ fireEvent.click(screen.getByRole('button', {name: 'Add 5'}));+ expect(original.numRows()).toBe(8);+ fireEvent.click(screen.getByRole('button', {name: 'Insert tracker'}));+ const position = screen.getByLabelText('Tracker position and speed').textContent;+ fireEvent.click(screen.getByRole('button', {name: 'Move tracker'}));+ expect(original.numRows()).toBe(9);+ expect(screen.getByLabelText('Tracker position and speed').textContent).not.toBe(position);+ fireEvent.click(screen.getByRole('button', {name: 'Remove tracker'}));+ fireEvent.click(screen.getByRole('button', {name: 'Remove last'}));+ expect(original.numRows()).toBe(7);+ expect(dataset(ARROW_DATASET_ID).dataContainer).toBe(original);+ expect(dataset(ARROW_DATASET_ID).dataRevision).toBe(5);+ expect(state().layers[1].id).toBe(`${ARROW_DATASET_ID}-point`);+ expect(state().layers[1].config.visConfig.radius).toBe(17);+ expect(state().filters[0]).toMatchObject({id: filterId, value: [30, 50]});+ fireEvent.click(screen.getByRole('button', {name: 'Host rows'}));+ fireEvent.click(screen.getByRole('button', {name: 'Add 1'}));+ expect(dataset('live-vehicles').dataContainer.numRows()).toBe(4);+ expect(original.numRows()).toBe(7);+ fireEvent.click(screen.getByRole('button', {name: 'Arrow rows'}));+ expect(dataset(ARROW_DATASET_ID).dataContainer).toBe(original);+ expect(state().layers.map(layer => layer.config.isVisible)).toEqual([false, true]);+ expect(state().filters[0]).toMatchObject({id: filterId, value: [30, 50]});+});++test('auto trail stops on mode change and unmount without changing the other dataset', () => {+ jest.useFakeTimers();+ const {dataset, unmount} = setup();+ const original = dataset(ARROW_DATASET_ID).dataContainer;+ fireEvent.click(screen.getByRole('button', {name: 'Arrow rows'}));+ fireEvent.click(screen.getByRole('checkbox', {name: /Auto trail/}));+ act(() => jest.advanceTimersByTime(800));+ expect(original.numRows()).toBe(4);+ fireEvent.click(screen.getByRole('button', {name: 'Host rows'}));+ act(() => jest.advanceTimersByTime(2400));+ expect(original.numRows()).toBe(4);+ expect(dataset('live-vehicles').dataContainer.numRows()).toBe(3);+ fireEvent.click(screen.getByRole('button', {name: 'Arrow rows'}));+ expect(screen.getByRole('checkbox', {name: /Auto trail/})).not.toBeChecked();+ fireEvent.click(screen.getByRole('checkbox', {name: /Auto trail/}));+ unmount();+ act(() => jest.advanceTimersByTime(2400));+ expect(original.numRows()).toBe(4);+ jest.useRealTimers();+});diff --git a/test/node/reducers/vis-state-test.js b/test/node/reducers/vis-state-test.jsindex 540c63471..ebe2be215 100644--- a/test/node/reducers/vis-state-test.js+++ b/test/node/reducers/vis-state-test.js@@ -15,6 +15,7 @@ import {VisStateActions, MapStateActions} from '@kepler.gl/actions';import {visStateReducer as reducer,INITIAL_VIS_STATE,+ INITIAL_MAP_STATE,DEFAULT_ANIMATION_CONFIG,serializeLayer,validateLayerWithData,@@ -11187,9 +11188,9 @@ test('VisStateUpdater -> addToDataset no-ops', async t => {t.equal(reducer(state, VisStateActions.addToDataset('live', [3, 4])),state,- 'tables without append (Arrow/DuckDB) are a no-op'+ 'tables without append are a no-op');- t.ok(warn.called, 'should warn that in-place row edits are not implemented for Arrow/DuckDB');+ t.ok(warn.called, 'should warn that in-place row edits are not implemented');warn.restore();t.equal(state.datasets.live.dataContainer.numRows(), 1, 'no-op must not mutate rows');@@ -11294,9 +11295,9 @@ test('VisStateUpdater -> removeFromDataset deletes rows and keeps layers', asynct.equal(reducer(oneIndex, VisStateActions.removeFromDataset('live', 0)),oneIndex,- 'tables without remove (Arrow/DuckDB) are a no-op'+ 'tables without remove are a no-op');- t.ok(warn.called, 'should warn that in-place deletes are not implemented for Arrow/DuckDB');+ t.ok(warn.called, 'should warn that in-place deletes are not implemented');warn.restore();t.end();@@ -11432,3 +11433,520 @@ test('VisStateUpdater -> addToDataset and removeFromDataset clear table sort', at.end();});++async function createArrowLiveDatasets(rows) {+ const table = rows instanceof arrow.Table ? rows : arrow.tableFromJSON(rows);+ return createNewDataEntryMock({+ info: {id: 'live', label: 'live.arrow'},+ data: processArrowBatches(table.batches)+ });+}++test('VisStateUpdater -> addToDataset and removeFromDataset on Arrow tables', async t => {+ const datasets = await createArrowLiveDatasets([{lat: 37.77, lng: -122.42}]);+ t.ok(+ typeof datasets.live.dataContainer.append === 'function',+ 'Arrow dataset should expose append'+ );++ const {props} = PointLayer.findDefaultLayerProps(datasets.live);+ const pointLayer = new PointLayer({+ id: 'p1',+ dataId: 'live',+ isVisible: true,+ ...props[0]+ });+ const {layerData, layer} = calculateLayerData(+ pointLayer,+ {...INITIAL_VIS_STATE, datasets, layers: [pointLayer], layerData: [{}]},+ undefined+ );+ const state = {+ ...INITIAL_VIS_STATE,+ datasets,+ layers: [layer],+ layerData: [layerData],+ layerOrder: [layer.id]+ };++ const appended = reducer(+ state,+ VisStateActions.addToDataset('live', [+ [10, 20],+ [30, 40]+ ])+ );+ t.equal(appended.datasets.live.dataContainer.numRows(), 3, 'should append both Arrow rows');+ t.equal(appended.layers[0].id, 'p1', 'should keep the same layer id');+ t.equal(+ appended.layerData[0].data.numRows,+ 3,+ 'should rebuild Arrow layer data with the new row count'+ );+ t.notEqual(+ appended.layerData[0].data,+ state.layerData[0].data,+ 'should replace the previous Arrow table snapshot'+ );++ const keyed = await createArrowLiveDatasets([+ {id: 'a', lat: 1, lng: 2},+ {id: 'b', lat: 3, lng: 4}+ ]);+ const upserted = reducer(+ {...INITIAL_VIS_STATE, datasets: keyed},+ VisStateActions.addToDataset(+ 'live',+ [+ {id: 'b', lat: 30, lng: 40},+ {id: 'c', lat: 5, lng: 6}+ ],+ {upsertBy: 'id'}+ )+ );+ t.equal(+ upserted.datasets.live.dataContainer.numRows(),+ 3,+ 'should append only the new Arrow key'+ );+ t.equal(+ upserted.datasets.live.dataContainer.valueAt(1, 1),+ 30,+ 'should replace the matching Arrow key'+ );+ t.equal(+ upserted.datasets.live.dataContainer.valueAt(2, 0),+ 'c',+ 'should append the new Arrow key'+ );++ const removed = reducer(+ upserted,+ VisStateActions.removeFromDataset('live', {field: 'id', values: ['b']})+ );+ t.equal(removed.datasets.live.dataContainer.numRows(), 2, 'should drop the matching Arrow id');+ t.equal(+ removed.datasets.live.dataContainer.valueAt(0, 0),+ 'a',+ 'should keep unmatched Arrow ids'+ );+ t.equal(+ removed.datasets.live.dataContainer.valueAt(1, 0),+ 'c',+ 'should keep later unmatched Arrow ids'+ );++ t.end();+});++test('VisStateUpdater -> rejected Arrow upserts leave all rows and derived state unchanged', async t => {+ const datasets = await createArrowLiveDatasets(+ new arrow.Table({+ id: arrow.vectorFromArray(['a', 'b'], new arrow.Utf8()),+ amount: arrow.vectorFromArray([1n, 2n], new arrow.Int64())+ })+ );+ const state = reducer(+ {...INITIAL_VIS_STATE, datasets},+ VisStateActions.sortTableColumn('live', 'amount')+ );+ const dataset = state.datasets.live;+ const container = dataset.dataContainer;+ const snapshot = container.getTable();+ const revision = dataset.dataRevision;+ const indexes = dataset.allIndexes;+ const sortOrder = dataset.sortOrder;+ const warn = sinon.stub(Console, 'warn');+ try {+ for (const lastKey of ['b', 'c']) {+ const result = reducer(+ state,+ VisStateActions.addToDataset(+ 'live',+ [+ ['a', 10n],+ [lastKey, 'invalid-int64']+ ],+ {+ upsertBy: 'id'+ }+ )+ );+ t.equal(result, state, 'encoding failure is a reducer no-op');+ t.equal(container.getTable(), snapshot, 'no partial Arrow snapshot is committed');+ t.deepEqual(+ container.flattenData(),+ [+ ['a', 1n],+ ['b', 2n]+ ],+ 'no earlier replacement leaks'+ );+ t.equal(dataset.dataRevision, revision, 'revision stays unchanged');+ t.equal(dataset.allIndexes, indexes, 'indexes stay unchanged');+ t.equal(dataset.sortOrder, sortOrder, 'sorting stays unchanged');+ }+ } finally {+ warn.restore();+ }+ t.end();+});++test('VisStateUpdater -> Arrow upserts preserve accessors and last incoming values', async t => {+ const datasets = await createArrowLiveDatasets([+ {id: 'a', value: 1},+ {id: 'b', value: 2},+ {id: 'c', value: 3}+ ]);+ let state = {...INITIAL_VIS_STATE, datasets};+ const container = datasets.live.dataContainer;+ const snapshot = container.getTable();+ const accessor = datasets.live.fields[1].valueAccessor;+ const revision = datasets.live.dataRevision;+ state = reducer(+ state,+ VisStateActions.addToDataset(+ 'live',+ [+ ['c', 30],+ ['e', 5],+ ['a', 10],+ ['d', 4],+ ['c', 33],+ ['e', 55]+ ],+ {upsertBy: 'id'}+ )+ );+ t.equal(state.datasets.live.dataContainer, container, 'keeps the original container');+ t.deepEqual(+ container.flattenData(),+ [+ ['a', 10],+ ['b', 2],+ ['c', 33],+ ['e', 55],+ ['d', 4]+ ],+ 'replacements keep positions and new keys keep first incoming order'+ );+ t.equal(accessor({index: 2}), 33, 'existing accessors read updated values');+ t.equal(snapshot.getChildAt(1).get(2), 3, 'previous Arrow snapshots stay unchanged');+ t.equal(state.datasets.live.dataRevision, revision + 1, 'one revision for the whole batch');+ t.deepEqual(state.datasets.live.allIndexes, [0, 1, 2, 3, 4], 'indexes include appended rows');++ state = reducer(state, VisStateActions.addToDataset('live', [['b', null]], {upsertBy: 'id'}));+ t.equal(container.numRows(), 5, 'replacement-only upsert keeps row count');+ t.equal(container.valueAt(1, 1), null, 'replacement-only upsert accepts null values');+ state = reducer(state, VisStateActions.addToDataset('live', [['f', 6]], {upsertBy: 'id'}));+ t.deepEqual(container.rowAsArray(5), ['f', 6], 'append-only upsert preserves values');++ state = reducer(state, VisStateActions.removeFromDataset('live', [0, 1, 2, 3, 4, 5]));+ state = reducer(state, VisStateActions.addToDataset('live', [['g', 7]], {upsertBy: 'id'}));+ t.deepEqual(+ container.flattenData(),+ [['g', 7]],+ 'upsert retains the schema after deleting all rows'+ );+ t.end();+});++test('VisStateUpdater -> Arrow nested deletion preserves list values and IPC', async t => {+ for (const [type, values] of [+ [+ new arrow.Int32(),+ [+ [1, null, 3],+ [4, 5, 6],+ [7, null, 9]+ ]+ ],+ [+ new arrow.Float16(),+ [+ [1.5, 2.5],+ [3, 4],+ [5.5, 6.5]+ ]+ ]+ ]) {+ for (const structWrapped of [false, true]) {+ const list = arrow.vectorFromArray(+ values,+ new arrow.List(new arrow.Field('item', type, true))+ );+ const column = structWrapped+ ? arrow.makeVector({+ type: new arrow.Struct([new arrow.Field('items', list.type, true)]),+ length: values.length,+ children: list.data+ })+ : list;+ const source = new arrow.Table({+ id: arrow.vectorFromArray([1, 2, 3], new arrow.Int32()),+ values: column+ });+ const datasets = await createArrowLiveDatasets(source);+ const state = {...INITIAL_VIS_STATE, datasets};+ const readValues = table =>+ Array.from(table.getChildAt(1), value => Array.from(structWrapped ? value.items : value));++ const removed = reducer(+ state,+ VisStateActions.removeFromDataset('live', {field: 'id', values: [2]})+ );+ const edited = removed.datasets.live.dataContainer.getTable();+ t.notEqual(removed, state, `${column.type} deletion succeeds`);+ t.deepEqual(readValues(edited), [values[0], values[2]], 'retained list values are unchanged');+ t.deepEqual(+ readValues(arrow.tableFromIPC(arrow.tableToIPC(edited))),+ [values[0], values[2]],+ 'IPC preserves the retained logical values and nulls'+ );+ t.deepEqual(readValues(source), values, 'the original Arrow snapshot stays unchanged');+ }+ }+ t.end();+});++test('VisStateUpdater -> Arrow nested deletion preserves shared dictionaries and IPC', async t => {+ const dictionary = arrow.vectorFromArray(['a', 'b', 'c'], new arrow.Utf8());+ const dictionaryType = new arrow.Dictionary(dictionary.type, new arrow.Int8(), 17);+ const column = arrow.makeVector({+ type: new arrow.List(new arrow.Field('item', dictionaryType, true)),+ valueOffsets: new Int32Array([0, 1, 2, 3]),+ child: arrow.makeData({+ type: dictionaryType,+ data: new Int8Array([0, 1, 2]),+ dictionary+ })+ });+ const source = new arrow.Table({left: column, right: column});+ const datasets = await createArrowLiveDatasets(source);+ const state = {...INITIAL_VIS_STATE, datasets};+ const removed = reducer(state, VisStateActions.removeFromDataset('live', [1]));+ const edited = removed.datasets.live.dataContainer.getTable();+ const readValues = table =>+ table.schema.fields.map((_, i) => Array.from(table.getChildAt(i), value => Array.from(value)));+ const expected = [+ [['a'], ['c']],+ [['a'], ['c']]+ ];++ t.notEqual(removed, state, 'deletion succeeds for columns sharing a nested dictionary');+ t.deepEqual(readValues(edited), expected, 'both columns retain their unmatched values');+ for (const batch of edited.batches) {+ t.equal(+ batch.getChildAt(0).data[0].children[0].dictionary,+ batch.getChildAt(1).data[0].children[0].dictionary,+ 'the shared dictionary id retains one dictionary'+ );+ }+ t.deepEqual(+ readValues(arrow.tableFromIPC(arrow.tableToIPC(edited))),+ expected,+ 'IPC preserves both nested dictionary columns'+ );+ t.deepEqual(+ readValues(source),+ [+ [['a'], ['b'], ['c']],+ [['a'], ['b'], ['c']]+ ],+ 'the original Arrow snapshot stays unchanged'+ );+ t.end();+});++test('VisStateUpdater -> Arrow nested deletion keeps exact timestamps and point rendering', async t => {+ for (const [type, values] of [+ [+ new arrow.TimestampNanosecond('UTC'),+ [1700000000000000123n, 1700000000000000456n, 1700000000000000789n]+ ],+ [+ new arrow.TimestampMicrosecond('UTC'),+ [1700000000000123n, 1700000000000456n, 1700000000000789n]+ ]+ ]) {+ for (const listWrapped of [false, true]) {+ const timestamps = arrow.makeVector({type, data: new BigInt64Array(values)});+ const child = listWrapped+ ? arrow.makeVector({+ type: new arrow.List(new arrow.Field('item', type, true)),+ valueOffsets: new Int32Array([0, 1, 2, 3]),+ child: timestamps.data[0]+ })+ : timestamps;+ const history = arrow.makeVector({+ type: new arrow.Struct([new arrow.Field('time', child.type, true)]),+ length: values.length,+ children: child.data+ });+ const source = new arrow.Table({+ lat: arrow.vectorFromArray([37, 38, 39], new arrow.Float64()),+ lng: arrow.vectorFromArray([-122, -123, -124], new arrow.Float64()),+ history+ });+ const datasets = await createArrowLiveDatasets(source);+ const point = new PointLayer({+ id: 'points',+ dataId: 'live',+ isVisible: true,+ ...PointLayer.findDefaultLayerProps(datasets.live).props[0]+ });+ const initial = {+ ...INITIAL_VIS_STATE,+ datasets,+ layers: [point],+ layerData: [{}],+ layerOrder: [point.id]+ };+ const {layer, layerData} = calculateLayerData(point, initial, undefined);+ const state = {...initial, layers: [layer], layerData: [layerData]};+ const renderPoints = current => {+ const dataset = current.datasets.live;+ return current.layers[0]+ .renderLayer({+ data: current.layerData[0],+ dataset,+ idx: 0,+ layerInteraction: {},+ mapState: INITIAL_MAP_STATE,+ gpuFilter: dataset.gpuFilter,+ interactionConfig: current.interactionConfig,+ visible: true,+ layerCallbacks: {}+ })[0]+ .renderLayers();+ };+ t.equal(+ renderPoints(state).reduce((count, rendered) => count + rendered.props.data.length, 0),+ 3,+ 'the original point layer renders all rows'+ );++ const removed = reducer(state, VisStateActions.removeFromDataset('live', [1]));+ const edited = removed.datasets.live.dataContainer.getTable();+ const rawTimestamps = table =>+ listWrapped+ ? Array.from(table.getChildAt(2), value => value.time.toArray()[0])+ : Array.from(+ new arrow.Vector(table.getChildAt(2).data.map(data => data.children[0])).toArray()+ );+ t.notEqual(removed, state, `${history.type} deletion succeeds`);+ t.equal(edited.numRows, 2, 'only the requested row is deleted');+ t.deepEqual(+ rawTimestamps(edited),+ [values[0], values[2]],+ 'retained timestamp storage is exact'+ );+ t.deepEqual(+ rawTimestamps(arrow.tableFromIPC(arrow.tableToIPC(edited))),+ [values[0], values[2]],+ 'IPC preserves exact nested timestamp storage'+ );+ t.deepEqual(rawTimestamps(source), values, 'the original timestamp snapshot stays unchanged');+ let rendered;+ t.doesNotThrow(() => {+ rendered = renderPoints(removed);+ }, 'the ordinary lat/lng point layer still renders after deletion');+ if (rendered) {+ t.equal(+ rendered.reduce((count, sublayer) => count + sublayer.props.data.length, 0),+ 2,+ 'the rendered point layer contains both retained rows'+ );+ }+ }+ }+ t.end();+});++test('VisStateUpdater -> Arrow Int64 upserts match JSON keys and reject invalid batches', async t => {+ const largeId = 9007199254740993n;+ const source = new arrow.Table({+ id: arrow.vectorFromArray([1n, largeId], new arrow.Int64()),+ value: arrow.vectorFromArray([1, 2], new arrow.Int32())+ });+ const datasets = await createArrowLiveDatasets(source);+ let state = {...INITIAL_VIS_STATE, datasets};+ state = reducer(+ state,+ VisStateActions.addToDataset('live', [{id: String(largeId), value: 20}], {upsertBy: 'id'})+ );+ const container = state.datasets.live.dataContainer;+ t.deepEqual(+ container.flattenData(),+ [+ [1n, 1],+ [largeId, 20]+ ],+ 'a decimal string replaces its exact existing Int64 key without adding a duplicate'+ );++ state = reducer(+ state,+ VisStateActions.addToDataset(+ 'live',+ [+ {id: '01', value: 11},+ {id: '2', value: 21},+ {id: 1n, value: 12},+ {id: '02', value: 22},+ {id: '1', value: 13},+ {id: 2n, value: 23}+ ],+ {upsertBy: 'id'}+ )+ );+ const expected = [+ [1n, 13],+ [largeId, 20],+ [2n, 23]+ ];+ t.deepEqual(+ container.flattenData(),+ expected,+ 'accepted key representations share last-wins matching'+ );+ const restored = arrow.tableFromIPC(arrow.tableToIPC(container.getTable()));+ t.deepEqual(+ Array.from(restored.getChildAt(0)),+ [1n, largeId, 2n],+ 'IPC keeps exact unique Int64 keys'+ );++ const snapshot = container.getTable();+ const beforeRejectedRows = container.flattenData();+ const revision = state.datasets.live.dataRevision;+ const indexes = state.datasets.live.allIndexes;+ const warn = sinon.stub(Console, 'warn');+ try {+ const rejected = reducer(+ state,+ VisStateActions.addToDataset(+ 'live',+ [+ {id: String(largeId), value: 30},+ {id: 'invalid-int64', value: 40}+ ],+ {upsertBy: 'id'}+ )+ );+ t.equal(rejected, state, 'an invalid key rejects the entire public-action batch');+ t.equal(container.getTable(), snapshot, 'a rejected batch retains the Arrow snapshot');+ t.deepEqual(+ container.flattenData(),+ beforeRejectedRows,+ 'a rejected batch cannot leak an earlier replacement'+ );+ t.equal(state.datasets.live.dataRevision, revision, 'a rejected batch preserves the revision');+ t.equal(state.datasets.live.allIndexes, indexes, 'a rejected batch preserves row indexes');+ } finally {+ warn.restore();+ }+ t.deepEqual(Array.from(source.getChildAt(0)), [1n, largeId], 'source keys remain unchanged');+ t.deepEqual(Array.from(source.getChildAt(1)), [1, 2], 'source values remain unchanged');+ t.end();+});diff --git a/test/node/utils/data-container-test.js b/test/node/utils/data-container-test.jsindex 14ac06bb2..1161c9481 100644--- a/test/node/utils/data-container-test.js+++ b/test/node/utils/data-container-test.js@@ -190,6 +190,7 @@ test('RowDataContainer.append and remove', t => {t.equal(dc.append([[1]]), false, 'should reject a wrong column count');t.equal(dc.append([]), false, 'should reject an empty batch');+ t.equal(dc.append(null), false, 'should reject a missing batch');t.equal(dc.numRows(), 5, 'should leave rows unchanged after a rejected append');t.equal(dc.replace(0, [11, 22]), true, 'should overwrite a row in place');@@ -423,6 +424,15 @@ test('ArrowDataContainer -> worker GeoArrow cache follows progressive chunks', t'large compacted centroid caches should keep every center');+ t.equal(dc.remove([0]), true, 'should remove a row from the cached table');+ t.equal(dc.valueAt(0, 0), 2, 'should retain only the second row');+ t.equal(dc.getGeoArrowBinaryData(0), undefined, 'row deletion should discard cached geometry');++ dc.update(batchA, {geoArrowBinaryData: {0: {0: cachedA}}});+ t.equal(dc.remove([0]), true, 'should remove the final cached row');+ t.equal(dc.numRows(), 0, 'should leave an empty table');+ t.equal(dc.getGeoArrowBinaryData(0), undefined, 'empty tables must not retain cached geometry');+t.end();});@@ -675,3 +685,497 @@ test('compactArrowTable -> Builder failure keeps the original table', t => {t.end();});++function arrowContainerFromTable(table) {+ const cols = Array.from({length: table.numCols}, (_, i) => table.getChildAt(i)).filter(Boolean);+ return new ArrowDataContainer({+ cols,+ fields: table.schema.fields.map((field, fieldIdx) => ({+ name: field.name,+ fieldIdx+ })),+ arrowTable: table+ });+}++test('ArrowDataContainer.append replace and remove', t => {+ const dc = arrowContainerFromTable(+ arrow.tableFromJSON([+ {id: 'a', lat: 1},+ {id: 'b', lat: 2}+ ])+ );++ t.equal(+ dc.append([+ ['c', 3],+ ['d', 4]+ ]),+ true,+ 'should append matching primitive/dictionary rows'+ );+ t.equal(dc.numRows(), 4, 'should grow without replacing existing rows');+ t.equal(dc.valueAt(0, 0), 'a', 'should keep the original first id');+ t.equal(dc.valueAt(3, 1), 4, 'should read the last appended lat');+ t.equal(dc.numChunks(), 1, 'append should compact concat batches');++ t.equal(dc.append([[1]]), false, 'should reject a wrong column count');+ t.equal(dc.append([]), false, 'should reject an empty batch');+ t.equal(dc.append(null), false, 'should reject a missing batch');+ t.equal(dc.numRows(), 4, 'should leave rows unchanged after a rejected append');++ t.equal(dc.replace(1, ['B', 22]), true, 'should overwrite a row by concat');+ t.equal(dc.valueAt(1, 0), 'B', 'should read the replaced id');+ t.equal(dc.valueAt(1, 1), 22, 'should read the replaced lat');+ t.equal(dc.replace(99, ['z', 1]), false, 'should reject a replace with an out-of-range index');+ t.equal(dc.replace(0, ['z']), false, 'should reject a replace with the wrong column count');+ t.equal(dc.valueAt(0, 0), 'a', 'should leave the row unchanged after a rejected replace');++ t.equal(dc.remove([1, 1, 3]), true, 'should drop unique valid indexes in one rebuild');+ t.equal(dc.numRows(), 2, 'should keep the remaining rows');+ t.equal(dc.valueAt(0, 0), 'a', 'should keep the former row 0');+ t.equal(dc.valueAt(1, 0), 'c', 'should keep the former row 2');+ t.deepEqual(dc.getPlainIndex(), [0, 1], 'should rebuild indexes after remove');++ t.equal(dc.remove([99]), false, 'should reject an out-of-range index');+ t.equal(dc.remove([-1]), false, 'should reject a negative index');+ t.equal(dc.numRows(), 2, 'should leave rows unchanged after a rejected remove');++ t.equal(dc.remove([0, 1]), true, 'should allow removing every remaining row');+ t.equal(dc.numRows(), 0, 'should keep an empty table with the original schema');+ t.equal(+ dc.append([['e', 5]]),+ true,+ 'empty table should accept an append that matches the schema'+ );+ t.equal(dc.numRows(), 1, 'empty table should grow from the append');+ t.equal(dc.valueAt(0, 0), 'e', 'appended id should survive an empty-table concat');++ t.end();+});++test('ArrowDataContainer row edits preserve column positions and schema', t => {+ for (const names of [+ ['2', '1'],+ ['x', 'x', 'x_1'],+ ['__proto__', 'value']+ ]) {+ const original = names.map((_, i) => i + 1);+ const added = original.map(value => value * 10);+ const replaced = original.map(value => value * 100);+ const columns = original.map((value, i) =>+ arrow.vectorFromArray([value], i % 2 ? new arrow.Float64() : new arrow.Int32())+ );+ const fields = names.map(+ (name, i) => new arrow.Field(name, columns[i].type, true, new Map([['index', String(i)]]))+ );+ const schema = new arrow.Schema(fields, new Map([['source', 'row-edit']]));+ const batch = new arrow.RecordBatch(+ schema,+ arrow.makeData({+ type: new arrow.Struct(fields),+ length: 1,+ children: columns.map(column => column.data[0])+ })+ );+ fields.forEach((field, i) => {+ batch.schema.fields[i] = field;+ });+ const dc = arrowContainerFromTable(new arrow.Table(schema, [batch]));++ t.ok(dc.append([added]), `append accepts ${names.join(', ')}`);+ t.deepEqual(dc.flattenData(), [original, added], 'append keeps column values aligned');+ t.ok(dc.replace(0, replaced), 'replace succeeds');+ t.deepEqual(dc.flattenData(), [replaced, added], 'replace keeps every column');+ t.ok(dc.remove([0]), 'remove succeeds');+ t.deepEqual(dc.flattenData(), [added], 'remove keeps the remaining values');+ t.deepEqual(dc.getTable().schema, schema, 'names, types and metadata survive row edits');+ }+ t.end();+});++test('ArrowDataContainer.append rejects nested and geoarrow columns', t => {+ const originalWarn = console.warn;+ console.warn = () => {};++ try {+ const listTable = new arrow.Table({+ vals: arrow.vectorFromArray([+ [1, 2],+ [3, 4]+ ])+ });+ const listDc = arrowContainerFromTable(listTable);+ t.equal(listDc.append([[[5, 6]]]), false, 'should reject a List column');+ t.equal(listDc.numRows(), 2, 'rejected append must not mutate a List table');+ t.equal(listDc.remove([0]), true, 'remove should still concat slices of nested columns');+ t.equal(listDc.numRows(), 1, 'should drop the nested row by slice');++ const geoMeta = new Map();+ geoMeta.set(GEOARROW_METADATA_KEY, GEOARROW_EXTENSIONS.WKB);+ const wkbTable = tableFromColumnValues(+ 'geometry',+ new arrow.Binary(),+ [new Uint8Array([1, 2, 3])],+ geoMeta+ );+ const geoDc = arrowContainerFromTable(wkbTable);+ t.equal(geoDc.append([[new Uint8Array([4, 5, 6])]]), false, 'should reject a geoarrow column');+ t.equal(geoDc.numRows(), 1, 'rejected append must not mutate a geoarrow table');+ } finally {+ console.warn = originalWarn;+ }+ t.end();+});++test('Arrow row edits preserve retained raw timestamps through compaction', t => {+ for (const [type, values] of [+ [+ new arrow.TimestampMicrosecond('UTC'),+ [9007199254740993n, 9007199254740995n, 9007199254740997n]+ ],+ [+ new arrow.TimestampNanosecond('UTC'),+ [1700000000000000123n, 1700000000000000456n, 1700000000000000789n]+ ]+ ]) {+ const timestamp = arrow.makeVector({+ type,+ data: new BigInt64Array([0n, ...values]),+ nullBitmap: new Uint8Array([0b1011]),+ nullCount: 1+ });+ const source = new arrow.Table({+ id: arrow.vectorFromArray(['skip', 'a', 'b', 'c'], new arrow.Utf8()),+ time: timestamp+ }).slice(1);+ const expected = [values[0], null, values[2]];+ const raw = table => {+ const column = table.getChildAt(1);+ const storage = column.toArray();+ return Array.from({length: column.length}, (_, i) => (column.isValid(i) ? storage[i] : null));+ };+ for (const [name, mutate, retained] of [+ ['append', dc => dc.append([['d', 0]]), [...expected, 0n]],+ ['delete middle', dc => dc.remove([1]), [values[0], values[2]]]+ ]) {+ const dc = arrowContainerFromTable(source);+ t.ok(mutate(dc), `${type} ${name} succeeds`);+ t.deepEqual(raw(dc.getTable()), retained, `${type} ${name} retains exact raw values`);+ t.deepEqual(+ raw(arrow.tableFromIPC(arrow.tableToIPC(dc.getTable()))),+ retained,+ 'IPC retains raw storage'+ );+ t.deepEqual(raw(source), expected, 'original snapshot is unchanged');+ t.equal(dc.numChunks(), 1, 'row edits respect the compaction limit');+ }+ }+ t.end();+});++test('Arrow row edits preserve shared dictionary serialization', t => {+ const column = arrow.vectorFromArray(+ ['a', null, 'b'],+ new arrow.Dictionary(new arrow.Utf8(), new arrow.Int8(), 17)+ );+ const fields = ['left', '2020'].map(+ name => new arrow.Field(name, column.type, true, new Map([['source', name]]))+ );+ const schema = new arrow.Schema(fields, new Map([['source', 'shared-dictionary']]));+ const source = new arrow.Table(schema, {0: column, 1: column});+ const dc = arrowContainerFromTable(source);+ t.ok(dc.append([['c', 'd']]), 'append succeeds with separate incoming dictionary values');+ t.deepEqual(+ dc.flattenData(),+ [+ ['a', 'a'],+ [null, null],+ ['b', 'b'],+ ['c', 'd']+ ],+ 'values stay aligned'+ );+ const assertIPC = expected => {+ let restored;+ t.doesNotThrow(() => {+ restored = arrow.tableFromIPC(arrow.tableToIPC(dc.getTable()));+ }, 'edited table is serializable');+ if (restored) {+ t.deepEqual(+ restored.schema.fields.map((_, i) => Array.from(restored.getChildAt(i))),+ expected,+ 'IPC preserves every dictionary value'+ );+ }+ t.equal(+ dc.getTable().schema.metadata.get('source'),+ 'shared-dictionary',+ 'table metadata survives'+ );+ t.deepEqual(+ dc.getTable().schema.fields.map(field => field.metadata.get('source')),+ ['left', '2020'],+ 'field metadata survives'+ );+ };+ assertIPC([+ ['a', null, 'b', 'c'],+ ['a', null, 'b', 'd']+ ]);+ t.ok(dc.remove([0, 1, 2]), 'deletion succeeds');+ assertIPC([['c'], ['d']]);+ t.ok(dc.remove([0]), 'delete all succeeds');+ t.ok(dc.append([['e', 'f']]), 'refill succeeds');+ assertIPC([['e'], ['f']]);+ t.deepEqual(Array.from(source.getChildAt(0)), ['a', null, 'b'], 'source values stay untouched');+ t.doesNotThrow(() => arrow.tableToIPC(source), 'source remains serializable');+ t.end();+});++test('Arrow row edits reject dictionary index overflow atomically', t => {+ for (const [Index, capacity] of [+ [arrow.Int8, 128],+ [arrow.Uint8, 256]+ ]) {+ const type = new arrow.Dictionary(new arrow.Utf8(), new Index());+ for (const empty of [false, true]) {+ const column = arrow.vectorFromArray(empty ? [] : ['original'], type);+ const source = new arrow.Table({value: column});+ const dc = arrowContainerFromTable(source);+ const snapshot = dc.getTable();+ const rows = Array.from({length: capacity + 1}, (_, i) => [`value-${i}`]);+ t.notOk(dc.append(rows), 'oversized incoming dictionary is rejected before indexes wrap');+ t.equal(dc.getTable(), snapshot, 'rejection retains the snapshot');+ t.deepEqual(dc.flattenData(), empty ? [] : [['original']], 'rejection changes no values');+ }+ const initial = Array.from({length: capacity}, (_, i) => `value-${i}`);+ const dc = arrowContainerFromTable(+ new arrow.Table({value: arrow.vectorFromArray(initial, type)})+ );+ const snapshot = dc.getTable();+ t.notOk(dc.append([['overflow']]), 'combined dictionary overflow is rejected');+ t.equal(dc.getTable(), snapshot, 'combined overflow is atomic');+ }+ t.end();+});++test('Arrow deletion preserves nested timestamp storage without decoding it', t => {+ const values = [1700000000000000123n, 1700000000000000456n, 1700000000000000789n];+ const time = arrow.makeVector({+ type: new arrow.TimestampNanosecond(),+ data: new BigInt64Array(values)+ });+ const type = new arrow.Struct([new arrow.Field('time', time.type, true)]);+ const nested = arrow.makeVector(arrow.makeData({type, length: 3, children: time.data}));+ const source = new arrow.Table({nested});+ const dc = arrowContainerFromTable(source);+ t.ok(dc.remove([1]), 'nested deletion succeeds');+ const raw = table =>+ Array.from(new arrow.Vector(table.getChildAt(0).data.map(data => data.children[0])).toArray());+ t.deepEqual(raw(dc.getTable()), [values[0], values[2]], 'nested raw values are exact');+ t.deepEqual(+ raw(arrow.tableFromIPC(arrow.tableToIPC(dc.getTable()))),+ [values[0], values[2]],+ 'nested IPC is exact'+ );+ t.deepEqual(raw(source), values, 'original nested buffers are unchanged');+ t.end();+});++test('Arrow shared timestamp dictionaries preserve raw values and IPC after edits', t => {+ const type = new arrow.Dictionary(new arrow.TimestampNanosecond('UTC'), new arrow.Int8(), 17);+ const timestamp = arrow.makeVector({+ type: type.dictionary,+ data: new BigInt64Array([1700000000000000123n, 1700000000000000456n])+ });+ const data = arrow.makeData({type, data: new Int8Array([0, 1]), dictionary: timestamp});+ const column = arrow.makeVector(data);+ const source = new arrow.Table(+ new arrow.Schema(['left', 'right'].map(name => new arrow.Field(name, type, true))),+ {0: column, 1: column}+ );+ const dc = arrowContainerFromTable(source);+ t.ok(dc.append([[0, 1000]]), 'shared timestamp append succeeds');+ const ipc = arrow.tableFromIPC(arrow.tableToIPC(dc.getTable()));+ for (let i = 0; i < 2; i++) {+ const col = ipc.getChildAt(i);+ const raw = col.data.flatMap(chunk =>+ Array.from(+ chunk.values.slice(0, chunk.length),+ index => chunk.dictionary.data[0].values[index]+ )+ );+ t.deepEqual(+ raw,+ [1700000000000000123n, 1700000000000000456n, i ? 1000000000n : 0n],+ 'shared dictionary keeps exact raw timestamp entries'+ );+ }+ t.equal(+ dc.getColumn(0).data[0].dictionary,+ dc.getColumn(1).data[0].dictionary,+ 'output columns share the same dictionary'+ );+ t.doesNotThrow(() => arrow.tableToIPC(source), 'source dictionary stays serializable');+ t.end();+});++test('Arrow shared dictionaries retain differing physical index widths', t => {+ const dictionary = arrow.vectorFromArray(['a', 'b'], new arrow.Utf8());+ const columns = [new arrow.Int8(), new arrow.Int16()].map(indices =>+ arrow.makeVector({+ type: new arrow.Dictionary(dictionary.type, indices, 17),+ data: new indices.ArrayType([0, 1]),+ dictionary+ })+ );+ const source = new arrow.Table({narrow: columns[0], wide: columns[1]});+ const dc = arrowContainerFromTable(source);+ t.ok(dc.append([['c', 'd']]), 'mixed-width append succeeds');+ t.ok(dc.remove([0]), 'mixed-width deletion succeeds');+ const ipc = arrow.tableFromIPC(arrow.tableToIPC(dc.getTable()));+ t.deepEqual(Array.from(ipc.getChildAt(0)), ['b', 'c'], 'narrow values survive IPC');+ t.deepEqual(Array.from(ipc.getChildAt(1)), ['b', 'd'], 'wide values survive IPC');+ t.deepEqual(+ ipc.schema.fields.map(field => field.type.indices.bitWidth),+ [8, 16],+ 'physical index widths are preserved'+ );+ t.equal(+ dc.getColumn(0).data[0].dictionary,+ dc.getColumn(1).data[0].dictionary,+ 'both columns retain one shared dictionary'+ );+ t.deepEqual(Array.from(source.getChildAt(0)), ['a', 'b'], 'source stays unchanged');+ t.end();+});++test('Arrow deletion retains timestamp storage inside nested dictionaries', t => {+ const time = new arrow.TimestampNanosecond('UTC');+ const type = new arrow.List(new arrow.Field('item', time));+ const values = [1234567890123456789n, 1234567890123456791n];+ const dictionary = arrow.makeVector({+ type,+ valueOffsets: new Int32Array([0, 1, 2]),+ child: arrow.makeData({type: time, data: new BigInt64Array(values)})+ });+ const column = arrow.makeVector({+ type: new arrow.Dictionary(type, new arrow.Int8(), 17),+ data: new Int8Array([0, 1, 0, 1]),+ dictionary+ });+ const source = new arrow.Table({nested: column});+ const dc = arrowContainerFromTable(source);+ t.ok(dc.remove([1]), 'nested dictionary deletion succeeds');+ const ipc = arrow.tableFromIPC(arrow.tableToIPC(dc.getTable()));+ t.deepEqual(+ Array.from(ipc.getChildAt(0), value => value.toArray()[0]),+ [values[0], values[0], values[1]],+ 'nested dictionary IPC keeps exact timestamp values'+ );+ t.deepEqual(+ Array.from(source.getChildAt(0), value => value.toArray()[0]),+ [values[0], values[1], values[0], values[1]],+ 'source nested dictionary is unchanged'+ );+ t.end();+});++test('Arrow deletion reconciles nested dictionary mappings across progressive batches', t => {+ const batch = values => {+ const dictionary = arrow.vectorFromArray(values, new arrow.Utf8());+ const type = new arrow.Dictionary(dictionary.type, new arrow.Int8(), 17);+ const column = indexes =>+ arrow.makeVector({+ type: new arrow.List(new arrow.Field('item', type, true)),+ valueOffsets: new Int32Array([0, 1, 2, 3]),+ child: arrow.makeData({type, data: new Int8Array(indexes), dictionary})+ });+ return new arrow.Table({left: column([0, 1, 0]), right: column([1, 0, 1])});+ };+ const first = batch(['a', 'b']);+ const source = first.concat(batch(['b', 'a']));+ const before = arrow.tableToIPC(source);+ const dc = arrowContainerFromTable(first);+ dc.update(source);+ t.ok(dc.remove([1, 4]), 'deletion accepts different dictionaries in progressive batches');+ const rows = table =>+ Array.from({length: table.numRows}, (_, index) =>+ table.schema.fields.map((_, column) => Array.from(table.getChildAt(column).get(index)))+ );+ const expected = [+ [['a'], ['b']],+ [['a'], ['b']],+ [['b'], ['a']],+ [['b'], ['a']]+ ];+ t.deepEqual(rows(dc.getTable()), expected, 'retained indexes use the unified dictionary');+ t.equal(dc.numChunks(), 1, 'edited table has one batch');+ t.deepEqual(+ rows(arrow.tableFromIPC(arrow.tableToIPC(dc.getTable()))),+ expected,+ 'shared dictionary survives IPC despite equivalent distinct source type objects'+ );+ t.deepEqual(arrow.tableToIPC(source), before, 'progressive source buffers remain unchanged');+ t.end();+});++test('compactArrowTable -> nested timestamp fallback keeps exact source storage', t => {+ const type = new arrow.TimestampNanosecond();+ const nested = new arrow.List(new arrow.Field('time', type, true));+ const raw = [1700000000000000123n, 1700000000000000456n];+ const batches = raw.map(+ value =>+ new arrow.Table({+ times: arrow.makeVector({+ type: nested,+ valueOffsets: new Int32Array([0, 1]),+ child: arrow.makeData({type, data: new BigInt64Array([value])})+ })+ })+ );+ const source = batches[0].concat(batches[1]);+ const compacted = compactArrowTable(source, 1);+ t.deepEqual(+ compacted.getChildAt(0).data.flatMap(data => Array.from(data.children[0].values)),+ raw,+ 'ordinary import compaction must not round nested timestamps through milliseconds'+ );+ t.end();+});++test('Arrow deletion counts only retained nested dictionary entries toward capacity', t => {+ const batch = prefix => {+ const dictionary = arrow.vectorFromArray(+ Array.from({length: 128}, (_, i) => `${prefix}${i}`),+ new arrow.Utf8()+ );+ const type = new arrow.Dictionary(dictionary.type, new arrow.Int8(), 17);+ const column = arrow.makeVector({+ type: new arrow.List(new arrow.Field('item', type, true)),+ valueOffsets: Int32Array.from({length: 129}, (_, i) => i),+ child: arrow.makeData({+ type,+ data: Int8Array.from({length: 128}, (_, i) => i),+ dictionary+ })+ });+ return new arrow.Table({values: column});+ };+ const first = batch('a');+ const source = first.concat(batch('b'));+ const original = arrow.tableToIPC(source);+ const dc = arrowContainerFromTable(first);+ dc.update(source);+ const indexes = Array.from({length: 256}, (_, i) => i).filter(i => i !== 0 && i !== 128);+ t.ok(dc.remove(indexes), 'two retained dictionary values fit Int8 despite deleted entries');+ const restored = arrow.tableFromIPC(arrow.tableToIPC(dc.getTable()));+ t.deepEqual(+ Array.from(restored.getChildAt(0), value => Array.from(value)),+ [['a0'], ['b0']],+ 'only the retained values survive IPC'+ );+ t.deepEqual(arrow.tableToIPC(source), original, 'source dictionaries are unchanged');+ t.end();+});diff --git a/test/node/utils/duckdb-utils-test.js b/test/node/utils/duckdb-utils-test.jsindex c59e29147..4e714cae5 100644--- a/test/node/utils/duckdb-utils-test.js+++ b/test/node/utils/duckdb-utils-test.js@@ -3,6 +3,7 @@import test from 'tape';import * as arrow from 'apache-arrow';+import Console from 'global/console';import {SUPPORTED_DUCKDB_DROP_EXTENSIONS,@@ -23,6 +24,7 @@ import {sanitizeDuckDBTableName,quoteTableName} from '../../../src/duckdb/src/table/duckdb-table-utils';+import {KeplerGlDuckDbTable} from '../../../src/duckdb/src/table/duckdb-table';import {GEOARROW_EXTENSIONS, GEOARROW_METADATA_KEY} from '@kepler.gl/constants';@@ -628,3 +630,33 @@ test('duckdb-utils -> SUPPORTED_DUCKDB_DROP_EXTENSIONS', t => {t.end();});++test('KeplerGlDuckDbTable -> row edits stay a warned no-op', t => {+ const snapshot = arrow.tableFromJSON([{id: 'a', value: 1}]);+ const table = new KeplerGlDuckDbTable({info: {id: 'duck', label: 'duck'}}).updateSchema({+ fields: [+ {name: 'id', type: 'string'},+ {name: 'value', type: 'real'}+ ],+ cols: snapshot.schema.fields.map((_, i) => snapshot.getChildAt(i)),+ arrowTable: snapshot,+ rows: []+ });+ const originalWarn = Console.warn;+ const warnings = [];+ Console.warn = message => warnings.push(String(message));++ try {+ t.equal(table.appendRows([['b', 2]]), false, 'appendRows should no-op');+ t.equal(table.upsertRows([['a', 2]], 'id'), false, 'upsertRows should no-op');+ t.equal(table.removeRows([0]), false, 'removeRows should no-op');+ t.deepEqual(table.dataContainer.flattenData(), [['a', 1]], 'snapshot stays unchanged');+ t.ok(+ warnings.some(message => message.includes('DuckDB INSERT')),+ 'should warn that DuckDB INSERT is not implemented'+ );+ } finally {+ Console.warn = originalWarn;+ }+ t.end();+});diff --git a/test/node/utils/kepler-table-test.js b/test/node/utils/kepler-table-test.jsindex 7aa6b5a20..d5025bb3a 100644--- a/test/node/utils/kepler-table-test.js+++ b/test/node/utils/kepler-table-test.js@@ -671,6 +671,378 @@ test('KeplerTable -> Arrow snapshots rebuild cached timestamps and reuse them bet.end();});+test('KeplerTable -> Arrow row edits and snapshots keep timestamp filters current', async t => {+ const firstTime = Date.UTC(2024, 1, 1);+ const lastTime = firstTime + 86400000;+ const snapshot = rows => {+ const source = new arrow.Table({+ id: arrow.vectorFromArray(+ rows.map(row => row[0]),+ new arrow.Utf8()+ ),+ time: arrow.vectorFromArray(+ rows.map(row => row[1]),+ new arrow.TimestampMillisecond()+ )+ });+ return processArrowBatches(source.batches);+ };+ const table = new KeplerTable({info: {id: 'row-edit-timestamps'}, color: [0, 0, 0]});+ await table.importData({+ data: snapshot([+ ['a', firstTime],+ ['b', lastTime]+ ])+ });+ const container = table.dataContainer;+ const accessor = table.fields[1].valueAccessor;+ const filters = [+ {+ ...table.getColumnFilterProps('time'),+ id: 'time',+ dataId: [table.id],+ name: ['time'],+ fieldIdx: [1],+ value: [firstTime, firstTime],+ enabled: true,+ gpu: true,+ gpuChannel: [0]+ }+ ];+ const check = (values, expected) => {+ t.equal(table.dataContainer, container, 'container identity survives edits and refresh');+ t.deepEqual(table.fields[1].filterProps.mappedValue, values, 'timestamp cache follows rows');+ t.deepEqual(+ values.map((_, index) => accessor({index})),+ values,+ 'original accessor stays bound'+ );+ table.filterTable(filters, []);+ table.filterTableCPU(filters, []);+ t.deepEqual(table.filteredIdxCPU, expected, 'CPU/export filter follows edited timestamps');+ const getValue = table.gpuFilter.filterValueAccessor(container)();+ const visible = table.filteredIndex.filter(index =>+ getValue({index}).every((value, channel) => {+ const [min, max] = table.gpuFilter.filterRange[channel];+ return value >= min && value <= max;+ })+ );+ t.deepEqual(visible, expected, 'GPU filter follows edited timestamps');+ };+ check([firstTime, lastTime], [0]);+ t.ok(table.appendRows([['c', firstTime]]), 'append succeeds');+ check([firstTime, lastTime, firstTime], [0, 2]);+ t.ok(+ table.upsertRows(+ [+ ['a', lastTime],+ ['b', firstTime]+ ],+ 'id'+ ),+ 'upsert succeeds'+ );+ check([lastTime, firstTime, firstTime], [1, 2]);+ t.ok(table.removeRows([1]), 'delete succeeds');+ check([lastTime, firstTime], [1]);+ await table.update(+ snapshot([+ ['c', firstTime],+ ['a', lastTime],+ ['d', null]+ ])+ );+ check([firstTime, lastTime, null], [0]);+ t.ok(table.removeRows([0, 1, 2]), 'delete all succeeds');+ check([], []);+ t.ok(table.appendRows([['e', firstTime]]), 'append after empty succeeds');+ check([firstTime], [0]);+ t.end();+});++test('KeplerTable -> Arrow upserts match and deduplicate encoded Int64 keys', t => {+ const existingKey = 9007199254740993n;+ const newKey = existingKey + 1n;+ const source = new arrow.Table({+ id: arrow.vectorFromArray([existingKey], new arrow.Int64()),+ value: arrow.vectorFromArray([10], new arrow.Int32())+ });+ const table = new KeplerTable({info: {id: 'encoded-keys'}, color: [0, 0, 0]}).updateSchema(+ processArrowBatches(source.batches)+ );+ const container = table.dataContainer;+ const snapshot = container.getTable();+ const accessor = table.fields[1].valueAccessor;+ const revision = table.dataRevision;++ t.ok(+ table.upsertRows([[String(existingKey), 20]], 'id'),+ 'accepts the JSON decimal-string representation of an Int64 key'+ );+ t.deepEqual(container.flattenData(), [[existingKey, 20]], 'replaces the existing encoded key');+ t.ok(+ table.upsertRows(+ [+ [String(existingKey), 30],+ [String(newKey), 40],+ [existingKey, 50],+ [newKey, 60]+ ],+ 'id'+ ),+ 'accepts equivalent string and BigInt keys in one edit'+ );+ t.deepEqual(+ container.flattenData(),+ [+ [existingKey, 50],+ [newKey, 60]+ ],+ 'last equivalent key wins for both replacements and appended rows'+ );+ t.equal(table.dataContainer, container, 'retains the container');+ t.equal(accessor({index: 0}), 50, 'the original accessor reads the replacement');+ t.equal(snapshot.getChild('value').get(0), 10, 'retains the previous Arrow snapshot');+ t.equal(table.dataRevision, revision + 2, 'increments the revision once per accepted edit');++ const currentSnapshot = container.getTable();+ const currentIndexes = table.allIndexes;+ const currentRevision = table.dataRevision;+ for (const invalidKey of ['invalid-int64', '9223372036854775808', '-9223372036854775809']) {+ t.notOk(+ table.upsertRows(+ [+ [String(existingKey), 70],+ [invalidKey, 80]+ ],+ 'id'+ ),+ 'an invalid or out-of-range later key rejects the entire edit'+ );+ t.equal(container.getTable(), currentSnapshot, 'rejected keys retain the Arrow snapshot');+ t.equal(table.allIndexes, currentIndexes, 'rejected keys retain the indexes');+ t.equal(table.dataRevision, currentRevision, 'rejected keys retain the revision');+ t.equal(accessor({index: 0}), 50, 'no replacement leaks from the rejected edit');+ }+ t.end();+});++test('KeplerTable -> Arrow upserts reject invalid integer keys before matching', t => {+ const uint64Max = (1n << 64n) - 1n;+ for (const {type, existingKeys, invalidKeys} of [+ {+ type: new arrow.Int8(),+ existingKeys: [-128, 127],+ invalidKeys: [128, '128', -129, 1.5, NaN, Infinity]+ },+ {+ type: new arrow.Uint64(),+ existingKeys: [0n, uint64Max],+ invalidKeys: [uint64Max + 1n, String(uint64Max + 1n), -1n, '-1']+ }+ ]) {+ const source = new arrow.Table({+ id: arrow.vectorFromArray(existingKeys, type),+ value: arrow.vectorFromArray([10, 20], new arrow.Int32())+ });+ const table = new KeplerTable({info: {id: 'integer-keys'}, color: [0, 0, 0]}).updateSchema(+ processArrowBatches(source.batches)+ );+ t.ok(+ table.upsertRows([[String(existingKeys[1]), 30]], 'id'),+ `accepts a numeric string at the ${type} upper boundary`+ );+ const expected = [+ [existingKeys[0], 10],+ [existingKeys[1], 30]+ ];+ t.deepEqual(table.dataContainer.flattenData(), expected, 'matches the stored boundary key');+ const snapshot = table.dataContainer.getTable();+ const revision = table.dataRevision;+ for (const key of invalidKeys) {+ t.notOk(+ table.upsertRows(+ [+ [existingKeys[0], 40],+ [key, 50]+ ],+ 'id'+ ),+ `rejects invalid ${type} key ${String(key)} before matching`+ );+ t.equal(table.dataContainer.getTable(), snapshot, 'retains the snapshot');+ t.equal(table.dataRevision, revision, 'retains the revision');+ t.deepEqual(table.dataContainer.flattenData(), expected, 'does not overwrite any row');+ }+ }+ t.end();+});++test('KeplerTable -> Arrow upserts match encoded Date and timestamp keys', t => {+ const first = Date.UTC(2024, 0, 1);+ const next = first + 86400000;+ for (const type of [new arrow.DateMillisecond(), new arrow.TimestampMillisecond()]) {+ const source = new arrow.Table({+ time: arrow.vectorFromArray([first], type),+ value: arrow.vectorFromArray([1], new arrow.Int32())+ });+ const table = new KeplerTable({info: {id: 'date-keys'}, color: [0, 0, 0]}).updateSchema(+ processArrowBatches(source.batches)+ );+ t.ok(+ table.upsertRows(+ [+ [new Date(first), 2],+ [new Date(next), 3],+ [next, 4]+ ],+ 'time'+ ),+ `accepts Date keys for ${type}`+ );+ t.deepEqual(+ table.dataContainer.flattenData(),+ [+ [first, 2],+ [next, 4]+ ],+ 'matches existing keys and deduplicates equivalent Date and millisecond keys'+ );+ }+ t.end();+});++test('KeplerTable -> row upserts keep distinct number and string keys', t => {+ const table = new KeplerTable({info: {id: 'row-keys'}, color: [0, 0, 0]}).updateSchema({+ fields: [+ {name: 'id', type: 'string'},+ {name: 'value', type: 'integer'}+ ],+ rows: [[1, 10]]+ });+ t.ok(+ table.upsertRows(+ [+ ['1', 20],+ [1, 30]+ ],+ 'id'+ ),+ 'accepts row keys without encoding'+ );+ t.deepEqual(+ table.dataContainer.flattenData(),+ [+ [1, 30],+ ['1', 20]+ ],+ 'row containers retain exact key equality'+ );+ t.end();+});++test('KeplerTable -> Arrow upserts preserve untouched raw timestamps and nulls', t => {+ const rawValues = vector =>+ vector.data.flatMap(chunk =>+ Array.from({length: chunk.length}, (_, index) =>+ chunk.getValid(index) ? chunk.values[index] : null+ )+ );++ for (const [Timestamp, first, unitsPerMillisecond] of [+ [arrow.TimestampMicrosecond, 9007199254740993n, 1000n],+ [arrow.TimestampNanosecond, 1700000000000000123n, 1000000n]+ ]) {+ const source = new arrow.Table({+ id: arrow.vectorFromArray(['padding', 'a', 'b', 'c', 'd'], new arrow.Utf8()),+ time: arrow.makeVector({+ type: new Timestamp('America/New_York'),+ data: new BigInt64Array([0n, first, first + 2n, first + 4n, 0n]),+ nullBitmap: new Uint8Array([0b01111]),+ nullCount: 1+ })+ }).slice(1);+ const original = [first, first + 2n, first + 4n, null];+ t.ok(source.getChild('time').data[0].offset > 0, 'fixture uses a nonzero Arrow offset');+ t.deepEqual(rawValues(source.getChild('time')), original, 'fixture keeps raw timestamp units');+ const table = new KeplerTable({info: {id: 'raw-timestamps'}, color: [0, 0, 0]}).updateSchema(+ processArrowBatches(source.batches)+ );+ const container = table.dataContainer;+ const accessor = table.fields[1].valueAccessor;+ t.ok(+ table.upsertRows(+ [+ ['new-b', 2000],+ ['a', 1000],+ ['new-a', 3000],+ ['a', 4000],+ ['new-b', 5000]+ ],+ 'id'+ ),+ 'mixed replacement and insertion succeeds'+ );+ const expected = [+ 4000n * unitsPerMillisecond,+ first + 2n,+ first + 4n,+ null,+ 5000n * unitsPerMillisecond,+ 3000n * unitsPerMillisecond+ ];+ t.equal(table.dataContainer, container, 'upsert retains the container');+ t.equal(accessor({index: 0}), 4000, 'the existing accessor reads the last replacement');+ t.deepEqual(+ Array.from(container.getColumn(0)),+ ['a', 'b', 'c', 'd', 'new-b', 'new-a'],+ 'existing positions and first incoming key order survive repeated keys'+ );+ t.deepEqual(+ rawValues(container.getColumn(1)),+ expected,+ 'untouched raw timestamps survive upsert'+ );+ const restored = arrow.tableFromIPC(arrow.tableToIPC(container.getTable()));+ t.deepEqual(+ rawValues(restored.getChild('time')),+ expected,+ 'IPC preserves raw timestamps and nulls'+ );+ t.equal(+ restored.schema.fields[1].type.timezone,+ 'America/New_York',+ 'IPC preserves the timezone'+ );+ t.deepEqual(+ rawValues(source.getChild('time')),+ original,+ 'upsert does not mutate the source snapshot'+ );++ const snapshot = container.getTable();+ const revision = table.dataRevision;+ t.notOk(+ table.upsertRows(+ [+ ['a', 6000],+ ['invalid', 'not-a-timestamp']+ ],+ 'id'+ ),+ 'a later invalid timestamp rejects the whole batch'+ );+ t.equal(container.getTable(), snapshot, 'a rejected upsert retains the Arrow snapshot');+ t.equal(table.dataRevision, revision, 'a rejected upsert does not advance the revision');+ t.deepEqual(+ rawValues(container.getColumn(1)),+ expected,+ 'a rejected upsert changes no raw values'+ );+ }+ t.end();+});+test('KeplerTable -> update DuckDB cols payload keeps fields and gpuFilter', async t => {const first = arrow.tableFromArrays({lat: [1, 2],