Skip to content

Kepler.gl is a powerful open source geospatial analysis tool for large-scale data sets.

feat: in-place row edits for Arrow tables (#3688)

Igor Dykhtaab6cbe3f6eb4
diff --git a/CHANGELOG.md b/CHANGELOG.md
index e8f38d489..9d84b0079 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -15,6 +15,10 @@ Each version should:
Ref: http://keepachangelog.com/en/0.3.0/
-->
+## [Unreleased]
+
+- feat: support in-place append, upsert, and delete for primitive Arrow tables (#3688). DuckDB row edits remain a warned no-op.
+
## [3.3.0-alpha.18] - Oct 19 2026
- c7a6399f fix(export): restore annotation and chart panels (#3853)
diff --git a/docs/api-reference/actions/actions.md b/docs/api-reference/actions/actions.md
index 1f00bcc3c..0e53cd654 100644
--- a/docs/api-reference/actions/actions.md
+++ b/docs/api-reference/actions/actions.md
@@ -528,9 +528,9 @@ Returns **{type: ActionTypes.ADD_LAYER, props: props}**
### addToDataset
-Append or upsert rows on an existing in-memory row dataset without `addDataToMap`. Keeps layer identity and style. Pass `options.upsertBy` to replace rows that share that key and append the rest.
+Append or upsert rows on an existing in-memory row or Arrow dataset without `addDataToMap`. Keeps layer identity and style. Pass `options.upsertBy` to replace rows that share that key and append the rest.
-Not implemented for Arrow or DuckDB tables (no INSERT / concat yet); those calls warn and leave the table unchanged. Use `addDataToMap` with `keepExistingConfig` for a full replace.
+DuckDB tables (no INSERT yet) warn and leave the table unchanged. Arrow tables support primitive columns; nested, binary, and geoarrow columns are rejected. Encoding failures and dictionary index overflow reject the whole edit batch. Use `addDataToMap` with `keepExistingConfig` for a full replace.
- **ActionTypes**: `ActionTypes.ADD_TO_DATASET`
- **Updaters**: `visStateUpdaters.addToDatasetUpdater`
@@ -744,9 +744,9 @@ Returns **{type: ActionTypes.REMOVE_DATASET, key: key}**
### removeFromDataset
-Delete rows from an existing in-memory row dataset without restyling layers. The second argument is either row indexes or `{field, values}` to match a column (for example an id).
+Delete rows from an existing in-memory row or Arrow dataset without restyling layers. The second argument is either row indexes or `{field, values}` to match a column (for example an id).
-Not implemented for Arrow or DuckDB tables; those calls warn and leave the table unchanged.
+DuckDB tables warn and leave the table unchanged. Arrow tables compact the retained rows while preserving nested values, exact timestamp storage, and shared dictionaries.
- **ActionTypes**: `ActionTypes.REMOVE_FROM_DATASET`
- **Updaters**: `visStateUpdaters.removeFromDatasetUpdater`
diff --git a/docs/api-reference/reducers/vis-state.md b/docs/api-reference/reducers/vis-state.md
index 365c0cd48..80446747d 100644
--- a/docs/api-reference/reducers/vis-state.md
+++ b/docs/api-reference/reducers/vis-state.md
@@ -107,7 +107,7 @@ Returns **[Object][69]** nextState
### addToDatasetUpdater
-Append or upsert rows on an existing in-memory row dataset. Keeps layers and re-runs filters. Arrow/DuckDB tables warn and are left unchanged (INSERT / concat not implemented).
+Append or upsert rows on an existing in-memory row or Arrow dataset. Keeps layers and re-runs filters. DuckDB tables warn and are left unchanged (INSERT not implemented). Nested, binary, and geoarrow Arrow columns are rejected.
- **Action**: [`addToDataset`](../actions/actions.md#addtodataset)
@@ -338,7 +338,7 @@ Returns **[Object][69]** nextState
### removeFromDatasetUpdater
-Delete rows by index or by field values from an existing in-memory row dataset. Keeps layers. Arrow/DuckDB tables warn and are left unchanged.
+Delete rows by index or by field values from an existing in-memory row or Arrow dataset. Keeps layers. DuckDB tables warn and are left unchanged.
- **Action**: [`removeFromDataset`](../actions/actions.md#removefromdataset)
diff --git a/examples/live-data/README.md b/examples/live-data/README.md
index d84373aea..9e789a142 100644
--- a/examples/live-data/README.md
+++ b/examples/live-data/README.md
@@ -1,7 +1,7 @@
# Live remotely hosted data
Loads a CORS CSV of points orbiting San Francisco as a remotely hosted dataset
-and **polls it every 300 ms**. The right sidebar switches between three host-app
+and **polls it every 300 ms**. The right sidebar switches between four host-app
update styles:
- **Poll URL** — HTTP snapshot replace (`refreshDataset` /
@@ -15,12 +15,17 @@ update styles:
`ws://localhost:4010/vehicles.ws` and maps each JSON upsert to
`addToDataset(..., {upsertBy: 'id'})`. Kepler has no websocket client and
still rebuilds layer data on every message.
+- **Arrow rows** — pauses polling and opens a separate local Arrow dataset.
+ The same host controls append, upsert and delete through the public actions.
+ Mode changes hide the inactive dataset's layers; Arrow rows and layer/filter
+ settings remain available when you return.
This bundles local monorepo `src/` (not a published npm package). Host-row
-and websocket edits only work for in-memory CSV (`RowDataContainer`), which this
-example uses. Arrow and DuckDB tables log a warning and leave the data unchanged
-(INSERT / concat are not implemented yet). The **Keyed by id** controls
-exercise `addToDataset(id, rows, {upsertBy: 'id'})` and
+and websocket edits use in-memory CSV (`RowDataContainer`); **Arrow rows**
+uses an explicitly typed Arrow table (`ArrowDataContainer`).
+The same actions support primitive Arrow columns; nested, binary, and GeoArrow
+columns cannot be appended or upserted. DuckDB tables remain a warned no-op.
+The **Keyed by id** controls exercise `addToDataset(id, rows, {upsertBy: 'id'})` and
`removeFromDataset(id, {field: 'id', values})`.
## Pre-requirements
@@ -77,6 +82,25 @@ run next to `yarn start` on 8080). The CSV server listens at
pattern: `ws → addToDataset`. It is **not** a Kepler websocket layer and does
**not** skip a full layer-data rebuild.
+### Arrow rows
+
+- Select **Arrow rows**. It starts with 3 local points and requires no CSV or
+ WebSocket connection. Storage should read **ArrowDataContainer · 1 batch(es)**.
+ The first visit fits the map to include the outer rings used by new points.
+- **Add 1 / Add 5** append rows. **Insert tracker**, then **Move tracker**
+ repeatedly: its coordinates and speed change while the row count stays put.
+ **Remove tracker / Remove veh-01 / Remove last host** delete by id;
+ **Remove last / Remove random** delete by index.
+- Change the **Live Arrow** layer radius/color or add a numeric **speed** filter.
+ Repeat the edits and check that the layer id, styling and filter selection
+ remain. The revision counter advances after each successful edit.
+- **Auto trail** adds every 800ms and keeps at most 20 host points. Switching
+ modes turns it off; returning to Arrow keeps the edited rows without restarting
+ the timer. The original CSV dataset keeps its existing mode behavior.
+- The Arrow table is initialized once. Edit controls never call `addDataToMap`
+ or replace its full snapshot. Sample columns are Utf8, Float64 and Int32;
+ nested/binary/GeoArrow append/upsert is deliberately outside this example.
+
## CSV server only (Add Data URL)
From this folder, or the repo root:
diff --git a/examples/live-data/esbuild.config.mjs b/examples/live-data/esbuild.config.mjs
index 75097ea45..05bcb6d92 100644
--- a/examples/live-data/esbuild.config.mjs
+++ b/examples/live-data/esbuild.config.mjs
@@ -57,7 +57,7 @@ const config = {
},
absWorkingDir: __dirname,
loader: {'.js': 'jsx', '.css': 'css'},
- entryPoints: [path.join(__dirname, 'src/app.tsx')],
+ entryPoints: [path.join(__dirname, 'src/index.tsx')],
outfile: path.join(DIST_DIR, 'bundle.js'),
bundle: true,
alias: localKeplerAliases(),
diff --git a/examples/live-data/src/app.tsx b/examples/live-data/src/app.tsx
index 5bfad6be0..140e96ea7 100644
--- a/examples/live-data/src/app.tsx
+++ b/examples/live-data/src/app.tsx
@@ -3,8 +3,7 @@
import * as React from 'react';
import {useCallback, useEffect, useRef, useState} from 'react';
-import ReactDOM from 'react-dom/client';
-import {Provider, useDispatch, useSelector} from 'react-redux';
+import {Provider, useDispatch, useSelector, useStore} from 'react-redux';
import {applyMiddleware, combineReducers, compose, createStore} from 'redux';
import 'maplibre-gl/dist/maplibre-gl.css';
@@ -13,13 +12,16 @@ import KeplerGl from '@kepler.gl/components';
import {
addDataToMap,
addToDataset,
+ fitBounds,
+ layerConfigChange,
refreshDataset,
removeFromDataset,
updateDatasetProps,
wrapTo
} from '@kepler.gl/actions';
import {DatasetType} from '@kepler.gl/constants';
-import {initApplicationConfig} from '@kepler.gl/utils';
+import {ArrowDataContainer, initApplicationConfig} from '@kepler.gl/utils';
+import {ARROW_DATASET_ID, createArrowDataset} from './arrow-data';
initApplicationConfig({enableAnnotations: false});
@@ -39,17 +41,19 @@ const COS_LAT = Math.cos((SF.lat * Math.PI) / 180);
const KM_PER_DEG_LAT = 111.32;
const GOLDEN_ANGLE = Math.PI * (3 - Math.sqrt(5));
-type LiveMode = 'poll' | 'host' | 'ws';
+type LiveMode = 'poll' | 'host' | 'ws' | 'arrow';
type WsStatus = 'idle' | 'connecting' | 'open' | 'closed' | 'error';
type KeplerRootState = {
keplerGl: {
[id: string]: {
visState: {
+ layers: Parameters<typeof layerConfigChange>[0][];
datasets: Record<
string,
{
fields?: {name: string}[];
+ dataRevision?: number;
dataContainer?: {
numRows: () => number;
valueAt: (row: number, col: number) => unknown;
@@ -87,12 +91,12 @@ const enhancers = applyMiddleware(...middlewares);
const composeEnhancers = (window as any).__REDUX_DEVTOOLS_EXTENSION_COMPOSE__ || compose;
const store = createStore(reducers, {}, composeEnhancers(enhancers));
-function fieldValue(dataset: LiveDataset | undefined, name: string) {
+function fieldValue(dataset: LiveDataset | undefined, name: string, row = 0) {
const col = dataset?.fields?.findIndex(field => field.name === name) ?? -1;
if (col < 0 || !dataset?.dataContainer?.numRows()) {
return undefined;
}
- return dataset.dataContainer.valueAt(0, col);
+ return dataset.dataContainer.valueAt(row, col);
}
function columnIndex(dataset: LiveDataset | undefined, name: string) {
@@ -121,7 +125,7 @@ function hostRowIndexes(dataset: LiveDataset | undefined): number[] {
}, []);
}
-function makeHostVehicle(seq: number, options?: {id?: string; ring?: number}) {
+function makeHostVehicle(seq: number, options?: {id?: string; ring?: number; speed?: number}) {
const angle = seq * GOLDEN_ANGLE;
const radiusKm = 5.2 + (seq % 8) * 0.35;
const lat = SF.lat + (radiusKm / KM_PER_DEG_LAT) * Math.sin(angle);
@@ -131,7 +135,7 @@ function makeHostVehicle(seq: number, options?: {id?: string; ring?: number}) {
lat,
lng,
heading: ((angle * 180) / Math.PI + 360) % 360,
- speed: 0,
+ speed: options?.speed ?? 0,
ring: options?.ring ?? 4 + (seq % 3),
progress: 0,
orbit_s: 0,
@@ -184,7 +188,8 @@ function ModeToggle({
return (
<div
style={{
- display: 'flex',
+ display: 'grid',
+ gridTemplateColumns: '1fr 1fr',
marginBottom: 14,
border: '1px solid #4a5160',
borderRadius: 6,
@@ -195,13 +200,15 @@ function ModeToggle({
[
['poll', 'Poll URL', true],
['host', 'Host rows', hostEnabled],
- ['ws', 'WebSocket', hostEnabled]
+ ['ws', 'WebSocket', hostEnabled],
+ ['arrow', 'Arrow rows', true]
] as const
).map(([value, label, enabled]) => (
<button
key={value}
type="button"
disabled={!enabled}
+ aria-pressed={mode === value}
onClick={() => onChange(value)}
style={{
flex: 1,
@@ -281,6 +288,7 @@ function PollSidebar({dataset}: {dataset: LiveDataset | undefined}) {
function HostSidebar({
dataset,
+ arrowMode,
autoTrail,
keyedNote,
onAutoTrail,
@@ -293,6 +301,7 @@ function HostSidebar({
onRemoveLastHostById
}: {
dataset: LiveDataset | undefined;
+ arrowMode: boolean;
autoTrail: boolean;
keyedNote: string;
onAutoTrail: (next: boolean) => void;
@@ -311,6 +320,8 @@ function HostSidebar({
const hasPollVehicle = ids.includes(POLL_VEHICLE_ID);
const hostIds = ids.filter(id => id.startsWith('host-'));
const lastHostId = hostIds[hostIds.length - 1];
+ const trackerIndex = ids.indexOf(TRACKER_ID);
+ const container = dataset?.dataContainer;
return (
<>
@@ -319,6 +330,18 @@ function HostSidebar({
<code>removeFromDataset</code> so rows change in place — same layer id and style. Host
points land on outer rings (4–6) and color by <code>ring</code>.
</p>
+ <p style={{margin: '0 0 12px', color: '#c3c9d5'}}>
+ Storage:{' '}
+ <output aria-label="Data storage">
+ {container instanceof ArrowDataContainer
+ ? `ArrowDataContainer · ${container.numChunks()} batch(es)`
+ : container
+ ? 'RowDataContainer'
+ : 'loading'}
+ </output>
+ {' · Revision '}
+ <output aria-label="Dataset revision">{dataset?.dataRevision ?? 0}</output>
+ </p>
<div style={{display: 'grid', gap: 8, marginBottom: 12}}>
<div style={{display: 'flex', justifyContent: 'space-between', gap: 12}}>
<span style={{color: '#c3c9d5'}}>rows</span>
@@ -389,6 +412,16 @@ function HostSidebar({
{lastHostId ? ` · last host ${lastHostId}` : ''}
</p>
{keyedNote ? <p style={{margin: '6px 0 0', color: '#9ad0a8'}}>{keyedNote}</p> : null}
+ {hasTracker ? (
+ <p style={{margin: '6px 0 0', color: '#c3c9d5'}}>
+ <output aria-label="Tracker position and speed">
+ {Number(fieldValue(dataset, 'lat', trackerIndex)).toFixed(5)},{' '}
+ {Number(fieldValue(dataset, 'lng', trackerIndex)).toFixed(5)}
+ {' · speed '}
+ {String(fieldValue(dataset, 'speed', trackerIndex))}
+ </output>
+ </p>
+ ) : null}
<label
style={{
display: 'flex',
@@ -408,8 +441,17 @@ function HostSidebar({
Auto trail (add every 800ms, keep {MAX_HOST_ROWS} host rows)
</label>
<p style={{margin: '16px 0 0', color: '#c3c9d5'}}>
- Switch back to <strong>Poll URL</strong> to resume HTTP snapshot replace — injected rows
- disappear on the next fetch.
+ {arrowMode ? (
+ <>
+ This local Arrow dataset keeps its edits when you switch modes. Change its point radius
+ or add a speed filter, then repeat edits to check the settings stay put.
+ </>
+ ) : (
+ <>
+ Switch back to <strong>Poll URL</strong> to resume HTTP snapshot replace — injected rows
+ disappear on the next fetch.
+ </>
+ )}
</p>
</>
);
@@ -481,12 +523,18 @@ function WsSidebar({
);
}
-function LiveDataSidebar() {
+export function LiveDataSidebar() {
const dispatch = useDispatch();
- const dataset = useSelector(
- (state: KeplerRootState) => state.keplerGl[MAP_ID]?.visState?.datasets?.[DATASET_ID]
+ const reduxStore = useStore<KeplerRootState>();
+ const datasets = useSelector(
+ (state: KeplerRootState) => state.keplerGl[MAP_ID]?.visState?.datasets
);
const [mode, setMode] = useState<LiveMode>('poll');
+ const datasetId = mode === 'arrow' ? ARROW_DATASET_ID : DATASET_ID;
+ const dataset = datasets?.[datasetId];
+ const datasetReady = Boolean(dataset);
+ const hasRowDataset = Boolean(datasets?.[DATASET_ID]);
+ const hasArrowDataset = Boolean(datasets?.[ARROW_DATASET_ID]);
const [autoTrail, setAutoTrail] = useState(false);
const [keyedNote, setKeyedNote] = useState('');
const [wsStatus, setWsStatus] = useState<WsStatus>('idle');
@@ -495,17 +543,30 @@ function LiveDataSidebar() {
const [wsError, setWsError] = useState<string | null>(null);
const hostSeq = useRef(1);
const trackSeq = useRef(1);
+ const arrowViewInitialized = useRef(false);
const datasetRef = useRef(dataset);
datasetRef.current = dataset;
const toKepler = useCallback(
- (action: Parameters<typeof wrapTo>[1]) => dispatch(wrapTo(MAP_ID, action)),
+ (action: Parameters<ReturnType<typeof wrapTo>>[0]) => dispatch(wrapTo(MAP_ID, action)),
[dispatch]
);
const setLiveMode = useCallback(
(next: LiveMode) => {
setMode(next);
+ setKeyedNote('');
+ if (next === 'arrow' && !arrowViewInitialized.current) {
+ toKepler(
+ fitBounds([SF.lng - 0.1, SF.lat - 0.08, SF.lng + 0.1, SF.lat + 0.08], {
+ left: 320,
+ right: 24,
+ top: 24,
+ bottom: 24
+ })
+ );
+ arrowViewInitialized.current = true;
+ }
if (next === 'poll') {
setAutoTrail(false);
setKeyedNote('');
@@ -528,12 +589,29 @@ function LiveDataSidebar() {
[toKepler]
);
+ useEffect(() => {
+ // Only switch visibility on mode/load transitions, so user styling and
+ // visibility choices stay untouched during edits.
+ const layers = reduxStore.getState().keplerGl[MAP_ID]?.visState.layers || [];
+ layers.forEach(layer => {
+ if (layer.config.dataId !== DATASET_ID && layer.config.dataId !== ARROW_DATASET_ID) return;
+ const isVisible = layer.config.dataId === datasetId;
+ if (layer.config.isVisible !== isVisible) {
+ toKepler(layerConfigChange(layer, {isVisible}));
+ }
+ });
+ }, [datasetId, hasRowDataset, hasArrowDataset, reduxStore, toKepler]);
+
const addHostRows = useCallback(
(count: number) => {
- const rows = Array.from({length: count}, () => makeHostVehicle(hostSeq.current++));
- toKepler(addToDataset(DATASET_ID, rows));
+ const rows = Array.from({length: count}, () =>
+ makeHostVehicle(hostSeq.current++, {
+ speed: mode === 'arrow' ? 20 + (hostSeq.current % 8) * 5 : undefined
+ })
+ );
+ toKepler(addToDataset(datasetId, rows));
},
- [toKepler]
+ [datasetId, mode, toKepler]
);
const removeLast = useCallback(() => {
@@ -541,38 +619,42 @@ function LiveDataSidebar() {
if (rows < 1) {
return;
}
- toKepler(removeFromDataset(DATASET_ID, rows - 1));
- }, [dataset, toKepler]);
+ toKepler(removeFromDataset(datasetId, rows - 1));
+ }, [dataset, datasetId, toKepler]);
const removeRandom = useCallback(() => {
const rows = dataset?.dataContainer?.numRows?.() ?? 0;
if (rows < 1) {
return;
}
- toKepler(removeFromDataset(DATASET_ID, Math.floor(Math.random() * rows)));
- }, [dataset, toKepler]);
+ toKepler(removeFromDataset(datasetId, Math.floor(Math.random() * rows)));
+ }, [dataset, datasetId, toKepler]);
const upsertTracker = useCallback(() => {
const ids = rowIds(datasetRef.current);
const existed = ids.includes(TRACKER_ID);
- const row = makeHostVehicle(trackSeq.current++, {id: TRACKER_ID, ring: 7});
- toKepler(addToDataset(DATASET_ID, row, {upsertBy: 'id'}));
+ const row = makeHostVehicle(trackSeq.current++, {
+ id: TRACKER_ID,
+ ring: 7,
+ speed: mode === 'arrow' ? 20 + ((trackSeq.current * 7) % 60) : undefined
+ });
+ toKepler(addToDataset(datasetId, row, {upsertBy: 'id'}));
setKeyedNote(
existed
? `upsert ${TRACKER_ID} (replaced — row count should stay ${ids.length})`
: `upsert ${TRACKER_ID} (inserted — row count ${ids.length + 1})`
);
- }, [toKepler]);
+ }, [datasetId, mode, toKepler]);
const removeByFieldId = useCallback(
(id: string) => {
if (!rowIds(datasetRef.current).includes(id)) {
return;
}
- toKepler(removeFromDataset(DATASET_ID, {field: 'id', values: id}));
+ toKepler(removeFromDataset(datasetId, {field: 'id', values: id}));
setKeyedNote(`removeFromDataset({field: 'id', values: '${id}'})`);
},
- [toKepler]
+ [datasetId, toKepler]
);
const removeLastHostById = useCallback(() => {
@@ -585,18 +667,18 @@ function LiveDataSidebar() {
}, [removeByFieldId]);
useEffect(() => {
- if (mode !== 'host' || !autoTrail) {
+ if ((mode !== 'host' && mode !== 'arrow') || !autoTrail || !datasetReady) {
return undefined;
}
const timer = window.setInterval(() => {
const extras = hostRowIndexes(datasetRef.current);
if (extras.length >= MAX_HOST_ROWS) {
- toKepler(removeFromDataset(DATASET_ID, extras[0]));
+ toKepler(removeFromDataset(datasetId, extras[0]));
}
- toKepler(addToDataset(DATASET_ID, makeHostVehicle(hostSeq.current++)));
+ addHostRows(1);
}, AUTO_INTERVAL_MS);
return () => window.clearInterval(timer);
- }, [autoTrail, mode, toKepler]);
+ }, [autoTrail, mode, datasetId, datasetReady, addHostRows, toKepler]);
useEffect(() => {
if (mode !== 'ws') {
@@ -621,13 +703,13 @@ function LiveDataSidebar() {
}
};
socket.onmessage = event => {
- let message: {op?: string; rows?: unknown; upsertBy?: string};
+ let message: {op?: string; rows?: Parameters<typeof addToDataset>[1]; upsertBy?: string};
try {
message = JSON.parse(String(event.data));
} catch {
return;
}
- if (message.op !== 'upsert' || !message.rows) {
+ if (cancelled || message.op !== 'upsert' || !message.rows) {
return;
}
toKepler(addToDataset(DATASET_ID, message.rows, {upsertBy: message.upsertBy || 'id'}));
@@ -664,7 +746,7 @@ function LiveDataSidebar() {
<h2 style={{margin: '0 0 8px', fontSize: 16}}>Live data</h2>
<ModeToggle
mode={mode}
- hostEnabled={(dataset?.dataContainer?.numRows?.() ?? 0) > 0}
+ hostEnabled={(datasets?.[DATASET_ID]?.dataContainer?.numRows?.() ?? 0) > 0}
onChange={setLiveMode}
/>
{mode === 'poll' ? (
@@ -680,6 +762,7 @@ function LiveDataSidebar() {
) : (
<HostSidebar
dataset={dataset}
+ arrowMode={mode === 'arrow'}
autoTrail={autoTrail}
keyedNote={keyedNote}
onAutoTrail={setAutoTrail}
@@ -705,19 +788,22 @@ const App = () => {
wrapTo(
MAP_ID,
addDataToMap({
- datasets: {
- info: {
- id: DATASET_ID,
- label: 'Live orbit',
- type: DatasetType.EXTERNALLY_HOSTED
+ datasets: [
+ {
+ info: {
+ id: DATASET_ID,
+ label: 'Live orbit',
+ type: DatasetType.EXTERNALLY_HOSTED
+ },
+ data: {fields: [], rows: []},
+ metadata: {
+ source: LIVE_DATA_URL,
+ sourceFormat: 'csv',
+ refreshIntervalMs: REFRESH_INTERVAL_MS
+ }
},
- data: {fields: [], rows: []},
- metadata: {
- source: LIVE_DATA_URL,
- sourceFormat: 'csv',
- refreshIntervalMs: REFRESH_INTERVAL_MS
- }
- },
+ createArrowDataset()
+ ],
options: {
centerMap: true,
autoCreateLayers: false,
@@ -734,7 +820,7 @@ const App = () => {
config: {
dataId: DATASET_ID,
label: 'Live orbit',
- columns: {lat: 'lat', lng: 'lng', altitude: null},
+ columns: {lat: 'lat', lng: 'lng'},
isVisible: true,
visConfig: {radius: 24, filled: true, opacity: 0.85}
},
@@ -742,6 +828,21 @@ const App = () => {
colorField: {name: 'ring', type: 'integer'},
colorScale: 'quantize'
}
+ },
+ {
+ id: 'live-arrow-point',
+ type: 'point',
+ config: {
+ dataId: ARROW_DATASET_ID,
+ label: 'Live Arrow',
+ columns: {lat: 'lat', lng: 'lng'},
+ isVisible: false,
+ visConfig: {radius: 24, filled: true, opacity: 0.85}
+ },
+ visualChannels: {
+ colorField: {name: 'ring', type: 'integer'},
+ colorScale: 'quantize'
+ }
}
]
},
@@ -783,6 +884,4 @@ const Root = () => (
</Provider>
);
-const container = document.getElementById('root');
-const root = ReactDOM.createRoot(container!);
-root.render(<Root />);
+export default Root;
diff --git a/examples/live-data/src/arrow-data.ts b/examples/live-data/src/arrow-data.ts
new file mode 100644
index 000000000..62e956061
--- /dev/null
+++ b/examples/live-data/src/arrow-data.ts
@@ -0,0 +1,31 @@
+// SPDX-License-Identifier: MIT
+// Copyright contributors to the kepler.gl project
+
+import {Table, Vector, Utf8, Float64, Int32, vectorFromArray} from 'apache-arrow';
+import {arrowSchemaToFields} from '@kepler.gl/processors';
+
+export const ARROW_DATASET_ID = 'live-arrow-vehicles';
+
+/** Build once; subsequent changes use addToDataset/removeFromDataset. */
+export function createArrowDataset() {
+ const table = new Table({
+ id: vectorFromArray(['veh-01', 'veh-02', 'veh-03'], new Utf8()),
+ lat: vectorFromArray([37.79, 37.76, 37.78], new Float64()),
+ lng: vectorFromArray([-122.45, -122.42, -122.39], new Float64()),
+ heading: vectorFromArray([0, 120, 240], new Float64()),
+ speed: vectorFromArray([25, 40, 55], new Float64()),
+ ring: vectorFromArray([1, 2, 3], new Int32()),
+ progress: vectorFromArray([0, 0, 0], new Float64()),
+ orbit_s: vectorFromArray([0, 0, 0], new Float64()),
+ updated_at: vectorFromArray(Array(3).fill(new Date().toISOString()), new Utf8())
+ });
+ return {
+ info: {id: ARROW_DATASET_ID, label: 'Live Arrow'},
+ data: {
+ fields: arrowSchemaToFields(table),
+ cols: table.schema.fields.map((_, index) => table.getChildAt(index) as Vector),
+ rows: [],
+ arrowTable: table
+ }
+ };
+}
diff --git a/examples/live-data/src/index.tsx b/examples/live-data/src/index.tsx
new file mode 100644
index 000000000..b61605842
--- /dev/null
+++ b/examples/live-data/src/index.tsx
@@ -0,0 +1,10 @@
+// SPDX-License-Identifier: MIT
+// Copyright contributors to the kepler.gl project
+
+import * as React from 'react';
+import ReactDOM from 'react-dom/client';
+import Root from './app';
+
+const container = document.getElementById('root') as HTMLElement;
+const root = ReactDOM.createRoot(container);
+root.render(<Root />);
diff --git a/src/actions/src/actions.ts b/src/actions/src/actions.ts
index aca490910..8f74debd3 100644
--- a/src/actions/src/actions.ts
+++ b/src/actions/src/actions.ts
@@ -51,7 +51,7 @@ export type ActionHandlers<T extends {[k: string]: Handler}> = {
* @param {boolean} data.options.readOnly `default: false` if `readOnly` is set to `true`
* the left setting panel will be hidden
* @param {boolean} data.options.keepExistingConfig whether to keep exiting map data and associated layer filter interaction config `default: false`.
- * To append, upsert, or delete rows on an existing in-memory row dataset without replacing it or creating layers, use `addToDataset` / `removeFromDataset`. Arrow and DuckDB tables do not support those actions yet.
+ * To append, upsert, or delete rows on an existing in-memory row or Arrow dataset without replacing it or creating layers, use `addToDataset` / `removeFromDataset`. DuckDB tables do not support those actions yet.
* @param {Object} data.config this object will contain the full kepler.gl instance configuration {mapState, mapStyle, visState}
* @public
* @example
diff --git a/src/actions/src/vis-state-actions.ts b/src/actions/src/vis-state-actions.ts
index ec41f54e4..94b5a36f6 100644
--- a/src/actions/src/vis-state-actions.ts
+++ b/src/actions/src/vis-state-actions.ts
@@ -1125,7 +1125,8 @@ export type AddToDatasetOptions = {
/**
* Field name used as a unique key. Matching rows are replaced in place;
* keys that are not in the table are appended. Last incoming row wins when
- * the same key appears twice. Row tables only — Arrow/DuckDB no-op with a warning.
+ * the same key appears twice. DuckDB tables no-op with a warning. Nested, binary, and
+ * geoarrow Arrow columns also no-op (primitive Arrow columns concat in place).
*/
upsertBy?: string;
};
@@ -1136,13 +1137,14 @@ export type AddToDatasetUpdaterAction = {
options?: AddToDatasetOptions;
};
/**
- * Append rows to an existing in-memory row dataset without `addDataToMap`.
+ * Append rows to an existing in-memory row or Arrow dataset without `addDataToMap`.
* Keeps layer identity and style. Pass `options.upsertBy` to replace rows that
* share that key and append the rest.
*
- * Not implemented for Arrow or DuckDB tables (no INSERT / concat yet); those
- * calls warn and leave the table unchanged. Use `addDataToMap` with
- * `keepExistingConfig` for a full replace.
+ * DuckDB tables (no INSERT yet) warn and leave the table unchanged. Arrow
+ * tables concat primitive columns in place; nested, binary, and geoarrow
+ * columns are rejected. Use `addDataToMap` with `keepExistingConfig` for a
+ * full replace.
* @memberof visStateActions
* @param dataId dataset id
* @param rows one row or an array of rows (arrays in field order, or objects keyed by field name)
@@ -1174,12 +1176,12 @@ export type RemoveFromDatasetUpdaterAction = {
byField?: RemoveFromDatasetByField;
};
/**
- * Delete rows from an existing in-memory row dataset without restyling layers.
+ * Delete rows from an existing in-memory row or Arrow dataset without restyling layers.
* The second argument is either row indexes or `{field, values}` to match a
* column (e.g. an id). Out-of-range indexes are a no-op for the whole action.
*
- * Not implemented for Arrow or DuckDB tables; those calls warn and leave the
- * table unchanged. For a full table replace use `addDataToMap` with
+ * DuckDB tables warn and leave the table unchanged. Arrow tables concat the
+ * kept slices in place. For a full table replace use `addDataToMap` with
* `keepExistingConfig`.
* @memberof visStateActions
* @param dataId dataset id
diff --git a/src/duckdb/src/table/duckdb-table.ts b/src/duckdb/src/table/duckdb-table.ts
index b2086bbb9..220dd550e 100644
--- a/src/duckdb/src/table/duckdb-table.ts
+++ b/src/duckdb/src/table/duckdb-table.ts
@@ -2,6 +2,7 @@
// Copyright contributors to the kepler.gl project
import * as arrow from 'apache-arrow';
+import Console from 'global/console';
import {
ALL_FIELD_TYPES,
@@ -75,6 +76,13 @@ const DUCKDB_WKB_COLUMN = 'wkb_geometry';
*/
const KEPLER_GEOM_FROM_GEOJSON_COLUMN = '_geojson';
+const DUCKDB_ROW_EDIT_NOT_IMPLEMENTED =
+ 'In-place row edits (append, upsert, delete) are not implemented for DuckDB tables. DuckDB INSERT is not supported yet. Use addDataToMap with keepExistingConfig to replace the table.';
+
+function warnDuckDbRowEdits(method: string): void {
+ Console.warn(`KeplerGlDuckDbTable.${method}: ${DUCKDB_ROW_EDIT_NOT_IMPLEMENTED}`);
+}
+
/**
* Names of columns that most likely contain binary wkb geometry
*/
@@ -341,6 +349,25 @@ export class KeplerGlDuckDbTable extends KeplerTable {
return super.update({cols, rows: [], fields: []});
}
+ /**
+ * DuckDB's Arrow view is a query snapshot. Mutating it without SQL INSERT
+ * would desync the WASM table, so row edits stay a warned no-op.
+ */
+ appendRows(_rows: any[][]): boolean {
+ warnDuckDbRowEdits('appendRows');
+ return false;
+ }
+
+ upsertRows(_rows: any[][], _keyField: string): boolean {
+ warnDuckDbRowEdits('upsertRows');
+ return false;
+ }
+
+ removeRows(_indexes: number[]): boolean {
+ warnDuckDbRowEdits('removeRows');
+ return false;
+ }
+
static getFileProcessor = function (data: any, inputFormat?: string) {
let processor;
let format;
diff --git a/src/reducers/src/vis-state-updaters.ts b/src/reducers/src/vis-state-updaters.ts
index 18ab26b3e..1ef6f66b3 100644
--- a/src/reducers/src/vis-state-updaters.ts
+++ b/src/reducers/src/vis-state-updaters.ts
@@ -4097,8 +4097,8 @@ function normalizeDatasetRows(
}
/**
- * Append or upsert rows on an existing in-memory row dataset. Keeps layers and re-runs filters.
- * Arrow/DuckDB tables warn and are left unchanged (INSERT / concat not implemented).
+ * Append or upsert rows on an existing in-memory row or Arrow dataset. Keeps layers and re-runs filters.
+ * DuckDB tables warn and are left unchanged (INSERT not implemented). Nested, binary, and geoarrow Arrow columns are rejected.
* @memberof visStateUpdaters
* @public
*/
@@ -4131,8 +4131,8 @@ export function addToDatasetUpdater(
}
/**
- * Delete rows by index or by field values from an existing in-memory row dataset.
- * Keeps layers. Arrow/DuckDB tables warn and are left unchanged.
+ * Delete rows by index or by field values from an existing in-memory row or Arrow dataset.
+ * Keeps layers. DuckDB tables warn and are left unchanged.
* @memberof visStateUpdaters
* @public
*/
diff --git a/src/table/src/kepler-table.ts b/src/table/src/kepler-table.ts
index bb4a9847f..e06fa5232 100644
--- a/src/table/src/kepler-table.ts
+++ b/src/table/src/kepler-table.ts
@@ -34,6 +34,7 @@ import {
getSortingFunction,
timeToUnixMilli,
createDataContainer,
+ ArrowDataContainer,
DataForm,
diffFilters,
filterDataByFilterTypes,
@@ -109,7 +110,7 @@ export type TimeFieldFilterProps = TimeRangeFieldDomain & {
const FID_KEY = 'name';
const ROW_EDIT_NOT_IMPLEMENTED =
- 'In-place row edits (append, upsert, delete) are not implemented for Arrow/DuckDB tables. DuckDB INSERT and Arrow concat are not supported yet. Use addDataToMap with keepExistingConfig to replace the table.';
+ 'In-place row edits (append, upsert, delete) are not implemented for this table. DuckDB tables omit row edits (no INSERT yet). Use addDataToMap with keepExistingConfig to replace the table.';
function warnRowEditsUnavailable(method: string): void {
Console.warn(`KeplerTable.${method}: ${ROW_EDIT_NOT_IMPLEMENTED}`);
@@ -366,8 +367,8 @@ class KeplerTable<F extends Field = Field> {
}
/**
- * Append column-ordered rows in place. No-op for Arrow/DuckDB (no `append`).
- * Accessors stay bound: the container instance is not replaced.
+ * Append column-ordered rows in place. Accessors stay bound: the container
+ * instance is not replaced. No-op when the container has no `append`.
* @returns false when the payload is rejected and the table is unchanged.
*/
appendRows(rows: any[][]): boolean {
@@ -398,7 +399,7 @@ class KeplerTable<F extends Field = Field> {
/**
* Replace rows that share `keyField` and append the rest. Last incoming row
- * wins for a repeated key. No-op for Arrow/DuckDB.
+ * wins for a repeated key. No-op when the container has no `append`/`replace`.
* @returns false when the payload is rejected and the table is unchanged.
*/
upsertRows(rows: any[][], keyField: string): boolean {
@@ -418,6 +419,16 @@ class KeplerTable<F extends Field = Field> {
return false;
}
+ // Match Arrow keys after encoding so accepted representations such as an
+ // Int64 decimal string or a Date compare equal to the stored column value.
+ const keys =
+ this.dataContainer instanceof ArrowDataContainer
+ ? this.dataContainer.getRowEditKeys(rows, keyIdx)
+ : rows.map(row => row[keyIdx]);
+ if (!keys) {
+ return false;
+ }
+
const start = this.dataContainer.numRows();
const keyToIndex = new Map<unknown, number>();
for (let i = 0; i < start; i++) {
@@ -428,31 +439,30 @@ class KeplerTable<F extends Field = Field> {
}
const lastByKey = new Map<unknown, any[]>();
- const keyOrder: unknown[] = [];
- for (const row of rows) {
- const key = row[keyIdx];
- if (!lastByKey.has(key)) {
- keyOrder.push(key);
- }
- lastByKey.set(key, row);
- }
+ rows.forEach((row, index) => lastByKey.set(keys[index], row));
+ const replacements = new Map<number, any[]>();
const toAppend: any[][] = [];
- for (const key of keyOrder) {
- const row = lastByKey.get(key);
- if (!row) {
- continue;
- }
+ for (const [key, row] of lastByKey) {
const existing = keyToIndex.get(key);
if (existing !== undefined) {
- this.dataContainer.replace(existing, row);
+ replacements.set(existing, row);
} else {
toAppend.push(row);
}
}
- if (toAppend.length && !this.dataContainer.append(toAppend)) {
- return false;
+ if (this.dataContainer instanceof ArrowDataContainer) {
+ if (!this.dataContainer.replaceRows(replacements, toAppend)) {
+ return false;
+ }
+ } else {
+ for (const [index, row] of replacements) {
+ this.dataContainer.replace(index, row);
+ }
+ if (toAppend.length && !this.dataContainer.append(toAppend)) {
+ return false;
+ }
}
const nextIndexes =
@@ -466,7 +476,7 @@ class KeplerTable<F extends Field = Field> {
}
/**
- * Remove rows by index in place. No-op for Arrow/DuckDB (no `remove`).
+ * Remove rows by index in place. No-op when the container has no `remove`.
* @returns false when any index is invalid and the table is unchanged.
*/
removeRows(indexes: number[]): boolean {
diff --git a/src/utils/src/arrow-data-container.ts b/src/utils/src/arrow-data-container.ts
index 46ae8e6f3..e74a235af 100644
--- a/src/utils/src/arrow-data-container.ts
+++ b/src/utils/src/arrow-data-container.ts
@@ -1,7 +1,7 @@
// SPDX-License-Identifier: MIT
// Copyright contributors to the kepler.gl project
-import {ALL_FIELD_TYPES} from '@kepler.gl/constants';
+import {ALL_FIELD_TYPES, GEOARROW_METADATA_KEY} from '@kepler.gl/constants';
import {GeoArrowBinaryData, ProtoDatasetField} from '@kepler.gl/types';
import * as arrow from 'apache-arrow';
import {console as globalConsole} from 'global/window';
@@ -251,13 +251,18 @@ function tableFromCompactedColumns(
* Convert an Arrow cell from Vector.get() into a value makeBuilder.append accepts.
* Nested GeoArrow (FixedSizeList / List) cells are Vectors, not arrays.
*/
-function arrowCellToBuilderValue(value: unknown): unknown {
+function arrowCellToBuilderValue(value: unknown, logicalChildren = false): unknown {
if (value == null) {
return null;
}
if (ArrayBuffer.isView(value)) {
return Array.from(value as any);
}
+ if (logicalChildren && (value instanceof arrow.StructRow || value instanceof arrow.MapRow)) {
+ return new Map(
+ Array.from(value, ([key, child]) => [key, arrowCellToBuilderValue(child, true)])
+ );
+ }
const nested = value as {
toArray?: () => ArrayLike<unknown>;
@@ -267,8 +272,9 @@ function arrowCellToBuilderValue(value: unknown): unknown {
isValid?: (index: number) => boolean;
};
- // Primitive numeric child of a FixedSizeList (lng/lat).
- if (typeof nested.toArray === 'function' && nested.numChildren === 0) {
+ // Keep ordinary import compaction unchanged. Edited snapshots reinterpret
+ // timestamp storage first, so their children can safely use logical getters.
+ if (!logicalChildren && typeof nested.toArray === 'function' && nested.numChildren === 0) {
return Array.from(nested.toArray());
}
@@ -276,7 +282,7 @@ function arrowCellToBuilderValue(value: unknown): unknown {
const out: unknown[] = [];
for (let i = 0; i < nested.length; i++) {
const valid = typeof nested.isValid === 'function' ? nested.isValid(i) : true;
- out.push(valid ? arrowCellToBuilderValue(nested.get(i)) : null);
+ out.push(valid ? arrowCellToBuilderValue(nested.get(i), logicalChildren) : null);
}
return out;
}
@@ -286,9 +292,11 @@ function arrowCellToBuilderValue(value: unknown): unknown {
function compactArrowVector(
column: arrow.Vector,
- type: arrow.DataType = column.type
+ type: arrow.DataType = column.type,
+ logicalChildren = false,
+ force = false
): arrow.Vector {
- if (!column || !column.data || column.data.length <= 1) {
+ if (!column || !column.data || (!force && column.data.length <= 1)) {
return column;
}
@@ -304,7 +312,25 @@ function compactArrowVector(
}
}
- const builder = arrow.makeBuilder({type, nullValues: [null]});
+ const builder = arrow.makeBuilder({
+ type,
+ nullValues: [null],
+ ...(logicalChildren && arrow.DataType.isDictionary(type)
+ ? {
+ // Struct/Map cells become Maps for builders; their default string key
+ // is always "[object Map]". Hash their typed contents instead.
+ dictionaryHashFunction: value =>
+ JSON.stringify(value, (_, child) =>
+ child instanceof Map
+ ? Array.from(child)
+ : typeof child === 'bigint' ||
+ (typeof child === 'number' && !Number.isFinite(child))
+ ? String(child)
+ : child
+ )
+ }
+ : {})
+ });
const length = column.length;
for (let i = 0; i < length; i++) {
const isValid =
@@ -314,7 +340,7 @@ function compactArrowVector(
// FixedSizeList/List .get() returns nested Vectors. Appending those
// silently writes NaN (points) or throws (lines/polygons). Plain JS
// arrays are what makeBuilder expects.
- builder.append(isValid ? arrowCellToBuilderValue(column.get(i)) : null);
+ builder.append(isValid ? arrowCellToBuilderValue(column.get(i), logicalChildren) : null);
}
const finished = builder.finish() as arrow.Vector | {toVector: () => arrow.Vector};
if (finished instanceof arrow.Vector) {
@@ -326,6 +352,215 @@ function compactArrowVector(
return column;
}
+/**
+ * Primitive (and dictionary-of-primitive) columns can be rebuilt from JS row
+ * arrays via vectorFromArray. Nested, binary, and geoarrow columns cannot.
+ */
+function isSupportedArrowRowEditType(type: arrow.DataType): boolean {
+ if (
+ arrow.DataType.isUtf8(type) ||
+ arrow.DataType.isNull(type) ||
+ arrow.DataType.isInt(type) ||
+ arrow.DataType.isFloat(type) ||
+ arrow.DataType.isBool(type) ||
+ arrow.DataType.isDate(type) ||
+ arrow.DataType.isTimestamp(type) ||
+ arrow.DataType.isTime(type)
+ ) {
+ return true;
+ }
+ if (arrow.DataType.isDictionary(type)) {
+ const dictionary = (type as arrow.Dictionary).dictionary;
+ return Boolean(dictionary && isSupportedArrowRowEditType(dictionary));
+ }
+ return false;
+}
+
+/** Reinterpret timestamp storage for builders without rounding through JS milliseconds. */
+function rowEditStorageType(type: arrow.DataType): arrow.DataType {
+ if (arrow.DataType.isTimestamp(type)) return new arrow.Int64();
+ if (arrow.DataType.isDictionary(type)) return type.indices;
+ if (type.children?.length) {
+ // Retain the Arrow type's prototype and layout (list size, union mode, etc.).
+ // Field.clone preserves names, nullability and extension metadata.
+ return Object.assign(Object.create(Object.getPrototypeOf(type)), type, {
+ children: type.children.map(field => field.clone({type: rowEditStorageType(field.type)}))
+ });
+ }
+ return type;
+}
+
+function rowEditStorageVector(
+ column: arrow.Vector,
+ type: arrow.DataType,
+ remapped = new Map<arrow.Data, arrow.Data>(),
+ dictionaries = new Map<number, arrow.Vector>()
+): arrow.Vector {
+ const convert = (original: arrow.Data, nextType: arrow.DataType): arrow.Data => {
+ const data = remapped.get(original) || original;
+ const copy = data.clone(
+ nextType,
+ data.offset,
+ data.length,
+ data.nullCount,
+ data.buffers,
+ data.children.map((child, i) => convert(child, nextType.children[i].type))
+ );
+ if (arrow.DataType.isDictionary(nextType)) {
+ const canonical = dictionaries.get(nextType.id);
+ if (canonical) {
+ // Equivalent field types from different batches need the same object,
+ // even when their DataType instances differ.
+ copy.dictionary = canonical;
+ } else {
+ if (!data.dictionary) throw new Error('Missing Arrow dictionary');
+ copy.dictionary = rowEditStorageVector(
+ data.dictionary,
+ nextType.dictionary,
+ remapped,
+ dictionaries
+ );
+ }
+ }
+ return copy;
+ };
+ return new arrow.Vector(column.data.map(data => convert(data, type)));
+}
+
+function rowEditDictionaryCapacity(type: arrow.Dictionary): number {
+ return 2 ** (type.indices.bitWidth - (type.indices.isSigned ? 1 : 0));
+}
+
+function collectRowEditDictionaries(data: arrow.Data, groups: Map<number, Set<arrow.Data>>): void {
+ data.children.forEach(child => collectRowEditDictionaries(child, groups));
+ if (!arrow.DataType.isDictionary(data.type) || !data.dictionary) return;
+ // Inner dictionaries must be reconciled before their enclosing dictionary.
+ data.dictionary.data.forEach(child => collectRowEditDictionaries(child, groups));
+ const group = groups.get(data.type.id) || new Set<arrow.Data>();
+ group.add(data);
+ groups.set(data.type.id, group);
+}
+
+function checkRowEditDictionaryCapacity(column: arrow.Vector): void {
+ if (!arrow.DataType.isDictionary(column.type)) return;
+ const capacity = rowEditDictionaryCapacity(column.type);
+ if (column.data.some(data => (data.dictionary?.length || 0) > capacity)) {
+ throw new Error('Row edit exceeds the Arrow dictionary index capacity');
+ }
+}
+
+/** Compact only edited snapshots; progressive loading and other rebuilds keep their behavior. */
+function compactRowEditColumns(columns: arrow.Vector[]): arrow.Vector[] {
+ columns = columns.map(
+ column =>
+ new arrow.Vector(
+ column.data.map(data => {
+ if (!data.children.length) return data;
+ // List/Map slices retain their entire child buffer. Normalize each
+ // retained chunk before merging dictionaries so deleted/null child
+ // entries cannot cause a false index-capacity overflow.
+ const sourceGroups = new Map<number, Set<arrow.Data>>();
+ collectRowEditDictionaries(data, sourceGroups);
+ const sourceDictionaries = new Map<number, arrow.Vector>();
+ sourceGroups.forEach((entries, id) => {
+ const dictionary = entries.values().next().value?.dictionary;
+ if (dictionary) sourceDictionaries.set(id, dictionary);
+ });
+ const type = rowEditStorageType(data.type);
+ const compacted = compactArrowVector(
+ rowEditStorageVector(new arrow.Vector([data]), type),
+ type,
+ true,
+ true
+ );
+ return rowEditStorageVector(compacted, data.type, new Map(), sourceDictionaries).data[0];
+ })
+ )
+ );
+ const groups = new Map<number, Set<arrow.Data>>();
+ columns.forEach(column => column.data.forEach(data => collectRowEditDictionaries(data, groups)));
+ const dictionaries = new Map<number, arrow.Vector>();
+ const remapped = new Map<arrow.Data, arrow.Data>();
+ for (const entries of groups.values()) {
+ // Narrow indexes get the first dictionary entries before wider columns add values.
+ const group = [...entries].sort(
+ (a, b) =>
+ rowEditDictionaryCapacity(a.type as arrow.Dictionary) -
+ rowEditDictionaryCapacity(b.type as arrow.Dictionary)
+ );
+ const type = group[group.length - 1].type as arrow.Dictionary;
+ if (
+ group.some(
+ data => String((data.type as arrow.Dictionary).dictionary) !== String(type.dictionary)
+ )
+ ) {
+ throw new Error('Shared Arrow dictionary value types must match');
+ }
+ const original = group[0].dictionary;
+ if (!original) throw new Error('Missing Arrow dictionary');
+ if (group.every(data => data.dictionary === original)) {
+ // Deletion usually keeps one original mapping. Preserve its exact payload,
+ // including nested timestamps, and only compact the retained indices.
+ dictionaries.set(
+ type.id,
+ rowEditStorageVector(original, original.type, remapped, dictionaries)
+ );
+ continue;
+ }
+ const storageType = new arrow.Dictionary(
+ rowEditStorageType(type.dictionary),
+ type.indices,
+ type.id,
+ type.isOrdered
+ );
+ const combined = new arrow.Vector(
+ group.flatMap(
+ data =>
+ rowEditStorageVector(new arrow.Vector([data]), storageType, remapped, dictionaries).data
+ )
+ );
+ const compacted = compactArrowVector(combined, storageType, true);
+ checkRowEditDictionaryCapacity(compacted);
+ const compactedDictionary = compacted.data[0].dictionary;
+ if (!compactedDictionary) throw new Error('Missing compacted Arrow dictionary');
+ const dictionary = rowEditStorageVector(
+ compactedDictionary,
+ type.dictionary,
+ new Map(),
+ dictionaries
+ );
+ dictionaries.set(type.id, dictionary);
+ let offset = 0;
+ for (const originalData of group) {
+ const columnType = originalData.type as arrow.Dictionary;
+ const chunk = compacted.slice(offset, offset + originalData.length).data[0];
+ const capacity = rowEditDictionaryCapacity(columnType);
+ for (let index = 0; index < chunk.length; index++) {
+ if (chunk.getValid(index) && Number(chunk.values[index]) >= capacity) {
+ throw new Error('Row edit exceeds the Arrow dictionary index capacity');
+ }
+ }
+ const buffers = chunk.buffers;
+ buffers[arrow.BufferType.DATA] = new columnType.indices.ArrayType(chunk.values);
+ const data = chunk.clone(columnType, chunk.offset, chunk.length, chunk.nullCount, buffers);
+ data.dictionary = dictionary;
+ remapped.set(originalData, data);
+ offset += originalData.length;
+ }
+ }
+ return columns.map(column => {
+ // Builders see raw timestamp units and canonical dictionary indices, even
+ // inside lists/structs. Restore the original types after one-batch compaction.
+ const type = rowEditStorageType(column.type);
+ return rowEditStorageVector(
+ compactArrowVector(rowEditStorageVector(column, type, remapped, dictionaries), type, true),
+ column.type,
+ new Map(),
+ dictionaries
+ );
+ });
+}
+
/**
* A data container where all data is stored in raw Arrow table
*/
@@ -433,6 +668,160 @@ export class ArrowDataContainer implements DataContainerInterface {
// this._colData = this._cols.map(c => c.toArray());
}
+ /** Append typed rows atomically while retaining the container and prior snapshots. */
+ append(rows: any[][]): boolean {
+ if (!Array.isArray(rows)) return false;
+ return this.replaceRows(new Map(), rows);
+ }
+
+ replace(index: number, row: any[]): boolean {
+ return this.replaceRows(new Map([[index, row]]), []);
+ }
+
+ /** @internal Match upsert keys in the same representation returned by valueAt. */
+ getRowEditKeys(rows: any[][], keyIndex: number): unknown[] | null {
+ const type = this._arrowTable.schema.fields[keyIndex]?.type;
+ if (!type) return null;
+ let valueType = type;
+ while (arrow.DataType.isDictionary(valueType)) valueType = valueType.dictionary;
+ try {
+ if (arrow.DataType.isInt(valueType)) {
+ const bits = valueType.bitWidth;
+ const limit = 1n << BigInt(bits - (valueType.isSigned ? 1 : 0));
+ const min = valueType.isSigned ? -limit : 0n;
+ const max = limit - 1n;
+ for (const row of rows) {
+ const value = row[keyIndex];
+ if (value === null) continue;
+ // Arrow's typed array assignment wraps out-of-range integers. A key
+ // must not silently turn into a different existing row's identity.
+ if (typeof value === 'string' && !value.trim()) return null;
+ const integer = BigInt(bits === 64 ? value : Number(value));
+ if (integer < min || integer > max) return null;
+ }
+ }
+ const keys = arrow.vectorFromArray(
+ rows.map(row => row[keyIndex]),
+ type
+ );
+ checkRowEditDictionaryCapacity(keys);
+ return Array.from(keys);
+ } catch {
+ return null;
+ }
+ }
+
+ /** @internal Encode a complete edit batch before replacing any live columns. */
+ replaceRows(replacements: Map<number, any[]>, appended: any[][]): boolean {
+ for (const index of replacements.keys()) {
+ if (!Number.isInteger(index) || index < 0 || index >= this._numRows) return false;
+ }
+ const incoming = this._columnsFromRows([...replacements.values(), ...appended]);
+ if (!incoming) return false;
+ const ordered = [...replacements.keys()]
+ .map((index, incomingIndex) => ({index, incomingIndex}))
+ .sort((a, b) => a.index - b.index);
+ try {
+ const columns = this._cols.map((column, i) => {
+ const chunks: arrow.Data[] = [];
+ let start = 0;
+ for (const {index, incomingIndex} of ordered) {
+ chunks.push(
+ ...column.slice(start, index).data,
+ ...incoming[i].slice(incomingIndex, incomingIndex + 1).data
+ );
+ start = index + 1;
+ }
+ chunks.push(...column.slice(start).data, ...incoming[i].slice(replacements.size).data);
+ return new arrow.Vector(chunks);
+ });
+ this._commitColumns(columns);
+ return true;
+ } catch {
+ return false;
+ }
+ }
+
+ /** Delete unique valid indexes without decoding retained values. */
+ remove(indexes: number[]): boolean {
+ if (!Array.isArray(indexes) || !indexes.length) return false;
+ if (indexes.some(index => !Number.isInteger(index) || index < 0 || index >= this._numRows))
+ return false;
+ const sorted = [...new Set(indexes)].sort((a, b) => a - b);
+ try {
+ const columns = this._cols.map(column => {
+ const chunks: arrow.Data[] = [];
+ let start = 0;
+ for (const index of sorted) {
+ if (start < index) chunks.push(...column.slice(start, index).data);
+ start = index + 1;
+ }
+ if (start < this._numRows) chunks.push(...column.slice(start).data);
+ return chunks.length ? new arrow.Vector(chunks) : column.slice(0, 0);
+ });
+ this._commitColumns(columns);
+ return true;
+ } catch {
+ return false;
+ }
+ }
+
+ private _commitColumns(columns: arrow.Vector[]): void {
+ const {schema} = this._arrowTable;
+ const compacted = compactRowEditColumns(columns);
+ // Ordinal keys let Arrow align chunks without interpreting real column names.
+ const ordered = new arrow.Table(Object.fromEntries(compacted.map((column, i) => [i, column])));
+ const batches = ordered.batches.map(batch => {
+ const result = new arrow.RecordBatch(schema, batch.data);
+ // Arrow reconciles fields by name, even for positional input. Retain the
+ // source fields when an external schema contains duplicate names.
+ schema.fields.forEach((field, i) => {
+ result.schema.fields[i] = field;
+ });
+ // Validate shared dictionary identities before publishing the snapshot.
+ result.dictionaries;
+ return result;
+ });
+ this._assignTable(new arrow.Table(schema, batches));
+ this._geoArrowBinaryData = {};
+ }
+
+ private _columnsFromRows(rows: any[][]): arrow.Vector[] | null {
+ const {fields} = this._arrowTable.schema;
+ if (
+ !rows.length ||
+ !fields.length ||
+ rows.some(row => !Array.isArray(row) || row.length !== fields.length)
+ )
+ return null;
+ for (const field of fields) {
+ if (
+ field.metadata.get(GEOARROW_METADATA_KEY)?.startsWith('geoarrow') ||
+ !isSupportedArrowRowEditType(field.type)
+ ) {
+ globalConsole.warn(
+ `ArrowDataContainer: row edits do not support column "${field.name}" (${field.type})`
+ );
+ return null;
+ }
+ }
+ try {
+ return fields.map((field, i) => {
+ const column = arrow.vectorFromArray(
+ rows.map(row => row[i]),
+ field.type
+ );
+ checkRowEditDictionaryCapacity(column);
+ return column;
+ });
+ } catch {
+ globalConsole.warn(
+ 'ArrowDataContainer: could not encode row edits using the existing column types'
+ );
+ return null;
+ }
+ }
+
numChunks(): number {
return this._numChunks;
}
diff --git a/src/utils/src/data-container-interface.ts b/src/utils/src/data-container-interface.ts
index 61b437ef9..1d2a48b1a 100644
--- a/src/utils/src/data-container-interface.ts
+++ b/src/utils/src/data-container-interface.ts
@@ -20,19 +20,19 @@ export interface DataContainerInterface {
update?(updateData: any[] | arrow.Table, options?: Record<string, unknown>): void;
/**
- * Append rows in place. Implemented by row containers; Arrow/DuckDB omit this.
- * @returns false when the rows are rejected (wrong shape) and the container is unchanged.
+ * Append rows in place. Implemented by row and Arrow containers.
+ * @returns false when the rows are rejected (wrong shape or unsupported Arrow type) and the container is unchanged.
*/
append?(rows: any[][]): boolean;
/**
- * Replace one row in place. Implemented by row containers; Arrow/DuckDB omit this.
+ * Replace one row in place. Implemented by row and Arrow containers.
* @returns false when the index or row is rejected and the container is unchanged.
*/
replace?(index: number, row: any[]): boolean;
/**
- * Remove rows by index in place. Implemented by row containers; Arrow/DuckDB omit this.
+ * Remove rows by index in place. Implemented by row and Arrow containers.
* @returns false when any index is invalid and the container is unchanged.
*/
remove?(indexes: number[]): boolean;
diff --git a/test/browser/live-data.spec.js b/test/browser/live-data.spec.js
new file mode 100644
index 000000000..f760c7d15
--- /dev/null
+++ b/test/browser/live-data.spec.js
@@ -0,0 +1,117 @@
+// SPDX-License-Identifier: MIT
+// Copyright contributors to the kepler.gl project
+
+import React from 'react';
+import {act, fireEvent, render, screen} from '@testing-library/react';
+import {Provider} from 'react-redux';
+import {combineReducers, createStore} from 'redux';
+import keplerGlReducer from '@kepler.gl/reducers';
+import {addFilter, registerEntry, setFilter, wrapTo} from '@kepler.gl/actions';
+import {KeplerGlLayers} from '@kepler.gl/layers';
+import {KeplerTable} from '@kepler.gl/table';
+import {ArrowDataContainer} from '@kepler.gl/utils';
+import {RowDataContainer} from '../../src/utils/src/row-data-container';
+import {LiveDataSidebar} from '../../examples/live-data/src/app';
+import {ARROW_DATASET_ID, createArrowDataset} from '../../examples/live-data/src/arrow-data';
+
+jest.mock('@kepler.gl/components', () => ({__esModule: true, default: () => null}));
+jest.mock('maplibre-gl/dist/maplibre-gl.css', () => ({}));
+
+const {PointLayer} = KeplerGlLayers;
+afterEach(() => jest.useRealTimers());
+
+function setup() {
+ const sample = createArrowDataset();
+ const arrow = new KeplerTable({info: sample.info, color: [0, 0, 0]}).updateSchema(sample.data);
+ const rows = new KeplerTable({info: {id: 'live-vehicles'}, color: [0, 0, 0]}).updateSchema({
+ fields: sample.data.fields,
+ rows: arrow.dataContainer.flattenData()
+ });
+ const layers = [rows, arrow].map(table => {
+ const {props} = PointLayer.findDefaultLayerProps(table);
+ const layer = new PointLayer({
+ ...props[0],
+ id: `${table.id}-point`,
+ dataId: table.id,
+ isVisible: table === rows
+ });
+ layer.updateLayerVisConfig({radius: 17});
+ return layer;
+ });
+ const reducer = combineReducers({keplerGl: keplerGlReducer});
+ const initial = reducer(undefined, registerEntry({id: 'map'}));
+ initial.keplerGl.map.visState = {
+ ...initial.keplerGl.map.visState,
+ datasets: {[rows.id]: rows, [arrow.id]: arrow},
+ layers,
+ layerData: [{}, {}],
+ layerOrder: [0, 1]
+ };
+ const store = createStore(reducer, initial);
+ const view = render(
+ <Provider store={store}>
+ <LiveDataSidebar />
+ </Provider>
+ );
+ const state = () => store.getState().keplerGl.map.visState;
+ const dataset = id => state().datasets[id];
+ return {...view, store, state, dataset};
+}
+
+test('Arrow controls edit the real container and retain settings across mode switches', () => {
+ const {store, state, dataset} = setup();
+ const original = dataset(ARROW_DATASET_ID).dataContainer;
+ expect(original).toBeInstanceOf(ArrowDataContainer);
+ expect(dataset('live-vehicles').dataContainer).toBeInstanceOf(RowDataContainer);
+ fireEvent.click(screen.getByRole('button', {name: 'Arrow rows'}));
+ act(() => {
+ store.dispatch(wrapTo('map', addFilter(ARROW_DATASET_ID)));
+ store.dispatch(wrapTo('map', setFilter(0, 'name', 'speed')));
+ store.dispatch(wrapTo('map', setFilter(0, 'value', [30, 50])));
+ });
+ const filterId = state().filters[0].id;
+ fireEvent.click(screen.getByRole('button', {name: 'Add 5'}));
+ expect(original.numRows()).toBe(8);
+ fireEvent.click(screen.getByRole('button', {name: 'Insert tracker'}));
+ const position = screen.getByLabelText('Tracker position and speed').textContent;
+ fireEvent.click(screen.getByRole('button', {name: 'Move tracker'}));
+ expect(original.numRows()).toBe(9);
+ expect(screen.getByLabelText('Tracker position and speed').textContent).not.toBe(position);
+ fireEvent.click(screen.getByRole('button', {name: 'Remove tracker'}));
+ fireEvent.click(screen.getByRole('button', {name: 'Remove last'}));
+ expect(original.numRows()).toBe(7);
+ expect(dataset(ARROW_DATASET_ID).dataContainer).toBe(original);
+ expect(dataset(ARROW_DATASET_ID).dataRevision).toBe(5);
+ expect(state().layers[1].id).toBe(`${ARROW_DATASET_ID}-point`);
+ expect(state().layers[1].config.visConfig.radius).toBe(17);
+ expect(state().filters[0]).toMatchObject({id: filterId, value: [30, 50]});
+ fireEvent.click(screen.getByRole('button', {name: 'Host rows'}));
+ fireEvent.click(screen.getByRole('button', {name: 'Add 1'}));
+ expect(dataset('live-vehicles').dataContainer.numRows()).toBe(4);
+ expect(original.numRows()).toBe(7);
+ fireEvent.click(screen.getByRole('button', {name: 'Arrow rows'}));
+ expect(dataset(ARROW_DATASET_ID).dataContainer).toBe(original);
+ expect(state().layers.map(layer => layer.config.isVisible)).toEqual([false, true]);
+ expect(state().filters[0]).toMatchObject({id: filterId, value: [30, 50]});
+});
+
+test('auto trail stops on mode change and unmount without changing the other dataset', () => {
+ jest.useFakeTimers();
+ const {dataset, unmount} = setup();
+ const original = dataset(ARROW_DATASET_ID).dataContainer;
+ fireEvent.click(screen.getByRole('button', {name: 'Arrow rows'}));
+ fireEvent.click(screen.getByRole('checkbox', {name: /Auto trail/}));
+ act(() => jest.advanceTimersByTime(800));
+ expect(original.numRows()).toBe(4);
+ fireEvent.click(screen.getByRole('button', {name: 'Host rows'}));
+ act(() => jest.advanceTimersByTime(2400));
+ expect(original.numRows()).toBe(4);
+ expect(dataset('live-vehicles').dataContainer.numRows()).toBe(3);
+ fireEvent.click(screen.getByRole('button', {name: 'Arrow rows'}));
+ expect(screen.getByRole('checkbox', {name: /Auto trail/})).not.toBeChecked();
+ fireEvent.click(screen.getByRole('checkbox', {name: /Auto trail/}));
+ unmount();
+ act(() => jest.advanceTimersByTime(2400));
+ expect(original.numRows()).toBe(4);
+ jest.useRealTimers();
+});
diff --git a/test/node/reducers/vis-state-test.js b/test/node/reducers/vis-state-test.js
index 540c63471..ebe2be215 100644
--- a/test/node/reducers/vis-state-test.js
+++ b/test/node/reducers/vis-state-test.js
@@ -15,6 +15,7 @@ import {VisStateActions, MapStateActions} from '@kepler.gl/actions';
import {
visStateReducer as reducer,
INITIAL_VIS_STATE,
+ INITIAL_MAP_STATE,
DEFAULT_ANIMATION_CONFIG,
serializeLayer,
validateLayerWithData,
@@ -11187,9 +11188,9 @@ test('VisStateUpdater -> addToDataset no-ops', async t => {
t.equal(
reducer(state, VisStateActions.addToDataset('live', [3, 4])),
state,
- 'tables without append (Arrow/DuckDB) are a no-op'
+ 'tables without append are a no-op'
);
- t.ok(warn.called, 'should warn that in-place row edits are not implemented for Arrow/DuckDB');
+ t.ok(warn.called, 'should warn that in-place row edits are not implemented');
warn.restore();
t.equal(state.datasets.live.dataContainer.numRows(), 1, 'no-op must not mutate rows');
@@ -11294,9 +11295,9 @@ test('VisStateUpdater -> removeFromDataset deletes rows and keeps layers', async
t.equal(
reducer(oneIndex, VisStateActions.removeFromDataset('live', 0)),
oneIndex,
- 'tables without remove (Arrow/DuckDB) are a no-op'
+ 'tables without remove are a no-op'
);
- t.ok(warn.called, 'should warn that in-place deletes are not implemented for Arrow/DuckDB');
+ t.ok(warn.called, 'should warn that in-place deletes are not implemented');
warn.restore();
t.end();
@@ -11432,3 +11433,520 @@ test('VisStateUpdater -> addToDataset and removeFromDataset clear table sort', a
t.end();
});
+
+async function createArrowLiveDatasets(rows) {
+ const table = rows instanceof arrow.Table ? rows : arrow.tableFromJSON(rows);
+ return createNewDataEntryMock({
+ info: {id: 'live', label: 'live.arrow'},
+ data: processArrowBatches(table.batches)
+ });
+}
+
+test('VisStateUpdater -> addToDataset and removeFromDataset on Arrow tables', async t => {
+ const datasets = await createArrowLiveDatasets([{lat: 37.77, lng: -122.42}]);
+ t.ok(
+ typeof datasets.live.dataContainer.append === 'function',
+ 'Arrow dataset should expose append'
+ );
+
+ const {props} = PointLayer.findDefaultLayerProps(datasets.live);
+ const pointLayer = new PointLayer({
+ id: 'p1',
+ dataId: 'live',
+ isVisible: true,
+ ...props[0]
+ });
+ const {layerData, layer} = calculateLayerData(
+ pointLayer,
+ {...INITIAL_VIS_STATE, datasets, layers: [pointLayer], layerData: [{}]},
+ undefined
+ );
+ const state = {
+ ...INITIAL_VIS_STATE,
+ datasets,
+ layers: [layer],
+ layerData: [layerData],
+ layerOrder: [layer.id]
+ };
+
+ const appended = reducer(
+ state,
+ VisStateActions.addToDataset('live', [
+ [10, 20],
+ [30, 40]
+ ])
+ );
+ t.equal(appended.datasets.live.dataContainer.numRows(), 3, 'should append both Arrow rows');
+ t.equal(appended.layers[0].id, 'p1', 'should keep the same layer id');
+ t.equal(
+ appended.layerData[0].data.numRows,
+ 3,
+ 'should rebuild Arrow layer data with the new row count'
+ );
+ t.notEqual(
+ appended.layerData[0].data,
+ state.layerData[0].data,
+ 'should replace the previous Arrow table snapshot'
+ );
+
+ const keyed = await createArrowLiveDatasets([
+ {id: 'a', lat: 1, lng: 2},
+ {id: 'b', lat: 3, lng: 4}
+ ]);
+ const upserted = reducer(
+ {...INITIAL_VIS_STATE, datasets: keyed},
+ VisStateActions.addToDataset(
+ 'live',
+ [
+ {id: 'b', lat: 30, lng: 40},
+ {id: 'c', lat: 5, lng: 6}
+ ],
+ {upsertBy: 'id'}
+ )
+ );
+ t.equal(
+ upserted.datasets.live.dataContainer.numRows(),
+ 3,
+ 'should append only the new Arrow key'
+ );
+ t.equal(
+ upserted.datasets.live.dataContainer.valueAt(1, 1),
+ 30,
+ 'should replace the matching Arrow key'
+ );
+ t.equal(
+ upserted.datasets.live.dataContainer.valueAt(2, 0),
+ 'c',
+ 'should append the new Arrow key'
+ );
+
+ const removed = reducer(
+ upserted,
+ VisStateActions.removeFromDataset('live', {field: 'id', values: ['b']})
+ );
+ t.equal(removed.datasets.live.dataContainer.numRows(), 2, 'should drop the matching Arrow id');
+ t.equal(
+ removed.datasets.live.dataContainer.valueAt(0, 0),
+ 'a',
+ 'should keep unmatched Arrow ids'
+ );
+ t.equal(
+ removed.datasets.live.dataContainer.valueAt(1, 0),
+ 'c',
+ 'should keep later unmatched Arrow ids'
+ );
+
+ t.end();
+});
+
+test('VisStateUpdater -> rejected Arrow upserts leave all rows and derived state unchanged', async t => {
+ const datasets = await createArrowLiveDatasets(
+ new arrow.Table({
+ id: arrow.vectorFromArray(['a', 'b'], new arrow.Utf8()),
+ amount: arrow.vectorFromArray([1n, 2n], new arrow.Int64())
+ })
+ );
+ const state = reducer(
+ {...INITIAL_VIS_STATE, datasets},
+ VisStateActions.sortTableColumn('live', 'amount')
+ );
+ const dataset = state.datasets.live;
+ const container = dataset.dataContainer;
+ const snapshot = container.getTable();
+ const revision = dataset.dataRevision;
+ const indexes = dataset.allIndexes;
+ const sortOrder = dataset.sortOrder;
+ const warn = sinon.stub(Console, 'warn');
+ try {
+ for (const lastKey of ['b', 'c']) {
+ const result = reducer(
+ state,
+ VisStateActions.addToDataset(
+ 'live',
+ [
+ ['a', 10n],
+ [lastKey, 'invalid-int64']
+ ],
+ {
+ upsertBy: 'id'
+ }
+ )
+ );
+ t.equal(result, state, 'encoding failure is a reducer no-op');
+ t.equal(container.getTable(), snapshot, 'no partial Arrow snapshot is committed');
+ t.deepEqual(
+ container.flattenData(),
+ [
+ ['a', 1n],
+ ['b', 2n]
+ ],
+ 'no earlier replacement leaks'
+ );
+ t.equal(dataset.dataRevision, revision, 'revision stays unchanged');
+ t.equal(dataset.allIndexes, indexes, 'indexes stay unchanged');
+ t.equal(dataset.sortOrder, sortOrder, 'sorting stays unchanged');
+ }
+ } finally {
+ warn.restore();
+ }
+ t.end();
+});
+
+test('VisStateUpdater -> Arrow upserts preserve accessors and last incoming values', async t => {
+ const datasets = await createArrowLiveDatasets([
+ {id: 'a', value: 1},
+ {id: 'b', value: 2},
+ {id: 'c', value: 3}
+ ]);
+ let state = {...INITIAL_VIS_STATE, datasets};
+ const container = datasets.live.dataContainer;
+ const snapshot = container.getTable();
+ const accessor = datasets.live.fields[1].valueAccessor;
+ const revision = datasets.live.dataRevision;
+ state = reducer(
+ state,
+ VisStateActions.addToDataset(
+ 'live',
+ [
+ ['c', 30],
+ ['e', 5],
+ ['a', 10],
+ ['d', 4],
+ ['c', 33],
+ ['e', 55]
+ ],
+ {upsertBy: 'id'}
+ )
+ );
+ t.equal(state.datasets.live.dataContainer, container, 'keeps the original container');
+ t.deepEqual(
+ container.flattenData(),
+ [
+ ['a', 10],
+ ['b', 2],
+ ['c', 33],
+ ['e', 55],
+ ['d', 4]
+ ],
+ 'replacements keep positions and new keys keep first incoming order'
+ );
+ t.equal(accessor({index: 2}), 33, 'existing accessors read updated values');
+ t.equal(snapshot.getChildAt(1).get(2), 3, 'previous Arrow snapshots stay unchanged');
+ t.equal(state.datasets.live.dataRevision, revision + 1, 'one revision for the whole batch');
+ t.deepEqual(state.datasets.live.allIndexes, [0, 1, 2, 3, 4], 'indexes include appended rows');
+
+ state = reducer(state, VisStateActions.addToDataset('live', [['b', null]], {upsertBy: 'id'}));
+ t.equal(container.numRows(), 5, 'replacement-only upsert keeps row count');
+ t.equal(container.valueAt(1, 1), null, 'replacement-only upsert accepts null values');
+ state = reducer(state, VisStateActions.addToDataset('live', [['f', 6]], {upsertBy: 'id'}));
+ t.deepEqual(container.rowAsArray(5), ['f', 6], 'append-only upsert preserves values');
+
+ state = reducer(state, VisStateActions.removeFromDataset('live', [0, 1, 2, 3, 4, 5]));
+ state = reducer(state, VisStateActions.addToDataset('live', [['g', 7]], {upsertBy: 'id'}));
+ t.deepEqual(
+ container.flattenData(),
+ [['g', 7]],
+ 'upsert retains the schema after deleting all rows'
+ );
+ t.end();
+});
+
+test('VisStateUpdater -> Arrow nested deletion preserves list values and IPC', async t => {
+ for (const [type, values] of [
+ [
+ new arrow.Int32(),
+ [
+ [1, null, 3],
+ [4, 5, 6],
+ [7, null, 9]
+ ]
+ ],
+ [
+ new arrow.Float16(),
+ [
+ [1.5, 2.5],
+ [3, 4],
+ [5.5, 6.5]
+ ]
+ ]
+ ]) {
+ for (const structWrapped of [false, true]) {
+ const list = arrow.vectorFromArray(
+ values,
+ new arrow.List(new arrow.Field('item', type, true))
+ );
+ const column = structWrapped
+ ? arrow.makeVector({
+ type: new arrow.Struct([new arrow.Field('items', list.type, true)]),
+ length: values.length,
+ children: list.data
+ })
+ : list;
+ const source = new arrow.Table({
+ id: arrow.vectorFromArray([1, 2, 3], new arrow.Int32()),
+ values: column
+ });
+ const datasets = await createArrowLiveDatasets(source);
+ const state = {...INITIAL_VIS_STATE, datasets};
+ const readValues = table =>
+ Array.from(table.getChildAt(1), value => Array.from(structWrapped ? value.items : value));
+
+ const removed = reducer(
+ state,
+ VisStateActions.removeFromDataset('live', {field: 'id', values: [2]})
+ );
+ const edited = removed.datasets.live.dataContainer.getTable();
+ t.notEqual(removed, state, `${column.type} deletion succeeds`);
+ t.deepEqual(readValues(edited), [values[0], values[2]], 'retained list values are unchanged');
+ t.deepEqual(
+ readValues(arrow.tableFromIPC(arrow.tableToIPC(edited))),
+ [values[0], values[2]],
+ 'IPC preserves the retained logical values and nulls'
+ );
+ t.deepEqual(readValues(source), values, 'the original Arrow snapshot stays unchanged');
+ }
+ }
+ t.end();
+});
+
+test('VisStateUpdater -> Arrow nested deletion preserves shared dictionaries and IPC', async t => {
+ const dictionary = arrow.vectorFromArray(['a', 'b', 'c'], new arrow.Utf8());
+ const dictionaryType = new arrow.Dictionary(dictionary.type, new arrow.Int8(), 17);
+ const column = arrow.makeVector({
+ type: new arrow.List(new arrow.Field('item', dictionaryType, true)),
+ valueOffsets: new Int32Array([0, 1, 2, 3]),
+ child: arrow.makeData({
+ type: dictionaryType,
+ data: new Int8Array([0, 1, 2]),
+ dictionary
+ })
+ });
+ const source = new arrow.Table({left: column, right: column});
+ const datasets = await createArrowLiveDatasets(source);
+ const state = {...INITIAL_VIS_STATE, datasets};
+ const removed = reducer(state, VisStateActions.removeFromDataset('live', [1]));
+ const edited = removed.datasets.live.dataContainer.getTable();
+ const readValues = table =>
+ table.schema.fields.map((_, i) => Array.from(table.getChildAt(i), value => Array.from(value)));
+ const expected = [
+ [['a'], ['c']],
+ [['a'], ['c']]
+ ];
+
+ t.notEqual(removed, state, 'deletion succeeds for columns sharing a nested dictionary');
+ t.deepEqual(readValues(edited), expected, 'both columns retain their unmatched values');
+ for (const batch of edited.batches) {
+ t.equal(
+ batch.getChildAt(0).data[0].children[0].dictionary,
+ batch.getChildAt(1).data[0].children[0].dictionary,
+ 'the shared dictionary id retains one dictionary'
+ );
+ }
+ t.deepEqual(
+ readValues(arrow.tableFromIPC(arrow.tableToIPC(edited))),
+ expected,
+ 'IPC preserves both nested dictionary columns'
+ );
+ t.deepEqual(
+ readValues(source),
+ [
+ [['a'], ['b'], ['c']],
+ [['a'], ['b'], ['c']]
+ ],
+ 'the original Arrow snapshot stays unchanged'
+ );
+ t.end();
+});
+
+test('VisStateUpdater -> Arrow nested deletion keeps exact timestamps and point rendering', async t => {
+ for (const [type, values] of [
+ [
+ new arrow.TimestampNanosecond('UTC'),
+ [1700000000000000123n, 1700000000000000456n, 1700000000000000789n]
+ ],
+ [
+ new arrow.TimestampMicrosecond('UTC'),
+ [1700000000000123n, 1700000000000456n, 1700000000000789n]
+ ]
+ ]) {
+ for (const listWrapped of [false, true]) {
+ const timestamps = arrow.makeVector({type, data: new BigInt64Array(values)});
+ const child = listWrapped
+ ? arrow.makeVector({
+ type: new arrow.List(new arrow.Field('item', type, true)),
+ valueOffsets: new Int32Array([0, 1, 2, 3]),
+ child: timestamps.data[0]
+ })
+ : timestamps;
+ const history = arrow.makeVector({
+ type: new arrow.Struct([new arrow.Field('time', child.type, true)]),
+ length: values.length,
+ children: child.data
+ });
+ const source = new arrow.Table({
+ lat: arrow.vectorFromArray([37, 38, 39], new arrow.Float64()),
+ lng: arrow.vectorFromArray([-122, -123, -124], new arrow.Float64()),
+ history
+ });
+ const datasets = await createArrowLiveDatasets(source);
+ const point = new PointLayer({
+ id: 'points',
+ dataId: 'live',
+ isVisible: true,
+ ...PointLayer.findDefaultLayerProps(datasets.live).props[0]
+ });
+ const initial = {
+ ...INITIAL_VIS_STATE,
+ datasets,
+ layers: [point],
+ layerData: [{}],
+ layerOrder: [point.id]
+ };
+ const {layer, layerData} = calculateLayerData(point, initial, undefined);
+ const state = {...initial, layers: [layer], layerData: [layerData]};
+ const renderPoints = current => {
+ const dataset = current.datasets.live;
+ return current.layers[0]
+ .renderLayer({
+ data: current.layerData[0],
+ dataset,
+ idx: 0,
+ layerInteraction: {},
+ mapState: INITIAL_MAP_STATE,
+ gpuFilter: dataset.gpuFilter,
+ interactionConfig: current.interactionConfig,
+ visible: true,
+ layerCallbacks: {}
+ })[0]
+ .renderLayers();
+ };
+ t.equal(
+ renderPoints(state).reduce((count, rendered) => count + rendered.props.data.length, 0),
+ 3,
+ 'the original point layer renders all rows'
+ );
+
+ const removed = reducer(state, VisStateActions.removeFromDataset('live', [1]));
+ const edited = removed.datasets.live.dataContainer.getTable();
+ const rawTimestamps = table =>
+ listWrapped
+ ? Array.from(table.getChildAt(2), value => value.time.toArray()[0])
+ : Array.from(
+ new arrow.Vector(table.getChildAt(2).data.map(data => data.children[0])).toArray()
+ );
+ t.notEqual(removed, state, `${history.type} deletion succeeds`);
+ t.equal(edited.numRows, 2, 'only the requested row is deleted');
+ t.deepEqual(
+ rawTimestamps(edited),
+ [values[0], values[2]],
+ 'retained timestamp storage is exact'
+ );
+ t.deepEqual(
+ rawTimestamps(arrow.tableFromIPC(arrow.tableToIPC(edited))),
+ [values[0], values[2]],
+ 'IPC preserves exact nested timestamp storage'
+ );
+ t.deepEqual(rawTimestamps(source), values, 'the original timestamp snapshot stays unchanged');
+ let rendered;
+ t.doesNotThrow(() => {
+ rendered = renderPoints(removed);
+ }, 'the ordinary lat/lng point layer still renders after deletion');
+ if (rendered) {
+ t.equal(
+ rendered.reduce((count, sublayer) => count + sublayer.props.data.length, 0),
+ 2,
+ 'the rendered point layer contains both retained rows'
+ );
+ }
+ }
+ }
+ t.end();
+});
+
+test('VisStateUpdater -> Arrow Int64 upserts match JSON keys and reject invalid batches', async t => {
+ const largeId = 9007199254740993n;
+ const source = new arrow.Table({
+ id: arrow.vectorFromArray([1n, largeId], new arrow.Int64()),
+ value: arrow.vectorFromArray([1, 2], new arrow.Int32())
+ });
+ const datasets = await createArrowLiveDatasets(source);
+ let state = {...INITIAL_VIS_STATE, datasets};
+ state = reducer(
+ state,
+ VisStateActions.addToDataset('live', [{id: String(largeId), value: 20}], {upsertBy: 'id'})
+ );
+ const container = state.datasets.live.dataContainer;
+ t.deepEqual(
+ container.flattenData(),
+ [
+ [1n, 1],
+ [largeId, 20]
+ ],
+ 'a decimal string replaces its exact existing Int64 key without adding a duplicate'
+ );
+
+ state = reducer(
+ state,
+ VisStateActions.addToDataset(
+ 'live',
+ [
+ {id: '01', value: 11},
+ {id: '2', value: 21},
+ {id: 1n, value: 12},
+ {id: '02', value: 22},
+ {id: '1', value: 13},
+ {id: 2n, value: 23}
+ ],
+ {upsertBy: 'id'}
+ )
+ );
+ const expected = [
+ [1n, 13],
+ [largeId, 20],
+ [2n, 23]
+ ];
+ t.deepEqual(
+ container.flattenData(),
+ expected,
+ 'accepted key representations share last-wins matching'
+ );
+ const restored = arrow.tableFromIPC(arrow.tableToIPC(container.getTable()));
+ t.deepEqual(
+ Array.from(restored.getChildAt(0)),
+ [1n, largeId, 2n],
+ 'IPC keeps exact unique Int64 keys'
+ );
+
+ const snapshot = container.getTable();
+ const beforeRejectedRows = container.flattenData();
+ const revision = state.datasets.live.dataRevision;
+ const indexes = state.datasets.live.allIndexes;
+ const warn = sinon.stub(Console, 'warn');
+ try {
+ const rejected = reducer(
+ state,
+ VisStateActions.addToDataset(
+ 'live',
+ [
+ {id: String(largeId), value: 30},
+ {id: 'invalid-int64', value: 40}
+ ],
+ {upsertBy: 'id'}
+ )
+ );
+ t.equal(rejected, state, 'an invalid key rejects the entire public-action batch');
+ t.equal(container.getTable(), snapshot, 'a rejected batch retains the Arrow snapshot');
+ t.deepEqual(
+ container.flattenData(),
+ beforeRejectedRows,
+ 'a rejected batch cannot leak an earlier replacement'
+ );
+ t.equal(state.datasets.live.dataRevision, revision, 'a rejected batch preserves the revision');
+ t.equal(state.datasets.live.allIndexes, indexes, 'a rejected batch preserves row indexes');
+ } finally {
+ warn.restore();
+ }
+ t.deepEqual(Array.from(source.getChildAt(0)), [1n, largeId], 'source keys remain unchanged');
+ t.deepEqual(Array.from(source.getChildAt(1)), [1, 2], 'source values remain unchanged');
+ t.end();
+});
diff --git a/test/node/utils/data-container-test.js b/test/node/utils/data-container-test.js
index 14ac06bb2..1161c9481 100644
--- a/test/node/utils/data-container-test.js
+++ b/test/node/utils/data-container-test.js
@@ -190,6 +190,7 @@ test('RowDataContainer.append and remove', t => {
t.equal(dc.append([[1]]), false, 'should reject a wrong column count');
t.equal(dc.append([]), false, 'should reject an empty batch');
+ t.equal(dc.append(null), false, 'should reject a missing batch');
t.equal(dc.numRows(), 5, 'should leave rows unchanged after a rejected append');
t.equal(dc.replace(0, [11, 22]), true, 'should overwrite a row in place');
@@ -423,6 +424,15 @@ test('ArrowDataContainer -> worker GeoArrow cache follows progressive chunks', t
'large compacted centroid caches should keep every center'
);
+ t.equal(dc.remove([0]), true, 'should remove a row from the cached table');
+ t.equal(dc.valueAt(0, 0), 2, 'should retain only the second row');
+ t.equal(dc.getGeoArrowBinaryData(0), undefined, 'row deletion should discard cached geometry');
+
+ dc.update(batchA, {geoArrowBinaryData: {0: {0: cachedA}}});
+ t.equal(dc.remove([0]), true, 'should remove the final cached row');
+ t.equal(dc.numRows(), 0, 'should leave an empty table');
+ t.equal(dc.getGeoArrowBinaryData(0), undefined, 'empty tables must not retain cached geometry');
+
t.end();
});
@@ -675,3 +685,497 @@ test('compactArrowTable -> Builder failure keeps the original table', t => {
t.end();
});
+
+function arrowContainerFromTable(table) {
+ const cols = Array.from({length: table.numCols}, (_, i) => table.getChildAt(i)).filter(Boolean);
+ return new ArrowDataContainer({
+ cols,
+ fields: table.schema.fields.map((field, fieldIdx) => ({
+ name: field.name,
+ fieldIdx
+ })),
+ arrowTable: table
+ });
+}
+
+test('ArrowDataContainer.append replace and remove', t => {
+ const dc = arrowContainerFromTable(
+ arrow.tableFromJSON([
+ {id: 'a', lat: 1},
+ {id: 'b', lat: 2}
+ ])
+ );
+
+ t.equal(
+ dc.append([
+ ['c', 3],
+ ['d', 4]
+ ]),
+ true,
+ 'should append matching primitive/dictionary rows'
+ );
+ t.equal(dc.numRows(), 4, 'should grow without replacing existing rows');
+ t.equal(dc.valueAt(0, 0), 'a', 'should keep the original first id');
+ t.equal(dc.valueAt(3, 1), 4, 'should read the last appended lat');
+ t.equal(dc.numChunks(), 1, 'append should compact concat batches');
+
+ t.equal(dc.append([[1]]), false, 'should reject a wrong column count');
+ t.equal(dc.append([]), false, 'should reject an empty batch');
+ t.equal(dc.append(null), false, 'should reject a missing batch');
+ t.equal(dc.numRows(), 4, 'should leave rows unchanged after a rejected append');
+
+ t.equal(dc.replace(1, ['B', 22]), true, 'should overwrite a row by concat');
+ t.equal(dc.valueAt(1, 0), 'B', 'should read the replaced id');
+ t.equal(dc.valueAt(1, 1), 22, 'should read the replaced lat');
+ t.equal(dc.replace(99, ['z', 1]), false, 'should reject a replace with an out-of-range index');
+ t.equal(dc.replace(0, ['z']), false, 'should reject a replace with the wrong column count');
+ t.equal(dc.valueAt(0, 0), 'a', 'should leave the row unchanged after a rejected replace');
+
+ t.equal(dc.remove([1, 1, 3]), true, 'should drop unique valid indexes in one rebuild');
+ t.equal(dc.numRows(), 2, 'should keep the remaining rows');
+ t.equal(dc.valueAt(0, 0), 'a', 'should keep the former row 0');
+ t.equal(dc.valueAt(1, 0), 'c', 'should keep the former row 2');
+ t.deepEqual(dc.getPlainIndex(), [0, 1], 'should rebuild indexes after remove');
+
+ t.equal(dc.remove([99]), false, 'should reject an out-of-range index');
+ t.equal(dc.remove([-1]), false, 'should reject a negative index');
+ t.equal(dc.numRows(), 2, 'should leave rows unchanged after a rejected remove');
+
+ t.equal(dc.remove([0, 1]), true, 'should allow removing every remaining row');
+ t.equal(dc.numRows(), 0, 'should keep an empty table with the original schema');
+ t.equal(
+ dc.append([['e', 5]]),
+ true,
+ 'empty table should accept an append that matches the schema'
+ );
+ t.equal(dc.numRows(), 1, 'empty table should grow from the append');
+ t.equal(dc.valueAt(0, 0), 'e', 'appended id should survive an empty-table concat');
+
+ t.end();
+});
+
+test('ArrowDataContainer row edits preserve column positions and schema', t => {
+ for (const names of [
+ ['2', '1'],
+ ['x', 'x', 'x_1'],
+ ['__proto__', 'value']
+ ]) {
+ const original = names.map((_, i) => i + 1);
+ const added = original.map(value => value * 10);
+ const replaced = original.map(value => value * 100);
+ const columns = original.map((value, i) =>
+ arrow.vectorFromArray([value], i % 2 ? new arrow.Float64() : new arrow.Int32())
+ );
+ const fields = names.map(
+ (name, i) => new arrow.Field(name, columns[i].type, true, new Map([['index', String(i)]]))
+ );
+ const schema = new arrow.Schema(fields, new Map([['source', 'row-edit']]));
+ const batch = new arrow.RecordBatch(
+ schema,
+ arrow.makeData({
+ type: new arrow.Struct(fields),
+ length: 1,
+ children: columns.map(column => column.data[0])
+ })
+ );
+ fields.forEach((field, i) => {
+ batch.schema.fields[i] = field;
+ });
+ const dc = arrowContainerFromTable(new arrow.Table(schema, [batch]));
+
+ t.ok(dc.append([added]), `append accepts ${names.join(', ')}`);
+ t.deepEqual(dc.flattenData(), [original, added], 'append keeps column values aligned');
+ t.ok(dc.replace(0, replaced), 'replace succeeds');
+ t.deepEqual(dc.flattenData(), [replaced, added], 'replace keeps every column');
+ t.ok(dc.remove([0]), 'remove succeeds');
+ t.deepEqual(dc.flattenData(), [added], 'remove keeps the remaining values');
+ t.deepEqual(dc.getTable().schema, schema, 'names, types and metadata survive row edits');
+ }
+ t.end();
+});
+
+test('ArrowDataContainer.append rejects nested and geoarrow columns', t => {
+ const originalWarn = console.warn;
+ console.warn = () => {};
+
+ try {
+ const listTable = new arrow.Table({
+ vals: arrow.vectorFromArray([
+ [1, 2],
+ [3, 4]
+ ])
+ });
+ const listDc = arrowContainerFromTable(listTable);
+ t.equal(listDc.append([[[5, 6]]]), false, 'should reject a List column');
+ t.equal(listDc.numRows(), 2, 'rejected append must not mutate a List table');
+ t.equal(listDc.remove([0]), true, 'remove should still concat slices of nested columns');
+ t.equal(listDc.numRows(), 1, 'should drop the nested row by slice');
+
+ const geoMeta = new Map();
+ geoMeta.set(GEOARROW_METADATA_KEY, GEOARROW_EXTENSIONS.WKB);
+ const wkbTable = tableFromColumnValues(
+ 'geometry',
+ new arrow.Binary(),
+ [new Uint8Array([1, 2, 3])],
+ geoMeta
+ );
+ const geoDc = arrowContainerFromTable(wkbTable);
+ t.equal(geoDc.append([[new Uint8Array([4, 5, 6])]]), false, 'should reject a geoarrow column');
+ t.equal(geoDc.numRows(), 1, 'rejected append must not mutate a geoarrow table');
+ } finally {
+ console.warn = originalWarn;
+ }
+ t.end();
+});
+
+test('Arrow row edits preserve retained raw timestamps through compaction', t => {
+ for (const [type, values] of [
+ [
+ new arrow.TimestampMicrosecond('UTC'),
+ [9007199254740993n, 9007199254740995n, 9007199254740997n]
+ ],
+ [
+ new arrow.TimestampNanosecond('UTC'),
+ [1700000000000000123n, 1700000000000000456n, 1700000000000000789n]
+ ]
+ ]) {
+ const timestamp = arrow.makeVector({
+ type,
+ data: new BigInt64Array([0n, ...values]),
+ nullBitmap: new Uint8Array([0b1011]),
+ nullCount: 1
+ });
+ const source = new arrow.Table({
+ id: arrow.vectorFromArray(['skip', 'a', 'b', 'c'], new arrow.Utf8()),
+ time: timestamp
+ }).slice(1);
+ const expected = [values[0], null, values[2]];
+ const raw = table => {
+ const column = table.getChildAt(1);
+ const storage = column.toArray();
+ return Array.from({length: column.length}, (_, i) => (column.isValid(i) ? storage[i] : null));
+ };
+ for (const [name, mutate, retained] of [
+ ['append', dc => dc.append([['d', 0]]), [...expected, 0n]],
+ ['delete middle', dc => dc.remove([1]), [values[0], values[2]]]
+ ]) {
+ const dc = arrowContainerFromTable(source);
+ t.ok(mutate(dc), `${type} ${name} succeeds`);
+ t.deepEqual(raw(dc.getTable()), retained, `${type} ${name} retains exact raw values`);
+ t.deepEqual(
+ raw(arrow.tableFromIPC(arrow.tableToIPC(dc.getTable()))),
+ retained,
+ 'IPC retains raw storage'
+ );
+ t.deepEqual(raw(source), expected, 'original snapshot is unchanged');
+ t.equal(dc.numChunks(), 1, 'row edits respect the compaction limit');
+ }
+ }
+ t.end();
+});
+
+test('Arrow row edits preserve shared dictionary serialization', t => {
+ const column = arrow.vectorFromArray(
+ ['a', null, 'b'],
+ new arrow.Dictionary(new arrow.Utf8(), new arrow.Int8(), 17)
+ );
+ const fields = ['left', '2020'].map(
+ name => new arrow.Field(name, column.type, true, new Map([['source', name]]))
+ );
+ const schema = new arrow.Schema(fields, new Map([['source', 'shared-dictionary']]));
+ const source = new arrow.Table(schema, {0: column, 1: column});
+ const dc = arrowContainerFromTable(source);
+ t.ok(dc.append([['c', 'd']]), 'append succeeds with separate incoming dictionary values');
+ t.deepEqual(
+ dc.flattenData(),
+ [
+ ['a', 'a'],
+ [null, null],
+ ['b', 'b'],
+ ['c', 'd']
+ ],
+ 'values stay aligned'
+ );
+ const assertIPC = expected => {
+ let restored;
+ t.doesNotThrow(() => {
+ restored = arrow.tableFromIPC(arrow.tableToIPC(dc.getTable()));
+ }, 'edited table is serializable');
+ if (restored) {
+ t.deepEqual(
+ restored.schema.fields.map((_, i) => Array.from(restored.getChildAt(i))),
+ expected,
+ 'IPC preserves every dictionary value'
+ );
+ }
+ t.equal(
+ dc.getTable().schema.metadata.get('source'),
+ 'shared-dictionary',
+ 'table metadata survives'
+ );
+ t.deepEqual(
+ dc.getTable().schema.fields.map(field => field.metadata.get('source')),
+ ['left', '2020'],
+ 'field metadata survives'
+ );
+ };
+ assertIPC([
+ ['a', null, 'b', 'c'],
+ ['a', null, 'b', 'd']
+ ]);
+ t.ok(dc.remove([0, 1, 2]), 'deletion succeeds');
+ assertIPC([['c'], ['d']]);
+ t.ok(dc.remove([0]), 'delete all succeeds');
+ t.ok(dc.append([['e', 'f']]), 'refill succeeds');
+ assertIPC([['e'], ['f']]);
+ t.deepEqual(Array.from(source.getChildAt(0)), ['a', null, 'b'], 'source values stay untouched');
+ t.doesNotThrow(() => arrow.tableToIPC(source), 'source remains serializable');
+ t.end();
+});
+
+test('Arrow row edits reject dictionary index overflow atomically', t => {
+ for (const [Index, capacity] of [
+ [arrow.Int8, 128],
+ [arrow.Uint8, 256]
+ ]) {
+ const type = new arrow.Dictionary(new arrow.Utf8(), new Index());
+ for (const empty of [false, true]) {
+ const column = arrow.vectorFromArray(empty ? [] : ['original'], type);
+ const source = new arrow.Table({value: column});
+ const dc = arrowContainerFromTable(source);
+ const snapshot = dc.getTable();
+ const rows = Array.from({length: capacity + 1}, (_, i) => [`value-${i}`]);
+ t.notOk(dc.append(rows), 'oversized incoming dictionary is rejected before indexes wrap');
+ t.equal(dc.getTable(), snapshot, 'rejection retains the snapshot');
+ t.deepEqual(dc.flattenData(), empty ? [] : [['original']], 'rejection changes no values');
+ }
+ const initial = Array.from({length: capacity}, (_, i) => `value-${i}`);
+ const dc = arrowContainerFromTable(
+ new arrow.Table({value: arrow.vectorFromArray(initial, type)})
+ );
+ const snapshot = dc.getTable();
+ t.notOk(dc.append([['overflow']]), 'combined dictionary overflow is rejected');
+ t.equal(dc.getTable(), snapshot, 'combined overflow is atomic');
+ }
+ t.end();
+});
+
+test('Arrow deletion preserves nested timestamp storage without decoding it', t => {
+ const values = [1700000000000000123n, 1700000000000000456n, 1700000000000000789n];
+ const time = arrow.makeVector({
+ type: new arrow.TimestampNanosecond(),
+ data: new BigInt64Array(values)
+ });
+ const type = new arrow.Struct([new arrow.Field('time', time.type, true)]);
+ const nested = arrow.makeVector(arrow.makeData({type, length: 3, children: time.data}));
+ const source = new arrow.Table({nested});
+ const dc = arrowContainerFromTable(source);
+ t.ok(dc.remove([1]), 'nested deletion succeeds');
+ const raw = table =>
+ Array.from(new arrow.Vector(table.getChildAt(0).data.map(data => data.children[0])).toArray());
+ t.deepEqual(raw(dc.getTable()), [values[0], values[2]], 'nested raw values are exact');
+ t.deepEqual(
+ raw(arrow.tableFromIPC(arrow.tableToIPC(dc.getTable()))),
+ [values[0], values[2]],
+ 'nested IPC is exact'
+ );
+ t.deepEqual(raw(source), values, 'original nested buffers are unchanged');
+ t.end();
+});
+
+test('Arrow shared timestamp dictionaries preserve raw values and IPC after edits', t => {
+ const type = new arrow.Dictionary(new arrow.TimestampNanosecond('UTC'), new arrow.Int8(), 17);
+ const timestamp = arrow.makeVector({
+ type: type.dictionary,
+ data: new BigInt64Array([1700000000000000123n, 1700000000000000456n])
+ });
+ const data = arrow.makeData({type, data: new Int8Array([0, 1]), dictionary: timestamp});
+ const column = arrow.makeVector(data);
+ const source = new arrow.Table(
+ new arrow.Schema(['left', 'right'].map(name => new arrow.Field(name, type, true))),
+ {0: column, 1: column}
+ );
+ const dc = arrowContainerFromTable(source);
+ t.ok(dc.append([[0, 1000]]), 'shared timestamp append succeeds');
+ const ipc = arrow.tableFromIPC(arrow.tableToIPC(dc.getTable()));
+ for (let i = 0; i < 2; i++) {
+ const col = ipc.getChildAt(i);
+ const raw = col.data.flatMap(chunk =>
+ Array.from(
+ chunk.values.slice(0, chunk.length),
+ index => chunk.dictionary.data[0].values[index]
+ )
+ );
+ t.deepEqual(
+ raw,
+ [1700000000000000123n, 1700000000000000456n, i ? 1000000000n : 0n],
+ 'shared dictionary keeps exact raw timestamp entries'
+ );
+ }
+ t.equal(
+ dc.getColumn(0).data[0].dictionary,
+ dc.getColumn(1).data[0].dictionary,
+ 'output columns share the same dictionary'
+ );
+ t.doesNotThrow(() => arrow.tableToIPC(source), 'source dictionary stays serializable');
+ t.end();
+});
+
+test('Arrow shared dictionaries retain differing physical index widths', t => {
+ const dictionary = arrow.vectorFromArray(['a', 'b'], new arrow.Utf8());
+ const columns = [new arrow.Int8(), new arrow.Int16()].map(indices =>
+ arrow.makeVector({
+ type: new arrow.Dictionary(dictionary.type, indices, 17),
+ data: new indices.ArrayType([0, 1]),
+ dictionary
+ })
+ );
+ const source = new arrow.Table({narrow: columns[0], wide: columns[1]});
+ const dc = arrowContainerFromTable(source);
+ t.ok(dc.append([['c', 'd']]), 'mixed-width append succeeds');
+ t.ok(dc.remove([0]), 'mixed-width deletion succeeds');
+ const ipc = arrow.tableFromIPC(arrow.tableToIPC(dc.getTable()));
+ t.deepEqual(Array.from(ipc.getChildAt(0)), ['b', 'c'], 'narrow values survive IPC');
+ t.deepEqual(Array.from(ipc.getChildAt(1)), ['b', 'd'], 'wide values survive IPC');
+ t.deepEqual(
+ ipc.schema.fields.map(field => field.type.indices.bitWidth),
+ [8, 16],
+ 'physical index widths are preserved'
+ );
+ t.equal(
+ dc.getColumn(0).data[0].dictionary,
+ dc.getColumn(1).data[0].dictionary,
+ 'both columns retain one shared dictionary'
+ );
+ t.deepEqual(Array.from(source.getChildAt(0)), ['a', 'b'], 'source stays unchanged');
+ t.end();
+});
+
+test('Arrow deletion retains timestamp storage inside nested dictionaries', t => {
+ const time = new arrow.TimestampNanosecond('UTC');
+ const type = new arrow.List(new arrow.Field('item', time));
+ const values = [1234567890123456789n, 1234567890123456791n];
+ const dictionary = arrow.makeVector({
+ type,
+ valueOffsets: new Int32Array([0, 1, 2]),
+ child: arrow.makeData({type: time, data: new BigInt64Array(values)})
+ });
+ const column = arrow.makeVector({
+ type: new arrow.Dictionary(type, new arrow.Int8(), 17),
+ data: new Int8Array([0, 1, 0, 1]),
+ dictionary
+ });
+ const source = new arrow.Table({nested: column});
+ const dc = arrowContainerFromTable(source);
+ t.ok(dc.remove([1]), 'nested dictionary deletion succeeds');
+ const ipc = arrow.tableFromIPC(arrow.tableToIPC(dc.getTable()));
+ t.deepEqual(
+ Array.from(ipc.getChildAt(0), value => value.toArray()[0]),
+ [values[0], values[0], values[1]],
+ 'nested dictionary IPC keeps exact timestamp values'
+ );
+ t.deepEqual(
+ Array.from(source.getChildAt(0), value => value.toArray()[0]),
+ [values[0], values[1], values[0], values[1]],
+ 'source nested dictionary is unchanged'
+ );
+ t.end();
+});
+
+test('Arrow deletion reconciles nested dictionary mappings across progressive batches', t => {
+ const batch = values => {
+ const dictionary = arrow.vectorFromArray(values, new arrow.Utf8());
+ const type = new arrow.Dictionary(dictionary.type, new arrow.Int8(), 17);
+ const column = indexes =>
+ arrow.makeVector({
+ type: new arrow.List(new arrow.Field('item', type, true)),
+ valueOffsets: new Int32Array([0, 1, 2, 3]),
+ child: arrow.makeData({type, data: new Int8Array(indexes), dictionary})
+ });
+ return new arrow.Table({left: column([0, 1, 0]), right: column([1, 0, 1])});
+ };
+ const first = batch(['a', 'b']);
+ const source = first.concat(batch(['b', 'a']));
+ const before = arrow.tableToIPC(source);
+ const dc = arrowContainerFromTable(first);
+ dc.update(source);
+ t.ok(dc.remove([1, 4]), 'deletion accepts different dictionaries in progressive batches');
+ const rows = table =>
+ Array.from({length: table.numRows}, (_, index) =>
+ table.schema.fields.map((_, column) => Array.from(table.getChildAt(column).get(index)))
+ );
+ const expected = [
+ [['a'], ['b']],
+ [['a'], ['b']],
+ [['b'], ['a']],
+ [['b'], ['a']]
+ ];
+ t.deepEqual(rows(dc.getTable()), expected, 'retained indexes use the unified dictionary');
+ t.equal(dc.numChunks(), 1, 'edited table has one batch');
+ t.deepEqual(
+ rows(arrow.tableFromIPC(arrow.tableToIPC(dc.getTable()))),
+ expected,
+ 'shared dictionary survives IPC despite equivalent distinct source type objects'
+ );
+ t.deepEqual(arrow.tableToIPC(source), before, 'progressive source buffers remain unchanged');
+ t.end();
+});
+
+test('compactArrowTable -> nested timestamp fallback keeps exact source storage', t => {
+ const type = new arrow.TimestampNanosecond();
+ const nested = new arrow.List(new arrow.Field('time', type, true));
+ const raw = [1700000000000000123n, 1700000000000000456n];
+ const batches = raw.map(
+ value =>
+ new arrow.Table({
+ times: arrow.makeVector({
+ type: nested,
+ valueOffsets: new Int32Array([0, 1]),
+ child: arrow.makeData({type, data: new BigInt64Array([value])})
+ })
+ })
+ );
+ const source = batches[0].concat(batches[1]);
+ const compacted = compactArrowTable(source, 1);
+ t.deepEqual(
+ compacted.getChildAt(0).data.flatMap(data => Array.from(data.children[0].values)),
+ raw,
+ 'ordinary import compaction must not round nested timestamps through milliseconds'
+ );
+ t.end();
+});
+
+test('Arrow deletion counts only retained nested dictionary entries toward capacity', t => {
+ const batch = prefix => {
+ const dictionary = arrow.vectorFromArray(
+ Array.from({length: 128}, (_, i) => `${prefix}${i}`),
+ new arrow.Utf8()
+ );
+ const type = new arrow.Dictionary(dictionary.type, new arrow.Int8(), 17);
+ const column = arrow.makeVector({
+ type: new arrow.List(new arrow.Field('item', type, true)),
+ valueOffsets: Int32Array.from({length: 129}, (_, i) => i),
+ child: arrow.makeData({
+ type,
+ data: Int8Array.from({length: 128}, (_, i) => i),
+ dictionary
+ })
+ });
+ return new arrow.Table({values: column});
+ };
+ const first = batch('a');
+ const source = first.concat(batch('b'));
+ const original = arrow.tableToIPC(source);
+ const dc = arrowContainerFromTable(first);
+ dc.update(source);
+ const indexes = Array.from({length: 256}, (_, i) => i).filter(i => i !== 0 && i !== 128);
+ t.ok(dc.remove(indexes), 'two retained dictionary values fit Int8 despite deleted entries');
+ const restored = arrow.tableFromIPC(arrow.tableToIPC(dc.getTable()));
+ t.deepEqual(
+ Array.from(restored.getChildAt(0), value => Array.from(value)),
+ [['a0'], ['b0']],
+ 'only the retained values survive IPC'
+ );
+ t.deepEqual(arrow.tableToIPC(source), original, 'source dictionaries are unchanged');
+ t.end();
+});
diff --git a/test/node/utils/duckdb-utils-test.js b/test/node/utils/duckdb-utils-test.js
index c59e29147..4e714cae5 100644
--- a/test/node/utils/duckdb-utils-test.js
+++ b/test/node/utils/duckdb-utils-test.js
@@ -3,6 +3,7 @@
import test from 'tape';
import * as arrow from 'apache-arrow';
+import Console from 'global/console';
import {
SUPPORTED_DUCKDB_DROP_EXTENSIONS,
@@ -23,6 +24,7 @@ import {
sanitizeDuckDBTableName,
quoteTableName
} from '../../../src/duckdb/src/table/duckdb-table-utils';
+import {KeplerGlDuckDbTable} from '../../../src/duckdb/src/table/duckdb-table';
import {GEOARROW_EXTENSIONS, GEOARROW_METADATA_KEY} from '@kepler.gl/constants';
@@ -628,3 +630,33 @@ test('duckdb-utils -> SUPPORTED_DUCKDB_DROP_EXTENSIONS', t => {
t.end();
});
+
+test('KeplerGlDuckDbTable -> row edits stay a warned no-op', t => {
+ const snapshot = arrow.tableFromJSON([{id: 'a', value: 1}]);
+ const table = new KeplerGlDuckDbTable({info: {id: 'duck', label: 'duck'}}).updateSchema({
+ fields: [
+ {name: 'id', type: 'string'},
+ {name: 'value', type: 'real'}
+ ],
+ cols: snapshot.schema.fields.map((_, i) => snapshot.getChildAt(i)),
+ arrowTable: snapshot,
+ rows: []
+ });
+ const originalWarn = Console.warn;
+ const warnings = [];
+ Console.warn = message => warnings.push(String(message));
+
+ try {
+ t.equal(table.appendRows([['b', 2]]), false, 'appendRows should no-op');
+ t.equal(table.upsertRows([['a', 2]], 'id'), false, 'upsertRows should no-op');
+ t.equal(table.removeRows([0]), false, 'removeRows should no-op');
+ t.deepEqual(table.dataContainer.flattenData(), [['a', 1]], 'snapshot stays unchanged');
+ t.ok(
+ warnings.some(message => message.includes('DuckDB INSERT')),
+ 'should warn that DuckDB INSERT is not implemented'
+ );
+ } finally {
+ Console.warn = originalWarn;
+ }
+ t.end();
+});
diff --git a/test/node/utils/kepler-table-test.js b/test/node/utils/kepler-table-test.js
index 7aa6b5a20..d5025bb3a 100644
--- a/test/node/utils/kepler-table-test.js
+++ b/test/node/utils/kepler-table-test.js
@@ -671,6 +671,378 @@ test('KeplerTable -> Arrow snapshots rebuild cached timestamps and reuse them be
t.end();
});
+test('KeplerTable -> Arrow row edits and snapshots keep timestamp filters current', async t => {
+ const firstTime = Date.UTC(2024, 1, 1);
+ const lastTime = firstTime + 86400000;
+ const snapshot = rows => {
+ const source = new arrow.Table({
+ id: arrow.vectorFromArray(
+ rows.map(row => row[0]),
+ new arrow.Utf8()
+ ),
+ time: arrow.vectorFromArray(
+ rows.map(row => row[1]),
+ new arrow.TimestampMillisecond()
+ )
+ });
+ return processArrowBatches(source.batches);
+ };
+ const table = new KeplerTable({info: {id: 'row-edit-timestamps'}, color: [0, 0, 0]});
+ await table.importData({
+ data: snapshot([
+ ['a', firstTime],
+ ['b', lastTime]
+ ])
+ });
+ const container = table.dataContainer;
+ const accessor = table.fields[1].valueAccessor;
+ const filters = [
+ {
+ ...table.getColumnFilterProps('time'),
+ id: 'time',
+ dataId: [table.id],
+ name: ['time'],
+ fieldIdx: [1],
+ value: [firstTime, firstTime],
+ enabled: true,
+ gpu: true,
+ gpuChannel: [0]
+ }
+ ];
+ const check = (values, expected) => {
+ t.equal(table.dataContainer, container, 'container identity survives edits and refresh');
+ t.deepEqual(table.fields[1].filterProps.mappedValue, values, 'timestamp cache follows rows');
+ t.deepEqual(
+ values.map((_, index) => accessor({index})),
+ values,
+ 'original accessor stays bound'
+ );
+ table.filterTable(filters, []);
+ table.filterTableCPU(filters, []);
+ t.deepEqual(table.filteredIdxCPU, expected, 'CPU/export filter follows edited timestamps');
+ const getValue = table.gpuFilter.filterValueAccessor(container)();
+ const visible = table.filteredIndex.filter(index =>
+ getValue({index}).every((value, channel) => {
+ const [min, max] = table.gpuFilter.filterRange[channel];
+ return value >= min && value <= max;
+ })
+ );
+ t.deepEqual(visible, expected, 'GPU filter follows edited timestamps');
+ };
+ check([firstTime, lastTime], [0]);
+ t.ok(table.appendRows([['c', firstTime]]), 'append succeeds');
+ check([firstTime, lastTime, firstTime], [0, 2]);
+ t.ok(
+ table.upsertRows(
+ [
+ ['a', lastTime],
+ ['b', firstTime]
+ ],
+ 'id'
+ ),
+ 'upsert succeeds'
+ );
+ check([lastTime, firstTime, firstTime], [1, 2]);
+ t.ok(table.removeRows([1]), 'delete succeeds');
+ check([lastTime, firstTime], [1]);
+ await table.update(
+ snapshot([
+ ['c', firstTime],
+ ['a', lastTime],
+ ['d', null]
+ ])
+ );
+ check([firstTime, lastTime, null], [0]);
+ t.ok(table.removeRows([0, 1, 2]), 'delete all succeeds');
+ check([], []);
+ t.ok(table.appendRows([['e', firstTime]]), 'append after empty succeeds');
+ check([firstTime], [0]);
+ t.end();
+});
+
+test('KeplerTable -> Arrow upserts match and deduplicate encoded Int64 keys', t => {
+ const existingKey = 9007199254740993n;
+ const newKey = existingKey + 1n;
+ const source = new arrow.Table({
+ id: arrow.vectorFromArray([existingKey], new arrow.Int64()),
+ value: arrow.vectorFromArray([10], new arrow.Int32())
+ });
+ const table = new KeplerTable({info: {id: 'encoded-keys'}, color: [0, 0, 0]}).updateSchema(
+ processArrowBatches(source.batches)
+ );
+ const container = table.dataContainer;
+ const snapshot = container.getTable();
+ const accessor = table.fields[1].valueAccessor;
+ const revision = table.dataRevision;
+
+ t.ok(
+ table.upsertRows([[String(existingKey), 20]], 'id'),
+ 'accepts the JSON decimal-string representation of an Int64 key'
+ );
+ t.deepEqual(container.flattenData(), [[existingKey, 20]], 'replaces the existing encoded key');
+ t.ok(
+ table.upsertRows(
+ [
+ [String(existingKey), 30],
+ [String(newKey), 40],
+ [existingKey, 50],
+ [newKey, 60]
+ ],
+ 'id'
+ ),
+ 'accepts equivalent string and BigInt keys in one edit'
+ );
+ t.deepEqual(
+ container.flattenData(),
+ [
+ [existingKey, 50],
+ [newKey, 60]
+ ],
+ 'last equivalent key wins for both replacements and appended rows'
+ );
+ t.equal(table.dataContainer, container, 'retains the container');
+ t.equal(accessor({index: 0}), 50, 'the original accessor reads the replacement');
+ t.equal(snapshot.getChild('value').get(0), 10, 'retains the previous Arrow snapshot');
+ t.equal(table.dataRevision, revision + 2, 'increments the revision once per accepted edit');
+
+ const currentSnapshot = container.getTable();
+ const currentIndexes = table.allIndexes;
+ const currentRevision = table.dataRevision;
+ for (const invalidKey of ['invalid-int64', '9223372036854775808', '-9223372036854775809']) {
+ t.notOk(
+ table.upsertRows(
+ [
+ [String(existingKey), 70],
+ [invalidKey, 80]
+ ],
+ 'id'
+ ),
+ 'an invalid or out-of-range later key rejects the entire edit'
+ );
+ t.equal(container.getTable(), currentSnapshot, 'rejected keys retain the Arrow snapshot');
+ t.equal(table.allIndexes, currentIndexes, 'rejected keys retain the indexes');
+ t.equal(table.dataRevision, currentRevision, 'rejected keys retain the revision');
+ t.equal(accessor({index: 0}), 50, 'no replacement leaks from the rejected edit');
+ }
+ t.end();
+});
+
+test('KeplerTable -> Arrow upserts reject invalid integer keys before matching', t => {
+ const uint64Max = (1n << 64n) - 1n;
+ for (const {type, existingKeys, invalidKeys} of [
+ {
+ type: new arrow.Int8(),
+ existingKeys: [-128, 127],
+ invalidKeys: [128, '128', -129, 1.5, NaN, Infinity]
+ },
+ {
+ type: new arrow.Uint64(),
+ existingKeys: [0n, uint64Max],
+ invalidKeys: [uint64Max + 1n, String(uint64Max + 1n), -1n, '-1']
+ }
+ ]) {
+ const source = new arrow.Table({
+ id: arrow.vectorFromArray(existingKeys, type),
+ value: arrow.vectorFromArray([10, 20], new arrow.Int32())
+ });
+ const table = new KeplerTable({info: {id: 'integer-keys'}, color: [0, 0, 0]}).updateSchema(
+ processArrowBatches(source.batches)
+ );
+ t.ok(
+ table.upsertRows([[String(existingKeys[1]), 30]], 'id'),
+ `accepts a numeric string at the ${type} upper boundary`
+ );
+ const expected = [
+ [existingKeys[0], 10],
+ [existingKeys[1], 30]
+ ];
+ t.deepEqual(table.dataContainer.flattenData(), expected, 'matches the stored boundary key');
+ const snapshot = table.dataContainer.getTable();
+ const revision = table.dataRevision;
+ for (const key of invalidKeys) {
+ t.notOk(
+ table.upsertRows(
+ [
+ [existingKeys[0], 40],
+ [key, 50]
+ ],
+ 'id'
+ ),
+ `rejects invalid ${type} key ${String(key)} before matching`
+ );
+ t.equal(table.dataContainer.getTable(), snapshot, 'retains the snapshot');
+ t.equal(table.dataRevision, revision, 'retains the revision');
+ t.deepEqual(table.dataContainer.flattenData(), expected, 'does not overwrite any row');
+ }
+ }
+ t.end();
+});
+
+test('KeplerTable -> Arrow upserts match encoded Date and timestamp keys', t => {
+ const first = Date.UTC(2024, 0, 1);
+ const next = first + 86400000;
+ for (const type of [new arrow.DateMillisecond(), new arrow.TimestampMillisecond()]) {
+ const source = new arrow.Table({
+ time: arrow.vectorFromArray([first], type),
+ value: arrow.vectorFromArray([1], new arrow.Int32())
+ });
+ const table = new KeplerTable({info: {id: 'date-keys'}, color: [0, 0, 0]}).updateSchema(
+ processArrowBatches(source.batches)
+ );
+ t.ok(
+ table.upsertRows(
+ [
+ [new Date(first), 2],
+ [new Date(next), 3],
+ [next, 4]
+ ],
+ 'time'
+ ),
+ `accepts Date keys for ${type}`
+ );
+ t.deepEqual(
+ table.dataContainer.flattenData(),
+ [
+ [first, 2],
+ [next, 4]
+ ],
+ 'matches existing keys and deduplicates equivalent Date and millisecond keys'
+ );
+ }
+ t.end();
+});
+
+test('KeplerTable -> row upserts keep distinct number and string keys', t => {
+ const table = new KeplerTable({info: {id: 'row-keys'}, color: [0, 0, 0]}).updateSchema({
+ fields: [
+ {name: 'id', type: 'string'},
+ {name: 'value', type: 'integer'}
+ ],
+ rows: [[1, 10]]
+ });
+ t.ok(
+ table.upsertRows(
+ [
+ ['1', 20],
+ [1, 30]
+ ],
+ 'id'
+ ),
+ 'accepts row keys without encoding'
+ );
+ t.deepEqual(
+ table.dataContainer.flattenData(),
+ [
+ [1, 30],
+ ['1', 20]
+ ],
+ 'row containers retain exact key equality'
+ );
+ t.end();
+});
+
+test('KeplerTable -> Arrow upserts preserve untouched raw timestamps and nulls', t => {
+ const rawValues = vector =>
+ vector.data.flatMap(chunk =>
+ Array.from({length: chunk.length}, (_, index) =>
+ chunk.getValid(index) ? chunk.values[index] : null
+ )
+ );
+
+ for (const [Timestamp, first, unitsPerMillisecond] of [
+ [arrow.TimestampMicrosecond, 9007199254740993n, 1000n],
+ [arrow.TimestampNanosecond, 1700000000000000123n, 1000000n]
+ ]) {
+ const source = new arrow.Table({
+ id: arrow.vectorFromArray(['padding', 'a', 'b', 'c', 'd'], new arrow.Utf8()),
+ time: arrow.makeVector({
+ type: new Timestamp('America/New_York'),
+ data: new BigInt64Array([0n, first, first + 2n, first + 4n, 0n]),
+ nullBitmap: new Uint8Array([0b01111]),
+ nullCount: 1
+ })
+ }).slice(1);
+ const original = [first, first + 2n, first + 4n, null];
+ t.ok(source.getChild('time').data[0].offset > 0, 'fixture uses a nonzero Arrow offset');
+ t.deepEqual(rawValues(source.getChild('time')), original, 'fixture keeps raw timestamp units');
+ const table = new KeplerTable({info: {id: 'raw-timestamps'}, color: [0, 0, 0]}).updateSchema(
+ processArrowBatches(source.batches)
+ );
+ const container = table.dataContainer;
+ const accessor = table.fields[1].valueAccessor;
+ t.ok(
+ table.upsertRows(
+ [
+ ['new-b', 2000],
+ ['a', 1000],
+ ['new-a', 3000],
+ ['a', 4000],
+ ['new-b', 5000]
+ ],
+ 'id'
+ ),
+ 'mixed replacement and insertion succeeds'
+ );
+ const expected = [
+ 4000n * unitsPerMillisecond,
+ first + 2n,
+ first + 4n,
+ null,
+ 5000n * unitsPerMillisecond,
+ 3000n * unitsPerMillisecond
+ ];
+ t.equal(table.dataContainer, container, 'upsert retains the container');
+ t.equal(accessor({index: 0}), 4000, 'the existing accessor reads the last replacement');
+ t.deepEqual(
+ Array.from(container.getColumn(0)),
+ ['a', 'b', 'c', 'd', 'new-b', 'new-a'],
+ 'existing positions and first incoming key order survive repeated keys'
+ );
+ t.deepEqual(
+ rawValues(container.getColumn(1)),
+ expected,
+ 'untouched raw timestamps survive upsert'
+ );
+ const restored = arrow.tableFromIPC(arrow.tableToIPC(container.getTable()));
+ t.deepEqual(
+ rawValues(restored.getChild('time')),
+ expected,
+ 'IPC preserves raw timestamps and nulls'
+ );
+ t.equal(
+ restored.schema.fields[1].type.timezone,
+ 'America/New_York',
+ 'IPC preserves the timezone'
+ );
+ t.deepEqual(
+ rawValues(source.getChild('time')),
+ original,
+ 'upsert does not mutate the source snapshot'
+ );
+
+ const snapshot = container.getTable();
+ const revision = table.dataRevision;
+ t.notOk(
+ table.upsertRows(
+ [
+ ['a', 6000],
+ ['invalid', 'not-a-timestamp']
+ ],
+ 'id'
+ ),
+ 'a later invalid timestamp rejects the whole batch'
+ );
+ t.equal(container.getTable(), snapshot, 'a rejected upsert retains the Arrow snapshot');
+ t.equal(table.dataRevision, revision, 'a rejected upsert does not advance the revision');
+ t.deepEqual(
+ rawValues(container.getColumn(1)),
+ expected,
+ 'a rejected upsert changes no raw values'
+ );
+ }
+ t.end();
+});
+
test('KeplerTable -> update DuckDB cols payload keeps fields and gpuFilter', async t => {
const first = arrow.tableFromArrays({
lat: [1, 2],