Skip to content

Kepler.gl is a powerful open source geospatial analysis tool for large-scale data sets.

[RFC] table class RFC (#1109)

Shan He306bb926cf47
diff --git a/docs/RFC/table-class.md b/docs/RFC/table-class.md
new file mode 100644
index 000000000..14e703ea2
--- /dev/null
+++ b/docs/RFC/table-class.md
@@ -0,0 +1,126 @@
+Replace kepler.gl dataset with table class
+
+## Current Dataset structure
+
+```js
+export type KeplerDataset = {
+ id: string,
+ label: string,
+ color: number[], // rgb
+
+ // fields and row
+ fields: KeplerField[],
+ allData: any[][],
+
+ // for Filter
+ allIndexes: number[],
+ filteredIndex: number[],
+ filteredIdxCPU: number[],
+ filteredIndexForDomain: number[],
+
+ gpuFilter: {
+ filterRange: number[][],
+ filterValueUpdateTriggers: any,
+ filterValueAccessor: any
+ },
+ filterRecord: {
+ dynamicDomain: any,
+ fixedDomain: any,
+ cpu: any,
+ gpu: any
+ },
+ filterRecordCPU: {
+ dynamicDomain: any,
+ fixedDomain: any,
+ cpu: any,
+ gpu: any
+ }
+ changedFilters: any,
+
+ // for suggest lat + lng column pairs, and can be extend to also support auto assign visuals
+ fieldPairs: {
+ defaultName: string,
+ pair: any,
+ suffix: string[]
+ }[],
+
+ sortColumn?: {
+ // column name: sorted idx
+ [key: string]: number[]
+ },
+ sortOrder?: string, // ASCENDING | DESCENDING | UNSORT
+
+ pinnedColumns?: string[],
+ // table-injected metadata
+ metadata?: object
+};
+
+export type KeplerField = {
+ analyzerType: string,
+ id: string,
+ name: string,
+ format: string,
+ tableFieldIndex: numberstring,
+ type: string,
+
+ // meta data, storing domain and mappedValues
+ filterProps?: any
+};
+```
+
+### Table class should handle:
+
+#### Add and remove columns (partially supported)
+
+Missing features
+
+- Copy over column meta, if replacing an existing column
+- Recompute fieldPairs
+
+#### Current kepler.gl methods:
+
+1. `initalize`
+
+to replace utils/dataset-utils `createNewDataEntry` This table method takes a `ProtoTable` and return a `Table` to be saved in kepler state.
+
+```js
+table.initalize(protoTable);
+
+// following meta data should be initialized
+table.allIndexes,
+ table.filteredIndex,
+ table.filteredIndexForDomain,
+ table.fieldPairs,
+ table.gpuFilter;
+```
+
+2. `getValue(columnName, rowIdx)`
+ Get the value of a cell
+ Should be a curried function
+ getValue(columName) = idx => value
+
+3. `getColumnDomain(columnName)`
+ to replace `filter-utils.js` `getFieldDomain`
+ Get the value domain of a specific column used in filter
+
+4. `getColumnScaleDomain(columnName, scaleType)`
+ to replace `base-layer.js` `layer.calculateLayerDomain`
+ Get the domain of this column based on scale type
+
+5. `getSampleData(rows)`
+ Get a sample of rows to calculate layer boundaries
+
+6. `parseFieldValue(value, type) => string`
+ Parse cell value based on column type and return a string representation
+ Value the field value, type the field type
+
+7. `sortDatasetByColumn`
+ Sort a column, return an sorted index, assign to `sortColumn`, `sortOrder`
+
+8. `filterDataset(filters, layers, opt)`
+ to replace `filter-utils.js` `filterDataset`
+ Apply filters to dataset, return the filtered dataset with updated `gpuFilter`, `filterRecord`, `filteredIndex`, `filteredIndexForDomain`
+
+9. `filterDatasetCPU(filters)`
+ to replace `filter-utils.js` `filterDatasetCPU`
+ Apply filters to a dataset all on CPU, assign to `filteredIdxCPU`, `filterRecordCPU`
diff --git a/src/utils/dataset-utils.js b/src/utils/dataset-utils.js
index 7883b24d1..42f7a032d 100644
--- a/src/utils/dataset-utils.js
+++ b/src/utils/dataset-utils.js
@@ -18,26 +18,6 @@
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
// THE SOFTWARE.
-// Copyright (c) 2020 Uber Technologies, Inc.
-//
-// Permission is hereby granted, free of charge, to any person obtaining a copy
-// of this software and associated documentation files (the "Software"), to deal
-// in the Software without restriction, including without limitation the rights
-// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-// copies of the Software, and to permit persons to whom the Software is
-// furnished to do so, subject to tdahe following conditions:
-//
-// The above copyright notice and this permission notice shall be included in
-// all copies or substantial portions of the Software.
-//
-// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-// THE SOFTWARE.
-
import {hexToRgb} from './color-utils';
import uniq from 'lodash.uniq';
import {TRIP_POINT_FIELDS, SORT_ORDER} from 'constants/default-settings';