Kepler.gl is a powerful open source geospatial analysis tool for large-scale data sets.
[RFC] table class RFC (#1109)
Shan He306bb926cf47
diff --git a/docs/RFC/table-class.md b/docs/RFC/table-class.mdnew file mode 100644index 000000000..14e703ea2--- /dev/null+++ b/docs/RFC/table-class.md@@ -0,0 +1,126 @@+Replace kepler.gl dataset with table class++## Current Dataset structure++```js+export type KeplerDataset = {+ id: string,+ label: string,+ color: number[], // rgb++ // fields and row+ fields: KeplerField[],+ allData: any[][],++ // for Filter+ allIndexes: number[],+ filteredIndex: number[],+ filteredIdxCPU: number[],+ filteredIndexForDomain: number[],++ gpuFilter: {+ filterRange: number[][],+ filterValueUpdateTriggers: any,+ filterValueAccessor: any+ },+ filterRecord: {+ dynamicDomain: any,+ fixedDomain: any,+ cpu: any,+ gpu: any+ },+ filterRecordCPU: {+ dynamicDomain: any,+ fixedDomain: any,+ cpu: any,+ gpu: any+ }+ changedFilters: any,++ // for suggest lat + lng column pairs, and can be extend to also support auto assign visuals+ fieldPairs: {+ defaultName: string,+ pair: any,+ suffix: string[]+ }[],++ sortColumn?: {+ // column name: sorted idx+ [key: string]: number[]+ },+ sortOrder?: string, // ASCENDING | DESCENDING | UNSORT++ pinnedColumns?: string[],+ // table-injected metadata+ metadata?: object+};++export type KeplerField = {+ analyzerType: string,+ id: string,+ name: string,+ format: string,+ tableFieldIndex: numberstring,+ type: string,++ // meta data, storing domain and mappedValues+ filterProps?: any+};+```++### Table class should handle:++#### Add and remove columns (partially supported)++Missing features++- Copy over column meta, if replacing an existing column+- Recompute fieldPairs++#### Current kepler.gl methods:++1. `initalize`++to replace utils/dataset-utils `createNewDataEntry` This table method takes a `ProtoTable` and return a `Table` to be saved in kepler state.++```js+table.initalize(protoTable);++// following meta data should be initialized+table.allIndexes,+ table.filteredIndex,+ table.filteredIndexForDomain,+ table.fieldPairs,+ table.gpuFilter;+```++2. `getValue(columnName, rowIdx)`+ Get the value of a cell+ Should be a curried function+ getValue(columName) = idx => value++3. `getColumnDomain(columnName)`+ to replace `filter-utils.js` `getFieldDomain`+ Get the value domain of a specific column used in filter++4. `getColumnScaleDomain(columnName, scaleType)`+ to replace `base-layer.js` `layer.calculateLayerDomain`+ Get the domain of this column based on scale type++5. `getSampleData(rows)`+ Get a sample of rows to calculate layer boundaries++6. `parseFieldValue(value, type) => string`+ Parse cell value based on column type and return a string representation+ Value the field value, type the field type++7. `sortDatasetByColumn`+ Sort a column, return an sorted index, assign to `sortColumn`, `sortOrder`++8. `filterDataset(filters, layers, opt)`+ to replace `filter-utils.js` `filterDataset`+ Apply filters to dataset, return the filtered dataset with updated `gpuFilter`, `filterRecord`, `filteredIndex`, `filteredIndexForDomain`++9. `filterDatasetCPU(filters)`+ to replace `filter-utils.js` `filterDatasetCPU`+ Apply filters to a dataset all on CPU, assign to `filteredIdxCPU`, `filterRecordCPU`diff --git a/src/utils/dataset-utils.js b/src/utils/dataset-utils.jsindex 7883b24d1..42f7a032d 100644--- a/src/utils/dataset-utils.js+++ b/src/utils/dataset-utils.js@@ -18,26 +18,6 @@// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN// THE SOFTWARE.-// Copyright (c) 2020 Uber Technologies, Inc.-//-// Permission is hereby granted, free of charge, to any person obtaining a copy-// of this software and associated documentation files (the "Software"), to deal-// in the Software without restriction, including without limitation the rights-// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell-// copies of the Software, and to permit persons to whom the Software is-// furnished to do so, subject to tdahe following conditions:-//-// The above copyright notice and this permission notice shall be included in-// all copies or substantial portions of the Software.-//-// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR-// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,-// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE-// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER-// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,-// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN-// THE SOFTWARE.-import {hexToRgb} from './color-utils';import uniq from 'lodash.uniq';import {TRIP_POINT_FIELDS, SORT_ORDER} from 'constants/default-settings';