Kepler.gl is a powerful open source geospatial analysis tool for large-scale data sets.
feat: add KML, GPX, TCX, and GeoJSONL loaders (#3697)
Igor Dykhtac3aeb45b40df
diff --git a/docs/user-guides/b-kepler-gl-workflow/a-add-data-to-the-map.md b/docs/user-guides/b-kepler-gl-workflow/a-add-data-to-the-map.mdindex fa1d67788..ba83a27c5 100644--- a/docs/user-guides/b-kepler-gl-workflow/a-add-data-to-the-map.md+++ b/docs/user-guides/b-kepler-gl-workflow/a-add-data-to-the-map.md@@ -11,8 +11,8 @@| | ||---|---|-| **Local files** | <span style="font-weight:normal">Upload CSV / GeoJSON files. Because data is only stored in your browser, there is a **250mb** limit on how much data Chrome allows you to upload into a browser. For datasets larger than **250mb** you should directly load them from a remote URL. See below.<span> |-| **From URL** | Directly load data or map json by pasting a remote URL. You can link it to CSV | JSON | Kepler.gl config json. Make sure the url contains the file extension. CORS policy must be defined on your custom url domain. |+| **Local files** | <span style="font-weight:normal">Upload CSV, GeoJSON, GeoJSONL, KML, GPX, TCX, Arrow, or Parquet files. Files are parsed in your browser, so available memory limits how large a file you can load. For data that does not fit in memory, use a tileset instead.<span> |+| **From URL** | Directly load data or map json by pasting a remote URL. Supported extensions include `csv`, `geojson`, `json`, `geojsonl`, `kml`, `gpx`, `tcx`, `arrow`, and `parquet`. Make sure the url contains the file extension. CORS policy must be defined on your custom url domain. || **Sample data** | Load one of kepler.gl’s sample datasets. The sample map data and config are directly loaded from [kepler.gl-data github][kepler.gl-data-github] repo |@@ -24,6 +24,10 @@ Geometry coordinates should be presented with a geographic coordinate reference## Supported File Formats- [CSV](#csv)- [GeoJSON](#geojson)+ - [GeoJSONL / NDJSON](#geojsonl--ndjson)+ - [KML](#kml)+ - [GPX](#gpx)+ - [TCX](#tcx)- [GeoArrow](#geoarrow)- [kepler.gl Json](#keplergl-json)@@ -191,6 +195,31 @@ kepler.gl will read styles from GeoJSON files. If you are a GeoJSON expert, you}```+### GeoJSONL / NDJSON++kepler.gl accepts newline-delimited GeoJSON: one [Feature](https://tools.ietf.org/html/rfc7946#section-3.2) per line. Common extensions are `.geojsonl`, `.ndgeojson`, `.ldgeojson`, `.jsonl`, and `.ndjson`. Features are combined into a FeatureCollection and visualized like a GeoJSON file.++Generic NDJSON (one JSON object per line, without GeoJSON `geometry`) is loaded as a table, the same way as CSV.++Example `places.geojsonl`:++```json+{"type":"Feature","properties":{"name":"alpha"},"geometry":{"type":"Point","coordinates":[-122.4,37.8]}}+{"type":"Feature","properties":{"name":"beta"},"geometry":{"type":"Point","coordinates":[-122.5,37.9]}}+```++### KML++kepler.gl accepts `.kml` files. Placemarks are converted to GeoJSON features and visualized with a **Polygon** layer. `.kmz` (zipped KML) is not supported; unzip the archive and upload the `.kml` file. KML styles, overlays, and network links are not preserved.++### GPX++kepler.gl accepts `.gpx` files. Waypoints, tracks, and routes are converted to GeoJSON features. NMEA log files are not supported.++### TCX++kepler.gl accepts Garmin `.tcx` activity files. Track points are converted to GeoJSON features.+### GeoArrow[GeoArrow](https://geoarrow.org/) file, a binary data format which can be visualized with the [PolygonLayer](https://docs.kepler.gl/docs/user-guides/c-types-of-layers/e-polygon).@@ -201,9 +230,9 @@ JSON file exported from kepler.gl. See "[Export Map as JSON](https://docs.kepler### Load Map Using URL-You load data or map through custom URL. It currently supports URLs with file extension of `csv`, `json` and `kepler.gl.json`+You load data or map through custom URL. It currently supports URLs with file extension of `csv`, `json`, `geojson`, `geojsonl`, `kml`, `gpx`, `tcx`, `arrow`, `parquet`, and `kepler.gl.json`.-In addition, this also by-passes 250mb file upload size limit which allows you to upload larger file to Kepler.+A remote URL still loads the full file into memory. For datasets that do not fit in memory, use a tileset (vector tiles, COG, or PMTiles).@@ -227,7 +256,7 @@ To add additional datasets to your map:-2. Choose one of the options above: upload a JSON/CSV file, or use Kepler.gl’s sample data.+2. Choose one of the options above: upload a supported file, paste a URL, or use Kepler.gl’s sample data.3. Repeat as needed. There is no limit on the number of datasets you can add. However, adding too many might cause its performance to suffer.diff --git a/docs/user-guides/c-types-of-layers/e-polygon.md b/docs/user-guides/c-types-of-layers/e-polygon.mdindex 61a6a0415..ab71995d0 100644--- a/docs/user-guides/c-types-of-layers/e-polygon.md+++ b/docs/user-guides/c-types-of-layers/e-polygon.md@@ -2,7 +2,7 @@Polygon layer can display all geometry types defined by [RFC 7946 (GeoJSON)](https://tools.ietf.org/html/rfc7946): `Point`, `LineString`, `Polygon`, `MultiPoint`, `MultiLineString`, `MultiPolygon`.-You can load a GeoJSON file (with a single [`Feature`](https://tools.ietf.org/html/rfc7946#section-3.2) or a [`FeatureCollection`](https://tools.ietf.org/html/rfc7946#section-3.3)) or a GeoArrow file.+You can load a GeoJSON file (with a single [`Feature`](https://tools.ietf.org/html/rfc7946#section-3.2) or a [`FeatureCollection`](https://tools.ietf.org/html/rfc7946#section-3.3)), newline-delimited GeoJSON (`.geojsonl`), KML, GPX, TCX, or a GeoArrow file. KML, GPX, TCX, and GeoJSONL are converted to GeoJSON features.diff --git a/docs/user-guides/i-FAQ.md b/docs/user-guides/i-FAQ.mdindex de3ce9285..b9f799771 100644--- a/docs/user-guides/i-FAQ.md+++ b/docs/user-guides/i-FAQ.md@@ -10,7 +10,7 @@ You can create a react redux app with kepler.gl as your map display component, rThere is no limit, but the more datasets you add, the more likely it is that performance will suffer.## What is the maximum file upload size?-Kepler.gl accepts files no larger than 250mb in chrome. You can load bigger files in safari, but performance will be limited.+kepler.gl does not enforce a file size cap. Files are parsed in the browser, so the limit is available memory. Very large files can freeze or crash the tab. For data that does not fit in memory, use a tileset (vector tiles, COG, or PMTiles) instead of a single file.## How many layers can I add to a map?There is no limit on the amount of layers you can add. However, note that the more layers you have on your map, the more likely it is that performance will suffer.diff --git a/docs/user-guides/j-get-started.md b/docs/user-guides/j-get-started.mdindex b6ea73c40..94d99202f 100644--- a/docs/user-guides/j-get-started.md+++ b/docs/user-guides/j-get-started.md@@ -7,7 +7,7 @@ Kepler.gl is a tool designed for geospatial data analysis. This guide will help-Kepler.gl will prompt you to add data to your map as soon as you open the web page. Upload your own CSV or GEOJSON file, or add kepler.gl sample data.+Kepler.gl will prompt you to add data to your map as soon as you open the web page. Upload your own CSV, GeoJSON, KML, or other [supported file](./b-kepler-gl-workflow/a-add-data-to-the-map.md#supported-file-formats), or add kepler.gl sample data.Sample data is a great way to explore and get familiar with kepler.gl’s features.Read more about adding [Add data to the map](./b-kepler-gl-workflow/a-add-data-to-the-map.md).diff --git a/package.json b/package.jsonindex 3cc0d0c72..5c137770d 100644--- a/package.json+++ b/package.json@@ -263,6 +263,7 @@"@loaders.gl/draco": "4.4.1","@loaders.gl/gltf": "4.4.1","@loaders.gl/json": "4.4.1",+ "@loaders.gl/kml": "4.4.1","@loaders.gl/loader-utils": "4.4.1","@loaders.gl/polyfills": "4.4.1","@loaders.gl/arrow": "4.4.1",diff --git a/src/components/src/common/file-uploader/file-upload.tsx b/src/components/src/common/file-uploader/file-upload.tsxindex f74a5f072..169659e02 100644--- a/src/components/src/common/file-uploader/file-upload.tsx+++ b/src/components/src/common/file-uploader/file-upload.tsx@@ -10,15 +10,16 @@ import FileUploadProgress from './file-upload-progress';import FileDrop from './file-drop';import {FileLoading, FileLoadingProgress} from '@kepler.gl/types';-import {GUIDES_FILE_FORMAT_DOC, REMOTE_FILE_FORMATS} from '@kepler.gl/constants';+import {GUIDES_FILE_FORMAT_DOC} from '@kepler.gl/constants';import {FormattedMessage} from '@kepler.gl/localization';import {fetchRemoteFileAsKeplerFile,+ getAcceptedRemoteFileFormats,getFileNameForRemoteUrl,isRemoteDatasetUrl} from '@kepler.gl/processors';import {media} from '@kepler.gl/styles';-import {isChrome, getApplicationConfig} from '@kepler.gl/utils';+import {getApplicationConfig} from '@kepler.gl/utils';import Markdown from 'markdown-to-jsx';import {Button, InputLight} from '../styled-components';@@ -103,6 +104,10 @@ const StyledDragNDropIcon = styled.div`const StyledFileTypeFow = styled.div`width: 100%;+ display: flex;+ flex-wrap: wrap;+ justify-content: center;+ gap: 8px 12px;`;const StyledFileUpload = styled.div`@@ -224,6 +229,8 @@ type FileUploadProps = {fileFormatNames?: string[];/** A list of typically 3 letter extensions (without '.') for file matching */fileExtensions?: string[];+ /** Extensions shown as icons. Defaults to `fileExtensions`. */+ displayedFileExtensions?: string[];/** Set to true if app wants to do its own file filtering */disableExtensionFilter?: boolean;} & WrappedComponentProps;@@ -414,7 +421,10 @@ function FileUploadFactory() {remoteProgress} = this.state;const {fileLoading, fileLoadingProgress, theme, intl} = this.props;- const {fileExtensions = [], fileFormatNames = []} = this.props;+ const {fileExtensions = [], fileFormatNames = [], displayedFileExtensions} = this.props;+ const iconExtensions = displayedFileExtensions?.length+ ? displayedFileExtensions+ : fileExtensions;const showFormatSelector = getApplicationConfig().enableRemoteFileFormatSelector;const fileUploadInfoText = `${intl.formatMessage({@@ -450,7 +460,7 @@ function FileUploadFactory() {<StyledFileDrop $dragOver={dragOver}><StyledDropBody><StyledFileTypeFow className="file-type-row">- {fileExtensions.map(ext => (+ {iconExtensions.map(ext => (<FileType key={ext} ext={ext} height="50px" fontSize="9px" />))}</StyledFileTypeFow>@@ -519,7 +529,7 @@ function FileUploadFactory() {onChange={this._onRemoteFormatChange}disabled={remoteLoading}>- {REMOTE_FILE_FORMATS.map(format => (+ {getAcceptedRemoteFileFormats().map(format => (<option key={format} value={format}>{format === 'auto'? intl.formatMessage({id: 'fileUploader.formatAuto'})@@ -555,7 +565,7 @@ function FileUploadFactory() {) : null}<WarningMsg>- {isChrome() ? <FormattedMessage id={'fileUploader.chromeMessage'} /> : ''}+ <FormattedMessage id={'fileUploader.chromeMessage'} /></WarningMsg></StyledFileUpload>);diff --git a/src/components/src/modal-container.tsx b/src/components/src/modal-container.tsxindex 251ac8b2f..4deff7ee5 100644--- a/src/components/src/modal-container.tsx+++ b/src/components/src/modal-container.tsx@@ -11,6 +11,7 @@ import {exportData,getFileFormatNames,getFileExtensions,+ getDisplayedFileExtensions,MapStyle,ProviderState} from '@kepler.gl/reducers';@@ -375,6 +376,7 @@ export default function ModalContainerFactory(fileLoadingProgress={visState.fileLoadingProgress}fileFormatNames={getFileFormatNames(this.props.visState)}fileExtensions={getFileExtensions(this.props.visState)}+ displayedFileExtensions={getDisplayedFileExtensions(this.props.visState)}/>);modalProps = {diff --git a/src/components/src/modals/load-data-modal.tsx b/src/components/src/modals/load-data-modal.tsxindex c456c2c61..ea5032974 100644--- a/src/components/src/modals/load-data-modal.tsx+++ b/src/components/src/modals/load-data-modal.tsx@@ -50,6 +50,8 @@ type LoadDataModalProps = {fileFormatNames: string[];/** A list of typically 3 letter extensions (without '.') for file matching */fileExtensions: string[];+ /** Extensions shown as icons in the uploader. Defaults to `fileExtensions`. */+ displayedFileExtensions?: string[];isCloudMapLoading: boolean;/** Set to true if app wants to do its own file filtering */disableExtensionFilter?: boolean;diff --git a/src/constants/src/dataset.ts b/src/constants/src/dataset.tsindex 7692ce548..5c1250117 100644--- a/src/constants/src/dataset.ts+++ b/src/constants/src/dataset.ts@@ -11,7 +11,19 @@ export enum DatasetType {BITMAP = 'bitmap'}-export const REMOTE_FILE_FORMATS = ['auto', 'csv', 'geojson', 'json', 'arrow', 'parquet'] as const;+export const REMOTE_FILE_FORMATS = [+ 'auto',+ 'csv',+ 'geojson',+ 'json',+ 'arrow',+ 'parquet',+ 'geojsonl',+ 'ndjson',+ 'kml',+ 'gpx',+ 'tcx'+] as const;export type RemoteFileFormat = (typeof REMOTE_FILE_FORMATS)[number];export const REMOTE_FILE_MIME_TYPES: Record<Exclude<RemoteFileFormat, 'auto'>, string> = {@@ -19,7 +31,12 @@ export const REMOTE_FILE_MIME_TYPES: Record<Exclude<RemoteFileFormat, 'auto'>, sgeojson: 'application/geo+json',json: 'application/json',arrow: 'application/vnd.apache.arrow.file',- parquet: 'application/vnd.apache.parquet'+ parquet: 'application/vnd.apache.parquet',+ geojsonl: 'application/geo+x-ndjson',+ ndjson: 'application/x-ndjson',+ kml: 'application/vnd.google-earth.kml+xml',+ gpx: 'application/gpx+xml',+ tcx: 'application/vnd.garmin.tcx+xml'};export const REMOTE_FILE_EXTENSIONS: Record<Exclude<RemoteFileFormat, 'auto'>, string> = {@@ -27,7 +44,12 @@ export const REMOTE_FILE_EXTENSIONS: Record<Exclude<RemoteFileFormat, 'auto'>, sgeojson: 'geojson',json: 'json',arrow: 'arrow',- parquet: 'parquet'+ parquet: 'parquet',+ geojsonl: 'geojsonl',+ ndjson: 'ndjson',+ kml: 'kml',+ gpx: 'gpx',+ tcx: 'tcx'};export const MIME_TO_REMOTE_FILE_EXTENSION: Record<string, string> = {@@ -39,7 +61,16 @@ export const MIME_TO_REMOTE_FILE_EXTENSION: Record<string, string> = {'application/vnd.apache.arrow.file': 'arrow','application/vnd.apache.arrow.stream': 'arrow','application/vnd.apache.parquet': 'parquet',- 'application/x-parquet': 'parquet'+ 'application/x-parquet': 'parquet',+ 'application/geo+x-ndjson': 'geojsonl',+ 'application/geo+x-ldjson': 'geojsonl',+ 'application/geo+json-seq': 'geojsonl',+ 'application/x-ndjson': 'ndjson',+ 'application/jsonlines': 'ndjson',+ 'application/json-seq': 'ndjson',+ 'application/vnd.google-earth.kml+xml': 'kml',+ 'application/gpx+xml': 'gpx',+ 'application/vnd.garmin.tcx+xml': 'tcx'};export const DATASET_REFRESH_INTERVAL_OPTIONS: {value: number; labelId: string}[] = [diff --git a/src/duckdb/src/table/duckdb-table.ts b/src/duckdb/src/table/duckdb-table.tsindex 4d5c38a0c..f052173b1 100644--- a/src/duckdb/src/table/duckdb-table.ts+++ b/src/duckdb/src/table/duckdb-table.ts@@ -12,7 +12,7 @@ import {import {arrowSchemaToFields,isArrowData,- isGeoJson,+ getGeoJsonFromLoaderResult,isKeplerGlMap,isRowObject,processArrowBatches@@ -303,19 +303,20 @@ export class KeplerGlDuckDbTable extends KeplerTable {static getFileProcessor = function (data: any, inputFormat?: string) {let processor;let format;+ const geojsonData = getGeoJsonFromLoaderResult(data);if (inputFormat === DATASET_FORMATS.arrow || isArrowData(data)) {format = DATASET_FORMATS.arrow;processor = processArrowBatches;} else if (inputFormat === DATASET_FORMATS.keplergl || isKeplerGlMap(data)) {format = DATASET_FORMATS.keplergl;processor = processKeplerglJSONforDuckDb;+ } else if (inputFormat === DATASET_FORMATS.geojson || geojsonData) {+ format = DATASET_FORMATS.geojson;+ processor = (raw: unknown) => processGeojson(getGeoJsonFromLoaderResult(raw) ?? raw);} else if (inputFormat === DATASET_FORMATS.row || isRowObject(data)) {// csv file goes hereformat = DATASET_FORMATS.row;processor = processCsvRowObject; // directly import json object into duckdb-wasm- } else if (inputFormat === DATASET_FORMATS.geojson || isGeoJson(data)) {- format = DATASET_FORMATS.geojson;- processor = processGeojson;}return {processor, format};};diff --git a/src/localization/src/translations/ca.ts b/src/localization/src/translations/ca.tsindex 5bbd99004..88ff99b7e 100644--- a/src/localization/src/translations/ca.ts+++ b/src/localization/src/translations/ca.ts@@ -699,7 +699,7 @@ export default {fileUploader: {message: "Arrossega i deixa anar l'arxiu aquí",chromeMessage:- '*usuari de Chrome: la mida màxima són 250mb, si has de carrgar un arxiu més gran fes servir Safari',+ "*Els fitxers s'analitzen al navegador. Els fitxers grans poden esgotar la memòria; useu un tileset per a conjunts de dades molt grans.",disclaimer:'*kepler.gl és una aplicació a la banda client que no es recolza en cap servidor. Les dades només existeixen a la teva màquina/navegador. ' +"No s'envien dades ni mapes a cap servidor.",diff --git a/src/localization/src/translations/cn.ts b/src/localization/src/translations/cn.tsindex 100f1cd7f..920753e23 100644--- a/src/localization/src/translations/cn.ts+++ b/src/localization/src/translations/cn.ts@@ -679,8 +679,7 @@ export default {},fileUploader: {message: '将您的文件拖放到此处(可多个)',- chromeMessage:- '*对于 Chrome 用户:文件大小最大为 250mb。如果需要上传更多文件,请尝试使用 Safari。',+ chromeMessage: '*文件在浏览器中解析。过大的文件可能耗尽内存;超大数据集请使用瓦片(tileset)。',disclaimer:'* kepler.gl 在客户端上工作。数据仅保留在您自己的设备/浏览器中。' +'没有信息或地图数据被发送到任何服务器。',diff --git a/src/localization/src/translations/en.ts b/src/localization/src/translations/en.tsindex c1ba3d63b..5f46062bb 100644--- a/src/localization/src/translations/en.ts+++ b/src/localization/src/translations/en.ts@@ -808,7 +808,7 @@ ${'```'}fileUploader: {message: 'Drag & Drop Your File(s) Here',chromeMessage:- '*Chrome user: Limit file size to 250mb, if need to upload larger file, try Safari',+ '*Files are parsed in your browser. Large files can run out of memory; use a tileset for very large datasets.',disclaimer:'*kepler.gl is a client-side application with no server backend. Data lives only on your machine/browser. ' +'No information or map data is sent to any server.',diff --git a/src/localization/src/translations/es.ts b/src/localization/src/translations/es.tsindex 491385734..149346d99 100644--- a/src/localization/src/translations/es.ts+++ b/src/localization/src/translations/es.ts@@ -700,7 +700,7 @@ export default {fileUploader: {message: 'Arrastra y suelta el archivo aquí',chromeMessage:- '*usuario de Chrome: la medida máxima son 250mb, si debes cargar un archivo más grande utiliza Safari',+ '*Los archivos se analizan en el navegador. Los archivos grandes pueden agotar la memoria; use un tileset para conjuntos de datos muy grandes.',disclaimer:'*kepler.gl es una aplicación al lado cliente que no utiliza ningún servidor. Los datos sólo existen en tu máquina/navegador. ' +'No se envian datos ni mapas a ningún servidor.',diff --git a/src/localization/src/translations/fi.ts b/src/localization/src/translations/fi.tsindex fa56b72bb..592e84a48 100644--- a/src/localization/src/translations/fi.ts+++ b/src/localization/src/translations/fi.ts@@ -697,7 +697,7 @@ export default {fileUploader: {message: 'Raahaa ja pudota tiedostosi tänne',chromeMessage:- '*Chromen käyttäjä: Rajoita tiedostokokosi 250Mb:hen. Jos haluat suurempia tiedostoja, kokeile Safaria',+ '*Tiedostot käsitellään selaimessa. Suuret tiedostot voivat loppua muistista; käytä tilesetiä hyvin suurille aineistoille.',disclaimer:'*kepler.gl on client-side sovellus, data pysyy vain selaimessasi...' +'Tietoja ei lähetetä palvelimelle.',diff --git a/src/localization/src/translations/ja.ts b/src/localization/src/translations/ja.tsindex 225590d03..4d40edf5f 100644--- a/src/localization/src/translations/ja.ts+++ b/src/localization/src/translations/ja.ts@@ -696,7 +696,7 @@ export default {fileUploader: {message: 'ここにファイルをドロップ(複数可)',chromeMessage:- '*Chromeユーザーの場合: ファイルサイズは250mbまでにしてください。それ以上のファイルをアップロードする必要がある場合、Safariを試してください。',+ '*ファイルはブラウザ内で解析されます。大きなファイルはメモリ不足になることがあります。非常に大きなデータセットにはタイルセットを使用してください。',disclaimer:'*kepler.glはクライアント上で動作します。データは自身の機器・ブラウザにのみ保持されます。' +'情報や地図データは、いかなるサーバーにも送信されません。',diff --git a/src/localization/src/translations/pt.ts b/src/localization/src/translations/pt.tsindex 203bb3fce..ff6886a19 100644--- a/src/localization/src/translations/pt.ts+++ b/src/localization/src/translations/pt.ts@@ -701,7 +701,7 @@ export default {fileUploader: {message: 'Arraste e solte seu(s) arquivo(s) aqui',chromeMessage:- '*Usuários do chrome: O limite de tamanho de arquivo é 250mb, se você precisa fazer upload de arquivos maiores tente o Safari',+ '*Os arquivos são processados no navegador. Arquivos grandes podem esgotar a memória; use um tileset para conjuntos de dados muito grandes.',disclaimer:'*kepler.gl é uma aplicação client-side, sem um servidor backend. Os dados ficam apenas na sua máquina/browser. ' +'Nenhuma informação ou dados de mapa é enviado para qualquer server.',diff --git a/src/localization/src/translations/ru.ts b/src/localization/src/translations/ru.tsindex a08dd2b89..3ae3d7567 100644--- a/src/localization/src/translations/ru.ts+++ b/src/localization/src/translations/ru.ts@@ -701,7 +701,7 @@ export default {fileUploader: {message: 'Перетащите сюда ваши файлы',chromeMessage:- '*Пользователь Chrome: ограничьте размер файла до 250 МБ, если нужно загрузить файл большего размера, попробуйте Safari',+ '*Файлы обрабатываются в браузере. Большие файлы могут исчерпать память; для очень больших наборов данных используйте тайлы (tileset).',disclaimer:'*kepler.gl - это клиентское приложение без серверной части. Данные живут только на вашем компьютере. ' +'Никакая информация или данные карты не отправляются ни на один сервер.',diff --git a/src/processors/package.json b/src/processors/package.jsonindex 8269561b2..c75ab2da0 100644--- a/src/processors/package.json+++ b/src/processors/package.json@@ -40,6 +40,7 @@"@loaders.gl/csv": "^4.4.1","@loaders.gl/gis": "^4.4.1","@loaders.gl/json": "^4.4.1",+ "@loaders.gl/kml": "^4.4.1","@loaders.gl/loader-utils": "^4.4.1","@loaders.gl/parquet": "^4.4.1","@loaders.gl/schema": "^4.4.1",diff --git a/src/processors/src/file-handler.ts b/src/processors/src/file-handler.tsindex 0fd1756aa..cffaac54f 100644--- a/src/processors/src/file-handler.ts+++ b/src/processors/src/file-handler.ts@@ -12,9 +12,9 @@ import {} from '@kepler.gl/utils';import {generateHashId} from '@kepler.gl/common-utils';import {DATASET_FORMATS, DatasetType, REMOTE_FILE_EXTENSIONS} from '@kepler.gl/constants';-import {AddDataToMapPayload, Feature, LoadedMap, ProcessorResult} from '@kepler.gl/types';+import {AddDataToMapPayload, LoadedMap, ProcessorResult} from '@kepler.gl/types';import {KeplerTable} from '@kepler.gl/table';-import type {FeatureCollection} from 'geojson';+import type {Feature, FeatureCollection} from 'geojson';import {processArrowBatches,@@ -56,6 +56,10 @@ const JSON_LOADER_OPTIONS = {]};+const KML_LOADER_OPTIONS = {+ shape: 'geojson-table'+};+export type ProcessFileDataContent = {data: unknown;fileName: string;@@ -105,6 +109,52 @@ export function isFeatureCollection(json: unknown): json is FeatureCollection {return isPlainObject(json) && json.type === 'FeatureCollection' && Boolean(json.features);}+export function isGeoJsonFeatureArray(json: unknown): json is Feature[] {+ return Array.isArray(json) && json.length > 0 && isFeature(json[0]);+}++/**+ * Normalize loaders.gl GIS output (Feature arrays, geojson-table, object-row+ * tables of Features) into a GeoJSON Feature or FeatureCollection Kepler can process.+ */+export function getGeoJsonFromLoaderResult(data: unknown): Feature | FeatureCollection | null {+ if (isGeoJson(data)) {+ return data;+ }+ if (isGeoJsonFeatureArray(data)) {+ return {type: 'FeatureCollection', features: data};+ }+ if (!isPlainObject(data)) {+ return null;+ }+ if (data.shape === 'geojson-table' && Array.isArray(data.features)) {+ return {type: 'FeatureCollection', features: data.features as Feature[]};+ }+ if (+ (data.shape === 'object-row-table' || data.shape === 'row-table') &&+ isGeoJsonFeatureArray(data.data)+ ) {+ return {type: 'FeatureCollection', features: data.data};+ }+ return null;+}++function getBatchRows(batch: {+ shape?: string;+ type?: string;+ features?: unknown;+ data?: unknown;+}): unknown[] | null {+ if (+ Array.isArray(batch?.features) &&+ (batch.shape === 'geojson-table' || batch.type === 'FeatureCollection')+ ) {+ return batch.features;+ }+ const batchData = isArrowTable(batch.data) ? batch.data.batches : batch.data;+ return Array.isArray(batchData) ? batchData : null;+}+export function isRowObject(json: any): boolean {return Array.isArray(json) && isPlainObject(json[0]);}@@ -153,10 +203,18 @@ function getPersistedRemoteFormat(keplerFormat?: string, fileName?: string): strreturn keplerFormat;}const ext = fileName?.match(/\.([a-z0-9]+)$/i)?.[1]?.toLowerCase();- if (ext && Object.prototype.hasOwnProperty.call(REMOTE_FILE_EXTENSIONS, ext)) {+ if (!ext) {+ return undefined;+ }+ if (Object.prototype.hasOwnProperty.call(REMOTE_FILE_EXTENSIONS, ext)) {return ext;}- return undefined;+ const extensionAliases: Record<string, string> = {+ jsonl: 'ndjson',+ ndgeojson: 'geojsonl',+ ldgeojson: 'geojsonl'+ };+ return extensionAliases[ext];}export async function* readBatch(@@ -168,7 +226,7 @@ export async function* readBatch(etag?: string,lastModified?: string): AsyncGenerator {- let result = null;+ let result: any = null;const batches = <any>[];for await (const batch of asyncIterator) {// Last batch will have this special type and will provide all the root@@ -190,9 +248,14 @@ export async function* readBatch(result = batches;}} else {- const batchData = isArrowTable(batch.data) ? batch.data.batches : batch.data;- for (let i = 0; i < batchData?.length; i++) {- batches.push(batchData[i]);+ const batchRows = getBatchRows(batch);+ if (batchRows) {+ for (let i = 0; i < batchRows.length; i++) {+ batches.push(batchRows[i]);+ }+ }+ if (batch.shape === 'geojson-table') {+ result = {type: 'FeatureCollection', features: [...batches]};}}@@ -237,6 +300,9 @@ export async function readFileInBatches({arrow: ARROW_LOADER_OPTIONS,json: JSON_LOADER_OPTIONS,parquet: PARQUET_LOADER_OPTIONS,+ kml: KML_LOADER_OPTIONS,+ gpx: KML_LOADER_OPTIONS,+ tcx: KML_LOADER_OPTIONS,metadata: true,...(mimeType ? {mimeType} : {}),...loadOptions@@ -286,19 +352,20 @@ export async function processFileData({processor = processorResult.processor;} else {// use default processors+ const geojsonData = getGeoJsonFromLoaderResult(data);if (isArrowData(data)) {format = DATASET_FORMATS.arrow;processor = processArrowBatches;} else if (isKeplerGlMap(data)) {format = DATASET_FORMATS.keplergl;processor = processKeplerglJSON;+ } else if (geojsonData) {+ format = DATASET_FORMATS.geojson;+ processor = () => processGeojson(geojsonData);} else if (isRowObject(data)) {// csv file goes hereformat = DATASET_FORMATS.row;processor = processRowObject;- } else if (isGeoJson(data)) {- format = DATASET_FORMATS.geojson;- processor = processGeojson;}}if (format && processor) {diff --git a/src/processors/src/loader-registry.ts b/src/processors/src/loader-registry.tsindex 189d00b31..990bc81bd 100644--- a/src/processors/src/loader-registry.ts+++ b/src/processors/src/loader-registry.ts@@ -2,6 +2,8 @@// Copyright contributors to the kepler.gl projectimport type {Loader} from '@loaders.gl/loader-utils';+import {REMOTE_FILE_FORMATS, type RemoteFileFormat} from '@kepler.gl/constants';+import {getApplicationConfig} from '@kepler.gl/utils';import {KeplerCSVLoader} from './kepler-csv-loader';@@ -22,6 +24,16 @@ function getFileExtension(fileName = ''): string {return match ? match[1].toLowerCase() : '';}+const NDJSON_EXTENSIONS = ['ndjson', 'jsonl', 'ndgeojson', 'geojsonl', 'ldgeojson'];+const NDJSON_MIME_TYPES = [+ 'application/x-ndjson',+ 'application/jsonlines',+ 'application/json-seq',+ 'application/geo+x-ndjson',+ 'application/geo+x-ldjson',+ 'application/geo+json-seq'+];+const DEFAULT_LOADER_ENTRIES: KeplerLoaderEntry[] = [{id: 'csv',@@ -35,6 +47,37 @@ const DEFAULT_LOADER_ENTRIES: KeplerLoaderEntry[] = [mimeTypes: ['application/json', 'application/geo+json'],load: async () => (await import('@loaders.gl/json')).JSONLoader},+ {+ id: 'ndjson',+ extensions: NDJSON_EXTENSIONS,+ mimeTypes: NDJSON_MIME_TYPES,+ load: async () => {+ const {NDJSONLoader} = await import('@loaders.gl/json');+ return {+ ...NDJSONLoader,+ extensions: NDJSON_EXTENSIONS,+ mimeTypes: NDJSON_MIME_TYPES+ };+ }+ },+ {+ id: 'kml',+ extensions: ['kml'],+ mimeTypes: ['application/vnd.google-earth.kml+xml'],+ load: async () => (await import('@loaders.gl/kml')).KMLLoader+ },+ {+ id: 'gpx',+ extensions: ['gpx'],+ mimeTypes: ['application/gpx+xml'],+ load: async () => (await import('@loaders.gl/kml')).GPXLoader+ },+ {+ id: 'tcx',+ extensions: ['tcx'],+ mimeTypes: ['application/vnd.garmin.tcx+xml'],+ load: async () => (await import('@loaders.gl/kml')).TCXLoader+ },{id: 'arrow',extensions: ['arrow', 'feather'],@@ -51,6 +94,81 @@ const DEFAULT_LOADER_ENTRIES: KeplerLoaderEntry[] = [export const KEPLER_LOADER_ENTRIES = DEFAULT_LOADER_ENTRIES;+/** Maps user-facing format ids and extensions to a loader-registry id. */+const TOKEN_TO_LOADER_ID: Record<string, string> = {+ csv: 'csv',+ tsv: 'csv',+ dsv: 'csv',+ json: 'json',+ geojson: 'json',+ ndjson: 'ndjson',+ jsonl: 'ndjson',+ ndgeojson: 'ndjson',+ geojsonl: 'ndjson',+ ldgeojson: 'ndjson',+ kml: 'kml',+ gpx: 'gpx',+ tcx: 'tcx',+ arrow: 'arrow',+ feather: 'arrow',+ parquet: 'parquet'+};++const REMOTE_FORMAT_TO_LOADER_ID: Record<Exclude<RemoteFileFormat, 'auto'>, string> = {+ csv: 'csv',+ geojson: 'json',+ json: 'json',+ arrow: 'arrow',+ parquet: 'parquet',+ geojsonl: 'ndjson',+ ndjson: 'ndjson',+ kml: 'kml',+ gpx: 'gpx',+ tcx: 'tcx'+};++function getAcceptedLoaderIdSet(): Set<string> | null {+ const configured = getApplicationConfig().acceptedFileFormats;+ if (configured == null) {+ return null;+ }+ const ids = new Set<string>();+ for (const token of configured) {+ const id = TOKEN_TO_LOADER_ID[String(token).trim().toLowerCase()];+ if (id) {+ ids.add(id);+ }+ }+ return ids;+}++export function isKeplerFileFormatAccepted(token: string): boolean {+ const allowed = getAcceptedLoaderIdSet();+ if (!allowed) {+ return true;+ }+ const id = TOKEN_TO_LOADER_ID[token.trim().toLowerCase()];+ return Boolean(id && allowed.has(id));+}++export function getAcceptedKeplerLoaderEntries(): KeplerLoaderEntry[] {+ const allowed = getAcceptedLoaderIdSet();+ if (!allowed) {+ return KEPLER_LOADER_ENTRIES;+ }+ return KEPLER_LOADER_ENTRIES.filter(entry => allowed.has(entry.id));+}++export function getAcceptedRemoteFileFormats(): readonly RemoteFileFormat[] {+ const allowed = getAcceptedLoaderIdSet();+ if (!allowed) {+ return REMOTE_FILE_FORMATS;+ }+ return REMOTE_FILE_FORMATS.filter(+ format => format === 'auto' || allowed.has(REMOTE_FORMAT_TO_LOADER_ID[format])+ );+}+function matchesFile(entry: KeplerLoaderEntry, file: FileMetadata): boolean {const extension = getFileExtension(file.name);const mimeType = file.type?.split(';')[0].trim().toLowerCase();@@ -69,8 +187,9 @@ export async function getKeplerLoaders(file: FileMetadata,customLoaders: Loader[] = []): Promise<Loader[]> {- const matchingEntries = KEPLER_LOADER_ENTRIES.filter(entry => matchesFile(entry, file));- const entries = matchingEntries.length ? matchingEntries : KEPLER_LOADER_ENTRIES;+ const availableEntries = getAcceptedKeplerLoaderEntries();+ const matchingEntries = availableEntries.filter(entry => matchesFile(entry, file));+ const entries = matchingEntries.length ? matchingEntries : availableEntries;const defaultLoaders = await Promise.all(entries.map(entry => entry.load()));const customLoaderIds = new Set(customLoaders.map(loader => loader.id));diff --git a/src/reducers/src/vis-state-selectors.ts b/src/reducers/src/vis-state-selectors.tsindex 68b858523..1cbab21b6 100644--- a/src/reducers/src/vis-state-selectors.ts+++ b/src/reducers/src/vis-state-selectors.ts@@ -2,10 +2,51 @@// Copyright contributors to the kepler.gl projectimport {createSelector} from 'reselect';+import {isKeplerFileFormatAccepted} from '@kepler.gl/processors';+import {getApplicationConfig} from '@kepler.gl/utils';// NOTE: default formats must match file-handler-test.js-const DEFAULT_FILE_EXTENSIONS = ['csv', 'tsv', 'dsv', 'json', 'geojson', 'arrow', 'parquet'];-const DEFAULT_FILE_FORMATS = ['CSV', 'TSV', 'DSV', 'Json', 'GeoJSON', 'Arrow', 'Parquet'];+const DEFAULT_FILE_EXTENSIONS = [+ 'csv',+ 'tsv',+ 'dsv',+ 'json',+ 'geojson',+ 'arrow',+ 'parquet',+ 'ndjson',+ 'jsonl',+ 'ndgeojson',+ 'geojsonl',+ 'ldgeojson',+ 'kml',+ 'gpx',+ 'tcx'+];+// One chip per format family. Aliases (tsv, jsonl, ndgeojson, …) stay accepted+// via DEFAULT_FILE_EXTENSIONS but are not shown as separate icons/labels.+const DISPLAY_FILE_EXTENSIONS = [+ 'csv',+ 'json',+ 'geojson',+ 'arrow',+ 'parquet',+ 'geojsonl',+ 'kml',+ 'gpx',+ 'tcx'+];+const DISPLAY_FILE_FORMATS = [+ 'CSV',+ 'Json',+ 'GeoJSON',+ 'Arrow',+ 'Parquet',+ 'GeoJSONL',+ 'KML',+ 'GPX',+ 'TCX'+];interface LoaderInfo {name: string;@@ -14,10 +55,24 @@ interface LoaderInfo {export const getFileFormatNames = createSelector((state: {loaders: LoaderInfo[]}) => state.loaders,- loaders => [...DEFAULT_FILE_FORMATS, ...loaders.map(loader => loader.name)]+ () => getApplicationConfig().acceptedFileFormats,+ (loaders, _acceptedFileFormats) => [+ ...DISPLAY_FILE_FORMATS.filter(isKeplerFileFormatAccepted),+ ...loaders.map(loader => loader.name)+ ]);export const getFileExtensions = createSelector((state: {loaders: LoaderInfo[]}) => state.loaders,- loaders => [...DEFAULT_FILE_EXTENSIONS, ...loaders.flatMap(loader => loader.extensions)]+ () => getApplicationConfig().acceptedFileFormats,+ (loaders, _acceptedFileFormats) => [+ ...DEFAULT_FILE_EXTENSIONS.filter(isKeplerFileFormatAccepted),+ ...loaders.flatMap(loader => loader.extensions)+ ]);++export const getDisplayedFileExtensions = createSelector(getFileExtensions, extensions => {+ const shown = DISPLAY_FILE_EXTENSIONS.filter(ext => extensions.includes(ext));+ const custom = extensions.filter(ext => !DEFAULT_FILE_EXTENSIONS.includes(ext));+ return [...shown, ...custom];+});diff --git a/src/utils/src/application-config.ts b/src/utils/src/application-config.tsindex ed01dd2a8..58cf581b6 100644--- a/src/utils/src/application-config.ts+++ b/src/utils/src/application-config.ts@@ -76,6 +76,25 @@ export type BaseMapLibraryConfig = {*/export type ReduxLogLevel = 0 | 1 | 2;+/** Built-in file formats that can be listed in {@link KeplerApplicationConfig.acceptedFileFormats}. */+export type KeplerAcceptedFileFormat =+ | 'csv'+ | 'tsv'+ | 'dsv'+ | 'json'+ | 'geojson'+ | 'ndjson'+ | 'jsonl'+ | 'geojsonl'+ | 'ndgeojson'+ | 'ldgeojson'+ | 'kml'+ | 'gpx'+ | 'tcx'+ | 'arrow'+ | 'feather'+ | 'parquet';+export type KeplerApplicationConfig = {/** Default name of export HTML file, can be overridden by user */defaultHtmlName?: string;@@ -190,6 +209,18 @@ export type KeplerApplicationConfig = {*/enableRemoteFileFormatSelector?: boolean;+ /**+ * File formats Kepler may load from drag-and-drop, the file picker, and remote URLs.+ * Values are format ids or extensions (`csv`, `geojson`, `kml`, `geojsonl`, …).+ * When omitted or `null`, every built-in format is accepted.+ *+ * @example+ * ```+ * initApplicationConfig({acceptedFileFormats: ['csv', 'geojson', 'parquet']});+ * ```+ */+ acceptedFileFormats?: KeplerAcceptedFileFormat[] | null;+/*** Custom SVG icons to be made available in the icon layer.* These icons will be merged with the default icons fetched from CDN.@@ -338,6 +369,8 @@ const DEFAULT_APPLICATION_CONFIG: Required<KeplerApplicationConfig> = {enableRemoteFileFormatSelector: false,+ acceptedFileFormats: null,+customIcons: [],customIconUrl: '',diff --git a/src/utils/src/index.ts b/src/utils/src/index.tsindex f730968ee..ff6cc91b1 100644--- a/src/utils/src/index.ts+++ b/src/utils/src/index.ts@@ -201,6 +201,7 @@ export {DataRow} from './data-row';export {getApplicationConfig, initApplicationConfig} from './application-config';export type {KeplerApplicationConfig,+ KeplerAcceptedFileFormat,BaseMapLibraryConfig,MapLibInstance,GetMapRef,diff --git a/test/node/processors/file-handler-test.js b/test/node/processors/file-handler-test.jsindex 2360076ca..18d57a54f 100644--- a/test/node/processors/file-handler-test.js+++ b/test/node/processors/file-handler-test.js@@ -7,7 +7,8 @@ import {makeProgressIterator,filesToDataPayload,processFileData,- processArrowBatches+ processArrowBatches,+ getGeoJsonFromLoaderResult} from '@kepler.gl/processors';import {getDatasetRefreshIntervalMs} from '@kepler.gl/constants';import * as arrow from 'apache-arrow';@@ -369,6 +370,59 @@ test('#file-handler -> processArrowBatches skip compact for incremental loads',t.end();});+test('#file-handler -> processFileData Feature array is geojson', async t => {+ const features = [+ {+ type: 'Feature',+ properties: {name: 'alpha'},+ geometry: {type: 'Point', coordinates: [-122.4, 37.8]}+ },+ {+ type: 'Feature',+ properties: {name: 'beta'},+ geometry: {type: 'Point', coordinates: [-122.5, 37.9]}+ }+ ];+ const processed = await processFileData({+ content: {fileName: 'places.geojsonl', data: features},+ fileCache: []+ });++ t.equal(+ processed[0].info.format,+ 'geojson',+ 'an array of Features should not be treated as rows'+ );+ t.equal(processed[0].data.rows.length, 2, 'should keep both features');+ t.end();+});++test('#file-handler -> rows with type Feature but no geometry stay tables', async t => {+ const rows = [+ {type: 'Feature', name: 'alpha', lat: 37.8, lng: -122.4},+ {type: 'Feature', name: 'beta', lat: 37.9, lng: -122.5}+ ];++ t.equal(+ getGeoJsonFromLoaderResult(rows),+ null,+ 'should not wrap tabular rows as a FeatureCollection'+ );++ const processed = await processFileData({+ content: {fileName: 'places.csv', data: rows},+ fileCache: []+ });++ t.equal(+ processed[0].info.format,+ 'row',+ 'a type column equal to Feature should not force geojson'+ );+ t.equal(processed[0].data.rows.length, 2, 'should keep both rows');+ t.end();+});+test('#file-handler -> processFileData skipArrowCompact', async t => {const batchA = arrow.tableFromJSON([{lng: -122.4, lat: 37.8}]);const batchB = arrow.tableFromJSON([{lng: -122.5, lat: 37.9}]);diff --git a/test/node/processors/loader-registry-test.js b/test/node/processors/loader-registry-test.jsindex d5480104d..72c446fd1 100644--- a/test/node/processors/loader-registry-test.js+++ b/test/node/processors/loader-registry-test.js@@ -2,7 +2,21 @@// Copyright contributors to the kepler.gl projectimport test from 'tape';-import {getKeplerLoaders, KeplerCSVLoader, readFileInBatches} from '@kepler.gl/processors';+import {+ getAcceptedKeplerLoaderEntries,+ getAcceptedRemoteFileFormats,+ getKeplerLoaders,+ isKeplerFileFormatAccepted,+ KeplerCSVLoader,+ processFileData,+ readFileInBatches+} from '@kepler.gl/processors';+import {+ getDisplayedFileExtensions,+ getFileExtensions,+ getFileFormatNames+} from '@kepler.gl/reducers';+import {initApplicationConfig} from '@kepler.gl/utils';test('#loader-registry -> resolves matching loaders lazily', async t => {const loaders = await getKeplerLoaders({name: 'data.csv', type: ''});@@ -21,6 +35,25 @@ test('#loader-registry -> resolves JSON loaders from MIME metadata', async t =>t.end();});+test('#loader-registry -> resolves NDJSON and GIS loaders by extension', async t => {+ const ndjson = await getKeplerLoaders({name: 'rows.jsonl', type: ''});+ t.equal(ndjson[0].id, 'ndjson', 'jsonl should resolve to NDJSON');++ const geojsonl = await getKeplerLoaders({name: 'places.geojsonl', type: ''});+ t.equal(geojsonl[0].id, 'ndjson', 'geojsonl should resolve to NDJSON');+ t.ok(geojsonl[0].extensions.includes('geojsonl'), 'NDJSON adapter should accept geojsonl');++ const kml = await getKeplerLoaders({name: 'places.kml', type: ''});+ t.equal(kml[0].id, 'kml', 'kml should resolve to the KML loader');++ const gpx = await getKeplerLoaders({name: 'track.gpx', type: ''});+ t.equal(gpx[0].id, 'gpx', 'gpx should resolve to the GPX loader');++ const tcx = await getKeplerLoaders({name: 'activity.tcx', type: ''});+ t.equal(tcx[0].id, 'tcx', 'tcx should resolve to the TCX loader');+ t.end();+});+test('#loader-registry -> custom loaders take precedence', async t => {const customLoader = {...KeplerCSVLoader, name: 'Custom CSV'};const loaders = await getKeplerLoaders({name: 'data.csv', type: ''}, [customLoader]);@@ -38,6 +71,15 @@ test('#loader-registry -> CSV parser is async and dynamically loaded', async t =t.end();});+async function readLastBatch(file) {+ const generator = await readFileInBatches({file, fileCache: [], loaders: [], loadOptions: {}});+ let last;+ for await (const batch of generator) {+ last = batch;+ }+ return last;+}+test('#loader-registry -> file loading uses the async loader path', async t => {const file = new File(['name,value\nalpha,1\n'], 'data.csv', {type: 'text/csv'});const generator = await readFileInBatches({file, fileCache: [], loaders: [], loadOptions: {}});@@ -52,3 +94,172 @@ test('#loader-registry -> file loading uses the async loader path', async t => {t.deepEqual(dataBatch.data, [{name: 'alpha', value: '1'}], 'should preserve CSV rows');t.end();});++test('#loader-registry -> geojsonl files become GeoJSON datasets', async t => {+ const geojsonl = [+ '{"type":"Feature","properties":{"name":"alpha"},"geometry":{"type":"Point","coordinates":[-122.4,37.8]}}',+ '{"type":"Feature","properties":{"name":"beta"},"geometry":{"type":"Point","coordinates":[-122.5,37.9]}}'+ ].join('\n');+ const file = new File([geojsonl], 'places.geojsonl', {type: ''});+ const batch = await readLastBatch(file);+ const processed = await processFileData({content: batch, fileCache: []});++ t.equal(processed[0].info.format, 'geojson', 'geojsonl should process as geojson');+ t.equal(processed[0].data.rows.length, 2, 'should keep both features');+ t.end();+});++test('#loader-registry -> ndjson row files stay row datasets', async t => {+ const ndjson = '{"name":"alpha","value":1}\n{"name":"beta","value":2}\n';+ const file = new File([ndjson], 'rows.ndjson', {type: ''});+ const batch = await readLastBatch(file);+ const processed = await processFileData({content: batch, fileCache: []});++ t.equal(processed[0].info.format, 'row', 'generic NDJSON should process as rows');+ t.equal(processed[0].data.rows.length, 2, 'should keep both rows');+ t.end();+});++test('#loader-registry -> kml files become GeoJSON datasets', async t => {+ const kml = `<?xml version="1.0" encoding="UTF-8"?>+<kml xmlns="http://www.opengis.net/kml/2.2">+ <Placemark>+ <name>alpha</name>+ <Point><coordinates>-122.4,37.8</coordinates></Point>+ </Placemark>+</kml>`;+ const file = new File([kml], 'places.kml', {type: ''});+ const batch = await readLastBatch(file);+ const processed = await processFileData({content: batch, fileCache: []});++ t.equal(processed[0].info.format, 'geojson', 'KML should process as geojson');+ t.equal(processed[0].data.rows.length, 1, 'should parse the KML placemark');+ t.end();+});++test('#loader-registry -> gpx files become GeoJSON datasets', async t => {+ const gpx = `<?xml version="1.0" encoding="UTF-8"?>+<gpx version="1.1" creator="kepler-test">+ <wpt lat="37.8" lon="-122.4"><name>alpha</name></wpt>+</gpx>`;+ const file = new File([gpx], 'track.gpx', {type: ''});+ const batch = await readLastBatch(file);+ const processed = await processFileData({content: batch, fileCache: []});++ t.equal(processed[0].info.format, 'geojson', 'GPX should process as geojson');+ t.equal(processed[0].data.rows.length, 1, 'should parse the GPX waypoint');+ t.end();+});++test('#loader-registry -> tcx files become GeoJSON datasets', async t => {+ const tcx = `<?xml version="1.0" encoding="UTF-8"?>+<TrainingCenterDatabase xmlns="http://www.garmin.com/xmlschemas/TrainingCenterDatabase/v2">+ <Activities>+ <Activity Sport="Running">+ <Id>2020-01-01T00:00:00Z</Id>+ <Lap StartTime="2020-01-01T00:00:00Z">+ <Track>+ <Trackpoint>+ <Time>2020-01-01T00:00:00Z</Time>+ <Position>+ <LatitudeDegrees>37.8</LatitudeDegrees>+ <LongitudeDegrees>-122.4</LongitudeDegrees>+ </Position>+ </Trackpoint>+ <Trackpoint>+ <Time>2020-01-01T00:01:00Z</Time>+ <Position>+ <LatitudeDegrees>37.81</LatitudeDegrees>+ <LongitudeDegrees>-122.41</LongitudeDegrees>+ </Position>+ </Trackpoint>+ </Track>+ </Lap>+ </Activity>+ </Activities>+</TrainingCenterDatabase>`;+ const file = new File([tcx], 'activity.tcx', {type: ''});+ const batch = await readLastBatch(file);+ const processed = await processFileData({content: batch, fileCache: []});++ t.equal(processed[0].info.format, 'geojson', 'TCX should process as geojson');+ t.ok(processed[0].data.rows.length >= 1, 'should parse the TCX track');+ t.end();+});++test('#loader-registry -> acceptedFileFormats defaults to all formats', t => {+ t.ok(isKeplerFileFormatAccepted('kml'), 'kml is accepted by default');+ t.ok(isKeplerFileFormatAccepted('geojsonl'), 'geojsonl is accepted by default');+ t.ok(+ getAcceptedKeplerLoaderEntries().some(entry => entry.id === 'kml'),+ 'KML stays in the default loader list'+ );+ t.ok(+ getAcceptedRemoteFileFormats().includes('kml'),+ 'remote format list includes kml by default'+ );+ t.ok(getFileExtensions({loaders: []}).includes('kml'), 'file picker includes kml by default');+ t.ok(+ getFileExtensions({loaders: []}).includes('geojsonl'),+ 'aliases stay accepted even when not shown as icons'+ );+ t.deepEqual(+ getDisplayedFileExtensions({loaders: []}),+ ['csv', 'json', 'geojson', 'arrow', 'parquet', 'geojsonl', 'kml', 'gpx', 'tcx'],+ 'Add Data icons show one chip per format family'+ );+ t.deepEqual(+ getFileFormatNames({loaders: []}),+ ['CSV', 'Json', 'GeoJSON', 'Arrow', 'Parquet', 'GeoJSONL', 'KML', 'GPX', 'TCX'],+ 'upload copy lists format families, not every alias'+ );+ t.end();+});++test('#loader-registry -> acceptedFileFormats restricts loaders and UI lists', async t => {+ initApplicationConfig({acceptedFileFormats: ['csv', 'geojson']});+ try {+ t.ok(isKeplerFileFormatAccepted('csv'), 'csv remains accepted');+ t.ok(isKeplerFileFormatAccepted('tsv'), 'csv aliases remain accepted');+ t.ok(isKeplerFileFormatAccepted('GeoJSON'), 'geojson remains accepted');+ t.notOk(isKeplerFileFormatAccepted('kml'), 'kml is rejected');+ t.notOk(isKeplerFileFormatAccepted('parquet'), 'parquet is rejected');++ const ids = getAcceptedKeplerLoaderEntries().map(entry => entry.id);+ t.deepEqual(ids.sort(), ['csv', 'json'], 'only matching built-in loaders remain');++ const remote = getAcceptedRemoteFileFormats();+ t.deepEqual(+ remote,+ ['auto', 'csv', 'geojson', 'json'],+ 'remote dropdown keeps auto plus accepted formats'+ );++ t.deepEqual(+ getFileExtensions({loaders: []}),+ ['csv', 'tsv', 'dsv', 'json', 'geojson'],+ 'file picker extensions follow the allowlist'+ );+ t.deepEqual(+ getFileFormatNames({loaders: []}),+ ['CSV', 'Json', 'GeoJSON'],+ 'file picker labels follow the allowlist'+ );+ t.deepEqual(+ getDisplayedFileExtensions({loaders: []}),+ ['csv', 'json', 'geojson'],+ 'Add Data icons follow the allowlist without aliases'+ );++ const kmlLoaders = await getKeplerLoaders({name: 'places.kml', type: ''});+ t.ok(+ kmlLoaders.every(loader => loader.id !== 'kml'),+ 'KML files do not resolve the KML loader when it is not accepted'+ );+ const csvLoaders = await getKeplerLoaders({name: 'data.csv', type: ''});+ t.equal(csvLoaders[0], KeplerCSVLoader, 'CSV still resolves when accepted');+ } finally {+ initApplicationConfig({acceptedFileFormats: null});+ }+ t.end();+});diff --git a/test/node/processors/remote-file-test.js b/test/node/processors/remote-file-test.jsindex 39512a659..4ac41b70f 100644--- a/test/node/processors/remote-file-test.js+++ b/test/node/processors/remote-file-test.js@@ -32,6 +32,8 @@ test('#remote-file -> getMimeTypeForFormat', t => {t.equal(getMimeTypeForFormat('csv'), 'text/csv');t.equal(getMimeTypeForFormat('parquet'), 'application/vnd.apache.parquet');t.equal(getMimeTypeForFormat('arrow'), 'application/vnd.apache.arrow.file');+ t.equal(getMimeTypeForFormat('kml'), 'application/vnd.google-earth.kml+xml');+ t.equal(getMimeTypeForFormat('geojsonl'), 'application/geo+x-ndjson');t.end();});diff --git a/yarn.lock b/yarn.lockindex 305e1bd75..8ef8f4090 100644--- a/yarn.lock+++ b/yarn.lock@@ -4100,6 +4100,7 @@ __metadata:"@loaders.gl/csv": "npm:^4.4.1""@loaders.gl/gis": "npm:^4.4.1""@loaders.gl/json": "npm:^4.4.1"+ "@loaders.gl/kml": "npm:^4.4.1""@loaders.gl/loader-utils": "npm:^4.4.1""@loaders.gl/parquet": "npm:^4.4.1""@loaders.gl/schema": "npm:^4.4.1"@@ -5261,6 +5262,21 @@ __metadata:languageName: nodelinkType: hard+"@loaders.gl/kml@npm:4.4.1":+ version: 4.4.1+ resolution: "@loaders.gl/kml@npm:4.4.1"+ dependencies:+ "@loaders.gl/gis": "npm:4.4.1"+ "@loaders.gl/loader-utils": "npm:4.4.1"+ "@loaders.gl/schema": "npm:4.4.1"+ "@tmcw/togeojson": "npm:^4.5.0"+ "@xmldom/xmldom": "npm:^0.7.13"+ peerDependencies:+ "@loaders.gl/core": ~4.4.0+ checksum: 10c0/a854b49cedcfee31e5ff193c42b901ebd9499b4c1a4f0630a55093a9607bc46c78c5c7543be7e1dd0d9a626ab7ed82a3d0a423b9f0ef7db02409fdfc4aa522b5+ languageName: node+ linkType: hard+"@loaders.gl/loader-utils@npm:4.4.1":version: 4.4.1resolution: "@loaders.gl/loader-utils@npm:4.4.1"@@ -7267,6 +7283,13 @@ __metadata:languageName: nodelinkType: hard+"@tmcw/togeojson@npm:^4.5.0":+ version: 4.7.0+ resolution: "@tmcw/togeojson@npm:4.7.0"+ checksum: 10c0/6c9585c4a50e28a187602348923500474b599ed459cef44b728f9e3aa6781f0c1bb7f96879c3047fe037f58a97ca13d2d8f4ddebd97e374d40098a9611715447+ languageName: node+ linkType: hard+"@tootallnate/once@npm:2":version: 2.0.1resolution: "@tootallnate/once@npm:2.0.1"@@ -9443,6 +9466,13 @@ __metadata:languageName: nodelinkType: hard+"@xmldom/xmldom@npm:^0.7.13":+ version: 0.7.13+ resolution: "@xmldom/xmldom@npm:0.7.13"+ checksum: 10c0/cb02e4e8d986acf18578a5f25d1bce5e18d08718f40d8a0cdd922a4c112c8e00daf94de4e43f9556ed147c696b135f2ab81fa9a2a8a0416f60af15d156b60e40+ languageName: node+ linkType: hard+"@xtuc/ieee754@npm:^1.2.0":version: 1.2.0resolution: "@xtuc/ieee754@npm:1.2.0"