web search done right
Remove git lfs for the time being. The data folder was starting to get too large, which led to some difficulties with LFS timeouting (I think) when uploading the tar file. We will need a better strategy for this in the future, but for the time being it is disabled
Mikkel Denker0cbca3363cff
diff --git a/.gitattributes b/.gitattributesdeleted file mode 100644index f6bd2897..00000000--- a/.gitattributes+++ /dev/null@@ -1 +0,0 @@-data.tar.gz filter=lfs diff=lfs merge=lfs -textdiff --git a/.gitignore b/.gitignoreindex 1ae26420..4d2e4b53 100644--- a/.gitignore+++ b/.gitignore@@ -7,3 +7,5 @@ package-lock.jsonnotes*.log/data/**/*+*.tar.gz+diff --git a/CONTRIBUTING.md b/CONTRIBUTING.mdindex aaa1dd11..10f6bcbf 100644--- a/CONTRIBUTING.md+++ b/CONTRIBUTING.md@@ -31,8 +31,6 @@ The process is simple and fast. Upon your first pull request, you will be prompt* Update ulimit. RocksDB tends to exceed the max number of allowed open files, so you will have to run `ulimit -n 10240` to increase the allowed max number of open files.* (Optional) Install cargo-watch by running `cargo install cargo-watch`. This makes frontend development easier.* (Optional) Install just by running `cargo install just`. This allows you to run the scripts in the Justfile. A Justfile is basically a simple Makefile, so this is just for convenience.-* (Optional) Install git-lfs and download the data.- * `git lfs pull` downloads a compressed sample index.- * `just unpack-data` un-packs the data.-After the non optional steps you can now run `cargo test` and should see all tests passing. If you have followed the optional steps, you should be able to run `just frontend` which will launch the frontend on `0.0.0.0:3000`.\ No newline at end of file+After the non optional steps you can now run `cargo test` and should see all tests passing. If you have followed the optional steps, you should be able to run `just local` which will index some warc files and launch the frontend on `0.0.0.0:3000`.+After you have run the intial indexing, you can run `just frontend` to reuse the index.diff --git a/data.tar.gz b/data.tar.gzdeleted file mode 100644index 9ec479b1..00000000--- a/data.tar.gz+++ /dev/null@@ -1,3 +0,0 @@-version https://git-lfs.github.com/spec/v1-oid sha256:d770c6f965cdc6dbc2bac3c97bb413b9c820b718eebfee04053b579fd451e8e4-size 1132332917