Skip to content

web search done right

Remove git lfs for the time being. The data folder was starting to get too large, which led to some difficulties with LFS timeouting (I think) when uploading the tar file. We will need a better strategy for this in the future, but for the time being it is disabled

Mikkel Denker0cbca3363cff
diff --git a/.gitattributes b/.gitattributes
deleted file mode 100644
index f6bd2897..00000000
--- a/.gitattributes
+++ /dev/null
@@ -1 +0,0 @@
-data.tar.gz filter=lfs diff=lfs merge=lfs -text
diff --git a/.gitignore b/.gitignore
index 1ae26420..4d2e4b53 100644
--- a/.gitignore
+++ b/.gitignore
@@ -7,3 +7,5 @@ package-lock.json
notes
*.log
/data/**/*
+*.tar.gz
+
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index aaa1dd11..10f6bcbf 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -31,8 +31,6 @@ The process is simple and fast. Upon your first pull request, you will be prompt
* Update ulimit. RocksDB tends to exceed the max number of allowed open files, so you will have to run `ulimit -n 10240` to increase the allowed max number of open files.
* (Optional) Install cargo-watch by running `cargo install cargo-watch`. This makes frontend development easier.
* (Optional) Install just by running `cargo install just`. This allows you to run the scripts in the Justfile. A Justfile is basically a simple Makefile, so this is just for convenience.
-* (Optional) Install git-lfs and download the data.
- * `git lfs pull` downloads a compressed sample index.
- * `just unpack-data` un-packs the data.
-After the non optional steps you can now run `cargo test` and should see all tests passing. If you have followed the optional steps, you should be able to run `just frontend` which will launch the frontend on `0.0.0.0:3000`.
\ No newline at end of file
+After the non optional steps you can now run `cargo test` and should see all tests passing. If you have followed the optional steps, you should be able to run `just local` which will index some warc files and launch the frontend on `0.0.0.0:3000`.
+After you have run the intial indexing, you can run `just frontend` to reuse the index.
diff --git a/data.tar.gz b/data.tar.gz
deleted file mode 100644
index 9ec479b1..00000000
--- a/data.tar.gz
+++ /dev/null
@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:d770c6f965cdc6dbc2bac3c97bb413b9c820b718eebfee04053b579fd451e8e4
-size 1132332917