web search done right
we also need to set the libtorch environment variables to be able to run the tests. We can now run 'just cargo ...' which makes sure cargo has all the necesarry env variables set
Mikkel Denkerb08bf68cba22
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.mdindex adcc7437..9d92adfc 100644--- a/CONTRIBUTING.md+++ b/CONTRIBUTING.md@@ -27,8 +27,8 @@ The process is simple and fast. Upon your first pull request, you will be prompt* Install rust by following the steps outlined [here](https://www.rust-lang.org/tools/install)* Install clang and npm* Update ulimit. RocksDB tends to exceed the max number of allowed open files, so you will have to run `ulimit -n 10240` to increase the allowed max number of open files.-* Install just by running `cargo install just`. This allows you to run the scripts in the `justfile` file. A justfile is basically a simple Makefile, so this is just for convenience.+* Install just by running `cargo install just`. This allows you to run the scripts in the `justfile` file. A justfile is basically a simple Makefile. Since libtorch requires some specific environment variables to be set, you need to preface all `cargo run` and `cargo test` commands with `just` (so they becom `just cargo run` and `just cargo test`) .* Run the command `just configure` which should automatically configures the rest of your dev environment. The script creates a python virtual environment, installs relevant dependencies, traces and exports the ML models and creates a small local index which you can use for development.* (Optional) Install cargo-watch by running `cargo install cargo-watch`. This makes frontend development easier.-After the non optional steps you can now run `cargo test` and should see all tests passing. If you have followed the optional steps, you should be able to run `just frontend` to start the search server and launch the frontend at `0.0.0.0:3000`.+After the non optional steps you can now run `just cargo test` and should see all tests passing. If you have followed the optional steps, you should be able to run `just frontend` to start the search server and launch the frontend at `0.0.0.0:3000`.diff --git a/justfile b/justfileindex 38565a0c..50c31557 100644--- a/justfile+++ b/justfile@@ -1,10 +1,10 @@@profile-indexer:sudo rm -rf data/index- LIBTORCH="libtorch" LD_LIBRARY_PATH="libtorch/lib" DYLD_LIBRARY_PATH="libtorch/lib" cargo flamegraph --root -- indexer local configs/indexer/profile.toml+ just cargo flamegraph --root -- indexer local configs/indexer/profile.toml@webgraph:rm -rf data/webgraph- LIBTORCH="libtorch" LD_LIBRARY_PATH="libtorch/lib" DYLD_LIBRARY_PATH="libtorch/lib" cargo run --release -- webgraph local configs/webgraph/local.toml+ just cargo run --release -- webgraph local configs/webgraph/local.toml@frontend-rerun *ARGS:cd frontend; npm run build@@ -29,14 +29,14 @@@configure *ARGS:just setup {{ARGS}}- LIBTORCH="libtorch" LD_LIBRARY_PATH="libtorch/lib" DYLD_LIBRARY_PATH="libtorch/lib" cargo run --release --all-features -- configure {{ARGS}}+ just cargo run --release --all-features -- configure {{ARGS}}@centrality webgraph output:./scripts/build_harmonic {{webgraph}} {{output}}@entity:rm -rf data/entity- LIBTORCH="libtorch" LD_LIBRARY_PATH="libtorch/lib" DYLD_LIBRARY_PATH="libtorch/lib" cargo run --release -- indexer entity data/enwiki_subset.xml.bz2 data/entity+ just cargo run --release -- indexer entity data/enwiki_subset.xml.bz2 data/entity@setup_python_env:rm -rf .venv@@ -44,4 +44,7 @@.venv/bin/pip install -r scripts/requirements.txt@download_libtorch *ARGS:- .venv/bin/python3 scripts/download_libtorch.py {{ARGS}}\ No newline at end of file+ .venv/bin/python3 scripts/download_libtorch.py {{ARGS}}++@cargo *ARGS:+ LIBTORCH="libtorch" LD_LIBRARY_PATH="libtorch/lib" DYLD_LIBRARY_PATH="libtorch/lib" cargo {{ARGS}}\ No newline at end of file