Skip to content
Python
54.4 KB
Raw Blame

Architecture

  • llm_adapters.py - LLM provider integrations
  • grid_manager.py - Dynamic grid and image placement
  • collab_sketch_with_label.py - Core sketch application logic
  • utils.py - Utility functions
  • prompts.py - LLM prompts and templates

Frequently Used

Baseline:

--no-grid

removes grid from the image that is sent to model

--no-system-prompt

removes system prompt from being sent to model

Inference:

GPT-5 (medium) - Connect Dots

python collab_sketch_with_label.py --llm gpt --model gpt-5 --mixed-dir {dataset_directory} --max-tokens 20000 --reasoning-effort medium --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20

Ex:

python collab_sketch_with_label.py --llm gpt --model gpt-5 --mixed-dir connecting_dots_dataset/worksheets_source --max-tokens 20000 --reasoning-effort medium --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20

GPT-5 (medium) - Physics ball drop (exact same but change dataset)

python collab_sketch_with_label.py --llm gpt --model gpt-5 --mixed-dir {dataset_directory} --max-tokens 20000 --reasoning-effort medium --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20

Ex:

python collab_sketch_with_label.py --llm gpt --model gpt-5 --mixed-dir datasets/ball_path --max-tokens 20000 --reasoning-effort medium --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20

python collab_sketch_with_label.py --llm gpt --model gpt-5 --mixed-dir datasets/second_batch_ball_path --max-tokens 20000 --reasoning-effort low --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20

python collab_sketch_with_label.py --llm gpt --model gpt-5 --mixed-dir datasets/second_batch_ball_number --max-tokens 20000 --reasoning-effort low --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20

To get the ball_path dataset (so it's formatted in same way as other datasets) from the large_run_split, run gather_ball_prompts.py. This will create almost two equivalent folders, where one just adds a small bit to the prompt that requires the model to sketch the path.

Gemini-2.5-Pro - Connect Dots (just changed model)

python collab_sketch_with_label.py --llm gemini --model gemini-2.5-pro --mixed-dir {dataset_directory} --max-tokens 20000 --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20

Gemini-2.5-Pro - Physics ball drop

python collab_sketch_with_label.py --llm gemini --model gemini-2.5-flash --mixed-dir datasets/second_batch_ball_path --max-tokens 20000 --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20

python collab_sketch_with_label.py --llm gemini --model gemini-2.5-flash --mixed-dir datasets/maze_v2/sketch_invalid_flattened --max-tokens 20000 --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20

python collab_sketch_with_label.py --llm gemini --model gemini-2.5-flash --mixed-dir datasets/maze_v2/sketch_valid_flattened --max-tokens 20000 --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20

python collab_sketch_with_label.py --llm gemini --model gemini-2.5-flash --mixed-dir datasets/maze_v2/invalid_flattened --max-tokens 20000 --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20 --no-system-prompt --no-grid

python collab_sketch_with_label.py --llm gemini --model gemini-2.5-flash --mixed-dir datasets/maze_v2/valid_flattened --max-tokens 20000 --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20 --no-system-prompt --no-grid

python collab_sketch_with_label.py --llm gemini --model gemini-3-pro-preview --mixed-dir datasets/maze_v2/valid_flattened --max-tokens 20000 --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20 --no-system-prompt --no-grid

python collab_sketch_with_label.py --llm gemini --model gemini-3-pro-preview --mixed-dir datasets/maze_v2/valid_flattened --max-tokens 20000 --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20 --no-system-prompt --no-grid

python collab_sketch_with_label.py --llm gemini --model gemini-3-pro-preview --mixed-dir datasets/maze_v2/sketch_valid_flattened --max-tokens 20000 --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20

python collab_sketch_with_label.py --llm gemini --model gemini-3-pro-preview --mixed-dir datasets/maze_v2/sketch_invalid_flattened --max-tokens 20000 --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20 --only "164"

python collab_sketch_with_label.py --llm gemini --model gemini-3-pro-preview --mixed-dir datasets/maze_v2/invalid_flattened --max-tokens 20000 --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20 --no-system-prompt --no-grid --only "114,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199"

python collab_sketch_with_label.py --llm gpt --model gpt-5 --reasoning-effort low --mixed-dir datasets/maze_v2/sketch_invalid_flattened --max-tokens 20000 --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20

python collab_sketch_with_label.py --llm gpt --model gpt-5 --reasoning-effort low --mixed-dir datasets/maze_v2/sketch_valid_flattened --max-tokens 20000 --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20

python collab_sketch_with_label.py --llm gpt --model gpt-5 --reasoning-effort low --mixed-dir datasets/maze_v2/sketch_valid_flattened --max-tokens 20000 --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20

python collab_sketch_with_label.py --llm gpt --model gpt-5 --reasoning-effort low --mixed-dir datasets/ball_path --max-tokens 20000 --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20 --two-turn


Comparisons / Accuracy:

Sometimes will take a minute for html to finish generating. Htmls include accuracy

*Html compare + accuracy - Physics ball drop; VQA (no sketch) vs Sketch

python ball_drop_compare_report.py --gt-root large_run_split --raw-jsonl results/mix_eval/ball_number_gpt5_medium.jsonl --grid-dir results/mix_eval/gpt_ball_drop_curves --out gpt_ball_drop_report.html --thumb-width 480

*Html compare + accuracy - Physics ball drop; Sketch1 vs Sketch2

python ball_drop_compare_report.py --gt-root large_run_split --grid-dir results/mix_eval/gpt_ball_drop_curves --grid-dir-b results/mix_eval/gpt_ball_drop_lines --grid-a-name "Curves + lines" --grid-b-name "Straight lines only" --no-raw --out meta_ball_better.html --thumb-width 480

*Grid directory results are assumed to be in the file layout that would be saved after running inference

The raw jsonl needed for html compare (where raw result is when doing VQA i.e. no sketch) is in format:

{"index": 0, "file": "datasets\ball_number\run_001_1.png", "prompt": "Bucket Physics (boxed)", "model": "openai::gpt-5-medium", "raw_text": "\boxed{1}", "parsed_label": "1", "parsed_int": 1, "gold": null, "gold_int": null, "correct": null}

{"index": 1, "file": "datasets\ball_number\run_004.png", "prompt": "Bucket Physics (boxed)", "model": "openai::gpt-5-medium", "raw_text": "$\boxed{3}$", "parsed_label": "3", "parsed_int": 3, "gold": null, "gold_int": null, "correct": null}

... etc. new line for each sample

Where really just the "file" variable path (to match the result with correct gt) and the "parsed_label" is needed for each row. This is shoddy and should probably be made better.

Code Execution Examples

Model Types

Claude

python sketch_app.py --llm claude --model claude-3-5-sonnet-20240620

OpenAI GPT

python sketch_app.py --llm gpt --model o3

Gemini 2.5 Pro

python collab_sketch_with_label.py --llm gemini --model gemini-2.5-pro

New Grid Fix (for higher res images)

example:

python collab_sketch_with_label.py --llm gemini --model gemini-2.5-pro --mixed-dir datasets/ball_path --count-only-text --max-tokens 10000 --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20

Task Types

sketch_app.py is now collab_sketch_all.py

Counting

Claude example:

python collab_sketch_all.py --llm claude --model claude-3-5-sonnet-20240620 --eval-dataset vikhyatk/CountBenchQA --eval-split test --count-only-text

OpenAI example:

python collab_sketch_all.py --llm gpt --model o3 --eval-dataset vikhyatk/CountBenchQA --eval-split test --count-only-text

python collab_sketch_with_label.py --llm gemini --model gemini-2.5-pro --eval-dataset vikhyatk/CountBenchQA --eval-split test --count-only-text --api-delay 4

Labeling

Gemini 2.5 Pro, label everything in datasets/labeling, concept from filenames:

python collab_sketch_with_label.py --llm gemini --model gemini-2.5-pro --api-delay 0.0 --label-dir datasets/labeling

…or force a constant concept for every image + custom hints:

python collab_sketch_with_label.py --llm gemini --model gemini-2.5-pro --api-delay 0.0
--label-dir datasets/labeling
--concept-mode constant --concept "human body"
--labels-hint "head, torso, left_arm, right_arm, left_leg, right_leg, hand, foot"

---- Eval one stroke at a time ----

python collab_sketch_with_label.py --llm gemini --model gemini-2.5-pro --count-stepwise-dir datasets/biased --count-only-text --count-stepwise-max-turns 30 --api-delay 0.2
python collab_sketch_with_label.py --llm gemini --model gemini-2.5-pro --eval-stepwise --eval-dataset vikhyatk/CountBenchQA --count-only-text --max-examples 100 --api-delay 0.2

---- Mixed testing -----

#all strokes in one call python collab_sketch_with_label.py --llm gemini --model gemini-2.5-pro --mixed-dir datasets/mix --count-only-text

#one stroke per turn python collab_sketch_with_label.py --llm gemini --model gemini-2.5-pro --mixed-dir datasets/mix --mixed-stepwise --mixed-max-turns 40 --count-only-text --api-delay 0.2

--- TallyQA

python collab_sketch_with_label.py --llm gemini --model gemini-2.5-pro --tallyqa-json TallyQA_dataset/test_sample_500.json --vg-root data --tallyqa-outdir results/tallyqa_eval --max-examples 200 --count-only-text --api-delay 5

#multi-turn python collab_sketch_with_label.py --llm gemini --model gemini-2.5-pro --tallyqa-json TallyQA_dataset/test_sample_500.json --vg-root data --tallyqa-outdir results/tallyqa_eval --tallyqa-stepwise --tallyqa-max-turns 40 --max-examples 200 --count-only-text --api-delay 4

Open Router

Providers for Qwen2.5-VL 7B Instruct Hyperbolic is the only provider for this model

python collab_sketch_with_label.py --llm qwen3 --model qwen/qwen-2.5-vl-7b-instruct --mixed-dir datasets/vpct_ball_drop_vqa --max-tokens 20000 --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20 --no-system-prompt --no-grid

python collab_sketch_with_label.py --llm qwen3 --model qwen/qwen-2.5-vl-7b-instruct --mixed-dir datasets/vpct_ball_drop_sketch --max-tokens 20000 --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20

python collab_sketch_with_label.py --llm qwen3 --model qwen/qwen-2.5-vl-7b-instruct --mixed-dir datasets/maze_v2/sketch_invalid_flattened --max-tokens 20000 --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20

python collab_sketch_with_label.py --llm qwen3 --model qwen/qwen-2.5-vl-7b-instruct --mixed-dir datasets/maze_v2/sketch_valid_flattened --max-tokens 20000 --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20

python collab_sketch_with_label.py --llm qwen3 --model qwen/qwen-2.5-vl-7b-instruct --mixed-dir datasets/maze_v2/invalid_flattened --max-tokens 20000 --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20 --no-system-prompt --no-grid

python collab_sketch_with_label.py --llm qwen3 --model qwen/qwen-2.5-vl-7b-instruct --mixed-dir datasets/maze_v2/valid_flattened --max-tokens 20000 --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20 --no-system-prompt --no-grid

python collab_sketch_with_label.py --llm openrouter --model google/gemini-3-pro-image-preview --mixed-dir datasets/ball_path --max-tokens 20000 --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20 --no-system-prompt --no-grid --only "73,82,90"

python collab_sketch_with_label.py --llm openrouter --model google/gemini-3-pro-image-preview --mixed-dir datasets/second_batch_ball_path --max-tokens 20000 --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20 --no-system-prompt --no-grid --only "3,34,36,54,62,65,76,84,94"

python collab_sketch_with_label.py --llm openrouter --model google/gemini-3-pro-preview --mixed-dir datasets/ball_path --max-tokens 20000 --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20

python collab_sketch_with_label.py --llm openrouter --model google/gemini-3-pro-preview --mixed-dir datasets/second_batch_ball_path --max-tokens 20000 --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20

python collab_sketch_with_label.py --llm openrouter --model google/gemini-3-pro-preview --mixed-dir datasets/ball_number --max-tokens 20000 --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20 --no-system-prompt --no-grid

python collab_sketch_with_label.py --llm openrouter --model google/gemini-3-pro-preview --mixed-dir datasets/second_batch_ball_number --max-tokens 20000 --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20 --no-system-prompt --no-grid

python3 /Users/log/Github/sketchvlm/report_vpct_ball_drop_compare.py
--gt-root /Users/log/Github/sketchvlm/datasets/vpct-1
--raw-dir /Users/log/Github/sketchvlm/results/mix_eval/vpct/vpct_qwen25vl_7b_sketch
--grid-dir /Users/log/Github/sketchvlm/results/mix_eval/vpct_qwen25vl_7b_sketch
--out /Users/log/Github/sketchvlm/analysis/vpct/report_sketch_qwen25vl_7b.html
--raw-fallback-last-number

python consistency/process_with_openrouter.py \
--input consistency/image_questions_thinkmorph.json \
--output consistency/consistency_results_thinkmorph.json 


python consistency/process_with_openrouter.py \
--input consistency/image_questions_vilasr.json \
--output consistency/consistency_results_vilasr.json 

python consistency/process_with_openrouter.py \
--input consistency/source_data/image_questions_geminipro3.json \
--output consistency/judge_output/consistency_results_geminipro3.json 



python consistency/process_with_openrouter.py \
--input consistency/source_data/gemini3_image_two_turn_batch2.json \
--output consistency/judge_output/gemini3_image_two_turn_batch2.json 

python consistency/process_with_openrouter.py \
--input consistency/source_data/gemini3_image_two_turn_batch1.json \
--output consistency/judge_output/gemini3_image_two_turn_batch1.json 



python consistency/process_with_openrouter.py \
--input consistency/source_data/gemini3_ball_paths_batch2.json \
--output consistency/judge_output/gemini3_ball_paths_batch2.json 

python consistency/process_with_openrouter.py \
--input consistency/source_data/gemini3_ball_paths_batch1.json \
--output consistency/judge_output/gemini3_ball_paths_batch1.json 

python consistency/calculate_consistency.py --judge-dir consistency/judge_output 





python consistency/process_with_openrouter.py \
--input consistency/source_data/vpct_thinkmorph.json \
--output consistency/judge_output/vpct_thinkmorph.json 


python consistency/process_with_openrouter.py \
--input consistency/source_data/vpct_vilasr.json \
--output consistency/judge_output/vpct_vilasr.json 

python consistency/process_with_openrouter.py \
--input consistency/source_data/vpct_geminipro3.json \
--output consistency/judge_output/vpct_geminipro3.json 



python /Users/log/Github/sketchvlm/consistency/process_with_openrouter.py \
  --input /Users/log/Github/sketchvlm/consistency/normal_gemini3/ball_batch2_gemini3_image_pro_response.json \
  --output /Users/log/Github/sketchvlm/consistency/normal_gemini3/output_ball_batch2_gemini3_image_pro_response.json \
  --model google/gemini-3-pro-preview




python /Users/log/Github/sketchvlm/consistency/process_with_openrouter.py \
     --input /Users/log/Github/sketchvlm/consistency/source_data/ball_quality/ball_batch1_nano_banana.json \
     --output /Users/log/Github/sketchvlm/consistency/judge_output/ball_quality/ball_batch1_nano_banana.json \
     --model google/gemini-3-flash-preview


python consistency/process_with_openrouter.py \
    --input consistency/source_data/normal_gemini3/source_ball_batch2_gemini3_image_pro_response.json \
     --output /Users/log/Github/sketchvlm/consistency/batch2_g3_pro_nb_results.json \
    --model google/gemini-3-pro-preview
o



  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch2_gemini_25_flash.json \
      --output consistency/judge_output/ball_quality/batch2_gemini_25_flash.json \
      --model google/gemini-3-flash-preview

  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch2_gemini3_pro.json \
      --output consistency/judge_output/ball_quality/batch2_gemini3_pro.json \
      --model google/gemini-3-flash-preview


  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch2_gpt5_low.json \
      --output consistency/judge_output/ball_quality/batch2_gpt5_low.json \
      --model google/gemini-3-flash-preview


      

  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch1_gemini_25_flash.json \
      --output consistency/judge_output/ball_quality/batch2_gemini_25_flash.json \
      --model google/gemini-3-flash-preview

  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch1_gemini3_pro.json \
      --output consistency/judge_output/ball_quality/batch2_gemini3_pro.json \
      --model google/gemini-3-flash-preview


  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch1_gpt5_low.json \
      --output consistency/judge_output/ball_quality/batch2_gpt5_low.json \
      --model google/gemini-3-flash-preview









  # Gemini 2.5 Flash
  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch2_gemini_25_flash.json \
      --output consistency/judge_output/ball_quality/batch2_gemini_25_flash.json \
      --model google/gemini-3-flash-preview

  # Gemini 2.5 Pro
  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch2_gemini_25_pro.json \
      --output consistency/judge_output/ball_quality/batch2_gemini_25_pro.json \
      --model google/gemini-3-flash-preview

  # Gemini 3 Flash
  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch2_gemini3_flash.json \
      --output consistency/judge_output/ball_quality/batch2_gemini3_pro.json \
      --model google/gemini-3-flash-preview

  # GPT-5 Low
  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch2_gpt5_low.json \
      --output consistency/judge_output/ball_quality/batch2_gpt5_low.json \
      --model google/gemini-3-flash-preview

  # GPT-5 Med
  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch2_gpt5_med.json \
      --output consistency/judge_output/ball_quality/batch2_gpt5_med.json \
      --model google/gemini-3-flash-preview

  # Qwen2.5 7B
  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch2_qwen25_7b.json \
      --output consistency/judge_output/ball_quality/batch2_qwen25_7b.json \
      --model google/gemini-3-flash-preview

  # ThinkMorph
  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch2_thinkmorph.json \
      --output consistency/judge_output/ball_quality/batch2_thinkmorph.json \
      --model google/gemini-3-flash-preview

  # ViLaSR
  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch2_vilasr.json \
      --output consistency/judge_output/ball_quality/batch2_vilasr.json \
      --model google/gemini-3-flash-preview












 python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch1_gemini_25_flash.json \
      --output consistency/judge_output/ball_quality/batch1_gemini_25_flash.json \
      --model google/gemini-3-flash-preview

  # Gemini 2.5 Pro
  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch2_gemini_25_pro.json \
      --output consistency/judge_output/ball_quality/batch2_gemini_25_pro.json \
      --model google/gemini-3-flash-preview

  # Gemini 3 Flash
  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch1_gemini3_pro.json \
      --output consistency/judge_output/ball_quality/batch1_gemini3_pro.json \
      --model google/gemini-3-flash-preview

  # GPT-5 Low
  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch1_gpt5_low.json \
      --output consistency/judge_output/ball_quality/batch1_gpt5_low.json \
      --model google/gemini-3-flash-preview

  # GPT-5 Med
  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch1_gpt5_med.json \
      --output consistency/judge_output/ball_quality/batch1_gpt5_med.json \
      --model google/gemini-3-flash-preview

  # Qwen2.5 7B
  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch1_qwen25_7b.json \
      --output consistency/judge_output/ball_quality/batch1_qwen25_7b.json \
      --model google/gemini-3-flash-preview

  # Qwen3 235B Thinking
  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch1_qwen3_235b_thinking.json \
      --output consistency/judge_output/ball_quality/batch1_qwen3_235b_thinking.json \
      --model google/gemini-3-flash-preview

  # Qwen3 8B Thinking
  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch1_qwen3_8b_thinking.json \
      --output consistency/judge_output/ball_quality/batch1_qwen3_8b_thinking.json \
      --model google/gemini-3-flash-preview

  # ThinkMorph
  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch2_thinkmorph.json \
      --output consistency/judge_output/ball_quality/batch2_thinkmorph.json \
      --model google/gemini-3-flash-preview

  # ViLaSR
  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch2_vilasr.json \
      --output consistency/judge_output/ball_quality/batch2_vilasr.json \
      --model google/gemini-3-flash-preview


python /Users/log/Github/sketchvlm/consistency/process_with_openrouter.py \
     --input /Users/log/Github/sketchvlm/consistency/source_data/ball_quality/ball_batch1_nano_banana.json \
     --output /Users/log/Github/sketchvlm/consistency/judge_output/ball_quality/ball_batch1_nano_banana.json \
     --model google/gemini-3-flash-preview




# Maze Quality Evaluation Commands

# Gemini 2.5 Flash Invalid
python consistency/process_with_openrouter.py \
    --input consistency/source_data/grid_world_quality/source_maze_quality_gemini25_flash_invalid.json \
    --output consistency/judge_output/grid_world_quality/gemini25_flash_invalid.json \
    --model google/gemini-3-flash-preview

# Gemini 2.5 Flash Valid
python consistency/process_with_openrouter.py \
    --input consistency/source_data/grid_world_quality/source_maze_quality_gemini25_flash_valid.json \
    --output consistency/judge_output/grid_world_quality/gemini25_flash_valid.json \
    --model google/gemini-3-flash-preview

# Gemini 2.5 Pro Invalid
python consistency/process_with_openrouter.py \
    --input consistency/source_data/grid_world_quality/source_maze_quality_gemini25_pro_invalid.json \
    --output consistency/judge_output/grid_world_quality/gemini25_pro_invalid.json \
    --model google/gemini-3-flash-preview

# Gemini 2.5 Pro Valid
python consistency/process_with_openrouter.py \
    --input consistency/source_data/grid_world_quality/source_maze_quality_gemini25_pro_valid.json \
    --output consistency/judge_output/grid_world_quality/gemini25_pro_valid.json \
    --model google/gemini-3-flash-preview

# Gemini 3 Pro Invalid
python consistency/process_with_openrouter.py \
    --input consistency/source_data/grid_world_quality/source_maze_quality_gemini3_pro_invalid.json \
    --output consistency/judge_output/grid_world_quality/gemini3_pro_invalid.json \
    --model google/gemini-3-flash-preview

# Gemini 3 Pro Valid
python consistency/process_with_openrouter.py \
    --input consistency/source_data/grid_world_quality/source_maze_quality_gemini3_pro_valid.json \
    --output consistency/judge_output/grid_world_quality/gemini3_pro_valid.json \
    --model google/gemini-3-flash-preview

# GPT-5 Low Invalid
python consistency/process_with_openrouter.py \
    --input consistency/source_data/grid_world_quality/source_maze_quality_gpt5_low_invalid.json \
    --output consistency/judge_output/grid_world_quality/gpt5_low_invalid.json \
    --model google/gemini-3-flash-preview

# GPT-5 Low Valid
python consistency/process_with_openrouter.py \
    --input consistency/source_data/grid_world_quality/source_maze_quality_gpt5_low_valid.json \
    --output consistency/judge_output/grid_world_quality/gpt5_low_valid.json \
    --model google/gemini-3-flash-preview

# Qwen 2.5 7B Invalid
python consistency/process_with_openrouter.py \
    --input consistency/source_data/grid_world_quality/source_maze_quality_qwen25_7b_invalid.json \
    --output consistency/judge_output/grid_world_quality/qwen25_7b_invalid.json \
    --model google/gemini-3-flash-preview

# Qwen 2.5 7B Valid
python consistency/process_with_openrouter.py \
    --input consistency/source_data/grid_world_quality/source_maze_quality_qwen25_7b_valid.json \
    --output consistency/judge_output/grid_world_quality/qwen25_7b_valid.json \
    --model google/gemini-3-flash-preview

# ThinkMorph Invalid
python consistency/process_with_openrouter.py \
    --input consistency/source_data/grid_world_quality/source_maze_quality_thinkmorph_invalid.json \
    --output consistency/judge_output/grid_world_quality/thinkmorph_invalid.json \
    --model google/gemini-3-flash-preview

# ThinkMorph Valid
python consistency/process_with_openrouter.py \
    --input consistency/source_data/grid_world_quality/source_maze_quality_thinkmorph_valid.json \
    --output consistency/judge_output/grid_world_quality/thinkmorph_valid.json \
    --model google/gemini-3-flash-preview

# ViLaSR Invalid
python consistency/process_with_openrouter.py \
    --input consistency/source_data/grid_world_quality/source_maze_quality_vilasr_invalid.json \
    --output consistency/judge_output/grid_world_quality/vilasr_invalid.json \
    --model google/gemini-3-flash-preview

# ViLaSR Valid
python consistency/process_with_openrouter.py \
    --input consistency/source_data/grid_world_quality/source_maze_quality_vilasr_valid.json \
    --output consistency/judge_output/grid_world_quality/vilasr_valid.json \
    --model google/gemini-3-flash-preview



python collab_sketch_with_label.py --llm gpt --model gpt-5 --reasoning-effort low --mixed-dir datasets/maze_v2/sketch_valid_flattened --no-grid --res-x 1000 --res-y 1000 --prompt-origin top_left --only "1,8,30,44,52,53,60,62,71,74,75,79,82,106,111,114,119,125,127,130,135,138,147,151,154,165,170,174,179,184"

python collab_sketch_with_label.py --llm gemini --model gpt-5 --reasoning-effort low --mixed-dir datasets/maze_v2/sketch_invalid_flattened --no-grid --res-x 1000 --res-y 1000 --prompt-origin top_left


python collab_sketch_with_label.py --llm qwen3 --model qwen/qwen-2.5-vl-7b-instruct --mixed-dir datasets/maze_v2/invalid_flattened --max-tokens 20000 --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20 --no-system-prompt --no-grid

python collab_sketch_with_label.py --llm openrouter --model google/gemini-3-pro-preview --mixed-dir datasets/ball_path --max-tokens 20000 --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20

python collab_sketch_with_label.py --llm openrouter --model google/gemini-3-pro-image-preview --mixed-dir datasets/vpct_ball_drop_sketch --max-tokens 20000 --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20 --no-system-prompt --no-grid --only "8,37,39,42,47,48,52,65,73,80,86,88,97,98"

python collab_sketch_with_label.py --llm openrouter --model google/gemini-3-pro-image-preview --mixed-dir datasets/maze_v2/sketch_invalid_flattened_nanob --max-tokens 20000 --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20 --no-system-prompt --no-grid 

python collab_sketch_with_label.py --llm openrouter --model google/gemini-3-pro-image-preview --mixed-dir datasets/maze_v2/sketch_valid_flattened_nanob --max-tokens 20000 --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20 --no-system-prompt --no-grid 


python collab_sketch_with_label.py --llm openrouter --model google/gemini-2.5-pro --mixed-dir datasets/maze_v2/sketch_invalid_flattened --no-grid --res-x 1000 --res-y 1000 --prompt-origin top_left


  python /Users/log/Github/sketchvlm/consistency/process_with_openrouter.py \
      --input /Users/log/Github/sketchvlm/consistency/normal_gemini3/source_data/image_questions_vpct_nanobanana.json \
      --model google/gemini-3-pro-preview


  # For maze valid paths
  python /Users/log/Github/sketchvlm/consistency/process_with_openrouter.py \
      --input /Users/log/Github/sketchvlm/consistency/normal_gemini3/source_data/image_questions_maze_nanob_valid.json \
      --model google/gemini-3-pro-preview

  python /Users/log/Github/sketchvlm/consistency/process_with_openrouter.py \
      --input /Users/log/Github/sketchvlm/consistency/normal_gemini3/source_data/image_questions_maze_nanob_invalid.json \
      --model google/gemini-3-pro-preview






# VPCT

  python consistency/process_with_openrouter.py \
      --input consistency/source_data/vpct_quality/source_ball_quality_vpct_gemini_flash.json \
      --output consistency/judge_output/vpct_quality/vpct_gemini_flash.json \
      --model google/gemini-3-flash-preview

  python consistency/process_with_openrouter.py \
      --input consistency/source_data/vpct_quality/source_ball_quality_vpct_gemini_pro25.json \
      --output consistency/judge_output/vpct_quality/vpct_gemini_pro25.json \
      --model google/gemini-3-flash-preview

  python consistency/process_with_openrouter.py \
      --input consistency/source_data/vpct_quality/source_ball_quality_vpct_gpt5low.json \
      --output consistency/judge_output/vpct_quality/vpct_gpt5low.json \
      --model google/gemini-3-flash-preview


  python consistency/process_with_openrouter.py \
      --input consistency/source_data/vpct_quality/source_ball_quality_vpct_gpt5med.json \
      --output consistency/judge_output/vpct_quality/vpct_gpt5med.json \
      --model google/gemini-3-flash-preview

  python consistency/process_with_openrouter.py \
      --input consistency/source_data/vpct_quality/source_ball_quality_vpct_nanobanana.json \
      --output consistency/judge_output/vpct_quality/vpct_nanobanana.json \
      --model google/gemini-3-flash-preview

  python consistency/process_with_openrouter.py \
      --input consistency/source_data/vpct_quality/source_ball_quality_vpct_thinkmorph.json \
      --output consistency/judge_output/vpct_quality/vpct_thinkmorph.json \
      --model google/gemini-3-flash-preview

  python consistency/process_with_openrouter.py \
      --input consistency/source_data/vpct_quality/source_ball_quality_vpct_vilasr.json \
      --output consistency/judge_output/vpct_quality/vpct_vilasr.json \
      --model google/gemini-3-flash-preview






# ball path
⏺ # Batch 1 - Gemini 3 Pro
  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch1_gemini3_pro.json \
      --output consistency/judge_output/ball_quality/batch1_gemini3_pro.json \
      --model google/gemini-3-flash-preview

  # Batch 1 - Gemini 2.5 Flash
  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch1_gemini_25_flash.json \
      --output consistency/judge_output/ball_quality/batch1_gemini_25_flash.json \
      --model google/gemini-3-flash-preview

  # Batch 1 - Gemini 2.5 Pro
  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch1_gemini_25_pro.json \
      --output consistency/judge_output/ball_quality/batch1_gemini_25_pro.json \
      --model google/gemini-3-flash-preview

  # Batch 1 - GPT-5 Low
  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch1_gpt5_low.json \
      --output consistency/judge_output/ball_quality/batch1_gpt5_low.json \
      --model google/gemini-3-flash-preview

  # Batch 1 - GPT-5 Med
  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch1_gpt5_med.json \
      --output consistency/judge_output/ball_quality/batch1_gpt5_med.json \
      --model google/gemini-3-flash-preview

  # Batch 1 - Nano Banana
  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch1_nano_banana.json \
      --output consistency/judge_output/ball_quality/batch1_nano_banana.json \
      --model google/gemini-3-flash-preview

  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch2_nano_banana.json \
      --output consistency/judge_output/ball_quality/batch2_nano_banana.json \
      --model google/gemini-3-flash-preview

  # Batch 1 - Qwen 2.5 7B
  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch1_qwen25_7b.json \
      --output consistency/judge_output/ball_quality/batch1_qwen25_7b.json \
      --model google/gemini-3-flash-preview

  # Batch 1 - Qwen 3 235B Thinking
  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch1_qwen3_235b_thinking.json \
      --output consistency/judge_output/ball_quality/batch1_qwen3_235b_thinking.json \
      --model google/gemini-3-flash-preview

  # Batch 1 - Qwen 3 8B Thinking
  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch1_qwen3_8b_thinking.json \
      --output consistency/judge_output/ball_quality/batch1_qwen3_8b_thinking.json \
      --model google/gemini-3-flash-preview

  # Batch 1 - ThinkMorph
  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch1_thinkmorph.json \
      --output consistency/judge_output/ball_quality/batch1_thinkmorph.json \
      --model google/gemini-3-flash-preview

  # Batch 1 - ViLaSR
  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch1_vilasr.json \
      --output consistency/judge_output/ball_quality/batch1_vilasr.json \
      --model google/gemini-3-flash-preview

  # Batch 2 - Gemini 3 Pro
  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch2_gemini3_pro.json \
      --output consistency/judge_output/ball_quality/batch2_gemini3_pro.json \
      --model google/gemini-3-flash-preview

  # Batch 2 - Gemini 2.5 Flash
  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch2_gemini_25_flash.json \
      --output consistency/judge_output/ball_quality/batch2_gemini_25_flash.json \
      --model google/gemini-3-flash-preview

  # Batch 2 - Gemini 2.5 Pro
  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch2_gemini_25_pro.json \
      --output consistency/judge_output/ball_quality/batch2_gemini_25_pro.json \
      --model google/gemini-3-flash-preview

  # Batch 2 - GPT-5 Low
  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch2_gpt5_low.json \
      --output consistency/judge_output/ball_quality/batch2_gpt5_low.json \
      --model google/gemini-3-flash-preview

  # Batch 2 - GPT-5 Med
  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch2_gpt5_med.json \
      --output consistency/judge_output/ball_quality/batch2_gpt5_med.json \
      --model google/gemini-3-flash-preview

  # Batch 2 - Qwen 2.5 7B
  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch2_qwen25_7b.json \
      --output consistency/judge_output/ball_quality/batch2_qwen25_7b.json \
      --model google/gemini-3-flash-preview

  # Batch 2 - ThinkMorph
  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch2_thinkmorph.json \
      --output consistency/judge_output/ball_quality/batch2_thinkmorph.json \
      --model google/gemini-3-flash-preview

  # Batch 2 - ViLaSR
  python consistency/process_with_openrouter.py \
      --input consistency/source_data/ball_quality/source_ball_quality_batch2_vilasr.json \
      --output consistency/judge_output/ball_quality/batch2_vilasr.json \
      --model google/gemini-3-flash-preview




  # Gemini 3 Pro - Valid Paths (0-1000)
  python consistency/process_with_openrouter.py \
      --input consistency/source_data/grid_world_quality/source_maze_quality_gemini3pro_validpaths_0_1000.json \
      --output consistency/judge_output/grid_world_quality/gemini3pro_validpaths_0_1000.json \
      --model google/gemini-3-flash-preview

  # Gemini 3 Pro - Invalid Paths (0-1000)
  python consistency/process_with_openrouter.py \
      --input consistency/source_data/grid_world_quality/source_maze_quality_gemini3pro_invalidpaths_0_1000.json \
      --output consistency/judge_output/grid_world_quality/gemini3pro_invalidpaths_0_1000.json \
      --model google/gemini-3-flash-preview






  python consistency/process_with_openrouter.py \
    --input consistency/normal_gemini3/source_data/image_questions_maze_nanob_invalid.json \
    --output consistency/normal_gemini3/judge_output/maze_nanob_gemini3pro_invalid.json \
    --model google/gemini-3-pro-preview

  python consistency/process_with_openrouter.py \
    --input consistency/normal_gemini3/source_data/image_questions_maze_nanob_valid.json \
    --output consistency/normal_gemini3/judge_output/maze_nanob_gemini3pro_valid.json \
    --model google/gemini-3-pro-preview


python collab_sketch_with_label.py --llm openrouter --model google/gemini-3-pro-image-preview --mixed-dir datasets/maze_v2/sketch_valid_flattened_nanob --max-tokens 20000 --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20 --no-system-prompt --no-grid --only "57,71,80"


python collab_sketch_with_label.py --llm openrouter --model google/gemini-3-pro-image-preview --mixed-dir datasets/maze_v2/sketch_invalid_flattened_nanob --max-tokens 20000 --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20 --no-system-prompt --no-grid --only "37,41,57,60,91,100,106"





                                                                                                                                                                              
python consistency/process_with_openrouter.py --input /Users/log/Github/sketchvlm/consistency/source_data/consistency/grid_world/gpt5_low_invalid.json --output /Users/log/Github/sketchvlm/consistency/judge_output/grid_world_consistency/consistency_results_gpt5_low_invalid.json --model google/gemini-3-flash-preview       

python consistency/process_with_openrouter.py --input /Users/log/Github/sketchvlm/consistency/source_data/consistency/grid_world/gpt5_low_invalid.json --output /Users/log/Github/sketchvlm/consistency/judge_output/grid_world_consistency/consistency_results_gpt5_low_invalid.json --model google/gemini-3-flash-preview       


  python consistency/process_with_openrouter.py --input /Users/log/Github/sketchvlm/consistency/source_data/consistency/grid_world/gemini3pro_invalid.json --output /Users/log/Github/sketchvlm/consistency/judge_output/grid_world_consistency/consistency_results_gemini3pro_invalid.json --model google/gemini-3-flash-preview           

python consistency/process_with_openrouter.py --input /Users/log/Github/sketchvlm/consistency/source_data/consistency/grid_world/gemini3pro_valid.json --output /Users/log/Github/sketchvlm/consistency/judge_output/grid_world_consistency/consistency_results_gemini3pro_valid.json --model google/gemini-3-flash-preview                         


  python consistency/process_with_openrouter.py --input /Users/log/Github/sketchvlm/consistency/source_data/consistency/grid_world/gemini25_flash_invalid.json --output /Users/log/Github/sketchvlm/consistency/judge_output/grid_world_consistency/consistency_results_gemini25_flash_invalid.json --model google/gemini-3-flash-preview  

  python consistency/process_with_openrouter.py --input /Users/log/Github/sketchvlm/consistency/source_data/consistency/grid_world/gemini25_flash_valid.json --output /Users/log/Github/sketchvlm/consistency/judge_output/grid_world_consistency/consistency_results_gemini25_flash_valid.json --model google/gemini-3-flash-preview     


python consistency/process_with_openrouter.py --input /Users/log/Github/sketchvlm/consistency/source_data/consistency/grid_world/gemini25_pro_invalid.json --output /Users/log/Github/sketchvlm/consistency/judge_output/grid_world_consistency/consistency_results_gemini25_pro_invalid.json --model google/gemini-3-flash-preview    

python consistency/process_with_openrouter.py --input /Users/log/Github/sketchvlm/consistency/source_data/consistency/grid_world/gemini25_pro_valid.json --output /Users/log/Github/sketchvlm/consistency/judge_output/grid_world_consistency/consistency_results_gemini25_pro_valid.json --model google/gemini-3-flash-preview   

  python consistency/process_with_openrouter.py --input /Users/log/Github/sketchvlm/consistency/source_data/consistency/grid_world/thinkmorph_invalid.json --output /Users/log/Github/sketchvlm/consistency/judge_output/grid_world_consistency/consistency_results_thinkmorph_invalid.json --model google/gemini-3-flash-preview       
  
  python consistency/process_with_openrouter.py --input /Users/log/Github/sketchvlm/consistency/source_data/consistency/grid_world/thinkmorph_valid.json --output /Users/log/Github/sketchvlm/consistency/judge_output/grid_world_consistency/consistency_results_thinkmorph_valid.json --model google/gemini-3-flash-preview       


python consistency/process_with_openrouter.py --input /Users/log/Github/sketchvlm/consistency/source_data/consistency/grid_world/vilasr_invalid.json --output /Users/log/Github/sketchvlm/consistency/judge_output/grid_world_consistency/consistency_results_vilasr_invalid.json --model google/gemini-3-flash-preview        

python consistency/process_with_openrouter.py --input /Users/log/Github/sketchvlm/consistency/source_data/consistency/grid_world/vilasr_valid.json --output /Users/log/Github/sketchvlm/consistency/judge_output/grid_world_consistency/consistency_results_vilasr_valid.json --model google/gemini-3-flash-preview        


python consistency/process_with_openrouter.py --input /Users/log/Github/sketchvlm/consistency/source_data/consistency/grid_world/nanob_gemini3_pro_invalid.json --output /Users/log/Github/sketchvlm/consistency/judge_output/grid_world_consistency/consistency_results_nanob_gemini3_pro_invalid.json --model google/gemini-3-flash-preview              
        
python consistency/process_with_openrouter.py --input /Users/log/Github/sketchvlm/consistency/source_data/consistency/grid_world/nanob_gemini3_pro_valid.json --output /Users/log/Github/sketchvlm/consistency/judge_output/grid_world_consistency/consistency_results_nanob_gemini3_pro_valid.json --model google/gemini-3-flash-preview                



python consistency/process_with_openrouter.py --input /Users/log/Github/sketchvlm/consistency/source_data/consistency/grid_world/nanob_gemini3_pro_invalid_retry.json --output /Users/log/Github/sketchvlm/consistency/judge_output/grid_world_consistency/consistency_results_nanob_gemini3_pro_invalid_retry.json --model google/gemini-3-flash-preview              
        
python consistency/process_with_openrouter.py --input /Users/log/Github/sketchvlm/consistency/source_data/consistency/grid_world/nanob_gemini3_pro_valid_retry.json --output /Users/log/Github/sketchvlm/consistency/judge_output/grid_world_consistency/consistency_results_nanob_gemini3_pro_valid_retry.json --model google/gemini-3-flash-preview                


python consistency/process_with_openrouter.py --input /Users/log/Github/sketchvlm/consistency/source_data/consistency/grid_world/nanob_gemini3_pro_invalid_missing.json --output /Users/log/Github/sketchvlm/consistency/judge_output/grid_world_consistency/nanob_gemini3_pro_invalid_missing_results.json --model google/gemini-3-pro-preview      


python consistency/process_with_openrouter.py --input /Users/log/Github/sketchvlm/consistency/source_data/consistency/grid_world/nanob_gemini3_pro_valid_missing.json --output /Users/log/Github/sketchvlm/consistency/judge_output/grid_world_consistency/nanob_gemini3_pro_valid_missing_results.json --model google/gemini-3-pro-preview            



python consistency/process_with_openrouter.py --input /Users/log/Github/sketchvlm/consistency/source_data/consistency/grid_world/nanob_gemini3_pro_invalid_newly_available.json --output /Users/log/Github/sketchvlm/consistency/judge_output/grid_world_consistency/nanob_gemini3_pro_invalid_newly_available_results.json --model google/gemini-3-pro-preview     
                             
python consistency/process_with_openrouter.py --input /Users/log/Github/sketchvlm/consistency/source_data/consistency/grid_world/nanob_gemini3_pro_valid_newly_available.json --output /Users/log/Github/sketchvlm/consistency/judge_output/grid_world_consistency/nanob_gemini3_pro_valid_newly_available_results.json --model google/gemini-3-pro-preview       
       

python consistency/process_with_openrouter.py --input /Users/log/Github/sketchvlm/consistency/source_data/consistency/grid_world/nanob_gemini3_pro_invalid.json --output /Users/log/Github/sketchvlm/consistency/judge_output/grid_world_consistency/consistency_results_nanob_gemini3_pro_invalid.json --model google/gemini-3-flash-preview                
               
python consistency/process_with_openrouter.py --input /Users/log/Github/sketchvlm/consistency/source_data/consistency/grid_world/nanob_gemini3_pro_valid.json --output /Users/log/Github/sketchvlm/consistency/judge_output/grid_world_consistency/consistency_results_nanob_gemini3_pro_valid.json --model google/gemini-3-flash-preview                  






python collab_sketch_with_label.py --llm gpt --model gpt-5 --mixed-dir datasets/maze_v2/sketch_valid_flattened --max-tokens 20000 --reasoning-effort medium --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20 

python collab_sketch_with_label.py --llm gpt --model gpt-5 --mixed-dir datasets/maze_v2/sketch_invalid_flattened --max-tokens 20000 --reasoning-effort medium --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20 

python collab_sketch_with_label.py --llm gpt --model gpt-5 --mixed-dir datasets/maze_v2/sketch_invalid_flattened --max-tokens 20000 --reasoning-effort medium --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20 


export DYLD_FALLBACK_LIBRARY_PATH=/opt/homebrew/opt/cairolib:$DYLD_FALLBACK_LIBRARY_PATH && python collab_sketch_with_label.py --llm gpt --model gpt-5 --mixed-dir datasets/maze_v2/sketch_valid_flattened --max-tokens 20000 --reasoning-effort medium --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20



python consistency/process_with_openrouter.py --input /Users/log/Github/sketchvlm/consistency/source_data/consistency/thinkmorph_ball_paths_batch1.json --output /Users/log/Github/sketchvlm/consistency/judge_output/gemini3_flash_judge/consistency_results_thinkmorph_ball_paths_batch1.json --model google/gemini-3-flash-preview    

python consistency/process_with_openrouter.py --input /Users/log/Github/sketchvlm/consistency/source_data/consistency/thinkmorph_ball_paths_batch2.json --output /Users/log/Github/sketchvlm/consistency/judge_output/gemini3_flash_judge/consistency_results_thinkmorph_ball_paths_batch2.json --model google/gemini-3-flash-preview    



python consistency/process_with_openrouter.py --input /Users/log/Github/sketchvlm/consistency/source_data/consistency/vilasr_ball_paths_batch1.json --output /Users/log/Github/sketchvlm/consistency/judge_output/gemini3_flash_judge/consistency_results_vilasr_ball_paths_batch1.json --model google/gemini-3-flash-preview

python consistency/process_with_openrouter.py --input /Users/log/Github/sketchvlm/consistency/source_data/consistency/vilasr_ball_paths_batch1.json --output /Users/log/Github/sketchvlm/consistency/judge_output/gemini3_flash_judge/consistency_results_vilasr_ball_paths_batch1.json --model google/gemini-3-flash-preview
     

python consistency/process_with_openrouter.py --input /Users/log/Github/sketchvlm/consistency/source_data/consistency/vilasr_ball_paths_batch1.json --output /Users/log/Github/sketchvlm/consistency/judge_output/gemini3_flash_judge/consistency_results_vilasr_ball_paths_batch1.json --model google/gemini-3-flash-preview    

python consistency/process_with_openrouter.py --input /Users/log/Github/sketchvlm/consistency/source_data/consistency/vilasr_ball_paths_batch2.json --output /Users/log/Github/sketchvlm/consistency/judge_output/gemini3_flash_judge/consistency_results_vilasr_ball_paths_batch2.json --model google/gemini-3-flash-preview    



python /Users/log/Github/sketchvlm/consistency/process_with_openrouter.py --input /Users/log/Github/sketchvlm/consistency/source_data/grid_world_quality/nano_banana_invalid.json --output /Users/log/Github/sketchvlm/consistency/judge_output/grid_world_quality/consistency_results_nano_banana_invalid.json --model google/gemini-3-flash-preview   

python /Users/log/Github/sketchvlm/consistency/process_with_openrouter.py --input /Users/log/Github/sketchvlm/consistency/source_data/grid_world_quality/nano_banana_valid.json --output /Users/log/Github/sketchvlm/consistency/judge_output/grid_world_quality/consistency_results_nano_banana_valid.json --model google/gemini-3-flash-preview   



python collab_sketch_with_label.py --llm gpt --model gpt-5 --mixed-dir datasets/maze_v2/valid_flattened --max-tokens 20000 --reasoning-effort medium --no-system-prompt --no-grid 

python collab_sketch_with_label.py --llm gpt --model gpt-5 --mixed-dir datasets/maze_v2/invalid_flattened --max-tokens 20000 --reasoning-effort medium --no-system-prompt --no-grid 


python collab_sketch_with_label.py --llm openrouter --model openai/gpt-5 --mixed-dir datasets/maze_v2/sketch_invalid_flattened --max-tokens 20000 --reasoning-effort medium --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20 --only "119,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199"


python collab_sketch_with_label.py --llm openrouter --model openai/gpt-5 --mixed-dir datasets/maze_v2/sketch_valid_flattened --max-tokens 20000 --reasoning-effort medium --adaptive-grid --target-cols 50 --target-rows 50 --min-cell-px 20 --only "48,123,125,126,127,128,129,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199"

python collab_sketch_with_label.py --llm openrouter --model openai/gpt-5 --mixed-dir datasets/maze_v2/valid_flattened --max-tokens 20000 --reasoning-effort medium 

python collab_sketch_with_label.py --llm openrouter --model openai/gpt-5 --mixed-dir datasets/maze_v2/invalid_flattened --max-tokens 20000 --reasoning-effort medium 





python consistency/process_with_openrouter.py --input consistency/normal_gemini3/source_data/image_questions_maze_nanob_valid.json --output consistency/normal_gemini3/judge_output/maze_nanob_gemini3pro_valid.json --model google/gemini-3-pro-preview



python consistency/process_with_openrouter.py --input consistency/normal_gemini3/source_data/image_questions_maze_nanob_invalid.json --output consistency/normal_gemini3/judge_output/maze_nanob_gemini3pro_invalid.json --model google/gemini-3-pro-preview


python consistency/process_with_openrouter.py --input consistency/normal_gemini3/source_data/image_questions_maze_nanob_invalid_retry.json --output consistency/normal_gemini3/judge_output/maze_nanob_gemini3pro_invalid_retry.json --model google/gemini-3-pro-preview

python consistency/process_with_openrouter.py --input consistency/normal_gemini3/source_data/image_questions_maze_nanob_valid_retry.json --output consistency/normal_gemini3/judge_output/maze_nanob_gemini3pro_valid_retry.json --model google/gemini-3-pro-preview


python3 consistency/process_with_openrouter.py --input consistency/source_data/consistency/grid_world/consistency_check_nanob_gemini3_invalid.json --output consistency/judge_output/grid_world_consistency/consistency_results_nanob_gemini3_invalid.json --model google/gemini-3-flash-preview   

python3 consistency/process_with_openrouter.py --input consistency/source_data/consistency/grid_world/consistency_check_nanob_gemini3_valid.json --output consistency/judge_output/grid_world_consistency/consistency_results_nanob_gemini3_valid.json --model google/gemini-3-flash-preview   







file:///Users/log/Github/sketchvlm/consistency/html_output/vpct_quality_scores.html
vpct_gemini3pro_0_1000
vpct_gpt5low
vpct_thinkmorph
vpct_nanobanana
vpct_gemini_flash
vpct_vilasr

/Users/log/Github/sketchvlm/consistency/compare_figure_pdf.py
^^ want to be in same style as this


/Users/log/Github/sketchvlm/consistency
Please make a different visualization for vpct_quality that works very similarly to compare_figure_pdf. I want it in latex. Don't include rows where there's any sort of errors. 




                                                                  
python consistency/process_with_openrouter.py \
  --input consistency/source_data/count_retry_500_errors.json \
  --output consistency/judge_output/count_retry_500_errors_results.json \
  --model google/gemini-3-pro-preview