Building 70 Projects ranging from beginner to advanced so anyone can β learn from, build upon, use as a reference, or even copy directly. Owned by my cybersecurity company π
Go aicertificationcloud-engineeringcybercybersecuritycybersecurity-analystcybersecurity-certificationcybersecurity-certificationscybersecurity-educationcybersecurity-jobscybersecurity-portfoliocybersecurity-projectscybersecurity-toolsgrchackinglearningpentestingproject-repositoryprojetcspython
feat(lisdex): B83 - the authors did call it, in a target the ladder did not pick
CarterPerez-dev8d5374bdf127
diff --git a/PROJECTS/advanced/zero-day-vulnerability-scanner/crates/lisdex-harness/src/synth.rs b/PROJECTS/advanced/zero-day-vulnerability-scanner/crates/lisdex-harness/src/synth.rsindex c9d74647..b91a6071 100644--- a/PROJECTS/advanced/zero-day-vulnerability-scanner/crates/lisdex-harness/src/synth.rs+++ b/PROJECTS/advanced/zero-day-vulnerability-scanner/crates/lisdex-harness/src/synth.rs@@ -1127,13 +1127,26 @@ pub struct Extension {// The mirror of `setup_run`, and the asymmetry is the point. An entry point the authors never// call has NO tail: `position` returning None yields an empty slice rather than the whole// sequence, so nothing is invented for a candidate the authors said nothing about. The setup-// side does the opposite deliberately - it scans the whole sequence for replayable plumbing --// which is why `archive_read_open_memory` replays six calls while having no tail at all.-fn tail_run<'a>(sequence: &'a [String], entry: &str) -> &'a [String] {- match sequence.iter().position(|call| call == entry) {- Some(index) => &sequence[index + 1..],- None => &[],+// side does the opposite deliberately - it scans the whole sequence for replayable plumbing.+//+// B83. "The authors never call it" was answered against ONE sequence - the tree-wide winner of+// the ladder - and that is a different question from whether they called it at all. libarchive+// ships 25 extracted fuzz targets; the ladder chose `libarchive_seek_fuzzer.cc`, which opens+// with `archive_read_open1`, so `archive_read_open_memory` read as uncalled and had no tail.+// `libarchive_roundtrip_fuzzer.cc` calls it directly, and the tail there is the real one:+// `archive_read_next_header, archive_entry_pathname, archive_entry_size, archive_read_data`.+// Every one of those call lists was already extracted and 24 of 25 were discarded.+//+// The chosen sequence is tried FIRST so nothing that has a tail today gets a different one -+// this only ever fills in an empty answer. Censused over ten tree-runs before it was built:+// 4 runs and 6 candidates gain a tail, and no candidate that already had one changes.+fn tail_run<'a>(sequences: &[&'a [String]], entry: &str) -> &'a [String] {+ for sequence in sequences {+ if let Some(index) = sequence.iter().position(|call| call == entry) {+ return &sequence[index + 1..];+ }}+ &[]}// D4's oracle 3b measures a candidate against the shorter call sequence it EXTENDS, so the@@ -1157,7 +1170,7 @@ fn tail_run<'a>(sequence: &'a [String], entry: &str) -> &'a [String] {pub fn upward_steps(candidate: &Candidate,vocabulary: &Vocabulary,- sequence: &[String],+ sequences: &[&[String]],) -> Vec<Extension> {let Ok(Contexts {locals,@@ -1187,14 +1200,27 @@ pub fn upward_steps(// `resolve_contexts` names locals `ctx`, `ctx1`, `ctx2` off its own count, so a step's own// declaration has to start past every one of them AND past every earlier step's.let mut opened = 0usize;- for name in tail_run(sequence, &candidate.name) {+ for name in tail_run(sequences, &candidate.name) {if steps.len() == MAX_UPWARD_STEPS {break;}// The author's teardown ends the sequence: everything they wrote after it runs on a// handle they had finished with, and appending it would run on one we had freed.+ //+ // B83, and the exception is narrow on purpose. The miner flattens SOURCE order, so an+ // early-return cleanup branch lands in the middle of the success path: libarchive's+ // `if (archive_read_open_memory(...) != ARCHIVE_OK) { archive_read_free(a); return 0; }`+ // puts a free between the opener and every call worth making. That free is one THIS+ // plan already schedules for the same handle, which is exactly what tells the two+ // cases apart - a teardown we emit ourselves at the end is not new knowledge and not a+ // gap, and skipping it here leaves the single teardown the plan always had. A teardown+ // we do NOT schedule is still the end of the road, because it disposes of something we+ // did not know we were holding.if is_teardown(name, vocabulary) {- break;+ if !scheduled.contains(name) {+ break;+ }+ continue;}// Skipped rather than stopped on, because none of the three is a gap in the author's// order: the entry point may appear twice in one sequence, a producer this plan@@ -1902,10 +1928,14 @@ mod tests {fn upward_of(sequence: &[&str]) -> Vec<String> {let owned: Vec<String> = sequence.iter().map(|call| (*call).to_owned()).collect();- upward_steps(&open_memory(), &replayable_archive_vocabulary(), &owned)- .into_iter()- .map(|step| step.function)- .collect()+ upward_steps(+ &open_memory(),+ &replayable_archive_vocabulary(),+ &sole(&owned),+ )+ .into_iter()+ .map(|step| step.function)+ .collect()}// B51(b), the accepting direction, and no tree in the validation set offers it: a call@@ -1947,18 +1977,59 @@ mod tests {}// `archive_read_finish` takes our handle and binds perfectly, so nothing but the teardown- // guard stops it. The plan schedules `archive_read_free`, so keying the guard on the- // scheduled set alone would let this one through and free the handle in front of every- // step after it.+ // guard stops it. The plan schedules `archive_read_free`, NOT this, and a teardown we do+ // not schedule disposes of something we did not know we were holding - so it still ends+ // the climb, taking every call written after it.+ //+ // `archive_read_data` BEHIND the teardown is the whole control and was put there by a+ // gutting pass: with the teardown last, breaking on it and stepping over it both yield an+ // empty list, so making the skip unconditional reddened nothing. A call after it is what+ // makes the two outcomes different.#[test]- fn a_teardown_in_the_tail_is_never_appended() {+ fn a_teardown_the_plan_does_not_schedule_still_ends_the_climb() {assert!(upward_of(&["archive_read_new","archive_read_open_memory","archive_read_finish",+ "archive_read_data",])- .is_empty()+ .is_empty(),+ "a teardown we never scheduled frees a handle every later step would use"+ );+ }++ // B83. The miner flattens SOURCE order, so libarchive's early-return cleanup -+ // `if (archive_read_open_memory(...) != ARCHIVE_OK) { archive_read_free(a); return 0; }` -+ // lands between the opener and every call worth making. That free is the one this plan+ // already schedules for the same handle, so it is not new knowledge and not a gap: skipping+ // it leaves the single teardown the plan always emitted, at the end where it belongs.+ #[test]+ fn a_teardown_the_plan_already_schedules_is_stepped_over_rather_than_stopped_on() {+ assert_eq!(+ upward_of(&[+ "archive_read_new",+ "archive_read_open_memory",+ "archive_read_free",+ "archive_read_data",+ ]),+ vec!["archive_read_data".to_owned()],+ "the author's error-branch cleanup is not the end of their sequence"+ );+ }++ // And the skip does not put a SECOND free in the harness - it drops the call entirely,+ // because the plan's own teardown already covers that handle.+ #[test]+ fn stepping_over_a_scheduled_teardown_never_appends_it_as_a_step() {+ assert!(+ !upward_of(&[+ "archive_read_new",+ "archive_read_open_memory",+ "archive_read_free",+ "archive_read_data",+ ])+ .contains(&"archive_read_free".to_owned()));}@@ -1999,11 +2070,11 @@ mod tests {let steps = upward_steps(&open_memory(),&vocabulary,- &[+ &sole(&["archive_read_open_memory".to_owned(),"LZ4_decompress_safe_partial".to_owned(),"archive_read_data".to_owned(),- ],+ ]),);assert!(steps.is_empty(), "{steps:?}");}@@ -2032,11 +2103,11 @@ mod tests {let steps = upward_steps(&open_memory(),&vocabulary,- &[+ &sole(&["archive_read_open_memory".to_owned(),"archive_read_planes".to_owned(),"archive_read_data".to_owned(),- ],+ ]),);assert!(steps.is_empty(), "{steps:?}");}@@ -2067,10 +2138,10 @@ mod tests {let steps = upward_steps(&open_memory(),&vocabulary,- &[+ &sole(&["archive_read_open_memory".to_owned(),"archive_read_next_header".to_owned(),- ],+ ]),);assert_eq!(steps.len(), 1, "{steps:?}");assert_eq!(steps[0].function, "archive_read_next_header");@@ -2178,7 +2249,7 @@ mod tests {plans[0].body);- let steps = upward_steps(&opener, &vocabulary, &sequence);+ let steps = upward_steps(&opener, &vocabulary, &sole(&sequence));assert_eq!(steps.len(), 1, "{steps:?}");assert_eq!(steps[0].declares,@@ -2219,10 +2290,10 @@ mod tests {upward_steps(&open_memory(),&vocabulary,- &[+ &sole(&["archive_read_open_memory".to_owned(),"archive_read_set_format".to_owned(),- ],+ ]),).is_empty());@@ -2253,7 +2324,7 @@ mod tests {sequence.push(name);}assert_eq!(- upward_steps(&open_memory(), &vocabulary, &sequence).len(),+ upward_steps(&open_memory(), &vocabulary, &sole(&sequence)).len(),MAX_UPWARD_STEPS);}@@ -2273,10 +2344,10 @@ mod tests {let steps = upward_steps(&open_memory(),&replayable_archive_vocabulary(),- &[+ &sole(&["archive_read_open_memory".to_owned(),"archive_read_data".to_owned(),- ],+ ]),);assert_eq!(steps.len(), 1, "the fixture must offer exactly one step");extend(&plans[0], &steps)@@ -3033,6 +3104,60 @@ mod tests {]}+ // B83 let the tail come from any of the author's targets, so `upward_steps` takes a list.+ // Most tests are about one sequence and say so with this.+ fn sole(sequence: &[String]) -> [&[String]; 1] {+ [sequence]+ }++ fn owned(calls: &[&str]) -> Vec<String> {+ calls.iter().map(|call| (*call).to_owned()).collect()+ }++ fn steps_over(sequences: &[&[String]]) -> Vec<String> {+ upward_steps(&open_memory(), &replayable_archive_vocabulary(), sequences)+ .into_iter()+ .map(|step| step.function)+ .collect()+ }++ // B83. libarchive ships 25 extracted fuzz targets; the ladder picks ONE tree-wide, and it+ // picked `libarchive_seek_fuzzer.cc`, which opens with `archive_read_open1`. Asking that+ // one sequence whether the authors ever called `archive_read_open_memory` answers "no",+ // while `libarchive_roundtrip_fuzzer.cc` calls it directly. The call lists were all+ // extracted already; 24 of 25 were being thrown away.+ #[test]+ fn a_candidate_the_chosen_sequence_never_names_takes_its_tail_from_one_that_does() {+ let chosen = owned(&["archive_read_new", "archive_read_open1"]);+ let other = owned(&["archive_read_open_memory", "archive_read_data"]);+ assert_eq!(+ steps_over(&[&chosen, &other]),+ vec!["archive_read_data".to_owned()]+ );+ }++ // The chosen sequence is tried FIRST, so nothing that already had a tail gets a different+ // one. Gut the ordering and this goes red while the test above stays green.+ #[test]+ fn the_chosen_sequence_still_wins_when_both_name_the_candidate() {+ let chosen = owned(&["archive_read_open_memory", "archive_read_data"]);+ let other = owned(&["archive_read_open_memory", "archive_read_next_header"]);+ assert_eq!(+ steps_over(&[&chosen, &other]),+ vec!["archive_read_data".to_owned()],+ "the winner of the ladder is still the harness this plan is shaped like"+ );+ }++ // The isolating control for the search itself: a candidate NO target names still has no+ // tail. Without this, returning the whole of some other sequence would pass the two above.+ #[test]+ fn a_candidate_no_target_names_at_all_still_has_no_tail() {+ let chosen = owned(&["archive_read_new", "archive_read_open1"]);+ let other = owned(&["archive_read_open_filename", "archive_read_data"]);+ assert!(steps_over(&[&chosen, &other]).is_empty());+ }+// The accepting direction. `buffer` names no handle and carries no extent, so it reaches// the same dead end `png_read_image` does - except that libpng's own header computes the// number, and libpng's own example writes `malloc(PNG_IMAGE_SIZE(image))` on the line above.@@ -3040,7 +3165,7 @@ mod tests {fn the_climb_sizes_an_out_param_the_headers_own_macro_states() {let mut vocabulary = sizing_vocabulary(&[("PNG_IMAGE_SIZE", 1)]);with_tail(&mut vocabulary, vec![image_handle(), void_out("buffer")]);- let steps = upward_steps(&begin_read(), &vocabulary, &simplified_read());+ let steps = upward_steps(&begin_read(), &vocabulary, &sole(&simplified_read()));assert_eq!(steps.len(), 1, "{steps:?}");assert_eq!(steps[0].declares,@@ -3081,7 +3206,7 @@ mod tests {fn an_out_param_no_macro_states_the_size_of_still_stops_the_climb() {let mut vocabulary = sizing_vocabulary(&[]);with_tail(&mut vocabulary, vec![image_handle(), void_out("buffer")]);- let steps = upward_steps(&begin_read(), &vocabulary, &simplified_read());+ let steps = upward_steps(&begin_read(), &vocabulary, &sole(&simplified_read()));assert!(steps.is_empty(), "{steps:?}");}@@ -3092,7 +3217,7 @@ mod tests {fn a_macro_that_needs_a_number_the_caller_owns_states_nothing() {let mut vocabulary = sizing_vocabulary(&[("PNG_IMAGE_BUFFER_SIZE", 2)]);with_tail(&mut vocabulary, vec![image_handle(), void_out("buffer")]);- let steps = upward_steps(&begin_read(), &vocabulary, &simplified_read());+ let steps = upward_steps(&begin_read(), &vocabulary, &sole(&simplified_read()));assert!(steps.is_empty(), "{steps:?}");}@@ -3136,10 +3261,10 @@ mod tests {let steps = upward_steps(&open_memory(),&vocabulary,- &[+ &sole(&["archive_read_open_memory".to_owned(),"archive_read_block".to_owned(),- ],+ ]),);assert!(steps.is_empty(), "{steps:?}");}@@ -3181,7 +3306,7 @@ mod tests {),],);- let steps = upward_steps(&begin_read(), &vocabulary, &simplified_read());+ let steps = upward_steps(&begin_read(), &vocabulary, &sole(&simplified_read()));assert_eq!(steps.len(), 1, "{steps:?}");assert_eq!(steps[0].call,@@ -3211,7 +3336,7 @@ mod tests {&mut vocabulary,vec![image_handle(), void_out("buffer"), void_out("colormap")],);- let steps = upward_steps(&begin_read(), &vocabulary, &simplified_read());+ let steps = upward_steps(&begin_read(), &vocabulary, &sole(&simplified_read()));assert_eq!(steps.len(), 1, "{steps:?}");assert_eq!(steps[0].declares,@@ -3251,7 +3376,7 @@ mod tests {let mut vocabulary = sizing_vocabulary(&[("PNG_IMAGE_SIZE", 1)]);with_tail(&mut vocabulary, vec![image_handle(), void_out("buffer")]);let plans = plan_candidate(&begin_read(), "png.h", &vocabulary).unwrap();- let steps = upward_steps(&begin_read(), &vocabulary, &simplified_read());+ let steps = upward_steps(&begin_read(), &vocabulary, &sole(&simplified_read()));let source = emit::render(&extend(&plans[0], &steps), "libpng_finish.c");assert!(source.contains("size_t lisdex_sized_len = (size_t)(PNG_IMAGE_SIZE(ctx));"),@@ -3313,7 +3438,7 @@ mod tests {#[test]fn a_step_declares_the_context_the_entry_point_never_took() {let vocabulary = context_vocabulary();- let steps = upward_steps(&begin_read(), &vocabulary, &simplified_read());+ let steps = upward_steps(&begin_read(), &vocabulary, &sole(&simplified_read()));assert_eq!(steps.len(), 1, "{steps:?}");assert_eq!(steps[0].declares[0],@@ -3336,7 +3461,7 @@ mod tests {fn a_record_the_library_hands_back_from_a_constructor_is_not_declared_by_a_step() {let mut vocabulary = context_vocabulary();vocabulary.produced.insert("png_color_struct".to_owned());- let steps = upward_steps(&begin_read(), &vocabulary, &simplified_read());+ let steps = upward_steps(&begin_read(), &vocabulary, &sole(&simplified_read()));assert!(steps.is_empty(), "{steps:?}");}@@ -3349,7 +3474,7 @@ mod tests {"png_color_struct".to_owned(),BTreeSet::from(["png_color_free".to_owned()]),);- let steps = upward_steps(&begin_read(), &vocabulary, &simplified_read());+ let steps = upward_steps(&begin_read(), &vocabulary, &sole(&simplified_read()));assert!(steps.is_empty(), "{steps:?}");}@@ -3373,7 +3498,7 @@ mod tests {),],);- let steps = upward_steps(&begin_read(), &vocabulary, &simplified_read());+ let steps = upward_steps(&begin_read(), &vocabulary, &sole(&simplified_read()));assert_eq!(steps.len(), 1, "{steps:?}");assert_eq!(steps[0].declares.len(), 1, "{:?}", steps[0].declares);let Stmt::Call { args, .. } = &steps[0].call else {@@ -3395,7 +3520,7 @@ mod tests {param("row_stride", "png_int_32", Shape::Integral),],);- let steps = upward_steps(&begin_read(), &vocabulary, &simplified_read());+ let steps = upward_steps(&begin_read(), &vocabulary, &sole(&simplified_read()));assert_eq!(steps.len(), 1, "{steps:?}");assert_eq!(steps[0].call,@@ -3422,7 +3547,7 @@ mod tests {param("row_stride", "png_int_32", Shape::Integral),],);- let steps = upward_steps(&begin_read(), &vocabulary, &simplified_read());+ let steps = upward_steps(&begin_read(), &vocabulary, &sole(&simplified_read()));assert!(steps.is_empty(), "{steps:?}");}@@ -3436,7 +3561,7 @@ mod tests {&mut vocabulary,vec![image_handle(), param("size", "size_t", Shape::Integral)],);- let steps = upward_steps(&begin_read(), &vocabulary, &simplified_read());+ let steps = upward_steps(&begin_read(), &vocabulary, &sole(&simplified_read()));assert_eq!(steps.len(), 1, "{steps:?}");let Stmt::Call { args, .. } = &steps[0].call else {panic!("the step is a call");diff --git a/PROJECTS/advanced/zero-day-vulnerability-scanner/crates/lisdex/src/main.rs b/PROJECTS/advanced/zero-day-vulnerability-scanner/crates/lisdex/src/main.rsindex e9632ed6..4f4a836d 100644--- a/PROJECTS/advanced/zero-day-vulnerability-scanner/crates/lisdex/src/main.rs+++ b/PROJECTS/advanced/zero-day-vulnerability-scanner/crates/lisdex/src/main.rs@@ -373,6 +373,7 @@ fn harness(args: &HarnessArgs) -> Result<(), Box<dyn Error>> {}let sequence = authored_sequence(&analysis);+ let sequences = authored_sequences(&analysis);let ranked = ladder_ordered(&analysis);let attempts: Vec<search::Attempt> = ranked.into_iter()@@ -383,7 +384,7 @@ fn harness(args: &HarnessArgs) -> Result<(), Box<dyn Error>> {Ok(variants) => search::Attempt::Planned {name: candidate.name.clone(),variants,- steps: synth::upward_steps(candidate, &vocabulary, sequence),+ steps: synth::upward_steps(candidate, &vocabulary, &sequences),},Err(refusal) => search::Attempt::Refused {name: candidate.name.clone(),@@ -549,6 +550,23 @@ fn authored_sequence(analysis: &surface::Surface) -> &[String] {.map_or(&[][..], |sequence| sequence.names())}+// B83. The setup replay works from the ONE sequence the ladder chose, because a harness is+// assembled in the shape of a single author's target. The tail is a different question - "did+// the authors ever call this, and what did they do next" - and answering it from the winner+// alone made `archive_read_open_memory` read as uncalled while `libarchive_roundtrip_fuzzer.cc`+// calls it directly. The chosen sequence stays FIRST so a candidate that already has a tail+// keeps exactly the one it had.+fn authored_sequences(analysis: &surface::Surface) -> Vec<&[String]> {+ let mut sequences = vec![authored_sequence(analysis)];+ sequences.extend(+ analysis+ .harnesses+ .iter()+ .map(|harness| harness.calls.as_slice()),+ );+ sequences+}+fn report_ladder(ladder: &lisdex_surface::ladder::Ladder) {match &ladder.chosen {Some(sequence) => println!(@@ -585,6 +603,7 @@ fn plan_only(top: usize,) {let sequence = authored_sequence(analysis);+ let sequences = authored_sequences(analysis);for (index, candidate) in ladder_ordered(analysis).into_iter().take(top).enumerate() {let include = include_of(candidate, root);let order = index + 1;@@ -604,7 +623,7 @@ fn plan_only(// On its own line, because the replay line is what the M3 gate asserts a// teardown never appears in, and a tail that legitimately contains one would// have turned that assertion red from the other side of the entry point.- let steps = synth::upward_steps(candidate, vocabulary, sequence);+ let steps = synth::upward_steps(candidate, vocabulary, &sequences);if !steps.is_empty() {println!(" extending {:?}", step_names(&steps));}diff --git a/PROJECTS/advanced/zero-day-vulnerability-scanner/scripts/m3-gate.sh b/PROJECTS/advanced/zero-day-vulnerability-scanner/scripts/m3-gate.shindex 8439d054..9216fbba 100755--- a/PROJECTS/advanced/zero-day-vulnerability-scanner/scripts/m3-gate.sh+++ b/PROJECTS/advanced/zero-day-vulnerability-scanner/scripts/m3-gate.sh@@ -514,15 +514,35 @@ elseok "the climb stops at the author's own teardown"fi# The measured gap, on a real tree: libarchive's tail is `archive_read_next_header,-# archive_entry_pathname, archive_entry_size, archive_read_data`. The first takes-# `struct archive_entry **` and cannot bind, and appending `archive_read_data` over that gap-# would read from a stream whose header was never read - B66's semantically dead harness-# arriving through the mechanism meant to add depth.-if echo "$LADDER_ARCHIVE" | grep -q "extending"; then+# archive_entry_pathname, archive_entry_size, archive_read_data`.+#+# This assertion used to be "libarchive appends NOTHING", on the ground that+# `archive_read_next_header` takes `struct archive_entry **` and could not bind. B81 made that+# binding real - the library allocates the entry and writes its address into a slot the harness+# declares as a NULL local - and B83 gave the candidate a tail to read at all, so the premise is+# gone and the assertion is now the specific climb rather than its absence.+#+# What must NOT change is what the old assertion was protecting: `archive_read_data` may never+# be appended. `archive_entry_pathname` needs the entry the STEP declared rather than anything+# the plan holds, so the climb stops there - and appending `read_data` past it would read from a+# stream whose header was never read, which is B66's semantically dead harness arriving through+# the mechanism meant to add depth. Both halves are asserted, because "it climbed" and "it did+# not climb too far" are different claims and only one of them is about B83.+if echo "$LADDER_ARCHIVE" | grep -q 'extending \["archive_read_next_header"\]'; then+ ok "libarchive climbs to the author's next call, which only binds because the library fills it"+else+ bad "libarchive did not append the one call its author's target makes next: \+$(echo "$LADDER_ARCHIVE" | grep -E 'extending|PLANS archive_read_open_memory' | head -1)"+fi+# Scoped to the `extending` line alone. The plan log also carries+# `refused archive_read_data - ...`, so grepping the whole of it matched the REFUSAL and made+# this fail while the climb was correct - a control that reads the wrong line asserts nothing+# about the right one.+if echo "$LADDER_ARCHIVE" | grep extending | grep -q "archive_read_data"; thenbad "libarchive appended a step across a call it cannot copy: \$(echo "$LADDER_ARCHIVE" | grep extending | head -1)"else- ok "libarchive appends nothing, because the call in front of its tail cannot bind"+ ok "and stops before archive_read_data, which no call it can copy has made reachable"firm -rf "$WORK/upward"