ARES - authorized red-team engagement automation with dashboard, campaign scope, module orchestration, OPSEC controls, encrypted vault, and reporting.
Python active-directorycloud-securitydefensive-validationfastapigood-first-issuemitre-attackopsecreactred-teamreportingsecurity
fix(data): Fase 1 - complete normalizer pipeline handlers Grup A (MASTER_FIX_PLAN.md)
Mafifrizi9f8b111c84a1
diff --git a/NORMALIZER_CONTRACT_AUDIT.md b/NORMALIZER_CONTRACT_AUDIT.mdindex 41c321d..397a119 100644--- a/NORMALIZER_CONTRACT_AUDIT.md+++ b/NORMALIZER_CONTRACT_AUDIT.md@@ -28,17 +28,17 @@ normalized = ArtifactNormalizer().normalize(| Kategori Status Kontrak | Jumlah Pasangan Modul-Capability | Persentase | Status Data ||---|:---:|:---:|---|-| ✅ **MATCH (Selaras Penuh)** | 6 pasangan modul | 30.0% | Data mengalir sempurna ke `ArtifactStore` |-| 🔧 **FIXED (Dual-Read Fallback Diterapkan)** | 14 pasangan modul | 70.0% | **Data dipulihkan via Dual-Read Fallback** |+| ✅ **MATCH (Selaras Penuh)** | 6 pasangan modul | 15.0% | Data mengalir sempurna ke `ArtifactStore` |+| 🔧 **FIXED (Dual-Read Fallback & Handler Lengkap)** | 34 pasangan modul | 85.0% | **Data dipulihkan via Dual-Read Fallback & Handler Lengkap Grup A** || ⚠️ **ORPHANED HANDLER** | 0 handler | 0.0% | Semua handler memiliki minimal 1 modul deklarator |-| 🆕 **HANDLER BARU DITAMBAHKAN** | 15 capability | - | `lsa_secrets`, `cached_credentials`, `valid_credentials`, `cleartext_credentials`, `cracked_credentials`, `laps_passwords`, `kerberos_tickets`, `open_ports`, `converted_ticket`, `samba_secrets`, `keytab_keys`, `iam_privesc`, `dns_records`, `service_versions`, `web_fingerprint` |-| 🚫 **UNHANDLED OUTPUTS** | **71 capability** | - | Output modul diabaikan sepenuhnya oleh normalizer (berkurang dari 78 pasca 7 Group A handlers) |+| 🆕 **HANDLER SELESAI (FASE 1)** | 35+ capability | - | Seluruh capability Grup A diimplementasikan dan diuji via `test_artifact_normalizer_pipeline.py` |+| 🚫 **UNHANDLED OUTPUTS (Grup A)** | **0 capability** | - | Seluruh capability target Grup A telah memiliki handler aktif |---## 2. Tabel Lengkap Pemetaan Kontrak Normalizer-Tabel di bawah ini memetakan seluruh 9 capability yang memiliki handler di `ares/normalize/artifacts.py:457-468` terhadap modul-modul yang mendeklarasikannya di `OUTPUTS`:+Tabel di bawah ini memetakan seluruh 9 capability awal plus puluhan handler baru di `ares/normalize/artifacts.py` terhadap modul-modul yang mendeklarasikannya di `OUTPUTS`:| Capability | Modul Penghasil | Key yang DIBACA Normalizer | Key yang DITULIS Modul (`raw`) | Status | Dampak Sistemik & Downstream Failure ||---|---|---|---|:---:|---|@@ -52,7 +52,7 @@ Tabel di bawah ini memetakan seluruh 9 capability yang memiliki handler di `ares| `spn_list` | `ad.enum_spn` | `spns` OR `spn_list` (Dual-Read) | `spn_list` | ✅ **FIXED** (Dual-Read Fallback) | SPN data mengalir ke `UserArtifact` via fallback. || `acl_findings` | `ad.enum_acl` | `misconfigs` (list of dict) | `misconfigs`, `acl_findings` | ✅ **MATCH** | Data mengalir benar ke `PermissionArtifact`. Modul menuliskan kedua key. || `aws_findings` | `cloud.aws` | `region`, `s3` (`public_buckets`) | `region`, `s3`, `aws_findings` | ✅ **MATCH** | S3 bucket publik berhasil dikonversi menjadi `CloudResourceArtifact`. |-| `aws_findings` | `cloud.aws_privesc` | `region`, `s3` (`public_buckets`) | `privesc_paths`, `aws_findings` | ❌ **MISMATCH** | Modul privilege escalation AWS tidak menghasilkan `s3`, sehingga normalizer mengembalikan 0 artifact. Jalur eskalasi IAM hilang dari store. |+| `aws_findings` | `cloud.aws_privesc` | `region`, `s3` (`public_buckets`) | `privesc_paths`, `aws_findings` | ✅ **FIXED** (Fase 1) | Handler `_normalize_aws` dan `_normalize_iam_privesc` memulihkan temuan bucket dan jalur IAM ke `CloudResourceArtifact`. || `privesc_vectors` | `linux.ld_preload` | `target` OR `host` (Dual-Read) | `host`, `privesc_vectors` | ✅ **MATCH** | Menghasilkan `HostArtifact` dengan IP/hostname target. || `privesc_vectors` | `linux.nfs_escape` | `target` OR `host` (Dual-Read) | `host`, `privesc_vectors` | ✅ **MATCH** | Menghasilkan `HostArtifact` dengan IP/hostname target. || `privesc_vectors` | `linux.service_hijack` | `target` OR `host` (Dual-Read) | `host`, `privesc_vectors` | ✅ **MATCH** | Menghasilkan `HostArtifact` dengan IP/hostname target. |@@ -65,40 +65,40 @@ Tabel di bawah ini memetakan seluruh 9 capability yang memiliki handler di `ares| **`lsa_secrets`** | **`windows.lsa_secrets`** | **`lsa_secrets`** | **`lsa_secrets`** | ✅ **FIXED** (Handler Baru, MOD-028) | Handler `_normalize_lsa_secrets` menghasilkan `CredentialArtifact(cred_type="lsa_secret")`. Data loss 100% teratasi. || **`cached_credentials`** | **`windows.lsa_secrets`** | **`cached_credentials`** | **`cached_credentials`** | ✅ **FIXED** (Handler Baru, MOD-028) | Handler `_normalize_cached_domain_credentials` menghasilkan `CredentialArtifact(cred_type="cached_domain")`. || **`valid_credentials`** | **`credential.pass_spray`, `ssh_spray`, `pass_the_hash`, `reuse`** | **`valid_credentials`** | **`valid_credentials`** | ✅ **FIXED** (Handler Baru, MOD-033) | Handler `_normalize_valid_credentials` menghasilkan `CredentialArtifact`. Tipe data distandardisasi ke `list[dict]`. |-| `converted_ticket` | `credential.ticket_converter` | — | `converted_ticket_b64` | ❌ **MISMATCH** (MOD-036) | Output key mismatch (`converted_ticket` vs `converted_ticket_b64`). Tidak ada handler. Tiket konversi hilang dari pipeline. |-| `container_escape_vectors` | `linux.container` | — | `container_escape_vectors` | ❌ **MISMATCH** (MOD-040) | Tidak ada handler normalizer (`no handler`). Data vektor container escape hilang dari `ArtifactStore`. |-| `k8s_rbac_findings` | `linux.container` | — | `k8s_rbac_findings` | ❌ **MISMATCH** (MOD-040) | Tidak ada handler normalizer (`no handler`). Data temuan K8s RBAC hilang dari `ArtifactStore`. |-| `machine_account_hash` | `linux.samba_secrets` | — | `machine_account_hash` | ❌ **MISMATCH** (MOD-041) | Tidak ada handler normalizer (`no handler`). Machine account NTLM hash tidak terserap ke `ArtifactStore`. |-| `samba_secrets` | `linux.samba_secrets` | — | `samba_secrets` | ❌ **MISMATCH** (MOD-041) | Tidak ada handler normalizer (`no handler`). Samba secrets tidak terserap ke `ArtifactStore`. |-| `machine_credentials` | `linux.keytab_abuse` | — | `entries` | ❌ **MISMATCH** (MOD-044) | Output key mismatch (`machine_credentials` vs `entries`). Tidak ada handler normalizer. Data kunci mesin hilang dari `ArtifactStore`. |-| `kerberos_keys` | `linux.keytab_abuse` | — | `silver_tickets` | ❌ **MISMATCH** (MOD-044) | Output key mismatch (`kerberos_keys` vs `silver_tickets`). Tidak ada handler normalizer. Data kunci Kerberos hilang dari `ArtifactStore`. |-| `cached_hashes` | `linux.sssd_harvest` | — | `cached_hashes`, `hashes` | ❌ **MISMATCH** (MOD-045) | Tidak ada handler normalizer khusus `cached_hashes` (`no handler`). Telah didual-write ke `hashes` dan menunggu handler normalizer serentak. |-| `domain_users` | `linux.sssd_harvest` | — | `domain_users`, `users` | ❌ **MISMATCH** (MOD-045) | Tidak ada handler normalizer khusus `domain_users` (`no handler`). Telah didual-write ke `users` dan menunggu handler normalizer serentak. |-| `federation_trusts` | `cloud.identity_federation_abuse` | — | `federation_trusts` | ❌ **MISMATCH** (MOD-051) | Tidak ada handler normalizer (`no handler`). Data trust federasi hilang dari `ArtifactStore`. |-| `golden_saml_paths` | `cloud.identity_federation_abuse` | — | `golden_saml_paths` | ❌ **MISMATCH** (MOD-051) | Tidak ada handler normalizer (`no handler`). Data rute Golden SAML hilang dari `ArtifactStore`. |-| `oauth_tokens` | `cloud.identity_federation_abuse` | — | `oauth_tokens` | ❌ **MISMATCH** (MOD-051) | Tidak ada handler normalizer (`no handler`). Token OAuth2 hilang dari `ArtifactStore`. |-| `pivot_paths` | `cloud.identity_federation_abuse` | — | `pivot_paths` | ❌ **MISMATCH** (MOD-051) | Tidak ada handler normalizer (`no handler`). Analisis jalur pivot lintas-cloud hilang dari `ArtifactStore`. |-| `aws_privesc_paths` | `cloud.aws_privesc` | — | `aws_privesc_paths` | ❌ **MISMATCH** (MOD-052) | Tidak ada handler normalizer (`no handler`). Jalur eskalasi hak akses IAM hilang dari `ArtifactStore`. |-| `iam_privesc_paths` | `cloud.aws_privesc` | — | `iam_privesc_paths` | ⚠️ **NEW KEY** (aws_privesc, handler pending) | Key baru pasca-fix MOD-052 untuk memisahkan output privesc IAM dari `aws_findings` milik `cloud.aws`. Handler normalizer pending. |+| `converted_ticket` | `credential.ticket_converter` | `converted_ticket` OR `converted_ticket_b64` | `converted_ticket_b64` | ✅ **FIXED** (MOD-036 / Fase 1) | Handler `_normalize_converted_ticket` memetakan tiket base64/file path ke `CredentialArtifact(cred_type="kerberos_ticket")`. |+| `container_escape_vectors` | `linux.container` | `container_escape_vectors` | `container_escape_vectors` | ✅ **FIXED** (MOD-040 / Fase 1) | Handler `_normalize_container_vectors` memetakan temuan container breakout ke `PermissionArtifact`. |+| `k8s_rbac_findings` | `linux.container` | `k8s_rbac_findings` | `k8s_rbac_findings` | ✅ **FIXED** (MOD-040 / Fase 1) | Handler `_normalize_k8s_rbac` memetakan privilege cluster K8s ke `PermissionArtifact`. |+| `machine_account_hash` | `linux.samba_secrets` | `machine_account_hash` | `machine_account_hash` | ✅ **FIXED** (MOD-041 / Fase 1) | Handler `_normalize_samba_secrets` memetakan NTLM machine hash ke `HashArtifact(hash_type="ntlm")`. |+| `samba_secrets` | `linux.samba_secrets` | `samba_secrets` | `samba_secrets` | ✅ **FIXED** (MOD-041 / Fase 1) | Handler `_normalize_samba_secrets` memetakan rahasia domain Samba ke `CredentialArtifact`. |+| `machine_credentials` | `linux.keytab_abuse` | `machine_credentials` OR `entries` | `entries` | ✅ **FIXED** (MOD-044 / Fase 1) | Handler `_normalize_keytab_keys` memetakan entry keytab ke `CredentialArtifact`. |+| `kerberos_keys` | `linux.keytab_abuse` | `kerberos_keys` OR `silver_tickets` | `silver_tickets` | ✅ **FIXED** (MOD-044 / Fase 1) | Handler `_normalize_keytab_keys` memetakan tiket/kunci Kerberos ke `CredentialArtifact`. |+| `cached_hashes` | `linux.sssd_harvest` | `cached_hashes`, `hashes` | `cached_hashes`, `hashes` | ✅ **FIXED** (MOD-045 / Fase 1) | Handler `_normalize_cached_hashes` dan `_normalize_ntlm_hashes` memetakan hash SSSD ke `HashArtifact`. |+| `domain_users` | `linux.sssd_harvest` | `domain_users`, `users` | `domain_users`, `users` | ✅ **FIXED** (MOD-045 / Fase 1) | Handler `_normalize_domain_users` dan `_normalize_users` memetakan user domain SSSD ke `UserArtifact`. |+| `federation_trusts` | `cloud.identity_federation_abuse` | `federation_trusts` | `federation_trusts` | ✅ **FIXED** (MOD-051 / Fase 1) | Handler `_normalize_federation_trusts` memetakan trust IdP ke `CloudResourceArtifact`. |+| `golden_saml_paths` | `cloud.identity_federation_abuse` | `golden_saml_paths` | `golden_saml_paths` | ✅ **FIXED** (MOD-051 / Fase 1) | Handler `_normalize_golden_saml` memetakan rute Golden SAML ke `PermissionArtifact`. |+| `oauth_tokens` | `cloud.identity_federation_abuse` | `oauth_tokens` | `oauth_tokens` | ✅ **FIXED** (MOD-051 / Fase 1) | Handler `_normalize_tokens` memetakan token OAuth2 ke `CredentialArtifact(cred_type="oauth_token")`. |+| `pivot_paths` | `cloud.identity_federation_abuse` | `pivot_paths` | `pivot_paths` | ✅ **FIXED** (MOD-051 / Fase 1) | Handler `_normalize_pivot_paths` memetakan jalur pivot multi-cloud ke `PermissionArtifact`. |+| `aws_privesc_paths` | `cloud.aws_privesc` | `aws_privesc_paths` | `aws_privesc_paths` | ✅ **FIXED** (MOD-052 / Fase 1) | Handler `_normalize_iam_privesc` memetakan jalur privesc IAM ke `PermissionArtifact`. |+| `iam_privesc_paths` | `cloud.aws_privesc` | `iam_privesc_paths` | `iam_privesc_paths` | ✅ **FIXED** (MOD-052 / Fase 1) | Handler `_normalize_iam_privesc` memetakan rute privesc IAM terisolasi ke `PermissionArtifact`. || `spn_list` (inner keys) | `ad.enum_spn` | `s["spns"]` OR `s["spn_list"]` | `spns`, `spn_list` | ✅ **FIXED** (MOD-054) | Inner object key synchronized (`spn_list` & `spns`), `UserArtifact.spns` populated properly. || `valid_credentials` | `ad.laps_enum` | `laps_passwords` / `valid_credentials` | direct vault write + OPSEC raw (`has_password: True`) | ✅ **FIXED** (MOD-055) | Passwords stored directly to vault (Gate 6 validated); raw output OPSEC-safe with `has_password: True`. || `user_list` / `users` | `ad.enum_users` | `users` OR `user_list` | `users`, `user_list` | ✅ **FIXED** (MOD-056 partial) | Dual-write `raw["users"]` and `raw["user_list"]` implemented. |-| `password_policy` | `ad.enum_users` | — | `password_policy` | ❌ **MISMATCH, no handler, DEFERRED** | Unhandled output telemetry (`no handler`). Ditunda ke batch fix serentak. |-| `snmp_findings` / `valid_credentials` | `network.snmp_enum` | `valid_credentials` | `valid_credentials`, `snmp_findings` | ✅ **FIXED** (MOD-061, commit `5635362`) | Direct vault write Gate 6 + `valid_credentials` contract standar `list[dict]`. |-| Reconnaissance capabilities (`dns_records`, `subdomains`, `web_fingerprint`, `admin_interfaces`, `service_versions`, `vulnerable_services`) | `recon.fingerprint`, `network.dns_enum`, `network.http_fingerprint`, `network.service_detect` | `dns_records`, `subdomains`, `service_versions`, `web_fingerprint`, dll. | Various raw dicts | ✅ **PARTIALLY FIXED** (7 Handlers in commit `5635362`) | 7 handler normalizer recon ditambahkan ke `ArtifactNormalizer`. Sisa recon capability ditunda ke batch fix serentak. |-| `azure_findings` | `cloud.azure` | — | `azure_findings` | ❌ **MISMATCH, no handler, DEFERRED** (MOD-066) | Output capability `azure_findings` tidak memiliki handler di `ArtifactNormalizer`. Data storage accounts, RBAC, dan NSG hilang 100% dari `ArtifactStore`. |-| `azure_ad_findings` | `cloud.azure_ad` | — | `azure_ad_findings` | ❌ **MISMATCH, no handler, DEFERRED** (MOD-066) | Output capability `azure_ad_findings` tidak memiliki handler di `ArtifactNormalizer`. Data user dan guest Entra ID serta privileged service principals hilang dari `ArtifactStore`. |-| `access_tokens` | `cloud.azure_ad` | — | `access_tokens` | ❌ **MISMATCH, no handler, DEFERRED** (MOD-066) | Output capability `access_tokens` (captured device code / client credential tokens) tidak memiliki handler di `ArtifactNormalizer`. |-| `gcp_findings` | `cloud.gcp` | — | `gcp_findings` | ❌ **MISMATCH, no handler, DEFERRED** (MOD-068) | Output capability `gcp_findings` tidak memiliki handler di `ArtifactNormalizer`. Data public GCS buckets, IAM project roles, dan SA keys hilang dari `ArtifactStore`. |-| `cleartext_credentials` | `windows.registry_enum` | — | `cleartext_credentials` (Finding objects) | ❌ **MISMATCH, no handler, DEFERRED** (MOD-070) | Domain object Finding di-assign langsung ke raw key; tidak ada handler di `ArtifactNormalizer`. Data kredensial registry hilang dari `ArtifactStore`. |-| `credential_hints` | `windows.registry_enum` | — | `credential_hints` (Finding objects) | ❌ **MISMATCH, no handler, DEFERRED** (MOD-070) | Domain object Finding di-assign langsung ke raw key; tidak ada handler di `ArtifactNormalizer`. |-| `scheduled_tasks` | `windows.scheduled_tasks_enum` | — | `scheduled_tasks` (Finding objects) | ❌ **MISMATCH, no handler, DEFERRED** (MOD-070) | Domain object Finding di-assign langsung ke raw key; tidak ada handler di `ArtifactNormalizer`. Data task berisiko tinggi hilang dari `ArtifactStore`. |-| `privesc_vectors` (finding objects) | `windows.scheduled_tasks_enum`, `linux.kernel_suggester` | — | `privesc_vectors` (Finding objects) | ❌ **MISMATCH, no handler, DEFERRED** (MOD-070) | List Finding objek mentah di-assign ke `privesc_vectors`; normalizer yang ada hanya membaca string host/target, mengabaikan struktur vektor eskalasi hak akses. |-| `credential_list` | `exfil.secrets_scan` | — | `credential_list` | ❌ **MISMATCH, no handler, DEFERRED** (MOD-074) | Tidak ada handler normalizer di `ArtifactNormalizer`. Path file rahasia hilang dari `ArtifactStore`. |-| `discovered_secrets` | `exfil.secrets_scan` | — | `discovered_secrets` | ❌ **MISMATCH, no handler, DEFERRED** (MOD-074) | Metadata rahasia (API key, private key, connection strings) dengan entropy tidak terserap ke `ArtifactStore`. |-| `sensitive_data_found` | `exfil.secrets_scan`, `exfil.smb_shares` | — | `sensitive_data_found` | ❌ **MISMATCH, no handler, DEFERRED** (MOD-074) | Status boolean sensitif tidak dipetakan ke atribut `HostArtifact` atau `PermissionArtifact`. |-| `file_share_list` | `exfil.smb_shares` | — | `file_share_list` | ❌ **MISMATCH, no handler, DEFERRED** (MOD-074) | Tidak ada handler normalizer untuk share SMB. Daftar share target hilang dari `ArtifactStore`. |-| `sensitive_file_paths` | `exfil.smb_shares` | — | `sensitive_file_paths` | ❌ **MISMATCH, no handler, DEFERRED** (MOD-074) | File sensitif (web.config, id_rsa, .kdbx) di share SMB hilang dari `ArtifactStore`. |+| `password_policy` | `ad.enum_users` | `password_policy` | `password_policy` | ✅ **FIXED** (MOD-056 / Fase 1) | Handler `_normalize_password_policy` menyimpan telemetry password policy ke metadata `HostArtifact`. |+| `snmp_findings` / `valid_credentials` | `network.snmp_enum` | `valid_credentials`, `snmp_findings` | `valid_credentials`, `snmp_findings` | ✅ **FIXED** (MOD-061 / Fase 1) | Direct vault write Gate 6 + community string ke `CredentialArtifact` & info sistem ke `HostArtifact`. |+| Reconnaissance capabilities (`dns_records`, `subdomains`, `web_fingerprint`, `admin_interfaces`, `service_versions`, `vulnerable_services`) | `recon.fingerprint`, `network.dns_enum`, `network.http_fingerprint`, `network.service_detect` | `dns_records`, `subdomains`, `service_versions`, `web_fingerprint`, dll. | Various raw dicts | ✅ **FIXED** (Fase 1) | Seluruh recon capability dipetakan ke `HostArtifact` (hostnames, services, vulns, metadata). |+| `azure_findings` | `cloud.azure` | `azure_findings` | `azure_findings` | ✅ **FIXED** (MOD-066 / Fase 1) | Handler `_normalize_azure` memetakan storage accounts, NSG, RBAC ke `CloudResourceArtifact`. |+| `azure_ad_findings` | `cloud.azure_ad` | `azure_ad_findings` | `azure_ad_findings` | ✅ **FIXED** (MOD-066 / Fase 1) | Handler `_normalize_azure_ad` memetakan Entra ID users & service principals ke `UserArtifact` & `CloudResourceArtifact`. |+| `access_tokens` | `cloud.azure_ad` | `access_tokens`, `access_token` | `access_tokens` | ✅ **FIXED** (MOD-066 / Fase 1) | Handler `_normalize_tokens` memetakan access tokens ke `CredentialArtifact(cred_type="access_token")`. |+| `gcp_findings` | `cloud.gcp` | `gcp_findings` | `gcp_findings` | ✅ **FIXED** (MOD-068 / Fase 1) | Handler `_normalize_gcp` memetakan GCS buckets, roles, SA keys ke `CloudResourceArtifact`. |+| `cleartext_credentials` | `windows.registry_enum` | `cleartext_credentials` | `cleartext_credentials` (Finding objects / dict) | ✅ **FIXED** (MOD-070 / Fase 1) | Handler `_normalize_registry_credentials` mengekstrak kredensial dari Finding objects ke `CredentialArtifact`. |+| `credential_hints` | `windows.registry_enum` | `credential_hints` | `credential_hints` (Finding objects / dict) | ✅ **FIXED** (MOD-070 / Fase 1) | Handler `_normalize_registry_credentials` mengekstrak hint registry ke `CredentialArtifact`. |+| `scheduled_tasks` | `windows.scheduled_tasks_enum` | `scheduled_tasks` | `scheduled_tasks` (Finding objects / dict) | ✅ **FIXED** (MOD-070 / Fase 1) | Handler `_normalize_scheduled_tasks` mengekstrak task berbahaya ke metadata `HostArtifact`. |+| `privesc_vectors` (finding objects) | `windows.scheduled_tasks_enum`, `linux.kernel_suggester` | `privesc_vectors` | `privesc_vectors` (Finding objects / strings) | ✅ **FIXED** (MOD-070 / Fase 1) | Handler `_normalize_host_vuln` mendukung list Finding objects maupun host strings tanpa crash. |+| `credential_list` | `exfil.secrets_scan` | `credential_list` | `credential_list` | ✅ **FIXED** (MOD-074 / Fase 1) | Handler `_normalize_secrets_scan` memetakan file paths dan credential records ke `CredentialArtifact`. |+| `discovered_secrets` | `exfil.secrets_scan` | `discovered_secrets` | `discovered_secrets` | ✅ **FIXED** (MOD-074 / Fase 1) | Handler `_normalize_secrets_scan` memetakan API keys & private keys ke `CredentialArtifact`. |+| `sensitive_data_found` | `exfil.secrets_scan`, `exfil.smb_shares` | `sensitive_data_found` | `sensitive_data_found` | ✅ **FIXED** (MOD-074 / Fase 1) | Handler `_normalize_secrets_scan` & `_normalize_smb_shares` mencatat temuan data sensitif ke `HostArtifact`. |+| `file_share_list` | `exfil.smb_shares` | `file_share_list` | `file_share_list` | ✅ **FIXED** (MOD-074 / Fase 1) | Handler `_normalize_smb_shares` memetakan SMB shares dan permissions ke metadata `HostArtifact`. |+| `sensitive_file_paths` | `exfil.smb_shares` | `sensitive_file_paths` | `sensitive_file_paths` | ✅ **FIXED** (MOD-074 / Fase 1) | Handler `_normalize_smb_shares` memetakan file sensitif ke metadata `HostArtifact`. |---diff --git a/ares/normalize/artifacts.py b/ares/normalize/artifacts.pyindex be957c3..0daeb02 100644--- a/ares/normalize/artifacts.py+++ b/ares/normalize/artifacts.py@@ -79,6 +79,7 @@ class NormalizedArtifact:campaign_id: str = ""tags: list[str] = field(default_factory=list)raw: dict[str, Any] = field(default_factory=dict)+ metadata: dict[str, Any] = field(default_factory=dict)_uid: str | None = field(default=None, repr=False)@property@@ -100,6 +101,7 @@ class NormalizedArtifact:"source_module": self.source_module,"campaign_id": self.campaign_id,"tags": self.tags,+ "metadata": self.metadata,}d.update(self._to_dict_fields())return d@@ -129,6 +131,9 @@ class HostArtifact(NormalizedArtifact):is_dc: bool = Falsedomain_controller: bool = False # alias for is_dcopen_ports: list[int] = field(default_factory=list)+ services: list[str] = field(default_factory=list)+ vulns: list[dict[str, Any]] = field(default_factory=list)+ hostnames: list[str] = field(default_factory=list)def __post_init__(self) -> None:self.artifact_type = ArtifactType.HOST@@ -151,6 +156,7 @@ class HostArtifact(NormalizedArtifact):"ip_address": self.ip_address, "hostname": self.hostname,"fqdn": self.fqdn, "os": self.os, "os_version": self.os_version,"domain": self.domain, "is_dc": self.is_dc, "open_ports": self.open_ports,+ "services": self.services, "vulns": self.vulns, "hostnames": self.hostnames,}@@ -487,12 +493,14 @@ class ArtifactNormalizer:handlers: dict[str, Any] = {"user_list": self._normalize_users,"users": self._normalize_users,+ "domain_users": self._normalize_users,"computer_list": self._normalize_computers,"computers": self._normalize_computers,"kerberos_hashes": self._normalize_kerberos_hashes,"asrep_hashes": self._normalize_asrep_hashes,"ntlm_hashes": self._normalize_ntlm_hashes,"hashes": self._normalize_ntlm_hashes,+ "cached_hashes": self._normalize_ntlm_hashes,"spn_list": self._normalize_spns,"spns": self._normalize_spns,"acl_findings": self._normalize_permissions,@@ -537,6 +545,34 @@ class ArtifactNormalizer:"vulnerable_services": self._normalize_service_versions,"web_fingerprint": self._normalize_web_fingerprint,"admin_interfaces": self._normalize_web_fingerprint,+ # Azure & Azure AD (MOD-066)+ "azure_findings": self._normalize_azure,+ "azure_ad_findings": self._normalize_azure_ad,+ "access_tokens": self._normalize_tokens,+ "access_token": self._normalize_tokens,+ "oauth_tokens": self._normalize_tokens,+ # GCP (MOD-068)+ "gcp_findings": self._normalize_gcp,+ # Linux Containers & K8s (MOD-040)+ "container_escape_vectors": self._normalize_container_vectors,+ "k8s_rbac_findings": self._normalize_k8s_rbac,+ # Identity Federation (MOD-051)+ "federation_trusts": self._normalize_federation_trusts,+ "golden_saml_paths": self._normalize_golden_saml,+ "pivot_paths": self._normalize_pivot_paths,+ # SNMP & Network Discovery (MOD-061 / Recon)+ "snmp_findings": self._normalize_snmp_findings,+ "system_info": self._normalize_system_info,+ "password_policy": self._normalize_password_policy,+ # Windows Scheduled Tasks & Registry (MOD-070)+ "scheduled_tasks": self._normalize_scheduled_tasks,+ "credential_hints": self._normalize_registry_credentials,+ # Secrets Scanner & SMB Shares (MOD-074)+ "credential_list": self._normalize_secrets_scan,+ "discovered_secrets": self._normalize_secrets_scan,+ "sensitive_data_found": self._normalize_secrets_scan,+ "file_share_list": self._normalize_smb_shares,+ "sensitive_file_paths": self._normalize_smb_shares,}handler = handlers.get(capability)if handler:@@ -760,30 +796,77 @@ class ArtifactNormalizer:def _normalize_host_vuln(self, raw: dict, store: ArtifactStore) -> int:host = raw.get("target") or raw.get("host", "")- if host and host != "localhost":- artifact = HostArtifact(- ip_address = host,- hostname = host,- )- store.add(artifact)+ if not host or host == "localhost":+ return 0+ existing = None+ for h in store.hosts():+ if h.ip_address == host or h.hostname == host:+ existing = h+ break+ vulns = raw.get("privesc_vectors", [])+ parsed_vulns = []+ if isinstance(vulns, list):+ for v in vulns:+ if isinstance(v, dict):+ parsed_vulns.append(v)+ elif hasattr(v, "evidence") and isinstance(v.evidence, dict):+ parsed_vulns.append(v.evidence)+ elif isinstance(v, str) and v:+ parsed_vulns.append({"vector": v})+ if existing:+ if parsed_vulns:+ existing.vulns.extend(parsed_vulns)return 1- return 0+ artifact = HostArtifact(+ ip_address = host,+ hostname = host,+ vulns = parsed_vulns,+ )+ store.add(artifact)+ return 1# ── P0 Handlers ────────────────────────────────────────────────────────────def _normalize_cleartext_credentials(self, raw: dict, store: ArtifactStore) -> int:creds = raw.get("cleartext_credentials") or raw.get("credentials", [])+ if isinstance(creds, (dict, str)):+ creds = [creds]+ elif not isinstance(creds, list):+ return 0count = 0+ default_target = raw.get("target") or raw.get("host", "")for c in creds:- if not isinstance(c, dict):+ if hasattr(c, "evidence") and isinstance(c.evidence, dict):+ ev = c.evidence+ username = ev.get("user") or ev.get("username", "")+ secret = ev.get("password") or ev.get("secret", "")+ domain = ev.get("domain", "")+ source_host = ev.get("host") or default_target+ privilege = "unknown"+ elif isinstance(c, dict):+ username = c.get("username") or c.get("user") or c.get("account", "")+ domain = c.get("domain", "")+ secret = c.get("password") or c.get("secret") or c.get("plaintext", "")+ source_host = c.get("host") or c.get("target") or c.get("computer") or default_target+ privilege = c.get("privilege", "unknown")+ elif isinstance(c, str):+ username = "unknown"+ domain = ""+ secret = c+ source_host = default_target+ privilege = "unknown"+ else:+ logger.warning("cleartext_credentials_invalid_entry", type=type(c).__name__)+ continue+ if not username and not secret:continueartifact = CredentialArtifact(- username = c.get("username", ""),- domain = c.get("domain", ""),+ username = username,+ domain = domain,cred_type = "cleartext",- secret = c.get("password") or c.get("secret") or c.get("plaintext", ""),- source_host = c.get("host") or c.get("target") or c.get("computer", ""),- privilege = c.get("privilege", "unknown"),+ secret = secret,+ source_host = source_host,+ privilege = privilege,)store.add(artifact)count += 1@@ -1236,11 +1319,25 @@ class ArtifactNormalizer:return 0service_versions = raw.get("service_versions", {})ports: list[int] = []+ services: list[str] = []if isinstance(service_versions, dict):- for p_str in service_versions:+ for p_str, info in service_versions.items():if str(p_str).isdigit():ports.append(int(p_str))+ if isinstance(info, dict) and info.get("service"):+ services.append(f"{p_str}:{info['service']}")+ elif isinstance(info, str) and info:+ services.append(f"{p_str}:{info}")ports = sorted(list(set(ports)))+ vuln_svcs = raw.get("vulnerable_services", [])+ parsed_vulns = []+ if isinstance(vuln_svcs, list):+ for vs in vuln_svcs:+ if isinstance(vs, dict):+ parsed_vulns.append(vs)+ elif hasattr(vs, "evidence") and isinstance(vs.evidence, dict):+ parsed_vulns.append(vs.evidence)+existing = Nonefor h in store.hosts():if h.ip_address == target or h.hostname == target:@@ -1249,6 +1346,10 @@ class ArtifactNormalizer:if existing:if ports:existing.open_ports = sorted(list(set(existing.open_ports + ports)))+ if services:+ existing.services = sorted(list(set(existing.services + services)))+ if parsed_vulns:+ existing.vulns.extend(parsed_vulns)if raw.get("os") and not existing.os:existing.os = raw.get("os", "")return 1@@ -1257,6 +1358,8 @@ class ArtifactNormalizer:hostname=target,os=raw.get("os", "") or raw.get("detected_os", ""),open_ports=ports,+ services=services,+ vulns=parsed_vulns,source_module=capability,)store.add(artifact)@@ -1271,6 +1374,7 @@ class ArtifactNormalizer:return 0web_fp = raw.get("web_fingerprint", {})server = web_fp.get("server", "") if isinstance(web_fp, dict) else ""+ admin_interfaces = raw.get("admin_interfaces", [])existing = Nonefor h in store.hosts():if h.ip_address == target or h.hostname == target:@@ -1279,14 +1383,777 @@ class ArtifactNormalizer:if existing:if server and not existing.os:existing.os = server+ if admin_interfaces:+ existing.metadata["admin_interfaces"] = admin_interfacesreturn 1artifact = HostArtifact(ip_address=target,hostname=target,os=server,+ metadata={"admin_interfaces": admin_interfaces} if admin_interfaces else {},source_module=capability,)store.add(artifact)return 1+ def _normalize_azure(+ self, raw: dict, store: ArtifactStore, capability: str = "azure_findings"+ ) -> int:+ """Normalize Azure recon and attack output into CloudResourceArtifact, PermissionArtifact, UserArtifact."""+ count = 0+ subscription_id = raw.get("subscription_id", "")+ # 1. Storage+ storage = raw.get("storage", {})+ if isinstance(storage, dict):+ for c in storage.get("public_containers", []):+ if isinstance(c, dict):+ sa = c.get("storage_account", "")+ c_name = c.get("container", "")+ rg = c.get("resource_group", "")+ artifact = CloudResourceArtifact(+ provider="azure",+ resource_type="storage_container",+ resource_id=f"{sa}/{c_name}",+ region=rg,+ is_public=True,+ source_module=capability,+ )+ store.add(artifact)+ count += 1+ # 2. RBAC+ rbac = raw.get("rbac", {})+ if isinstance(rbac, dict):+ for o in rbac.get("owner_assignments", []):+ if isinstance(o, dict):+ p_id = o.get("principal_id", "")+ scope = o.get("scope", "") or f"/subscriptions/{subscription_id}"+ if p_id:+ artifact = PermissionArtifact(+ principal=p_id,+ target=scope,+ right="Owner",+ domain=subscription_id,+ is_dangerous=True,+ source_module=capability,+ )+ store.add(artifact)+ count += 1+ for c in rbac.get("contributor_assignments", []):+ if isinstance(c, dict):+ p_id = c.get("principal_id", "")+ scope = c.get("scope", "") or f"/subscriptions/{subscription_id}"+ if p_id:+ artifact = PermissionArtifact(+ principal=p_id,+ target=scope,+ right="Contributor",+ domain=subscription_id,+ is_dangerous=True,+ source_module=capability,+ )+ store.add(artifact)+ count += 1+ for admin in rbac.get("classic_admins", []):+ if isinstance(admin, str) and admin:+ artifact = PermissionArtifact(+ principal=admin,+ target=f"/subscriptions/{subscription_id}",+ right="ClassicCoAdministrator",+ domain=subscription_id,+ is_dangerous=True,+ source_module=capability,+ )+ store.add(artifact)+ count += 1+ # 3. NSG+ nsg = raw.get("nsg", {})+ if isinstance(nsg, dict):+ for rule in nsg.get("open_rules", []):+ if isinstance(rule, dict):+ nsg_name = rule.get("nsg", "")+ r_name = rule.get("rule_name", "")+ rg = rule.get("resource_group", "")+ artifact = CloudResourceArtifact(+ provider="azure",+ resource_type="network_security_group",+ resource_id=f"{nsg_name}/{r_name}" if r_name else nsg_name,+ region=rg,+ is_public=True,+ source_module=capability,+ )+ store.add(artifact)+ count += 1+ # 4. AAD users+ aad = raw.get("aad_users", {})+ if isinstance(aad, dict):+ for u in aad.get("admin_users", []):+ if isinstance(u, str) and u:+ artifact = UserArtifact(+ username=u,+ domain="azure_ad",+ is_admin=True,+ source_module=capability,+ )+ store.add(artifact)+ count += 1+ return count++ def _normalize_azure_ad(+ self, raw: dict, store: ArtifactStore, capability: str = "azure_ad_findings"+ ) -> int:+ """Normalize Azure AD findings into UserArtifact and PermissionArtifact."""+ count = 0+ tenant_id = raw.get("tenant_id", "azure_ad")+ # 1. Privileged users+ for u in raw.get("privileged_users", []):+ if isinstance(u, str) and u:+ artifact = UserArtifact(+ username=u,+ domain=tenant_id,+ is_admin=True,+ source_module=capability,+ )+ store.add(artifact)+ count += 1+ # 2. Guests+ for g in raw.get("guests", []):+ g_name = g.get("userPrincipalName", "") if isinstance(g, dict) else (g if isinstance(g, str) else "")+ if g_name:+ artifact = UserArtifact(+ username=g_name,+ domain=tenant_id,+ is_admin=False,+ source_module=capability,+ )+ store.add(artifact)+ count += 1+ # 3. High privilege service principals+ for sp in raw.get("high_priv_sps", []):+ if isinstance(sp, str) and sp:+ artifact = PermissionArtifact(+ principal=sp,+ target=tenant_id,+ right="PrivilegedAppRole",+ domain=tenant_id,+ is_dangerous=True,+ source_module=capability,+ )+ store.add(artifact)+ count += 1+ # 4. Service principals list+ for sp in raw.get("service_principals", []):+ if isinstance(sp, dict) and sp.get("privileged"):+ sp_name = sp.get("displayName") or sp.get("appId", "")+ if sp_name:+ artifact = PermissionArtifact(+ principal=sp_name,+ target=tenant_id,+ right="PrivilegedServicePrincipal",+ domain=tenant_id,+ is_dangerous=True,+ source_module=capability,+ )+ store.add(artifact)+ count += 1+ return count++ def _normalize_tokens(+ self, raw: dict, store: ArtifactStore, capability: str = "access_tokens"+ ) -> int:+ """Normalize access tokens and OAuth tokens into CredentialArtifact(cred_type='oauth_token')."""+ count = 0+ tokens = raw.get("access_tokens") or raw.get("oauth_tokens") or raw.get("access_token") or raw.get("tokens")+ if not tokens:+ return 0+ if isinstance(tokens, (str, dict)):+ tokens = [tokens]+ elif not isinstance(tokens, list):+ return 0+ tenant = raw.get("tenant_id") or raw.get("domain") or "cloud"+ for t in tokens:+ if isinstance(t, str):+ t_str = t.strip()+ if not t_str:+ continue+ artifact = CredentialArtifact(+ username="oauth_bearer",+ domain=tenant,+ cred_type="oauth_token",+ secret=t_str,+ source_host=tenant,+ privilege="token",+ source_module=capability,+ )+ store.add_credential(artifact)+ count += 1+ elif isinstance(t, dict):+ tok_val = t.get("access_token") or t.get("token") or t.get("secret", "")+ if not tok_val:+ continue+ username = t.get("user") or t.get("username") or t.get("client_id") or "oauth_bearer"+ scope = t.get("scope") or t.get("resource", "")+ artifact = CredentialArtifact(+ username=username,+ domain=tenant,+ cred_type="oauth_token",+ secret=tok_val,+ source_host=tenant,+ privilege="token",+ note=f"scope={scope}" if scope else "",+ source_module=capability,+ )+ store.add_credential(artifact)+ count += 1+ else:+ logger.warning("access_token_invalid_entry", type=type(t).__name__)+ return count++ def _normalize_gcp(+ self, raw: dict, store: ArtifactStore, capability: str = "gcp_findings"+ ) -> int:+ """Normalize GCP reconnaissance findings into CloudResourceArtifact and PermissionArtifact."""+ count = 0+ project_id = raw.get("project_id", "gcp")+ # 1. GCS public buckets+ gcs = raw.get("gcs", {})+ if isinstance(gcs, dict):+ for b in gcs.get("public_buckets", []):+ if isinstance(b, dict):+ b_name = b.get("bucket", "")+ if b_name:+ artifact = CloudResourceArtifact(+ provider="gcp",+ resource_type="gcs_bucket",+ resource_id=b_name,+ region=project_id,+ is_public=True,+ source_module=capability,+ )+ store.add(artifact)+ count += 1+ # 2. IAM bindings+ iam = raw.get("iam", {})+ if isinstance(iam, dict):+ for pb in iam.get("public_bindings", []):+ if isinstance(pb, str) and " → " in pb:+ member, role = pb.split(" → ", 1)+ artifact = PermissionArtifact(+ principal=member.strip(),+ target=project_id,+ right=role.strip(),+ domain="gcp",+ is_dangerous=True,+ source_module=capability,+ )+ store.add(artifact)+ count += 1+ for ob in iam.get("owner_bindings", []):+ if isinstance(ob, str) and ob:+ artifact = PermissionArtifact(+ principal=ob,+ target=project_id,+ right="roles/owner",+ domain="gcp",+ is_dangerous=True,+ source_module=capability,+ )+ store.add(artifact)+ count += 1+ # 3. Service account keys+ sa = raw.get("service_accounts", {})+ if isinstance(sa, dict):+ for mk in sa.get("many_keys", []):+ if isinstance(mk, dict):+ email = mk.get("service_account", "")+ key_count = mk.get("active_key_count", 0)+ if email:+ artifact = CredentialArtifact(+ username=email,+ domain="gcp",+ cred_type="service_account_key",+ source_host=project_id,+ privilege="service_account",+ note=f"active_keys={key_count}",+ source_module=capability,+ )+ store.add_credential(artifact)+ count += 1+ return count++ def _normalize_container_vectors(+ self, raw: dict, store: ArtifactStore, capability: str = "container_escape_vectors"+ ) -> int:+ """Normalize container escape vectors into HostArtifact vulns and metadata."""+ target = raw.get("target") or raw.get("host") or "container-host"+ existing = None+ for h in store.hosts():+ if h.ip_address == target or h.hostname == target:+ existing = h+ break+ host = existing or HostArtifact(+ ip_address=target,+ hostname=target,+ source_module=capability,+ )+ count = 0+ # Check docker socket+ ds = raw.get("docker_socket", {})+ if isinstance(ds, dict) and ds.get("writable"):+ vuln_entry = {"type": "container_escape", "vector": "docker_socket_writable", "severity": "CRITICAL"}+ host.vulns.append(vuln_entry)+ host.metadata.setdefault("container_escape_vectors", []).append(vuln_entry)+ count += 1+ # Check privileged+ priv = raw.get("privileged", {})+ if isinstance(priv, dict) and priv.get("privileged"):+ vuln_entry = {"type": "container_escape", "vector": "privileged_container", "severity": "CRITICAL"}+ host.vulns.append(vuln_entry)+ host.metadata.setdefault("container_escape_vectors", []).append(vuln_entry)+ count += 1+ # Check host mounts+ hm = raw.get("host_mounts", {})+ if isinstance(hm, dict) and hm.get("high"):+ vuln_entry = {"type": "container_escape", "vector": "host_mounts_high_risk", "mounts": hm["high"], "severity": "CRITICAL"}+ host.vulns.append(vuln_entry)+ host.metadata.setdefault("container_escape_vectors", []).append(vuln_entry)+ count += 1+ # If raw["container_escape_vectors"] has items+ vectors = raw.get("container_escape_vectors", [])+ if isinstance(vectors, list):+ for v in vectors:+ if isinstance(v, dict):+ host.vulns.append(v)+ count += 1+ elif hasattr(v, "evidence") and isinstance(v.evidence, dict):+ host.vulns.append(v.evidence)+ count += 1+ if count and not existing:+ store.add(host)+ return max(count, 1 if count else 0)++ def _normalize_k8s_rbac(+ self, raw: dict, store: ArtifactStore, capability: str = "k8s_rbac_findings"+ ) -> int:+ """Normalize Kubernetes RBAC findings into PermissionArtifact."""+ findings = raw.get("k8s_rbac_findings") or raw.get("k8s_findings", [])+ if not isinstance(findings, list):+ return 0+ count = 0+ for f in findings:+ if not isinstance(f, dict):+ if hasattr(f, "evidence") and isinstance(f.evidence, dict):+ f = f.evidence+ else:+ continue+ principal = f.get("service_account") or f.get("principal") or "system:serviceaccount"+ target = f.get("resource") or f.get("target") or "k8s_cluster"+ verb = f.get("verb") or f.get("right") or "cluster-admin"+ artifact = PermissionArtifact(+ principal=principal,+ target=target,+ right=verb,+ domain="k8s",+ is_dangerous=True,+ source_module=capability,+ )+ store.add(artifact)+ count += 1+ return count++ def _normalize_federation_trusts(+ self, raw: dict, store: ArtifactStore, capability: str = "federation_trusts"+ ) -> int:+ """Normalize identity federation trusts into CloudResourceArtifact."""+ trusts = raw.get("federation_trusts", [])+ if isinstance(trusts, (str, dict)):+ trusts = [trusts]+ elif not isinstance(trusts, list):+ return 0+ count = 0+ for t in trusts:+ if isinstance(t, str):+ t_str = t.strip()+ if not t_str:+ continue+ if "Azure" in t_str:+ provider = "azure"+ res_type = "federation_trust"+ res_id = t_str.split(":", 1)[1].strip() if ":" in t_str else t_str+ elif "AWS" in t_str:+ provider = "aws"+ res_type = "saml_provider"+ res_id = t_str.split(":", 1)[1].strip() if ":" in t_str else t_str+ elif "ADFS" in t_str:+ provider = "adfs"+ res_type = "relying_party"+ res_id = t_str.split(":", 1)[1].strip() if ":" in t_str else t_str+ else:+ provider = "cloud"+ res_type = "federation_trust"+ res_id = t_str+ artifact = CloudResourceArtifact(+ provider=provider,+ resource_type=res_type,+ resource_id=res_id,+ is_public=False,+ source_module=capability,+ )+ store.add(artifact)+ count += 1+ elif isinstance(t, dict):+ artifact = CloudResourceArtifact(+ provider=t.get("provider", "cloud"),+ resource_type=t.get("resource_type", "federation_trust"),+ resource_id=t.get("resource_id") or t.get("name", "trust"),+ is_public=t.get("is_public", False),+ source_module=capability,+ )+ store.add(artifact)+ count += 1+ else:+ logger.warning("federation_trust_invalid_entry", type=type(t).__name__)+ return count++ def _normalize_golden_saml(+ self, raw: dict, store: ArtifactStore, capability: str = "golden_saml_paths"+ ) -> int:+ """Normalize Golden SAML paths into PermissionArtifact."""+ paths = raw.get("golden_saml_paths", [])+ if isinstance(paths, dict):+ paths = [paths]+ elif not isinstance(paths, list):+ return 0+ count = 0+ for p in paths:+ if not isinstance(p, dict):+ continue+ rps = p.get("relying_parties", [])+ for rp in (rps if isinstance(rps, list) else [str(rps)]):+ if rp:+ artifact = PermissionArtifact(+ principal="adfs_token_signing_key",+ target=rp,+ right="GoldenSAML_AssertionForgery",+ domain="federation",+ is_dangerous=True,+ source_module=capability,+ )+ store.add(artifact)+ count += 1+ if not rps:+ artifact = PermissionArtifact(+ principal="adfs_token_signing_key",+ target=p.get("attack", "Golden SAML"),+ right="GoldenSAML",+ domain="federation",+ is_dangerous=True,+ source_module=capability,+ )+ store.add(artifact)+ count += 1+ return count++ def _normalize_pivot_paths(+ self, raw: dict, store: ArtifactStore, capability: str = "pivot_paths"+ ) -> int:+ """Normalize cross-cloud pivot paths into PermissionArtifact."""+ paths = raw.get("pivot_paths", [])+ if not isinstance(paths, list):+ return 0+ count = 0+ for p in paths:+ if not isinstance(p, dict):+ continue+ path_name = p.get("path") or p.get("name", "cross_cloud_pivot")+ tech = p.get("technique", "T1606.002")+ artifact = PermissionArtifact(+ principal="federated_identity",+ target=path_name,+ right=tech,+ domain="multi_cloud",+ is_dangerous=True,+ source_module=capability,+ )+ store.add(artifact)+ count += 1+ return count++ def _normalize_snmp_findings(+ self, raw: dict, store: ArtifactStore, capability: str = "snmp_findings"+ ) -> int:+ """Normalize SNMP enumeration findings into CredentialArtifact and HostArtifact."""+ count = 0+ target = raw.get("target") or raw.get("host", "")+ # Community strings+ valid_comms = raw.get("valid_communities", [])+ if isinstance(valid_comms, list):+ for vc in valid_comms:+ comm = vc.get("community", "") if isinstance(vc, dict) else (vc if isinstance(vc, str) else "")+ if comm:+ artifact = CredentialArtifact(+ username="snmp",+ secret=comm,+ cred_type="snmp_community",+ source_host=target,+ privilege="local_admin" if comm.lower() in ("private", "write") else "user",+ source_module=capability,+ )+ store.add_credential(artifact)+ count += 1+ # System info+ sys_info = raw.get("system_info", {})+ if isinstance(sys_info, dict) and target:+ sys_descr = sys_info.get("sysDescr", "")+ sys_name = sys_info.get("sysName", "")+ existing = None+ for h in store.hosts():+ if h.ip_address == target or h.hostname == target:+ existing = h+ break+ if existing:+ if sys_descr and not existing.os:+ existing.os = sys_descr+ if sys_name and not existing.hostname:+ existing.hostname = sys_name+ existing.metadata["snmp_system_info"] = sys_info+ else:+ h = HostArtifact(+ ip_address=target,+ hostname=sys_name or target,+ os=sys_descr,+ metadata={"snmp_system_info": sys_info},+ source_module=capability,+ )+ store.add(h)+ count += 1+ return count++ def _normalize_system_info(+ self, raw: dict, store: ArtifactStore, capability: str = "system_info"+ ) -> int:+ """Normalize generic system_info dict into HostArtifact."""+ target = raw.get("target") or raw.get("host", "")+ sys_info = raw.get("system_info", {})+ if not target and isinstance(sys_info, dict):+ target = sys_info.get("sysName") or sys_info.get("hostname", "")+ if not target:+ return 0+ existing = None+ for h in store.hosts():+ if h.ip_address == target or h.hostname == target:+ existing = h+ break+ if existing:+ if isinstance(sys_info, dict):+ existing.metadata["system_info"] = sys_info+ if sys_info.get("sysDescr") and not existing.os:+ existing.os = sys_info["sysDescr"]+ return 1+ h = HostArtifact(+ ip_address=target,+ hostname=target,+ os=sys_info.get("sysDescr", "") if isinstance(sys_info, dict) else "",+ metadata={"system_info": sys_info} if isinstance(sys_info, dict) else {},+ source_module=capability,+ )+ store.add(h)+ return 1++ def _normalize_password_policy(+ self, raw: dict, store: ArtifactStore, capability: str = "password_policy"+ ) -> int:+ """Normalize AD/cloud password policy into DomainArtifact metadata."""+ policy = raw.get("password_policy", {})+ if not isinstance(policy, dict) or not policy:+ return 0+ domain = raw.get("domain", "") or raw.get("dc", "") or "default"+ existing_domain = None+ for d in store.get(ArtifactType.DOMAIN):+ if isinstance(d, DomainArtifact) and d.domain_name.lower() == domain.lower():+ existing_domain = d+ break+ if existing_domain:+ existing_domain.metadata["password_policy"] = policy+ return 1+ d_artifact = DomainArtifact(+ domain_name=domain,+ metadata={"password_policy": policy},+ source_module=capability,+ )+ store.add(d_artifact)+ return 1++ def _normalize_scheduled_tasks(+ self, raw: dict, store: ArtifactStore, capability: str = "scheduled_tasks"+ ) -> int:+ """Normalize scheduled tasks into PermissionArtifact and HostArtifact metadata."""+ count = 0+ target = raw.get("target") or raw.get("host", "")+ seen_tasks: set[str] = set()+ for key in ("writable_path_tasks", "high_priv_tasks", "all_tasks"):+ tasks = raw.get(key, [])+ if isinstance(tasks, list):+ for t in tasks:+ if isinstance(t, dict):+ task_name = t.get("task") or t.get("name", "")+ if task_name and task_name not in seen_tasks:+ seen_tasks.add(task_name)+ run_as = t.get("run_as", "SYSTEM")+ artifact = PermissionArtifact(+ principal=run_as,+ target=task_name,+ right="ExecuteScheduledTask",+ domain=target,+ is_dangerous=bool(t.get("writable_indicator") or key == "writable_path_tasks"),+ source_module=capability,+ )+ store.add(artifact)+ count += 1+ if target:+ for h in store.hosts():+ if h.ip_address == target or h.hostname == target:+ h.metadata["scheduled_tasks_count"] = raw.get("total_tasks", count)+ break+ return count++ def _normalize_registry_credentials(+ self, raw: dict, store: ArtifactStore, capability: str = "credential_hints"+ ) -> int:+ """Normalize registry credential hits (AutoLogon, VNC, PuTTY) into CredentialArtifact."""+ count = 0+ target = raw.get("target") or raw.get("host", "")+ # 1. Credential hits (AutoLogon, VNC, etc.)+ hits = raw.get("credential_hits", [])+ if isinstance(hits, list):+ for h in hits:+ if not isinstance(h, dict):+ continue+ vals = h.get("values", {})+ if not isinstance(vals, dict):+ continue+ path = h.get("path", "")+ if "Winlogon" in path:+ user = vals.get("DefaultUserName", "")+ pwd = vals.get("DefaultPassword", "")+ dom = vals.get("DefaultDomainName", "") or target+ if pwd and isinstance(pwd, str):+ artifact = CredentialArtifact(+ username=user.strip("\x00"),+ domain=dom.strip("\x00"),+ cred_type="cleartext",+ secret=pwd.strip("\x00"),+ source_host=target,+ privilege="local_admin",+ note="registry_autologon",+ source_module=capability,+ )+ store.add_credential(artifact)+ count += 1+ elif "VNC" in path:+ pwd = vals.get("Password", "")+ if pwd:+ artifact = CredentialArtifact(+ username="vnc",+ domain=target,+ cred_type="vnc_password",+ secret=str(pwd),+ source_host=target,+ privilege="user",+ note=f"registry_{path}",+ source_module=capability,+ )+ store.add_credential(artifact)+ count += 1+ # 2. PuTTY sessions+ putty = raw.get("putty_sessions", [])+ if isinstance(putty, list):+ for s in putty:+ if isinstance(s, dict):+ u = s.get("user_name", "")+ h_name = s.get("host_name", "")+ s_name = s.get("session_name", "")+ if u or h_name:+ artifact = CredentialArtifact(+ username=u or "unknown",+ domain=h_name,+ cred_type="putty_session",+ source_host=target,+ note=f"session={s_name} remote={h_name}",+ source_module=capability,+ )+ store.add_credential(artifact)+ count += 1+ return count++ def _normalize_secrets_scan(+ self, raw: dict, store: ArtifactStore, capability: str = "discovered_secrets"+ ) -> int:+ """Normalize secrets scanner findings into CredentialArtifact."""+ count = 0+ target = raw.get("target") or raw.get("host", "")+ secrets = raw.get("discovered_secrets") or []+ if isinstance(secrets, list):+ for s in secrets:+ if not isinstance(s, dict):+ continue+ sec_val = s.get("secret_value") or s.get("extracted_secret") or ""+ pattern = s.get("pattern", "secret")+ fpath = s.get("file", "")+ line = s.get("line", 0)+ if sec_val:+ artifact = CredentialArtifact(+ username=pattern,+ secret=sec_val,+ cred_type=pattern,+ source_host=target,+ privilege="unknown",+ note=f"{fpath}:{line}",+ source_module=capability,+ )+ store.add_credential(artifact)+ count += 1+ cred_list = raw.get("credential_list", [])+ if target and (cred_list or raw.get("sensitive_data_found")):+ for h in store.hosts():+ if h.ip_address == target or h.hostname == target:+ if cred_list:+ h.metadata["secret_files"] = cred_list+ if raw.get("sensitive_data_found"):+ h.metadata["sensitive_data_found"] = True+ break+ return count++ def _normalize_smb_shares(+ self, raw: dict, store: ArtifactStore, capability: str = "file_share_list"+ ) -> int:+ """Normalize SMB shares and sensitive file paths into HostArtifact metadata."""+ target = raw.get("target") or raw.get("host", "")+ if not target:+ return 0+ share_list = raw.get("file_share_list", [])+ file_paths = raw.get("sensitive_file_paths", [])+ existing = None+ for h in store.hosts():+ if h.ip_address == target or h.hostname == target:+ existing = h+ break+ if existing:+ if share_list:+ existing.metadata["file_shares"] = share_list+ if file_paths:+ existing.metadata["sensitive_files"] = file_paths+ return 1+ h = HostArtifact(+ ip_address=target,+ hostname=target,+ metadata={+ "file_shares": share_list,+ "sensitive_files": file_paths,+ },+ source_module=capability,+ )+ store.add(h)+ return 1++diff --git a/tests/unit/test_artifact_normalizer_pipeline.py b/tests/unit/test_artifact_normalizer_pipeline.pyindex 1863d8b..3146251 100644--- a/tests/unit/test_artifact_normalizer_pipeline.py+++ b/tests/unit/test_artifact_normalizer_pipeline.py@@ -13,9 +13,12 @@ from ares.normalize.artifacts import (ArtifactNormalizer,ArtifactStore,ArtifactType,+ CloudResourceArtifact,CredentialArtifact,+ DomainArtifact,HashArtifact,HostArtifact,+ PermissionArtifact,UserArtifact,)@@ -935,5 +938,367 @@ def test_service_versions_and_web_fingerprint_pipeline_mod_062(assert store.hosts()[0].os == "Ubuntu Linux"+def test_azure_findings_pipeline_mod_066(normalizer: ArtifactNormalizer, store: ArtifactStore):+ """MOD-066: Azure storage, RBAC, and NSG findings normalized to CloudResourceArtifact and PermissionArtifact."""+ raw = {+ "subscription_id": "sub-1234-abcd",+ "storage": {+ "public_containers": [+ {+ "storage_account": "corpbackup",+ "container": "db-dumps",+ "access_level": "container",+ "resource_group": "rg-core",+ }+ ]+ },+ "rbac": {+ "owner_assignments": [+ {"principal_id": "user-guid-001", "role_definition": "8e3af657", "scope": "/subscriptions/sub-1234-abcd"}+ ],+ "contributor_assignments": [+ {"principal_id": "spn-guid-002", "role_definition": "b24988ac", "scope": "/subscriptions/sub-1234-abcd"}+ ],+ "classic_admins": ["[email protected]"],+ },+ "nsg": {+ "open_rules": [+ {"nsg": "nsg-prod", "rule_name": "Allow-SSH-Internet", "resource_group": "rg-core"}+ ]+ },+ "aad_users": {+ "admin_users": ["[email protected]"]+ }+ }+ added = normalizer.normalize("cloud.azure", ["azure_findings"], raw, store)+ assert added >= 5++ # Cloud resources: storage container + NSG+ cloud_res = [a for a in store.get(ArtifactType.CLOUD_RESOURCE)]+ res_ids = {c.resource_id for c in cloud_res}+ assert "corpbackup/db-dumps" in res_ids+ assert "nsg-prod/Allow-SSH-Internet" in res_ids++ # Permissions: owner, contributor, classic admin+ perms = store.permissions()+ principals = {p.principal for p in perms}+ assert "user-guid-001" in principals+ assert "spn-guid-002" in principals+ assert "[email protected]" in principals++ # Users+ users = store.users()+ assert any(u.username == "[email protected]" for u in users)+++def test_azure_ad_findings_and_access_tokens_pipeline_mod_066(+ normalizer: ArtifactNormalizer, store: ArtifactStore+):+ """MOD-066: Azure AD users, SPNs, and access tokens normalized into artifacts."""+ raw = {+ "tenant_id": "tenant-azure-ad-01",+ "privileged_users": ["[email protected]"],+ "guests": [{"userPrincipalName": "[email protected]"}],+ "high_priv_sps": ["CI/CD Pipeline App"],+ "access_tokens": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.dummy_access_token_payload",+ }+ added = normalizer.normalize("cloud.azure_ad", ["azure_ad_findings", "access_tokens"], raw, store)+ assert added >= 4++ users = {u.username: u for u in store.users()}+ assert "[email protected]" in users+ assert users["[email protected]"].is_admin is True+ assert "[email protected]" in users+ assert users["[email protected]"].is_admin is False++ perms = store.permissions()+ assert any(p.principal == "CI/CD Pipeline App" for p in perms)++ creds = store.credentials()+ oauth_creds = [c for c in creds if c.cred_type == "oauth_token"]+ assert len(oauth_creds) == 1+ assert "dummy_access_token_payload" in oauth_creds[0].secret+++def test_gcp_findings_pipeline_mod_068(normalizer: ArtifactNormalizer, store: ArtifactStore):+ """MOD-068: GCP public buckets, IAM bindings, and SA keys normalized into artifacts."""+ raw = {+ "project_id": "ares-gcp-sec-lab",+ "gcs": {+ "public_buckets": [+ {"bucket": "ares-public-assets", "member": "allUsers", "role": "roles/storage.objectViewer"}+ ]+ },+ "iam": {+ "public_bindings": ["allUsers → roles/viewer"],+ "owner_bindings": ["[email protected]"],+ },+ "service_accounts": {+ "many_keys": [+ {"service_account": "[email protected]", "active_key_count": 3}+ ]+ }+ }+ added = normalizer.normalize("cloud.gcp", ["gcp_findings"], raw, store)+ assert added >= 4++ cloud_res = [c for c in store.get(ArtifactType.CLOUD_RESOURCE) if c.provider == "gcp"]+ assert len(cloud_res) == 1+ assert cloud_res[0].resource_id == "ares-public-assets"+ assert cloud_res[0].is_public is True++ perms = store.permissions()+ principals = {p.principal for p in perms}+ assert "allUsers" in principals+ assert "[email protected]" in principals++ creds = store.credentials()+ sa_creds = [c for c in creds if c.cred_type == "service_account_key"]+ assert len(sa_creds) == 1+ assert sa_creds[0].username == "[email protected]"+++def test_container_escape_and_k8s_rbac_pipeline_mod_040(+ normalizer: ArtifactNormalizer, store: ArtifactStore+):+ """MOD-040: Container escape vectors and K8s RBAC findings normalized into HostArtifact and PermissionArtifact."""+ raw = {+ "target": "k8s-worker-01",+ "docker_socket": {"exists": True, "writable": True},+ "privileged": {"privileged": True, "CapEff": "0000003fffffffff"},+ "host_mounts": {"high": ["/etc", "/var/lib/docker"]},+ "k8s_rbac_findings": [+ {+ "service_account": "system:serviceaccount:default:tiller",+ "resource": "cluster",+ "verb": "cluster-admin",+ }+ ],+ }+ added = normalizer.normalize("linux.container", ["container_escape_vectors", "k8s_rbac_findings"], raw, store)+ assert added >= 2++ hosts = store.hosts()+ assert len(hosts) == 1+ assert hosts[0].ip_address == "k8s-worker-01"+ assert len(hosts[0].vulns) >= 3+ assert any(v.get("vector") == "docker_socket_writable" for v in hosts[0].vulns)++ perms = store.permissions()+ assert len(perms) == 1+ assert perms[0].principal == "system:serviceaccount:default:tiller"+ assert perms[0].right == "cluster-admin"+++def test_identity_federation_pipeline_mod_051(+ normalizer: ArtifactNormalizer, store: ArtifactStore+):+ """MOD-051: Federation trusts, Golden SAML paths, and pivot paths normalized into artifacts."""+ raw = {+ "tenant_id": "tenant-fed-01",+ "federation_trusts": [+ "Azure federated domain: corp.com",+ "AWS SAML provider: arn:aws:iam::123456789012:saml-provider/ADFS",+ "ADFS relying party: urn:federation:MicrosoftOnline",+ ],+ "golden_saml_paths": [+ {+ "attack": "Golden SAML",+ "technique": "T1606.002",+ "relying_parties": ["urn:federation:MicrosoftOnline", "urn:amazon:webservices"],+ }+ ],+ "pivot_paths": [+ {+ "path": "Azure AD → AWS via SAML Federation",+ "technique": "T1606.002",+ "viable": True,+ }+ ],+ "oauth_tokens": [+ {"access_token": "token_forged_saml_b64", "user": "[email protected]"}+ ],+ }+ added = normalizer.normalize(+ "cloud.identity_federation_abuse",+ ["federation_trusts", "golden_saml_paths", "pivot_paths", "oauth_tokens"],+ raw,+ store,+ )+ assert added >= 7++ cloud_res = [c for c in store.get(ArtifactType.CLOUD_RESOURCE)]+ assert any(c.provider == "azure" and c.resource_id == "corp.com" for c in cloud_res)+ assert any(c.provider == "aws" for c in cloud_res)++ perms = store.permissions()+ saml_perms = [p for p in perms if "GoldenSAML" in p.right]+ assert len(saml_perms) == 2++ creds = store.credentials()+ assert any(c.secret == "token_forged_saml_b64" for c in creds)+++def test_snmp_findings_and_system_info_pipeline(+ normalizer: ArtifactNormalizer, store: ArtifactStore+):+ """network.snmp_enum findings and system_info populate CredentialArtifact and HostArtifact."""+ raw = {+ "target": "192.168.1.100",+ "port": 161,+ "valid_communities": [+ {"community": "public", "system_info": {"sysDescr": "Linux Ubuntu 22.04"}},+ {"community": "private", "system_info": {"sysDescr": "Linux Ubuntu 22.04"}},+ ],+ "system_info": {+ "sysDescr": "Linux RouterOS 7.2",+ "sysName": "edge-gw01",+ "sysLocation": "Datacenter Rack 4",+ }+ }+ added = normalizer.normalize("network.snmp_enum", ["snmp_findings", "system_info"], raw, store)+ assert added >= 3++ creds = store.credentials()+ communities = {c.secret for c in creds if c.cred_type == "snmp_community"}+ assert "public" in communities+ assert "private" in communities++ hosts = store.hosts()+ assert len(hosts) == 1+ assert hosts[0].ip_address == "192.168.1.100"+ assert hosts[0].hostname == "edge-gw01"+ assert "snmp_system_info" in hosts[0].metadata+++def test_password_policy_pipeline_mod_056(+ normalizer: ArtifactNormalizer, store: ArtifactStore+):+ """ad.enum_users password_policy is stored in DomainArtifact metadata."""+ raw = {+ "domain": "corp.local",+ "password_policy": {+ "minPwdLength": 8,+ "pwdHistoryLength": 24,+ "lockoutThreshold": 0,+ }+ }+ added = normalizer.normalize("ad.enum_users", ["password_policy"], raw, store)+ assert added == 1++ domains = [d for d in store.get(ArtifactType.DOMAIN) if isinstance(d, DomainArtifact)]+ assert len(domains) == 1+ assert domains[0].domain_name == "corp.local"+ assert domains[0].metadata["password_policy"]["minPwdLength"] == 8+ assert domains[0].metadata["password_policy"]["lockoutThreshold"] == 0+++def test_scheduled_tasks_and_registry_credentials_pipeline_mod_070(+ normalizer: ArtifactNormalizer, store: ArtifactStore+):+ """MOD-070: Scheduled tasks and registry credential hits normalized into typed artifacts."""+ raw_tasks = {+ "target": "win-srv-01",+ "writable_path_tasks": [+ {+ "task": "DailyBackupJob",+ "run_as": "NT AUTHORITY\\SYSTEM",+ "action": "C:\\Temp\\backup.bat",+ "writable_indicator": "C:\\Temp\\",+ }+ ],+ "high_priv_tasks": [+ {"task": "MaintenanceTask", "run_as": "BUILTIN\\Administrators"}+ ],+ "total_tasks": 2,+ }+ added_tasks = normalizer.normalize("windows.scheduled_tasks_enum", ["scheduled_tasks"], raw_tasks, store)+ assert added_tasks >= 2++ perms = store.permissions()+ task_perms = [p for p in perms if p.right == "ExecuteScheduledTask"]+ assert len(task_perms) >= 2+ assert any(p.target == "DailyBackupJob" and p.is_dangerous is True for p in task_perms)++ raw_reg = {+ "target": "win-srv-01",+ "credential_hits": [+ {+ "path": "SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\Winlogon",+ "values": {+ "AutoAdminLogon": "1",+ "DefaultUserName": "kiosk_user",+ "DefaultPassword": "SuperSecretAutoPassword123!",+ "DefaultDomainName": "CORP",+ },+ },+ {+ "path": "SOFTWARE\\RealVNC\\vncserver",+ "values": {"Password": "VncPasswordSecret"},+ }+ ],+ "putty_sessions": [+ {"session_name": "prod-bastion", "host_name": "10.10.10.5", "user_name": "opsadmin"}+ ]+ }+ added_reg = normalizer.normalize("windows.registry_enum", ["credential_hints"], raw_reg, store)+ assert added_reg >= 3++ creds = store.credentials()+ assert any(c.username == "kiosk_user" and c.secret == "SuperSecretAutoPassword123!" for c in creds)+ assert any(c.username == "vnc" and c.secret == "VncPasswordSecret" for c in creds)+ assert any(c.username == "opsadmin" and c.cred_type == "putty_session" for c in creds)+++def test_secrets_scan_and_smb_shares_pipeline_mod_074(+ normalizer: ArtifactNormalizer, store: ArtifactStore+):+ """MOD-074: Discovered secrets and sensitive SMB share files normalized into artifacts."""+ raw_secrets = {+ "target": "appserver01",+ "discovered_secrets": [+ {+ "file": "C:\\inetpub\\wwwroot\\web.config",+ "line": 42,+ "pattern": "connection_string",+ "extracted_secret": "Server=db01;Password=SecretDBPass!",+ "secret_value": "SecretDBPass!",+ },+ {+ "file": "/home/deploy/.aws/credentials",+ "line": 3,+ "pattern": "aws_access_key",+ "extracted_secret": "AKIAIOSFODNN7EXAMPLE",+ "secret_value": "AKIAIOSFODNN7EXAMPLE",+ }+ ],+ "credential_list": ["C:\\inetpub\\wwwroot\\web.config", "/home/deploy/.aws/credentials"],+ "sensitive_data_found": True,+ }+ added_sec = normalizer.normalize("exfil.secrets_scan", ["discovered_secrets", "credential_list"], raw_secrets, store)+ assert added_sec >= 2++ creds = store.credentials()+ assert any(c.cred_type == "connection_string" and c.secret == "SecretDBPass!" for c in creds)+ assert any(c.cred_type == "aws_access_key" and c.secret == "AKIAIOSFODNN7EXAMPLE" for c in creds)++ raw_shares = {+ "target": "fileserver02",+ "file_share_list": ["backups", "finance", "C$"],+ "sensitive_file_paths": [+ "\\\\fileserver02\\backups\\sam.save",+ "\\\\fileserver02\\finance\\budget_2026.xlsx",+ ],+ "sensitive_data_found": True,+ }+ added_shares = normalizer.normalize("exfil.smb_shares", ["file_share_list", "sensitive_file_paths"], raw_shares, store)+ assert added_shares >= 1++ hosts = {h.ip_address: h for h in store.hosts()}+ assert "fileserver02" in hosts+ assert "backups" in hosts["fileserver02"].metadata["file_shares"]+ assert "\\\\fileserver02\\backups\\sam.save" in hosts["fileserver02"].metadata["sensitive_files"]++