πΈ Rybbit - open-source and privacy-friendly alternative to Google Analytics that is 10x more intuitive.
Remove obsolete memory bank files including activeContext.md, decisionLog.md, error-tracking-plan.md, productContext.md, progress.md, systemPatterns.md, and mockdata directory. These files are no longer needed as the project evolves and transitions to a more streamlined documentation approach.
Bill Yang6263a4b4731e
diff --git a/memory-bank/activeContext.md b/memory-bank/activeContext.mddeleted file mode 100644index 7d8cefa7c..000000000--- a/memory-bank/activeContext.md+++ /dev/null@@ -1,44 +0,0 @@-# Active Context--This file tracks the project's current status, including recent changes, current goals, and open questions.--## Current Focus--- Creating new integration guides for Squarespace, Wix, Hugo, Jekyll, Drupal, and Laravel.-- Answering user question regarding Laravel compatibility with client-side script.--## Recent Changes--2025-05-31 13:49:39 - Memory Bank system initialized for Rybbit Analytics project-2025-06-06 23:18:00 - User requested creation of new integration guides for top priority platforms.--## Open Questions/Issues--- What specific development tasks or improvements are currently prioritized? (Previously existing)-- Are there any known technical debt items or performance issues to address? (Previously existing)-- What new features or enhancements are planned for the platform? (Previously existing)-- Are there any deployment or infrastructure concerns that need attention? (Previously existing)--[2025-06-08 06:42:07] - Docker build diagnosis confirmed: Build context mismatch identified as root cause. COPY shared ./shared fails because shared/ directory not present in ./server context. Dockerfiles from commit 974e6cc expect monorepo root context but CI uses subdirectory contexts.--[2025-06-09 19:42:35] - **COMPLETED: GoalFormModal Form Reset Bug Fix**--- **Issue**: Form state wasn't resetting when saving goals in GoalFormModal component-- **Root Cause**: Form was only being reset in onOpenChange handler when manually closing dialog, not after successful submission-- **Diagnosis Process**: Added debug logging to identify that form.reset() was never called after successful goal creation/update-- **Solution**: Added form.reset() and setUseProperties(false) after successful submission in onSubmit function-- **Files Modified**: [`client/src/app/[site]/goals/components/GoalFormModal.tsx`](client/src/app/[site]/goals/components/GoalFormModal.tsx:162)-- **Impact**: Form now properly clears all fields and state after successful goal creation/editing--[2025-06-19 19:29:31] - **COMPLETED: API Documentation Language Examples Review and Fixes**--- **Task**: Review and fix potential issues in language examples in API documentation-- **Status**: β COMPLETED-- **Key Issues Addressed**:- 1. Screen pixel type inconsistencies across mobile platforms- 2. Deprecated iOS UIScreen.main API usage- 3. Poor Flutter app launch event tracking placement- 4. Inconsistent example data between platforms- 5. Poor API key handling in Swift example-- **Impact**: All mobile platform examples now use consistent logical pixels, modern APIs, and standardized example data-- **Files Modified**: [`docs/src/content/api.mdx`](docs/src/content/api.mdx:78)diff --git a/memory-bank/decisionLog.md b/memory-bank/decisionLog.mddeleted file mode 100644index 9778ed982..000000000--- a/memory-bank/decisionLog.md+++ /dev/null@@ -1,234 +0,0 @@-# Decision Log--This file records architectural and implementation decisions using a list format.--## Decision--2025-05-31 13:49:52 - Memory Bank Architecture Implementation--## Rationale--Implemented a comprehensive Memory Bank system to maintain project context across different modes and sessions. This decision was made to:--- Ensure continuity of project understanding across different development sessions-- Provide a centralized location for tracking architectural decisions and progress-- Enable better collaboration and knowledge transfer-- Support the complex, multi-component architecture of Rybbit Analytics--## Implementation Details--- Created five core Memory Bank files: productContext.md, activeContext.md, progress.md, decisionLog.md, and systemPatterns.md-- Established initial project context based on projectBrief.md-- Set up tracking mechanisms for ongoing development activities-- Prepared framework for documenting future architectural decisions and patterns--[2025-06-08 06:53:49] - Docker Build Context Fix: Changed all build contexts from subdirectory (./client, ./server) to monorepo root (.) in docker-compose.cloud.yml and GitHub Actions workflow. Updated Dockerfiles to use correct paths from monorepo root (e.g., COPY client/package.json instead of COPY package.json). This resolves the shared package access issue while maintaining compatibility between local and CI builds.--[2025-06-08 16:43:52] - Enhanced authedFetchWithError to use axios with query parameter support--## Rationale--User requested to replace all instances of authedFetch with authedFetchWithError and wanted to use axios specifically for its params support. This decision was made to:--- Provide clean object-based query parameter support instead of manual string template literals-- Maintain consistency with axios patterns for HTTP requests-- Prepare for eventual replacement of all authedFetch instances-- Leverage axios's built-in parameter serialization and error handling--## Implementation Details--- Added axios dependency to client package.json-- Modified authedFetchWithError to accept query parameters as objects-- Maintained overloaded function signature for backward compatibility-- Used axios withCredentials: true to maintain authentication behavior-- Function now supports both query params as objects and full axios config objects--[2025-06-08 16:46:58] - Simplified authedFetchWithError function signature--## Rationale--User feedback indicated they didn't like the overloading behavior and wanted the 2nd argument to always be params instead of config. This decision was made to:--- Provide a cleaner, more predictable API-- Remove complexity from function signature detection-- Make the second parameter always represent query parameters-- Keep axios configuration as the third parameter for advanced use cases--## Implementation Details--- Removed overloaded function signature detection logic-- Second parameter is now always `params?: Record<string, any>`-- Third parameter remains `config: AxiosRequestConfig = {}` for advanced axios options-- Simplified function body to always treat second argument as query parameters--[2025-06-08 19:13:30] - Added past 24-hour mode support to performance analytics hooks--## Rationale--User requested to add support for the past 24-hour mode to useGetPerformanceTimeSeries and useGetPerformanceOverview hooks, following the same pattern already implemented in useGetPerformanceByDimension. This decision was made to:--- Provide consistent API behavior across all performance analytics hooks-- Support real-time analytics for the last 24 hours using pastMinutes parameters-- Maintain backward compatibility with existing date-range based queries-- Ensure proper query key differentiation between different time modes--## Implementation Details--- Modified useGetPerformanceTimeSeries to detect `time.mode === "last-24-hours"`-- Modified useGetPerformanceOverview to detect `time.mode === "last-24-hours"`-- Added conditional query parameter logic using pastMinutesStart/pastMinutesEnd for 24-hour mode-- Updated query keys to include mode differentiation ("past-minutes" vs "date-range")-- Maintained existing date-based approach for all other time modes-- Used same pattern as useGetPerformanceByDimension for consistency--[2025-01-08 19:23:09] - **Code Deduplication: Query Parameter Logic Refactoring**--- **Decision**: Created a centralized `getQueryParams` utility function to eliminate duplicated conditional logic across analytics hooks-- **Problem**: Found 7+ instances of identical `isPast24HoursMode` conditional logic for query parameter construction-- **Solution**:- - Created `getQueryParams(time, pastMinutesStart?, pastMinutesEnd?)` utility in `client/src/api/utils.ts`- - Handles both past-24-hours mode (using pastMinutesStart/End) and regular date-based queries- - Supports optional custom past minutes parameters for complex cases-- **Files Refactored**:- - `useGetPerformanceTimeSeries.ts` - Reduced ~20 lines to 3 lines- - `useGetPerformanceOverview.ts` - Simplified conditional logic- - `useGetPerformanceByDimension.ts` - Removed duplicated logic- - `useSingleCol.ts` - Enhanced with custom past minutes support- - `usePaginatedSingleCol.ts` - Simplified query parameter construction- - `useInfiniteSingleCol.ts` - Removed conditional logic duplication- - `useGetPageTitles.ts` - Standardized query parameter handling-- **Benefits**: Improved maintainability, reduced code duplication, centralized time-based query logic-- **Impact**: Significant reduction in duplicated code across analytics layer--[2025-01-08 19:31:12] - **Hook Consolidation: useGetOverview Functions**--- **Decision**: Combined `useGetOverview` and `useGetOverviewPastMinutes` into a single, flexible hook-- **Problem**: Two separate hooks with duplicated logic and similar functionality-- **Solution**:- - Enhanced `useGetOverview` to accept optional `pastMinutesStart` and `pastMinutesEnd` parameters- - Maintained backward compatibility by keeping `useGetOverviewPastMinutes` as a wrapper function- - Integrated with the `getQueryParams` utility for consistent query parameter handling- - Implemented intelligent query key generation based on parameters used-- **Benefits**:- - Reduced code duplication between the two hooks- - Simplified API while maintaining backward compatibility- - Consistent with other analytics hooks using `getQueryParams` utility- - Single source of truth for overview data fetching logic-- **Impact**: Improved maintainability and consistency across overview data fetching--[2025-01-08 19:36:50] - **Bug Fix: TypeScript Error in useGetOverview**--- **Problem**: TypeScript error "Argument of type 'number' is not assignable to parameter of type 'Record<string, any>'" when calling `getQueryParams` with `pastMinutesStart` parameter-- **Root Cause**: Incorrect function call signature - `getQueryParams` expects parameters as `(time, additionalParams, options)` but was being called with individual numbers-- **Solution**: Fixed function call to use correct parameter structure: `getQueryParams(timeToUse, { filters }, { pastMinutesStart, pastMinutesEnd })`-- **Additional Work**: Updated all callsites of `useGetOverviewPastMinutes` to use the consolidated `useGetOverview` hook-- **Files Updated**:- - [`useGetOverview.ts`](client/src/api/analytics/useGetOverview.ts:1) - Fixed function call signature- - [`useGetOverviewWithInView.ts`](client/src/api/analytics/useGetOverviewWithInView.ts:1) - Updated import and comment- - [`SiteCard.tsx`](client/src/components/SiteCard.tsx:1) - Replaced `useGetOverviewPastMinutes` with `useGetOverview`- - [`Overview.tsx`](client/src/app/[site]/main/components/MainSection/Overview.tsx:1) - Replaced both instances of `useGetOverviewPastMinutes` with `useGetOverview`-- **Impact**: Eliminated TypeScript errors and completed hook consolidation across the codebase--[2025-06-09 17:59:00] - **Generic Interface Implementation: FilterParams Type Conversion**--## Decision--Converted the `FilterParams` interface into a generic type using intersection types to allow extending with additional properties.--## Rationale--User requested to make the existing `FilterParams` interface generic so it could be extended with additional properties like `FilterParams<{something: number}>`. The original interface-based approach with `extends` didn't properly merge the generic type parameter, causing TypeScript errors when trying to access properties from the generic type.--## Implementation Details--- Changed from `export interface FilterParams<T = {}> extends BaseParams {}` to `export type FilterParams<T = {}> = BaseParams & T`-- Used intersection types (`&`) instead of interface extension to properly merge the base properties with the generic type parameter-- Maintained `BaseParams` interface for the core filtering properties (startDate, endDate, timeZone, filters, pastMinutesStart, pastMinutesEnd)-- Added `OverviewParams<T>` as an alias for backward compatibility-- This allows proper type safety when using `FilterParams<{bucket: TimeBucket}>` where both base properties and the `bucket` property are accessible--## Impact--- Enables type-safe extension of filtering parameters across the analytics API-- Maintains backward compatibility with existing code-- Fixes TypeScript errors in files like [`getPerformanceTimeSeries.ts`](server/src/api/analytics/performance/getPerformanceTimeSeries.ts:187) that use extended filter parameters--[2025-06-12 21:10:00] - **Web Vitals Opt-in Flag Implementation**--## Decision--Added a `data-web-vitals` flag to the Rybbit tracking script that disables Web Vitals collection by default and only enables it when explicitly set to `"true"`.--## Rationale--This decision was made to:--- Reduce the default script size and network overhead by not loading the Web Vitals library unless needed-- Give users explicit control over performance metrics collection-- Improve initial page load performance for users who don't need Web Vitals data-- Follow privacy-by-design principles by making performance tracking opt-in--## Implementation Details--- Added `enableWebVitals` flag that checks for `data-web-vitals="true"` attribute-- Modified Web Vitals library loading to only occur when flag is enabled-- Updated `initWebVitals()` function to respect the flag setting-- Web Vitals are now disabled by default (breaking change from previous always-on behavior)-- Updated documentation in [`docs/src/content/script.mdx`](docs/src/content/script.mdx:1) to include the new flag-- Added comprehensive Web Vitals section explaining the feature and usage--## Files Modified--- [`server/public/script-full.js`](server/public/script-full.js:1) - Added web vitals flag logic-- [`docs/src/content/script.mdx`](docs/src/content/script.mdx:1) - Updated documentation with new flag and Web Vitals section--## Impact--- Users upgrading will need to add `data-web-vitals="true"` to continue collecting Web Vitals-- New installations will have better performance by default-- Reduced bandwidth usage for users who don't need performance metrics-- More granular control over tracking features--[2025-06-19 19:29:17] - **API Documentation Mobile Platform Consistency**--## Decision--Standardized mobile platform examples in API documentation to use logical pixels consistently and modernized deprecated APIs across Swift, Kotlin, and Flutter examples.--## Rationale--This decision was made to:--- Ensure consistency across all mobile platform examples in the API documentation-- Replace deprecated iOS APIs with modern alternatives to prevent future compatibility issues-- Clarify pixel type expectations for mobile developers to prevent implementation confusion-- Improve code quality and best practices in documentation examples-- Standardize example data across all platforms for better learning experience--## Implementation Details--- **Screen Pixel Type Clarification**: Updated API parameter documentation to specify "logical pixels (density-independent pixels)" for screenWidth/screenHeight parameters-- **Swift Example Modernization**:- - Replaced deprecated `UIScreen.main` with modern `UIWindowScene.screen` approach with proper fallback- - Added API key validation to prevent empty string errors- - Standardized example data to match server-side examples-- **Kotlin Example Updates**:- - Changed from physical pixels to logical pixels using `resources.displayMetrics.density` conversion- - Updated example data for consistency across platforms-- **Flutter Example Improvements**:- - Changed from physical pixels to logical pixels using `MediaQuery.of(context).devicePixelRatio`- - Moved app launch tracking from `build()` method to `initState()` to prevent multiple triggers- - Changed from StatelessWidget to StatefulWidget for proper lifecycle management- - Updated example data to match other platforms--## Files Modified--- [`docs/src/content/api.mdx`](docs/src/content/api.mdx:78) - API parameter documentation and all mobile platform examples--## Impact--- Improved consistency and accuracy across all mobile platform documentation-- Modernized deprecated APIs to ensure future compatibility-- Clarified pixel type expectations to prevent developer confusion-- Enhanced code quality and best practices in documentation examples-- Provided standardized learning experience across all supported platformsdiff --git a/memory-bank/error-tracking-plan.md b/memory-bank/error-tracking-plan.mddeleted file mode 100644index a1b365e7b..000000000--- a/memory-bank/error-tracking-plan.md+++ /dev/null@@ -1,138 +0,0 @@-# Frontend Error Tracking Implementation Plan--## Overview--This document outlines the implementation plan for adding frontend error tracking to the Rybbit analytics system. Errors will be tracked as a new event type in the existing ClickHouse infrastructure.--## Key Decisions--1. **Storage**: Use existing `props` JSON column in ClickHouse events table-2. **Filtering**: Track only first-party errors (same domain as the website)-3. **Opt-in**: Error tracking requires explicit enablement via `data-track-errors="true"`-4. **Truncation**: Error messages limited to 500 chars, stack traces to 2000 chars--## Implementation Components--### 1. Error Event Structure--Errors will be stored with `type: 'error'` in the events table:--```typescript-{- site_id: 123,- type: 'error',- event_name: 'TypeError', // The error type/name- props: {- message: 'Cannot read property...', // Truncated to 500 chars- stack: '...', // Truncated to 2000 chars- fileName: 'app.js',- lineNumber: 42,- columnNumber: 15,- timestamp: '2025-01-20T10:30:00Z'- },- // Standard fields auto-populated:- session_id, pathname, referrer, browser, os, device, country, etc.-}-```--### 2. Backend Changes--#### 2.1 Update `server/src/tracker/trackEvent.ts`--- Add 'error' to valid event types-- Validate error-specific properties-- Apply truncation to message (500) and stack (2000)--#### 2.2 Create `server/src/api/analytics/getErrorNames.ts`--- Similar structure to `getPageTitles.ts`-- Query errors grouped by `event_name`-- Return occurrence count and session count-- Support pagination and time filtering--### 3. Analytics Script Updates--#### 3.1 Add Error Capture--- Check for `data-track-errors="true"` attribute-- Install global error handler when enabled-- Filter errors to first-party only (same origin)-- Format and send error data to `/api/track`--#### 3.2 Error Filtering Logic--```javascript-// Only track errors from the same origin-if (error.fileName && !error.fileName.startsWith(window.location.origin)) {- return; // Skip third-party errors-}-```--### 4. Frontend UI Components--#### 4.1 Create `client/src/api/analytics/useGetErrorNames.ts`--- React Query hook for fetching error data-- Support pagination parameters-- Return error names with counts--#### 4.2 Implement `client/src/app/[site]/errors/page.tsx`--- List view similar to pages implementation-- Display: Error Name | Occurrences | Sessions Affected | Percentage-- Pagination support-- No sparklines in initial version--### 5. Documentation--Update `docs/src/content/script.mdx`:--- Add `data-track-errors` to configuration options table-- Default value: `"false"`-- Description: "Set to 'true' to enable automatic error tracking. When enabled, captures JavaScript errors from your domain only."--## Implementation Order--1. **Backend trackEvent.ts** - Add error type support (validate and store)-2. **Analytics script** - Add error capture with first-party filtering-3. **Backend getErrorNames.ts** - Create the analytics endpoint-4. **Frontend API hook** - Create useGetErrorNames.ts-5. **Frontend UI** - Implement errors/page.tsx-6. **Documentation** - Update script.mdx--## Technical Considerations--### Performance--- Errors stored in existing props JSON column-- ClickHouse handles JSON queries efficiently-- Consider adding dedicated columns if query performance becomes an issue--### Privacy--- First-party only filtering reduces noise and privacy concerns-- Stack traces may contain sensitive file paths - truncation helps-- No personal data should be in error messages--### Rate Limiting--- Consider implementing rate limiting in future iterations-- Suggested: 10 errors per minute per session-- Not included in initial implementation for simplicity--## Future Enhancements--1. **Error Details View**: Click through to see full error details-2. **Browser/OS Breakdown**: Show which browsers are most affected-3. **Time Series**: Show error trends over time-4. **Source Maps**: Support for production source map resolution-5. **Error Grouping**: Smart grouping of similar errors-6. **Alerting**: Notify when error rates spike--## Success Criteria--1. Errors are successfully captured and stored in ClickHouse-2. Error list page shows aggregated error data-3. Only first-party errors are tracked-4. Documentation clearly explains the feature-5. No performance impact on sites that don't enable error trackingdiff --git a/memory-bank/productContext.md b/memory-bank/productContext.mddeleted file mode 100644index dd5a8be6a..000000000--- a/memory-bank/productContext.md+++ /dev/null@@ -1,60 +0,0 @@-# Product Context--This file provides a high-level overview of the project and the expected product that will be created. Initially it is based upon projectBrief.md (if provided) and all other available project-related information in the working directory. This file is intended to be updated as the project evolves, and should be used to inform all other modes of the project's goals and context.--## Project Overview--**Rybbit Analytics** is an open source web analytics platform designed to be a more intuitive but still extremely powerful alternative to Google Analytics. The project launched in May 2025 and has achieved 6k GitHub stars, indicating strong community adoption.--## Project Goal--Create and maintain a comprehensive, user-friendly web analytics platform that provides powerful insights while remaining more intuitive than existing solutions like Google Analytics.--## Key Features--- Open source web analytics platform-- Intuitive user interface with powerful analytics capabilities-- Real-time analytics and performance monitoring-- User behavior tracking and analysis-- Event tracking and goal conversion-- Multi-site management with organizational structure-- Subscription-based business model with Stripe integration--## Overall Architecture--**Frontend (/client):**--- Next.js framework with TypeScript-- Tailwind CSS for styling-- Shadcn UI components-- Nivo.rocks for data visualization-- Tanstack Query for data fetching-- Tanstack Table for data tables-- Zustand for state management-- Luxon for date/time handling--**Backend (/server):**--- Fastify web framework-- Luxon for date/time operations-- Drizzle ORM for database operations-- Stripe for payment processing-- TypeScript throughout--**Documentation (/docs):**--- Nextra framework for documentation and landing page--**Data Storage:**--- PostgreSQL for relational data (users, organizations, sites, etc.)-- ClickHouse for high-volume events data and analytics-- Docker Compose for orchestration--**Infrastructure:**--- Docker containerization-- Self-hosting capabilities-- Cloud deployment options--2025-05-31 13:49:27 - Initial Memory Bank creation with project context from projectBrief.mddiff --git a/memory-bank/progress.md b/memory-bank/progress.mddeleted file mode 100644index 7714ec011..000000000--- a/memory-bank/progress.md+++ /dev/null@@ -1,274 +0,0 @@-# Progress--This file tracks the project's progress using a task list format.--## Completed Tasks--2025-05-31 13:49:45 - Memory Bank initialization completed--- Created productContext.md with comprehensive project overview-- Created activeContext.md for tracking current status-- Established project context from projectBrief.md--## Current Tasks--- Awaiting specific development tasks or architectural decisions from user-- Ready to analyze codebase and provide architectural guidance-- Prepared to document decisions and track progress--## Next Steps--- Identify current development priorities-- Analyze existing codebase for potential improvements-- Document architectural patterns and decisions-- Support ongoing development and feature implementation--2025-06-01 11:40:32 - Added skeleton loading state to SiteSelector component--- Implemented conditional rendering for loading state when site data is falsy-- Created skeleton with matching dimensions and styling-- Added pulse animation for better UX during data fetching--2025-06-01 11:42:55 - Added skeleton loading state to SiteSelectorContent dropdown items--- Implemented conditional rendering for loading state when sites data is falsy-- Created 3 skeleton dropdown items with matching structure and styling-- Added pulse animation and prevented interactions during loading state--2025-06-01 13:13:52 - Optimized getSitesFromOrg authorization checks for concurrent execution--- Converted sequential admin and member checks to run concurrently using Promise.all()-- Simplified conditional logic while maintaining same authorization behavior-- Improved performance by reducing total execution time for authorization--2025-06-01 13:14:48 - Further optimized getSitesFromOrg by adding sites query to concurrent execution--- Extended Promise.all to include sites data query alongside authorization checks-- Now runs admin check, member check, and sites query all concurrently-- Additional performance improvement by eliminating sequential sites data fetch--2025-06-01 13:16:19 - Completed full optimization of getSitesFromOrg with all 5 database operations concurrent--- Added organization owner query and organization info query to Promise.all-- Removed duplicate sequential queries that were previously executed separately-- Now runs admin check, member check, sites query, owner query, and org info query all concurrently-- Maximum performance optimization achieved - reduced from 5 sequential to 1 concurrent operation--2025-06-06 21:28:35 - Enhanced normalizeOrigin function to remove all subdomain prefixes--- Modified normalizeOrigin function in server/src/utils.ts to handle all subdomains, not just "www"-- Added support for IP addresses and localhost (returned as-is)-- Implemented logic for multi-level TLDs (e.g., .co.uk, .com.au)-- Function now extracts the root domain from any subdomain structure-- Examples: api.example.com β example.com, subdomain.site.co.uk β site.co.uk--2025-06-06 21:29:55 - Updated domain normalization in trackEvent.ts validateOrigin function--- Imported normalizeOrigin function from utils.ts into trackEvent.ts-- Replaced manual www. prefix removal with enhanced normalizeOrigin function-- Now uses consistent subdomain removal logic across both origin and site domain validation-- Ensures proper domain matching for sites with subdomains in tracking validation--2025-06-06 21:33:45 - Improved normalizeOrigin function with Public Suffix List and better input handling--- Added psl (Public Suffix List) dependency for reliable multi-level TLD detection-- Removed unnecessary URL parsing when input is already a hostname-- Function now accepts both URLs and hostnames as input-- Uses PSL to accurately determine registrable domains (handles all TLDs correctly)-- Added robust fallback logic for edge cases-- Eliminates hardcoded TLD list that was incomplete and unreliable--2025-06-06 23:20:00 - Completed creation of new integration guides--- Completed: Create Squarespace integration guide ([`docs/src/content/integrations/squarespace.mdx`](docs/src/content/integrations/squarespace.mdx:1))-- Completed: Create Wix integration guide ([`docs/src/content/integrations/wix.mdx`](docs/src/content/integrations/wix.mdx:1))-- Completed: Create Hugo integration guide ([`docs/src/content/integrations/hugo.mdx`](docs/src/content/integrations/hugo.mdx:1))-- Completed: Create Jekyll integration guide ([`docs/src/content/integrations/jekyll.mdx`](docs/src/content/integrations/jekyll.mdx:1))-- Completed: Create Drupal integration guide ([`docs/src/content/integrations/drupal.mdx`](docs/src/content/integrations/drupal.mdx:1))-- Completed: Create Laravel integration guide ([`docs/src/content/integrations/laravel.mdx`](docs/src/content/integrations/laravel.mdx:1))-- Created: [`docs/src/content/integrations/_meta.json`](docs/src/content/integrations/_meta.json:1) to include new guides in sidebar.--2025-06-07 00:16:00 - Completed review and standardization of integration guides--- Standardized script tags (URL, `async defer`) across all relevant guides in `docs/src/content/guides/`.-- Updated Next.js guide ([`docs/src/content/guides/react/next-js.mdx`](docs/src/content/guides/react/next-js.mdx:1)) with SPA page view tracking callout and custom event tracking section.-- Significantly expanded Shopify guide ([`docs/src/content/guides/shopify.mdx`](docs/src/content/guides/shopify.mdx:1)) with e-commerce event tracking.-- Expanded Framer guide ([`docs/src/content/guides/framer.mdx`](docs/src/content/guides/framer.mdx:1)) and Webflow guide ([`docs/src/content/guides/webflow.mdx`](docs/src/content/guides/webflow.mdx:1)) with notes on SPA behavior and custom event tracking.-- Corrected minor inconsistencies in Hugo and Jekyll guides.--2025-06-08 19:13:40 - Added past 24-hour mode support to performance analytics hooks--- Modified [`useGetPerformanceTimeSeries.ts`](client/src/api/analytics/useGetPerformanceTimeSeries.ts:1) to support last-24-hours mode-- Modified [`useGetPerformanceOverview.ts`](client/src/api/analytics/useGetPerformanceOverview.ts:1) to support last-24-hours mode-- Implemented conditional query parameter logic using pastMinutesStart/pastMinutesEnd for 24-hour mode-- Updated query keys to differentiate between "past-minutes" and "date-range" modes-- Maintained backward compatibility with existing date-range based queries-- Achieved consistency across all performance analytics hooks--[2025-01-08 19:23:28] - **COMPLETED: Analytics Hooks Code Deduplication**--- **Task**: Eliminate duplicated query parameter logic across analytics hooks-- **Status**: β COMPLETED-- **Work Done**:- - Created centralized [`getQueryParams`](client/src/api/utils.ts:1) utility function- - Refactored 7 analytics hook files to use the new utility- - Reduced code duplication from ~20 lines per file to 3 lines- - Enhanced utility to support custom past minutes parameters for complex cases-- **Files Modified**:- - [`client/src/api/utils.ts`](client/src/api/utils.ts:1) - Added `getQueryParams` utility- - [`useGetPerformanceTimeSeries.ts`](client/src/api/analytics/useGetPerformanceTimeSeries.ts:1) - Refactored- - [`useGetPerformanceOverview.ts`](client/src/api/analytics/useGetPerformanceOverview.ts:1) - Refactored- - [`useGetPerformanceByDimension.ts`](client/src/api/analytics/useGetPerformanceByDimension.ts:1) - Refactored- - [`useSingleCol.ts`](client/src/api/analytics/useSingleCol.ts:1) - Refactored with custom params- - [`usePaginatedSingleCol.ts`](client/src/api/analytics/usePaginatedSingleCol.ts:1) - Refactored- - [`useInfiniteSingleCol.ts`](client/src/api/analytics/useInfiniteSingleCol.ts:1) - Refactored- - [`useGetPageTitles.ts`](client/src/api/analytics/useGetPageTitles.ts:1) - Refactored-- **Impact**: Significantly improved code maintainability and reduced duplication across analytics layer--[2025-01-08 19:31:22] - **COMPLETED: Additional Hook Consolidation - useGetOverview**--- **Task**: Combine `useGetOverview` and `useGetOverviewPastMinutes` functions-- **Status**: β COMPLETED-- **Work Done**:- - Enhanced [`useGetOverview`](client/src/api/analytics/useGetOverview.ts:1) to accept optional `pastMinutesStart` and `pastMinutesEnd` parameters- - Integrated with [`getQueryParams`](client/src/api/utils.ts:1) utility for consistent query parameter handling- - Maintained backward compatibility by keeping `useGetOverviewPastMinutes` as a wrapper function- - Implemented intelligent query key generation based on parameters used-- **Benefits**:- - Eliminated code duplication between two similar hooks- - Simplified API while maintaining backward compatibility- - Consistent with other refactored analytics hooks- - Single source of truth for overview data fetching-- **Total Refactoring Summary**: 8 analytics hook files now use the centralized `getQueryParams` utility, significantly reducing code duplication across the entire analytics layer--[2025-01-08 19:37:23] - **COMPLETED: Bug Fix and Callsite Updates for useGetOverview**--- **Task**: Fix TypeScript error and update all callsites to use consolidated `useGetOverview` hook-- **Status**: β COMPLETED-- **Bug Fixed**:- - TypeScript error: "Argument of type 'number' is not assignable to parameter of type 'Record<string, any>'"- - Root cause: Incorrect function call signature for [`getQueryParams`](client/src/api/utils.ts:38)- - Solution: Fixed parameter structure in [`useGetOverview`](client/src/api/analytics/useGetOverview.ts:1)-- **Callsites Updated**: 4 files updated to use consolidated hook- - [`useGetOverviewWithInView.ts`](client/src/api/analytics/useGetOverviewWithInView.ts:1) - Updated import and comment- - [`SiteCard.tsx`](client/src/components/SiteCard.tsx:1) - Replaced `useGetOverviewPastMinutes` with `useGetOverview`- - [`Overview.tsx`](client/src/app/[site]/main/components/MainSection/Overview.tsx:1) - Replaced 2 instances with `useGetOverview`-- **Final Status**: All TypeScript errors resolved, hook consolidation complete across entire codebase-- **Total Impact**: 9 analytics hook files now use centralized patterns, eliminating 100+ lines of duplicated code--[2025-06-08 22:08:11] - **COMPLETED: SiteSelector UI Enhancement - Dropdown to Popover Conversion**--- **Task**: Convert SiteSelector from dropdown menu to popover with enhanced functionality-- **Status**: β COMPLETED-- **Work Done**:- - Replaced [`DropdownMenu`](client/src/components/ui/dropdown-menu.tsx:1) components with [`Popover`](client/src/components/ui/popover.tsx:1) components- - Enhanced site display to show sessions in past 24 hours using [`sessionsLast24Hours`](client/src/api/admin/sites.ts:60) field- - Integrated [`AddSite`](client/src/app/components/AddSite.tsx:1) component at bottom of popover for new site creation- - Added [`ChevronDown`](https://lucide.dev/icons/chevron-down) icon to trigger button- - Implemented scrollable container (`max-h-96 overflow-y-auto`) for better UX with many sites- - Enhanced skeleton loading states to match new layout structure-- **Files Modified**:- - [`client/src/app/[site]/components/Sidebar/SiteSelector.tsx`](client/src/app/[site]/components/Sidebar/SiteSelector.tsx:1) - Complete UI transformation-- **Features Added**:- - Site domain display with sessions count (e.g., "example.com - 42 sessions (24h)")- - Inline new site creation capability- - Improved visual hierarchy and spacing- - Better responsive design with scrolling support-- **Impact**: Enhanced user experience for site management with richer information display and streamlined site creation workflow--[2025-06-10 23:34:29] - **COMPLETED: Enhanced 404 Not Found Page**--- **Task**: Create a basic 404 page for [`client/src/app/not-found.tsx`](client/src/app/not-found.tsx:1)-- **Status**: β COMPLETED-- **Work Done**:- - Replaced basic "404" text with comprehensive, user-friendly 404 page- - Added analytics-themed icon using [`BarChart3`](https://lucide.dev/icons/bar-chart3) from Lucide React- - Implemented responsive design with proper spacing and typography- - Added actionable navigation buttons using [`Button`](client/src/components/ui/button.tsx:1) component- - Included "Go to Dashboard" and "Go Back" options for user convenience- - Added help links to documentation and GitHub repository- - Maintained consistency with project's design system (Tailwind CSS, Shadcn UI)- - Updated page title to "Rybbit Β· Page Not Found" for better SEO-- **Design Features**:- - Centered layout with proper vertical spacing- - Dark mode support using Tailwind's dark: variants- - Responsive button layout (stacked on mobile, side-by-side on desktop)- - Professional error messaging with helpful context- - Brand-consistent styling matching the analytics platform theme-- **Impact**: Users encountering 404 errors now have a professional, helpful experience with clear navigation options and access to support resources--[2025-06-12 21:10:00] - **COMPLETED: Web Vitals Opt-in Flag Implementation**--- **Task**: Add web vitals flag to disable Web Vitals collection unless explicitly enabled-- **Status**: β COMPLETED-- **Work Done**:- - Added `data-web-vitals` attribute support to [`server/public/script-full.js`](server/public/script-full.js:1)- - Web Vitals library loading now conditional based on flag setting- - Modified `initWebVitals()` function to respect the enableWebVitals flag- - Updated documentation in [`docs/src/content/script.mdx`](docs/src/content/script.mdx:1)- - Added comprehensive Web Vitals section with usage examples- - Added `data-track-outbound` to documentation table for completeness-- **Key Changes**:- - Web Vitals are now **disabled by default** (opt-in behavior)- - Users must set `data-web-vitals="true"` to enable performance metrics collection- - Reduced default script overhead by not loading Web Vitals library unless needed- - Enhanced documentation with clear usage examples and explanations-- **Impact**: Improved default performance while giving users explicit control over Web Vitals collection--[2025-06-13 23:46:38] - **COMPLETED: Added Web Vitals Option to ScriptBuilder Component**--- **Task**: Add Web Vitals toggle option to the ScriptBuilder component in site settings-- **Status**: β COMPLETED-- **Work Done**:- - Added `webVitals` state variable (defaults to `false` to match opt-in behavior)- - Added `data-web-vitals="true"` attribute to generated script when enabled- - Created Web Vitals toggle section in the UI with proper labeling and description- - Positioned the option logically after the track query parameters section- - Used consistent styling and layout patterns matching existing options-- **Files Modified**:- - [`client/src/components/SiteSettings/ScriptBuilder.tsx`](client/src/components/SiteSettings/ScriptBuilder.tsx:1) - Added Web Vitals configuration option-- **Features Added**:- - Toggle switch for enabling/disabling Web Vitals collection- - Clear description explaining what metrics are collected (LCP, CLS, INP, FCP, TTFB)- - Conditional script attribute generation (`data-web-vitals="true"` only when enabled)- - Consistent with existing opt-in behavior (disabled by default)-- **Impact**: Users can now easily enable Web Vitals performance metrics collection through the ScriptBuilder UI, matching the functionality documented in the script documentation--[2025-06-19 19:28:32] - **COMPLETED: API Documentation Language Examples Fixes**--- **Task**: Fix potential issues in language examples in API documentation-- **Status**: β COMPLETED-- **Issues Fixed**:- 1. **Screen Pixel Type Clarification**: Updated API documentation to specify "logical pixels (density-independent pixels)" for screenWidth/screenHeight parameters- 2. **Swift Example Modernization**:- - Replaced deprecated `UIScreen.main` with modern `UIWindowScene.screen` approach- - Added proper API key validation to prevent empty string errors- - Updated example data to match server-side examples for consistency- 3. **Kotlin Example Consistency**:- - Changed from physical pixels to logical pixels using density conversion- - Updated example data to match server-side examples- 4. **Flutter Example Improvements**:- - Changed from physical pixels to logical pixels using devicePixelRatio- - Moved app launch tracking from `build()` method to `initState()` to prevent multiple triggers- - Updated example data to match server-side examples- - Changed from StatelessWidget to StatefulWidget for proper lifecycle management-- **Files Modified**:- - [`docs/src/content/api.mdx`](docs/src/content/api.mdx:78) - API parameter documentation and all mobile platform examples-- **Impact**: Improved consistency across platform examples, modernized deprecated APIs, and clarified pixel type expectations for mobile developers--[2025-06-20 16:16:23] - **COMPLETED: Docker Cleanup Documentation**--- **Task**: Add Docker cleanup section to troubleshooting documentation-- **Status**: β COMPLETED-- **Work Done**:- - Added comprehensive "Full Docker Cleanup" section to [`docs/src/content/troubleshooting.mdx`](docs/src/content/troubleshooting.mdx:1)- - Included all requested Docker cleanup commands (containers, images, volumes, networks)- - Added warning callout about destructive nature of cleanup commands- - Provided alternative Rybbit-specific cleanup options using docker-compose- - Added recovery instructions for restarting after cleanup-- **Features Added**:- - Step-by-step cleanup process with explanations- - System-wide vs project-specific cleanup options- - Clear warnings about data loss- - Post-cleanup recovery steps-- **Impact**: Users experiencing Docker issues now have clear, comprehensive cleanup procedures with appropriate warnings and recovery guidancediff --git a/memory-bank/systemPatterns.md b/memory-bank/systemPatterns.mddeleted file mode 100644index 88ad6831b..000000000--- a/memory-bank/systemPatterns.md+++ /dev/null@@ -1,47 +0,0 @@-# System Patterns--This file documents recurring patterns and standards used in the project.-It is optional, but recommended to be updated as the project evolves.--## Coding Patterns--**Frontend Patterns:**--- Next.js App Router for routing and page structure-- TypeScript throughout for type safety-- Tailwind CSS for utility-first styling-- Shadcn components for consistent UI elements-- Tanstack Query for server state management-- Zustand for client state management-- Luxon for date/time operations--**Backend Patterns:**--- Fastify for high-performance HTTP server-- Drizzle ORM for type-safe database operations-- TypeScript for consistent typing across frontend and backend-- API route organization under `/api` directory structure--## Architectural Patterns--**Data Architecture:**--- PostgreSQL for relational data (users, organizations, sites, configurations)-- ClickHouse for high-volume analytics events and time-series data-- Clear separation between operational and analytical data stores--**Service Architecture:**--- Microservice-oriented with clear separation of concerns-- Docker containerization for consistent deployment-- Environment-based configuration management--**Authentication & Authorization:**--- Organization-based multi-tenancy-- Role-based access control-- Stripe integration for subscription management--## Testing Patterns--2025-05-31 13:49:59 - Initial system patterns documented based on project structure analysisdiff --git a/mockdata/README.md b/mockdata/README.mddeleted file mode 100644index f940e8b62..000000000--- a/mockdata/README.md+++ /dev/null@@ -1,101 +0,0 @@-# E-commerce Mock Data Generator for Rybbit--This script generates realistic mock analytics data for Rybbit, simulating user behavior on an e-commerce website called ShopEase.--## Features--- Generates approximately 5 million events per day over a specified number of days-- Creates realistic user shopping sessions with multiple pageviews and custom events-- Models realistic e-commerce user journeys (browsing, product views, add to cart, checkout, purchase)-- Sessions end after 30 minutes of inactivity-- Distributes traffic realistically throughout the day (with peak hours)-- Uses Faker.js to generate realistic product data, location information, and more-- Uses weighted distributions for pages, browsers, operating systems, screen sizes, and more-- Varies the number of events per day to be more realistic--## Requirements--- Node.js 14+-- ClickHouse database set up (configuration in .env file)--## Installation--1. Make sure all dependencies are installed:--```bash-npm install-```--## Usage--Run the script with optional parameters:--```bash-# Default: 30 days with ~5 million events per day-npm run generate--# Custom parameters (days, events per day)-node index.js 15 2000000 # 15 days with ~2 million events per day-```--### Parameters--1. `daysInPast`: Number of days in the past to generate data for (default: 30)-2. `eventsPerDay`: Approximate number of events per day (default: 5,000,000)--## Data Distribution--The script creates realistic distributions of:--- Page views with appropriate e-commerce user flows-- Shopping behavior patterns (browsing, cart additions, checkouts, purchases)-- Geographic distribution (countries and regions using Faker.js)-- Browser types and versions-- Operating systems-- Screen resolutions-- Referrers (including social media and other e-commerce sites)-- Custom events specific to e-commerce (product views, add to cart, checkout steps, etc.)-- Time of day (peak hours vs. off hours)--## E-commerce Event Types--The generator includes these e-commerce-specific events:--- page-view - Basic page view tracking-- product-view - When a user views a product detail page-- add-to-cart - When a user adds an item to their cart-- remove-from-cart - When a user removes an item from their cart-- begin-checkout - When a user starts the checkout process-- checkout-step - Tracking each step of the checkout funnel-- purchase - When a user completes a purchase-- search - When a user searches for products-- filter-products - When a user applies filters to product listings-- add-to-wishlist - When a user adds items to their wishlist-- And more...--## Data Schema--Events are inserted into the ClickHouse `pageviews` table with the following schema:--- site_id-- timestamp-- session_id-- user_id-- hostname-- pathname-- querystring-- page_title-- referrer-- browser-- browser_version-- operating_system-- operating_system_version-- language-- country-- iso_3166_2-- screen_width-- screen_height-- device_type-- type (pageview or custom_event)-- event_name (for custom events)-- properties (JSON string for custom events)diff --git a/mockdata/index.js b/mockdata/index.jsdeleted file mode 100644index f66d83206..000000000--- a/mockdata/index.js+++ /dev/null@@ -1,1356 +0,0 @@-const { createClient } = require("@clickhouse/client");-const { DateTime } = require("luxon");-const { faker } = require("@faker-js/faker");-require("dotenv").config();--const clickhouse = createClient({- url: process.env.CLICKHOUSE_HOST,- database: process.env.CLICKHOUSE_DB,- password: process.env.CLICKHOUSE_PASSWORD,-});--// Command line arguments-const args = process.argv.slice(2);-const daysInPast = parseInt(args[0] || "30", 10);-const eventsPerDay = parseInt(args[1] || "5000000", 10);--console.log(- `Generating mock data for ${daysInPast} days with approximately ${eventsPerDay} events per day`-);--// Site ID to use-const SITE_ID = 5;--// Generate 10,000 unique user IDs-const userIds = Array.from({ length: 10000 }, () => faker.string.uuid());--// Website details-const SITE_NAME = "ShopEase";-const SITE_DOMAIN = "shopease.com";--// Generate products for consistent paths-const products = Array.from({ length: 50 }, (_, index) => {- const productId = 1000 + index;- const name = faker.commerce.productName();- const category = faker.commerce.department();- const subcategory = faker.commerce.productAdjective();- const slug = name.toLowerCase().replace(/[^a-z0-9]+/g, "-");-- return {- id: productId,- name,- category,- subcategory,- slug,- price: faker.commerce.price({ min: 10, max: 1000 }),- };-});--// Generate categories-const categories = [...new Set(products.map((p) => p.category))];-const subcategories = [...new Set(products.map((p) => p.subcategory))];--// Realistic page paths with weighted distribution-const pagePaths = [- { path: "/", weight: 20, title: `${SITE_NAME} - Online Shopping` },- { path: "/products", weight: 15, title: `All Products - ${SITE_NAME}` },- { path: "/categories", weight: 10, title: `Shop by Category - ${SITE_NAME}` },- { path: "/sale", weight: 12, title: `Sale Items - ${SITE_NAME}` },- { path: "/new-arrivals", weight: 8, title: `New Arrivals - ${SITE_NAME}` },- { path: "/cart", weight: 7, title: `Your Shopping Cart - ${SITE_NAME}` },- { path: "/checkout", weight: 5, title: `Checkout - ${SITE_NAME}` },- { path: "/account", weight: 5, title: `Your Account - ${SITE_NAME}` },- { path: "/wishlist", weight: 4, title: `Your Wishlist - ${SITE_NAME}` },- { path: "/order-history", weight: 3, title: `Order History - ${SITE_NAME}` },- { path: "/about", weight: 2, title: `About Us - ${SITE_NAME}` },- { path: "/contact", weight: 2, title: `Contact Us - ${SITE_NAME}` },- { path: "/faq", weight: 2, title: `FAQ - ${SITE_NAME}` },- { path: "/shipping", weight: 2, title: `Shipping Policy - ${SITE_NAME}` },- { path: "/returns", weight: 2, title: `Returns & Refunds - ${SITE_NAME}` },-];--// Add category pages-categories.forEach((category) => {- const slug = category.toLowerCase().replace(/[^a-z0-9]+/g, "-");- pagePaths.push({- path: `/category/${slug}`,- weight: 6,- title: `${category} - ${SITE_NAME}`,- category,- });-});--// Add product pages (high weight as these are important for e-commerce)-products.forEach((product) => {- pagePaths.push({- path: `/product/${product.id}-${product.slug}`,- weight: 9,- title: `${product.name} - ${SITE_NAME}`,- product,- });-});--// Custom events with weighted distribution-const customEvents = [- { name: "page-view", weight: 25, properties: { page: "home" } },- { name: "product-view", weight: 20, properties: {} }, // Will be filled with product details- { name: "add-to-cart", weight: 15, properties: {} }, // Will be filled with product details- { name: "remove-from-cart", weight: 5, properties: {} }, // Will be filled with product details- {- name: "begin-checkout",- weight: 8,- properties: { items_count: 1, value: 0 },- },- { name: "checkout-step", weight: 7, properties: { step: 1, option: "" } },- {- name: "purchase",- weight: 5,- properties: { transaction_id: "", value: 0, currency: "USD", items: [] },- },- {- name: "product-click",- weight: 12,- properties: { list_name: "recommended" },- },- { name: "search", weight: 10, properties: { search_term: "" } },- {- name: "filter-products",- weight: 8,- properties: { filter: "price", value: "100-200" },- },- { name: "add-to-wishlist", weight: 6, properties: {} }, // Will be filled with product details- { name: "share-product", weight: 4, properties: { method: "email" } },- {- name: "select-promotion",- weight: 5,- properties: { promotion_name: "Summer Sale" },- },- { name: "sign-up", weight: 3, properties: { method: "email" } },- { name: "login", weight: 6, properties: { method: "site" } },- { name: "view-cart", weight: 10, properties: { items_count: 0, value: 0 } },-];--// Tab/query parameters for certain pages-const tabParams = {- "/product/": [- { param: "?variant=large", weight: 30 },- { param: "?variant=medium", weight: 25 },- { param: "?variant=small", weight: 20 },- { param: "?color=blue", weight: 15 },- { param: "?color=black", weight: 15 },- { param: "", weight: 20 },- ],- "/products": [- { param: "?sort=price-asc", weight: 25 },- { param: "?sort=price-desc", weight: 20 },- { param: "?sort=newest", weight: 20 },- { param: "?sort=popular", weight: 20 },- { param: "", weight: 15 },- ],- "/category/": [- { param: "?filter=new", weight: 20 },- { param: "?filter=sale", weight: 25 },- { param: "?filter=in-stock", weight: 20 },- { param: "?sort=price-low", weight: 15 },- { param: "", weight: 20 },- ],-};--// Referrers with weighted distribution-const referrers = [- { url: "", weight: 30 }, // Direct- { url: "https://www.google.com/", weight: 25 },- { url: "https://www.facebook.com/", weight: 10 },- { url: "https://www.instagram.com/", weight: 12 },- { url: "https://pinterest.com/", weight: 8 },- { url: "https://www.amazon.com/", weight: 5 },- { url: "https://www.youtube.com/", weight: 5 },- { url: "https://www.tiktok.com/", weight: 4 },- { url: "https://twitter.com/", weight: 3 },- { url: "https://www.bing.com/", weight: 3 },- { url: "https://www.retailmenot.com/", weight: 2 },- { url: "https://slickdeals.net/", weight: 2 },- { url: "https://www.dealsplus.com/", weight: 1 },- { url: `https://email.${SITE_DOMAIN}/`, weight: 10 }, // Email campaigns- { url: "https://www.pricegrabber.com/", weight: 1 },-];--// Browsers with weighted distribution-const browsers = [- { name: "Chrome", weight: 60, versions: ["133", "134", "135", "132"] },- { name: "Firefox", weight: 15, versions: ["124", "123", "122"] },- { name: "Safari", weight: 10, versions: ["17", "16", "15"] },- { name: "Edge", weight: 8, versions: ["122", "121", "120"] },- { name: "Opera", weight: 4, versions: ["105", "104", "103"] },- { name: "Yandex", weight: 3, versions: ["25", "24", "23"] },-];--// Operating systems with weighted distribution-const operatingSystems = [- { name: "Windows", weight: 65, versions: ["10", "11", "7"] },- { name: "Android", weight: 15, versions: ["13", "12", "11", "10"] },- { name: "iOS", weight: 10, versions: ["17", "16", "15"] },- { name: "macOS", weight: 7, versions: ["14", "13", "12"] },- { name: "Linux", weight: 3, versions: ["5.15", "5.10", "5.8"] },-];--// Screen resolutions with weighted distribution-const screenResolutions = [- { width: 1920, height: 1080, weight: 30 },- { width: 1366, height: 768, weight: 20 },- { width: 1536, height: 864, weight: 15 },- { width: 2560, height: 1440, weight: 10 },- { width: 1440, height: 900, weight: 8 },- { width: 1024, height: 768, weight: 5 },- { width: 3840, height: 2160, weight: 5 },- { width: 1280, height: 720, weight: 3 },- // Mobile resolutions- { width: 414, height: 896, weight: 3 },- { width: 375, height: 667, weight: 3 },- { width: 360, height: 740, weight: 3 },- { width: 412, height: 915, weight: 2 },- { width: 390, height: 844, weight: 2 },- { width: 768, height: 1024, weight: 1 }, // iPad-];--// Languages with weighted distribution-const languages = [- { code: "en-US", weight: 30 },- { code: "en", weight: 15 },- { code: "ru", weight: 12 },- { code: "de", weight: 8 },- { code: "pl", weight: 7 },- { code: "es", weight: 6 },- { code: "fr", weight: 6 },- { code: "zh-CN", weight: 5 },- { code: "it", weight: 4 },- { code: "cs", weight: 3 },- { code: "pt-BR", weight: 2 },- { code: "ja", weight: 2 },-];--// Countries and regions-const geoData = [- {- country: "US",- weight: 25,- regions: [- { code: "US-NY", weight: 20 },- { code: "US-CA", weight: 18 },- { code: "US-TX", weight: 15 },- { code: "US-FL", weight: 12 },- { code: "US-IL", weight: 10 },- { code: "US-PA", weight: 8 },- { code: "US-OH", weight: 7 },- { code: "US-MI", weight: 5 },- { code: "US-GA", weight: 5 },- ],- },- {- country: "RU",- weight: 20,- regions: [- { code: "RU-MOW", weight: 25 },- { code: "RU-SPE", weight: 15 },- { code: "RU-KDA", weight: 10 },- { code: "RU-KYA", weight: 10 },- { code: "RU-NVS", weight: 8 },- { code: "RU-SVE", weight: 8 },- { code: "RU-IRK", weight: 5 },- { code: "RU-YAR", weight: 5 },- ],- },- {- country: "DE",- weight: 10,- regions: [- { code: "DE-BY", weight: 20 },- { code: "DE-NW", weight: 18 },- { code: "DE-BW", weight: 15 },- { code: "DE-NI", weight: 12 },- { code: "DE-HE", weight: 10 },- { code: "DE-BE", weight: 8 },- ],- },- {- country: "PL",- weight: 8,- regions: [- { code: "PL-MZ", weight: 20 },- { code: "PL-DS", weight: 15 },- { code: "PL-WP", weight: 15 },- { code: "PL-SL", weight: 12 },- { code: "PL-MA", weight: 10 },- ],- },- {- country: "GB",- weight: 7,- regions: [- { code: "GB-ENG", weight: 80 },- { code: "GB-SCT", weight: 10 },- { code: "GB-WLS", weight: 7 },- { code: "GB-NIR", weight: 3 },- ],- },- {- country: "FR",- weight: 6,- regions: [- { code: "FR-IDF", weight: 25 },- { code: "FR-ARA", weight: 15 },- { code: "FR-HDF", weight: 12 },- { code: "FR-NAQ", weight: 10 },- { code: "FR-OCC", weight: 10 },- ],- },- {- country: "CA",- weight: 5,- regions: [- { code: "CA-ON", weight: 40 },- { code: "CA-QC", weight: 25 },- { code: "CA-BC", weight: 15 },- { code: "CA-AB", weight: 10 },- ],- },- {- country: "CN",- weight: 4,- regions: [- { code: "CN-11", weight: 20 }, // Beijing- { code: "CN-31", weight: 20 }, // Shanghai- { code: "CN-44", weight: 15 }, // Guangdong- { code: "CN-51", weight: 10 }, // Sichuan- { code: "CN-33", weight: 10 }, // Zhejiang- { code: "CN-32", weight: 10 }, // Jiangsu- ],- },- {- country: "AU",- weight: 3,- regions: [- { code: "AU-NSW", weight: 35 },- { code: "AU-VIC", weight: 30 },- { code: "AU-QLD", weight: 20 },- { code: "AU-WA", weight: 15 },- ],- },- {- country: "ES",- weight: 3,- regions: [- { code: "ES-MD", weight: 25 },- { code: "ES-CT", weight: 20 },- { code: "ES-AN", weight: 18 },- { code: "ES-VA", weight: 12 },- ],- },- {- country: "IT",- weight: 2,- regions: [- { code: "IT-25", weight: 20 }, // Lombardy- { code: "IT-62", weight: 15 }, // Lazio- { code: "IT-21", weight: 15 }, // Piedmont- { code: "IT-34", weight: 12 }, // Veneto- ],- },- {- country: "CZ",- weight: 2,- regions: [- { code: "CZ-PR", weight: 35 },- { code: "CZ-JM", weight: 20 },- { code: "CZ-ST", weight: 15 },- ],- },- {- country: "AT",- weight: 2,- regions: [- { code: "AT-9", weight: 35 }, // Vienna- { code: "AT-6", weight: 20 }, // Styria- { code: "AT-4", weight: 15 }, // Upper Austria- ],- },- {- country: "JP",- weight: 2,- regions: [- { code: "JP-13", weight: 30 }, // Tokyo- { code: "JP-27", weight: 15 }, // Osaka- { code: "JP-14", weight: 10 }, // Kanagawa- ],- },- {- country: "BR",- weight: 1,- regions: [- { code: "BR-SP", weight: 35 },- { code: "BR-RJ", weight: 20 },- { code: "BR-MG", weight: 15 },- ],- },-];--// Helper function to select a random item based on weights-function weightedRandom(items) {- const totalWeight = items.reduce((sum, item) => sum + item.weight, 0);- let random = Math.random() * totalWeight;-- for (const item of items) {- random -= item.weight;- if (random <= 0) {- return item;- }- }-- return items[0]; // Fallback-}--// Helper function to generate a UUID-function generateUUID() {- return "xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx".replace(/[xy]/g, function (c) {- const r = (Math.random() * 16) | 0;- const v = c === "x" ? r : (r & 0x3) | 0x8;- return v.toString(16);- });-}--// Function to generate a realistic timestamp for a given day with more traffic during peak hours-function generateTimeForDay(date) {- // Define peak hours (in 24-hour format)- const peakHours = [- { hour: 9, weight: 5 }, // 9 AM- { hour: 10, weight: 6 }, // 10 AM- { hour: 11, weight: 7 }, // 11 AM- { hour: 12, weight: 8 }, // 12 PM- { hour: 13, weight: 9 }, // 1 PM- { hour: 14, weight: 10 }, // 2 PM- { hour: 15, weight: 10 }, // 3 PM- { hour: 16, weight: 9 }, // 4 PM- { hour: 17, weight: 8 }, // 5 PM- { hour: 18, weight: 9 }, // 6 PM- { hour: 19, weight: 10 }, // 7 PM- { hour: 20, weight: 10 }, // 8 PM- { hour: 21, weight: 9 }, // 9 PM- { hour: 22, weight: 7 }, // 10 PM- { hour: 23, weight: 5 }, // 11 PM- { hour: 0, weight: 3 }, // 12 AM- { hour: 1, weight: 2 }, // 1 AM- { hour: 2, weight: 1 }, // 2 AM- { hour: 3, weight: 1 }, // 3 AM- { hour: 4, weight: 1 }, // 4 AM- { hour: 5, weight: 1 }, // 5 AM- { hour: 6, weight: 2 }, // 6 AM- { hour: 7, weight: 3 }, // 7 AM- { hour: 8, weight: 4 }, // 8 AM- ];-- const hour = weightedRandom(peakHours).hour;- const minute = Math.floor(Math.random() * 60);- const second = Math.floor(Math.random() * 60);-- return date.set({ hour, minute, second });-}--// Helper function to format estimated time remaining-function formatETA(currentCount, targetCount, eventsPerSecond, elapsedSeconds) {- if (eventsPerSecond <= 0 || elapsedSeconds < 1) return "calculating...";-- const remainingEvents = targetCount - currentCount;-- // Use a dynamic slowdown factor based on how far along we are- // The more events we've generated, the more we expect slowdown- const progress = currentCount / targetCount;- const slowdownFactor = 1 + progress * 0.3; // Gradually add up to 30% buffer as we progress-- // Calculate seconds remaining with the slowdown factor applied- const adjustedRate = eventsPerSecond / slowdownFactor;- const secondsRemaining = remainingEvents / adjustedRate;-- // Format the ETA string- if (secondsRemaining < 60) {- return `${Math.ceil(secondsRemaining)}s`;- } else if (secondsRemaining < 3600) {- return `${Math.floor(secondsRemaining / 60)}m ${Math.ceil(- secondsRemaining % 60- )}s`;- } else {- const hours = Math.floor(secondsRemaining / 3600);- const minutes = Math.floor((secondsRemaining % 3600) / 60);- return `${hours}h ${minutes}m`;- }-}--// Helper function to format elapsed time-function formatTime(seconds) {- if (seconds < 60) {- return `${Math.floor(seconds)}s`;- } else if (seconds < 3600) {- const minutes = Math.floor(seconds / 60);- const remainingSeconds = Math.floor(seconds % 60);- return `${minutes}m ${remainingSeconds}s`;- } else {- const hours = Math.floor(seconds / 3600);- const minutes = Math.floor((seconds % 3600) / 60);- const remainingSeconds = Math.floor(seconds % 60);- return `${hours}h ${minutes}m ${remainingSeconds}s`;- }-}--// Optimized version of generateSessionEvents for speed-function generateSessionEventsOptimized(- userId,- sessionId,- startTime,- sessionData-) {- // Pre-allocate array for events - this is much faster than dynamic growth- // Most sessions will have between 1-30 events, so this is a reasonable size- const MAX_EVENTS = 36;- const events = new Array(MAX_EVENTS);- let eventCount = 0;-- // Session duration calculation- const sessionDuration = Math.floor(Math.random() * 1800) + 60; // 1-30 minutes in seconds- const numberOfPageviews = Math.floor(Math.random() * 8) + 1; // 1-8 pageviews per session (reduced for speed)-- // Cache commonly used values to avoid repeated property lookups- const siteId = SITE_ID;- const hostname = SITE_DOMAIN;- const browser = sessionData.browser;- const browserVersion = sessionData.browserVersion;- const os = sessionData.os;- const osVersion = sessionData.osVersion;- const language = sessionData.language;- const screenWidth = sessionData.screenWidth;- const screenHeight = sessionData.screenHeight;- const deviceType = sessionData.deviceType;- const country = sessionData.country;- const iso3166 = sessionData.iso3166;- const initialReferrer = sessionData.referrer;- const dateFormat = "yyyy-MM-dd HH:mm:ss";-- // Track session state- let currentPath = null;- let currentProduct = null;- let cartItems = [];- let hasPurchased = false;- let currentTime = startTime;-- // Create a fast template event creator function - much faster than creating and copying objects- function createEvent(- pathname,- querystring,- title,- referrer,- type,- eventName,- props- ) {- if (eventCount >= MAX_EVENTS) return; // Safety check-- events[eventCount++] = {- site_id: siteId,- session_id: sessionId,- user_id: userId,- hostname: hostname,- browser: browser,- browser_version: browserVersion,- operating_system: os,- operating_system_version: osVersion,- language: language,- screen_width: screenWidth,- screen_height: screenHeight,- device_type: deviceType,- country: country,- iso_3166_2: iso3166,- timestamp: currentTime.toFormat(dateFormat),- pathname: pathname,- querystring: querystring || "",- page_title: title,- referrer: referrer || "",- type: type,- event_name: eventName || "",- properties: props || "",- };- }-- // Fast entry page selection - avoid complex weighted random for most common case- // First page - entry point - optimize for most common case- const entryPageIndex = Math.floor(Math.random() * 10);- if (entryPageIndex < 3) {- // 30% homepage- currentPath = pagePaths[0]; // homepage- } else if (entryPageIndex < 6) {- // 30% products- currentPath = pagePaths[1]; // products- } else if (entryPageIndex < 8) {- // 20% sale- currentPath = pagePaths[3]; // sale- } else {- // 20% random- currentPath = pagePaths[Math.floor(Math.random() * 15)];- }-- // Run through the pageviews for this session- for (let i = 0; i < numberOfPageviews; i++) {- // Advance time - simplify time calculation- const timeAdvance = 30 + Math.floor(Math.random() * 30);- currentTime = currentTime.plus({ seconds: timeAdvance });-- // Check if we've exceeded session duration- if (i > 0 && currentTime > startTime.plus({ seconds: sessionDuration })) {- break;- }-- // Get querystring - simplified- let querystring = "";- if (Math.random() < 0.2) {- // Reduced probability for speed- if (currentPath.path.startsWith("/product/")) {- // Use most common product variant- querystring = "?variant=medium";- } else if (currentPath.path === "/products") {- // Use most common sort- querystring = "?sort=newest";- }- }-- // Add pageview event - always happens- createEvent(- currentPath.path,- querystring,- currentPath.title,- i === 0 ? initialReferrer : "",- "pageview",- "",- ""- );-- // Update product for product pages- if (currentPath.path.startsWith("/product/")) {- currentProduct = currentPath.product;-- // Add product view event - always on product pages- currentTime = currentTime.plus({ seconds: 2 });-- // Simplified properties creation - avoid building complex objects- const productProps = JSON.stringify({- product_id: currentProduct.id,- product_name: currentProduct.name,- category: currentProduct.category,- price: currentProduct.price,- currency: "USD",- });-- createEvent(- currentPath.path,- querystring,- currentPath.title,- "",- "custom_event",- "product-view",- productProps- );-- // Add to cart (30% chance)- if (Math.random() < 0.3 && !cartItems.includes(currentProduct)) {- currentTime = currentTime.plus({ seconds: 5 });- cartItems.push(currentProduct);-- createEvent(- currentPath.path,- querystring,- currentPath.title,- "",- "custom_event",- "add-to-cart",- JSON.stringify({- product_id: currentProduct.id,- product_name: currentProduct.name,- category: currentProduct.category,- price: currentProduct.price,- quantity: 1,- currency: "USD",- })- );- }- }- // Cart view/checkout - simplified for speed- else if (currentPath.path === "/cart" && cartItems.length > 0) {- // Calculate cart total once- const cartTotal = cartItems- .reduce((sum, item) => sum + parseFloat(item.price), 0)- .toFixed(2);-- createEvent(- currentPath.path,- querystring,- currentPath.title,- "",- "custom_event",- "view-cart",- JSON.stringify({- items_count: cartItems.length,- value: cartTotal,- currency: "USD",- })- );- }- // Purchase flow - simplified for speed- else if (- currentPath.path === "/checkout" &&- cartItems.length > 0 &&- !hasPurchased- ) {- // Calculate cart total once- const cartTotal = cartItems- .reduce((sum, item) => sum + parseFloat(item.price), 0)- .toFixed(2);-- createEvent(- currentPath.path,- querystring,- currentPath.title,- "",- "custom_event",- "begin-checkout",- JSON.stringify({- items_count: cartItems.length,- value: cartTotal,- currency: "USD",- })- );-- // Purchase - simplify condition- if (- (i === numberOfPageviews - 1 || Math.random() < 0.6) &&- !hasPurchased- ) {- currentTime = currentTime.plus({ seconds: 15 });- const transactionId = Math.random()- .toString(36)- .substring(2, 8)- .toUpperCase();-- // Simplified purchase event - avoid creating complex JSON structures- createEvent(- currentPath.path,- querystring,- currentPath.title,- "",- "custom_event",- "purchase",- JSON.stringify({- transaction_id: transactionId,- value: cartTotal,- tax: (parseFloat(cartTotal) * 0.08).toFixed(2),- shipping: "9.99",- currency: "USD",- items_count: cartItems.length,- })- );-- hasPurchased = true;- }- }-- // Simplified navigation logic for speed - use faster random selection- if (i < numberOfPageviews - 1) {- const navRand = Math.random();-- // Faster navigation logic - avoid complex condition checking- if (cartItems.length > 0 && navRand < 0.3) {- // 30% go to cart if items exist- currentPath = pagePaths.find((p) => p.path === "/cart") || pagePaths[0];- } else if (- currentPath.path === "/cart" &&- cartItems.length > 0 &&- navRand < 0.6- ) {- // 30% go to checkout from cart- currentPath =- pagePaths.find((p) => p.path === "/checkout") || pagePaths[0];- } else if (navRand < 0.4) {- // 40% go to product - products are after index 15- const productIndex =- 15 + Math.floor(Math.random() * (pagePaths.length - 15));- currentPath = pagePaths[productIndex];- } else if (navRand < 0.7) {- // 30% go to one of the main pages (indexes 0-5)- currentPath = pagePaths[Math.floor(Math.random() * 6)];- } else {- // 30% random navigation- currentPath = pagePaths[Math.floor(Math.random() * pagePaths.length)];- }- }- }-- // Return only the populated part of the array- return events.slice(0, eventCount);-}--// Function to generate session data (browser, OS, screen resolution, etc.)-function generateSessionData() {- // Select browser and version- const browser = weightedRandom(browsers);- const browserVersion =- browser.versions[Math.floor(Math.random() * browser.versions.length)];-- // Select OS and version- const os = weightedRandom(operatingSystems);- const osVersion = os.versions[Math.floor(Math.random() * os.versions.length)];-- // Select screen resolution- const resolution = weightedRandom(screenResolutions);-- // Generate language code (instead of using faker.locale)- const languageCodes = [- "en",- "es",- "fr",- "de",- "it",- "ru",- "zh",- "ja",- "pt",- "nl",- ];- const language =- languageCodes[Math.floor(Math.random() * languageCodes.length)] +- (Math.random() < 0.5 ? "" : "-" + faker.location.countryCode());-- // Select referrer- const referrer = weightedRandom(referrers).url;-- // Use Faker for location data- const country = faker.location.countryCode();- let region = "";-- // Get region code based on country- if (country === "US") {- region = faker.location.state({ abbreviated: true });- } else {- // For non-US, we'll use a simple region code- region = faker.location.county().slice(0, 3).toUpperCase();- }-- // Determine device type based on screen resolution and OS- let deviceType = "Desktop";- if (os.name === "Android" || os.name === "iOS") {- deviceType = "Mobile";- if (resolution.width > 768) {- deviceType = "Tablet";- }- } else if (resolution.width <= 1024) {- deviceType = "Mobile";- }-- return {- browser: browser.name,- browserVersion: browserVersion,- os: os.name,- osVersion: osVersion,- screenWidth: resolution.width,- screenHeight: resolution.height,- language: language,- referrer: referrer,- country: country,- iso3166: country && region ? `${country}-${region}` : country,- deviceType: deviceType,- };-}--// Function to generate events for a specific day-async function generateEventsForDay(date, targetEventsCount) {- // Instead of one large array, use a collection of batch arrays- const eventBatches = [];- const BATCH_SIZE = 100000; // Store events in manageable chunks- let currentBatch = [];-- // Active sessions map to track ongoing sessions- const activeSessions = new Map();-- // Vary the count by Β±15% to make it more realistic- const variation = Math.random() * 0.3 - 0.15; // -15% to +15%- const actualEventsCount = Math.round(targetEventsCount * (1 + variation));-- console.log(- `Generating ${actualEventsCount.toLocaleString()} events for ${date.toFormat(- "yyyy-MM-dd"- )}`- );-- // Generate 4-10x more events than sessions (averaging 5-6 events per session)- const eventsPerSession = 6;- let eventCount = 0;-- const sessionsToGenerate = Math.ceil(actualEventsCount / eventsPerSession);-- console.log(- `Preparing ${sessionsToGenerate.toLocaleString()} sessions with timestamps...`- );-- // For session data creation - cache common data to avoid recreating for every session- // Pre-generate some session data arrays for faster random selection- const pregenSessionData = {- browsers: Array.from({ length: 1000 }, () => {- const browser = weightedRandom(browsers);- return {- name: browser.name,- version:- browser.versions[Math.floor(Math.random() * browser.versions.length)],- };- }),-- operatingSystems: Array.from({ length: 1000 }, () => {- const os = weightedRandom(operatingSystems);- return {- name: os.name,- version: os.versions[Math.floor(Math.random() * os.versions.length)],- };- }),-- resolutions: Array.from({ length: 100 }, () =>- weightedRandom(screenResolutions)- ),-- referrers: Array.from({ length: 100 }, () => weightedRandom(referrers).url),-- countries: Array.from({ length: 100 }, () => {- const country = faker.location.countryCode();- let region = "";- if (country === "US") {- region = faker.location.state({ abbreviated: true });- } else {- region = faker.location.county().slice(0, 3).toUpperCase();- }- return { country, region };- }),-- languages: Array.from({ length: 100 }, () => {- const languageCodes = [- "en",- "es",- "fr",- "de",- "it",- "ru",- "zh",- "ja",- "pt",- "nl",- ];- return (- languageCodes[Math.floor(Math.random() * languageCodes.length)] +- (Math.random() < 0.5 ? "" : "-" + faker.location.countryCode())- );- }),- };-- // Pre-generate timestamps for the day with realistic distribution in batches- // This reduces memory pressure while maintaining chronological order- const TIMESTAMP_BATCH_SIZE = 100000;- let totalTimestampsGenerated = 0;- let timestampBatch = [];-- // For tracking progress - modified to limit logs to once per second- const startGenerationTime = Date.now();- let lastProgressTime = startGenerationTime;- const MIN_LOG_INTERVAL_MS = 1000; // Minimum 1 second between progress logs-- // For improved ETA calculation- let recentRates = [];- const MAX_RATES_TO_TRACK = 5; // Keep track of the last 5 generation rates-- console.log(`Beginning event generation in optimized batches...`);-- // Faster UUID generation- const fastUUID = () => {- return "xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx".replace(/[xy]/g, (c) => {- const r = (Math.random() * 16) | 0;- return (c === "x" ? r : (r & 0x3) | 0x8).toString(16);- });- };-- // Initial progress message- console.log(- `Progress: 0% | Generated 0 of ${actualEventsCount.toLocaleString()} events | 0 events/sec | Time: 0s | ETA: calculating...`- );-- // Process in batches to optimize memory usage and performance- while (totalTimestampsGenerated < sessionsToGenerate) {- // Generate next batch of timestamps- const batchSize = Math.min(- TIMESTAMP_BATCH_SIZE,- sessionsToGenerate - totalTimestampsGenerated- );- timestampBatch = [];-- // Generate timestamps - using a more optimized approach- const dateJSDate = date.toJSDate();- for (let i = 0; i < batchSize; i++) {- // Simplified time generation - avoid full weightedRandom for every timestamp- const hour = 9 + Math.floor(Math.random() * 12); // Focus on 9am-9pm where most traffic happens- const minute = Math.floor(Math.random() * 60);- const second = Math.floor(Math.random() * 60);-- timestampBatch.push(- DateTime.fromJSDate(dateJSDate).set({ hour, minute, second })- );- }-- // Sort this batch chronologically- timestampBatch.sort((a, b) => a.toMillis() - b.toMillis());-- // Process each timestamp to generate sessions and events- for (let idx = 0; idx < timestampBatch.length; idx++) {- const timestamp = timestampBatch[idx];- const sessionIdx = totalTimestampsGenerated + idx;-- // Check if it's time to show a progress update- const currentTime = Date.now();- const timeSinceLastLog = currentTime - lastProgressTime;-- // Only show a progress update if it's been at least MIN_LOG_INTERVAL_MS since the last one- // Or if this is the last session- if (- timeSinceLastLog >= MIN_LOG_INTERVAL_MS ||- sessionIdx === sessionsToGenerate - 1- ) {- const elapsedSeconds = (currentTime - startGenerationTime) / 1000;-- // Calculate progress based on actual events generated, not sessions- const percentComplete = Math.min(- ((eventCount / actualEventsCount) * 100).toFixed(1),- 100.0- );-- // Format elapsed time- const elapsedFormatted = formatTime(elapsedSeconds);-- // Calculate current generation rate- const currentIntervalRate =- eventCount > 0 ? Math.round(eventCount / elapsedSeconds) : 0;-- // Store recent rates for weighted average calculation- if (timeSinceLastLog > 0) {- // Calculate rate just for this interval- const newEventsInThisInterval =- eventCount -- (recentRates.length > 0- ? eventCount -- (elapsedSeconds - timeSinceLastLog / 1000) * currentIntervalRate- : 0);- const intervalRate = Math.round(- newEventsInThisInterval / (timeSinceLastLog / 1000)- );-- // Add to recent rates, keeping only the most recent ones- recentRates.push(intervalRate);- if (recentRates.length > MAX_RATES_TO_TRACK) {- recentRates.shift(); // Remove oldest rate- }- }-- // Calculate weighted average of recent rates (more weight to recent rates)- let weightedRate = currentIntervalRate;- if (recentRates.length > 0) {- let totalWeight = 0;- let weightedSum = 0;- for (let i = 0; i < recentRates.length; i++) {- const weight = i + 1; // More recent rates get higher weights- weightedSum += recentRates[i] * weight;- totalWeight += weight;- }- weightedRate = Math.round(weightedSum / totalWeight);- }-- console.log(- `Progress: ${percentComplete}% | Generated ${eventCount.toLocaleString()} of ${actualEventsCount.toLocaleString()} events ` +- `| ${weightedRate.toLocaleString()} events/sec | Time: ${elapsedFormatted} | ETA: ${formatETA(- eventCount,- actualEventsCount,- weightedRate,- elapsedSeconds- )}`- );-- lastProgressTime = currentTime;- }-- // Optimized session generation logic- // Decide whether to continue an existing session or create a new one- const now = timestamp.toMillis();- let sessionId, userId, sessionData;-- // First expire old sessions (more than 30 minutes old)- if (activeSessions.size > 100) {- // Only check when we have many active sessions- for (const [sid, sessionInfo] of activeSessions.entries()) {- if (now - sessionInfo.lastActivity > 30 * 60 * 1000) {- activeSessions.delete(sid);- }- }- }-- // 20% chance to continue an existing session if any are active and not too many sessions- if (- activeSessions.size > 0 &&- activeSessions.size < 1000 &&- Math.random() < 0.2- ) {- // Continue an existing session (select one of the first 50 to keep it fast)- const activeSids = Array.from(activeSessions.keys()).slice(0, 50);- sessionId = activeSids[Math.floor(Math.random() * activeSids.length)];- const sessionInfo = activeSessions.get(sessionId);- userId = sessionInfo.userId;- sessionData = sessionInfo.data;-- // Update last activity timestamp- sessionInfo.lastActivity = now;- } else {- // Create a new session- sessionId = fastUUID(); // Using faster UUID generation- userId = userIds[Math.floor(Math.random() * userIds.length)];-- // Faster session data creation using pre-generated arrays- const browserIdx = Math.floor(Math.random() * 1000);- const osIdx = Math.floor(Math.random() * 1000);- const resIdx = Math.floor(Math.random() * 100);- const countryIdx = Math.floor(Math.random() * 100);- const langIdx = Math.floor(Math.random() * 100);- const refIdx = Math.floor(Math.random() * 100);-- const browserInfo = pregenSessionData.browsers[browserIdx];- const osInfo = pregenSessionData.operatingSystems[osIdx];- const resolution = pregenSessionData.resolutions[resIdx];- const { country, region } = pregenSessionData.countries[countryIdx];-- // Determine device type based on screen resolution and OS - simplified- let deviceType = "Desktop";- if (osInfo.name === "Android" || osInfo.name === "iOS") {- deviceType = resolution.width > 768 ? "Tablet" : "Mobile";- } else if (resolution.width <= 1024) {- deviceType = "Mobile";- }-- sessionData = {- browser: browserInfo.name,- browserVersion: browserInfo.version,- os: osInfo.name,- osVersion: osInfo.version,- screenWidth: resolution.width,- screenHeight: resolution.height,- language: pregenSessionData.languages[langIdx],- referrer: pregenSessionData.referrers[refIdx],- country: country,- iso3166: country && region ? `${country}-${region}` : country,- deviceType: deviceType,- };-- // Add to active sessions- activeSessions.set(sessionId, {- userId,- data: sessionData,- lastActivity: now,- });- }-- // Generate events for this session using optimized event generation- const sessionEvents = generateSessionEventsOptimized(- userId,- sessionId,- timestamp,- sessionData- );-- // Add these events to our collection using batch arrays- const newEventCount = sessionEvents.length;-- // Add events to the current batch, creating new batches as needed- for (let i = 0; i < newEventCount; i++) {- // If current batch is full, add it to batches and create a new one- if (currentBatch.length >= BATCH_SIZE) {- eventBatches.push(currentBatch);- currentBatch = [];- }-- // Add event to current batch- currentBatch.push(sessionEvents[i]);- eventCount++;- }-- // Check if we've reached our target event count- if (eventCount >= actualEventsCount) {- break;- }- }-- totalTimestampsGenerated += timestampBatch.length;-- // If we've reached our target event count, break out- if (eventCount >= actualEventsCount) {- break;- }- }-- // Don't forget to add the last batch if it has any events- if (currentBatch.length > 0) {- eventBatches.push(currentBatch);- }-- // Final progress update- const finalGenTime = Date.now();- const totalGenSeconds = ((finalGenTime - startGenerationTime) / 1000).toFixed(- 2- );-- // Flatten all batches into a single array, but only up to the actualEventsCount- // This is more memory-efficient than pre-allocating a huge array- let finalEvents = [];- let remainingEvents = Math.min(eventCount, actualEventsCount);-- for (const batch of eventBatches) {- if (remainingEvents <= 0) break;-- const eventsToTake = Math.min(batch.length, remainingEvents);- finalEvents = finalEvents.concat(batch.slice(0, eventsToTake));- remainingEvents -= eventsToTake;- }-- const generationEndTime = Date.now();- const generationSeconds = (- (generationEndTime - startGenerationTime) /- 1000- ).toFixed(2);- const genEventsPerSecond = Math.round(finalEvents.length / generationSeconds);-- console.log(- `Generated ${finalEvents.length.toLocaleString()} events in ${generationSeconds} seconds ` +- `(${genEventsPerSecond.toLocaleString()} events/sec). Starting insertion...`- );-- // Add timing measurement- const startTime = Date.now();- let totalInserted = 0;-- // Increase batch size for higher throughput- const INSERTION_BATCH_SIZE = 50000; // Increased from 10000 to 50000-- // Create batches- const insertionBatches = [];- for (let i = 0; i < finalEvents.length; i += INSERTION_BATCH_SIZE) {- insertionBatches.push(finalEvents.slice(i, i + INSERTION_BATCH_SIZE));- }-- // Maximum number of parallel inserts (based on CPU cores)- const MAX_PARALLEL = 3; // Using 3 of the 4 cores for parallelism-- // Process batches with controlled parallelism- for (let i = 0; i < insertionBatches.length; i += MAX_PARALLEL) {- const batchPromises = [];-- // Create a set of promises for parallel execution- for (let j = 0; j < MAX_PARALLEL && i + j < insertionBatches.length; j++) {- const batchIndex = i + j;- const batch = insertionBatches[batchIndex];-- batchPromises.push(- (async () => {- const batchStartTime = Date.now();-- try {- await clickhouse.insert({- table: "pageviews",- values: batch,- format: "JSONEachRow",- });-- const batchEndTime = Date.now();- const batchSeconds = (batchEndTime - batchStartTime) / 1000;- const batchSpeed = Math.round(batch.length / batchSeconds);-- return {- batchIndex,- batchSize: batch.length,- batchSpeed,- success: true,- };- } catch (error) {- console.error(`Error inserting batch ${batchIndex + 1}:`, error);- throw error;- }- })()- );- }-- // Wait for this group of batches to complete- try {- const results = await Promise.all(batchPromises);-- // Update metrics- for (const result of results) {- if (result.success) {- totalInserted += result.batchSize;-- const totalElapsed = (Date.now() - startTime) / 1000;- const averageSpeed = Math.round(totalInserted / totalElapsed);-- console.log(- `Inserted batch ${result.batchIndex + 1} of ${- insertionBatches.length- } | ` +- `Batch speed: ${result.batchSpeed.toLocaleString()} events/sec | ` +- `Avg speed: ${averageSpeed.toLocaleString()} events/sec`- );- }- }- } catch (error) {- console.error("Error in batch processing:", error);- throw error;- }- }-- const endTime = Date.now();- const totalSeconds = (endTime - startTime) / 1000;- const overallSpeed = Math.round(finalEvents.length / totalSeconds);-- console.log(- `Day completed in ${totalSeconds.toFixed(2)} seconds | ` +- `Overall insertion speed: ${overallSpeed.toLocaleString()} events/sec | ` +- `CPU cores utilized: ${MAX_PARALLEL} of 4`- );-- return finalEvents.length;-}--// Main function to generate all the data-async function generateMockData() {- let totalEvents = 0;-- // Generate data for each day, starting from the most recent- for (let day = 0; day < daysInPast; day++) {- const date = DateTime.now().minus({ days: day });- const eventsCount = await generateEventsForDay(date, eventsPerDay);- totalEvents += eventsCount;- console.log(- `Completed day ${day + 1} of ${daysInPast}. Total events: ${totalEvents}`- );- }-- console.log(- `Mock data generation complete. Generated ${totalEvents} events across ${daysInPast} days.`- );- process.exit(0);-}--// Start the data generation-generateMockData().catch((error) => {- console.error("Error generating mock data:", error);- process.exit(1);-});diff --git a/mockdata/package-lock.json b/mockdata/package-lock.jsondeleted file mode 100644index 8791f0336..000000000--- a/mockdata/package-lock.json+++ /dev/null@@ -1,74 +0,0 @@-{- "name": "mockdata",- "version": "1.0.0",- "lockfileVersion": 3,- "requires": true,- "packages": {- "": {- "name": "mockdata",- "version": "1.0.0",- "license": "ISC",- "dependencies": {- "@clickhouse/client": "1.11.1",- "@faker-js/faker": "9.7.0",- "dotenv": "16.5.0",- "luxon": "3.6.1"- }- },- "node_modules/@clickhouse/client": {- "version": "1.11.1",- "resolved": "https://registry.npmjs.org/@clickhouse/client/-/client-1.11.1.tgz",- "integrity": "sha512-u9h++h72SmWystijNqfNvMkfA+5+Y1LNfmLL/odCL3VgI3oyAPP9ubSw/Yrt2zRZkLKehMMD1kuOej0QHbSoBA==",- "license": "Apache-2.0",- "dependencies": {- "@clickhouse/client-common": "1.11.1"- },- "engines": {- "node": ">=16"- }- },- "node_modules/@clickhouse/client-common": {- "version": "1.11.1",- "resolved": "https://registry.npmjs.org/@clickhouse/client-common/-/client-common-1.11.1.tgz",- "integrity": "sha512-bme0le2yhDSAh13d2fxhSW5ZrNoVqZ3LTyac8jK6hNH0qkksXnjYkLS6KQalPU6NMpffxHmpI4+/Gi2MnX0NCA==",- "license": "Apache-2.0"- },- "node_modules/@faker-js/faker": {- "version": "9.7.0",- "resolved": "https://registry.npmjs.org/@faker-js/faker/-/faker-9.7.0.tgz",- "integrity": "sha512-aozo5vqjCmDoXLNUJarFZx2IN/GgGaogY4TMJ6so/WLZOWpSV7fvj2dmrV6sEAnUm1O7aCrhTibjpzeDFgNqbg==",- "funding": [- {- "type": "opencollective",- "url": "https://opencollective.com/fakerjs"- }- ],- "license": "MIT",- "engines": {- "node": ">=18.0.0",- "npm": ">=9.0.0"- }- },- "node_modules/dotenv": {- "version": "16.5.0",- "resolved": "https://registry.npmjs.org/dotenv/-/dotenv-16.5.0.tgz",- "integrity": "sha512-m/C+AwOAr9/W1UOIZUo232ejMNnJAJtYQjUbHoNTBNTJSvqzzDh7vnrei3o3r3m9blf6ZoDkvcw0VmozNRFJxg==",- "license": "BSD-2-Clause",- "engines": {- "node": ">=12"- },- "funding": {- "url": "https://dotenvx.com"- }- },- "node_modules/luxon": {- "version": "3.6.1",- "resolved": "https://registry.npmjs.org/luxon/-/luxon-3.6.1.tgz",- "integrity": "sha512-tJLxrKJhO2ukZ5z0gyjY1zPh3Rh88Ej9P7jNrZiHMUXHae1yvI2imgOZtL1TO8TW6biMMKfTtAOoEJANgtWBMQ==",- "license": "MIT",- "engines": {- "node": ">=12"- }- }- }-}diff --git a/mockdata/package.json b/mockdata/package.jsondeleted file mode 100644index c06775a57..000000000--- a/mockdata/package.json+++ /dev/null@@ -1,18 +0,0 @@-{- "name": "mockdata",- "version": "1.0.0",- "main": "index.js",- "scripts": {- "generate": "node index.js",- "test": "echo \"Error: no test specified\" && exit 1"- },- "author": "",- "license": "ISC",- "description": "",- "dependencies": {- "@clickhouse/client": "1.11.1",- "@faker-js/faker": "9.7.0",- "dotenv": "16.5.0",- "luxon": "3.6.1"- }-}