Papermark is the open-source DocSend alternative and secure data rooms with built-in analytics and custom domains.
chore: remove
Marc Seitz02a505b9601e
diff --git a/EXPORT_VISITS_BACKGROUND_TASK.md b/EXPORT_VISITS_BACKGROUND_TASK.mddeleted file mode 100644index 9df9832d..00000000--- a/EXPORT_VISITS_BACKGROUND_TASK.md+++ /dev/null@@ -1,180 +0,0 @@-# Export Visits Background Task System--This document explains the new background task system for exporting visits data, which helps prevent rate limiting issues with the Tinybird API when processing large datasets (200+ views).--## Overview--The export visits functionality has been converted from a synchronous API call to an asynchronous background task system using Trigger.dev v3. This provides several benefits:--- **Rate Limiting**: Tinybird API calls are now rate-limited to 5 concurrent requests with 200ms minimum interval-- **Reliability**: Background tasks have retry logic and can handle failures gracefully-- **Scalability**: Can process large datasets without hitting API timeouts-- **User Experience**: Users get immediate feedback and are notified when exports are ready--## Components--### 1. Background Task (`lib/trigger/export-visits.ts`)-- **Task ID**: `export-visits`-- **Rate Limiting**: Uses Bottleneck to limit Tinybird API calls-- **Retry Logic**: 3 attempts with exponential backoff-- **Timeout**: 15 minutes maximum duration-- **Machine**: Medium-1x preset for more resources--### 2. Redis Job Storage-Export jobs are stored in Redis for fast access and automatic cleanup:--```typescript-interface ExportJob {- id: string;- type: "document" | "dataroom" | "dataroom-group";- status: "PENDING" | "PROCESSING" | "COMPLETED" | "FAILED";- resourceId: string;- resourceName?: string;- groupId?: string;- result?: string; // CSV data- error?: string;- userId: string;- teamId: string;- createdAt: string;- updatedAt: string;- completedAt?: string;-}-```--**Benefits of Redis storage:**-- **Fast access**: Redis is much faster than database queries-- **Automatic cleanup**: Jobs expire after 7 days (configurable)-- **No database migrations**: Uses existing Redis infrastructure-- **Sorted job lists**: Jobs are automatically sorted by creation time--### 3. API Endpoints--#### Create Export Job-- **Document**: `POST /api/teams/[teamId]/documents/[id]/export-visits`-- **Dataroom**: `POST /api/teams/[teamId]/datarooms/[id]/export-visits`-- **Dataroom Group**: `POST /api/teams/[teamId]/datarooms/[id]/groups/[groupId]/export-visits`--#### Job Management-- **List Jobs**: `GET /api/teams/[teamId]/export-jobs`-- **Job Status**: `GET /api/teams/[teamId]/export-jobs/[exportId]`-- **Download Result**: `GET /api/teams/[teamId]/export-jobs/[exportId]?download=true`-- **Delete Job**: `DELETE /api/teams/[teamId]/export-jobs/[exportId]`--### 4. Frontend Integration-The frontend components now:-- Trigger background jobs instead of direct API calls-- Poll for job completion every 3 seconds-- Automatically download results when ready-- Show appropriate loading/error states--## Setup Instructions--### 1. Environment Variables-Ensure your Trigger.dev environment variables are set:--```bash-TRIGGER_PROJECT_ID=your_project_id-TRIGGER_ACCESS_TOKEN=your_access_token-```--### 2. Start Trigger.dev Development Server-Run the Trigger.dev development server:--```bash-npm run trigger:v3:dev-```--### 3. Deploy to Production-When ready for production, deploy the Trigger.dev tasks:--```bash-npm run trigger:v3:deploy-```--## Rate Limiting Configuration--The background task uses Bottleneck to limit Tinybird API calls:--```typescript-const tinybirdLimiter = new Bottleneck({- maxConcurrent: 5, // Maximum 5 concurrent requests- minTime: 200, // Minimum 200ms between requests-});-```--You can adjust these values based on your Tinybird API limits and requirements.--## Monitoring--Export jobs can be monitored through:-- Redis commands to inspect job data (`HGETALL export_job:*`)-- API endpoints for job listing and status-- Trigger.dev dashboard (if using Trigger.dev Cloud)-- Application logs--## Error Handling--The system handles various error scenarios:-- Tinybird API rate limiting-- Network timeouts-- Data processing errors-- Job failures with retry logic--Failed jobs are marked with status `FAILED` and include error details.--## Usage Examples--### Document Export-```javascript-const response = await fetch(- `/api/teams/${teamId}/documents/${documentId}/export-visits`,- { method: "POST" }-);-const { exportId } = await response.json();-```--### Dataroom Export-```javascript-const response = await fetch(- `/api/teams/${teamId}/datarooms/${dataroomId}/export-visits`,- { method: "POST" }-);-const { exportId } = await response.json();-```--### Check Job Status-```javascript-const response = await fetch(- `/api/teams/${teamId}/export-jobs/${exportId}`-);-const jobStatus = await response.json();-```--### Download Results-```javascript-const response = await fetch(- `/api/teams/${teamId}/export-jobs/${exportId}?download=true`-);-const blob = await response.blob();-```--## Migration Notes--**No database migrations required!** This solution uses Redis for job storage.--The following endpoints have been changed from GET to POST:-- `/api/teams/[teamId]/documents/[id]/export-visits`-- `/api/teams/[teamId]/datarooms/[id]/export-visits`-- `/api/teams/[teamId]/datarooms/[id]/groups/[groupId]/export-visits`--The frontend components have been updated to use the new background task system with polling for job completion.--## Performance Improvements--With the new system, you should see:-- Reduced Tinybird API rate limiting errors-- Better handling of large datasets (200+ views)-- Improved user experience with background processing-- More reliable exports with retry logic--The system is designed to handle the rate limits gracefully while providing a responsive user experience.\ No newline at end of file