Papermark is the open-source DocSend alternative and secure data rooms with built-in analytics and custom domains.
Implement background export jobs with Trigger.dev for visits data
Cursor Agentd03c318a878e
diff --git a/EXPORT_VISITS_BACKGROUND_TASK.md b/EXPORT_VISITS_BACKGROUND_TASK.mdnew file mode 100644index 00000000..ce6776a7--- /dev/null+++ b/EXPORT_VISITS_BACKGROUND_TASK.md@@ -0,0 +1,182 @@+# Export Visits Background Task System++This document explains the new background task system for exporting visits data, which helps prevent rate limiting issues with the Tinybird API when processing large datasets (200+ views).++## Overview++The export visits functionality has been converted from a synchronous API call to an asynchronous background task system using Trigger.dev v3. This provides several benefits:++- **Rate Limiting**: Tinybird API calls are now rate-limited to 5 concurrent requests with 200ms minimum interval+- **Reliability**: Background tasks have retry logic and can handle failures gracefully+- **Scalability**: Can process large datasets without hitting API timeouts+- **User Experience**: Users get immediate feedback and are notified when exports are ready++## Components++### 1. Background Task (`lib/trigger/export-visits.ts`)+- **Task ID**: `export-visits`+- **Rate Limiting**: Uses Bottleneck to limit Tinybird API calls+- **Retry Logic**: 3 attempts with exponential backoff+- **Timeout**: 15 minutes maximum duration+- **Machine**: Medium-1x preset for more resources++### 2. Database Schema+A new `ExportJob` table has been added to track export jobs:++```sql+CREATE TABLE "ExportJob" (+ "id" TEXT NOT NULL,+ "type" TEXT NOT NULL,+ "status" "ExportStatus" NOT NULL DEFAULT 'PENDING',+ "resourceId" TEXT NOT NULL,+ "resourceName" TEXT,+ "groupId" TEXT,+ "result" TEXT,+ "error" TEXT,+ "userId" TEXT NOT NULL,+ "teamId" TEXT NOT NULL,+ "createdAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,+ "updatedAt" TIMESTAMP(3) NOT NULL,+ "completedAt" TIMESTAMP(3),++ CONSTRAINT "ExportJob_pkey" PRIMARY KEY ("id")+);++CREATE TYPE "ExportStatus" AS ENUM ('PENDING', 'PROCESSING', 'COMPLETED', 'FAILED');+```++### 3. API Endpoints++#### Create Export Job+- **Document**: `POST /api/teams/[teamId]/documents/[id]/export-visits`+- **Dataroom**: `POST /api/teams/[teamId]/datarooms/[id]/export-visits`+- **Dataroom Group**: `POST /api/teams/[teamId]/datarooms/[id]/groups/[groupId]/export-visits`++#### Job Management+- **List Jobs**: `GET /api/teams/[teamId]/export-jobs`+- **Job Status**: `GET /api/teams/[teamId]/export-jobs/[exportId]`+- **Download Result**: `GET /api/teams/[teamId]/export-jobs/[exportId]?download=true`+- **Delete Job**: `DELETE /api/teams/[teamId]/export-jobs/[exportId]`++### 4. Frontend Integration+The frontend components now:+- Trigger background jobs instead of direct API calls+- Poll for job completion every 3 seconds+- Automatically download results when ready+- Show appropriate loading/error states++## Setup Instructions++### 1. Database Migration+Apply the database migration to add the export jobs table:++```bash+npx prisma migrate dev --name add-export-jobs --schema prisma/schema/schema.prisma+```++### 2. Environment Variables+Ensure your Trigger.dev environment variables are set:++```bash+TRIGGER_PROJECT_ID=your_project_id+TRIGGER_ACCESS_TOKEN=your_access_token+```++### 3. Start Trigger.dev Development Server+Run the Trigger.dev development server:++```bash+npm run trigger:v3:dev+```++### 4. Deploy to Production+When ready for production, deploy the Trigger.dev tasks:++```bash+npm run trigger:v3:deploy+```++## Rate Limiting Configuration++The background task uses Bottleneck to limit Tinybird API calls:++```typescript+const tinybirdLimiter = new Bottleneck({+ maxConcurrent: 5, // Maximum 5 concurrent requests+ minTime: 200, // Minimum 200ms between requests+});+```++You can adjust these values based on your Tinybird API limits and requirements.++## Monitoring++Export jobs can be monitored through:+- Database queries on the `ExportJob` table+- Trigger.dev dashboard (if using Trigger.dev Cloud)+- Application logs++## Error Handling++The system handles various error scenarios:+- Tinybird API rate limiting+- Network timeouts+- Data processing errors+- Job failures with retry logic++Failed jobs are marked with status `FAILED` and include error details.++## Usage Examples++### Document Export+```javascript+const response = await fetch(+ `/api/teams/${teamId}/documents/${documentId}/export-visits`,+ { method: "POST" }+);+const { exportId } = await response.json();+```++### Dataroom Export+```javascript+const response = await fetch(+ `/api/teams/${teamId}/datarooms/${dataroomId}/export-visits`,+ { method: "POST" }+);+const { exportId } = await response.json();+```++### Check Job Status+```javascript+const response = await fetch(+ `/api/teams/${teamId}/export-jobs/${exportId}`+);+const jobStatus = await response.json();+```++### Download Results+```javascript+const response = await fetch(+ `/api/teams/${teamId}/export-jobs/${exportId}?download=true`+);+const blob = await response.blob();+```++## Migration Notes++The following endpoints have been changed from GET to POST:+- `/api/teams/[teamId]/documents/[id]/export-visits`+- `/api/teams/[teamId]/datarooms/[id]/export-visits`+- `/api/teams/[teamId]/datarooms/[id]/groups/[groupId]/export-visits`++The frontend components have been updated to use the new background task system with polling for job completion.++## Performance Improvements++With the new system, you should see:+- Reduced Tinybird API rate limiting errors+- Better handling of large datasets (200+ views)+- Improved user experience with background processing+- More reliable exports with retry logic++The system is designed to handle the rate limits gracefully while providing a responsive user experience.\ No newline at end of filediff --git a/components/documents/document-header.tsx b/components/documents/document-header.tsxindex 7be7d6ad..70e71cb0 100644--- a/components/documents/document-header.tsx+++ b/components/documents/document-header.tsx@@ -320,39 +320,82 @@ export default function DocumentHeader({toast.error("This feature is not available for your plan");return;}+try {+ // Trigger the background export jobconst response = await fetch(`/api/teams/${teamId}/documents/${document.id}/export-visits`,- { method: "GET" },+ { method: "POST" },);if (!response.ok) {throw new Error(`HTTP error! status: ${response.status}`);}const data = await response.json();- // Create and download the CSV file- const blob = new Blob([data.visits], { type: "text/csv;charset=utf-8;" });- const url = window.URL.createObjectURL(blob);- const link = window.document.createElement("a");- link.href = url;- link.setAttribute(- "download",- `${data.documentName}_visits_${formattedTime}.csv`,- );- link.rel = "noopener noreferrer";- window.document.body.appendChild(link);- link.click();-- setTimeout(() => {- window.URL.revokeObjectURL(url);- window.document.body.removeChild(link);- }, 100);+ if (data.exportId) {+ toast.success("Export job started. You'll be notified when it's ready.");++ // Poll for export completion+ const pollInterval = setInterval(async () => {+ try {+ const statusResponse = await fetch(+ `/api/teams/${teamId}/export-jobs/${data.exportId}`,+ { method: "GET" },+ );++ if (statusResponse.ok) {+ const statusData = await statusResponse.json();++ if (statusData.status === "COMPLETED" && statusData.isReady) {+ clearInterval(pollInterval);++ // Download the CSV file+ const downloadResponse = await fetch(+ `/api/teams/${teamId}/export-jobs/${data.exportId}?download=true`,+ { method: "GET" },+ );++ if (downloadResponse.ok) {+ const blob = await downloadResponse.blob();+ const url = window.URL.createObjectURL(blob);+ const link = window.document.createElement("a");+ link.href = url;+ link.setAttribute(+ "download",+ `${statusData.resourceName}_visits_${formattedTime}.csv`,+ );+ link.rel = "noopener noreferrer";+ window.document.body.appendChild(link);+ link.click();++ setTimeout(() => {+ window.URL.revokeObjectURL(url);+ window.document.body.removeChild(link);+ }, 100);++ toast.success("CSV file downloaded successfully");+ } else {+ toast.error("Failed to download the export file");+ }+ } else if (statusData.status === "FAILED") {+ clearInterval(pollInterval);+ toast.error(`Export failed: ${statusData.error || "Unknown error"}`);+ }+ }+ } catch (error) {+ console.error("Error polling export status:", error);+ }+ }, 3000); // Poll every 3 seconds- toast.success("CSV file downloaded successfully");+ // Clear interval after 5 minutes to prevent indefinite polling+ setTimeout(() => {+ clearInterval(pollInterval);+ }, 300000);+ }} catch (error) {console.error("Error:", error);toast.error(- "An error occurred while downloading the CSV. Please try again.",+ "An error occurred while starting the export. Please try again.",);}};diff --git a/components/visitors/dataroom-visitors-table.tsx b/components/visitors/dataroom-visitors-table.tsxindex 0e0caf3d..7c7991ec 100644--- a/components/visitors/dataroom-visitors-table.tsx+++ b/components/visitors/dataroom-visitors-table.tsx@@ -51,34 +51,76 @@ export default function DataroomVisitorsTable({const exportVisitCounts = async (dataroomId: string) => {const formattedTime = new Date().toISOString().replace(/[-:Z]/g, "");try {+ // Trigger the background export jobconst response = await fetch(`/api/teams/${teamId}/datarooms/${dataroomId}${groupId ? `/groups/${groupId}` : ""}/export-visits`,- { method: "GET" },+ { method: "POST" },);if (!response.ok) {throw new Error(`HTTP error! status: ${response.status}`);}const data = await response.json();- // Create blob and download- const blob = new Blob([data.visits], { type: "text/csv;charset=utf-8;" });- const url = URL.createObjectURL(blob);- const link = document.createElement("a");- link.href = url;- link.setAttribute(- "download",- `${data.dataroomName}_${name ? `${name}_` : ""}visits_${formattedTime}.csv`,- );- document.body.appendChild(link);- link.click();- document.body.removeChild(link);- URL.revokeObjectURL(url);+ if (data.exportId) {+ toast.success("Export job started. You'll be notified when it's ready.");++ // Poll for export completion+ const pollInterval = setInterval(async () => {+ try {+ const statusResponse = await fetch(+ `/api/teams/${teamId}/export-jobs/${data.exportId}`,+ { method: "GET" },+ );++ if (statusResponse.ok) {+ const statusData = await statusResponse.json();++ if (statusData.status === "COMPLETED" && statusData.isReady) {+ clearInterval(pollInterval);++ // Download the CSV file+ const downloadResponse = await fetch(+ `/api/teams/${teamId}/export-jobs/${data.exportId}?download=true`,+ { method: "GET" },+ );++ if (downloadResponse.ok) {+ const blob = await downloadResponse.blob();+ const url = URL.createObjectURL(blob);+ const link = document.createElement("a");+ link.href = url;+ link.setAttribute(+ "download",+ `${statusData.resourceName}_${name ? `${name}_` : ""}visits_${formattedTime}.csv`,+ );+ document.body.appendChild(link);+ link.click();+ document.body.removeChild(link);+ URL.revokeObjectURL(url);++ toast.success("CSV file downloaded successfully");+ } else {+ toast.error("Failed to download the export file");+ }+ } else if (statusData.status === "FAILED") {+ clearInterval(pollInterval);+ toast.error(`Export failed: ${statusData.error || "Unknown error"}`);+ }+ }+ } catch (error) {+ console.error("Error polling export status:", error);+ }+ }, 3000); // Poll every 3 seconds- toast.success("CSV file downloaded successfully");+ // Clear interval after 5 minutes to prevent indefinite polling+ setTimeout(() => {+ clearInterval(pollInterval);+ }, 300000);+ }} catch (error) {console.error("Error:", error);toast.error(- "An error occurred while downloading the CSV. Please try again.",+ "An error occurred while starting the export. Please try again.",);}};diff --git a/export_jobs_migration.sql b/export_jobs_migration.sqlnew file mode 100644index 00000000..0247c824--- /dev/null+++ b/export_jobs_migration.sql@@ -0,0 +1,37 @@+-- Migration: Add Export Jobs Table+-- This file adds the necessary tables and enums for the export visits background task system++-- Create the ExportStatus enum+CREATE TYPE "ExportStatus" AS ENUM ('PENDING', 'PROCESSING', 'COMPLETED', 'FAILED');++-- Create the ExportJob table+CREATE TABLE "ExportJob" (+ "id" TEXT NOT NULL,+ "type" TEXT NOT NULL,+ "status" "ExportStatus" NOT NULL DEFAULT 'PENDING',+ "resourceId" TEXT NOT NULL,+ "resourceName" TEXT,+ "groupId" TEXT,+ "result" TEXT,+ "error" TEXT,+ "userId" TEXT NOT NULL,+ "teamId" TEXT NOT NULL,+ "createdAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,+ "updatedAt" TIMESTAMP(3) NOT NULL,+ "completedAt" TIMESTAMP(3),++ CONSTRAINT "ExportJob_pkey" PRIMARY KEY ("id")+);++-- Create indexes for better performance+CREATE INDEX "ExportJob_userId_idx" ON "ExportJob"("userId");+CREATE INDEX "ExportJob_teamId_idx" ON "ExportJob"("teamId");+CREATE INDEX "ExportJob_status_idx" ON "ExportJob"("status");+CREATE INDEX "ExportJob_createdAt_idx" ON "ExportJob"("createdAt");++-- Add foreign key constraints+ALTER TABLE "ExportJob" ADD CONSTRAINT "ExportJob_userId_fkey" FOREIGN KEY ("userId") REFERENCES "User"("id") ON DELETE CASCADE ON UPDATE CASCADE;+ALTER TABLE "ExportJob" ADD CONSTRAINT "ExportJob_teamId_fkey" FOREIGN KEY ("teamId") REFERENCES "Team"("id") ON DELETE CASCADE ON UPDATE CASCADE;++-- Update the default timestamp for updatedAt+ALTER TABLE "ExportJob" ALTER COLUMN "updatedAt" SET DEFAULT CURRENT_TIMESTAMP;\ No newline at end of filediff --git a/lib/trigger/export-visits.ts b/lib/trigger/export-visits.tsnew file mode 100644index 00000000..481d4b46--- /dev/null+++ b/lib/trigger/export-visits.ts@@ -0,0 +1,586 @@+import { logger, task } from "@trigger.dev/sdk/v3";+import Bottleneck from "bottleneck";++import prisma from "@/lib/prisma";+import {+ getViewPageDuration,+ getViewUserAgent,+ getViewUserAgent_v2,+} from "@/lib/tinybird";++// Create a bottleneck instance to limit tinybird API calls+const tinybirdLimiter = new Bottleneck({+ maxConcurrent: 5, // Maximum 5 concurrent requests+ minTime: 200, // Minimum 200ms between requests+});++export type ExportVisitsPayload = {+ type: "document" | "dataroom" | "dataroom-group";+ teamId: string;+ resourceId: string; // document ID or dataroom ID+ groupId?: string; // for dataroom groups+ userId: string;+ exportId: string; // unique identifier for this export job+};++export const exportVisitsTask = task({+ id: "export-visits",+ retry: { maxAttempts: 3 },+ maxDuration: 900, // 15 minutes to handle large datasets+ machine: {+ preset: "medium-1x", // More resources for processing+ },+ run: async (payload: ExportVisitsPayload) => {+ const { type, teamId, resourceId, groupId, userId, exportId } = payload;++ logger.info("Starting export visits task", { payload });++ try {+ // Update job status to processing+ await prisma.exportJob.update({+ where: { id: exportId },+ data: { status: "PROCESSING" },+ });++ // Verify team access+ const team = await prisma.team.findUnique({+ where: {+ id: teamId,+ users: {+ some: {+ userId: userId,+ },+ },+ },+ select: { plan: true },+ });++ if (!team) {+ throw new Error("Team not found or access denied");+ }++ if (team.plan === "free") {+ throw new Error("This feature is not available for your plan");+ }++ let csvData: string;+ let resourceName: string;++ if (type === "document") {+ ({ csvData, resourceName } = await exportDocumentVisits(resourceId, teamId));+ } else if (type === "dataroom") {+ ({ csvData, resourceName } = await exportDataroomVisits(resourceId, teamId, groupId));+ } else {+ throw new Error("Invalid export type");+ }++ // Store the result in the database+ await prisma.exportJob.update({+ where: { id: exportId },+ data: {+ status: "COMPLETED",+ result: csvData,+ resourceName,+ completedAt: new Date(),+ },+ });++ logger.info("Export visits task completed successfully", {+ exportId,+ type,+ resourceId,+ csvSize: csvData.length,+ });++ return {+ success: true,+ exportId,+ resourceName,+ csvSize: csvData.length,+ };+ } catch (error) {+ logger.error("Export visits task failed", {+ exportId,+ error: error instanceof Error ? error.message : String(error),+ });++ // Update job status to failed+ await prisma.exportJob.update({+ where: { id: exportId },+ data: {+ status: "FAILED",+ error: error instanceof Error ? error.message : String(error),+ },+ });++ throw error;+ }+ },+});++async function exportDocumentVisits(docId: string, teamId: string): Promise<{+ csvData: string;+ resourceName: string;+}> {+ // Fetch Document+ const document = await prisma.document.findUnique({+ where: { id: docId, teamId: teamId },+ select: {+ id: true,+ name: true,+ numPages: true,+ versions: {+ orderBy: { createdAt: "desc" },+ select: {+ versionNumber: true,+ createdAt: true,+ numPages: true,+ },+ },+ team: {+ select: {+ plan: true,+ },+ },+ },+ });++ if (!document) {+ throw new Error("Document not found");+ }++ // Fetch views+ const views = await prisma.view.findMany({+ where: { documentId: docId },+ include: {+ link: { select: { name: true } },+ agreementResponse: {+ include: {+ agreement: {+ select: {+ name: true,+ content: true,+ },+ },+ },+ },+ customFieldResponse: {+ select: {+ data: true,+ },+ },+ },+ orderBy: {+ viewedAt: "desc",+ },+ });++ if (!views || views.length === 0) {+ throw new Error("Document has no views");+ }++ const isProPlan = document.team.plan.includes("pro");++ // Create CSV rows array starting with headers+ const csvRows: string[] = [];+ const headers = [+ "Viewed at",+ "Name",+ "Email",+ "Link Name",+ "Total Visit Duration (s)",+ "Total Document Completion (%)",+ "Document version",+ "Downloaded at",+ "Verified",+ "Agreement Accepted",+ "Agreement Name",+ "Agreement Content",+ "Agreement Accepted At",+ "Viewed from dataroom",+ "Browser",+ "OS",+ "Device",+ ];++ if (!isProPlan) {+ headers.push("Country", "City", "Custom Fields");+ }++ csvRows.push(headers.join(","));++ // Process views with rate limiting+ logger.info("Processing document views with rate limiting", {+ viewCount: views.length,+ });++ for (let i = 0; i < views.length; i++) {+ const view = views[i];++ logger.info(`Processing view ${i + 1}/${views.length}`, {+ viewId: view.id,+ viewedAt: view.viewedAt,+ });++ // Rate-limited calls to tinybird+ const [duration, userAgentData] = await Promise.all([+ tinybirdLimiter.schedule(() =>+ getViewPageDuration({+ documentId: docId,+ viewId: view.id,+ since: 0,+ }),+ ),+ tinybirdLimiter.schedule(async () => {+ const result = await getViewUserAgent({+ viewId: view.id,+ });++ if (!result || result.rows === 0) {+ return getViewUserAgent_v2({+ documentId: docId,+ viewId: view.id,+ since: 0,+ });+ }++ return result;+ }),+ ]);++ const relevantDocumentVersion = document.versions.find(+ (version) => version.createdAt <= view.viewedAt,+ );++ const numPages =+ relevantDocumentVersion?.numPages || document.numPages || 0;+ const completionRate = numPages+ ? (duration.data.length / numPages) * 100+ : 0;++ const totalDuration = duration.data.reduce(+ (total, data) => total + data.sum_duration,+ 0,+ );++ const rowData = [+ view.viewedAt.toISOString(),+ view.viewerName || "NaN",+ view.viewerEmail || "NaN",+ view.link?.name || "NaN",+ (totalDuration / 1000.0).toFixed(1),+ completionRate.toFixed(2) + "%",+ relevantDocumentVersion?.versionNumber ||+ document.versions[0]?.versionNumber ||+ "NaN",+ view.downloadedAt ? view.downloadedAt.toISOString() : "NaN",+ view.verified ? "Yes" : "No",+ view.agreementResponse ? "Yes" : "NaN",+ view.agreementResponse?.agreement.name || "NaN",+ view.agreementResponse?.agreement.content || "NaN",+ view.agreementResponse?.createdAt.toISOString() || "NaN",+ view.dataroomId ? "Yes" : "No",+ userAgentData?.data[0]?.browser || "NaN",+ userAgentData?.data[0]?.os || "NaN",+ userAgentData?.data[0]?.device || "NaN",+ ];++ if (!isProPlan) {+ rowData.push(+ userAgentData?.data[0]?.country || "NaN",+ userAgentData?.data[0]?.city || "NaN",+ view.customFieldResponse?.data+ ? `"${JSON.stringify(view.customFieldResponse.data).replace(/"/g, '""')}"`+ : "NaN",+ );+ }++ csvRows.push(rowData.join(","));+ }++ return {+ csvData: csvRows.join("\n"),+ resourceName: document.name,+ };+}++async function exportDataroomVisits(+ dataroomId: string,+ teamId: string,+ groupId?: string+): Promise<{+ csvData: string;+ resourceName: string;+}> {+ // Fetch Dataroom+ const dataroom = await prisma.dataroom.findUnique({+ where: { id: dataroomId, teamId: teamId },+ select: {+ id: true,+ name: true,+ documents: {+ select: {+ id: true,+ document: {+ select: {+ name: true,+ numPages: true,+ versions: {+ orderBy: { createdAt: "desc" },+ select: {+ versionNumber: true,+ createdAt: true,+ numPages: true,+ },+ },+ },+ },+ },+ },+ },+ });++ if (!dataroom) {+ throw new Error("Dataroom not found");+ }++ // Fetch views+ const views = await prisma.view.findMany({+ where: {+ dataroomId: dataroomId,+ ...(groupId && { groupId }),+ },+ include: {+ link: { select: { name: true } },+ agreementResponse: {+ include: {+ agreement: {+ select: {+ name: true,+ content: true,+ },+ },+ },+ },+ document: {+ select: {+ id: true,+ name: true,+ numPages: true,+ versions: {+ orderBy: { createdAt: "desc" },+ select: {+ versionNumber: true,+ createdAt: true,+ numPages: true,+ },+ },+ },+ },+ customFieldResponse: {+ select: {+ data: true,+ },+ },+ },+ orderBy: {+ viewedAt: "desc",+ },+ });++ if (!views || views.length === 0) {+ throw new Error("Dataroom has no views");+ }++ // First get all dataroom views+ const dataroomViews = views.filter((view) => view.viewType === "DATAROOM_VIEW");+ const documentViews = views.filter((view) => view.viewType === "DOCUMENT_VIEW");++ logger.info("Processing dataroom views with rate limiting", {+ dataroomViewCount: dataroomViews.length,+ documentViewCount: documentViews.length,+ });++ // Process dataroom views+ const exportData = [];+ for (let i = 0; i < dataroomViews.length; i++) {+ const dataroomView = dataroomViews[i];++ logger.info(`Processing dataroom view ${i + 1}/${dataroomViews.length}`, {+ viewId: dataroomView.id,+ });++ // Find associated document views+ const associatedDocViews = documentViews.filter(+ (docView) => docView.dataroomViewId === dataroomView.id+ );++ // Process document views with rate limiting+ const documentViewDetails = [];+ for (let j = 0; j < associatedDocViews.length; j++) {+ const docView = associatedDocViews[j];++ logger.info(`Processing document view ${j + 1}/${associatedDocViews.length} for dataroom view ${i + 1}`, {+ docViewId: docView.id,+ });++ const duration = await tinybirdLimiter.schedule(() =>+ getViewPageDuration({+ documentId: docView.document?.id || "null",+ viewId: docView.id,+ since: 0,+ })+ );++ const relevantVersion = docView.document?.versions.find(+ (version) => version.createdAt <= docView.viewedAt+ );++ const numPages =+ relevantVersion?.numPages || docView.document?.numPages || 0;+ const completionRate = numPages+ ? (duration.data.length / numPages) * 100+ : 0;++ documentViewDetails.push({+ documentName: docView.document?.name || "NaN",+ viewedAt: docView.viewedAt.toISOString(),+ downloadedAt: docView.downloadedAt?.toISOString() || "NaN",+ duration: duration.data.reduce(+ (total, data) => total + data.sum_duration,+ 0+ ),+ completionRate: completionRate.toFixed(2) + "%",+ documentVersion:+ relevantVersion?.versionNumber ||+ docView.document?.versions[0]?.versionNumber ||+ "NaN",+ viewId: docView.id,+ });+ }++ exportData.push({+ dataroomViewedAt: dataroomView.viewedAt.toISOString(),+ dataroomDownloadedAt: dataroomView.downloadedAt?.toISOString() || "NaN",+ viewerName: dataroomView.viewerName || "NaN",+ viewerEmail: dataroomView.viewerEmail || "NaN",+ linkName: dataroomView.link?.name || "NaN",+ verified: dataroomView.verified ? "Yes" : "NaN",+ agreementStatus: dataroomView.agreementResponse ? "Yes" : "NaN",+ agreementName: dataroomView.agreementResponse?.agreement.name || "NaN",+ agreementAcceptedAt:+ dataroomView.agreementResponse?.createdAt.toISOString() || "NaN",+ agreementContent: dataroomView.agreementResponse?.agreement.content || "NaN",+ documentViews: documentViewDetails,+ });+ }++ // Get user agent data for all document views at once with rate limiting+ const userAgentDataMap = new Map();+ for (const docView of documentViews) {+ const userAgentData = await tinybirdLimiter.schedule(async () => {+ const result = await getViewUserAgent({+ viewId: docView.id,+ });++ if (!result || result.rows === 0) {+ return getViewUserAgent_v2({+ documentId: docView.document?.id || "null",+ viewId: docView.id,+ since: 0,+ });+ }++ return result;+ });++ userAgentDataMap.set(docView.id, userAgentData);+ }++ // Create CSV+ const csvRows: string[] = [];+ csvRows.push([+ "Dataroom Viewed At",+ "Dataroom Downloaded At",+ "Visitor Name",+ "Visitor Email",+ "Link Name",+ "Verified",+ "Agreement Accepted",+ "Agreement Name",+ "Agreement Content",+ "Agreement Accepted At",+ "Document Name",+ "Document Viewed At",+ "Document Downloaded At",+ "Total Visit Duration (s)",+ "Total Document Completion (%)",+ "Document Version",+ "Browser",+ "OS",+ "Device",+ "Country",+ "City",+ ].join(","));++ exportData.forEach((view) => {+ if (view.documentViews.length === 0) {+ csvRows.push([+ view.dataroomViewedAt,+ view.dataroomDownloadedAt,+ view.viewerName,+ view.viewerEmail,+ view.linkName,+ view.verified,+ view.agreementStatus,+ view.agreementName,+ view.agreementContent,+ view.agreementAcceptedAt,+ "NaN",+ "NaN",+ "NaN",+ "NaN",+ "NaN",+ "NaN",+ "NaN",+ "NaN",+ "NaN",+ "NaN",+ "NaN",+ ].join(","));+ } else {+ view.documentViews.forEach((docView) => {+ const userAgentData = userAgentDataMap.get(docView.viewId);++ csvRows.push([+ view.dataroomViewedAt,+ view.dataroomDownloadedAt,+ view.viewerName,+ view.viewerEmail,+ view.linkName,+ view.verified,+ view.agreementStatus,+ view.agreementName,+ view.agreementContent,+ view.agreementAcceptedAt,+ docView.documentName,+ docView.viewedAt,+ docView.downloadedAt,+ (docView.duration / 1000).toFixed(1),+ docView.completionRate,+ docView.documentVersion,+ userAgentData?.data[0]?.browser || "NaN",+ userAgentData?.data[0]?.os || "NaN",+ userAgentData?.data[0]?.device || "NaN",+ userAgentData?.data[0]?.country || "NaN",+ userAgentData?.data[0]?.city || "NaN",+ ].join(","));+ });+ }+ });++ return {+ csvData: csvRows.join("\n"),+ resourceName: dataroom.name,+ };+}\ No newline at end of filediff --git a/pages/api/teams/[teamId]/datarooms/[id]/export-visits.ts b/pages/api/teams/[teamId]/datarooms/[id]/export-visits.tsindex d662c693..89d243d6 100644--- a/pages/api/teams/[teamId]/datarooms/[id]/export-visits.ts+++ b/pages/api/teams/[teamId]/datarooms/[id]/export-visits.ts@@ -4,24 +4,16 @@ import { authOptions } from "@/pages/api/auth/[...nextauth]";import { getServerSession } from "next-auth/next";import prisma from "@/lib/prisma";-import {- getViewPageDuration,- getViewUserAgent,- getViewUserAgent_v2,-} from "@/lib/tinybird";+import { exportVisitsTask } from "@/lib/trigger/export-visits";import { CustomUser } from "@/lib/types";-export const config = {- maxDuration: 180,-};-export default async function handler(req: NextApiRequest,res: NextApiResponse,) {- if (req.method !== "GET") {- // GET /api/teams/:teamId/datarooms/:id/export-visits- res.setHeader("Allow", ["GET"]);+ if (req.method !== "POST") {+ // Changed to POST to trigger background job+ res.setHeader("Allow", ["POST"]);return res.status(405).end(`Method ${req.method} Not Allowed`);}@@ -68,25 +60,6 @@ export default async function handler(select: {id: true,name: true,- documents: {- select: {- id: true,- document: {- select: {- name: true,- numPages: true,- versions: {- orderBy: { createdAt: "desc" },- select: {- versionNumber: true,- createdAt: true,- numPages: true,- },- },- },- },- },- },},});@@ -94,249 +67,44 @@ export default async function handler(return res.status(404).end("Dataroom not found");}- // First fetch the dataroom views- const views = await prisma.view.findMany({- where: {- dataroomId: dataroomId,- },- include: {- link: { select: { name: true } },- agreementResponse: {- include: {- agreement: {- select: {- name: true,- content: true,- },- },- },- },- document: {- select: {- id: true,- name: true,- numPages: true,- versions: {- orderBy: { createdAt: "desc" },- select: {- versionNumber: true,- createdAt: true,- numPages: true,- },- },- },- },- customFieldResponse: {- select: {- data: true,- },- },- },- orderBy: {- viewedAt: "desc",+ // Create export job record+ const exportJob = await prisma.exportJob.create({+ data: {+ type: "dataroom",+ resourceId: dataroomId,+ resourceName: dataroom.name,+ userId,+ teamId,+ status: "PENDING",},});- if (!views || views.length === 0) {- return res.status(404).end("Dataroom has no views");- }-- // First get all dataroom views- const dataroomViews = views.filter(- (view) => view.viewType === "DATAROOM_VIEW",- );-- // Get all document views for this dataroom- const documentViews = views.filter(- (view) => view.viewType === "DOCUMENT_VIEW",- );-- // For each dataroom view, get its associated document views and build export data- const exportData = await Promise.all(- dataroomViews.map(async (dataroomView) => {- // Find all document views that belong to this dataroom view- const associatedDocViews = documentViews.filter(- (docView) => docView.dataroomViewId === dataroomView.id,- );-- // Get document view details including durations from Tinybird- const documentViewDetails = await Promise.all(- associatedDocViews.map(async (docView) => {- const duration = await getViewPageDuration({- documentId: docView.document?.id || "null",- viewId: docView.id,- since: 0,- });-- const relevantVersion = docView.document?.versions.find(- (version) => version.createdAt <= docView.viewedAt,- );-- const numPages =- relevantVersion?.numPages || docView.document?.numPages || 0;- const completionRate = numPages- ? (duration.data.length / numPages) * 100- : 0;-- return {- documentName: docView.document?.name || "NaN",- viewedAt: docView.viewedAt.toISOString(),- downloadedAt: docView.downloadedAt?.toISOString() || "NaN",- duration: duration.data.reduce(- (total, data) => total + data.sum_duration,- 0,- ),- completionRate: completionRate.toFixed(2) + "%",- documentVersion:- relevantVersion?.versionNumber ||- docView.document?.versions[0]?.versionNumber ||- "NaN",- };- }),- );-- // Return the complete view data- return {- // Dataroom view details- dataroomViewedAt: dataroomView.viewedAt.toISOString(),- dataroomDownloadedAt:- dataroomView.downloadedAt?.toISOString() || "NaN",- viewerName: dataroomView.viewerName || "NaN",- viewerEmail: dataroomView.viewerEmail || "NaN",- linkName: dataroomView.link?.name || "NaN",- verified: dataroomView.verified ? "Yes" : "NaN",-- // Agreement details- agreementStatus: dataroomView.agreementResponse ? "Yes" : "NaN",- agreementName:- dataroomView.agreementResponse?.agreement.name || "NaN",- agreementAcceptedAt:- dataroomView.agreementResponse?.createdAt.toISOString() || "NaN",- agreementContent:- dataroomView.agreementResponse?.agreement.content || "NaN",-- // Document view details- documentViews: documentViewDetails,- };- }),- );-- // Create CSV with nested document views- const csvRows: string[] = [];-- // Add headers- csvRows.push(- [- "Dataroom Viewed At",- "Dataroom Downloaded At",- "Visitor Name",- "Visitor Email",- "Link Name",- "Verified",- "Agreement Accepted",- "Agreement Name",- "Agreement Content",- "Agreement Accepted At",- "Document Name",- "Document Viewed At",- "Document Downloaded At",- "Total Visit Duration (s)",- "Total Document Completion (%)",- "Document Version",- "Browser",- "OS",- "Device",- "Country",- "City",- ].join(","),- );-- // Get user agent data for all views- const userAgentData = await Promise.all(- documentViews.map(async (view) => {- const result = await getViewUserAgent({- viewId: view.id,- });-- if (!result || result.rows === 0) {- return getViewUserAgent_v2({- documentId: view.document?.id!,- viewId: view.id,- since: 0,- });- }-- return result;- }),+ // Trigger the background task+ await exportVisitsTask.trigger(+ {+ type: "dataroom",+ teamId,+ resourceId: dataroomId,+ userId,+ exportId: exportJob.id,+ },+ {+ idempotencyKey: exportJob.id,+ tags: [+ `team_${teamId}`,+ `user_${userId}`,+ `export_${exportJob.id}`,+ ],+ },);- // Process each view and add to CSV rows- exportData.forEach((view) => {- if (view.documentViews.length === 0) {- // Add a row for dataroom view without document views- csvRows.push(- [- view.dataroomViewedAt,- view.dataroomDownloadedAt,- view.viewerName,- view.viewerEmail,- view.linkName,- view.verified,- view.agreementStatus,- view.agreementName,- view.agreementContent,- view.agreementAcceptedAt,- "No documents viewed",- "",- "",- "",- "",- "",- "", // Browser- "", // OS- "", // Device- "", // Country- "", // City- ].join(","),- );- } else {- // Add a row for each document view- view.documentViews.forEach((docView, index) => {- csvRows.push(- [- view.dataroomViewedAt,- view.dataroomDownloadedAt,- view.viewerName,- view.viewerEmail,- view.linkName,- view.verified,- view.agreementStatus,- view.agreementName,- view.agreementContent,- view.agreementAcceptedAt,- docView.documentName,- docView.viewedAt,- docView.downloadedAt,- (docView.duration / 1000).toFixed(1),- docView.completionRate,- docView.documentVersion,- userAgentData[index]?.data[0]?.browser || "NaN",- userAgentData[index]?.data[0]?.os || "NaN",- userAgentData[index]?.data[0]?.device || "NaN",- userAgentData[index]?.data[0]?.country || "NaN",- userAgentData[index]?.data[0]?.city || "NaN",- ].join(","),- );- });- }- });-return res.status(200).json({- dataroomName: dataroom.name,- visits: csvRows.join("\n"),+ exportId: exportJob.id,+ status: exportJob.status,+ message: "Export job created successfully. You will be notified when it's ready.",});} catch (error) {- console.error(error);+ console.error("Error creating export job:", error);return res.status(500).json({ message: "Something went wrong" });}}diff --git a/pages/api/teams/[teamId]/datarooms/[id]/groups/[groupId]/export-visits.ts b/pages/api/teams/[teamId]/datarooms/[id]/groups/[groupId]/export-visits.tsindex 94beadca..29efada7 100644--- a/pages/api/teams/[teamId]/datarooms/[id]/groups/[groupId]/export-visits.ts+++ b/pages/api/teams/[teamId]/datarooms/[id]/groups/[groupId]/export-visits.ts@@ -4,24 +4,16 @@ import { authOptions } from "@/pages/api/auth/[...nextauth]";import { getServerSession } from "next-auth/next";import prisma from "@/lib/prisma";-import {- getViewPageDuration,- getViewUserAgent,- getViewUserAgent_v2,-} from "@/lib/tinybird";+import { exportVisitsTask } from "@/lib/trigger/export-visits";import { CustomUser } from "@/lib/types";-export const config = {- maxDuration: 180,-};-export default async function handler(req: NextApiRequest,res: NextApiResponse,) {- if (req.method !== "GET") {- // GET /api/teams/:teamId/datarooms/:id/groups/:groupId/export-visits- res.setHeader("Allow", ["GET"]);+ if (req.method !== "POST") {+ // Changed to POST to trigger background job+ res.setHeader("Allow", ["POST"]);return res.status(405).end(`Method ${req.method} Not Allowed`);}@@ -73,25 +65,6 @@ export default async function handler(select: {id: true,name: true,- documents: {- select: {- id: true,- document: {- select: {- name: true,- numPages: true,- versions: {- orderBy: { createdAt: "desc" },- select: {- versionNumber: true,- createdAt: true,- numPages: true,- },- },- },- },- },- },},});@@ -99,270 +72,46 @@ export default async function handler(return res.status(404).end("Dataroom not found");}- // First fetch the dataroom views- const views = await prisma.view.findMany({- where: {- dataroomId: dataroomId,- groupId: groupId,- },- include: {- link: { select: { name: true } },- agreementResponse: {- include: {- agreement: {- select: {- name: true,- content: true,- },- },- },- },- document: {- select: {- id: true,- name: true,- numPages: true,- versions: {- orderBy: { createdAt: "desc" },- select: {- versionNumber: true,- createdAt: true,- numPages: true,- },- },- },- },- customFieldResponse: {- select: {- data: true,- },- },- },- orderBy: {- viewedAt: "desc",+ // Create export job record+ const exportJob = await prisma.exportJob.create({+ data: {+ type: "dataroom",+ resourceId: dataroomId,+ resourceName: dataroom.name,+ groupId,+ userId,+ teamId,+ status: "PENDING",},});- if (!views || views.length === 0) {- return res.status(404).end("Dataroom has no views");- }-- // First get all dataroom views- const dataroomViews = views.filter(- (view) => view.viewType === "DATAROOM_VIEW",- );-- // Get all document views for this dataroom- const documentViews = views.filter(- (view) => view.viewType === "DOCUMENT_VIEW",- );-- // For each dataroom view, get its associated document views and build export data- const exportData = await Promise.all(- dataroomViews.map(async (dataroomView) => {- // Find all document views that belong to this dataroom view- const associatedDocViews = documentViews.filter(- (docView) => docView.dataroomViewId === dataroomView.id,- );-- // Get document view details including durations from Tinybird- const documentViewDetails = await Promise.all(- associatedDocViews.map(async (docView) => {- const duration = await getViewPageDuration({- documentId: docView.document?.id || "null",- viewId: docView.id,- since: 0,- });-- const relevantVersion = docView.document?.versions.find(- (version) => version.createdAt <= docView.viewedAt,- );-- const numPages =- relevantVersion?.numPages || docView.document?.numPages || 0;- const completionRate = numPages- ? (duration.data.length / numPages) * 100- : 0;-- return {- documentName: docView.document?.name || "NaN",- viewedAt: docView.viewedAt.toISOString(),- downloadedAt: docView.downloadedAt?.toISOString() || "NaN",- duration: duration.data.reduce(- (total, data) => total + data.sum_duration,- 0,- ),- completionRate: completionRate.toFixed(2) + "%",- documentVersion:- relevantVersion?.versionNumber ||- docView.document?.versions[0]?.versionNumber ||- "NaN",- };- }),- );-- // Return the complete view data- return {- // Dataroom view details- dataroomViewedAt: dataroomView.viewedAt.toISOString(),- dataroomDownloadedAt:- dataroomView.downloadedAt?.toISOString() || "NaN",- viewerName: dataroomView.viewerName || "NaN",- viewerEmail: dataroomView.viewerEmail || "NaN",- linkName: dataroomView.link?.name || "NaN",- verified: dataroomView.verified ? "Yes" : "NaN",-- // Agreement details- agreementStatus: dataroomView.agreementResponse ? "Yes" : "NaN",- agreementName:- dataroomView.agreementResponse?.agreement.name || "NaN",- agreementAcceptedAt:- dataroomView.agreementResponse?.createdAt.toISOString() || "NaN",- agreementContent:- dataroomView.agreementResponse?.agreement.content || "NaN",-- // Document view details- documentViews: documentViewDetails,- };- }),- );-- // Create CSV with nested document views- const csvRows: string[] = [];-- // Add headers- csvRows.push(- [- "Dataroom Viewed At",- "Dataroom Downloaded At",- "Visitor Name",- "Visitor Email",- "Link Name",- "Verified",- "Agreement Accepted",- "Agreement Name",- "Agreement Content",- "Agreement Accepted At",- "Document Name",- "Document Viewed At",- "Document Downloaded At",- "Total Visit Duration (s)",- "Total Document Completion (%)",- "Document Version",- "Browser",- "OS",- "Device",- "Country",- "City",- ].join(","),- );-- // Create a map of viewId to userAgent data for efficient lookup- const userAgentDataMap = new Map();-- await Promise.all(- documentViews.map(async (view) => {- const result = await getViewUserAgent({- viewId: view.id,- });-- let userAgentResult;- if (!result || result.rows === 0) {- // Only call v2 if document and its id exist- if (view.document?.id) {- userAgentResult = await getViewUserAgent_v2({- documentId: view.document.id,- viewId: view.id,- since: 0,- });- } else {- // Set default empty result if document/id is missing- userAgentResult = { data: [] };- }- } else {- userAgentResult = result;- }-- userAgentDataMap.set(view.id, userAgentResult);- }),+ // Trigger the background task+ await exportVisitsTask.trigger(+ {+ type: "dataroom",+ teamId,+ resourceId: dataroomId,+ groupId,+ userId,+ exportId: exportJob.id,+ },+ {+ idempotencyKey: exportJob.id,+ tags: [+ `team_${teamId}`,+ `user_${userId}`,+ `export_${exportJob.id}`,+ ],+ },);- // Process each view and add to CSV rows- exportData.forEach((view) => {- if (view.documentViews.length === 0) {- // Add a row for dataroom view without document views- csvRows.push(- [- view.dataroomViewedAt,- view.dataroomDownloadedAt,- view.viewerName,- view.viewerEmail,- view.linkName,- view.verified,- view.agreementStatus,- view.agreementName,- view.agreementContent,- view.agreementAcceptedAt,- "No documents viewed",- "",- "",- "",- "",- "",- "", // Browser- "", // OS- "", // Device- "", // Country- "", // City- ].join(","),- );- } else {- // Add a row for each document view- view.documentViews.forEach((docView) => {- // Find the corresponding document view to get the correct viewId- const correspondingDocView = documentViews.find(- (dv) => dv.viewedAt.toISOString() === docView.viewedAt,- );-- const userAgentData = correspondingDocView- ? userAgentDataMap.get(correspondingDocView.id)- : { data: [] };-- csvRows.push(- [- view.dataroomViewedAt,- view.dataroomDownloadedAt,- view.viewerName,- view.viewerEmail,- view.linkName,- view.verified,- view.agreementStatus,- view.agreementName,- view.agreementContent,- view.agreementAcceptedAt,- docView.documentName,- docView.viewedAt,- docView.downloadedAt,- (docView.duration / 1000).toFixed(1),- docView.completionRate,- docView.documentVersion,- userAgentData?.data[0]?.browser || "NaN",- userAgentData?.data[0]?.os || "NaN",- userAgentData?.data[0]?.device || "NaN",- userAgentData?.data[0]?.country || "NaN",- userAgentData?.data[0]?.city || "NaN",- ].join(","),- );- });- }- });-return res.status(200).json({- dataroomName: dataroom.name,- visits: csvRows.join("\n"),+ exportId: exportJob.id,+ status: exportJob.status,+ message: "Export job created successfully. You will be notified when it's ready.",});} catch (error) {- console.error(error);+ console.error("Error creating export job:", error);return res.status(500).json({ message: "Something went wrong" });}}diff --git a/pages/api/teams/[teamId]/documents/[id]/export-visits.ts b/pages/api/teams/[teamId]/documents/[id]/export-visits.tsindex 812d68aa..799689df 100644--- a/pages/api/teams/[teamId]/documents/[id]/export-visits.ts+++ b/pages/api/teams/[teamId]/documents/[id]/export-visits.ts@@ -4,24 +4,16 @@ import { authOptions } from "@/pages/api/auth/[...nextauth]";import { getServerSession } from "next-auth/next";import prisma from "@/lib/prisma";-import {- getViewPageDuration,- getViewUserAgent,- getViewUserAgent_v2,-} from "@/lib/tinybird";+import { exportVisitsTask } from "@/lib/trigger/export-visits";import { CustomUser } from "@/lib/types";-export const config = {- maxDuration: 180,-};-export default async function handler(req: NextApiRequest,res: NextApiResponse,) {- if (req.method !== "GET") {- // GET /api/teams/:teamId/documents/:id/export-visits- res.setHeader("Allow", ["GET"]);+ if (req.method !== "POST") {+ // Changed to POST to trigger background job+ res.setHeader("Allow", ["POST"]);return res.status(405).end(`Method ${req.method} Not Allowed`);}@@ -65,15 +57,6 @@ export default async function handler(select: {id: true,name: true,- numPages: true,- versions: {- orderBy: { createdAt: "desc" },- select: {- versionNumber: true,- createdAt: true,- numPages: true,- },- },},});@@ -81,152 +64,44 @@ export default async function handler(return res.status(404).end("Document not found");}- // Fetch views data from the database- const views = await prisma.view.findMany({- where: { documentId: docId },- include: {- link: { select: { name: true } },- agreementResponse: {- include: {- agreement: {- select: {- name: true,- content: true,- },- },- },- },- customFieldResponse: {- select: {- data: true,- },- },- },- orderBy: {- viewedAt: "desc",+ // Create export job record+ const exportJob = await prisma.exportJob.create({+ data: {+ type: "document",+ resourceId: docId,+ resourceName: document.name,+ userId,+ teamId,+ status: "PENDING",},});- if (!views || views.length === 0) {- return res.status(404).end("Document has no views");- }-- const isProPlan = team.plan.includes("pro");-- // Create CSV rows array starting with headers- const csvRows: string[] = [];- const headers = [- "Viewed at",- "Name",- "Email",- "Link Name",- "Total Visit Duration (s)",- "Total Document Completion (%)",- "Document version",- "Downloaded at",- "Verified",- "Agreement Accepted",- "Agreement Name",- "Agreement Content",- "Agreement Accepted At",- "Viewed from dataroom",- "Browser",- "OS",- "Device",- ];-- if (!isProPlan) {- headers.push("Country", "City", "Custom Fields");- }-- csvRows.push(headers.join(","));-- // Fetch all durations in parallel- const durations = await Promise.all(- views.map((view) =>- getViewPageDuration({- documentId: docId,- viewId: view.id,- since: 0,- }),- ),- );-- const userAgentData = await Promise.all(- views.map(async (view) => {- const result = await getViewUserAgent({- viewId: view.id,- });-- if (!result || result.rows === 0) {- return getViewUserAgent_v2({- documentId: docId,- viewId: view.id,- since: 0,- });- }-- return result;- }),+ // Trigger the background task+ await exportVisitsTask.trigger(+ {+ type: "document",+ teamId,+ resourceId: docId,+ userId,+ exportId: exportJob.id,+ },+ {+ idempotencyKey: exportJob.id,+ tags: [+ `team_${teamId}`,+ `user_${userId}`,+ `export_${exportJob.id}`,+ ],+ },);- // Process each view and add to CSV rows- views.forEach((view, index) => {- const relevantDocumentVersion = document.versions.find(- (version) => version.createdAt <= view.viewedAt,- );-- const numPages =- relevantDocumentVersion?.numPages || document.numPages || 0;- const completionRate = numPages- ? (durations[index].data.length / numPages) * 100- : 0;-- const totalDuration = durations[index].data.reduce(- (total, data) => total + data.sum_duration,- 0,- );-- const rowData = [- view.viewedAt.toISOString(),- view.viewerName || "NaN",- view.viewerEmail || "NaN",- view.link?.name || "NaN",- (totalDuration / 1000.0).toFixed(1),- completionRate.toFixed(2) + "%",- relevantDocumentVersion?.versionNumber ||- document.versions[0]?.versionNumber ||- "NaN",- view.downloadedAt ? view.downloadedAt.toISOString() : "NaN",- view.verified ? "Yes" : "No",- view.agreementResponse ? "Yes" : "NaN",- view.agreementResponse?.agreement.name || "NaN",- view.agreementResponse?.agreement.content || "NaN",- view.agreementResponse?.createdAt.toISOString() || "NaN",- view.dataroomId ? "Yes" : "No",- userAgentData[index]?.data[0]?.browser || "NaN",- userAgentData[index]?.data[0]?.os || "NaN",- userAgentData[index]?.data[0]?.device || "NaN",- ];-- if (!isProPlan) {- rowData.push(- userAgentData[index]?.data[0]?.country || "NaN",- userAgentData[index]?.data[0]?.city || "NaN",- view.customFieldResponse?.data- ? `"${JSON.stringify(view.customFieldResponse.data).replace(/"/g, '""')}"`- : "NaN",- );- }-- csvRows.push(rowData.join(","));- });return res.status(200).json({- documentName: document.name,- visits: csvRows.join("\n"),+ exportId: exportJob.id,+ status: exportJob.status,+ message: "Export job created successfully. You will be notified when it's ready.",});} catch (error) {- console.error(error);+ console.error("Error creating export job:", error);return res.status(500).json({ message: "Something went wrong" });}}diff --git a/pages/api/teams/[teamId]/export-jobs.ts b/pages/api/teams/[teamId]/export-jobs.tsnew file mode 100644index 00000000..4752165a--- /dev/null+++ b/pages/api/teams/[teamId]/export-jobs.ts@@ -0,0 +1,125 @@+import { NextApiRequest, NextApiResponse } from "next";+import { nanoid } from "nanoid";++import { authOptions } from "@/pages/api/auth/[...nextauth]";+import { getServerSession } from "next-auth/next";++import { exportVisitsTask } from "@/lib/trigger/export-visits";+import prisma from "@/lib/prisma";+import { CustomUser } from "@/lib/types";++export default async function handler(+ req: NextApiRequest,+ res: NextApiResponse,+) {+ const session = await getServerSession(req, res, authOptions);+ if (!session) {+ return res.status(401).end("Unauthorized");+ }++ const { teamId } = req.query as { teamId: string };+ const userId = (session.user as CustomUser).id;++ if (req.method === "POST") {+ // Trigger a new export job+ const { type, resourceId, groupId } = req.body as {+ type: "document" | "dataroom" | "dataroom-group";+ resourceId: string;+ groupId?: string;+ };++ if (!type || !resourceId) {+ return res.status(400).json({ error: "Missing required fields" });+ }++ try {+ // Verify team access+ const team = await prisma.team.findUnique({+ where: {+ id: teamId,+ users: {+ some: {+ userId: userId,+ },+ },+ },+ select: { plan: true },+ });++ if (!team) {+ return res.status(404).json({ error: "Team not found" });+ }++ if (team.plan === "free") {+ return res.status(403).json({+ error: "This feature is not available for your plan"+ });+ }++ // Create export job record+ const exportJob = await prisma.exportJob.create({+ data: {+ type,+ resourceId,+ groupId,+ userId,+ teamId,+ status: "PENDING",+ },+ });++ // Trigger the background task+ await exportVisitsTask.trigger(+ {+ type,+ teamId,+ resourceId,+ groupId,+ userId,+ exportId: exportJob.id,+ },+ {+ idempotencyKey: exportJob.id,+ tags: [+ `team_${teamId}`,+ `user_${userId}`,+ `export_${exportJob.id}`,+ ],+ },+ );++ return res.status(200).json({+ exportId: exportJob.id,+ status: exportJob.status,+ message: "Export job created successfully",+ });+ } catch (error) {+ console.error("Error creating export job:", error);+ return res.status(500).json({ error: "Failed to create export job" });+ }+ }++ if (req.method === "GET") {+ // Get export jobs for the team+ try {+ const exportJobs = await prisma.exportJob.findMany({+ where: {+ teamId,+ userId,+ },+ orderBy: {+ createdAt: "desc",+ },+ take: 20, // Limit to last 20 jobs+ });++ return res.status(200).json(exportJobs);+ } catch (error) {+ console.error("Error fetching export jobs:", error);+ return res.status(500).json({ error: "Failed to fetch export jobs" });+ }+ }++ res.setHeader("Allow", ["POST", "GET"]);+ return res.status(405).end(`Method ${req.method} Not Allowed`);+}\ No newline at end of filediff --git a/pages/api/teams/[teamId]/export-jobs/[exportId].ts b/pages/api/teams/[teamId]/export-jobs/[exportId].tsnew file mode 100644index 00000000..9cbe53c1--- /dev/null+++ b/pages/api/teams/[teamId]/export-jobs/[exportId].ts@@ -0,0 +1,95 @@+import { NextApiRequest, NextApiResponse } from "next";++import { authOptions } from "@/pages/api/auth/[...nextauth]";+import { getServerSession } from "next-auth/next";++import prisma from "@/lib/prisma";+import { CustomUser } from "@/lib/types";++export default async function handler(+ req: NextApiRequest,+ res: NextApiResponse,+) {+ const session = await getServerSession(req, res, authOptions);+ if (!session) {+ return res.status(401).end("Unauthorized");+ }++ const { teamId, exportId } = req.query as {+ teamId: string;+ exportId: string;+ };+ const userId = (session.user as CustomUser).id;++ if (req.method === "GET") {+ try {+ // Get export job details+ const exportJob = await prisma.exportJob.findUnique({+ where: {+ id: exportId,+ teamId,+ userId,+ },+ });++ if (!exportJob) {+ return res.status(404).json({ error: "Export job not found" });+ }++ // Check if client wants to download the CSV+ const { download } = req.query;+ if (download === "true" && exportJob.status === "COMPLETED" && exportJob.result) {+ const timestamp = new Date().toISOString().replace(/[-:Z]/g, "");+ const filename = `${exportJob.resourceName || "export"}_visits_${timestamp}.csv`;++ res.setHeader("Content-Type", "text/csv");+ res.setHeader(+ "Content-Disposition",+ `attachment; filename="${filename}"`+ );+ res.setHeader("Content-Length", Buffer.byteLength(exportJob.result));++ return res.status(200).send(exportJob.result);+ }++ // Return job status+ return res.status(200).json({+ id: exportJob.id,+ type: exportJob.type,+ status: exportJob.status,+ resourceId: exportJob.resourceId,+ resourceName: exportJob.resourceName,+ groupId: exportJob.groupId,+ error: exportJob.error,+ createdAt: exportJob.createdAt,+ updatedAt: exportJob.updatedAt,+ completedAt: exportJob.completedAt,+ isReady: exportJob.status === "COMPLETED" && !!exportJob.result,+ });+ } catch (error) {+ console.error("Error fetching export job:", error);+ return res.status(500).json({ error: "Failed to fetch export job" });+ }+ }++ if (req.method === "DELETE") {+ try {+ // Delete export job+ await prisma.exportJob.delete({+ where: {+ id: exportId,+ teamId,+ userId,+ },+ });++ return res.status(200).json({ message: "Export job deleted successfully" });+ } catch (error) {+ console.error("Error deleting export job:", error);+ return res.status(500).json({ error: "Failed to delete export job" });+ }+ }++ res.setHeader("Allow", ["GET", "DELETE"]);+ return res.status(405).end(`Method ${req.method} Not Allowed`);+}\ No newline at end of filediff --git a/prisma/schema/schema.prisma b/prisma/schema/schema.prismaindex ee594ee7..82ec6596 100644--- a/prisma/schema/schema.prisma+++ b/prisma/schema/schema.prisma@@ -62,6 +62,9 @@ model User {// conversationparticipatedConversations ConversationParticipant[]messages Message[]++ // export jobs+ exportJobs ExportJob[]}model Team {@@ -98,6 +101,9 @@ model Team {restrictedTokens RestrictedToken[]webhooks Webhook[]+ // export jobs+ exportJobs ExportJob[]+conversations Conversation[]uploadedDocuments DocumentUpload[]@@ -132,6 +138,13 @@ enum Role {MEMBER}+enum ExportStatus {+ PENDING+ PROCESSING+ COMPLETED+ FAILED+}+model UserTeam {role Role @default(MEMBER)status String @default("ACTIVE")@@ -642,3 +655,29 @@ model TagItem {@@index([tagId, documentId])@@index([tagId, dataroomId])}++model ExportJob {+ id String @id @default(cuid())+ type String // "document" | "dataroom" | "dataroom-group"+ status ExportStatus @default(PENDING)+ resourceId String // document ID or dataroom ID+ resourceName String? // document name or dataroom name+ groupId String? // for dataroom groups+ result String? @db.Text // CSV data+ error String? // error message if failed++ userId String+ user User @relation(fields: [userId], references: [id], onDelete: Cascade)++ teamId String+ team Team @relation(fields: [teamId], references: [id], onDelete: Cascade)++ createdAt DateTime @default(now())+ updatedAt DateTime @updatedAt+ completedAt DateTime?++ @@index([userId])+ @@index([teamId])+ @@index([status])+ @@index([createdAt])+}