Skip to content

Papermark is the open-source DocSend alternative and secure data rooms with built-in analytics and custom domains.

Implement background export jobs with Trigger.dev for visits data

Cursor Agentd03c318a878e
diff --git a/EXPORT_VISITS_BACKGROUND_TASK.md b/EXPORT_VISITS_BACKGROUND_TASK.md
new file mode 100644
index 00000000..ce6776a7
--- /dev/null
+++ b/EXPORT_VISITS_BACKGROUND_TASK.md
@@ -0,0 +1,182 @@
+# Export Visits Background Task System
+
+This document explains the new background task system for exporting visits data, which helps prevent rate limiting issues with the Tinybird API when processing large datasets (200+ views).
+
+## Overview
+
+The export visits functionality has been converted from a synchronous API call to an asynchronous background task system using Trigger.dev v3. This provides several benefits:
+
+- **Rate Limiting**: Tinybird API calls are now rate-limited to 5 concurrent requests with 200ms minimum interval
+- **Reliability**: Background tasks have retry logic and can handle failures gracefully
+- **Scalability**: Can process large datasets without hitting API timeouts
+- **User Experience**: Users get immediate feedback and are notified when exports are ready
+
+## Components
+
+### 1. Background Task (`lib/trigger/export-visits.ts`)
+- **Task ID**: `export-visits`
+- **Rate Limiting**: Uses Bottleneck to limit Tinybird API calls
+- **Retry Logic**: 3 attempts with exponential backoff
+- **Timeout**: 15 minutes maximum duration
+- **Machine**: Medium-1x preset for more resources
+
+### 2. Database Schema
+A new `ExportJob` table has been added to track export jobs:
+
+```sql
+CREATE TABLE "ExportJob" (
+ "id" TEXT NOT NULL,
+ "type" TEXT NOT NULL,
+ "status" "ExportStatus" NOT NULL DEFAULT 'PENDING',
+ "resourceId" TEXT NOT NULL,
+ "resourceName" TEXT,
+ "groupId" TEXT,
+ "result" TEXT,
+ "error" TEXT,
+ "userId" TEXT NOT NULL,
+ "teamId" TEXT NOT NULL,
+ "createdAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
+ "updatedAt" TIMESTAMP(3) NOT NULL,
+ "completedAt" TIMESTAMP(3),
+
+ CONSTRAINT "ExportJob_pkey" PRIMARY KEY ("id")
+);
+
+CREATE TYPE "ExportStatus" AS ENUM ('PENDING', 'PROCESSING', 'COMPLETED', 'FAILED');
+```
+
+### 3. API Endpoints
+
+#### Create Export Job
+- **Document**: `POST /api/teams/[teamId]/documents/[id]/export-visits`
+- **Dataroom**: `POST /api/teams/[teamId]/datarooms/[id]/export-visits`
+- **Dataroom Group**: `POST /api/teams/[teamId]/datarooms/[id]/groups/[groupId]/export-visits`
+
+#### Job Management
+- **List Jobs**: `GET /api/teams/[teamId]/export-jobs`
+- **Job Status**: `GET /api/teams/[teamId]/export-jobs/[exportId]`
+- **Download Result**: `GET /api/teams/[teamId]/export-jobs/[exportId]?download=true`
+- **Delete Job**: `DELETE /api/teams/[teamId]/export-jobs/[exportId]`
+
+### 4. Frontend Integration
+The frontend components now:
+- Trigger background jobs instead of direct API calls
+- Poll for job completion every 3 seconds
+- Automatically download results when ready
+- Show appropriate loading/error states
+
+## Setup Instructions
+
+### 1. Database Migration
+Apply the database migration to add the export jobs table:
+
+```bash
+npx prisma migrate dev --name add-export-jobs --schema prisma/schema/schema.prisma
+```
+
+### 2. Environment Variables
+Ensure your Trigger.dev environment variables are set:
+
+```bash
+TRIGGER_PROJECT_ID=your_project_id
+TRIGGER_ACCESS_TOKEN=your_access_token
+```
+
+### 3. Start Trigger.dev Development Server
+Run the Trigger.dev development server:
+
+```bash
+npm run trigger:v3:dev
+```
+
+### 4. Deploy to Production
+When ready for production, deploy the Trigger.dev tasks:
+
+```bash
+npm run trigger:v3:deploy
+```
+
+## Rate Limiting Configuration
+
+The background task uses Bottleneck to limit Tinybird API calls:
+
+```typescript
+const tinybirdLimiter = new Bottleneck({
+ maxConcurrent: 5, // Maximum 5 concurrent requests
+ minTime: 200, // Minimum 200ms between requests
+});
+```
+
+You can adjust these values based on your Tinybird API limits and requirements.
+
+## Monitoring
+
+Export jobs can be monitored through:
+- Database queries on the `ExportJob` table
+- Trigger.dev dashboard (if using Trigger.dev Cloud)
+- Application logs
+
+## Error Handling
+
+The system handles various error scenarios:
+- Tinybird API rate limiting
+- Network timeouts
+- Data processing errors
+- Job failures with retry logic
+
+Failed jobs are marked with status `FAILED` and include error details.
+
+## Usage Examples
+
+### Document Export
+```javascript
+const response = await fetch(
+ `/api/teams/${teamId}/documents/${documentId}/export-visits`,
+ { method: "POST" }
+);
+const { exportId } = await response.json();
+```
+
+### Dataroom Export
+```javascript
+const response = await fetch(
+ `/api/teams/${teamId}/datarooms/${dataroomId}/export-visits`,
+ { method: "POST" }
+);
+const { exportId } = await response.json();
+```
+
+### Check Job Status
+```javascript
+const response = await fetch(
+ `/api/teams/${teamId}/export-jobs/${exportId}`
+);
+const jobStatus = await response.json();
+```
+
+### Download Results
+```javascript
+const response = await fetch(
+ `/api/teams/${teamId}/export-jobs/${exportId}?download=true`
+);
+const blob = await response.blob();
+```
+
+## Migration Notes
+
+The following endpoints have been changed from GET to POST:
+- `/api/teams/[teamId]/documents/[id]/export-visits`
+- `/api/teams/[teamId]/datarooms/[id]/export-visits`
+- `/api/teams/[teamId]/datarooms/[id]/groups/[groupId]/export-visits`
+
+The frontend components have been updated to use the new background task system with polling for job completion.
+
+## Performance Improvements
+
+With the new system, you should see:
+- Reduced Tinybird API rate limiting errors
+- Better handling of large datasets (200+ views)
+- Improved user experience with background processing
+- More reliable exports with retry logic
+
+The system is designed to handle the rate limits gracefully while providing a responsive user experience.
\ No newline at end of file
diff --git a/components/documents/document-header.tsx b/components/documents/document-header.tsx
index 7be7d6ad..70e71cb0 100644
--- a/components/documents/document-header.tsx
+++ b/components/documents/document-header.tsx
@@ -320,39 +320,82 @@ export default function DocumentHeader({
toast.error("This feature is not available for your plan");
return;
}
+
try {
+ // Trigger the background export job
const response = await fetch(
`/api/teams/${teamId}/documents/${document.id}/export-visits`,
- { method: "GET" },
+ { method: "POST" },
);
if (!response.ok) {
throw new Error(`HTTP error! status: ${response.status}`);
}
const data = await response.json();
- // Create and download the CSV file
- const blob = new Blob([data.visits], { type: "text/csv;charset=utf-8;" });
- const url = window.URL.createObjectURL(blob);
- const link = window.document.createElement("a");
- link.href = url;
- link.setAttribute(
- "download",
- `${data.documentName}_visits_${formattedTime}.csv`,
- );
- link.rel = "noopener noreferrer";
- window.document.body.appendChild(link);
- link.click();
-
- setTimeout(() => {
- window.URL.revokeObjectURL(url);
- window.document.body.removeChild(link);
- }, 100);
+ if (data.exportId) {
+ toast.success("Export job started. You'll be notified when it's ready.");
+
+ // Poll for export completion
+ const pollInterval = setInterval(async () => {
+ try {
+ const statusResponse = await fetch(
+ `/api/teams/${teamId}/export-jobs/${data.exportId}`,
+ { method: "GET" },
+ );
+
+ if (statusResponse.ok) {
+ const statusData = await statusResponse.json();
+
+ if (statusData.status === "COMPLETED" && statusData.isReady) {
+ clearInterval(pollInterval);
+
+ // Download the CSV file
+ const downloadResponse = await fetch(
+ `/api/teams/${teamId}/export-jobs/${data.exportId}?download=true`,
+ { method: "GET" },
+ );
+
+ if (downloadResponse.ok) {
+ const blob = await downloadResponse.blob();
+ const url = window.URL.createObjectURL(blob);
+ const link = window.document.createElement("a");
+ link.href = url;
+ link.setAttribute(
+ "download",
+ `${statusData.resourceName}_visits_${formattedTime}.csv`,
+ );
+ link.rel = "noopener noreferrer";
+ window.document.body.appendChild(link);
+ link.click();
+
+ setTimeout(() => {
+ window.URL.revokeObjectURL(url);
+ window.document.body.removeChild(link);
+ }, 100);
+
+ toast.success("CSV file downloaded successfully");
+ } else {
+ toast.error("Failed to download the export file");
+ }
+ } else if (statusData.status === "FAILED") {
+ clearInterval(pollInterval);
+ toast.error(`Export failed: ${statusData.error || "Unknown error"}`);
+ }
+ }
+ } catch (error) {
+ console.error("Error polling export status:", error);
+ }
+ }, 3000); // Poll every 3 seconds
- toast.success("CSV file downloaded successfully");
+ // Clear interval after 5 minutes to prevent indefinite polling
+ setTimeout(() => {
+ clearInterval(pollInterval);
+ }, 300000);
+ }
} catch (error) {
console.error("Error:", error);
toast.error(
- "An error occurred while downloading the CSV. Please try again.",
+ "An error occurred while starting the export. Please try again.",
);
}
};
diff --git a/components/visitors/dataroom-visitors-table.tsx b/components/visitors/dataroom-visitors-table.tsx
index 0e0caf3d..7c7991ec 100644
--- a/components/visitors/dataroom-visitors-table.tsx
+++ b/components/visitors/dataroom-visitors-table.tsx
@@ -51,34 +51,76 @@ export default function DataroomVisitorsTable({
const exportVisitCounts = async (dataroomId: string) => {
const formattedTime = new Date().toISOString().replace(/[-:Z]/g, "");
try {
+ // Trigger the background export job
const response = await fetch(
`/api/teams/${teamId}/datarooms/${dataroomId}${groupId ? `/groups/${groupId}` : ""}/export-visits`,
- { method: "GET" },
+ { method: "POST" },
);
if (!response.ok) {
throw new Error(`HTTP error! status: ${response.status}`);
}
const data = await response.json();
- // Create blob and download
- const blob = new Blob([data.visits], { type: "text/csv;charset=utf-8;" });
- const url = URL.createObjectURL(blob);
- const link = document.createElement("a");
- link.href = url;
- link.setAttribute(
- "download",
- `${data.dataroomName}_${name ? `${name}_` : ""}visits_${formattedTime}.csv`,
- );
- document.body.appendChild(link);
- link.click();
- document.body.removeChild(link);
- URL.revokeObjectURL(url);
+ if (data.exportId) {
+ toast.success("Export job started. You'll be notified when it's ready.");
+
+ // Poll for export completion
+ const pollInterval = setInterval(async () => {
+ try {
+ const statusResponse = await fetch(
+ `/api/teams/${teamId}/export-jobs/${data.exportId}`,
+ { method: "GET" },
+ );
+
+ if (statusResponse.ok) {
+ const statusData = await statusResponse.json();
+
+ if (statusData.status === "COMPLETED" && statusData.isReady) {
+ clearInterval(pollInterval);
+
+ // Download the CSV file
+ const downloadResponse = await fetch(
+ `/api/teams/${teamId}/export-jobs/${data.exportId}?download=true`,
+ { method: "GET" },
+ );
+
+ if (downloadResponse.ok) {
+ const blob = await downloadResponse.blob();
+ const url = URL.createObjectURL(blob);
+ const link = document.createElement("a");
+ link.href = url;
+ link.setAttribute(
+ "download",
+ `${statusData.resourceName}_${name ? `${name}_` : ""}visits_${formattedTime}.csv`,
+ );
+ document.body.appendChild(link);
+ link.click();
+ document.body.removeChild(link);
+ URL.revokeObjectURL(url);
+
+ toast.success("CSV file downloaded successfully");
+ } else {
+ toast.error("Failed to download the export file");
+ }
+ } else if (statusData.status === "FAILED") {
+ clearInterval(pollInterval);
+ toast.error(`Export failed: ${statusData.error || "Unknown error"}`);
+ }
+ }
+ } catch (error) {
+ console.error("Error polling export status:", error);
+ }
+ }, 3000); // Poll every 3 seconds
- toast.success("CSV file downloaded successfully");
+ // Clear interval after 5 minutes to prevent indefinite polling
+ setTimeout(() => {
+ clearInterval(pollInterval);
+ }, 300000);
+ }
} catch (error) {
console.error("Error:", error);
toast.error(
- "An error occurred while downloading the CSV. Please try again.",
+ "An error occurred while starting the export. Please try again.",
);
}
};
diff --git a/export_jobs_migration.sql b/export_jobs_migration.sql
new file mode 100644
index 00000000..0247c824
--- /dev/null
+++ b/export_jobs_migration.sql
@@ -0,0 +1,37 @@
+-- Migration: Add Export Jobs Table
+-- This file adds the necessary tables and enums for the export visits background task system
+
+-- Create the ExportStatus enum
+CREATE TYPE "ExportStatus" AS ENUM ('PENDING', 'PROCESSING', 'COMPLETED', 'FAILED');
+
+-- Create the ExportJob table
+CREATE TABLE "ExportJob" (
+ "id" TEXT NOT NULL,
+ "type" TEXT NOT NULL,
+ "status" "ExportStatus" NOT NULL DEFAULT 'PENDING',
+ "resourceId" TEXT NOT NULL,
+ "resourceName" TEXT,
+ "groupId" TEXT,
+ "result" TEXT,
+ "error" TEXT,
+ "userId" TEXT NOT NULL,
+ "teamId" TEXT NOT NULL,
+ "createdAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
+ "updatedAt" TIMESTAMP(3) NOT NULL,
+ "completedAt" TIMESTAMP(3),
+
+ CONSTRAINT "ExportJob_pkey" PRIMARY KEY ("id")
+);
+
+-- Create indexes for better performance
+CREATE INDEX "ExportJob_userId_idx" ON "ExportJob"("userId");
+CREATE INDEX "ExportJob_teamId_idx" ON "ExportJob"("teamId");
+CREATE INDEX "ExportJob_status_idx" ON "ExportJob"("status");
+CREATE INDEX "ExportJob_createdAt_idx" ON "ExportJob"("createdAt");
+
+-- Add foreign key constraints
+ALTER TABLE "ExportJob" ADD CONSTRAINT "ExportJob_userId_fkey" FOREIGN KEY ("userId") REFERENCES "User"("id") ON DELETE CASCADE ON UPDATE CASCADE;
+ALTER TABLE "ExportJob" ADD CONSTRAINT "ExportJob_teamId_fkey" FOREIGN KEY ("teamId") REFERENCES "Team"("id") ON DELETE CASCADE ON UPDATE CASCADE;
+
+-- Update the default timestamp for updatedAt
+ALTER TABLE "ExportJob" ALTER COLUMN "updatedAt" SET DEFAULT CURRENT_TIMESTAMP;
\ No newline at end of file
diff --git a/lib/trigger/export-visits.ts b/lib/trigger/export-visits.ts
new file mode 100644
index 00000000..481d4b46
--- /dev/null
+++ b/lib/trigger/export-visits.ts
@@ -0,0 +1,586 @@
+import { logger, task } from "@trigger.dev/sdk/v3";
+import Bottleneck from "bottleneck";
+
+import prisma from "@/lib/prisma";
+import {
+ getViewPageDuration,
+ getViewUserAgent,
+ getViewUserAgent_v2,
+} from "@/lib/tinybird";
+
+// Create a bottleneck instance to limit tinybird API calls
+const tinybirdLimiter = new Bottleneck({
+ maxConcurrent: 5, // Maximum 5 concurrent requests
+ minTime: 200, // Minimum 200ms between requests
+});
+
+export type ExportVisitsPayload = {
+ type: "document" | "dataroom" | "dataroom-group";
+ teamId: string;
+ resourceId: string; // document ID or dataroom ID
+ groupId?: string; // for dataroom groups
+ userId: string;
+ exportId: string; // unique identifier for this export job
+};
+
+export const exportVisitsTask = task({
+ id: "export-visits",
+ retry: { maxAttempts: 3 },
+ maxDuration: 900, // 15 minutes to handle large datasets
+ machine: {
+ preset: "medium-1x", // More resources for processing
+ },
+ run: async (payload: ExportVisitsPayload) => {
+ const { type, teamId, resourceId, groupId, userId, exportId } = payload;
+
+ logger.info("Starting export visits task", { payload });
+
+ try {
+ // Update job status to processing
+ await prisma.exportJob.update({
+ where: { id: exportId },
+ data: { status: "PROCESSING" },
+ });
+
+ // Verify team access
+ const team = await prisma.team.findUnique({
+ where: {
+ id: teamId,
+ users: {
+ some: {
+ userId: userId,
+ },
+ },
+ },
+ select: { plan: true },
+ });
+
+ if (!team) {
+ throw new Error("Team not found or access denied");
+ }
+
+ if (team.plan === "free") {
+ throw new Error("This feature is not available for your plan");
+ }
+
+ let csvData: string;
+ let resourceName: string;
+
+ if (type === "document") {
+ ({ csvData, resourceName } = await exportDocumentVisits(resourceId, teamId));
+ } else if (type === "dataroom") {
+ ({ csvData, resourceName } = await exportDataroomVisits(resourceId, teamId, groupId));
+ } else {
+ throw new Error("Invalid export type");
+ }
+
+ // Store the result in the database
+ await prisma.exportJob.update({
+ where: { id: exportId },
+ data: {
+ status: "COMPLETED",
+ result: csvData,
+ resourceName,
+ completedAt: new Date(),
+ },
+ });
+
+ logger.info("Export visits task completed successfully", {
+ exportId,
+ type,
+ resourceId,
+ csvSize: csvData.length,
+ });
+
+ return {
+ success: true,
+ exportId,
+ resourceName,
+ csvSize: csvData.length,
+ };
+ } catch (error) {
+ logger.error("Export visits task failed", {
+ exportId,
+ error: error instanceof Error ? error.message : String(error),
+ });
+
+ // Update job status to failed
+ await prisma.exportJob.update({
+ where: { id: exportId },
+ data: {
+ status: "FAILED",
+ error: error instanceof Error ? error.message : String(error),
+ },
+ });
+
+ throw error;
+ }
+ },
+});
+
+async function exportDocumentVisits(docId: string, teamId: string): Promise<{
+ csvData: string;
+ resourceName: string;
+}> {
+ // Fetch Document
+ const document = await prisma.document.findUnique({
+ where: { id: docId, teamId: teamId },
+ select: {
+ id: true,
+ name: true,
+ numPages: true,
+ versions: {
+ orderBy: { createdAt: "desc" },
+ select: {
+ versionNumber: true,
+ createdAt: true,
+ numPages: true,
+ },
+ },
+ team: {
+ select: {
+ plan: true,
+ },
+ },
+ },
+ });
+
+ if (!document) {
+ throw new Error("Document not found");
+ }
+
+ // Fetch views
+ const views = await prisma.view.findMany({
+ where: { documentId: docId },
+ include: {
+ link: { select: { name: true } },
+ agreementResponse: {
+ include: {
+ agreement: {
+ select: {
+ name: true,
+ content: true,
+ },
+ },
+ },
+ },
+ customFieldResponse: {
+ select: {
+ data: true,
+ },
+ },
+ },
+ orderBy: {
+ viewedAt: "desc",
+ },
+ });
+
+ if (!views || views.length === 0) {
+ throw new Error("Document has no views");
+ }
+
+ const isProPlan = document.team.plan.includes("pro");
+
+ // Create CSV rows array starting with headers
+ const csvRows: string[] = [];
+ const headers = [
+ "Viewed at",
+ "Name",
+ "Email",
+ "Link Name",
+ "Total Visit Duration (s)",
+ "Total Document Completion (%)",
+ "Document version",
+ "Downloaded at",
+ "Verified",
+ "Agreement Accepted",
+ "Agreement Name",
+ "Agreement Content",
+ "Agreement Accepted At",
+ "Viewed from dataroom",
+ "Browser",
+ "OS",
+ "Device",
+ ];
+
+ if (!isProPlan) {
+ headers.push("Country", "City", "Custom Fields");
+ }
+
+ csvRows.push(headers.join(","));
+
+ // Process views with rate limiting
+ logger.info("Processing document views with rate limiting", {
+ viewCount: views.length,
+ });
+
+ for (let i = 0; i < views.length; i++) {
+ const view = views[i];
+
+ logger.info(`Processing view ${i + 1}/${views.length}`, {
+ viewId: view.id,
+ viewedAt: view.viewedAt,
+ });
+
+ // Rate-limited calls to tinybird
+ const [duration, userAgentData] = await Promise.all([
+ tinybirdLimiter.schedule(() =>
+ getViewPageDuration({
+ documentId: docId,
+ viewId: view.id,
+ since: 0,
+ }),
+ ),
+ tinybirdLimiter.schedule(async () => {
+ const result = await getViewUserAgent({
+ viewId: view.id,
+ });
+
+ if (!result || result.rows === 0) {
+ return getViewUserAgent_v2({
+ documentId: docId,
+ viewId: view.id,
+ since: 0,
+ });
+ }
+
+ return result;
+ }),
+ ]);
+
+ const relevantDocumentVersion = document.versions.find(
+ (version) => version.createdAt <= view.viewedAt,
+ );
+
+ const numPages =
+ relevantDocumentVersion?.numPages || document.numPages || 0;
+ const completionRate = numPages
+ ? (duration.data.length / numPages) * 100
+ : 0;
+
+ const totalDuration = duration.data.reduce(
+ (total, data) => total + data.sum_duration,
+ 0,
+ );
+
+ const rowData = [
+ view.viewedAt.toISOString(),
+ view.viewerName || "NaN",
+ view.viewerEmail || "NaN",
+ view.link?.name || "NaN",
+ (totalDuration / 1000.0).toFixed(1),
+ completionRate.toFixed(2) + "%",
+ relevantDocumentVersion?.versionNumber ||
+ document.versions[0]?.versionNumber ||
+ "NaN",
+ view.downloadedAt ? view.downloadedAt.toISOString() : "NaN",
+ view.verified ? "Yes" : "No",
+ view.agreementResponse ? "Yes" : "NaN",
+ view.agreementResponse?.agreement.name || "NaN",
+ view.agreementResponse?.agreement.content || "NaN",
+ view.agreementResponse?.createdAt.toISOString() || "NaN",
+ view.dataroomId ? "Yes" : "No",
+ userAgentData?.data[0]?.browser || "NaN",
+ userAgentData?.data[0]?.os || "NaN",
+ userAgentData?.data[0]?.device || "NaN",
+ ];
+
+ if (!isProPlan) {
+ rowData.push(
+ userAgentData?.data[0]?.country || "NaN",
+ userAgentData?.data[0]?.city || "NaN",
+ view.customFieldResponse?.data
+ ? `"${JSON.stringify(view.customFieldResponse.data).replace(/"/g, '""')}"`
+ : "NaN",
+ );
+ }
+
+ csvRows.push(rowData.join(","));
+ }
+
+ return {
+ csvData: csvRows.join("\n"),
+ resourceName: document.name,
+ };
+}
+
+async function exportDataroomVisits(
+ dataroomId: string,
+ teamId: string,
+ groupId?: string
+): Promise<{
+ csvData: string;
+ resourceName: string;
+}> {
+ // Fetch Dataroom
+ const dataroom = await prisma.dataroom.findUnique({
+ where: { id: dataroomId, teamId: teamId },
+ select: {
+ id: true,
+ name: true,
+ documents: {
+ select: {
+ id: true,
+ document: {
+ select: {
+ name: true,
+ numPages: true,
+ versions: {
+ orderBy: { createdAt: "desc" },
+ select: {
+ versionNumber: true,
+ createdAt: true,
+ numPages: true,
+ },
+ },
+ },
+ },
+ },
+ },
+ },
+ });
+
+ if (!dataroom) {
+ throw new Error("Dataroom not found");
+ }
+
+ // Fetch views
+ const views = await prisma.view.findMany({
+ where: {
+ dataroomId: dataroomId,
+ ...(groupId && { groupId }),
+ },
+ include: {
+ link: { select: { name: true } },
+ agreementResponse: {
+ include: {
+ agreement: {
+ select: {
+ name: true,
+ content: true,
+ },
+ },
+ },
+ },
+ document: {
+ select: {
+ id: true,
+ name: true,
+ numPages: true,
+ versions: {
+ orderBy: { createdAt: "desc" },
+ select: {
+ versionNumber: true,
+ createdAt: true,
+ numPages: true,
+ },
+ },
+ },
+ },
+ customFieldResponse: {
+ select: {
+ data: true,
+ },
+ },
+ },
+ orderBy: {
+ viewedAt: "desc",
+ },
+ });
+
+ if (!views || views.length === 0) {
+ throw new Error("Dataroom has no views");
+ }
+
+ // First get all dataroom views
+ const dataroomViews = views.filter((view) => view.viewType === "DATAROOM_VIEW");
+ const documentViews = views.filter((view) => view.viewType === "DOCUMENT_VIEW");
+
+ logger.info("Processing dataroom views with rate limiting", {
+ dataroomViewCount: dataroomViews.length,
+ documentViewCount: documentViews.length,
+ });
+
+ // Process dataroom views
+ const exportData = [];
+ for (let i = 0; i < dataroomViews.length; i++) {
+ const dataroomView = dataroomViews[i];
+
+ logger.info(`Processing dataroom view ${i + 1}/${dataroomViews.length}`, {
+ viewId: dataroomView.id,
+ });
+
+ // Find associated document views
+ const associatedDocViews = documentViews.filter(
+ (docView) => docView.dataroomViewId === dataroomView.id
+ );
+
+ // Process document views with rate limiting
+ const documentViewDetails = [];
+ for (let j = 0; j < associatedDocViews.length; j++) {
+ const docView = associatedDocViews[j];
+
+ logger.info(`Processing document view ${j + 1}/${associatedDocViews.length} for dataroom view ${i + 1}`, {
+ docViewId: docView.id,
+ });
+
+ const duration = await tinybirdLimiter.schedule(() =>
+ getViewPageDuration({
+ documentId: docView.document?.id || "null",
+ viewId: docView.id,
+ since: 0,
+ })
+ );
+
+ const relevantVersion = docView.document?.versions.find(
+ (version) => version.createdAt <= docView.viewedAt
+ );
+
+ const numPages =
+ relevantVersion?.numPages || docView.document?.numPages || 0;
+ const completionRate = numPages
+ ? (duration.data.length / numPages) * 100
+ : 0;
+
+ documentViewDetails.push({
+ documentName: docView.document?.name || "NaN",
+ viewedAt: docView.viewedAt.toISOString(),
+ downloadedAt: docView.downloadedAt?.toISOString() || "NaN",
+ duration: duration.data.reduce(
+ (total, data) => total + data.sum_duration,
+ 0
+ ),
+ completionRate: completionRate.toFixed(2) + "%",
+ documentVersion:
+ relevantVersion?.versionNumber ||
+ docView.document?.versions[0]?.versionNumber ||
+ "NaN",
+ viewId: docView.id,
+ });
+ }
+
+ exportData.push({
+ dataroomViewedAt: dataroomView.viewedAt.toISOString(),
+ dataroomDownloadedAt: dataroomView.downloadedAt?.toISOString() || "NaN",
+ viewerName: dataroomView.viewerName || "NaN",
+ viewerEmail: dataroomView.viewerEmail || "NaN",
+ linkName: dataroomView.link?.name || "NaN",
+ verified: dataroomView.verified ? "Yes" : "NaN",
+ agreementStatus: dataroomView.agreementResponse ? "Yes" : "NaN",
+ agreementName: dataroomView.agreementResponse?.agreement.name || "NaN",
+ agreementAcceptedAt:
+ dataroomView.agreementResponse?.createdAt.toISOString() || "NaN",
+ agreementContent: dataroomView.agreementResponse?.agreement.content || "NaN",
+ documentViews: documentViewDetails,
+ });
+ }
+
+ // Get user agent data for all document views at once with rate limiting
+ const userAgentDataMap = new Map();
+ for (const docView of documentViews) {
+ const userAgentData = await tinybirdLimiter.schedule(async () => {
+ const result = await getViewUserAgent({
+ viewId: docView.id,
+ });
+
+ if (!result || result.rows === 0) {
+ return getViewUserAgent_v2({
+ documentId: docView.document?.id || "null",
+ viewId: docView.id,
+ since: 0,
+ });
+ }
+
+ return result;
+ });
+
+ userAgentDataMap.set(docView.id, userAgentData);
+ }
+
+ // Create CSV
+ const csvRows: string[] = [];
+ csvRows.push([
+ "Dataroom Viewed At",
+ "Dataroom Downloaded At",
+ "Visitor Name",
+ "Visitor Email",
+ "Link Name",
+ "Verified",
+ "Agreement Accepted",
+ "Agreement Name",
+ "Agreement Content",
+ "Agreement Accepted At",
+ "Document Name",
+ "Document Viewed At",
+ "Document Downloaded At",
+ "Total Visit Duration (s)",
+ "Total Document Completion (%)",
+ "Document Version",
+ "Browser",
+ "OS",
+ "Device",
+ "Country",
+ "City",
+ ].join(","));
+
+ exportData.forEach((view) => {
+ if (view.documentViews.length === 0) {
+ csvRows.push([
+ view.dataroomViewedAt,
+ view.dataroomDownloadedAt,
+ view.viewerName,
+ view.viewerEmail,
+ view.linkName,
+ view.verified,
+ view.agreementStatus,
+ view.agreementName,
+ view.agreementContent,
+ view.agreementAcceptedAt,
+ "NaN",
+ "NaN",
+ "NaN",
+ "NaN",
+ "NaN",
+ "NaN",
+ "NaN",
+ "NaN",
+ "NaN",
+ "NaN",
+ "NaN",
+ ].join(","));
+ } else {
+ view.documentViews.forEach((docView) => {
+ const userAgentData = userAgentDataMap.get(docView.viewId);
+
+ csvRows.push([
+ view.dataroomViewedAt,
+ view.dataroomDownloadedAt,
+ view.viewerName,
+ view.viewerEmail,
+ view.linkName,
+ view.verified,
+ view.agreementStatus,
+ view.agreementName,
+ view.agreementContent,
+ view.agreementAcceptedAt,
+ docView.documentName,
+ docView.viewedAt,
+ docView.downloadedAt,
+ (docView.duration / 1000).toFixed(1),
+ docView.completionRate,
+ docView.documentVersion,
+ userAgentData?.data[0]?.browser || "NaN",
+ userAgentData?.data[0]?.os || "NaN",
+ userAgentData?.data[0]?.device || "NaN",
+ userAgentData?.data[0]?.country || "NaN",
+ userAgentData?.data[0]?.city || "NaN",
+ ].join(","));
+ });
+ }
+ });
+
+ return {
+ csvData: csvRows.join("\n"),
+ resourceName: dataroom.name,
+ };
+}
\ No newline at end of file
diff --git a/pages/api/teams/[teamId]/datarooms/[id]/export-visits.ts b/pages/api/teams/[teamId]/datarooms/[id]/export-visits.ts
index d662c693..89d243d6 100644
--- a/pages/api/teams/[teamId]/datarooms/[id]/export-visits.ts
+++ b/pages/api/teams/[teamId]/datarooms/[id]/export-visits.ts
@@ -4,24 +4,16 @@ import { authOptions } from "@/pages/api/auth/[...nextauth]";
import { getServerSession } from "next-auth/next";
import prisma from "@/lib/prisma";
-import {
- getViewPageDuration,
- getViewUserAgent,
- getViewUserAgent_v2,
-} from "@/lib/tinybird";
+import { exportVisitsTask } from "@/lib/trigger/export-visits";
import { CustomUser } from "@/lib/types";
-export const config = {
- maxDuration: 180,
-};
-
export default async function handler(
req: NextApiRequest,
res: NextApiResponse,
) {
- if (req.method !== "GET") {
- // GET /api/teams/:teamId/datarooms/:id/export-visits
- res.setHeader("Allow", ["GET"]);
+ if (req.method !== "POST") {
+ // Changed to POST to trigger background job
+ res.setHeader("Allow", ["POST"]);
return res.status(405).end(`Method ${req.method} Not Allowed`);
}
@@ -68,25 +60,6 @@ export default async function handler(
select: {
id: true,
name: true,
- documents: {
- select: {
- id: true,
- document: {
- select: {
- name: true,
- numPages: true,
- versions: {
- orderBy: { createdAt: "desc" },
- select: {
- versionNumber: true,
- createdAt: true,
- numPages: true,
- },
- },
- },
- },
- },
- },
},
});
@@ -94,249 +67,44 @@ export default async function handler(
return res.status(404).end("Dataroom not found");
}
- // First fetch the dataroom views
- const views = await prisma.view.findMany({
- where: {
- dataroomId: dataroomId,
- },
- include: {
- link: { select: { name: true } },
- agreementResponse: {
- include: {
- agreement: {
- select: {
- name: true,
- content: true,
- },
- },
- },
- },
- document: {
- select: {
- id: true,
- name: true,
- numPages: true,
- versions: {
- orderBy: { createdAt: "desc" },
- select: {
- versionNumber: true,
- createdAt: true,
- numPages: true,
- },
- },
- },
- },
- customFieldResponse: {
- select: {
- data: true,
- },
- },
- },
- orderBy: {
- viewedAt: "desc",
+ // Create export job record
+ const exportJob = await prisma.exportJob.create({
+ data: {
+ type: "dataroom",
+ resourceId: dataroomId,
+ resourceName: dataroom.name,
+ userId,
+ teamId,
+ status: "PENDING",
},
});
- if (!views || views.length === 0) {
- return res.status(404).end("Dataroom has no views");
- }
-
- // First get all dataroom views
- const dataroomViews = views.filter(
- (view) => view.viewType === "DATAROOM_VIEW",
- );
-
- // Get all document views for this dataroom
- const documentViews = views.filter(
- (view) => view.viewType === "DOCUMENT_VIEW",
- );
-
- // For each dataroom view, get its associated document views and build export data
- const exportData = await Promise.all(
- dataroomViews.map(async (dataroomView) => {
- // Find all document views that belong to this dataroom view
- const associatedDocViews = documentViews.filter(
- (docView) => docView.dataroomViewId === dataroomView.id,
- );
-
- // Get document view details including durations from Tinybird
- const documentViewDetails = await Promise.all(
- associatedDocViews.map(async (docView) => {
- const duration = await getViewPageDuration({
- documentId: docView.document?.id || "null",
- viewId: docView.id,
- since: 0,
- });
-
- const relevantVersion = docView.document?.versions.find(
- (version) => version.createdAt <= docView.viewedAt,
- );
-
- const numPages =
- relevantVersion?.numPages || docView.document?.numPages || 0;
- const completionRate = numPages
- ? (duration.data.length / numPages) * 100
- : 0;
-
- return {
- documentName: docView.document?.name || "NaN",
- viewedAt: docView.viewedAt.toISOString(),
- downloadedAt: docView.downloadedAt?.toISOString() || "NaN",
- duration: duration.data.reduce(
- (total, data) => total + data.sum_duration,
- 0,
- ),
- completionRate: completionRate.toFixed(2) + "%",
- documentVersion:
- relevantVersion?.versionNumber ||
- docView.document?.versions[0]?.versionNumber ||
- "NaN",
- };
- }),
- );
-
- // Return the complete view data
- return {
- // Dataroom view details
- dataroomViewedAt: dataroomView.viewedAt.toISOString(),
- dataroomDownloadedAt:
- dataroomView.downloadedAt?.toISOString() || "NaN",
- viewerName: dataroomView.viewerName || "NaN",
- viewerEmail: dataroomView.viewerEmail || "NaN",
- linkName: dataroomView.link?.name || "NaN",
- verified: dataroomView.verified ? "Yes" : "NaN",
-
- // Agreement details
- agreementStatus: dataroomView.agreementResponse ? "Yes" : "NaN",
- agreementName:
- dataroomView.agreementResponse?.agreement.name || "NaN",
- agreementAcceptedAt:
- dataroomView.agreementResponse?.createdAt.toISOString() || "NaN",
- agreementContent:
- dataroomView.agreementResponse?.agreement.content || "NaN",
-
- // Document view details
- documentViews: documentViewDetails,
- };
- }),
- );
-
- // Create CSV with nested document views
- const csvRows: string[] = [];
-
- // Add headers
- csvRows.push(
- [
- "Dataroom Viewed At",
- "Dataroom Downloaded At",
- "Visitor Name",
- "Visitor Email",
- "Link Name",
- "Verified",
- "Agreement Accepted",
- "Agreement Name",
- "Agreement Content",
- "Agreement Accepted At",
- "Document Name",
- "Document Viewed At",
- "Document Downloaded At",
- "Total Visit Duration (s)",
- "Total Document Completion (%)",
- "Document Version",
- "Browser",
- "OS",
- "Device",
- "Country",
- "City",
- ].join(","),
- );
-
- // Get user agent data for all views
- const userAgentData = await Promise.all(
- documentViews.map(async (view) => {
- const result = await getViewUserAgent({
- viewId: view.id,
- });
-
- if (!result || result.rows === 0) {
- return getViewUserAgent_v2({
- documentId: view.document?.id!,
- viewId: view.id,
- since: 0,
- });
- }
-
- return result;
- }),
+ // Trigger the background task
+ await exportVisitsTask.trigger(
+ {
+ type: "dataroom",
+ teamId,
+ resourceId: dataroomId,
+ userId,
+ exportId: exportJob.id,
+ },
+ {
+ idempotencyKey: exportJob.id,
+ tags: [
+ `team_${teamId}`,
+ `user_${userId}`,
+ `export_${exportJob.id}`,
+ ],
+ },
);
- // Process each view and add to CSV rows
- exportData.forEach((view) => {
- if (view.documentViews.length === 0) {
- // Add a row for dataroom view without document views
- csvRows.push(
- [
- view.dataroomViewedAt,
- view.dataroomDownloadedAt,
- view.viewerName,
- view.viewerEmail,
- view.linkName,
- view.verified,
- view.agreementStatus,
- view.agreementName,
- view.agreementContent,
- view.agreementAcceptedAt,
- "No documents viewed",
- "",
- "",
- "",
- "",
- "",
- "", // Browser
- "", // OS
- "", // Device
- "", // Country
- "", // City
- ].join(","),
- );
- } else {
- // Add a row for each document view
- view.documentViews.forEach((docView, index) => {
- csvRows.push(
- [
- view.dataroomViewedAt,
- view.dataroomDownloadedAt,
- view.viewerName,
- view.viewerEmail,
- view.linkName,
- view.verified,
- view.agreementStatus,
- view.agreementName,
- view.agreementContent,
- view.agreementAcceptedAt,
- docView.documentName,
- docView.viewedAt,
- docView.downloadedAt,
- (docView.duration / 1000).toFixed(1),
- docView.completionRate,
- docView.documentVersion,
- userAgentData[index]?.data[0]?.browser || "NaN",
- userAgentData[index]?.data[0]?.os || "NaN",
- userAgentData[index]?.data[0]?.device || "NaN",
- userAgentData[index]?.data[0]?.country || "NaN",
- userAgentData[index]?.data[0]?.city || "NaN",
- ].join(","),
- );
- });
- }
- });
-
return res.status(200).json({
- dataroomName: dataroom.name,
- visits: csvRows.join("\n"),
+ exportId: exportJob.id,
+ status: exportJob.status,
+ message: "Export job created successfully. You will be notified when it's ready.",
});
} catch (error) {
- console.error(error);
+ console.error("Error creating export job:", error);
return res.status(500).json({ message: "Something went wrong" });
}
}
diff --git a/pages/api/teams/[teamId]/datarooms/[id]/groups/[groupId]/export-visits.ts b/pages/api/teams/[teamId]/datarooms/[id]/groups/[groupId]/export-visits.ts
index 94beadca..29efada7 100644
--- a/pages/api/teams/[teamId]/datarooms/[id]/groups/[groupId]/export-visits.ts
+++ b/pages/api/teams/[teamId]/datarooms/[id]/groups/[groupId]/export-visits.ts
@@ -4,24 +4,16 @@ import { authOptions } from "@/pages/api/auth/[...nextauth]";
import { getServerSession } from "next-auth/next";
import prisma from "@/lib/prisma";
-import {
- getViewPageDuration,
- getViewUserAgent,
- getViewUserAgent_v2,
-} from "@/lib/tinybird";
+import { exportVisitsTask } from "@/lib/trigger/export-visits";
import { CustomUser } from "@/lib/types";
-export const config = {
- maxDuration: 180,
-};
-
export default async function handler(
req: NextApiRequest,
res: NextApiResponse,
) {
- if (req.method !== "GET") {
- // GET /api/teams/:teamId/datarooms/:id/groups/:groupId/export-visits
- res.setHeader("Allow", ["GET"]);
+ if (req.method !== "POST") {
+ // Changed to POST to trigger background job
+ res.setHeader("Allow", ["POST"]);
return res.status(405).end(`Method ${req.method} Not Allowed`);
}
@@ -73,25 +65,6 @@ export default async function handler(
select: {
id: true,
name: true,
- documents: {
- select: {
- id: true,
- document: {
- select: {
- name: true,
- numPages: true,
- versions: {
- orderBy: { createdAt: "desc" },
- select: {
- versionNumber: true,
- createdAt: true,
- numPages: true,
- },
- },
- },
- },
- },
- },
},
});
@@ -99,270 +72,46 @@ export default async function handler(
return res.status(404).end("Dataroom not found");
}
- // First fetch the dataroom views
- const views = await prisma.view.findMany({
- where: {
- dataroomId: dataroomId,
- groupId: groupId,
- },
- include: {
- link: { select: { name: true } },
- agreementResponse: {
- include: {
- agreement: {
- select: {
- name: true,
- content: true,
- },
- },
- },
- },
- document: {
- select: {
- id: true,
- name: true,
- numPages: true,
- versions: {
- orderBy: { createdAt: "desc" },
- select: {
- versionNumber: true,
- createdAt: true,
- numPages: true,
- },
- },
- },
- },
- customFieldResponse: {
- select: {
- data: true,
- },
- },
- },
- orderBy: {
- viewedAt: "desc",
+ // Create export job record
+ const exportJob = await prisma.exportJob.create({
+ data: {
+ type: "dataroom",
+ resourceId: dataroomId,
+ resourceName: dataroom.name,
+ groupId,
+ userId,
+ teamId,
+ status: "PENDING",
},
});
- if (!views || views.length === 0) {
- return res.status(404).end("Dataroom has no views");
- }
-
- // First get all dataroom views
- const dataroomViews = views.filter(
- (view) => view.viewType === "DATAROOM_VIEW",
- );
-
- // Get all document views for this dataroom
- const documentViews = views.filter(
- (view) => view.viewType === "DOCUMENT_VIEW",
- );
-
- // For each dataroom view, get its associated document views and build export data
- const exportData = await Promise.all(
- dataroomViews.map(async (dataroomView) => {
- // Find all document views that belong to this dataroom view
- const associatedDocViews = documentViews.filter(
- (docView) => docView.dataroomViewId === dataroomView.id,
- );
-
- // Get document view details including durations from Tinybird
- const documentViewDetails = await Promise.all(
- associatedDocViews.map(async (docView) => {
- const duration = await getViewPageDuration({
- documentId: docView.document?.id || "null",
- viewId: docView.id,
- since: 0,
- });
-
- const relevantVersion = docView.document?.versions.find(
- (version) => version.createdAt <= docView.viewedAt,
- );
-
- const numPages =
- relevantVersion?.numPages || docView.document?.numPages || 0;
- const completionRate = numPages
- ? (duration.data.length / numPages) * 100
- : 0;
-
- return {
- documentName: docView.document?.name || "NaN",
- viewedAt: docView.viewedAt.toISOString(),
- downloadedAt: docView.downloadedAt?.toISOString() || "NaN",
- duration: duration.data.reduce(
- (total, data) => total + data.sum_duration,
- 0,
- ),
- completionRate: completionRate.toFixed(2) + "%",
- documentVersion:
- relevantVersion?.versionNumber ||
- docView.document?.versions[0]?.versionNumber ||
- "NaN",
- };
- }),
- );
-
- // Return the complete view data
- return {
- // Dataroom view details
- dataroomViewedAt: dataroomView.viewedAt.toISOString(),
- dataroomDownloadedAt:
- dataroomView.downloadedAt?.toISOString() || "NaN",
- viewerName: dataroomView.viewerName || "NaN",
- viewerEmail: dataroomView.viewerEmail || "NaN",
- linkName: dataroomView.link?.name || "NaN",
- verified: dataroomView.verified ? "Yes" : "NaN",
-
- // Agreement details
- agreementStatus: dataroomView.agreementResponse ? "Yes" : "NaN",
- agreementName:
- dataroomView.agreementResponse?.agreement.name || "NaN",
- agreementAcceptedAt:
- dataroomView.agreementResponse?.createdAt.toISOString() || "NaN",
- agreementContent:
- dataroomView.agreementResponse?.agreement.content || "NaN",
-
- // Document view details
- documentViews: documentViewDetails,
- };
- }),
- );
-
- // Create CSV with nested document views
- const csvRows: string[] = [];
-
- // Add headers
- csvRows.push(
- [
- "Dataroom Viewed At",
- "Dataroom Downloaded At",
- "Visitor Name",
- "Visitor Email",
- "Link Name",
- "Verified",
- "Agreement Accepted",
- "Agreement Name",
- "Agreement Content",
- "Agreement Accepted At",
- "Document Name",
- "Document Viewed At",
- "Document Downloaded At",
- "Total Visit Duration (s)",
- "Total Document Completion (%)",
- "Document Version",
- "Browser",
- "OS",
- "Device",
- "Country",
- "City",
- ].join(","),
- );
-
- // Create a map of viewId to userAgent data for efficient lookup
- const userAgentDataMap = new Map();
-
- await Promise.all(
- documentViews.map(async (view) => {
- const result = await getViewUserAgent({
- viewId: view.id,
- });
-
- let userAgentResult;
- if (!result || result.rows === 0) {
- // Only call v2 if document and its id exist
- if (view.document?.id) {
- userAgentResult = await getViewUserAgent_v2({
- documentId: view.document.id,
- viewId: view.id,
- since: 0,
- });
- } else {
- // Set default empty result if document/id is missing
- userAgentResult = { data: [] };
- }
- } else {
- userAgentResult = result;
- }
-
- userAgentDataMap.set(view.id, userAgentResult);
- }),
+ // Trigger the background task
+ await exportVisitsTask.trigger(
+ {
+ type: "dataroom",
+ teamId,
+ resourceId: dataroomId,
+ groupId,
+ userId,
+ exportId: exportJob.id,
+ },
+ {
+ idempotencyKey: exportJob.id,
+ tags: [
+ `team_${teamId}`,
+ `user_${userId}`,
+ `export_${exportJob.id}`,
+ ],
+ },
);
- // Process each view and add to CSV rows
- exportData.forEach((view) => {
- if (view.documentViews.length === 0) {
- // Add a row for dataroom view without document views
- csvRows.push(
- [
- view.dataroomViewedAt,
- view.dataroomDownloadedAt,
- view.viewerName,
- view.viewerEmail,
- view.linkName,
- view.verified,
- view.agreementStatus,
- view.agreementName,
- view.agreementContent,
- view.agreementAcceptedAt,
- "No documents viewed",
- "",
- "",
- "",
- "",
- "",
- "", // Browser
- "", // OS
- "", // Device
- "", // Country
- "", // City
- ].join(","),
- );
- } else {
- // Add a row for each document view
- view.documentViews.forEach((docView) => {
- // Find the corresponding document view to get the correct viewId
- const correspondingDocView = documentViews.find(
- (dv) => dv.viewedAt.toISOString() === docView.viewedAt,
- );
-
- const userAgentData = correspondingDocView
- ? userAgentDataMap.get(correspondingDocView.id)
- : { data: [] };
-
- csvRows.push(
- [
- view.dataroomViewedAt,
- view.dataroomDownloadedAt,
- view.viewerName,
- view.viewerEmail,
- view.linkName,
- view.verified,
- view.agreementStatus,
- view.agreementName,
- view.agreementContent,
- view.agreementAcceptedAt,
- docView.documentName,
- docView.viewedAt,
- docView.downloadedAt,
- (docView.duration / 1000).toFixed(1),
- docView.completionRate,
- docView.documentVersion,
- userAgentData?.data[0]?.browser || "NaN",
- userAgentData?.data[0]?.os || "NaN",
- userAgentData?.data[0]?.device || "NaN",
- userAgentData?.data[0]?.country || "NaN",
- userAgentData?.data[0]?.city || "NaN",
- ].join(","),
- );
- });
- }
- });
-
return res.status(200).json({
- dataroomName: dataroom.name,
- visits: csvRows.join("\n"),
+ exportId: exportJob.id,
+ status: exportJob.status,
+ message: "Export job created successfully. You will be notified when it's ready.",
});
} catch (error) {
- console.error(error);
+ console.error("Error creating export job:", error);
return res.status(500).json({ message: "Something went wrong" });
}
}
diff --git a/pages/api/teams/[teamId]/documents/[id]/export-visits.ts b/pages/api/teams/[teamId]/documents/[id]/export-visits.ts
index 812d68aa..799689df 100644
--- a/pages/api/teams/[teamId]/documents/[id]/export-visits.ts
+++ b/pages/api/teams/[teamId]/documents/[id]/export-visits.ts
@@ -4,24 +4,16 @@ import { authOptions } from "@/pages/api/auth/[...nextauth]";
import { getServerSession } from "next-auth/next";
import prisma from "@/lib/prisma";
-import {
- getViewPageDuration,
- getViewUserAgent,
- getViewUserAgent_v2,
-} from "@/lib/tinybird";
+import { exportVisitsTask } from "@/lib/trigger/export-visits";
import { CustomUser } from "@/lib/types";
-export const config = {
- maxDuration: 180,
-};
-
export default async function handler(
req: NextApiRequest,
res: NextApiResponse,
) {
- if (req.method !== "GET") {
- // GET /api/teams/:teamId/documents/:id/export-visits
- res.setHeader("Allow", ["GET"]);
+ if (req.method !== "POST") {
+ // Changed to POST to trigger background job
+ res.setHeader("Allow", ["POST"]);
return res.status(405).end(`Method ${req.method} Not Allowed`);
}
@@ -65,15 +57,6 @@ export default async function handler(
select: {
id: true,
name: true,
- numPages: true,
- versions: {
- orderBy: { createdAt: "desc" },
- select: {
- versionNumber: true,
- createdAt: true,
- numPages: true,
- },
- },
},
});
@@ -81,152 +64,44 @@ export default async function handler(
return res.status(404).end("Document not found");
}
- // Fetch views data from the database
- const views = await prisma.view.findMany({
- where: { documentId: docId },
- include: {
- link: { select: { name: true } },
- agreementResponse: {
- include: {
- agreement: {
- select: {
- name: true,
- content: true,
- },
- },
- },
- },
- customFieldResponse: {
- select: {
- data: true,
- },
- },
- },
- orderBy: {
- viewedAt: "desc",
+ // Create export job record
+ const exportJob = await prisma.exportJob.create({
+ data: {
+ type: "document",
+ resourceId: docId,
+ resourceName: document.name,
+ userId,
+ teamId,
+ status: "PENDING",
},
});
- if (!views || views.length === 0) {
- return res.status(404).end("Document has no views");
- }
-
- const isProPlan = team.plan.includes("pro");
-
- // Create CSV rows array starting with headers
- const csvRows: string[] = [];
- const headers = [
- "Viewed at",
- "Name",
- "Email",
- "Link Name",
- "Total Visit Duration (s)",
- "Total Document Completion (%)",
- "Document version",
- "Downloaded at",
- "Verified",
- "Agreement Accepted",
- "Agreement Name",
- "Agreement Content",
- "Agreement Accepted At",
- "Viewed from dataroom",
- "Browser",
- "OS",
- "Device",
- ];
-
- if (!isProPlan) {
- headers.push("Country", "City", "Custom Fields");
- }
-
- csvRows.push(headers.join(","));
-
- // Fetch all durations in parallel
- const durations = await Promise.all(
- views.map((view) =>
- getViewPageDuration({
- documentId: docId,
- viewId: view.id,
- since: 0,
- }),
- ),
- );
-
- const userAgentData = await Promise.all(
- views.map(async (view) => {
- const result = await getViewUserAgent({
- viewId: view.id,
- });
-
- if (!result || result.rows === 0) {
- return getViewUserAgent_v2({
- documentId: docId,
- viewId: view.id,
- since: 0,
- });
- }
-
- return result;
- }),
+ // Trigger the background task
+ await exportVisitsTask.trigger(
+ {
+ type: "document",
+ teamId,
+ resourceId: docId,
+ userId,
+ exportId: exportJob.id,
+ },
+ {
+ idempotencyKey: exportJob.id,
+ tags: [
+ `team_${teamId}`,
+ `user_${userId}`,
+ `export_${exportJob.id}`,
+ ],
+ },
);
- // Process each view and add to CSV rows
- views.forEach((view, index) => {
- const relevantDocumentVersion = document.versions.find(
- (version) => version.createdAt <= view.viewedAt,
- );
-
- const numPages =
- relevantDocumentVersion?.numPages || document.numPages || 0;
- const completionRate = numPages
- ? (durations[index].data.length / numPages) * 100
- : 0;
-
- const totalDuration = durations[index].data.reduce(
- (total, data) => total + data.sum_duration,
- 0,
- );
-
- const rowData = [
- view.viewedAt.toISOString(),
- view.viewerName || "NaN",
- view.viewerEmail || "NaN",
- view.link?.name || "NaN",
- (totalDuration / 1000.0).toFixed(1),
- completionRate.toFixed(2) + "%",
- relevantDocumentVersion?.versionNumber ||
- document.versions[0]?.versionNumber ||
- "NaN",
- view.downloadedAt ? view.downloadedAt.toISOString() : "NaN",
- view.verified ? "Yes" : "No",
- view.agreementResponse ? "Yes" : "NaN",
- view.agreementResponse?.agreement.name || "NaN",
- view.agreementResponse?.agreement.content || "NaN",
- view.agreementResponse?.createdAt.toISOString() || "NaN",
- view.dataroomId ? "Yes" : "No",
- userAgentData[index]?.data[0]?.browser || "NaN",
- userAgentData[index]?.data[0]?.os || "NaN",
- userAgentData[index]?.data[0]?.device || "NaN",
- ];
-
- if (!isProPlan) {
- rowData.push(
- userAgentData[index]?.data[0]?.country || "NaN",
- userAgentData[index]?.data[0]?.city || "NaN",
- view.customFieldResponse?.data
- ? `"${JSON.stringify(view.customFieldResponse.data).replace(/"/g, '""')}"`
- : "NaN",
- );
- }
-
- csvRows.push(rowData.join(","));
- });
return res.status(200).json({
- documentName: document.name,
- visits: csvRows.join("\n"),
+ exportId: exportJob.id,
+ status: exportJob.status,
+ message: "Export job created successfully. You will be notified when it's ready.",
});
} catch (error) {
- console.error(error);
+ console.error("Error creating export job:", error);
return res.status(500).json({ message: "Something went wrong" });
}
}
diff --git a/pages/api/teams/[teamId]/export-jobs.ts b/pages/api/teams/[teamId]/export-jobs.ts
new file mode 100644
index 00000000..4752165a
--- /dev/null
+++ b/pages/api/teams/[teamId]/export-jobs.ts
@@ -0,0 +1,125 @@
+import { NextApiRequest, NextApiResponse } from "next";
+import { nanoid } from "nanoid";
+
+import { authOptions } from "@/pages/api/auth/[...nextauth]";
+import { getServerSession } from "next-auth/next";
+
+import { exportVisitsTask } from "@/lib/trigger/export-visits";
+import prisma from "@/lib/prisma";
+import { CustomUser } from "@/lib/types";
+
+export default async function handler(
+ req: NextApiRequest,
+ res: NextApiResponse,
+) {
+ const session = await getServerSession(req, res, authOptions);
+ if (!session) {
+ return res.status(401).end("Unauthorized");
+ }
+
+ const { teamId } = req.query as { teamId: string };
+ const userId = (session.user as CustomUser).id;
+
+ if (req.method === "POST") {
+ // Trigger a new export job
+ const { type, resourceId, groupId } = req.body as {
+ type: "document" | "dataroom" | "dataroom-group";
+ resourceId: string;
+ groupId?: string;
+ };
+
+ if (!type || !resourceId) {
+ return res.status(400).json({ error: "Missing required fields" });
+ }
+
+ try {
+ // Verify team access
+ const team = await prisma.team.findUnique({
+ where: {
+ id: teamId,
+ users: {
+ some: {
+ userId: userId,
+ },
+ },
+ },
+ select: { plan: true },
+ });
+
+ if (!team) {
+ return res.status(404).json({ error: "Team not found" });
+ }
+
+ if (team.plan === "free") {
+ return res.status(403).json({
+ error: "This feature is not available for your plan"
+ });
+ }
+
+ // Create export job record
+ const exportJob = await prisma.exportJob.create({
+ data: {
+ type,
+ resourceId,
+ groupId,
+ userId,
+ teamId,
+ status: "PENDING",
+ },
+ });
+
+ // Trigger the background task
+ await exportVisitsTask.trigger(
+ {
+ type,
+ teamId,
+ resourceId,
+ groupId,
+ userId,
+ exportId: exportJob.id,
+ },
+ {
+ idempotencyKey: exportJob.id,
+ tags: [
+ `team_${teamId}`,
+ `user_${userId}`,
+ `export_${exportJob.id}`,
+ ],
+ },
+ );
+
+ return res.status(200).json({
+ exportId: exportJob.id,
+ status: exportJob.status,
+ message: "Export job created successfully",
+ });
+ } catch (error) {
+ console.error("Error creating export job:", error);
+ return res.status(500).json({ error: "Failed to create export job" });
+ }
+ }
+
+ if (req.method === "GET") {
+ // Get export jobs for the team
+ try {
+ const exportJobs = await prisma.exportJob.findMany({
+ where: {
+ teamId,
+ userId,
+ },
+ orderBy: {
+ createdAt: "desc",
+ },
+ take: 20, // Limit to last 20 jobs
+ });
+
+ return res.status(200).json(exportJobs);
+ } catch (error) {
+ console.error("Error fetching export jobs:", error);
+ return res.status(500).json({ error: "Failed to fetch export jobs" });
+ }
+ }
+
+ res.setHeader("Allow", ["POST", "GET"]);
+ return res.status(405).end(`Method ${req.method} Not Allowed`);
+}
\ No newline at end of file
diff --git a/pages/api/teams/[teamId]/export-jobs/[exportId].ts b/pages/api/teams/[teamId]/export-jobs/[exportId].ts
new file mode 100644
index 00000000..9cbe53c1
--- /dev/null
+++ b/pages/api/teams/[teamId]/export-jobs/[exportId].ts
@@ -0,0 +1,95 @@
+import { NextApiRequest, NextApiResponse } from "next";
+
+import { authOptions } from "@/pages/api/auth/[...nextauth]";
+import { getServerSession } from "next-auth/next";
+
+import prisma from "@/lib/prisma";
+import { CustomUser } from "@/lib/types";
+
+export default async function handler(
+ req: NextApiRequest,
+ res: NextApiResponse,
+) {
+ const session = await getServerSession(req, res, authOptions);
+ if (!session) {
+ return res.status(401).end("Unauthorized");
+ }
+
+ const { teamId, exportId } = req.query as {
+ teamId: string;
+ exportId: string;
+ };
+ const userId = (session.user as CustomUser).id;
+
+ if (req.method === "GET") {
+ try {
+ // Get export job details
+ const exportJob = await prisma.exportJob.findUnique({
+ where: {
+ id: exportId,
+ teamId,
+ userId,
+ },
+ });
+
+ if (!exportJob) {
+ return res.status(404).json({ error: "Export job not found" });
+ }
+
+ // Check if client wants to download the CSV
+ const { download } = req.query;
+ if (download === "true" && exportJob.status === "COMPLETED" && exportJob.result) {
+ const timestamp = new Date().toISOString().replace(/[-:Z]/g, "");
+ const filename = `${exportJob.resourceName || "export"}_visits_${timestamp}.csv`;
+
+ res.setHeader("Content-Type", "text/csv");
+ res.setHeader(
+ "Content-Disposition",
+ `attachment; filename="${filename}"`
+ );
+ res.setHeader("Content-Length", Buffer.byteLength(exportJob.result));
+
+ return res.status(200).send(exportJob.result);
+ }
+
+ // Return job status
+ return res.status(200).json({
+ id: exportJob.id,
+ type: exportJob.type,
+ status: exportJob.status,
+ resourceId: exportJob.resourceId,
+ resourceName: exportJob.resourceName,
+ groupId: exportJob.groupId,
+ error: exportJob.error,
+ createdAt: exportJob.createdAt,
+ updatedAt: exportJob.updatedAt,
+ completedAt: exportJob.completedAt,
+ isReady: exportJob.status === "COMPLETED" && !!exportJob.result,
+ });
+ } catch (error) {
+ console.error("Error fetching export job:", error);
+ return res.status(500).json({ error: "Failed to fetch export job" });
+ }
+ }
+
+ if (req.method === "DELETE") {
+ try {
+ // Delete export job
+ await prisma.exportJob.delete({
+ where: {
+ id: exportId,
+ teamId,
+ userId,
+ },
+ });
+
+ return res.status(200).json({ message: "Export job deleted successfully" });
+ } catch (error) {
+ console.error("Error deleting export job:", error);
+ return res.status(500).json({ error: "Failed to delete export job" });
+ }
+ }
+
+ res.setHeader("Allow", ["GET", "DELETE"]);
+ return res.status(405).end(`Method ${req.method} Not Allowed`);
+}
\ No newline at end of file
diff --git a/prisma/schema/schema.prisma b/prisma/schema/schema.prisma
index ee594ee7..82ec6596 100644
--- a/prisma/schema/schema.prisma
+++ b/prisma/schema/schema.prisma
@@ -62,6 +62,9 @@ model User {
// conversation
participatedConversations ConversationParticipant[]
messages Message[]
+
+ // export jobs
+ exportJobs ExportJob[]
}
model Team {
@@ -98,6 +101,9 @@ model Team {
restrictedTokens RestrictedToken[]
webhooks Webhook[]
+ // export jobs
+ exportJobs ExportJob[]
+
conversations Conversation[]
uploadedDocuments DocumentUpload[]
@@ -132,6 +138,13 @@ enum Role {
MEMBER
}
+enum ExportStatus {
+ PENDING
+ PROCESSING
+ COMPLETED
+ FAILED
+}
+
model UserTeam {
role Role @default(MEMBER)
status String @default("ACTIVE")
@@ -642,3 +655,29 @@ model TagItem {
@@index([tagId, documentId])
@@index([tagId, dataroomId])
}
+
+model ExportJob {
+ id String @id @default(cuid())
+ type String // "document" | "dataroom" | "dataroom-group"
+ status ExportStatus @default(PENDING)
+ resourceId String // document ID or dataroom ID
+ resourceName String? // document name or dataroom name
+ groupId String? // for dataroom groups
+ result String? @db.Text // CSV data
+ error String? // error message if failed
+
+ userId String
+ user User @relation(fields: [userId], references: [id], onDelete: Cascade)
+
+ teamId String
+ team Team @relation(fields: [teamId], references: [id], onDelete: Cascade)
+
+ createdAt DateTime @default(now())
+ updatedAt DateTime @updatedAt
+ completedAt DateTime?
+
+ @@index([userId])
+ @@index([teamId])
+ @@index([status])
+ @@index([createdAt])
+}