Every analytics tool wants a subscription. And your data. I built Twiligent instead. YouTube stats. Instagram stats. One dashboard. Runs locally on your machine.
# Architecture Overview
System Topology
┌─────────────────────────────────────────────────────────────────┐
│ Public site and docs - Cloudflare Pages "twiligent-site" │
│ https://twiligent.ujjwal.fyi │
│ No API calls; Sign in and Get started link to the app │
└────────────────────────┬────────────────────────────────────────┘
│ links
▼
┌─────────────────────────────────────────────────────────────────┐
│ Browser (User) │
│ React SPA (the app) - Cloudflare Pages "twiligent" │
│ https://app.twiligent.ujjwal.fyi │
└────────────────────────┬────────────────────────────────────────┘
│ HTTPS + Bearer JWT
▼
┌─────────────────────────────────────────────────────────────────┐
│ Cloudflare Worker - "twiligent" │
│ https://twiligent.ujjwalkrai.workers.dev/api │
│ │
│ Hono v4 app (backend/app.ts) │
│ ├── Middleware: CORS, request id, security headers, clients, │
│ │ per-IP rate limit │
│ ├── Public: GET /api/health, GET /api/health/ready, │
│ │ GET /api/auth/instagram/callback, │
│ │ POST /api/auth/email-hook (signed by Supabase Auth), │
│ │ GET/POST /api/webhooks/instagram (signed by Meta) │
│ ├── MCP: POST /mcp (Claude clients): requireMcpAuth (access │
│ │ token or approved OAuth client) -> mcp limit -> tools │
│ ├── Protected (/api/*): requireAuth -> rate limits │
│ │ -> route table -> controller -> service -> repository │
│ └── Errors: errorHandler -> { success, code, message, ... } │
│ │
│ Cron handlers (wrangler.toml triggers, jobs/scheduled.ts): │
│ ├── * * * * * -> automationProcessor.run(), then │
│ │ automationFollowups.run(), then │
│ │ automationPublicReplies.run() (dispatch) │
│ ├── */5 * * * * -> on :00 :15 :30 :45 publishDue(); on the │
│ │ other minutes automationReconciler.run() │
│ └── 0 0 * * * -> token refresh, retention purge │
└──────┬──────────┬──────────────┬────────────────────────────────┘
│ │ │
▼ ▼ ▼
┌──────────┐ ┌─────────┐ ┌──────────────────────────────────────┐
│ Supabase │ │ Upstash │ │ External APIs │
│ Postgres │ │ Redis │ │ ├── YouTube Data API v3 │
│ │ │ (cache, │ │ ├── Instagram Graph API v25 │
│ tables: │ │ rate │ │ ├── Cloudinary Upload API │
│ accounts │ │ limits,│ │ └── Resend Email API (auth emails) │
│ │ │ email │ │ │
│ │ │ counts)│ │ │
│ settings │ │ │ └──────────────────────────────────────┘
│ sched_ │ │ keys: │
│ posts │ │ videos: │
│ tracked_ │ │ ig: │
│ posts │ │ │
│ automa- │ │ │
│ tion_* │ │ │
│ audit_ │ │ │
│ events │ │ │
│ mcp_* │ │ │
│ │ │ │
│ auth: │ │ │
│ users │ │ optional│
└──────────┘ └─────────┘
Deployment Topology
| Component | Platform | URL |
|---|---|---|
| Backend API | Cloudflare Workers | twiligent.ujjwalkrai.workers.dev |
| Public site and docs | Cloudflare Pages, project twiligent-site |
twiligent.ujjwal.fyi (twiligent.spacesdrive.cc redirects here with a 301 rule on the spacesdrive.cc zone) |
| Frontend SPA (the app) | Cloudflare Pages, project twiligent |
app.twiligent.ujjwal.fyi (also twiligent.pages.dev) |
| Database + Auth | Supabase; the schema is applied from supabase/migrations/ with npm run db:push |
nheuatbcevaxgxyvahtr.supabase.co |
| Media cache | Upstash Redis | Configured via UPSTASH_REDIS_REST_URL |
| Media hosting | Cloudinary | CLOUDINARY_CLOUD_NAME |
| Auth emails | Resend, through Supabase's Send Email hook | RESEND_API_KEY, EMAIL_FROM |
| CI/CD | GitHub Actions | spacesdrive/twiligent |
Request Lifecycle (Protected Route)
The full pipeline, with the job of each layer, is in docs/architecture/backend/REQUEST_PIPELINE.md.
Browser
-> fetch(`${VITE_API_URL}/accounts`, { Authorization: Bearer <jwt> })
↓
Cloudflare Worker edge node (nearest to user)
-> CORS middleware (allows FRONTEND_URL and ALLOWED_ORIGINS only)
-> request id (X-Request-Id), cancellation signal, access log line
-> security headers
-> client injection middleware (creates Supabase + Redis clients from env)
-> rateLimit(ip) middleware (per client IP; 429 when over the limit)
-> requireAuth middleware
-> extracts Bearer token from Authorization header
-> supabaseAuth.auth.getUser(token) -> Supabase validates JWT
-> sets userId, userEmail on the Hono context (401 otherwise)
-> rateLimit(user), and rateLimit(quota) or rateLimit(publish) on the routes they cover
-> route table (routes/accountRoutes.ts) -> validator, if the route takes input
-> controller (accountController.listAccounts)
-> servicesFor(c).accounts.list(userId) [services/accountService.ts]
-> accounts.listForUser(userId) [repositories/accountRepository.ts]
-> supabase.from('accounts').select(...).eq('user_id', userId)
-> safeAccount() on every account
-> returns JSON response
-> a thrown error at any step -> errorHandler -> { success: false, code, message, requestId }
Cron Lifecycle
Cloudflare scheduler fires cron trigger
-> Worker.scheduled(event, env, ctx)
-> ctx.waitUntil(runScheduledJob(event.cron, env, event.scheduledTime)) [jobs/scheduled.ts]
-> createServices(service-role Supabase, Redis, config, cron context)
-> cronPlan(cron, scheduledTime)
-> '* * * * *': dispatch
automationProcessor.run() [50 subrequests, stops starting sends after 50 seconds]
-> claim due reply jobs (accounts take turns; paused, cooling down or limited
accounts are left out), reserve within limits, send the private reply,
record the outcome [see data-flows/AUTOMATION.md]
automationFollowups.run() [the subrequests the replies did not use, keeping 11]
-> claim due follow-ups, check the follow, send the link or a reminder
automationPublicReplies.run() [the rest, at least 11 subrequests]
-> claim due public replies (accounts take turns; paused accounts and public
cooldowns or limits are left out), reserve within limits, post the reply
under the comment, record the outcome
-> '*/5 * * * *' at the other minutes: reconcile
automationReconciler.run()
-> expire overdue replies (7 days), follow-ups (24 hours) and public replies (7 days)
-> look for up to 4 Needs review replies in the Instagram conversation;
a match is marked accepted, nothing is ever sent
-> look for up to 5 public replies of unknown outcome among the comment's replies;
found: accepted; missing on two full reads: back in the queue once for the dispatch
-> log the queue's health, and a warning when replies wait too long
-> '*/5 * * * *' at minutes 0, 15, 30, 45: publish
scheduledPublisher.publishDue()
-> posts.listDue(), posts.listStale(cutoff) [no userId filter - scans all users;
stale = publishing, claim older than 20 minutes]
-> for each post: posts.claim(post) [publishing, claimed_at = now, attempts + 1;
null when another run won]
-> check a saved container first, else create it and save data.containerId,
then wait for and publish the container
-> posts.updateById(id, { status: 'published' })
or back to 'pending' for a temporary failure (up to 3 attempts), or 'failed'
-> '0 0 * * *':
tokenRefresh.refreshExpiringInstagramTokens()
-> accounts.listByPlatform('instagram') [all users, filtered in SQL]
-> for each expiring token: refreshLongLivedToken()
-> accounts.updateById(id, { accessToken, tokenExpiresAt })
retention.purge()
-> delete automation comments older than 90 days in batches of 1,000 (up to 20,000 a day),
audit events older than a year
Webhook Lifecycle (Instagram Comments)
Meta -> POST /api/webhooks/instagram
-> CORS, request id, security headers, clients, per-IP rate limit (no requireAuth)
-> verifyMetaWebhook: 1 MB limit, X-Hub-Signature-256 over the raw body [401 when wrong]
-> automationWebhook.ingest(): rules for the Instagram account, match, store every comment once;
a message answering a followers-only reply creates a follow-up job
-> 200 { success: true } [5xx when storing fails; Meta retries]
-> ctx.waitUntil(automationProcessor.run({ ids }), then automationFollowups.run({ ids })) after the response
The full flow, limits and failure handling are in docs/architecture/data-flows/AUTOMATION.md.
Data Storage Strategy
| Data type | Where stored | Why |
|---|---|---|
| User accounts (auth) | Supabase Auth | Built-in JWT issuance |
| YouTube/Instagram accounts | accounts table, data jsonb |
Flexible schema across platforms |
| User settings (GitHub repository) | settings table, value jsonb |
Generic key-value per user |
| Scheduled posts | scheduled_posts table |
Queryable by status + scheduled_at |
| Comment automation rules, received comments, reply attempts, follow-up messages, account controls | automation_rules, automation_comments, automation_attempts, automation_followups, automation_accounts tables |
The comment rows are also the reply queue, so dedupe, claims and limits are enforced in one database |
| Audit events | audit_events table |
Who connected, removed or changed what; kept a year |
| Video/media analytics | Upstash Redis (videos: / ig: keys) |
Avoids repeated heavy API calls |
| Auth email counts | Upstash Redis (email:day: / email:to: keys, addresses hashed) |
Caps verification emails at the provider's free daily quota |
| Media files (for publishing) | Cloudinary CDN | Instagram requires public URL |
| API secrets | Cloudflare Worker secrets | Encrypted at rest, never in responses |
Environment Configuration
See docs/architecture/cloudflare/WORKERS.md for the full list of required Worker secrets and how to set them.
Frontend env vars are documented in docs/architecture/cloudflare/PAGES.md.
The public marketing site and the user and self-hosting documentation are a separate static site in the site/ workspace, deployed to the twiligent-site Pages project at https://twiligent.ujjwal.fyi (docs under /docs). It makes no API calls and needs no secrets; its sign-in and sign-up buttons link to the app. The app keeps a /docs/* route that sends old links to the same page on the site. Self-hosters do not need the site to run Twiligent. See docs/architecture/site/SITE.md.
Key Architectural Invariants
The Worker never receives binary file uploads. Cloudinary handles file ingestion directly from the browser. The Worker only receives Cloudinary URLs.
Redis is always optional. Every cache call goes through
analyticsCacheRepository, which catches Redis failures, so a miss falls back to live APIs, and rate limits fall back to per-isolate memory. The email counters inemailQuotaRepositoryfall back to Resend's own daily quota. Removing Upstash credentials degrades performance and loosens rate limits but never breaks functionality.The service-role key is never used on the frontend. The frontend uses only the anon key (for Supabase Auth client-side). The service-role key is a Worker secret. Sign-up and sign-in go from the browser to Supabase Auth directly; the Worker only sends the emails Supabase hands it through the signed Send Email hook (see
docs/architecture/data-flows/AUTHENTICATION.md).safeAccount()is the only place access tokens are stripped. Repositories return full accounts; every account in a response passes through this function first.Repositories are the only files that contain Supabase queries. Each table has one repository in
backend/repositories/; services call its methods and never callsupabase.from()directly. User-scoped methods end inForUserand always filter by the verified user id.Every
/apirequest is rate limited. Per IP before authentication, per user after it, and tighter on routes that spend platform API quota. Seedocs/architecture/backend/RATE_LIMITING.md.api.tsis the only file that callsfetch()in the frontend. Components callapi.methodName()- they never construct fetch calls.The API contract is typed once, in
shared/types/. Controllers check every response against it withsatisfies, andapi.tsmethods return the same types, sonpm run typecheckcatches a shape change on both sides. CI runs the type check before every deploy, because Wrangler and Vite strip types without checking them. Seedocs/guidelines/TYPESCRIPT.md.Every failure has one shape. Services throw typed errors from
backend/lib/errors.ts, andmiddleware/errorHandler.tsanswers each with a 4xx or 5xx status and{ success: false, code, message, requestId }. A 2xx response is always a success. Seedocs/architecture/backend/REQUEST_PIPELINE.md.