Skip to content

AutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.

fix(platform): split public and private media storage (#15193)

Krzysztof Czerwinski666e3304d198
diff --git a/autogpt_platform/backend/.env.default b/autogpt_platform/backend/.env.default
index acbcaa8be6..7ba576e516 100644
--- a/autogpt_platform/backend/.env.default
+++ b/autogpt_platform/backend/.env.default
@@ -82,9 +82,13 @@ FRONTEND_BASE_URL=http://localhost:3000
# same format as BACKEND_CORS_ALLOW_ORIGINS. Self-hosting needs nothing here.
# TRUSTED_FRONTEND_ORIGINS=["regex:https://autogpt-pr-\\d+\\.vercel\\.app"]
-# Optional GCS media bucket. When empty, marketplace media uses local store-media/
-# beside WORKSPACE_STORAGE_DIR (or the backend data directory when unset).
+# Optional split GCS buckets. Public media contains only explicitly published
+# marketplace assets and OAuth app logos. Private storage contains user media,
+# workspaces, transcripts, and temporary uploads. MEDIA_GCS_BUCKET_NAME remains
+# a backwards-compatible fallback for self-hosted single-bucket deployments.
MEDIA_GCS_BUCKET_NAME=
+PUBLIC_SITE_MEDIA_BUCKET=
+PRIVATE_USER_DATA_BUCKET=
## ===== API KEYS AND OAUTH CREDENTIALS ===== ##
# All API keys below are optional - only add what you need
diff --git a/autogpt_platform/backend/agents/StoreAgent_rows.csv b/autogpt_platform/backend/agents/StoreAgent_rows.csv
index 44a5e052fc..1b11331c1e 100644
--- a/autogpt_platform/backend/agents/StoreAgent_rows.csv
+++ b/autogpt_platform/backend/agents/StoreAgent_rows.csv
@@ -1,7 +1,7 @@
listing_id,storeListingVersionId,slug,agent_name,agent_video,agent_image,featured,sub_heading,description,categories,useForOnboarding,is_available
-6e60a900-9d7d-490e-9af2-a194827ed632,d85882b8-633f-44ce-a315-c20a8c123d19,flux-ai-image-generator,Flux AI Image Generator,,"[""https://storage.googleapis.com/agpt-prod-website-artifacts/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/ca154dd1-140e-454c-91bd-2d8a00de3f08.jpg"",""https://storage.googleapis.com/agpt-prod-website-artifacts/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/577d995d-bc38-40a9-a23f-1f30f5774bdb.jpg"",""https://storage.googleapis.com/agpt-prod-website-artifacts/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/415db1b7-115c-43ab-bd6c-4e9f7ef95be1.jpg""]",false,Transform ideas into breathtaking images,"Transform ideas into breathtaking images with this AI-powered Image Generator. Using cutting-edge Flux AI technology, the tool crafts highly detailed, photorealistic visuals from simple text prompts. Perfect for artists, marketers, and content creators, this generator produces unique images tailored to user specifications. From fantastical scenes to lifelike portraits, users can unleash creativity with professional-quality results in seconds. Easy to use and endlessly versatile, bring imagination to life with the AI Image Generator today!","[""creative""]",false,true
-f11fc6e9-6166-4676-ac5d-f07127b270c1,c775f60d-b99f-418b-8fe0-53172258c3ce,youtube-transcription-scraper,YouTube Transcription Scraper,https://youtu.be/H8S3pU68lGE,"[""https://storage.googleapis.com/agpt-prod-website-artifacts/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/65bce54b-0124-4b0d-9e3e-f9b89d0dc99e.jpg""]",false,Fetch the transcriptions from the most popular YouTube videos in your chosen topic,"Effortlessly gather transcriptions from multiple YouTube videos with this agent. It scrapes and compiles video transcripts into a clean, organized list, making it easy to extract insights, quotes, or content from various sources in one go. Ideal for researchers, content creators, and marketers looking to quickly analyze or repurpose video content.","[""writing""]",false,true
-17908889-b599-4010-8e4f-bed19b8f3446,6e16e65a-ad34-4108-b4fd-4a23fced5ea2,business-ownerceo-finder,Decision Maker Lead Finder,,"[""https://storage.googleapis.com/agpt-prod-website-artifacts/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/1020d94e-b6a2-4fa7-bbdf-2c218b0de563.jpg""]",false,Contact CEOs today,"Find the key decision-makers you need, fast.
+6e60a900-9d7d-490e-9af2-a194827ed632,d85882b8-633f-44ce-a315-c20a8c123d19,flux-ai-image-generator,Flux AI Image Generator,,"[""https://storage.googleapis.com/agpt-prod-public-site-media/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/ca154dd1-140e-454c-91bd-2d8a00de3f08.jpg"",""https://storage.googleapis.com/agpt-prod-public-site-media/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/577d995d-bc38-40a9-a23f-1f30f5774bdb.jpg"",""https://storage.googleapis.com/agpt-prod-public-site-media/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/415db1b7-115c-43ab-bd6c-4e9f7ef95be1.jpg""]",false,Transform ideas into breathtaking images,"Transform ideas into breathtaking images with this AI-powered Image Generator. Using cutting-edge Flux AI technology, the tool crafts highly detailed, photorealistic visuals from simple text prompts. Perfect for artists, marketers, and content creators, this generator produces unique images tailored to user specifications. From fantastical scenes to lifelike portraits, users can unleash creativity with professional-quality results in seconds. Easy to use and endlessly versatile, bring imagination to life with the AI Image Generator today!","[""creative""]",false,true
+f11fc6e9-6166-4676-ac5d-f07127b270c1,c775f60d-b99f-418b-8fe0-53172258c3ce,youtube-transcription-scraper,YouTube Transcription Scraper,https://youtu.be/H8S3pU68lGE,"[""https://storage.googleapis.com/agpt-prod-public-site-media/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/65bce54b-0124-4b0d-9e3e-f9b89d0dc99e.jpg""]",false,Fetch the transcriptions from the most popular YouTube videos in your chosen topic,"Effortlessly gather transcriptions from multiple YouTube videos with this agent. It scrapes and compiles video transcripts into a clean, organized list, making it easy to extract insights, quotes, or content from various sources in one go. Ideal for researchers, content creators, and marketers looking to quickly analyze or repurpose video content.","[""writing""]",false,true
+17908889-b599-4010-8e4f-bed19b8f3446,6e16e65a-ad34-4108-b4fd-4a23fced5ea2,business-ownerceo-finder,Decision Maker Lead Finder,,"[""https://storage.googleapis.com/agpt-prod-public-site-media/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/1020d94e-b6a2-4fa7-bbdf-2c218b0de563.jpg""]",false,Contact CEOs today,"Find the key decision-makers you need, fast.
This agent identifies business owners or CEOs of local companies in any area you choose. Simply enter what kind of businesses you’re looking for and where, and it will:
@@ -16,14 +16,14 @@ Perfect for:
* Researchers looking to connect with business leaders
Save hours of manual searching and get straight to the people who matter most.","[""business""]",true,true
-72beca1d-45ea-4403-a7ce-e2af168ee428,415b7352-0dc6-4214-9d87-0ad3751b711d,smart-meeting-brief,Smart Meeting Prep,https://youtu.be/9ydZR2hkxaY,"[""https://storage.googleapis.com/agpt-prod-website-artifacts/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/2f116ce1-63ae-4d39-a5cd-f514defc2b97.png"",""https://storage.googleapis.com/agpt-prod-website-artifacts/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/0a71a60a-2263-4f12-9836-9c76ab49f155.png"",""https://storage.googleapis.com/agpt-prod-website-artifacts/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/95327695-9184-403c-907a-a9d3bdafa6a5.png"",""https://storage.googleapis.com/agpt-prod-website-artifacts/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/2bc77788-790b-47d4-8a61-ce97b695e9f5.png""]",true,Business meeting briefings delivered daily,"Never walk into a meeting unprepared again. Every day at 4 pm, the Smart Meeting Prep Agent scans your calendar for tomorrow's external meetings. It reviews your past email exchanges, researches each participant's background and role, and compiles the insights into a concise briefing, so you can close your workday ready for tomorrow's calls.
+72beca1d-45ea-4403-a7ce-e2af168ee428,415b7352-0dc6-4214-9d87-0ad3751b711d,smart-meeting-brief,Smart Meeting Prep,https://youtu.be/9ydZR2hkxaY,"[""https://storage.googleapis.com/agpt-prod-public-site-media/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/2f116ce1-63ae-4d39-a5cd-f514defc2b97.png"",""https://storage.googleapis.com/agpt-prod-public-site-media/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/0a71a60a-2263-4f12-9836-9c76ab49f155.png"",""https://storage.googleapis.com/agpt-prod-public-site-media/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/95327695-9184-403c-907a-a9d3bdafa6a5.png"",""https://storage.googleapis.com/agpt-prod-public-site-media/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/2bc77788-790b-47d4-8a61-ce97b695e9f5.png""]",true,Business meeting briefings delivered daily,"Never walk into a meeting unprepared again. Every day at 4 pm, the Smart Meeting Prep Agent scans your calendar for tomorrow's external meetings. It reviews your past email exchanges, researches each participant's background and role, and compiles the insights into a concise briefing, so you can close your workday ready for tomorrow's calls.
How It Works
1. At 4 pm, the agent scans your calendar and identifies external meetings scheduled for the next day.
2. It reviews recent email threads with each participant to surface key relationship history and communication context.
3. It conducts online research to gather publicly available information on roles, company backgrounds, and relevant professional data.
4. It produces a unified briefing for each participant, including past exchange highlights, profile notes, and strategic conversation points.","[""personal""]",true,true
-9fa5697a-617b-4fae-aea0-7dbbed279976,b8ceb480-a7a2-4c90-8513-181a49f7071f,automated-support-ai,Automated Support Agent,https://youtu.be/nBMfu_5sgDA,"[""https://storage.googleapis.com/agpt-prod-website-artifacts/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/ed56febc-2205-4179-9e7e-505d8500b66c.png""]",true,Automate up to 80 percent of inbound support emails,"Overview:
+9fa5697a-617b-4fae-aea0-7dbbed279976,b8ceb480-a7a2-4c90-8513-181a49f7071f,automated-support-ai,Automated Support Agent,https://youtu.be/nBMfu_5sgDA,"[""https://storage.googleapis.com/agpt-prod-public-site-media/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/ed56febc-2205-4179-9e7e-505d8500b66c.png""]",true,Automate up to 80 percent of inbound support emails,"Overview:
Support teams spend countless hours on basic tickets. This agent automates repetitive customer support tasks. It reads incoming requests, researches your knowledge base, and responds automatically when confident. When unsure, it escalates to a human for final resolution.
How it Works:
@@ -33,8 +33,8 @@ It measures confidence in the answer found and either replies directly or escala
Business Value:
Automating the easy 80 percent of support tickets allows your team to focus on high-value, complex customer issues, improving efficiency and response times.","[""business""]",false,true
-2bdac92b-a12c-4131-bb46-0e3b89f61413,31daf49d-31d3-476b-aa4c-099abc59b458,unspirational-poster-maker,Unspirational Poster Maker,,"[""https://storage.googleapis.com/agpt-prod-website-artifacts/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/6a490dac-27e5-405f-a4c4-8d1c55b85060.jpg"",""https://storage.googleapis.com/agpt-prod-website-artifacts/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/d343fbb5-478c-4e38-94df-4337293b61f1.jpg""]",false,Because adulting is hard,"This witty AI agent generates hilariously relatable ""motivational"" posters that tackle the everyday struggles of procrastination, overthinking, and workplace chaos with a blend of absurdity and sarcasm. From goldfish facing impossible tasks to cats in existential crises, The Unspirational Poster Maker designs tongue-in-cheek graphics and captions that mock productivity clichés and embrace our collective struggles to ""get it together."" Perfect for adding a touch of humour to the workday, these posters remind us that sometimes, all we can do is laugh at the chaos.","[""creative""]",false,true
-9adf005e-2854-4cc7-98cf-f7103b92a7b7,a03b0d8c-4751-43d6-a54e-c3b7856ba4e3,ai-shortform-video-generator-create-viral-ready-content,AI Video Generator,,"[""https://storage.googleapis.com/agpt-prod-website-artifacts/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/8d2670b9-fea5-4966-a597-0a4511bffdc3.png"",""https://storage.googleapis.com/agpt-prod-website-artifacts/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/aabe8aec-0110-4ce7-a259-4f86fe8fe07d.png""]",false,Create Viral-Ready Shorts Content in Seconds,"OVERVIEW
+2bdac92b-a12c-4131-bb46-0e3b89f61413,31daf49d-31d3-476b-aa4c-099abc59b458,unspirational-poster-maker,Unspirational Poster Maker,,"[""https://storage.googleapis.com/agpt-prod-public-site-media/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/6a490dac-27e5-405f-a4c4-8d1c55b85060.jpg"",""https://storage.googleapis.com/agpt-prod-public-site-media/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/d343fbb5-478c-4e38-94df-4337293b61f1.jpg""]",false,Because adulting is hard,"This witty AI agent generates hilariously relatable ""motivational"" posters that tackle the everyday struggles of procrastination, overthinking, and workplace chaos with a blend of absurdity and sarcasm. From goldfish facing impossible tasks to cats in existential crises, The Unspirational Poster Maker designs tongue-in-cheek graphics and captions that mock productivity clichés and embrace our collective struggles to ""get it together."" Perfect for adding a touch of humour to the workday, these posters remind us that sometimes, all we can do is laugh at the chaos.","[""creative""]",false,true
+9adf005e-2854-4cc7-98cf-f7103b92a7b7,a03b0d8c-4751-43d6-a54e-c3b7856ba4e3,ai-shortform-video-generator-create-viral-ready-content,AI Video Generator,,"[""https://storage.googleapis.com/agpt-prod-public-site-media/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/8d2670b9-fea5-4966-a597-0a4511bffdc3.png"",""https://storage.googleapis.com/agpt-prod-public-site-media/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/aabe8aec-0110-4ce7-a259-4f86fe8fe07d.png""]",false,Create Viral-Ready Shorts Content in Seconds,"OVERVIEW
Transform any trending headline or broad topic into a polished, vertical short-form video in a single run.
The agent automates research, scriptwriting, metadata creation, and Revid.ai rendering, returning one ready-to-publish MP4 plus its title, script and hashtags.
@@ -63,7 +63,7 @@ BUSINESS VALUE
IMPORTANT NOTES
- The agent outputs exactly one video per execution. Run it again for additional shorts.
- Video rendering time varies; AI-generated footage may take several minutes.","[""writing""]",false,true
-864e48ef-fee5-42c1-b6a4-2ae139db9fc1,55d40473-0f31-4ada-9e40-d3a7139fcbd4,automated-blog-writer,Automated SEO Blog Writer,https://youtu.be/nKcDCbDVobs,"[""https://storage.googleapis.com/agpt-prod-website-artifacts/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/2dd5f95b-5b30-4bf8-a11b-bac776c5141a.jpg""]",true,"Automate research, writing, and publishing for high-ranking blog posts","Scale your blog with a fully automated content engine. The Automated SEO Blog Writer learns your brand voice, finds high-demand keywords, and creates SEO-optimized articles that attract organic traffic and boost visibility.
+864e48ef-fee5-42c1-b6a4-2ae139db9fc1,55d40473-0f31-4ada-9e40-d3a7139fcbd4,automated-blog-writer,Automated SEO Blog Writer,https://youtu.be/nKcDCbDVobs,"[""https://storage.googleapis.com/agpt-prod-public-site-media/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/2dd5f95b-5b30-4bf8-a11b-bac776c5141a.jpg""]",true,"Automate research, writing, and publishing for high-ranking blog posts","Scale your blog with a fully automated content engine. The Automated SEO Blog Writer learns your brand voice, finds high-demand keywords, and creates SEO-optimized articles that attract organic traffic and boost visibility.
How it works:
@@ -80,7 +80,7 @@ Use cases:
• Strategists: Automate execution, focus on strategy.
• Marketers: Drive steady organic growth.
• Local businesses: Capture nearby search traffic.","[""writing""]",false,true
-6046f42e-eb84-406f-bae0-8e052064a4fa,a548e507-09a7-4b30-909c-f63fcda10fff,lead-finder-local-businesses,Lead Finder,,"[""https://storage.googleapis.com/agpt-prod-website-artifacts/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/abd6605f-d5f8-426b-af36-052e8ba5044f.webp""]",false,Auto-Prospect Like a Pro,"Turbo-charge your local lead generation with the AutoGPT Marketplace’s top Google Maps prospecting agent. “Lead Finder: Local Businesses” delivers verified, ready-to-contact prospects in any niche and city—so you can focus on closing, not searching.
+6046f42e-eb84-406f-bae0-8e052064a4fa,a548e507-09a7-4b30-909c-f63fcda10fff,lead-finder-local-businesses,Lead Finder,,"[""https://storage.googleapis.com/agpt-prod-public-site-media/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/abd6605f-d5f8-426b-af36-052e8ba5044f.webp""]",false,Auto-Prospect Like a Pro,"Turbo-charge your local lead generation with the AutoGPT Marketplace’s top Google Maps prospecting agent. “Lead Finder: Local Businesses” delivers verified, ready-to-contact prospects in any niche and city—so you can focus on closing, not searching.
**WHAT IT DOES**
• Searches Google Maps via the official API (no scraping)
@@ -103,7 +103,7 @@ Use cases:
Stop scrolling directories—start filling your pipeline. Start now and let AI prospect while you profit.
→ Click *Add to Library* and own your market today.","[""business""]",true,true
-f623c862-24e9-44fc-8ce8-d8282bb51ad2,eafa21d3-bf14-4f63-a97f-a5ee41df83b3,linkedin-post-generator,LinkedIn Post Generator,,"[""https://storage.googleapis.com/agpt-prod-website-artifacts/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/297f6a8e-81a8-43e2-b106-c7ad4a5662df.png"",""https://storage.googleapis.com/agpt-prod-website-artifacts/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/fceebdc1-aef6-4000-97fc-4ef587f56bda.png""]",false,Auto‑craft LinkedIn gold,"Create research‑driven, high‑impact LinkedIn posts in minutes. This agent searches YouTube for the best videos on your chosen topic, pulls their transcripts, and distils the most valuable insights into a polished post ready for your company page or personal feed.
+f623c862-24e9-44fc-8ce8-d8282bb51ad2,eafa21d3-bf14-4f63-a97f-a5ee41df83b3,linkedin-post-generator,LinkedIn Post Generator,,"[""https://storage.googleapis.com/agpt-prod-public-site-media/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/297f6a8e-81a8-43e2-b106-c7ad4a5662df.png"",""https://storage.googleapis.com/agpt-prod-public-site-media/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/fceebdc1-aef6-4000-97fc-4ef587f56bda.png""]",false,Auto‑craft LinkedIn gold,"Create research‑driven, high‑impact LinkedIn posts in minutes. This agent searches YouTube for the best videos on your chosen topic, pulls their transcripts, and distils the most valuable insights into a polished post ready for your company page or personal feed.
FEATURES
• Automated YouTube research – discovers and analyses top‑ranked videos so you don’t have to
@@ -125,7 +125,7 @@ USE CASES
WHY YOU’LL LOVE IT
Save hours of manual research, avoid surface‑level hot‑takes, and publish posts that showcase real expertise—without the heavy lift.","[""writing""]",true,true
-7d4120ad-b6b3-4419-8bdb-7dd7d350ef32,e7bb29a1-23c7-4fee-aa3b-5426174b8c52,youtube-to-linkedin-post-converter,YouTube to LinkedIn Post Converter,,"[""https://storage.googleapis.com/agpt-prod-website-artifacts/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/f084b326-a708-4396-be51-7ba59ad2ef32.png""]",false,Transform Your YouTube Videos into Engaging LinkedIn Posts with AI,"WHAT IT DOES:
+7d4120ad-b6b3-4419-8bdb-7dd7d350ef32,e7bb29a1-23c7-4fee-aa3b-5426174b8c52,youtube-to-linkedin-post-converter,YouTube to LinkedIn Post Converter,,"[""https://storage.googleapis.com/agpt-prod-public-site-media/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/f084b326-a708-4396-be51-7ba59ad2ef32.png""]",false,Transform Your YouTube Videos into Engaging LinkedIn Posts with AI,"WHAT IT DOES:
This agent converts YouTube video content into a LinkedIn post by analyzing the video's transcript. It provides you with a tailored post that reflects the core ideas, key takeaways, and tone of the original video, optimizing it for engagement on LinkedIn.
HOW IT WORKS:
@@ -146,7 +146,7 @@ OUTPUT:
- LinkedIn Post – A well-crafted, AI-generated LinkedIn post with a professional tone, based on the video content and your specified preferences
Perfect for content creators, marketers, and professionals who want to repurpose YouTube videos for LinkedIn and boost their professional branding.","[""writing""]",false,true
-c61d6a83-ea48-4df8-b447-3da2d9fe5814,00fdd42c-a14c-4d19-a567-65374ea0e87f,personalized-morning-coffee-newsletter,Personal Newsletter,,"[""https://storage.googleapis.com/agpt-prod-website-artifacts/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/f4b38e4c-8166-4caf-9411-96c9c4c82d4c.png""]",false,Start your day with personalized AI newsletters that deliver credibility and context for every interest or mood.,"This Personal Newsletter Agent provides a bespoke daily digest on your favorite topics and tone. Whether you prefer industry insights, lighthearted reads, or breaking news, this agent crafts your own unique newsletter to keep you informed and entertained.
+c61d6a83-ea48-4df8-b447-3da2d9fe5814,00fdd42c-a14c-4d19-a567-65374ea0e87f,personalized-morning-coffee-newsletter,Personal Newsletter,,"[""https://storage.googleapis.com/agpt-prod-public-site-media/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/f4b38e4c-8166-4caf-9411-96c9c4c82d4c.png""]",false,Start your day with personalized AI newsletters that deliver credibility and context for every interest or mood.,"This Personal Newsletter Agent provides a bespoke daily digest on your favorite topics and tone. Whether you prefer industry insights, lighthearted reads, or breaking news, this agent crafts your own unique newsletter to keep you informed and entertained.
How It Works
@@ -162,7 +162,7 @@ Use Cases
• Executives: Get a daily digest of market updates and leadership insights.
• Marketers: Receive curated creative trends and campaign inspiration.
• Entrepreneurs: Stay updated on your industry without information overload.","[""research""]",true,true
-e2e49cfc-4a39-4d62-a6b3-c095f6d025ff,fc2c9976-0962-4625-a27b-d316573a9e7f,email-address-finder,Email Scout - Contact Finder Assistant,,"[""https://storage.googleapis.com/agpt-prod-website-artifacts/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/da8a690a-7a8b-4c1d-b6f8-e2f840c0205d.jpg"",""https://storage.googleapis.com/agpt-prod-website-artifacts/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/6a2ac25c-1609-4881-8140-e6da2421afb3.jpg"",""https://storage.googleapis.com/agpt-prod-website-artifacts/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/26179263-fe06-45bd-b6a0-0754660a0a46.jpg""]",false,Find contact details from name and location using AI search,"Finding someone's professional email address can be time-consuming and frustrating. Manual searching across multiple websites, social profiles, and business directories often leads to dead ends or outdated information.
+e2e49cfc-4a39-4d62-a6b3-c095f6d025ff,fc2c9976-0962-4625-a27b-d316573a9e7f,email-address-finder,Email Scout - Contact Finder Assistant,,"[""https://storage.googleapis.com/agpt-prod-public-site-media/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/da8a690a-7a8b-4c1d-b6f8-e2f840c0205d.jpg"",""https://storage.googleapis.com/agpt-prod-public-site-media/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/6a2ac25c-1609-4881-8140-e6da2421afb3.jpg"",""https://storage.googleapis.com/agpt-prod-public-site-media/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/26179263-fe06-45bd-b6a0-0754660a0a46.jpg""]",false,Find contact details from name and location using AI search,"Finding someone's professional email address can be time-consuming and frustrating. Manual searching across multiple websites, social profiles, and business directories often leads to dead ends or outdated information.
Email Scout automates this process by intelligently searching across publicly available sources when you provide a person's name and location. Simply input basic information like ""Tim Cook, USA"" or ""Sarah Smith, London"" and let the AI assistant do the work of finding potential contact details.
@@ -175,7 +175,7 @@ Key Features:
Perfect for recruiters, business development professionals, researchers, and anyone needing to establish professional contact.
Note: This tool searches only publicly available information. Search results depend on what contact information people have made public. Some searches may not yield results if the information isn't publicly accessible.","[""""]",false,true
-81bcc372-0922-4a36-bc35-f7b1e51d6939,e437cc95-e671-489d-b915-76561fba8c7f,ai-youtube-to-blog-converter,YouTube Video to SEO Blog Writer,,"[""https://storage.googleapis.com/agpt-prod-website-artifacts/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/239e5a41-2515-4e1c-96ef-31d0d37ecbeb.webp"",""https://storage.googleapis.com/agpt-prod-website-artifacts/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/c7d96966-786f-4be6-ad7d-3a51c84efc0e.png"",""https://storage.googleapis.com/agpt-prod-website-artifacts/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/0275a74c-e2c2-4e29-a6e4-3a616c3c35dd.png""]",false,One link. One click. One powerful blog post.,"Effortlessly transform your YouTube videos into high-quality, SEO-optimized blog posts.
+81bcc372-0922-4a36-bc35-f7b1e51d6939,e437cc95-e671-489d-b915-76561fba8c7f,ai-youtube-to-blog-converter,YouTube Video to SEO Blog Writer,,"[""https://storage.googleapis.com/agpt-prod-public-site-media/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/239e5a41-2515-4e1c-96ef-31d0d37ecbeb.webp"",""https://storage.googleapis.com/agpt-prod-public-site-media/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/c7d96966-786f-4be6-ad7d-3a51c84efc0e.png"",""https://storage.googleapis.com/agpt-prod-public-site-media/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/0275a74c-e2c2-4e29-a6e4-3a616c3c35dd.png""]",false,One link. One click. One powerful blog post.,"Effortlessly transform your YouTube videos into high-quality, SEO-optimized blog posts.
Your videos deserve a second life—in writing.
Make your content work twice as hard by repurposing it into engaging, searchable articles.
@@ -198,8 +198,8 @@ FEATURES
Whether you're looking to grow your blog, boost SEO, or simply get more out of your content, the AI YouTube-to-Blog Converter makes it effortless.
","[""writing""]",true,true
-5c3510d2-fc8b-4053-8e19-67f53c86eb1a,f2cc74bb-f43f-4395-9c35-ecb30b5b4fc9,ai-webpage-copy-improver,AI Webpage Copy Improver,,"[""https://storage.googleapis.com/agpt-prod-website-artifacts/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/d562d26f-5891-4b09-8859-fbb205972313.jpg""]",false,Boost Your Website's Search Engine Performance,"Elevate your web content with this powerful AI Webpage Copy Improver. Designed for marketers, SEO specialists, and web developers, this tool analyses and enhances website copy for maximum impact. Using advanced language models, it optimizes text for better clarity, SEO performance, and increased conversion rates. The AI examines your existing content, identifies areas for improvement, and generates refined copy that maintains your brand voice while boosting engagement. From homepage headlines to product descriptions, transform your web presence with AI-driven insights. Improve readability, incorporate targeted keywords, and craft compelling calls-to-action - all with the click of a button. Take your digital marketing to the next level with the AI Webpage Copy Improver.","[""marketing""]",true,true
-94d03bd3-7d44-4d47-b60c-edb2f89508d6,b6f6f0d3-49f4-4e3b-8155-ffe9141b32c0,domain-name-finder,Domain Name Finder,,"[""https://storage.googleapis.com/agpt-prod-website-artifacts/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/28545e09-b2b8-4916-b4c6-67f982510a78.jpeg""]",false,Instantly generate brand-ready domain names that are actually available,"Overview:
+5c3510d2-fc8b-4053-8e19-67f53c86eb1a,f2cc74bb-f43f-4395-9c35-ecb30b5b4fc9,ai-webpage-copy-improver,AI Webpage Copy Improver,,"[""https://storage.googleapis.com/agpt-prod-public-site-media/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/d562d26f-5891-4b09-8859-fbb205972313.jpg""]",false,Boost Your Website's Search Engine Performance,"Elevate your web content with this powerful AI Webpage Copy Improver. Designed for marketers, SEO specialists, and web developers, this tool analyses and enhances website copy for maximum impact. Using advanced language models, it optimizes text for better clarity, SEO performance, and increased conversion rates. The AI examines your existing content, identifies areas for improvement, and generates refined copy that maintains your brand voice while boosting engagement. From homepage headlines to product descriptions, transform your web presence with AI-driven insights. Improve readability, incorporate targeted keywords, and craft compelling calls-to-action - all with the click of a button. Take your digital marketing to the next level with the AI Webpage Copy Improver.","[""marketing""]",true,true
+94d03bd3-7d44-4d47-b60c-edb2f89508d6,b6f6f0d3-49f4-4e3b-8155-ffe9141b32c0,domain-name-finder,Domain Name Finder,,"[""https://storage.googleapis.com/agpt-prod-public-site-media/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/28545e09-b2b8-4916-b4c6-67f982510a78.jpeg""]",false,Instantly generate brand-ready domain names that are actually available,"Overview:
Finding a domain name that fits your brand shouldn’t take hours of searching and failed checks. The Domain Name Finder Agent turns your pitch into hundreds of creative, brand-ready domain ideas—filtered by live availability so every result is actionable.
How It Works
@@ -217,7 +217,7 @@ Key Use Cases
• Startup Founders: Quickly find brand-ready domains for MVP launches or rebrands.
• Marketers: Test name options across campaigns with instant availability data.
• Entrepreneurs: Validate ideas faster with instant domain options.","[""business""]",false,true
-7a831906-daab-426f-9d66-bcf98d869426,516d813b-d1bc-470f-add7-c63a4b2c2bad,ai-function,AI Function,,"[""https://storage.googleapis.com/agpt-prod-website-artifacts/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/620e8117-2ee1-4384-89e6-c2ef4ec3d9c9.webp"",""https://storage.googleapis.com/agpt-prod-website-artifacts/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/476259e2-5a79-4a7b-8e70-deeebfca70d7.png""]",false,Never Code Again,"AI FUNCTION MAGIC
+7a831906-daab-426f-9d66-bcf98d869426,516d813b-d1bc-470f-add7-c63a4b2c2bad,ai-function,AI Function,,"[""https://storage.googleapis.com/agpt-prod-public-site-media/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/620e8117-2ee1-4384-89e6-c2ef4ec3d9c9.webp"",""https://storage.googleapis.com/agpt-prod-public-site-media/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/476259e2-5a79-4a7b-8e70-deeebfca70d7.png""]",false,Never Code Again,"AI FUNCTION MAGIC
Your AI‑powered assistant for turning plain‑English descriptions into working Python functions.
HOW IT WORKS
diff --git a/autogpt_platform/backend/backend/api/features/oauth.py b/autogpt_platform/backend/backend/api/features/oauth.py
index 023a433951..2790a7b41c 100644
--- a/autogpt_platform/backend/backend/api/features/oauth.py
+++ b/autogpt_platform/backend/backend/api/features/oauth.py
@@ -19,7 +19,7 @@ import os
import uuid
from datetime import datetime
from typing import Literal, Optional
-from urllib.parse import urlencode
+from urllib.parse import unquote, urlencode, urlparse
from autogpt_libs.auth import get_user_id
from fastapi import APIRouter, Body, HTTPException, Security, UploadFile, status
@@ -637,9 +637,6 @@ async def update_app_logo(
detail="OAuth App not found",
)
- # Delete the current app logo file (if any and it's in our cloud storage)
- await _delete_app_current_logo_file(app)
-
updated_app = await update_oauth_application(
app_id=app_id,
owner_id=user_id,
@@ -652,6 +649,8 @@ async def update_app_logo(
detail="Application not found or you don't have permission to update it",
)
+ await _delete_app_current_logo_file(app)
+
logger.info(
f"OAuth app {updated_app.name} (#{app_id}) logo updated by user #{user_id}"
)
@@ -696,7 +695,7 @@ async def upload_app_logo(
)
# Check GCS configuration
- if not settings.config.media_gcs_bucket_name:
+ if not settings.config.resolved_public_site_media_bucket:
raise HTTPException(
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
detail="Media storage is not configured",
@@ -775,7 +774,7 @@ async def upload_app_logo(
# Upload to GCS
try:
async with async_storage.Storage() as async_client:
- bucket_name = settings.config.media_gcs_bucket_name
+ bucket_name = settings.config.resolved_public_site_media_bucket
await async_client.upload(
bucket_name, storage_path, file_bytes, content_type=content_type
@@ -789,9 +788,6 @@ async def upload_app_logo(
detail="Failed to upload logo",
)
- # Delete the current app logo file (if any and it's in our cloud storage)
- await _delete_app_current_logo_file(app)
-
# Update the app with the new logo URL
updated_app = await update_oauth_application(
app_id=app_id,
@@ -805,6 +801,8 @@ async def upload_app_logo(
detail="Application not found or you don't have permission to update it",
)
+ await _delete_app_current_logo_file(app)
+
logger.info(
f"OAuth app {updated_app.name} (#{app_id}) logo uploaded by user #{user_id}"
)
@@ -816,18 +814,35 @@ async def _delete_app_current_logo_file(app: OAuthApplicationInfo):
"""
Delete the current logo file for the given app, if there is one in our cloud storage
"""
- bucket_name = settings.config.media_gcs_bucket_name
- storage_base_url = f"https://storage.googleapis.com/{bucket_name}/"
+ if not app.logo_url:
+ return
+
+ parsed = urlparse(app.logo_url)
+ if parsed.scheme != "https" or parsed.netloc != "storage.googleapis.com":
+ return
+
+ bucket_and_path = unquote(parsed.path).lstrip("/").split("/", 1)
+ if len(bucket_and_path) != 2:
+ return
+ bucket_name, object_path = bucket_and_path
+
+ allowed_buckets = {
+ settings.config.resolved_public_site_media_bucket,
+ settings.config.resolved_private_user_data_bucket,
+ settings.config.media_gcs_bucket_name,
+ }
+ allowed_buckets.discard("")
+ expected_prefix = f"oauth-apps/{app.id}/logo/"
+ if bucket_name not in allowed_buckets or not object_path.startswith(
+ expected_prefix
+ ):
+ return
- if app.logo_url and app.logo_url.startswith(storage_base_url):
- # Parse blob path from URL: https://storage.googleapis.com/{bucket}/{path}
- old_path = app.logo_url.replace(storage_base_url, "")
- try:
- async with async_storage.Storage() as async_client:
- await async_client.delete(bucket_name, old_path)
- logger.info(f"Deleted old logo for OAuth app #{app.id}: {old_path}")
- except Exception as e:
- # Log but don't fail - the new logo was uploaded successfully
- logger.warning(
- f"Failed to delete old logo for OAuth app #{app.id}: {e}", exc_info=e
- )
+ try:
+ async with async_storage.Storage() as async_client:
+ await async_client.delete(bucket_name, object_path)
+ logger.info(f"Deleted old logo for OAuth app #{app.id}: {object_path}")
+ except Exception as e:
+ logger.warning(
+ f"Failed to delete old logo for OAuth app #{app.id}: {e}", exc_info=e
+ )
diff --git a/autogpt_platform/backend/backend/api/features/oauth_storage_test.py b/autogpt_platform/backend/backend/api/features/oauth_storage_test.py
new file mode 100644
index 0000000000..29251c7e49
--- /dev/null
+++ b/autogpt_platform/backend/backend/api/features/oauth_storage_test.py
@@ -0,0 +1,172 @@
+import io
+from types import SimpleNamespace
+from unittest.mock import AsyncMock, MagicMock, patch
+
+import pytest
+from fastapi import UploadFile
+from starlette.datastructures import Headers
+
+from backend.api.features import oauth
+from backend.api.features.oauth import _delete_app_current_logo_file, settings
+
+
+async def test_logo_url_updates_before_previous_object_is_deleted():
+ app = SimpleNamespace(id="app-123", owner_id="owner", logo_url="old-logo")
+ updated_app = SimpleNamespace(name="Test app")
+ order = []
+
+ async def update(**kwargs):
+ order.append("update")
+ return updated_app
+
+ async def delete(previous):
+ order.append("delete")
+
+ with (
+ patch.object(oauth, "get_oauth_application_by_id", AsyncMock(return_value=app)),
+ patch.object(oauth, "update_oauth_application", side_effect=update),
+ patch.object(oauth, "_delete_app_current_logo_file", side_effect=delete),
+ ):
+ result = await oauth.update_app_logo(
+ "app-123", oauth.UpdateAppLogoRequest(logo_url="new-logo"), user_id="owner"
+ )
+
+ assert result is updated_app
+ assert order == ["update", "delete"]
+
+
+async def test_failed_logo_url_update_keeps_previous_object():
+ app = SimpleNamespace(id="app-123", owner_id="owner", logo_url="old-logo")
+
+ with (
+ patch.object(oauth, "get_oauth_application_by_id", AsyncMock(return_value=app)),
+ patch.object(oauth, "update_oauth_application", AsyncMock(return_value=None)),
+ patch.object(
+ oauth, "_delete_app_current_logo_file", new_callable=AsyncMock
+ ) as delete,
+ pytest.raises(oauth.HTTPException) as error,
+ ):
+ await oauth.update_app_logo(
+ "app-123", oauth.UpdateAppLogoRequest(logo_url="new-logo"), user_id="owner"
+ )
+
+ assert error.value.status_code == 404
+ delete.assert_not_awaited()
+
+
+async def test_logo_reference_updates_before_previous_object_is_deleted(monkeypatch):
+ monkeypatch.setattr(settings.config, "public_site_media_bucket", "public-media")
+ monkeypatch.setattr(settings.config, "private_user_data_bucket", "private-data")
+ monkeypatch.setattr(settings.config, "media_gcs_bucket_name", "legacy-media")
+ app = SimpleNamespace(
+ id="app-123",
+ owner_id="owner",
+ logo_url=(
+ "https://storage.googleapis.com/legacy-media/"
+ "oauth-apps/app-123/logo/old.png"
+ ),
+ )
+ updated_app = SimpleNamespace(name="Test app")
+ order = []
+ update_arguments = {}
+
+ async def update(**kwargs):
+ order.append("update")
+ update_arguments.update(kwargs)
+ return updated_app
+
+ async def delete(previous):
+ order.append("delete")
+
+ client = AsyncMock()
+ context = MagicMock()
+ context.__aenter__ = AsyncMock(return_value=client)
+ context.__aexit__ = AsyncMock(return_value=None)
+ file = UploadFile(
+ filename="logo.png",
+ file=io.BytesIO(b"image"),
+ headers=Headers({"content-type": "image/png"}),
+ )
+
+ with (
+ patch.object(oauth, "get_oauth_application_by_id", AsyncMock(return_value=app)),
+ patch.object(oauth, "update_oauth_application", side_effect=update),
+ patch.object(oauth, "_delete_app_current_logo_file", side_effect=delete),
+ patch.object(oauth, "scan_content_safe", AsyncMock()),
+ patch.object(
+ oauth.Image, "open", return_value=SimpleNamespace(size=(512, 512))
+ ),
+ patch.object(oauth.async_storage, "Storage", return_value=context),
+ ):
+ result = await oauth.upload_app_logo("app-123", file, user_id="owner")
+
+ assert result is updated_app
+ assert order == ["update", "delete"]
+ assert client.upload.await_args.args[0] == "public-media"
+ assert update_arguments["logo_url"].startswith(
+ "https://storage.googleapis.com/public-media/oauth-apps/app-123/logo/"
+ )
+
+
+ ("bucket_name", "setting_name"),
+ [
+ ("public-media", "public_site_media_bucket"),
+ ("legacy-media", "media_gcs_bucket_name"),
+ ],
+)
+async def test_delete_app_logo_supports_current_and_legacy_buckets(
+ monkeypatch, bucket_name, setting_name
+):
+ monkeypatch.setattr(settings.config, "public_site_media_bucket", "public-media")
+ monkeypatch.setattr(settings.config, "private_user_data_bucket", "private-data")
+ monkeypatch.setattr(settings.config, "media_gcs_bucket_name", "legacy-media")
+ assert getattr(settings.config, setting_name) == bucket_name
+
+ client = AsyncMock()
+ context = MagicMock()
+ context.__aenter__ = AsyncMock(return_value=client)
+ context.__aexit__ = AsyncMock(return_value=None)
+ app = SimpleNamespace(
+ id="app-123",
+ logo_url=(
+ f"https://storage.googleapis.com/{bucket_name}/"
+ "oauth-apps/app-123/logo/logo.png"
+ ),
+ )
+
+ with patch(
+ "backend.api.features.oauth.async_storage.Storage", return_value=context
+ ):
+ await _delete_app_current_logo_file(app)
+
+ client.delete.assert_awaited_once_with(
+ bucket_name, "oauth-apps/app-123/logo/logo.png"
+ )
+
+
+ "logo_url",
+ [
+ "https://example.com/public-media/oauth-apps/app-123/logo/logo.png",
+ "https://storage.googleapis.com/other-bucket/oauth-apps/app-123/logo/logo.png",
+ "https://storage.googleapis.com/public-media/oauth-apps/other/logo/logo.png",
+ "https://storage.googleapis.com/public-media/users/app-123/logo/logo.png",
+ ],
+)
+async def test_delete_app_logo_rejects_unmanaged_locations(monkeypatch, logo_url):
+ monkeypatch.setattr(settings.config, "public_site_media_bucket", "public-media")
+ monkeypatch.setattr(settings.config, "private_user_data_bucket", "private-data")
+ monkeypatch.setattr(settings.config, "media_gcs_bucket_name", "legacy-media")
+ app = SimpleNamespace(id="app-123", logo_url=logo_url)
+
+ storage = MagicMock()
+ with patch("backend.api.features.oauth.async_storage.Storage", storage):
+ await _delete_app_current_logo_file(app)
+
+ storage.assert_not_called()
diff --git a/autogpt_platform/backend/backend/api/features/store/db.py b/autogpt_platform/backend/backend/api/features/store/db.py
index 04bef68439..2ed3eb9692 100644
--- a/autogpt_platform/backend/backend/api/features/store/db.py
+++ b/autogpt_platform/backend/backend/api/features/store/db.py
@@ -25,6 +25,7 @@ from backend.util.settings import Settings
from . import exceptions as store_exceptions
from . import model as store_model
+from . import public_media
from .categories import category_filter_values
from .embeddings import ensure_embedding
from .hybrid_search import hybrid_search
@@ -1305,6 +1306,11 @@ async def update_profile(
logger.error(f"Failed to update profile for user {user_id}")
raise DatabaseError("Failed to update profile")
+ if profile.avatar_url is not None:
+ updated_profile = await _publish_live_creator_avatar(
+ user_id, updated_profile
+ )
+
return store_model.ProfileDetails.from_db(updated_profile)
except prisma.errors.PrismaError as e:
@@ -1557,6 +1563,8 @@ async def review_store_submission(
"ActiveVersion": {"connect": {"id": other_approved.id}},
},
)
+ if public_media.publishing_enabled():
+ await _publish_reactivated_media(other_approved.id)
submission_status = (
prisma.enums.SubmissionStatus.APPROVED
@@ -1588,6 +1596,9 @@ async def review_store_submission(
f"Failed to update store listing version {store_listing_version_id}"
)
+ if is_approved:
+ reviewed_submission = await _publish_approved_media(reviewed_submission)
+
try:
await _send_submission_review_notification(
creator_user_id,
@@ -1610,6 +1621,162 @@ async def review_store_submission(
raise DatabaseError("Failed to create store submission review") from e
+async def _publish_approved_media(
+ version: prisma.models.StoreListingVersion,
+) -> prisma.models.StoreListingVersion:
+ """
+ Copy an approved version's media and its creator's avatar to the public site
+ media bucket, and point the rows at the copies. Never fails the review. The
+ profile and the version are written separately on purpose: each copy is
+ valid on its own, so a failed version write keeps the published avatar.
+ """
+ if not public_media.publishing_enabled():
+ return version
+ try:
+ assert version.StoreListing is not None
+ owner_id = version.StoreListing.owningUserId
+ await publish_creator_avatar(owner_id)
+ published = await public_media.publish_urls(
+ [*version.imageUrls, version.videoUrl, version.agentOutputDemoUrl],
+ await _listing_media_owner_ids(version.StoreListing),
+ )
+
+ media_update = _published_media_update(version, published)
+ if not media_update:
+ return version
+
+ updated = await prisma.models.StoreListingVersion.prisma().update(
+ where={"id": version.id},
+ data=media_update,
+ include={"StoreListing": True, "Reviewer": True},
+ )
+ return updated or version
+ except Exception:
+ logger.exception(
+ f"Failed to publish media of store listing version {version.id}"
+ )
+ return version
+
+
+async def _publish_reactivated_media(version_id: str) -> None:
+ """Publish the media of a version that became active again. Never raises."""
+ try:
+ version = await prisma.models.StoreListingVersion.prisma().find_unique(
+ where={"id": version_id}, include={"StoreListing": True}
+ )
+ except Exception:
+ logger.exception(f"Failed to load store listing version {version_id}")
+ return
+ if version:
+ await _publish_approved_media(version)
+
+
+def _published_media_update(
+ version: prisma.models.StoreListingVersion, published: dict[str, str]
+) -> prisma.types.StoreListingVersionUpdateInput:
+ """The version's media columns rewritten to their published URLs."""
+ media_update: prisma.types.StoreListingVersionUpdateInput = {}
+ if any(url in published for url in version.imageUrls):
+ media_update["imageUrls"] = [
+ published.get(url, url) for url in version.imageUrls
+ ]
+ if version.videoUrl in published:
+ media_update["videoUrl"] = published[version.videoUrl]
+ if version.agentOutputDemoUrl in published:
+ media_update["agentOutputDemoUrl"] = published[version.agentOutputDemoUrl]
+ return media_update
+
+
+async def _listing_media_owner_ids(listing: prisma.models.StoreListing) -> set[str]:
+ """
+ Users whose uploads a listing may publish: its owner and, for an org listing,
+ the org's active members, who can edit it and upload under their own path.
+ """
+ owner_ids = {listing.owningUserId}
+ if listing.owningOrgId:
+ members = await prisma.models.OrgMember.prisma().find_many(
+ where={
+ "orgId": listing.owningOrgId,
+ "status": prisma.enums.OrgMemberStatus.ACTIVE,
+ }
+ )
+ owner_ids.update(member.userId for member in members)
+ return owner_ids
+
+
+async def publish_creator_avatar(user_id: str) -> None:
+ """Publish the avatar of a creator who has just gained a public listing."""
+ if not public_media.publishing_enabled():
+ return
+ try:
+ profile = await prisma.models.Profile.prisma().find_unique(
+ where={"userId": user_id}
+ )
+ except Exception:
+ logger.exception(f"Failed to load the profile of user {user_id}")
+ return
+ if profile:
+ await _publish_live_creator_avatar(user_id, profile)
+
+
+async def _has_public_listing(user_id: str) -> bool:
+ """Whether the marketplace shows this user as a creator to anyone."""
+ store_listing = await prisma.models.StoreListing.prisma().find_first(
+ where={
+ "owningUserId": user_id,
+ "isDeleted": False,
+ "hasApprovedVersion": True,
+ }
+ )
+ if store_listing:
+ return True
+ skill_listing = await prisma.models.SkillListing.prisma().find_first(
+ where={
+ "owningUserId": user_id,
+ "isDeleted": False,
+ "hasApprovedVersion": True,
+ "ActiveVersion": {
+ "is": {
+ "submissionStatus": prisma.enums.SubmissionStatus.APPROVED,
+ "isDeleted": False,
+ "isAvailable": True,
+ }
+ },
+ }
+ )
+ return skill_listing is not None
+
+
+async def _publish_live_creator_avatar(
+ user_id: str, profile: prisma.models.Profile
+) -> prisma.models.Profile:
+ """
+ Copy the avatar of a creator with a public listing to the public site media
+ bucket and point the profile at the copy. Only writes while the profile
+ still holds the avatar it copied. Never fails the caller.
+ """
+ if not profile.avatarUrl or not public_media.publishing_enabled():
+ return profile
+ try:
+ if not await _has_public_listing(user_id):
+ return profile
+
+ published = await public_media.publish_urls([profile.avatarUrl], [user_id])
+ if profile.avatarUrl not in published:
+ return profile
+ await prisma.models.Profile.prisma().update_many(
+ where={"id": profile.id, "avatarUrl": profile.avatarUrl},
+ data={"avatarUrl": published[profile.avatarUrl]},
+ )
+ updated = await prisma.models.Profile.prisma().find_unique(
+ where={"id": profile.id}
+ )
+ return updated or profile
+ except Exception:
+ logger.exception(f"Failed to publish avatar of user {user_id}")
+ return profile
+
+
async def _approve_sub_agent(
tx,
sub_graph: prisma.models.AgentGraph,
diff --git a/autogpt_platform/backend/backend/api/features/store/db_test.py b/autogpt_platform/backend/backend/api/features/store/db_test.py
index 85e82894b4..6c047d61f4 100644
--- a/autogpt_platform/backend/backend/api/features/store/db_test.py
+++ b/autogpt_platform/backend/backend/api/features/store/db_test.py
@@ -1,6 +1,6 @@
import json
from datetime import datetime
-from unittest.mock import AsyncMock, patch
+from unittest.mock import AsyncMock, call, patch
import prisma.enums
import prisma.errors
@@ -1283,3 +1283,468 @@ async def test_get_available_graph_never_carries_picked_files(mocker):
graph = await db.get_available_graph("slv-1", hide_nodes=False)
assert graph.nodes[0].input_default == {"spreadsheet": None, "range": "A1"}
+
+
+PRIVATE_URL = "https://storage.googleapis.com/media-bucket/users/owner-id/images"
+PUBLIC_URL = "https://storage.googleapis.com/public-bucket/users/owner-id/images"
+
+
+def public_media_bucket(mocker):
+ mocker.patch(
+ "backend.api.features.store.db.public_media.publishing_enabled",
+ return_value=True,
+ )
+
+
+def _publish_mock(mocker, **kwargs):
+ return mocker.patch(
+ "backend.api.features.store.db.public_media.publish_urls",
+ new_callable=AsyncMock,
+ **kwargs,
+ )
+
+
+def review_mocks(mocker):
+ """review_store_submission without a database. The version's media and the
+ owner's avatar are all in the private media bucket."""
+ now = datetime.now()
+ listing = prisma.models.StoreListing(
+ id="listing-id",
+ createdAt=now,
+ updatedAt=now,
+ isDeleted=False,
+ hasApprovedVersion=False,
+ slug="test-agent",
+ agentGraphId="agent-id",
+ owningUserId="owner-id",
+ useForOnboarding=False,
+ )
+ graph = prisma.models.AgentGraph(
+ id="agent-id",
+ version=1,
+ userId="owner-id",
+ createdAt=now,
+ isActive=True,
+ visibility=prisma.enums.ResourceVisibility.PRIVATE,
+ )
+ version = prisma.models.StoreListingVersion(
+ id="version-id",
+ agentGraphId="agent-id",
+ agentGraphVersion=1,
+ name="Test Agent",
+ description="Test description",
+ createdAt=now,
+ updatedAt=now,
+ subHeading="",
+ imageUrls=[f"{PRIVATE_URL}/a.png", "https://example.com/b.png"],
+ videoUrl=f"{PRIVATE_URL}/../videos/v.mp4",
+ agentOutputDemoUrl=f"{PRIVATE_URL}/demo.png",
+ categories=[],
+ isFeatured=False,
+ isDeleted=False,
+ version=1,
+ storeListingId="listing-id",
+ submissionStatus=prisma.enums.SubmissionStatus.PENDING,
+ isAvailable=True,
+ submittedAt=now,
+ StoreListing=listing,
+ AgentGraph=graph,
+ )
+ profile = prisma.models.Profile(
+ id="profile-id",
+ userId="owner-id",
+ name="Owner",
+ username="owner",
+ description="",
+ links=[],
+ avatarUrl=f"{PRIVATE_URL}/avatar.png",
+ isFeatured=False,
+ createdAt=now,
+ updatedAt=now,
+ )
+
+ stored = [version]
+
+ def update_version(where, data, include=None):
+ stored[0] = stored[0].model_copy(
+ update={key: value for key, value in data.items() if key != "Reviewer"}
+ )
+ return stored[0]
+
+ mock_slv = mocker.patch("prisma.models.StoreListingVersion.prisma")
+ mock_slv.return_value.find_unique = AsyncMock(return_value=version)
+ mock_slv.return_value.find_first = AsyncMock(return_value=None)
+ mock_slv.return_value.update = AsyncMock(side_effect=update_version)
+ mock_listing = mocker.patch("prisma.models.StoreListing.prisma")
+ mock_listing.return_value.update = AsyncMock()
+ mock_listing.return_value.find_first = AsyncMock(return_value=listing)
+ mocker.patch("prisma.models.AgentGraph.prisma").return_value.update = AsyncMock()
+ mock_members = mocker.patch("prisma.models.OrgMember.prisma")
+ mock_members.return_value.find_many = AsyncMock(return_value=[])
+ mock_profile = mocker.patch("prisma.models.Profile.prisma")
+ mock_profile.return_value.find_unique = AsyncMock(return_value=profile)
+ mock_profile.return_value.update_many = AsyncMock()
+ mocker.patch(
+ "backend.api.features.store.db.transaction",
+ return_value=AsyncMock(
+ __aenter__=AsyncMock(return_value=mocker.MagicMock()),
+ __aexit__=AsyncMock(return_value=False),
+ ),
+ )
+ mocker.patch(
+ "backend.api.features.store.db.get_sub_graphs",
+ new_callable=AsyncMock,
+ return_value=[],
+ )
+ mocker.patch(
+ "backend.api.features.store.db.ensure_embedding", new_callable=AsyncMock
+ )
+ mocker.patch(
+ "backend.api.features.store.db._send_submission_review_notification",
+ new_callable=AsyncMock,
+ )
+ return mocker.MagicMock(
+ slv=mock_slv.return_value,
+ profile=mock_profile.return_value,
+ listing=mock_listing.return_value,
+ members=mock_members.return_value,
+ version=version,
+ )
+
+
+async def _review(is_approved: bool):
+ return await db.review_store_submission(
+ store_listing_version_id="version-id",
+ is_approved=is_approved,
+ external_comments="",
+ internal_comments="",
+ reviewer_id="reviewer-id",
+ )
+
+
[email protected](loop_scope="session")
+async def test_approve_publishes_media_and_rewrites_the_rows(
+ mocker, public_media_bucket, review_mocks
+):
+ publish = _publish_mock(
+ mocker,
+ return_value={
+ f"{PRIVATE_URL}/a.png": f"{PUBLIC_URL}/a.png",
+ f"{PRIVATE_URL}/demo.png": f"{PUBLIC_URL}/demo.png",
+ f"{PRIVATE_URL}/avatar.png": f"{PUBLIC_URL}/avatar.png",
+ },
+ )
+
+ result = await _review(is_approved=True)
+
+ assert publish.await_args_list == [
+ call([f"{PRIVATE_URL}/avatar.png"], ["owner-id"]),
+ call(
+ [
+ f"{PRIVATE_URL}/a.png",
+ "https://example.com/b.png",
+ f"{PRIVATE_URL}/../videos/v.mp4",
+ f"{PRIVATE_URL}/demo.png",
+ ],
+ {"owner-id"},
+ ),
+ ]
+ media_update = review_mocks.slv.update.await_args_list[-1].kwargs
+ assert media_update["where"] == {"id": "version-id"}
+ assert media_update["data"] == {
+ "imageUrls": [f"{PUBLIC_URL}/a.png", "https://example.com/b.png"],
+ "agentOutputDemoUrl": f"{PUBLIC_URL}/demo.png",
+ }
+ review_mocks.profile.update_many.assert_awaited_once_with(
+ where={"id": "profile-id", "avatarUrl": f"{PRIVATE_URL}/avatar.png"},
+ data={"avatarUrl": f"{PUBLIC_URL}/avatar.png"},
+ )
+ assert result.status == prisma.enums.SubmissionStatus.APPROVED
+ assert result.image_urls == [f"{PUBLIC_URL}/a.png", "https://example.com/b.png"]
+ assert result.video_url == f"{PRIVATE_URL}/../videos/v.mp4"
+ assert result.agent_output_demo_url == f"{PUBLIC_URL}/demo.png"
+
+
[email protected](loop_scope="session")
+async def test_reject_does_not_publish_media(mocker, public_media_bucket, review_mocks):
+ publish = _publish_mock(mocker, return_value={})
+
+ result = await _review(is_approved=False)
+
+ publish.assert_not_awaited()
+ assert review_mocks.slv.update.await_count == 1
+ review_mocks.profile.update_many.assert_not_awaited()
+ assert result.status == prisma.enums.SubmissionStatus.REJECTED
+
+
[email protected](loop_scope="session")
+async def test_approve_does_nothing_with_media_when_the_bucket_is_unset(
+ mocker, review_mocks
+):
+ publish = _publish_mock(mocker, return_value={})
+
+ result = await _review(is_approved=True)
+
+ publish.assert_not_awaited()
+ assert review_mocks.slv.update.await_count == 1
+ assert result.image_urls[0] == f"{PRIVATE_URL}/a.png"
+
+
[email protected](loop_scope="session")
+async def test_approve_succeeds_when_publishing_fails(
+ mocker, public_media_bucket, review_mocks
+):
+ _publish_mock(mocker, side_effect=Exception("GCS is down"))
+
+ result = await _review(is_approved=True)
+
+ assert result.status == prisma.enums.SubmissionStatus.APPROVED
+ assert result.image_urls[0] == f"{PRIVATE_URL}/a.png"
+ assert review_mocks.slv.update.await_count == 1
+ review_mocks.profile.update_many.assert_not_awaited()
+
+
+def _avatar_profile(avatar_url: str) -> prisma.models.Profile:
+ return prisma.models.Profile(
+ id="profile-id",
+ userId="owner-id",
+ name="Owner",
+ username="owner",
+ description="",
+ links=[],
+ avatarUrl=avatar_url,
+ isFeatured=False,
+ createdAt=datetime.now(),
+ updatedAt=datetime.now(),
+ )
+
+
+ "store_listing, skill_listing, published",
+ [(True, False, True), (False, True, True), (False, False, False)],
+)
[email protected](loop_scope="session")
+async def test_update_profile_publishes_a_live_creators_new_avatar(
+ mocker, public_media_bucket, store_listing, skill_listing, published
+):
+ old = _avatar_profile(f"{PRIVATE_URL}/old.png")
+ saved = _avatar_profile(f"{PRIVATE_URL}/new.png")
+ mock_profile = mocker.patch("prisma.models.Profile.prisma")
+ mock_profile.return_value.find_first = AsyncMock(return_value=old)
+ mock_profile.return_value.update = AsyncMock(
+ side_effect=lambda where, data: saved.model_copy(update=dict(data))
+ )
+ mock_profile.return_value.update_many = AsyncMock(return_value=1)
+ mock_profile.return_value.find_unique = AsyncMock(
+ return_value=_avatar_profile(f"{PUBLIC_URL}/new.png")
+ )
+ mock_listing = mocker.patch("prisma.models.StoreListing.prisma")
+ mock_listing.return_value.find_first = AsyncMock(
+ return_value=mocker.MagicMock() if store_listing else None
+ )
+ mocker.patch("prisma.models.SkillListing.prisma").return_value.find_first = (
+ AsyncMock(return_value=mocker.MagicMock() if skill_listing else None)
+ )
+ publish = _publish_mock(
+ mocker, return_value={f"{PRIVATE_URL}/new.png": f"{PUBLIC_URL}/new.png"}
+ )
+
+ result = await db.update_profile(
+ "owner-id",
+ Profile(
+ name="Owner",
+ username="owner",
+ description="",
+ links=[],
+ avatar_url=f"{PRIVATE_URL}/new.png",
+ ),
+ )
+
+ live_where = mock_listing.return_value.find_first.await_args.kwargs["where"]
+ assert live_where == {
+ "owningUserId": "owner-id",
+ "isDeleted": False,
+ "hasApprovedVersion": True,
+ }
+ if published:
+ publish.assert_awaited_once_with([f"{PRIVATE_URL}/new.png"], ["owner-id"])
+ mock_profile.return_value.update_many.assert_awaited_once_with(
+ where={"id": "profile-id", "avatarUrl": f"{PRIVATE_URL}/new.png"},
+ data={"avatarUrl": f"{PUBLIC_URL}/new.png"},
+ )
+ assert result.avatar_url == f"{PUBLIC_URL}/new.png"
+ else:
+ publish.assert_not_awaited()
+ assert result.avatar_url == f"{PRIVATE_URL}/new.png"
+
+
[email protected](loop_scope="session")
+async def test_update_profile_saves_when_publishing_the_avatar_fails(
+ mocker, public_media_bucket
+):
+ profile = _avatar_profile(f"{PRIVATE_URL}/new.png")
+ mock_profile = mocker.patch("prisma.models.Profile.prisma")
+ mock_profile.return_value.find_first = AsyncMock(return_value=profile)
+ mock_profile.return_value.update = AsyncMock(return_value=profile)
+ mocker.patch("prisma.models.StoreListing.prisma").return_value.find_first = (
+ AsyncMock(return_value=mocker.MagicMock())
+ )
+ _publish_mock(mocker, side_effect=Exception("GCS is down"))
+
+ result = await db.update_profile(
+ "owner-id",
+ Profile(
+ name="Owner",
+ username="owner",
+ description="",
+ links=[],
+ avatar_url=f"{PRIVATE_URL}/new.png",
+ ),
+ )
+
+ assert result.avatar_url == f"{PRIVATE_URL}/new.png"
+ mock_profile.return_value.update.assert_awaited_once()
+
+
[email protected](loop_scope="session")
+async def test_avatar_publish_does_not_overwrite_a_newer_avatar(
+ mocker, public_media_bucket
+):
+ saved = _avatar_profile(f"{PRIVATE_URL}/first.png")
+ row = [saved]
+
+ def update(where, data):
+ row[0] = row[0].model_copy(update=dict(data))
+ return row[0]
+
+ def update_many(where, data):
+ if all(getattr(row[0], key) == value for key, value in where.items()):
+ update(where, data)
+ return 1
+ return 0
+
+ mock_profile = mocker.patch("prisma.models.Profile.prisma")
+ mock_profile.return_value.update = AsyncMock(side_effect=update)
+ mock_profile.return_value.update_many = AsyncMock(side_effect=update_many)
+ mock_profile.return_value.find_unique = AsyncMock(side_effect=lambda where: row[0])
+ mocker.patch("prisma.models.StoreListing.prisma").return_value.find_first = (
+ AsyncMock(return_value=mocker.MagicMock())
+ )
+
+ async def publish_while_a_newer_avatar_is_saved(urls, user_id):
+ row[0] = row[0].model_copy(update={"avatarUrl": f"{PUBLIC_URL}/second.png"})
+ return {f"{PRIVATE_URL}/first.png": f"{PUBLIC_URL}/first.png"}
+
+ _publish_mock(mocker, side_effect=publish_while_a_newer_avatar_is_saved)
+
+ result = await db._publish_live_creator_avatar("owner-id", saved)
+
+ assert row[0].avatarUrl == f"{PUBLIC_URL}/second.png"
+ assert result.avatarUrl == f"{PUBLIC_URL}/second.png"
+
+
[email protected](loop_scope="session")
+async def test_approve_keeps_the_published_avatar_when_the_version_write_fails(
+ mocker, public_media_bucket, review_mocks
+):
+ _publish_mock(
+ mocker,
+ return_value={
+ f"{PRIVATE_URL}/a.png": f"{PUBLIC_URL}/a.png",
+ f"{PRIVATE_URL}/avatar.png": f"{PUBLIC_URL}/avatar.png",
+ },
+ )
+ save_review = review_mocks.slv.update.side_effect
+
+ def fail_the_media_write(where, data, include=None):
+ if review_mocks.slv.update.await_count > 1:
+ raise Exception("database is down")
+ return save_review(where, data, include)
+
+ review_mocks.slv.update.side_effect = fail_the_media_write
+
+ result = await _review(is_approved=True)
+
+ assert result.status == prisma.enums.SubmissionStatus.APPROVED
+ assert result.image_urls[0] == f"{PRIVATE_URL}/a.png"
+ review_mocks.profile.update_many.assert_awaited_once_with(
+ where={"id": "profile-id", "avatarUrl": f"{PRIVATE_URL}/avatar.png"},
+ data={"avatarUrl": f"{PUBLIC_URL}/avatar.png"},
+ )
+
+
[email protected](loop_scope="session")
+async def test_approve_publishes_media_uploaded_by_org_members(
+ mocker, public_media_bucket, review_mocks
+):
+ review_mocks.version.StoreListing.owningOrgId = "org-id"
+ review_mocks.members.find_many.return_value = [
+ mocker.MagicMock(userId="owner-id"),
+ mocker.MagicMock(userId="member-id"),
+ ]
+ publish = _publish_mock(mocker, return_value={})
+
+ await _review(is_approved=True)
+
+ assert review_mocks.members.find_many.await_args.kwargs["where"] == {
+ "orgId": "org-id",
+ "status": prisma.enums.OrgMemberStatus.ACTIVE,
+ }
+ assert publish.await_args_list[-1].args[1] == {"owner-id", "member-id"}
+
+
[email protected](loop_scope="session")
+async def test_reject_publishes_the_media_of_the_reactivated_version(
+ mocker, public_media_bucket, review_mocks
+):
+ approved = review_mocks.version.model_copy(
+ update={"submissionStatus": prisma.enums.SubmissionStatus.APPROVED}
+ )
+ previous = review_mocks.version.model_copy(
+ update={"id": "previous-id", "imageUrls": [f"{PRIVATE_URL}/old.png"]}
+ )
+ review_mocks.slv.find_unique.side_effect = lambda where, include=None: (
+ previous if where["id"] == "previous-id" else approved
+ )
+ review_mocks.slv.find_first.return_value = previous
+ publish = _publish_mock(mocker, return_value={})
+
+ await _review(is_approved=False)
+
+ assert (
+ call(
+ [
+ f"{PRIVATE_URL}/old.png",
+ f"{PRIVATE_URL}/../videos/v.mp4",
+ f"{PRIVATE_URL}/demo.png",
+ ],
+ {"owner-id"},
+ )
+ in publish.await_args_list
+ )
+
+
[email protected](loop_scope="session")
+async def test_reject_completes_when_the_reactivated_version_fails_to_load(
+ mocker, public_media_bucket, review_mocks
+):
+ approved = review_mocks.version.model_copy(
+ update={"submissionStatus": prisma.enums.SubmissionStatus.APPROVED}
+ )
+ previous = review_mocks.version.model_copy(update={"id": "previous-id"})
+
+ def find_unique(where, include=None):
+ if where["id"] == "previous-id":
+ raise prisma.errors.PrismaError("database is down")
+ return approved
+
+ review_mocks.slv.find_unique.side_effect = find_unique
+ review_mocks.slv.find_first.return_value = previous
+ publish = _publish_mock(mocker, return_value={})
+
+ result = await _review(is_approved=False)
+
+ assert result.status == prisma.enums.SubmissionStatus.REJECTED
+ publish.assert_not_awaited()
diff --git a/autogpt_platform/backend/backend/api/features/store/generated_media_test.py b/autogpt_platform/backend/backend/api/features/store/generated_media_test.py
index 33c5732987..1cdf53eb8d 100644
--- a/autogpt_platform/backend/backend/api/features/store/generated_media_test.py
+++ b/autogpt_platform/backend/backend/api/features/store/generated_media_test.py
@@ -15,9 +15,14 @@ from backend.util.settings import Settings
@pytest.fixture
def generated_media_io(monkeypatch, tmp_path):
settings = Settings()
- settings.config.use_agent_image_generation_v2 = True
- settings.config.workspace_storage_dir = str(tmp_path / "workspaces")
- settings.config.platform_base_url = ""
+ monkeypatch.setattr(settings.config, "use_agent_image_generation_v2", True)
+ monkeypatch.setattr(settings.config, "media_gcs_bucket_name", "")
+ monkeypatch.setattr(settings.config, "public_site_media_bucket", "")
+ monkeypatch.setattr(settings.config, "private_user_data_bucket", "")
+ monkeypatch.setattr(
+ settings.config, "workspace_storage_dir", str(tmp_path / "workspaces")
+ )
+ monkeypatch.setattr(settings.config, "platform_base_url", "")
monkeypatch.setattr(image_gen, "settings", settings)
monkeypatch.setattr(image_gen, "ideogram_credentials", TEST_CREDENTIALS)
monkeypatch.setattr(media, "Settings", lambda: settings)
@@ -165,3 +170,26 @@ async def test_flux_still_requests_and_preserves_jpeg(
result = await image_gen.generate_agent_image(graph)
assert result.read() == source.getvalue()
assert run.call_args.kwargs["input"]["output_format"] == "jpg"
+
+
[email protected]("provider_format,mode", [("PNG", "RGB")])
+async def test_generate_image_regenerates_when_the_existence_check_fails(
+ generated_media_io,
+ graph_and_library_update,
+ ideogram_response,
+ monkeypatch,
+ provider_format,
+ mode,
+):
+ settings, _, _ = generated_media_io
+ settings.config.media_gcs_bucket_name = "test-bucket"
+ graph, _ = graph_and_library_update
+ _, generate, _ = ideogram_response
+ monkeypatch.setattr(
+ media, "check_media_exists", AsyncMock(side_effect=RuntimeError("GCS is down"))
+ )
+
+ result = await routes.generate_image(graph.id, user_id="test-user")
+
+ generate.assert_awaited_once()
+ assert result.image_url.endswith("users/test-user/images/agent_graph-1.jpeg")
diff --git a/autogpt_platform/backend/backend/api/features/store/local_media.py b/autogpt_platform/backend/backend/api/features/store/local_media.py
index 50a8b1a9b7..9a409ae55f 100644
--- a/autogpt_platform/backend/backend/api/features/store/local_media.py
+++ b/autogpt_platform/backend/backend/api/features/store/local_media.py
@@ -3,6 +3,7 @@ import os
import re
import tempfile
import uuid
+from collections.abc import AsyncIterator
from pathlib import Path
from backend.util.data import get_data_path
@@ -37,6 +38,41 @@ async def check_media_exists(user_id: str, filename: str) -> str | None:
return await asyncio.to_thread(_check_media_exists, user_id, filename)
+async def media_metadata(
+ user_id: str, media_type: str, filename: str
+) -> dict[str, str]:
+ return await asyncio.to_thread(_media_metadata, user_id, media_type, filename)
+
+
+async def stream_media(
+ user_id: str,
+ media_type: str,
+ filename: str,
+ byte_range: tuple[int, int] | None = None,
+) -> AsyncIterator[bytes]:
+ path = get_media_path(user_id, media_type, filename)
+ if not path.is_file():
+ raise FileNotFoundError("Media not found")
+
+ file = await asyncio.to_thread(path.open, "rb")
+ try:
+ remaining: int | None = None
+ if byte_range is not None:
+ start, end = byte_range
+ await asyncio.to_thread(file.seek, start)
+ remaining = end - start + 1
+ while remaining is None or remaining > 0:
+ read_size = 64 * 1024 if remaining is None else min(64 * 1024, remaining)
+ chunk = await asyncio.to_thread(file.read, read_size)
+ if not chunk:
+ break
+ yield chunk
+ if remaining is not None:
+ remaining -= len(chunk)
+ finally:
+ await asyncio.to_thread(file.close)
+
+
def get_media_path(user_id: str, media_type: str, filename: str) -> Path:
"""Resolve a file inside the media root, rejecting traversal and escaping symlinks."""
if media_type not in MEDIA_TYPES:
@@ -47,9 +83,9 @@ def get_media_path(user_id: str, media_type: str, filename: str) -> Path:
os.path.join(
base_dir,
"users",
- _validate_path_component(user_id),
+ validate_path_component(user_id),
media_type,
- _validate_path_component(filename),
+ validate_path_component(filename),
)
)
if not candidate.startswith(base_dir + os.sep):
@@ -66,7 +102,7 @@ def stored_filename(filename: str, content_type: str, use_file_name: bool) -> st
"""Derive the extension from validated content rather than a client-supplied name."""
extension = CONTENT_TYPE_EXTENSIONS[content_type]
if use_file_name:
- return f"{Path(_validate_path_component(filename)).stem}{extension}"
+ return f"{Path(validate_path_component(filename)).stem}{extension}"
return f"{uuid.uuid4()}{extension}"
@@ -81,13 +117,13 @@ def media_root() -> Path:
def media_url(user_id: str, media_type: str, filename: str) -> str:
"""Return a public backend URL or a same-origin path for local installations."""
path = (
- f"/api/store/media/{_validate_path_component(user_id)}"
- f"/{media_type}/{_validate_path_component(filename)}"
+ f"/api/store/media/{validate_path_component(user_id)}"
+ f"/{media_type}/{validate_path_component(filename)}"
)
return f"{Settings().config.platform_base_url.rstrip('/')}{path}"
-def _validate_path_component(value: str) -> str:
+def validate_path_component(value: str) -> str:
if value in {".", ".."} or not _SAFE_PATH_COMPONENT.fullmatch(value):
raise ValueError("Invalid media path component")
return value
@@ -95,8 +131,8 @@ def _validate_path_component(value: str) -> str:
def _check_media_exists(user_id: str, filename: str) -> str | None:
try:
- _validate_path_component(user_id)
- _validate_path_component(filename)
+ validate_path_component(user_id)
+ validate_path_component(filename)
content_type = content_type_for_filename(filename)
if content_type is None:
return None
@@ -109,6 +145,14 @@ def _check_media_exists(user_id: str, filename: str) -> str | None:
return None
+def _media_metadata(user_id: str, media_type: str, filename: str) -> dict[str, str]:
+ path = get_media_path(user_id, media_type, filename)
+ try:
+ return {"size": str(path.stat().st_size)}
+ except FileNotFoundError as error:
+ raise FileNotFoundError("Media not found") from error
+
+
def _write_media(user_id: str, media_type: str, filename: str, content: bytes) -> None:
file_path = get_media_path(user_id, media_type, filename)
file_path.parent.mkdir(parents=True, exist_ok=True)
diff --git a/autogpt_platform/backend/backend/api/features/store/local_media_test.py b/autogpt_platform/backend/backend/api/features/store/local_media_test.py
index 582e2f94cd..a67c057595 100644
--- a/autogpt_platform/backend/backend/api/features/store/local_media_test.py
+++ b/autogpt_platform/backend/backend/api/features/store/local_media_test.py
@@ -224,3 +224,24 @@ async def test_named_jpeg_aliases_reuse_upload(local_settings, filename):
@pytest.mark.parametrize("filename", ["../image.png", "image.html", "missing.png"])
async def test_invalid_or_missing_lookup_returns_none(local_settings, filename):
assert await local_media.check_media_exists("user", filename) is None
+
+
+ "byte_range, expected",
+ [(None, b"0123456789"), ((2, 5), b"2345"), ((7, 9), b"789"), ((0, 0), b"0")],
+)
+async def test_stream_media_returns_exactly_the_requested_bytes(
+ local_settings, byte_range, expected
+):
+ path = local_media.get_media_path("owner", "videos", "clip.mp4")
+ path.parent.mkdir(parents=True)
+ path.write_bytes(b"0123456789")
+
+ chunks = [
+ chunk
+ async for chunk in local_media.stream_media(
+ "owner", "videos", "clip.mp4", byte_range
+ )
+ ]
+
+ assert b"".join(chunks) == expected
diff --git a/autogpt_platform/backend/backend/api/features/store/media.py b/autogpt_platform/backend/backend/api/features/store/media.py
index bca44f6e40..662374ea78 100644
--- a/autogpt_platform/backend/backend/api/features/store/media.py
+++ b/autogpt_platform/backend/backend/api/features/store/media.py
@@ -1,15 +1,14 @@
import logging
-import os
-import uuid
import fastapi
from gcloud.aio import storage as async_storage
-from backend.util.settings import Settings
+from backend.util.gcs_utils import is_not_found_error
+from backend.util.settings import Config, Settings
from backend.util.virus_scanner import scan_content_safe
from . import exceptions as store_exceptions
-from . import local_media
+from . import local_media, submission_media
logger = logging.getLogger(__name__)
@@ -30,32 +29,48 @@ async def check_media_exists(user_id: str, filename: str) -> str | None:
Returns:
str | None: URL of the blob if it exists, None otherwise
"""
- settings = Settings()
- if not settings.config.media_gcs_bucket_name:
+ config = Settings().config
+ bucket_name = config.resolved_private_user_data_bucket
+ if not bucket_name:
return await local_media.check_media_exists(user_id, filename)
+ try:
+ safe_user_id = local_media.validate_path_component(user_id)
+ safe_filename = local_media.validate_path_component(filename)
+ except ValueError:
+ return None
+ content_type = local_media.content_type_for_filename(safe_filename)
+ if content_type is None:
+ return None
+ safe_filename = local_media.stored_filename(safe_filename, content_type, True)
async with async_storage.Storage() as async_client:
- bucket_name = settings.config.media_gcs_bucket_name
-
- # Check images
- image_path = f"users/{user_id}/images/{filename}"
+ image_path = f"users/{safe_user_id}/images/{safe_filename}"
try:
await async_client.download_metadata(bucket_name, image_path)
- # If we get here, the file exists - construct public URL
- return f"https://storage.googleapis.com/{bucket_name}/{image_path}"
- except Exception:
- # File doesn't exist, continue to check videos
- pass
-
- # Check videos
- video_path = f"users/{user_id}/videos/{filename}"
+ return _stored_media_url(
+ _serves_private_urls(config),
+ bucket_name,
+ safe_user_id,
+ "images",
+ safe_filename,
+ )
+ except Exception as error:
+ if not _is_missing_media_error(error):
+ raise
+
+ video_path = f"users/{safe_user_id}/videos/{safe_filename}"
try:
await async_client.download_metadata(bucket_name, video_path)
- # If we get here, the file exists - construct public URL
- return f"https://storage.googleapis.com/{bucket_name}/{video_path}"
- except Exception:
- # File doesn't exist
- pass
+ return _stored_media_url(
+ _serves_private_urls(config),
+ bucket_name,
+ safe_user_id,
+ "videos",
+ safe_filename,
+ )
+ except Exception as error:
+ if not _is_missing_media_error(error):
+ raise
return None
@@ -119,8 +134,9 @@ async def upload_media(
else:
raise store_exceptions.InvalidFileTypeError("Invalid video file signature")
- settings = Settings()
- use_local_storage = not settings.config.media_gcs_bucket_name
+ config = Settings().config
+ bucket_name = config.resolved_private_user_data_bucket
+ use_local_storage = not bucket_name
try:
# Validate file type
@@ -164,47 +180,40 @@ async def upload_media(
# Reset file pointer
await file.seek(0)
- # Generate unique filename
- filename = file.filename or ""
- file_ext = os.path.splitext(filename)[1].lower()
- if use_file_name and not is_avatar:
- unique_filename = filename
- else:
- unique_filename = f"{uuid.uuid4()}{file_ext}"
-
- # Construct storage path
media_type = "images" if content_type in ALLOWED_IMAGE_TYPES else "videos"
+ unique_filename = local_media.stored_filename(
+ file.filename or "", content_type, use_file_name and not is_avatar
+ )
if use_local_storage:
- unique_filename = local_media.stored_filename(
- unique_filename, content_type, use_file_name and not is_avatar
- )
file_bytes = await file.read()
await scan_content_safe(file_bytes, filename=unique_filename)
return await local_media.store_media(
user_id, media_type, unique_filename, file_bytes
)
- storage_path = f"users/{user_id}/{media_type}/{unique_filename}"
+ storage_path = submission_media.object_path(
+ user_id, media_type, unique_filename
+ )
try:
async with async_storage.Storage() as async_client:
- bucket_name = settings.config.media_gcs_bucket_name
-
file_bytes = await file.read()
await scan_content_safe(file_bytes, filename=unique_filename)
- public_url = (
- f"https://storage.googleapis.com/{bucket_name}/{storage_path}"
- )
-
# Upload using pure async client
await async_client.upload(
bucket_name, storage_path, file_bytes, content_type=content_type
)
logger.info(f"Successfully uploaded file to: {storage_path}")
- return public_url
+ return _stored_media_url(
+ _serves_private_urls(config),
+ bucket_name,
+ user_id,
+ media_type,
+ unique_filename,
+ )
except fastapi.HTTPException:
raise
@@ -221,3 +230,28 @@ async def upload_media(
raise store_exceptions.MediaUploadError(
"Unexpected error during media upload"
) from e
+
+
+def _serves_private_urls(config: Config) -> bool:
+ """Only the legacy single-bucket setup still hands out direct GCS URLs."""
+ return (
+ config.resolved_private_user_data_bucket
+ != config.resolved_public_site_media_bucket
+ )
+
+
+def _stored_media_url(
+ buckets_are_split: bool,
+ bucket_name: str,
+ user_id: str,
+ media_type: str,
+ filename: str,
+) -> str:
+ if buckets_are_split:
+ return submission_media.url(user_id, media_type, filename)
+ object_path = submission_media.object_path(user_id, media_type, filename)
+ return f"https://storage.googleapis.com/{bucket_name}/{object_path}"
+
+
+def _is_missing_media_error(error: Exception) -> bool:
+ return isinstance(error, FileNotFoundError) or is_not_found_error(error)
diff --git a/autogpt_platform/backend/backend/api/features/store/media_test.py b/autogpt_platform/backend/backend/api/features/store/media_test.py
index 315a199f70..586906d422 100644
--- a/autogpt_platform/backend/backend/api/features/store/media_test.py
+++ b/autogpt_platform/backend/backend/api/features/store/media_test.py
@@ -6,6 +6,7 @@ import fastapi
import pytest
import starlette.datastructures
+from backend.util.gcs_utils_test import _URL_CONTAINING_404, _gcs_http_error
from backend.util.settings import Settings
from . import exceptions as store_exceptions
@@ -16,9 +17,13 @@ from . import media as store_media
@pytest.fixture
def mock_settings(monkeypatch):
settings = Settings()
- settings.config.media_gcs_bucket_name = "test-bucket"
- settings.config.google_application_credentials = "test-credentials"
+ monkeypatch.setattr(settings.config, "media_gcs_bucket_name", "test-bucket")
+ monkeypatch.setattr(settings.config, "public_site_media_bucket", "")
+ monkeypatch.setattr(settings.config, "private_user_data_bucket", "")
monkeypatch.setattr("backend.api.features.store.media.Settings", lambda: settings)
+ monkeypatch.setattr(
+ "backend.api.features.store.public_media.Settings", lambda: settings
+ )
return settings
@@ -46,6 +51,60 @@ def mock_storage_client(mocker):
return mock_client
+async def test_upload_media_uses_private_bucket_when_storage_is_split(
+ mock_settings, mock_storage_client
+):
+ mock_settings.config.platform_base_url = "https://backend.test"
+ mock_settings.config.public_site_media_bucket = "public-media"
+ mock_settings.config.private_user_data_bucket = "private-media"
+ test_file = fastapi.UploadFile(
+ filename="private.jpeg",
+ file=io.BytesIO(b"\xff\xd8\xffprivate"),
+ headers=starlette.datastructures.Headers({"content-type": "image/jpeg"}),
+ )
+
+ result = await store_media.upload_media("test-user", test_file)
+
+ assert mock_storage_client.upload.await_args.args[0] == "private-media"
+ assert result.startswith("/api/store/submissions/media/test-user/images/")
+
+
+async def test_check_media_exists_uses_private_bucket_when_storage_is_split(
+ mock_settings, mock_storage_client
+):
+ mock_settings.config.platform_base_url = "https://backend.test"
+ mock_settings.config.public_site_media_bucket = "public-media"
+ mock_settings.config.private_user_data_bucket = "private-media"
+ mock_storage_client.download_metadata.return_value = {}
+
+ result = await store_media.check_media_exists("test-user", "agent.jpeg")
+
+ mock_storage_client.download_metadata.assert_awaited_once_with(
+ "private-media", "users/test-user/images/agent.jpeg"
+ )
+ assert result == "/api/store/submissions/media/test-user/images/agent.jpeg"
+
+
+async def test_check_media_exists_preserves_non_not_found_errors(
+ mock_settings, mock_storage_client
+):
+ mock_storage_client.download_metadata.side_effect = _gcs_http_error(
+ 503, _URL_CONTAINING_404
+ )
+
+ with pytest.raises(Exception) as error:
+ await store_media.check_media_exists("test-user", "agent.jpeg")
+
+ assert getattr(error.value, "status", None) == 503
+
+
+async def test_check_media_exists_rejects_invalid_path_before_storage_access(
+ mock_settings, mock_storage_client
+):
+ assert await store_media.check_media_exists("test-user", "../private.jpeg") is None
+ mock_storage_client.download_metadata.assert_not_awaited()
+
+
async def test_upload_media_success(mock_settings, mock_storage_client):
# Create test JPEG data with valid signature
test_data = b"\xff\xd8\xff" + b"test data"
@@ -58,9 +117,8 @@ async def test_upload_media_success(mock_settings, mock_storage_client):
result = await store_media.upload_media("test-user", test_file)
- assert result.startswith(
- "https://storage.googleapis.com/test-bucket/users/test-user/images/"
- )
+ object_path = mock_storage_client.upload.await_args.args[1]
+ assert result == f"https://storage.googleapis.com/test-bucket/{object_path}"
assert result.endswith(".jpeg")
mock_storage_client.upload.assert_called_once()
@@ -96,17 +154,41 @@ async def test_upload_media_missing_content_type_accepts_valid_jpeg(
filename="image.jpeg", file=io.BytesIO(b"\xff\xd8\xffimage")
)
result = await store_media.upload_media("test-user", upload)
- assert result.startswith("https://storage.googleapis.com/test-bucket/")
+ object_path = mock_storage_client.upload.await_args.args[1]
+ assert result == f"https://storage.googleapis.com/test-bucket/{object_path}"
assert mock_storage_client.upload.call_args.kwargs["content_type"] == "image/jpeg"
[email protected]("use_file_name", [False, True])
+async def test_upload_media_normalizes_extension_from_validated_content(
+ mock_settings, mock_storage_client, use_file_name
+):
+ upload = fastapi.UploadFile(
+ filename="misleading.png",
+ file=io.BytesIO(b"\xff\xd8\xffimage"),
+ headers=starlette.datastructures.Headers({"content-type": "image/jpeg"}),
+ )
+
+ result = await store_media.upload_media(
+ "test-user", upload, use_file_name=use_file_name
+ )
+
+ object_path = mock_storage_client.upload.await_args.args[1]
+ assert object_path.endswith(".jpeg")
+ assert not object_path.endswith(".png")
+ assert result == f"https://storage.googleapis.com/test-bucket/{object_path}"
+
+
@pytest.fixture
def local_storage_settings(monkeypatch, tmp_path):
settings = Settings()
- settings.config.media_gcs_bucket_name = ""
- settings.config.google_application_credentials = ""
- settings.config.workspace_storage_dir = str(tmp_path / "workspaces")
- settings.config.platform_base_url = ""
+ monkeypatch.setattr(settings.config, "media_gcs_bucket_name", "")
+ monkeypatch.setattr(settings.config, "public_site_media_bucket", "")
+ monkeypatch.setattr(settings.config, "private_user_data_bucket", "")
+ monkeypatch.setattr(
+ settings.config, "workspace_storage_dir", str(tmp_path / "workspaces")
+ )
+ monkeypatch.setattr(settings.config, "platform_base_url", "")
monkeypatch.setattr("backend.api.features.store.media.Settings", lambda: settings)
monkeypatch.setattr(
"backend.api.features.store.local_media.Settings", lambda: settings
@@ -180,9 +262,8 @@ async def test_upload_media_video_type(mock_settings, mock_storage_client):
result = await store_media.upload_media("test-user", test_file)
- assert result.startswith(
- "https://storage.googleapis.com/test-bucket/users/test-user/videos/"
- )
+ object_path = mock_storage_client.upload.await_args.args[1]
+ assert result == f"https://storage.googleapis.com/test-bucket/{object_path}"
assert result.endswith(".mp4")
mock_storage_client.upload.assert_called_once()
@@ -221,9 +302,8 @@ async def test_upload_media_png_success(mock_settings, mock_storage_client):
)
result = await store_media.upload_media("test-user", test_file)
- assert result.startswith(
- "https://storage.googleapis.com/test-bucket/users/test-user/images/"
- )
+ object_path = mock_storage_client.upload.await_args.args[1]
+ assert result == f"https://storage.googleapis.com/test-bucket/{object_path}"
assert result.endswith(".png")
@@ -235,9 +315,8 @@ async def test_upload_media_gif_success(mock_settings, mock_storage_client):
)
result = await store_media.upload_media("test-user", test_file)
- assert result.startswith(
- "https://storage.googleapis.com/test-bucket/users/test-user/images/"
- )
+ object_path = mock_storage_client.upload.await_args.args[1]
+ assert result == f"https://storage.googleapis.com/test-bucket/{object_path}"
assert result.endswith(".gif")
@@ -249,9 +328,8 @@ async def test_upload_media_webp_success(mock_settings, mock_storage_client):
)
result = await store_media.upload_media("test-user", test_file)
- assert result.startswith(
- "https://storage.googleapis.com/test-bucket/users/test-user/images/"
- )
+ object_path = mock_storage_client.upload.await_args.args[1]
+ assert result == f"https://storage.googleapis.com/test-bucket/{object_path}"
assert result.endswith(".webp")
@@ -263,9 +341,8 @@ async def test_upload_media_webm_success(mock_settings, mock_storage_client):
)
result = await store_media.upload_media("test-user", test_file)
- assert result.startswith(
- "https://storage.googleapis.com/test-bucket/users/test-user/videos/"
- )
+ object_path = mock_storage_client.upload.await_args.args[1]
+ assert result == f"https://storage.googleapis.com/test-bucket/{object_path}"
assert result.endswith(".webm")
@@ -341,3 +418,51 @@ async def test_expert_avatar_still_enforces_type_and_size(
await store_media.upload_media("owner", upload, is_avatar=True)
assert error.value.status_code == status
mock_storage_client.upload.assert_not_awaited()
+
+
+async def test_upload_media_is_private_when_only_the_public_bucket_is_set(
+ mock_settings, mock_storage_client
+):
+ mock_settings.config.public_site_media_bucket = "public-media"
+ test_file = fastapi.UploadFile(
+ filename="private.jpeg",
+ file=io.BytesIO(b"\xff\xd8\xffprivate"),
+ headers=starlette.datastructures.Headers({"content-type": "image/jpeg"}),
+ )
+
+ result = await store_media.upload_media("test-user", test_file)
+
+ assert mock_storage_client.upload.await_args.args[0] == "test-bucket"
+ assert result.startswith("/api/store/submissions/media/test-user/images/")
+
+
+async def test_check_media_exists_looks_up_the_name_uploads_are_stored_under(
+ mock_settings, mock_storage_client
+):
+ mock_settings.config.public_site_media_bucket = "public-media"
+ mock_settings.config.private_user_data_bucket = "private-media"
+ mock_storage_client.download_metadata.return_value = {}
+
+ result = await store_media.check_media_exists("test-user", "agent.jpg")
+
+ mock_storage_client.download_metadata.assert_awaited_once_with(
+ "private-media", "users/test-user/images/agent.jpeg"
+ )
+ assert result == "/api/store/submissions/media/test-user/images/agent.jpeg"
+
+
+async def test_upload_media_is_private_when_only_the_private_bucket_is_set(
+ mock_settings, mock_storage_client
+):
+ mock_settings.config.media_gcs_bucket_name = ""
+ mock_settings.config.private_user_data_bucket = "private-media"
+ test_file = fastapi.UploadFile(
+ filename="private.jpeg",
+ file=io.BytesIO(b"\xff\xd8\xffprivate"),
+ headers=starlette.datastructures.Headers({"content-type": "image/jpeg"}),
+ )
+
+ result = await store_media.upload_media("test-user", test_file)
+
+ assert mock_storage_client.upload.await_args.args[0] == "private-media"
+ assert result.startswith("/api/store/submissions/media/test-user/images/")
diff --git a/autogpt_platform/backend/backend/api/features/store/public_media.py b/autogpt_platform/backend/backend/api/features/store/public_media.py
new file mode 100644
index 0000000000..abfd0e63b7
--- /dev/null
+++ b/autogpt_platform/backend/backend/api/features/store/public_media.py
@@ -0,0 +1,140 @@
+import logging
+import re
+from collections.abc import Iterable
+from urllib.parse import parse_qs, unquote, urlsplit
+
+from gcloud.aio import storage as async_storage
+
+from backend.util.settings import Settings
+
+from . import submission_media
+
+logger = logging.getLogger(__name__)
+
+_GCS_URL_FORMS = (
+ re.compile(
+ r"https?://(?:www|storage)\.googleapis\.com/(?:download/)?storage/v1"
+ r"/b/(?P<bucket>[^/?#]+)/o/(?P<path>[^?#]+)(?:[?#].*)?",
+ re.IGNORECASE,
+ ),
+ re.compile(
+ r"https?://(?:storage\.googleapis\.com|storage\.cloud\.google\.com"
+ r"|commondatastorage\.googleapis\.com)/(?P<bucket>[^/?#]+)/(?P<path>[^?#]+)"
+ r"(?:[?#].*)?",
+ re.IGNORECASE,
+ ),
+ re.compile(
+ r"https?://(?P<bucket>[a-z0-9._-]+)\.storage\.googleapis\.com"
+ r"/(?P<path>[^?#]+)(?:[?#].*)?",
+ re.IGNORECASE,
+ ),
+ re.compile(r"gs://(?P<bucket>[^/?#]+)/(?P<path>[^?#]+)", re.IGNORECASE),
+)
+_PUBLISHABLE_MEDIA_PATH = re.compile(
+ r"users/(?P<owner>[^/]+)/(?P<media_type>images|videos)/(?P<filename>[^/]+)"
+)
+
+
+def publishing_enabled() -> bool:
+ """True when public and private media live in different buckets."""
+ config = Settings().config
+ private_bucket = config.resolved_private_user_data_bucket
+ public_bucket = config.resolved_public_site_media_bucket
+ return bool(private_bucket and public_bucket and private_bucket != public_bucket)
+
+
+async def publish_urls(
+ urls: Iterable[str | None], owner_ids: Iterable[str]
+) -> dict[str, str]:
+ """
+ Copy the private media among `urls` that belongs to one of `owner_ids` to
+ the public bucket and map each published source URL to its public URL.
+ """
+ if not publishing_enabled():
+ return {}
+ config = Settings().config
+ private_bucket = config.resolved_private_user_data_bucket
+ public_bucket = config.resolved_public_site_media_bucket
+
+ paths = _publishable_paths(urls, frozenset(owner_ids), private_bucket)
+ if not paths:
+ return {}
+
+ published: dict[str, str] = {}
+ async with async_storage.Storage() as async_client:
+ for source_url, path in paths.items():
+ try:
+ await async_client.copy(
+ private_bucket, path, public_bucket, new_name=path
+ )
+ except Exception:
+ logger.exception(f"Failed to publish {path} to {public_bucket}")
+ continue
+ published[source_url] = (
+ f"https://storage.googleapis.com/{public_bucket}/{path}"
+ )
+ logger.info(f"Published {path} to {public_bucket}")
+ return published
+
+
+def _publishable_paths(
+ urls: Iterable[str | None], owner_ids: frozenset[str], private_bucket: str
+) -> dict[str, str]:
+ paths: dict[str, str] = {}
+ for source_url in dict.fromkeys(url for url in urls if url):
+ path = object_path_from_url(source_url, private_bucket)
+ if path is None:
+ continue
+ match = _PUBLISHABLE_MEDIA_PATH.fullmatch(path)
+ if (
+ not match
+ or match["owner"] not in owner_ids
+ or not _is_valid_path(match, path)
+ ):
+ logger.error(f"Not publishing {path!r}: not media of the listing owners")
+ continue
+ paths[source_url] = path
+ return paths
+
+
+def _is_valid_path(match: re.Match[str], path: str) -> bool:
+ try:
+ canonical = submission_media.object_path(
+ match["owner"], match["media_type"], match["filename"]
+ )
+ except ValueError:
+ return False
+ return canonical == path
+
+
+def object_path_from_url(url: str, bucket: str) -> str | None:
+ url = url.strip()
+ wrapped = urlsplit(url)
+ if wrapped.path == "/_next/image":
+ url = parse_qs(wrapped.query).get("url", [""])[0].strip()
+ if private_path := _private_api_object_path(url):
+ return private_path
+
+ url = unquote(url)
+ for form in _GCS_URL_FORMS:
+ match = form.fullmatch(url)
+ if match:
+ return match["path"] if match["bucket"] == bucket else None
+ return None
+
+
+def _private_api_object_path(url: str) -> str | None:
+ parsed = urlsplit(url)
+ if parsed.scheme or parsed.netloc or parsed.query or parsed.fragment:
+ return None
+ path = unquote(parsed.path)
+ if not path.startswith(submission_media.PRIVATE_MEDIA_PREFIX):
+ return None
+ parts = path.removeprefix(submission_media.PRIVATE_MEDIA_PREFIX).split("/")
+ if len(parts) != 3:
+ return None
+ owner_id, media_type, filename = parts
+ try:
+ return submission_media.object_path(owner_id, media_type, filename)
+ except ValueError:
+ return None
diff --git a/autogpt_platform/backend/backend/api/features/store/public_media_test.py b/autogpt_platform/backend/backend/api/features/store/public_media_test.py
new file mode 100644
index 0000000000..ee33602a29
--- /dev/null
+++ b/autogpt_platform/backend/backend/api/features/store/public_media_test.py
@@ -0,0 +1,179 @@
+import unittest.mock
+from unittest.mock import AsyncMock
+
+import pytest
+
+from backend.util.settings import Settings
+
+from . import public_media
+
+OWNER = "owner-1"
+OWN_IMAGE = "users/owner-1/images/shot.png"
+OWN_VIDEO = "users/owner-1/videos/demo.mp4"
+
+
+def mock_settings(monkeypatch):
+ settings = Settings()
+ monkeypatch.setattr(settings.config, "media_gcs_bucket_name", "test-bucket")
+ monkeypatch.setattr(settings.config, "public_site_media_bucket", "")
+ monkeypatch.setattr(settings.config, "private_user_data_bucket", "")
+ monkeypatch.setattr(
+ "backend.api.features.store.public_media.Settings", lambda: settings
+ )
+ return settings
+
+
+def mock_storage_client(mocker):
+ client = AsyncMock()
+ client.__aenter__ = AsyncMock(return_value=client)
+ client.__aexit__ = AsyncMock(return_value=None)
+ mocker.patch(
+ "backend.api.features.store.public_media.async_storage.Storage",
+ return_value=client,
+ )
+ return client
+
+
+def public_bucket(mock_settings):
+ mock_settings.config.public_site_media_bucket = "public-bucket"
+ return mock_settings
+
+
+async def test_publish_urls_is_a_noop_when_unset(mock_settings, mock_storage_client):
+ result = await public_media.publish_urls(
+ [f"https://storage.googleapis.com/test-bucket/{OWN_IMAGE}"], [OWNER]
+ )
+
+ assert result == {}
+ mock_storage_client.copy.assert_not_called()
+
+
+async def test_publish_urls_copies_own_media_to_the_same_path(
+ public_bucket, mock_storage_client
+):
+ image = f"https://storage.googleapis.com/test-bucket/{OWN_IMAGE}"
+ video = f"https://storage.googleapis.com/test-bucket/{OWN_VIDEO}"
+
+ result = await public_media.publish_urls([image, video, None, image], [OWNER])
+
+ assert result == {
+ image: f"https://storage.googleapis.com/public-bucket/{OWN_IMAGE}",
+ video: f"https://storage.googleapis.com/public-bucket/{OWN_VIDEO}",
+ }
+ assert mock_storage_client.copy.await_args_list == [
+ unittest.mock.call(
+ "test-bucket", OWN_IMAGE, "public-bucket", new_name=OWN_IMAGE
+ ),
+ unittest.mock.call(
+ "test-bucket", OWN_VIDEO, "public-bucket", new_name=OWN_VIDEO
+ ),
+ ]
+
+
+ "url",
+ [
+ f" https://storage.googleapis.com/test-bucket/{OWN_IMAGE} ",
+ f"https://storage.cloud.google.com/test-bucket/{OWN_IMAGE}?authuser=0",
+ f"https://commondatastorage.googleapis.com/test-bucket/{OWN_IMAGE}",
+ "https://storage.googleapis.com/download/storage/v1/b/test-bucket/o/"
+ "users%2Fowner-1%2Fimages%2Fshot.png?alt=media",
+ f"https://test-bucket.storage.googleapis.com/{OWN_IMAGE}",
+ f"gs://test-bucket/{OWN_IMAGE}",
+ "https://storage.googleapis.com/test-bucket/users/owner-1/images/shot%2Epng",
+ "/_next/image?url=https%3A%2F%2Fstorage.googleapis.com%2Ftest-bucket%2F"
+ "users%2Fowner-1%2Fimages%2Fshot.png&w=640&q=75",
+ "/api/store/submissions/media/owner-1/images/shot.png",
+ ],
+)
+async def test_publish_urls_accepts_every_managed_url_form(
+ public_bucket, mock_storage_client, url
+):
+ result = await public_media.publish_urls([url], [OWNER])
+
+ assert result == {url: f"https://storage.googleapis.com/public-bucket/{OWN_IMAGE}"}
+ mock_storage_client.copy.assert_awaited_once_with(
+ "test-bucket", OWN_IMAGE, "public-bucket", new_name=OWN_IMAGE
+ )
+
+
+ "url",
+ [
+ "https://storage.googleapis.com/test-bucket/users/someone-else/images/a.png",
+ "https://storage.googleapis.com/test-bucket/uploads/owner-1/secret.pdf",
+ "https://storage.googleapis.com/test-bucket/workspaces/owner-1/notes.md",
+ "https://storage.googleapis.com/test-bucket/users/owner-1/files/a.png",
+ "https://storage.googleapis.com/test-bucket/users/owner-1/images/sub/a.png",
+ "https://storage.googleapis.com/test-bucket/users/owner-1/images/..",
+ "https://storage.googleapis.com/test-bucket/users/../images/a.png",
+ "https://storage.googleapis.com/test-bucket/users/owner-1/images/a%20b.png",
+ f"https://storage.googleapis.com/public-bucket/{OWN_IMAGE}",
+ f"https://storage.googleapis.com/other-bucket/{OWN_IMAGE}",
+ f"https://example.com/test-bucket/{OWN_IMAGE}",
+ f"https://example.com/?u=https://storage.googleapis.com/test-bucket/{OWN_IMAGE}",
+ "https://attacker.test/api/store/submissions/media/owner-1/images/shot.png",
+ "/api/store/submissions/media/someone-else/images/shot.png",
+ "/api/store/media/owner-1/images/shot.png",
+ "",
+ ],
+)
+async def test_publish_urls_refuses_everything_else(
+ public_bucket, mock_storage_client, url
+):
+ assert await public_media.publish_urls([url], [OWNER]) == {}
+ mock_storage_client.copy.assert_not_called()
+
+
+async def test_publish_urls_skips_objects_that_fail_to_copy(
+ public_bucket, mock_storage_client
+):
+ image = f"https://storage.googleapis.com/test-bucket/{OWN_IMAGE}"
+ video = f"https://storage.googleapis.com/test-bucket/{OWN_VIDEO}"
+ mock_storage_client.copy.side_effect = [Exception("404 not found"), {}]
+
+ result = await public_media.publish_urls([image, video], [OWNER])
+
+ assert result == {
+ video: f"https://storage.googleapis.com/public-bucket/{OWN_VIDEO}"
+ }
+
+
+async def test_publish_urls_is_idempotent(public_bucket, mock_storage_client):
+ image = f"https://storage.googleapis.com/test-bucket/{OWN_IMAGE}"
+
+ first = await public_media.publish_urls([image], [OWNER])
+ second = await public_media.publish_urls(list(first.values()), [OWNER])
+
+ assert second == {}
+ mock_storage_client.copy.assert_awaited_once()
+
+
+async def test_publish_urls_accepts_media_of_every_listed_owner(
+ public_bucket, mock_storage_client
+):
+ member_image = "users/member-2/images/shot.png"
+ url = f"https://storage.googleapis.com/test-bucket/{member_image}"
+
+ result = await public_media.publish_urls([url], [OWNER, "member-2"])
+
+ assert result == {
+ url: f"https://storage.googleapis.com/public-bucket/{member_image}"
+ }
+
+
+async def test_publish_urls_falls_back_to_the_legacy_bucket_as_public(
+ mock_settings, mock_storage_client
+):
+ mock_settings.config.private_user_data_bucket = "private-bucket"
+ image = f"https://storage.googleapis.com/private-bucket/{OWN_IMAGE}"
+
+ result = await public_media.publish_urls([image], [OWNER])
+
+ assert result == {image: f"https://storage.googleapis.com/test-bucket/{OWN_IMAGE}"}
+ mock_storage_client.copy.assert_awaited_once_with(
+ "private-bucket", OWN_IMAGE, "test-bucket", new_name=OWN_IMAGE
+ )
diff --git a/autogpt_platform/backend/backend/api/features/store/routes.py b/autogpt_platform/backend/backend/api/features/store/routes.py
index 7affe4679a..711a438c8e 100644
--- a/autogpt_platform/backend/backend/api/features/store/routes.py
+++ b/autogpt_platform/backend/backend/api/features/store/routes.py
@@ -1,5 +1,6 @@
import logging
import urllib.parse
+from collections.abc import Mapping
from typing import Literal
import autogpt_libs.auth
@@ -24,6 +25,7 @@ from . import image_gen as store_image_gen
from . import local_media
from . import media as store_media
from . import model as store_model
+from . import submission_media
logger = logging.getLogger(__name__)
@@ -578,6 +580,115 @@ async def upload_submission_media(
)
+ "/submissions/media/{owner_user_id}/{media_type}/{filename}",
+ summary="Read private submission media",
+ response_class=fastapi.responses.StreamingResponse,
+ responses={
+ 200: {
+ "content": {
+ content_type: {"schema": {"type": "string", "format": "binary"}}
+ for content_type in local_media.CONTENT_TYPE_EXTENSIONS
+ }
+ },
+ 206: {
+ "description": "Requested byte range",
+ "content": {
+ content_type: {"schema": {"type": "string", "format": "binary"}}
+ for content_type in local_media.CONTENT_TYPE_EXTENSIONS
+ },
+ },
+ 404: {"description": "Media not found"},
+ 416: {"description": "Requested range is not satisfiable"},
+ },
+ tags=["store", "private"],
+)
+async def get_private_submission_media(
+ owner_user_id: str,
+ media_type: str,
+ filename: str,
+ request: fastapi.Request,
+ user: autogpt_libs.auth.User = Security(autogpt_libs.auth.requires_user),
+) -> fastapi.responses.StreamingResponse:
+ """
+ Serve a user's private media to them, to members of an organization they
+ share, and to admins. Anyone else gets a 404, as if it did not exist.
+ """
+ content_type = local_media.content_type_for_filename(filename)
+ if content_type is None:
+ raise NotFoundError("Media not found")
+ if not await submission_media.can_read(user, owner_user_id):
+ raise NotFoundError("Media not found")
+ try:
+ metadata = await submission_media.metadata(owner_user_id, media_type, filename)
+ except (FileNotFoundError, ValueError):
+ raise NotFoundError("Media not found")
+
+ headers = {
+ "Cache-Control": "private, no-store",
+ "X-Content-Type-Options": "nosniff",
+ "Accept-Ranges": "bytes",
+ }
+ total_size = _private_media_size(metadata)
+ byte_range: tuple[int, int] | None = None
+ status_code = 200
+ if range_header := request.headers.get("range"):
+ try:
+ byte_range = _parse_private_media_range(range_header, total_size)
+ except ValueError:
+ headers["Content-Range"] = f"bytes */{total_size}"
+ raise fastapi.HTTPException(
+ status_code=416,
+ detail="Requested range is not satisfiable",
+ headers=headers,
+ )
+ start, end = byte_range
+ headers["Content-Range"] = f"bytes {start}-{end}/{total_size}"
+ headers["Content-Length"] = str(end - start + 1)
+ status_code = 206
+ else:
+ headers["Content-Length"] = str(total_size)
+ return fastapi.responses.StreamingResponse(
+ submission_media.stream(owner_user_id, media_type, filename, byte_range),
+ status_code=status_code,
+ media_type=content_type,
+ headers=headers,
+ )
+
+
+def _private_media_size(metadata: Mapping[str, object]) -> int:
+ try:
+ size = int(str(metadata["size"]))
+ except (KeyError, TypeError, ValueError) as error:
+ raise NotFoundError("Media not found") from error
+ if size < 0:
+ raise NotFoundError("Media not found")
+ return size
+
+
+def _parse_private_media_range(value: str, total_size: int) -> tuple[int, int]:
+ if total_size <= 0 or len(value) > 64 or not value.startswith("bytes="):
+ raise ValueError("Invalid range")
+ specification = value.removeprefix("bytes=").strip()
+ if "," in specification or specification.count("-") != 1:
+ raise ValueError("Invalid range")
+ start_text, end_text = specification.split("-", 1)
+ if start_text:
+ if not start_text.isdigit() or (end_text and not end_text.isdigit()):
+ raise ValueError("Invalid range")
+ start = int(start_text)
+ end = int(end_text) if end_text else total_size - 1
+ if start >= total_size or end < start:
+ raise ValueError("Invalid range")
+ return start, min(end, total_size - 1)
+ if not end_text.isdigit():
+ raise ValueError("Invalid range")
+ suffix_length = int(end_text)
+ if suffix_length <= 0:
+ raise ValueError("Invalid range")
+ return max(0, total_size - suffix_length), total_size - 1
+
+
class ImageURLResponse(BaseModel):
image_url: str
@@ -605,7 +716,11 @@ async def generate_image(
# Use .jpeg here since we are generating JPEG images
filename = f"agent_{graph_id}.jpeg"
- existing_url = await store_media.check_media_exists(user_id, filename)
+ try:
+ existing_url = await store_media.check_media_exists(user_id, filename)
+ except Exception:
+ logger.exception(f"Could not check for an existing image of graph {graph_id}")
+ existing_url = None
if existing_url:
logger.info(f"Using existing image for agent graph {graph_id}")
return ImageURLResponse(image_url=existing_url)
diff --git a/autogpt_platform/backend/backend/api/features/store/skill_submission_db.py b/autogpt_platform/backend/backend/api/features/store/skill_submission_db.py
index b7969cb37c..b37f5ae356 100644
--- a/autogpt_platform/backend/backend/api/features/store/skill_submission_db.py
+++ b/autogpt_platform/backend/backend/api/features/store/skill_submission_db.py
@@ -24,6 +24,7 @@ from backend.copilot.tools.skills import (
from backend.data.db import transaction
from backend.util.exceptions import NotFoundError, PreconditionFailed
+from . import db as store_db
from . import skill_db, skill_model
_MAX_LICENSE_CHARS = 256
@@ -199,6 +200,8 @@ async def review_skill_submission(
)
or listing
)
+ if is_approved and listing.owningUserId:
+ await store_db.publish_creator_avatar(listing.owningUserId)
return skill_model.SkillSubmission.from_db(updated, listing)
diff --git a/autogpt_platform/backend/backend/api/features/store/skill_submission_db_test.py b/autogpt_platform/backend/backend/api/features/store/skill_submission_db_test.py
index 6b451f5668..f927f1a74f 100644
--- a/autogpt_platform/backend/backend/api/features/store/skill_submission_db_test.py
+++ b/autogpt_platform/backend/backend/api/features/store/skill_submission_db_test.py
@@ -171,8 +171,11 @@ async def test_a_slug_another_creator_holds_is_refused(creator):
async def test_approval_promotes_the_version_and_puts_it_on_the_shelf(
- creator, reviewer
+ creator, reviewer, mocker
):
+ publish_avatar = mocker.patch.object(
+ skill_submission_db.store_db, "publish_creator_avatar", autospec=True
+ )
submission = await skill_submission_db.submit_skill(creator, _request())
reviewed = await skill_submission_db.review_skill_submission(
@@ -186,6 +189,7 @@ async def test_approval_promotes_the_version_and_puts_it_on_the_shelf(
assert reviewed.is_live is True
browse = await skill_db.get_marketplace_skills()
assert [s.slug for s in browse.skills] == ["brand-voice-guide"]
+ publish_avatar.assert_awaited_once_with(creator)
async def test_rejection_keeps_the_skill_off_the_shelf(creator, reviewer):
diff --git a/autogpt_platform/backend/backend/api/features/store/submission_media.py b/autogpt_platform/backend/backend/api/features/store/submission_media.py
new file mode 100644
index 0000000000..c1de379058
--- /dev/null
+++ b/autogpt_platform/backend/backend/api/features/store/submission_media.py
@@ -0,0 +1,112 @@
+from collections.abc import AsyncIterator, Mapping
+
+import aiohttp
+import prisma.enums
+import prisma.models
+from autogpt_libs.auth import User
+from gcloud.aio import storage as async_storage
+
+from backend.util.gcs_utils import is_not_found_error
+from backend.util.settings import Settings
+
+from . import local_media
+
+PRIVATE_MEDIA_PREFIX = "/api/store/submissions/media/"
+
+# The library's default is a 10-second total timeout, which covers the whole
+# body and so cuts off any video a client can't download in 10 seconds.
+_STREAM_TIMEOUT = aiohttp.ClientTimeout(total=None, sock_connect=10, sock_read=60)
+
+
+async def can_read(user: User, owner_user_id: str) -> bool:
+ """
+ Whether `user` may read media stored under `owner_user_id`: their own, an
+ admin, or someone sharing an active organization with the owner, who can
+ see that owner's org listings, org avatar and shared agents. Leaving the
+ organization ends access.
+ """
+ if user.user_id == owner_user_id or user.role == "admin":
+ return True
+ shared_membership = await prisma.models.OrgMember.prisma().find_first(
+ where={
+ "userId": user.user_id,
+ "status": prisma.enums.OrgMemberStatus.ACTIVE,
+ "Org": {
+ "is": {
+ "deletedAt": None,
+ "Members": {
+ "some": {
+ "userId": owner_user_id,
+ "status": prisma.enums.OrgMemberStatus.ACTIVE,
+ }
+ },
+ }
+ },
+ }
+ )
+ return shared_membership is not None
+
+
+def url(user_id: str, media_type: str, filename: str) -> str:
+ object_path(user_id, media_type, filename)
+ return (
+ f"{PRIVATE_MEDIA_PREFIX}{local_media.validate_path_component(user_id)}"
+ f"/{media_type}/{local_media.validate_path_component(filename)}"
+ )
+
+
+async def metadata(
+ user_id: str, media_type: str, filename: str
+) -> Mapping[str, object]:
+ bucket_name = Settings().config.resolved_private_user_data_bucket
+ if not bucket_name:
+ return await local_media.media_metadata(user_id, media_type, filename)
+
+ storage_path = object_path(user_id, media_type, filename)
+ try:
+ async with async_storage.Storage() as async_client:
+ return await async_client.download_metadata(bucket_name, storage_path)
+ except Exception as error:
+ if is_not_found_error(error):
+ raise FileNotFoundError("Private media not found") from error
+ raise
+
+
+async def stream(
+ user_id: str,
+ media_type: str,
+ filename: str,
+ byte_range: tuple[int, int] | None = None,
+) -> AsyncIterator[bytes]:
+ bucket_name = Settings().config.resolved_private_user_data_bucket
+ if not bucket_name:
+ async for chunk in local_media.stream_media(
+ user_id, media_type, filename, byte_range
+ ):
+ yield chunk
+ return
+
+ storage_path = object_path(user_id, media_type, filename)
+ headers = None
+ if byte_range is not None:
+ start, end = byte_range
+ headers = {"Range": f"bytes={start}-{end}"}
+ async with async_storage.Storage() as async_client:
+ stream = await async_client.download_stream(
+ bucket_name,
+ storage_path,
+ headers=headers,
+ timeout=_STREAM_TIMEOUT, # type: ignore[arg-type]
+ )
+ while chunk := await stream.read(64 * 1024):
+ yield chunk
+
+
+def object_path(user_id: str, media_type: str, filename: str) -> str:
+ if media_type not in local_media.MEDIA_TYPES:
+ raise ValueError("Invalid media type")
+ safe_user_id = local_media.validate_path_component(user_id)
+ safe_filename = local_media.validate_path_component(filename)
+ if local_media.content_type_for_filename(safe_filename) is None:
+ raise ValueError("Invalid media filename")
+ return f"users/{safe_user_id}/{media_type}/{safe_filename}"
diff --git a/autogpt_platform/backend/backend/api/features/store/submission_media_test.py b/autogpt_platform/backend/backend/api/features/store/submission_media_test.py
new file mode 100644
index 0000000000..5be6c8c35a
--- /dev/null
+++ b/autogpt_platform/backend/backend/api/features/store/submission_media_test.py
@@ -0,0 +1,292 @@
+from unittest.mock import AsyncMock
+
+import autogpt_libs.auth
+import fastapi
+import prisma.enums
+import pytest
+
+from backend.util.exceptions import NotFoundError
+from backend.util.settings import Settings
+
+from . import routes as store_routes
+from . import submission_media
+
+
+def _user(user_id: str, role: str = "authenticated") -> autogpt_libs.auth.User:
+ return autogpt_libs.auth.User(user_id=user_id, email="", phone_number="", role=role)
+
+
+OWNER = _user("owner")
+
+
+def mock_settings(monkeypatch):
+ settings = Settings()
+ monkeypatch.setattr(settings.config, "media_gcs_bucket_name", "test-bucket")
+ monkeypatch.setattr(settings.config, "private_user_data_bucket", "private-media")
+ monkeypatch.setattr(
+ "backend.api.features.store.submission_media.Settings", lambda: settings
+ )
+ return settings
+
+
+def mock_storage_client(mocker):
+ client = AsyncMock()
+ client.__aenter__ = AsyncMock(return_value=client)
+ client.__aexit__ = AsyncMock(return_value=None)
+ mocker.patch(
+ "backend.api.features.store.submission_media.async_storage.Storage",
+ return_value=client,
+ )
+ return client
+
+
+def test_private_media_route_requires_an_authenticated_user():
+ route = next(
+ route
+ for route in store_routes.router.routes
+ if getattr(route, "path", "")
+ == "/submissions/media/{owner_user_id}/{media_type}/{filename}"
+ )
+
+ assert autogpt_libs.auth.requires_user in {
+ dependency.call for dependency in route.dependant.dependencies
+ }
+
+
+async def test_private_media_read_serves_the_owner(mocker):
+ mocker.patch.object(
+ submission_media, "metadata", new=AsyncMock(return_value={"size": "13"})
+ )
+
+ async def chunks(*args):
+ yield b"private-image"
+
+ mocker.patch.object(submission_media, "stream", side_effect=chunks)
+
+ response = await store_routes.get_private_submission_media(
+ owner_user_id="owner",
+ media_type="images",
+ filename="image.jpeg",
+ request=fastapi.Request({"type": "http", "headers": []}),
+ user=OWNER,
+ )
+
+ assert (
+ b"".join([chunk async for chunk in response.body_iterator]) == b"private-image"
+ )
+ assert response.headers["cache-control"] == "private, no-store"
+ assert response.headers["x-content-type-options"] == "nosniff"
+ assert response.headers["content-length"] == "13"
+
+
+async def test_private_media_read_returns_not_found_for_missing_object(mocker):
+ mocker.patch.object(
+ submission_media,
+ "metadata",
+ new=AsyncMock(side_effect=FileNotFoundError("missing")),
+ )
+
+ with pytest.raises(NotFoundError):
+ await store_routes.get_private_submission_media(
+ owner_user_id="owner",
+ media_type="images",
+ filename="image.jpeg",
+ request=fastapi.Request({"type": "http", "headers": []}),
+ user=OWNER,
+ )
+
+
+ ("value", "total_size", "expected"),
+ [
+ ("bytes=0-4", 10, (0, 4)),
+ ("bytes=5-", 10, (5, 9)),
+ ("bytes=-3", 10, (7, 9)),
+ ("bytes=0-99", 10, (0, 9)),
+ ],
+)
+def test_private_media_range_parsing(value, total_size, expected):
+ assert store_routes._parse_private_media_range(value, total_size) == expected
+
+
+ ("value", "total_size"),
+ [
+ ("items=0-1", 10),
+ ("bytes=", 10),
+ ("bytes=5-4", 10),
+ ("bytes=10-", 10),
+ ("bytes=-0", 10),
+ ("bytes=0-1,3-4", 10),
+ ("bytes=0-0", 0),
+ ],
+)
+def test_private_media_range_rejects_invalid_or_unsatisfiable_values(value, total_size):
+ with pytest.raises(ValueError):
+ store_routes._parse_private_media_range(value, total_size)
+
+
+async def test_private_media_read_serves_single_byte_range(mocker):
+ mocker.patch.object(
+ submission_media, "metadata", new=AsyncMock(return_value={"size": "13"})
+ )
+
+ async def chunks(*args):
+ assert args[-1] == (2, 5)
+ yield b"ivat"
+
+ stream = mocker.patch.object(submission_media, "stream", side_effect=chunks)
+ request = fastapi.Request({"type": "http", "headers": [(b"range", b"bytes=2-5")]})
+
+ response = await store_routes.get_private_submission_media(
+ owner_user_id="owner",
+ media_type="videos",
+ filename="preview.mp4",
+ request=request,
+ user=OWNER,
+ )
+
+ assert response.status_code == 206
+ assert response.headers["content-range"] == "bytes 2-5/13"
+ assert response.headers["content-length"] == "4"
+ assert response.headers["accept-ranges"] == "bytes"
+ assert b"".join([chunk async for chunk in response.body_iterator]) == b"ivat"
+ assert stream.call_count == 1
+
+
+async def test_private_media_read_rejects_unsatisfiable_range(mocker):
+ mocker.patch.object(
+ submission_media, "metadata", new=AsyncMock(return_value={"size": "13"})
+ )
+ stream = mocker.patch.object(submission_media, "stream")
+ request = fastapi.Request({"type": "http", "headers": [(b"range", b"bytes=99-")]})
+
+ with pytest.raises(fastapi.HTTPException) as error:
+ await store_routes.get_private_submission_media(
+ owner_user_id="owner",
+ media_type="videos",
+ filename="preview.mp4",
+ request=request,
+ user=OWNER,
+ )
+
+ assert error.value.status_code == 416
+ assert error.value.headers["Content-Range"] == "bytes */13"
+ assert error.value.headers["Cache-Control"] == "private, no-store"
+ stream.assert_not_called()
+
+
+async def test_private_media_stream_reads_only_from_private_bucket(
+ mock_settings, mock_storage_client
+):
+ class Stream:
+ def __init__(self):
+ self.chunks = [b"private", b"-image", b""]
+
+ async def read(self, size):
+ return self.chunks.pop(0)
+
+ mock_storage_client.download_stream.return_value = Stream()
+
+ chunks = [
+ chunk
+ async for chunk in submission_media.stream("owner", "images", "image.jpeg")
+ ]
+
+ assert b"".join(chunks) == b"private-image"
+ mock_storage_client.download_stream.assert_awaited_once_with(
+ "private-media",
+ "users/owner/images/image.jpeg",
+ headers=None,
+ timeout=submission_media._STREAM_TIMEOUT,
+ )
+ # A total timeout would cut off any body the client reads slowly.
+ assert submission_media._STREAM_TIMEOUT.total is None
+
+
+async def test_private_media_range_stream_uses_gcs_range_header(
+ mock_settings, mock_storage_client
+):
+ class Stream:
+ def __init__(self):
+ self.chunks = [b"part", b""]
+
+ async def read(self, size):
+ return self.chunks.pop(0)
+
+ mock_storage_client.download_stream.return_value = Stream()
+
+ chunks = [
+ chunk
+ async for chunk in submission_media.stream(
+ "owner", "videos", "preview.mp4", (2, 5)
+ )
+ ]
+
+ assert b"".join(chunks) == b"part"
+ mock_storage_client.download_stream.assert_awaited_once_with(
+ "private-media",
+ "users/owner/videos/preview.mp4",
+ headers={"Range": "bytes=2-5"},
+ timeout=submission_media._STREAM_TIMEOUT,
+ )
+
+
+ "user, shares_an_org, allowed",
+ [
+ (_user("owner"), False, True),
+ (_user("reviewer", role="admin"), False, True),
+ (_user("colleague"), True, True),
+ (_user("stranger"), False, False),
+ ],
+)
+async def test_private_media_is_readable_by_owner_admins_and_colleagues(
+ mocker, user, shares_an_org, allowed
+):
+ find_membership = mocker.patch("prisma.models.OrgMember.prisma")
+ find_membership.return_value.find_first = AsyncMock(
+ return_value=mocker.MagicMock() if shares_an_org else None
+ )
+
+ assert await submission_media.can_read(user, "owner") is allowed
+
+
+async def test_private_media_read_hides_media_from_other_users(mocker):
+ mocker.patch.object(submission_media, "can_read", new=AsyncMock(return_value=False))
+ metadata = mocker.patch.object(submission_media, "metadata", new=AsyncMock())
+
+ with pytest.raises(NotFoundError):
+ await store_routes.get_private_submission_media(
+ owner_user_id="owner",
+ media_type="images",
+ filename="image.jpeg",
+ request=fastapi.Request({"type": "http", "headers": []}),
+ user=_user("stranger"),
+ )
+ metadata.assert_not_awaited()
+
+
+async def test_colleague_access_requires_active_memberships_in_a_live_org(mocker):
+ find_membership = mocker.patch("prisma.models.OrgMember.prisma")
+ find_membership.return_value.find_first = AsyncMock(return_value=None)
+
+ await submission_media.can_read(_user("colleague"), "owner")
+
+ assert find_membership.return_value.find_first.await_args.kwargs["where"] == {
+ "userId": "colleague",
+ "status": prisma.enums.OrgMemberStatus.ACTIVE,
+ "Org": {
+ "is": {
+ "deletedAt": None,
+ "Members": {
+ "some": {
+ "userId": "owner",
+ "status": prisma.enums.OrgMemberStatus.ACTIVE,
+ }
+ },
+ }
+ },
+ }
diff --git a/autogpt_platform/backend/backend/util/cloud_storage.py b/autogpt_platform/backend/backend/util/cloud_storage.py
index 0597abad46..aef6c47258 100644
--- a/autogpt_platform/backend/backend/util/cloud_storage.py
+++ b/autogpt_platform/backend/backend/util/cloud_storage.py
@@ -30,7 +30,7 @@ class CloudStorageConfig:
config = Config()
# GCS configuration from settings - uses Application Default Credentials
- self.gcs_bucket_name = config.media_gcs_bucket_name
+ self.gcs_bucket_name = config.resolved_private_user_data_bucket
# Future providers can be added here
# self.aws_bucket_name = config.aws_bucket_name
@@ -207,6 +207,9 @@ class CloudStorageHandler:
self.config.gcs_bucket_name,
blob_name,
content,
+ # Top-level keys are not stored as custom metadata, so the expiry
+ # cleanup never fires. Nesting them under "metadata" would switch it
+ # on and delete files that saved schedules and presets still use.
metadata={
"uploaded_at": upload_time.isoformat(),
"expires_at": expiration_time.isoformat(),
@@ -256,12 +259,7 @@ class CloudStorageHandler:
f"in_task: {current_task is not None}"
)
- # Parse bucket and blob name from path (path already has gcs:// prefix removed)
- parts = path.split("/", 1)
- if len(parts) != 2:
- raise ValueError(f"Invalid GCS path: {path}")
-
- bucket_name, blob_name = parts
+ bucket_name, blob_name = self._parse_configured_gcs_path(path)
# Authorization check
self._validate_file_access(blob_name, user_id, graph_exec_id)
@@ -415,12 +413,7 @@ class CloudStorageHandler:
graph_exec_id: str | None = None,
) -> str:
"""Generate signed URL for GCS with authorization."""
- # Parse bucket and blob name from path (path already has gcs:// prefix removed)
- parts = path.split("/", 1)
- if len(parts) != 2:
- raise ValueError(f"Invalid GCS path: {path}")
-
- bucket_name, blob_name = parts
+ bucket_name, blob_name = self._parse_configured_gcs_path(path)
# Authorization check
self._validate_file_access(blob_name, user_id, graph_exec_id)
@@ -430,6 +423,18 @@ class CloudStorageHandler:
sync_client, bucket_name, blob_name, expiration_hours * 3600
)
+ def _parse_configured_gcs_path(self, path: str) -> tuple[str, str]:
+ parts = path.split("/", 1)
+ if len(parts) != 2:
+ raise ValueError(f"Invalid GCS path: {path}")
+ bucket_name, blob_name = parts
+ configured_bucket = self.config.gcs_bucket_name
+ if not configured_bucket:
+ raise ValueError("GCS_BUCKET_NAME not configured")
+ if bucket_name != configured_bucket:
+ raise PermissionError("Access denied: not the configured private bucket")
+ return bucket_name, blob_name
+
async def delete_expired_files(self, provider: str = "gcs") -> int:
"""
Delete files that have passed their expiration time.
@@ -529,11 +534,7 @@ class CloudStorageHandler:
async def _check_file_expired_gcs(self, path: str) -> bool:
"""Check if a GCS file has expired."""
- parts = path.split("/", 1)
- if len(parts) != 2:
- raise ValueError(f"Invalid GCS path: {path}")
-
- bucket_name, blob_name = parts
+ bucket_name, blob_name = self._parse_configured_gcs_path(path)
async_client = await self._get_async_gcs_client()
diff --git a/autogpt_platform/backend/backend/util/cloud_storage_test.py b/autogpt_platform/backend/backend/util/cloud_storage_test.py
index 92bd4d2b71..3db4f87181 100644
--- a/autogpt_platform/backend/backend/util/cloud_storage_test.py
+++ b/autogpt_platform/backend/backend/util/cloud_storage_test.py
@@ -26,6 +26,14 @@ class TestCloudStorageHandler:
"""Create a test handler."""
return CloudStorageHandler(config)
+ @patch("backend.util.cloud_storage.Config")
+ def test_cloud_storage_config_uses_private_bucket(self, mock_config):
+ mock_config.return_value.resolved_private_user_data_bucket = "private-data"
+
+ config = CloudStorageConfig()
+
+ assert config.gcs_bucket_name == "private-data"
+
def test_parse_cloud_path_gcs(self, handler):
"""Test parsing GCS paths."""
provider, path = handler.parse_cloud_path("gcs://bucket/path/to/file.txt")
@@ -72,7 +80,10 @@ class TestCloudStorageHandler:
assert call_args[0][0] == "test-bucket" # bucket name
assert call_args[0][1].startswith("uploads/system/") # blob name
assert call_args[0][2] == content # file content
- assert "metadata" in call_args[1] # metadata argument
+ # Top level on purpose: see the comment in store_file.
+ upload_metadata = call_args.kwargs["metadata"]
+ assert upload_metadata["expires_at"]
+ assert "metadata" not in upload_metadata
@patch("backend.util.cloud_storage.async_gcs_storage.Storage")
@pytest.mark.asyncio
@@ -154,6 +165,22 @@ class TestCloudStorageHandler:
assert result == "https://signed-url.example.com"
mock_blob.generate_signed_url.assert_called_once()
+ @pytest.mark.asyncio
+ async def test_retrieve_rejects_a_different_bucket(self, handler):
+ with pytest.raises(PermissionError, match="configured private bucket"):
+ await handler.retrieve_file(
+ "gcs://other-bucket/uploads/users/user123/id/file.txt",
+ user_id="user123",
+ )
+
+ @pytest.mark.asyncio
+ async def test_signed_url_rejects_a_different_bucket(self, handler):
+ with pytest.raises(PermissionError, match="configured private bucket"):
+ await handler.generate_signed_url(
+ "gcs://other-bucket/uploads/users/user123/id/file.txt",
+ user_id="user123",
+ )
+
@pytest.mark.asyncio
async def test_unsupported_provider(self, handler):
"""Test unsupported provider error."""
diff --git a/autogpt_platform/backend/backend/util/settings.py b/autogpt_platform/backend/backend/util/settings.py
index 593ba36db1..adab5e13a4 100644
--- a/autogpt_platform/backend/backend/util/settings.py
+++ b/autogpt_platform/backend/backend/util/settings.py
@@ -1,9 +1,10 @@
+import functools
import json
import logging
import os
import re
from enum import Enum
-from typing import Any, Dict, Generic, List, Literal, Set, Tuple, Type, TypeVar
+from typing import Any, Dict, Generic, List, Literal, Self, Set, Tuple, Type, TypeVar
from pydantic import (
AliasChoices,
@@ -12,6 +13,7 @@ from pydantic import (
PrivateAttr,
ValidationInfo,
field_validator,
+ model_validator,
)
from pydantic_settings import (
BaseSettings,
@@ -24,6 +26,16 @@ from backend.util.data import get_data_path
logger = logging.getLogger(__name__)
+
+def _warn_single_bucket() -> None:
+ logger.warning(
+ "Private user data and public media share the legacy "
+ "MEDIA_GCS_BUCKET_NAME bucket. Configure PRIVATE_USER_DATA_BUCKET and "
+ "PUBLIC_SITE_MEDIA_BUCKET to keep user data out of the public bucket."
+ )
+
+
T = TypeVar("T", bound=BaseSettings)
_SERVICE_NAME = "MainProcess"
@@ -481,9 +493,57 @@ class Config(UpdateTrackingModel["Config"], BaseSettings):
media_gcs_bucket_name: str = Field(
default="",
- description="The name of the Google Cloud Storage bucket for media files",
+ description="Legacy single GCS bucket for public media and private user "
+ "data. Split deployments should configure public_site_media_bucket and "
+ "private_user_data_bucket.",
)
+ public_site_media_bucket: str = Field(
+ default="",
+ description="GCS bucket for approved marketplace media and OAuth app logos",
+ )
+
+ private_user_data_bucket: str = Field(
+ default="",
+ description="Private GCS bucket for user media, workspaces, transcripts, "
+ "and temporary uploads",
+ )
+
+ @property
+ def resolved_public_site_media_bucket(self) -> str:
+ return self.public_site_media_bucket or self.media_gcs_bucket_name
+
+ @property
+ def resolved_private_user_data_bucket(self) -> str:
+ return self.private_user_data_bucket or self.media_gcs_bucket_name
+
+ @model_validator(mode="after")
+ def validate_cloud_storage_bucket_separation(self) -> Self:
+ split_configured = bool(
+ self.public_site_media_bucket or self.private_user_data_bucket
+ )
+ if not split_configured:
+ if self.media_gcs_bucket_name:
+ _warn_single_bucket()
+ return self
+ if self.behave_as != BehaveAs.CLOUD:
+ return self
+ if not self.public_site_media_bucket or not self.private_user_data_bucket:
+ raise ValueError(
+ "Cloud deployments must configure both PUBLIC_SITE_MEDIA_BUCKET "
+ "and PRIVATE_USER_DATA_BUCKET"
+ )
+ if self.public_site_media_bucket == self.private_user_data_bucket:
+ raise ValueError(
+ "PUBLIC_SITE_MEDIA_BUCKET and PRIVATE_USER_DATA_BUCKET must be different"
+ )
+ if self.public_site_media_bucket == self.media_gcs_bucket_name:
+ raise ValueError(
+ "PUBLIC_SITE_MEDIA_BUCKET must not be the legacy "
+ "MEDIA_GCS_BUCKET_NAME bucket, which holds private user data"
+ )
+ return self
+
workspace_storage_dir: str = Field(
default="",
description="Local directory for workspace file storage when GCS is not configured. "
diff --git a/autogpt_platform/backend/backend/util/settings_storage_test.py b/autogpt_platform/backend/backend/util/settings_storage_test.py
new file mode 100644
index 0000000000..8c18ce57d5
--- /dev/null
+++ b/autogpt_platform/backend/backend/util/settings_storage_test.py
@@ -0,0 +1,115 @@
+import pytest
+
+import backend.util.settings as settings_module
+from backend.util.settings import Config
+
+
[email protected](autouse=True)
+def no_bucket_env(monkeypatch):
+ # Config reads the environment before keyword arguments, and backend.data.db
+ # copies .env into os.environ on import.
+ for name in (
+ "BEHAVE_AS",
+ "MEDIA_GCS_BUCKET_NAME",
+ "PUBLIC_SITE_MEDIA_BUCKET",
+ "PRIVATE_USER_DATA_BUCKET",
+ ):
+ monkeypatch.delenv(name, raising=False)
+
+
+def test_storage_bucket_settings_use_distinct_public_and_private_names():
+ config = Config(
+ _env_file=None,
+ PUBLIC_SITE_MEDIA_BUCKET="public-media",
+ PRIVATE_USER_DATA_BUCKET="private-data",
+ )
+
+ assert config.resolved_public_site_media_bucket == "public-media"
+ assert config.resolved_private_user_data_bucket == "private-data"
+
+
+def test_legacy_media_bucket_env_remains_compatible_for_self_hosting():
+ config = Config(
+ _env_file=None,
+ MEDIA_GCS_BUCKET_NAME="legacy-media",
+ )
+
+ assert config.resolved_public_site_media_bucket == "legacy-media"
+ assert config.resolved_private_user_data_bucket == "legacy-media"
+
+
+def test_public_media_env_takes_precedence_over_legacy_name():
+ config = Config(
+ _env_file=None,
+ PUBLIC_SITE_MEDIA_BUCKET="public-media",
+ PRIVATE_USER_DATA_BUCKET="private-data",
+ MEDIA_GCS_BUCKET_NAME="legacy-media",
+ )
+
+ assert config.resolved_public_site_media_bucket == "public-media"
+ assert config.resolved_private_user_data_bucket == "private-data"
+
+
+def test_each_explicit_bucket_keeps_the_other_legacy_fallback():
+ public_override = Config(
+ _env_file=None,
+ PUBLIC_SITE_MEDIA_BUCKET="public-media",
+ MEDIA_GCS_BUCKET_NAME="legacy-media",
+ )
+ private_override = Config(
+ _env_file=None,
+ PRIVATE_USER_DATA_BUCKET="private-data",
+ MEDIA_GCS_BUCKET_NAME="legacy-media",
+ )
+
+ assert public_override.resolved_public_site_media_bucket == "public-media"
+ assert public_override.resolved_private_user_data_bucket == "legacy-media"
+ assert private_override.resolved_public_site_media_bucket == "legacy-media"
+ assert private_override.resolved_private_user_data_bucket == "private-data"
+
+
+ "values",
+ [
+ {"PUBLIC_SITE_MEDIA_BUCKET": "public-media"},
+ {"PRIVATE_USER_DATA_BUCKET": "private-data"},
+ {
+ "PUBLIC_SITE_MEDIA_BUCKET": "same-bucket",
+ "PRIVATE_USER_DATA_BUCKET": "same-bucket",
+ },
+ {
+ "PUBLIC_SITE_MEDIA_BUCKET": "legacy-media",
+ "PRIVATE_USER_DATA_BUCKET": "private-data",
+ "MEDIA_GCS_BUCKET_NAME": "legacy-media",
+ },
+ ],
+)
+def test_cloud_split_configuration_fails_closed(values):
+ with pytest.raises(ValueError):
+ Config(_env_file=None, BEHAVE_AS="cloud", **values)
+
+
+def test_cloud_split_configuration_accepts_distinct_buckets():
+ config = Config(
+ _env_file=None,
+ BEHAVE_AS="cloud",
+ PUBLIC_SITE_MEDIA_BUCKET="public-media",
+ PRIVATE_USER_DATA_BUCKET="private-data",
+ MEDIA_GCS_BUCKET_NAME="legacy-media",
+ )
+
+ assert config.resolved_public_site_media_bucket == "public-media"
+ assert config.resolved_private_user_data_bucket == "private-data"
+
+
[email protected]("behave_as", ["cloud", "local"])
+def test_a_deployment_on_the_legacy_bucket_warns_once(caplog, behave_as):
+ settings_module._warn_single_bucket.cache_clear()
+
+ for _ in range(2):
+ Config(
+ _env_file=None, BEHAVE_AS=behave_as, MEDIA_GCS_BUCKET_NAME="legacy-media"
+ )
+
+ warnings = [r for r in caplog.records if "PRIVATE_USER_DATA_BUCKET" in r.message]
+ assert len(warnings) == 1
diff --git a/autogpt_platform/backend/backend/util/workspace_storage.py b/autogpt_platform/backend/backend/util/workspace_storage.py
index d821eb4866..ee3af757b6 100644
--- a/autogpt_platform/backend/backend/util/workspace_storage.py
+++ b/autogpt_platform/backend/backend/util/workspace_storage.py
@@ -179,9 +179,11 @@ class GCSWorkspaceStorage(WorkspaceStorageBackend):
blob_name,
content,
metadata={
- "uploaded_at": upload_time.isoformat(),
- "workspace_id": workspace_id,
- "file_id": file_id,
+ "metadata": {
+ "uploaded_at": upload_time.isoformat(),
+ "workspace_id": workspace_id,
+ "file_id": file_id,
+ }
},
)
@@ -189,21 +191,21 @@ class GCSWorkspaceStorage(WorkspaceStorageBackend):
async def retrieve(self, storage_path: str) -> bytes:
"""Retrieve file from GCS."""
- bucket_name, blob_name = parse_gcs_path(storage_path)
- return await download_with_fresh_session(bucket_name, blob_name)
+ blob_name = self._parse_storage_path(storage_path)
+ return await download_with_fresh_session(self.bucket_name, blob_name)
async def retrieve_partial(self, storage_path: str, max_bytes: int) -> bytes:
"""Retrieve the first ``max_bytes`` of a GCS file via a Range request."""
- bucket_name, blob_name = parse_gcs_path(storage_path)
- return await download_range(bucket_name, blob_name, max_bytes)
+ blob_name = self._parse_storage_path(storage_path)
+ return await download_range(self.bucket_name, blob_name, max_bytes)
async def delete(self, storage_path: str) -> None:
"""Delete file from GCS."""
- bucket_name, blob_name = parse_gcs_path(storage_path)
+ blob_name = self._parse_storage_path(storage_path)
client = await self._get_async_client()
try:
- await client.delete(bucket_name, blob_name)
+ await client.delete(self.bucket_name, blob_name)
except Exception as e:
if not is_not_found_error(e):
raise
@@ -217,7 +219,7 @@ class GCSWorkspaceStorage(WorkspaceStorageBackend):
Falls back to an API proxy endpoint if signed URL generation fails
(e.g., when running locally with user OAuth credentials).
"""
- bucket_name, blob_name = parse_gcs_path(storage_path)
+ blob_name = self._parse_storage_path(storage_path)
# Extract file_id from blob_name for fallback: workspaces/{workspace_id}/{file_id}/{filename}
blob_parts = blob_name.split("/")
@@ -227,7 +229,7 @@ class GCSWorkspaceStorage(WorkspaceStorageBackend):
try:
sync_client = self._get_sync_client()
return await generate_signed_url(
- sync_client, bucket_name, blob_name, expires_in
+ sync_client, self.bucket_name, blob_name, expires_in
)
except AttributeError as e:
# Signed URL generation requires service account with private key.
@@ -240,6 +242,12 @@ class GCSWorkspaceStorage(WorkspaceStorageBackend):
return f"/api/workspace/files/{file_id}/download"
raise
+ def _parse_storage_path(self, storage_path: str) -> str:
+ bucket_name, blob_name = parse_gcs_path(storage_path)
+ if bucket_name != self.bucket_name:
+ raise PermissionError("Access denied: not the configured private bucket")
+ return blob_name
+
class LocalWorkspaceStorage(WorkspaceStorageBackend):
"""Local filesystem implementation for workspace storage (self-hosted deployments)."""
@@ -406,7 +414,9 @@ async def get_workspace_storage() -> WorkspaceStorageBackend:
config = Config()
# --- Local storage (shared) ---
- if not config.media_gcs_bucket_name:
+ bucket_name = config.resolved_private_user_data_bucket
+
+ if not bucket_name:
if _local_storage is None:
storage_dir = (
config.workspace_storage_dir if config.workspace_storage_dir else None
@@ -418,11 +428,8 @@ async def get_workspace_storage() -> WorkspaceStorageBackend:
# --- GCS storage (per event loop) ---
loop_id = id(asyncio.get_running_loop())
if loop_id not in _gcs_storages:
- logger.info(
- f"Creating GCS workspace storage for loop {loop_id}: "
- f"{config.media_gcs_bucket_name}"
- )
- _gcs_storages[loop_id] = GCSWorkspaceStorage(config.media_gcs_bucket_name)
+ logger.info(f"Creating GCS workspace storage for loop {loop_id}: {bucket_name}")
+ _gcs_storages[loop_id] = GCSWorkspaceStorage(bucket_name)
return _gcs_storages[loop_id]
diff --git a/autogpt_platform/backend/backend/util/workspace_storage_test.py b/autogpt_platform/backend/backend/util/workspace_storage_test.py
index d41a20e674..877d3b2fdf 100644
--- a/autogpt_platform/backend/backend/util/workspace_storage_test.py
+++ b/autogpt_platform/backend/backend/util/workspace_storage_test.py
@@ -3,6 +3,7 @@ from unittest.mock import AsyncMock, MagicMock
import aiohttp
import pytest
+from backend.util import workspace_storage
from backend.util.gcs_utils_test import _URL_CONTAINING_404, _gcs_http_error
from backend.util.workspace_storage import GCSWorkspaceStorage, LocalWorkspaceStorage
@@ -49,6 +50,52 @@ async def test_gcs_retrieve_partial_delegates_to_download_range(mocker):
download_range.assert_awaited_once_with("my-bucket", "path/file.txt", 4)
+async def test_gcs_store_wraps_custom_metadata(mocker):
+ storage = GCSWorkspaceStorage(bucket_name="my-bucket")
+ client = MagicMock()
+ client.upload = AsyncMock()
+ mocker.patch.object(storage, "_get_async_client", AsyncMock(return_value=client))
+
+ await storage.store("workspace", "file", "data.txt", b"content")
+
+ custom_metadata = client.upload.await_args.kwargs["metadata"]["metadata"]
+ assert custom_metadata["uploaded_at"]
+ assert custom_metadata["workspace_id"] == "workspace"
+ assert custom_metadata["file_id"] == "file"
+
+
+async def test_workspace_storage_uses_private_bucket(mocker):
+ config = MagicMock()
+ config.resolved_private_user_data_bucket = "private-data"
+ mocker.patch("backend.util.workspace_storage.Config", return_value=config)
+ workspace_storage._gcs_storages.clear()
+
+ storage = await workspace_storage.get_workspace_storage()
+
+ assert isinstance(storage, GCSWorkspaceStorage)
+ assert storage.bucket_name == "private-data"
+ workspace_storage._gcs_storages.clear()
+
+
[email protected]("operation", ["retrieve", "partial", "delete", "url"])
+async def test_gcs_workspace_operations_reject_a_different_bucket(operation):
+ storage = GCSWorkspaceStorage(bucket_name="private-bucket")
+ path = "gcs://other-bucket/workspaces/ws/file/data.txt"
+
+ with pytest.raises(PermissionError, match="configured private bucket"):
+ if operation == "retrieve":
+ await storage.retrieve(path)
+ elif operation == "partial":
+ await storage.retrieve_partial(path, 10)
+ elif operation == "delete":
+ await storage.delete(path)
+ else:
+ await storage.get_download_url(path)
+
+
def _gcs_storage_whose_delete_raises(mocker, error: Exception) -> GCSWorkspaceStorage:
storage = GCSWorkspaceStorage(bucket_name="my-bucket")
client = MagicMock()
diff --git a/autogpt_platform/backend/scripts/backfill_private_media_urls.py b/autogpt_platform/backend/scripts/backfill_private_media_urls.py
new file mode 100644
index 0000000000..127f93b17a
--- /dev/null
+++ b/autogpt_platform/backend/scripts/backfill_private_media_urls.py
@@ -0,0 +1,266 @@
+"""Safely rewrite legacy private-bucket media URLs to authenticated API paths.
+
+The default mode is a read-only dry run. ``--apply`` executes the rewrites in
+short transactions of a few hundred rows and protects every row with
+compare-and-swap, so an interrupted run keeps its progress and a re-run picks
+up where it stopped. Only GCS URLs below ``users/<user>/(images|videos)/<file>``
+in the legacy bucket are eligible; references that are served publicly are
+held, so run ``publish_live_media.py --apply`` first. Output is aggregate-only
+and never contains URLs, object names, row IDs, or user IDs.
+
+Usage::
+
+ poetry run python scripts/backfill_private_media_urls.py
+ poetry run python scripts/backfill_private_media_urls.py --apply
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import os
+import sys
+from datetime import timedelta
+from pathlib import Path
+
+from prisma import Prisma, get_client
+
+if __package__:
+ from scripts.media_url_backfill import (
+ ApplyProgress,
+ BackfillReport,
+ Candidate,
+ Outcome,
+ Transaction,
+ print_report,
+ process_candidates,
+ resolve_source_bucket,
+ )
+ from scripts.media_url_backfill_queries import CREATOR_IS_PUBLIC, VERSION_IS_PUBLIC
+else:
+ from media_url_backfill import (
+ ApplyProgress,
+ BackfillReport,
+ Candidate,
+ Outcome,
+ Transaction,
+ print_report,
+ process_candidates,
+ resolve_source_bucket,
+ )
+ from media_url_backfill_queries import CREATOR_IS_PUBLIC, VERSION_IS_PUBLIC
+
+BATCH_TRANSACTION_TIMEOUT = timedelta(seconds=30)
+
+CANDIDATE_QUERY = f"""
+WITH candidates AS (
+ SELECT
+ 'Profile.avatarUrl' AS target,
+ p.id AS record_id,
+ p."userId" AS owner_user_id,
+ ARRAY[p."avatarUrl"]::text[] AS values,
+ false AS is_array,
+ CASE WHEN {CREATOR_IS_PUBLIC} THEN 'public' END AS hold_reason,
+ ARRAY[]::text[] AS co_owner_ids
+ FROM platform."Profile" AS p
+ WHERE p."avatarUrl" IS NOT NULL
+
+ UNION ALL
+ SELECT
+ 'Expert.avatarUrl', e.id, e."ownerUserId",
+ ARRAY[e."avatarUrl"]::text[], false,
+ CASE WHEN e."ownerUserId" IS NULL THEN 'ambiguous' END, ARRAY[]::text[]
+ FROM platform."Expert" AS e
+ WHERE e."avatarUrl" IS NOT NULL
+
+ UNION ALL
+ SELECT
+ 'LibraryAgent.imageUrl', la.id, la."userId",
+ ARRAY[la."imageUrl"]::text[], false, NULL,
+ ARRAY(
+ SELECT colleague."userId"
+ FROM platform."OrgMember" AS me
+ JOIN platform."OrgMember" AS colleague ON colleague."orgId" = me."orgId"
+ JOIN platform."Organization" AS org ON org.id = me."orgId"
+ WHERE me."userId" = la."userId"
+ AND me.status = 'ACTIVE'
+ AND colleague.status = 'ACTIVE'
+ AND org."deletedAt" IS NULL
+ )
+ FROM platform."LibraryAgent" AS la
+ WHERE la."imageUrl" IS NOT NULL
+
+ UNION ALL
+ SELECT
+ 'StoreListingVersion.imageUrls', slv.id, sl."owningUserId",
+ slv."imageUrls", true,
+ CASE WHEN {VERSION_IS_PUBLIC} THEN 'public' END,
+ ARRAY(
+ SELECT om."userId"
+ FROM platform."OrgMember" AS om
+ WHERE om."orgId" = sl."owningOrgId" AND om.status = 'ACTIVE'
+ )
+ FROM platform."StoreListingVersion" AS slv
+ JOIN platform."StoreListing" AS sl ON sl.id = slv."storeListingId"
+ WHERE cardinality(slv."imageUrls") > 0
+
+ UNION ALL
+ SELECT
+ 'StoreListingVersion.videoUrl', slv.id, sl."owningUserId",
+ ARRAY[slv."videoUrl"]::text[], false,
+ CASE WHEN {VERSION_IS_PUBLIC} THEN 'public' END,
+ ARRAY(
+ SELECT om."userId"
+ FROM platform."OrgMember" AS om
+ WHERE om."orgId" = sl."owningOrgId" AND om.status = 'ACTIVE'
+ )
+ FROM platform."StoreListingVersion" AS slv
+ JOIN platform."StoreListing" AS sl ON sl.id = slv."storeListingId"
+ WHERE slv."videoUrl" IS NOT NULL
+
+ UNION ALL
+ SELECT
+ 'StoreListingVersion.agentOutputDemoUrl', slv.id, sl."owningUserId",
+ ARRAY[slv."agentOutputDemoUrl"]::text[], false,
+ CASE WHEN {VERSION_IS_PUBLIC} THEN 'public' END,
+ ARRAY(
+ SELECT om."userId"
+ FROM platform."OrgMember" AS om
+ WHERE om."orgId" = sl."owningOrgId" AND om.status = 'ACTIVE'
+ )
+ FROM platform."StoreListingVersion" AS slv
+ JOIN platform."StoreListing" AS sl ON sl.id = slv."storeListingId"
+ WHERE slv."agentOutputDemoUrl" IS NOT NULL
+
+ UNION ALL
+ SELECT
+ 'Organization.avatarUrl', o.id, NULL,
+ ARRAY[o."avatarUrl"]::text[], false, NULL,
+ ARRAY(
+ SELECT om."userId"
+ FROM platform."OrgMember" AS om
+ WHERE om."orgId" = o.id AND om.status = 'ACTIVE'
+ )
+ FROM platform."Organization" AS o
+ WHERE o."avatarUrl" IS NOT NULL
+
+ UNION ALL
+ SELECT
+ 'OrganizationProfile.avatarUrl', op."organizationId", NULL,
+ ARRAY[op."avatarUrl"]::text[], false, NULL,
+ ARRAY(
+ SELECT om."userId"
+ FROM platform."OrgMember" AS om
+ WHERE om."orgId" = op."organizationId" AND om.status = 'ACTIVE'
+ )
+ FROM platform."OrganizationProfile" AS op
+ WHERE op."avatarUrl" IS NOT NULL
+)
+SELECT target, record_id, owner_user_id, values, is_array, hold_reason, co_owner_ids
+FROM candidates
+WHERE EXISTS (
+ SELECT 1 FROM unnest(values) AS candidate_url
+ WHERE position($1 IN candidate_url) > 0
+)
+"""
+
+
+async def main(*, apply: bool, bucket_override: str | None = None) -> int:
+ if not os.environ.get("DATABASE_URL"):
+ raise SystemExit("DATABASE_URL must be set")
+
+ backend_root = str(Path(__file__).resolve().parent.parent)
+ if backend_root not in sys.path:
+ sys.path.insert(0, backend_root)
+
+ from backend.data.db import connect, disconnect, transaction
+ from backend.util.settings import Settings
+
+ config = Settings().config
+ private_bucket = config.resolved_private_user_data_bucket
+ bucket = resolve_source_bucket(
+ private_bucket=private_bucket,
+ legacy_bucket=config.media_gcs_bucket_name,
+ override=bucket_override,
+ )
+ if bucket != private_bucket:
+ print(
+ "Warning: --bucket differs from PRIVATE_USER_DATA_BUCKET. Rewritten "
+ "references are served from PRIVATE_USER_DATA_BUCKET, so their "
+ "objects must already exist there under the same paths."
+ )
+ public_bucket = config.public_site_media_bucket
+ if public_bucket == bucket:
+ public_bucket = ""
+
+ def batch_transaction():
+ return transaction(timeout=BATCH_TRANSACTION_TIMEOUT)
+
+ progress = ApplyProgress()
+ await connect()
+ try:
+ report = await _run(
+ get_client(),
+ bucket,
+ public_bucket,
+ apply=apply,
+ transaction=batch_transaction,
+ progress=progress,
+ )
+ except Exception:
+ raise SystemExit(
+ "Backfill failed without logging identifiers after "
+ f"{progress.applied_rows} committed row updates; the batch in "
+ "progress was rolled back. Re-running is safe."
+ ) from None
+ finally:
+ await disconnect()
+
+ print_report(report, apply=apply)
+ return 2 if report.cas_conflicts or stranded_references(report) else 0
+
+
+def stranded_references(report: BackfillReport) -> int:
+ """References left on the legacy bucket, which stop loading once it is private."""
+ return sum(
+ count
+ for outcome, count in report.counts.items()
+ if outcome not in {Outcome.REWRITE, Outcome.ALREADY_PUBLIC}
+ )
+
+
+async def _run(
+ client: Prisma,
+ bucket: str,
+ public_bucket: str,
+ *,
+ apply: bool,
+ transaction: Transaction,
+ progress: ApplyProgress,
+) -> BackfillReport:
+ rows = await client.query_raw(CANDIDATE_QUERY, bucket)
+ candidates = [Candidate.model_validate(row) for row in rows]
+ return await process_candidates(
+ candidates,
+ bucket,
+ public_bucket=public_bucket or None,
+ apply=apply,
+ transaction=transaction,
+ progress=progress,
+ )
+
+
+if __name__ == "__main__":
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument(
+ "--apply",
+ action="store_true",
+ help="Apply owner-proven rewrites. Without this flag the script is read-only.",
+ )
+ parser.add_argument(
+ "--bucket",
+ help="Legacy bucket the stored URLs point at. Defaults to "
+ "PRIVATE_USER_DATA_BUCKET (or MEDIA_GCS_BUCKET_NAME).",
+ )
+ args = parser.parse_args()
+ raise SystemExit(asyncio.run(main(apply=args.apply, bucket_override=args.bucket)))
diff --git a/autogpt_platform/backend/scripts/download_transcripts.py b/autogpt_platform/backend/scripts/download_transcripts.py
index 57e6e2ceba..36dcf2c3aa 100644
--- a/autogpt_platform/backend/scripts/download_transcripts.py
+++ b/autogpt_platform/backend/scripts/download_transcripts.py
@@ -2,15 +2,15 @@
"""Download CoPilot transcripts from prod GCS and load into local dev environment.
Usage:
- # Step 1: Download from prod GCS (needs MEDIA_GCS_BUCKET_NAME + gcloud auth)
- MEDIA_GCS_BUCKET_NAME=<prod-bucket> USER_ID=<user-uuid> \
+ # Step 1: Download from prod GCS (needs PRIVATE_USER_DATA_BUCKET + gcloud auth)
+ PRIVATE_USER_DATA_BUCKET=<prod-bucket> USER_ID=<user-uuid> \
poetry run python scripts/download_transcripts.py download <session_id> ...
# Step 2: Load downloaded transcripts into local storage + DB
poetry run python scripts/download_transcripts.py load <session_id> ...
# Or do both in one step (if you have GCS access):
- MEDIA_GCS_BUCKET_NAME=<prod-bucket> USER_ID=<user-uuid> \
+ PRIVATE_USER_DATA_BUCKET=<prod-bucket> USER_ID=<user-uuid> \
poetry run python scripts/download_transcripts.py both <session_id> ...
The "download" step saves transcripts to transcripts/<session_id>.jsonl.
@@ -67,9 +67,14 @@ async def cmd_download(session_ids: list[str]) -> None:
print(" You can find it in Sentry breadcrumbs or the DB.")
sys.exit(1)
- bucket = os.environ.get("MEDIA_GCS_BUCKET_NAME", "")
+ from backend.util.settings import Config
+
+ bucket = Config().resolved_private_user_data_bucket
if not bucket:
- print("ERROR: Set MEDIA_GCS_BUCKET_NAME to the prod GCS bucket.")
+ print(
+ "ERROR: Set PRIVATE_USER_DATA_BUCKET (or legacy "
+ "MEDIA_GCS_BUCKET_NAME) to the prod GCS bucket."
+ )
sys.exit(1)
os.makedirs(TRANSCRIPTS_DIR, exist_ok=True)
diff --git a/autogpt_platform/backend/scripts/media_url_backfill.py b/autogpt_platform/backend/scripts/media_url_backfill.py
new file mode 100644
index 0000000000..767a2a7e9e
--- /dev/null
+++ b/autogpt_platform/backend/scripts/media_url_backfill.py
@@ -0,0 +1,431 @@
+from __future__ import annotations
+
+import re
+from collections import Counter
+from collections.abc import Callable, Mapping, Sequence
+from contextlib import AbstractAsyncContextManager
+from enum import StrEnum
+from pathlib import PurePosixPath
+from typing import LiteralString
+from urllib.parse import parse_qs, unquote, urlsplit
+
+from prisma import Prisma
+from pydantic import BaseModel, ConfigDict
+
+if __package__:
+ from scripts.media_url_backfill_queries import PATH_OWNER_TARGETS, UPDATE_QUERIES
+else:
+ from media_url_backfill_queries import PATH_OWNER_TARGETS, UPDATE_QUERIES
+
+PRIVATE_MEDIA_PREFIX = "/api/store/submissions/media"
+APPLY_BATCH_SIZE = 200
+
+Transaction = Callable[[], AbstractAsyncContextManager[Prisma]]
+
+# Same URL forms as backend.api.features.store.public_media (parity-tested).
+_GCS_URL_FORMS = (
+ re.compile(
+ r"https?://(?:www|storage)\.googleapis\.com/(?:download/)?storage/v1"
+ r"/b/(?P<bucket>[^/?#]+)/o/(?P<path>[^?#]+)(?:[?#].*)?",
+ re.IGNORECASE,
+ ),
+ re.compile(
+ r"https?://(?:storage\.googleapis\.com|storage\.cloud\.google\.com"
+ r"|commondatastorage\.googleapis\.com)/(?P<bucket>[^/?#]+)/(?P<path>[^?#]+)"
+ r"(?:[?#].*)?",
+ re.IGNORECASE,
+ ),
+ re.compile(
+ r"https?://(?P<bucket>[a-z0-9._-]+)\.storage\.googleapis\.com"
+ r"/(?P<path>[^?#]+)(?:[?#].*)?",
+ re.IGNORECASE,
+ ),
+ re.compile(r"gs://(?P<bucket>[^/?#]+)/(?P<path>[^?#]+)", re.IGNORECASE),
+)
+_SAFE_PATH_COMPONENT = re.compile(r"^[A-Za-z0-9_.-]+$")
+_CONTENT_TYPE_BY_SUFFIX = {
+ ".gif": "image",
+ ".jpeg": "image",
+ ".jpg": "image",
+ ".png": "image",
+ ".webp": "image",
+ ".mp4": "video",
+ ".webm": "video",
+}
+
+
+class MalformedPrivateMediaUrl(ValueError):
+ pass
+
+
+class Target(StrEnum):
+ PROFILE_AVATAR = "Profile.avatarUrl"
+ EXPERT_AVATAR = "Expert.avatarUrl"
+ LIBRARY_IMAGE = "LibraryAgent.imageUrl"
+ LISTING_IMAGES = "StoreListingVersion.imageUrls"
+ LISTING_VIDEO = "StoreListingVersion.videoUrl"
+ LISTING_DEMO = "StoreListingVersion.agentOutputDemoUrl"
+ ORGANIZATION_AVATAR = "Organization.avatarUrl"
+ ORGANIZATION_PROFILE_AVATAR = "OrganizationProfile.avatarUrl"
+ OAUTH_LOGO = "OAuthApplication.logoUrl"
+
+
+class HoldReason(StrEnum):
+ PUBLIC = "public"
+ AMBIGUOUS = "ambiguous"
+
+
+class Outcome(StrEnum):
+ REWRITE = "rewrite"
+ HOLD_PUBLIC = "hold_public"
+ HOLD_AMBIGUOUS = "hold_ambiguous"
+ HOLD_CROSS_USER = "hold_cross_user"
+ HOLD_MALFORMED = "hold_malformed"
+ UNRECOGNIZED = "unrecognized"
+ ALREADY_PUBLIC = "already_public"
+
+
+class ParsedPrivateMedia(BaseModel):
+ model_config = ConfigDict(frozen=True)
+
+ owner_user_id: str
+ media_type: str
+ filename: str
+ private_url: str
+
+
+class Candidate(BaseModel):
+ model_config = ConfigDict(frozen=True)
+
+ target: Target
+ record_id: str
+ owner_user_id: str | None
+ values: list[str]
+ is_array: bool
+ hold_reason: HoldReason | None = None
+ # Users whose uploads the row's readers can open through the private media
+ # endpoint: active members of a listing's or avatar's org, or the org
+ # colleagues of a library owner.
+ co_owner_ids: list[str] = []
+
+
+class Mutation(BaseModel):
+ model_config = ConfigDict(frozen=True)
+
+ target: Target
+ record_id: str
+ owner_user_id: str | None
+ old_values: list[str]
+ new_values: list[str]
+ is_array: bool
+
+
+class BackfillPlan(BaseModel):
+ mutations: list[Mutation]
+ counts: Counter[Outcome]
+ target_counts: dict[Target, Counter[Outcome]]
+
+
+class ApplyProgress(BaseModel):
+ applied_rows: int = 0
+ cas_conflicts: int = 0
+
+
+class BackfillReport(ApplyProgress):
+ counts: Counter[Outcome]
+ target_counts: dict[Target, Counter[Outcome]]
+ planned_rows: int
+
+ @classmethod
+ def from_plan(cls, plan: BackfillPlan) -> BackfillReport:
+ return cls(
+ counts=plan.counts,
+ target_counts=plan.target_counts,
+ planned_rows=len(plan.mutations),
+ )
+
+
+def resolve_source_bucket(
+ *, private_bucket: str, legacy_bucket: str, override: str | None
+) -> str:
+ """The bucket whose stored URLs are processed: the legacy bucket."""
+ if override:
+ return override
+ if legacy_bucket and private_bucket and legacy_bucket != private_bucket:
+ raise SystemExit(
+ "PRIVATE_USER_DATA_BUCKET differs from MEDIA_GCS_BUCKET_NAME, so stored "
+ "URLs may point at either bucket. Re-run with --bucket set to the "
+ "bucket the stored URLs point at."
+ )
+ if not private_bucket:
+ raise SystemExit(
+ "PRIVATE_USER_DATA_BUCKET or legacy MEDIA_GCS_BUCKET_NAME must be set"
+ )
+ return private_bucket
+
+
+def gcs_object_path(url: str, bucket: str) -> str | None:
+ """Object path of a GCS URL in ``bucket``, in any form the backend accepts."""
+ url = unquote(_unwrap(url))
+ for form in _GCS_URL_FORMS:
+ match = form.fullmatch(url)
+ if match:
+ return match["path"] if match["bucket"] == bucket else None
+ return None
+
+
+def private_api_object_path(url: str) -> str | None:
+ """Object path of a private media API URL, validated like the endpoint."""
+ parsed = urlsplit(_unwrap(url))
+ if parsed.scheme or parsed.netloc or parsed.query or parsed.fragment:
+ return None
+ path = unquote(parsed.path)
+ if not path.startswith(f"{PRIVATE_MEDIA_PREFIX}/"):
+ return None
+ parts = path.removeprefix(f"{PRIVATE_MEDIA_PREFIX}/").split("/")
+ if len(parts) != 3:
+ return None
+ owner_user_id, media_type, filename = parts
+ if (
+ media_type not in {"images", "videos"}
+ or not _safe_path_component(owner_user_id)
+ or not _safe_path_component(filename)
+ or PurePosixPath(filename).suffix.lower() not in _CONTENT_TYPE_BY_SUFFIX
+ ):
+ return None
+ return f"users/{owner_user_id}/{media_type}/{filename}"
+
+
+def parse_private_media_url(url: str, private_bucket: str) -> ParsedPrivateMedia | None:
+ object_path = gcs_object_path(url, private_bucket)
+ if object_path is None:
+ return None
+ return parse_private_media_path(object_path)
+
+
+def parse_private_media_path(object_path: str) -> ParsedPrivateMedia:
+ parts = object_path.split("/")
+ if len(parts) != 4 or parts[0] != "users":
+ raise MalformedPrivateMediaUrl("Managed object path is malformed")
+ _, owner_user_id, media_type, filename = parts
+ if not _safe_path_component(owner_user_id) or not _safe_path_component(filename):
+ raise MalformedPrivateMediaUrl("Managed object path is malformed")
+ if media_type not in {"images", "videos"}:
+ raise MalformedPrivateMediaUrl("Managed object has an invalid media type")
+
+ content_kind = _CONTENT_TYPE_BY_SUFFIX.get(PurePosixPath(filename).suffix.lower())
+ if content_kind != media_type.removesuffix("s"):
+ raise MalformedPrivateMediaUrl(
+ "Managed object filename does not match its type"
+ )
+ private_url = f"{PRIVATE_MEDIA_PREFIX}/{owner_user_id}/{media_type}/{filename}"
+ return ParsedPrivateMedia(
+ owner_user_id=owner_user_id,
+ media_type=media_type,
+ filename=filename,
+ private_url=private_url,
+ )
+
+
+def build_plan(
+ candidates: list[Candidate],
+ private_bucket: str,
+ public_bucket: str | None = None,
+) -> BackfillPlan:
+ counts: Counter[Outcome] = Counter()
+ target_counts: dict[Target, Counter[Outcome]] = {}
+ mutations: list[Mutation] = []
+ for candidate in candidates:
+ if not candidate.is_array and len(candidate.values) != 1:
+ raise ValueError("Scalar candidate must have exactly one value")
+ new_values = list(candidate.values)
+ changed = False
+ for index, value in enumerate(candidate.values):
+ outcome, replacement = _classify_reference(
+ candidate, value, private_bucket, public_bucket
+ )
+ if outcome is None:
+ continue
+ counts[outcome] += 1
+ target_counts.setdefault(candidate.target, Counter())[outcome] += 1
+ if replacement is not None and replacement != value:
+ new_values[index] = replacement
+ changed = True
+ if changed:
+ owner_user_id = None
+ if candidate.target.value not in PATH_OWNER_TARGETS:
+ if candidate.owner_user_id is None:
+ raise RuntimeError("A rewrite candidate has no proven owner")
+ owner_user_id = candidate.owner_user_id
+ mutations.append(
+ Mutation(
+ target=candidate.target,
+ record_id=candidate.record_id,
+ owner_user_id=owner_user_id,
+ old_values=candidate.values,
+ new_values=new_values,
+ is_array=candidate.is_array,
+ )
+ )
+ return BackfillPlan(
+ mutations=mutations,
+ counts=counts,
+ target_counts=target_counts,
+ )
+
+
+async def process_candidates(
+ candidates: list[Candidate],
+ private_bucket: str,
+ *,
+ public_bucket: str | None = None,
+ apply: bool,
+ transaction: Transaction | None = None,
+ progress: ApplyProgress | None = None,
+) -> BackfillReport:
+ plan = build_plan(candidates, private_bucket, public_bucket)
+ report = BackfillReport.from_plan(plan)
+ if not apply:
+ return report
+ if transaction is None:
+ raise ValueError("Applying a backfill needs a transaction factory")
+
+ await apply_mutations(
+ plan.mutations, UPDATE_QUERIES, transaction, progress or report
+ )
+ if progress is not None:
+ report.applied_rows = progress.applied_rows
+ report.cas_conflicts = progress.cas_conflicts
+ return report
+
+
+async def apply_mutations(
+ mutations: Sequence[Mutation],
+ queries: Mapping[str, LiteralString],
+ transaction: Transaction,
+ progress: ApplyProgress,
+ *,
+ batch_size: int = APPLY_BATCH_SIZE,
+) -> None:
+ """Apply compare-and-swap updates in short transactions of ``batch_size``.
+
+ Committed batches stay committed if a later one fails; re-running is safe
+ because every update matches only the value it was planned from.
+ """
+ for start in range(0, len(mutations), batch_size):
+ applied = conflicts = 0
+ async with transaction() as client:
+ for mutation in mutations[start : start + batch_size]:
+ changed = await client.execute_raw(
+ queries[mutation.target.value], *_update_args(mutation)
+ )
+ if changed == 1:
+ applied += 1
+ elif changed == 0:
+ conflicts += 1
+ else:
+ raise RuntimeError(
+ "A compare-and-swap update affected multiple rows"
+ )
+ progress.applied_rows += applied
+ progress.cas_conflicts += conflicts
+
+
+def print_report(report: BackfillReport, *, apply: bool) -> None:
+ print("Private media URL backfill summary (aggregate counts only):")
+ for outcome in Outcome:
+ print(f" {outcome.value}: {report.counts[outcome]}")
+ for target in Target:
+ target_counts = report.target_counts.get(target)
+ if not target_counts:
+ continue
+ summary = ", ".join(
+ f"{outcome.value}={target_counts[outcome]}"
+ for outcome in Outcome
+ if target_counts[outcome]
+ )
+ print(f" {target.value}: {summary}")
+ print(f" planned_row_updates: {report.planned_rows}")
+ if apply:
+ print(f" applied_row_updates: {report.applied_rows}")
+ print(f" compare_and_swap_conflicts: {report.cas_conflicts}")
+ else:
+ print("Dry run: no database writes were attempted. Re-run with --apply.")
+
+
+def _update_args(mutation: Mutation) -> list[str | list[str]]:
+ if mutation.is_array:
+ args: list[str | list[str]] = [
+ mutation.record_id,
+ mutation.new_values,
+ mutation.old_values,
+ ]
+ else:
+ args = [mutation.record_id, mutation.new_values[0], mutation.old_values[0]]
+ if mutation.owner_user_id is not None:
+ args.append(mutation.owner_user_id)
+ return args
+
+
+def _unwrap(url: str) -> str:
+ url = url.strip()
+ wrapped = urlsplit(url)
+ if wrapped.path == "/_next/image":
+ url = parse_qs(wrapped.query).get("url", [""])[0].strip()
+ return url
+
+
+def _safe_path_component(value: str) -> bool:
+ return (
+ value not in {".", ".."} and _SAFE_PATH_COMPONENT.fullmatch(value) is not None
+ )
+
+
+def _classify_reference(
+ candidate: Candidate,
+ value: str,
+ private_bucket: str,
+ public_bucket: str | None,
+) -> tuple[Outcome | None, str | None]:
+ if private_bucket not in value:
+ return None, None
+ if public_bucket and gcs_object_path(value, public_bucket) is not None:
+ return Outcome.ALREADY_PUBLIC, None
+ object_path = gcs_object_path(value, private_bucket)
+ if object_path is None:
+ if private_api_object_path(value) is not None:
+ return None, None
+ return Outcome.UNRECOGNIZED, None
+ if candidate.hold_reason is not None:
+ return _HOLD_OUTCOMES[candidate.hold_reason], None
+ try:
+ parsed = parse_private_media_path(object_path)
+ except MalformedPrivateMediaUrl:
+ return Outcome.HOLD_MALFORMED, None
+ if not _target_accepts_media_type(candidate.target, parsed.media_type):
+ return Outcome.HOLD_MALFORMED, None
+ if candidate.target.value in PATH_OWNER_TARGETS:
+ if parsed.owner_user_id not in candidate.co_owner_ids:
+ return Outcome.HOLD_CROSS_USER, None
+ else:
+ if candidate.owner_user_id is None:
+ return Outcome.HOLD_AMBIGUOUS, None
+ if parsed.owner_user_id != candidate.owner_user_id and (
+ parsed.owner_user_id not in candidate.co_owner_ids
+ ):
+ return Outcome.HOLD_CROSS_USER, None
+ return Outcome.REWRITE, parsed.private_url
+
+
+def _target_accepts_media_type(target: Target, media_type: str) -> bool:
+ if target == Target.LISTING_VIDEO:
+ return media_type == "videos"
+ if target == Target.LISTING_DEMO:
+ return True
+ return media_type == "images"
+
+
+_HOLD_OUTCOMES = {
+ HoldReason.PUBLIC: Outcome.HOLD_PUBLIC,
+ HoldReason.AMBIGUOUS: Outcome.HOLD_AMBIGUOUS,
+}
diff --git a/autogpt_platform/backend/scripts/media_url_backfill_queries.py b/autogpt_platform/backend/scripts/media_url_backfill_queries.py
new file mode 100644
index 0000000000..e64a3e90c8
--- /dev/null
+++ b/autogpt_platform/backend/scripts/media_url_backfill_queries.py
@@ -0,0 +1,160 @@
+from typing import LiteralString
+
+# A listing version is served on the public marketplace while it is approved
+# and neither it nor its listing is deleted: superseded approved versions are
+# re-activated when a newer version is rejected, and get_store_agent_details
+# serves an active version even when it is not available. Expects aliases
+# ``slv`` (StoreListingVersion) and ``sl`` (StoreListing); never NULL.
+VERSION_IS_PUBLIC: LiteralString = """(
+ slv."submissionStatus" = 'APPROVED'
+ AND NOT slv."isDeleted"
+ AND NOT sl."isDeleted"
+)"""
+
+# A creator profile is served publicly while its owner has a non-deleted
+# listing that has (or had) an approved version, or a live skill listing.
+# Expects alias ``p`` (Profile); EXISTS keeps it non-NULL when a listing has
+# no active version.
+CREATOR_IS_PUBLIC: LiteralString = """(
+ EXISTS (
+ SELECT 1
+ FROM platform."StoreListing" AS public_listing
+ WHERE public_listing."owningUserId" = p."userId"
+ AND NOT public_listing."isDeleted"
+ AND (
+ public_listing."hasApprovedVersion"
+ OR EXISTS (
+ SELECT 1
+ FROM platform."StoreListingVersion" AS public_version
+ WHERE public_version."storeListingId" = public_listing.id
+ AND public_version."submissionStatus" = 'APPROVED'
+ AND NOT public_version."isDeleted"
+ )
+ )
+ )
+ OR EXISTS (
+ SELECT 1
+ FROM platform."SkillListing" AS skill
+ JOIN platform."SkillListingVersion" AS skill_version
+ ON skill_version.id = skill."activeVersionId"
+ WHERE skill."owningUserId" = p."userId"
+ AND NOT skill."isDeleted"
+ AND skill."hasApprovedVersion"
+ AND skill_version."submissionStatus" = 'APPROVED'
+ AND skill_version."isAvailable"
+ AND NOT skill_version."isDeleted"
+ )
+)"""
+
+# Org avatars have no single owner. The private media endpoint serves a file to
+# members of an active org its uploader belongs to, so these rows are rewritten
+# only when the uploader is an active member of that org, and the UPDATE takes
+# no owner parameter.
+PATH_OWNER_TARGETS = frozenset(
+ {
+ "Organization.avatarUrl",
+ "OrganizationProfile.avatarUrl",
+ }
+)
+
+UPDATE_QUERIES: dict[str, LiteralString] = {
+ "Profile.avatarUrl": f"""
+ UPDATE platform."Profile" AS p
+ SET "avatarUrl" = $2
+ WHERE p.id = $1
+ AND p."avatarUrl" = $3
+ AND p."userId" = $4
+ AND NOT {CREATOR_IS_PUBLIC}
+ """,
+ "Expert.avatarUrl": """
+ UPDATE platform."Expert" AS e
+ SET "avatarUrl" = $2
+ WHERE e.id = $1
+ AND e."avatarUrl" = $3
+ AND e."ownerUserId" = $4
+ """,
+ "LibraryAgent.imageUrl": """
+ UPDATE platform."LibraryAgent" AS la
+ SET "imageUrl" = $2
+ WHERE la.id = $1
+ AND la."imageUrl" = $3
+ AND la."userId" = $4
+ """,
+ "StoreListingVersion.imageUrls": f"""
+ UPDATE platform."StoreListingVersion" AS slv
+ SET "imageUrls" = $2::text[]
+ FROM platform."StoreListing" AS sl
+ WHERE slv.id = $1
+ AND slv."imageUrls" = $3::text[]
+ AND sl.id = slv."storeListingId"
+ AND sl."owningUserId" = $4
+ AND NOT {VERSION_IS_PUBLIC}
+ """,
+ "StoreListingVersion.videoUrl": f"""
+ UPDATE platform."StoreListingVersion" AS slv
+ SET "videoUrl" = $2
+ FROM platform."StoreListing" AS sl
+ WHERE slv.id = $1
+ AND slv."videoUrl" = $3
+ AND sl.id = slv."storeListingId"
+ AND sl."owningUserId" = $4
+ AND NOT {VERSION_IS_PUBLIC}
+ """,
+ "StoreListingVersion.agentOutputDemoUrl": f"""
+ UPDATE platform."StoreListingVersion" AS slv
+ SET "agentOutputDemoUrl" = $2
+ FROM platform."StoreListing" AS sl
+ WHERE slv.id = $1
+ AND slv."agentOutputDemoUrl" = $3
+ AND sl.id = slv."storeListingId"
+ AND sl."owningUserId" = $4
+ AND NOT {VERSION_IS_PUBLIC}
+ """,
+ "Organization.avatarUrl": """
+ UPDATE platform."Organization" AS o
+ SET "avatarUrl" = $2
+ WHERE o.id = $1
+ AND o."avatarUrl" = $3
+ """,
+ "OrganizationProfile.avatarUrl": """
+ UPDATE platform."OrganizationProfile" AS op
+ SET "avatarUrl" = $2
+ WHERE op."organizationId" = $1
+ AND op."avatarUrl" = $3
+ """,
+}
+
+# Publishing only swaps a reference for the public copy of the same object,
+# so a compare-and-swap on the old value is the whole guard.
+PUBLISH_UPDATE_QUERIES: dict[str, LiteralString] = {
+ "StoreListingVersion.imageUrls": """
+ UPDATE platform."StoreListingVersion"
+ SET "imageUrls" = $2::text[]
+ WHERE id = $1 AND "imageUrls" = $3::text[]
+ """,
+ "StoreListingVersion.videoUrl": """
+ UPDATE platform."StoreListingVersion"
+ SET "videoUrl" = $2
+ WHERE id = $1 AND "videoUrl" = $3
+ """,
+ "StoreListingVersion.agentOutputDemoUrl": """
+ UPDATE platform."StoreListingVersion"
+ SET "agentOutputDemoUrl" = $2
+ WHERE id = $1 AND "agentOutputDemoUrl" = $3
+ """,
+ "Profile.avatarUrl": """
+ UPDATE platform."Profile"
+ SET "avatarUrl" = $2
+ WHERE id = $1 AND "avatarUrl" = $3
+ """,
+ "LibraryAgent.imageUrl": """
+ UPDATE platform."LibraryAgent"
+ SET "imageUrl" = $2
+ WHERE id = $1 AND "imageUrl" = $3
+ """,
+ "OAuthApplication.logoUrl": """
+ UPDATE platform."OAuthApplication"
+ SET "logoUrl" = $2
+ WHERE id = $1 AND "logoUrl" = $3
+ """,
+}
diff --git a/autogpt_platform/backend/scripts/media_url_publish.py b/autogpt_platform/backend/scripts/media_url_publish.py
new file mode 100644
index 0000000000..1a9d571759
--- /dev/null
+++ b/autogpt_platform/backend/scripts/media_url_publish.py
@@ -0,0 +1,313 @@
+from __future__ import annotations
+
+import asyncio
+import re
+from collections import Counter
+from collections.abc import Iterable
+from enum import StrEnum
+from typing import NamedTuple, Protocol
+from urllib.parse import quote
+
+from pydantic import BaseModel, ConfigDict
+
+if __package__:
+ from scripts.media_url_backfill import (
+ PRIVATE_MEDIA_PREFIX,
+ ApplyProgress,
+ Mutation,
+ Target,
+ gcs_object_path,
+ private_api_object_path,
+ )
+else:
+ from media_url_backfill import (
+ PRIVATE_MEDIA_PREFIX,
+ ApplyProgress,
+ Mutation,
+ Target,
+ gcs_object_path,
+ private_api_object_path,
+ )
+
+COPY_CONCURRENCY = 8
+
+_MEDIA_PATH = re.compile(
+ r"users/(?P<owner>[^/]+)/(?:images|videos)/(?P<filename>[^/]+)"
+)
+_LOGO_PATH = re.compile(r"oauth-apps/(?P<app_id>[^/]+)/logo/(?P<filename>[^/]+)")
+
+
+class PublishOutcome(StrEnum):
+ PUBLISH = "publish"
+ ALREADY_PUBLIC = "already_public"
+ COPY_FAILED = "copy_failed"
+ SKIP_FOREIGN_OWNER = "skip_foreign_owner"
+ SKIP_MALFORMED = "skip_malformed"
+ SKIP_UNRECOGNIZED = "skip_unrecognized"
+
+
+class Buckets(BaseModel):
+ model_config = ConfigDict(frozen=True)
+
+ source: str
+ private: str
+ public: str
+
+
+class SourceObject(NamedTuple):
+ bucket: str
+ path: str
+
+
+class PublishCandidate(BaseModel):
+ model_config = ConfigDict(frozen=True)
+
+ target: Target
+ record_id: str
+ values: list[str]
+ is_array: bool
+ owner_ids: list[str] = []
+
+
+class PlannedReference(BaseModel):
+ model_config = ConfigDict(frozen=True)
+
+ candidate: PublishCandidate
+ index: int
+ source: SourceObject
+
+
+class PublishPlan(BaseModel):
+ references: list[PlannedReference]
+ counts: dict[Target, Counter[PublishOutcome]]
+ already_public_image_paths: set[str]
+
+ @property
+ def objects(self) -> set[SourceObject]:
+ return {reference.source for reference in self.references}
+
+
+class PublishReport(ApplyProgress):
+ counts: dict[Target, Counter[PublishOutcome]]
+ planned_objects: int
+ copied_objects: int = 0
+ planned_rows: int = 0
+
+
+class ObjectCopier(Protocol):
+ async def copy(
+ self,
+ bucket: str,
+ object_name: str,
+ destination_bucket: str,
+ *,
+ new_name: str | None = None,
+ ) -> object: ...
+
+
+def public_url(public_bucket: str, path: str) -> str:
+ return f"https://storage.googleapis.com/{public_bucket}/{quote(path, safe='/')}"
+
+
+def locate_object(value: str, buckets: Buckets) -> tuple[str, str] | None:
+ """The bucket and object path a reference to private data points at."""
+ for bucket in dict.fromkeys((buckets.source, buckets.private)):
+ if (path := gcs_object_path(value, bucket)) is not None:
+ return bucket, path
+ if (path := private_api_object_path(value)) is not None:
+ return buckets.private, path
+ return None
+
+
+def plan_publication(
+ candidates: Iterable[PublishCandidate], buckets: Buckets
+) -> PublishPlan:
+ references: list[PlannedReference] = []
+ counts: dict[Target, Counter[PublishOutcome]] = {}
+ already_public_image_paths: set[str] = set()
+ for candidate in candidates:
+ if not candidate.is_array and len(candidate.values) != 1:
+ raise ValueError("Scalar candidate must have exactly one value")
+ for index, value in enumerate(candidate.values):
+ if (public_path := gcs_object_path(value, buckets.public)) is not None:
+ _count(counts, candidate.target, PublishOutcome.ALREADY_PUBLIC)
+ if candidate.target == Target.LISTING_IMAGES:
+ already_public_image_paths.add(public_path)
+ continue
+ located = locate_object(value, buckets)
+ if located is None:
+ if _mentions_private_data(value, buckets):
+ _count(counts, candidate.target, PublishOutcome.SKIP_UNRECOGNIZED)
+ continue
+ bucket, path = located
+ if skip := _check_path(candidate, path):
+ _count(counts, candidate.target, skip)
+ continue
+ references.append(
+ PlannedReference(
+ candidate=candidate,
+ index=index,
+ source=SourceObject(bucket=bucket, path=path),
+ )
+ )
+ return PublishPlan(
+ references=references,
+ counts=counts,
+ already_public_image_paths=already_public_image_paths,
+ )
+
+
+async def copy_objects(
+ objects: Iterable[SourceObject], public_bucket: str, copier: ObjectCopier
+) -> set[SourceObject]:
+ """Copy objects to the public bucket under the same path; return successes."""
+ semaphore = asyncio.Semaphore(COPY_CONCURRENCY)
+
+ async def copy_one(source: SourceObject) -> SourceObject | None:
+ async with semaphore:
+ try:
+ await copier.copy(
+ source.bucket, source.path, public_bucket, new_name=source.path
+ )
+ except Exception:
+ return None
+ return source
+
+ results = await asyncio.gather(*(copy_one(source) for source in set(objects)))
+ return {source for source in results if source is not None}
+
+
+def build_mutations(
+ plan: PublishPlan, copied: set[SourceObject], public_bucket: str
+) -> tuple[list[Mutation], dict[Target, Counter[PublishOutcome]]]:
+ counts = {target: Counter(counter) for target, counter in plan.counts.items()}
+ new_values: dict[tuple[Target, str], list[str]] = {}
+ candidates: dict[tuple[Target, str], PublishCandidate] = {}
+ for reference in plan.references:
+ candidate = reference.candidate
+ if reference.source not in copied:
+ _count(counts, candidate.target, PublishOutcome.COPY_FAILED)
+ continue
+ _count(counts, candidate.target, PublishOutcome.PUBLISH)
+ key = (candidate.target, candidate.record_id)
+ candidates[key] = candidate
+ values = new_values.setdefault(key, list(candidate.values))
+ values[reference.index] = public_url(public_bucket, reference.source.path)
+ mutations = [
+ Mutation(
+ target=candidate.target,
+ record_id=candidate.record_id,
+ owner_user_id=None,
+ old_values=candidate.values,
+ new_values=new_values[key],
+ is_array=candidate.is_array,
+ )
+ for key, candidate in candidates.items()
+ ]
+ return mutations, counts
+
+
+def published_image_paths(plan: PublishPlan, copied: set[SourceObject]) -> set[str]:
+ """Listing image objects that are public once this run's copies are done."""
+ return plan.already_public_image_paths | {
+ reference.source.path
+ for reference in plan.references
+ if reference.candidate.target == Target.LISTING_IMAGES
+ and reference.source in copied
+ }
+
+
+def library_mutations(
+ rows: Iterable[tuple[str, str]],
+ image_paths: set[str],
+ buckets: Buckets,
+ counts: dict[Target, Counter[PublishOutcome]],
+) -> list[Mutation]:
+ """Point library copies of a published listing image at its public copy."""
+ mutations: list[Mutation] = []
+ for record_id, value in rows:
+ located = locate_object(value, buckets)
+ if located is None or located[1] not in image_paths:
+ continue
+ _count(counts, Target.LIBRARY_IMAGE, PublishOutcome.PUBLISH)
+ mutations.append(
+ Mutation(
+ target=Target.LIBRARY_IMAGE,
+ record_id=record_id,
+ owner_user_id=None,
+ old_values=[value],
+ new_values=[public_url(buckets.public, located[1])],
+ is_array=False,
+ )
+ )
+ return mutations
+
+
+def print_report(report: PublishReport, *, apply: bool) -> None:
+ print("Live media publication summary (aggregate counts only):")
+ totals: Counter[PublishOutcome] = Counter()
+ for target_counts in report.counts.values():
+ totals.update(target_counts)
+ for outcome in PublishOutcome:
+ print(f" {outcome.value}: {totals[outcome]}")
+ for target in Target:
+ target_counts = report.counts.get(target)
+ if not target_counts:
+ continue
+ summary = ", ".join(
+ f"{outcome.value}={target_counts[outcome]}"
+ for outcome in PublishOutcome
+ if target_counts[outcome]
+ )
+ print(f" {target.value}: {summary}")
+ print(f" planned_object_copies: {report.planned_objects}")
+ print(f" planned_row_updates: {report.planned_rows}")
+ if apply:
+ print(f" copied_objects: {report.copied_objects}")
+ print(f" applied_row_updates: {report.applied_rows}")
+ print(f" compare_and_swap_conflicts: {report.cas_conflicts}")
+ else:
+ print(
+ "Dry run: no objects were copied and no database writes were "
+ "attempted. Re-run with --apply."
+ )
+
+
+def _check_path(candidate: PublishCandidate, path: str) -> PublishOutcome | None:
+ if candidate.target == Target.OAUTH_LOGO:
+ match = _LOGO_PATH.fullmatch(path)
+ if not match or not _is_segment(match["filename"]):
+ return PublishOutcome.SKIP_MALFORMED
+ if match["app_id"] != candidate.record_id:
+ return PublishOutcome.SKIP_FOREIGN_OWNER
+ return None
+ match = _MEDIA_PATH.fullmatch(path)
+ if (
+ not match
+ or not _is_segment(match["owner"])
+ or not _is_segment(match["filename"])
+ ):
+ return PublishOutcome.SKIP_MALFORMED
+ if match["owner"] not in candidate.owner_ids:
+ return PublishOutcome.SKIP_FOREIGN_OWNER
+ return None
+
+
+def _is_segment(value: str) -> bool:
+ return value not in {".", ".."}
+
+
+def _mentions_private_data(value: str, buckets: Buckets) -> bool:
+ return (
+ buckets.source in value
+ or buckets.private in value
+ or PRIVATE_MEDIA_PREFIX in value
+ )
+
+
+def _count(
+ counts: dict[Target, Counter[PublishOutcome]],
+ target: Target,
+ outcome: PublishOutcome,
+) -> None:
+ counts.setdefault(target, Counter())[outcome] += 1
diff --git a/autogpt_platform/backend/scripts/publish_live_media.py b/autogpt_platform/backend/scripts/publish_live_media.py
new file mode 100644
index 0000000000..b97fa1efec
--- /dev/null
+++ b/autogpt_platform/backend/scripts/publish_live_media.py
@@ -0,0 +1,316 @@
+"""Publish media that the public site already serves from the legacy bucket.
+
+Before public access on the legacy bucket is revoked, every reference the
+public site serves must point at the public site media bucket. This copies
+each such object from the private (legacy) bucket to the public bucket under
+the same object path and rewrites the reference with compare-and-swap:
+
+- media of every approved, non-deleted version on a non-deleted listing;
+- avatars of creators with such a listing or a live skill listing;
+- library copies of a published listing image;
+- OAuth application logos stored under ``oauth-apps/<app id>/logo/``.
+
+Only ``users/<owner>/(images|videos)/<file>`` objects owned by the row's owner
+(or, for listing media, an active member of the listing's organization) and
+logos under the app's own prefix are copied. The default mode is a read-only
+dry run; ``--apply`` copies and writes in short transactions. Already-public
+references are skipped, so the script is safe to re-run and repairs a failed
+approval-time copy. Output is aggregate-only and never contains URLs, object
+names, row IDs, or user IDs.
+
+Usage::
+
+ poetry run python scripts/publish_live_media.py
+ poetry run python scripts/publish_live_media.py --apply
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import os
+import sys
+from collections.abc import Callable
+from contextlib import AbstractAsyncContextManager
+from datetime import timedelta
+from pathlib import Path
+
+from prisma import Prisma, get_client
+
+if __package__:
+ from scripts.media_url_backfill import (
+ PRIVATE_MEDIA_PREFIX,
+ ApplyProgress,
+ Target,
+ Transaction,
+ apply_mutations,
+ resolve_source_bucket,
+ )
+ from scripts.media_url_backfill_queries import (
+ CREATOR_IS_PUBLIC,
+ PUBLISH_UPDATE_QUERIES,
+ VERSION_IS_PUBLIC,
+ )
+ from scripts.media_url_publish import (
+ Buckets,
+ ObjectCopier,
+ PublishCandidate,
+ PublishOutcome,
+ PublishReport,
+ build_mutations,
+ copy_objects,
+ library_mutations,
+ plan_publication,
+ print_report,
+ published_image_paths,
+ )
+else:
+ from media_url_backfill import (
+ PRIVATE_MEDIA_PREFIX,
+ ApplyProgress,
+ Target,
+ Transaction,
+ apply_mutations,
+ resolve_source_bucket,
+ )
+ from media_url_backfill_queries import (
+ CREATOR_IS_PUBLIC,
+ PUBLISH_UPDATE_QUERIES,
+ VERSION_IS_PUBLIC,
+ )
+ from media_url_publish import (
+ Buckets,
+ ObjectCopier,
+ PublishCandidate,
+ PublishOutcome,
+ PublishReport,
+ build_mutations,
+ copy_objects,
+ library_mutations,
+ plan_publication,
+ print_report,
+ published_image_paths,
+ )
+
+BATCH_TRANSACTION_TIMEOUT = timedelta(seconds=30)
+
+BLOCKING_OUTCOMES = (
+ PublishOutcome.COPY_FAILED,
+ PublishOutcome.SKIP_FOREIGN_OWNER,
+ PublishOutcome.SKIP_MALFORMED,
+ PublishOutcome.SKIP_UNRECOGNIZED,
+)
+
+CopierFactory = Callable[[], AbstractAsyncContextManager[ObjectCopier]]
+
+LISTING_QUERY = f"""
+SELECT
+ slv.id AS record_id,
+ slv."imageUrls" AS image_urls,
+ slv."videoUrl" AS video_url,
+ slv."agentOutputDemoUrl" AS demo_url,
+ ARRAY[sl."owningUserId"] || ARRAY(
+ SELECT om."userId"
+ FROM platform."OrgMember" AS om
+ WHERE om."orgId" = sl."owningOrgId"
+ AND om.status = 'ACTIVE'
+ ) AS owner_ids
+FROM platform."StoreListingVersion" AS slv
+JOIN platform."StoreListing" AS sl ON sl.id = slv."storeListingId"
+WHERE {VERSION_IS_PUBLIC}
+"""
+
+PROFILE_QUERY = f"""
+SELECT p.id AS record_id, p."userId" AS owner_id, p."avatarUrl" AS value
+FROM platform."Profile" AS p
+WHERE p."avatarUrl" IS NOT NULL
+ AND {CREATOR_IS_PUBLIC}
+"""
+
+OAUTH_LOGO_QUERY = """
+SELECT oa.id AS record_id, oa."logoUrl" AS value
+FROM platform."OAuthApplication" AS oa
+WHERE oa."logoUrl" IS NOT NULL
+"""
+
+LIBRARY_QUERY = """
+SELECT la.id AS record_id, la."imageUrl" AS value
+FROM platform."LibraryAgent" AS la
+WHERE la."imageUrl" IS NOT NULL
+ AND EXISTS (
+ SELECT 1 FROM unnest($1::text[]) AS needle
+ WHERE position(needle IN la."imageUrl") > 0
+ )
+"""
+
+
+async def main(*, apply: bool, bucket_override: str | None = None) -> int:
+ if not os.environ.get("DATABASE_URL"):
+ raise SystemExit("DATABASE_URL must be set")
+
+ backend_root = str(Path(__file__).resolve().parent.parent)
+ if backend_root not in sys.path:
+ sys.path.insert(0, backend_root)
+
+ from gcloud.aio import storage as async_storage
+
+ from backend.data.db import connect, disconnect, transaction
+ from backend.util.settings import Settings
+
+ config = Settings().config
+ private_bucket = config.resolved_private_user_data_bucket
+ buckets = Buckets(
+ source=resolve_source_bucket(
+ private_bucket=private_bucket,
+ legacy_bucket=config.media_gcs_bucket_name,
+ override=bucket_override,
+ ),
+ private=private_bucket,
+ public=config.public_site_media_bucket,
+ )
+ if not buckets.public:
+ raise SystemExit("PUBLIC_SITE_MEDIA_BUCKET must be set")
+ if buckets.public in {buckets.source, buckets.private}:
+ raise SystemExit(
+ "PUBLIC_SITE_MEDIA_BUCKET must differ from the private and legacy buckets"
+ )
+
+ def batch_transaction():
+ return transaction(timeout=BATCH_TRANSACTION_TIMEOUT)
+
+ progress = ApplyProgress()
+ await connect()
+ try:
+ report = await _run(
+ get_client(),
+ buckets,
+ apply=apply,
+ transaction=batch_transaction,
+ copier_factory=async_storage.Storage,
+ progress=progress,
+ )
+ except Exception:
+ raise SystemExit(
+ "Publication failed without logging identifiers after "
+ f"{progress.applied_rows} committed row updates; the batch in "
+ "progress was rolled back. Re-running is safe."
+ ) from None
+ finally:
+ await disconnect()
+
+ print_report(report, apply=apply)
+ return 2 if report.cas_conflicts or unpublished_references(report) else 0
+
+
+def unpublished_references(report: PublishReport) -> int:
+ """Live references that would break once the legacy bucket goes private."""
+ return sum(
+ counts[outcome]
+ for counts in report.counts.values()
+ for outcome in BLOCKING_OUTCOMES
+ )
+
+
+async def _run(
+ client: Prisma,
+ buckets: Buckets,
+ *,
+ apply: bool,
+ transaction: Transaction,
+ copier_factory: CopierFactory,
+ progress: ApplyProgress,
+) -> PublishReport:
+ plan = plan_publication(await _load_candidates(client), buckets)
+ objects = plan.objects
+ if apply and objects:
+ async with copier_factory() as copier:
+ copied = await copy_objects(objects, buckets.public, copier)
+ else:
+ copied = objects
+ mutations, counts = build_mutations(plan, copied, buckets.public)
+
+ library_rows = await client.query_raw(
+ LIBRARY_QUERY, [buckets.source, buckets.private, PRIVATE_MEDIA_PREFIX]
+ )
+ mutations += library_mutations(
+ ((row["record_id"], row["value"]) for row in library_rows),
+ published_image_paths(plan, copied),
+ buckets,
+ counts,
+ )
+
+ report = PublishReport(
+ counts=counts,
+ planned_objects=len(objects),
+ copied_objects=len(copied) if apply else 0,
+ planned_rows=len(mutations),
+ )
+ if apply:
+ await apply_mutations(mutations, PUBLISH_UPDATE_QUERIES, transaction, progress)
+ report.applied_rows = progress.applied_rows
+ report.cas_conflicts = progress.cas_conflicts
+ return report
+
+
+async def _load_candidates(client: Prisma) -> list[PublishCandidate]:
+ candidates: list[PublishCandidate] = []
+ for row in await client.query_raw(LISTING_QUERY):
+ common = {"record_id": row["record_id"], "owner_ids": row["owner_ids"]}
+ if row["image_urls"]:
+ candidates.append(
+ PublishCandidate(
+ target=Target.LISTING_IMAGES,
+ values=row["image_urls"],
+ is_array=True,
+ **common,
+ )
+ )
+ for target, column in (
+ (Target.LISTING_VIDEO, "video_url"),
+ (Target.LISTING_DEMO, "demo_url"),
+ ):
+ if row[column]:
+ candidates.append(
+ PublishCandidate(
+ target=target, values=[row[column]], is_array=False, **common
+ )
+ )
+ for row in await client.query_raw(PROFILE_QUERY):
+ candidates.append(
+ PublishCandidate(
+ target=Target.PROFILE_AVATAR,
+ record_id=row["record_id"],
+ values=[row["value"]],
+ is_array=False,
+ owner_ids=[row["owner_id"]],
+ )
+ )
+ for row in await client.query_raw(OAUTH_LOGO_QUERY):
+ candidates.append(
+ PublishCandidate(
+ target=Target.OAUTH_LOGO,
+ record_id=row["record_id"],
+ values=[row["value"]],
+ is_array=False,
+ )
+ )
+ return candidates
+
+
+if __name__ == "__main__":
+ parser = argparse.ArgumentParser(
+ description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
+ )
+ parser.add_argument(
+ "--apply",
+ action="store_true",
+ help="Copy objects and rewrite references. Without this flag the script "
+ "is read-only.",
+ )
+ parser.add_argument(
+ "--bucket",
+ help="Legacy bucket the stored URLs point at. Defaults to "
+ "PRIVATE_USER_DATA_BUCKET (or MEDIA_GCS_BUCKET_NAME).",
+ )
+ args = parser.parse_args()
+ raise SystemExit(asyncio.run(main(apply=args.apply, bucket_override=args.bucket)))
diff --git a/autogpt_platform/backend/scripts/test_backfill_private_media_urls.py b/autogpt_platform/backend/scripts/test_backfill_private_media_urls.py
new file mode 100644
index 0000000000..292dfed171
--- /dev/null
+++ b/autogpt_platform/backend/scripts/test_backfill_private_media_urls.py
@@ -0,0 +1,672 @@
+from collections import Counter
+from contextlib import asynccontextmanager
+from unittest.mock import AsyncMock
+
+import pytest
+
+import scripts.backfill_private_media_urls as backfill_cli
+import scripts.media_url_backfill as backfill
+import scripts.media_url_backfill_queries as queries
+
+PRIVATE_BUCKET = "private-media"
+
+
+ "url",
+ [
+ "https://storage.googleapis.com/private-media/users/owner/images/photo.jpeg",
+ "https://private-media.storage.googleapis.com/users/owner/images/photo.jpeg",
+ "gs://private-media/users/owner/images/photo.jpeg",
+ "https://storage.cloud.google.com/private-media/users/owner/images/photo.jpeg",
+ "https://commondatastorage.googleapis.com/private-media/"
+ "users/owner/images/photo.jpeg",
+ "https://storage.googleapis.com/storage/v1/b/private-media/o/"
+ "users%2Fowner%2Fimages%2Fphoto.jpeg?alt=media",
+ "https://www.googleapis.com/download/storage/v1/b/private-media/o/"
+ "users%2Fowner%2Fimages%2Fphoto.jpeg",
+ " https://storage.googleapis.com/private-media/users/owner/images/photo.jpeg ",
+ "https://storage.googleapis.com/private-media/users/owner/images/photo.jpeg?v=2",
+ "/_next/image?url=https%3A%2F%2Fstorage.googleapis.com%2Fprivate-media"
+ "%2Fusers%2Fowner%2Fimages%2Fphoto.jpeg&w=640&q=75",
+ ],
+)
+def test_parse_accepts_every_backend_gcs_url_form(url: str):
+ parsed = backfill.parse_private_media_url(url, PRIVATE_BUCKET)
+
+ assert parsed is not None
+ assert parsed.owner_user_id == "owner"
+ assert parsed.media_type == "images"
+ assert parsed.filename == "photo.jpeg"
+ assert parsed.private_url == (
+ "/api/store/submissions/media/owner/images/photo.jpeg"
+ )
+
+
+ "url",
+ [
+ "https://storage.googleapis.com/other/users/owner/images/photo.jpeg",
+ "/api/store/submissions/media/owner/images/photo.jpeg",
+ "https://example.test/private-media/users/owner/images/photo.jpeg",
+ "https://storage.googleapis.com/private-media-public/users/o/images/a.png",
+ ],
+)
+def test_parse_ignores_urls_not_raw_private_bucket_urls(url: str):
+ assert backfill.parse_private_media_url(url, PRIVATE_BUCKET) is None
+
+
+ "url",
+ [
+ "https://storage.googleapis.com/private-media/users/owner/images/a/b.jpeg",
+ "https://storage.googleapis.com/private-media/users/owner/images/%2e%2e",
+ "https://storage.googleapis.com/private-media/users/owner/videos/photo.jpeg",
+ "https://storage.googleapis.com/private-media/users/owner/images/my photo.png",
+ "https://storage.googleapis.com/private-media/oauth-apps/app/logo/logo.png",
+ ],
+)
+def test_parse_holds_malformed_managed_urls(url: str):
+ with pytest.raises(backfill.MalformedPrivateMediaUrl):
+ backfill.parse_private_media_url(url, PRIVATE_BUCKET)
+
+
+ "url",
+ [
+ "https://storage.googleapis.com/private-media/users/owner/images/photo.jpeg",
+ "https://storage.googleapis.com/other/users/owner/images/photo.jpeg",
+ "https://private-media.storage.googleapis.com/users/owner/videos/clip.mp4",
+ "https://storage.cloud.google.com/private-media/users/o/images/a.png#x",
+ "https://commondatastorage.googleapis.com/private-media/users/o/images/a.png",
+ "https://storage.googleapis.com/storage/v1/b/private-media/o/"
+ "users%2Fo%2Fimages%2Fa.png?alt=media",
+ "https://www.googleapis.com/download/storage/v1/b/private-media/o/"
+ "users%2Fo%2Fimages%2Fa.png",
+ "gs://private-media/users/owner/images/photo.jpeg",
+ " gs://private-media/users/owner/images/photo.jpeg\n",
+ "/_next/image?url=https%3A%2F%2Fstorage.googleapis.com%2Fprivate-media"
+ "%2Fusers%2Fo%2Fimages%2Fa.png&w=640",
+ "https://platform.example/_next/image?url=gs%3A%2F%2Fprivate-media"
+ "%2Fusers%2Fo%2Fimages%2Fa.png",
+ "/api/store/submissions/media/owner/images/photo.jpeg",
+ "/api/store/submissions/media/owner/images/photo.txt",
+ "/api/store/submissions/media/owner/images/a/b.png",
+ "/_next/image?url=%2Fapi%2Fstore%2Fsubmissions%2Fmedia%2Fo%2Fimages%2Fa.png",
+ "https://storage.googleapis.com/private-media/users/o/images/legacy name.png",
+ "https://example.test/private-media/users/owner/images/photo.jpeg",
+ "not a url",
+ ],
+)
+def test_url_parser_matches_backend_parser(url: str):
+ from backend.api.features.store import public_media
+
+ ours = backfill.private_api_object_path(url) or backfill.gcs_object_path(
+ url, PRIVATE_BUCKET
+ )
+
+ assert ours == public_media.object_path_from_url(url, PRIVATE_BUCKET)
+
+
+def test_scalar_rewrite_requires_owner_proof():
+ owned = _candidate(
+ target=backfill.Target.PROFILE_AVATAR,
+ owner="owner",
+ values=[_raw_url("owner", "images", "photo.jpeg")],
+ )
+ cross_user = _candidate(
+ target=backfill.Target.EXPERT_AVATAR,
+ owner="owner",
+ values=[_raw_url("someone-else", "images", "photo.jpeg")],
+ )
+
+ plan = backfill.build_plan([owned, cross_user], PRIVATE_BUCKET)
+
+ assert plan.counts == Counter(
+ {
+ backfill.Outcome.REWRITE: 1,
+ backfill.Outcome.HOLD_CROSS_USER: 1,
+ }
+ )
+ assert len(plan.mutations) == 1
+ assert plan.mutations[0].new_values == [
+ "/api/store/submissions/media/owner/images/photo.jpeg"
+ ]
+
+
+def test_array_rewrites_only_owned_well_formed_references():
+ owned = _raw_url("owner", "images", "owned.png")
+ cross_user = _raw_url("other", "images", "cross.png")
+ malformed = (
+ "https://storage.googleapis.com/private-media/"
+ "users/owner/images/nested/file.png"
+ )
+ external = "https://example.test/external.png"
+ candidate = _candidate(
+ target=backfill.Target.LISTING_IMAGES,
+ owner="owner",
+ values=[owned, cross_user, malformed, external],
+ is_array=True,
+ )
+
+ plan = backfill.build_plan([candidate], PRIVATE_BUCKET)
+
+ assert plan.counts == Counter(
+ {
+ backfill.Outcome.REWRITE: 1,
+ backfill.Outcome.HOLD_CROSS_USER: 1,
+ backfill.Outcome.HOLD_MALFORMED: 1,
+ }
+ )
+ assert plan.mutations[0].new_values == [
+ "/api/store/submissions/media/owner/images/owned.png",
+ cross_user,
+ malformed,
+ external,
+ ]
+
+
+ ("reason", "outcome"),
+ [
+ (backfill.HoldReason.PUBLIC, backfill.Outcome.HOLD_PUBLIC),
+ (backfill.HoldReason.AMBIGUOUS, backfill.Outcome.HOLD_AMBIGUOUS),
+ ],
+)
+def test_classification_holds_unsafe_reference_classes(reason, outcome):
+ candidate = _candidate(
+ target=backfill.Target.PROFILE_AVATAR,
+ owner="owner",
+ values=[_raw_url("owner", "images", "photo.jpeg")],
+ hold_reason=reason,
+ )
+
+ plan = backfill.build_plan([candidate], PRIVATE_BUCKET)
+
+ assert plan.counts == Counter({outcome: 1})
+ assert plan.mutations == []
+
+
+ "value",
+ [
+ "https://storage.googleapis.com/private-media/users/other/images/a.png",
+ "https://storage.googleapis.com/private-media/users/owner/images/a/b.png",
+ "https://storage.googleapis.com/private-media/users/owner/videos/a.png",
+ ],
+)
+def test_hold_reason_is_reported_before_owner_and_shape_checks(value: str):
+ candidate = _candidate(
+ target=backfill.Target.LISTING_IMAGES,
+ owner="owner",
+ values=[value],
+ is_array=True,
+ hold_reason=backfill.HoldReason.PUBLIC,
+ )
+
+ plan = backfill.build_plan([candidate], PRIVATE_BUCKET)
+
+ assert plan.counts == Counter({backfill.Outcome.HOLD_PUBLIC: 1})
+ assert plan.mutations == []
+
+
+def test_selected_but_unparsed_references_are_counted_as_unrecognized():
+ candidate = _candidate(
+ target=backfill.Target.LISTING_IMAGES,
+ owner="owner",
+ values=[
+ "https://example.test/private-media/users/owner/images/photo.jpeg",
+ "https://cdn.example/private-media.png",
+ "https://example.test/unrelated.png",
+ ],
+ is_array=True,
+ )
+
+ plan = backfill.build_plan([candidate], PRIVATE_BUCKET)
+
+ assert plan.counts == Counter({backfill.Outcome.UNRECOGNIZED: 2})
+ assert plan.mutations == []
+
+
+def test_public_bucket_urls_are_not_unrecognized_when_names_overlap():
+ candidate = _candidate(
+ target=backfill.Target.PROFILE_AVATAR,
+ owner="owner",
+ values=[
+ "https://storage.googleapis.com/private-media-public/"
+ "users/owner/images/photo.jpeg"
+ ],
+ )
+
+ plan = backfill.build_plan(
+ [candidate], PRIVATE_BUCKET, public_bucket="private-media-public"
+ )
+
+ assert plan.counts == Counter({backfill.Outcome.ALREADY_PUBLIC: 1})
+ assert plan.mutations == []
+
+
+def test_rewrite_accepts_query_and_wrapper_forms():
+ candidate = _candidate(
+ target=backfill.Target.PROFILE_AVATAR,
+ owner="owner",
+ values=[
+ "/_next/image?url=https%3A%2F%2Fstorage.googleapis.com%2Fprivate-media"
+ "%2Fusers%2Fowner%2Fimages%2Fphoto.jpeg%3Fv%3D3&w=640"
+ ],
+ )
+
+ plan = backfill.build_plan([candidate], PRIVATE_BUCKET)
+
+ assert plan.counts == Counter({backfill.Outcome.REWRITE: 1})
+ assert plan.mutations[0].new_values == [
+ "/api/store/submissions/media/owner/images/photo.jpeg"
+ ]
+
+
+ "target",
+ [
+ backfill.Target.ORGANIZATION_AVATAR,
+ backfill.Target.ORGANIZATION_PROFILE_AVATAR,
+ ],
+)
+def test_org_avatars_are_rewritten_only_while_the_uploader_is_a_member(target):
+ member_upload = _raw_url("member", "images", "avatar.jpeg")
+ former_member_upload = _raw_url("former", "images", "avatar.jpeg")
+ candidates = [
+ _candidate(target=target, owner=None, values=[url], co_owners=["member"])
+ for url in (member_upload, former_member_upload)
+ ]
+
+ plan = backfill.build_plan(candidates, PRIVATE_BUCKET)
+
+ assert plan.counts == Counter(
+ {backfill.Outcome.REWRITE: 1, backfill.Outcome.HOLD_CROSS_USER: 1}
+ )
+ assert plan.mutations[0].owner_user_id is None
+ assert plan.mutations[0].new_values == [
+ "/api/store/submissions/media/member/images/avatar.jpeg"
+ ]
+
+
+def test_library_rows_are_rewritten_only_to_media_their_owner_can_open():
+ own = _raw_url("owner", "images", "own.png")
+ colleague = _raw_url("colleague", "images", "shared.png")
+ creator = _raw_url("creator", "images", "listing.png")
+ candidates = [
+ _candidate(
+ target=backfill.Target.LIBRARY_IMAGE,
+ owner="owner",
+ values=[url],
+ co_owners=["colleague"],
+ )
+ for url in (own, colleague, creator)
+ ]
+
+ plan = backfill.build_plan(candidates, PRIVATE_BUCKET)
+
+ assert plan.counts == Counter(
+ {backfill.Outcome.REWRITE: 2, backfill.Outcome.HOLD_CROSS_USER: 1}
+ )
+ assert [mutation.owner_user_id for mutation in plan.mutations] == [
+ "owner",
+ "owner",
+ ]
+
+
+def test_candidate_query_allows_owner_proven_shared_rows():
+ assert "e.visibility" not in backfill_cli.CANDIDATE_QUERY
+ assert "la.visibility" not in backfill_cli.CANDIDATE_QUERY
+
+
+def test_candidate_query_leaves_logos_and_attribution_urls_alone():
+ assert "OAuthApplication" not in backfill_cli.CANDIDATE_QUERY
+ assert "SkillListingVersion.sourceUrl" not in backfill_cli.CANDIDATE_QUERY
+ assert "marketplace" not in backfill_cli.CANDIDATE_QUERY
+
+
+def test_candidate_query_covers_every_update_query():
+ for target in queries.UPDATE_QUERIES:
+ assert f"'{target}'" in backfill_cli.CANDIDATE_QUERY
+
+
+def test_select_and_update_share_null_safe_public_predicates():
+ assert "activeVersionId" not in queries.VERSION_IS_PUBLIC
+ assert backfill_cli.CANDIDATE_QUERY.count(queries.VERSION_IS_PUBLIC) == 3
+ assert queries.CREATOR_IS_PUBLIC in backfill_cli.CANDIDATE_QUERY
+ assert queries.CREATOR_IS_PUBLIC in queries.UPDATE_QUERIES["Profile.avatarUrl"]
+ for column in ("imageUrls", "videoUrl", "agentOutputDemoUrl"):
+ query = queries.UPDATE_QUERIES[f"StoreListingVersion.{column}"]
+ assert f"AND NOT {queries.VERSION_IS_PUBLIC}" in query
+ assert "SkillListing" in queries.CREATOR_IS_PUBLIC
+
+
+def test_updates_never_touch_updated_at():
+ for query in queries.UPDATE_QUERIES.values():
+ assert "updatedAt" not in query
+
+
+def test_only_owner_checked_updates_take_an_owner_parameter():
+ for target, query in queries.UPDATE_QUERIES.items():
+ assert ("$4" in query) == (target not in queries.PATH_OWNER_TARGETS)
+
+
+async def test_dry_run_never_executes_updates():
+ client = AsyncMock()
+ transaction, opened = _transactions(client)
+ candidate = _candidate(
+ target=backfill.Target.PROFILE_AVATAR,
+ owner="owner",
+ values=[_raw_url("owner", "images", "photo.jpeg")],
+ )
+
+ report = await backfill.process_candidates(
+ [candidate], PRIVATE_BUCKET, apply=False, transaction=transaction
+ )
+
+ assert report.planned_rows == 1
+ assert report.applied_rows == 0
+ assert opened == []
+ client.execute_raw.assert_not_awaited()
+
+
+async def test_apply_uses_compare_and_swap_and_reports_conflict():
+ client = AsyncMock()
+ client.execute_raw.return_value = 0
+ transaction, _ = _transactions(client)
+ old_url = _raw_url("owner", "images", "photo.jpeg")
+ candidate = _candidate(
+ target=backfill.Target.PROFILE_AVATAR,
+ owner="owner",
+ values=[old_url],
+ )
+
+ report = await backfill.process_candidates(
+ [candidate], PRIVATE_BUCKET, apply=True, transaction=transaction
+ )
+
+ assert report.applied_rows == 0
+ assert report.cas_conflicts == 1
+ query, row_id, replacement, expected, owner = client.execute_raw.await_args.args
+ assert 'AND p."avatarUrl" = $3' in query
+ assert 'AND p."userId" = $4' in query
+ assert "AND NOT (" in query
+ assert row_id == "record-secret"
+ assert replacement == "/api/store/submissions/media/owner/images/photo.jpeg"
+ assert expected == old_url
+ assert owner == "owner"
+
+
+async def test_apply_updates_owned_array_with_whole_row_compare_and_swap():
+ client = AsyncMock()
+ client.execute_raw.return_value = 1
+ transaction, _ = _transactions(client)
+ old_urls = [
+ _raw_url("owner", "images", "first.png"),
+ "https://example.test/unchanged.png",
+ ]
+ candidate = _candidate(
+ target=backfill.Target.LISTING_IMAGES,
+ owner="owner",
+ values=old_urls,
+ is_array=True,
+ )
+
+ report = await backfill.process_candidates(
+ [candidate], PRIVATE_BUCKET, apply=True, transaction=transaction
+ )
+
+ assert report.applied_rows == 1
+ assert report.cas_conflicts == 0
+ query, _row_id, replacement, expected, owner = client.execute_raw.await_args.args
+ assert 'AND slv."imageUrls" = $3::text[]' in query
+ assert 'AND sl."owningUserId" = $4' in query
+ assert "AND NOT (" in query
+ assert replacement == [
+ "/api/store/submissions/media/owner/images/first.png",
+ "https://example.test/unchanged.png",
+ ]
+ assert expected == old_urls
+ assert owner == "owner"
+
+
+async def test_apply_allows_owner_proven_shared_expert_media():
+ client = AsyncMock()
+ client.execute_raw.return_value = 1
+ transaction, _ = _transactions(client)
+ old_url = _raw_url("owner", "images", "shared.png")
+ candidate = _candidate(
+ target=backfill.Target.EXPERT_AVATAR, owner="owner", values=[old_url]
+ )
+
+ report = await backfill.process_candidates(
+ [candidate], PRIVATE_BUCKET, apply=True, transaction=transaction
+ )
+
+ assert report.applied_rows == 1
+ query = client.execute_raw.await_args.args[0]
+ assert "visibility" not in query
+
+
+async def test_apply_rewrites_a_library_image_only_for_its_owner():
+ client = AsyncMock()
+ client.execute_raw.return_value = 1
+ transaction, _ = _transactions(client)
+ old_url = _raw_url("owner", "images", "agent.png")
+ candidate = _candidate(
+ target=backfill.Target.LIBRARY_IMAGE, owner="owner", values=[old_url]
+ )
+
+ report = await backfill.process_candidates(
+ [candidate], PRIVATE_BUCKET, apply=True, transaction=transaction
+ )
+
+ assert report.applied_rows == 1
+ query, row_id, replacement, expected, owner = client.execute_raw.await_args.args
+ assert 'la."userId" = $4' in query
+ assert row_id == "record-secret"
+ assert replacement == "/api/store/submissions/media/owner/images/agent.png"
+ assert expected == old_url
+ assert owner == "owner"
+
+
+async def test_apply_commits_in_short_batches_and_keeps_progress_on_failure():
+ client = AsyncMock()
+ client.execute_raw.side_effect = [1] * 400 + [RuntimeError("db down")]
+ transaction, opened = _transactions(client)
+ candidates = [
+ _candidate(
+ target=backfill.Target.LIBRARY_IMAGE,
+ owner="creator",
+ values=[_raw_url("creator", "images", f"{index}.png")],
+ )
+ for index in range(450)
+ ]
+ progress = backfill.ApplyProgress()
+
+ with pytest.raises(RuntimeError):
+ await backfill.process_candidates(
+ candidates,
+ PRIVATE_BUCKET,
+ apply=True,
+ transaction=transaction,
+ progress=progress,
+ )
+
+ assert len(opened) == 3
+ assert progress.applied_rows == 400
+
+
+async def test_apply_splits_mutations_into_batches_of_the_configured_size():
+ client = AsyncMock()
+ client.execute_raw.return_value = 1
+ transaction, opened = _transactions(client)
+ candidates = [
+ _candidate(
+ target=backfill.Target.LIBRARY_IMAGE,
+ owner="creator",
+ values=[_raw_url("creator", "images", f"{index}.png")],
+ )
+ for index in range(backfill.APPLY_BATCH_SIZE * 2 + 1)
+ ]
+
+ report = await backfill.process_candidates(
+ candidates, PRIVATE_BUCKET, apply=True, transaction=transaction
+ )
+
+ assert len(opened) == 3
+ assert report.applied_rows == backfill.APPLY_BATCH_SIZE * 2 + 1
+
+
+async def test_apply_is_idempotent_for_already_rewritten_url():
+ client = AsyncMock()
+ transaction, opened = _transactions(client)
+ candidate = _candidate(
+ target=backfill.Target.PROFILE_AVATAR,
+ owner="owner",
+ values=["/api/store/submissions/media/owner/images/photo.jpeg"],
+ )
+
+ report = await backfill.process_candidates(
+ [candidate], PRIVATE_BUCKET, apply=True, transaction=transaction
+ )
+
+ assert report.planned_rows == 0
+ assert opened == []
+ client.execute_raw.assert_not_awaited()
+
+
+def test_summary_never_prints_identifiers_or_urls(capsys: pytest.CaptureFixture[str]):
+ candidate = _candidate(
+ target=backfill.Target.PROFILE_AVATAR,
+ owner="owner-secret",
+ values=[_raw_url("owner-secret", "images", "filename-secret.jpeg")],
+ )
+ plan = backfill.build_plan([candidate], PRIVATE_BUCKET)
+ report = backfill.BackfillReport.from_plan(plan)
+
+ backfill.print_report(report, apply=False)
+
+ output = capsys.readouterr().out
+ assert "owner-secret" not in output
+ assert "filename-secret" not in output
+ assert "record-secret" not in output
+ assert "storage.googleapis.com" not in output
+ assert "rewrite" in output
+ assert "unrecognized" in output
+ assert "Profile.avatarUrl" in output
+
+
+def test_source_bucket_defaults_to_the_private_bucket():
+ assert (
+ backfill.resolve_source_bucket(
+ private_bucket="legacy", legacy_bucket="", override=None
+ )
+ == "legacy"
+ )
+ assert (
+ backfill.resolve_source_bucket(
+ private_bucket="legacy", legacy_bucket="legacy", override=None
+ )
+ == "legacy"
+ )
+
+
+def test_source_bucket_refuses_to_guess_when_buckets_diverge():
+ with pytest.raises(SystemExit, match="--bucket"):
+ backfill.resolve_source_bucket(
+ private_bucket="new-private", legacy_bucket="legacy", override=None
+ )
+
+ assert (
+ backfill.resolve_source_bucket(
+ private_bucket="new-private", legacy_bucket="legacy", override="legacy"
+ )
+ == "legacy"
+ )
+
+
+def test_source_bucket_requires_a_bucket():
+ with pytest.raises(SystemExit):
+ backfill.resolve_source_bucket(
+ private_bucket="", legacy_bucket="", override=None
+ )
+
+
+def _transactions(client):
+ opened: list[int] = []
+
+ @asynccontextmanager
+ async def transaction():
+ opened.append(len(opened))
+ yield client
+
+ return transaction, opened
+
+
+def _candidate(
+ *,
+ target: "backfill.Target",
+ owner: str | None,
+ values: list[str],
+ is_array: bool = False,
+ hold_reason: "backfill.HoldReason | None" = None,
+ co_owners: list[str] | None = None,
+):
+ return backfill.Candidate(
+ target=target,
+ record_id="record-secret",
+ owner_user_id=owner,
+ values=values,
+ is_array=is_array,
+ hold_reason=hold_reason,
+ co_owner_ids=co_owners or [],
+ )
+
+
+def _raw_url(owner: str, media_type: str, filename: str) -> str:
+ return (
+ f"https://storage.googleapis.com/{PRIVATE_BUCKET}/"
+ f"users/{owner}/{media_type}/{filename}"
+ )
+
+
+def test_org_listing_rewrites_media_uploaded_by_an_active_member():
+ member_image = _raw_url("member", "images", "shot.png")
+ stranger_image = _raw_url("stranger", "images", "shot.png")
+ candidate = _candidate(
+ target=backfill.Target.LISTING_IMAGES,
+ owner="owner",
+ values=[member_image, stranger_image],
+ is_array=True,
+ co_owners=["owner", "member"],
+ )
+
+ plan = backfill.build_plan([candidate], PRIVATE_BUCKET)
+
+ assert plan.mutations[0].owner_user_id == "owner"
+ assert plan.mutations[0].new_values == [
+ "/api/store/submissions/media/member/images/shot.png",
+ stranger_image,
+ ]
+ assert plan.counts[backfill.Outcome.HOLD_CROSS_USER] == 1
+
+
+ "outcome, blocks",
+ [
+ (backfill.Outcome.HOLD_CROSS_USER, True),
+ (backfill.Outcome.HOLD_MALFORMED, True),
+ (backfill.Outcome.HOLD_AMBIGUOUS, True),
+ (backfill.Outcome.HOLD_PUBLIC, True),
+ (backfill.Outcome.UNRECOGNIZED, True),
+ (backfill.Outcome.REWRITE, False),
+ (backfill.Outcome.ALREADY_PUBLIC, False),
+ ],
+)
+def test_references_left_on_the_legacy_bucket_fail_the_run(outcome, blocks):
+ report = backfill.BackfillReport(
+ counts=Counter({outcome: 1}), target_counts={}, planned_rows=0
+ )
+
+ assert bool(backfill_cli.stranded_references(report)) is blocks
diff --git a/autogpt_platform/backend/scripts/test_publish_live_media.py b/autogpt_platform/backend/scripts/test_publish_live_media.py
new file mode 100644
index 0000000000..5919f5c795
--- /dev/null
+++ b/autogpt_platform/backend/scripts/test_publish_live_media.py
@@ -0,0 +1,473 @@
+import re
+from collections import Counter
+from contextlib import asynccontextmanager
+from unittest.mock import AsyncMock
+
+import pytest
+
+import scripts.media_url_backfill_queries as queries
+import scripts.media_url_publish as publish
+import scripts.publish_live_media as publish_cli
+from scripts.media_url_backfill import Target
+
+PRIVATE = "legacy-media"
+PUBLIC = "site-media"
+BUCKETS = publish.Buckets(source=PRIVATE, private=PRIVATE, public=PUBLIC)
+Outcome = publish.PublishOutcome
+
+
+def test_plan_publishes_only_media_owned_by_the_listing_or_its_org_members():
+ owned = _private_url("owner", "images", "a.png")
+ member = _private_url("member", "videos", "clip.mp4")
+ foreign = _private_url("stranger", "images", "b.png")
+ nested = f"https://storage.googleapis.com/{PRIVATE}/users/owner/images/x/y.png"
+ traversal = f"https://storage.googleapis.com/{PRIVATE}/users/owner/images/%2e%2e"
+ external = "https://example.test/c.png"
+ candidate = _candidate(
+ Target.LISTING_IMAGES,
+ [owned, member, foreign, nested, traversal, external],
+ owner_ids=["owner", "member"],
+ is_array=True,
+ )
+
+ plan = publish.plan_publication([candidate], BUCKETS)
+
+ assert [reference.source.path for reference in plan.references] == [
+ "users/owner/images/a.png",
+ "users/member/videos/clip.mp4",
+ ]
+ assert plan.counts == {
+ Target.LISTING_IMAGES: Counter(
+ {Outcome.SKIP_FOREIGN_OWNER: 1, Outcome.SKIP_MALFORMED: 2}
+ )
+ }
+
+
+def test_plan_accepts_legacy_filenames_and_url_forms_the_new_validator_rejects():
+ candidate = _candidate(
+ Target.PROFILE_AVATAR,
+ [
+ f"/_next/image?url=gs%3A%2F%2F{PRIVATE}%2Fusers%2Fowner%2Fimages"
+ "%2FMy%20Photo%20(1).PNG&w=64"
+ ],
+ owner_ids=["owner"],
+ )
+
+ plan = publish.plan_publication([candidate], BUCKETS)
+
+ assert plan.objects == {
+ publish.SourceObject(bucket=PRIVATE, path="users/owner/images/My Photo (1).PNG")
+ }
+
+
+def test_plan_skips_already_public_and_counts_unrecognized_private_references():
+ candidate = _candidate(
+ Target.LISTING_IMAGES,
+ [
+ _public_url("users/owner/images/a.png"),
+ f"https://cdn.example/{PRIVATE}/users/owner/images/a.png",
+ "/api/store/submissions/media/owner/images/legacy name.png",
+ ],
+ owner_ids=["owner"],
+ is_array=True,
+ )
+
+ plan = publish.plan_publication([candidate], BUCKETS)
+
+ assert plan.references == []
+ assert plan.counts == {
+ Target.LISTING_IMAGES: Counter(
+ {Outcome.ALREADY_PUBLIC: 1, Outcome.SKIP_UNRECOGNIZED: 2}
+ )
+ }
+ assert plan.already_public_image_paths == {"users/owner/images/a.png"}
+
+
+def test_plan_repairs_private_api_references_from_the_private_bucket():
+ buckets = publish.Buckets(source="legacy", private="new-private", public=PUBLIC)
+ candidate = _candidate(
+ Target.LISTING_VIDEO,
+ ["/api/store/submissions/media/owner/videos/clip.mp4"],
+ owner_ids=["owner"],
+ )
+
+ plan = publish.plan_publication([candidate], buckets)
+
+ assert plan.objects == {
+ publish.SourceObject(bucket="new-private", path="users/owner/videos/clip.mp4")
+ }
+
+
+def test_plan_publishes_oauth_logos_only_under_their_own_app_prefix():
+ own = _candidate(
+ Target.OAUTH_LOGO,
+ [f"https://storage.googleapis.com/{PRIVATE}/oauth-apps/app-1/logo/l.png"],
+ record_id="app-1",
+ )
+ other = _candidate(
+ Target.OAUTH_LOGO,
+ [f"https://storage.googleapis.com/{PRIVATE}/oauth-apps/app-1/logo/l.png"],
+ record_id="app-2",
+ )
+ user_media = _candidate(
+ Target.OAUTH_LOGO, [_private_url("owner", "images", "a.png")], record_id="app-3"
+ )
+
+ plan = publish.plan_publication([own, other, user_media], BUCKETS)
+
+ assert plan.objects == {
+ publish.SourceObject(bucket=PRIVATE, path="oauth-apps/app-1/logo/l.png")
+ }
+ assert plan.counts == {
+ Target.OAUTH_LOGO: Counter(
+ {Outcome.SKIP_FOREIGN_OWNER: 1, Outcome.SKIP_MALFORMED: 1}
+ )
+ }
+
+
+async def test_copy_objects_copies_each_object_once_and_reports_failures():
+ copier = _Copier(fail={"users/owner/images/missing.png"})
+ objects = [
+ publish.SourceObject(bucket=PRIVATE, path="users/owner/images/a.png"),
+ publish.SourceObject(bucket=PRIVATE, path="users/owner/images/a.png"),
+ publish.SourceObject(bucket=PRIVATE, path="users/owner/images/missing.png"),
+ ]
+
+ copied = await publish.copy_objects(objects, PUBLIC, copier)
+
+ assert copied == {
+ publish.SourceObject(bucket=PRIVATE, path="users/owner/images/a.png")
+ }
+ assert sorted(copier.calls) == [
+ (PRIVATE, "users/owner/images/a.png", PUBLIC, "users/owner/images/a.png"),
+ (
+ PRIVATE,
+ "users/owner/images/missing.png",
+ PUBLIC,
+ "users/owner/images/missing.png",
+ ),
+ ]
+
+
+def test_build_mutations_rewrites_only_copied_references():
+ copied_url = _private_url("owner", "images", "a b.png")
+ failed_url = _private_url("owner", "images", "missing.png")
+ candidate = _candidate(
+ Target.LISTING_IMAGES,
+ [copied_url, failed_url, "https://example.test/x.png"],
+ owner_ids=["owner"],
+ is_array=True,
+ )
+ plan = publish.plan_publication([candidate], BUCKETS)
+ copied = {publish.SourceObject(bucket=PRIVATE, path="users/owner/images/a b.png")}
+
+ mutations, counts = publish.build_mutations(plan, copied, PUBLIC)
+
+ assert len(mutations) == 1
+ assert mutations[0].owner_user_id is None
+ assert mutations[0].old_values == candidate.values
+ assert mutations[0].new_values == [
+ f"https://storage.googleapis.com/{PUBLIC}/users/owner/images/a%20b.png",
+ failed_url,
+ "https://example.test/x.png",
+ ]
+ assert counts == {
+ Target.LISTING_IMAGES: Counter({Outcome.PUBLISH: 1, Outcome.COPY_FAILED: 1})
+ }
+
+
+async def test_dry_run_copies_nothing_and_writes_nothing():
+ client = _client(
+ listings=[_listing_row([_private_url("owner", "images", "a.png")])],
+ library=[("lib-1", _private_url("owner", "images", "a.png"))],
+ )
+ copier_factory, copier = _copier_factory()
+ transaction, opened = _transactions(client)
+
+ report = await publish_cli._run(
+ client,
+ BUCKETS,
+ apply=False,
+ transaction=transaction,
+ copier_factory=copier_factory,
+ progress=publish.ApplyProgress(),
+ )
+
+ assert copier.calls == []
+ assert opened == []
+ client.execute_raw.assert_not_awaited()
+ assert report.planned_objects == 1
+ assert report.planned_rows == 2
+ assert report.counts[Target.LIBRARY_IMAGE] == Counter({Outcome.PUBLISH: 1})
+
+
+async def test_apply_copies_then_rewrites_listing_profile_logo_and_library_rows():
+ image = _private_url("owner", "images", "a.png")
+ avatar = _private_url("owner", "images", "avatar.png")
+ logo = f"https://storage.googleapis.com/{PRIVATE}/oauth-apps/app-1/logo/l.png"
+ client = _client(
+ listings=[
+ _listing_row([image], video_url=_private_url("owner", "videos", "v.mp4"))
+ ],
+ profiles=[{"record_id": "profile-1", "owner_id": "owner", "value": avatar}],
+ logos=[{"record_id": "app-1", "value": logo}],
+ library=[
+ ("lib-1", image),
+ ("lib-2", "/api/store/submissions/media/owner/images/a.png"),
+ ("lib-3", _private_url("owner", "images", "not-listed.png")),
+ ],
+ )
+ client.execute_raw.return_value = 1
+ copier_factory, copier = _copier_factory()
+ transaction, opened = _transactions(client)
+
+ report = await publish_cli._run(
+ client,
+ BUCKETS,
+ apply=True,
+ transaction=transaction,
+ copier_factory=copier_factory,
+ progress=publish.ApplyProgress(),
+ )
+
+ assert {call[1] for call in copier.calls} == {
+ "users/owner/images/a.png",
+ "users/owner/videos/v.mp4",
+ "users/owner/images/avatar.png",
+ "oauth-apps/app-1/logo/l.png",
+ }
+ assert all(call[0] == PRIVATE and call[2] == PUBLIC for call in copier.calls)
+ assert len(opened) == 1
+ updates = {
+ (args[1], _set_column(args[0])): args[2:]
+ for args in (call.args for call in client.execute_raw.await_args_list)
+ }
+ assert updates == {
+ ("version-1", "imageUrls"): (
+ [_public_url("users/owner/images/a.png")],
+ [image],
+ ),
+ ("version-1", "videoUrl"): (
+ _public_url("users/owner/videos/v.mp4"),
+ _private_url("owner", "videos", "v.mp4"),
+ ),
+ ("profile-1", "avatarUrl"): (
+ _public_url("users/owner/images/avatar.png"),
+ avatar,
+ ),
+ ("app-1", "logoUrl"): (_public_url("oauth-apps/app-1/logo/l.png"), logo),
+ ("lib-1", "imageUrl"): (_public_url("users/owner/images/a.png"), image),
+ ("lib-2", "imageUrl"): (
+ _public_url("users/owner/images/a.png"),
+ "/api/store/submissions/media/owner/images/a.png",
+ ),
+ }
+ assert client.execute_raw.await_count == 6
+ assert report.applied_rows == 6
+ assert report.copied_objects == 4
+
+
+async def test_apply_skips_rows_whose_copy_failed():
+ image = _private_url("owner", "images", "missing.png")
+ client = _client(listings=[_listing_row([image])], library=[("lib-1", image)])
+ copier_factory, _ = _copier_factory(fail={"users/owner/images/missing.png"})
+ transaction, opened = _transactions(client)
+
+ report = await publish_cli._run(
+ client,
+ BUCKETS,
+ apply=True,
+ transaction=transaction,
+ copier_factory=copier_factory,
+ progress=publish.ApplyProgress(),
+ )
+
+ client.execute_raw.assert_not_awaited()
+ assert report.counts == {Target.LISTING_IMAGES: Counter({Outcome.COPY_FAILED: 1})}
+
+
+async def test_rerun_is_idempotent_and_repairs_stale_library_copies():
+ published = _public_url("users/owner/images/a.png")
+ client = _client(
+ listings=[_listing_row([published])],
+ library=[("lib-1", _private_url("owner", "images", "a.png"))],
+ )
+ client.execute_raw.return_value = 1
+ copier_factory, copier = _copier_factory()
+ transaction, _ = _transactions(client)
+
+ report = await publish_cli._run(
+ client,
+ BUCKETS,
+ apply=True,
+ transaction=transaction,
+ copier_factory=copier_factory,
+ progress=publish.ApplyProgress(),
+ )
+
+ assert copier.calls == []
+ assert client.execute_raw.await_count == 1
+ query, row_id, new_value, old_value = client.execute_raw.await_args.args
+ assert 'platform."LibraryAgent"' in query
+ assert (row_id, new_value) == ("lib-1", published)
+ assert old_value == _private_url("owner", "images", "a.png")
+ assert report.counts[Target.LISTING_IMAGES] == Counter({Outcome.ALREADY_PUBLIC: 1})
+
+
+def test_report_never_prints_identifiers_or_urls(capsys: pytest.CaptureFixture[str]):
+ report = publish.PublishReport(
+ counts={Target.PROFILE_AVATAR: Counter({Outcome.PUBLISH: 1})},
+ planned_objects=1,
+ planned_rows=1,
+ )
+
+ publish.print_report(report, apply=True)
+
+ output = capsys.readouterr().out
+ assert "storage.googleapis.com" not in output
+ assert "users/" not in output
+ assert "Profile.avatarUrl: publish=1" in output
+ assert "copy_failed: 0" in output
+
+
+def test_publish_queries_are_compare_and_swap_without_updated_at():
+ for target, query in queries.PUBLISH_UPDATE_QUERIES.items():
+ column = target.split(".")[1]
+ assert "updatedAt" not in query
+ assert "WHERE id = $1" in query
+ assert f'"{column}" = $3' in query
+ assert "$4" not in query
+
+
+def test_selection_matches_the_backfill_holds():
+ assert queries.VERSION_IS_PUBLIC in publish_cli.LISTING_QUERY
+ assert queries.CREATOR_IS_PUBLIC in publish_cli.PROFILE_QUERY
+ assert "om.status = 'ACTIVE'" in publish_cli.LISTING_QUERY
+ assert 'om."orgId" = sl."owningOrgId"' in publish_cli.LISTING_QUERY
+
+
+def _set_column(query: str) -> str:
+ match = re.search(r'SET "(\w+)"', query)
+ assert match
+ return match[1]
+
+
+class _Copier:
+ def __init__(self, fail: set[str] | None = None):
+ self.fail = fail or set()
+ self.calls: list[tuple[str, str, str, str | None]] = []
+
+ async def copy(self, bucket, object_name, destination_bucket, *, new_name=None):
+ self.calls.append((bucket, object_name, destination_bucket, new_name))
+ if object_name in self.fail:
+ raise RuntimeError("404")
+ return {}
+
+
+def _copier_factory(fail: set[str] | None = None):
+ copier = _Copier(fail)
+
+ @asynccontextmanager
+ async def factory():
+ yield copier
+
+ return factory, copier
+
+
+def _transactions(client):
+ opened: list[int] = []
+
+ @asynccontextmanager
+ async def transaction():
+ opened.append(len(opened))
+ yield client
+
+ return transaction, opened
+
+
+def _client(
+ *,
+ listings: list[dict] | None = None,
+ profiles: list[dict] | None = None,
+ logos: list[dict] | None = None,
+ library: list[tuple[str, str]] | None = None,
+):
+ results = {
+ publish_cli.LISTING_QUERY: listings or [],
+ publish_cli.PROFILE_QUERY: profiles or [],
+ publish_cli.OAUTH_LOGO_QUERY: logos or [],
+ publish_cli.LIBRARY_QUERY: [
+ {"record_id": record_id, "value": value}
+ for record_id, value in library or []
+ ],
+ }
+ client = AsyncMock()
+
+ async def query_raw(query, *args):
+ return results[query]
+
+ client.query_raw.side_effect = query_raw
+ return client
+
+
+def _listing_row(
+ image_urls: list[str],
+ *,
+ video_url: str | None = None,
+ demo_url: str | None = None,
+ owner_ids: list[str] | None = None,
+):
+ return {
+ "record_id": "version-1",
+ "image_urls": image_urls,
+ "video_url": video_url,
+ "demo_url": demo_url,
+ "owner_ids": owner_ids or ["owner"],
+ }
+
+
+def _candidate(
+ target: Target,
+ values: list[str],
+ *,
+ owner_ids: list[str] | None = None,
+ is_array: bool = False,
+ record_id: str = "record-secret",
+):
+ return publish.PublishCandidate(
+ target=target,
+ record_id=record_id,
+ values=values,
+ is_array=is_array,
+ owner_ids=owner_ids or [],
+ )
+
+
+def _private_url(owner: str, media_type: str, filename: str) -> str:
+ return f"https://storage.googleapis.com/{PRIVATE}/users/{owner}/{media_type}/{filename}"
+
+
+def _public_url(path: str) -> str:
+ return f"https://storage.googleapis.com/{PUBLIC}/{path}"
+
+
+ "outcome, blocks",
+ [
+ (publish.PublishOutcome.SKIP_FOREIGN_OWNER, True),
+ (publish.PublishOutcome.SKIP_MALFORMED, True),
+ (publish.PublishOutcome.SKIP_UNRECOGNIZED, True),
+ (publish.PublishOutcome.COPY_FAILED, True),
+ (publish.PublishOutcome.ALREADY_PUBLIC, False),
+ (publish.PublishOutcome.PUBLISH, False),
+ ],
+)
+def test_every_unpublished_live_reference_fails_the_run(outcome, blocks):
+ report = publish.PublishReport(
+ counts={Target.LISTING_IMAGES: Counter({outcome: 1})},
+ planned_objects=0,
+ copied_objects=0,
+ planned_rows=0,
+ )
+
+ assert bool(publish_cli.unpublished_references(report)) is blocks
diff --git a/autogpt_platform/backend/test/load_store_agents.py b/autogpt_platform/backend/test/load_store_agents.py
index b7b79151e5..987276894d 100644
--- a/autogpt_platform/backend/test/load_store_agents.py
+++ b/autogpt_platform/backend/test/load_store_agents.py
@@ -179,7 +179,7 @@ async def create_user_and_profile(db: Prisma) -> None:
username=AUTOGPT_USERNAME,
description="Official AutoGPT agents and templates",
links=["https://agpt.co"],
- avatarUrl="https://storage.googleapis.com/agpt-prod-website-artifacts/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/4b5781a6-49e1-433c-9a75-65af1be5c02d.png",
+ avatarUrl="https://storage.googleapis.com/agpt-prod-public-site-media/users/b3e41ea4-2f4c-4964-927c-fe682d857bad/images/4b5781a6-49e1-433c-9a75-65af1be5c02d.png",
)
)
diff --git a/autogpt_platform/frontend/next.config.mjs b/autogpt_platform/frontend/next.config.mjs
index 9bef02203b..ad1463686c 100644
--- a/autogpt_platform/frontend/next.config.mjs
+++ b/autogpt_platform/frontend/next.config.mjs
@@ -19,6 +19,10 @@ const nextConfig = {
poweredByHeader: false,
async rewrites() {
return [
+ {
+ source: "/api/store/submissions/media/:path*",
+ destination: "/api/proxy/api/store/submissions/media/:path*",
+ },
{
source: "/api/store/media/:path*",
destination: "/api/proxy/api/store/media/:path*",
diff --git a/autogpt_platform/frontend/src/app/api/openapi.json b/autogpt_platform/frontend/src/app/api/openapi.json
index 30cb238083..35fccb7398 100644
--- a/autogpt_platform/frontend/src/app/api/openapi.json
+++ b/autogpt_platform/frontend/src/app/api/openapi.json
@@ -16138,6 +16138,96 @@
}
}
},
+ "/api/store/submissions/media/{owner_user_id}/{media_type}/{filename}": {
+ "get": {
+ "tags": ["v2", "store", "private"],
+ "summary": "Read private submission media",
+ "description": "Serve a user's private media to them, to members of an organization they\nshare, and to admins. Anyone else gets a 404, as if it did not exist.",
+ "operationId": "getV2Read private submission media",
+ "security": [{ "HTTPBearerJWT": [] }],
+ "parameters": [
+ {
+ "name": "owner_user_id",
+ "in": "path",
+ "required": true,
+ "schema": { "type": "string", "title": "Owner User Id" }
+ },
+ {
+ "name": "media_type",
+ "in": "path",
+ "required": true,
+ "schema": { "type": "string", "title": "Media Type" }
+ },
+ {
+ "name": "filename",
+ "in": "path",
+ "required": true,
+ "schema": { "type": "string", "title": "Filename" }
+ }
+ ],
+ "responses": {
+ "200": {
+ "description": "Successful Response",
+ "content": {
+ "image/jpeg": {
+ "schema": { "type": "string", "format": "binary" }
+ },
+ "image/png": {
+ "schema": { "type": "string", "format": "binary" }
+ },
+ "image/gif": {
+ "schema": { "type": "string", "format": "binary" }
+ },
+ "image/webp": {
+ "schema": { "type": "string", "format": "binary" }
+ },
+ "video/mp4": {
+ "schema": { "type": "string", "format": "binary" }
+ },
+ "video/webm": {
+ "schema": { "type": "string", "format": "binary" }
+ }
+ }
+ },
+ "206": {
+ "description": "Requested byte range",
+ "content": {
+ "image/jpeg": {
+ "schema": { "type": "string", "format": "binary" }
+ },
+ "image/png": {
+ "schema": { "type": "string", "format": "binary" }
+ },
+ "image/gif": {
+ "schema": { "type": "string", "format": "binary" }
+ },
+ "image/webp": {
+ "schema": { "type": "string", "format": "binary" }
+ },
+ "video/mp4": {
+ "schema": { "type": "string", "format": "binary" }
+ },
+ "video/webm": {
+ "schema": { "type": "string", "format": "binary" }
+ }
+ }
+ },
+ "401": {
+ "$ref": "#/components/responses/HTTP401NotAuthenticatedError"
+ },
+ "404": { "description": "Media not found" },
+ "416": { "description": "Requested range is not satisfiable" },
+ "422": {
+ "description": "Validation Error",
+ "content": {
+ "application/json": {
+ "schema": { "$ref": "#/components/schemas/HTTPValidationError" }
+ }
+ }
+ }
+ }
+ }
+ },
"/api/store/submissions/{store_listing_version_id}": {
"put": {
"tags": ["v2", "store", "private"],
diff --git a/autogpt_platform/frontend/src/app/api/proxy/[...path]/__tests__/route.helpers.test.ts b/autogpt_platform/frontend/src/app/api/proxy/[...path]/__tests__/route.helpers.test.ts
index 3763e8d33c..b5e7856479 100644
--- a/autogpt_platform/frontend/src/app/api/proxy/[...path]/__tests__/route.helpers.test.ts
+++ b/autogpt_platform/frontend/src/app/api/proxy/[...path]/__tests__/route.helpers.test.ts
@@ -12,8 +12,69 @@ import {
RESPONSE_START_TIMEOUT_MS,
CODEX_LOGIN_RESPONSE_START_TIMEOUT_MS,
getResponseStartTimeoutMs,
+ getSafePrivateMediaRange,
+ isPrivateStoreMediaRequest,
+ PRIVATE_MEDIA_RANGE_CHUNK_BYTES,
+ isPrivateStoreVideoRequest,
} from "../route.helpers";
+describe("private store media request matching", () => {
+ const imagePath = [
+ "api",
+ "store",
+ "submissions",
+ "media",
+ "user-123",
+ "images",
+ "thumbnail.png",
+ ];
+
+ it("recognizes exact private image and video paths", () => {
+ expect(isPrivateStoreMediaRequest(imagePath)).toBe(true);
+ expect(
+ isPrivateStoreMediaRequest([
+ ...imagePath.slice(0, 5),
+ "videos",
+ "preview.mp4",
+ ]),
+ ).toBe(true);
+ expect(isPrivateStoreVideoRequest(imagePath)).toBe(false);
+ expect(
+ isPrivateStoreVideoRequest([
+ ...imagePath.slice(0, 5),
+ "videos",
+ "preview.mp4",
+ ]),
+ ).toBe(true);
+ });
+
+ it.each([
+ [imagePath.slice(0, -1)],
+ [[...imagePath, "extra"]],
+ [["api", "store", "media", "user-123", "images", "thumbnail.png"]],
+ [[...imagePath.slice(0, 4), "..", "images", "thumbnail.png"]],
+ [[...imagePath.slice(0, 5), "documents", "thumbnail.png"]],
+ [[...imagePath.slice(0, 6), "nested/thumbnail.png"]],
+ ])("rejects a non-private-media path: %j", (path) => {
+ expect(isPrivateStoreMediaRequest(path)).toBe(false);
+ });
+
+ it.each([
+ ["bytes=0-1023", "bytes=0-1023"],
+ ["bytes=1024-", `bytes=1024-${1023 + PRIVATE_MEDIA_RANGE_CHUNK_BYTES}`],
+ ["bytes=0-999999999", `bytes=0-${PRIVATE_MEDIA_RANGE_CHUNK_BYTES - 1}`],
+ ["bytes=-512", "bytes=-512"],
+ ["bytes=-999999999", `bytes=-${PRIVATE_MEDIA_RANGE_CHUNK_BYTES}`],
+ [" bytes=0-1023 ", "bytes=0-1023"],
+ ["bytes=0-1,4-5", null],
+ ["items=0-1", null],
+ ["bytes=-", null],
+ [null, null],
+ ])("sanitizes a Range header: %s", (value, expected) => {
+ expect(getSafePrivateMediaRange(value)).toBe(expected);
+ });
+});
+
describe("isWorkspaceDownloadRequest", () => {
const VALID_UUID = "550e8400-e29b-41d4-a716-446655440000";
const VALID_UUID_2 = "6ba7b810-9dad-11d1-80b4-00c04fd430c8";
diff --git a/autogpt_platform/frontend/src/app/api/proxy/[...path]/__tests__/route.test.ts b/autogpt_platform/frontend/src/app/api/proxy/[...path]/__tests__/route.test.ts
index 9532bce787..c98f026f59 100644
--- a/autogpt_platform/frontend/src/app/api/proxy/[...path]/__tests__/route.test.ts
+++ b/autogpt_platform/frontend/src/app/api/proxy/[...path]/__tests__/route.test.ts
@@ -18,6 +18,14 @@ vi.mock("@/services/environment", () => ({
import { getCountryToken } from "@/lib/auth/country-token";
import { getServerAuthToken } from "@/lib/auth/server/getServerAuthToken";
import { GET, POST } from "../route";
+import { PRIVATE_MEDIA_RANGE_CHUNK_BYTES } from "../route.helpers";
+
+const PRIVATE_IMAGE_HEADER_KEYS = [
+ "accept",
+ "accept-encoding",
+ "authorization",
+ "cache-control",
+];
const BACKEND = "https://backend.test";
@@ -105,6 +113,7 @@ describe("proxy route — handler pass-through", () => {
"Content-Type": "application/json",
"Content-Length": "1234",
"Content-Encoding": "gzip",
+ Authorization: "Bearer upstream-secret",
"Cache-Control": "no-store",
"X-Custom-Header": "preserved",
},
@@ -116,6 +125,7 @@ describe("proxy route — handler pass-through", () => {
expect(res.headers.get("content-encoding")).toBeNull();
expect(res.headers.get("content-length")).toBeNull();
+ expect(res.headers.get("authorization")).toBeNull();
expect(res.headers.get("cache-control")).toBe("no-store");
expect(res.headers.get("x-custom-header")).toBe("preserved");
expect(res.headers.get("content-type")).toBe("application/json");
@@ -375,6 +385,137 @@ describe("proxy route — handler pass-through", () => {
);
});
+ it("preserves private video range responses without allowing shared caching", async () => {
+ vi.mocked(fetch).mockResolvedValue(
+ new Response("video bytes", {
+ status: 206,
+ headers: {
+ "Content-Type": "video/mp4",
+ "Content-Length": "11",
+ "Content-Range": "bytes 0-10/100",
+ "Accept-Ranges": "bytes",
+ "Cache-Control": "public, max-age=3600",
+ },
+ }),
+ );
+ const path = [
+ "api",
+ "store",
+ "submissions",
+ "media",
+ "user-123",
+ "videos",
+ "preview.mp4",
+ ];
+ const req = new NextRequest(
+ `https://app.test/api/proxy/${path.join("/")}`,
+ {
+ headers: { Range: "bytes=0-10" },
+ },
+ );
+ const res = await GET(req, makeParams(path));
+
+ const init = vi.mocked(fetch).mock.calls[0][1]!;
+ const sentHeaders = init.headers as Headers;
+ expect(sentHeaders.get("range")).toBe("bytes=0-10");
+ expect(sentHeaders.get("accept-encoding")).toBe("identity");
+ expect(init.cache).toBe("no-store");
+ expect(res.status).toBe(206);
+ expect(await res.text()).toBe("video bytes");
+ expect(res.headers.get("content-length")).toBe("11");
+ expect(res.headers.get("content-range")).toBe("bytes 0-10/100");
+ expect(res.headers.get("accept-ranges")).toBe("bytes");
+ expect(res.headers.get("cache-control")).toBe("private, no-store");
+ expect(res.headers.get("x-content-type-options")).toBe("nosniff");
+ });
+
+ it("forwards only allowlisted headers for a private image", async () => {
+ vi.mocked(fetch).mockResolvedValue(
+ new Response("image", {
+ status: 200,
+ headers: { "Content-Type": "image/png", "Content-Length": "5" },
+ }),
+ );
+ const path = [
+ "api",
+ "store",
+ "submissions",
+ "media",
+ "user-123",
+ "images",
+ "thumbnail.png",
+ ];
+ const req = new NextRequest(
+ `https://app.test/api/proxy/${path.join("/")}`,
+ {
+ headers: {
+ Accept: "image/*",
+ Authorization: "Bearer client-supplied",
+ Range: "bytes=0-10",
+ "X-Forwarded-For": "203.0.113.7",
+ },
+ },
+ );
+ await GET(req, makeParams(path));
+
+ const sentHeaders = vi.mocked(fetch).mock.calls[0][1]!.headers as Headers;
+ expect([...sentHeaders.keys()].sort()).toEqual(PRIVATE_IMAGE_HEADER_KEYS);
+ expect(sentHeaders.get("authorization")).not.toBe("Bearer client-supplied");
+ });
+
+ it("streams private media too large to buffer", async () => {
+ const size = PRIVATE_MEDIA_RANGE_CHUNK_BYTES + 1;
+ vi.mocked(fetch).mockResolvedValue(
+ new Response("large image", {
+ status: 200,
+ headers: {
+ "Content-Type": "image/png",
+ "Content-Length": String(size),
+ },
+ }),
+ );
+ const path = [
+ "api",
+ "store",
+ "submissions",
+ "media",
+ "user-123",
+ "images",
+ "large.png",
+ ];
+ const res = await GET(
+ new NextRequest(`https://app.test/api/proxy/${path.join("/")}`),
+ makeParams(path),
+ );
+
+ expect(res.headers.get("content-length")).toBeNull();
+ expect(res.headers.get("cache-control")).toBe("private, no-store");
+ expect(await res.text()).toBe("large image");
+ });
+
+ it("does not expose private-media proxy exception details", async () => {
+ vi.mocked(fetch).mockRejectedValue(
+ new Error("upstream failed with bearer super-secret"),
+ );
+ vi.spyOn(console, "error").mockImplementation(() => {});
+ const path = [
+ "api",
+ "store",
+ "submissions",
+ "media",
+ "user-123",
+ "images",
+ "thumbnail.png",
+ ];
+ const req = new NextRequest(`https://app.test/api/proxy/${path.join("/")}`);
+ const res = await GET(req, makeParams(path));
+
+ expect(res.status).toBe(502);
+ expect(await res.json()).toEqual({ error: "Proxy request failed" });
+ expect(res.headers.get("cache-control")).toBe("private, no-store");
+ expect(res.headers.get("x-content-type-options")).toBe("nosniff");
+ });
+
it("hardens a public shared-file response after proxying storage", async () => {
vi.mocked(fetch).mockResolvedValue(
new Response("<script>window.name = 'executed'</script>", {
diff --git a/autogpt_platform/frontend/src/app/api/proxy/[...path]/route.helpers.ts b/autogpt_platform/frontend/src/app/api/proxy/[...path]/route.helpers.ts
index d02633d7b5..9f35b49031 100644
--- a/autogpt_platform/frontend/src/app/api/proxy/[...path]/route.helpers.ts
+++ b/autogpt_platform/frontend/src/app/api/proxy/[...path]/route.helpers.ts
@@ -1,5 +1,58 @@
const UUID_RE =
/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
+const SAFE_MEDIA_PATH_COMPONENT_RE = /^(?!\.{1,2}$)[A-Za-z0-9_.-]+$/;
+const SINGLE_BYTE_RANGE_RE =
+ /^bytes=(?:([0-9]{1,15})-([0-9]{0,15})|-([0-9]{1,15}))$/;
+// Private media up to this size is buffered before it is forwarded, below
+// Vercel's 4.5 MB limit for buffered bodies, and each video range is capped to
+// it so every video response stays buffered.
+export const PRIVATE_MEDIA_RANGE_CHUNK_BYTES = 4 * 1024 * 1024;
+
+export function isPrivateStoreMediaRequest(path: string[]): boolean {
+ return (
+ path.length === 7 &&
+ path[0] === "api" &&
+ path[1] === "store" &&
+ path[2] === "submissions" &&
+ path[3] === "media" &&
+ SAFE_MEDIA_PATH_COMPONENT_RE.test(path[4]) &&
+ (path[5] === "images" || path[5] === "videos") &&
+ SAFE_MEDIA_PATH_COMPONENT_RE.test(path[6])
+ );
+}
+
+export function isPrivateStoreVideoRequest(path: string[]): boolean {
+ return isPrivateStoreMediaRequest(path) && path[5] === "videos";
+}
+
+export function getSafePrivateMediaRange(value: string | null): string | null {
+ if (!value) return null;
+ const match = SINGLE_BYTE_RANGE_RE.exec(value.trim());
+ if (!match) return null;
+ const [, startText, endText, suffixText] = match;
+ if (suffixText !== undefined) {
+ const suffix = Math.min(
+ Number(suffixText),
+ PRIVATE_MEDIA_RANGE_CHUNK_BYTES,
+ );
+ return `bytes=-${suffix}`;
+ }
+ const start = Number(startText);
+ const chunkEnd = start + PRIVATE_MEDIA_RANGE_CHUNK_BYTES - 1;
+ const end = endText ? Math.min(Number(endText), chunkEnd) : chunkEnd;
+ return `bytes=${start}-${end}`;
+}
+
+export function shouldBufferPrivateMedia(headers: Headers): boolean {
+ const header = headers.get("content-length");
+ if (!header) return false;
+ const length = Number(header);
+ return (
+ Number.isFinite(length) &&
+ length >= 0 &&
+ length <= PRIVATE_MEDIA_RANGE_CHUNK_BYTES
+ );
+}
export function isWorkspaceDownloadRequest(path: string[]): boolean {
// api/workspace/files/{id}/download
diff --git a/autogpt_platform/frontend/src/app/api/proxy/[...path]/route.ts b/autogpt_platform/frontend/src/app/api/proxy/[...path]/route.ts
index 3bfd19b47b..04fee6c515 100644
--- a/autogpt_platform/frontend/src/app/api/proxy/[...path]/route.ts
+++ b/autogpt_platform/frontend/src/app/api/proxy/[...path]/route.ts
@@ -13,7 +13,11 @@ import {
buildSafeWorkspaceDownloadHeaders,
fetchWorkspaceDownloadWithRetry,
getResponseStartTimeoutMs,
+ getSafePrivateMediaRange,
+ shouldBufferPrivateMedia,
getWorkspaceDownloadErrorMessage,
+ isPrivateStoreMediaRequest,
+ isPrivateStoreVideoRequest,
isWorkspaceDownloadRequest,
watchResponseStart,
} from "./route.helpers";
@@ -79,6 +83,7 @@ const STRIPPED_RESPONSE_HEADERS: ReadonlySet<string> = new Set([
"upgrade",
"content-encoding",
"content-length",
+ "authorization",
"set-cookie",
]);
@@ -127,6 +132,12 @@ function filterResponseHeaders(src: Headers): Headers {
return out;
}
+function hardenPrivateMediaResponseHeaders(headers: Headers): Headers {
+ headers.set("Cache-Control", "private, no-store");
+ headers.set("X-Content-Type-Options", "nosniff");
+ return headers;
+}
+
const METHODS_WITHOUT_BODY: ReadonlySet<string> = new Set([
"GET",
"HEAD",
@@ -207,8 +218,8 @@ async function createWorkspaceDownloadErrorResponse(
* pass through without parse → re-serialise round-trips. Status codes and
* non-hop-by-hop response headers are preserved.
*
- * Workspace file downloads have a dedicated buffering path because Vercel's
- * stream forwarding silently truncates large binary responses.
+ * Workspace file downloads and private store media are buffered because
+ * Vercel's stream forwarding silently truncates large binary responses.
*/
async function handler(
req: NextRequest,
@@ -219,6 +230,7 @@ async function handler(
const queryString = url.search;
const backendUrl = buildBackendUrl(path, queryString);
const method = req.method;
+ const isPrivateMedia = isPrivateStoreMediaRequest(path);
try {
const token = await getServerAuthToken();
@@ -228,6 +240,16 @@ async function handler(
}
const headers = await buildForwardHeaders(req, token);
+ if (isPrivateMedia) {
+ headers.set("cache-control", "no-store");
+ }
+ if (method === "GET" && isPrivateStoreVideoRequest(path)) {
+ const range = getSafePrivateMediaRange(req.headers.get("range"));
+ if (range) {
+ headers.set("range", range);
+ headers.set("accept-encoding", "identity");
+ }
+ }
const hasBody = !METHODS_WITHOUT_BODY.has(method);
// Two-phase timeout: the overall ceiling covers the whole hop, while the
@@ -249,6 +271,7 @@ async function handler(
// ReadableStream body. Cast because TS lib types haven't caught up.
...(responseStart.body ? ({ duplex: "half" } as RequestInit) : {}),
redirect: "manual",
+ ...(isPrivateMedia ? { cache: "no-store" } : {}),
signal: AbortSignal.any([
AbortSignal.timeout(BACKEND_FETCH_TIMEOUT_MS),
responseStart.signal,
@@ -258,17 +281,47 @@ async function handler(
responseStart.clear();
}
+ const responseHeaders = filterResponseHeaders(backendResponse.headers);
+ if (isPrivateMedia) {
+ hardenPrivateMediaResponseHeaders(responseHeaders);
+ // Vercel silently drops the tail of large streamed binary bodies, so
+ // private media is buffered like workspace downloads. Larger images keep
+ // streaming: a buffered body over Vercel's 4.5 MB limit fails outright.
+ if (shouldBufferPrivateMedia(backendResponse.headers)) {
+ const body = await backendResponse.arrayBuffer();
+ responseHeaders.set("content-length", String(body.byteLength));
+ return new NextResponse(body, {
+ status: backendResponse.status,
+ statusText: backendResponse.statusText,
+ headers: responseHeaders,
+ });
+ }
+ }
+
return new NextResponse(backendResponse.body, {
status: backendResponse.status,
statusText: backendResponse.statusText,
- headers: filterResponseHeaders(backendResponse.headers),
+ headers: responseHeaders,
});
} catch (error) {
const timedOut = isTimeoutError(error);
+ const errorMessage = timedOut
+ ? "Proxy request timed out"
+ : "Proxy request failed";
+ if (isPrivateMedia) {
+ console.error(`Private media proxy ${method} failed: ${errorMessage}`);
+ const response = NextResponse.json(
+ { error: errorMessage },
+ { status: timedOut ? 504 : 502 },
+ );
+ hardenPrivateMediaResponseHeaders(response.headers);
+ return response;
+ }
+
console.error(`Proxy error for ${method} /${path.join("/")}:`, error);
return NextResponse.json(
{
- error: timedOut ? "Proxy request timed out" : "Proxy request failed",
+ error: errorMessage,
detail: error instanceof Error ? error.message : "Unknown error",
},
{ status: timedOut ? 504 : 502 },
diff --git a/autogpt_platform/frontend/src/lib/__tests__/store-media-routing.test.ts b/autogpt_platform/frontend/src/lib/__tests__/store-media-routing.test.ts
index 95f21b33e1..f6daeb8914 100644
--- a/autogpt_platform/frontend/src/lib/__tests__/store-media-routing.test.ts
+++ b/autogpt_platform/frontend/src/lib/__tests__/store-media-routing.test.ts
@@ -21,9 +21,11 @@ vi.mock("@/services/environment", () => ({
}));
import { GET } from "@/app/api/proxy/[...path]/route";
+import { getServerAuthToken } from "@/lib/auth/server/getServerAuthToken";
beforeEach(() => {
vi.stubEnv("NEXT_PUBLIC_AGPT_SERVER_URL", "http://localhost:8006/api");
+ vi.mocked(getServerAuthToken).mockResolvedValue(null);
vi.stubGlobal("fetch", vi.fn());
});
afterEach(() => {
@@ -102,6 +104,54 @@ it.each([
},
);
+it("rewrites private media through the authenticated proxy without caching it", async () => {
+ vi.mocked(getServerAuthToken).mockResolvedValue("private-token");
+ const { default: nextConfig } = await vi.importActual<{
+ default: NextConfig;
+ }>(join(process.cwd(), "next.config.mjs"));
+ const relativeURL =
+ "/api/store/submissions/media/user-123/images/thumbnail.png";
+ const browserURL = `http://localhost:3000${relativeURL}?v=2`;
+ const routing = await unstable_getResponseFromNextConfig({
+ url: browserURL,
+ nextConfig,
+ });
+ const proxyURL = getRewrittenUrl(routing);
+ expect(proxyURL).toBe(`http://localhost:3000/api/proxy${relativeURL}?v=2`);
+
+ vi.mocked(fetch).mockResolvedValue(
+ new Response("private image", {
+ headers: {
+ "content-type": "image/png",
+ "cache-control": "public, max-age=3600",
+ },
+ }),
+ );
+ const response = await GET(new NextRequest(proxyURL!), {
+ params: Promise.resolve({
+ path: [
+ "api",
+ "store",
+ "submissions",
+ "media",
+ "user-123",
+ "images",
+ "thumbnail.png",
+ ],
+ }),
+ });
+
+ expect(fetch).toHaveBeenCalledWith(
+ `http://internal-backend:8006${relativeURL}?v=2`,
+ expect.anything(),
+ );
+ const requestHeaders = vi.mocked(fetch).mock.calls[0][1]?.headers as Headers;
+ expect(requestHeaders.get("authorization")).toBe("Bearer private-token");
+ expect(response.headers.get("cache-control")).toBe("private, no-store");
+ expect(response.headers.get("x-content-type-options")).toBe("nosniff");
+ expect(isLocalStoreMediaUrl(relativeURL)).toBe(true);
+});
+
it("does not rewrite unrelated frontend API routes", async () => {
const { default: nextConfig } = await vi.importActual<{
default: NextConfig;
diff --git a/autogpt_platform/frontend/src/lib/__tests__/store-media.test.ts b/autogpt_platform/frontend/src/lib/__tests__/store-media.test.ts
index 6724f5821c..cbea5c4689 100644
--- a/autogpt_platform/frontend/src/lib/__tests__/store-media.test.ts
+++ b/autogpt_platform/frontend/src/lib/__tests__/store-media.test.ts
@@ -3,6 +3,8 @@ import { isLocalStoreMediaUrl } from "@/lib/store-media";
import { isRenderableImageUrl } from "@/lib/next-image";
const mediaPath = "/store/media/user-123/images/thumbnail.png";
+const privateMediaPath =
+ "/store/submissions/media/user-123/images/thumbnail.png";
const defaultImageURLs = [
`http://localhost:8006/api${mediaPath}`,
`http://localhost:8006/api${mediaPath}?version=2#preview`,
@@ -11,6 +13,12 @@ const defaultImageURLs = [
`http://localhost:3000/api${mediaPath}`,
`/api/proxy/api${mediaPath}`,
`http://localhost:3000/api/proxy/api${mediaPath}`,
+ `http://localhost:8006/api${privateMediaPath}`,
+ `http://localhost:8006/api${privateMediaPath}?version=2#preview`,
+ `/api${privateMediaPath}`,
+ `http://localhost:3000/api${privateMediaPath}`,
+ `/api/proxy/api${privateMediaPath}`,
+ `http://localhost:3000/api/proxy/api${privateMediaPath}`,
];
beforeEach(() => {
@@ -148,6 +156,10 @@ describe("malformed and ordinary URLs", () => {
"/images/thumbnail.png",
"https://cdn.example.com/images/thumbnail.png",
"/api/store/media/user-123/videos/preview.mp4",
+ "/api/store/submissions/media/user-123/videos/preview.mp4",
+ "/api/store/submissions/media/user-123/images/thumbnail.png/extra",
+ "/api/store/submissions/media//images/thumbnail.png",
+ "/api/store/submissions/media/user-123/images/%2e%2e",
"/api/store/media/user-123/images/thumbnail.png/extra",
"/api/store/media/user-123/images/thumbnail.png/",
"/api/store/media/user-123/images/",
diff --git a/autogpt_platform/frontend/src/lib/direct-upload.ts b/autogpt_platform/frontend/src/lib/direct-upload.ts
index fe15f566fc..14e5b6e837 100644
--- a/autogpt_platform/frontend/src/lib/direct-upload.ts
+++ b/autogpt_platform/frontend/src/lib/direct-upload.ts
@@ -141,7 +141,7 @@ export async function uploadFileDirect(
/**
* Uploads store submission media (agent thumbnails, profile avatars) directly
- * to the backend. Returns the public URL of the stored media.
+ * to private storage through the backend. Returns its authenticated media URL.
*/
export async function uploadSubmissionMediaDirect(
file: File,
diff --git a/autogpt_platform/frontend/src/lib/store-media.ts b/autogpt_platform/frontend/src/lib/store-media.ts
index 421e2e94f1..c8b77ab22e 100644
--- a/autogpt_platform/frontend/src/lib/store-media.ts
+++ b/autogpt_platform/frontend/src/lib/store-media.ts
@@ -1,5 +1,7 @@
const LOCAL_STORE_IMAGE_PATH =
- /\/api\/store\/media\/(?!\.{1,2}\/)[A-Za-z0-9_.-]+\/images\/(?!\.{1,2}$)[A-Za-z0-9_.-]+$/;
+ /\/api\/store\/(?:submissions\/)?media\/(?!\.{1,2}\/)[A-Za-z0-9_.-]+\/images\/(?!\.{1,2}$)[A-Za-z0-9_.-]+$/;
+
+const STORE_MEDIA_PATH_MARKERS = ["/store/submissions/media/", "/store/media/"];
export function isLocalStoreMediaUrl(src: string | null | undefined): boolean {
if (!src || src.startsWith("//")) return false;
@@ -19,10 +21,12 @@ export function isLocalStoreMediaUrl(src: string | null | undefined): boolean {
return false;
}
- const apiPath = url.pathname.slice(
- 0,
- url.pathname.lastIndexOf("/store/media/"),
+ const marker = STORE_MEDIA_PATH_MARKERS.find((candidate) =>
+ url.pathname.includes(candidate),
);
+ if (!marker) return false;
+
+ const apiPath = url.pathname.slice(0, url.pathname.lastIndexOf(marker));
if (
url.origin === frontendURL.origin &&
(apiPath === "/api" || apiPath === "/api/proxy/api")
diff --git a/docs/engineering/workspace-media-architecture.md b/docs/engineering/workspace-media-architecture.md
index f5e9f6d60a..da6b7617eb 100644
--- a/docs/engineering/workspace-media-architecture.md
+++ b/docs/engineering/workspace-media-architecture.md
@@ -122,9 +122,69 @@ WorkspaceManager delegates to `WorkspaceStorageBackend`:
| Backend | When Used | Storage Path Format |
|---------|-----------|---------------------|
-| `GCSWorkspaceStorage` | `media_gcs_bucket_name` is configured | `gcs://bucket/workspaces/{ws_id}/{file_id}/{filename}` |
+| `GCSWorkspaceStorage` | `private_user_data_bucket` (or the legacy bucket fallback) is configured | `gcs://bucket/workspaces/{ws_id}/{file_id}/{filename}` |
| `LocalWorkspaceStorage` | No GCS bucket configured | `local://{ws_id}/{file_id}/{filename}` |
+### Public and private media
+
+Hosted storage has two trust boundaries:
+
+- `PRIVATE_USER_DATA_BUCKET` is the default destination for user uploads,
+ generated library images, custom Expert avatars, workspaces, transcripts,
+ and temporary agent inputs. Private media is read through an authenticated,
+ non-cacheable API that serves a file to its owner, admins and members of
+ an organization the owner belongs to.
+- `PUBLIC_SITE_MEDIA_BUCKET` contains only objects that a trusted publication
+ flow explicitly copied after approval, plus media whose purpose is inherently
+ public such as OAuth consent-screen logos. Anonymous exact-object reads are
+ allowed, but anonymous bucket listing is not.
+
+An upload caller cannot select the public destination. Publication is a
+separate privileged operation; sharing a private resource grants access through
+its opaque application URL and does not make its storage prefix public.
+
+### Moving a single-bucket deployment to split buckets
+
+Stored rows hold `gcs://<bucket>/...` paths and full GCS URLs, and the storage
+code only reads from the configured private bucket, so the existing bucket must
+become the private one:
+
+1. Create the new public bucket. Deploy with `PRIVATE_USER_DATA_BUCKET` set to
+ the existing `MEDIA_GCS_BUCKET_NAME` bucket and `PUBLIC_SITE_MEDIA_BUCKET`
+ set to the new one. Pointing `PRIVATE_USER_DATA_BUCKET` at a new bucket
+ instead makes every existing workspace file and transcript unreadable.
+ Deploy the frontend first: once the backend has both names set, uploads
+ return `/api/store/submissions/media/...`, which a frontend without the new
+ rewrite answers with a 404.
+ If anonymous users hold `roles/storage.objectViewer` on the old bucket,
+ they can list every object in it until step 4. Swap that binding for
+ `roles/storage.legacyObjectReader` first: existing links keep working and
+ listing stops.
+2. Copy everything that is already public to the public bucket and repoint its
+ rows: `poetry run python scripts/publish_live_media.py` (dry run), then
+ again with `--apply` until it exits 0. A non-zero exit means a live
+ reference was not published (copy failure, conflict, an object outside the
+ listing's owners or an unrecognised URL) and would break in step 4. This
+ covers approved listing media, the avatars of creators with a public
+ listing, library copies of listing images and OAuth app logos. It is also
+ the repair tool when a copy at approval time failed.
+3. Rewrite the remaining private media URLs to the authenticated API path:
+ `poetry run python scripts/backfill_private_media_urls.py` (dry run), then
+ `--apply`, re-running until it reports no conflicts. It exits 2 while any
+ reference stays on the old bucket (held as public, cross-user, ambiguous,
+ malformed or unrecognised): those stop loading in step 4, so check the
+ counts before going on. Both scripts commit in small batches and can be
+ re-run.
+4. Run step 2's dry run once more and check it exits 0. Then remove every
+ public binding from the old bucket and turn on public access prevention, so
+ no object-level grant can expose a file again.
+
+The private media endpoint serves a file to its owner, to admins and to
+members of an organization the owner belongs to, so a leaked URL is useless to
+anyone else. Published copies are never deleted automatically: when a listing
+is taken down or a creator changes their avatar, the old public copy stays in
+the public bucket until someone removes it by hand.
+
---
## store_media_file()
@@ -333,7 +393,9 @@ async def upload_file(file: UploadFile, user_id: str, workspace_id: str):
| Setting | Purpose | Default |
|---------|---------|---------|
-| `media_gcs_bucket_name` | GCS bucket for workspace storage | None (uses local) |
+| `private_user_data_bucket` | Private GCS bucket for user media, workspace storage, transcripts, and temporary uploads | Falls back to `media_gcs_bucket_name`, then local storage |
+| `public_site_media_bucket` | Public GCS bucket for explicitly published marketplace media and OAuth app logos | Falls back to `media_gcs_bucket_name` for compatibility |
+| `media_gcs_bucket_name` | Legacy single-bucket setting for self-hosted compatibility | None |
| `workspace_storage_dir` | Local storage directory | `{app_data}/workspaces` |
| `max_file_size_mb` | Maximum file size in MB | 100 |
| `clamav_service_enabled` | Enable virus scanning | true |