Open-source Python library for AI browser agents, plus Browser Use Cloud: stealth browsers and a hosted agent API.
Python ai-agentsai-toolsbrowser-agentbrowser-automationbrowser-usecdpcloud-browsercomputer-useheadless-browserllmmcpplaywrightpythonremote-browserstealth-browserweb-automationweb-scraping
added chatOciRaw
Sandeep3eee52338d8e
diff --git a/browser_use/llm/oci_raw/README.md b/browser_use/llm/oci_raw/README.mdnew file mode 100644index 000000000..5122d7005--- /dev/null+++ b/browser_use/llm/oci_raw/README.md@@ -0,0 +1,157 @@+# OCI Raw API Integration++This module provides direct integration with Oracle Cloud Infrastructure's Generative AI service using raw API calls, without Langchain dependencies.++## Features++- **Direct API Integration**: Uses OCI's native Python SDK for direct API calls+- **Async Support**: Full async/await support for non-blocking operations+- **Structured Output**: Support for Pydantic model validation of responses+- **Error Handling**: Comprehensive error handling with proper exception types+- **Authentication**: Support for multiple OCI authentication methods++## Installation++Make sure you have the required OCI dependencies installed:++```bash+pip install oci+```++## Usage++### Basic Usage++```python+from browser_use import Agent+from browser_use.llm import ChatOCIRaw++# Configure the model+model = ChatOCIRaw(+ model_id="ocid1.generativeaimodel.oc1.us-chicago-1.amaaaaaask7dceya...",+ service_endpoint="https://inference.generativeai.us-chicago-1.oci.oraclecloud.com",+ compartment_id="ocid1.tenancy.oc1..aaaaaaaayeiis5uk2nuubznrekd...",+ provider="meta", # or "cohere"+ temperature=1.0,+ max_tokens=600,+ top_p=0.75,+ auth_type="API_KEY",+ auth_profile="DEFAULT"+)++# Use with browser-use Agent+agent = Agent(+ task="Search for Python tutorials and summarize them",+ llm=model+)++history = await agent.run()+```++### Structured Output++```python+from pydantic import BaseModel++class SearchResult(BaseModel):+ title: str+ summary: str+ relevance_score: float++# Use structured output+response = await model.ainvoke(messages, output_format=SearchResult)+result = response.completion # This is a SearchResult instance+```++## Configuration++### Authentication Types++The integration supports multiple OCI authentication methods:++- `API_KEY`: Uses API key authentication (default)+- `INSTANCE_PRINCIPAL`: Uses instance principal authentication+- `RESOURCE_PRINCIPAL`: Uses resource principal authentication++### Model Parameters++- `model_id`: The OCID of your OCI GenAI model+- `service_endpoint`: The OCI service endpoint URL+- `compartment_id`: The OCID of your OCI compartment+- `provider`: Model provider ("meta" or "cohere")+- `temperature`: Response randomness (0.0-2.0)+- `max_tokens`: Maximum tokens in response+- `top_p`: Top-p sampling parameter+- `frequency_penalty`: Frequency penalty for repetition+- `presence_penalty`: Presence penalty for repetition++## Error Handling++The integration provides proper error handling with specific exception types:++- `ModelRateLimitError`: For rate limiting (429 errors)+- `ModelProviderError`: For other API errors (4xx, 5xx)++## Comparison with Langchain Integration++| Feature | OCI Raw API | Langchain Integration |+|---------|-------------|----------------------|+| Dependencies | OCI SDK only | Langchain + OCI SDK |+| Performance | Direct API calls | Additional abstraction layer |+| Control | Full control over requests | Limited by Langchain interface |+| Updates | Direct OCI SDK updates | Dependent on Langchain updates |+| Complexity | Lower complexity | Higher complexity |++## Example Response Format++The OCI GenAI API returns responses in this format:++```json+{+ "chat_response": {+ "api_format": "GENERIC",+ "choices": [+ {+ "finish_reason": "stop",+ "index": 0,+ "message": {+ "content": [+ {+ "text": "Response text here",+ "type": "TEXT"+ }+ ],+ "role": "ASSISTANT"+ }+ }+ ],+ "usage": {+ "completion_tokens": 18,+ "prompt_tokens": 38,+ "total_tokens": 56+ }+ }+}+```++## Troubleshooting++### Common Issues++1. **Authentication Errors**: Ensure your OCI configuration is correct and you have the necessary permissions+2. **Model Not Found**: Verify your model OCID and ensure it's available in your compartment+3. **Rate Limiting**: The integration handles rate limits automatically with proper error types++### Debug Mode++Enable verbose logging by setting the `verbose` parameter to `True` (not implemented in this version but can be added).++## Contributing++When contributing to this module:++1. Follow the existing code style+2. Add proper type hints+3. Include comprehensive error handling+4. Add tests for new features+5. Update documentationdiff --git a/browser_use/llm/oci_raw/__init__.py b/browser_use/llm/oci_raw/__init__.pynew file mode 100644index 000000000..ecf250c45--- /dev/null+++ b/browser_use/llm/oci_raw/__init__.py@@ -0,0 +1,10 @@+"""+OCI Raw API integration for browser-use.++This module provides direct integration with Oracle Cloud Infrastructure's+Generative AI service using the raw API endpoints, without Langchain dependencies.+"""++from .chat import ChatOCIRaw++__all__ = ['ChatOCIRaw']diff --git a/browser_use/llm/oci_raw/chat.py b/browser_use/llm/oci_raw/chat.pynew file mode 100644index 000000000..afe6982e3--- /dev/null+++ b/browser_use/llm/oci_raw/chat.py@@ -0,0 +1,378 @@+"""+OCI Raw API chat model integration for browser-use.++This module provides direct integration with Oracle Cloud Infrastructure's+Generative AI service using raw API calls without Langchain dependencies.+"""++import json+import asyncio+from dataclasses import dataclass+from typing import Any, Dict, TypeVar, overload++import oci+from oci.generative_ai_inference import GenerativeAiInferenceClient+from oci.generative_ai_inference.models import (+ ChatDetails,+ GenericChatRequest,+ OnDemandServingMode,+ BaseChatRequest+)+from pydantic import BaseModel++from browser_use.llm.base import BaseChatModel+from browser_use.llm.exceptions import ModelProviderError, ModelRateLimitError+from browser_use.llm.messages import BaseMessage+from .serializer import OCIRawMessageSerializer+from browser_use.llm.schema import SchemaOptimizer+from browser_use.llm.views import ChatInvokeCompletion, ChatInvokeUsage++T = TypeVar('T', bound=BaseModel)+++@dataclass+class ChatOCIRaw(BaseChatModel):+ """+ A direct OCI Raw API integration for browser-use that bypasses Langchain.++ This class provides a browser-use compatible interface for OCI GenAI models+ using direct API calls to Oracle Cloud Infrastructure.++ Args:+ model_id: The OCI GenAI model OCID+ service_endpoint: The OCI service endpoint URL+ compartment_id: The OCI compartment OCID+ provider: The model provider (e.g., "meta", "cohere" ,"grok")+ temperature: Temperature for response generation (0.0-2.0)+ max_tokens: Maximum tokens in response+ frequency_penalty: Frequency penalty for response generation+ presence_penalty: Presence penalty for response generation+ top_p: Top-p sampling parameter+ auth_type: Authentication type (e.g., "API_KEY")+ auth_profile: Authentication profile name+ timeout: Request timeout in seconds+ """++ # Model configuration+ model_id: str+ service_endpoint: str+ compartment_id: str+ provider: str = "meta"++ # Model parameters+ temperature: float | None = 1.0+ max_tokens: int | None = 600+ frequency_penalty: float | None = 0.0+ presence_penalty: float | None = 0.0+ top_p: float | None = 0.75++ # Authentication+ auth_type: str = "API_KEY"+ auth_profile: str = "DEFAULT"++ # Client configuration+ timeout: float = 60.0++ # Static properties+ @property+ def provider_name(self) -> str:+ return 'oci-raw'++ @property+ def name(self) -> str:+ # Return a shorter name for telemetry (max 100 chars)+ if len(self.model_id) > 90:+ # Extract the model name from the OCID+ parts = self.model_id.split('.')+ if len(parts) >= 4:+ return f"oci-{self.provider}-{parts[3]}" # e.g., "oci-meta-us-chicago-1"+ else:+ return f"oci-{self.provider}-model"+ return self.model_id++ @property+ def model(self) -> str:+ return self.model_id++ @property+ def model_name(self) -> str:+ # Override for telemetry - return shorter name (max 100 chars)+ if len(self.model_id) > 90:+ # Extract the model name from the OCID+ parts = self.model_id.split('.')+ if len(parts) >= 4:+ return f"oci-{self.provider}-{parts[3]}" # e.g., "oci-meta-us-chicago-1"+ else:+ return f"oci-{self.provider}-model"+ return self.model_id++ def _get_oci_client(self) -> GenerativeAiInferenceClient:+ """Get the OCI GenerativeAiInferenceClient following your working example."""+ if not hasattr(self, '_client'):+ # Configure OCI client based on auth_type (following your working example)+ if self.auth_type == "API_KEY":+ config = oci.config.from_file('~/.oci/config', self.auth_profile)+ self._client = GenerativeAiInferenceClient(+ config=config,+ service_endpoint=self.service_endpoint,+ retry_strategy=oci.retry.NoneRetryStrategy(),+ timeout=(10, 240) # Following your working example+ )+ elif self.auth_type == "INSTANCE_PRINCIPAL":+ config = {}+ signer = oci.auth.signers.InstancePrincipalsSecurityTokenSigner()+ self._client = GenerativeAiInferenceClient(+ config=config,+ signer=signer,+ service_endpoint=self.service_endpoint,+ retry_strategy=oci.retry.NoneRetryStrategy(),+ timeout=(10, 240)+ )+ elif self.auth_type == "RESOURCE_PRINCIPAL":+ config = {}+ signer = oci.auth.signers.get_resource_principals_signer()+ self._client = GenerativeAiInferenceClient(+ config=config,+ signer=signer,+ service_endpoint=self.service_endpoint,+ retry_strategy=oci.retry.NoneRetryStrategy(),+ timeout=(10, 240)+ )+ else:+ # Fallback to API_KEY+ config = oci.config.from_file('~/.oci/config', self.auth_profile)+ self._client = GenerativeAiInferenceClient(+ config=config,+ service_endpoint=self.service_endpoint,+ retry_strategy=oci.retry.NoneRetryStrategy(),+ timeout=(10, 240)+ )++ return self._client++ def _extract_usage(self, response) -> ChatInvokeUsage | None:+ """Extract usage information from OCI response."""+ try:+ # The response is the direct OCI response object, not a dict+ if hasattr(response, 'data') and hasattr(response.data, 'chat_response'):+ chat_response = response.data.chat_response+ if hasattr(chat_response, 'usage'):+ usage = chat_response.usage+ return ChatInvokeUsage(+ prompt_tokens=getattr(usage, 'prompt_tokens', 0),+ prompt_cached_tokens=None,+ prompt_cache_creation_tokens=None,+ prompt_image_tokens=None,+ completion_tokens=getattr(usage, 'completion_tokens', 0),+ total_tokens=getattr(usage, 'total_tokens', 0),+ )+ return None+ except Exception:+ return None++ def _extract_content(self, response) -> str:+ """Extract text content from OCI response."""+ try:+ # The response is the direct OCI response object, not a dict+ if not hasattr(response, 'data'):+ raise ModelProviderError(+ message="Invalid response format: no data attribute",+ status_code=500,+ model=self.name+ )++ chat_response = response.data.chat_response+ if not hasattr(chat_response, 'choices') or not chat_response.choices:+ raise ModelProviderError(+ message="No choices in response",+ status_code=500,+ model=self.name+ )++ choice = chat_response.choices[0]+ message = choice.message+ content_parts = message.content++ # Extract text from content parts+ text_parts = []+ for part in content_parts:+ if hasattr(part, 'text'):+ text_parts.append(part.text)++ return '\n'.join(text_parts) if text_parts else ''++ except Exception as e:+ raise ModelProviderError(+ message=f"Failed to extract content from response: {str(e)}",+ status_code=500,+ model=self.name+ ) from e++ async def _make_request(self, messages: list[BaseMessage]):+ """Make async request to OCI API using proper OCI SDK models."""+ # Serialize messages to OCI Message objects+ oci_messages = OCIRawMessageSerializer.serialize_messages(messages)++ # Create chat request using proper OCI SDK models (following your working example)+ chat_request = GenericChatRequest()+ chat_request.api_format = BaseChatRequest.API_FORMAT_GENERIC+ chat_request.messages = oci_messages+ chat_request.max_tokens = self.max_tokens+ chat_request.temperature = self.temperature+ chat_request.frequency_penalty = self.frequency_penalty+ chat_request.presence_penalty = self.presence_penalty+ chat_request.top_p = self.top_p++ # Create serving mode+ serving_mode = OnDemandServingMode(model_id=self.model_id)++ # Create chat details+ chat_details = ChatDetails()+ chat_details.serving_mode = serving_mode+ chat_details.chat_request = chat_request+ chat_details.compartment_id = self.compartment_id++ # Make the request in a thread to avoid blocking+ def _sync_request():+ try:+ client = self._get_oci_client()+ response = client.chat(chat_details)+ return response # Return the raw response object+ except Exception as e:+ # Handle OCI-specific exceptions+ status_code = getattr(e, 'status', 500)+ if status_code == 429:+ raise ModelRateLimitError(+ message=f"Rate limit exceeded: {str(e)}",+ model=self.name+ ) from e+ else:+ raise ModelProviderError(+ message=str(e),+ status_code=status_code,+ model=self.name+ ) from e++ # Run in thread pool to make it async+ loop = asyncio.get_event_loop()+ return await loop.run_in_executor(None, _sync_request)++ @overload+ async def ainvoke(self, messages: list[BaseMessage], output_format: None = None) -> ChatInvokeCompletion[str]: ...++ @overload+ async def ainvoke(self, messages: list[BaseMessage], output_format: type[T]) -> ChatInvokeCompletion[T]: ...++ async def ainvoke(+ self, messages: list[BaseMessage], output_format: type[T] | None = None+ ) -> ChatInvokeCompletion[T] | ChatInvokeCompletion[str]:+ """+ Invoke the OCI GenAI model with the given messages using raw API.++ Args:+ messages: List of chat messages+ output_format: Optional Pydantic model class for structured output++ Returns:+ Either a string response or an instance of output_format+ """+ try:+ if output_format is None:+ # Return string response+ response = await self._make_request(messages)+ content = self._extract_content(response)+ usage = self._extract_usage(response)++ return ChatInvokeCompletion(+ completion=content,+ usage=usage,+ )+ else:+ # For structured output, add JSON schema instructions+ optimized_schema = SchemaOptimizer.create_optimized_json_schema(output_format)++ # Add JSON schema instruction to messages+ system_instruction = f"""+You must respond with ONLY a valid JSON object that matches this exact schema:+{json.dumps(optimized_schema, indent=2)}++IMPORTANT:+- Your response must be ONLY the JSON object, no additional text+- The JSON must be valid and parseable+- All required fields must be present+- No extra fields are allowed+- Use proper JSON syntax with double quotes+"""++ # Clone messages and add system instruction+ modified_messages = messages.copy()++ # Add or modify system message+ from browser_use.llm.messages import SystemMessage+ if modified_messages and hasattr(modified_messages[0], 'role') and modified_messages[0].role == 'system':+ # Modify existing system message+ existing_content = modified_messages[0].content+ if isinstance(existing_content, str):+ modified_messages[0].content = existing_content + "\n\n" + system_instruction+ else:+ # Handle list content+ modified_messages[0].content = str(existing_content) + "\n\n" + system_instruction+ else:+ # Insert new system message at the beginning+ modified_messages.insert(0, SystemMessage(content=system_instruction))++ response = await self._make_request(modified_messages)+ response_text = self._extract_content(response)++ # Clean and parse the JSON response+ try:+ # Clean the response text+ cleaned_text = response_text.strip()++ # Remove markdown code blocks if present+ if cleaned_text.startswith('```json'):+ cleaned_text = cleaned_text[7:]+ if cleaned_text.startswith('```'):+ cleaned_text = cleaned_text[3:]+ if cleaned_text.endswith('```'):+ cleaned_text = cleaned_text[:-3]++ cleaned_text = cleaned_text.strip()++ # Try to find JSON object in the response+ if not cleaned_text.startswith('{'):+ start_idx = cleaned_text.find('{')+ end_idx = cleaned_text.rfind('}')+ if start_idx != -1 and end_idx != -1 and end_idx > start_idx:+ cleaned_text = cleaned_text[start_idx:end_idx + 1]++ # Parse the JSON+ parsed_data = json.loads(cleaned_text)+ parsed = output_format.model_validate(parsed_data)++ usage = self._extract_usage(response)+ return ChatInvokeCompletion(+ completion=parsed,+ usage=usage,+ )++ except (json.JSONDecodeError, ValueError) as e:+ raise ModelProviderError(+ message=f'Failed to parse structured output: {str(e)}. Response was: {response_text[:200]}...',+ status_code=500,+ model=self.name,+ ) from e++ except ModelRateLimitError:+ # Re-raise rate limit errors as-is+ raise+ except ModelProviderError:+ # Re-raise provider errors as-is+ raise+ except Exception as e:+ # Handle any other exceptions+ raise ModelProviderError(+ message=f"Unexpected error: {str(e)}",+ status_code=500,+ model=self.name,+ ) from ediff --git a/browser_use/llm/oci_raw/serializer.py b/browser_use/llm/oci_raw/serializer.pynew file mode 100644index 000000000..4ac3b90e9--- /dev/null+++ b/browser_use/llm/oci_raw/serializer.py@@ -0,0 +1,151 @@+"""+Message serializer for OCI Raw API integration.++This module handles the conversion between browser-use message formats+and the OCI Raw API message format using proper OCI SDK models.+"""++from typing import List++from oci.generative_ai_inference.models import Message, TextContent, ImageContent, ImageUrl+from browser_use.llm.messages import (+ BaseMessage,+ UserMessage,+ SystemMessage,+ AssistantMessage,+ ContentPartImageParam,+)+++class OCIRawMessageSerializer:+ """+ Serializer for converting between browser-use message types and OCI Raw API message formats.+ Uses proper OCI SDK model objects as shown in the working example.+ """++ @staticmethod+ def _is_base64_image(url: str) -> bool:+ """Check if the URL is a base64 encoded image."""+ return url.startswith('data:image/')++ @staticmethod+ def _parse_base64_url(url: str) -> str:+ """Parse base64 URL and return the base64 data."""+ if not OCIRawMessageSerializer._is_base64_image(url):+ raise ValueError(f"Not a base64 image URL: {url}")++ # Extract the base64 data from data:image/png;base64,<data>+ try:+ header, data = url.split(',', 1)+ return data+ except ValueError:+ raise ValueError(f"Invalid base64 image URL format: {url}")++ @staticmethod+ def _create_image_content(part: ContentPartImageParam) -> ImageContent:+ """Convert ContentPartImageParam to OCI ImageContent."""+ url = part.image_url.url++ if OCIRawMessageSerializer._is_base64_image(url):+ # Handle base64 encoded images - OCI expects data URLs as-is+ image_url = ImageUrl(url=url)+ else:+ # Handle regular URLs+ image_url = ImageUrl(url=url)++ return ImageContent(image_url=image_url)++ @staticmethod+ def serialize_messages(messages: List[BaseMessage]) -> List[Message]:+ """+ Serialize a list of browser-use messages to OCI Raw API Message objects.++ Args:+ messages: List of browser-use messages++ Returns:+ List of OCI Message objects+ """+ oci_messages = []++ for message in messages:+ oci_message = Message()++ if isinstance(message, UserMessage):+ oci_message.role = "USER"+ content = message.content+ if isinstance(content, str):+ text_content = TextContent()+ text_content.text = content+ oci_message.content = [text_content]+ elif isinstance(content, list):+ # Handle content parts - text and images+ contents = []+ for part in content:+ if part.type == 'text':+ text_content = TextContent()+ text_content.text = part.text+ contents.append(text_content)+ elif part.type == 'image_url':+ image_content = OCIRawMessageSerializer._create_image_content(part)+ contents.append(image_content)+ if contents:+ oci_message.content = contents++ elif isinstance(message, SystemMessage):+ oci_message.role = "SYSTEM"+ content = message.content+ if isinstance(content, str):+ text_content = TextContent()+ text_content.text = content+ oci_message.content = [text_content]+ elif isinstance(content, list):+ # Handle content parts - typically just text for system messages+ contents = []+ for part in content:+ if part.type == 'text':+ text_content = TextContent()+ text_content.text = part.text+ contents.append(text_content)+ elif part.type == 'image_url':+ # System messages can theoretically have images too+ image_content = OCIRawMessageSerializer._create_image_content(part)+ contents.append(image_content)+ if contents:+ oci_message.content = contents++ elif isinstance(message, AssistantMessage):+ oci_message.role = "ASSISTANT"+ content = message.content+ if isinstance(content, str):+ text_content = TextContent()+ text_content.text = content+ oci_message.content = [text_content]+ elif isinstance(content, list):+ # Handle content parts - text, images, and refusals+ contents = []+ for part in content:+ if part.type == 'text':+ text_content = TextContent()+ text_content.text = part.text+ contents.append(text_content)+ elif part.type == 'image_url':+ # Assistant messages can have images in responses+ image_content = OCIRawMessageSerializer._create_image_content(part)+ contents.append(image_content)+ elif part.type == 'refusal':+ text_content = TextContent()+ text_content.text = f"[Refusal] {part.refusal}"+ contents.append(text_content)+ if contents:+ oci_message.content = contents+ else:+ # Fallback for any message format issues+ oci_message.role = "USER"+ text_content = TextContent()+ text_content.text = str(message)+ oci_message.content = [text_content]++ oci_messages.append(oci_message)++ return oci_messages