Added support for Google Custom Search API

This pull request adds functionality to the project to allow for a choice between the original Google search method and the Google Custom Search API. The google_search method uses the original method of scraping the HTML from the search results page, using googlesearch-python, while the google_official_search method uses the Google Custom Search API to retrieve search results.

How to test:

To test the functionality, ensure that you have valid API keys and search engine IDs for both the Google search method and the Google Custom Search API. You can set these values in your environment variables as described in the README.md file.

Additional Notes:

This pull request only adds functionality and makes improvements to existing code. No new features or major changes have been introduced.

.env.template

@@ -1,4 +0,0 @@
-OPENAI_API_KEY=your-openai-api-key
-ELEVENLABS_API_KEY=your-elevenlabs-api-key
-SMART_LLM_MODEL="gpt-4"
-FAST_LLM_MODEL="gpt-3.5-turbo"

README.md

@@ -29,13 +29,22 @@ Your support is greatly appreciated
 
 ## Table of Contents
 
-- [Features](#-features)
-- [Requirements](#-requirements)
-- [Installation](#-installation)
-- [Usage](#-usage)
-- [Limitations](#-limitations)
-- [Disclaimer](#-disclaimer)
-- [Connect with Us on Twitter ](#-connect-with-us-on-twitter)
+- [Auto-GPT: An Autonomous GPT-4 Experiment](#auto-gpt-an-autonomous-gpt-4-experiment)
+    - [Demo (30/03/2023):](#demo-30032023)
+  - [💖 Help Fund Auto-GPT's Development](#-help-fund-auto-gpts-development)
+  - [Table of Contents](#table-of-contents)
+  - [🚀 Features](#-features)
+  - [📋 Requirements](#-requirements)
+  - [💾 Installation](#-installation)
+  - [🔧 Usage](#-usage)
+  - [🗣️ Speech Mode](#️-speech-mode)
+  - [🔍 Google API Keys Configuration](#-google-api-keys-configuration)
+    - [Setting up environment variables](#setting-up-environment-variables)
+  - [💀 Continuous Mode ⚠️](#-continuous-mode-️)
+  - [GPT3.5 ONLY Mode](#gpt35-only-mode)
+  - [⚠️ Limitations](#️-limitations)
+  - [🛡 Disclaimer](#-disclaimer)
+  - [🐦 Connect with Us on Twitter](#-connect-with-us-on-twitter)
 
 
 ## 🚀 Features
@@ -100,6 +109,35 @@ python scripts/main.py --speak
 
 ```
 
+## 🔍 Google API Keys Configuration
+
+This section is optional, use the official google api if you are having issues with error 429 when running google search.
+To use the `google_official_search` command, you need to set up your Google API keys in your environment variables.
+
+1. Go to the [Google Cloud Console](https://console.cloud.google.com/).
+2. If you don't already have an account, create one and log in.
+3. Create a new project by clicking on the "Select a Project" dropdown at the top of the page and clicking "New Project". Give it a name and click "Create".
+4. Go to the [APIs & Services Dashboard](https://console.cloud.google.com/apis/dashboard) and click "Enable APIs and Services". Search for "Custom Search API" and click on it, then click "Enable".
+5. Go to the [Credentials](https://console.cloud.google.com/apis/credentials) page and click "Create Credentials". Choose "API Key".
+6. Copy the API key and set it as an environment variable named `GOOGLE_API_KEY` on your machine. See setting up environment variables below.
+7. Go to the [Custom Search Engine](https://cse.google.com/cse/all) page and click "Add".
+8. Set up your search engine by following the prompts. You can choose to search the entire web or specific sites.
+9.  Once you've created your search engine, click on "Control Panel" and then "Basics". Copy the "Search engine ID" and set it as an environment variable named `CUSTOM_SEARCH_ENGINE_ID` on your machine. See setting up environment variables below.
+
+### Setting up environment variables
+   For Windows Users:
+```
+setx GOOGLE_API_KEY "YOUR_GOOGLE_API_KEY"
+setx CUSTOM_SEARCH_ENGINE_ID "YOUR_CUSTOM_SEARCH_ENGINE_ID"
+
+```
+For macOS and Linux users:
+```
+export GOOGLE_API_KEY="YOUR_GOOGLE_API_KEY"
+export CUSTOM_SEARCH_ENGINE_ID="YOUR_CUSTOM_SEARCH_ENGINE_ID"
+
+```
+
 ## 💀 Continuous Mode ⚠️
 Run the AI **without** user authorisation, 100% automated.
 Continuous mode is not recommended. 

requirements.txt

@@ -9,4 +9,5 @@ requests
 tiktoken==0.3.3
 docker
 googlesearch-python
+google-api-python-client # If using Google's Custom Search JSON API (https://developers.google.com/custom-search/v1/overview) Won't result in a 403 error
 # Googlesearch python seems to be a bit cursed, anyone good at fixing thigns like this?

scripts/commands.py

@@ -10,6 +10,9 @@ from file_operations import read_file, write_to_file, append_to_file, delete_fil
 from execute_code import execute_python_file
 from json_parser import fix_and_parse_json
 from googlesearch import search
+from googleapiclient.discovery import build
+from googleapiclient.errors import HttpError
+
 cfg = Config()
 
 
@@ -44,6 +47,12 @@ def get_command(response):
 def execute_command(command_name, arguments):
     try:
         if command_name == "google":
+            print("Using Google search method")
+            # Check if the Google API key is set and use the official search method
+            # If the API key is not set or has only whitespaces, use the unofficial search method
+            if cfg.google_api_key and (cfg.google_api_key.strip() if cfg.google_api_key else None):
+                return google_official_search(arguments["input"])
+            else:
                 return google_search(arguments["input"])
         elif command_name == "memory_add":
             return commit_memory(arguments["string"])
@@ -108,6 +117,40 @@ def google_search(query, num_results=8):
 
     return json.dumps(search_results, ensure_ascii=False, indent=4)
 
+def google_official_search(query, num_results=8):
+    from googleapiclient.discovery import build
+    from googleapiclient.errors import HttpError
+    import json
+
+    try:
+        # Get the Google API key and Custom Search Engine ID from the config file
+        api_key = cfg.google_api_key
+        custom_search_engine_id = cfg.custom_search_engine_id
+
+        # Initialize the Custom Search API service
+        service = build("customsearch", "v1", developerKey=api_key)
+        
+        # Send the search query and retrieve the results
+        result = service.cse().list(q=query, cx=custom_search_engine_id, num=num_results).execute()
+
+        # Extract the search result items from the response
+        search_results = result.get("items", [])
+        
+        # Create a list of only the URLs from the search results
+        search_results_links = [item["link"] for item in search_results]
+
+    except HttpError as e:
+        # Handle errors in the API call
+        error_details = json.loads(e.content.decode())
+        
+        # Check if the error is related to an invalid or missing API key
+        if error_details.get("error", {}).get("code") == 403 and "invalid API key" in error_details.get("error", {}).get("message", ""):
+            return "Error: The provided Google API key is invalid or missing."
+        else:
+            return f"Error: {e}"
+
+    # Return the list of search result URLs
+    return search_results_links
 
 def browse_website(url):
     summary = get_text_summary(url)

scripts/config.py

@@ -36,10 +36,12 @@ class Config(metaclass=Singleton):
         self.openai_api_key = os.getenv("OPENAI_API_KEY")
         self.elevenlabs_api_key = os.getenv("ELEVENLABS_API_KEY")
         
+        self.google_api_key = os.getenv("GOOGLE_API_KEY")
+        self.custom_search_engine_id = os.getenv("CUSTOM_SEARCH_ENGINE_ID")
+
         # Initialize the OpenAI API client
         openai.api_key = self.openai_api_key
 
-
     def set_continuous_mode(self, value: bool):
         self.continuous_mode = value
 
@@ -64,5 +66,8 @@ class Config(metaclass=Singleton):
     def set_elevenlabs_api_key(self, value: str):
         self.elevenlabs_api_key = value
         
+    def set_google_api_key(self, value: str):
+        self.google_api_key = value
     
-        
+    def set_custom_search_engine_id(self, value: str):
+        self.custom_search_engine_id = value