The Ultimate Guide to Building AI Agents with Exa

Oh wait you still use Google to search the internet? In this glorious age of reasoning AI. Come on now.

Here’s the thing, if you’re building AI agents, Google won’t cut it. You don’t need 500,000 search results. You need just the right amount of information for your agent to do its job. Your agent needs something more powerful. That’s where Exa comes in.

Exa is a specialized web API designed specifically for AI applications. In this guide I will walk you through everything you need to know about leveraging Exa to create powerful, web-aware AI agents that can perform complex tasks with real-world data. There’s code examples that you can copy and paste too.

But first, let’s look under the hood to see how it works.

Why Exa for AI Agents?

Exa positions itself as “a search engine made for AIs” and excels at retrieving high-quality, relevant web content to enhance AI applications.

Key Advantages

1. Designed for AI Integration: Unlike traditional search APIs, Exa is optimized for AI consumption, returning clean, parsed content that’s ready for processing by large language models (LLMs).
2. Semantic Understanding: Exa offers neural search capabilities that understand the meaning behind queries, not just keywords, making it ideal for natural language interactions.
3. Comprehensive Web Coverage: With very high availability across crucial categories like research papers, personal pages, news, and company information, Exa provides broad access to the web’s knowledge.
4. Focused Results: Exa excels at finding specific entities like people, companies, and research papers—often delivering 20x more correct results than traditional search engines for complex queries.
5. Verification Capabilities: Through its AI agent integration, Exa can verify and validate search results, ensuring higher accuracy for critical applications.

How Exa Works Behind the Scenes

At its core, Exa uses embeddings to transform web content into numerical vector representations. This allows it to understand the conceptual meaning of content rather than just matching keywords.

Exa’s search engine consists of three main components:

(1) Crawling & Indexing

• Exa crawls the web to collect data, just like Google.
• They identify and process URLs, storing document content in a structured format.
• Unlike Google, which focuses on keyword-based indexing, Exa processes documents using AI models to understand content semantically.

(2) AI Processing (Neural Search)

• Instead of relying on traditional PageRank (which ranks results based on backlinks and domain authority), Exa uses a neural-based approach.
• Exa’s link prediction model is trained to predict which links follow a given piece of text.
• This means Exa learns the relationships between documents, similar to how transformers predict words in a sentence.

Example: If an article says “Check out this aerospace startup” and links to spacex.com, Exa learns that “aerospace startup” is semantically related to SpaceX—even if the text doesn’t explicitly say “SpaceX.”

(3) Search & Retrieval

• At query time, Exa’s model predicts the most relevant documents, rather than relying on exact keyword matches.
• The search query is processed like an LLM prompt, allowing for natural language queries instead of just keywords.

Comprehensive Web Index

Exa has indexed billions of web pages, focusing on high-quality content across various categories like research papers, personal websites, company information, and news sources. While this is smaller than Google’s trillion-page index, Exa has prioritized quality and relevance for specific use cases over sheer quantity.

The index is maintained with special attention to categories that are particularly valuable for AI applications. For example, research papers, personal websites, and LinkedIn profiles have very high availability in Exa’s index, making it especially useful for finding specific entities and specialized information.

Search Types and Processing

Exa offers three main search types:

1. Neural Search: This leverages the embedding technology for semantic understanding. When you use neural search, Exa finds content that is conceptually related to your query, even if it doesn’t contain the exact words. This is particularly effective for exploratory searches and complex concepts.
2. Keyword Search: This more traditional approach focuses on finding exact word matches, which is useful for proper nouns or specific terminology. It’s optimized for precision when you know exactly what terms should appear.
3. Auto Search: This lets Exa decide whether to use neural or keyword search based on the query, combining the advantages of both approaches.

Content Processing

When retrieving full content from web pages, Exa doesn’t just return raw HTML. It processes the content to extract the most relevant text, removing navigational elements, ads, and other noise. This clean parsing makes the content immediately usable by language models without requiring additional cleaning steps.

Exa can also generate AI summaries of content, identify highlights based on queries, and extract structured data from web pages, further enhancing its utility for AI applications.

Technical Infrastructure

Under the hood, Exa likely uses a combination of:

1. Vector Databases: To store and efficiently query the embedded representations of web pages.
2. Large Language Models: For query understanding, content summarization, and result verification.
3. Distributed Computing: To handle the computational demands of embedding and searching billions of web pages.
4. Crawlers and Parsers: To continually update its index with fresh content from the web.

API Functionality

The Exa API exposes this functionality through endpoints like:

• /search for finding relevant web pages
• /contents for retrieving and processing the content of specific pages
• Features to filter by domain, date, category, and other parameters

The API is designed to be easily integrated with language models through SDKs for Python and JavaScript, making it straightforward to incorporate web data into AI workflows.

Getting Started with Exa

Now that you have an understanding of how Exa works, we can get to building agents with it. To begin building with Exa, you’ll need to:

1. Create an Account: Visit dashboard.exa.ai to register for an account.
2. Obtain an API Key: Generate your API key from the dashboard at dashboard.exa.ai/api-keys.
3. Install the SDK: Choose between the Python or JavaScript SDK based on your preferred development environment. When building pure AI agents, I use Python, so all code examples in this post will use Python.
4. Set Up Environment Variables: Create a .env file to securely store your API key

Basic Exa Implementation

Here’s an example of how to initiate an Exa search in Python:

import os
from exa import Exa
from dotenv import load_dotenv

# Load environment variables
load_dotenv()

# Initialize the Exa client
exa = Exa(api_key=os.getenv("EXA_API_KEY"))

# Perform a basic search
results = exa.search("Latest research in LLMs", type="auto")

# Print the results
for result in results:
    print(f"Title: {result.title}")
    print(f"URL: {result.url}")
    print(f"Published Date: {result.published_date}")
    print(f"Score: {result.score}")
    print("---")

As you can see, it’s pretty simple. You initialize the Exa client, pass it a string to search, and then it returns a bunch of results. There are options to customize how the results are returned (how many, what level of detail, etc.) We’ll explore this with a few more examples.

Building Your First AI Agent with Exa

Let’s walk through creating a basic research agent that can gather information on a specific topic and provide a summary.

import os
from exa import Exa
from dotenv import load_dotenv
from openai import OpenAI

# Load environment variables
load_dotenv()

# Initialize clients
exa = Exa(api_key=os.getenv("EXA_API_KEY"))
openai_client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

def research_agent(topic, num_results=5):
    """
    A simple research agent that gathers information on a topic and summarizes it.
    
    Args:
        topic: The research topic
        num_results: Number of search results to consider
    
    Returns:
        A summary of findings
    """
    # Step 1: Search for relevant information
    print(f"Searching for information on: {topic}")
    search_results = exa.search_and_contents(
        query=topic,
        num_results=num_results,
        text=True  # Retrieve the full text content
    )
    
    # Step 2: Extract and compile the content
    all_content = ""
    sources = []
    
    for i, result in enumerate(search_results):
        if hasattr(result, 'text') and result.text:
            all_content += f"\nSource {i+1}: {result.title}\n{result.text[:1000]}...\n"
            sources.append(f"{i+1}. {result.title} - {result.url}")
    
    # Step 3: Summarize with LLM
    prompt = f"""
    Based on the following information about "{topic}", provide a comprehensive summary:
    
    {all_content}
    
    Your summary should be well-structured, factual, and highlight the most important points.
    """
    
    response = openai_client.chat.completions.create(
        model="gpt-4-turbo",
        messages=[
            {"role": "system", "content": "You are a research assistant that summarizes information accurately."},
            {"role": "user", "content": prompt}
        ]
    )
    
    summary = response.choices[0].message.content
    
    # Compile final report
    final_report = f"""
    # Research Summary: {topic}
    
    {summary}
    
    ## Sources
    {'\n'.join(sources)}
    """
    
    return final_report

# Example usage
if __name__ == "__main__":
    result = research_agent("Advances in quantum computing in 2024")
    print(result)

This agent demonstrates the core workflow of using Exa for AI agents:

1. Our agent starts by taking a query from the user and searching with Exa
2. We use the search_and_contents function to not just get back a list of URLs that best match our query but also the content within those pages.
3. We then use an LLM (in this case, GPT-4) to analyze and summarize the findings and format it into a report.

More Complex AI Agents

As you become more comfortable with basic implementations, you can build more sophisticated AI agents for specific use cases. Let’s explore three powerful examples.

1. Competitor Research Agent

This agent automatically discovers and analyzes competitors for a given company, compiling insights into a structured report.

import os
from exa import Exa
from dotenv import load_dotenv
from openai import OpenAI

load_dotenv()
exa = Exa(api_key=os.getenv("EXA_API_KEY"))
openai_client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

class CompetitorResearchAgent:
    def __init__(self):
        self.exa_client = exa
        self.openai_client = openai_client
        
    def find_competitors(self, company_name):
        """Find potential competitors using Exa search"""
        query = f"Top competitors of {company_name}"
        results = self.exa_client.search(query, num_results=10, exclude_domains = ["mydomain.com"])  # you can change this
        return results
        
    def analyze_competitor(self, competitor_result):
        """Analyze a specific competitor based on web content"""
        # Get detailed content about the competitor
        content_result = self.exa_client.get_contents(competitor_result.id, text=True)
        
        if not hasattr(content_result, 'text') or not content_result.text:
            return {
                "name": competitor_result.title,
                "url": competitor_result.url,
                "overview": "No detailed information available",
                "products": [],
                "strengths": [],
                "weaknesses": []
            }
            
        # Use LLM to extract structured information
        prompt = f"""
        Based on the following content about a company, extract:
        1. Company name
        2. Brief overview (2-3 sentences)
        3. Main product offerings (up to 5)
        4. Key strengths (up to 3)
        5. Potential weaknesses (up to 3)
        
        Content:
        {content_result.text[:4000]}
        
        Format your response as JSON with the following structure:
        {{
            "name": "Company Name",
            "overview": "Brief overview",
            "products": ["Product 1", "Product 2", ...],
            "strengths": ["Strength 1", "Strength 2", ...],
            "weaknesses": ["Weakness 1", "Weakness 2", ...]
        }}
        """
        
        response = self.openai_client.chat.completions.create(
            model="gpt-4-turbo",
            response_format={"type": "json_object"},
            messages=[{"role": "user", "content": prompt}]
        )
        
        try:
            import json
            analysis = json.loads(response.choices[0].message.content)
            analysis["url"] = competitor_result.url
            return analysis
        except:
            return {
                "name": competitor_result.title,
                "url": competitor_result.url,
                "overview": "Analysis failed",
                "products": [],
                "strengths": [],
                "weaknesses": []
            }
    
    def generate_report(self, company_name):
        """Generate a complete competitor analysis report"""
        print(f"Finding competitors for {company_name}...")
        competitors = self.find_competitors(company_name)
        
        if not competitors:
            return f"No competitors found for {company_name}"
            
        print(f"Found {len(competitors)} competitors. Analyzing each...")
        
        analyses = []
        for competitor in competitors:
            print(f"Analyzing {competitor.title}...")
            analysis = self.analyze_competitor(competitor)
            analyses.append(analysis)
            
        # Generate the final report
        report = f"# Competitor Analysis Report for {company_name}\n\n"
        report += f"## Executive Summary\n\n"
        
        # Use LLM to generate executive summary
        companies_list = ", ".join([a["name"] for a in analyses])
        summary_prompt = f"""
        Create a brief executive summary for a competitor analysis report for {company_name}.
        The identified competitors are: {companies_list}.
        Keep it under 150 words and highlight key insights.
        """
        
        summary_response = self.openai_client.chat.completions.create(
            model="gpt-4-turbo",
            messages=[{"role": "user", "content": summary_prompt}]
        )
        
        report += f"{summary_response.choices[0].message.content}\n\n"
        
        # Add detailed competitor analyses
        report += f"## Detailed Competitor Analysis\n\n"
        
        for analysis in analyses:
            report += f"### {analysis['name']}\n\n"
            report += f"**Website**: {analysis['url']}\n\n"
            report += f"**Overview**: {analysis['overview']}\n\n"
            
            report += "**Product Offerings**:\n"
            for product in analysis['products']:
                report += f"- {product}\n"
            report += "\n"
            
            report += "**Key Strengths**:\n"
            for strength in analysis['strengths']:
                report += f"- {strength}\n"
            report += "\n"
            
            report += "**Potential Weaknesses**:\n"
            for weakness in analysis['weaknesses']:
                report += f"- {weakness}\n"
            report += "\n"
            
        return report

# Example usage
if __name__ == "__main__":
    agent = CompetitorResearchAgent()
    report = agent.generate_report("MyAwesomeStartup") #insert your company name here
    print(report)
    
    # Save report to file
    with open("competitor_analysis.md", "w") as f:
        f.write(report)

There’s a lot going on here so let’s break it down.

First we run a basic Exa search for “Top competitors of {company_name}” which in this case would be your company. We’re also setting a parameter for number of search results (in this case 10), which you can customize.

We also exclude our own company just in case but you can add more domains to exclude here.

We get the results back in a list which we then loop through. At this point, you can also filter it further but for each URL, we run exa.get_contents() which simply scrapes that URL and returns all the page content.

We then use GPT 4 to analyze the content and turn it into a comprehensive competitor research report. This section is mostly prompt engineering so feel free to play around and try some other prompts.

2. Newsletter Generation Agent with CrewAI

For more complex workflows, you can combine Exa with agent orchestration frameworks like CrewAI. This example creates a team of specialized agents that work together to generate a newsletter.

import os
from dotenv import load_dotenv
from crewai import Agent, Task, Crew, Process
from exa import Exa
from openai import OpenAI

load_dotenv()
exa = Exa(api_key=os.getenv("EXA_API_KEY"))
openai_client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

# Create specialized agents
researcher = Agent(
    role="Research Specialist",
    goal="Find the latest and most relevant news on a given topic",
    backstory="You are an expert researcher who excels at finding accurate information online",
    verbose=True,
    allow_delegation=True,
    tools=[exa.search, exa.get_contents],
    llm=openai_client,
)

fact_checker = Agent(
    role="Fact Checker",
    goal="Verify information accuracy and source credibility",
    backstory="You're a meticulous fact-checker with years of experience in journalism",
    verbose=True,
    allow_delegation=True,
    tools=[exa.search, exa.get_contents],
    llm=openai_client,
)

writer = Agent(
    role="Newsletter Writer",
    goal="Create engaging, informative newsletter content",
    backstory="You're a talented writer who specializes in distilling complex topics into readable content",
    verbose=True,
    allow_delegation=False,
    llm=openai_client,
)

# Create tasks for each agent
def create_newsletter_crew(topic):
    research_task = Task(
        description=f"Research the latest news about {topic} from the past week. Find at least 5 important developments or stories.",
        expected_output="A list of news items with titles, brief summaries, and source URLs",
        agent=researcher,
    )

    verification_task = Task(
        description="Verify the accuracy of each news item and evaluate the credibility of sources",
        expected_output="Verified list of news items with credibility scores for each source",
        agent=fact_checker,
        context=[research_task],
    )

    writing_task = Task(
        description=f"Create a newsletter about {topic} based on the verified research. Include an introduction, summaries of the top stories, and a conclusion.",
        expected_output="Complete newsletter in HTML format, ready to be sent",
        agent=writer,
        context=[verification_task],
    )

    # Create and run the crew
    crew = Crew(
        agents=[researcher, fact_checker, writer],
        tasks=[research_task, verification_task, writing_task],
        verbose=True,
        process=Process.sequential,
    )
    
    return crew

# Example usage
if __name__ == "__main__":
    topic = "Artificial Intelligence Advancements"
    crew = create_newsletter_crew(topic)
    result = crew.kickoff()
    
    with open(f"{topic.replace(' ', '_')}_newsletter.html", "w") as f:
        f.write(result)
    
    print(f"Newsletter generated and saved to {topic.replace(' ', '_')}_newsletter.html")

The research agent uses Exa to find relevant news, the fact-checker verifies information, and the writer compiles everything into a cohesive newsletter. You can tweak the instructions to structure the newsletter in a specific way.

3. Recruiting Agent with Exa and OpenAI

This agent automates the process of discovering, researching, and evaluating exceptional candidates for recruitment purposes.

It’s a lot more complicated than the previous two and the code requires multiple files, so I haven’t added it here. I will describe the logic to it so that you can try it out on your own. If you need help, contact me!

1. Candidate Discovery: We start with an Exa search for qualified professionals based on a job title and required skills, focusing on LinkedIn profiles and GitHub accounts. In the Exa API call you can limit results to certain domains.
2. Comprehensive Research: For each potential candidate, it gathers information from their LinkedIn profile and then tries to find their personal websites or GitHub profiles for additional context.
3. Intelligent Evaluation: Using GPT-4o, it evaluates each candidate against job requirements, scoring them on technical skills, experience, education, communication, and overall fit.
4. Similar Candidate Finding: After identifying top performers, it uses Exa’s semantic search to find similar professionals, expanding the talent pool.
5. Structured Reporting: It generates a comprehensive markdown report with an executive summary, detailed evaluations of each candidate, and recommendations.

Additional Exa Features and Best Practices

As you build more complex agents with Exa, try using these additional features and best practices to maximize effectiveness:

Filtering and Refinement

Exa offers powerful filtering options to narrow down search results:

# Date filtering
results = exa.search(
    "AI ethics developments",
    start_published_date="2024-01-01",
    end_published_date="2024-12-31"
)

# Domain filtering
results = exa.search(
    "Climate change research",
    include_domains=["nature.com", "science.org"],
    exclude_domains=["wikipedia.org"]
)

# Category filtering
results = exa.search(
    "Quantum computing advances",
    category="research paper"
)

Content Retrieval

For deeper analysis, retrieve the full content of web pages, or an AI-generated summary:

# Retrieve text content
content = exa.get_contents(result_id, text=True)

# Retrieve highlights based on a query
highlights = exa.get_contents(result_id, highlights="quantum advantage")

# Retrieve AI-generated summaries
summary = exa.get_contents(result_id, summary=True)

Working with Websets

Exa’s Websets feature, launched in December 2024, transforms complex searches into structured datasets. This powerful tool helps you find specific entities like people, companies, and research papers with greater precision than traditional search engines.

I’ve played around with it and I’ve found it to be really good for use cases like sales, HR and recruiting, and even finding founders to invest in. It doesn’t just bring you a list, it also researches them and verifies all the information

Advanced AI Agents with Exa

If you’ve made it this far, congrats. Hopefully you’ve tried some of the code samples above and even adapted it to your use case. In this section, I’m going to talk about even more complex builds. These are large projects I’ve built out for clients and there’s too much code to add to this post, so I’m just going to explain the intuition behind them.

Chain-of-Thought Research Agent

One powerful pattern for research agents is the chain-of-thought approach, where the agent breaks down complex questions into sub-questions, researches each one separately, and then synthesizes the findings. Here’s how it works:

1. Question Decomposition: When given a complex research question, the agent uses GPT-4 to break it down into 3-5 more focused sub-questions. Each sub-question targets a specific aspect of the main question, making the research more manageable and thorough.
2. Sub-Question Research: For each sub-question, the agent uses Exa to search the web for relevant information. It gathers content from multiple sources, extracts the most important passages, and then uses GPT-4 to formulate a concise but comprehensive answer based strictly on the information found.
3. Synthesis of Findings: After researching all sub-questions, the agent compiles the individual answers and uses GPT-4 to synthesize them into a cohesive response to the original question. This step ensures that connections between different aspects are identified and incorporated.
4. Report Generation: Finally, the agent creates a structured research report with an executive summary containing the synthesized answer, followed by detailed findings for each sub-question with their respective sources properly cited.

This approach mirrors how human researchers tackle complex topics—breaking down big questions into manageable parts, researching each thoroughly, and then connecting the dots to form a complete picture. It’s particularly effective for multifaceted questions that require exploring different angles or domains of knowledge.

Here’s an example of a research agent I built:

Multi-Modal Generation with Exa and LLMs

This agent implements a comprehensive multi-modal content production pipeline that mimics the process a professional content creator might follow. The workflow consists of six carefully orchestrated steps:

1. Topic Research: When given a topic, the agent first conducts thorough research using Exa’s search and content retrieval capabilities. It gathers information from multiple sources, extracting relevant text and organizing it with proper attribution.

2. Visual Element Research: Recognizing that engaging content isn’t just text, the agent searches for relevant imagery related to the topic. Again, with Exa, we can search specifically for images. We can also limit the search to sites like Unsplash.

3. Structured Outline Generation:With research in hand, the agent uses GPT-4 to create a comprehensive outline for the article with the proposed title, introduction concept, detailed section breakdowns with subheadings, key points for each section, and a conclusion approach.

4. Data Visualization Creation: Here, the agent generates custom Python code for a data visualization relevant to the article topic, if applicable. It analyzes the research data to identify key concepts that would benefit from visual representation, then creates complete, executable code using matplotlib or seaborn.

5. Article Writing: The agent then synthesizes all the previous elements – research, outline, images, data viz – into a complete article. It follows the outline structure precisely, incorporates references to the suggested images, maintains an engaging writing style, and includes proper citations to the original sources.

6. Content Package Assembly: Finally, the agent compiles everything into a comprehensive content package containing the original topic, structured outline, finished article text, data visualization code, image descriptions, and source references. This modular approach makes it easy to use the outputs in various ways – publishing the article as-is, extracting just the visualization code, or using the outline as a starting point for further development.

What will you build with Exa?

As we’ve explored throughout this guide, Exa represents a powerful evolution in how AI agents interact with the web. By providing a dedicated search API optimized specifically for AI consumption, Exa bridges a crucial gap between large language models and the vast, constantly updating knowledge contained on the internet.

The agents we’ve examined demonstrate how this integration creates systems that are not merely intelligent but also well-informed. These agents ground their reasoning in current, relevant information rather than relying solely on their training data, which inevitably becomes outdated over time.

I hope this guide serves as a starting point for you, demonstrating what’s possible today. So, tell me, what will you build with Exa?