Create Agent from Website Knowledge

​

What you can do with this

• Build customer support agents that answer questions using your help documentation
• Create internal knowledge assistants to help employees find information quickly
• Develop product information bots that provide detailed specs and features
• Design training assistants for interactive learning experiences
• Ensure compliance with accurate information delivery for regulatory requirements

​

What you get

• Instant expertise: Your Agent immediately knows your content and can answer related questions
• Contextual responses: Provides accurate answers based on your specific information
• Content updates: Automatically stays current as you update your Knowledge Base
• Source citations: Can reference specific documents or sections when answering
• Domain expertise: Specialized knowledge in your specific field or industry

​

Step-by-step creation process

1

Click Create from Knowledge card

Navigate to the Create Agent page and click on the Create from Knowledge card to begin the process.

This opens a modal dialog where you can configure your knowledge-powered Agent.

2

Select Website data source and customize Agent

In the modal, select Website as the data source for training your Agent’s Knowledge Base and customize the Agent’s appearance:

Customization options:

• Data Source: Choose Website to train from web content
• Agent Icon: Select or change the icon representing your Agent

You can choose from various data sources and customize your Agent’s icon. Additional data source options will be introduced in future updates.

3

Configure Knowledge Base settings

Enter the Knowledge Base Name and Website Base URL, then configure advanced scraping settings if needed.

• Basic Configuration
• Advanced Settings

Advanced configuration options: - Number of URLs to scrape: Set the maximum number of URLs to crawl (default: 200, maximum: 10,000) - Number of levels deep: Define how deep the scraper should go into the website structure (default: 20, maximum: 100) - Custom user Agent: Optional custom user Agent string for web requests - Crawl using only base URLs: Ensures crawling goes only one level deep, ignoring the depth setting - Use browser: Enable browser-rendered content in the background instead of raw HTML for dynamic content - Use legacy scraper: Choose between Scrapy (legacy) and Crawl4AI (non-legacy) scraping engines

Optimal Settings: For most websites, the default settings (200 URLs, 20 levels deep) work well. Use the browser option for JavaScript-heavy sites, and choose legacy scraper if you encounter compatibility issues with the modern Crawl4AI engine.

4

Fetch and process content

Click the Fetch Links button to begin the web scraping process.

• Pre-fetch
• While fetching

Scraping process: - Content discovery: System discovers and crawls web pages, extracting content from each page - Data processing: Cleans and organizes extracted content for optimal training preparation

The fetching process may take several minutes depending on the size of your website and the number of pages to scrape.

5

Train the Knowledge Base

After content is successfully fetched, click Train to process and train your Knowledge Base.

• Pre-training
• While training
• Post-training

Page selection options: - Select the pages you want to train - Deselect the pages you don’t need - View metadata about the number of words fetched for each page

Monitor the training progress as the system: - Processes and indexes the scraped content - Creates searchable knowledge embeddings - Optimizes the content for AI understanding - Shows detailed training logs and progress

Training completed: - View the content that has been trained in the knowledge base - See the pages and content the Agent now has knowledge of - Training is complete and ready for Agent creation

Training process: - Vector storage: Content is converted to vector embeddings and stored in vector databases for fast retrieval - Index optimization: Creates efficient search and retrieval system for AI responses

6

Agent creation complete

Your knowledge-powered Agent is now ready!

What happens automatically: - Agent creation: Agent is created with your specified name and Knowledge Base gambit connected - Ready to use: Agent can immediately answer questions based on your website content