How To Enable Auto Refresh for Knowledge Base Links in HighLevel

Keeping your knowledge base up to date is essential when you rely on live URLs—websites, funnels, landing pages, or blog posts that change over time. HighLevel’s auto refresh for knowledge base links ensures that the content your AI agents reference stays current without manual maintenance. This saves time, prevents stale or broken answers, and helps your assistants remain reliable as pages evolve.

Why auto refresh matters for your AI agents

Knowledge bases are only as useful as the information inside them. If you train an AI agent on a set of web pages and those pages change—prices update, copy is revised, pages are removed—the agent can start returning outdated or incorrect responses. Auto refresh automates that maintenance by periodically re-crawling the links you have trained your knowledge base with and updating the stored content.

Auto refresh is particularly valuable for:

• Marketing funnels and landing pages that receive frequent copy or offer updates.
• Product pages where pricing, features, or stock status change regularly.
• Support documentation that evolves with new product releases or processes.
• Large knowledge bases where manually refreshing hundreds or thousands of links is impractical.

Where to find and enable auto refresh in HighLevel

The setting lives inside the Knowledge Base area of HighLevel’s AI Agents. The path is straightforward:

1. Open HighLevel and go to AI Agents.
2. Select Knowledge Bases.
3. Choose the knowledge base you want to edit and click Edit.
4. Within the knowledge base, navigate to the Web Crawler section where your list of trained links appears.
5. Flip the Auto Refresh toggle to on and select a refresh frequency: daily, weekly, or monthly.
6. Click Save to apply the setting.

Once enabled, HighLevel will automatically revisit each trained URL at the interval you selected and update the knowledge base content accordingly.

Choosing the right refresh frequency

The refresh cadence determines how often the web crawler re-reads your links. There is no one-size-fits-all setting; choose the option that balances content freshness with system load and bandwidth.

Guidelines for selecting a frequency

• Daily: Best for small sets of pages or pages that change daily—promotions, pricing, or event pages. If your knowledge base contains a handful of essential pages, daily refresh ensures near real-time accuracy.
• Weekly: Good for moderate-sized knowledge bases or pages that change regularly but not daily. Weekly strikes a balance between freshness and resource usage.
• Monthly: Ideal for very large knowledge bases or when pages rarely change. If you have thousands of links, monthly refresh reduces bandwidth and system strain while still keeping content reasonably up to date.

Example: If your knowledge base includes around 1,500 links, a monthly refresh is often the practical choice. Refreshing that many URLs every day or even weekly could cause unnecessary load and might create performance issues.

How auto refresh behaves and what to expect

Understanding what happens during a refresh helps you anticipate outcomes and avoid surprises.

• Automatic updates: When the scheduled refresh runs, HighLevel’s web crawler revisits each URL and updates the stored content based on the live version of the page.
• Manual vs automatic: Manually refreshing a knowledge base link does the same work as auto refresh. The only difference is that auto refresh runs on the schedule you chose without you having to intervene.
• Failure handling: If a page is no longer live—server returns a 404, the domain is removed, or the page is blocked—the content from that link will be taken down during the next refresh cycle. In short, the knowledge base will reflect the current live state of the web.
• Cost and credits: Refreshes and auto refreshes are rolled into your Conversation AI usage. They do not consume separate credits or incur extra costs beyond your AI usage allocation.

Best practices to avoid issues

A few practical tips will help you get the most from auto refresh while avoiding common pitfalls.

• Start conservative with large link sets. If your knowledge base has many links, begin with monthly refresh and monitor performance. If content changes more frequently than expected, move to weekly.
• Segment knowledge bases. Create separate knowledge bases for fast-changing content (daily/weekly) and static content (monthly). This allows tailored refresh schedules that optimize resource usage.
• Watch for pages behind login. The web crawler cannot access pages that require authentication or are blocked by robots.txt. For members-only content, export or provide public versions where appropriate.
• Prefer server-rendered content. Sites that heavily rely on client-side JavaScript may not render fully for the crawler. Where possible, use server-rendered pages or static copies of important content.
• Monitor failed refreshes. If a trained link fails during a refresh, check whether the page is temporarily down, blocked, or intentionally removed. If the page should exist, work with the site owner to restore it or replace the trained link.

How auto refresh fits with HighLevel workflows and scaling

Auto refresh is not just a maintenance tool. It can be woven into larger HighLevel agency systems and automations to create smarter, more resilient processes.

For example, when product pages or support docs are updated, those content changes can trigger internal workflows:

• Notify team members via automation when a refresh removes a page or finds a broken link.
• Use updated knowledge base content to retrain or fine-tune responses in shared AI agents across multiple client accounts.
• Combine auto refresh with HighLevel automations to send alerts or update CRM records when critical knowledge base content changes.

These integrations make maintaining consistent and accurate AI-driven customer interactions easier as you scale your agency operations.

When to manually refresh a link

There are times a manual refresh is the right move:

• After publishing an urgent update that needs to be available to AI agents immediately.
• When testing how the AI uses newly added content before the next scheduled auto refresh.
• If a trained link was temporarily down and is now restored and you want the updated content immediately.

Manual refresh does the same job as auto refresh but gives you immediate control when timing matters.

Troubleshooting common problems

If auto refresh is not behaving as expected, check these points:

• Robots.txt and crawl blocking: Verify the target site allows crawling. If robots.txt disallows crawlers, the link will fail to refresh.
• Authentication: Pages behind authentication or paywalls will not be crawled. Provide a public version or include the content another way.
• Dynamic content: Sites that load content via client-side JS may not present the full content to the crawler. Consider server-side rendering or static snapshots for training.
• Link changes: If a URL structure changed (for example, a page moved to a new path), update the trained link to the new URL.
• Large refresh loads: If you notice systemic slowdowns after enabling refreshes on very large knowledge bases, reduce frequency or split links across multiple knowledge bases.

Checklist: Enabling and optimizing auto refresh

1. Open HighLevel > AI Agents > Knowledge Bases and choose the target knowledge base.
2. Under Web Crawler, review the list of trained links and remove any obsolete URLs.
3. Toggle Auto Refresh on.
4. Select a refresh frequency based on the size and volatility of your content.
5. Save the setting and monitor for any failed refreshes.
6. Segment content into multiple knowledge bases if you need different refresh cadences.
7. Set up automations to notify stakeholders about removed or updated pages if desired.

Additional resources and support

For more details about the auto refresh feature, consult HighLevel’s help documentation at https://help.gohighlevel.com/en/support/solutions/articles/155000006539-auto-refresh-of-knowledge-base-trained-links or reach out to HighLevel support for assistance with specific cases.

If you want hands-on support, consider scheduling a walkthrough with the HighLevel success team or joining the Nexus Hub community for templates, implementation guidance, and agency-focused resources.

Ready to test this yourself? Start a free 14-day HighLevel trial at http://gohighlevel.com/army-offer and experiment with auto refresh in a sandbox environment before applying it to production knowledge bases.

Summary

Auto refresh for knowledge base links in HighLevel is a simple but powerful feature. It keeps your AI agents aligned with live content, reduces manual maintenance, and fits neatly into your broader HighLevel workflows and agency systems. Choose the refresh frequency that matches your content volatility and link volume, monitor refresh results, and use segmentation and automations to scale safely.

With a little setup and the right cadence, auto refresh can dramatically improve the accuracy and reliability of AI-driven customer interactions across your business.

FAQ

How do I enable auto refresh for a knowledge base in HighLevel?

Open HighLevel, go to AI Agents > Knowledge Bases, edit the knowledge base, find the Web Crawler section, toggle Auto Refresh on, choose daily, weekly, or monthly, and click Save.

Will auto refresh cost additional credits?

No. Auto refreshes and manual refreshes are rolled into your Conversation AI usage and do not consume separate credits beyond your existing allocation.

What happens if a page is removed or returns an error during refresh?

If a page fails to load during a refresh because it is offline or has been removed, the content from that link will be removed from the knowledge base when the refresh runs. Monitor failed refreshes and update or replace links as needed.

How should I choose between daily, weekly, and monthly refresh?

Choose daily for a small number of frequently changing pages, weekly for moderately sized or moderately changing content, and monthly for very large knowledge bases or mostly static content. Segment knowledge bases to use different cadences for different content sets.

Can the web crawler access pages behind logins or paywalls?

No. Pages that require authentication or are blocked by robots.txt will not be crawled. Provide public versions of important content or use alternative training methods for gated content.

How can I minimize system load when refreshing many links?

Reduce refresh frequency, split links into multiple knowledge bases with staggered schedules, and only include essential links in a given knowledge base to avoid large simultaneous crawls.