aljaz/Tutorial-Codebase-Knowledge
1
0
Fork 0
mirror of https://github.com/aljazceru/Tutorial-Codebase-Knowledge.git synced 2025-12-19 15:34:23 +01:00
Code Issues Packages Projects Releases Wiki Activity

update nav

Browse Source
This commit is contained in:
zachary62
2025-04-04 14:03:22 -04:00
parent 2fa60fe7d5
commit 0426110e66
24 changed files with 261 additions and 32 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
docs/Crawl4AI/10_basedispatcher.md
View File

@@ -1,3 +1,10 @@
---
layout: default
title: "BaseDispatcher"
parent: "Crawl4AI"
nav_order: 10
---
# Chapter 10: Orchestrating the Crawl - BaseDispatcher
In [Chapter 9: Smart Fetching with Caching - CacheContext / CacheMode](09_cachecontext___cachemode.md), we learned how Crawl4AI uses caching to cleverly avoid re-fetching the same webpage multiple times, which is especially helpful when crawling many URLs. We've also seen how methods like `arun_many()` ([Chapter 2: Meet the General Manager - AsyncWebCrawler](02_asyncwebcrawler.md)) or strategies like [DeepCrawlStrategy](08_deepcrawlstrategy.md) can lead to potentially hundreds or thousands of individual URLs needing to be crawled.