fix: continue with failures when bulk loading urls with WebBaseLoader

9ed1a315 · Jun Siang Cheah · 7538dc05 · 9ed1a315
Commit 9ed1a315 authored May 12, 2024 by Jun Siang Cheah
Hide whitespace changes
Inline Side-by-side

Showing with 1 addition and 0 deletions

backend/apps/rag/main.py backend/apps/rag/main.py +1 -0

No files found.
--- a/backend/apps/rag/main.py
+++ b/backend/apps/rag/main.py
@@ -584,6 +584,7 @@ def get_web_loader(url: Union[str, Sequence[str]], verify_ssl: bool = True):
        url,
        verify_ssl=verify_ssl,
        requests_per_second=RAG_WEB_SEARCH_CONCURRENT_REQUESTS,
+        continue_on_failure=True,
    )