Web Scrape QnA
Last updated
Last updated
Let's say you have a website (could be a store, an ecommerce site, a blog), and you want to scrap all the relative links of that website and have LLM answer any question on your website. In this tutorial, we are going to go through how to achieve that.
This flow is used to upsert all information from a website to a vector database, then have LLM answer user's question by looking up from the vector database.
You can find the example flow called - Conversational Retrieval QA Chain from the marketplace templates.
Here, we are going to use Cheerio Web Scraper node to scrape links from a given URL. Also replacing RecursiveTextSplitter to HtmlToMarkdown for cleaner data preparation.
If you do not specify anything, by default only the given URL page will be scraped. If you want to crawl the rest of relative links, click Additional Parameters.
Get Relative Links Method - how to crawl all relative links, Web Crawl or Sitemap
Get Relative Links Limit - how many links to crawl, set 0 to crawl all
When you open the chat and start asking question, all links will be scraped and upserted into vector database (Pinecone in this case).
From the console/terminal, you can see all the links that are being scraped:
Navigate to Pinecone dashboard, you will be able to see new vectors being added under the namespace you have specified in the flow.
Few things to keep in mind
Once the flow is set, documents will only get upserted once when user start asking first question from the UI or API or Embedded Chat.
Documents will not get upserted again whenever user ask another question.
The only condition where documents get upserted again is when the flow configuration (like different file, different models, different pinecone index, etc) have changed, and you have to save the chatflow again.
In the other words, we store the flow state, and if a new save is done, we check if existing state == new state, if not, do upsert, if yes, ignore.
However, sometimes you might want to change some settings like metadata but you don't want another upsert to be done again.
Therefore, it is generally recommended to create another flow to load the existing index from vector store.
Understanding the exact cause of this issue can be quite intricate, so let's focus on the solution for now. As you follow these steps, the underlying reasons should become clearer.
Restarting and the Issue: You've observed that restarting seems to lead to the problem. While this action appears problematic, it's not necessarily a fault in the true sense.
Checking Pinecone:
Begin by heading to the Pinecone dashboard.
If you started with an empty Pinecone index, you should still see vectors from before the restart.
Adjusting in the Flow:
If the above vectors are visible, return to the flow.
Here, remove the splitter and scraper nodes.
Modifying Pinecone Nodes:
Replace the "Pinecone Upsert Document" node with the "Pinecone Load Existing Index" node.
Ensure you update the parameters of this new node, especially if you've assigned specific names to your index or have added metadata during your scrape.
Finalizing the Setup:
Conclude by saving your changes and refreshing the system. After this, everything should be back on track!
This flow is used to load an existing index/collection from vector store, typically after you have upserted the documents to that particular index/collection.
If you have specified namespace or metadata from the upsert flow, remember to specify here as well, under the Additional Parameters in Pinecone node.
It is recommended to specify a system message for the Conversational Retrieval QA Chain. For example, you can specify the name of AI, the language to answer, the response when answer its not found (to prevent hallucination).
That's it! You can start asking question 🤔
You can also turn on the Return Source Documents option to return a list of document chunks where the AI's response is coming from.
The same logic can be applied to any document use cases, not just limited to web scraping.
If you have any suggestion on how to improve the performance, we'd love your contribution!
