Skip to content

๐Ÿ” An LLM-based Multi-agent Framework of Web Search Engine (like Perplexity.ai Pro and SearchGPT)

License

Notifications You must be signed in to change notification settings

starkwpgots/MindSearch

Folders and files

NameName
Last commit message
Last commit date

Latest commit

b2e9b8e ยท Aug 14, 2024

History

36 Commits
Jul 29, 2024
Aug 12, 2024
Aug 13, 2024
Aug 12, 2024
Aug 12, 2024
Aug 12, 2024
Jul 28, 2024
Jul 29, 2024
Jul 29, 2024
Aug 1, 2024
Jul 28, 2024
Aug 14, 2024
Aug 14, 2024
Aug 5, 2024

Repository files navigation

โœจ MindSearch: Mimicking Human Minds Elicits Deep AI Searcher

MindSearch is an open-source AI Search Engine Framework with Perplexity.ai Pro performance. You can simply deploy it with your own perplexity.ai style search engine with either close-source LLMs (GPT, Claude) or open-source LLMs (InternLM2.5 series are specifically optimized to provide superior performance within the MindSearch framework; other open-source models have not been specifically tested). It owns following features:

  • ๐Ÿค” Ask everything you want to know: MindSearch is designed to solve any question in your life and use web knowledge.
  • ๐Ÿ“š In-depth Knowledge Discovery: MindSearch browses hundreds of web pages to answer your question, providing deeper and wider knowledge base answer.
  • ๐Ÿ” Detailed Solution Path: MindSearch exposes all details, allowing users to check everything they want. This greatly improves the credibility of its final response as well as usability.
  • ๐Ÿ’ป Optimized UI Experience: Providing all kinds of interfaces for users, including React, Gradio, Streamlit and Terminal. Choose any type based on your need.
  • ๐Ÿง  Dynamic Graph Construction Process: MindSearch decomposes the user query into atomic sub-questions as nodes in the graph and progressively extends the graph based on the search result from WebSearcher.

โšก๏ธ MindSearch vs other AI Search Engines

Comparison on human preference based on depth, breadth, factuality of the response generated by ChatGPT-Web, Perplexity.ai (Pro), and MindSearch. Results are obtained on 100 human-crafted real-world questions and evaluated by 5 human experts*.

* All experiments are done before July.7 2024.

โšฝ๏ธ Build Your Own MindSearch

Step1: Dependencies Installation

git clone https://github.com/InternLM/MindSearch
cd MindSearch
pip install -r requirements.txt

Step2: Setup MindSearch API

Setup FastAPI Server.

python -m mindsearch.app --lang en --model_format internlm_server
  • --lang: language of the model, en for English and cn for Chinese.
  • --model_format: format of the model.
    • internlm_server for InternLM2.5-7b-chat with local server. (InternLM2.5-7b-chat has been better optimized for Chinese.)
    • gpt4 for GPT4. if you want to use other models, please modify models

Step3: Setup MindSearch Frontend

Providing following frontend interfaces,

  • React
# Install Node.js and npm
# for Ubuntu
sudo apt install nodejs npm

# for windows
# download from https://nodejs.org/zh-cn/download/prebuilt-installer

# Install dependencies

cd frontend/React
npm install
npm start

Details can be found in React

  • Gradio
python frontend/mindsearch_gradio.py
  • Streamlit
streamlit run frontend/mindsearch_streamlit.py

๐Ÿž Debug Locally

python -m mindsearch.terminal

๐Ÿ“ License

This project is released under the Apache 2.0 license.

Citation

If you find this project useful in your research, please consider cite:

@article{chen2024mindsearch,
  title={MindSearch: Mimicking Human Minds Elicits Deep AI Searcher},
  author={Chen, Zehui and Liu, Kuikun and Wang, Qiuchen and Liu, Jiangning and Zhang, Wenwei and Chen, Kai and Zhao, Feng},
  journal={arXiv preprint arXiv:2407.20183},
  year={2024}
}

Our Projects

Explore our additional research on large language models, focusing on LLM agents.

  • Lagent: A lightweight framework for building LLM-based agents
  • AgentFLAN: An innovative approach for constructing and training with high-quality agent datasets (ACL 2024 Findings)
  • T-Eval: A Fine-grained tool utilization evaluation benchmark (ACL 2024)

About

๐Ÿ” An LLM-based Multi-agent Framework of Web Search Engine (like Perplexity.ai Pro and SearchGPT)

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 54.1%
  • TypeScript 25.8%
  • Less 17.6%
  • Dockerfile 2.2%
  • HTML 0.3%