Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: dev note features & langchain core(pt.1) #95

Merged
merged 29 commits into from
Apr 1, 2024
Merged
Changes from 1 commit
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
0e8d6ad
refactor: update configuration files and database processing logic
codeacme17 Mar 27, 2024
979441a
feat: upload create note api related files
codeacme17 Mar 27, 2024
bbfc84d
feat: updated upload form component and database handler
codeacme17 Mar 27, 2024
72473fa
feat: optimize the file deletion logic of the drag-and-drop upload co…
codeacme17 Mar 27, 2024
581b09b
feat: add handle file upload and deletion in note creation route, and…
codeacme17 Mar 27, 2024
6c60489
fix: fix drag-upload file list issue
codeacme17 Mar 27, 2024
94f0f9b
feat: add check for existing note before creating
codeacme17 Mar 27, 2024
d09ea22
ui: refactor question type selection in UploadForm component
codeacme17 Mar 27, 2024
ca68c56
refactor: rename and refactor functions
codeacme17 Mar 27, 2024
bf20b6a
feat: refactor file upload logic and fix bug in note creation
codeacme17 Mar 28, 2024
fe0ff14
chore: add langchain related deps
codeacme17 Mar 28, 2024
44aaf14
feat: add markdown loader
codeacme17 Mar 28, 2024
360b207
refactor: refactor markdownSpitter function
codeacme17 Mar 28, 2024
dbfbfae
chore: add tiktoken dep
codeacme17 Mar 28, 2024
ee3aa42
feat: add lenToken method
codeacme17 Mar 28, 2024
18a9c2f
feat: add limit methods for parse doc
codeacme17 Mar 28, 2024
dd853d6
feat: add document db method
codeacme17 Mar 28, 2024
2edee51
feat: fix file update bug and add file upload status
codeacme17 Mar 28, 2024
48bb5a6
feat: add qg prompt and refactor QuestionType
codeacme17 Mar 28, 2024
bdfd911
feat: add getFirst function to profileHandler
codeacme17 Mar 28, 2024
e43f804
feat: add intergration llms
codeacme17 Mar 30, 2024
1175d6f
test: add vitest and llm unit test
codeacme17 Mar 30, 2024
794141f
refactor: update intergration llm
codeacme17 Mar 30, 2024
5f4d82e
feat: add PureLlm class and test case
codeacme17 Mar 30, 2024
1783977
chore: add alias to vitest
codeacme17 Mar 30, 2024
3dbc8ae
feat: update configuration parameters and prompt templates
codeacme17 Mar 31, 2024
0353028
refactor: add splitQuestions function and import questionHandler
codeacme17 Mar 31, 2024
f3c194b
refactor: note creation route and chain generation
codeacme17 Apr 1, 2024
56130d4
refacotr: update question type and remove unnecessary code
codeacme17 Apr 1, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
refactor: add splitQuestions function and import questionHandler
  • Loading branch information
codeacme17 committed Mar 31, 2024
commit 0353028453abd64785db983d5ff8bfd1ea86879f
31 changes: 21 additions & 10 deletions next/langchain/chain/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,12 @@ import {
removePrefixNumbers,
isLegalQuestionStructure,
extractScore,
splitQuestions,
getPushDate,
} from './util'
import { documentHandler, fileHandler } from '@/lib/db-handler'
import type { PromptType, QuestionType, RoleType } from '@/types/global'
import { questionHandler } from '@/lib/db-handler/question'

export class Chain {
profile: TProfile
Expand Down Expand Up @@ -53,7 +55,7 @@ export class Chain {
async agenerateQuestions(
docs: Document[],
title: string,
questionType: string
questionType: QuestionType
): Promise<number> {
const tasks = []
const llmChain = this.initLlmChain(questionType)
Expand Down Expand Up @@ -91,21 +93,26 @@ export class Chain {
doc: Document,
title: string,
docId: number,
questionType: string
questionType: QuestionType
) {
await this.semaphore.acquire()

try {
const res = await llmChain.predict(title, doc.pageContent)
for (const question of spiteQuestions(res, questionType)) {
if (!isLegalQuestionStructure(question, questionType)) continue
const res = await llmChain.invoke({
title,
context: doc.pageContent,
})

question.saveQuestionToDb(
removePrefixNumbers(question),
for (const question of splitQuestions(res, questionType)) {
if (!isLegalQuestionStructure(question, questionType)) continue
const { currentRole } = this.profile
questionHandler.create(
docId,
questionType,
this.profile.currentRole
removePrefixNumbers(question),
currentRole
)

this.questionCount += 1
}
} finally {
Expand All @@ -114,17 +121,21 @@ export class Chain {
}

private initLlmChain(
roleType: RoleType,
questionType: QuestionType,
timeout: number = 10
): LLMChain {
const { currentRole } = this.profile
const llmInstance = new IntergrationLlm(this.profile, {
temperature: this.temperature,
streaming: this.streaming,
maxRetries: adjustRetriesByPaymentStatus(),
timeout,
})
const prompt = choicePrompt(this.promptType, roleType, questionType)
const prompt = choicePrompt(
this.promptType,
currentRole as RoleType,
questionType
)
return new LLMChain({ prompt, llm: llmInstance.llm! })
}
}