Skip to content

Commit 9b4975c

Browse files
Java-EdgeJava-Edge
committed
docs:新增文档
1 parent 6c44e71 commit 9b4975c

6 files changed

Lines changed: 641 additions & 2 deletions

‎docs/.vuepress/config.js‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1323,6 +1323,7 @@ module.exports = {
13231323
children: [
13241324
"Claude3到底多强",
13251325
"GPTs推荐",
1326+
"ChatGPT为啥不用Websocket而是EventSource？",
13261327
]
13271328
},
13281329
{
@@ -1346,7 +1347,7 @@ module.exports = {
13461347
"03-为啥LLM还没能完全替代你？",
13471348
"04-产业拆解",
13481349
"05-应用级开发者 AI 时代破局点",
1349-
"06-智能体(agent)命理大师虚拟项目",
1350+
"06-智能体项目案例",
13501351
]
13511352
},
13521353
{
@@ -1359,6 +1360,8 @@ module.exports = {
13591360
"02-LangChain实战：用prompts模板调教LLM的输入出",
13601361
"03-示例选择器",
13611362
"04-LLMs和Chat Models",
1363+
"05-Loader机制",
1364+
"06-文档转换实战",
13621365
]
13631366
},
13641367
{
@@ -1396,6 +1399,7 @@ module.exports = {
13961399
"模板方法设计模式（Template Pattern）",
13971400
"策略模式Strategy Pattern",
13981401
"建造者模式",
1402+
"代理模式Proxy Pattern",
13991403
]
14001404
},
14011405
],

‎docs/md/AI/05-Loader机制.md‎

Lines changed: 150 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,150 @@
1+
# 05-Loader机制
2+
3+
loader机制让大模型具备实时学习的能力：
4+
5+
![](https://my-img.javaedge.com.cn/javaedge-blog/2024/06/cbf5df8cd1b29adecec27fc658f7f3fd.png)
6+
7+
## 0 Loader机制
8+
9+
案例环境准备：
10+
11+
```python
12+
import os
13+
14+
os.environ["OPENAI_API_KEY"] = "sk-javaedge"
15+
os.environ["OPENAI_PROXY"] = "https://api.chatanywhere.tech"
16+
17+
18+
import os
19+
from dotenv import load_dotenv
20+
# Load environment variables from openai.env file
21+
load_dotenv("openai.env")
22+
23+
# Read the OPENAI_API_KEY from the environment
24+
api_key = os.getenv("OPENAI_API_KEY")
25+
api_base = os.getenv("OPENAI_API_BASE")
26+
os.environ["OPENAI_API_KEY"] = api_key
27+
os.environ["OPENAI_API_BASE"] = api_base
28+
```
29+
30+
## 1 加载markdown
31+
32+
准备一个 md 文件：
33+
34+
```markdown
35+
# 我是一个markdown加载示例
36+
- 第一项目
37+
- 第二个项目
38+
- 第三个项目
39+
40+
## 第一个项目
41+
编程严选网，最厉害专业的AI研究基地
42+
43+
## 第二个项目
44+
AIGC打造未来AI应用天地
45+
46+
## 第三个项目
47+
编程严选网是一个非常牛逼的AI媒体
48+
```
49+
50+
```python
51+
#使用loader来加载markdown文本
52+
from langchain.document_loaders import TextLoader
53+
54+
loader = TextLoader("loader.md")
55+
loader.load()
56+
```
57+
58+
![](https://my-img.javaedge.com.cn/javaedge-blog/2024/06/afda3b9852f22f5db86ae87d4596f409.png)
59+
60+
## 2 加载cvs
61+
62+
```csv
63+
Project,DES,Price,People,Location
64+
AI GC培训,培训课程,500,100,北京
65+
AI工程师认证,微软AI认证,6000,200,西安
66+
AI应用大会,AI应用创新大会,200门票,300,深圳
67+
AI 应用咨询服务,AI与场景结合,1000/小时,50,香港
68+
AI项目可研,可行性报告,20000,60,上海
69+
```
70+
71+
```python
72+
#使用 CSVLoader 来加载 csv 文件
73+
from langchain.document_loaders.csv_loader import CSVLoader
74+
75+
#loader = Loader(file_path="loader.")
76+
loader = CSVLoader(file_path="loader.csv")
77+
data = loader.load()
78+
print(data)
79+
```
80+
81+
![](https://my-img.javaedge.com.cn/javaedge-blog/2024/06/b73e5f8fc222172accbb969f244e34c8.png)
82+
83+
## 3 加载html
84+
85+
先下包：
86+
87+
```python
88+
! pip install "unstructured[xlsx]"
89+
```
90+
91+
加载文件目录
92+
93+
```python
94+
from langchain.document_loaders import UnstructuredHTMLLoader
95+
96+
loader = UnstructuredHTMLLoader("loader.html")
97+
data = loader.load()
98+
data
99+
```
100+
101+
会加载 html 所有内容。
102+
103+
```python
104+
from langchain.document_loaders import BSHTMLLoader
105+
loader = BSHTMLLoader("loader.html")
106+
data = loader.load()
107+
data
108+
```
109+
110+
只加载去除标签后的关键内容：
111+
112+
![](https://my-img.javaedge.com.cn/javaedge-blog/2024/06/c4bf9ca3d5fa7552b4d146074a3d2a3b.png)
113+
114+
## 4 加载JSON
115+
116+
先装 jq 包：
117+
118+
```python
119+
! pip install jq
120+
```
121+
122+
123+
124+
```python
125+
from langchain.document_loaders import JSONLoader
126+
loader = JSONLoader(
127+
file_path = "simple_prompt.json",jq_schema=".template",text_content=True
128+
)
129+
data = loader.load()
130+
print(data)
131+
```
132+
133+
![](https://my-img.javaedge.com.cn/javaedge-blog/2024/06/8dc7055c72a7d75008d45b9dcc395a1e.png)
134+
135+
## 5 加载PDF
136+
137+
先装包：
138+
139+
```python
140+
! pip install pypdf
141+
```
142+
143+
```python
144+
from langchain.document_loaders import PyPDFLoader
145+
loader = PyPDFLoader("loader.pdf")
146+
pages = loader.load_and_split()
147+
pages[0]
148+
```
149+
150+
![](https://my-img.javaedge.com.cn/javaedge-blog/2024/06/fc1407f642d6549421f3c4cf10dbddc0.png)

‎docs/md/AI/06-文档转换实战.md‎

Lines changed: 175 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,175 @@
1+
# 06-文档转换实战
2+
3+
## 1 文档切割器和按字符分割
4+
5+
### 1.1 原理
6+
7+
1. 将文档分成小的、有意义的块(句子).
8+
2. 将小的块组合成为一个更大的块，直到达到一定的大小.
9+
3. 一旦达到一定的大小，接着开始创建与下一个块重叠的部分.
10+
11+
### 1.2 示例
12+
13+
#### 第一个文档分割
14+
15+
```python
16+
from langchain.text_splitter import RecursiveCharacterTextSplitter
17+
18+
#加载要切割的文档
19+
with open("test.txt") as f:
20+
zuizhonghuanxiang = f.read()
21+
22+
#初始化切割器
23+
text_splitter = RecursiveCharacterTextSplitter(
24+
chunk_size=50,#切分的文本块大小，一般通过长度函数计算
25+
chunk_overlap=20,#切分的文本块重叠大小，一般通过长度函数计算
26+
length_function=len,#长度函数,也可以传递tokenize函数
27+
add_start_index=True,#是否添加起始索引
28+
)
29+
30+
text = text_splitter.create_documents([zuizhonghuanxiang])
31+
text[0]
32+
text[1]
33+
```
34+
35+
#### 按字符切割
36+
37+
```python
38+
from langchain.text_splitter import CharacterTextSplitter
39+
40+
#加载要切分的文档
41+
with open("test.txt") as f:
42+
zuizhonghuanxiang = f.read()
43+
44+
#初始化切分器
45+
text_splitter = CharacterTextSplitter(
46+
separator="。",#切割的标志字符，默认是\n\n
47+
chunk_size=50,#切分的文本块大小，一般通过长度函数计算
48+
chunk_overlap=20,#切分的文本块重叠大小，一般通过长度函数计算
49+
length_function=len,#长度函数,也可以传递tokenize函数
50+
add_start_index=True,#是否添加起始索引
51+
is_separator_regex=False,#是否是正则表达式
52+
)
53+
text = text_splitter.create_documents([zuizhonghuanxiang])
54+
print(text[0])
55+
```
56+
57+
58+
59+
![](https://my-img.javaedge.com.cn/javaedge-blog/2024/06/38130dccedf0a3a1791edf13b05ab4bc.png)
60+
61+
## 2 代码文档分割器
62+
63+
```python
64+
from langchain.text_splitter import (
65+
RecursiveCharacterTextSplitter,
66+
Language,
67+
)
68+
69+
#支持解析的编程语言
70+
#[e.value for e in Language]
71+
72+
#要切割的代码文档
73+
PYTHON_CODE = """
74+
def hello_world():
75+
print("Hello, World!")
76+
#调用函数
77+
hello_world()
78+
"""
79+
py_spliter = RecursiveCharacterTextSplitter.from_language(
80+
language=Language.PYTHON,
81+
chunk_size=50,
82+
chunk_overlap=10,
83+
)
84+
python_docs = py_spliter.create_documents([PYTHON_CODE])
85+
python_docs
86+
```
87+
88+
## 3 按token分割文档
89+
90+
```python
91+
from langchain.text_splitter import CharacterTextSplitter
92+
93+
#要切割的文档
94+
with open("test.txt") as f:
95+
zuizhonghuanxiang = f.read()
96+
97+
#初始化切分器
98+
text_splitter = CharacterTextSplitter.from_tiktoken_encoder(
99+
chunk_size=4000,#切分的文本块大小，一般通过长度函数计算
100+
chunk_overlap=30,#切分的文本块重叠大小，一般通过长度函数计算
101+
)
102+
103+
text = text_splitter.create_documents([zuizhonghuanxiang])
104+
print(text[0])
105+
```
106+
107+
108+
109+
![](https://my-img.javaedge.com.cn/javaedge-blog/2024/06/d4f2372aa6f8bf137c49ecfcd7c14e42.png)
110+
111+
112+
113+
## 4 文档总结、精炼、翻译
114+
115+
先装包：
116+
117+
```python
118+
! pip install doctran==0.0.14
119+
```
120+
121+
![](https://my-img.javaedge.com.cn/javaedge-blog/2024/06/b4296386a8f14c5b5d0257b5056e8d9b.png)
122+
123+
先加载文档：
124+
125+
```python
126+
with open("letter.txt") as f:
127+
content = f.read()
128+
```
129+
130+
```python
131+
from dotenv import load_dotenv
132+
import os
133+
load_dotenv("openai.env")
134+
OPENAI_API_KEY = os.environ.get("OPEN_API_KEY")
135+
OPENAI_API_BASE = os.environ.get("OPENAI_API_BASE")
136+
OPENAI_MODEL = "gpt-3.5-turbo-16k"
137+
OPENAI_TOKEN_LIMIT = 8000
138+
139+
from doctran import Doctran
140+
doctrans = Doctran(
141+
openai_api_key=OPENAI_API_KEY,
142+
openai_model=OPENAI_MODEL,
143+
openai_token_limit=OPENAI_TOKEN_LIMIT,
144+
)
145+
documents = doctrans.parse(content=content)
146+
```
147+
148+
### 4.1 总结
149+
150+
```python
151+
summary = documents.summarize(token_limit=100).execute()
152+
print(summary.transformed_content)
153+
```
154+
155+
![](https://my-img.javaedge.com.cn/javaedge-blog/2024/06/25f938b757391d3193f7d905f1f9903f.png)
156+
157+
### 4.2 翻译
158+
159+
```python
160+
translation = documents.translate(language="chinese").execute()
161+
print(translation.transformed_content)
162+
```
163+
164+
![](https://my-img.javaedge.com.cn/javaedge-blog/2024/06/348287dc54a058c2373bf61b6b4fc0dd.png)
165+
166+
### 4.3 精炼
167+
168+
删除除了某个主题或关键词之外的内容，仅保留与主题相关的内容
169+
170+
```python
171+
refined = documents.refine(topics=["marketing","Development"]).execute()
172+
print(refined.transformed_content)
173+
```
174+
175+
![](https://my-img.javaedge.com.cn/javaedge-blog/2024/06/4ca5b18bc96b5a497f08acdc6380ac9a.png)

docs/md/AI/06-智能体(agent)命理大师虚拟项目.md renamed to docs/md/AI/06-智能体项目案例.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# 06-智能体(agent)命理大师虚拟项目
1+
# 06-智能体项目案例
22

33
需求分析、技术选型、技术分解
44

0 commit comments

Comments
 (0)