aljaz/Auto-GPT
1
0
Fork 0
mirror of https://github.com/aljazceru/Auto-GPT.git synced 2025-12-21 07:54:21 +01:00
Code Issues Packages Projects Releases Wiki Activity

split_file() rework

Browse Source 
rework the split_file function to make it simple and only have one yield while providing an overlap at the start and end of each chunk
This commit is contained in:
Maiko Bossuyt
2023-04-12 23:31:26 +02:00
parent 2f1181f9a1
commit 36d455c20e
1 changed files with 4 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

11
scripts/file_operations.py
View File

@@ -35,15 +35,12 @@ def split_file(content, max_length=4000, overlap=0):
while start < content_length: while start < content_length:
end = start + max_length end = start + max_length
chunk = content[start:end] if end + overlap < content_length:
chunk = content[start:end+overlap]
else:
chunk = content[start:content_length]
yield chunk yield chunk
start += max_length - overlap start += max_length - overlap
if start + max_length > content_length:
start = content_length - max_length
end = content_length
chunk = content[start:end]
yield chunk
break
def read_file(filename): def read_file(filename):
"""Read a file and return the contents""" """Read a file and return the contents"""