Maiko Bossuyt
36d455c20e
split_file() rework
...
rework the split_file function to make it simple and only have one yield while providing an overlap at the start and end of each chunk
2023-04-12 23:31:26 +02:00
Maiko Bossuyt
2c8b42307b
Merge branch 'Torantulino:master' into add_ingest_documents_script
2023-04-12 22:52:08 +02:00
Maiko Bossuyt
2e0b44ae05
fix chunk creation
...
the last chunk wasn't correctly created, this commit fix that issue.
2023-04-12 22:46:49 +02:00
Drikus Roor
1f837968ff
fix: Fix flake8 errors based on the flake8 command with a narrower definition of errors
2023-04-12 22:12:25 +02:00
Maiko Bossuyt
4465486ea3
Update file_operations.py
...
move the search_file function inside the data_ingestion script
2023-04-12 20:19:27 +02:00
Maiko Bossuyt
c91117616f
Update file_operations.py
...
revert change in import as we don't need them
2023-04-12 19:46:58 +02:00
Maiko Bossuyt
0dddc94bda
Add file ingestion methode in file_operations.py
...
Add the following functions to ingest data into memory before Auto-GPT run.
- split_file: given a content, split it in chunks of max_length with (or without) a specified overlap
- ingest_file: read a file, use split_file to split it in chunks and load each chunk in memory
- ingest_directory: ingest all files in a directory in memory
2023-04-12 19:13:04 +02:00
BillSchumacher
1b128a1ef0
Use UTF-8 encoding for Windows users.
2023-04-10 18:50:54 -05:00
Andres Caicedo
26abb30674
Merge branch 'master' into dev
2023-04-09 15:42:53 +02:00
Andres Caicedo
011699e6a1
Code review changes
2023-04-09 15:39:11 +02:00
Toran Bruce Richards
b8f2dd59c8
Fixes "/" search not working.
2023-04-06 09:23:03 +01:00
slavakurilyak
1e47328079
Add search files command
2023-04-04 20:32:15 -05:00
Andres Caicedo
1d10236a63
Update file_operations.py
...
Introduces spaces between code blocks.
2023-04-04 11:32:55 +02:00
Andres Caicedo
e2892a92a1
Merge branch 'master' into dev
2023-04-04 10:37:42 +02:00
yousefissa
9ef4fab084
error handling back
2023-04-03 08:50:07 -07:00
yousefissa
4416aa1aa1
create file dir if it doesnt exist during write_to_file
2023-04-03 08:48:43 -07:00
Andres Caicedo
765210f0cd
Add extra documentation
2023-04-03 14:10:02 +02:00
Andres Caicedo
eac5c1f6e6
Add documentation
2023-04-02 19:03:37 +02:00
Torantulino
b4685f67e6
Apply autopep8 formatting to entire codebase
2023-04-02 09:13:15 +01:00
Torantulino
a608d8dbdc
Adds full file I/O
2023-04-01 04:08:30 +01:00
