This PR is the next evolution of MyGPT. It uses unstructured.io for Document loading, extending support from text files to CSV, PDF, Email, Office, Markdown, CSV, and more.
Also included is upgrading gradio, langchain, chromadb dependencies and taking advantage of collection.upsert() when putting embeddings into chroma.
And finally, small platform specific tweaks needed to get things running on Windows. Also cleaned up a bunch of code so hopefully easier to follow.
This PR is the next evolution of MyGPT. It uses unstructured.io for Document loading, extending support from text files to CSV, PDF, Email, Office, Markdown, CSV, and more.
Also included is upgrading gradio, langchain, chromadb dependencies and taking advantage of `collection.upsert()` when putting embeddings into chroma.
And finally, small platform specific tweaks needed to get things running on Windows. Also cleaned up a bunch of code so hopefully easier to follow.
- use unstructured package for reading documents, many more document
loaders
- support calling Index.ingest() repeatedly, only ingests files not
already ingested into vector database.
- cleaned up Index code a lot, took out older stuff.
rajatarya
changed title from Document Loading updates, Windows tweaks, many dependency updates to WIP: Document Loading updates, Windows tweaks, many dependency updates10 months ago
rajatarya
changed title from WIP: Document Loading updates, Windows tweaks, many dependency updates to Document Loading updates, Windows tweaks, many dependency updates10 months ago
rajatarya
requested review from zach 10 months ago
zach
This PR is the next evolution of MyGPT. It uses unstructured.io for Document loading, extending support from text files to CSV, PDF, Email, Office, Markdown, CSV, and more.
Also included is upgrading gradio, langchain, chromadb dependencies and taking advantage of
collection.upsert()when putting embeddings into chroma.And finally, small platform specific tweaks needed to get things running on Windows. Also cleaned up a bunch of code so hopefully easier to follow.
Document Loading updates, Windows tweaks, many dependency updatesto WIP: Document Loading updates, Windows tweaks, many dependency updates 10 months agoWIP: Document Loading updates, Windows tweaks, many dependency updatesto Document Loading updates, Windows tweaks, many dependency updates 10 months agoReviewed over screenshare with @zach, merging manually.
Reviewers