What

1. Added a new table to store the document related details.
2. Added a new table to store the indexing related details.
3. UI changes to display the indexing status for each combination of document and LLM profile.
4. Other UI/UX improvements.

Why

Previously, we were not storing the index status. As a result, we were unable to display it in the UI. Because of the limited information, the user-experience in the prompt studio didn't feel great.

How

NA

Database Migrations

Made the following changes in DB migrations:

1. Added a new table to persist document details: This table will primarily store the document name and the prompt studio ID.
2. Added a new table to persist index details: This table will primarily store the following: Vector DB IDs for raw and summarized content, history of Vector DB IDs for all versions of the corresponding profile manager, and a unique combination of document ID and profile manager ID.
3. Modified the output manager table: Previously, the document name was stored directly to uniquely identify the document. Now, I'm associating it with the document manager table. Also, changed on_delete values to CASCADE wherever necessary.

Env Config

NA

Relevant Docs

NA

Related Issues or PRs

NA

Dependencies Versions

NA

Notes on Testing

Two major things needed to be tested in this PR:

1. Check if the functionalities related to documents are working fine, since there has been some substantial amount of work that has gone it.
2. Check if the index status for each combination of document and LLM profile is properly displayed in the UI.

Here are some of the steps I followed to make sure if everything worked fine,

1. Created a new LLM profile, named profile1.
2. Uploaded the document and indexed it for profile1.
3. The status was successfully updated in the Document Manager table.
4. Then created a new LLM profile, named profile2.
5. Entered the prompt and ran it before indexing. This will help us check if it is indexed before fetching the result.
6. After the result is generated, checked the document manager to see if it was indexed for profile2.
7. Enabled summarization and reindexed inside the document manager to check if it gets indexed properly.
8. Cleared the database and repeated the same steps to ensure everything worked fine.

Screenshots

Checklist

I have read and understood the Contribution Guidelines.