Paginate connector page #2328
Paginate connector page #2328
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
| stmt = stmt.join(SearchSettings).where( | ||
| SearchSettings.status == IndexModelStatus.PRESENT | ||
| ) | ||
|
|
There was a problem hiding this comment.
pretty identical except for this pagination logic
| @@ -33,6 +39,38 @@ | |||
| router = APIRouter(prefix="/manage") | |||
|
|
|||
|
|
|||
There was a problem hiding this comment.
I created a seperate endpoint to get the index attempts for a couple reasons:
- enable pagination of the index attempts
- allow the front end to load the rest of the connector data without having to wait for the index attempts
| num_docs_indexed: int, # not ideal, but this must be computed separately | ||
| is_editable_for_current_user: bool, | ||
| ) -> "CCPairFullInfo": | ||
| # figure out if we need to artificially deflate the number of docs indexed. |
There was a problem hiding this comment.
This logic was originally in the front end but i thought it made more sense to be here
| and number_of_index_attempts == 1 | ||
| ): | ||
| num_docs_indexed = last_index_attempt.new_docs_indexed | ||
|
|
There was a problem hiding this comment.
we now return the last indexing status and the number of docs to facilitate some of the original display logic on the frontend without having to send back the indexattempts themselves
| const [indexAttemptTracePopupId, setIndexAttemptTracePopupId] = useState< | ||
| number | null | ||
| >(null); | ||
| const indexAttemptToDisplayTraceFor = ccPair.index_attempts.find( | ||
| const [currentPageData, setCurrentPageData] = |
There was a problem hiding this comment.
I cooked from here down. 🧑🍳
The main features are:
- preloading pages
- debouncing (in the form of a 200 ms delay from clicking the button to actually calling the api)
- instant loading of preloaded pages
There was a problem hiding this comment.
READ THE COMMENTS FOR EACH PART, THEY ARE INFORMATIVE
| const indexAttemptToDisplayTraceFor = ccPair.index_attempts.find( | ||
|
|
||
| const totalPages = Math.ceil(ccPair.number_of_index_attempts / NUM_IN_PAGE); | ||
|
|
There was a problem hiding this comment.
this allows navigation to the different index_attempt pages through the url
| const [indexAttemptTracePopupId, setIndexAttemptTracePopupId] = useState< | ||
| number | null | ||
| >(null); | ||
| const indexAttemptToDisplayTraceFor = ccPair.index_attempts.find( | ||
| const [currentPageData, setCurrentPageData] = |
There was a problem hiding this comment.
READ THE COMMENTS FOR EACH PART, THEY ARE INFORMATIVE
| setCurrentPageData(cachedBatches[batchNum][batchPageNum]); | ||
| setIsCurrentPageLoading(false); | ||
| } else { | ||
| setIsCurrentPageLoading(true); |
There was a problem hiding this comment.
Generally, we should try to avoid using effects transform state (React's docs are great - https://react.dev/learn/you-might-not-need-an-effect). I'd try to use some memoization for expensive calculations and avoid using effects to continuously update state - (but fine for pagination)
| // we use it to avoid duplicate requests | ||
| const ongoingRequestsRef = useRef<Set<number>>(new Set()); | ||
|
|
||
| const urlBuilder = (batchNum: number) => |
There was a problem hiding this comment.
nit: more descriptive function naming
* Added pagination to individual connector pages * I cooked * Gordon Ramsay in this b * meepe * properly calculated max chunk and switch dict to array * chunks -> batches * increased max page size * renmaed var
* Added pagination to individual connector pages * I cooked * Gordon Ramsay in this b * meepe * properly calculated max chunk and switch dict to array * chunks -> batches * increased max page size * renmaed var
This should greatly improve load times by: