THR-1 h5p editor: temporary file storage #4147
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…loud-server into THR-18-H5P-Editor
…nto THR-18-H5P-Editor
Create temporary file folder
casparneumann-cap
added a commit
that referenced
this pull request
Nov 8, 2023
* THR-1 h5p editor: temporary file storage (#4147) * Created H5P microservice * Add kubernetes files for h5p microservice * Added @lumieducation/h5p-server package * Don't use mock auth for testing * Fix api tests to use new TestApiClient * WIP: H5P editor: temp file storage * WIP: H5P editor: added data classes * resolve merge conflict leftovers * h5p editor: temp file storage implementation completed * h5p editor: temp file storage tests added * h5p editor: temp file storage moved * h5p editor: temp storage files moved * Update temporary-file-storage.ts Create temporary file folder --------- Co-authored-by: Marvin Rode <marvin.rode@capgemini.com> Co-authored-by: Marvin Rode (Cap) <127723478+marode-cap@users.noreply.github.com> Co-authored-by: Andre Blome <andre-david.blome@capgemini.com> Co-authored-by: Stephan Krause <101647440+SteKrause@users.noreply.github.com> * THR-2 local content storage (#4140) * Created H5P microservice * Add kubernetes files for h5p microservice * Added @lumieducation/h5p-server package * add ContentStorage with interface IContentStorage * add addContent() * implement addFile(), contentExists(), deleteContent() * implement deleteFile(), fileExists() * add getFileStats(), getFileStream() * add getMetadata(), getParameters(), getUsage() * add getUserPermissions(), listContent() * add listFiles(), sanitizeFilename() * refactor sanitizeFileneame * add user check for getMetadata and getParameters * fix existsOrCreateDir * fix deleteContent and fileExists * fix fileExists and contentExists, refactor methods * add unit tests for contentStorage * delete comments in addFile * fix addFile test and change descriptions * fix getUsage * fix getUsage test * delete TODO * add error message to contentExists + tests * add test for empty contentId at fileEsists * remove unused private sanitize method * adjust errorhandling + add tests for error cases * add tests and fix error cases * replace math.random * change regex in checkfilename * refactor tests and content storage implementation * change any values to unknown * fix create content id * refactor and fix exist boolean in createContentId * fix test can not createContentId * create folder if content will be created * Allow files in subdirectories --------- Co-authored-by: Marvin Rode <marvin.rode@capgemini.com> Co-authored-by: Marvin Rode (Cap) <127723478+marode-cap@users.noreply.github.com> * THR-3 local library storage (#4133) * Implemented library storage * Tests for librray storage * full test coverage * Moved files --------- Co-authored-by: Stephan Krause <101647440+SteKrause@users.noreply.github.com> * h5p editor: added deployment which uses existing template (#4136) Co-authored-by: Andre Blome <andre-david.blome@capgemini.com> Co-authored-by: Stephan Krause <101647440+SteKrause@users.noreply.github.com> * Thr 5 h5p implementation endpoints (#4169) * Created H5P microservice * Add kubernetes files for h5p microservice * Added @lumieducation/h5p-server package * Don't use mock auth for testing * add ContentStorage with interface IContentStorage * Fix api tests to use new TestApiClient * WIP: H5P editor: temp file storage * WIP: H5P editor: added data classes * add addContent() * implement addFile(), contentExists(), deleteContent() * implement deleteFile(), fileExists() * add getFileStats(), getFileStream() * add getMetadata(), getParameters(), getUsage() * add getUserPermissions(), listContent() * add listFiles(), sanitizeFilename() * refactor sanitizeFileneame * Implemented library storage * add user check for getMetadata and getParameters * fix existsOrCreateDir * h5p editor: added deployment which uses existing template * fix deleteContent and fileExists * Tests for librray storage * resolve merge conflict leftovers * fix fileExists and contentExists, refactor methods * add unit tests for contentStorage * delete comments in addFile * fix addFile test and change descriptions * fix getUsage * fix getUsage test * delete TODO * full test coverage * Moved files * Implemented library storage * Tests for librray storage * full test coverage * Moved files * add error message to contentExists + tests * add test for empty contentId at fileEsists * remove unused private sanitize method * adjust errorhandling + add tests for error cases * add tests and fix error cases * replace math.random * change regex in checkfilename * delete unused code * change regex in checkfilename * refactor tests and content storage implementation * change any values to unknown * fix create content id * Fix issues from merging main into THR-2 * refactor and fix exist boolean in createContentId * fix test can not createContentId * Stubbed endpoints * Editor and Player service * Added H5P Usecase * h5p editor: temp file storage implementation completed * h5p editor: temp file storage tests added * h5p editor: temp file storage moved * h5p editor: temp storage files moved * File streaming * Post Ajax endpoint * Merge branch 'THR-8-h5p-api-endpoints' of github.com:hpi-schul-cloud/schulcloud-server into THR-5-h5p-implementation-endpoints * Remove @nestjs/serve-static dependency * Editor working more reliably * Disable saving of userstate * Organized code * More organization * Tests for getAjax usecase * add tests for h5p controller * Tests and organization * add import of FilesStorageAMQPModule to h5p-editor module * Switched over to storage implementations * Organized modules * change stringPath of createNewEditor * Testing for files in UC * Testing content parameters in UC * change AjaxPostBodyParams to undefined if empty * change save api endpoint * Fixed file streaming * Fixed h5p uploading * Update temporary-file-storage.ts Create temporary file folder * merge save and create endpoints to one get and post endpoint * create folder if content will be created * adjust paths at api tests * rename usecase to getH5pEditor * Use real user * Allow files in subdirectories * Testing for h5p files * API tests for internal AJAX endpoint * Test range requests for storages * add unit-tests to usecases * add api tests for h5p endpoints * add api test save or create * adjust create or save api test --------- Co-authored-by: Marvin Rode <marvin.rode@capgemini.com> Co-authored-by: Marvin Rode (Cap) <127723478+marode-cap@users.noreply.github.com> Co-authored-by: Andre Blome <andre-david.blome@capgemini.com> * THR-7 ContentStorage: store data on S3 via S3ClientAdapter (#4198) * Add "List" and "Head" Methods to S3ClientAdapter * H5P Content Storage now uses S3 Bucket and database * Added database repo for H5P metadata * Thr 20 create UI elements (#4190) * Necessary changes to support H5P on the client * h5p editor: library storage s3 implementation * h5p editor: library storage updated; unit tests; bugfixes * change to correct Buckets / Key names * add config keys * change default value for env var S3_REGION * Fix API Validation * THR-25 S3 implementation of temporary file storage (#4204) * Created H5P microservice * Add kubernetes files for h5p microservice * Added @lumieducation/h5p-server package * Don't use mock auth for testing * Fix api tests to use new TestApiClient * WIP: H5P editor: temp file storage * WIP: H5P editor: added data classes * resolve merge conflict leftovers * h5p editor: temp file storage implementation completed * h5p editor: temp file storage tests added * h5p editor: temp file storage moved * h5p editor: temp storage files moved * h5p editor: temp file storage changed to use s3 * h5p editor: s3 config and adapter injection handled * h5p editor: temp file storage fixes * h5p editor: temp file storage: repo method renamed to findByUserAndFilename() * change to correct Buckets names * change to correct Bucket name * initial commit * Merge remote-tracking branch 'origin/THR-18-dev-feature-h5p-editor' into THR-25-temp-storage-s3 * Update configs * Update module and fix all tests * Fix module config * Fix Library Entity from THR-18 * Updated Tests * Remove console output * Remove config from development.json * Remove unused config file * Organized all files for H5P module --------- Co-authored-by: Marvin Rode <marvin.rode@capgemini.com> Co-authored-by: Marvin Rode (Cap) <127723478+marode-cap@users.noreply.github.com> Co-authored-by: Andre Blome <andre-david.blome@capgemini.com> Co-authored-by: Majed Mak <132336669+MajedAlaitwniCap@users.noreply.github.com> Co-authored-by: Majed Aitwni <majed.alaitwni@capgemini.com> * Thr 27 h5p language parameter (#4283) * adjust controller and uc for language params * refactor h5p-editor service * use userService in getUserLanguage() * use env available languages * Refactor merging * resolve bugs in uc tests * change h5p-editor dto * delete await from editor + player declarations --------- Co-authored-by: Marvin Rode <marvin.rode@capgemini.com> Co-authored-by: Marvin Rode (Cap) <127723478+marode-cap@users.noreply.github.com> Co-authored-by: Andre Blome <andre-david.blome@capgemini.com> Co-authored-by: Majed Aitwni <majed.alaitwni@capgemini.com> * THR-42 fixes to library storage (#4332) * Library storage THR-34 hotfixes * Fix small bugs in LibraryStorageService * Update Tests * resolved PR comments * delete h5p-editor static files * Remove deployment from main * remove static server * add LibraryStorage to uc tests * expect LanguageType in editor uc test * add missing language param at api tests * add field name to response type IGetFileResponse * change library entity constructor * refactor library entity constructor * add list of files (#4349) * add list of files * change s3 command type * fix tests * move recursive part to private methode * fix limits --------- Co-authored-by: Marvin Rode (Cap) <127723478+marode-cap@users.noreply.github.com> * code Smells corrector * THR-6 H5P editor authorization (#4364) * Preparing entities for authorization * Add entities * save and create tests * get player tests * get editor tests * files uc tests * last tests for changed UC * better error message * Document custom transform pipe * Fix PR comments * BaseEntityWithTimestamps for entities * Address more PR comments * pr comments * pr comments * Better file handling * Missing file * fix changes from s3 module * h5p editor: change api and static files urls * refactor library tests and imports * refactor imports of contentStorage and temporary file storage * refactor h5p content tests * delete unused imports * refactor h5p api tests * refactor imports in module * delete blank line * restructure api tests h5p-save-create * h5p editor: temporarily dont map errors * refactor tests regarding authorization * fix save-create api tests * quick fixes service tests * fix test in contentStorage service * refactor api editor files tests * set default parameter rangeStart = 0 * minor code structure adjustments * refactor: only one return statement * fix commit dd41731: redefine type of returnValue * resolve type error for response stream * revert minor code changes * review comments controller * Revert "review comments controller" * Fix s3 client mocks * Fix tests * fix file storage service test * Fix integration tests * add empty line before return * refactor: await below ifelse block * refactor: set default for rangeStart * Merge remote-tracking branch 'origin/main' into THR-18-dev-feature-h5p-editor * Fix imports * Remove import from files-storage * do not overwrite input parameter * change Error Type to NotAcceptable * create own h5pfile dto * fix remaining review comments in temporary-file-storage.service * refactor h5peditor/ h5pplayer service to provider Hint: npm ci * add test for new Repo Method * fix unit test for Temporary-file-storage * fix test Module imports * fix review comments: return method and error types * create seperate h5p error mapper - unit tests still WIP * update unit tests for uc.getAjax & uc.postAjax * resolve ts expect error & Test corrector * resolve rest review comments h5p.uc.ts * remove unused import * move PostBodyParamsTransformPipe to extra file * rename TemporaryFile to BaseEntityWithTimestamp * add private method and use cause error * refactor from Service to H5PAjaxEndpointProvider * use explicit return type * use cause error types * initial commit * create library.repo.spec test setup and first test * create tests for library.repo * revert * revert zu main * last review comments * fix library.repo.spec code coverage * Add library.entity.spec * Add test for existsOne to increase CodeCoverage * add h5p-content mapper Test * Fix eslint * fix Test * fix saveFile mockImplementation * fix unnecessary type assertion * import Correction * add tests libraryStorage.service * remove uneccessary * Remove uneccessary import * correction and add test for h5p mapper * remove uneccessary imports * remove unused import * test structure * add tests for library.entity * WIP | create tests post.body.params.transform-pipe * add tests for post.body.params.transform-pipe * remove unused parameter * Create Test for H5PContentMetadata * refactor Method * define method return type * Add test for temporary-file-storage.service * remove unused import * added H5p File Response interface * create tests h5p-content.entity * add GetLibraryFile dto * rename entity file * Create own GetLibraryFile Dto for UC * Revert post.body.params.transform-pipe & logging * bugfix post.body.params.transform-pipe * add return undefined * quick fix: transformed can be undefined * adapt userMcok to updated ICurrentUser Interface * Restructure AjaxPostBodyParamsTransformPipe * quick fix nest-cli.json * revert wrong Changes * Comment out end-to-end tests in push * Update push.yml * Update push.yml: add legacy e2e test again --------- Co-authored-by: André Blome <43345275+ssmid@users.noreply.github.com> Co-authored-by: Marvin Rode <marvin.rode@capgemini.com> Co-authored-by: Marvin Rode (Cap) <127723478+marode-cap@users.noreply.github.com> Co-authored-by: Andre Blome <andre-david.blome@capgemini.com> Co-authored-by: Stephan Krause <101647440+SteKrause@users.noreply.github.com> Co-authored-by: Majed Mak <132336669+MajedAlaitwniCap@users.noreply.github.com> Co-authored-by: stekrause <stephan.krause.ext@capgemini.com> Co-authored-by: Majed Aitwni <majed.alaitwni@capgemini.com> Co-authored-by: Sergej Hoffmann <97111299+SevenWaysDP@users.noreply.github.com> Co-authored-by: SevenWaysDP <sergej.hoffmann@dataport.de>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This is an implementation of interface ITemporaryFileStorage from the Lumi H5P editor. It is used by Lumi to store temporary files per user that are used inside a running H5P editor instance.
The implementation stores the files in the local filesystem, similar to the reference implementation of Lumi.
Links to Tickets or other pull requests
https://ticketsystem.dbildungscloud.de/browse/THR-1
Changes
temporary file storage added and corresponding data classes
Datasecurity
Deployment
New Repos, NPM pakages or vendor scripts
Approval for review