Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Contributions from PCO #67

Merged
merged 59 commits into from
May 6, 2024
Merged

Contributions from PCO #67

merged 59 commits into from
May 6, 2024

Conversation

danielaboost
Copy link
Contributor

@danielaboost danielaboost commented May 5, 2024 •
edited
Loading

  • PDF Extraction
  • S3 extraction example
  • Fix time calculation for when creating a pipeline that uses an existing extraction

richardmatthewsdev and others added 30 commits March 5, 2024 13:46
@richardmatthewsdev
add PDF as an available option for the extraction definition
49021ad
@richardmatthewsdev
feat(pdf_extraction): Add the ability to extract text from a PDF
7b79fd0
@richardmatthewsdev
(refactor) rename the PDF extraction worker to a text extraction work…
6910597 
…er as it is going to work with more formats than just PDF
@richardmatthewsdev
feat(extract_from_pdf): Initial Working for extracting text from a PD…
07c5b5e 
…F through the UI
@richardmatthewsdev
Add new fields to the extraction definition to determine if a file ne…
a22bf83 
…eds to have its text extracted
@richardmatthewsdev
Remove the credentials file from Git
f11fc4d
@richardmatthewsdev
Update the harvester to be able to display and transform text extract…
b2f8163 
…ed from a document
@richardmatthewsdev
Add Text from file extraction to enrichments
a4fe99c
@richardmatthewsdev
feat(pdf_extraction): Allow the File Extraction to run as part of an …
9e2ddab 
…enrichment
@eoin-boost
ci: Remove mirror to Gitlab workflow
ea20ebf
@richardmatthewsdev
ERB lint and rubocop
17e84ea
@richardmatthewsdev
Merge branch 'rm/extract-data-from-documents' of github.com:boost/pco…
42ce3b9 
…_harvester into rm/extract-data-from-documents
@richardmatthewsdev
Bundle audit update
a630943
@richardmatthewsdev
Update brakeman
cf5aeed
@richardmatthewsdev
Rubocop and erb lint
034ec7a
@richardmatthewsdev
Remove needing the credentials file
6e0bfd8
@richardmatthewsdev
feat(extract_pdf): Do not display the preview for extractions that ne…
8b3d978 
…ed to be extracted from a file
@richardmatthewsdev
refactor(extract_pdf): Reduce duplication in transformed record class
674a2ea
@richardmatthewsdev
Prettier
36981a6
@richardmatthewsdev
refactor(extract_pdf): Write binary files
3c30fca 
Remove need for checking if the file is a binary when writing it to the filesystem.
@richardmatthewsdev
Update dockerfile
72ee9ca
@richardmatthewsdev
Add the secret key as an argument to the harvester pipeline
3244a56
@richardmatthewsdev
rename secret key to secret key base
5fd76d5
@richardmatthewsdev
Correct typo
535a41f
@richardmatthewsdev
Fix issue with secret key base naming
04f47db
@richardmatthewsdev
fix(pdf_extraction): Improve disabled checking for enrichment and har…
a9bec5c 
…vest definition templates
@richardmatthewsdev
Add missing question mark to completed method
bb1c864
@richardmatthewsdev
Stop the extraction show page from crashing while the text extraction…
6840a66 
… is running
@richardmatthewsdev
Dont mark the extraction as complete when it is not
19ef18c
@richardmatthewsdev
Fix for files that need to be extracted saying that they are complete…
9e0d3ab 
… when the extraction is still running
richardmatthewsdev and others added 25 commits April 15, 2024 13:59
@richardmatthewsdev
feat(s3_extraction): Add first implementation of the S3 extraction to…
36cf54f 
… the harvester
@richardmatthewsdev
Update S3 extraction to take an argument of the bucket path
672e550
@richardmatthewsdev
Bundle audit and rubocop
7b03d6f
@richardmatthewsdev
Merge pull request #4 from boost/rm/lenz-ingest
0b87afc 
Full LENZ ingest
@richardmatthewsdev
Create batches of 100 files rather than 1 file per record
1602354
@richardmatthewsdev
rubocop
f8b8adf
@richardmatthewsdev
Merge pull request #5 from boost/rm/lenz-ingest-perf
540f89c 
S3 performance
@richardmatthewsdev
fix(load-retry): Correct issue where Load Worker wasnt retrying when …
6a4cbc4 
…the API responded with a 500
@richardmatthewsdev
rspec
79f3802
@richardmatthewsdev
Rspec
ca9e361
@richardmatthewsdev
Merge pull request #6 from boost/rm/load-retry
b6e53e4 
Load Retry
@richardmatthewsdev
Stagger load processes from harvests that use an existing extraction …
1bb481f 
…so not to smash the API
@richardmatthewsdev
Merge branch 'main' into rm/load-retry
ef66e8a
@richardmatthewsdev
Rubocop
206d63e
@richardmatthewsdev
Fix specs
9ccc5ae
@richardmatthewsdev
Merge pull request #7 from boost/rm/load-retry
ba2461a 
Fix issue with harvests that use an existing extraction
@richardmatthewsdev
Increase delay between jobs that are scheduled when using an existing…
f53f4c7 
… extraction definition
@richardmatthewsdev
Merge branch 'main' into rm/load-retry
263cc91
@richardmatthewsdev
Merge pull request #8 from boost/rm/load-retry
f60ba10 
Increase delay between jobs that are scheduled when using an existing extraction definition
@richardmatthewsdev
fix(harvest-report-time): Correct issue when the transformation and l…
7efe875 
…oad start times were being overwritten everytime a job started
@richardmatthewsdev
fix failing spec
f00d9fa
@richardmatthewsdev
Merge pull request #9 from boost/rm/existing-extractions-time-wrong
655fcbf 
Harvests using existing extractions duration time is wrong
@richardmatthewsdev
remove PCO LENZ sync task from source code
48f6b86
@richardmatthewsdev
Update comments to be more useful for future users
8af685d
@richardmatthewsdev
Merge pull request #10 from boost/rm/move-lenz-data-extract-example
076c450 
remove PCO LENZ sync task from source code
@github-actions GitHub Actions
Copy link

github-actions bot commented May 5, 2024

Code quality score

Nice work!! The code quality has improved for this PR! ✨ 🌈 🎉 🌟

Ruby file count Similarity score (flay) ABC complexity (flog) Code smells (reek) TOTALS
base 86 6.5 5.36 16.8 28.66
this branch 88 6.12 5.36 17.1 28.58
difference 2 -0.38 0.0 ⚠️ 0.3 -0.08

richardmatthewsdev
richardmatthewsdev approved these changes May 6, 2024
View reviewed changes
Copy link
Contributor

@richardmatthewsdev richardmatthewsdev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@danielaboost danielaboost merged commit ea7344c into main May 6, 2024
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@richardmatthewsdev richardmatthewsdev richardmatthewsdev approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@danielaboost @richardmatthewsdev @eoin-boost