Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ingestion attempt communication with PMPY #3049

Merged
merged 7 commits into from
Feb 3, 2023
Merged

Add ingestion attempt communication with PMPY #3049

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
35ff0b2
Add ingestion attempt communication with PMPY
lagoan Jan 26, 2023
f3087fb
Merge branch 'master' into add/entity_ingestion_attempts_priority
lagoan Jan 30, 2023
384d5d6
Add CHANGELOG entry
lagoan Jan 30, 2023
930cec8
Merge branch 'add/entity_ingestion_attempts_priority' of github.com:u…
lagoan Jan 30, 2023
5a3d545
Remove brakeman's EOLRuby check.
lagoan Jan 31, 2023
8f0f3b8
Improve code documentation for pmpy communication.
lagoan Feb 2, 2023
40fbc8b
Merge branch 'master' into add/entity_ingestion_attempts_priority
lagoan Feb 3, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ New entries in this file should aim to provide a meaningful amount of informatio
### Added
- Push collections and communities to preservation on save along with a rake task to do so [#255](https://github.com/ualbertalib/pushmi_pullyu/issues/255)
- Added minitest-retry gem to retry flapping tests that are able to pass through retries [#3044](https://github.com/ualbertalib/jupiter/pull/3044)
- Add attempt count for entity ingestion on depositable model. Related to [pushmi_pullyu#297](https://github.com/ualbertalib/pushmi_pullyu/issues/297)
pgwillia marked this conversation as resolved.
Show resolved Hide resolved
## [2.4.3] - 2022-12-14

### Security
Expand Down
5 changes: 3 additions & 2 deletions app/models/jupiter_core/depositable.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -170,8 +170,9 @@ def push_entity_for_preservation

$queue.with do |connection|
# pushmi_pullyu requires both the id and type of the depositable
entity = { uuid: id, type: self.class.table_name }
connection.zadd queue_name, Time.now.to_f, entity.to_json
connection.zadd(queue_name, Time.now.to_f, { uuid: id, type: self.class.table_name }.to_json)
# Here we add the attempt pmpy will be trying to ingest the depositable. We start at 0
connection.set("#{Rails.application.secrets.attempt_ingest_prefix}#{id}", 0)
pgwillia marked this conversation as resolved.
Show resolved Hide resolved
end

true
Expand Down
1 change: 1 addition & 0 deletions config/brakeman.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,5 @@
:run_all_checks: true
:skip_checks:
- CheckUnscopedFind
- CheckEOLRuby
pgwillia marked this conversation as resolved.
Show resolved Hide resolved
:github_repo: ualbertalib/jupiter
6 changes: 5 additions & 1 deletion config/secrets.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,8 @@ shared:
google_analytics_token: <%= ENV['GOOGLE_ANALYTICS_TOKEN'] %>
allow_crawlers: <%= ENV['RAILS_ALLOW_CRAWLERS'] || false %>
preservation_queue_name: <%= ENV['PMPY_QUEUE_NAME'] || 'prod:pmpy_queue' %>
attempt_ingest_prefix: <%= ENV['ATTEMPT_INGEST_PREFIX'] || "'prod:pmpy_ingest_attempt:'" %>

fits_path: <%= ENV['FITS_PATH'] || 'fits.sh' %>
rack_attack_safelisted_ips: <%= ENV['RACK_ATTACK_SAFELISTED_IPS'] || '""' %>
system_user_api_key: <%= ENV['SYSTEM_USER_API_KEY'] %>
Expand Down Expand Up @@ -62,7 +64,8 @@ development:
fcrepo_password: fedoraAdmin
fcrepo_base_path: /dev

preservation_queue_name: <%= ENV['PMPY_QUEUE_NAME'] || 'dev:pmpy_queue'%>
preservation_queue_name: <%= ENV['PMPY_QUEUE_NAME'] || 'dev:pmpy_queue' %>
attempt_ingest_prefix: <%= ENV['ATTEMPT_INGEST_PREFIX'] || "'dev:pmpy_ingest_attempt:'" %>
system_user_api_key: <%= ENV['SYSTEM_USER_API_KEY'] || '3eeb395e-63b7-11ea-bc55-0242ac130003' %>

test:
Expand All @@ -85,6 +88,7 @@ test:
fcrepo_base_path: /test

preservation_queue_name: <%= ENV['PMPY_QUEUE_NAME'] || 'test:pmpy_queue' %>
attempt_ingest_prefix: <%= ENV['ATTEMPT_INGEST_PREFIX'] || "'test:pmpy_ingest_attempt:'" %>
system_user_api_key: <%= ENV['SYSTEM_USER_API_KEY'] || '3eeb395e-63b7-11ea-bc55-0242ac130003' %>

google_client_id: 'test-google-client-id'
Expand Down