FI-3271/FI-3272: Implement Mock Server and CARIN tests for required/m…

…ust-support elements for Client-Based CARIN Testing (#47)

* Implement basic structure and Mock Server for carin client tests

* Add tests that check for the presence of each of the Carin profile resources, fix error that was thrown if request with no search params sent in, split out EOB resource test into seperate tests for each profile

* Add 2.0.0 version to client tests, clean up code, switch to use SuiteEndpoint functionality instead of record_response_route

* mock operation outcome reference typo

* FI-3355: Integrate Carin Client Test Suite with Reference Client (#48)

* Update Carin Patient from 998 to 888, update to allow multiple _include params in query, update to include endpoint for queries in the format

* Add cpcds client ri preset

* FI-3356: Add Required Searches (#53)

* Add all US Core required searches

* Create first required search test to ensure functionality works correctly

* Add required search tests for each resource type, and update the way tagging works to allow for these tests to work

* Add postman collection for required search tests, and fix bug found when reference server returns operation outcome

* Make PR fixes, get rid of US Core 3.1.1 searches, update the way required searches get stored in scratch

* Get rid of initial scratch storing test in claims data request test group

* Fix how tagged requests are associated with tests so that they use the resource type to grab the correct requests, and add spec tests for the client tests

* Fix typo bug in resource_api_endpoint

* Associate requests with correct resource for search tests

* Add US Core searches back

* Update organization of tests so that there is only 1 wait test, and each claims data request test and search test is separated into their respective resource profile group. Also update spec tests to work with new changes

* FI-3359: Prepare Carin Client Test Suite for Release (#54)

* Code cleanup and add descriptions for each test, test group, and client test suite

* Make documentation improvements, update Postman collection to point to reference server, add verifies_requirements to tests

* Make PR documentation improvement updates, add client suite information to README

* Make additional PR documentation/test organization improvements

* update requirements tools and generate coverage

---------

Co-authored-by: Karl Naden <knaden@mitre.org>

* Run bundle install after rebase

---------

Co-authored-by: Karl Naden <knaden@mitre.org>

.env.development

@@ -1,2 +1,4 @@
 FHIR_RESOURCE_VALIDATOR_URL=http://localhost/hl7validatorapi
-REDIS_URL=redis://localhost:6379/0
+REDIS_URL=redis://localhost:6379/0
+FHIR_REFERENCE_SERVER=http://localhost:8080/reference-server/r4
+HOST_HEADER=localhost:8080

.env.production

@@ -1,3 +1,5 @@
 INFERNO_HOST=http://localhost
 REDIS_URL=redis://redis:6379/0
 FHIR_RESOURCE_VALIDATOR_URL=http://hl7_validator_service:3500
+FHIR_REFERENCE_SERVER=http://host.docker.internal:8080/reference-server/r4
+HOST_HEADER=host.docker.internal:8080

.env.test

@@ -1,2 +1,4 @@
 VALIDATOR_URL=https://example.com/validatorapi
+FHIR_REFERENCE_SERVER=http://example.com/reference-server/r4
+HOST_HEADER=localhost:8080
 ASYNC_JOBS=false

Gemfile

@@ -1,6 +1,6 @@
 # frozen_string_literal: true
 
-source "https://rubygems.org"
+source 'https://rubygems.org'
 
 gemspec
 
@@ -9,3 +9,11 @@ group :development, :test do
   gem 'foreman'
   gem 'roo', '~> 2.7.1'
 end
+
+group :test do
+  gem 'database_cleaner-sequel', '~> 1.8'
+  gem 'factory_bot', '~> 6.1'
+  gem 'rack-test'
+  gem 'rspec', '~> 3.10'
+  gem 'webmock', '~> 3.11'
+end

Gemfile.lock

@@ -8,7 +8,7 @@ PATH
 GEM
   remote: https://rubygems.org/
   specs:
-    activesupport (6.1.7.10)
+    activesupport (6.1.7.8)
       concurrent-ruby (~> 1.0, >= 1.0.2)
       i18n (>= 1.6, < 2)
       minitest (>= 5.1)
@@ -132,12 +132,12 @@ GEM
     hansi (0.2.1)
     hashdiff (1.1.1)
     http-accept (1.7.0)
-    http-cookie (1.0.7)
+    http-cookie (1.0.8)
       domain_name (~> 0.5)
     httpclient (2.8.3)
     i18n (1.14.6)
       concurrent-ruby (~> 1.0)
-    inferno_core (0.5.0)
+    inferno_core (0.5.2)
       activesupport (~> 6.1.7.5)
       base62-rb (= 0.3.1)
       blueprinter (= 0.25.2)
@@ -175,14 +175,14 @@ GEM
       httpclient
     jwt (2.9.3)
       base64
-    kramdown (2.4.0)
-      rexml
-    logger (1.6.1)
+    kramdown (2.5.0)
+      rexml (>= 3.3.6)
+    logger (1.6.2)
     method_source (1.1.0)
     mime-types (3.6.0)
       logger
       mime-types-data (~> 3.2015)
-    mime-types-data (3.2024.1105)
+    mime-types-data (3.2024.1203)
     mini_portile2 (2.8.7)
     minitest (5.25.1)
     multi_json (1.15.0)
@@ -214,19 +214,21 @@ GEM
     oj (3.11.0)
     pastel (0.8.0)
       tty-color (~> 0.5)
-    pry (0.14.2)
+    pry (0.15.0)
       coderay (~> 1.1)
       method_source (~> 1.0)
-    pry-byebug (3.10.1)
+    pry-byebug (3.8.0)
       byebug (~> 11.0)
-      pry (>= 0.13, < 0.15)
+      pry (~> 0.10)
     psych (5.1.2)
       stringio
     public_suffix (6.0.1)
     puma (5.6.9)
       nio4r (~> 2.0)
     racc (1.8.1)
-    rack (2.2.10)
+    rack (2.2.9)
+    rack-test (2.1.0)
+      rack (>= 1.3)
     rake (13.2.1)
     rdoc (6.7.0)
       psych (>= 4.0.0)
@@ -239,11 +241,11 @@ GEM
       http-cookie (>= 1.0.2, < 2.0)
       mime-types (>= 1.16, < 4.0)
       netrc (~> 0.8)
-    rexml (3.3.9)
+    rexml (3.3.8)
     roo (2.7.1)
       nokogiri (~> 1)
       rubyzip (~> 1.1, < 2.0.0)
-    rouge (4.4.0)
+    rouge (4.5.1)
     rspec (3.13.0)
       rspec-core (~> 3.13.0)
       rspec-expectations (~> 3.13.0)
@@ -317,6 +319,7 @@ DEPENDENCIES
   debug
   factory_bot (~> 6.1)
   foreman
+  rack-test
   roo (~> 2.7.1)
   rspec (~> 3.10)
   webmock (~> 3.11)

README.md

@@ -5,23 +5,94 @@ for the CARIN IG for Blue Button®
 [v1.1.0](http://hl7.org/fhir/us/carin-bb/STU1.1), and
 [v2.0.0](http://hl7.org/fhir/us/carin-bb/STU2).
 
+The test kit includes suites targeting the following actors from the specification:
+ - CARIN IG for Blue Button® Requestor: Inferno will act as a server implementing the CARIN IG for Blue Button
+ and respond to requests for patient claims data made by the client under test.
+ - CARIN IG for Blue Button® Responder: Inferno will act as a client and make a series of requests to the server under test requesting CARIN for Blue Button patient claims data.
+
 ## Instructions
 
 It is highly recommended that you use [Docker](https://www.docker.com/) to run
 these tests so that you don't have to configure ruby and the HL7® FHIR®
 validator service. For more information on how to run Inferno, visit [Inferno's
 documentation](https://inferno-framework.github.io/inferno-core/getting-started.html)
 
+### ONC Hosted Instance
+
+You can run the CARIN IG for Blue Button test kit via the [ONC Inferno](https://inferno.healthit.gov/test-kits/carin-for-blue-button/) website by choosing the "CARIN IG for Blue Button Test Kit" test kit.
+
+### Local Inferno Instance
+
 - Clone this repo.
 - Run `setup.sh` in this repo.
 - Run `run.sh` in this repo.
 - Navigate to `http://localhost`. The US Core test suite will be available.
 
+## CARIN IG for Blue Button Client Test Suite 
+
+The CARIN for Blue Button v2.0.0 Client test suite verifies the conformance of client systems to the STU2 HL7® FHIR®
+[CARIN for Blue Button® Implementation Guide](http://hl7.org/fhir/us/carin-bb/STU2). These tests validate that that a
+FHIR Client can play the role of a CARIN IG for Blue Button® Requestor and retrieve and use data defined by the CARIN for
+Blue Button® IG.
+
+### Sample Postman Collection
+
+To try out these tests without a CARIN for Blue Button client implementation, you may
+run them using [this Postman collection](https://github.com/inferno-framework/carin-for-blue-button-test-kit/blob/config/presets/C4BB%20Client%20Search%20Tests.postman_collection.json). This Postman collection includes all of the required CARIN for Blue Button profile resource requests and required search
+requests needed to pass all of the tests.
+
+To run the client tests against the Postman collection:
+1. Start an Inferno session of the CARIN for Blue Button Client test suite.
+3. Click the "Run All Tests" button in the upper right and type in "SAMPLE_TOKEN" for the `access_token` input in the dialog that appears.
+4. Click the "Submit" button. The simulated server will then be waiting for an interaction.
+4. Open Postman and import the `C4BB Client Search Tests` Postman collection.
+5. Send each of the requests listed under the `C4BB Client Search Tests` Postman collection and ensure a
+   200 response is received along with any requested CARIN for Blue Button resources.
+6. Once you have finished making requests, click the "Click here" link in the wait dialog to evaluate the requests.
+7. An attestation dialog will appear asking the client to attest that it was able to process each of the 
+   the resources it received. Click the first "Click here" link in the wait dialog to pass the test.
+
+### CPCDS Reference Implementation Client
+
+To try out these tests without a Carin for Blue Button client implementation, you may
+run them using the [CPCDS Reference Implementation Client](https://github.com/carin-alliance/cpcds-client-ri). Use this
+[forked repository](https://github.com/emichaud998/cpcds-client-ri), which been adjusted to work with this test kit.
+
+1. Follow the instructions listed [here](https://github.com/carin-alliance/cpcds-client-ri?tab=readme-ov-file#running-app-locally)
+   to get the CPCDS Reference Implementation Client running locally.
+      Note: If you install rails with the command `gem install rails`, you must run any of the rails commands in the README by prepending
+      it with `bundle exec rails`
+
+To run the client tests against the CPCDS Reference Implementation Client:
+1. Start an Inferno session of the Carin for Blue Button Client test suite.
+2. Select the *Carin CPCDS Reference Implementation Client* from the Preset dropdown in the
+   upper left.
+3. Click the "Wait for Claims Data and Search Requests" test group in the left navigation sidebar.
+4. Click the "Run Tests" button in the upper right and click the "Submit" button in the dialog
+   that appears. The simulated server will then be waiting for an interaction.
+5. Navigate to `localhost:3000`. Enter the Carin Client Suite FHIR endpoint URL, and then type the Carin patient id, `888`,
+   into the client secret and client id fields. Hit the "Connect" button.
+6. On the next page, hit the "Display" button to make a request to the Carin Client Suite .
+7. In the Inferno Client Suite, click the "Click here" link in the wait dialog to signal the client has finished submitting requests.
+8. Navigate to each Carin for Blue Button Profile test group in the left navigation sidebar, and for each test group hit the run
+   icon next all the tests listed except for the last required search parameters test
+   (search requests tests will not pass if ran with the CPCDS Client).
+
+### FHIR Server Simulation for the CARIN IG for Blue Button Client Test Suite
+
+The CARIN IG for Blue Button Client Test Suite test suite needs to be able to return responses to FHIR read and search APIs.
+These responses can be complex and so the suite relies on a full FHIR server to provide
+responses for it to provide back to systems under test. The test kit was written to work
+with the [Inferno Reference Server](https://github.com/inferno-framework/inferno-reference-server)
+
+- loaded with [patient 888](https://github.com/inferno-framework/inferno-reference-server/blob/main/resources/carin_bundle_patient_888.json)
+- accepting bearer token `SAMPLE_TOKEN` for read access.
+
 ## TEST GENERATION
 The CARIN IG for Blue Button® Test Kit has an implemeneted test generator. It
-extracts necessarry data elements from CARIN for Blue Button Implementation
+extracts necessary data elements from CARIN for Blue Button Implementation
 Guide archive files and generates tests accordingly. The repo currently contains
-suites for IG versions 1.1.0 and 2.0.0.
+generated test suites for IG versions 1.1.0 and 2.0.0.
 
 To generate a test suite for a different CARIN for Blue Button IG version:
 - Navigate to

Rakefile

@@ -28,34 +28,16 @@ namespace :requirements do
     require 'inferno'
     Inferno::Application.start(:suites)
 
-    require_relative 'lib/inferno_requirements_tools/tasks/map_requirements'
-    InfernoRequirementsTools::Tasks::MapRequirements.new.run
+    require_relative 'lib/inferno_requirements_tools/tasks/requirements_coverage'
+    InfernoRequirementsTools::Tasks::RequirementsCoverage.new.run
   end
-end
 
-namespace :requirements do
   desc 'Check if requirements coverage CSV is up-to-date'
   task :check_coverage do
     require 'inferno'
     Inferno::Application.start(:suites)
 
-    require_relative 'lib/inferno_requirements_tools/tasks/map_requirements'
-    InfernoRequirementsTools::Tasks::MapRequirements.new.run_check
-  end
-end
-
-namespace :requirements do
-  desc 'Collect requirements and planned not tested requirements into CSVs'
-  task :collect_requirements, [:input_directory] => [] do |_, args|
-    require_relative 'lib/inferno_requirements_tools/tasks/collect_requirements'
-    InfernoRequirementsTools::Tasks::CollectRequirements.new.run(args.input_directory)
-  end
-end
-
-namespace :requirements do
-  desc 'Check if requirements and planned not tested CSVs are up-to-date'
-  task :check_collected_requirements, [:input_directory] => [] do |_, args|
-    require_relative 'lib/inferno_requirements_tools/tasks/collect_requirements'
-    InfernoRequirementsTools::Tasks::CollectRequirements.new.run_check(args.input_directory)
+    require_relative 'lib/inferno_requirements_tools/tasks/requirements_coverage'
+    InfernoRequirementsTools::Tasks::RequirementsCoverage.new.run_check
   end
 end