Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IS-11: Test suites for Stream Compatibility Management #728

Open
wants to merge 86 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
86 commits
Select commit Hold shift + click to select a range
4b52d49
Placeholders for "IS-11 Stream Compatibility Management API", "IS-11 …
garethsb Oct 3, 2022
08b7444
Merge pull request #757 from AMWA-TV/master
garethsb Jan 27, 2023
2a2e7ce
Merge remote-tracking branch 'origin/master' into is-11
garethsb Mar 31, 2023
4da3b4a
* Initial submit for tests general (0), Senders (2), Outputs (3), Rec…
gwgeorgea Apr 27, 2023
1bb8da3
Merge remote-tracking branch 'origin/master' into is-11
garethsb Apr 27, 2023
90c9b24
Adding IS-11 in README.md
gwgeorgea Apr 27, 2023
1a56c8f
Merge pull request #801 from gwgeorgea/is-11-Readme
garethsb Apr 27, 2023
b745197
Fix Issue 793 and in general review all FAIL and UNCLEAR messages to …
gwgeorgea Apr 28, 2023
3515a39
Change wording on a few FAIL message items
gwgeorgea Apr 28, 2023
c2e4e7e
Check IS-11 control advertisement like other test suites do
N-Nagorny Apr 30, 2023
6da067a
Remove a blank line
N-Nagorny Apr 30, 2023
f2f4850
Fix gareth's mentions
gwgeorgea May 2, 2023
91fc894
Removing, accidentally commited file
gwgeorgea May 2, 2023
82113dd
Pull up Device control advertisement test to NMOSUtils
N-Nagorny May 5, 2023
ffb1650
Reorder args
N-Nagorny May 5, 2023
b9e00ca
Merge remote-tracking branch 'origin/master' into is-11-is-04-adverti…
garethsb May 6, 2023
b5ac172
Merge pull request #806 from N-Nagorny/is-11-is-04-advertisement
garethsb May 6, 2023
38cacef
Merge branch 'is-11' into is-11-Issues-793
garethsb May 22, 2023
c0f7812
Fixing comments from gareth in PR-802
gwgeorgea Jun 12, 2023
e0852ef
Remove accidentally change text regarding "The receivers"
gwgeorgea Jun 13, 2023
55e9007
Improve messages
garethsb Jun 13, 2023
6f36b1b
Merge remote-tracking branch 'origin/master' into is-11
garethsb Jun 13, 2023
ba3a49d
Adding 'valid' verification to Fix issue https://github.com/AMWA-TV/n…
gwgeorgea Jun 14, 2023
14bccb2
Removing urlparse unused at this time for this PR
gwgeorgea Jun 14, 2023
c6c0c91
Extract fetching IS-11 resources to set_up_tests
N-Nagorny Jun 16, 2023
208cd88
Remove unused functions
N-Nagorny Jun 16, 2023
ca2ffae
Merge pull request #816 from gwgeorgea/is-11-Fixing-790
N-Nagorny Jun 20, 2023
7413022
[IS-11] Add more tests (#818)
N-Nagorny Jun 20, 2023
1c2c704
Adding Input test and Sender test 2.3.x
gwgeorgea May 24, 2023
40629be
Remove Multipart payload
gwgeorgea Jun 16, 2023
afae62f
fixing flake8 issues
gwgeorgea Jun 16, 2023
04e7cf8
Merging with latest is-11 branch using IS11Utils
gwgeorgea Jul 4, 2023
0a42cfd
Fixing flake8 errors
gwgeorgea Jul 4, 2023
a3183d1
Remove extra space
gwgeorgea Jul 4, 2023
c450d61
Adding Output test cases 4.3 and 4.4
gwgeorgea May 12, 2023
db603d2
Fix flake8 blank line
gwgeorgea May 12, 2023
0939f57
Fix Flake8 E275 missing whitespace after keyword
gwgeorgea May 12, 2023
75cb766
Fixing various comments from Gareth
gwgeorgea May 24, 2023
99eca0f
- Changing DELAY Variable to STABLE_STATE_DELAY
gwgeorgea May 31, 2023
45c179b
Now using flowid instead of caps mediatype for the sender.
gwgeorgea Jun 14, 2023
1c3710d
Now verifying that the sender mediatype is either Audio or Video
gwgeorgea Jun 19, 2023
350d22f
merging with is-11 latest commits
gwgeorgea Jul 5, 2023
5118c03
Removing duplicate functions introduced during merged.
gwgeorgea Jul 5, 2023
acf44fd
Reverting unwanted changes
gwgeorgea Jul 5, 2023
275acfb
Fixes
N-Nagorny Jul 11, 2023
36ec684
Merge pull request #4 from gwgeorgea/is-11-fix-tests-of-statuses-whil…
gwgeorgea Jul 17, 2023
d0e7447
Merge pull request #811 from gwgeorgea/is-11-Adding-4-3-and-4-4
N-Nagorny Jul 17, 2023
8b7f7cc
Merge branch 'is-11' into is-11-NewInputTests
gwgeorgea Jul 17, 2023
49227c4
Merge pull request #813 from gwgeorgea/is-11-NewInputTests
N-Nagorny Jul 22, 2023
6d37b24
Fix typos and failure reasons
N-Nagorny Jul 22, 2023
44754c1
Rework output tests
N-Nagorny Jul 22, 2023
6215af8
Fix linting
N-Nagorny Jul 22, 2023
562a440
Update nmostesting/Config.py
N-Nagorny Jul 25, 2023
d9f3955
Merge pull request #823 from N-Nagorny/is-11-output-tests-rework
N-Nagorny Jul 27, 2023
7ac91e8
- Added Senders EDID Test 2.3.5.1, 2.3.5.2
gwgeorgea Jul 31, 2023
871d63c
- Added invalid invalid_edid in test data
gwgeorgea Jul 31, 2023
b584f33
Merge pull request #824 from gwgeorgea/is-11-LastSenderTests
N-Nagorny Aug 4, 2023
deb2b14
Merge branch 'master' into is-11
N-Nagorny Aug 8, 2023
412066f
- Fix Test 4.4 to support devices without output
gwgeorgea Oct 5, 2023
d569132
Rewrite tests of Inputs in the independent manner
N-Nagorny Oct 11, 2023
4b9debb
Add missed default Effective EDID get
N-Nagorny Oct 12, 2023
d3bb288
Rewrite tests of Outputs in the independent manner
N-Nagorny Oct 12, 2023
e994801
Return the valid and the invalid EDID examples to the filesystem
N-Nagorny Oct 26, 2023
cf451f0
Clean IS-11 helper functions
N-Nagorny Oct 26, 2023
5a64c55
Turn back the top-level sectioning
N-Nagorny Oct 31, 2023
1f83676
Insert waits between modifying Base EDID and GETting Effective EDID
N-Nagorny Oct 31, 2023
bfbd5df
Add "adjust_to_caps" into test 06.04
N-Nagorny Nov 1, 2023
4360422
Merge pull request #834 from gwgeorgea/is-11-Add-Support-without-output
N-Nagorny Nov 1, 2023
404c863
Merge branch 'master' into is-11
N-Nagorny Nov 1, 2023
d52c5f3
Merge branch 'master' into is-11-independent-test-cases
N-Nagorny Nov 1, 2023
d0bbe2b
- Fix occasional exceptions caused by undefined variables.
gwgeorgea Nov 1, 2023
d783ba2
Add tear_down_tests
N-Nagorny Nov 1, 2023
bf3999c
Merge remote-tracking branch 'upstream/is-11' into is-11
N-Nagorny Nov 2, 2023
11dba23
Merge branch 'is-11' into is-11-independent-test-cases
N-Nagorny Nov 2, 2023
57a4d79
Merge pull request #836 from N-Nagorny/is-11-independent-test-cases
N-Nagorny Nov 3, 2023
c6e0864
Use the GenericTest::do_request instead of raw request to enable HTTP…
lo-simon Mar 13, 2024
e92a93e
Fix indent and remove unused request import
lo-simon Mar 13, 2024
0201393
Fix indent
lo-simon Mar 13, 2024
4768170
Replace TestHelper with self
lo-simon Mar 14, 2024
31aba4a
do_request doesn't take 4 parameters
lo-simon Apr 18, 2024
5e015e2
Update tests description
lo-simon Apr 18, 2024
09fc19d
Correct test_02_03_05_02 warning message
lo-simon Apr 18, 2024
3882388
Shorten the lines to prevent flake8 line too lomg error
lo-simon Apr 18, 2024
2f2a5b1
Merge pull request #849 from lo-simon/fix-test_02_03_04
N-Nagorny Apr 27, 2024
adae812
Merge pull request #838 from gwgeorgea/is-11-Senders-outputs-exceptio…
N-Nagorny Apr 27, 2024
0dafb47
Merge branch 'master' into is-11
N-Nagorny May 14, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ The following test suites are currently supported.
| IS-09-01 | IS-09 System API | | (X) | | System Parameters Server |
| IS-09-02 | IS-09 System API Discovery | X | | | |
| IS-10-01 | IS-10 Authorization API | | | | Authorization Server |
| IS-11-01 | IS-11 Stream Compatibility Management API | X | | | |
| - | BCP-002-01 Natural Grouping | X | | | Included in IS-04 Node API suite |
| - | BCP-002-02 Asset Distinguishing Information | X | | | Included in IS-04 Node API suite |
| BCP-003-01 | BCP-003-01 Secure Communication | X | X | | See [Testing TLS](docs/2.2.%20Usage%20-%20Testing%20BCP-003-01%20TLS.md) |
Expand Down
17 changes: 17 additions & 0 deletions nmostesting/Config.py
Original file line number Diff line number Diff line change
Expand Up @@ -277,6 +277,17 @@
}
}
},
"is-11": {
"repo": "is-11",
"versions": ["v1.0"],
"default_version": "v1.0",
"apis": {
"streamcompatibility": {
"name": "Stream Compatibility Management API",
"raml": "StreamCompatibilityManagementAPI.raml"
}
}
},
"bcp-002-01": {
"repo": "bcp-002-01",
"versions": ["v1.0"],
Expand Down Expand Up @@ -313,6 +324,12 @@
}
}
},
"bcp-005-01": {
"repo": "bcp-005-01",
"versions": ["v1.0"],
"default_version": "v1.0",
"apis": {}
},
"nmos-parameter-registers": {
"repo": "nmos-parameter-registers",
"versions": ["main"],
Expand Down
100 changes: 98 additions & 2 deletions nmostesting/IS04Utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,11 @@
# See the License for the specific language governing permissions and
# limitations under the License.

import re
from copy import deepcopy
from fractions import Fraction
from . import TestHelper

from .NMOSUtils import NMOSUtils
from copy import deepcopy


class IS04Utils(NMOSUtils):
Expand Down Expand Up @@ -71,6 +72,101 @@ def get_resources(self, resource, url=None):
return toReturn

@staticmethod
def comparable_parameter_constraint_value(value):
if isinstance(value, dict):
return Fraction(value["numerator"], value.get("denominator", 1))
return value

@staticmethod
def comparable_parameter_constraint(param_constraint):
result = deepcopy(param_constraint)
if "minimum" in result:
result["minimum"] = IS04Utils.comparable_parameter_constraint_value(
result["minimum"]
)
if "maximum" in result:
result["maximum"] = IS04Utils.comparable_parameter_constraint_value(
result["maximum"]
)
if "enum" in result:
for i, value in enumerate(result["enum"]):
result["enum"][i] = IS04Utils.comparable_parameter_constraint_value(
value
)
return result

@staticmethod
def comparable_constraint_set(constraint_set):
result = {
k: IS04Utils.comparable_parameter_constraint(v)
if re.match("^urn:x-nmos:cap:(?!meta:)", k)
else v
for k, v in constraint_set.items()
}
# could also remove urn:x-nmos:cap:meta:label?
if "urn:x-nmos:cap:meta:preference" not in result:
result["urn:x-nmos:cap:meta:preference"] = 0
if "urn:x-nmos:cap:meta:enabled" not in result:
result["urn:x-nmos:cap:meta:enabled"] = True
return result

@staticmethod
def comparable_constraint_sets(constraint_sets):
return [IS04Utils.comparable_constraint_set(_) for _ in constraint_sets]

@staticmethod
def compare_constraint_sets(lhs, rhs):
"""Check that two Constraint Sets arrays are closely equivalent"""
return TestHelper.compare_json(
IS04Utils.comparable_constraint_sets(lhs),
IS04Utils.comparable_constraint_sets(rhs),
)

@staticmethod
def make_sampling(flow_components):
samplers = {
# Red-Green-Blue-Alpha
"RGBA": {"R": (1, 1), "G": (1, 1), "B": (1, 1), "A": (1, 1)},
# Red-Green-Blue
"RGB": {"R": (1, 1), "G": (1, 1), "B": (1, 1)},
# Non-constant luminance YCbCr
"YCbCr-4:4:4": {"Y": (1, 1), "Cb": (1, 1), "Cr": (1, 1)},
"YCbCr-4:2:2": {"Y": (1, 1), "Cb": (2, 1), "Cr": (2, 1)},
"YCbCr-4:2:0": {"Y": (1, 1), "Cb": (2, 2), "Cr": (2, 2)},
"YCbCr-4:1:1": {"Y": (1, 1), "Cb": (4, 1), "Cr": (4, 1)},
# Constant luminance YCbCr
"CLYCbCr-4:4:4": {"Yc": (1, 1), "Cbc": (1, 1), "Crc": (1, 1)},
"CLYCbCr-4:2:2": {"Yc": (1, 1), "Cbc": (2, 1), "Crc": (2, 1)},
"CLYCbCr-4:2:0": {"Yc": (1, 1), "Cbc": (2, 2), "Crc": (2, 2)},
# Constant intensity ICtCp
"ICtCp-4:4:4": {"I": (1, 1), "Ct": (1, 1), "Cp": (1, 1)},
"ICtCp-4:2:2": {"I": (1, 1), "Ct": (2, 1), "Cp": (2, 1)},
"ICtCp-4:2:0": {"I": (1, 1), "Ct": (2, 2), "Cp": (2, 2)},
# XYZ
"XYZ": {"X": (1, 1), "Y": (1, 1), "Z": (1, 1)},
# Key signal represented as a single component
"KEY": {"Key": (1, 1)},
# Sampling signaled by the payload
"UNSPECIFIED": {},
}

max_w, max_h = 0, 0
for component in flow_components:
w, h = component["width"], component["height"]
if w > max_w:
max_w = w
if h > max_h:
max_h = h

components_sampler = {}
for component in flow_components:
w, h = component["width"], component["height"]
components_sampler[component["name"]] = (max_w / w, max_h / h)

for sampling, sampler in samplers.items():
if sampler == components_sampler:
return sampling

def downgrade_resource(resource_type, resource_data, requested_version):
"""Downgrades given resource data to requested version"""
version_major, version_minor = [int(x) for x in requested_version[1:].split(".")]
Expand Down
24 changes: 24 additions & 0 deletions nmostesting/NMOSTesting.py
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,9 @@
from .suites import IS0901Test
from .suites import IS0902Test
# from .suites import IS1001Test
from .suites import IS1101Test
from .suites import BCP00301Test
from .suites import BCP0050101Test
from .suites import BCP0060101Test
from .suites import BCP0060102Test

Expand Down Expand Up @@ -340,6 +342,20 @@
# }],
# "class": IS1001Test.IS1001Test
# },
"IS-11-01": {
"name": "IS-11 Stream Compatibility Management API",
"specs": [{
"spec_key": "is-11",
"api_key": "streamcompatibility"
}, {
"spec_key": "is-04",
"api_key": "node"
}, {
"spec_key": "is-05",
"api_key": "connection"
}],
"class": IS1101Test.IS1101Test
},
"BCP-003-01": {
"name": "BCP-003-01 Secure Communication",
"specs": [{
Expand All @@ -348,6 +364,14 @@
}],
"class": BCP00301Test.BCP00301Test
},
"BCP-005-01-01": {
"name": "BCP-005-01 EDID to Receiver Capabilities Mapping",
"specs": [{
"spec_key": "is-04",
"api_key": "node"
}],
"class": BCP0050101Test.BCP0050101Test
},
"BCP-006-01-01": {
"name": "BCP-006-01 NMOS With JPEG XS",
"specs": [{
Expand Down
26 changes: 26 additions & 0 deletions nmostesting/suites/BCP0050101Test.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Copyright (C) 2022 Advanced Media Workflow Association
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

from ..GenericTest import GenericTest

NODE_API_KEY = "node"


class BCP0050101Test(GenericTest):
"""
Runs Node Tests covering BCP-005-01
"""
def __init__(self, apis, **kwargs):
GenericTest.__init__(self, apis, **kwargs)
self.node_url = self.apis[NODE_API_KEY]["url"]
Loading