Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update with refactoring and fixes #8

Merged
merged 4 commits into from
Sep 2, 2020
Merged

Update with refactoring and fixes #8

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
1eec707
Fix failing check as suggested by BruceM
RobH123 Aug 20, 2020
43ac4f4
Convert noticeList to objects rather than arrays
RobH123 Aug 21, 2020
460e52f
Start on annotation framework; bug fixes; some global renaming
RobH123 Aug 26, 2020
9127da1
Rename to characterIndex for better self-documentation
RobH123 Sep 2, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 9 additions & 4 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,11 +50,11 @@ The top-level checking demonstrations:
However, the lower-level checking functions provide only the list of success message strings and one list of `notices` (i.e., warnings/errors combined) typically consisting of the following eight fields:

1. A notice priority number in the range 1-1000. Each different type of warning/error has a unique number (but not each instance of those warnings/errors). By default, notice priority numbers 700 and over are considered `errors` and 0-699 are considered `warnings`.
1. The 3-character UPPERCASE [book code](http://ubsicap.github.io/usfm/identification/books.html) or [OBS](https://www.openbiblestories.org/) (if relevant)
1. The 3-character UPPERCASE [book identifier](http://ubsicap.github.io/usfm/identification/books.html) or [OBS](https://www.openbiblestories.org/) (if relevant)
1. The chapter number or story number (if relevant)
1. The verse number or frame number
1. The actual general descriptive text of the notice
1. A zero-based integer index which indicates the zero-based position of the error in the given text (line or file). -1 indicates that this index does not contain any useful information, e.g., for a global error.
1. A zero-based integer characterIndex which indicates the zero-based position of the error in the given text (line or file). -1 indicates that this index does not contain any useful information, e.g., for a global error.
1. An extract of the checked text which indicates the area containing the problem. Where helpful, some character substitutions have already been made, for example, if the notice is about spaces, it is generally helpful to display spaces as a visible character in an attempt to best highlight the issue to the user. (The length of the extract defaults to ten characters, but is settable as an option.)
1. A string indicating the context of the notice, e.g., `in line 17 of 'someBook.usfm'`.

Expand All @@ -71,18 +71,23 @@ However, the user is, of course, free to create their own alternative version of

Still unfinished (in rough priority order):

1. Standardise parameters according to best practice (i.e., dereferencing, etc.)
1. Document the API
1. Publish to NPM so that the functions may be easily used by other software -- this may involve some changes to the folder structure, etc. as we only want the core functions published in this way -- not the demo code
1. Checking of general markdown and naked links
1. Testing and fine-tuning of error messages (e.g., comparing with tX), especially suppression of false alarms
1. Improve documentation
1. Optimise various different file fetches and caches (incl. using zips) for the demos
1. Is our `RepoCheck` the same as `ResourceContainerCheck`? Or is the latter more specific?
1. Standardise parameters according to best practice (i.e., dereferencing, etc.)
1. Understand and standardise React stuff, e.g., e.g., withStyles, etc.
1. Understand and standardise React stuff in the demos, e.g., e.g., withStyles, etc.
1. Write unit tests (especially for the core functions) and get them passing
1. Check for and remove left-over (but unused) code from the source projects that the original code was copied from
1. Remove all debug code and console logging, and to consider possible speed and memory optimizations

Known bugs:

1. The line number in the USFM Grammar check doesn't account for blank lines, so the real line number may be larger.

## Functionality and Limitations

See component `README` for details.
Expand Down
2 changes: 1 addition & 1 deletion makeNoticeList.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
with open(os.path.join(root, name), 'rt') as sourceFile:
for line in sourceFile:
if 'addNotice' in line and 'function addNotice' not in line \
and 'console.assert' not in line and 'noticeEntry[0]' not in line:
and 'console.assert' not in line and 'noticeEntry.priority' not in line:
strippedLine = line.strip()
if strippedLine.endswith(');'): strippedLine = strippedLine[:-2]
if not strippedLine.startswith('//'):
Expand Down
144 changes: 72 additions & 72 deletions noticeList.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,74 +1,74 @@
./makeNoticeList.py
Got 96 notices:
996, bookCode, '', '', "Failed to load", -1, "", `${generalLocation} ${filename}: ${cBPgfError}`, repoCode
996, bookCode, "", "", "Failed to load", -1, "", `${generalLocation} ${thisPath}: ${tQerror}`, repoCode
996, BBBid,'','', "Failed to load", -1, "", `${generalLocation} ${thisFilepath}: ${cRgfError}`, repoCode
996, "Failed to load", -1, "", `${generalLocation} ${filename}: ${gcUHBerror}`, repoCode
996, "Failed to load", -1, "", `${generalLocation} ${filename}: ${gcUGNTerror}`, repoCode
993, "Unresolved GIT conflict", ix, extract
993, "Unresolved GIT conflict", ix, extract
992, "Unresolved GIT conflict", ix, extract
992, "Unresolved GIT conflict", ix, extract
991, "Unresolved GIT conflict", ix, extract
991, "Unresolved GIT conflict", ix, extract
979, "Invalid book code passed to checkTN_TSVDataRow", -1, "", ` '${BBB}' in first parameter: ${tlcNCerror}`
979, "Invalid book code passed to checkTNLinks", -1, "", ` '${BBB}' in first parameter: ${tnlcError}`
978, `Wrong '${B}' book code`, -1, "", ` (expected '${BBB}')`
977, "Missing book code", 0, ""
947, BBBid,'','', "Missing manifest.yaml", -1, "", 'MANIFEST'
946, BBBid,'','', "Missing LICENSE.md", -1, "", 'LICENSE'
944, `USFM3 toJSON Check doesn't pass`, -1, ""
944, `USFM3 Grammar Check (${strictnessString} mode) doesn't pass`, -1, ""
929, "'projects' key is missing", -1, ""
928, "'dublin_core' key is missing", -1, ""
916, yamlError.message, -1, "")
900, '','','', "Bad parameter: should be given a valid book abbreviation", -1,bookCodeList, ` (not '${bookCodeList}')${location}`
900, '', '', '', "Bad function call: should be given a valid book abbreviation", -1, bookCode, ` (not '${bookCode}')${location}`, ''
889, `Unable to find ${fieldName} TA link`, -1, fieldText, `${ourLocation} ${filepath}`
888, `Error loading ${fieldName} TA link`, -1, fieldText, `${ourLocation} ${filepath}: ${trcGCerror}`
887, `Linked ${fieldName} TA article seems empty`, -1, fieldText, `${ourLocation} ${filepath}`
886, `Unable to find ${fieldName} TA link`, -1, resultArray[0], `${ourLocation} ${filepath}`
885, `Error loading ${fieldName} TA link`, -1, resultArray[0], `${ourLocation} ${filepath}: ${trcGCerror}`
884, `Linked ${fieldName} TA article seems empty`, -1, resultArray[0], `${ourLocation} ${filepath}`
883, `Unable to find ${fieldName} TW link`, -1, resultArray[0], `${ourLocation} ${filepath}`
882, `Error loading ${fieldName} TW link`, -1, resultArray[0], `${ourLocation} ${filepath}: ${trcGCerror}`
881, `Linked ${fieldName} TW article seems empty`, -1, resultArray[0], `${ourLocation} ${filepath}`
861, `Found wrong number of TSV fields (expected ${NUM_EXPECTED_TSV_FIELDS})`, -1, `Found ${fields.length} field${fields.length === 1 ? '' : 's'}`
851, "Unable to load original language verse text", -1, ""
843, "Invalid chapter number", -1, resultArray[4], `${ourLocation}`
824, `Invalid zero chapter number`, -1, C
823, `Invalid large chapter number`, -1, C
822, "Unable to check chapter number", -1, "", ` '${C}'`
821, "Bad chapter number", -1, "", ` '${C}' with`
820, "Missing chapter number", -1, "", ` ?:${V}`
814, `Invalid zero '${V}' verse number`, -1, ""
813, `Invalid large '${V}' verse number`, ` for chapter ${C}`
812, "Unable to check verse number", -1, "", ` '${V}'`
811, "Bad verse number", -1, "", ` '${V}'`
810, "Missing verse number", -1, "", ` after ${C}:?`
792, `Invalid '${occurrence}' occurrence field`, -1, ""
779, "Missing ID field", -1, ""
778, "ID should be exactly 4 characters", -1, "", ` (not ${fieldID.length})`)
765, "Unexpected link", ix, extract
743, "Chapter numbers of Bible link don't match", -1, resultArray[0], `${ourLocation}`
742, "Verse numbers of Bible link don't match", -1, resultArray[0], `${ourLocation}`
722, "Unable to find original language quote in verse text", -1, extract
721, "Unable to find original language quote in verse text", -1, extract
663, "Mismatched " + leftChar + rightChar + " characters", -1, "(left=" + lCount.toLocaleString() + ", right=" + rCount.toLocaleString() + ")"
638, "Only found whitespace", -1, ""
638, "Only found whitespace", -1, ""
621, "Seems original language quote might not finish at the end of a word", fieldText.length, extract
620, "Seems original language quote might not start at the beginning of a word", 0, extract
600, `${regexResultsArray.length} link target${regexResultsArray.length === 1 ? ' is' : 's are'} still being checked…`, -1, "", ""
583, "Unexpected newLine character", ix, extract
582, "Unexpected carriageReturn character", ix, extract
581, "Unexpected non-break space character", ix, extract
580, "Unexpected narrow non-break space character", ix, extract
550, "Invalid zero occurrence field when we have an original quote", -1, ""
472, "Nesting seems confused", 0, ''
441, `Unknown linkType parameter`, -1, linkType, ""
439, "Error fetching link", -1, "", ` ${fetchLink}`
438, `Blank field / missing link (expected ${linkOptions.expectedCount} link${linkOptions.expectedCount === 1 ? "" : "s"})`, -1, ""
./makeNoticeList.py
Got 96 notices:
996, bookID, '', '', "Failed to load", -1, "", `${generalLocation} ${filename}: ${cBPgfError}`, repoCode
996, bookID, "", "", "Failed to load", -1, "", `${generalLocation} ${thisPath}: ${tQerror}`, repoCode
996, BBBid,'','', "Failed to load", -1, "", `${generalLocation} ${thisFilepath}: ${cRgfError}`, repoCode
996, "Failed to load", -1, "", `${generalLocation} ${filename}: ${gcUHBerror}`, repoCode
996, "Failed to load", -1, "", `${generalLocation} ${filename}: ${gcUGNTerror}`, repoCode
993, "Unresolved GIT conflict", ix, extract
993, "Unresolved GIT conflict", ix, extract
992, "Unresolved GIT conflict", ix, extract
992, "Unresolved GIT conflict", ix, extract
991, "Unresolved GIT conflict", ix, extract
991, "Unresolved GIT conflict", ix, extract
979, "Invalid book identifier passed to checkTN_TSVDataRow", -1, "", ` '${bookID}' in first parameter: ${tlcNCerror}`
979, "Invalid book identifier passed to checkTNLinks", -1, "", ` '${bookID}' in first parameter: ${tnlcError}`
978, `Wrong '${B}' book identifier`, -1, "", ` (expected '${bookID}')`
977, "Missing book identifier", 0, ""
947, BBBid,'','', "Missing manifest.yaml", -1, "", 'MANIFEST'
946, BBBid,'','', "Missing LICENSE.md", -1, "", 'LICENSE'
944, `USFM3 toJSON Check doesn't pass`, -1, ""
944, `USFM3 Grammar Check (${strictnessString} mode) doesn't pass`, -1, ""
929, "'projects' key is missing", -1, ""
928, "'dublin_core' key is missing", -1, ""
916, yamlError.message, -1, "")
900, '','','', "Bad parameter: should be given a valid book abbreviation", -1,bookIDList, ` (not '${bookIDList}')${location}`
900, '', '', '', "Bad function call: should be given a valid book abbreviation", -1, bookID, ` (not '${bookID}')${location}`, ''
889, `Unable to find ${fieldName} TA link`, -1, fieldText, `${ourLocation} ${filepath}`
888, `Error loading ${fieldName} TA link`, -1, fieldText, `${ourLocation} ${filepath}: ${trcGCerror}`
887, `Linked ${fieldName} TA article seems empty`, -1, fieldText, `${ourLocation} ${filepath}`
886, `Unable to find ${fieldName} TA link`, -1, resultArray[0], `${ourLocation} ${filepath}`
885, `Error loading ${fieldName} TA link`, -1, resultArray[0], `${ourLocation} ${filepath}: ${trcGCerror}`
884, `Linked ${fieldName} TA article seems empty`, -1, resultArray[0], `${ourLocation} ${filepath}`
883, `Unable to find ${fieldName} TW link`, -1, resultArray[0], `${ourLocation} ${filepath}`
882, `Error loading ${fieldName} TW link`, -1, resultArray[0], `${ourLocation} ${filepath}: ${trcGCerror}`
881, `Linked ${fieldName} TW article seems empty`, -1, resultArray[0], `${ourLocation} ${filepath}`
861, `Found wrong number of TSV fields (expected ${NUM_EXPECTED_TSV_FIELDS})`, -1, `Found ${fields.length} field${fields.length === 1 ? '' : 's'}`
851, "Unable to load original language verse text", -1, ""
843, "Invalid chapter number", -1, resultArray[4], `${ourLocation}`
824, `Invalid zero chapter number`, -1, C
823, `Invalid large chapter number`, -1, C
822, "Unable to check chapter number", -1, "", ` '${C}'`
821, "Bad chapter number", -1, "", ` '${C}' with`
820, "Missing chapter number", -1, "", ` ?:${V}`
814, `Invalid zero '${V}' verse number`, -1, ""
813, `Invalid large '${V}' verse number`, ` for chapter ${C}`
812, "Unable to check verse number", -1, "", ` '${V}'`
811, "Bad verse number", -1, "", ` '${V}'`
810, "Missing verse number", -1, "", ` after ${C}:?`
792, `Invalid '${occurrence}' occurrence field`, -1, ""
779, "Missing ID field", -1, ""
778, "ID should be exactly 4 characters", -1, "", ` (not ${fieldID.length})`)
765, "Unexpected link", ix, extract
743, "Chapter numbers of Bible link don't match", -1, resultArray[0], `${ourLocation}`
742, "Verse numbers of Bible link don't match", -1, resultArray[0], `${ourLocation}`
722, "Unable to find original language quote in verse text", -1, extract
721, "Unable to find original language quote in verse text", -1, extract
663, "Mismatched " + leftChar + rightChar + " characters", -1, "(left=" + lCount.toLocaleString() + ", right=" + rCount.toLocaleString() + ")"
638, "Only found whitespace", -1, ""
638, "Only found whitespace", -1, ""
621, "Seems original language quote might not finish at the end of a word", fieldText.length, extract
620, "Seems original language quote might not start at the beginning of a word", 0, extract
600, `${regexResultsArray.length} link target${regexResultsArray.length === 1 ? ' is' : 's are'} still being checked…`, -1, "", ""
583, "Unexpected newLine character", ix, extract
582, "Unexpected carriageReturn character", ix, extract
581, "Unexpected non-break space character", ix, extract
580, "Unexpected narrow non-break space character", ix, extract
550, "Invalid zero occurrence field when we have an original quote", -1, ""
472, "Nesting seems confused", 0, ''
441, `Unknown linkType parameter`, -1, linkType, ""
439, "Error fetching link", -1, "", ` ${fetchLink}`
438, `Blank field / missing link (expected ${linkOptions.expectedCount} link${linkOptions.expectedCount === 1 ? "" : "s"})`, -1, ""
375, "Ellipsis without surrounding snippet", -1, ""
287, `Not enough links (expected ${linkOptions.expectedCount} link${linkOptions.expectedCount === 1 ? "" : "s"})`, -1, "", ` (only found ${regexResultsArray.length})${ourAtString}`
276, "Missing OrigQuote field", -1, ""
Expand All @@ -95,5 +95,5 @@ Got 96 notices:
106, "Unexpected leading space" + (fieldText[1] === ' ' ? "s" : ""), 0, extract
105, "Unexpected trailing space(s)", fieldText.length - 1, extract
104, "Unexpected trailing break", fieldText.length - 1, extract
101, "USFMGrammar found: " + warningString, -1, ""
101, "USFMGrammar: " + warningString, -1, ""

2 changes: 1 addition & 1 deletion package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "content-validation",
"description": "Functions for Checking uW Content/Resources.",
"version": "0.7.4",
"version": "0.8.3",
"private": false,
"homepage": "https://unfoldingword.github.io/content-validation/",
"repository": {
Expand Down
Loading