WIP: Update readtags to support escape sequences and fix & improve its testing #1605

b4n · 2017-11-29T23:48:19Z

Following what I started in #1580, this PR aims at updating readtags to properly support the current format, as well as fixing & improving its automated test cases (roundtrip).

Follow Automake's verbosity instead of unconditionally silencing the rules.

* Don't perform shell expansion on tag names. * Actually check whether readtags found a tag matching. As readtags exits successfully even without matches, make sure it found at least one match for assuming success. * Code cleanup.

* Properly handle all input, not just the first line; * Un-escape also the low 0x01 - 0x1F and 0x7F bytes.

This restores backward compatibility with e.g. backslashes in the search string.

coveralls · 2017-11-30T00:23:58Z

Coverage decreased (-1.7%) to 84.325% when pulling 2b51abe on b4n:readtags-update-and-test into 248cffc on universal-ctags:master.

coveralls · 2017-11-30T02:11:02Z

Coverage decreased (-1.7%) to 84.325% when pulling 0ac3a99 on b4n:readtags-update-and-test into 248cffc on universal-ctags:master.

coveralls · 2017-11-30T03:34:04Z

Coverage decreased (-1.7%) to 84.325% when pulling 0d502b3 on b4n:readtags-update-and-test into 248cffc on universal-ctags:master.

coveralls · 2017-11-30T06:41:17Z

Coverage decreased (-1.7%) to 84.325% when pulling 0d502b3 on b4n:readtags-update-and-test into 248cffc on universal-ctags:master.

* don't stop all processing when one input line has no escaping * don't stop expansion after an unhandled escape sequence

Actually just revert it.

coveralls · 2017-12-02T05:17:55Z

Coverage decreased (-1.7%) to 84.328% when pulling 168eb2a on b4n:readtags-update-and-test into 248cffc on universal-ctags:master.

b4n · 2017-12-02T06:23:18Z

OK I don't know how to avoid the issue yet, e.g. :: leads to readtags receiving ;, leading /es to C:/…, etc. I tried escaping with \ and ^, doesn't seem to help.

coveralls · 2017-12-02T06:39:41Z

Coverage decreased (-1.7%) to 84.328% when pulling 0ed2741 on b4n:readtags-update-and-test into 248cffc on universal-ctags:master.

https://github.com/msys2/msys2/wiki/Porting#user-content-filesystem-namespaces

coveralls · 2017-12-02T09:37:55Z

Coverage decreased (-1.7%) to 84.328% when pulling 5d05ee6 on b4n:readtags-update-and-test into 248cffc on universal-ctags:master.

coveralls · 2017-12-02T09:44:47Z

Coverage decreased (-1.7%) to 84.328% when pulling 341db19 on b4n:readtags-update-and-test into 248cffc on universal-ctags:master.

b4n · 2017-12-02T21:41:45Z

OK I improved the situation with the last commit after finding https://github.com/msys2/msys2/wiki/Porting#user-content-filesystem-namespaces, but it's still not perfect: there's encoding issues, and some exclude patterns break legitimate stuff (like * and /). All in all, there's still 14 tests failing on MSYS2.

I don't really know what I can do, but add the ability for readtags to read the tag name from a file instead of the command line. It's not very nice because I can't really see a real use for it, but it would solve the whole problem at once I guess.

masatake · 2017-12-12T15:01:40Z

(I will work on this item in the new year day.)

@b4n

…put-format=u-ctags|e-ctags option To record meta characters like tab in a name, we have designed new file format. The file format itself was already introduced. TAG_OUTPUT_MODE pseudo tag was used as marker telling which format is used in a tag file: Exuberant-ctags(e-ctags) compatible format or Universal-ctags(u-ctags) file format. u-ctags was the default format. A user could choose the format with --output-format option. However, we recognize that the hack using TAG_OUTPUT_MODE pseudo doesn't work well with readtags command. A data structure of readtags doesn't have a field for recording TAG_OUTPUT_MODE. It means readtags cannot switch the file format version dynamically. The pseudo tags can be used in readtags for the purpose, recognizing the file format is only TAG_FILE_FORMAT. This commit introduces following SELF INCOMPATIBLE changes: 1. --output-format=u-ctags and --output-format=e-ctags are removed. They are unified to --output-format=ctags. 2. TAG_OUTPUT_MODE pseudo tag is removed. Instead TAG_FILE_FORMAT=3 is introduced. TAG_FILE_FORMAT=2 is compatible with e-ctags. TAG_FILE_FORMAT=3 is default. A user can choose one of the formats with --format=3 or --format=2. The new version 3 may break many client tools. If you are just user of such client tools, use --format=2 option to generate tags file with compatible file format. If you are a developer of such client tools, please, consider to support --format=3. The rest of work are: 1. more Tmain test cases, 2. more precise documentation about the format 3. 3. updating readtags to support format 3. About readtags, the most of all work is done by @b4n. I must merge his changes(universal-ctags#1605). Signed-off-by: Masatake YAMATO <yamato@redhat.com>

@b4n

…put-format=u-ctags|e-ctags option To record meta characters like tab in a name, we have designed new file format. The file format itself was already introduced. TAG_OUTPUT_MODE pseudo tag was used as marker telling which format is used in a tag file: Exuberant-ctags(e-ctags) compatible format or Universal-ctags(u-ctags) file format. u-ctags was the default format. A user could choose the format with --output-format option. However, we recognize that the hack using TAG_OUTPUT_MODE pseudo doesn't work well with readtags command. A data structure of readtags doesn't have a field for recording TAG_OUTPUT_MODE. It means readtags cannot switch the file format version dynamically. The pseudo tags can be used in readtags for the purpose, recognizing the file format is only TAG_FILE_FORMAT. This commit introduces following SELF INCOMPATIBLE changes: 1. --output-format=u-ctags and --output-format=e-ctags are removed. They are unified to --output-format=ctags. 2. TAG_OUTPUT_MODE pseudo tag is removed. Instead TAG_FILE_FORMAT=3 is introduced. TAG_FILE_FORMAT=2 is compatible with e-ctags. TAG_FILE_FORMAT=3 is default. A user can choose one of the formats with --format=3 or --format=2. The new version 3 may break many client tools. If you are just user of such client tools, use --format=2 option to generate tags file with compatible file format. If you are a developer of such client tools, please, consider to support --format=3. The rest of work are: 1. more Tmain test cases, 2. more precise documentation about the format 3. 3. updating readtags to support format 3. About readtags, the most of all work is done by @b4n. I must merge his changes(universal-ctags#1605). Signed-off-by: Masatake YAMATO <yamato@redhat.com>

@b4n

…put-format=u-ctags|e-ctags option To record meta characters like tab in a name, we have designed new file format. The file format itself was already introduced. TAG_OUTPUT_MODE pseudo tag was used as marker telling which format is used in a tag file: Exuberant-ctags(e-ctags) compatible format or Universal-ctags(u-ctags) file format. u-ctags was the default format. A user could choose the format with --output-format option. However, we recognize that the hack using TAG_OUTPUT_MODE pseudo doesn't work well with readtags command. A data structure of readtags doesn't have a field for recording TAG_OUTPUT_MODE. It means readtags cannot switch the file format version dynamically. The pseudo tags can be used in readtags for the purpose, recognizing the file format is only TAG_FILE_FORMAT. This commit introduces following SELF INCOMPATIBLE changes: 1. --output-format=u-ctags and --output-format=e-ctags are removed. They are unified to --output-format=ctags. 2. TAG_OUTPUT_MODE pseudo tag is removed. Instead TAG_FILE_FORMAT=3 is introduced. TAG_FILE_FORMAT=2 is compatible with e-ctags. TAG_FILE_FORMAT=3 is default. A user can choose one of the formats with --format=3 or --format=2. The new "version 3" may break many client tools. If you are just user of such client tools, use --format=2 option to generate tags file with compatible file format. If you are a developer of such client tools, please, consider to support --format=3. The rest of work are: 1. more Tmain test cases, 2. more precise documentation about the format 3. 3. updating readtags to support format 3. About readtags, the most of all work is done by @b4n. I must merge his changes(universal-ctags#1605). docs/news.rst are rewritten by @b4n and @codebrainz. Ussing stringfy for printing default version of file format in --help message is suggested by @b4n. Signed-off-by: Masatake YAMATO <yamato@redhat.com>

masatake · 2017-12-29T16:46:03Z

I will take time for this important one.

masatake · 2018-01-03T01:33:35Z

@b4n, I'm reading your patches.
I will devide your patches into two groups; one group(A) gathers the part of patches handling escape sequence, and the other group(B) gathers the part of patches general purpose, nothing about escape sequence. I will merge B into master branch first.

masatake · 2018-01-03T01:36:40Z

misc/roundtrip

@@ -112,8 +114,10 @@ OLD_IFS="$IFS"
 IFS='
 '
 set -x
-for tags in $(find "$UNITS" -name expected.tags); do
-    for name in $(sed -e 's/^\([^	]*\)	.*/\1/' "$tags"); do
+tagfiles="$(find "$UNITS" -name expected.tags)"


The first and the last double quote characters are needed.

Does the following code do the same?

tagfiles=$(find "$UNITS" -name expected.tags)

yes, doesn't matter

masatake · 2018-01-03T01:37:30Z

misc/roundtrip

-    for name in $(sed -e 's/^\([^	]*\)	.*/\1/' "$tags"); do
+tagfiles="$(find "$UNITS" -name expected.tags)"
+for tags in $tagfiles; do
+    tagnames="$(sed -e 's/^\([^	]*\)	.*/\1/' "$tags")"


The same as the last my comment.

masatake · 2018-01-03T01:41:01Z

I will merge following changes to master branch.
(The commit ids are changed because I make your branch be rebased on master.)

52c638a8 circleci: Install find for roundtrip
c079b0cc roundtirp: Improve message for failing tests
dc0bce22 readtags: Build with coverage support if requested
c7817168 roundtrip: Cleanup and fix so it actually does check something
85358ece roundtrip: Fix running on non-C locales
16260d46 testing: Define $(SILENT)
85525454 testing: Don't unconditionally silent test rules
                 roundtrip: Exit on command failure

masatake · 2018-01-06T09:22:42Z

misc/roundtrip

@@ -53,7 +53,7 @@ expandEscapeSequences()
 :again
    s/\\/__BACKSLASH__/
    T out
-    s/__BACKSLASH__\\/__LASTBACKSLASH__/;	t again
+    s/__BACKSLASH__\\/__LITBACKSLASH__/;	t again


Could you tell me why this change is needed?

It is not needed per se, but I changed the name for clarity as it's not only used for the last backslash but for all literal backslashes

masatake · 2018-01-06T09:33:30Z

@b4n, my understanding is that many changes are about \n and \t.
If we use GNU sed on MacOSX, can we make changes simpler?

b4n · 2018-01-06T10:25:44Z

@b4n, my understanding is that many changes are about \n and \t.
If we use GNU sed on MacOSX, can we make changes simpler?

Not really, having fixups here is just that I'm too used to GNU sed so I don't think about its extensions in supporting escape sequences; but ultimately the changes are the same just with a slightly different syntax (using the literal value rather than the GNU sed escape sequence).

masatake · 2018-02-06T08:22:41Z

This is privte note for reviewing task. I have been thinking about code for comparison.

My testing:

[yamato@master tmp]$ /usr/bin/ctags --version
/usr/bin/ctags --version
Exuberant Ctags 5.8, Copyright (C) 1996-2009 Darren Hiebert
  Compiled: Aug  2 2017, 19:41:49
  Addresses: <dhiebert@users.sourceforge.net>, http://ctags.sourceforge.net
  Optional compiled features: +wildcards, +regex
[yamato@master tmp]$ cat /tmp/foo.unknown 
cat /tmp/foo.unknown 
_a	b_ def
[yamato@master tmp]$ /usr/bin/ctags --langdef=Z --regex-Z='/_(.*)_ def$/\1/k,kind,kinds/' --language-force=Z  -o - /tmp/foo.unknown 
/usr/bin/ctags --langdef=Z --regex-Z='/_(.*)_ def$/\1/k,kind,kinds/' --language-force=Z  -o - /tmp/foo.unknown 
a	b	/tmp/foo.unknown	/^_a	b_ def$/;"	k
[yamato@master tmp]$ u-ctags --langdef=Z --regex-Z='/_(.*)_ def$/\1/k,kind,kinds/' --language-force=Z  -o - /tmp/foo.unknown 
u-ctags --langdef=Z --regex-Z='/_(.*)_ def$/\1/k,kind,kinds/' --language-force=Z  -o - /tmp/foo.unknown 
a\tb	/tmp/foo.unknown	/^_a	b_ def$/;"	k
[yamato@master tmp]$ u-ctags --format=2 --langdef=Z --regex-Z='/_(.*)_ def$/\1/k,kind,kinds/' --language-force=Z  -o - /tmp/foo.unknown 
u-ctags --format=2 --langdef=Z --regex-Z='/_(.*)_ def$/\1/k,kind,kinds/' --language-force=Z  -o - /tmp/foo.unknown 
a\tb	/tmp/foo.unknown	/^_a	b_ def$/;"	k

A tab character is put between a and b in input.unknown.

Let's see if a whitespace is placed instead of the tab char.

[yamato@master tmp]$ /usr/bin/ctags --format=2 --langdef=Z --regex-Z='/_(.*)_ def$/\1/k,kind,kinds/' --language-force=Z  -o - /tmp/bar.unknown 
/usr/bin/ctags --format=2 --langdef=Z --regex-Z='/_(.*)_ def$/\1/k,kind,kinds/' --language-force=Z  -o - /tmp/bar.unknown 
a b	/tmp/bar.unknown	/^_a b_ def$/;"	k
[yamato@master tmp]$ u-ctags --langdef=Z --regex-Z='/_(.*)_ def$/\1/k,kind,kinds/' --language-force=Z  -o - /tmp/bar.unknown 
u-ctags --langdef=Z --regex-Z='/_(.*)_ def$/\1/k,kind,kinds/' --language-force=Z  -o - /tmp/bar.unknown 
a b	/tmp/bar.unknown	/^_a b_ def$/;"	k
[yamato@master tmp]$ u-ctags --format=2 --langdef=Z --regex-Z='/_(.*)_ def$/\1/k,kind,kinds/' --language-force=Z  -o - /tmp/bar.unknown 
u-ctags --format=2 --langdef=Z --regex-Z='/_(.*)_ def$/\1/k,kind,kinds/' --language-force=Z  -o - /tmp/bar.unknown

e-ctags violates the v2 format. u-ctags --format=2 violates the format. u-ctags (version 3) violates the format.

I guess none seriously follows v2 format.
So thinking about following v2 format seriously in u-ctags doesn't make sense.
We should fix the output of v2 on-issue base.

Instead, we should refine v3 format as possible and make a reliable format.

o.k. About the tags generator side, I understand where I should go.

How about reader side?

I assume the original readtags works well with the output of e-ctags though the generator side doesn't follow the v2 format spec:-P.

So I think the original code of readtags code for handling e-ctags output as is.

My idea is that the code introduced by @b4n's patch should be activated for handling u-ctags v2 and v3 formats output only.

The good new is that tagFileInfo and TagFile data structures provide enough information to detect whether e-ctags, u-ctags v2 or u-ctags v3 generates the current output. In other word, using strcmp for e-ctags and tagcmp for u-ctags is my prefer.

Keeping the strcmp based code makes the source code (and test cases) complicated a bit.
This is trading off.

About v2 format.

format.rst which explains v2 tags format says:

{tagname}
	Any identifier, not containing white space..

However, as above output shows, e-ctags doesn't follow the v2 tags format at least using regex parser.

masatake · 2018-02-06T08:30:14Z

Comparing the output is the part of library. So I am conservative. However, about command line interface we can introduce changes, as we did in ctags command itself.

The question is whether we should print the result in escaped way or un'escaped way?
... This question is the out of scope of the changes. However, the output format has impacts of roundtrip test driver.

I will find more time for this issue.
Very sorry that it takes long time for merging this.

b4n · 2018-02-09T04:38:33Z

My idea is that the code introduced by @b4n's patch should be activated for handling u-ctags v2 and v3 formats output only.

As said, I think that my changes are backward compatible. And I think it is not only with the v2 format, but also the e-ctags format: the only thing that would risk being a problem is if e-ctags could output unescaped \es, in which case the updated code would erroneously try and decode it. However, IIRC e-ctags doesn't do so, in which case we should be safe.

masatake · 2018-02-17T20:58:58Z

Thank you very much. I merged all your changes.

b4n added 17 commits November 28, 2017 17:32

readtags: Add support for escape sequences in tag names

3d7f38f

testing: Don't unconditionally silent test rules

a6e4bde

Follow Automake's verbosity instead of unconditionally silencing the rules.

testing: Define $(SILENT)

72fc6df

roundtrip: Try and match unescaped names too

a9433ab

Run roundtrip on make check

5692578

roundtrip: Fix running on non-C locales

d59b3af

roundtrip: Try and fix macos compatibility

378c53e

readtags: Build with coverage support if requested

c814e65

WIP: roundtrip: add debugging output because of Windows

21a8b0e

roundtrip: Fix unescaping

247a93b

* Properly handle all input, not just the first line; * Un-escape also the low 0x01 - 0x1F and 0x7F bytes.

roundtrip: Fix handling of escaped newlines

df5e6bd

readtags: Don't expect escaped search string

e2dc477

This restores backward compatibility with e.g. backslashes in the search string.

roundtirp: Improve message for failing tests

0714968

fixup! roundtrip: Fix unescaping

4bae9ed

roundtrip: Fix handling of trailing newlines

cd6c100

fixup! readtags: Add support for escape sequences in tag names

2b51abe

b4n added the readtags label Nov 29, 2017

b4n requested a review from masatake November 29, 2017 23:48

b4n changed the title ~~WIP: Update readtags to support escap sequences and fix & improve its testing~~ WIP: Update readtags to support escape sequences and fix & improve its testing Nov 29, 2017

roundtrip: Exit on command failure

f7ade0c

b4n force-pushed the readtags-update-and-test branch from 0ac3a99 to f7ade0c Compare November 30, 2017 00:23

circleci: Install find for roundtrip

0d502b3

b4n added 3 commits November 30, 2017 00:25

roundtrip: Fix MacOS compatibility

e441795

roundtrip: Fix some escaping cases

15fe94c

* don't stop all processing when one input line has no escaping * don't stop expansion after an unhandled escape sequence

fixup! WIP: roundtrip: add debugging output because of Windows

3b19b37

Actually just revert it.

fixup! roundtrip: Try and workaround AppVeyor/MSYS2 weirdness

0ed2741

roundtrip: Try and actually fix erroneous MSYS2 path rewrites

341db19

https://github.com/msys2/msys2/wiki/Porting#user-content-filesystem-namespaces

b4n force-pushed the readtags-update-and-test branch from 5d05ee6 to 341db19 Compare December 2, 2017 09:29

masatake mentioned this pull request Dec 26, 2017

[SELF INCOMPATIBLE] main: introduce version 3 format and remove --out… #1621

Merged

masatake removed their request for review December 29, 2017 16:45

masatake self-assigned this Dec 29, 2017

masatake reviewed Jan 3, 2018

View reviewed changes

masatake mentioned this pull request Jan 3, 2018

Improve roundtrip test harness generic part of b4n s work #1641

Merged

masatake reviewed Jan 6, 2018

View reviewed changes

masatake closed this Feb 17, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Update readtags to support escape sequences and fix & improve its testing #1605

WIP: Update readtags to support escape sequences and fix & improve its testing #1605

b4n commented Nov 29, 2017

coveralls commented Nov 30, 2017

coveralls commented Nov 30, 2017

coveralls commented Nov 30, 2017

coveralls commented Nov 30, 2017

coveralls commented Dec 2, 2017

b4n commented Dec 2, 2017

coveralls commented Dec 2, 2017

coveralls commented Dec 2, 2017

coveralls commented Dec 2, 2017

b4n commented Dec 2, 2017 •

edited

Loading

masatake commented Dec 12, 2017

masatake commented Dec 29, 2017

masatake commented Jan 3, 2018

masatake Jan 3, 2018

b4n Jan 3, 2018

masatake Jan 3, 2018

masatake commented Jan 3, 2018

masatake Jan 6, 2018

b4n Jan 6, 2018

masatake commented Jan 6, 2018

b4n commented Jan 6, 2018

masatake commented Feb 6, 2018

masatake commented Feb 6, 2018

b4n commented Feb 9, 2018

masatake commented Feb 17, 2018

WIP: Update readtags to support escape sequences and fix & improve its testing #1605

WIP: Update readtags to support escape sequences and fix & improve its testing #1605

Conversation

b4n commented Nov 29, 2017

coveralls commented Nov 30, 2017

coveralls commented Nov 30, 2017

coveralls commented Nov 30, 2017

coveralls commented Nov 30, 2017

coveralls commented Dec 2, 2017

b4n commented Dec 2, 2017

coveralls commented Dec 2, 2017

coveralls commented Dec 2, 2017

coveralls commented Dec 2, 2017

b4n commented Dec 2, 2017 • edited Loading

masatake commented Dec 12, 2017

masatake commented Dec 29, 2017

masatake commented Jan 3, 2018

masatake Jan 3, 2018

Choose a reason for hiding this comment

b4n Jan 3, 2018

Choose a reason for hiding this comment

masatake Jan 3, 2018

Choose a reason for hiding this comment

masatake commented Jan 3, 2018

masatake Jan 6, 2018

Choose a reason for hiding this comment

b4n Jan 6, 2018

Choose a reason for hiding this comment

masatake commented Jan 6, 2018

b4n commented Jan 6, 2018

masatake commented Feb 6, 2018

masatake commented Feb 6, 2018

b4n commented Feb 9, 2018

masatake commented Feb 17, 2018

b4n commented Dec 2, 2017 •

edited

Loading