Add options to allow unicode character in identifier names

[eblis]

PR checklist

• Read the contribution guildelines.
• Ran the shell/batch script under ./bin/ to update Petstore sample so that CIs can verify the change. (For instance, only need to run ./bin/{LANG}-petstore.sh and ./bin/security/{LANG}-petstore.sh if updating the {LANG} (e.g. php, ruby, python, etc) code generator or {LANG} client's mustache templates)
• Filed the PR against the correct branch: master for non-breaking changes and 2.3.0 branch for breaking (non-backward compatible) changes.

Description of the PR

Added option to allow unicode characters in class names, method names etc.
Added option to allow keeping of underscore characters in class names, methods names, etc.

Unicode identifiers are supported by some languages, like Java, but the codegen will remove all non ASCII letters by default. Users might want to separate different parts of names using extended punctuation connector characters, like "_, ‿, ⁀, ⁔, ・, ︳, ︴, ﹍, ﹎, ﹏, ＿, ･"

Underscores cannot be used to separate different parts of the name as they're removed by default by the codegen, the second option allows underscore characters if the user so desires.

Both options can be used separately if needed, for example one might allow extended punctuation connector characters, but still disallow underscore. Sometimes you need to logically separate parts of the identifier names, which isn't supported yet, without these changes.

The default values for both options are set such that the generator will behave as it did before.

[wing328]

Thanks for the PR but your commit (as shown in the Commits tab) is not linked to your Github account, which means this PR won't count as your contribution in https://github.com/swagger-api/swagger-codegen/graphs/contributors.

Let me know if you need help fixing it.

Ref: https://github.com/swagger-api/swagger-codegen/wiki/FAQ#how-can-i-update-commits-that-are-not-linked-to-my-github-account

[eblis]

I've added the email address in the commits as a secondary email, hopefully that should fix it.
Thanks.

If that doesn't work i'll use the script to edit the history, please let me know if I should do that instead.

[wing328]

Now the commits are correctly linked to your account.

Can you elaborate with a use case in which you need to keep the underscore in the property name? As you probably aware, keeping the underscore in the variable name won't conform to C# style guide in terms of variable naming.

[eblis]

Sure.

The reason we need to keep the underscores is to logically separate parts of type names. We're building a proxy server to handle requests to a 3rd party system which has a very weird API and we're building a more RESTful interface for it. The problem is that the 3rd party system types use 2 different names, domain name and type name, and when calling the system we need to know which is which, as the proxy relies on reflection to do it's job (so as not to manually hard-code calls), reflection allows us to be future compatible, any new feature created in the 3rd party system just works without any further changes in the proxy. We've used _ to separate between the domain and type names, but those get removed by the swagger codegen. We'd like to have the option to keep those underscores in certain cases, if the user knows what he's doing and really wants underscores to be present, for any language they might be generating for.

The feature has been implemented at the base layer, so going forward the codegen could have different default values, depending on whether underscores conform to each language's style guides or not, or in cases users really want the underscores for other reasons, like the one I mentioned above.

Alternatively, some languages (like Java, which we're generating for) allow multiple unicode characters as punctuation connector characters, which would alleviate our need for underscores, as we could use one of these extended connectors to logically separate between domain and type names.
Also, some languages support unicode characters in type names, so this feature could be used to allow that support for languages that offer it.

Both feature default values are set to false, so they're not activated unless the user really wants to use either of them. And they can be turned on/off independently, which could be used to further configure the generators for all languages that swagger codegen supports, depending on the style guides for those specific languages.

[wing328]

@eblis thanks for the detailed explanation.

We've used _ to separate between the domain and type names, but those get removed by the swagger codegen

Let's say the name is "domain_type" and it will become "domainType" in Java, "DomainType" in C#. I would like to point out that these are just variable names but not the actual parameter (path, header, form, query) name. When the API client (e.g. C#) sends the request to the RESTful backend, the actual parameter name (baseName) is used as shown below:

https://github.com/swagger-api/swagger-codegen/blob/master/modules/swagger-codegen/src/main/resources/csharp/api.mustache#L240-L251

We definitely welcome enhancement to meet different use cases but I would like to point this out to make sure we're on the same page after merging the additional features you've built.

[eblis]

We are more interested in the variable and class names, as we're using reflection to handle the proxy-ing, and need to know the domain name and type name.
I probably didn't explain it good enough the first time, the data types we use have 2 different logical names in them (as the system we're doing proxy calls for uses this naming convention) so a full name is not just domain_type (or DomainType, domainType) but instead it is domain＿type, for example ActiveDirectory＿OrganizationalUnit. We'd like to have a separator there so we can determine, at runtime, that domain was ActiveDirectory and type was OrganizationalUnit.
(There are some other ways we could achieve this, like adding an enumerated value to each type definition, but that would just generate a lot of fluff)

We can do the logical separation using underscores, but we can also use a unicode extended separator character (e.g. ＿) to logically separate these parts of the name.

However, I thought that having the option to strip/not strip underscores might be useful for some people (I noticed other reported issues mentioning the fact that underscores get stripped) and also for some languages which may use/approve of this naming styling (C, Python, etc ?) The two features should allow for more customizing options for each supported language, e.g. enable unicode characters for languages that support it, enable underscores for languages (or companies) that use snake case.

[eblis]

I just noticed the name template variable for classes, which I missed the first time looking at the variables :(
Looks like it's holding the original name of the class, including any unicode characters and/or underscores so we could probably use this value and annotate all generated types with the original name, instead of relying on the name of the class.

However, I feel like the original pull request is still valid, as you can make use of unicode characters in variable names and allow languages (and companies) that use snake case to generate code using swagger codegen.

[bhuvnesharya]

Thanks for the excellent use cases @eblis . I am having a use case where data is coming from the MATLAB type system and the image is already created from that and used from a long time and the response is again send back to the other system to do some analysis. I have to write an application in between that do some transformations and send the data in the same format as it is coming from the first system( snake_case). If I will not have support of snake_case then I need to convert snake_case to camelCase ( already done by swagger codegen , while storing I need to store it in camelCase and while retrieving and send back data to other system , I need to convert it back to snake_case. Which creates complexity while writing code.

[bhuvnesharya]

@eblis : Can you provide more insight , how to use KEEP_UNDERSCORES parameter in maven codegen. Shall I use this as a config options.

[eblis]

Just a note, the KEEP_UNDERSCORES setting will not prevent camelization of the names, as I wasn't really interested in the case that much, I just wanted to keep the underscores in place. I guess it makes more sense to stop camelization if the KEEP_UNDERSCORES setting is enabled, as you probably don't want to have camel case + snake case.

The easiest way to use the new settings is to enable them in the configuration options, e.g.:
{ "allowUnicodeIdentifiers": "false", "keepUnderscores": "false", "interfaceOnly": false, "java8": true, "serializableModel": false, "async": false, "library": "spring-boot" }

[eblis]

I have modified the pull request such that it doesn't camelize the names if KEEP_UNDERSCORES option is selected. The settings acts like a case selector now, if KEEP_UNDERSCORES is false camel case will be used, if KEEP_UNDERSCORES is true snake case will be used instead.

[bhuvnesharya]

Thanks @eblis . I am currently using swagger-codegen-maven-plugin-2.2.1 which internally uses swagger codegen -2.2.1 . May I know when a new release will be coming where I can get these above changes merged into master.

[eblis]

It depends if and when this pull request is accepted, the code from this pull request isn't available in the swagger code.
Maybe if there's more demand for the feature it will get in :)

[wing328]

@eblis first of all, thank you for your work. The support for unicode identifier is definitely a nice one and we definitely want to include the enhancement in the upcoming release.

But for the "underscore" feature, I do have the following feedback:

1. the switch/flag is done inside the camelize function. IMO, that's not the best way to do it because a developer will expect the function to simply convert the input into camel case. A switch/flag inside the function makes the implementation relatively easy but that will completely change the behaviour of the function, which can be used elsewhere such as class naming, function naming, filename naming, parameter naming, etc.

2. For some generators (e.g. typescript) we do have an option to keep the original naming, e.g.

	modelPropertyNaming
	    Naming convention for the property: 'camelCase', 'PascalCase', 'snake_case' and 'original', which keeps the original name (Default: camelCase)

(ref: java -jar modules/swagger-codegen-cli/target/swagger-codegen-cli.jar config-help -l typescript-node)

If we implement this option for other generators, it should cover what the "KEEP_UNDERSCORE" option provides. Do you agree?

What I would propose is that I'll cherry pick your work related to ALLOW_UNICODE_IDENTIFIERS and add the feature to the latest master while we discuss further on the "KEEP_UNDERSCORE" feature. What do you think?

[eblis]

I agree with both statements. If swagger already has an option/flag for snake casing we can use that and make it available for multiple languages (sorry, didn't know about the feature). Cherry picking the changes for Unicode support also sounds like a very good idea. If you have any trouble I can try and modify the original pull request and only keep the Unicode chances. Thank you, Cristi.

…

On Wed, Jan 18, 2017, 19:16 wing328 ***@***.***> wrote: @eblis <https://github.com/eblis> first of all, thank you for your work. The support for unicode identifier is definitely a nice one and we definitely want to include the enhancement in the upcoming release. But for the "underscore" feature, I do have the following feedback: 1. the switch/flag is done inside the camelize function. IMO, that's not the best way to do it because a developer will expect the function to simply convert the input into camel case. A switch/flag inside the function makes the implementation relatively easy but that will completely change the behaviour of the function, which can be used elsewhere such as class naming, function naming, filename naming, parameter naming, etc. 2. For some generators (e.g. typescript) we do have an option to keep the original naming, e.g. modelPropertyNaming Naming convention for the property: 'camelCase', 'PascalCase', 'snake_case' and 'original', which keeps the original name (Default: camelCase) (ref: java -jar modules/swagger-codegen-cli/target/swagger-codegen-cli.jar config-help -l typescript-node) If we implement this option for other generators, it should cover what the "KEEP_UNDERSCORE" option provides. Do you agree? What I would propose is that I'll cherry pick your work related to ALLOW_UNICODE_IDENTIFIERS and add the feature to the latest master while we discuss further on the "KEEP_UNDERSCORE" feature. What do you think? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4508 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAg-HPtvo4bbKX25Hb0foFDlGYzpmiCdks5rTkjkgaJpZM4Lc5LO> .

[eblis]

I've reworked the pull request to only include changes related to unicode characters support.

[wing328]

@eblis thanks. Looks good to me. I'll perform some tests later and merge if no question/feedback from me or anyone in the community.

[wing328]

Btw, before I forgot, for your upcoming PRs (not this one), I would recommend creating a new branch for the change as per git best practice.

[wing328]

@eblis PR merged into master. Thanks for your contribution.

[eblis]

Thank you !

…

On Fri, Jan 20, 2017, 08:42 wing328 ***@***.***> wrote: @eblis <https://github.com/eblis> PR merged into master. Thanks for your contribution. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4508 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAg-HHF2SD7Ne7aRO2Zg0BgGiuWkfr6rks5rUFdYgaJpZM4Lc5LO> .