UNOB charset for EDIFACT output

[p-e-jak]

Hi,

i'm successfully using staedi for EDIFACT output in a customer project (many thanks for this implementation with simple & flexible interface)

The customer requirements dictate to use the (fairly limited) UNOA charset, comprising the characters A-Z, a-z, 0-9, and a few more special characters ()

EDIOutputFactory.createEDIStreamWriter(OutputStream stream, String encoding) already provides a way to specify an encoding.
However, to my understanding it's currently not possible - in other words: responsibility of the staedi client - to ensure the output only contains characters valid according to UNOA.

In my case the input chars to be written to EDIFACT may also be a wider variety of Unicode characters including German umlauts ÄÖÜ, "&" frequently needed for German addresses (....straße = street) and others not contained in UNOA.
Hence, i need some conversion from the input to UNOB. This conversion may be customer specific, eg. Ä -> AE, or "ß" -> sz or s.

Would it be useful for other EDIFACT clients too, to perform such conversion - eg. based on a mapping from input char in a frequently used charset to ->UNOA/UNOB output char in some config file?
One possible way of implementing this (transparently) i can think of: using java.io.FilterOutputStream

I'm looking forward to your feedback :)

[MikeEdgar]

@p-e-jak thank you for starting this discussion. I think using something like a configurable FilterOutputStream sounds like a great idea. I imagine that the stream could be configured with a set of mappings (perhaps an overloaded createEDIStreamWriter method) that allows you to provide a Map<Character, String>. I'm curious to hear if you have any thoughts on how it would integrate with the existing API.

[p-e-jak]

@MikeEdgar thank you for picking up this suggestion!
I think your initial approach is fine assuming that clients don't want to configure replacement mappings for a set of charsets (syntax IDs) at once:

Map<Character, String> srcToUnob = new HashMap<>();
srcToUnob.put(new Character('ß'), new String("sz"));
...
writer = factory.createEDIStreamWriter(out, srcToUnob);

If clients want to specify >1 charset at once (eg. the charset is not known at compile time yet, and overall different charsets are necessary) then they need to specify which replacement mapping to use for which respective syntax ID (UNOA, UNOB, and the complete range of possible values):

Map<Character, String> srcToUnoB = new HashMap<>();
srcToUnob.put(new Character('ß'), new String("sz"));
...
writer = factory.createEDIStreamWriter(out, "UNOB", srcToUnob);

Not sure if that's a common use case but in this case a config via the factory creation method looks a bit cumbersome. In this case the config could be done by setting a property on the factory:
EDIOutputFactory.setProperty(EDIStreamConstants.CharsetMapper.UNOB, srcToUnoB); EDIOutputFactory.setProperty(EDIStreamConstants.CharsetMapper.UNOW, srcToUnoW);
staedi could then automatically use the mapping after the following code:

writer.writeStartSegment("UNB");
writer.writeStartElement()
	// syntax ID -> from now on staedi knows which charset mapping to pick
	.writeComponent(header.getSyntaxId())
	// syntax version
	.writeComponent(header.getSyntaxVersion()).endElement();

Alternatively there could be a way to provide the mapping via a property file - eg. one per charset/syntaxId - in the classpath.

Another question regarding the implementation: How should characters be handled if they are not contained in the chosen charset and not mapped in the replacement mapping either. Strategies could be "simply escape using the release character" or "simply do nothing" (such as current impl., right?)

Also, it would be possible that the FilterOutputStream picking up the mapping performs some string normalization on the input. I'm very new to this topic and merely know it from university ;) Eg. a Unicode character can be written as precomposed character or as decomposed characters. (like here: Comparing precomposed and decomposed characters)
But that could also be responsibility of clients and maybe does not need to be considered in the inital approach.

Some test code could show how the different approaches/ideas act in practice :)

[MikeEdgar]

If clients want to specify >1 charset at once (eg. the charset is not known at compile time yet, and overall different charsets are necessary) then they need to specify which replacement mapping to use for which respective syntax ID (UNOA, UNOB, and the complete range of possible values):

I tend to prefer keeping staedi more low-level so maintaining the full set of mappings for multiple encodings seems to me more a concern of the client application. It also introduces some burden to the library to have more knowledge about the meaning of the UNB01-1 element (it has some limited knowledge about other elements already, so maybe not a bad thing in this case). I don't think it would be unreasonable for an application to create a writer for each character encoding it is dealing with. The application would know what it will write to the UNB01-1 element and could load the corresponding Map based on that.

I think your point about decomposed characters makes it seem better to use a Map<String, String> rather than Map<Character, String> for the mapping. That would allow a sequence of characters that represent a single character to be mapped properly.