[RFC FS-1001 POC] String Interpolation [WIP] #6770

yatli · 2019-05-18T08:28:55Z

Planning

This is the beginning of my take on FS-1001 (fsharp/fslang-design#6, fsharp/fslang-design#368) -- I have just started to code it, but I'd like to know what you think.

In this thread, I will interchangeably call the original formatting specifiers "deferred specifier", and inline interpolations "immediate specifier".

New Tokens and Types

We'll have to introduce new tokens & non-terminals in the parser to accommodate the interpolated string information. Given a string:

We want to tokenize the string into:

When we parse the token stream:

SIP_LEFT SynExpr SIP_MID SynExpr SIP_RIGHT

In the parser, we can deduce with the following rules:

STRING_INTEROP_HEAD:
| SIP_LEFT SynExpr
| STRING_INTEROP_HEAD SIP_MID SynExpr

STRING_INTEROP:
| STRING_INTEROP_HEAD SIP_RIGHT

Tokenizer Challenges

If we let the hierarchy of nested interpolation slip from the lexer to the parser, we will have to construct ad-hoc parser and start over again. This is not desirable. However, to get a clean token stream considering all the syntax/semantics is a bit challenging. Here's how it's done in Roslyn
https://github.com/dotnet/roslyn/blob/14e0068d9d529ade86c0ed1544477a221809d0e0/src/Compilers/CSharp/Portable/Parser/Lexer_StringLiteral.cs#L222

            // We have a string of the form
            //                $" ... "
            // or, if isVerbatim is true, of possible forms
            //                $@" ... "
            //                @$" ... "
            // Where the contents contains zero or more sequences
            //                { STUFF }
            // where these curly braces delimit STUFF in expression "holes".
            // In order to properly find the closing quote of the whole string,
            // we need to locate the closing brace of each hole, as strings
            // may appear in expressions in the holes. So we
            // need to match up any braces that appear between them.
            // But in order to do that, we also need to match up any
            // /**/ comments, ' characters quotes, () parens
            // [] brackets, and "" strings, including interpolated holes in the latter.

So basically they hoist a non-regular, mini parser that counts braces.

There are two approaches for us:

Follow Roslyn
Embed interpolation states into the main lexer, so that when an unmatched } is encountered, either yield RBRACE, or resume the string lexer to yield either SIP_MID or SIP_RIGHT.

Option 2 is perhaps more maintainable, because it does not drift away from the "real lexer".

Implementing FS-1001

Suppose we have the token stream properly parsed, and we have STRING_INTEROP nodes in the untyped AST. The string formatters are generated at a much later stage, in src/fsharp/CheckFormatStrings.fs, which is only called by the type checker

fsharp/src/fsharp/TypeChecker.fs

Line 6851 in 8397008

and TcConstStringExpr cenv overallTy env m tpenv s =

There are two situations here.

~~The interpolated string does not have a "deferred" specifier.~~
The interpolated string has one or more "deferred" specifier.

Case 1

No deferred specifier means that we don't have to worry about string formatters. Simply move away from the path stringOrKeywordString -> rawConstant -> constant -...-> later picked up by the type checker, and translate our node as appropriate.

~~After translation, we should get something like:~~

edit: for the completeness of the feature we have to go for case 2.

Case 2

The harder case. We will have to make a closure to capture the interpolated expressions, and return a string formatter for the "deferred" parts.

let ($) (x: Printf.StringFormat<'a>) = x
let bar = "baz"
// the format string should be StringFormat<string -> string>
let fmt = $ "foo %{bar} %s"

fsharp/src/fsharp/FSharp.Core/printf.fs

Line 1519 in 8397008

    
           parseFormatString s typeof<'T> :?> _, (2 * count + 1) - optimizedArgCount // second component is used in SprintfEnv as value for internal buffer

The runtime support is here.
If we capture the immediate values in the format, we can translate the format string to indicate the index of the captured values -- for example, %{s:1} means to extract from format.Captures.[1]

Currently, in the front end (printf), I'm putting the captures in the Format objects, and I assume it will be populated by the backend-generated code.
FormatSpecifier now optionally holds the index of the capture target.
The capture arrays are then forwarded to the ctor of PrintfEnv's.
The builder now handles "Begin-with-capture" arguments on the builder stack. (wip)

The Printf.Specialization class is updated with captured value builders. To prevent combinatorial explosion, we restrict that a capture must be the first argument in the argument pack.
In the main builder, each time an immediate spec is encountered, the pack should be "flushed".
New builders in the families FinalCaptureX, FinalFastCaptureX, ... , ChainedCaptureX , ... , LittleAChainedCaptureX, ... are added.
Note that a format specifier can be both immediate capture and special, so we have to extend the main builder for the special form to consider captures.

Your input is much appreciated!

yatli · 2019-05-18T08:56:50Z

dnfclas · 2019-05-20T11:35:40Z

All CLA requirements met.

forki · 2019-05-20T12:13:26Z

@cartermp looks @yatli is Microsoft employee. Can you please resolve this?

yatli · 2019-05-20T17:22:41Z

Anything I should do at the moment?

cartermp · 2019-05-20T17:25:40Z

@yatli you'll need to sign the CLA to allow any PR to be accepted to the repo. @dsyme can speak more towards this approach.

yatli · 2019-05-21T09:41:06Z

I've signed the CLA. Working on printf.fs.

…re indices

src/fsharp/xlf/FSComp.txt.es.xlf

src/fsharp/FSharp.Core/printf.fs

realvictorprm · 2019-06-17T18:02:28Z

cool stuff :)

cartermp · 2019-06-17T22:15:48Z

Tagging @dsyme for design review

yatli · 2019-06-20T10:42:15Z

@realvictorprm @forki I'm wondering if we could team up on this?
Considering that we all tried to attack 1001 before 😀

realvictorprm · 2019-06-20T11:11:45Z

@yatli convinced, are you on the F# Slack?

yatli · 2019-06-20T11:18:41Z

@realvictorprm yeah, I've just joined ~~FSF~~ FSSF :D

forki · 2019-06-20T11:24:11Z

Well, that was long time ago ;-) Any concrete questions or things you want me to do? Yatao Li <notifications@github.com> schrieb am Do., 20. Juni 2019, 13:18:

…

@realvictorprm <https://github.com/realvictorprm> yeah, I've just joined FSF FSSF :D — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#6770?email_source=notifications&email_token=AAAOANBEPBMCJSA4U5ENGCTP3NRRDA5CNFSM4HNZ2MP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYFDSZY#issuecomment-503986535>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAOANHF5LWWHP5PJT4LJYLP3NRRDANCNFSM4HNZ2MPQ> .

yatli · 2019-06-20T11:38:38Z

@forki no worry, one question for now:
Reading through your PR, I see that you've added some tests.
I'm fairly new to the codebase, so I'm wondering if you could give me some pointers -- where to put unit tests for FSharp.Core (now it's relevant in the interpolation), where to put end-to-end test files, where compiler tests, etc.

cartermp · 2019-07-02T17:51:11Z

There are a lot of design discussions going on here. Can we capture some of this in an RFC? I think I generally agree with the direction and @dsyme suggestions here. But there are some sub-points that would need a separate design discussion. For example, for verbatim interoplated strings, what about both $@ and @$ (as per C# here: dotnet/csharplang#1630)?

Additionally we'd have to adjust tooling to account for understanding when we're in an interoplated context (and complete the } with brace completion.

yatli · 2019-07-03T06:14:27Z

@cartermp yes I agree. There are details that are not covered in FS-1001, some of these points are already well-aligned, and some needs further convergence.

I'm listing some of the well aligned points here (may have missed something):

$"....{}..." as the string interpolation form
Support verbatim/multiline interpolation (we still have to fix down the exact syntax)
``%O` by-default for interpolated expressions
Extend %O to incorporate C# formatting specifiers, for both interpolated/conventional strings
Take cultural info into consideration, which further implies supporting IFormatProvider
And after some more thoughts I agree with @dsyme about separating the interpolated form and the normal form.

The thing that still doesn't come to its fixed-point in my head is, whether the interpolated form should be a Format<_,_,_,_,_>, or FormattableString (net46+ only)/IFormattable.

The former incorporates well with the printf family, while the latter is more of the "embracing the C# way" style. I prefer the former because, it implies that we plant our support for IFormatProvider into the printf family, which supports both the interpolated form and the conventional form.
On the other hand, if we decide to check interpolated string into FormattableString/IFormattable, then there will be a separation between the localization and custom formatting, for conventional/interpolated forms.

Some complexities in extending the printf family:
There is overlap between IFormatProvider and F# %a specifier, if we plan to extend the printf family, it has to be adjusted accordingly so that the custom formatter receives the format provider, in addition to the contextual writer and the payload.

dsyme · 2019-07-03T09:39:25Z

@cartermp I will rewrite FS-1001 today

yatli · 2019-07-03T10:14:55Z

@dsyme shall we split out the proposal to extend %O for conventional strings? (e.g. your %(N3)O example)

dsyme · 2019-07-03T10:36:44Z

@dsyme shall we split out the proposal to extend %O for conventional strings?

It would be great to get it all in together, as a complete package, but I think it depends on whether you feel it fits into the implementation and testing reasonably.

yatli · 2019-07-03T10:53:42Z

I'd love to do it altogether -- just want to make sure that this is indeed the plan :)

dsyme · 2019-07-03T10:53:43Z

The thing that still doesn't come to its fixed-point in my head is, whether the interpolated form should be a Format<_,_,_,_,_>, or FormattableString (net46+ only)/IFormattable.

Could we make it that both interpolated and non-interpolated strings are considered compatible with all of these, depending on the known type from type checking?

Existing string literals interpreted as PrintfFormat strings do not include any interpolation fills - the $ must be added to get this.

May I ask what's your concern on this? Indeed, I think, mixing captures and deferred specs is definitely bad when there're dozens of them. More chances of off-by-one (or more) errors.

I think it would just really surprise people if non-interoplated strings could ever have interpolation fills when used as printf-style format strings. Also not backwards compatible is it, if the format for interpolation is %printfFormat{<interpolationExpression>}? For example sprintf "%d{x}" 3 today produces 3{x}.

If possible, I'd very much want interpolated strings to incorporate with xprintf family for its stronger verbal functions (output to streams, print to the screen without first combining the pieces, etc)

Yes, I agree with this

Rule (2) would mean your example would still need $ to interpolate...

if fmt must be followed by a non-curried form, then csprintf becomes a shortcut for FormattableString.Format(some_culture_info), right?

My thinking was that

interpolated strings would support interpolation fills and be usable as any of Format<_,_,_,_,_>, or FormattableString or IFormattable.
regular strings would not support interpolation fills and are also usable as any of Format<_,_,_,_,_>, or FormattableString or IFormattable.

So you could have

let fmt = csprintf some_culture_info
fmt $"....%d{xyz}..."
fmt "....%d..." xyz

Additionally we'd have to adjust tooling to account for understanding when we're in an interoplated context (and complete the } with brace completion.

yes

There is overlap between IFormatProvider and F# %a specifier, if we plan to extend the printf family, it has to be adjusted accordingly so that the custom formatter receives the format provider, in addition to the contextual writer and the payload.

I think pretty much no one uses %a. If we do want to address this would it be possible to use %z or the like for an extension point also taking IFormatProvider? Would that need to be optional in case it hasn't been provided, or is there a default one that would be passed in.

BTW you can find me on F# Software Foundation Slack and we can chat there if you like

dsyme

Just adding a few comments

src/fsharp/FSharp.Core/.vim/coc-settings.json

src/fsharp/FSharp.Core/printf.fs

dsyme · 2019-07-31T11:23:53Z

@yatli Have you been able to make more progress here? Is there anything we can help with?

yatli · 2019-07-31T12:06:20Z

@dsyme thanks for getting back :)
was interrupted by a minibatch of deadlines, so I have to put the codebase back into my cache again

yatli · 2019-07-31T13:01:11Z

it'll be great if we can do some coding together :)

dsyme · 2019-07-31T13:30:11Z

it'll be great if we can do some coding together :)

Can you msg me on FSSF Slack? Maybe late August

naughtyGitCat · 2020-01-11T11:18:35Z

Now is 2020, Is there any new progress on string interpolation? It's quite a practical feature😀

varon · 2020-03-07T12:22:14Z

Please can we get this in. @dsyme / @yatli ?

dsyme · 2020-04-07T17:26:38Z

Hi @yatli, I'm thinking of taking another look at this. Would you like to still work on it with me for the next iteration?

dsyme · 2020-04-07T19:31:22Z

Hi @yatli, I'm thinking of taking another look at this. Would you like to still work on it with me for the next iteration?

(I'm starting with prototyping the lexer and parser changes)

yatli · 2020-04-08T05:57:24Z

Definitely! Sent from Outlook Mobile<https://aka.ms/blhgte>

…

________________________________ From: Don Syme <notifications@github.com> Sent: Wednesday, April 8, 2020 3:31:39 AM To: dotnet/fsharp <fsharp@noreply.github.com> Cc: Yatao Li <yatli@microsoft.com>; Mention <mention@noreply.github.com> Subject: Re: [dotnet/fsharp] [RFC FS-1001 POC] String Interpolation [WIP] (#6770) Hi @yatli<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.luolix.top%2Fyatli&data=02%7C01%7Cyatli%40microsoft.com%7C784ed6632c234024569108d7db2a4b97%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637218847012904482&sdata=gvD3Lhv9pBLpvIbwr%2FRNaQZM8oSnbV5kUpjdpN0pNes%3D&reserved=0>, I'm thinking of taking another look at this. Would you like to still work on it with me for the next iteration? (I'm starting with prototyping the lexer and parser changes) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.luolix.top%2Fdotnet%2Ffsharp%2Fpull%2F6770%23issuecomment-610578136&data=02%7C01%7Cyatli%40microsoft.com%7C784ed6632c234024569108d7db2a4b97%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637218847012904482&sdata=i2l9NKzqPfOT1IuoBc7YpZQDoHOqL0wT0Y645ggyPHo%3D&reserved=0>, or unsubscribe<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit.luolix.top%2Fnotifications%2Funsubscribe-auth%2FAE5Z7MAJZPLZUJ7SOKDHAHDRLN5RXANCNFSM4HNZ2MPQ&data=02%7C01%7Cyatli%40microsoft.com%7C784ed6632c234024569108d7db2a4b97%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637218847012904482&sdata=u2XbQTDqMCgDH4RMiw%2B%2Fs2UR59WR8Ah2RyTQ52Mvz9M%3D&reserved=0>.

dsyme · 2020-04-08T14:42:34Z

@yatli I'm going to close this PR and start a new one coming from a feature branch feature/string-interp in this repo (containing what you've done so far). You will be able to send work as PRs to that feature branch if you like.

dsyme · 2020-04-08T15:06:35Z

Replaced by #8907

yatli · 2020-04-08T15:10:51Z

sounds great!

dsyme · 2020-04-08T15:16:32Z

Continued at #8907

minor touch: typo

a52d9e7

forki approved these changes May 18, 2019

View reviewed changes

Yatao Li added 2 commits May 20, 2019 14:15

Merge branch 'master' of https://github.com/dotnet/fsharp

083f431

checkpoint.

19b33f8

captures are put in the envs. prepare to build specializers for captu…

f3fa739

…re indices

yatli commented May 21, 2019

View reviewed changes

src/fsharp/xlf/FSComp.txt.es.xlf Outdated Show resolved Hide resolved

Yatao Li added 2 commits May 21, 2019 18:46

findNextFormatSpecifier also needs to consider '{'

3fc2b79

add comments

30b8147

yatli commented May 21, 2019

View reviewed changes

src/fsharp/FSharp.Core/printf.fs Outdated Show resolved Hide resolved

checkpoint.

52648bf

Yatao Li added 6 commits June 18, 2019 20:20

checkpoint.

a669330

Final/Chain specializers for n>1

b293dcd

(temp) vim session file for the folds..

fcc1adc

checkpoint.

17f8788

checkpoint.

fa3301d

printf.fs: capture-related code complete

eec4129

expect Types from codegen, not runtime

075619f

Merge branch 'master' into master

f507ac8

dsyme reviewed Jul 3, 2019

View reviewed changes

src/fsharp/FSharp.Core/.vim/coc-settings.json Outdated Show resolved Hide resolved

src/fsharp/FSharp.Core/printf.fs Outdated Show resolved Hide resolved

src/fsharp/FSharp.Core/printf.fs Outdated Show resolved Hide resolved

dsyme mentioned this pull request Jul 3, 2019

Revised string interpolation spec fsharp/fslang-design#367

Merged

Yatao Li added 4 commits July 3, 2019 21:07

cleanup, ignore editor config files

6e23c9c

PopValueUnsafe uppercase

07d0fd8

PrintFormat: capture fields should be immutable

f75ab75

printf: Type is in System namespace

fadb047

inosik mentioned this pull request Mar 18, 2020

Support tagged template literals (e.g. for styled-components) fable-compiler/Fable#1973

Closed

dsyme mentioned this pull request Apr 8, 2020

[RFC FS-1001] String Interpolation #8907

Merged

29 tasks

dsyme closed this Apr 8, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RFC FS-1001 POC] String Interpolation [WIP] #6770

[RFC FS-1001 POC] String Interpolation [WIP] #6770

yatli commented May 18, 2019 •

edited

Loading

yatli commented May 18, 2019 •

edited

Loading

dnfclas commented May 20, 2019 •

edited

Loading

forki commented May 20, 2019

yatli commented May 20, 2019

cartermp commented May 20, 2019

yatli commented May 21, 2019

realvictorprm commented Jun 17, 2019

cartermp commented Jun 17, 2019

yatli commented Jun 20, 2019

realvictorprm commented Jun 20, 2019

yatli commented Jun 20, 2019

forki commented Jun 20, 2019 via email

yatli commented Jun 20, 2019

cartermp commented Jul 2, 2019

yatli commented Jul 3, 2019 •

edited

Loading

dsyme commented Jul 3, 2019

yatli commented Jul 3, 2019 •

edited

Loading

dsyme commented Jul 3, 2019

yatli commented Jul 3, 2019

dsyme commented Jul 3, 2019

dsyme left a comment

dsyme commented Jul 31, 2019

yatli commented Jul 31, 2019

yatli commented Jul 31, 2019

dsyme commented Jul 31, 2019

naughtyGitCat commented Jan 11, 2020

varon commented Mar 7, 2020

dsyme commented Apr 7, 2020

dsyme commented Apr 7, 2020

yatli commented Apr 8, 2020 via email

dsyme commented Apr 8, 2020 •

edited

Loading

dsyme commented Apr 8, 2020

yatli commented Apr 8, 2020

dsyme commented Apr 8, 2020

[RFC FS-1001 POC] String Interpolation [WIP] #6770

[RFC FS-1001 POC] String Interpolation [WIP] #6770

Conversation

yatli commented May 18, 2019 • edited Loading

Planning

New Tokens and Types

Tokenizer Challenges

Implementing FS-1001

Case 1

Case 2

yatli commented May 18, 2019 • edited Loading

Status

Scratchpad

dnfclas commented May 20, 2019 • edited Loading

forki commented May 20, 2019

yatli commented May 20, 2019

cartermp commented May 20, 2019

yatli commented May 21, 2019

realvictorprm commented Jun 17, 2019

cartermp commented Jun 17, 2019

yatli commented Jun 20, 2019

realvictorprm commented Jun 20, 2019

yatli commented Jun 20, 2019

forki commented Jun 20, 2019 via email

yatli commented Jun 20, 2019

cartermp commented Jul 2, 2019

yatli commented Jul 3, 2019 • edited Loading

dsyme commented Jul 3, 2019

yatli commented Jul 3, 2019 • edited Loading

dsyme commented Jul 3, 2019

yatli commented Jul 3, 2019

dsyme commented Jul 3, 2019

dsyme left a comment

Choose a reason for hiding this comment

dsyme commented Jul 31, 2019

yatli commented Jul 31, 2019

yatli commented Jul 31, 2019

dsyme commented Jul 31, 2019

naughtyGitCat commented Jan 11, 2020

varon commented Mar 7, 2020

dsyme commented Apr 7, 2020

dsyme commented Apr 7, 2020

yatli commented Apr 8, 2020 via email

dsyme commented Apr 8, 2020 • edited Loading

dsyme commented Apr 8, 2020

yatli commented Apr 8, 2020

dsyme commented Apr 8, 2020

yatli commented May 18, 2019 •

edited

Loading

yatli commented May 18, 2019 •

edited

Loading

dnfclas commented May 20, 2019 •

edited

Loading

yatli commented Jul 3, 2019 •

edited

Loading

yatli commented Jul 3, 2019 •

edited

Loading

dsyme commented Apr 8, 2020 •

edited

Loading