Feature/fixembedding (antlr#4176)

* feat: Createa n Init routine for BaseATNConfig so we can embed sructs rather than allocate to pointers Signed-off-by: Jim.Idle <jimi@idle.ws> * feat: Change BaseATNConfig to be properly embedded in other structs such as LexerATNConfig instead of by pointer This is the first of many changes that switches the embedded class structure that was copying Java class hieracrchy from allocations/new to proper embedding such that any struct is allocated with one allocation not two or more. Main PR will cover what this means. Signed-off-by: Jim.Idle <jimi@idle.ws> * feat: Change embedding for ATNBaseSimulator to true embedding instaed of pointer Saves an extra allocation and helps the GC Signed-off-by: Jim.Idle <jimi@idle.ws> * feat: Switch the use of pointers to embedded ATN states to true embeddding Saves many allocations and grbage collections Signed-off-by: Jim.Idle <jimi@idle.ws> * fix: Correct the way that PredictionContext is compared for merge Should reduce allocation count by tons. Signed-off-by: Jim.Idle <jimi@idle.ws> * Feature/docclean Greatly improve the godoc comments in the runtime (antlr#4169) * doc: Updates to some of the Go doc comments to start a ful ldocumentation cleanup Signed-off-by: Jim.Idle <jimi@idle.ws> * doc: More documentation fixes. Using this as a method of forcing myself to read every line of code in the runtime, and therefore discover mistakes in the original implementation. And, of course, actually working docs for the Go runtime, can only be a good thing. Signed-off-by: Jim.Idle <jimi@idle.ws> * doc: More documentation fixes Also changes the exporet level of a some variables and funcs that were not correct, even though no user has currently needed them it would seem. Signed-off-by: Jim.Idle <jimi@idle.ws> * doc: Many updates to document exported fuctions correctly and reformat the ingerited Java code It looks like a massive amount of changes, but it is almost all doc or changing exports or renaming unused paramters etc to make the Go linter happy. No actual code changes yet. Signed-off-by: Jim.Idle <jimi@idle.ws> * doc: More additions and corrections to the Go documentation for the runtime Signed-off-by: Jim.Idle <jimi@idle.ws> * doc: Final clean of exported func and type documentation There will be more to do here as there are a lot of things that are hidden internal to the antlr package that probably should not be. There are also a lot of exported funcs and types without any documentation, that will eventually need to be cleaned up. Signed-off-by: Jim.Idle <jimi@idle.ws> * Changed Parser typings (antlr#4149) Signed-off-by: Josua Frank <josua.frank@daimlertruck.com> Co-authored-by: Josua Frank <josua.frank@daimlertruck.com> Signed-off-by: Jim.Idle <jimi@idle.ws> * fix: Fixes the failing go runtime test suite which was missing the /v4 off the replace option on the go.mod file (antlr#4163) Arrrgh! Signed-off-by: Jim.Idle <jimi@idle.ws> * present antlr before versioning (antlr#4156) Signed-off-by: Jim.Idle <jimi@idle.ws> * fix: Prevent use of labels such as start= from generating code that clashes with builtin funcs (antlr#4161) Signed-off-by: Jim.Idle <jimi@idle.ws> * Feature/gotestfix (antlr#4168) * fix: Fixes the failing go runtime test suite which was missing the /v4 off the replace option on the go.mod file Arrrgh! Signed-off-by: Jim.Idle <jimi@idle.ws> * present antlr before versioning (antlr#4156) Signed-off-by: Jim.Idle <jimi@idle.ws> * fix: Prevent use of labels such as start= from generating code that clashes with builtin funcs (antlr#4161) Signed-off-by: Jim.Idle <jimi@idle.ws> * fix: Cater for the fact that some test rules use start as a label or rule name As a fix for other cvode gen errors when start, end, or exception are used as label names, they are now translated to have a suffix of `_` at code gen time. However, the runtime tests sometimes use start as a rule name and so we must now cater for this in the tests. Signed-off-by: Jim.Idle <jimi@idle.ws> --------- Signed-off-by: Jim.Idle <jimi@idle.ws> Co-authored-by: ericvergnaud <eric.vergnaud@wanadoo.fr> Signed-off-by: Jim.Idle <jimi@idle.ws> --------- Signed-off-by: Jim.Idle <jimi@idle.ws> Signed-off-by: Josua Frank <josua.frank@daimlertruck.com> Co-authored-by: Josua Frank <frank.josua@gmail.com> Co-authored-by: Josua Frank <josua.frank@daimlertruck.com> Co-authored-by: ericvergnaud <eric.vergnaud@wanadoo.fr> * feat: Change BaseATNConfig to be properly embedded in other structs such as LexerATNConfig instead of by pointer This is the first of many changes that switches the embedded class structure that was copying Java class hieracrchy from allocations/new to proper embedding such that any struct is allocated with one allocation not two or more. Main PR will cover what this means. Signed-off-by: Jim.Idle <jimi@idle.ws> * feat: Change embedding for ATNBaseSimulator to true embedding instaed of pointer Saves an extra allocation and helps the GC Signed-off-by: Jim.Idle <jimi@idle.ws> * fix: Correct the way that PredictionContext is compared for merge Should reduce allocation count by tons. Signed-off-by: Jim.Idle <jimi@idle.ws> * doc: Merge documentation updates Signed-off-by: Jim.Idle <jimi@idle.ws> * feat: Rework predictions tructs to use emedding instead of pointers Signed-off-by: Jim.Idle <jimi@idle.ws> * feat: more reworking of PredictionContext for embedding Signed-off-by: Jim.Idle <jimi@idle.ws> * feat: Ensure that EmptyPredictionContext is correctly initialized Rework of the variaous PredictionContexts has reduced memory allocations to between 30% and 50% of previous version. Signed-off-by: Jim.Idle <jimi@idle.ws> * feat: Change from use of type casting to using stored type Signed-off-by: Jim.Idle <jimi@idle.ws> * feat: Convert CommonToken to true emedding rather than pointers Signed-off-by: Jim.Idle <jimi@idle.ws> --------- Signed-off-by: Jim.Idle <jimi@idle.ws> Signed-off-by: Josua Frank <josua.frank@daimlertruck.com> Co-authored-by: Josua Frank <frank.josua@gmail.com> Co-authored-by: Josua Frank <josua.frank@daimlertruck.com> Co-authored-by: ericvergnaud <eric.vergnaud@wanadoo.fr> Signed-off-by: Jim.Idle <jimi@idle.ws>
jimidle · Mar 28, 2023 · ccd0ed4 · ccd0ed4
1 parent c466045
commit ccd0ed4
Show file tree

Hide file tree

Showing 22 changed files with 1,279 additions and 1,109 deletions.
diff --git a/runtime/Go/antlr/v4/array_prediction_context.go b/runtime/Go/antlr/v4/array_prediction_context.go
@@ -0,0 +1,115 @@
+package antlr
+
+import (
+	"golang.org/x/exp/slices"
+	"strconv"
+)
+
+type ArrayPredictionContext struct {
+	BasePredictionContext
+	parents      []PredictionContext
+	returnStates []int
+}
+
+func NewArrayPredictionContext(parents []PredictionContext, returnStates []int) *ArrayPredictionContext {
+	// Parent can be nil only if full ctx mode and we make an array
+	// from {@link //EMPTY} and non-empty. We merge {@link //EMPTY} by using
+	// nil parent and
+	// returnState == {@link //EmptyReturnState}.
+	hash := murmurInit(1)
+	for _, parent := range parents {
+		hash = murmurUpdate(hash, parent.Hash())
+	}
+	for _, returnState := range returnStates {
+		hash = murmurUpdate(hash, returnState)
+	}
+	hash = murmurFinish(hash, len(parents)<<1)
+
+	return &ArrayPredictionContext{
+		BasePredictionContext: BasePredictionContext{
+			cachedHash: hash,
+			pcType:     PredictionContextArray,
+		},
+		parents:      parents,
+		returnStates: returnStates,
+	}
+}
+
+func (a *ArrayPredictionContext) GetReturnStates() []int {
+	return a.returnStates
+}
+
+func (a *ArrayPredictionContext) hasEmptyPath() bool {
+	return a.getReturnState(a.length()-1) == BasePredictionContextEmptyReturnState
+}
+
+func (a *ArrayPredictionContext) isEmpty() bool {
+	// since EmptyReturnState can only appear in the last position, we
+	// don't need to verify that size==1
+	return a.returnStates[0] == BasePredictionContextEmptyReturnState
+}
+
+func (a *ArrayPredictionContext) length() int {
+	return len(a.returnStates)
+}
+
+func (a *ArrayPredictionContext) GetParent(index int) PredictionContext {
+	return a.parents[index]
+}
+
+func (a *ArrayPredictionContext) getReturnState(index int) int {
+	return a.returnStates[index]
+}
+
+// Equals is the default comparison function for ArrayPredictionContext when no specialized
+// implementation is needed for a collection
+func (a *ArrayPredictionContext) Equals(o interface{}) bool {
+	if a == o {
+		return true
+	}
+	other, ok := o.(*ArrayPredictionContext)
+	if !ok {
+		return false
+	}
+	if a.cachedHash != other.Hash() {
+		return false // can't be same if hash is different
+	}
+
+	// Must compare the actual array elements and not just the array address
+	//
+	return slices.Equal(a.returnStates, other.returnStates) &&
+		slices.EqualFunc(a.parents, other.parents, func(x, y PredictionContext) bool {
+			return x.Equals(y)
+		})
+}
+
+// Hash is the default hash function for ArrayPredictionContext when no specialized
+// implementation is needed for a collection
+func (a *ArrayPredictionContext) Hash() int {
+	return a.BasePredictionContext.cachedHash
+}
+
+func (a *ArrayPredictionContext) String() string {
+	if a.isEmpty() {
+		return "[]"
+	}
+
+	s := "["
+	for i := 0; i < len(a.returnStates); i++ {
+		if i > 0 {
+			s = s + ", "
+		}
+		if a.returnStates[i] == BasePredictionContextEmptyReturnState {
+			s = s + "$"
+			continue
+		}
+		s = s + strconv.Itoa(a.returnStates[i])
+		if a.parents[i] != nil {
+			s = s + " " + a.parents[i].String()
+		} else {
+			s = s + "nil"
+		}
+	}
+
+	return s + "]"
+}
diff --git a/runtime/Go/antlr/v4/atn_config.go b/runtime/Go/antlr/v4/atn_config.go
@@ -106,17 +106,23 @@ func NewBaseATNConfig1(c ATNConfig, state ATNState, context PredictionContext) *
 // are just wrappers around this one.
 func NewBaseATNConfig(c ATNConfig, state ATNState, context PredictionContext, semanticContext SemanticContext) *BaseATNConfig {
 	if semanticContext == nil {
-		panic("semanticContext cannot be nil")
+		panic("semanticContext cannot be nil") // TODO: Remove this - probably put here for some bug that is now fixed
 	}
 
-	return &BaseATNConfig{
-		state:                      state,
-		alt:                        c.GetAlt(),
-		context:                    context,
-		semanticContext:            semanticContext,
-		reachesIntoOuterContext:    c.GetReachesIntoOuterContext(),
-		precedenceFilterSuppressed: c.getPrecedenceFilterSuppressed(),
-	}
+	b := &BaseATNConfig{}
+	b.InitBaseATNConfig(c, state, c.GetAlt(), context, semanticContext)
+
+	return b
+}
+
+func (b *BaseATNConfig) InitBaseATNConfig(c ATNConfig, state ATNState, alt int, context PredictionContext, semanticContext SemanticContext) {
+
+	b.state = state
+	b.alt = alt
+	b.context = context
+	b.semanticContext = semanticContext
+	b.reachesIntoOuterContext = c.GetReachesIntoOuterContext()
+	b.precedenceFilterSuppressed = c.getPrecedenceFilterSuppressed()
 }
 
 func (b *BaseATNConfig) getPrecedenceFilterSuppressed() bool {
@@ -237,49 +243,74 @@ func (b *BaseATNConfig) String() string {
 // BaseATNConfig struct.
 // TODO: Stop using a pointer and embed the struct instead as this saves allocations. Same for the LexerATNConfig "constructors"
 type LexerATNConfig struct {
-	*BaseATNConfig
+	BaseATNConfig
 	lexerActionExecutor            *LexerActionExecutor
 	passedThroughNonGreedyDecision bool
 }
 
 func NewLexerATNConfig6(state ATNState, alt int, context PredictionContext) *LexerATNConfig {
-	return &LexerATNConfig{BaseATNConfig: NewBaseATNConfig5(state, alt, context, SemanticContextNone)}
+
+	return &LexerATNConfig{
+		BaseATNConfig: BaseATNConfig{
+			state:           state,
+			alt:             alt,
+			context:         context,
+			semanticContext: SemanticContextNone,
+		},
+	}
 }
 
 func NewLexerATNConfig5(state ATNState, alt int, context PredictionContext, lexerActionExecutor *LexerActionExecutor) *LexerATNConfig {
 	return &LexerATNConfig{
-		BaseATNConfig:       NewBaseATNConfig5(state, alt, context, SemanticContextNone),
+		BaseATNConfig: BaseATNConfig{
+			state:           state,
+			alt:             alt,
+			context:         context,
+			semanticContext: SemanticContextNone,
+		},
 		lexerActionExecutor: lexerActionExecutor,
 	}
 }
 
 func NewLexerATNConfig4(c *LexerATNConfig, state ATNState) *LexerATNConfig {
-	return &LexerATNConfig{
-		BaseATNConfig:                  NewBaseATNConfig(c, state, c.GetContext(), c.GetSemanticContext()),
+	lac := &LexerATNConfig{
+
 		lexerActionExecutor:            c.lexerActionExecutor,
 		passedThroughNonGreedyDecision: checkNonGreedyDecision(c, state),
 	}
+	lac.BaseATNConfig.InitBaseATNConfig(c, state, c.GetAlt(), c.GetContext(), c.GetSemanticContext())
+	return lac
 }
 
 func NewLexerATNConfig3(c *LexerATNConfig, state ATNState, lexerActionExecutor *LexerActionExecutor) *LexerATNConfig {
-	return &LexerATNConfig{
-		BaseATNConfig:                  NewBaseATNConfig(c, state, c.GetContext(), c.GetSemanticContext()),
+	lac := &LexerATNConfig{
 		lexerActionExecutor:            lexerActionExecutor,
 		passedThroughNonGreedyDecision: checkNonGreedyDecision(c, state),
 	}
+	lac.BaseATNConfig.InitBaseATNConfig(c, state, c.GetAlt(), c.GetContext(), c.GetSemanticContext())
+	return lac
 }
 
 func NewLexerATNConfig2(c *LexerATNConfig, state ATNState, context PredictionContext) *LexerATNConfig {
-	return &LexerATNConfig{
-		BaseATNConfig:                  NewBaseATNConfig(c, state, context, c.GetSemanticContext()),
+	lac := &LexerATNConfig{
 		lexerActionExecutor:            c.lexerActionExecutor,
 		passedThroughNonGreedyDecision: checkNonGreedyDecision(c, state),
 	}
+	lac.BaseATNConfig.InitBaseATNConfig(c, state, c.GetAlt(), context, c.GetSemanticContext())
+	return lac
 }
 
 //goland:noinspection GoUnusedExportedFunction
 func NewLexerATNConfig1(state ATNState, alt int, context PredictionContext) *LexerATNConfig {
-	return &LexerATNConfig{BaseATNConfig: NewBaseATNConfig5(state, alt, context, SemanticContextNone)}
+	lac := &LexerATNConfig{
+		BaseATNConfig: BaseATNConfig{
+			state:           state,
+			alt:             alt,
+			context:         context,
+			semanticContext: SemanticContextNone,
+		},
+	}
+	return lac
 }
 
 // Hash is the default hash function for LexerATNConfig objects, it can be used directly or via
@@ -330,7 +361,7 @@ func (l *LexerATNConfig) Equals(other Collectable[ATNConfig]) bool {
 		return false
 	}
 
-	return l.BaseATNConfig.Equals(otherT.BaseATNConfig)
+	return l.BaseATNConfig.Equals(&otherT.BaseATNConfig)
 }
 
 func checkNonGreedyDecision(source *LexerATNConfig, target ATNState) bool {