Exception in thread "main" overflowdb.traversal.filter.StringPropertyFilter$InvalidRegexException: invalid regular expression

[almaz045]

I've tried to test depscan on https://github.com/grpc/grpc by next command:

atom reachables -o app.atom -s reachables.json -l c .

p.s. bom.json is exist

And got next error:

Generating data-flow dependencies from atom. Please wait ...
Exception in thread "main" overflowdb.traversal.filter.StringPropertyFilter$InvalidRegexException: invalid regular expression: ` grpc_core.ChildPolicyHandler.ConfigChangeRequiresNewPolicyInstance:bool(grpc_core::LoadBalancingPolicy::Config *`
	at overflowdb.traversal.filter.StringPropertyFilter$.regexpCompile(StringPropertyFilter.scala:48)
	at overflowdb.traversal.filter.StringPropertyFilter$.regexp(StringPropertyFilter.scala:13)
	at io.shiftleft.codepropertygraph.generated.traversal.CallTraversalExtGen$.typeFullName$extension(Call.scala:597)
	at io.appthreat.x2cpg.passes.taggers.CdxPass.run$$anonfun$1$$anonfun$5$$anonfun$2$$anonfun$5(CdxPass.scala:285)
	at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
	at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
	at scala.collection.ArrayOps$.foreach$extension(ArrayOps.scala:1323)
	at io.appthreat.x2cpg.passes.taggers.CdxPass.run$$anonfun$1$$anonfun$5$$anonfun$2(CdxPass.scala:351)
	at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
	at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
	at scala.collection.immutable.Vector.foreach(Vector.scala:2124)
	at io.appthreat.x2cpg.passes.taggers.CdxPass.run$$anonfun$1$$anonfun$5(CdxPass.scala:353)
	at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
	at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
	at scala.collection.immutable.VectorStatics$.foreachRec(Vector.scala:2124)
	at scala.collection.immutable.Vector.foreach(Vector.scala:2130)
	at io.appthreat.x2cpg.passes.taggers.CdxPass.run$$anonfun$1(CdxPass.scala:354)
	at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
	at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
	at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:576)
	at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:574)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1300)
	at io.appthreat.x2cpg.passes.taggers.CdxPass.run(CdxPass.scala:355)
	at io.shiftleft.passes.CpgPass.runOnPart(CpgPass.scala:29)
	at io.shiftleft.passes.NewStyleCpgPassBase.runWithBuilder(CpgPass.scala:134)
	at io.shiftleft.passes.ForkJoinParallelCpgPass.createApplySerializeAndStore(CpgPass.scala:77)
	at io.shiftleft.passes.NewStyleCpgPassBase.createAndApply(CpgPass.scala:124)
	at io.appthreat.atom.Atom$.generateForLanguage(Atom.scala:606)
	at io.appthreat.atom.Atom$.generateAtom(Atom.scala:446)
	at io.appthreat.atom.Atom$.run(Atom.scala:300)
	at io.appthreat.atom.Atom$.run(Atom.scala:294)
	at io.appthreat.atom.Atom$.main(Atom.scala:275)
	at io.appthreat.atom.Atom.main(Atom.scala)
Caused by: java.util.regex.PatternSyntaxException: Unclosed group near index 117
(?s) grpc_core.ChildPolicyHandler.ConfigChangeRequiresNewPolicyInstance:bool(grpc_core::LoadBalancingPolicy::Config *
	at java.base/java.util.regex.Pattern.error(Pattern.java:2204)
	at java.base/java.util.regex.Pattern.accept(Pattern.java:2054)
	at java.base/java.util.regex.Pattern.group0(Pattern.java:3232)
	at java.base/java.util.regex.Pattern.sequence(Pattern.java:2300)
	at java.base/java.util.regex.Pattern.expr(Pattern.java:2245)
	at java.base/java.util.regex.Pattern.compile(Pattern.java:1945)
	at java.base/java.util.regex.Pattern.<init>(Pattern.java:1576)
	at java.base/java.util.regex.Pattern.compile(Pattern.java:1101)
	at scala.util.matching.Regex.<init>(Regex.scala:234)
	at scala.collection.StringOps$.r$extension(StringOps.scala:852)
	at overflowdb.traversal.filter.StringPropertyFilter$.regexpCompile(StringPropertyFilter.scala:45)
	... 32 more

[prabhu]

Interesting test case! There is a type that is confusing the regex during tagging.

[prabhu]

@almaz045 could you confirm the version of atom used, since I don't get this error with the latest.

[kittyandrew]

Hey, I've got very similar crash on ghcr.io/appthreat/atom:nightly (as of today) on my repo

Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8
Generating data-flow dependencies from atom. Please wait ...
Exception in thread "main" overflowdb.traversal.filter.StringPropertyFilter$InvalidRegexException: invalid regular expression: `,_a).py:<module>.*`
	at overflowdb.traversal.filter.StringPropertyFilter$.regexpCompile(StringPropertyFilter.scala:48)
	at overflowdb.traversal.filter.StringPropertyFilter$.regexp(StringPropertyFilter.scala:13)
	at io.shiftleft.codepropertygraph.generated.traversal.CallTraversalExtGen$.methodFullName$extension(Call.scala:361)
	at io.appthreat.x2cpg.passes.taggers.CdxPass.run$$anonfun$1$$anonfun$5$$anonfun$1$$anonfun$1$$anonfun$1(CdxPass.scala:118)
	at overflowdb.traversal.TraversalLogicExt$.where$extension$$anonfun$1(Traversal.scala:176)
	at scala.collection.Iterator$$anon$6.hasNext(Iterator.scala:479)
	at scala.collection.Iterator$$anon$9.hasNext(Iterator.scala:583)
	at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:619)
	at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:617)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1303)
	at io.shiftleft.semanticcpg.language.NewTagNodePairTraversal.store(NewTagNodePairTraversal.scala:9)
	at io.appthreat.x2cpg.passes.taggers.CdxPass.run$$anonfun$1$$anonfun$5$$anonfun$1$$anonfun$1(CdxPass.scala:119)
	at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
	at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
	at scala.collection.immutable.Set$Set2.foreach(Set.scala:210)
	at io.appthreat.x2cpg.passes.taggers.CdxPass.run$$anonfun$1$$anonfun$5$$anonfun$1(CdxPass.scala:114)
	at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
	at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
	at scala.collection.immutable.Set$Set1.foreach(Set.scala:177)
	at io.appthreat.x2cpg.passes.taggers.CdxPass.run$$anonfun$1$$anonfun$5(CdxPass.scala:112)
	at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
	at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
	at scala.collection.immutable.Vector.foreach(Vector.scala:2124)
	at io.appthreat.x2cpg.passes.taggers.CdxPass.run$$anonfun$1(CdxPass.scala:91)
	at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
	at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
	at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:619)
	at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:617)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1303)
	at io.appthreat.x2cpg.passes.taggers.CdxPass.run(CdxPass.scala:64)
	at io.shiftleft.passes.CpgPass.runOnPart(CpgPass.scala:29)
	at io.shiftleft.passes.NewStyleCpgPassBase.runWithBuilder(CpgPass.scala:134)
	at io.shiftleft.passes.ForkJoinParallelCpgPass.createApplySerializeAndStore(CpgPass.scala:77)
	at io.shiftleft.passes.NewStyleCpgPassBase.createAndApply(CpgPass.scala:124)
	at io.appthreat.atom.Atom$.generateForLanguage(Atom.scala:606)
	at io.appthreat.atom.Atom$.generateAtom(Atom.scala:446)
	at io.appthreat.atom.Atom$.run(Atom.scala:300)
	at io.appthreat.atom.Atom$.run(Atom.scala:294)
	at io.appthreat.atom.Atom$.main(Atom.scala:275)
	at io.appthreat.atom.Atom.main(Atom.scala)
Caused by: java.util.regex.PatternSyntaxException: Unmatched closing ')' near index 6
(?s),_a).py:<module>.*
      ^
	at java.base/java.util.regex.Pattern.error(Pattern.java:2204)
	at java.base/java.util.regex.Pattern.compile(Pattern.java:1949)
	at java.base/java.util.regex.Pattern.<init>(Pattern.java:1576)
	at java.base/java.util.regex.Pattern.compile(Pattern.java:1101)
	at scala.util.matching.Regex.<init>(Regex.scala:234)
	at scala.collection.StringOps$.r$extension(StringOps.scala:852)
	at overflowdb.traversal.filter.StringPropertyFilter$.regexpCompile(StringPropertyFilter.scala:45)
	... 39 more

[prabhu]

@kittyandrew, could you share the .py file or the code snippet?

[kittyandrew]

Hi, I've looked at it again today and realized that I might've included .venv directory when pointing to the "current working directory". Problem went away when I started pointing only to my source code.

I don't know if that makes sense to debug further

[prabhu]

Thank you for confirming. Will keep this issue open in case we face any other problematic pattern in the future.