RFC: [x2cpg] Added ForkJoinParallelCpgPassWithAccumulator

[max-leuthaeuser]

This allows to get rid of Global typesSeen contention. I adapted jssrc2cpg to demo this.

If thats something we want, this can go into https://github.com/ShiftLeftSecurity/codepropertygraph/blob/master/codepropertygraph/src/main/scala/io/shiftleft/passes/CpgPass.scala

@bbrehm This also revealed a double call of finish() both in ForkJoinParallelCpgPass.createAndApply() and ForkJoinParallelCpgPass.runWithBuilder()

[max-leuthaeuser]

Waiting for ShiftLeftSecurity/codepropertygraph#1841, then I can remove the "finish() only once" hack here.

[ml86]

Use of a thread local variable seems sub-optimal. Why not have runOnPart return a type X and accumulate this along side the diff graph in the stream.collect consumer function?

[max-leuthaeuser]

Because runOnPart runs on a single part of work (i.e., in our case a single, parsed file).
You would end up with thousands of HashSet instances carrying the result of the runOnPart invocation on that particular part of work. We want map-reduce style of merging at thread level.
Also, this approach works without touching any of the ForkJoinParallelCpgPass API so other frontends can adapt/switch at a later point in time.

[ml86]

It seems to depend on the accumulator use case. If one would end up writing lots of duplicate data in the accumulator like for our seen types than I would also expect the thread local implementation to be better.
Anyway I think we can go ahead with this but please put the new CPG pass along side the others.

[max-leuthaeuser]

Waiting for ShiftLeftSecurity/codepropertygraph#1844 to adapt this PR here.

[bbrehm]

I think the appropriate way would be for CpgPasses to be able to bring their own subclass of DiffGraphBuilder. That can then carry the hashset along.

This is exacly what the stream collector interface is made for, so we should use that.

[max-leuthaeuser]

But that would be per part again right? Anyway, feel free to open a PR in codepropertygraph. I guess the pass API will change too.

[max-leuthaeuser]

Will do a new PR for jssrc2pg using ForkJoinParallelCpgPassWithAccumulator from ShiftLeftSecurity/codepropertygraph#1847 once that release is available.