Is it possible? If yes, how to do it?
I wrote Designscript code to create multiple road routes, there are several performance issues:
I use loop to create them one by one, but if parallel operations can be done, I can create them in parallel.
for each route, it’s consists of many curves, I use loop to create them one by one. I can also create them in parallel if Parallel operations can be done.
Hello
you can try to use IronPython with Parallel LINQ
a small test
import clr
import time
clr.AddReference('ProtoGeometry')
from Autodesk.DesignScript.Geometry import *
from System.Threading.Tasks import *
clr.AddReference("System.Core")
import System
clr.ImportExtensions(System.Linq)
def drawlines(lstpt):
outlines = []
for idx, pt in enumerate(lstpt):
if idx > 0:
line = Line.ByStartPointEndPoint(lstpt[idx - 1], pt)
outlines.append(line)
return outlines
lstpoints = IN[0]
start = time.time()
if IN[1]:
threadResult = lstpoints.AsParallel().Select(lambda sublst: drawlines(sublst)).ToList()
out = threadResult
else:
out = []
for sublst in lstpoints:
out.append(drawlines(sublst))
# Calculate Run-time
time = ("%s s" % (time.time()-start))
OUT = time, outMy PC has i7-4790K,
the time with Parallel operations is 0.23 s,
the time without parallel operations is 0.36 s,
the gain (time consuming) is about (0.36-0.24) / 0.36 = 33.3%, while in your test the gain is about (0.5-0.24) / 0.5 = 50%
Better to time this with TuneUp as it will incorporate the other aspects of the Dynamo environment (ie: preview and data marshaling) if I’m not mistaken.
Also a note to anyone who visits later: This will likely not work with Generative Design (requires 6+ cores for running each of the Dynamo Core engines concurrently) or Revit interactions (limited to one thread as a DB). Worth testing anyway.
Hi @c.poupin!
Parallel LINQ is great, Dynamo refreshes several times faster on large dataset!
Trying to test PLINQ with Geometry.IntersectAll:
Almost working, no crashes. But i have problems with list recreate.
Could you help with my code, please?
Thanks in advance!
Linq2.dyn (78.1 KB)
Hello @Vladimir
it’s necessary to convert the result to List (to avoid overwriting the previous return variable)
import clr
import time
clr.AddReference('ProtoGeometry')
from Autodesk.DesignScript.Geometry import *
from System.Threading.Tasks import *
clr.AddReference("System.Core")
import System
clr.ImportExtensions(System.Linq)
def intrs(elA, elB):
return elA.IntersectAll(elB)
lstpoints = IN[0]
o = IN[1]
start = time.time()
res =[]
tst = 0
#oo = o
#ls = lstpoints
for ls,oo in zip(lstpoints,o):
threadResult = ls.AsParallel().Select(lambda l: (intrs(l,oo))).ToList()
res.append(threadResult)
tst=tst+1
time = ("%s s" % (time.time()-start))
OUT = time, res
Hello @c.poupin !
All works, but i feel, that for ls,oo in zip(lstpoints,o): in my code is bad for balancing quenue.
The idea is to send independant chunks (like [a[0], b[0]],[a[1], b[1]] ) to different threads.
Example:
The “a” chunks contains Rooms solids. The “b” chunks contains some walls solids.
Can i ask you, how to push two arguments into PLINQ?
Found .Zip here:
var a = new int[] { 1, 2 };
var b = new int[] { 3, 4 };
foreach (var n in a.Zip(b, (x, y) => x + y))
{
// enumerates 4 (1 + 3), 6 (2 + 4)
}
And the question is how to apply it in PLINQ in Python. (Without Python loops)
Attached sample dyn.
Thank you!
Linq3.dyn (41.1 KB)
UPD: find some similar discussion: .net - How is the PLINQ AsParallel function passing data to a function when called within the same scope - Stack Overflow
Hi,
you can unpack the arguments in the function
a simple example with AsParallel() and Zip()
for simple methods, Zip() is just as efficient
Hi @c.poupin!
The last problem is some nulls data after PLINQ.
I can easily fix this with recursive loops by processing my list until nulls are gone.
But think the problem is Safe Threading:
https://www.phidiax.com/blog/post/parallel-linq-plinq-thread-safety-using-asparallel-list-t-add-returns-object-reference-not-set-to-an-instance-of-an-object
As i have to use .AsOrdered() to keep my list, i think that ConcurrentQueue will be better than ConcurrentBag:
Trying to enqueue IN[0] with ConcurrentQueue.Enqueue(T) method, but lost in syntax.
Can you help, please?
Concurrent_.dyn (10.9 KB)
UPD: found out, that we can’t use ConcurrentQueue with PLINQ multithreading. The question is how we can make thread safe PLINQ to protect data and avoid data corruption. Thank you!
I don’t know all the PLINQ methods, but out of my curiosity, I compared some methods
here the results
for AsParallel() with ConcurrentQueue I used this code
import sys
import clr
import System
clr.AddReference('ProtoGeometry')
from Autodesk.DesignScript.Geometry import *
import Autodesk.DesignScript.Geometry as DS
from System.Collections.Concurrent import ConcurrentQueue
from System.Collections.Generic import List
clr.AddReference("System.Core")
clr.ImportExtensions(System.Linq)
def test(*args):
lstPoints = [p.Centroid() for p in args]
points = Point.PruneDuplicates(lstPoints)
return points
ziplst = zip(*IN)
out = []
lstRtn = ConcurrentQueue[System.Array[DS.Point]]()
#
ziplst.AsParallel()\
.WithDegreeOfParallelism(System.Environment.ProcessorCount)\
.WithExecutionMode(System.Linq.ParallelExecutionMode.ForceParallelism)\
.ForAll(lambda e : lstRtn.Enqueue(test(*e)))
OUT = lstRtn
Test AsParallel with Concurrent.dyn (27.8 KB)
Hello @c.poupin,
Noticed errors if try to switch to CPython3 in your previous post:
AttributeError : ‘zip’ object has no attribute ‘AsParallel’.
Do you think its possible to use PLINQ in CPython?
CPython use a PythonNet bridge, it does not support PLINQ extension, and CPython engine is executed from an imported (non-main) module.
Try to use multithreading with CPython3 engine