Python multiprocessing pool map and parallel loops: issues with Protogeometry functions

Hello,

I would like to speed-up a Python function that I have created to analyse how much sky is visible from building surfaces.
I tried to use both parallel loops and multiprocessing pool map. They work well when the function is simple, but it stops working when I add some particular Protogeometry methods like Line.ByStartPointEndPoint or Surface.ByRevolve.

This is the error I get with multiprocessing:

Warning: IronPythonEvaluator.EvaluateIronPythonScript operation failed.
Traceback (most recent call last):
File “”, line 129, in
File “C:\Program Files\Autodesk\Revit 2021\AddIns\DynamoForRevit\IronPython.StdLib.2.7.9\multiprocessing\pool.py”, line 253, in map
File “C:\Program Files\Autodesk\Revit 2021\AddIns\DynamoForRevit\IronPython.StdLib.2.7.9\multiprocessing\pool.py”, line 572, in get
TypeError: Exception of type ‘System.ArgumentNullException’ was thrown.
Parameter name: host

And this is the error I get with parallel loops:

Warning: IronPythonEvaluator.EvaluateIronPythonScript operation failed.
Traceback (most recent call last):
File “”, line 121, in <lambda$16095>
File “”, line 97, in SkyAccessAnalysis
File “”, line 121, in
Exception: One or more errors occurred.

If anyone expert of these kind of python processes can help, it would be much appreciated.
I have uploaded my Dynamo graph that can be run with no edit.

SkyAccessAnalysis.dyn (390.0 KB)

You will see that, unchecking both “Parallel” and “Multiprocessing”, the analysis works.

Thank you
Cristiano

try disposing your geometry on the same threads that created it.

Hi Michael,

Thanks for answering.
As per your suggestion, inside the function I disposed all geometries created, but I still get the same errors.

You can find the Python definition below. Am I doing it wrong?

def SkyAccessAnalysis(point, surface, solids, angleRange= 180, angleStep= 2, maxDist= 300000):
# Make vectors by surface and point
n = surface.NormalAtPoint(point)
u = CrossProduct(n, Vector.ByCoordinates(0,0,1))
v = CrossProduct(n, u)	
# Project normal to horizontal plane
nHor = Vector.ByCoordinates(n.X, n.Y, 0)
nHor= nHor.Scale(1/nHor.Length)
# Angle between surface and horizon
vertAngle = int(nHor.AngleWithVector(n) + 90)
# Make angle ranges to check
horRange = range(-angleRange/2, angleRange/2, step = angleStep)
vertRange = range(0, vertAngle, step = angleStep)
# Move slightly the point to avoid clash with itself
point = point.Translate(direction = nHor, distance = 10)
# Calculate max sky access score
skyValue = len(horRange) * len(vertRange)
# Make horizontal slice surface for following tests
bRevolveLine = Line.ByStartPointDirectionLength(point, nHor.Rotate(zVect, -angleRange/2), maxDist)
bSurfTest = Surface.ByRevolve(bRevolveLine, point, zVect, 0, angleRange) 

# Make a sphere wedge and check if it does intersect surrounding buildings.
# If the solid passes the test, the analysis is concluded with maximum score.
solidTest = Solid.ByRevolve(PolyCurve.ByJoinedCurves(bSurfTest.PerimeterCurves()), point, u, 0, vertAngle)
if not Geometry.DoesIntersect(solidTest, solids):
	solidTest.Dispose()
	bSurfTest.Dispose()
	bRevolveLine.Dispose()
	return skyValue
	
else:
	# make vertical slice for following tests
	aRevolveLine = Line.ByStartPointDirectionLength(point, nHor, maxDist)
	aSurfTest = Surface.ByRevolve(aRevolveLine, point, u, 0, vertAngle)		
	# Alpha angles test. If the slice passes the test, remove the angle from following analysis.
	for alpha in horRange:
		current_aSurfTest = aSurfTest.Rotate(origin = point, axis = zVect, degrees = alpha)
		if not Geometry.DoesIntersect(current_aSurfTest, solids):
			horRange.Remove(alpha)		
		current_aSurfTest.Dispose()
	# Beta angles test. If the slice passes the test, remove the angle from following analysis.
	for beta in vertRange:	
		current_bSurfTest = bSurfTest.Rotate(origin = point, axis = u, degrees = beta)
		if not Geometry.DoesIntersect(current_bSurfTest, solids):
			vertRange.Remove(beta)
		current_bSurfTest.Dispose()
	# Final test: check clashes between rays and surrounding builidings.
	# Every clash subtracts one point to the final score.
	for alpha in horRange:
		aVector = nHor.Rotate(axis = u, degrees =alpha)		
		for beta in vertRange:
			bVector = aVector.Rotate(axis = u, degrees =beta)			
			currentRay = Line.ByStartPointDirectionLength(point, bVector, maxDist)
			if currentRay.DoesIntersect(solids):
				skyValue -= 1	
			currentRay.Dispose()					
	
	solidTest.Dispose()
	aRevolveLine.Dispose()
	aSurfTest.Dispose()
	bSurfTest.Dispose()
	bRevolveLine.Dispose()				
	
	return skyValue

Thank you,
Cristiano

Does this works with normal threading?? i.e. the default thread that dynamo uses. If it does, it means that, due to the parallel work, some of the processes completed faster than the other, and its variables are dependent, hence it will be a null value for those dependant functions which will cause that error.

My take on multi-thread processing is to ensure that no functions are dependant and all objects within it are safe-object. Then you can use multi-threading

2 Likes

Hi @stillgotme @Michael_Kirschner2

Yes, it does work with normal threading. Unchecking both methods the function runs correctly.

Furthermore, there is no dependant variable. Each run can be done by itself, no matter the order.
I’m still not getting what’s wrong. Sometimes the multiprocessing works, but most of the times it doesn’t.
Looks like something improved after disposing geometries, because my ram use remains low.

Here you can find the updated graph, with the method dispose() inside the function.
SkyAccessAnalysis.dyn (390.5 KB)

Hello
you can try a variant using Parallel LINQ (System.Linq can be use in IronPython)
I had to relaunch Revit 2021 during the tests

# ACPV Python v.1.0
# Coded by: 

### IMPORTS ###
# Clr and Sys
import clr
import sys
sys.path.append('C:\Program Files (x86)\IronPython 2.7\Lib')
# Import Protogeometry
clr.AddReference('ProtoGeometry')
from Autodesk.DesignScript.Geometry import *
# Import Dynamo Core Nodes
clr.AddReference('DSCoreNodes')
import DSCore
# Import Time
import time
# Import System
import System
from System import Array
from System.Collections.Generic import *
# Import Parallel
from System.Threading.Tasks import *
clr.AddReference("System.Core")
import System
clr.ImportExtensions(System.Linq)


### DEFINITIONS ###
zVect = Vector.ByCoordinates(0,0,1)

def Flatten(x):
    result = []
    for el in x:
        if hasattr(el, "__iter__") and not isinstance(el, basestring):
            result.extend(Flatten(el))
        else:
            result.append(el)
    return result
    
def CrossProduct(vectA, vectB):
    a=[vectA.X , vectA.Y , vectA.Z]
    b=[vectB.X , vectB.Y , vectB.Z]
    c = [a[1]*b[2] - a[2]*b[1],
         a[2]*b[0] - a[0]*b[2],
         a[0]*b[1] - a[1]*b[0]]
    return Vector.ByCoordinates(c[0],c[1],c[2])


def SkyAccessAnalysis(point, surface, solids, angleRange= 180, angleStep= 2, maxDist= 300000):
	# Make vectors by surface and point
	n = surface.NormalAtPoint(point)
	u = CrossProduct(n, Vector.ByCoordinates(0,0,1))
	v = CrossProduct(n, u)	
	# Project normal to horizontal plane
	nHor = Vector.ByCoordinates(n.X, n.Y, 0)
	nHor= nHor.Scale(1/nHor.Length)
	# Angle between surface and horizon
	vertAngle = int(nHor.AngleWithVector(n) + 90)
	# Make angle ranges to check
	horRange = range(-angleRange/2, angleRange/2, step = angleStep)
	vertRange = range(0, vertAngle, step = angleStep)
	# Move slightly the point to avoid clash with itself
	point = point.Translate(direction = nHor, distance = 10)
	# Calculate max sky access score
	skyValue = len(horRange) * len(vertRange)
	# Make horizontal slice surface for following tests
	bRevolveLine = Line.ByStartPointDirectionLength(point, nHor.Rotate(zVect, -angleRange/2), maxDist)
	bSurfTest = Surface.ByRevolve(bRevolveLine, point, zVect, 0, angleRange) 

	# Make a sphere wedge and check if it does intersect surrounding buildings.
	# If the solid passes the test, the analysis is concluded with maximum score.
	solidTest = Solid.ByRevolve(PolyCurve.ByJoinedCurves(bSurfTest.PerimeterCurves()), point, u, 0, vertAngle)
	if not Geometry.DoesIntersect(solidTest, solids):
		solidTest.Dispose()
		bSurfTest.Dispose()
		bRevolveLine.Dispose()
		return skyValue
		
	else:
		# make vertical slice for following tests
		aRevolveLine = Line.ByStartPointDirectionLength(point, nHor, maxDist)
		aSurfTest = Surface.ByRevolve(aRevolveLine, point, u, 0, vertAngle)		
		# Alpha angles test. If the slice passes the test, remove the angle from following analysis.
		for alpha in horRange:
			current_aSurfTest = aSurfTest.Rotate(origin = point, axis = zVect, degrees = alpha)
			if not Geometry.DoesIntersect(current_aSurfTest, solids):
				horRange.Remove(alpha)		
			current_aSurfTest.Dispose()
		# Beta angles test. If the slice passes the test, remove the angle from following analysis.
		for beta in vertRange:	
			current_bSurfTest = bSurfTest.Rotate(origin = point, axis = u, degrees = beta)
			if not Geometry.DoesIntersect(current_bSurfTest, solids):
				vertRange.Remove(beta)
			current_bSurfTest.Dispose()
		# Final test: check clashes between rays and surrounding builidings.
		# Every clash subtracts one point to the final score.
		for alpha in horRange:
			aVector = nHor.Rotate(axis = u, degrees =alpha)		
			for beta in vertRange:
				bVector = aVector.Rotate(axis = u, degrees =beta)			
				currentRay = Line.ByStartPointDirectionLength(point, bVector, maxDist)
				if currentRay.DoesIntersect(solids):
					skyValue -= 1	
				currentRay.Dispose()					
		
		solidTest.Dispose()
		aRevolveLine.Dispose()
		aSurfTest.Dispose()
		bSurfTest.Dispose()
		bRevolveLine.Dispose()				
		
		return skyValue

### INPUTS ###
parallel = IN[0]
multiproc = IN[1]
points = [Flatten(lst) for lst in IN[2]]
surfaces = IN[3]
aStep = IN[4]
aRange = IN[5]
buildings =Solid.ByUnion(IN[6])

### CODE ###

# Start Run-time
start = time.time()

values = []
if parallel:
	values =[]
	for surf,pts in zip(surfaces,points):	
		threadResult = pts.AsParallel().Select(lambda pt: SkyAccessAnalysis(pt, surface = surf, solids = buildings, angleRange = aRange, angleStep = aStep)).ToList()
		values.append(threadResult)

else:
	for surf,pts in zip(surfaces,points):
		values.append([SkyAccessAnalysis(pt, surf, buildings, angleRange = aRange, angleStep = aStep) for pt in pts])


# Calculate Run-time
time = ("%s s" % (time.time()-start))

### OUTPUT ###
OUT = values, time
1 Like

looks like you don’t dispose these geometries:
PolyCurve.ByJoinedCurves(bSurfTest.PerimeterCurves())

Hi @Michael_Kirschner2,

Thank you!

After disposing those curves the script improved its performances: now it works more often than before.
However, most of the times it gets the same error. I am noticing that it depends on the amount of geometries to be calculated. The more points I pass through the function, the less is the chance of success. Furthermore, I noticed that, reducing the amount of phisical cores in the pool, it tends to be more successful.

Anyway, I’ll try splitting the function in subfunctions and doing separate poolmaps. I will let you know if that works.
Any other suggestion will be appreciated.

def SkyAccessAnalysis(point, surface, solids, angleRange= 180, angleStep= 2, maxDist= 300000):
	# Make vectors by surface and point
	n = surface.NormalAtPoint(point)
	z = Vector.ByCoordinates(0,0,1)
	u = CrossProduct(n, z)
	v = CrossProduct(n, u)	
	# Project normal to horizontal plane
	nHor = Vector.ByCoordinates(n.X, n.Y, 0)
	nHor= nHor.Scale(1/nHor.Length)
	# Angle between surface and horizon
	vertAngle = int(nHor.AngleWithVector(n) + 90)
	# Make angle ranges to check
	horRange = range(-angleRange/2, angleRange/2, step = angleStep)
	vertRange = range(0, vertAngle, step = angleStep)
	# Move slightly the point to avoid clash with itself
	newPoint = point.Translate(direction = nHor, distance = 10)
	# Calculate max sky access score
	skyValue = len(horRange) * len(vertRange)
	# Make horizontal slice surface for following tests
	nHorRotated = nHor.Rotate(zVect, -angleRange/2)
	bRevolveLine = Line.ByStartPointDirectionLength(newPoint, nHorRotated, maxDist)
	bSurfTest = Surface.ByRevolve(bRevolveLine, newPoint, zVect, 0, angleRange) 
​
	# Make a sphere wedge and check if it does intersect surrounding buildings.
	# If the solid passes the test, the analysis is concluded with maximum score.
	perimeterCurves = bSurfTest.PerimeterCurves()
	joinedCurves = PolyCurve.ByJoinedCurves(perimeterCurves)
	solidTest = Solid.ByRevolve(joinedCurves, newPoint, u, 0, vertAngle)
	if not Geometry.DoesIntersect(solidTest, solids):
		solidTest.Dispose()
		bSurfTest.Dispose()
		bRevolveLine.Dispose()
		z.Dispose()
		n.Dispose()
		u.Dispose()
		v.Dispose()
		nHor.Dispose()
		nHorRotated.Dispose()
		for ele in perimeterCurves:
			ele.Dispose()
		newPoint.Dispose()
		joinedCurves.Dispose()
		return skyValue
		
	else:
		# make vertical slice for following tests
		aRevolveLine = Line.ByStartPointDirectionLength(newPoint, nHor, maxDist)
		aSurfTest = Surface.ByRevolve(aRevolveLine, newPoint, u, 0, vertAngle)		
		# Alpha angles test. If the slice passes the test, remove the angle from following analysis.
		for alpha in horRange:
			current_aSurfTest = aSurfTest.Rotate(origin = newPoint, axis = zVect, degrees = alpha)
			if not Geometry.DoesIntersect(current_aSurfTest, solids):
				horRange.Remove(alpha)		
			current_aSurfTest.Dispose()
		# Beta angles test. If the slice passes the test, remove the angle from following analysis.
		for beta in vertRange:	
			current_bSurfTest = bSurfTest.Rotate(origin = newPoint, axis = u, degrees = beta)
			if not Geometry.DoesIntersect(current_bSurfTest, solids):
				vertRange.Remove(beta)
			current_bSurfTest.Dispose()
		# Final test: check clashes between rays and surrounding builidings.
		# Every clash subtracts one point to the final score.
		for alpha in horRange:
			aVector = nHor.Rotate(axis = u, degrees =alpha)		
			for beta in vertRange:
				bVector = aVector.Rotate(axis = u, degrees =beta)			
				currentRay = Line.ByStartPointDirectionLength(newPoint, bVector, maxDist)
				if currentRay.DoesIntersect(solids):
					skyValue -= 1	
				currentRay.Dispose()
				bVector.Dispose()
			aVector.Dispose()				
		
		solidTest.Dispose()
		aRevolveLine.Dispose()
		aSurfTest.Dispose()
		bSurfTest.Dispose()
		bRevolveLine.Dispose()				
		z.Dispose()
		n.Dispose()
		u.Dispose()
		v.Dispose()
		nHor.Dispose()
		nHorRotated
		for ele in perimeterCurves:
			ele.Dispose()
		newPoint.Dispose()
		joinedCurves.Dispose()

Hello @c.poupin,

Thank you! I just tested the script with your code.
Most of the times I still get errors, especially when I decrease the angle step or increase the angle range, because it has to create more geometry.

With the same slider settings as your screenshot I get an error. I guess my computer is less powerful than yours. Would you please send me your pc specifics so I can get an idea.

here are the characteristics except gpu

image

1 Like

Thanks @c.poupin! Your pc has 2 cores more than mine. I think that’s the reason.