Committing Transaction to create a large number of Revit Elements

Revit
1

Hi All,

I am just wondering if there is a Transaction limit when it comes to the number of Revit elements that can be created via the API when Dynamo is being used as the communicator.

I am using Dynamo Python to create,

collect = Autodesk.Revit.Creation.FamilyInstanceCreationData

and use it with the,

items = doc.Create.NewFamilyInstances2(collect)

with the purpose of placing approximately 10,000 Revit families (the family size is 1mb)

Code below:

## Import Reference Examples

import clr
clr.AddReference('RevitAPI')
import Autodesk
from Autodesk.Revit.DB import *
from Autodesk.Revit.DB.Structure import *

clr.AddReference('RevitServices')
import RevitServices
from RevitServices.Persistence import DocumentManager
from RevitServices.Transactions import TransactionManager

clr.AddReference('RevitAPIUI')
from  Autodesk.Revit.UI import *

clr.AddReference("RevitNodes")
import Revit
clr.ImportExtensions(Revit.GeometryConversion)

clr.AddReference('ProtoGeometry')
from Autodesk.DesignScript.Geometry import *


clr.AddReference('System')
from System.Collections.Generic import List as sList

clr.AddReference('DSCoreNodes') 
import DSCore
from DSCore import *


def flat(lst):
	return DSCore.List.Flatten(lst,2)
	
## Split list into chuncks
def chunks(data,sizes):
	it = iter(data)
	return [[next(it) for _ in range(size)] for size in sizes]

doc = DocumentManager.Instance.CurrentDBDocument
uiapp = DocumentManager.Instance.CurrentUIApplication 
app = uiapp.Application 
uidoc = uiapp.ActiveUIDocument

location_list = flat(IN[0])
family_list = flat(IN[1])
rotation_list = flat(IN[2])
lev_list = flat(IN[3])

result = []

collect = sList[Autodesk.Revit.Creation.FamilyInstanceCreationData]()
	
for location, family, rotation, lev in  zip(location_list,family_list,rotation_list,lev_list):

	location1 = location.ToXyz()
	family1 = UnwrapElement(family)
	rotation1 = Math.DegreesToRadians(rotation)
	lev1 = UnwrapElement(lev)
	
	create = Autodesk.Revit.Creation.FamilyInstanceCreationData(location1,family1,lev1,StructuralType.NonStructural)
	create.RotateAngle = rotation1
	create.Axis = Line.ByStartPointDirectionLength(location,Vector.ZAxis(),1).ToRevitType()
	
	collect.Add(create)

# Start
TransactionManager.Instance.EnsureInTransaction(doc)

items = doc.Create.NewFamilyInstances2(collect)
	
# Stop
TransactionManager.Instance.TransactionTaskDone()

elems = []

for i in items:
	elems.append(doc.GetElement(i))

counts = IN[4]

collect.Clear()

OUT = chunks(elems,counts)

I know this code works to place 100 Revit Families, but my current test is taking a very long time to do 10,000 ish.

Revit has been displaying the ‘Committing Transaction’ message in the bottom task bar for 10 hours now. The process of creating 100 Revit Families takes approximately 6 minutes, so I am wondering if there is a limit to the number of Revit Elements that can be rolled into being created in 1 Transaction within the API when using Dynamo.

Any thoughts or advice?

I guess I could split the creation data into chunks of 100 and commit for each chunk…

2

hi,
to have encountered the case (pull up a lot of floor points on a topo), I had to divide the work into several transactions, I think some operations consume much more memory than others)

3

I have just re-executed the code without the Transaction, to assess exactly how many Revit Families were going to be placed. 28,215, so a little over the 10,000 I thought :sweat_smile:

I split out a 100 sample data set and got the following:

100 Simple data container families (3d boxes about 200mm sq with some parameters, not set)

  • 3 min 30 seconds using the NewFamilyInstances2 method outside of the loop with a list of combined CreationData.

  • 2 min 55 seconds using the NewFamilyInstances2 method outside of the loop with individual CreationData.

I have remembered why I needed to use this method (instead of NewFamilyInstance) and it was because of the ability to set the rotation and axis as part of the creation process, and not have to adjust it after, which I think I did test and was very time intensive.

Will look to work on my recursive Transactions tmw, thanks @c.poupin

1 Like
4

@Ewan_Opie, look into FamilyInstance.ByPointsLevelsBatch code from Springs. I’ve successfully created 20000 family instaces with it.

1 Like
5

@Vladimir Thanks, the Springs node is using the same doc.Create.NewFamilyInstances2() method with creation data as my part of the code, but I’m adding in rotations. Perhaps I should interegate my Family a bit more to optimise its size/complexity, though its really not that complex.

Springs node code excerpt below:

st1 = StructuralType.NonStructural
FICD1 = List[FamilyInstanceCreationData]()
for i, p in enumerate(pts):
	t1 = types[0] if types_len else types[i]
	lvl = lvls[0] if lvl_len else lvls[i]
	FICD1.Add(FamilyInstanceCreationData(p.ToXyz(True), t1, lvl, st1) )

TransactionManager.Instance.EnsureInTransaction(doc)
new_inst = doc.Create.NewFamilyInstances2(FICD1)
TransactionManager.Instance.TransactionTaskDone()

OUT = [doc.GetElement(i).ToDSType(trackElems) for i in new_inst]
1 Like
6

How long did 20000 take? @Vladimir

7

About 15 minutes, i’ve rotated families later, so it was faster than create+rotate in one method, as i remember.

1 Like
8

I tried it that way before, but will see if it makes a difference. :+1:

9

Too long for just 100 pcs. Something wrong with family, maybe? Just try to place simple box, test, than add parameters step by step.

1 Like
10

Whoops, my bad. @Vladimir

3 min 30 was the whole supporting codebase, as there are a few pieces of logic to go through before the creation data is fed to the node.

I will go check TuneUp for the creation node run time shortly.

1 Like
11

I’ve managed 2x that before; the number of transactions isn’t a limiting factor, but other calculation issues (ie: if the family you are placing is room bounding then the rooms have to be recalculated as the item(s) are placed).

That said 100 families taking 6 minutes is VERY slow relative to the expected outcome - seeing your creation method you can likely utilize the OOTB “ByCoordinateSystem”. Is that any faster?

If so then I’d question the code you’re using - perhaps changing out the API calls for one of the other methods given as a start. I know rotation after creation can feel slow, but the speed and effectiveness of the by coordinate system tooling is achieved by ‘create > rotate’ directly. This works fairly well all things considered as it doesn’t have to duplicate marshaling efforts (Dynamo > Revit to make the families, Revit to Dynamo to return the results > Dynamo to Revit to set the rotation > Revit to Dynamo to return the results) which slows things down and adds an additional hit on the memory. My gut says that this will help with the speed here as you’re going to be memory constrained.

If the lead up to the family instance creation is taking a bit, consider writing out everything you need to an intermediate file - say a JSON to store the coordinate systems and parameter values; or a CSV and store the 12 numbers defining the coordinate system and the parameter values. When you’ve collected all the data run a second graph to create all the instances. Should save a bit on memory consumption reducing the page fault count, while simultaneously allowing for batching of groups of N if you decide to go that route.

4 Likes
12

I handle this in pyRevit using data chunking (not sure if correct term). Effectively I break down lists that could be very big into a number I’m confident I can handle, then I catch each chunk in a transaction. It’ll still take a while to finish, but I’ve found it’s a lot more stable for Revit to deal with. In pyRevit it also means I can use a loading bar to cancel it as it works through the chunks vs hoping it gets to the end.

# Break your list into pieces you can handle
def chunks(lst, n):
    for i in xrange(0, len(lst), n):
        yield lst[i:i + n]

# Your inputs
dataIn    = IN[0]
limitSize = IN[1]

# Your output collector
dataOut = []

# Chunk our data set
chunkData = chunks(IN[0],IN[1])

# For each chunk, do a thing
for chunk in chunkData:
	# Put a start transaction here
	# Your function here
	for d in chunk:
		dataOut.append(d*d)
	# Put a end/commit transaction here
	
OUT = dataOut
4 Likes
13

I gave the looping of transactions a try on Friday afternoon and got chunks of 10 to process for a test sample, but an error that was being ignored when the transaction was ‘en-mass’ outside the loop has popped up.

Depending on where I place the transaction, the ’ Duplicate Elements’ error gets thrown.

Now, I shouldn’t have duplicate elements in my final graph but the logic for creation data will need to be adjusted for this to happen. Might try that first, before trying to adjust the BuiltInFailures options to handle the error.

Thanks for the help all, bit of a long one but posting some of my learnings might help others :sweat_smile:

2 Likes
14

@jacob.small Storing the data as a separate file is something that I have been considering for additional change management purposes, as the logic to create my CreationData is extracted from other linked models, prepared by multiple sources, which get updated sporadically.

I’m thinking JSON as it’s easy enough to both pull and push between platforms / environments if I need to.

Also I am suspicious now that the lag I have experienced is a combined result of the creation and regeneration process, so many things to explore…

1 Like
15

I performed a quick test this morning to check how the family I am using stacks up against a generic family that only contains a 200mm Cube. I removed my pre-creation code (logic to work out point locations) and used generic points for creation.

Timings as below:

My Family:
Placed 1,000 instances in 30 seconds
Placed 5,000 instances in 13 minutes

200mm Cube:
Placed 1,000 instances in 3 seconds
Placed 5,000 instances in 5 seconds
Placed 10,000 instances in 12 seconds
Placed 50,000 instances in 9 minutes

Definately going to do an optimisation of Family parameters, geometry and formulas. :crazy_face:

Still working on iteration of Transactions, more soon.

1 Like
16

I think my family creation time using the FamilyInstances2 method will work overall, I am confident that batch execution using it will work. Just having a head scratcher with the below. Ideally it would run a Transaction for each chunk ignoring any warnings thrown.

Bit of a long one… but here goes.

So Code 1 will process everything as chunks but commit 1 transaction (I’m looking for a transaction per chunk) with no warnings, creates elements as expected. Some DuplicateInstances warnings are created for multiple elements, which is ok as I’m looking on how to ignore this programmatically.

Code 1

## Import Reference Examples

import clr
clr.AddReference('RevitAPI')
import Autodesk
from Autodesk.Revit.DB import *
from Autodesk.Revit.DB.Structure import *

clr.AddReference('RevitServices')
import RevitServices
from RevitServices.Persistence import DocumentManager
from RevitServices.Transactions import TransactionManager

clr.AddReference('RevitAPIUI')
from  Autodesk.Revit.UI import *

clr.AddReference("RevitNodes")
import Revit
clr.ImportExtensions(Revit.GeometryConversion)

clr.AddReference('ProtoGeometry')
from Autodesk.DesignScript.Geometry import *


clr.AddReference('System')
from System.Collections.Generic import List as sList

clr.AddReference('DSCoreNodes') 
import DSCore
from DSCore import *


def flat(lst):
	return DSCore.List.Flatten(lst,2)

doc = DocumentManager.Instance.CurrentDBDocument
uiapp = DocumentManager.Instance.CurrentUIApplication 
app = uiapp.Application 
uidoc = uiapp.ActiveUIDocument

location_list = flat(IN[0])
family_list = flat(IN[1])
rotation_list = flat(IN[2])
lev_list = flat(IN[3])

counts = IN[4]

## Split list into chuncks
def chunks(data,sizes):
	it = iter(data)
	return [[next(it) for _ in range(size)] for size in sizes]

chopLen = IN[5]

# Yield successive n-sized chunks from L.
def chop(L, n):
	for i in range(0, len(L), n):
		yield L[i:i+n]
        
# # Chop Inputs
chopLocation = chop(location_list,chopLen)
chopFamily = chop(family_list,chopLen)
chopRotation = chop(rotation_list,chopLen)
chopLevel = chop(lev_list,chopLen)

#OUT = chopLocation

class DuplicateInstancesWarningSwallower(IFailuresPreprocessor):
    def PreprocessFailures(self, failuresAccessor):
        fail_list = List[FailureMessageAccessor]()
        fail_acc_list = failuresAccessor.GetFailureMessages().GetEnumerator()
           
        for failure in fail_acc_list:
            failuresAccessor.ResolveFailure(failure)
            failure_id = failure.GetFailureDefinitionId()
            failure_severity = failure.GetSeverity()
            failure_type = BuiltInFailures.InaccurateFailures.DuplicateInstances
            if failure_id == failure_type:
                
                failuresAccessor.DeleteWarning(failure)

        return FailureProcessingResult.Continue

output = []
result = []
testing = []


collect = sList[Autodesk.Revit.Creation.FamilyInstanceCreationData]()

for cLocation, cFamily, cRotation, cLev in  zip(chopLocation,chopFamily,chopRotation,chopLevel):
	
	collect.Clear()

	TransactionManager.Instance.EnsureInTransaction(doc)
	
	for location, family, rotation, lev in  zip(cLocation, cFamily, cRotation, cLev):		
	
		location1 = location.ToXyz()
		family1 = UnwrapElement(family)
		rotation1 = Math.DegreesToRadians(rotation)
		lev1 = UnwrapElement(lev)
	
		create = Autodesk.Revit.Creation.FamilyInstanceCreationData(location1,family1,lev1,StructuralType.NonStructural)
		create.RotateAngle = rotation1
		create.Axis = Line.ByStartPointDirectionLength(location,Vector.ZAxis(),1).ToRevitType()
		
		collect.Add(create)
		
	items = doc.Create.NewFamilyInstances2(collect)	
	
	pack = []
	for i in items:
		pack.append(doc.GetElement(i))
	result.append(pack)

	TransactionManager.Instance.TransactionTaskDone()


OUT = result

Code 2 will process chunks as separate Transactions but throws the DuplicateInstances warnings when run, so not ideal for a ‘hands-free’ user execution.

Code 2

## Import Reference Examples

import clr
clr.AddReference('RevitAPI')
import Autodesk
from Autodesk.Revit.DB import *
from Autodesk.Revit.DB.Structure import *

clr.AddReference('RevitServices')
import RevitServices
from RevitServices.Persistence import DocumentManager
from RevitServices.Transactions import TransactionManager

clr.AddReference('RevitAPIUI')
from  Autodesk.Revit.UI import *

clr.AddReference("RevitNodes")
import Revit
clr.ImportExtensions(Revit.GeometryConversion)

clr.AddReference('ProtoGeometry')
from Autodesk.DesignScript.Geometry import *


clr.AddReference('System')
from System.Collections.Generic import List as sList

clr.AddReference('DSCoreNodes') 
import DSCore
from DSCore import *


def flat(lst):
	return DSCore.List.Flatten(lst,2)

doc = DocumentManager.Instance.CurrentDBDocument
uiapp = DocumentManager.Instance.CurrentUIApplication 
app = uiapp.Application 
uidoc = uiapp.ActiveUIDocument

location_list = flat(IN[0])
family_list = flat(IN[1])
rotation_list = flat(IN[2])
lev_list = flat(IN[3])

counts = IN[4]

## Split list into chuncks
def chunks(data,sizes):
	it = iter(data)
	return [[next(it) for _ in range(size)] for size in sizes]

chopLen = IN[5]

# Yield successive n-sized chunks from L.
def chop(L, n):
	for i in range(0, len(L), n):
		yield L[i:i+n]
        
# # Chop Inputs
chopLocation = chop(location_list,chopLen)
chopFamily = chop(family_list,chopLen)
chopRotation = chop(rotation_list,chopLen)
chopLevel = chop(lev_list,chopLen)

#OUT = chopLocation

class DuplicateInstancesWarningSwallower(IFailuresPreprocessor):
    def PreprocessFailures(self, failuresAccessor):
        fail_list = List[FailureMessageAccessor]()
        fail_acc_list = failuresAccessor.GetFailureMessages().GetEnumerator()
           
        for failure in fail_acc_list:
            failuresAccessor.ResolveFailure(failure)
            failure_id = failure.GetFailureDefinitionId()
            failure_severity = failure.GetSeverity()
            failure_type = BuiltInFailures.InaccurateFailures.DuplicateInstances
            if failure_id == failure_type:
                
                failuresAccessor.DeleteWarning(failure)

        return FailureProcessingResult.Continue

output = []
result = []
testing = []


collect = sList[Autodesk.Revit.Creation.FamilyInstanceCreationData]()

for cLocation, cFamily, cRotation, cLev in  zip(chopLocation,chopFamily,chopRotation,chopLevel):
	
	collect.Clear()
	
	TransactionManager.Instance.ForceCloseTransaction()
	
	
	t = Transaction(doc, "Batch Create Families")
	
	# Start
	t.Start()
	
	#tOpt = t.GetFailureHandlingOptions()
	#tOpt.SetFailuresPreprocessor(DuplicateInstancesWarningSwallower())
	#t.SetFailureHandlingOptions(tOpt)
	
	for location, family, rotation, lev in  zip(cLocation, cFamily, cRotation, cLev):		
	
		location1 = location.ToXyz()
		family1 = UnwrapElement(family)
		rotation1 = Math.DegreesToRadians(rotation)
		lev1 = UnwrapElement(lev)
	
		create = Autodesk.Revit.Creation.FamilyInstanceCreationData(location1,family1,lev1,StructuralType.NonStructural)
		create.RotateAngle = rotation1
		create.Axis = Line.ByStartPointDirectionLength(location,Vector.ZAxis(),1).ToRevitType()
		
					
		collect.Add(create)
		
	items = doc.Create.NewFamilyInstances2(collect)	
	
	
	pack = []
	for i in items:
		pack.append(doc.GetElement(i))
	result.append(pack)

	t.Commit()

OUT = result

And Code 3 enables FailurePreProcessor methods, shows no warning but creates nothing…

Code 3

## Import Reference Examples

import clr
clr.AddReference('RevitAPI')
import Autodesk
from Autodesk.Revit.DB import *
from Autodesk.Revit.DB.Structure import *

clr.AddReference('RevitServices')
import RevitServices
from RevitServices.Persistence import DocumentManager
from RevitServices.Transactions import TransactionManager

clr.AddReference('RevitAPIUI')
from  Autodesk.Revit.UI import *

clr.AddReference("RevitNodes")
import Revit
clr.ImportExtensions(Revit.GeometryConversion)

clr.AddReference('ProtoGeometry')
from Autodesk.DesignScript.Geometry import *


clr.AddReference('System')
from System.Collections.Generic import List as sList

clr.AddReference('DSCoreNodes') 
import DSCore
from DSCore import *


def flat(lst):
	return DSCore.List.Flatten(lst,2)

doc = DocumentManager.Instance.CurrentDBDocument
uiapp = DocumentManager.Instance.CurrentUIApplication 
app = uiapp.Application 
uidoc = uiapp.ActiveUIDocument

location_list = flat(IN[0])
family_list = flat(IN[1])
rotation_list = flat(IN[2])
lev_list = flat(IN[3])

counts = IN[4]

## Split list into chuncks
def chunks(data,sizes):
	it = iter(data)
	return [[next(it) for _ in range(size)] for size in sizes]

chopLen = IN[5]

# Yield successive n-sized chunks from L.
def chop(L, n):
	for i in range(0, len(L), n):
		yield L[i:i+n]
        
# # Chop Inputs
chopLocation = chop(location_list,chopLen)
chopFamily = chop(family_list,chopLen)
chopRotation = chop(rotation_list,chopLen)
chopLevel = chop(lev_list,chopLen)

#OUT = chopLocation

class DuplicateInstancesWarningSwallower(IFailuresPreprocessor):
    def PreprocessFailures(self, failuresAccessor):
        fail_list = List[FailureMessageAccessor]()
        fail_acc_list = failuresAccessor.GetFailureMessages().GetEnumerator()
           
        for failure in fail_acc_list:
            failuresAccessor.ResolveFailure(failure)
            failure_id = failure.GetFailureDefinitionId()
            failure_severity = failure.GetSeverity()
            failure_type = BuiltInFailures.InaccurateFailures.DuplicateInstances
            if failure_id == failure_type:
                
                failuresAccessor.DeleteWarning(failure)

        return FailureProcessingResult.Continue

output = []
result = []
testing = []


collect = sList[Autodesk.Revit.Creation.FamilyInstanceCreationData]()

for cLocation, cFamily, cRotation, cLev in  zip(chopLocation,chopFamily,chopRotation,chopLevel):
	
	collect.Clear()
	
	TransactionManager.Instance.ForceCloseTransaction()
	
	
	t = Transaction(doc, "Batch Create Families")
	
	# Start
	t.Start()
	
	tOpt = t.GetFailureHandlingOptions()
	tOpt.SetFailuresPreprocessor(DuplicateInstancesWarningSwallower())
	t.SetFailureHandlingOptions(tOpt)
	
	for location, family, rotation, lev in  zip(cLocation, cFamily, cRotation, cLev):		
	
		location1 = location.ToXyz()
		family1 = UnwrapElement(family)
		rotation1 = Math.DegreesToRadians(rotation)
		lev1 = UnwrapElement(lev)
	
		create = Autodesk.Revit.Creation.FamilyInstanceCreationData(location1,family1,lev1,StructuralType.NonStructural)
		create.RotateAngle = rotation1
		create.Axis = Line.ByStartPointDirectionLength(location,Vector.ZAxis(),1).ToRevitType()
		
					
		collect.Add(create)
		
	items = doc.Create.NewFamilyInstances2(collect)	
	
	
	pack = []
	for i in items:
		pack.append(doc.GetElement(i))
	result.append(pack)

	t.Commit()

OUT = result

At least I am making progress :sweat_smile:

17

For duplicate warnings to be produced and the new instance discarded I believe you’d have to roll back or rather discard the transaction which means each instances needs it’s own transaction, which would move from one transaction to N transactions. I can’t confirm now though as I am on vacation.

Why not prune the duplicates before creation? Seems that would be simple as you’ve got or could get a compatible item for each. Or are these issues with things already existing in the model but your automation data set doesn’t exclude those? If either is too hard at the moment this would favor the inclusion of an intermediate external file rather then a ‘one click’ process.

1 Like
18

An update on testing.

It would appear from large scale testing that Revit holding lots of Transactions in the ‘Undo’ stack may be causing a significant decrease in performance. Reducing these through TransactionGroups was something I wanted to try.

I have used a sample Transaction of placing a single Point based Foundation Pad element to compare whether using TransactionGroups provides any performance gains.

Result 1 - Small Scale:
10 Individual Transactions = 7.3 seconds

1 TransactionGroup containing the same 10 Transactions = 1.4 seconds

This trend of improving the runtime is mirrored in larger scale testing.

Result 2 - Large Scale:
10 Transactions each creating 100 elements = 202 seconds

1 TransactionGroup containing 10 Transactions, each creating 100 elements = 44 seconds

Result 3 - Larger Scale in Project:
I tested again inside my project environment with good results.

Yesterday = 1200 CreationData entries, 3692 Revit elements created, 160 minutes.

Today = 8000 CreationData entries, 8336 Revit Elements created, 160 seconds…

I’m doing something right :sunglasses:

I have also implemented an IFailuresPreprocessor, that now works properly to ignore/remove the ‘DuplicateInstances’ warning. So less warnings for Revit to keep track of as well.

Revit is performing much better now after the entire creation process has completed. So I am happy with that. :slightly_smiling_face:

More soon :+1:

7 Likes
19

Why don’t you eliminate duplicates before creating? Duplicates is redundant, no matter when you solve this problem.

1 Like
20

Agreed @Vladimir
I will be sorting that part of the logic next.

My priority was memory testing large load automation, so being able to fail fast is better right now for my proof of concept.

2 Likes