Lookign for explanation of how Dynamo instantiates geometry and manages resources

[stephen.lewis]

I have searched the forums and the web to see what information is out there when it comes to options for speeding up performance of Dynamo, and its instantiation of elements in REVIT.

Dynamo: performance

Hello, My latest Dynamo project is a ‘clash importer’ that creates a sphere with various attributes of the ‘clash’. There are some 3rd party applications that do the same thing- i.e import the XML report from Navisworks and place a sphere marker at the coordinates. Using the tools in Navisworks, it is difficult to get a sense of the big picture of where the clashes are and their status. Refer below, colour coded by status in Navisworks I’m using Pentaho to get the XML into a better format- this …

Efficiently Working With Large Data Sets In Dynamo

Transaction Explanation

In general Revit these days wants to handle Transactions on its own, and even the External Commands recommendation is to not manage them manually (not exactly but for the sake of this conversation let’s leave it at that) but instead let Revit handle them. You only need to do this manually if you are creating things that later need to serve as hosts for other things since they need to be commited to the DB before you can use them downstream. What are you creating?

I note the following but would like confirmation/explanation

1. It is best to generate geometry but not preview/tesselate the geometry. This makes sense per the first link above, I would expect if the user unchecks the preview request on a set of nodes this should not be an issue - unless Dynamo is still storing in memory the preview geometry when a node is set to not preview. Is this true. (Someone in the thread above also questioned this and asked if not selecting preview on a node this would dispose of any memory overhead).

2. I understand that Dynamo always stores a copy of the data output from a node and used downstream in subsequent nodes, therefor the more nodes containing data needed for a task, the more memory needed by Dynamo. Therefore it makes sense that condensing 10 nodes used to manipulate/create data to a single code block would be massively efficient as the construction geometry used in the code block does not need to be stored by Dynamo. Only the output of the node is managed by Dynamo. (Does one need to set the value of any internal variables used in the DS of a node block to null after looping for this to be true?)

3. If 10 nodes are used to manipulate geometry and data from A to B, if a user shrinks the 10 nodes to a single custom node, shouldn’t dynamo manage the memory much more efficiently as it does not need to have a copy of all the node data form A to B in the uncompressed definition on the canvas. It only needs to manage the output data of the custom node.

4. Is data managed more efficiently by Dynamo if instead of creating a custom node from 10 nodes say, the user instead reduces the nodes to a single node DS using “node to code”? That way we end up with option 2. If so, is this more efficient than option 3?

5. Is it correct to assume that using a scripting node in Python or scripting using DS in a code block will also be much more efficient as these nodes are unable to preview geometry and hence there is less overhead for Dynamo. If so, how does one preview such geometry? I don’t see a DS statement for this. (I’m not sure this is true as a point constructor defined using DS in a code block is previewed. Perhaps this is specific to a Python node)

6. What is the best practice for instantiating large numbers of elements in REVIT? Say if I have a list of a 1000 surfaces that I want to be represented as floor elements, is there a way to create/transaction all the 1000 surface geometry in one go into the REVIT environment and then assign floor element to all of these objects in one shot? The reason I ask is that Dynamo studio appears to in much faster with background threading whereas Dynamo REVIT is running single thread and it has to wait until all the REVIT transactions are done before it becomes responsive. (Maybe Dynamo V2 might be better with improved background/and or multi threading.

Apologies if some of these questions are obvious but I am coming

[Michael_Kirschner2]

1. this presupposes that preview geometry is heavier than the underlying solid geometry for regular dynamo OOTB geometry nodes. In most cases I don’t think this is true. You can see here in the code https://github.com/DynamoDS/Dynamo/blob/master/src/DynamoCoreWpf/ViewModels/Watch3D/HelixWatch3DViewModel.cs#L768 that when a node preview is changed the underly helix geometry3d object visibility is set true or false… to dig further and see what is done in that library - you’ll have to go look at helix3d which is also opensource on github.

2. Simply using a codeblock to represent the same graph will not save any memory as all outputs from the codeblock generate ports and can be used as downstream values by any other node. Condensing your calls into functions (def statements) should enable better memory management as the DS garbage collector should detect that these intermediate values cannot be referenced by anything and cleaned up. You should not need to set anything to null, @Yu_Ke may know more here. Have you seen this wiki post? https://github.com/DynamoDS/Dynamo/wiki/Efficiently-Working-With-Large-Data-Sets-In-Dynamo … I will conclude this post with some general tips about memory management when dealing with extremely large dataSets.

3. This could be true as custom nodes are functions - I am not sure if it is. One could profile this with dot memory or another memory profiler. Custom nodes are complex and it wouldn’t surprise me if they were not as efficient as they could be.

4. No, as said above node to code will not produce functions but instead produce codeblocks with outputs at every step.

5. This presupposes that tessellated geometry is the limiting factor - I don’t think it is in most cases unfortunately, though definitely it contributes to out of memory errors. Python nodes will only preview output geometry. The general case is this: if a type which implements IGraphicItem is present in the output of a node it will get previewed. The best performance is via C# - as it is compiled and DS is only responsible for calling the functions, you also have a high level of control over what you return.

6. Revit is single threaded in the addin context, can’t get around that except with separate processes. Dynamo already tries to wrap as many possible interactions into a single transaction. You can control this if required with nodes like Transaction.Start and End. For creating many revit elements I would recommend using directShape elements if possible. If you need actual revit elements, like floors etc, then you are best to avoid replication as there will be some function overhead. The absolute fastest thing you can do it to use a zero touch c# node that calls the revit api directly.

In general Dynamo’s solid geometry is heavy, previewing it is an added expense, but I think the brunt of memory use is actually the underlying geo. In python and c# you must manually dispose intermediate geometry objects you do not return into the graph. https://github.com/DynamoDS/Dynamo/wiki/Zero-Touch-Plugin-Development#dispose--using-statement

This will make Dynamo much more stable if you are currently using c# and python nodes and not doing this. In DS in codeblocks you should not have to.

To deal with large amounts of geometry and memory use in general you should limit what is returned into the graph from your functions (DS,Python or C#) - stream data when possible, avoid the built in Dynamo geometry types if you must, prefer meshes from meshtoolkit if you do not need solids. If you do not return geometry types into the graph you’ll avoid needing to preview them, and if you’re using c# or python you’ll have already manually disposed them.
This applies in general - if you have a giant list or dictionary of data which Dynamo is previewing - this is slow and memory consuming - you can write a ZT or Python type which wraps this data and override the toString() method to get a much faster preview value in the graph.