Dynamo: performance

Hello, My latest Dynamo project is a ‘clash importer’ that creates a sphere with various attributes of the ‘clash’. There are some 3rd party applications that do the same thing- i.e import the XML report from Navisworks and place a sphere marker at the coordinates. Using the tools in Navisworks, it is difficult to get a sense of the big picture of where the clashes are and their status. Refer below, colour coded by status in Navisworks I’m using Pentaho to get the XML into a better format- this tool is well worth a look if you do any database work or need to transform data between different formats & structures. kettle-header-dashboard If anyone is interested, I am happy to share the necessary Pentaho/Dynamo

 

As a plug for my blog- this post covers the Myth of clash detection In this project, there are a crazy amount of clashes- in the tens of thousands. However, Dynamo/Revit seems to hit a hard wall of how many records it can deal with.

  • 20,000: no problem, takes about 5 mins to run in Dynamo
  • 30,000: no-go, even if I let it run all weekend, Revit/Dynamo 'hangs', maxing out a processor
I would have expected 30,000 to take ~50% longer than 20,000. but just seems to be over some sort of threshold. There could be be some bad data, but looking through it it seems to be pretty consistent. The questions are:
  1. Is there any way to see the progress/status of Dynamo as it is running, particularly to identify any processor or memory intensive steps ?
  2. Is there any way to increase the threshold i.e configuration changes, assuming plenty of physical memory etc
(this is other than adding watches in Dynamo, or seeing various errors once it has run) It seems that as soon as you hit 'run', it is a bit of a mystery what Dynamo is doing. I am thinking of various activity/metrics/monitoring tools thanks Andrew ![20160307 clash spheres|1183x821](upload://y05RZpD6IdPVbLkBPdJ00CstLUJ.png)
1 Like

3rd question- is it just me, or does the formatting of the post go crazy once you submit ?

When it comes to the formatting, you are not alone :slight_smile:

Many a time I find myself editing posts to rectify the formatting.

I haven’t done much clash detection myself to fully understand the capabilities of your definition. However, found your blog post to be informative.

Hi Andrew,

I don’t know of any ways to add a progress bar to Dynamo. Tho you should probably check this out first:

https://github.com/DynamoDS/Dynamo/wiki/Efficiently-Working-With-Large-Data-Sets-In-Dynamo

I’m guessing that Revit is the biggest bottleneck in your process. Consider using a cube or even a tetrahedron instead of a sphere. Curved faces are bad. The less faces you have to render, the smaller the performance penalty.

 

And finally yes, the forum gobbles up your formatting every time you press edit.

 

P.S.

Not sure if this will help with performace but it should reduce your file size; try using transformed/translated typed direct shapes instead of actual families:

http://dynamonodes.com/2016/02/18/directshape-bygeometry-directshape-transform/

 

1 Like

thanks Vikram & Dimitar, I will take a look through the links above.

I had already tried cubes rather than spheres- but interestingly, it makes minimal difference to the Dynamo processing time.

There is some Revit display weirdness for both spheres & cubes, but I think this is just because the objects are a long way from the origin.

Refer link- spheres=blue,cubes=cyan. Yes they do change shape as you zoom in, and cubes appear as triangular prisms.

But once you bring it back into Navisworks, it is fine

I’ve also learnt that each time you edit a post in the forum, it gets even crazier…! So don’t even try !

 

 

I know this is an old post, but I wanted to follow up on the question regarding a progress bar. Does anyone know if there is a way to add a progress bar to function as part of a Dynamo script run?

Not that I’m aware of

I think the issue would be that Dynamo would need to first work out the total amount of processing, then the progress at a point of time as a proportion of that

Progress Bars are inherently difficult to implement - because they all pretty much fake it.

You can have a graph with 100 nodes, and most of those will execute in microseconds, but one Python node could take 20 minutes - therefore your progress bar, if implemented on a per-node basis would go super fast for 99% of it then hang for the residual 19.9 minutes. To get an accurate progress bar, you would have to pre-calculate to then post-calculate… which kind of defeats the purpose :slight_smile:

As a side note, we are developing a way to showcase how long each part of a graph takes to run using the TuneUp Extension - available to check progress on here: https://github.com/DynamoDS/TuneUp

This will showcase per-node runtimes (execution time) and also expose a Force Calculate All button as Dynamo uses Delta Compute - Where it will only re-run sections of the graph that need it (i.e have new data passing through). This allows us to be much more efficient in operations - especially with large datasets.

A good example is taking a bunch of geometry and booleaning them together, then translating them. If you translate again and again after it will be fast, but if you had to re-calculate the boolean operations over and over it would be slow!

We’ve had internal discussions around how to implement a feasible, but more importantly useful approach to showing that Dynamo is still running - happy to hear any ideas on this front too :blush: TuneUp is a nod towards starting to showcase this.

And more generically on Performance we’re looking at how to improve the overall performance of Dynamo right now - as mentioned in our Public Roadmap: https://github.com/DynamoDS/Dynamo/wiki/Dynamo-Public-Roadmap

Have no idea how this works in your code behind, but something like a property of the node you could write to in python or OOTB nodes that you give a count total based on the inputs and then just i++ for each element of the loop. This could be an selected thing like the player’s Is Input property. I’d selected and given a count and the individual increment a PG goes across the bottonnor too of the node based on items processed not tine. That way it wouldn’t be “fake” but also wouldn’t indicate time remaining either, just a way to tell it’s still working. I donthis often in C# Forms to make sure things are still running.

We’ve discussed this option also - more of a way to showcase that Dynamo is still working rather than a proper progress-bar :slight_smile:

3 Likes

@Andrew_Hannell -
is strikes me that your initial problem here might have had to do with this:

It will be difficult to test if the daily builds fix it since I know this was a long long time ago, AND in Revit - but something to look out for.

1 Like

Greating! I read that due to the incomplete implementation of the multiprocessing module in IPython it is impossible to use many processors. I can run IronPython without the need for a GIL but Python (CPython) requires a GIL and performance is improved through threading. Can statusbar be implemented in main thread? In the code sections call join and start and show statusbar? Anybody try?

Hello @ilgnv88,

Dynamo Sandbox is multi-threaded (Which is awesome!) and contains a UI thread and an execution thread (Also awesome), but some of our host environments are single-threaded (Such as Revit) that means both the UI and Execution are on the same thread. This pretty much stops us being able to implement UI features that still continue to function during the execution of the graph (i.e a progress bar).

As an aside, we’re looking into enabling CPython inside of Dynamo now (Firstly in tandem with IronPython) to expose a whole bunch of things :slight_smile:

It’s very great news!!! :slightly_smiling_face: When it is possible to see?

We’re just kicking off the work now - so not for a release or two sadly. Will release another Dynamo Public Roadmap in the not-to-distant future that will talk a little more about our plans :slight_smile:

@solamour , how can we use it in collision(intersection) detection? How Does.Intersect can be multi-threaded with Dynamo? We can use some packages for fast detection collisions in Revit, but solid-solid collisions in Dynamo are just brute-force.
Thanks!

1 Like

Hello @Vladimir, the nodes themselves are single threaded (We decouple the UI thread from the execution thread and enable delta-compute on the execution thread).

Spring Nodes by @Dimitar_Venkov has an example in this Mesh.ToPolysurface node that showcases one way to use Threading in Python inside Dynamo.

Please bear in mind that you will have to dispose of any geometry on those threads before they disappear as threading sits outside the scope of our recent work allowing for auto-garbage disposal in Python.

1 Like

I 've implemented threading in my script and it works differently on different computers, somewhere longer, somewhere faster :thinking:. Therefore, I am interested in the implementation of the multiprocessing module because now for large models the execution time of my script (script is used to find the shortest paths in electrical circuits) takes more than 12 minutes for 325 electrical circuits.