I have created a graph that performs solar analysis on 5 surfaces and outputs the results to a CSV file. I need to repeat this process for every sunlight hour of the year which obviously will result in a very large dataset.
So far, when I run the process for a short time span, say one or two days, the time necessary to complete the process remains reasonable, around 10 minutes, or less. But if I increase the timeframe of the study, the time needed to process doesn’t scale proportionally. I understand that this is a limitation of Dynamo as it has to keep all the data in memory. (I have turned off preview in both Dynamo and Revit)
Initially, I was outputting separate columns for x, y, and z but that necessitated list manipulation. So I have decided to trim down the process and tried to output to SQLite. One day, that is 12,000 lines of data took 5:40 (minutes, seconds) while going for two days with 24,000 lines of data took 33 minutes.
The difference with outputting to SQLite is that the data is exported sequentially, so monitoring the database in DB Browser for SQLite shows the table filling up gradually.
Clearly, the best way (at least for me) is to process in chunks as the more data that is generated, the slower Dynamo will become till it is no more reasonable. I also think that resetting the solar analysis node to clear its data is the way to go. Resetting the whole graph will force dynamo to load all geometries which will take additional time to reload them for each run.
I’m attaching the original graph that writes to CSV and the trimmed down version that writes to SQLite.
(Note that in the second graph, the lines that go off the screen go to frozen nodes).
Any hints on how to proceed from here will be appreciated.