Just wondering if anyone has experimented with tools such as ComfyUI in Dynamo ?
If you are not familiar with it, ComfyUI is well worth a look- I’d recommend this course: gen ai for architects
In the Rhino/Grasshopper world, there is the AIman package- it basically passes a Rhino geometry screenshot to ComfyUI, together with prompts for the generative image part (using diffusion models)
The geometry part can be very simple- just a massing/form, and the AI fills in the rest.
https://www.food4rhino.com/en/app/aiman-tools
As I see it- the same would be possible in Dynamo, but relies on being able to pass the geometry image to ComfyUI
2 Likes
Wouldn’t be too hard to do, and I have seen people talk about doing similar already although always in a .dyn > .rvt > AI rendering > jpg workflow. To pass the geometry wouldn’t be too ahrd. Easy way would be to export to a format that ComfyUI accepts - SAT, STL, DWG, or other format for which there is already a functional node for. Harder way but perhaps with more control, you could build something which uses Dynamo’s tessellation engine (how Dynamo generates the background geometry display) and pass that (a mesh) on as a string that ComfyUI accepts. The Dynamo Unfold Package from @Michael_Kirschner2 has a good example of the basics there, and it can be used in both Python or C#.
The hard part with any of these is that tools like ComfyUI et. al. require a license and the costs can add up quickly - passing around code quickly runs up a bill if you aren’t careful. As a result the ‘safe’ way to do this is for the service provider (ComfyUI) to provide the nodes so that the usage is properly logged.
Long term I’m hopeful that there will be a supported MCP from one of the Autodesk teams (perhaps 3dsMax?).
thanks @jacob.small
I’ll think it over- how it works with Grasshopper is giving me some ideas..
1 Like
The first method that comes to mind would be to programatically switch to geometry view, zoom extents and export a workspace image to send up to Comfy. Not sure how much of that is possible, but its how I’d go about it if it is probably.
I’m unsure how well Dynamo goes with waiting for things of unpredictable times, might be better to build a call/response pairing workflow or even just a round trip script where the user selects the workspace image export file and it sends it up if it hasnt been yet, and checks/returns the status from Comfy if it’s processing or finished.
Should do fine here so long as the mode which sends is also the node which receives - DynaWeb, DynaFetch (good new package from @CStorms if you haven’t heard of it yet), DynamoForForma, anything which touches APS, and more.
If you write one node to send and another to receive then you’re going to likely ask for the response at the wrong time or with the wrong keys… so I would avoid that.
4 Likes