It may be that your custom node isn’t compatible due to multi-threading or reliance on a tool which won’t allow concurrent API calls, or that the settings in Generative Design are over constrained.
First try doing a study with on of the samples provided to confirm that GD can run.
Then try doing a random study with your graph to see if the later is at fault. Upload the graph and custom nodes if you think it is related to the nodes you have built.