Isolated Pick Model Elements - Is there a way to make sure the selection isn't "lost" accidentally?

I absolutely love this node, the “Isolated Pick Model Elements” node. It makes selecting specific model elements so easy without needing to tab through and hold CTRL to add to selection. The ONLY thing I have an issue with is when you’re adding to the selection and you have a whole bunch of elements picked already, and you accidentally click in white space, your entire selection is reset, and you have to start again. Is there a way to get the node to just ignore the click into white space, and to only remove from selection when cancelling the command?