I tested it quickly and once you are in the document of the host family you can select any element inside it with a filtered element collector. Here I randomly chose a nested family based on its category and its comment value, then i obtained the document. From there on I think you can do what you want.
I did not do it all in one python code to make it more intuitive here on the forum
First code:
doc = DocumentManager.Instance.CurrentDBDocument
actually here you should input your doc as IN[1]
cat=UnwrapElement(IN[0]).Id
OUT=FilteredElementCollector(doc).OfCategoryId(cat).ToElements()
Second code:
fam=UnwrapElement(IN[0])
famdoc=doc.EditFamily(fam)
OUT=famdoc