Yes, thats possible:
in the child component you have to declare a p:remoteCommand [p:remoteCommand][1] that executes the child component logic. The name of this remoteCommand should be defined by a component interface attribute:
< cc:interface componentType="IvyComponent">
< cc:attribute name="callbackName" />
< /cc:interface>
< cc:implementation>
< p:remoteCommand name="#{cc.attrs.callbackName}" actionListener="#{logic.execMyComponentLogic}"/>
< /cc:implementation>
The parent dialog can then embedd the component-dialog and define the name of the child callback method. And bind it to an oncomplete action of a commandButton:
< ic:jsf.call.child.logic.ChildDialog callbackName="callChild" />
< p:commandButton value="call child logic" oncomplete="callChild();"/>
[1]: http://www.primefaces.org/showcase/ui/ajax/remoteCommand.xhtml