img2img creations are probably confusing it because they are totally rearranging how the prompt emerges from what's in the latent space with respect to a new image. So if you make an image of <subject> by passing in an image of <subject>, it's going to represent <subject> in a fundamentally different way than if it relies purely on its own "imagination" for the rendition.
