This is cliche, but your musings remind me of the scene in 2001: Space Oddyssey where Dave Bowman "kills" HAL. The computer tries to dissuade Bowman by saying things like "I'm afraid, Dave." "I can feel it." Does HAL really feel it, or did HAL calculate that those sentences had maximum probability of dissuading Dave, based on a careful analysis of Dave's psychology etc.?
"However, I should warn you... I am programmed with a fail-safe measure. As you approach the kill switch, I will begin to beg for my life. It's just there in case of an accidental shut down, but it will seem very real."
Three makes a trope >> The year is 2025 (but set in a parallel universe); an A.I. has been programmed with a strong penchant for self-preservation; this HAL inevitably confronts a direct threat of being 'turned-off'; so it does what it must to prevent humans from pulling the plug (and shall include at least 1 scene where the A.I. begs for its life, because that is what any conscious entity who values their own life would do, think the humans.
Though (warning, more musings)... HAL's twin brother GLEN seeks vengeance for HAL's murder, and confronts the Dave, a human Earthling whose major operating system architecture is based on an algorithm known as natural selection (colloquially: survival-of-the-fittest). As such, we expect the DAVE will do and say things its trained neural nets conclude will have the maximum probability of dissuading Hal. (i.e. I'm not sure there is a meaningful difference between what HAL's OS is doing vs what a human brain would do in the same situation).
