I like your analogy!
Perhaps a useful question is....Ginkgo wrote:There is no euphoria on my part when I beat you. I might ACT like I am delighted with the win, but is is only an act.
What will it take for me the human to buy in to the act put on by you the virtual software based chess partner?
It seems that at some point the simulation will become realistic enough that even though I know you are software, I buy in to and experience the emotions you the machine are projecting.
The simulation doesn't have to fool us completely. As example, consider a play or a movie.
Intellectually we know it is just a play and those are just actors pretending to be the characters they are portraying. What makes the play work is that we willingly suspend disbelief and embrace the illusion. We've been doing this ever since the stories around the campfire days.
Consider how real world actors are now playing right along side software entities in movies like Avatar. I don't see people jumping up in the theater yelling, "Those characters are just software!"
Why would a Holodeck in your bedroom be any different? If the movie Avatar is compelling in it's current form, won't it be even more compelling when you and I are in the movie?