There were two things going on: 1) I had already established in my own mind where I wanted to go with the next series, and having James around as a Grey Eminence would have complicated matters. He had had an amazing life and it was time to bid him good-bye.
Also, it's risky to try to duplicate earlier success. Magician had a certain charm to it, mostly due to my choice of lead characters, that I would be hard put to duplicate.