"We stayed in touch," Elijah says calmly. "But he preferred his privacy and so did I. If he'd asked, I would happily have visited, but..."
He didn't, so Elijah didn't. End of story.
Other options:
no subject
"We stayed in touch," Elijah says calmly. "But he preferred his privacy and so did I. If he'd asked, I would happily have visited, but..."
He didn't, so Elijah didn't. End of story.