"I think Mary and Carlisle were quite well-suited in some ways. But in the end he just wasn’t as nice as Matthew, really. And I think Lavinia was, in a way, a much sweeter person than Mary, and would have been a very loving and supportive wife. Whereas I think Mary is a much more prickly affair. He might have had an easier, if a more boring, time with Lavinia, and similarly, if Mary had married Carlisle, they would have really built something."

Julian Fellowes Overcomes His Scruples and Looks Back at Season 2 of ‘Downton Abbey’ - NYTimes.com