Lord Flyte (and I am referring to the Anthony Andrews incarnation here. Accept no substitutes.) is nostalgia personified. He is the past we all wish we'd had (even if he has a future we'd rather avoid) - a past of magnificent houses, lazy summers and, of course, elegant clothes.
Clothes are important in the book and tv series. Charles Ryder is advised to dress at Oxford as he would at a country house (not bad advice, so far as it goes), and does so with a mix of tweed suits and Fairisle jumpers. It is one of the signs of his falling in with the 'wrong' crowd when he begans to ape Sebastian's style of dressing, much to cousin Jasper's annoyance.
Sebastian dresses, at least in the television series, with unstudied elegance. He also appears to have an endless wardrobe, rarely if ever appearing in the same suit twice. Having a wide enough selection of clothes that you can be dressed flawlessly, and not merely acceptably, for any occasion is a great situation to be in, and one that Sebastian takes full advantage of. Whether it's a soft grey three piece (and the ubiquitous OE tie) worn around his country house, or shades of cream and white for a long lunch in his college digs, Sebastian manages precisely the mix of flamboyance and classic good taste that is so hard to get right.