I think if you're a seam-dominant pace attack (i.e. Australia in the 2000s) then yes, SL is a place where you'll struggle. Plenty of swing bowlers do fine there though. Vaas, the Pakistani bowlers. Hell, even someone like Kulasekara does all right.
Just gotta play to the conditions: pitch it up, get late swing. Not every pitch needs to play like Australia.