A ball that's got sideways spin imparted will tend to curve one way in the air and then cut back off the pitch in the opposite direction. Hence a leggie getting in-drift, and an offie getting a curve towards off. In The Art of Cricket Bradman referred to this as "swerve".
A ball that's bowled to swing - with the seam angled and stabilised by the backwards rotation of the ball - may hit the seam and move in a more or less random way. It could swing one way in the air and go even further that way off the pitch, or it could swing one way and seam the other. It can be absolutely bloody lethal - see
this example by Terry Alderman v John Stephenson in 1989.
From what I've read of SF Barnes he bowled the first kind of ball (basically quickish legspin) - and perhaps also the second, although I have my doubts - but as Migara says he wouldn't have bowled both in the one ball.
Edit: re-reading Migara's post, it is certainly also possible that a spinning delivery will also be affected by swing, depending on the angle of the seam. So if you bowl leggies with a bit of topspin, the seam is likely to be angled towards slip which could produce some out-swing. However because this is in the opposite direction to the natural swerve of the spinning ball, you'd be likely overall to see less rather than more movement due to the swing.