Yeah that would be fair enough if it wasn't cutting down the sample size by almost 90%. Smith played about 8 matches against Australia and rarely made the runs, but that's not many matches, it's not enough to say he couldn't. It could simply be the case that a lot of his bad days happened to come against Australia, that he got a disproportionate amount of literally unplayable ones, bad decisions or every mistake led to a dismissal in those games. I don't think this is the case for Smith, but the point still stands.
Also, aussie's suggestion was using game against Australia in Australia, which trims it down so much that it's not viable at all.
Not only that, but the definition of a good batsman isn't one who performs against the best team in the world at that time. Surely fans would prefer a batsman who performs generally well against everyone, rather than fantastically well against Australia and mediocre against everyone else. In truth, a poor record against Australia is just a glitch on a batsman's record, much like Warne in India.