Some quotes from team mates and articles
"Hammond was cut on the chin by a short ball, causing him to retire hurt. He again commented that he would quit rather than face such bowling; soon after his return, he was out. Les Ames, who played in the three-match series, won by England 2–0, believed that the West Indian pacemen worried Hammond, who showed a weakness against short, fast bowling."
"However, Australian bowlers such as O'Reilly and Grimmett troubled him by bowling at his leg stump, restricting his scoring as he had fewer effective leg-side shots.[208][214] Occasionally, he displayed discomfort against the fastest bowlers.[208] His team-mate Charlie Barnett said that he did not relish fast bowling, although he was capable of playing it well in the initial stages of his career.[214] Other colleagues, such as Les Ames, Bob Wyatt and Reg Sinfield, believed that he did not like to face the new ball,[122] and he was occasionally happy for the other batsmen to face the difficult bowling"
While the above is certainly subjective, what is certain is against the West Indies, the only strong pace attack he would have faced, he struggled mightily and in thirteen matches and twenty innings only scored one hundred at an average of 35 and was known to be uncomfortable against them much in the same way that Weekes struggled versus Lindwall and Miller.