Hang on, people can't attend five days of Test cricket all the time! Adults have work, children have school, and we have better things to do than declare unofficial holidays for Test cricket! We have to earn a living, or learn what we need to learn.
Some of the grounds are quite large, which may make crowds look small. Test match ticket prices are quite nominal, in comparison to ODI or IPL ticket rates, so buying them isn't an issue. Playing truant from school or work is.
Moreover, television access is another factor. While you get a live feel at the venue, it's reduced spectacularly, and comes out of spectator response rather than the game, when it's at a large, packed venue. Television takes you straight to the tip of the bat or blade of grass, and gives you so many options that you don't quite get at the venue. Some grounds had giant TV screens, but some of them, like the Hede's Megavision, are not in working condition- maybe it's been brought back in action.
The one funny thing is that in an advertisement for a television, Sachin Tendulkar (no less), said that people do anything to watch a cricket match- showing fans perched on trees, towers and terraces. He said that he can watch a match- in comfort- watching it on television. Not so encouraging for watching a game live, is it?