My kids went at age 3. They did much the same as I did that day, look around the crowd all gormless wondering what the f**k was going on.
I'd say that was too young. He'll probably end up cold and bored and it might put him off for life. To get him interested take him to a big game or the first game of the season, where there'll be some atmosphere and excitement about the place.