Newmarket itself, where I grew up, is now pretty sordid, by all accounts. The stable lads always were a rowdy lot.
If you like walking you can walk for ages across the Heath. Even if you're not into horseracing, a visit to the National Stud is worthwhile as Newmarket is more of a breeding centre than a racing venue. There is a horseracing museum in the High Street next to the jockey club.
If you can book a guided tour your will get to see Nell Gwynn's cottage and learn more about Charles II.
Lilly Langtry had a house in nearby Kentford, and Lord Byron lived with his sister Augusta (?) in Six Mile Bottom but there is nothing there these days to commemorate those colourful characters.
Lots of picturesque villages nearby, such as Clare and Lavendish. You need at least a day to see Cambridge, the Colleges, the Backs, the Botanical Gardens and to soak up the atmosphere. If you're lucky there might be a concert or at least Evensong in one of the chapels. Don't miss the grasshopper!