Bill and Carol McGann's The Story of the Tour De France, Volume 1 is a fairly comprehensive history of the Tour from its inception in 1903 to 1964.
The book takes the form of a play-by-play for the significant stages of each year's Tour, and discusses how the rules of the Tour evolved to adapt to the changing times. The creator of the Tour, Henri Desgranges, was quite the character. He wanted the Tour to be a show of the individual cyclist's strength and cunning, but he was in the end unable to keep the team element out of the sport.
The main takeaway from this volume, for me, is that the Tour de France used to have a lot more badassery than it does today. Consider, for example, Stage 2 of the 1909 Tour, 398 kilometers (247 mi) long and run entirely in freezing rain, in which François Faber broke away for the entire second half of the stage. Or the story of Eugène Christophe in the 1913 tour, in which he broke his fork on a descent during Stage 6. There were no follow cars back then; under the rules of the time, riders had to make their own repairs. Christophe carried his broken bike down the hill, walking 10 kilometers to the next town. Here he found a blacksmith, who was able to talk him through what he would need to do to repair his fork (this was as much as was permissible under the rules). Christophe worked at the forge for 3 hours. But at some point, he ran out of hands. He asked a little boy to work the bellows for him. For this the Tour referees gave him a 10-minute penalty. Eventually Christophe finished the repair and kept on riding. He finished the stage 29th— still ahead of 15 other riders.
If you are interested in the history of the Tour de France, there is probably not a better book. The Story of the Tour de France gave me some perspective about how competitive cycling works and about the historical evolution of the sport, but at times it seemed to me to be little more than so many names and numbers.