Guitarists Jeremy Bolen and Nathan Snydacker formed Chin Up Chin Up in Chicago in 2001, and eventually rounded out the quartet with drummer Chris Dye and bassist Chris Saathoff. In January 2002, the band released a self-titled EP on Flameshovel Records, before adding keyboardist Greg Sharp. After some successful touring they began on We Should Habe Never Lived Like We Were Skyscrapers, which was unfinished when bassist Chris Saathoff was killed hit-and-run traffic accident. After a hiatus, the band finished the LP using recorded practice material of Saathoff and eventually toured again, with Jesse Woghin from another Chicago band, the Narrator, on bass. In 2006 they released the acclaimed follow-up The Harness Can't Ride Anything on Suicide Squeeze.