Tofflemire grew up in Post Falls, Idaho, as one of eight siblings born to John and Anna Tofflemire. He was a 1984 graduate of Post Falls High School where he was a three-year letterman in both track and football and was named to the All State teams being named the Offensive Lineman of the year his senior year.
In high school, he was an All Intermountain League First Team kicker his junior and senior seasons. He kicked a 44 yard field goal his junior year in a State Playoff game versus Bishop Kelly which at that time was an Idaho high school state record. He was moved up to the varsity team his freshman year as a kicker. He wore a size 14, black Spotbilt square toe kicking boot and was given the name "Joe the Toe" by assistant coach Steve Long during a practice session. The nickname stuck, and throughout his life he was referred to by former high school, college and pro teammates simply as "Toe". With a powerful leg, he could regularly kick the football through the uprights on kickoffs.
At the University of Arizona Joe started four years at center, being named to the All Pac-10 Team three straight years. He mad All-American his junior and senior years. He won the Morris Trophy his senior year, which is given to the most outstanding Offensive Lineman in the Pac-10. He was also named captain of the team and MVP his senior year at the University of Arizona. Since Graduating from Arizona, he has been named to the Pac-10 all 80’s team, and has been elected to the University of Arizona Sports Hall of Fame. In 1997 he was named to the wildcat stadium ring of honor at the University of Arizona, this honor has been given to only 24 players that have played football at Arizona. In 2000, he was named one of the top 100 athletes to attend the University of Arizona and was also named to the All-Century team as well as the Arizona All-Time, All Pac-10 First Team.
Joe was the 44th player chosen in the 1989 NFL draft by the Seattle Seahawks, starting at center for seven years before retiring in 1995.