Joseph Phillips was the fifth child of Michael Phillips, Sr. and Ave Rosena Sheperda. He came to America with his parents in 1837 at the age of two. He married Catherine Swartz, they had 8 children: John, Frank, Edward, Mary, Helen, Rosa, Kate, and Michael.