It was founded in 1854 as Wisconsin Female College under the auspices of the Wisconsin Baptist Convention to prepare women for missionary service. It was poorly funded, and was reorganized in 1863, with control passing to the village of Fox Lake. For a while it operated as Fox Lake Seminary and was co-educational. In 1883 Judge Jason Downer of the Wisconsin Supreme Court, a strong advocate of educational opportunities for women, died and left the college $80,000, putting the school on a firmer foundation.
In July, 1895, Downer merged with Milwaukee College to form Milwaukee-Downer College under the presidency of Milwaukee College's Ellen Clara Sabin. A new site was purchased on a tract of about ten acres on the northern end of the city of Milwaukee, about half-way between Lake Michigan and the Milwaukee River.