Baker began on drums, but switched to trumpet during his teens. He started his career on riverboats and played with Don Redman in the mid-1930s. He also worked with Teddy Wilson and Andy Kirk before joining Duke Ellington. He married Kirk's pianist Mary Lou Williams and though the two separated shortly thereafter, they never officially divorced.
Baker worked on and off in Duke Ellington's Orchestra from 1942 to 1962. He also worked with Johnny Hodges's group in the early 1950s during the period when Hodges was not a member of Ellington's orchestra.