As surveys of galaxy populations at high redshifts progress, it becomes increasingly urgent to understand how observed galaxies at high redshift map into those at lower redshift. In this talk, I investigate how and where stellar mass builds up over cosmic time by showing recent studies of the changing star forming properties of galaxies as a function of stellar mass and environment from redshift three to present.