I think I understand it! I got 400! If the 40 dyed ones were replaced and you pick out 40 and 4 are dyed then that's 4/40 which is one 10th of the dyed ones, so 36 is 1/10 of the population of dyed ones so there's 360 non-dyed ones and 40 dyed ones so there are 400 fish :D Is that right?
But 40 were originally taken out so it can't be 36. So I added 36 to 40 and got 76, but the answer was above 300 :S I know, it's a very confusing question, but if 4 are already dyed this suggests the dyed ones were put back in.