That is true, but then again hardly any specimen we find is going to literally be your great^n grandparent. When an animal dies, it rarely leaves around a fossil; so most or even all of what we've found is stuff that is closely related to, but is not exactly, our ancestors.

A visual analogy would be that we're an apple from the top of the tree; the fossils we find are apples from lower branches. Our apple didn't grow from theirs, but the branches we grew from did grow out of the branches that their fruit grew from.