I don't want to spend money on gems to get the trees or animals. Does this mean I can't do the goals?

You can do all goals without needing to spend money. It just might take you a little longer.

Adventures from the keep sometimes give gem trees and animals as rewards. Truffles and Prime Cuts are sometimes rewards as well. You'll even get gems on occasion.

The Purple Cow and Sir Pigglesworth sometimes drop gems.

Bacon and Wool sometimes drop from the Stables.