The Fuji Gran Fondo 3.0 sounds like a nice bike as it is. Is it 10s or 11s? Did you get an old model?

Are you planning on doing the upgrades yourself, or having someone do them for you? If I was doing a scratch build, I might choose Ultegra over 105. However, I'm not sure the differences are great enough to justify rebuilding a bike that is currently configured with 105 just to install Ultegra.

If you can do your own work, check out some of the prices on groupsets, and perhaps wheels from some of the big overseas online dealers such as Ribble.

Are your racing, or just doing casual rides?

Anyway, I'd probably be a bit conservative, and keep the Fuji as-is, and simply replace/upgrade parts that went bad.