I am assuming the rattling is from the rear DR.
The first thing that comes to mind is you didn't thread the chain through the jockey wheels correctly. There is a tab of sorts on the upper jockey wheel frame that some folks often run the chain over instead of under. But I would imagine you would have major issues in all gears in that case.
The next is...did you install all required spacers on the cassette?
I have two bikes that came with the R8000 group set. In both cases I downloaded the install & set-up manuals from the Shimano web site, assumed they were set-up wrong to start with (they were), and just started from the beginning, step by step. Properly set-up they are smooth, and a joy to ride.
Other than that something might be bent or damaged.
Hopefully someone with a whole lot more knowledge and experience than me will chime in...
Can you post pictures or video?.