I take it you missed the "put on some gloves" part.

In any event, yeah, the vast majority of the stuff comes off easily. The dried up stuff wants to stick to itself, so you let it. A lot of it can be peeled. If patches resist peeling, you flatten the carcass on a table or work bench and rub it with the flat of your palm and it'll roll up. You don't want or need abrasiveness - I only wear gloves on the off chance that there's something sharp embedded in the tire. The sealant has one job, to gum up air gaps - breaking it up with bristles and abrasives is just going result in a bigger mess while gumming up and clogging your bristles and pads... but you already learned that.