The texture would actually give the stem less surface contact on the adapter.

Do forks have a textured clamp area for stems? no. so why would the adaptor need to?