I used to feel the same way you feel now.
I picked up a BLACK - very high quality bathroom hand towel (because of its size), folded it in half, and have had it draped i over the center arm rest for about 2 years. After a week, you'll never notice that it's there. (Blk to match the interior)