Even ignoring the privacy implications, the aggressive nature of the favicon cache annoys me. I have a personal site I like to have linked on my devices, including my iPhone. I made the mistake of bookmarking an early version without a favicon. Now, no matter what I do, short of a reset of the device I suppose, the bookmark in Safari has no favicon on it.
In the end it doesn't matter, but still, it annoys me every time I see it.