Does SSL affect performance of high traffic websites?

Is it true that I will lose the benefits of caching and see performance bottlenecks if I switch to site wide SSL for my website?