You may have heard that if you have a slow site, you can be penalized by Google for that. There is some truth to that.
Short answer: Yes, make sure the pages on your site load under 2-4 seconds.
Long answer: Let’s break down the actual fact from fiction.