By no means is Cal Poly's drug scene known far and wide. Surely, if you're looking for drugs, you'll be able to find them, but it's extremely easy to avoid exposure to the entire drug scene. This is probably due at least in part to the fact that Cal Poly students spend a lot of time studying and are generally in challenging classes that require their full brain to be working. That isn't to say that Cal Poly students don't make time for fun; it's just that when they do, they generally choose to drink alcohol.
It's incredibly reassuring to know that peer pressure is not generally an issue. This is obviously contingent upon the people, but students will generally respect others' wishes to either participate or not.