How do I restrict users from being able to view and book specific patients?

You can create and assign groups of patients to access groups. For more information, see here.