Your Youth Pastor would have been referring to the story/parable of Lazarus and the rich man. When the rich man asked Abraham to send Lazarus to dip his finger in water and cool his tongue, Abraham answered and told him that people could not travel from either side to the other. Now, note two things:
1. This was referring to being able to move from the abode of the righteous dead and the abode of the unrighteous dead; it has nothing to do with anything connected to our earthly existence.
2. While they could not travel back and forth, they obviously COULD see and hear from one side to the other, so comunication was possible.
Your warnings are valid; too many people think that all ‘spirits’ are benevolent, and that all ‘angels’ are God’s angels. We can open ‘doorways’ to the spirit realm, sometimes deliberately, and sometimes unwittingly, and subject ourselves to oppression and worse, regardless of our original intent.