You will be presented with many doors in your career. You don't always have to walk through them, but I'd advise you to at least consider each one. Maybe poke your head through and see what's there. If someone from an agency you don't think you want to work at wants to chat, why not at least chat?
Thinking back over my career, I can think of several opportunities that I completely passed on without a second thought. Some were offers made by people that I didn't really take seriously. When I think of these passed opportunities, I'd like to go back and kick my younger self.
Here's an excellent example of how this can work, by Jason Friedman at 37signals.