Of course you need to be concerned about more than just the cache, it's only one question. The idea is to make sure that the candidate understands the idea that the computer isn't just a black box and that how you program it can have a huge effect on performance. There are plenty of different kinds of performance hazards and they tend to vary pretty drastically depending on the architecture you are on. For example TLB misses, branch mispredicts and any other kind of funky penalties a particular machine has.