Kingdom Hearts fans!

I haven't played Chain of Memories. And I suddenly realized I need to know: are the amnesia powers of Castle Oblivion due to Naminé's manipulation only, or would anyone at all lose their memories inside, whether Naminé had been there or not? I got a mad scientist holing up in there, and he has his memories, and so does his prisoner. I was operating under the idea that with no one in the castle, and with Marluxia and company defeated, its powers are defunct. Then I also wondered that, if it still has those powers, if they could be manipulated by someone in control of the place or if the person would have no control over it. I really didn't want the memory loss stuff to play a part in the story, but if it is and will always be central to Castle Oblivion, then I guess I'll need to do something with it. Or else say that the scientist lied about their location.