There are 4 materia caves on the World Map and hold some of the best materia in the game inside. Here are the locations and what’s inside each.
Far south of Wutai, right at the end of the island, hidden behind a mountain, use a black or gold chocobo to get there. Go inside and examine the glowing light to receive the Quadra Magic materia. This casts a magic spell or summoning 4 times.
North of Mideel, on long island nearby. Go inside and examine the glowing light to receive the Mime materia. This mimes the previous command in battle.
Cave is next to the mountain lying on a square of brown land in the North Corel area. Need a black or gold chocobo to get there. Go inside and examine the glowing light to receive the HP<>MP materia which switches your HP and MP values.
Go to Goblin Island (its the elongated island with a forest on it to the northwest of the Chocobo farm). Ride a gold chocobo northwest from here. You will eventually come to an island that isn’t on the map. It is surrounded by mountains and trees. There is a forest in the middle. Dismount in here and enter the cave. Examine the glowing light to receive the best summon materia in the game – the Knights of the Round.