One speech sound can be associated with multiple meanings through iconicity, indexicality, and/or systematicity. It was not until recently that this “pluripotentiality” of sound symbolism attracted serious attention, and it remains uninvestigated how pluripotentiality may arise. In the current study, Japanese, Korean, Mandarin, and English speakers rated unfamiliar jewel names on three semantic scales: size, brightness, and hardness. The results showed language-specific and cross-linguistically shared pluripotential sound symbolism. Japanese speakers associated voiced stops with large and dark jewels, whereas Mandarin speakers associated [i] with small and bright jewels. Japanese, Mandarin, and English speakers also associated lip rounding with darkness and softness. These sound-symbolic meanings are unlikely to be obtained through metaphorical or metonymical extension, nor are they reported to colexify. Notably, in a purely semantic network without the mediation of lip rounding, softness can instead be associated with brightness, as illustrated by synesthetic metaphors such as yawaraka-na hizashi /jawaɾakanaçizaɕi/ “a gentle (lit. soft) sunshine” in Japanese. These findings suggest that the semantic networks of sound symbolism may not coincide with those of metaphor or metonymy. The current study summarizes the findings in the form of (phono)semantic maps to facilitate cross-linguistic comparisons of pluripotential sound symbolism.