Mark Balaguer's Entry on Mathematical Fictionalism

In a previous post about the book The Big Questions, I argued that the writer seem to naively and glibly treat a very serious and controversial current issue in philosophy (the "immutability" of mathematical truths). I looked up the Stanford Encyclopedia of Philosophy entry on mathematical fictionalism (written by Balaguer) and it has this passage ('CH' is Continuum Hypothesis):

For it could be that our conception of set is not entirely precise and, in particular, that it is consistent with both CH and ~CH, so that neither of these sentences is part of the story of mathematics.

I think Balaguer doesn't realize here that there is a much more radical and disturbing interpretation of that possibility other than that neither CH nor ~CH are "part of the story of mathematics". If both CH and ~CH are consistent with our conception of set, then that could also be interpreted as saying that our conception of set is inconsistent or that both are consequences of our conception of set (and thus by ex falso quodlibet, our notion, i.e., our "story" of just about all of mathematics is inconsistent). That would be disastrous for mathematics. How would we go about our mathematical business then? Should we give up and accept it or should we try to establish non set-theoretic foundations for math?