Here's one way to look a limited anti-realism of morality (more like a skepticism of current systems) summarizing my views that is starting to look more plausible to me though I am still quite doubtful.
What if most of the ethics or morality we know in philosophy or common sense ethical reasoning are seriously shaped by our building of fictions based on ethical intuitions? That's not to say that necessarily these
intuitions are wrong (error theory on these intuitions). Even if they are right these intuitions may be based on one or a few paradigm cases and though have some truth in them, small inaccuracies in the way morality is built up from them may magnify and cause ever more fictions to be built on top of earlier formulations. These fictions may in turn cause frictions with either each other or other aspects of what we know. That's not to say that harmonization of these fictions with each other and other aspects of our knowledge are not possible, it's just to say that any harmonization may be in some sense arbitrary and based on fictions.
So though we may possible eventually have a coherent and consistent moral system, it may be built on lies. The more elaborate, complex the society we live in, the more opportunity for differing intuitions. Novel situations and complex cultural milieus present more opportunities for intuitions to differ with each other since intuitions are a product of natural and cultural influences. When intuitions come in conflict we must invent fictions to harmonize them. The realization of moral luck and the truth of determinism may bring this idea out. (These are just two examples but I suspect many other notions in ethics such as rights may be susceptible to this kind of skepticism.)
As one example could it be that our intuitions regarding moral responsibility and especially contempt and its opposite, praise of moral character, are based on fictions we have of each others characters such as that these character traits are relatively permanent (maybe even across possible worlds) and rather consistent with other character traits of the individual? We may have invented conceptions of personhood as enbodying relatively permanent and consistent character traits based on justifying our practices of attributing blame and contempt, eg such as fictions regarding people's character or personality. It may also be based on the fiction of alternative possibilities assuming that it is a fiction (my point is that we don't know if determinism is true and so my calling it an invented fiction may be justified on that epistemic lacuna). As we know more about the world, we would have to come to ever more elaborate fictions to justify our practices or else risk not being able to justify them as they conflict either with each other or with other notions. If determinism is true, we may have to change our conceptions of what it means to be a person (i.e., come up with further fictions which in turn may or may not come into conflict with either other moral intuitions or what we will learn in the future about the world).
Let's say that one day we were to learn that the Nazis are very much like us. That is, that had most Nazis been raised in different environments like ours or we had been raised in theirs, they would have turned out very differently and not been subject to the contempt we have of them and may even be the objects of praise. Consider Hitler; it's possible that had he been raised differently, he might have turned out a good person. We may have invented a character type for him which is false and his fellow Nazis. Since many of our reasoning may be based on using paradigm cases of evil and these are our paradigms, many of our reasoning may be contaminated in a sense.
In fact, there is good evidence now that most people are not that different from Germans living in the 1930s. Since many of our paradigm cases of evil (as seen in examples of moral depravity) are Nazis, how will that square with the reality of Nazis being like us. We may have invented (at least implicitly) an evil aura or essence about Nazis. Further moral reasoning may be based on this fiction and thus be fictional themselves. It may be the case that Nazis displayed evil but it may also be the case that our vision of their souls as stained etc are not wholly justified in robust ways that is relevant to all our ethical beliefs implicit and explicit.
This is analogous to the fictions we sometimes invent (as the story goes) in mathematics such as inventing the fiction of sets, etc. Now it may the true that some of our “fictions” may actually turn out true in which case they would not be fictions but it seems that they could as well turn out to be fictions and the long history of ad hoc rationalizing and creating of fictions needed to form coherent pictures of either morality or mathematics may count against them as such. If they turn out true, that would be coincidental.
However, there may be moral facts out there. It's just that they may not square well at all with our moral common sense or reasoning as these maybe the products of our fiction building as well as our intuitions. Moral facts and a truthful moral theory may be far more nuanced and unrecognizable from all of our notions.
Some people may be worse than others in more deep ways than moral luck can damage. If we look at all the possible worlds in which a person inhabits, we may have a way to assess their moral worth by comparing people with each other. If person A turns out as a bad person in more of the possible worlds they inhabit than person B does, as measured by averaging out all finite subset's of possible worlds in which they inhabit, we may have an way of assessing their moral worth on some sense. But here we will have the usual problems with indeterminacy and weighting problems but that is where the problems should be located. The kind of anti-realism described here is not a complete anti-realism because it still leave open the possibility that there is a non fictional moral reality (facts of the matter) that can be made out of our intuitions and that our intuitions (at least some of them) are accurate or true. It merely suggests that the system(s) we all have now are fictions.