I was thinking about how similar neanderthal and our DNA is, and wondered if there were more stark epigenetic differences between us that could explain some of the reduced viability of hybrid offspring.
In any case, it got me wondering if, in general, epigenetic changes like DNA methylation were in part responsible for reproductive isolation and the beginnings of speciation, i.e. differential methylation that gives positive fitness for two different groups in the same species but negative fitness when hybridized could result in less successful crossing between the groups.
I found a paper that I think supports this idea, but I'm not an expert in this field and also don't know how much of an outlier this paper is. Also, even if this happens, I don't know how common it is or how big a role it is thought to play in speciation.