Finding frustration with food

I have spent the last few months frustrated. The weight loss is just not happening despite eating healthy at least 80% of the time and exercising regularly. I have recently been reading the Primal Blueprint and trying to implement his principles. He advocates clean foods as does The Eat Clean Diet, but he eliminates grains and limited dairy. I stumbled upon his blog and book trying to find recipes for gluten free, dairy free. I have been told several times that it appears I am intolerant to gluten and possibly dairy. Well I eliminated gluten and it is in fact a trigger for me and liquid dairy appears to be as well. I am still out on cheese as I have just been sticking to goat cheese. I wish I could say I have stuck to his eating style 100%, but I haven't. Regardless of gluten and dairy making me feel like crap I keep eating them. It is only a little on weekends, but alas it is still there. I still am not losing any weight and can't help but think it is these dang little cheats that are causing it. I will get down a few pounds and then put them right back on the weekend after a cheat. So I seem to KNOW what my problem is now if I could just knock it off. It is so hard when the family wants to go out to eat and all my favorite foods are there. I think I won't feel THAT bad and have a small amount. I don't feel horrible and I guess that's why I keep doing it, but than I get on the scale and ARGH!! Summer is really hard because we bbq a lot and while having a burger without the bun doesn't bother me I really love pasta salads. Oh and smores! So this is really just a rant and trying to get my thoughts out there.