I can either have smudges on my material an label, or, not on the material or label.
I cant seem to have it on the material but not the label.
Any ideas?
Sure. Use the label as an opacity map for the smudges. There's more than one way to do this, but basically you need to make the smudges invisible outside of the label. Using blending modes, the color composite node and the label texture node and your smudges node, this should be possible.