With all weight training you have have to build up to the weight to protect your joints, tendons and muscles. Doesn't matter whether your using a vest, sand, water, iron or resistance bands.
With a weight vest the pressure is on the tops of your shoulders pushing down. Here its pressing on bone - the strongest part of your body.
With a ruck sack it's pulling on your shoulder joint, both backwards and downwards. Its the backwards bit that can do the damage, especially if the straps are loose and the weight is low hanging around your arse. (Think of how school kids wear their rucksacks). If you have a cheap rucksack this is especially true as the straps are rubbish, thin and unpadded. You also don't have chest and waist straps. The chest strap is important for helping resist the backwards pull of the straps and evening the load out.
Military bags are very well designed, good straps and you keep everything tight. I'm sure you build up the weight to. Consequently you can carry much more.
So you are right in most of what you say, that it doesn't matter so long as you build up to it. But if you are using a rucksack the quality of that is much more important than any of the others due to how it sits and acts on you.