You need to kill every stick of that invasive bush honeysuckle,AKA autumn or Russian olive. Horrible stuff.I have plastic fence around the garden. Sounds like you have a very nice garden. We have a garden, plants and landscaping in a wooded area around the house. Even some " deer resistant" plants were eaten, along with some beautiful rose bushes.
You probably already know or are planning defenses. I put up some old Christmas lights on metal loop stakes to ward off the deer plant eating demons. It has worked. We will move larger items around such as the trash can or wheelbarrow to confuse and scare the deer off. Putting up some green plastic fencing here and there seems to work.
The deer bed down and even birth in the bush honeysuckle near the house. My wife enjoys that but would eat the deer if I hunted them. Everything else that is a problem is trapped or rimfired.