I think you could probably get by with just having the stalks as long as you're not caught in the process of stripping them of leaves and or buds...by the cops or the "owner". Most people wouldn't recognize one fibrous woody plant stalk from another anyway so if you gathered some of the hemp stalks and grouped them with some mullein stalks, some teasel stalks, I doubt anyone would even notice. All of these, by the way, have great properties for friction fire by hand drill, especially the hemp.
If you stumble into someones pot patch just be damned careful. Back in the days between the dwindling popularity of making moonshine and the growing popularity of making meth I used to wonder into them some times in the mountains here...the first few times I did so while wondering who had just butchered a damned skunk. I was young and dumb and only recognized it in it's dried form back then. I only put two and two together when I found patches rigged with trip wires and punji pits around the perimeter. I was lucky I didn't step all the way into the first one and was wearing issue jungle boots with steel shanks in the sole. They were lucky too because if I had been hurt I would have set up an over-watch on that patch from a ridge 300 meters away with a Remington 700 and 50mm of glass on top, and scared the shit out of somebody. But, since I wasn't hurt I thought it best to just get the hell out of dodge before the situation really went south.