Whitman, button-wood trees, and missing poems

My great grandmother used to tell a story of the crazy old man Walt Whitman, who walked around Camden, NJ. The kids, of whom she was one, would make fun of him, and throw things at him from the button-wood trees as he walked past.

When I was in college, I found a reference, in one of Whitman’s poems, to kids throwing things at him from the trees, and I thought it was pretty neat how my great grandmother’s story was corroborated by something in this great poet’s writings.

Well, this week my mom asked me what poem that was in, and I went looking for it. I’m sure it was somewhere in Leaves of Grass, but I can’t find it. I’ve done full text searches of the entire book, and I can’t find anything that comes close to matching my memory of the mention. It was only a single line in a poem. But I’m starting to think that I made it up, or that I imagined it.

Any Whitman scholars out there who can point me in the right direction?