But now police think they know where it's coming from.
Investigators say they found drugs and weapons inside the home at 902 Franklin Street when they raided it Thursday night, but that the drugs didn't always stay here.
Sometimes they migrated across the street to the Timberland Library.
"We did notice needles around library grounds," said library manager Donna Feddern. "Sometimes in the morning we'd have to pick those up just to keep this a safe place."
Police say the bathroom there became a safe haven for heroin and meth users.
"Heroin use inside the library is obviously very unusual and so we were glad to hear about it quickly so we could get on top of it," said Laura Wohl with Olympia Police.
Police arrested five men and one woman, and three could face felony charges.
Two of the suspects appeared in court Friday afternoon. A judge set bail for Dean Fulks at $2,500 while Laurie Pfeiffer is set to be released on her own recognizance.