Can social services reports affect your family years later?

[Dilbert]

Suppose, just hypothetically, that one of your children is 90% deaf, or autistic, or some other abnormality which requires you to behave in a way that some people might misconstrue as abuse. You might have to yell at them sometimes, or hold them while they struggle and scream, or whatever. And if strangers sometimes call social services to report the perceived abuse, investigation reports accumulate. The social workers come to visit you,. write the incident off, and go about their business. But their reports accumulate, year after year. And some of them aren't very well written, and contain vague statements that could be misunderstood.

My question is whether those reports can somehow be misused against you. Such as if you're in a divorce/custody case, or if someone is just looking for dirt on you, for whatever reason. And if so, how many years later can it happen?
Or what if at some point in the future a dumb/misguided social worker decides all those reports justify taking your kids, just to be sure they're safe? Because even if most of the reports are clearly false alarms, some of them might not be so clear, because they're too vague and poorly written.
And another part of the question is whether there is anything you can do to get the reports deleted after a certain amount of time, so they don't keep accumulating.