Is AI Evolving from Helpful Assistant to Permanent Spy?

“Their power derives from memory, and memory is where the risks lie.” - Kevin Frazier and Joshua Joseph

​Here’s a quick news item that will come as a surprise to absolutely no one, except perhaps for hermits who have been living in caves since AI went mainstream in 2022. Two new pieces of reporting, from Stanford and Tech Policy Press, confirm the fresh dangers to privacy emerging from the AI frontier.
 
First to Palo Alto, where researchers evaluated the privacy policies of six frontier AI developers. You can check out the complete analysis, but here are the takeaways from the abstract. Spoiler alert – they’re not a win for privacy:

• All six AI developers appear to employ their users' chat data to train and improve their models by default
  
  
• Some retain this data indefinitely
  
  
• Developers may collect and train on personal information disclosed in chats, including sensitive information such as biometric and health data, as well as files uploaded by users
  
  
• Four of the six companies examined appear to include children's chat data for model training, as well as customer data from other products
  ​
  
• On the whole, developers' privacy policies often lack essential information about their practices, highlighting the need for greater transparency and accountability.

 
The Tech Policy Press interview with experts sheds some light on why “agentic AI” is so dependent on user information. Agentic AI refers to generative AI with the ability to act independently. Generative AI says things. Agentic AI does things. Both are built on the large language models Stanford studied.
 
It’s a logical evolution – think of asking a restaurant chef to give you his recipe versus having a live-in chef who plans and prepares them. But it’s all built on memory. The more AI is allowed to remember about us, the more effective it will be at meeting our asks. “The central tension, then, is between convenience and control,” the experts told Tech Policy Press.
 
We would add that if you think you’re trusting AI what to remember about your prompts and interests and what not to remember, think again. We’re really talking about trusting companies like the ones in the Stanford study – because they’ll be the ones licensing the AI. As of now, then, the fate of your data ultimately rests in the hands of others. From the interview:
 
“Who, exactly, can access your agent’s memories – just you, or also the lab that designed it, a future employer who pays for integration, or even third-party developers who build on top of the agent’s platform?
 
In short, these experts say, the stakes are these:

“Deciding what should be remembered is not just a question of personal preference; it’s a question of governance. Without careful design and clear rules, we risk creating agents whose memories become less like a helpful assistant and more like a permanent surveillance file.”

We close with a refrain that will be familiar to our readers – now is the time for common-sense laws that privilege personal privacy. Without it, these experts warn, AI will become a tool of enclosure rather than empowerment.

DONATE & HELP US DEFEND YOUR FOURTH AMENDMENT RIGHTS