It’s not just Google photo’s. Apple is doing the exact same thing. These scanning processes are not full proof, and can have very severe consequences. Scanning billions of photos will always lead to mistakes.
Another reason why I moved all my cloud services off iCloud, and onto my self-hosted Nextcloud instance and other open source / more privacy friendly solutions.
The other addition that is worrisome as well, in regards to the behavior of Google afterwards. I can’t recall if this was one in the first or second story, I will have to read again to verify. But in one of the stories, the father was a paying G Suite member and still was unable to speak or get anything resolved. In this day of age, “too many eggs”, if even a ‘paid’ user has difficulty, then where should someone setup email, file-share, etc. I have been trying to transition away from gmail to G Suite (or anyone) with a custom family domain. But now, maybe G Suite isn’t the way to go (not just because of this) but always in the ‘afterwards’ part, can’t imagine being locked out of my account that literally holds the last 15+ years of my data currently.
This is not a part of this discussion, but if you want to find out, open a new discussion, and you will get a lot of ideas how to do this, without getting in touch with companies like Google or similar.
This was always a key concern when I administered AWS servers. For me, the biggest risk of service loss was not being hacked but an account issue - where Amazon took umbridge to data being stored or patterns of usage and closed the account. (We were legally storing copyright films and images, including some from Disney.)
Here I was able to mitigate by (at some considerable expense) backing up elsewhere, although services would have been disrupted for quite some time. The point remains, however, that these large providers can simply turn off accounts at will without any appeal process. While I get the concept that it’s their business and they can do what they like, it potentially results in disproportionate impact on the customer due simply, as in this case, due to a misunderstanding by an algorithm. Do they have no liability to ensure their algorithms don’t do undue harm?
I agree. I wonder how other sites handle this?
“Tech companies including Microsoft, Google, Facebook and others have for years been sharing digital fingerprints of known child sexual abuse images. Apple has used those to scan user files stored in its iCloud service, which is not as securely encrypted as its on-device data, for child pornography.” - npr.org 8/06/21
That’s why I thought it was interesting regarding the user in question who was actually paying for the email service. There’s a TOS involved, and Google banned them for a supposed violation of that TOS - but there was no appeal process.
The thing is, you could have a flagrant violation of the contract / TOS, and the user’s only real recourse is to sue Google for a contract violation. And that’s probably beyond the average person’s ability to afford.
I would like to believe that Apple would look at a police report concluding “nothing wrong”, and work with the user. I could be very, very wrong. But Apple at least has a phone number and infrastructure for dealing with paying customers. Google, not so much.
I would suggest that you have some sort of a separate backup of critical data, also shuttled off to a service like Backblaze for backups. Backblaze is end-to-end encrypted (if you set it up that way), so theoretically even if you had a problem with something like suspected CSAM or other content that they found objectionable, their servers couldn’t scan for or ban you because of it.
And it’s not tightly coupled with the rest of your digital universe, so “my email provider shut me off” wouldn’t also equal, “and all my cloud data is gone”.
That’s always a good idea. I’ve been using Gmail since the private beta and I also have a Google Workspace account for my private domains. All my files sync to my Mac which is backed up locally and to Backblaze B2. And I schedule Google to create automatic downloads of all my data, except photos. (My photos are synced to Apple in addition to Google).
I believe in a “belt and suspenders” approach to protecting data.
Arguments of the form X is harmful; Y can lead to X, so don’t do Y are rarely useful without a broader context and a more wholistic understanding of risks.
In this case it seems that the risk to the child is what’s in question: A child in medical distress needs help during a pandemic in which face to face interactions are necessarily limited. In-person medical care is limited and creates further risk for everyone, so the doctor has requested photographs to help decide if the issue is serious enough to warrant it. Photographs of this sort are an additional risk to the child, as they could fall into the hands of people who would misuse them. Unintended exposure of the photos is more likely for sufficiently technically naive people, but that’s nearly the whole population these days.
Not taking the photos also puts the child into imminent risk of either a poor diagnosis, or of exposure to COVID for an in-person visit (if that were even possible). (We can take the pandemic out of this and replace it with a child in a remote enough location that they could not access in-person care and the argument is more or less the same. Where I live, this is a not unlikely scenario, and one that baffles my European relatives).
One thing that distinguishes this case from more typical CSAM cases is that the child was not harmed during the production of the photo. That kind of harm is a major (but not the only) component of what’s terrible about CSAM.
The parents and physician have to weight the potential harm to the child of either course of action (photos nor none) and decide which presents the least potential for harm to the child. In this case they chose the remote option with a photo. Maybe it was the right choice; maybe it wasn’t, and maybe there was no completely “right” choice.
However, to conclude that you can keep yourself from experiencing this kind of issue by simply never taking such pictures is harmfully reductive. A better takeaway from this incident is that Google (or any similarly run company) can effectively cut you off from its services at any time and for (almost) any reason, and you an find yourself with little effective recourse. That is something that you should consider very carefully when such services underpin many important aspects of our lives.
Curious if you ever discussed this with AWS and what they said. We don’t host Disney-level potentially infringing material but we do talk through our architecture and use cases with them. We pay for a support tier, too, so would have quick remedy if accidentally suspended.
This. So much this. Also noting that figuring out what to do in a situation like this in retrospect is easier than figuring it out in the moment, because in retrospect we effectively can have perfect knowledge of consequences, alternatives, etc.
The hugely-important part of this is the “(almost) any reason” part. And that includes not only potentially-spurious interpretations of existing rules, but reasons that may not even be valid reasons now (legalese typically worded, “at Google’s sole discretion”).
It’s impossible to do research and risk mitigation if part of the agreement is, effectively, that one side can unilaterally change the nature of the agreement at any time. You have to implicitly trust your provider to not do things like that, or to figure out a way where your system is diversified enough that it’s a non-issue.
Agreed. These provider accounts (Google, etc.) are colossal “central points of failure” for somebody’s digital life. Worth noting, and attempting to act accordingly.
One option is to create a second free (gmail, outlook.com, etc) account and set your primary account to always forward a copy of your emails to it as a backup. And if you are concerned about your photos causing a problem upload them to the “backup” email account instead of your primary. That way your photos should never trigger a lockout of your main account, and if something else happens to your primary you still have a backup.
I don’t think it’s beyond Google to figure out the association between both accounts and lock the other one as well.
It also won’t solve the issue that your primary email address - the one you’re actually giving out to people - may be locked and you‘re prevented from receiving future mail on it.
I would look at this at a higher level not just about CSAM
I have long repeated my belief in the adage “If you are not the customer than you are the product.”
Thus I would never consider free software for any mission-critical business task, nor would I consider it for any personal use if I would be upset if the service suddenly disappeared.
I suspect this would have been resolved in a much more acceptable fashion if the user were a Google Workspace customer - $6 per month gives you a phone number to call and lots more
I already moved away from everything Google (except web search), Social Media, WhatsApp,…
While “protecting children” might be well intended, searching every file/message for “unwanted content” is a slippery slope. The potential for misuse is undeniable. And if history has taught us anything, power is often misused. Secrecy of correspondence was, for a long time, a basic human right and written into the laws and constitutions of many countries. And now, with the arguments of “terrorism” or “protecting children” we are willingly creating tools of mass observations. Trying not to turn this into a political debate, everybody is entitled to his stance on any topic. But if somebody proposed “let’s open every private letter”, because it MIGHT contain something “wrong”…that person would have been tarred and feathered.
I am, and always was, an early adopter of technology. I started using Facebook when it was still unknown in Europe. I started visualizing bike rides in Google Maps/Earth, when there were no apps for that (just GPX files),… and I love and enjoy our technology. But I find myself rejecting more and more services, applications and use-cases. And while it has been argued in this thread that Google is a “private company”, don’t forget that no nation state has more data on us than Google/Meta.
I don‘t see a 6$ monthly fee making much of a difference when they‘re locking an account as a matter of principle and policy.
For me, this affair underscores what I‘ve always thought:
If your files - or photos and data - really matter, you shouldn’t store them (unencrypted) on someone else’s computer. Especially when, in the case of online storage, that someone is often subject to no more stringent (legal) privacy and security requirements, regulation and supervision than your next-door neighbour is.
I basically treat these services the same way as Facebook: wouldn’t store any photo there that I wouldn’t be comfortable sharing on facebook.
Sure, but why would google lock out a second account? And what business would risk losing access to their Google Workspace account because one of their employees did something that got him locked out of gmail?
Apple may choose to lock out a doctor or health professional for violating their terms of service. Anything is possible.
“If you are a covered entity, business associate or representative of a covered entity or business associate (as those terms are defined at 45 C.F.R § 160.103), You agree that you will not use any component, function or other facility of iCloud to create, receive, maintain or transmit any “protected health information” (as such term is defined at 45 C.F.R § 160.103) or use iCloud in any manner that would make Apple (or any Apple Subsidiary) Your or any third party’s business associate.”
Though I wouldn’t be sure about the U.S., a country that Google/Facebook likely provide all data to (willingly or unwillingly).
Guilt by association, otherwise known as precaution.
Amazon has done it in some instances, AFAIK.
Yet another reason to own your own domain.
By what I read, one of the users was a paying customer.
These days, I think sometimes it’s not as much about actual choice as much as it is about certain behaviors being enabled by default, or set up by a well-meaning colleague / friend / family member.
Not the same, but IP “neighbors” get banned in email blacklists all the time. One guy sends a bunch of spam from a Linode (which doesn’t tolerate spammers!), and the blacklists clobber an entire Linode network allocation.