Help me kick iCloud into gear (stop me getting angry at Apple)

I have an incredibly annoying iCloud problem that Apple seem incapable of fixing (if they’re even trying). I’ll spare you the rant (well, mostly) that I’d love to make about trying to get Apple to sort it. I am hoping the clever hive mind here can suggest some things I haven’t tried.

I have a MacBook Pro (MBP), Mac mini (MM), iPad mini, and iPhone 11 Pro. The Mac mini is on the latest Catalina version, the others are all on the latest current OSes, though as I will explain the releases are probably immaterial.

I have iCloud Drive turned on everywhere, and I have “Store Documents and Desktop in iCloud” turned on on both Macs.

iCloud Drive works flawlessly, except: Any file that is created or modified on the MBP will not be synched up to the cloud. If you change any of the bolded terms, it works fine. Folders can be created, changed, deleted anywhere and sync everywhere. Files can be deleted anywhere and sync everywhere. Files can be created or modified anywhere except the MBP and they sync everywhere including to the MBP. “Anywhere” includes iCloud Drive on the web (including on the MBP). The successful syncing of any of the ones that work occurs in ~5 seconds to all devices. It seems the issue lies in the MBP sending data to iCloud. And when it “fails”, it usually looks like it is constantly trying but it never actually gets anywhere. I’ve even left it untouched for >24 hours and still nothing.

Here’s the fun part where I tell you what I have already tried, or at least, what I know doesn’t work. Obviously the focus is entirely on the MBP.

Formatting the SSD and reinstalling the OS does not fix it. Installing a different version of the OS does not fix it. Turning iCloud Drive off and on does not fix it. Turning iCloud Drive off for >24 hours then back on does not fix it. Changing networks (wifi in multiple locations plus Ethernet at home), uninstalling VPNs, starting in Safe Mode, and booting off an external drive do not work. Selling the laptop and using an iPad for 6 months, then buying a completely new laptop that uses a different CPU and an OS you’ve never touched before does not fix it!

OK, I should explain that last bit. I originally hit this problem with a 2016 MBP I bought refurbed in 2018. It happened late in the Catalina beta cycle and never went away through all of the above machinations, most of which were at the direction of Apple Support/Engineering. I even installed High Sierra (original OS for the laptop) and it was still there. After four months of no more than this (seemingly endless) basic troubleshooting and (seemingly endless) log collections, it was going nowhere and I gave up when Apple claimed they tried to contact me but my phone never rang. I wiped and sold the laptop and bought an iPad Pro.

Life (and iCloud) was great until the M1 Macs landed. I couldn’t resist. I sold the iPad Pro and bought an M1 MacBook Pro (now taking the MBP role in this story). After days spent setting it up as my primary computer (taking over from the MM) I realised the problem was back! And it is exactly the same. I had told an Apple Senior Adviser previously that I thought the problem surely had to lie in iCloud itself but he told me I wasn’t correct in that assumption. I think I am vindicated now! (Though I am flummoxed as to how it does not affect my Mac mini!)

I also found a few incantations that involved terminal commands and careful sequencing of steps to try to force some iCloud process to “re-evaluate” but nothing has worked. The problem maintains its exact same behaviour.

It’s one thing to use the MM as a workaround to get the files in the cloud, but apps like Ulysses are affected too. I basically cannot use Ulysses on MBP without resorting to “outside of iCloud” files, which lose me both the simplicity of Ulysses and some features. I opened a new call with Apple in December, but after a month with no response to my last email I rang Apple again earlier in the week and managed to get an update that “they’re still investigating” but I just don’t believe that’s true any more. I have another call scheduled in a week on which I expect to hear the same.

Hive mind, what can you suggest that I may have missed? Turning anything off and on has already been tried. I have already refused to do those actions again with Apple on account of all I did last time and the identical nature of its reoccurrence. It would just cause me more effort for, I am certain, no gain.

Do you have anything like Little Snitch or TripMode installed? It sounds to me like this could be a network issue, although it shouldn’t be.

1 Like

That sounds a lot like “the operation was a complete success, but the patient died”

It’s been my experience that some parts of iCloud work well, primarily photos and app syncing. I rarely have a problem with contacts, calendars, Drafts, etc. staying in sync. OTOH, after numerous attempts to get iCloud to keep standard files and folders synced, I’ve given up.

Apple has 11 data centers of their own and also uses Amazon AWS and Google for iCloud storage. I’ve been storing files on Google and AWS for many years and have never had a single problem so I suspect the problem is the way Apple manages file storage.

Apple introduced MobileMe in 2000 and iCloud in 2011. If they were really interested in making iCloud work as good as their competition they’ve had more than enough time.

IMO, it is time to keep the parts of iCloud that work for you and find another cloud storage provider for the rest.

3 Likes

Sadly, having also experienced periodic iCloud syncing issues with Apple Notes over several years and after extensive trouble shooting, including many listed above, and too many calls with Apple tech support I gave up on AN and moved all of my notes to Craft. Fortunately, I have not experienced iCloud syncing issues with other apps. :crossed_fingers:

2 Likes

Apple Notes! After many failed attempts to use AN, I gave up on it too. Apparently I had completely exorcised that painful memory. Or perhaps I just shouldn’t be posting at 3AM. :wink:

1 Like

These might be some of the same Terminal commands you mentioned trying, but a few months ago I had a similar issue, and, unlike past problems, it stuck around over the course of a week or two, so I tried a bunch of the stuff here:

My Desktop and Documents eventually started syncing, although I don’t know if it was anything I did, or just the passage of time.

And, yeah, whereas I’d love to drop Dropbox, unfortunately iCloud still has too many occasional issues like this for me to trust it.

1 Like

When dealing with any cloud based services always compare what is on the “cloud” and the connect devices. This will show immediately where the bottleneck is.

In your case: open iCloud.com in the web browser.
Change a file on your M1 mac and see if it propagates to icloud.com
Next verify if the file shows on the other devices once it has shown in icloud.com.
Repeat the steps in the reversed direction.
Ultimately you can create a file on iCloud.com and see if and where is appears.

Going trough this systematic approach you can quickly narrow down where the problem is and target your efforts.

1 Like

Thanks. I don’t remember trying the approach in the top answer there and it’s pretty simple to achieve so I think I will give that a go after I get the call next week to say they’re still working on it. Just in case they actually are working on it.

1 Like

I do not, but that is an interesting point. It is consistent across the wifi at work and my wifi and Ethernet at home. There was an answer in the post @Evan referred to that mentioned wifi conflicting with Ethernet so I have turned off wifi for a bit, but it has certainly survived wifi-only use at home and work. The work wifi is a wildly different network than at home, covering a large building and with plenty of security features.

The (multiple) networks in use are a consistent factor between the two laptops that had/have the problem, but the Mac mini also uses both home networks and has never put a foot wrong.

Wracking my brain the only two factors I can think of that have not been varied at all are:

  1. My iCloud account (I already told Apple I’m not going to use a different one because it is extremely disruptive to the computer).
  2. There are two Macs using the account… I’m now wondering what happens if I turn off iCloud on the Mac mini, though I am reticent about that because it is currently my lifeboat.

When you got the M1 Mac did you test the syncing before loading any other software? Just wondering, like Rosemary, if there could be something you have on the MBP that’s not on the Mini that might cause a problem.

Have you tried creating a new user account and testing iCloud from there? That might be able to rule out anything regarding the user profile.

1 Like

Sounds like you have tried a lot of different options. Have you tried safe mode as well?

I have tried twice to use iCloud as a Dropbox substitute but the syncing for anything in the files app was flaky to say the least. App files even external ones seem to sync fine as I believe they use a different system to the one users use.

Rule of thumb seems to be iCloud is fine for long term rarely changing files but if you change files regularly and need immediate sync then iCloud (file storage) is not for you.

As I was working on code files (basically small text files) on my MacBook Pro while in the coffee shop I would often get home a couple of hours later and find they had failed to sync to the always on macMini, they synced eventually but I could discern no fixed schedule or condition.

Back to Dropbox (without all the desktop app clutter) whose sync I have to say is pretty flawless and almost instantaneous even on shared folders.

2 Likes

I have to agree with the dropBox idea - having used multiple Macs phones and iPads for donkeys years - iCloud won’t cut it at all, it seems to regularly have issues file syncing - and is a pain if you do a re-install, it seems to decide if it’s going to sync or not, and only when it’s good and ready, on the other hand DropBox performs flawlessly at all times - shame really, as iCloud is pretty good with Photos and calendars etc. - and seems that Apple seem to deny there is ever a problem :frowning:

Not something I had thought of, but I would think I have (or at least had when I identified this) less software on the M1 by virtue of it being brand new. I know on the old MBP I did clean installs with no third party software at all and it was still happening.

This was carried out on both old and new laptops and still the problem persisted.

I do not recall if we tried it on the current MBP but it was most certainly tested on the old one. I should try it again on the new one to be sure.

That’s the crazy part. Every action that does work is syncing within 5 seconds! But I’d take slow syncing over no syncing any day.

I went off DropBox a while back when they messed with the app, plus it’s a cost I’d really rather not wear when I already pay for Apple One and get 200GB which is pretty much all I need. It might be more tempting if they offered something between 5GB and 2TB.

Thank you, everyone, for your thoughts. I still have a couple of things to try. I’m not confident at all they will work, but it beats sitting on my hands.

So I tried the terminal-fu and it seems to have done nothing at all. The documents didn’t even disappear.

The CloudDocs folder was rebuilt but with less content. Only the sessions directory.

I tried looking at brctl and it seems mine is regularly spouting… stuff… into the logs. Next I tried a carefully timed file create while watching the log and I think I see where the issue might lie, though have no idea what to do about it.

You can see the block that shows upload, needs, 1 which must be the file I created. Then I see this:

Progress: start tracking upload for doc 135466 because document is needs-upload (versionSize: 71610, transferSize: 75190)

The versionSize number exactly matches the size of the file in the Get Info window, so it’s trying to send it. But then…

Sync-Up[<private>]: finishing sync up

Rescheduling upload for <private>

I had let the log run until it paused then stopped. Searching what was dumped in that time I could only find 3 more mentions of doc 135466 which all read like this.

revisiond[364] -[GSClient getFileIdForDocumentId:onDevice:reply:] query(135466, 16777234)

So it seems it sees the change, attempts the sync and then for whatever reason fails and decides to do it later, but later never comes.

[dbg  2021-03-07 15:42:51.312+1300] bird[65794]   o Got coordinated read
[dbg  2021-03-07 15:42:51.312+1300] bird[65794]  ┃  Obtained coordinated read at '<private>' in 0.006283 seconds
[dbg  2021-03-07 15:42:51.312+1300] bird[65794]  ┃  ┏  acquired coordination for (reading)
  at '<private>'
[dbg  2021-03-07 15:42:51.313+1300] bird[65794]  ┃  ┃  openat(22, '<private>') failed [2: No such file or directory]
[dbg  2021-03-07 15:42:51.313+1300] bird[65794]  ┃  ┃  ┏  match lookup with disk <private>
[dbg  2021-03-07 15:42:51.313+1300] bird[65794]  ┃  ┃  ┃  Considering the live item
[dbg  2021-03-07 15:42:51.313+1300] bird[65794]  ┃  ┃  ┗  end
[dbg  2021-03-07 15:42:51.313+1300] bird[65794]  ┃  ┃  ┏  reading <private> at <private>
[dbg  2021-03-07 15:42:51.313+1300] bird[65794]  ┃  ┃  ┃  ┏  diffing item <private>
against <private>
[dbg  2021-03-07 15:42:51.313+1300] bird[65794]  ┃  ┃  ┃  ┃  Item is marked needs-reading, so it must have changed
[dbg  2021-03-07 15:42:51.313+1300] bird[65794]  ┃  ┃  ┃  ┗  end
[dbg  2021-03-07 15:42:51.313+1300] bird[65794]  ┃  ┃  ┃  item has unscanned changes on disk, reading the file
[dbg  2021-03-07 15:42:51.313+1300] bird[65794]  ┃  ┃  ┃  serialized xattr <private> (blob size:89)
[dbg  2021-03-07 15:42:51.313+1300] bird[65794]  ┃  ┃  ┃  serialized xattr <private> (blob size:231)
[dbg  2021-03-07 15:42:51.313+1300] bird[65794]  ┃  ┃  ┃  serialized xattr <private> (blob size:331)
[dbg  2021-03-07 15:42:51.313+1300] bird[65794]  ┃  ┃  ┃  structural xattr signature: <private>
[dbg  2021-03-07 15:42:51.314+1300] bird[65794]  ┃  ┃  ┃  updated item from FS: <private>
diffs:()
[dbg  2021-03-07 15:42:51.314+1300] bird[65794]  ┃  ┃  ┃  we're a document missing a content signature
[dbg  2021-03-07 15:42:51.314+1300] bird[65794]  ┃  ┃  ┃  Clearing signatures: <private>
[dbg  2021-03-07 15:42:51.314+1300] bird[65794]  ┃  ┃  ┃  ┏  Saving item <private>
[dbg  2021-03-07 15:42:51.314+1300] bird[65794]  ┃  ┃  ┃  ┃  ┏  updating upload jobs
[dbg  2021-03-07 15:42:51.314+1300] bird[65794]  ┃  ┃  ┃  ┃  ┃  ┏  Uploader[<private>]: uploading
[dbg  2021-03-07 15:42:51.314+1300] bird[65794]  ┃  ┃  ┃  ┃  ┃  ┃  Uploader[<private>]: created job with state pending-sync-dn
[dbg  2021-03-07 15:42:51.314+1300] bird[65794]  ┃  ┃  ┃  ┃  ┃  ┗  end
[dbg  2021-03-07 15:42:51.314+1300] bird[65794]  ┃  ┃  ┃  ┃  ┗  end
[dbg  2021-03-07 15:42:51.314+1300] bird[65794]  ┃  ┃  ┃  ┃  updating recursive properties:
 fault:    {needs:0 done:0}
 quota:    0
 upload:   {needs:(count:1 size:71610) done:(count:0 size:0)}
 download: {needs:(count:0 size:0) done:(count:0 size:0)}
 unstage:  {needs:0 done:0}
 sync-up:  {needs:1 done:0}
[dbg  2021-03-07 15:42:51.315+1300] bird[65794]  ┃  ┃  ┃  ┃  recursive update of 95222
[dbg  2021-03-07 15:42:51.315+1300] bird[65794]  ┃  ┃  ┃  ┃  recursive update of 416
[dbg  2021-03-07 15:42:51.315+1300] bird[65794]  ┃  ┃  ┃  ┃  Reader[<private>]: Updated 1 jobs to state inactive
[dbg  2021-03-07 15:42:51.315+1300] bird[65794]  ┃  ┃  ┃  ┃  Progress: start tracking upload for doc 135466 because document is needs-upload (versionSize: 71610, transferSize: 75190)
[dbg  2021-03-07 15:42:51.315+1300] bird[65794]  ┃  ┃  ┃  ┃  Detected sync up state change
[dbg  2021-03-07 15:42:51.315+1300] bird[65794]  ┃  ┃  ┃  ┃  Finishing sync up for <private> in 1
[dbg  2021-03-07 15:42:51.315+1300] bird[65794]  ┃  ┃  ┃  ┃  ┏  Sync-Up[<private>]: finishing sync up
[dbg  2021-03-07 15:42:51.315+1300] bird[65794]  ┃  ┃  ┃  ┃  ┗  end
[dbg  2021-03-07 15:42:51.315+1300] bird[65794]  ┃  ┃  ┃  ┃  Rescheduling upload for <private>
[dbg  2021-03-07 15:42:51.315+1300] bird[65794]  ┃  ┃  ┃  ┃  Saved into DB. changedFields:<private> sendNotif:YES serverUpdate:NO
[dbg  2021-03-07 15:42:51.315+1300] bird[65794]  ┃  ┃  ┃  ┃  Queueing notification <private>
[dbg  2021-03-07 15:42:51.315+1300] bird[65794]  ┃  ┃  ┃  ┃  [ dispatching to com.apple.brc.notififcation-mgr
[dbg  2021-03-07 15:42:51.315+1300] bird[65794]   o continuing on com.apple.brc.notififcation-mgr
[dbg  2021-03-07 15:42:51.316+1300] bird[65794]  ┃  Merging <private> with <private>
[dbg  2021-03-07 15:42:51.316+1300] bird[65794]  ┗  end
[dbg  2021-03-07 15:42:51.316+1300] bird[65794]  ┃  ┃  ┃  ┗  end
[dbg  2021-03-07 15:42:51.316+1300] bird[65794]  ┃  ┃  ┗  end
[dbg  2021-03-07 15:42:51.316+1300] bird[65794]  ┃  ┗  end
[dbg  2021-03-07 15:42:51.316+1300] bird[65794]  ┃  Releasing coordinated read at '<private>' after 0.004103 seconds
[dbg  2021-03-07 15:42:51.316+1300] bird[65794]  ┃  Releasing coordinated I/O slot for r:95222 for <private>
[dbg  2021-03-07 15:42:51.316+1300] bird[65794]  ┗  end
[dbg  2021-03-07 15:42:51.316+1300] bird[65794]  ┗  end
[info 2021-03-07 15:42:51.317+1300] bird[65794]  ┏  <private> (<private>) -[BRCXPCRegularIPCsClient getAttributeValues:forItemAtURL:reply:] <private>
[dbg  2021-03-07 15:42:51.317+1300] bird[65794]  ┃  [ dispatching to com.apple.bird.read-only-workloop
[info 2021-03-07 15:42:51.317+1300] bird[65794]  ┗  end
[dbg  2021-03-07 15:42:51.317+1300] bird[65794]   o continuing on com.apple.bird.read-only-workloop
[dbg  2021-03-07 15:42:51.317+1300] bird[65794]  ┃  open('<private>') failed [1: Operation not permitted]
[dbg  2021-03-07 15:42:51.318+1300] bird[65794]  ┃  lookup: <private>
[dbg  2021-03-07 15:42:51.318+1300] bird[65794]  ┃  No value for attribute: <private>
[dbg  2021-03-07 15:42:51.318+1300] bird[65794]  ┃  No value for attribute: <private>
[dbg  2021-03-07 15:42:51.318+1300] bird[65794]  ┃  No value for attribute: <private>
[dbg  2021-03-07 15:42:51.318+1300] bird[65794]  ┃  No value for attribute: <private>
[dbg  2021-03-07 15:42:51.318+1300] bird[65794]  ┃  No value for attribute: <private>
[dbg  2021-03-07 15:42:51.318+1300] bird[65794]  ┃  returning <private>
[info 2021-03-07 15:42:51.318+1300] bird[65794]  ┃  <private>: reply(<private>, (null))
[dbg  2021-03-07 15:42:51.318+1300] bird[65794]  ┗  end
[dbg  2021-03-07 15:42:51.319+1300] bird[65794]  non-purgeable space was updated to 47364629592
[dbg  2021-03-07 15:42:51.319+1300] bird[65794]  ┏  db pre-flush: flushing client-state <private>
[dbg  2021-03-07 15:42:51.319+1300] bird[65794]  ┗  end
[dbg  2021-03-07 15:42:51.320+1300] bird[65794]  ┏  db post flush
[dbg  2021-03-07 15:42:51.320+1300] bird[65794]  ┃  ┏  collecting 2 additional folder updates
[dbg  2021-03-07 15:42:51.320+1300] bird[65794]  ┃  ┃  Queueing notification <private>
[dbg  2021-03-07 15:42:51.320+1300] bird[65794]  ┃  ┃  [ dispatching to com.apple.brc.notififcation-mgr
[dbg  2021-03-07 15:42:51.321+1300] bird[65794]   o continuing on com.apple.brc.notififcation-mgr
[dbg  2021-03-07 15:42:51.321+1300] bird[65794]  ┃  Merging <private> with <private>
[dbg  2021-03-07 15:42:51.321+1300] bird[65794]  ┗  end
[dbg  2021-03-07 15:42:51.321+1300] bird[65794]  ┃  ┃  Queueing notification <private>
[dbg  2021-03-07 15:42:51.321+1300] bird[65794]  ┃  ┃  [ dispatching to com.apple.brc.notififcation-mgr
[dbg  2021-03-07 15:42:51.321+1300] bird[65794]   o continuing on com.apple.brc.notififcation-mgr
[dbg  2021-03-07 15:42:51.321+1300] bird[65794]  ┗  end
[dbg  2021-03-07 15:42:51.321+1300] bird[65794]  ┃  ┗  end
[dbg  2021-03-07 15:42:51.322+1300] bird[65794]  ┃  [ dispatching to com.apple.brc.notififcation-mgr
[dbg  2021-03-07 15:42:51.322+1300] bird[65794]   o continuing on com.apple.brc.notififcation-mgr
[dbg  2021-03-07 15:42:51.322+1300] bird[65794]  ┗  end
[dbg  2021-03-07 15:42:51.322+1300] bird[65794]  ┃  signaling change on <private>
[dbg  2021-03-07 15:42:51.322+1300] bird[65794]  ┃  ┏  <private>: processing updates <private> [rank:2199504]
[dbg  2021-03-07 15:42:51.322+1300] bird[65794]  ┃  ┃  dropping all updates
[dbg  2021-03-07 15:42:51.322+1300] bird[65794]  ┃  ┗  end
[dbg  2021-03-07 15:42:51.322+1300] bird[65794]  ┃  ┏  <private>: processing updates <private> [rank:2199504]
[dbg  2021-03-07 15:42:51.322+1300] bird[65794]  ┃  ┃  dropping all updates
[dbg  2021-03-07 15:42:51.322+1300] bird[65794]  ┃  ┗  end
[dbg  2021-03-07 15:42:51.322+1300] bird[65794]  ┃  ┏  <private>: processing updates <private> [rank:2199504]
[dbg  2021-03-07 15:42:51.322+1300] bird[65794]  ┃  ┃  dropping all updates
[dbg  2021-03-07 15:42:51.322+1300] bird[65794]  ┃  ┗  end
[dbg  2021-03-07 15:42:51.322+1300] bird[65794]  ┃  ┏  <private>: processing updates <private> [rank:2199504]
[dbg  2021-03-07 15:42:51.322+1300] bird[65794]  ┏  Reader: scheduling
[dbg  2021-03-07 15:42:51.323+1300] bird[65794]  ┃  ┃  dropping all updates
[dbg  2021-03-07 15:42:51.323+1300] bird[65794]  ┃  ┗  end
[dbg  2021-03-07 15:42:51.323+1300] bird[65794]  ┃  ┏  <private>: processing updates <private> [rank:2199504]
[dbg  2021-03-07 15:42:51.323+1300] bird[65794]  ┃  ┃  dropping all updates
[dbg  2021-03-07 15:42:51.323+1300] bird[65794]  ┃  Reader: nothing schedulable, going to sleep
[dbg  2021-03-07 15:42:51.323+1300] bird[65794]  ┗  end

Meanwhile, I discovered the brctl evict and brctl download commands which expunge or download the local copy of the file respectively. If I try to evict one of the troublesome files it returns an error saying it is not yet synced with the server. But I can freely evict and download other files.

I’m going to send this all to Apple. Do their job for them because I am utterly unconvinced they have even looked at my logs.

I sent the above analysis to Apple and I know it has been forwarded to Engineering. Nothing heard back so far, but I’m not holding my breath.

I realised later that iCloud Drive seemed pretty hosed with many, many items showing they needed to upload when that should not have been the case. I tried to reverse the terminal-fu and weird stuff was happening so I turned off iCloud Drive and shut down the computer. A day later (build up the courage) I turned it back on and it has now completed downloading, but still there are oddities. KeepIt has lost the plot completely saying iCloud is “turned off” and no longer has a Quit option!

I will wait some more and see if it rights itself. I fear I may be in for a nuke and pave soon. All because Apple can’t give me anything concrete.

APPLE CALLED ME BACK!

When I got over the shock, there were two “options” given.

  1. The problem is identified and there will be a fix “in a future version of macOS”.
  2. They can do an iCloud Drive “reset” which will wipe all of the contents.

I’m fine with iCloud Drive being backed up, wiped, and repopulated. But that’s just the visible part. There are countless other folders in ~/Library/Mobile Documents which are concerning me should I go down the reset path.

I have 240 folders in there, many for iOS-only apps, some not even installed any more, and I just don’t know how they will fare. I have sought clarification of these folders but I fear it will be a “yes they are wiped too” with no hint on how to deal with them or all the apps that rely on them.

Option 1 isn’t ideal because I do not know which “future release” will supposedly fix the problem. I’m torn.

3 Likes

Copy these folders from the ~Library to a new temporary User folder.