[macOS] DBRequestError "No such file or directory" on uploadUrl

[R4N]

We've been successfully utilizing the dropbox-sdk-obj-c client library in our iOS/macOS app for years now, so thank you for providing this. We're using the latest tagged release version (7.4.1).

Over the last year or so we've had intermittent reports from a few users running into an issue with one of our upload requests and wanted to ask here for guidance on further troubleshooting steps or possible causes.

Overview of usage

Our application allows users to upload pieces of data from our application to their personal Dropbox as a means of backing up their data. To do this we:

1. Check if the expected folder structure exists in Dropbox and create it if it doesn't (/<company name>/<sub-folder>)
2. Check for the existence of a text file named sync.lock (indicating that another client is in the process of uploading) and bail out with an error if one exists.
3. Upload a simple text file named sync.lock to indicate that one of the clients is in the process of uploading data and to block all other clients from uploading at the same time.
4. Perform the actual upload of data.
5. Delete the sync.lock file to indicate that this client has completed.

Where the issue occurs

The issue occurs at step 3 of above, within the uploadUrl completion block we're getting back nil for the DBFILESFileMetadata and a DBRequestError as follows:

DropboxClientError[{
NSError = "Error Domain=NSPOSIXErrorDomain Code=2 \"No such file or directory\" UserInfo={NSErrorFailingURLStringKey=https://api-content.dropbox.com/2/files/upload, NSErrorFailingURLKey=https://api-content.dropbox.com/2/files/upload, _NSURLErrorRelatedURLSessionTaskErrorKey=(\n \"BackgroundUploadTask <GUID>.<5>\",\n \"LocalDownloadTask <GUID>.<5>\"\n), _NSURLErrorFailingURLSessionTaskErrorKey=BackgroundUploadTask <GUID>.<5>}";
}

The users report that when they view the /<company name>/<sub-folder> location within the Dropbox webapp it appears that the sync.lock file was successfully created there.

Relevant Code

...
        // create our sync lock file
        SLog(@"Attempting to create sync.lock file locally");
        NSError *writeError = nil;
        NSString *randomStringToWrite = @"Blarg123";
        NSString *localPath = <local file path in container accessible to application>/sync.lock;
        BOOL success = [randomStringToWrite writeToFile:localPath atomically:YES encoding:NSUTF8StringEncoding error:&writeError];
        SLog(@"Result of attempting to write to file = %d", success);
        // as long as there was no write error for the sync file, let's send it out to Dropbox
        if (!writeError) {
            SLog(@"No write error, attempting to upload sync lock file");
            // self.rootPath is established as /<company name>/<sub-folder>
            NSString *destinationPath = [NSString stringWithFormat:@"%@/%@", self.rootPath, self.syncLockFileName];
            SLog(@"Does sync lock exist at path %d", [[NSFileManager defaultManager] fileExistsAtPath:localPath]);
            SLog(@"Uploading sync lock to %@ from local file %@", destinationPath, localPath);
            DBFILESWriteMode *addMode = [[DBFILESWriteMode alloc] initWithAdd];
            self.currentTask = [[client.filesRoutes uploadUrl:destinationPath
                                                         mode:addMode
                                                   autorename:@(NO)
                                               clientModified:nil
                                                         mute:@(YES)
                                               propertyGroups:nil
                                               strictConflict:nil
                                                  contentHash:nil
                                                     inputUrl:localPath]
                                setResponseBlock:^(DBFILESFileMetadata * _Nullable result, DBFILESUploadError * _Nullable routeError, DBRequestError * _Nullable error) {
                SLog(@"Response from upload, result != nil = %d, routeError = %@, requestError = %@", result != nil, routeError, error);
                if (result != nil) {
                    // Success!
                    // continue on with the rest of the process
                } else {
                    // Report error to the user
...

Log output

Attempting to create sync.lock file locally
Result of attempting to write to file = 1
No write error, attempting to upload sync lock file
Does sync lock exist at path 1
Uploading sync lock to /<company name>/<sub-folder>/sync.lock from local file <absolute path to container>/sync.lock
Response from upload, result != nil = 0, routeError = (null), requestError = DropboxClientError[{
NSError = "Error Domain=NSPOSIXErrorDomain Code=2 \"No such file or directory\" UserInfo={NSErrorFailingURLStringKey=https://api-content.dropbox.com/2/files/upload, NSErrorFailingURLKey=https://api-content.dropbox.com/2/files/upload, _NSURLErrorRelatedURLSessionTaskErrorKey=(\n \"BackgroundUploadTask <GUID>.<5>\",\n \"LocalDownloadTask <GUID>.<5>\"\n), _NSURLErrorFailingURLSessionTaskErrorKey=BackgroundUploadTask <GUID>.<5>}";

Additional Comments/Questions

• This issue has only been reported in our macOS app. Our iOS app which uses the same shared code-path for uploading the sync.lock file hasn't had any reports of a similar issue. And some of the users who report this issue are using both our iOS and macOS with the same Dropbox account.
• We haven't been able to reproduce this issue ourselves on any of our machines
• Are there any known causes of this issue (or how to avoid it)?
• We've asked reporting users to confirm that their Dropbox account has sufficient space, Is there anything related to the user's Dropbox account which could be causing it?
• Is there any additional logging we can add to try to get additional information as to the root cause?
• I've tried experimenting some with trying to reproduce the issue by programmatically using flock and chmod, with chmod I was able to create a separate error when completing removing read/write permissions which is expected (DBFILESWriteError)

[alyx-db]

Hi @R4N

Thanks for the report! We're investigating this issue. We'll follow up here once we have an update.

[jboulter11]

Hey @R4N, thanks for the detailed write up. It seems like from the error is indicating that the upload task can't find the file you're trying to upload.

Some things you could check:

1. Are you putting the file in a temp folder which could be getting deleted? MacOS might clean a cache folder or temp folder if disk space is low.
2. Is your lock file's contents the same every time, or are you writing something unique to each process each time? According to the docs, if file contents are the same, the upload will succeed as no conflict if another client was uploading that file at the same time with the same contents (since strictConflict = nil). If your file has unique contents, though, I'd expect you to get a failure from one of the two writes due to conflict. To test this scenario you might set this to loop on two machines or two processes on one machine and see if you manage to get a repro when two clients overlap uploads.

[jboulter11]

Whoops, I'm now realizing that second thing makes no sense if the error you're seeing is from the local file being missing.

I think it would be more like if you had two syncs running on one machine using the same local file. One finishes and deletes the local file while the other is about to use it.

[R4N]

Hi @jboulter11

Thanks for the response with the ideas.

1. Are you putting the file in a temp folder which could be getting deleted? MacOS might clean a cache folder or temp folder if disk space is low.

No. The sync.lock file is being created in the same folder the main data file of the application. For our macOS App that's:

~/Library/Group Containers/<appId>/<sub-folder>/sync.lock

The main data file of the application is being frequently written to/read from by the application and other complimentary files are being created/deleted from that same directory as needed.

I think it would be more like if you had two syncs running on one machine using the same local file. One finishes and deletes the local file while the other is about to use it.

The sync is triggered manually in the application interface by the user rather than automatically (or in the background). When a sync is triggered, the interface is blocked from proceeding until it completes (or is canceled). There are also programatic guards in place to prevent an additional sync from occurring while one is in progress.

My first thought was also that there was something wrong with the local file's existence or some other process holding onto it and preventing the Dropbox API from interacting with it.

Anecdotally a few of the users who've reported this issue have confirmed that they previously restored onto a new machine from a Time Machine backup prior to the issue occurring, so perhaps that could be related.

Are you able to confirm that this error message always references an issue with the local file on the file system? Or could it ever be something related to the remote?

I could add extra logging after the request completes to confirm that the sync.lock file still exists in the local file system to rule out something else removing it during the upload process.

[jboulter11]

I am able to confirm that error is definitely a client error, not a server error. NSPOSIXErrorDomain is MacOS and the error is a RequestError of type DropboxClientError. Definitely happening on the MacOS client machine. I think adding extra logging could be helpful.

[R4N]

@jboulter11

Wanted to follow up with some additional information from one user who tried the new build with the additional logging:

1. The logging within the completion block reports that the sync.lock file still exists at the local file path.
2. I had him check the permissions for the enclosing folder (the container folder for the application) and confirmed that it has the appropriate read/write permissions for his macOS user account.
3. I've had him try moving his data out of the way, deleting the entire application container and allowing the application to create it fresh.
4. I've asked him about whether he's running any backup software or antivirus software and if so, if he could temporarily disable it (maybe it's somehow interfering?). I'm still waiting on a response from that.

Is your lock file's contents the same every time, or are you writing something unique to each process each time?

I know you mentioned this one wasn't relevant anymore, but figured I would answer it for completeness:

Yes, the lock file's content is the same every time. We cleanup the lock (in their Dropbox and locally) at the end of sync process (or if it was created during the process and an error occurs).

It's definitely strange that it produces an error but the file is still successfully uploaded to their Dropbox.

Let me know if you have any other thoughts or troubleshooting suggestions. Thanks!

[jboulter11]

Seems like that error comes back from Apple's networking library (we fire off an NSURLSessionTask) and from what I can see we don't alter your file path in between you giving it to us and us handling it to Apple's networking code. If the file is still there, then I don't know what else could be giving us a file not found error. I can see that it's a background task from the error message you shared, but we are configuring the session to work with background tasks from what I can see in the code.

What kind of volume are you seeing here? Like is this 1% of your requests? Less? Maybe you can just retry your requests in this case? Not optimal, but we do many suboptimal things to work around Apple bugs in this line of work. 😂

Yes, the lock file's content is the same every time. We cleanup the lock (in their Dropbox and locally) at the end of sync process (or if it was created during the process and an error occurs).

I would recommend you put a guid in there that each client can use because if no changes are made during a conflicting upload, you will get back a success, based off what the docs say. So client 1 would set the lock to "A" then client 2 uses the same contents, also setting the lock to "A", then client 2 would receive a successful upload response rather than a conflict response, since there is no conflict in the contents of the file. If you used a GUID specific to each client, client 2 gets a conflict response since the contents are different. If you're ever in doubt about who has the lock, simply read the file from Dropbox and you can check the GUID against your client's lock GUID.

[R4N]

@jboulter11

Thanks for the response and the suggestion.

What kind of volume are you seeing here? Like is this 1% of your requests? Less?

Yes, it's a very low percentage of folks (less than 1%). I think we've had a total of 4-5 reports of it.

I've made the following modifications:

1. The sync.lock file now contains a UUID.
2. If an error occurs when attempting to upload it, try to download the sync.lock file from the dropbox remote.
3. If the sync.lock from the remote was successfully downloaded, compare the sync.lock.remote UUID agains the known UUID from our local file, if it matches assume the upload succeeded, the error was incorrect, and continue on.
4. If the sync.lock.remote file doesn't exist, or isn't the same UUID raise the error to the user.

I'll let you know if this does the trick once a user experiencing the issue is able to try it. Thanks!

[jboulter11]

Assuming the bug is in Apple's code, like it seems, I'd recommend just building a robust retry mechanism that can catch this error and retry the request. Track whether retries improve your success rate or whether the requests continue to fail. This is what we tend to do when these kinds of Apple bugs crop up for us. If we can't find something out via Hopper, trial and error, analytics, etc. we just build retry and verification mechanisms with good analytics/alerts to tell us if something goes wrong.

Regarding the sync.lock, I think that will serve you better than before. I still wouldn't expect it to be perfect, these systems are really complicated and we as far as I'm aware we don't make the kinds of guarantees with our upload/download APIs that you'd need to get to 100% no collisions with what you're trying to do. I don't expect it will help at all with this file not found error but it hopefully reduces collisions for you a little. :)