A few weeks back one of my customers started seeing a new issue with a recently updated Ionic app: uploading files resulted in a PHP Upload error #3, UPLOAD_ERR_PARTIAL. I'd come across the usual PHP upload errors before: needing to increase post_max_size, upload_max_filesize and max_execution_time. But these settings had never caused an UPLOAD_ERR_PARTIAL before.
One of the problems with the UPLOAD_ERR_PARTIAL error is how little information you're given. Apache and PHP report no errors, and there's no way to see the underlying web request that caused the problem. Using mod_security and mitmproxy gave me some interesting data, but nothing conclusive. mod_security reported a 408 Timeout and using mitmproxy actually corrected the problem altogether.
What had changed was replacing the standard Angular HTTP Client with a Cordova specific plugin. I'd found some evidence that iOS had timeout issues due to Keep-Alives, and thought that may be in play. Turning off Keep-Alive didn't help. I then swapped out the Ionic plugin, first for the FileTranser plugin, then for the native HTTP plugin. The problem persisted.
I then switched back to the Angular HTTP Client, and to my surprise, the issue showed up there as well.
I was just about to swap out Apache for Nginx, when it occurred to me that I hadn't investigated the issue from the Apache side of things. Attacking the problem from this angle yielded this hit on Google: Can't upload larger files to server. This individual ran into a similar issue and for once I got a new answer:
Found the problem. The Apache had a RequestReadTimeout header=20-40,MinRate=500 body=20-40,MinRate=500 setting which means the request's forced to timeout after max 40 sec... Another thing to watch out for.
I quick Google search of Apache RequestReadTimeout turned up a module I had no idea even existed, let alone one that was turned on by default. The mod_reqtimeout does what you think it does: if it takes too long to read a request, the Apache truncates the request. I enabled the logging the module suggested (LogLevel reqtimeout:info) and had my customer try again. Sure enough, the module reported that the request timed out.
Suddenly, the whole scenario made sense. My customer's slow connection was causing his uploads to be timed-out and truncated. From PHP's perspective the file was indeed a partial upload. Putting mitmproxy in between my client and Apache 'fixed' the problem because mitmproxy read the entire request slowly, reported it to me, and then delivered it all at once to Apache, which made Apache quite happy.
Not only am I pleased that I've fixed this bug, but I'm happy to have uncovered yet another critical Apache setting. Heck, just being reminded that you can turn up Apache's logging by setting LogLevel was an invaluable take away from all of this.
And here was I blaming Ionic, Cordova and iOS when the issue was fully due to my Apache ignorance.
No comments:
Post a Comment