large

  • 27 Dec

    Uploading large files using FTP Multipart option

    Dear users,

    A while ago we received the folllowing support question:

    “I am having problem transfering a big file. The file is 959MB. We get the following error: Exception Error Control channel transfer error 114690.”

    We found out that error 11469 = No data is being transferred via control channel. We asked our FTP code supplier for advice. 

    “Looks like a timeout-related issue. Error 114690 (SB_FTPS_ERROR_CONTROL_CHANNEL_NO_DATA) is returned if no reply was received from the control channel where it was expected to come in.  According to the log, the server does not perform any activity after the file is being sent. I believe there may be a delay between the file is sent to the server and the actual response the server is sending back. This delay may originate, for example, from the server copying the uploaded file from a temporary location to its final destination. The solution is quite straightforward – instead of uploading the file as a whole, upload it in pieces. Please start with splitting the file into 100MB chunks and see how it goes.”

    And that is what we did in version ‘v2018.12.27.0’. The Multipart Upload option was added in the ‘File & Directory’ tab of the FTP destination.

    • Enabled this option using the ‘Use Multipart Upload’ checkbox.
    • Set the part size. Default is 102.400 KiByte (100 MiByte). The minimum is 5 MiByte.
    • Set the treshold size. Default is 512.000 KiByte (500 MiByte).

    So what does this mean. When a source filesize is 500 MiByte or higher then this file will be uploaded in parts (chunks) of 100 MiByte. So not as a whole but in parts. Each part will be appened at the FTP server side. We use the APPE (Append) FTP command to achieve this.

    A second option to achieve a multipart upload is using the COMB command (‘Use COMB’ checkbox). Important, not so many FTP servers support the COMB command (non RFC). The source file is send in segments and afterwards the FTP server will combine them when it receives the COMB command. We tested this using the trial version of Titan FTP server.

    Even better is when you combine this with the Retry option. So if one of the parts would fail during the FTP upload then it will retry to upload this part again (and not the whole file).

    Feedback would be much appreciated,

    Happy Holidays,

    Limagito Team

     

    By Limagito New Option ,