[bug] Support for send 25Mb+ size files

[binsee]

Previously implemented to send any type of file, but there is a limit: send or forward more than 25Mb size file will fail.

The reason for the failure is that the server requires that the Signature field must be included when sending a file of size above 25Mb. This field is from the json data returned after the API_checkuploadwas called before the file was uploaded. The currentpuppet-web` does not implement this.

When the ATTACH type of message is received, the Signature field is not included, so the server will fail when forwarding a file of size above 25Mb.

The solution is as follows:

1. In PuppetWeb.uploadMedia (), add a call to API_checkupload
2. In the data returned by PuppetWeb.uploadMedia (), add the Signature field
3. In MediaMessage, addreupload (), download the file and callPuppetWeb.uploadMedia (). Used to re-upload the file, the continuation of file validity.
4. In MediaMessage.forward (), when the file size is greater than 25Mb, call reupload () to re-upload the file

If there are better suggestions, please comment.

---

Send details about 25Mb + size file messages:

Received data for file messages:

{
"MsgType": 49,
"Content": "<?xml version='1.0' ?>
<msg>
    <appmsg appid='wx6618f1cfc6c132f8' sdkver='0'>
        <title>25.6Mb文件.zip</title>
        <des />
        <action>view</action>
        <type>6</type>
        <showtype>0</showtype>
        <content />
        <url />
        <dataurl />
        <lowurl />
        <lowdataurl />
        <recorditem>
            <![CDATA[]]>
        </recorditem>
        <thumburl />
        <extinfo />
        <sourceusername />
        <sourcedisplayname />
        <commenturl />
        <appattach>
            <totallen>26902123</totallen>
            <attachid>@cdn_304b02010004443042020100020495970f8b02032f54cd0204313ca17b020459a33b78042038336532366461353330646638646232633439653766353565653630376561300201000201000400_b14bba4fd61c486f82a7bf83dd4e4430_1</attachid>
            <emoticonmd5 />
            <fileext>zip</fileext>
            <cdnattachurl>304b02010004443042020100020495970f8b02032f54cd0204313ca17b020459a33b78042038336532366461353330646638646232633439653766353565653630376561300201000201000400</cdnattachurl>
            <aeskey>b14bba4fd61c486f82a7bf83dd4e4430</aeskey>
            <encryver>1</encryver>
        </appattach>
        <md5>86e8affd5f2374143f791054a2a2d40d</md5>
    </appmsg>
    <fromusername>wxid_jqr6vfax27il21</fromusername>
    <scene>0</scene>
    <appinfo>
        <version>7</version>
        <appname>微信电脑版</appname>
    </appinfo>
    <commenturl></commenturl>
</msg>",
"FileName": "25.6Mb文件.zip",
"FileSize": "26902123",
"MediaId": "@crypt_88535d4c_0dc2fe8674491a53d4f9ca6620202d0f078b1fd9f64781b02e185c95eb68e4b1e8907f26b8bf6c711550f78273ef786a0df3af60fd281030a907f141d283f3ade20cefb2fd5ab12ef2bf882f0d646e8c59e8fbb3e0b587b08c1557c8d9cff4b15b998ac3adca2eb4348679ec187db18f9487946f4f6e366f9db2b6d3605d6c4cb4cff91be824e638e3774514437d4ef8da738cbb74a998bd9c2afe04a2d6e076cc80ed609cd5572e893affa844208bfb5e0a026d6be491b339dabe56103675f7535eb989be6b824e28b9724771dfac33",
}

Post data for forward file: forward failed

{
    "BaseRequest": {
        "Uin": 12345678,
        "Sid": "xxxxxxxxx",
        "Skey": "@crypt_xxxx_xxxxxxxxxxxx",
        "DeviceID": "xxxxxxxxxxxxx"
    },
    "Msg": {
        "Type": 6,
        "Content": "<appmsg appid='wxeb7ec651dd0aefa9' sdkver=''>
    <title>25.6Mb文件.zip</title>
    <des></des>
    <action></action>
    <type>6</type>
    <content></content>
    <url></url>
    <lowurl></lowurl>
    <appattach>
        <totallen>26902123</totallen>
        <attachid>@crypt_88535d4c_0dc2fe8674491a53d4f9ca6620202d0f078b1fd9f64781b02e185c95eb68e4b1e8907f26b8bf6c711550f78273ef786a0df3af60fd281030a907f141d283f3ade20cefb2fd5ab12ef2bf882f0d646e8c59e8fbb3e0b587b08c1557c8d9cff4b15b998ac3adca2eb4348679ec187db18f9487946f4f6e366f9db2b6d3605d6c4cb4cff91be824e638e3774514437d4ef8da738cbb74a998bd9c2afe04a2d6e076cc80ed609cd5572e893affa844208bfb5e0a026d6be491b339dabe56103675f7535eb989be6b824e28b9724771dfac33</attachid>
        <fileext>txt</fileext>
    </appattach>
    <extinfo></extinfo>
</appmsg>",
        "FromUserName": "@xxxxxxxxxxxx",
        "ToUserName": "@xxxxxxxxxxxxxxxxxx",
        "LocalID": "15038701009240769",
        "ClientMsgId": "15038701009240769"
    },
    "Scene": 2
}

After uploading the 25Mb file, post data for send message:

{
    "BaseRequest": {
        "Uin": 12345678,
        "Sid": "xxxxxxxxx",
        "Skey": "@crypt_xxxx_xxxxxxxxxxxx",
        "DeviceID": "xxxxxxxxxxxxx"
    },
    "Msg": {
        "Signature": "epNkkyx8r8W74ULeofJEJSWFfxXKRup7vxTbaDu2ivKWVlCC6qKVx82qHlfyAVEb93TZpJOCKD2uzs3/TffjCAHxfN7ziNdemp/lJwTElrf/MjG0sK4MYasbxBccEE/TwsprceXNtC6IhYIAFmOF05z6f7bBbq7rjq8umvmbRgE=",
        "Type": 6,
        "Content": "<appmsg appid='wxeb7ec651dd0aefa9' sdkver=''>
    <title>25.6Mb文件.zip</title>
    <des></des>
    <action></action>
    <type>6</type>
    <content></content>
    <url></url>
    <lowurl></lowurl>
    <appattach>
        <totallen>26902123</totallen>
        <attachid>@crypt_a5d9e417_838c97580d69d876ec912143f0959956f63b3772ea6a1fff2f1ef7cf61ab0f8edb810fb41f55c4990df11c129e0a56f29a487258c34d11372955c0f929db7bc6a236ed0226ede3f1584a24915f1a84393cc8232fee528d5a383c1f2828084f0bf1edb785973f1457d36fec9e690287a10111622c1f7835a9fe8c61826f10c8c06b19a30c00742cfa92f61e520d73d6e5f6bea0afeceafba67926dfd5b033225582036f9bb268d91cdb36c2e821a8665ea0601014b06ea626e52dded4827dd8d1ef61bb1f49b1329cb8abba9110d8670e797fb97b6ea10f357f786ca47eb43768d6bad2fa008c5029f365413d03396bb0205cabb2585c8532523171b4054cf7a516c987d1690cc1e8c4f071b181a96f884ef38b90e92203f103c5839424f381065d2ce036d2db17173ac190a208353f9014ae7ed8294965ca43a9a0349dc7c343</attachid>
        <fileext>zip</fileext>
    </appattach>
    <extinfo></extinfo>
</appmsg>",
        "FromUserName": "@xxxxxxxxxxxx",
        "ToUserName": "@xxxxxxxxxxxxxxxxxx",
        "LocalID": "15038705888070503",
        "ClientMsgId": "15038705888070503"
    },
    "Scene": 0
}

Call the upload file API to get the data:

{
    "BaseResponse": {
        "Ret": 0,
        "ErrMsg": ""
    },
    "AESKey": "6a6e696f666e69766b78716e65766c77",
    "Signature": "epNkkyx8r8W74ULeofJEJSWFfxXKRup7vxTbaDu2ivKWVlCC6qKVx82qHlfyAVEb93TZpJOCKD2uzs3/TffjCAHxfN7ziNdemp/lJwTElrf/MjG0sK4MYasbxBccEE/TwsprceXNtC6IhYIAFmOF05z6f7bBbq7rjq8umvmbRgE=",
    "MediaId": ""
}

[huan]

I agree that we should support for sending 25Mb+ size files in Wechaty automatically, and your analysis looks very clear and will be useful.

It would be very convenient for the user that we can just do the following operation without considering the file size:

1. send a file: contact.say(new MediaMessage('/filepath')
2. forward a file: message.forward(contact)

To contrast, the user will have to write lots of code if we leave those jobs out of our Wechaty framework.

[binsee]

@zixia
We need to merge #727 , and then I can increase in MediaMessage.forward() for large file processing

When forwarding large files, you need to download the file to the local temporary directory.
Do you have any suggestions for this temporary catalog?

[huan]

Got it. I had already approved the PR, and so do @lijiarui. Could you please invite another reviewer to approve the PR, so we will meet the 3 approvements that I can merge it?

Appreciate for that because that's the necessary step to keep the high quality of our code base.

About the local temporary directory, I'd like to recommend to use the standard nodejs library function: fs.mkdtemp(prefix[, options], callback).

See: https://nodejs.org/api/fs.html#fs_fs_mkdtemp_prefix_options_callback

[binsee]

current progress:
Has been achieved forward on the handling of large files, and change the forward so that it can bulk forwarding.

Current question:
Forward upload large files, the generated message can not be captured, so it can not be reused. This will cause a file to be repeatedly uploaded.

Hope solution:

1. Change PuppetWeb.uploadMedia () to public.
2. In MediaMessage.forward () directly call uploadMedia () upload files, reuse it return MediaData.

Can I change uploadMedia () to public?

[huan]

Hi binsee,

Great to hear that the forwarding and the large files all can be handled in your new code!

I think it worth to change the PuppetWeb.uploadMedia() to the public if we can reuse the MediaData and prevent to re-upload file multiple times.

Also please notice that: PuppetWeb.uploadMedia() should only be called inside the puppet-web/ directory, because it should not be used outside.

Have a nice weekend!

[olifeolove]

Is it possible to fix a bug with file size is 0Kb when downloading 25mb+ files?

[binsee]

@olifeolove Please tell me the details.

[olifeolove]

@binsee I use bot receiving some files such as *.pdf or *.zip. The file will be saved as a 0 byte file when the file size is more than 25mb.

[binsee]

@olifeolove
Who sent the file?
bot or phone?

If it is bot, please see #771 .

[olifeolove]

@binsee Sorry, it's phone. I just use the bot to receive files.

[huan]

Please notice that when we are running Wechaty, if you use the browser that controlled by Wechaty to send attachment files, you will get a zero sized file, because it is not an attachment from the network, but a local data, which is not supported by Wechaty yet.

[binsee]

@olifeolove Because webWx restrictions, more than 25 MB of files can not be downloaded.

[huan]

Landed at v0.8.240