Error by deleting a file on a Windows share

[rumen-vasilev]

Hello guys,

I am trying to delete a file with the "filedelete" microservice. Using an absolute path on the windows C directory works fine. But trying to delete a file on a Windows share (UNC path) or on a mapped drive doesn't work. I am getting the following error.

I used this manual to set up the API Calls.

Thanks for your support!

BR,
Rumen

[mountaindude]

My guess would be the following.

Does the user doing the reload have access to the UNC file share?

If reloading the app/script from a QMC task, the file deletion will be done in the name of the user running the Qlik Sense services, i.e. the Sense service account. Does this user have access to the file share?
If doing a manual reload from the script editor I think the currently logged in user will be used. Not sure though.

But for reload tasks started from the QMC I've seen this problem happen when the service account didn't have appropriate permissions on the target folders/files.

[rumen-vasilev]

Hello Göran,

Both the system user and the reload user have access to the UNC file share I am testing. My last test shows if I have a share on the same server where the Qlik Sense Engine is installed on, then it works. But if I have a share on another server, than I get the error I posted above. Is there any restriction known, if the share is not located on the same server?

BR,
Rumen

[mountaindude]

Not any restrictions that I am aware of, no.

I have had very similar issues in the past, but in those cases it always turned out to be permissions related.
When I finally resorted to logging into the Windows server using the Sense service account, it turned out I couldn't delete the files in question. So if you haven't already tried that I would encourage that as the next step.

If things still fail... Not sure.
I can maybe try to set up a similar scenario using a SMB file share on a remote server and see if I can replicate the problem.

[rumen-vasilev]

Thank you for your suggestion. I could successfully delete the files manually from the share by logging on the windows server. It would be grateful if you can test that on your site. I am using Butler 8.3.3 and Qlik Sense February 2022 Patch 2.

[mountaindude]

Some other thoughts

Have you white-listed the folder in the Butler config file? All directories from which files should be deleted must be whitelisted, as a security precaution. The config file section is Butler.fileDeleteApprovedDirectories

If the directory has not been whitelisted you should see an error in the Butler console logs (or its log files, if enabled).
Here's an example:
error: FILEDELETE: File delete request //192.168.1.124/qlik/testdata/deletefile/file.csv is not in any approved directories.

Silly maybe, but still:
Does the host name you use resolve for the service account? I.e. when logged in as the service account, does ping lr-analytics-test.lr-world.com resolve to an IP address?
If not that will for sure be a problem.

[mountaindude]

@rumen-vasilev Wait.... there seems to be an issue with UNC paths..
Pretty sure this is a regression, i.e. it used to work but something has changed => this no longer works.

Need to dig into this a bit more to be sure, stay tuned.

[mountaindude]

@rumen-vasilev
Where are you running Butler? Windows or Linux or something else?

[rumen-vasilev]

Hi,

I am running Butler 8.3.3 on a Windows Server 2019 using the standalone butler.exe.

[mountaindude]

I have a new version of Butler brewing, ETA a day or two. Should fix your problem.

…

On Sun, 7 Aug 2022, 13:54 Rumen Vasilev, ***@***.***> wrote: Hi, I am running Butler 8.3.3 on a Windows Server 2019 using the standalone butler.exe. — Reply to this email directly, view it on GitHub <#520 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAH3JDX275YB6IYQJYWA6QDVX6PYRANCNFSM55SDNKAQ> . You are receiving this because you commented.Message ID: ***@***.***>

[mountaindude]

@rumen-vasilev

Butler 8.4.0 is now available from the release page.

• The entire file copy/move/delete handling has been reworked, with better path handling in general - including handling of UNC paths.
• Log warnings have also been added if you're running Butler on an OS that doesn't support UNC paths (Linux, Docker, macOS...).

The Butler doc site has also been updated:
https://butler.ptarmiganlabs.com/docs/examples/file-copy-move-delete/

[mountaindude]

Here's a sample run to show that a file on a remote SMB file share can be deleted using a UNC path.
Butler (standalone exe binary) is running on Windows Server 2016, on one of the servers in a QSEoW May 2022 cluster.

SET ThousandSep=',';
SET DecimalSep='.';
SET MoneyThousandSep=',';
SET MoneyDecimalSep='.';
SET MoneyFormat='$#,##0.00;-$#,##0.00';
SET TimeFormat='h:mm:ss TT';
SET DateFormat='M/D/YYYY';
SET TimestampFormat='M/D/YYYY h:mm:ss[.fff] TT';
SET FirstWeekDay=6;
SET BrokenWeeks=1;
SET ReferenceDay=0;
SET FirstMonthOfYear=1;
SET CollationLocale='en-US';
SET CreateSearchIndexOnReload=1;
SET MonthNames='Jan;Feb;Mar;Apr;May;Jun;Jul;Aug;Sep;Oct;Nov;Dec';
SET LongMonthNames='January;February;March;April;May;June;July;August;September;October;November;December';
SET DayNames='Mon;Tue;Wed;Thu;Fri;Sat;Sun';
SET LongDayNames='Monday;Tuesday;Wednesday;Thursday;Friday;Saturday;Sunday';
SET NumericalAbbreviation='3:k;6:M;9:G;12:T;15:P;18:E;21:Z;24:Y;-3:m;-6:μ;-9:n;-12:p;-15:f;-18:a;-21:z;-24:y';


// ------------------------------------------------------------
// ** Delete file **
//
// Files can only be deleted in folders (and subfolders of) directories that 
// have been approved in the Butler config file.
//
// Paramaters:
// vFile                : File to be deleted. 
// ------------------------------------------------------------
sub DeleteFile(vFile)
    let vFile = Replace('$(vFile)', '\', '/');
    let vFile = Replace('$(vFile)', '#', '%23');

    let vRequestBody = '{""deleteFile"":""$(vFile)""}';

    LIB CONNECT TO 'Butler_POST';

    RestConnectorMasterTable:
    SQL SELECT 
        "vFile"
    FROM JSON (wrap on) "root"
    WITH CONNECTION (
    Url "$(vButlerHost):$(vButlerPort)/v4/filedelete",
    BODY "$(vRequestBody)",
    HTTPHEADER "X-HTTP-Method-Override" "DELETE"
    );

    set vFile=;
    set vRequestBody=;
    DROP TABLE RestConnectorMasterTable;
end sub


let vButlerHost = 'http://192.168.100.110';
let vButlerPort = '8081';

Call DeleteFile('//192.168.1.124/qlik/testdata/deletefile/file1.csv')
//Call DeleteFile('\\192.168.1.124\qlik\testdata\deletefile\file1.csv')

In the first example forward slashes ("/") were used:

But you use backslash ("") too if you prefer:

Butler console log shows now warnings or errors:

[rumen-vasilev]

Hello Göran,

It works! The main problem was, that Butler was running all the time as a Windows service (Created via NSSM tool). I just switched from local system account to the Qlik Sense system account and than the file deletion has worked also via the windows service. With both version 8.3.3 and 8.4. Unfortunately, it didn't occur to me earlier that the problem could be in the service.

Thank you so much supporting me.

BR,
Rumen

[mountaindude]

Great!

NSSM is an awesome tool, indeed. Very useful.

Still good that all the path handling logic got some much needed attention, it's now (per 8.4) in much better shape than before.