error on installation

[0MAL]

On local dev site, it works fine now. For the site, I don't use plugins, I use custom integration. Then I tried to install mosparo into live site on subdomain, it fails due to failing create cache directories or something. Then I gave up clean installing, instead of that, I tried copy database and files/directories to the production site. I import sql file with the same dbname, same user id, same password. I rsync all files to the webroot/. And now I still have errors.
The current error is like this.
webroot/var/log/prod-todaysdate.log says
"[2024-03-26T08:44:27.941084+00:00] app.CRITICAL: An exception occurred while executing a query: SQLSTATE[23000]: Integrity constraint violation: 1451 Cannot delete or update a parent row: a foreign key constraint fails (lamp.submission, CONSTRAINT FK_DB055AF32B4057C1 FOREIGN KEY (submit_token_id) REFERENCES submit_token (id)) [] []"
Browser displays error below when opening mysubdomain/index.php:
"
Oops! An Error Occurred
The server returned a "500 Internal Server Error".

Something is broken. Please let us know what you were doing when this error occurred. We will fix it as soon as possible. Sorry for any inconvenience caused.
"

[zepich]

Hi @0MAL

Thank you very much for your question.

Are you sure that this error message is from right now? The timestamp shows 08:44:27+00:00, which would be 6 hours ago. Can you maybe delete the log file and request the page again to make sure that this is the error message that is happening right now?

In general, this error should not be visible (of course). If the error is still visible in the log file after trying it again, please look in the database in the table submission and submit_token. Can you delete all rows from these two tables?

To be honest, I personally recommend a clean install. Do you remember what error message you saw when you tried the clean install? In general, you should only need to extract or upload the files from the Zip file and then set the correct access rights (if you extract/upload the files with a different user than your webserver or PHP process is running).

If you want to try that again, please let me know what error you see.

Kind regards,

zepich

[0MAL]

.log file in /var/log/ doesn't update when I reproduce the issue, so it looks like nothing to do with this issue.
Error log on production server doesn't catch any error, it is weird.
I checked submission and submit_token table, and those number of rows are same for local and prod. Then I tried to delete data in the two tables, I have an error:
#1701 - Cannot truncate a table referenced in a foreign key constraint (lamp.submit_token, CONSTRAINT FK_6C047AC88DF22AA4 FOREIGN KEY (last_submission_id) REFERENCES lamp.submission (id))
So it seems I can't delete those data from database for both local and prod. But I can delete them when I uncheck "foreign key" option on phpmyadmin. Deleting those two tables didn't solve the issue. Same error is displayed.

I'll try clean install again.

[zepich]

Hi @0MAL

Thank you very much for your feedback.

Both issues (the issue with the clean install and the one with the synced folder) sound like an issue with the access rights.

Do you use a virtual server to which you have SSH access, or do you have standard web hosting?

If you have a virtual server, can you please check which user runs your webserver and (if you use) the PHP process? The user who runs the PHP process must have write access to at least some of the files (config/env.mosparo.php, var/, /public/resources/).

Please let me know when you have the error message, and maybe you can tell me a bit more about the server (kind of hosting/server, operating system, web server type).

Kind regards,

zepich

[0MAL]

Adjusting owner of unzipped files solved the "couldn't create folder" issue. But I have another error:
Got error 'PHP message: PHP Warning: tempnam(): open_basedir restriction in effect. File(/tmp) is not within the allowed path(s) in vendor/symfony/dependency-injection/Loader/XmlFileLoader.php on line 684PHP message: PHP Fatal error: Uncaught ValueError: Path cannot be empty in vendor/symfony/dependency-injection/Loader/XmlFileLoader.php:692

[zepich]

Hi @0MAL

Okay, perfect. Thank you for the update.

Can you add /tmp to the PHP config open_basedir? You must add it to the existing value delimited with a colon (:).

Kind regards,

zepich

[0MAL]

This is the first time to allow access to /tmp folder in the server. Is this just for install? Or the permission is needed after installation?

[zepich]

Hi @0MAL

Yes, the open_basedir must always include the temporary directory.

But there is a different option. You can also set sys_temp_dir to something like /path/to/mosparo/var/tmp. After that, you must create an empty directory with the name tmp in var/.

With that, you tell PHP to store the temporary files in the var/tmp directory of your mosparo directory, which is inside the open_basedir setting, and therefore, there is no problem for PHP to access it.

I hope that helps you to solve the issue. Please let me know if you have more questions.

Kind regards,

zepich

[0MAL]

I solved issue of /tmp folder with creating temporary folder only for mosparo and set open_basedir for it, but mosparo still recognizes the sys_temp_dir as /tmp, so I end up with modifying sys_get_temp_dir() to literal path for the temporary folder. I modified XmlFileLoader.php,XliffUtils.php and FlockStore.php. Then errors disappeared.

I have another question, for installation requirement, php zip module is listed, is this module still needed after installation?

[zepich]

Hi @0MAL

Thank you very much for the update.

Access to the /tmp and the zip module is required for the setup and update process (as far as I can remember right now). The zip module is required to extract the update with the updater in mosparo. After the update (and before the setup), mosparo needs to create temporary files to build the cache. When I tested the issue yesterday, I had no problems after creating the cache with the enabled open_basedir setting without the /tmp included.

Since you changed some files, you probably don't want to use the update functionality in mosparo anyway. If so, you also don't need the zip module.

If you have shell access to your server (by SSH, for example), you can also use the CLI command to update mosparo. If you do that, you must enable the zip module only for the CLI interface (and the open_basedir setting, which is probably not active for CLI).

Please let me know if you have more questions.

Kind regards,

zepich

[0MAL]

Ok.
Another issue is, all files are accessble to the internet, for example, https://mymosparosubdomain.mydomain.com/composer.json opens composer.json file to the public by default...

[zepich]

Hi @0MAL

Thank you very much for your feedback.

Did you set the document root of your web host to the public directory of mosparo? Because then, it should not be accessible at all since it's one level down.

If you have set the document root to the root directory of mosparo, you have to remove two lines from the .htaccess file. Please edit the .htaccess file in the root of the mosparo code and remove these two lines:

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d

The adjusted .htaccess file should look like this:

<IfModule mod_rewrite.c>
    RewriteEngine On

    RewriteBase /

    RewriteRule ^build(.*)$ public/build/$1 [L]
    RewriteRule ^bundles(.*)$ public/bundles/$1 [L]
    RewriteRule ^resources(.*)$ public/resources/$1 [L]

    RewriteRule ^robots.txt$ public/robots.txt [L]

    RewriteCond %{HTTP:Authorization} ^(.*)
    RewriteRule .* - [e=HTTP_AUTHORIZATION:%1]

    RewriteRule ^(.*)$ public/index.php [QSA,L]
</IfModule>

Important: only remove these two lines in the root directory, not in the /public directory. That should restrict access to all files outside of /public. I will remove these two lines from the .htaccess in the source code, so that future release do not have this issue anymore.

Kind regards,

zepich

[0MAL]

Thanks.
BTW, mosparo's example of custom integration, verifyFormDataWithMosparo(), works in my local devsite, but not work in prod. site. That cause timeout error like this:
PHP Fatal error: Uncaught GuzzleHttp\Exception\ConnectException: cURL error 7: Failed to connect to mysite port 443: Connection timed out (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://mysite/api/v1/verification/verify in mysite/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php:210

JS input validation works fine before submission, but Guzzle request during submission fails.
Do you have any idea?

[zepich]

Hi @0MAL

Do you see the request for the API from your server? The request should be visible in the Apache access log file for the mosparo installation.

A timeout causes the error message, so it sounds like it can resolve the domain but not really connect, or while connecting, something takes too long. I guess either the request is not reaching Apache2 and is not in the access log, or Apache2 processes the request, but something times out. In this second case, you should see the request in the access log.

Please let me know if you can see the request in the access log.

Kind regards,

zepich

[0MAL]

[Edited]

I see the request in access log for mosparo subdomain:

"GET /resources/*****************(uiid).css HTTP/1.1" 307
"GET /resources/**************/************.css HTTP/1.1" 200
"POST /api/v1/frontend/request-submit-token HTTP/1.1" 200
"POST /api/v1/frontend/check-form-data HTTP/1.1" 200

After the POST request for check-form-data, submit button is clicked and several minutes has passed and it timed out .

access log for website domain:
"POST /aaa.php HTTP/1.1" 500 1168 "https://mydomain/form.html"
aaa.php which include verifyFormDataWithMosparo(), is called from form.html.

[zepich]

Hi @0MAL

Thank you very much for your log lines.

You see the frontend request but not the one to the backend API (/api/v1/verification/verify). This means that the request to the backend API does not reach your Apache2.

I guess the problem is that the server cannot resolve your hostname. Is it a virtual server where you have SSH access? If so, can you ping the mosparo host from the terminal?

For example:

ping mosparo.example.com

Kind regards,

zepich

[0MAL]

The issue occurs on test domain of production server, and Guzzle package is different from the prod. site. I'll try setting the program on
prod site later.

[zepich]

Hi @0MAL

And what happens if you request a URL from mosparo from the terminal/SSH? For example:

wget https://mosparo.example.com/login

Kind regards,

zepich

[0MAL]

wget is waiting response for minutes. Same thing happens for "wget https://mywebsite.com/form.html".

[zepich]

Hi @0MAL

Okay, thank you very much.

It sounds like it's some firewall issue. Do you have a firewall enabled on the server, and if so, which one?

Kind regards,

zepich

[0MAL]

That was my mistake, wget for mosparo/login works. I used wrong terminal.

[zepich]

Hi @0MAL

Okay, thank you very much for your update.

Can you test if a PHP script can connect to github.com, for example (to ensure that PHP can connect to other web servers)?

Kind regards,

zepich

[0MAL]

I confirmed I can GET and POST request to other servers from the program with the GuzzleHttp.

[zepich]

Hi @0MAL

Okay. To which IP address is the mosparo host resolving? Is it the internal IP address (127.0.0.1) or the external one (you don't have to post your IP address of course).

Kind regards,

zepich

[0MAL]

This looks like server setting issue, not mosparo. The program can't success GET request between subdomains in the server.

[0MAL]

I solved the issue after adjusting server settings. Now everything is working perfectly I think.
And wget from internal server terminal failed before, but now it works.
Thanks a lot!

[zepich]

Hi @0MAL

Awesome, thank you very much for the information! I'm happy that it's working now!

Please let me know if you have any other questions or issues.

Kind regards,

zepich

[0MAL]

Hi. I had another issue.
When I put the contact form in real site, it didn't work. The error is:
AH01071: Got error 'PHP message: PHP Fatal error: Uncaught GuzzleHttp\\Exception\\ClientException: Client error: POST https://mymosparo-sub-domain/api/v1/verification/verify` resulted in a 400 Bad Request response:\n{"error":true,"errorMessage":"Request invalid.","debugInformation":{"reason":"hmac_hash_invalid","expectedHmacHash":"ed7 (truncated...)\n in /mydomain-src/guzzlehttp/guzzle/src/Exception/RequestException.php:113\nStack trace:\n#0 /mydomain-src/guzzlehttp/guzzle/src/Middleware.php(66): GuzzleHttp\Exception\RequestException::create(Object(GuzzleHttp\Psr7\Request), Object(GuzzleHttp\Psr7\Response))\n#1 /mydomain-src/guzzlehttp/promises/src/Promise.php(203): GuzzleHttp\Middleware::GuzzleHttp\{closure}(Object(GuzzleHttp\Psr7\Response))\n#2 /mydomain-src/guzzlehttp/promises/src/Promise.php(156): GuzzleHttp\Promise\Pr...\n', referer: mydomain/mosparo-form.html`
The version of PHP is 7 for the site, so it may not meet the requirement, but I want to use contact form at the site if possible. Because the site is running CMS which has good XSS protection. I can create another subdomain for contact form with php8 but I have to prepare XSS protection program so I don't want to do that.
The version of PHP for mosparo subdomain is 8.1.

[zepich]

Hi @0MAL

Thank you very much for your question.

PHP 7, at least PHP 7.4, should be fine for the frontend. The most important functionality should be available since 5.1.2+ (https://www.php.net/manual/de/function.hash-hmac.php). I used PHP 7.4 for the verification, too. Only the backend of mosparo requires PHP 8.1.10+.

Your problem is that the request is not accepted at all, which means there is an issue with the request signature. Are you using the official PHP mosparo API client, or did you build your own integration?

Can you maybe share your integration code?

Have you checked the complete response from mosparo? Since you've enabled the API debug mode, you should see the expected HMAC hash and the one that was sent. You should additionally see the expected payload. If you add some debug output to your verification, you should see the difference between the payload you sent and the payload mosparo expected.

Kind regards,

zepich

[0MAL]

I'm using mosparo's example of custom integration, verifyFormDataWithMosparo().
It works with previous test with PHP 8.
Now I downgraded php version to 7.0 for the previous test site, and same issue occurs.

[0MAL]

I found out Guzzle (Guzzle 7) requires PHP 7.2.5.
My production site is PHP 7.0 so I have to install Guzzle 6 and adapter if I use Guzzle for mosparo submission.

I'll test Guzzle 6, and if it fails, I would give up using mosparo submission. For that case, I can still use JS verification. I can't see submission history for contact form submission on Guzzle console, though.

Another option is creating subdomain only for contact form, then I can use PHP 8.

[0MAL]

Why expectedHmacHash and receivedHmacHash is different? My code is based on mosparo's example of custom integration, verifyFormDataWithMosparo(). I tried curl_setopt function instead of GuzzleHttp but still same error happens. I modified php files to change /tmp setting before, so do you know if Authentication process of mosparo is related to /tmp settings? Is it possible that is a culprit?

[zepich]

Hi @0MAL

Thank you very much for your question.

The different hash has nothing to do with the HTTP client (Guzzle/curl) you use and the /tmp change also has nothing to do with it.

The hash issue has two main reasons:

1. The payload is not the same
2. The private key is wrong

The payload is not the same

If the API debug mode is enabled, you will receive the two hashes from mosparo and the expected payload in the error response. Please compare this expected payload with the payload your code uses. The actual payload is in the variable $combinedApiEndpointJsonRequestData:

    // ...
    $requestData = [
        'submitToken' => $submitToken,
        'validationSignature' => $validationSignature,
        'formSignature' => $formDataSignature,
        'formData' => $preparedFormData,
    ];

    // 9. Generate the request signature
    $jsonRequestData = json_encode($requestData);
    $combinedApiEndpointJsonRequestData = $apiEndpoint . $jsonRequestData;

    // ^- THIS is your actual payload
    var_dump($combinedApiEndpointJsonRequestData); // Add this and then compare the values

    $requestSignature = hash_hmac('sha256', $combinedApiEndpointJsonRequestData, $projectPrivateKey);

    // 10. Send the API request
    $projectPublicKey = '<publicKey>'; // You can find this value in the project settings in mosparo
    $client = new \GuzzleHttp\Client([
        'base_uri' => 'https://mosparo.example.com', // The host of your mosparo installation
    ]);
    // ...

Output the actual payload and compare it with the expected payload you receive from the API. With these two values, you should see the difference. If there is no difference, then it's the private key.

The private key is wrong

Since the hash is calculated with the private key, using the right key is crucial. Can you verify that you use the correct private key and that there are no additional characters in it, such as spaces or anything else?

Kind regards,

zepich

[0MAL]

Thank you zepich.
I compared payloads and proceeded with debugging, I found out the polyfill code for PHP7 I borrowed for str_starts_with() function was wrong.
Now submission of the site is working fine.

[zepich]

Thank you very much for the update! I'm happy that it's working now!

[0MAL]

With the sample code of mosparo documents, varidation is run before submission and spam is filtered at that time then every submission is sent without spam, right? I am OK with that, but I am just curious about if it is possible to filter spam after submission. Because spammers notice spam protection when validation is run before submission, I think.

[zepich]

Hi @0MAL

Thank you very much for your question. That's a very important question.

As you know, mosparo has two steps:

1. The frontend validation (to validate the form data)
2. The backend verification (to verify that nobody changed the validated form data)

These two steps are intended to maximize the usability in the frontend, directly showing the user that something is not good (frontend validation) while ensuring that there is no option to bypass the protection (backend verification).

When the user clicks on the checkbox in the frontend, the mosparo frontend script sends the data to the mosparo validation API. If everything is good, mosparo stores the hashes of the form fields and returns a validation token to the frontend. This validation token is then stored in a hidden form field.

As soon as the user submits the form, the validation token, along with the other form data, is sent to your website's backend.

Now, your backend will verify the submitted data. This step is done with the backend verification code (which you now use to do the verification). This code uses the validation token, which the mosparo validation API generated earlier. This token is valid only once. Your backend generates the hashes for all the form fields and sends these hashes together with the submit and the validation token to the mosparo verification API.

The mosparo verification API now compares the hashes of all verifiable fields (which mosparo stores in the database when the user clicks the checkbox in the frontend) with the hashes that your backend sends to the verification API. If one of the hashes is not the same, the whole verification fails, and the submission is blocked. If all the hashes are the same and the validation token is correct, the submission is valid and not spam. If that's the case, your backend can now process the submission (for example, send the email or store the data in a database).

With these two methods, we can ensure that the fields cannot be manipulated after the frontend validation validates them as "no-spam" (as far as we can tell, please let us know if you see a possibility to exploit it).

So, to answer your question, technically, not every submission sent to your backend is valid. It is possible that a bad actor tried to manipulate the form fields after mosparo validated the form. However, after the backend verification, we can ensure the submission is not spam.

(Of course, mosparo can detect something as spam only if there is a rule or a security setting to detect it as spam. Otherwise, everything is "no-spam")

I hope that explanation answers your question. Otherwise, please let me know.

Kind regards,

zepich