Large number of Transactions for the logs when using WASB

[ahmadfarhan97]

We are using WASB (azure storage account - file share) for the loggings and it shows that the transactions between Airflow and the file share (solely used for loggings) is very large - more than 1 million transaction per hour.

I am wondering what is happening in the app that continuously require the read/write transaction and how to reduce that number.

Currently we have 4 DAGs running including 2 only run on hourly schedule. The Airflow is deployed to Kubernetes cluster using Helm Chart (v1.10.0). We are using persistent volume to mount to storage account.

The PV and PVC looks like this

apiVersion: v1
kind: PersistentVolume
metadata:
  name: logs-pv
  namespace: namespace
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteMany
  storageClassName: azurefile
  azureFile:
    secretName: secret-name
    secretNamespace: namespace
    shareName: logs
    readOnly: false
  mountOptions:
    - dir_mode=0777
    - file_mode=0777
    - uid=1000
    - gid=1000
    - mfsymlinks
    - cache=strict
    - actimeo=30
  claimRef:
    name: logs-pvc
    namespace: namespace

This is what we have in the values.yaml:

logs:
  persistence:
    # Enable persistent volume for storing logs
    enabled: true
    # Volume size for logs

    size: 1Gi
    # If using a custom storageClass, pass name here

    storageClassName:
    ## the name of an existing PVC to use
    existingClaim: logs-pvc

[Taragolis]

Remote shares, especially cloud based, could turned life into the pain, see: What resources might limit Scheduler’s performance

In additional there is report about this issue in SO in the past, seems like some stat calls could spawn additional Azure File Share transaction (whatever it was)

• https://stackoverflow.com/questions/66682516/too-many-transactions-to-azure-fileshare-from-airflow
• https://stackoverflow.com/questions/65274774/im-getting-a-lot-of-transactions-while-idling-airflow-and-azure-file-share

[ahmadfarhan97]

I tried changing file_parsing_sort_mode setting for scheduler to alphabetical but that has not helped. Bear in mind that we are using AKS and the file share is only used for logging and not DAGs files.
I am not sure what is there to change as well. I will appreciate any guidance.

[alex-bes-ccl]

Hi @ahmadfarhan97! did you manage to find a solution for this?