Dialog is an accessible modal extension which is the default modal for Shadows. The CSS is part of /themes/%STORE_ID%/THEME_NAME/core.css
. If you want to make styling changes, it is recommended you add them to your /themes/%STORE_ID%/THEME_NAME/theme.css
file.
If you are using Shadows DE, you can modify the CSS in build/core/css/components/_dialog.scss
and update as part of your build process.