Awesome Machine learning for discovery of physical laws (ML4PhysicsLAW)

A curated list of awesome resources on using machine learning and data science for discovery of physical laws,inspired by awesome-computer-vision.

For a list people in ML4PhysicsLaw, please visit here

Edited by Machine Learning and Evolution Laboratory at University of South Carolina

Contributing

Please feel free to send me pull requests or email Dr. Jianjun Hu at University of South Carolina(hujianju@gmail.com) to add links.

---

Table of Contents

Reading list to get inspired

• Artificial Intelligence and Augmented Intelligence for Automated Investigations for Scientific Discovery network
• Nuclear Fusion and Artificial Intelligence: the Dream of Limitless Energy from AI daily
• Is Fusion Really Close To Reality? Yes, Thanks To Machine Learning. Forbes
• Artificial Intelligence Accelerates Development of Limitless Fusion Energy. article
• Scientists think we’ll finally solve nuclear fusion thanks to cutting-edge AI. link
• Containing the sun using Deep-learning AI. Harvard link
• AI for global health. Talk
• Blending physics with artificial intelligence.
• Advancing Fusion with Machine Learning workshop report identified several areas that ML can help fusion physics.
• Machine Learning Reveals Quantum Phases of Matter
• AI helps unlock 'dark matter' of bizarre superconductors
• Can AI help crack the code of fusion power? theverge
• AI could be the perfect tool for exploring the Universe
• Tackling Climate Change with Machine Learning ICLR2020 workshop papers
• A machine-learning revolution for physics and materials science...
• Machine learning meets quantum physics
• Machine learning versus physics-based modeling
• AI Copernicus ‘discovers’ that Earth orbits the Sun
• AI Teaches Itself Laws of Physics

Exemplary research works

• "Distilling free-form natural laws from experimental data." Science 324, no. 5923 (2009): 81-85. Link and Citations by Schmidt, Michael, and Hod Lipson.
• "AI Feynman: A physics-inspired method for symbolic regression." link Science Advances 6, no. 16 (2020): eaay2631. Udrescu, Silviu-Marian, and Max Tegmark.
• Predicting disruptive instabilities in controlled fusion plasmas through deep learning. Nature 2020. link
• K. T. Schütt, M. Gastegger, A. Tkatchenko, K.-R. Müller, R. J. Maurer. Unifying machine learning and quantum chemistry with a deep neural network for molecular wavefunctions. Nature Communications, 2019; 10 (1) DOI: 10.1038/s41467-019-12875-2
• Discovering Physical Concepts with Neural Networks

Applications

• Finding Strong Gravitational Lenses in the Kilo Degree Survey with Convolutional Neural Networks. arxiv
• Physics-Guided Machine Learning for Scientific Discovery: An Application in Simulating Lake Temperature Profiles

Survey/Review papers

• "Integrating physics-based modeling with machine learning: A survey." PDF 2020 by Willard, Jared, Xiaowei Jia, Shaoming Xu, Michael Steinbach, and Vipin Kumar
• Achuta Kadambi "Blending physics with artificial intelligence", Proc. SPIE 11396, Computational Imaging V, 113960B (24 April 2020); https://doi.org/10.1117/12.2565099
• Machine learning and the physical sciences arxiv
• The power of machine learning. Nature physics. link
• Fast Differentiable Sorting and Ranking. arxiv
• Gradient Boosting Neural Networks: GrowNet. arxiv
• Learning with Differentiable Perturbed Optimizers. arxiv
• The Costs and Benefits of Goal-Directed Attention in Deep Convolutional Neural Networks. pdf
• The Geometry of Sign Gradient Descent. arxiv
• The large learning rate phase of deep learning: the catapult mechanism. arxiv

Symbolic regression for equation search/discovery

Causal relationship learning/discovery

• Causal machine learning papers link

• Causal Discovery with Attention-Based Convolutional Neural Networks” by Meike Nauta * , Doina Bucur and Christin Seifert

• https://aip.scitation.org/doi/full/10.1063/1.5025050

• https://academic.oup.com/bioinformatics/article/32/6/875/1744279

• https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0103812

• https://github.com/khurramjaved96/online-causal-models

• https://github.com/capybaralet/REx_code_release

• https://github.com/reiinakano/invariant-risk-minimization

• https://github.com/IBM/IRM-games

• https://github.com/logangraham/arXausality arxiv causal ML algorithms papers

• https://github.com/robertness/causalML

• https://github.com/FenTechSolutions/CausalDiscoveryToolbox

• https://github.com/microsoft/dowhy causal inference

ODE derivation, equation inference

-Physics Based Deep Learning

Bayesian networks

Interpretable AI/ML, including Inpretable deep neural networks

Topological analysis

feature/representation learning (DL, ISOMAP, ICA,ALAE)

• GLAD: Learning Sparse Graph Recovery

Uncertainty quantification

• Physics-Guided Machine Learning for Scientific Discovery: An Application in Simulating Lake Temperature Profiles. link

Graph neural networks, graph deep learning

Reinforcement learning for active learning and experiment planning/design

ML/DL for design of experiments, active learning to do minimum no. of experiments/sampling

Disentangling of features by deep learning

Multi-scale issues

Dimension reduction and visualization

Active learning for experiments design

active learning, sequential decision making, experimental design, reinforcement learning, interactive learning or generative learning

• Workshop I: From Passive to Active: Generative and Reinforcement Learning with Physics link talks

Extrapolation capability and model robustness

• building a model and try to break it to make it robust

Deep learning resources'

• github Awesome deep learning resources

AutoML

• When NAS Meets Robustness: In Search of Robust Architectures against Adversarial Attacks. arxiv code
• Gryffin: An algorithm for Bayesian optimization for categorical variables informed by physical intuition with applications to chemistry. arxiv
• Uncertainty Quantification for Bayesian Optimization. pdf

Dimension reduction

• An Idea From Physics Helps AI See in Higher Dimensions Blog
• Unbalanced GANs: Pre-training the Generator of Generative Adversarial Network using Variational Autoencoder. pdf

GNN

• Bridging the Gap Between Spectral and Spatial Domains in Graph Neural Networks. arxiv code
• Generalization and Representational Limits of Graph Neural Networks. arxiv
• SIGN: Scalable Inception Graph Neural Networks. arxiv
• StickyPillars: Robust feature matching on point clouds using Graph Neural Networks. arxiv
• Supervised Learning on Relational Databases with Graph Neural Networks. arxiv code

Meta Learning

• A Comprehensive Overview and Survey of Recent Advances in Meta-Learning. arxiv
• Meta-Learning in Neural Networks: A Survey. arxiv
• Regularizing Meta-Learning via Gradient Dropout. arxiv

Research opportunities of ML for physics

• 'Science Discovery with Machine Learning' involves bridging gaps in theoretical understanding via identification of missing effects using large datasets; the acceleration of hypothesis generation and testing and the optimisation of experimental planning. Essentially, machine learning is used to support and accelerate the scientific process itself.
• 'Machine Learning Boosted Diagnostics' is where machine learning methods are used to maximise the information extracted from measurements, systematically fuse multiple data sources and infer quantities that are not directly measured. Classifcation techniques, such as supervised learning, could be used on data that is extracted from the diagnostic measurements.
• 'Model Extraction and Reduction' includes the construction of models of fusion systems and the acceleration of computational algorithms. Effective model reduction can result in shorten computation times and mean that simulations (for the tokamak fusion reactor for example) happen faster than real-time execution.
• 'Control Augmentation with Machine Learning'. Three broad areas of plasma control research would benefit significantly from machine learning: control-level models, real-time data analysis algorithms; optimisation of plasma discharge trajectories for control scenarios. Using AI to improve control mathematics could manage the uncertainty in calculations and ensure better operational performance.
• 'Extreme Data Algorithms' involves finding methods to manage the amount and speed of data that will be generated during the fusion models.
• 'Data-Enhanced Prediction' will help monitor the health of the plant system and predict any faults, such as disruptions which are essential to be mitigated.
• 'Fusion Data Machine Learning Platform' is a system that can manage, format, curate and enable the access to experimental and simulation data from fusion models for optimal usability when used by machine learning algorithms.

Conferences and Workshops related

• UCLA IPAM Machine Learning for Physics and the Physics of Learning Link talks
• UCLA IPAM Machine Learning for Physics and the Physics of Learning Tutorials link
• Workshop II: PDE and Inverse Problem Methods in Machine Learning IPAM link
• Workshop IV: Using Physical Insights for Machine Learning link
• Workshop III: Validation and Guarantees in Learning Physical Models: from Patterns to Governing Equations to Laws of Nature link
• Workshop II: Interpretable Learning in Physical Sciences link talks
• Workshop I: From Passive to Active: Generative and Reinforcement Learning with Physics link
• Workshop IV: Deep Geometric Learning of Big Data and Applications. Non-Euclidean domain deep learning. videos link 3D point clouds and 3D shapes in computer graphics, functional MRI signals on the brain structural connectivity network, the DNA of the gene regulatory network in genomics, drugs design in quantum chemistry, neutrino detection in high energy physics, and knowledge graph for common sense understanding of visual scenes. graphs and manifolds. Fundamental operations such as convolution, coarsening, multi-resolution, causality have been redefined through spectral and spatial approaches.
• ML4science workshop

Journals

• Machine Learning: Science and Technology

software toolls

• AI Feynman symbolic regression package
• Eureka commercial
• Integration of Neural Network-Based Symbolic Regression in Deep Learning for Scientific Discovery
• [S. S. Sahoo, C. H. Lampert, and G. Martius, “Learning Equations for Extrapolation and Control,” jun 2018. [Online](https://arxiv.org/abs/1806.07259）
• fast symbolic regression

Tutorials

• Physics-Informed Learning Machines

Resources for students

Resource link collection

• Resources for students - Frédo Durand (MIT)
• Advice for Graduate Students - Aaron Hertzmann (Adobe Research)
• Graduate Skills Seminars - Yashar Ganjali, Aaron Hertzmann (University of Toronto)
• Research Skills - Simon Peyton Jones (Microsoft Research)
• Resource collection - Tao Xie (UIUC) and Yuan Xie (UCSB)

Writing

• Write Good Papers - Frédo Durand (MIT)
• Notes on writing - Frédo Durand (MIT)
• How to Write a Bad Article - Frédo Durand (MIT)
• How to write a good CVPR submission - William T. Freeman (MIT)
• How to write a great research paper - Simon Peyton Jones (Microsoft Research)
• How to write a SIGGRAPH paper - SIGGRAPH ASIA 2011 Course
• Writing Research Papers - Aaron Hertzmann (Adobe Research)
• How to Write a Paper for SIGGRAPH - Jim Blinn
• How to Get Your SIGGRAPH Paper Rejected - Jim Kajiya (Microsoft Research)
• How to write a SIGGRAPH paper - Li-Yi Wei (The University of Hong Kong)
• How to Write a Great Paper - Martin Martin Hering Hering--Bertram (Hochschule Bremen University of Applied Sciences)
• How to have a paper get into SIGGRAPH? - Takeo Igarashi (The University of Tokyo)
• Good Writing - Marc H. Raibert (Boston Dynamics, Inc.)
• How to Write a Computer Vision Paper - Derek Hoiem (UIUC)
• Common mistakes in technical writing - Wojciech Jarosz (Dartmouth College)

Presentation

• Giving a Research Talk - Frédo Durand (MIT)
• How to give a good talk - David Fleet (University of Toronto) and Aaron Hertzmann (Adobe Research)
• Designing conference posters - Colin Purrington
• Physics-Based-Deep-Learning

Research

• How to do research - William T. Freeman (MIT)
• You and Your Research - Richard Hamming
• Warning Signs of Bogus Progress in Research in an Age of Rich Computation and Information - Yi Ma (UIUC)
• Seven Warning Signs of Bogus Science - Robert L. Park
• Five Principles for Choosing Research Problems in Computer Graphics - Thomas Funkhouser (Cornell University)
• How To Do Research In the MIT AI Lab - David Chapman (MIT)
• Recent Advances in Computer Vision - Ming-Hsuan Yang (UC Merced)
• How to Come Up with Research Ideas in Computer Vision? - Jia-Bin Huang (UIUC)
• How to Read Academic Papers - Jia-Bin Huang (UIUC)

Time Management

• Time Management - Randy Pausch (CMU)

Blogs

• Learn OpenCV - Satya Mallick
• Tombone's Computer Vision Blog - Tomasz Malisiewicz
• Computer vision for dummies - Vincent Spruyt
• Andrej Karpathy blog - Andrej Karpathy
• AI Shack - Utkarsh Sinha
• Computer Vision Talks - Eugene Khvedchenya
• Computer Vision Basics with Python Keras and OpenCV - Jason Chin (University of Western Ontario)

Links

• The Computer Vision Industry - David Lowe
• German Computer Vision Research Groups & Companies
• awesome-deep-learning
• awesome-machine-learning
• Cat Paper Collection
• Computer Vision News
• Most cited researchers on Google scholars

Licenses

License

Acknowledgement

This awesome list is made possible by the NSF HDR Grant. Award #1940099 Collaborative Research: Integrating Physics and Generative Machine Learning Models for Inverse Materials Design. and the NSF HDR PI workshop April28-30, 2020.