This is a collection of questions to ask for given a paper, problem, or even a application design.

Questions

Did you like the paper? Did you find it interesting? Be honest!
What are the most important things you learned from the paper? Why are they important?
Do the lessons learned generalize beyond the specific task? Do they promote our understanding of language? Do they contribute towards building an important system or application?
Is the experimental setup satisfying? Any experiments missing? Any obvious or important baseline missing? Is the ablation analysis sufficient?
If a theoretical analysis is included, do you find it satisfying? If none is included, is it missing?
Is the problem/approach well motivated?
Are you convinced by the results? Why?
Is the writing clear? Is the paper well structured?

Data analysis

Why is this task difficult?
What are the hard cases?
What are the easy cases?
Can you think of a simple baseline? How well will it perform?
Why are the models discussed in class and readings appropriate?
Do these models make assumptions that hurt performance? How much do these assumptions hurt?
Is there an upper bound on performance?
What about the assumptions built into the annotation scheme? Any of them arbitrary?

What is the required technical background that is not in the paper and is required for understanding the method?
What are the main equations and/or algorithms in the paper?
What is the main innovation of the paper? How does it relate to previous work?
(Is the author's description of the previous work accurate? or misleading?)
What problem is the approach trying to solve?
Why is this a hard problem?
Can you think of a simpler baseline that will achieve simialr results?
Can you relate the approach and solution to specific data examples? (see "data analysis guidelines" above)
Is the approach specific to the current work, or can it be applied elsewhere?