In this repository are found some additional materials about the research.
Seed Papers: CSV | Seed Papers Page
Each search engine has its own form of advanced search, so the search string must be adjusted for each of the devices used in the review. Therefore, the search strings used in each machine are listed below.
-
ACM (CSV):
(Title:("code comprehension") OR Title:("code understandability") OR Title:("code understanding") OR Title:("code readability") OR Title:("code complexity") OR Title:("code misunderstanding") OR Title:("code patterns") OR Title:("program comprehension") OR Title:("program understandability") OR Title:("program understanding") OR Title:("program readability") OR Title:("program idioms") OR Title:("program style") OR Title:("program patterns") OR Title:("programmer experience")) OR (Keyword:"code comprehension" OR Keyword:"code understandability" OR Keyword:"code understanding" OR Keyword:"code readability" OR Keyword:"code complexity" OR Keyword:"code misunderstanding" OR Keyword:"code patterns" OR Keyword:"program comprehension" OR Keyword:"program understandability" OR Keyword:"program understanding" OR Keyword:"program readability" OR Keyword:"program idioms" OR Keyword:"program style" OR Keyword:"program patterns" OR Keyword:"programmer experience") -
IEEE (CSV):
("Document Title":"code comprehension" OR "Document Title":"code understandability" OR "Document Title":"code understanding" OR "Document Title":"code readability" OR "Document Title":"code complexity" OR "Document Title":"code misunderstanding" OR "Document Title":"code patterns" OR "Document Title":"program comprehension" OR "Document Title":"program understandability" OR "Document Title":"program understanding" OR "Document Title":"program readability" OR "Document Title":"program idioms" OR "Document Title":"program style" OR "Document Title":"program patterns" OR "Document Title":"programmer experience") OR ("Author Keywords":"code comprehension" OR "Author Keywords":"code understandability" OR "Author Keywords":"code understanding" OR "Author Keywords":"code readability" OR "Author Keywords":"code complexity" OR "Author Keywords":"code misunderstanding" OR "Author Keywords":"code patterns" OR "Author Keywords":"program comprehension" OR "Author Keywords":"program understandability" OR "Author Keywords":"program understanding" OR "Author Keywords":"program readability" OR "Author Keywords":"program idioms" OR "Author Keywords":"program style" OR "Author Keywords":"program patterns" OR "Author Keywords":"programmer experience") -
Scopus (CSV):
((TITLE("code comprehension") OR TITLE("code understandability") OR TITLE("code understanding") OR TITLE("code readability") OR TITLE("program comprehension") OR TITLE("program understandability") OR TITLE("program understanding") OR TITLE("program readability") OR TITLE("programmer experience")) OR (KEY("code comprehension") OR KEY("code understandability") OR KEY("code understanding") OR KEY("code readability") OR KEY("program comprehension") OR KEY("program understandability") OR KEY("program understanding") OR KEY("program readability") OR KEY("programmer experience"))) AND ( LIMIT-TO ( DOCTYPE,"cp" ) OR LIMIT-TO ( DOCTYPE,"ar" ) ) AND ( LIMIT-TO ( SUBJAREA,"COMP" ))
In this directory, there is a list of all the papers found in each search engine. This directory is organized according to the exclusion phase and inclusion phase. The criteria evaluated in each phase are listed below.
-
All Papers (after merge all engines results)
| ID | Description |
|---|---|
| EC1 | The study is clearly irrelevant to our research questions, i.e., it is not primarily related to source code comprehension, readability, legibility, or hard-to-understand code, or does not involve any comparison of different ways of writing code. |
| EC2 | The study is not a full paper (e.g., PhD theses or 4-page papers) or is not written in English: not considering these types of documents in systematic reviews is a common practice. |
| EC3 | The study is about legibility, readability, or understandability metrics. |
| EC4 | The study is about program comprehension aids, such as visualizations or other forms of analysis or sensory aids, e.g., trace-based execution and code summarization. |
| EC5 | The study focuses on accessibility, e.g., targets individuals with visual impairments or neurodiverse developers. |
| ID | Description |
|---|---|
| IC1 (Scope) | The study must be primarily related to legibility. |
| IC2 (Methodology) | The study must be or contain a controlled experiment, quasi-experiment, or survey involving human subjects. |
| IC3 (Comparison) | The study must directly compare alternative formatting elements in terms of code legibility and the alternatives must be clearly identifiable. |
| IC4 (Granularity) | The study must target fine-grained program elements and low-level/limited-scope programming activities. Not design or comments, but implementation. |
In the quality assessment stage of primary studies included in the systematic review, we used a questionnaire as a guide. This questions used in this phase are listed below.
- Questions about design: QA1, QA2, QA3
- Questions about analysis and rigor: QA4, QA5, QA6, QA7
- Questions about conclusion: QA8, QA9
| ID | Questions for quality assessment for study |
|---|---|
| QA1 | Are the aims clearly stated? |
| QA2 | Are the independent variables adequately defined? |
| QA3 | Are the dependent variables adequately defined? |
| QA4 | Are the data collection methods adequately described? |
| QA5 | Are the study participants or observational units adequately described? For example, SE experience, type (student, practitioner, consultant), nationality, task experience and other relevant variables. |
| QA6 | Were the collected data adequately described (e.g., descriptive statistics)? |
| QA7 | Did the study employ statistical methods to analyze the data? |
| QA8 | Is there a clear statement of findings, i.e., are all study questions answered? |
| QA9 | Do the researchers explain the threats to the study validity? |
- Data (csv | page-level-1 | page-level-2)
| Level | # | Source paper | Link | Backward | Forward |
|---|---|---|---|---|---|
| 1 | 1 | an experimental investigation of the effect of program structure on program understanding | Link | 13 | 66 |
| 1 | 2 | an eye tracking study on camelcase and underscore identifier styles | Link | 27 | 170 |
| 1 | 3 | an investigation of misunderstanding code patterns in c opensource software projects | Link | 47 | 19 |
| 1 | 4 | atoms of confusion in java | Link | 27 | 4 |
| 1 | 5 | enhancing program comprehension formatting and documenting | Link | 19 | 18 |
| 1 | 6 | impacts of coding practices on readability | Link | 29 | 40 |
| 1 | 7 | indentation simply a matter of style or support for program comprehension | Link | 40 | 19 |
| 1 | 8 | measuring neural efficiency of program comprehension | Link | 48 | 110 |
| 1 | 9 | program indentation and comprehensibility | Link | 22 | 219 |
| 1 | 10 | software readability practices and the importance of their teaching | Link | 22 | 7 |
| 1 | 11 | the effect of scope delimiters on program comprehension | Link | 9 | 6 |
| 1 | 12 | the impact of identifier style on effort and comprehension | Link | 47 | 156 |
| 1 | 13 | typographic style is more than cosmetic | Link | 16 | 138 |
| 1 | 14 | understanding misunderstandings in source code | Link | 43 | 51 |
| 2 | 15 | to camelcase or under_score | Link | 21 | 154 |
| 2 | 16 | extended models on the impact of identifier style on effort and comprehension | Link | 44 | 1 |
| 2 | 17 | a look at programmers communicating through program indentation | Link | 20 | 0 |
| Total | References + Citations = 1672 | 494 | 1178 |