Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Running a Subset of the Test Suite #544

Merged
merged 3 commits into from
Dec 7, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/full-test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -33,5 +33,5 @@ jobs:
timeout-minutes: 25
r-version: ${{ matrix.r-version }}
image: ${{ matrix.os }}
action: test-full
action: test-full -- --forbid-only
coverage: false
2 changes: 1 addition & 1 deletion .github/workflows/qa.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ jobs:
if: ${{ github.event_name != 'pull_request' || github.event.pull_request.base.ref != 'main' }}
uses: Code-Inspect/flowr/.github/workflows/run.yaml@main
with:
action: test-full
action: test-full -- --forbid-only
coverage: true


Expand Down
6 changes: 1 addition & 5 deletions wiki/Capabilities.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,6 @@ and the grouping with `{`<sup>T61</sup> and `}`<sup>T62</sup>, the normalization

<li> <details> <summary><b>Separators</b>&emsp;(full)</summary>


Separators help R to distinguish between otherwise ambiguous expressions
like `a <- 1` and `a < -1`. They are partially defined by [Section 10.3.5](https://cran.r-project.org/doc/manuals/r-release/R-lang.html#Separators),
with more separator tokens on the [Tokens](https://github.com/Code-Inspect/flowr/wiki/Tokens#tokens-used-to-delimit-parts-of-expressions) wiki page. The most common
Expand All @@ -142,7 +141,6 @@ this is [discouraged](https://style.tidyverse.org/syntax.html#semicolons), and n
Nevertheless, the normalization of *flowR* does handle the use of semicolons, as well as `,`<sup>T59</sup>, in combination with arguments and parameters.
</details></li>


</ol>

### Values
Expand Down Expand Up @@ -274,13 +272,12 @@ special way (although we prevent assignments and references, similar to
our handling of `NULL`).
</details></li>


</ol>

### Data Types

R is dynamically typed and assigns a type to each R object, which can be
queried at runtime with the typeof function. The list of existing types is
queried at runtime with the `typeof` function. The list of existing types is
documented at the beginning of [Section 2](https://cran.r-project.org/doc/manuals/r-release/R-lang.html#Objects).
These types include symbols, primitive types like characters, booleans (logical values), integers,
doubles, complex numbers, and more. Since the intricate details of these
Expand Down Expand Up @@ -420,7 +417,6 @@ corresponding change in the token type from `EQ_ASSIGN` to `EQ_SUB`
and `EQ_FORMALS`.
For another important differentiation when used in arguments, we recommend Section&nbsp;8.2.26 of the great [The R Inferno](https://www.burns-stat.com/pages/Tutor/R_inferno.pdf) by Burns.


Besides using `=`, there is the older variant of the local equal assignment
using an additional colon: `:=`. While (from [lintr](https://github.com/r-lib/lintr/blob/3d9e6d78efe7fc41d8b545a845a984a0821cfbbe/R/assignment_linter.R#L95-L96))
> it’s [`:=`] extremely uncommon as a normal assignment operator
Expand Down
1 change: 0 additions & 1 deletion wiki/Core.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,4 +32,3 @@ If you want to execute a single step (for whatever reason) there are two ways to
#### Step 4: Slicing on the Dataflow Graph

#### Step 5: Reconstruct the R Code

1 change: 0 additions & 1 deletion wiki/Home.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
[![flowR Logo](img/flowR.png)](https://github.com/Code-Inspect/flowr)


***This wiki page is currently under construction***

*flowR* is a static [dataflow analyzer](https://en.wikipedia.org/wiki/Data-flow_analysis) and [program slicer](https://github.com/Code-Inspect/flowr/wiki/Terminology#program-slice) for the [*R*](https://www.r-project.org/) programming language (currently tested for versions `4.x`), and you have stumbled upon its wiki pages!
Expand Down
4 changes: 1 addition & 3 deletions wiki/Interface.md
Original file line number Diff line number Diff line change
Expand Up @@ -693,7 +693,6 @@ Within the started session, type the following message and press enter to see th
{"type": "request-file-analysis","id":"0","filetoken":"x","content":"x <- 1\nx + 1"}
```


##### Using Python

In python, a similar process would look like this.
Expand Down Expand Up @@ -735,6 +734,7 @@ sequenceDiagram
end
deactivate Server
```

</details>

In order to slice, you have to send a file analysis request first. The `filetoken` you assign is of use here as you can re-use it to repeatedly slice the same file.
Expand Down Expand Up @@ -953,7 +953,6 @@ With a shell object (let's call it `shell`), you can execute R code by using `RS

Besides that, the command `RShell::tryToInjectHomeLibPath` may be of interest, as it enables all libraries available on the host system.


### Slicing With The `SteppingSlicer`

The main class that represents *flowR*'s slicing is the [`SteppingSlicer`](https://code-inspect.github.io/flowr/doc/classes/src_core_slicer.SteppingSlicer.html) class. With *flowR*, this allows you to slice code like this:
Expand Down Expand Up @@ -1095,7 +1094,6 @@ Whenever this name appears, you may substitute this with whatever name fits your
3. **Add it to the feature-mapping**\
Now, in the `feature.ts` file in `src/statistics/features`, add your feature to the `ALL_FEATURES` object.


Now, we want to extract something. For the *example* feature created in the previous steps, we choose to count the amount of `COMMENT` tokens.
So we define a corresponding [XPath](https://developer.mozilla.org/en-US/docs/Web/XPath) query:

Expand Down
12 changes: 9 additions & 3 deletions wiki/Linting and Testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ For the latest code-coverage information, see [codecov.io](https://codecov.io/gh
- [Functionality Tests](#functionality-tests)
- [Test Structure](#test-structure)
- [Writing a Test](#writing-a-test)
- [Running Only Some Tests](#running-only-some-tests)
- [Performance Tests](#performance-tests)
- [CI Pipeline](#ci-pipeline)
- [Linting](#linting)
Expand All @@ -30,7 +31,7 @@ What may be counter-intuitive is that this does not run *all* tests by default b
npm run test-full
```

However, depending on your local R version, your network connection and potentially other factors, some tests may be skipped automatically as they do not apply to your current system setup (or can't be tested with the current prerequisites). Each test can speciy such requirements as part of the `TestConfiguration`, which is then enforced by the [`ensureConfig`](https://github.com/Code-Inspect/flowr/blob/main/test/functionality/_helper/shell.ts) function.
However, depending on your local R version, your network connection and potentially other factors, some tests may be skipped automatically as they do not apply to your current system setup (or can't be tested with the current prerequisites). Each test can specify such requirements as part of the `TestConfiguration`, which is then enforced by the [`ensureConfig`](https://github.com/Code-Inspect/flowr/blob/main/test/functionality/_helper/shell.ts) function.
It is up to the [ci](#ci-pipeline) to run the tests on different systems to ensure that those tests are ensured to run.

#### Test Structure
Expand All @@ -49,7 +50,12 @@ Besides folders can (theoretically) arbitrarily structure their tests. We use th

#### Writing a Test

Currently this is heavily dependend on what you want to test (normalization, dataflow, quad-export, ...) and it is probably best to have a look at existing tests in that area to get an idea of what comfort functionality is available.
Currently this is heavily dependent on what you want to test (normalization, dataflow, quad-export, ...) and it is probably best to have a look at existing tests in that area to get an idea of what comfort functionality is available.

#### Running Only Some Tests

To run only some of the tests, the mocha test runner allows several options.
You can specify so [in the code](https://mochajs.org/#exclusive-tests) or [via the command line](https://mochajs.org/#-grep-regexp-g-regexp). If you modify the code, please make sure to remove the changes before committing!

### Performance Tests

Expand All @@ -70,7 +76,7 @@ We have several workflows defined in [.github/workflows](../.github/workflows/)
We explain the most important workflows in the following:

- [run.yaml](../.github/workflows/run.yaml) is a [reusable workflow](https://docs.github.com/en/actions/using-workflows/reusing-workflows) that we use *whenever* we want to run a *flowR* npm script in the pipeline (furthermore, this hides necessary extra steps for coverage, documentation, and benchmarks).
- [qa.yaml](../.github/workflows/qa.yaml) is the main workflow which will run different steps depending on several factors. It is repsonsible for:
- [qa.yaml](../.github/workflows/qa.yaml) is the main workflow which will run different steps depending on several factors. It is responsible for:
- running the [functionality](#functionality-tests) and [performance tests](#performance-tests)
- uploading the results to the [benchmark page](https://code-inspect.github.io/flowr/wiki/stats/benchmark) for releases
- running the [functionality tests](#functionality-tests) on different operating systems (Windows, macOS, Linux) and with different versions of R
Expand Down
3 changes: 0 additions & 3 deletions wiki/Overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,6 @@ The [core](https://github.com/Code-Inspect/flowr/tree/main/src/core) module cont

The [utility](https://github.com/Code-Inspect/flowr/tree/main/src/util) module is of no further interest for the usage of *flowR*


The following sections explain how to use these features.

## Using *flowR*
Expand Down Expand Up @@ -129,7 +128,6 @@ Sleep 200ms

The server allows to access the REPL as well (see the [interface](https://github.com/Code-Inspect/flowr/wiki/Interface) wiki page for more information).


## Calling the Scripts Directly

This describes the old way of using *flowR* by creating and calling the respective scripts directly. Although this is no longer necessary, the scripts still remain, fully integrated into the REPL of *flowR* (you can access them simply by adding a colon `:` before the name).
Expand Down Expand Up @@ -216,7 +214,6 @@ See
npm run stats -- --help
```


### Export RDF N-Quads

If you know what the RDF N-Quads refer to, then you are good to go! If not, you do not have to worry - you probably do not need them (they are for a graph search that is based on *flowR*). See the following for more information:
Expand Down
2 changes: 0 additions & 2 deletions wiki/Setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,6 @@ This should drop you into *flowR*'s read-evaluate-print loop.
Enter `:help` to receive more information and `:quit` to leave.
Please remember, that you have to link external directories to make them available within the running container:


## ⚒️ Building From Scratch

To use *flowR*, you need [*R*](https://www.r-project.org/) installed and on your path.
Expand Down Expand Up @@ -55,7 +54,6 @@ for(i in 1:(N-1)) product <- product * i
cat("Product:", product, "\n")
```


At the time of writing this, there is currently no page for frequently encountered errors. So just [message me](mailto:florian.sihler@uni-ulm.de) in case of problems.

## 📜 Developing for *flowR*
Expand Down
11 changes: 2 additions & 9 deletions wiki/Terminology.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
Based on the explanations in my [master's thesis](http://dx.doi.org/10.18725/OPARU-50107), this page collects the definitions for terms which are of interest in the context of *flowR*.


<!-- TOC -->
- [Program Slicing](#program-slicing)
- [Slicing Criterion](#slicing-criterion)
Expand All @@ -13,7 +12,6 @@ Based on the explanations in my [master's thesis](http://dx.doi.org/10.18725/OPA
- [Dynamic](#dynamic)
<!-- TOC -->


## Program Slicing

For more information on different slicing variants, refer to Section&nbsp;2.1.3 of my [master's thesis](http://dx.doi.org/10.18725/OPARU-50107).
Expand All @@ -36,7 +34,6 @@ Weiser originally defined it as a combination of a line number and a set of vari

Furthermore, slicing criteria can be joined by the use of a `;`-semicolon. `10@sum;12@product` refers to the first occurrence of `sum` in line 10 and the first occurrence of `product` in line 12 as two variables to be used for slicing.


### Program Slice

Based on a slicing criterion and a program, a program slice is a subset of the program that either
Expand Down Expand Up @@ -64,7 +61,7 @@ print(x + z)

Let's slice for the last definition of `x` in line 9 (`x <- y`).

* the smallest [backward slice](#backward-slice) is:
- the smallest [backward slice](#backward-slice) is:

```R
if(x > 4) {
Expand All @@ -79,17 +76,14 @@ Let's slice for the last definition of `x` in line 9 (`x <- y`).
It does not contain `z <- 3` as it is not necessary to define `x` in the last line, however it includes the if's condition, as it decides the active branch and hence the active definition for `y` in the definition.
If `x <- y` itself is to be included in the slice depends on the program slicer. *flowR* includes it.


* the smallest [forward slice](#forward-slice) is:
- the smallest [forward slice](#forward-slice) is:

```R
print(x + z)
```

It contains all statements affected by the definition of `x`. *flowR* does not support forward slicing in its current form.



### Slicing Direction

#### Backward Slice
Expand All @@ -102,7 +96,6 @@ See the explanation of [program slices](#program-slice) for an example.
Based on a [slicing criterion](#slicing-criterion) and a program, a forward slice is a subset of the program that might be influenced by the [slicing criterion](#slicing-criterion).
See the explanation of [program slices](#program-slice) for an example.


### Information Used for Slicing

#### Static
Expand Down
8 changes: 4 additions & 4 deletions wiki/Thesis.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ The (sorted) results should be versioned and can be found in [scripts/top-r-down
cd scripts/ && Rscript top-downloads.R
```

1. If you haven't done so already, or updated the package list in the previous step, **download** the package sources.
2. If you haven't done so already, or updated the package list in the previous step, **download** the package sources.
For this, you can use the [scripts/download-top-pkg-sources.R](../scripts/download-top-pkg-sources.R) script.
But **make sure**, that you set the `to` variable to the output path you want.

Expand All @@ -33,13 +33,13 @@ The (sorted) results should be versioned and can be found in [scripts/top-r-down

Downloading and extracting the sources can take a while.

2. Make sure you have the latest version of the *flowr* package installed.
3. Make sure you have the latest version of the *flowr* package installed.

```shell
npm ci
```

3. Now you can run the statistics program on the downloaded sources.
4. Now you can run the statistics program on the downloaded sources.
You can do this in two ways (check `npm run stats -- --help` for more information).
In any case, the extraction may take a long time, so be prepared for that!
Furthermore, you may want to store the output of the tool as it provides additional information.
Expand All @@ -63,7 +63,7 @@ The (sorted) results should be versioned and can be found in [scripts/top-r-down
Theoretically, you should be able to stop the extraction at any time and still get usable information with the next step,
of course limited to only those files that have been processed so far.

4. Afterward, your output folder should contain several folders with the recorded stats of all extracted features.
5. Afterward, your output folder should contain several folders with the recorded stats of all extracted features.
To make sense of them, you need to use the post-processor, which prints the summarized information to the command-line:

```shell
Expand Down