Skip to content

Commit

Permalink
Static analysis check based on imports in source files (#126)
Browse files Browse the repository at this point in the history 
* Fix MavenInvokerTest

* Add explicit encoding for the failsafe-plugin to avoid log warnings

* Edit description

* Update to JDK17 and prepare for new release

* Fix IT

* Consider classes from imports in the source code of the Maven project as used

* Add test

* Fix IT

* bugfix: class files in dependencies where not analyzed

* Remove logs

* Workaround

* Fix IT and remove logs

* Fix #125

* Fix checkstyle error

* Refactoring workaround

* Fix test

* Test DefaultCallGraph
  • Loading branch information
@cesarsotovalero
cesarsotovalero authored Apr 5, 2022
1 parent 3e15d36 commit 5c13f9d
Show file tree
Hide file tree
Showing 42 changed files with 638 additions and 151 deletions.
23 changes: 12 additions & 11 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -89,17 +89,18 @@ If all the tests pass, and the project builds correctly after these changes, the

The Maven plugin can be configured with the following additional parameters.

| Name | Type | Description |
|:----------|:-------------:| :-------------|
| `<ignoreDependencies>` | `Set<String>` | Add a list of dependencies, identified by their coordinates, to be ignored by DepClean during the analysis and considered as used dependencies. Useful to override incomplete result caused by bytecode-level analysis. **Dependency format is:** `groupId:artifactId:version:scope`.|
| `<ignoreScopes>` | `Set<String>` | Add a list of scopes, to be ignored by DepClean during the analysis. Useful to not analyze dependencies with scopes that are not needed at runtime. **Valid scopes are:** `compile`, `provided`, `test`, `runtime`, `system`, `import`. An Empty string indicates no scopes (default).|
| `<ignoreTests>` | `boolean` | If this is true, DepClean will not analyze the test classes in the project, and, therefore, the dependencies that are only used for testing will be considered unused. This parameter is useful to detect dependencies that have `compile` scope but are only used for testing. **Default value is:** `false`.|
| `<createPomDebloated>` | `boolean` | If this is true, DepClean creates a debloated version of the pom without unused dependencies called `debloated-pom.xml`, in the root of the project. **Default value is:** `false`.|
| `<createResultJson>` | `boolean` | If this is true, DepClean creates a JSON file of the dependency tree along with metadata of each dependency. The file is called `depclean-results.json`, and is located in the root of the project. **Default value is:** `false`.|
| `<failIfUnusedDirect>` | `boolean` | If this is true, and DepClean reported any unused direct dependency in the dependency tree, the build fails immediately after running DepClean. **Default value is:** `false`.|
| `<failIfUnusedTransitive>` | `boolean` | If this is true, and DepClean reported any unused transitive dependency in the dependency tree, the build fails immediately after running DepClean. **Default value is:** `false`.|
| `<failIfUnusedInherited>` | `boolean` | If this is true, and DepClean reported any unused inherited dependency in the dependency tree, the build fails immediately after running DepClean. **Default value is:** `false`.|
| `<skipDepClean>` | `boolean` | Skip plugin execution completely. **Default value is:** `false`.|
| Name | Type | Description |
|:---------------------------|:-------------:|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `<ignoreDependencies>` | `Set<String>` | Add a list of dependencies, identified by their coordinates, to be ignored by DepClean during the analysis and considered as used dependencies. Useful to override incomplete result caused by bytecode-level analysis. **Dependency format is:** `groupId:artifactId:version:scope`. |
| `<ignoreScopes>` | `Set<String>` | Add a list of scopes, to be ignored by DepClean during the analysis. Useful to not analyze dependencies with scopes that are not needed at runtime. **Valid scopes are:** `compile`, `provided`, `test`, `runtime`, `system`, `import`. An Empty string indicates no scopes (default). |
| `<ignoreTests>` | `boolean` | If this is true, DepClean will not analyze the test classes in the project, and, therefore, the dependencies that are only used for testing will be considered unused. This parameter is useful to detect dependencies that have `compile` scope but are only used for testing. **Default value is:** `false`. |
| `<createPomDebloated>` | `boolean` | If this is true, DepClean creates a debloated version of the pom without unused dependencies called `debloated-pom.xml`, in the root of the project. **Default value is:** `false`. |
| `<createResultJson>` | `boolean` | If this is true, DepClean creates a JSON file of the dependency tree along with metadata of each dependency. The file is called `depclean-results.json`, and is located in the `target` directory of the project. **Default value is:** `false`. |
| `<createCallGraphCsv>` | `boolean` | If this is true, DepClean creates a CSV file with the static call graph of the API members used in the project. The file is called `depclean-callgraph.csv`, and is located in the `target` directory of the project. **Default value is:** `false`. |
| `<failIfUnusedDirect>` | `boolean` | If this is true, and DepClean reported any unused direct dependency in the dependency tree, the build fails immediately after running DepClean. **Default value is:** `false`. |
| `<failIfUnusedTransitive>` | `boolean` | If this is true, and DepClean reported any unused transitive dependency in the dependency tree, the build fails immediately after running DepClean. **Default value is:** `false`. |
| `<failIfUnusedInherited>` | `boolean` | If this is true, and DepClean reported any unused inherited dependency in the dependency tree, the build fails immediately after running DepClean. **Default value is:** `false`. |
| `<skipDepClean>` | `boolean` | Skip plugin execution completely. **Default value is:** `false`. |

For example, to fail the build in the presence of unused direct dependencies and ignore all the scopes except the
`compile` scope, use the following plugin configuration.
Expand Down
2 changes: 1 addition & 1 deletion checkstyle.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@

<module name="LineLength">
<property name="fileExtensions" value="java"/>
<property name="max" value="120"/>
<property name="max" value="160"/>
<property name="ignorePattern" value="^package.*|^import.*|a href|href|http://|https://|ftp://"/>
</module>

Expand Down
14 changes: 14 additions & 0 deletions depclean-core/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -72,10 +72,24 @@
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>11</source>
<target>11</target>
</configuration>
</plugin>
</plugins>
</build>

<dependencies>
<!-- Source code manipulation -->
<dependency>
<groupId>com.thoughtworks.qdox</groupId>
<artifactId>qdox</artifactId>
<version>2.0.1</version>
</dependency>
<!-- Bytecode manipulation -->
<dependency>
<groupId>org.ow2.asm</groupId>
Expand Down
56 changes: 39 additions & 17 deletions depclean-core/src/main/java/se/kth/depclean/core/DepCleanManager.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
import java.io.File;
import java.io.IOException;
import java.nio.charset.Charset;
import java.util.HashSet;
import java.util.Objects;
import java.util.Set;
import java.util.concurrent.TimeUnit;
Expand Down Expand Up @@ -41,7 +42,7 @@ public class DepCleanManager {
private final boolean failIfUnusedInherited;
private final boolean createPomDebloated;
private final boolean createResultJson;
private final boolean createClassUsageCsv;
private final boolean createCallGraphCsv;

/**
* Execute the depClean manager.
Expand All @@ -58,7 +59,7 @@ public void execute() throws AnalysisFailureException {
getLog().info("Starting DepClean dependency analysis");

if (dependencyManager.isMaven() && dependencyManager.isPackagingPom()) {
getLog().info("Skipping because packaging type pom.");
getLog().info("Skipping because packaging type is pom.");
return;
}

Expand Down Expand Up @@ -93,7 +94,7 @@ public void execute() throws AnalysisFailureException {
dependencyManager.getDebloater(analysis).write();
}

/* Writing the JSON file with the debloat results */
/* Writing the JSON file with the depclean results */
if (createResultJson) {
createResultJson(analysis);
}
Expand All @@ -106,7 +107,7 @@ private void createResultJson(ProjectDependencyAnalysis analysis) {
printString("Creating depclean-results.json, please wait...");
final File jsonFile = new File(dependencyManager.getBuildDirectory() + File.separator + "depclean-results.json");
final File treeFile = new File(dependencyManager.getBuildDirectory() + File.separator + "tree.txt");
final File classUsageFile = new File(dependencyManager.getBuildDirectory() + File.separator + "class-usage.csv");
final File csvFile = new File(dependencyManager.getBuildDirectory() + File.separator + "depclean-callgraph.csv");
try {
dependencyManager.generateDependencyTree(treeFile);
} catch (IOException | InterruptedException e) {
Expand All @@ -115,30 +116,31 @@ private void createResultJson(ProjectDependencyAnalysis analysis) {
Thread.currentThread().interrupt();
return;
}
if (createClassUsageCsv) {
printString("Creating class-usage.csv, please wait...");
if (createCallGraphCsv) {
printString("Creating " + csvFile.getName() + ", please wait...");
try {
FileUtils.write(classUsageFile, "OriginClass,TargetClass,Dependency\n", Charset.defaultCharset());
FileUtils.write(csvFile, "OriginClass,TargetClass,OriginDependency,TargetDependency\n", Charset.defaultCharset());
} catch (IOException e) {
getLog().error("Error writing the CSV header.");
}
}
String treeAsJson = dependencyManager.getTreeAsJson(treeFile,
String treeAsJson = dependencyManager.getTreeAsJson(
treeFile,
analysis,
classUsageFile,
createClassUsageCsv
csvFile,
createCallGraphCsv
);

try {
FileUtils.write(jsonFile, treeAsJson, Charset.defaultCharset());
} catch (IOException e) {
getLog().error("Unable to generate JSON file.");
getLog().error("Unable to generate " + jsonFile.getName() + " file.");
}
if (jsonFile.exists()) {
getLog().info("depclean-results.json file created in: " + jsonFile.getAbsolutePath());
getLog().info(jsonFile.getName() + " file created in: " + jsonFile.getAbsolutePath());
}
if (classUsageFile.exists()) {
getLog().info("class-usage.csv file created in: " + classUsageFile.getAbsolutePath());
if (csvFile.exists()) {
getLog().info(csvFile.getName() + " file created in: " + csvFile.getAbsolutePath());
}
}

Expand All @@ -162,21 +164,42 @@ private ProjectContext buildProjectContext() {
ignoreScopes.add("test");
}

// Consider are used all the classes declared in Maven processors
Set<ClassName> allUsedClasses = new HashSet<>();
Set<ClassName> usedClassesFromProcessors = dependencyManager
.collectUsedClassesFromProcessors().stream()
.map(ClassName::new)
.collect(Collectors.toSet());

// Consider as used all the classes located in the imports of the source code
Set<ClassName> usedClassesFromSource = dependencyManager.collectUsedClassesFromSource(
dependencyManager.getSourceDirectory(),
dependencyManager.getTestDirectory())
.stream()
.map(ClassName::new)
.collect(Collectors.toSet());

allUsedClasses.addAll(usedClassesFromProcessors);
allUsedClasses.addAll(usedClassesFromSource);

final DependencyGraph dependencyGraph = dependencyManager.dependencyGraph();
return new ProjectContext(
dependencyGraph,
dependencyManager.getOutputDirectory(),
dependencyManager.getTestOutputDirectory(),
dependencyManager.getSourceDirectory(),
dependencyManager.getTestDirectory(),
dependencyManager.getDependenciesDirectory(),
ignoreScopes.stream().map(Scope::new).collect(Collectors.toSet()),
toDependency(dependencyGraph.allDependencies(), ignoreDependencies),
dependencyManager.collectUsedClassesFromProcessors().stream().map(ClassName::new).collect(Collectors.toSet())
allUsedClasses
);
}

/**
* Returns a set of {@code DependencyCoordinate}s that match given string representations.
*
* @param allDependencies all known dependencies
* @param allDependencies all known dependencies
* @param ignoreDependencies string representation of dependencies to return
* @return a set of {@code Dependency} that match given string representations
*/
Expand All @@ -197,7 +220,6 @@ private Dependency findDependency(Set<Dependency> allDependencies, String depend
private String getTime(long millis) {
long minutes = TimeUnit.MILLISECONDS.toMinutes(millis);
long seconds = (TimeUnit.MILLISECONDS.toSeconds(millis) % 60);

return String.format("%smin %ss", minutes, seconds);
}

Expand Down
4 changes: 1 addition & 3 deletions depclean-core/src/main/java/se/kth/depclean/core/analysis/ActualUsedClasses.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,20 +12,18 @@
@Slf4j
public class ActualUsedClasses {

final Set<ClassName> classes = new HashSet<>();
private final ProjectContext context;
private final Set<ClassName> classes = new HashSet<>();

public ActualUsedClasses(ProjectContext context) {
this.context = context;
}

private void registerClass(ClassName className) {

// Do not register class unknown to dependencies
if (context.hasNoDependencyOnClass(className)) {
return;
}

log.trace("## Register class {}", className);
classes.add(className);
}
Expand Down
41 changes: 29 additions & 12 deletions depclean-core/src/main/java/se/kth/depclean/core/analysis/ClassFileVisitorUtils.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@
import java.net.URI;
import java.net.URISyntaxException;
import java.net.URL;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.jar.JarEntry;
import java.util.jar.JarInputStream;
import org.codehaus.plexus.util.DirectoryScanner;
Expand All @@ -50,10 +52,8 @@ private ClassFileVisitorUtils() {
*
* @param url The jar or directory
* @param visitor A {@link ClassFileVisitor}.
* @throws IOException In case of any I/O problems.
*/
public static void accept(URL url, ClassFileVisitor visitor)
throws IOException {
public static void accept(URL url, ClassFileVisitor visitor) {
if (url.getPath().endsWith(".jar")) {
acceptJar(url, visitor);
} else {
Expand All @@ -80,19 +80,19 @@ public static void accept(URL url, ClassFileVisitor visitor)
*
* @param url URL of jar
* @param visitor A {@link ClassFileVisitor}.
* @throws IOException In case of IO issues.
*/
private static void acceptJar(URL url, ClassFileVisitor visitor)
throws IOException {
private static void acceptJar(URL url, ClassFileVisitor visitor) {
try (JarInputStream in = new JarInputStream(url.openStream())) {
JarEntry entry = null;
JarEntry entry;
while ((entry = in.getNextJarEntry()) != null) { //NOSONAR
String name = entry.getName();
// ignore files like package-info.class and module-info.class
if (name.endsWith(CLASS) && name.indexOf('-') == -1) {
visitClass(name, in, visitor);
}
}
} catch (IOException e) {
e.printStackTrace();
}
}

Expand All @@ -101,10 +101,8 @@ private static void acceptJar(URL url, ClassFileVisitor visitor)
*
* @param directory Directory or File to be analyzed.
* @param visitor A {@link ClassFileVisitor}.
* @throws IOException In case of IO issues.
*/
private static void acceptDirectory(File directory, ClassFileVisitor visitor)
throws IOException {
private static void acceptDirectory(File directory, ClassFileVisitor visitor) {
if (!directory.isDirectory()) {
throw new IllegalArgumentException("File is not a directory");
}
Expand All @@ -118,21 +116,40 @@ private static void acceptDirectory(File directory, ClassFileVisitor visitor)
File file = new File(directory, path);
try (FileInputStream in = new FileInputStream(file)) {
visitClass(path, in, visitor);
} catch (IOException e) {
e.printStackTrace();
}
}
}

/**
* Removes the root folder of the dependency from the path.
*
* @param path the dependency folder
* @return path without the dependency folder
*/
public static String getChild(String path) {
Path tmp = Paths.get(path);
if (tmp.getNameCount() > 1) {
return tmp.subpath(1, tmp.getNameCount()).toString();
} else {
// impossible to extract child's path
return path;
}
}

/**
* Visits the classes.
*
* @param path Path of the class.
* @param in read the input bytes.
* @param path Path of the class.
* @param in read the input bytes.
* @param visitor A {@link ClassFileVisitor}.
*/
private static void visitClass(String path, InputStream in, ClassFileVisitor visitor) {
if (!path.endsWith(CLASS)) {
throw new IllegalArgumentException("Path is not a class");
}
path = getChild(path);
String className = path.substring(0, path.length() - CLASS.length());
className = className.replace('/', '.');
visitor.visitClass(className, in);
Expand Down
8 changes: 2 additions & 6 deletions depclean-core/src/main/java/se/kth/depclean/core/analysis/DefaultClassAnalyzer.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -41,12 +41,8 @@ public Set<String> analyze(URL url) throws IOException {
CollectorClassFileVisitor visitor = new CollectorClassFileVisitor();
try {
ClassFileVisitorUtils.accept(url, visitor);
} catch (ZipException e) {
// since the current ZipException gives no indication what jar file is corrupted
// we prefer to wrap another ZipException for better error visibility
ZipException ze = new ZipException("Cannot process Jar entry on URL: " + url + " due to " + e.getMessage());
ze.initCause(e);
throw ze;
} catch (Exception e) {
e.printStackTrace();
}
return visitor.getClasses();
}
Expand Down
Loading

0 comments on commit 5c13f9d

Please sign in to comment.