Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gson: initial integration #6742

Merged
merged 5 commits into from
Nov 6, 2021
Merged

gson: initial integration #6742

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
b8c3229
gson: initial integration
DavidKorczynski Nov 2, 2021
ceabe31
gson: fix project yaml
DavidKorczynski Nov 2, 2021
3c2a8d0
gson: update maven
DavidKorczynski Nov 5, 2021
6aa34b7
gson: include reader fuzzer
DavidKorczynski Nov 6, 2021
dcbdce4
gson: finalize integraiton
DavidKorczynski Nov 6, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions projects/gson/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# Copyright 2021 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
################################################################################

FROM gcr.io/oss-fuzz-base/base-builder-jvm
RUN apt-get update && apt-get install -y make autoconf automake libtool wget

RUN curl -L https://downloads.apache.org/maven/maven-3/3.8.3/binaries/apache-maven-3.8.3-bin.zip -o maven.zip && \
unzip maven.zip -d $SRC/maven && \
rm -rf maven.zip
ENV MVN $SRC/maven/apache-maven-3.8.3/bin/mvn

RUN git clone --depth 1 https://github.com/google/gson gson
WORKDIR gson
COPY build.sh $SRC/
COPY pom.xml $SRC/gson/pom.xml
COPY gson/pom.xml $SRC/gson/gson/pom.xml
Comment on lines +28 to +29
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should probably have asked this earlier, but why are you providing custom POM files?
It appears the only difference is that the copy-rename-maven-plugin and proguard-maven-plugin have been removed. Was your intention to make building Gson more efficient because you are skipping the tests anyways?
(maybe you could then also run Maven only inside the gson/gson folder to avoid building the extras Maven module)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually I had a similar question. When I tried building with local sources it didn't work until I hacked at the pom.xml files to adjust things like the source and target levels.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In short, I did this to make it build without issues.

The poms I added have the following changes from the original: change target/source levels and avoid the rename+proguard plugins.

I did this because I first ran into source number issues, which suggested I increase from 1.6 to 1.6. Then, I ran into the following issue when compiling the tests: /src/gson/gson/src/test/java/com/google/gson/internal/LinkedTreeMapTest.java:[164,20] Invalid java.lang.SafeVarargs annotation. Instance method <T>assertIterationOrd er(java.lang.Iterable<T>,T...) is not final.

and I ran into this issue when I skip tests:

[ERROR] Failed to execute goal com.coderplus.maven.plugins:copy-rename-maven-plugin:1.0.1:rename (pre-obfuscate-class) on project gson: sourceFile /src/gson/gson/tar
get/test-classes/com/google/gson/functional/EnumWithObfuscatedTest.class does not exist

As such, I moved forward from there until the errors didn't persist. Let me know if this complicates things in terms of adding false positives or similar.

Notice my knowledge of Maven is very limited.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

change target/source levels

Makes sense; your docker image is apparently using JDK 15, but Gson is compiled against Java 1.6 for which compilation support was dropped in JDK 12.
But hopefully future Gson versions will be compiled against at least Java 1.7 (see google/gson#2018) which is still supported by JDK 17.

avoid the rename+proguard plugins

The problem seems to be that you are building Gson with -Dmaven.test.skip=true, so even compilation of tests is disabled (see documentation). Therefore the copy-rename-maven-plugin which is needed for the tests fails. Unfortunately that plugin has no skip parameter or similar. I assume it could be solved for Gson by putting that plugin execution into a Maven profile which is enabled by default (and disabled when maven.test.skip=true), but not sure if that is worth it.

I am not familiar with how oss-fuzz or Jazzer works, but would it be an issue if the Gson tests were compiled (but not executed)? Then maybe you can remove these copies of the Gson POMs and run:

// Move into main Gson Maven module
cd gson
mvn clean package -DskipTests "-Dmaven.compiler.release=8"

COPY *.java $SRC/
27 changes: 27 additions & 0 deletions projects/gson/FuzzParse.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
// Copyright 2021 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//
////////////////////////////////////////////////////////////////////////////////
import com.code_intelligence.jazzer.api.FuzzedDataProvider;

import java.io.*;
import com.google.gson.*;

public class FuzzParse {
public static void fuzzerTestOneInput(FuzzedDataProvider data) {
try {
JsonParser.parseString(data.consumeRemainingAsString());
} catch (JsonSyntaxException expected) { }
DavidKorczynski marked this conversation as resolved.
Show resolved Hide resolved
}
}
35 changes: 35 additions & 0 deletions projects/gson/FuzzReader.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
// Copyright 2021 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//
////////////////////////////////////////////////////////////////////////////////
import com.code_intelligence.jazzer.api.FuzzedDataProvider;

import java.io.*;
import com.google.gson.*;
import com.google.gson.stream.JsonReader;
import com.google.gson.stream.JsonToken;

public class FuzzReader {
public static void fuzzerTestOneInput(FuzzedDataProvider data) {
TypeAdapter<JsonElement> adapter = new Gson().getAdapter(JsonElement.class);
boolean lenient = data.consumeBoolean();
JsonReader reader = new JsonReader(new StringReader(data.consumeRemainingAsString()));
reader.setLenient(lenient);
try {
while (reader.peek() != JsonToken.END_DOCUMENT) {
adapter.read(reader);
}
} catch (JsonSyntaxException | IOException expected) { }
}
}
42 changes: 42 additions & 0 deletions projects/gson/build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
#!/bin/bash -eu
# Copyright 2021 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
################################################################################

MAVEN_ARGS="-Dmaven.test.skip=true -Djavac.src.version=11 -Djavac.target.version=11 -X"
$MVN --batch-mode --update-snapshots verify ${MAVEN_ARGS}
find ./gson -name "gson-*.jar" -exec mv {} $OUT/gson.jar \;

ALL_JARS="gson.jar"
BUILD_CLASSPATH=$(echo $ALL_JARS | xargs printf -- "$OUT/%s:"):$JAZZER_API_PATH
RUNTIME_CLASSPATH=$(echo $ALL_JARS | xargs printf -- "\$this_dir/%s:"):.:\$this_dir

for fuzzer in $(find $SRC -name 'Fuzz*.java'); do
fuzzer_basename=$(basename -s .java $fuzzer)
javac -cp $BUILD_CLASSPATH $fuzzer
cp $SRC/$fuzzer_basename.class $OUT/

# Create an execution wrapper that executes Jazzer with the correct arguments.
echo "#!/bin/sh
# LLVMFuzzerTestOneInput for fuzzer detection.
this_dir=\$(dirname \"\$0\")
LD_LIBRARY_PATH=\"$JVM_LD_LIBRARY_PATH\":\$this_dir \
\$this_dir/jazzer_driver --agent_path=\$this_dir/jazzer_agent_deploy.jar \
--cp=$RUNTIME_CLASSPATH \
--target_class=$fuzzer_basename \
--jvm_args=\"-Xmx2048m\" \
\$@" > $OUT/$fuzzer_basename
chmod u+x $OUT/$fuzzer_basename
done
110 changes: 110 additions & 0 deletions projects/gson/gson/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>

<parent>
<groupId>com.google.code.gson</groupId>
<artifactId>gson-parent</artifactId>
<version>2.9.0-SNAPSHOT</version>
</parent>

<artifactId>gson</artifactId>
<name>Gson</name>

<properties>
<proguardVersion>7.1.1</proguardVersion>
</properties>

<licenses>
<license>
<name>Apache-2.0</name>
<url>https://www.apache.org/licenses/LICENSE-2.0.txt</url>
</license>
</licenses>

<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<scope>test</scope>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<configuration>
<includePackageNames>com.google.gson</includePackageNames>
<excludePackageNames>com.google.gson.internal:com.google.gson.internal.bind</excludePackageNames>
<links>
<link>https://docs.oracle.com/javase/6/docs/api/</link>
</links>
</configuration>
</plugin>
<plugin>
<groupId>biz.aQute.bnd</groupId>
<artifactId>bnd-maven-plugin</artifactId>
<version>6.0.0</version>
<executions>
<execution>
<goals>
<goal>bnd-process</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<archive>
<manifestFile>${project.build.outputDirectory}/META-INF/MANIFEST.MF</manifestFile>
</archive>
</configuration>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>templating-maven-plugin</artifactId>
<version>1.0.0</version>
<executions>
<execution>
<id>filtering-java-templates</id>
<goals>
<goal>filter-sources</goal>
</goals>
<configuration>
<sourceDirectory>${basedir}/src/main/java-templates</sourceDirectory>
<outputDirectory>${project.build.directory}/generated-sources/java-templates</outputDirectory>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-resources-plugin</artifactId>
<version>3.2.0</version>
<executions>
<execution>
<id>post-obfuscate-class</id>
<phase>process-test-classes</phase>
<goals>
<goal>copy-resources</goal>
</goals>
<configuration>
<outputDirectory>${project.build.directory}/test-classes/com/google/gson/functional</outputDirectory>
<resources>
<resource>
<directory>${project.build.directory}/test-classes-obfuscated-outjar/com/google/gson/functional</directory>
<includes>
<include>EnumWithObfuscatedTest.class</include>
<include>EnumWithObfuscatedTest$Gender.class</include>
</includes>
</resource>
</resources>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
Loading