Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix readme paths #180

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Fix readme paths #180

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
aeeb0dd
Fixed some paths to CodeBLEU
mavaa Feb 4, 2024
9419e42
Removed full URL from paths to work better in forks
mavaa Feb 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Code-Code/CodeCompletion-line/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ def __init__ ( self ) :

## Pipeline

We provide a pipeline that evaluate line completion on [CodeGPT](https://github.com/microsoft/CodeXGLUE/tree/main/Code-Code/CodeCompletion-token#codegpt) model. You could directly use the model trained on token level code completion to test on line-level completion.
We provide a pipeline that evaluate line completion on [CodeGPT](/Code-Code/CodeCompletion-token#codegpt) model. You could directly use the model trained on token level code completion to test on line-level completion.

### Dependency

Expand Down
8 changes: 4 additions & 4 deletions Code-Code/code-refinement/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

Code refinement aims to automatically fix bugs in the code, which can contribute to reducing the cost of bug-fixes for developers.
In CodeXGLUE, given a piece of Java code with bugs, the task is to remove the bugs to output the refined code.
Models are evaluated by BLEU scores, accuracy (exactly match) and [CodeBLEU](https://github.com/microsoft/CodeXGLUE/blob/main/code-to-code-trans/CodeBLEU.MD).
Models are evaluated by BLEU scores, accuracy (exactly match) and [CodeBLEU](/Code-Code/code-to-code-trans/CodeBLEU.MD).

## Dataset

Expand Down Expand Up @@ -119,16 +119,16 @@ The results on the test set are shown as below:

Small:

| Method | BLEU | Acc (100%) | [CodeBLEU](https://github.com/microsoft/CodeXGLUE/blob/main/Code-Code/code-to-code-trans/CodeBLEU.MD) |
| Method | BLEU | Acc (100%) | [CodeBLEU](/Code-Code/code-to-code-trans/CodeBLEU.MD) |
| ---------- | :-------: | :-------: | :-------: |
| Naive copy | 78.06 | 0.0 | - |
| LSTM | 76.76 | 10.0 | - |
| Transformer| 77.21 | 14.7 | 73.31 |
| Transformer| 77.21 | 14.7 | 73.31 |
| CodeBERT | **77.42** | **16.4** | **75.58** |

Medium:

| Method | BLEU | Acc (100%) | [CodeBLEU](https://github.com/microsoft/CodeXGLUE/blob/main/Code-Code/code-to-code-trans/CodeBLEU.MD) |
| Method | BLEU | Acc (100%) | [CodeBLEU](/Code-Code/code-to-code-trans/CodeBLEU.MD) |
| ---------- | :-------: | :-------: | :-------: |
| Naive copy | 90.91 | 0.0 | - |
| LSTM | 72.08 | 2.5 | - |
Expand Down
4 changes: 2 additions & 2 deletions Code-Code/code-to-code-trans/CodeBLEU.MD
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,13 +28,13 @@ BLEU, but can also consider the grammatical correctness and the logic correctnes

An ideal evaluation metric should consider the grammatical correctness and the logic correctness.
We propose weighted n-gram match and syntactic AST match to measure grammatical correctness, and introduce semantic data-flow match to calculate logic correctness.
![CodeBLEU](https://github.com/microsoft/CodeXGLUE/blob/main/code-to-code-trans/CodeBLEU.jpg)
![CodeBLEU](CodeBLEU.jpg)

### Examples

Here we will give two toy examples and show the qualitative advantages of CodeBLEU compared with the traditional BLEU score.

![Example](https://github.com/microsoft/CodeXGLUE/blob/main/code-to-code-trans/example.png)
![Example](example.png)



Expand Down
6 changes: 3 additions & 3 deletions Code-Code/code-to-code-trans/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

Code translation aims to migrate legacy software from one programming language in a platform toanother.
In CodeXGLUE, given a piece of Java (C#) code, the task is to translate the code into C# (Java) version.
Models are evaluated by BLEU scores, accuracy (exactly match), and [CodeBLEU](https://github.com/microsoft/CodeXGLUE/blob/main/code-to-code-trans/CodeBLEU.MD) scores.
Models are evaluated by BLEU scores, accuracy (exactly match), and [CodeBLEU](/Code-Code/code-to-code-trans/CodeBLEU.MD) scores.

## Dataset

Expand Down Expand Up @@ -119,7 +119,7 @@ The results on the test set are shown as below:

Java to C#:

| Method | BLEU | Acc (100%) | [CodeBLEU](https://github.com/microsoft/CodeXGLUE/blob/main/Code-Code/code-to-code-trans/CodeBLEU.MD) |
| Method | BLEU | Acc (100%) | [CodeBLEU](/Code-Code/code-to-code-trans/CodeBLEU.MD) |
| ---------- | :--------: | :-------: | :-------: |
| Naive copy | 18.54 | 0.0 | - |
| PBSMT | 43.53 | 12.5 | 42.71 |
Expand All @@ -129,7 +129,7 @@ Java to C#:

C# to Java:

| Method | BLEU | Acc (100%) | [CodeBLEU](https://github.com/microsoft/CodeXGLUE/blob/main/Code-Code/code-to-code-trans/CodeBLEU.MD) |
| Method | BLEU | Acc (100%) | [CodeBLEU](/Code-Code/code-to-code-trans/CodeBLEU.MD) |
| ---------- | :--------: | :-------: | :-------: |
| Naive copy | 18.69 | 0.0 | - |
| PBSMT | 40.06 | 16.1 | 43.48 |
Expand Down
8 changes: 4 additions & 4 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,10 @@ Recent years have seen a surge of applying of statistical models, including neur

To address this, researchers from Microsoft Research Asia, Developer Division, and Bing introduce CodeXGLUE, a benchmark dataset and open challenge for code intelligence. It includes a collection of code intelligence tasks and a platform for model evaluation and comparison. CodeXGLUE stands for General Language Understanding Evaluation benchmark for CODE. It includes 14 datasets for 10 diversified code intelligence tasks covering the following scenarios:

* **[code-code](https://github.com/microsoft/CodeXGLUE/tree/main/Code-Code)** (clone detection, defect detection, cloze test, code completion, code repair, and code-to-code translation)
* **[text-code](https://github.com/microsoft/CodeXGLUE/tree/main/Text-Code)** (natural language code search, text-to-code generation)
* **[code-text](https://github.com/microsoft/CodeXGLUE/tree/main/Code-Text/)** (code summarization)
* **[text-text](https://github.com/microsoft/CodeXGLUE/tree/main/Text-Text)** (documentation translation)
* **[code-code](/Code-Code)** (clone detection, defect detection, cloze test, code completion, code repair, and code-to-code translation)
* **[text-code](/Text-Code)** (natural language code search, text-to-code generation)
* **[code-text](/Code-Text/)** (code summarization)
* **[text-text](/Text-Text)** (documentation translation)

A brief summary of CodeXGLUE is given below, including tasks, datasets, language, sizes in various states, baseline systems, providers, and short definitions of each task. Datasets highlighted in BLUE are newly introduced.
![A brief summary of CodeXGLUE, including tasks, datasets, baseline systems, etc.](tasks.jpg)
Expand Down
4 changes: 2 additions & 2 deletions Text-Code/text-to-code/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ The outputs are:
BLEU: 16.68, EM: 17.0
```

The CodeBLEU score can be calculated by this [script](https://github.com/microsoft/CodeXGLUE/tree/main/Code-Code/code-to-code-trans/evaluator/CodeBLEU)
The CodeBLEU score can be calculated by this [script](/Code-Code/code-to-code-trans/evaluator/CodeBLEU)

### Input Format

Expand All @@ -69,7 +69,7 @@ public void inc ( ) { this . add ( 1 ) ; }

## Pipeline

We provide a pipeline for this task with [CodeGPT and CodeGPT-adapted](https://github.com/microsoft/CodeXGLUE/tree/main/Code-Code/CodeCompletion-token#codegpt) model.
We provide a pipeline for this task with [CodeGPT and CodeGPT-adapted](/Code-Code/CodeCompletion-token#codegpt) model.

## Dependency

Expand Down