Spaces:
Sleeping
Sleeping
Kewen Zhao
commited on
Commit
·
f5dea60
1
Parent(s):
0c0f1e7
update readme and description
Browse files- README.md +1 -1
- code_eval_stdio.py +5 -4
README.md
CHANGED
@@ -16,7 +16,7 @@ description: >-
|
|
16 |
(https://arxiv.org/abs/2107.03374).
|
17 |
---
|
18 |
|
19 |
-
# Metric Card for Code Eval
|
20 |
|
21 |
## Metric description
|
22 |
|
|
|
16 |
(https://arxiv.org/abs/2107.03374).
|
17 |
---
|
18 |
|
19 |
+
# Metric Card for Code Eval StdIO
|
20 |
|
21 |
## Metric description
|
22 |
|
code_eval_stdio.py
CHANGED
@@ -76,10 +76,11 @@ Returns:
|
|
76 |
pass_at_k: dict with pass rates for each k
|
77 |
results: dict with granular results of each unittest
|
78 |
Examples:
|
79 |
-
>>>
|
80 |
-
>>>
|
81 |
-
>>>
|
82 |
-
>>>
|
|
|
83 |
>>> print(pass_at_k)
|
84 |
{'pass@1': 0.5, 'pass@2': 1.0}
|
85 |
"""
|
|
|
76 |
pass_at_k: dict with pass rates for each k
|
77 |
results: dict with granular results of each unittest
|
78 |
Examples:
|
79 |
+
>>> code_eval_stdio = evaluate.load("hage2000/code_eval_stdio")
|
80 |
+
>>> inputs = ["2 3"]
|
81 |
+
>>> references = ["5"]
|
82 |
+
>>> candidates = [[ "nums = list(map(int, input().split()))\nprint(sum(nums))"]]
|
83 |
+
>>> pass_at_k, results = code_eval_stdio.compute(references=references, predictions=candidates, inputs = inputs, k=[1, 2])
|
84 |
>>> print(pass_at_k)
|
85 |
{'pass@1': 0.5, 'pass@2': 1.0}
|
86 |
"""
|