Kewen Zhao commited on
Commit
f5dea60
·
1 Parent(s): 0c0f1e7

update readme and description

Browse files
Files changed (2) hide show
  1. README.md +1 -1
  2. code_eval_stdio.py +5 -4
README.md CHANGED
@@ -16,7 +16,7 @@ description: >-
16
  (https://arxiv.org/abs/2107.03374).
17
  ---
18
 
19
- # Metric Card for Code Eval
20
 
21
  ## Metric description
22
 
 
16
  (https://arxiv.org/abs/2107.03374).
17
  ---
18
 
19
+ # Metric Card for Code Eval StdIO
20
 
21
  ## Metric description
22
 
code_eval_stdio.py CHANGED
@@ -76,10 +76,11 @@ Returns:
76
  pass_at_k: dict with pass rates for each k
77
  results: dict with granular results of each unittest
78
  Examples:
79
- >>> code_eval = evaluate.load("code_eval")
80
- >>> test_cases = ["assert add(2,3)==5"]
81
- >>> candidates = [["def add(a,b): return a*b", "def add(a, b): return a+b"]]
82
- >>> pass_at_k, results = code_eval.compute(references=test_cases, predictions=candidates, k=[1, 2])
 
83
  >>> print(pass_at_k)
84
  {'pass@1': 0.5, 'pass@2': 1.0}
85
  """
 
76
  pass_at_k: dict with pass rates for each k
77
  results: dict with granular results of each unittest
78
  Examples:
79
+ >>> code_eval_stdio = evaluate.load("hage2000/code_eval_stdio")
80
+ >>> inputs = ["2 3"]
81
+ >>> references = ["5"]
82
+ >>> candidates = [[ "nums = list(map(int, input().split()))\nprint(sum(nums))"]]
83
+ >>> pass_at_k, results = code_eval_stdio.compute(references=references, predictions=candidates, inputs = inputs, k=[1, 2])
84
  >>> print(pass_at_k)
85
  {'pass@1': 0.5, 'pass@2': 1.0}
86
  """