[parsec-users] Correct Benchmark Outputs

Christian Bienia cbienia at CS.Princeton.EDU
Thu Apr 10 22:02:37 EDT 2008


Hi James,

We don't offer reference outputs for PARSEC at the moment, however it's 
planned for the future. One particular difficulty is that many outputs are 
not architecture independent, so a simple comparison would not work in most 
cases. For example, programs with a lot of floating point computations can 
produce slightly different results on different machines.

In the case case of swaptions you have to distinguish between logically 
correct and factually correct. Swaptions employs Monte Carlo simulation which 
needs a high number of simulation runs in order to produce an accurate 
result. However, we had to reduce the number of simulation runs in order to 
create the smaller inputs. Only the 'native' input set generates an accurate 
result, the other ones are increasingly inaccurate as their size decreases 
but logically correct in the sense that they produce the correct result for 
the given input values. Especially 'test' and 'simdev' are so small that they 
wouldn't make any sense in a realistic setting. The only properties which we 
guarantee for these input sets are that the correct binary gets executed in 
the case of 'test' and the same execution path as in the larger input sets is 
taken in the case of 'simdev'. So don't be surprised if they produce results 
which look strange.

Furthermore, swaptions uses a synthetic input which is obtained by replicating 
a real swaption. You therefore get the same result for each swaption. The 
output which you described in your email seems to be correct, I get similar 
values on my machine.

- Chris



On Wednesday 09 April 2008 04:53 pm, James Poe wrote:
> I was wondering if there was an example of proper execution output for each
> of the benchmarks anywhere.  So far I haven't had any luck finding this.
> Specifically I am concerned about the output I receive for "swaptions".
> With the 'test' and 'simdev' input sets I get all 0's for the swaption
> prices and standard error, and with the simsmall input my output is:
>
> PARSEC Benchmark Suite Version 1.0
> Number of Simulations: 5000,  Number of threads: 4 Number of swaptions: 16
> Swaption0: [SwaptionPrice: 0.0000234735 StdError: 0.0000083599]
> Swaption1: [SwaptionPrice: 0.0000234735 StdError: 0.0000083599]
> Swaption2: [SwaptionPrice: 0.0000234735 StdError: 0.0000083599]
> Swaption3: [SwaptionPrice: 0.0000234735 StdError: 0.0000083599]
> Swaption4: [SwaptionPrice: 0.0000234735 StdError: 0.0000083599]
> Swaption5: [SwaptionPrice: 0.0000234735 StdError: 0.0000083599]
> Swaption6: [SwaptionPrice: 0.0000234735 StdError: 0.0000083599]
> Swaption7: [SwaptionPrice: 0.0000234735 StdError: 0.0000083599]
> Swaption8: [SwaptionPrice: 0.0000234735 StdError: 0.0000083599]
> Swaption9: [SwaptionPrice: 0.0000234735 StdError: 0.0000083599]
> Swaption10: [SwaptionPrice: 0.0000234735 StdError: 0.0000083599]
> Swaption11: [SwaptionPrice: 0.0000234735 StdError: 0.0000083599]
> Swaption12: [SwaptionPrice: 0.0000234735 StdError: 0.0000083599]
> Swaption13: [SwaptionPrice: 0.0000234735 StdError: 0.0000083599]
> Swaption14: [SwaptionPrice: 0.0000234735 StdError: 0.0000083599]
> Swaption15: [SwaptionPrice: 0.0000234735 StdError: 0.0000083599]
>
> Is this an acceptable output?  Does it matter that the values are both
> extremely small and uniform across all of the Swaptions?  Is there
> somewhere in the documentation a location for correct outputs that I have
> missed?
>
> Thank you in advance for your response,
>
> James



More information about the parsec-users mailing list