Reflections on Trusting TrustHUB

Hardware security verification is a critical task in evaluating algorithms and tools for effectiveness and efficiency, both from an organizational and a technical view. In the past decade, the TrustHUB benchmark suite evolved as the de-facto standard benchmark suite in order to perform experiments that allow quantitative analysis of countermeasures, but also to provide statistical data on effectiveness, and performance indicators to assess practical applicability. In this work, we study the effectiveness of the TrustHUB benchmark suite, and its ability to provide meaningful and reasonable experimental data on algorithms that target security properties of digital designs. We systematically elaborate fundamental properties of effective hardware Trojan benchmarks: Correctness, Maliciousness, Stealthiness, and Persistence. We utilize these properties to classify a reasonable subset of the TrustHUB benchmark suite by means of effectiveness. The results of our study are fairly surprising: Only three out of 83 studied benchmark designs can be considered actual hardware Trojans, all others entail undocumented limitations that may impact the results of experiments that evaluate countermeasures. We introduce a comprehensive, yet simple yes/no effectiveness classification framework, with which we identify potential peculiarities that may render most of these benchmarks ineffective. We identify spots of improvement, and provide recommendations for effective benchmark design.