The incorporation of artificial intelligence into the rapidly growing IoT devices demands a high level of built-in intelligence, e.g. machine learning capability at the device level. Affective computing offers a new degree of cognitive intelligence into edge processing IoT devices by inferring human emotion, stress levels for intelligent human assistance. This work explores the design space and runtime optimization opportunity for affective computing at the system-on-chip (SoC) level. A design optimization methodology for the neural network classifier and runtime power management schemes are proposed to achieve high energy efficiency on embedded low power devices. A test chip based on a 65nm CMOS process was used to demonstrate the proposed methodology on emotion and stress classification for affective computing. An average power saving of 45% is achieved with a peak power savings of 60% from the proposed emotion-driven adaptive power management scheme.