Input Structure Based Optimization for Privacy Preserving AI Systems