Over past years, the philosophy for designing the artificial intelligence algorithms has significantly shifted towards automatically extracting the composable systems from massive data volumes. This paradigm shift has been expedited by the big data booming which enables us to easily access and analyze the highly large data sets. The most well-known class of big data analysis techniques is called \textit{deep learning}. These models require significant computation power and extremely high memory accesses which necessitate the design of novel approaches to reduce the memory access and improve power efficiency while taking into account the development of domain-specific hardware accelerators to support the current and future data sizes and model structures. The current trends for designing application-specific integrated circuits \textit{barely consider the essential requirement for maintaining the complex neural network computation to be resilient in the presence of soft errors}. The soft errors might strike either memory storage or combinational logic in the hardware accelerator that can affect the architectural behavior such that the precision of the results fall behind the minimum allowable correctness. In this study, we demonstrate that the impact of soft errors on a customized deep learning algorithm called Binarized Neural Network might cause drastic image misclassification. Our experimental results shows that the accuracy of image classifier can drastically drop by 75.48\% and 15.5\% in lfcW1A1 and cnvW1A1 networks, respectively across CIFAR-10 and MNIST datasets during the fault injection for the worst-case scenarios.