Nowadays, Internet traffic encryption is rapidly increasing due to privacy and security concerns. This is because of the massive usage of end-to-end security protocols over Internet such as HTTPS and QUIC. The encryption trend will continue to rapidly increase in the future, and this trend concerns video streaming applications as well. Network providers face a serious challenge in managing their networks due to such widespread deployment of end-to-end security protocols. These operators need to have a clear visibility into traffic on their networks to monitor and manage both quality of experience (QoE)-and-service (QoS) impairments in popular video streaming services, in the most effective and efficient manner. Moreover, so many factors that influence QoE need to be taken care of to get an acceptable user experience. Most of the existing solutions use the deep packet inspection to infer these factors from the encrypted traffic. However, these solutions are inefficient, most of the time, leading to low QoE inference accuracy. To bridge this gap, we propose a machine-learning based solution that leverages a random forest classifier for a better QoE inference accuracy. The proposed solution uses network-and-transport layer information to infer QoE factors such as startup delay and stall events. It helps the network providers to react quickly and in real time for any impairments in the QoE of the encrypted video traffic. We evaluate our solution using an HTTP adaptive streaming service (YouTube) that uses HTTPS and QUIC protocols. Our experimental results show that our solution achieves up to 91.1% classification accuracy for HTTPS and up to 87.3% for QUIC.