Sustainable Energy and Artificial Intelligence

Unsupervised Video Summarization Using GAN and BiLSTM-based Self-Attention Network

Document Type : Original Article

Authors

Alireza Gilaki 1
Roozbeh Rajabi 2

1 Faculty of Electrical and Computer Engineering, Qom University of Technology, Qom, Iran

2 University of Genoa

https://doi.org/10.61882/seai.2411-1021
Abstract
This paper presents an approach for automated unsupervised video summarization, that means, nothing more than video is needed to train the model. The goal is to extract a sequence of frames from an input video and assign each frame a score between 0 and 1. By doing so, we can select a subset of the most informative and diverse shots to make a summarized video. We build upon the foundation of SUM-GAN, particularly SUM-GAN-SLA, which utilize Generative Adversarial Networks to compare and distinguish between the original video and its regenerated counterpart. A key contribution of our work lies in the novel biLSTM-based self-attention network that we introduce to handle the crucial scoring layer of our model. We adjusted several aspects of the model, particularly in the loss functions and learning steps, to enhance the training process and achieve superior performance compared to state-of-the-art unsupervised and even supervised methods. To ensure a fair comparison, we evaluate our proposed model using two widely used datasets: SumMe and TVSum. The experimental results highlight the effectiveness of our proposed approach in automated unsupervised video summarization, achieving a 1.2% improvement over the best-performing methods' average F-score on SumMe and TVSum datasets. Additionally, our method ranks second among state-of-the-art unsupervised methods on each dataset. Notably, the top-performing methods exhibited inconsistent results across datasets, underscoring the broader applicability of our approach to diverse types of videos. Furthermore, our method demonstrates competitive performance compared to supervised approaches, with the best supervised method surpassing our results by only 0.75%.

Keywords

Video summarization
generative adversarial networks
attention network
deep learning
BiLSTM
Subjects

  1. Tonge, A., & Thepade, S. D. (2022). Creating Video Visual Storyboard with Static Video Summarization using Fractional Energy of Orthogonal Transforms. International Journal of Advanced Computer Science and Applications, 13(9).

  2. Chen, S. N. (2017). Storyboard-based accurate automatic summary video editing system. Multimedia Tools and Applications, 76(18), 18409-18423.

  3. Mahasseni, B., Lam, M., & Todorovic, S. (2017). Unsupervised video summarization with adversarial lstm networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (pp. 202-211).

  4. Apostolidis, E., Metsai, A. I., Adamantidou, E., Mezaris, V., & Patras, I. (2019, October). A stepwise, label-based approach for improving the adversarial training in unsupervised video summarization. In Proceedings of the 1st International Workshop on AI for Smart TV Content Production, Access and Delivery (pp. 17-25).

  5. Apostolidis, E., Adamantidou, E., Metsai, A. I., Mezaris, V., & Patras, I. (2019, December). Unsupervised video summarization via attention-driven adversarial learning. In International Conference on multimedia modeling (pp. 492-504). Cham: Springer International Publishing.

  6. Apostolidis, E., Adamantidou, E., Metsai, A. I., Mezaris, V., & Patras, I. (2020). AC-SUM-GAN: Connecting actor-critic and generative adversarial networks for unsupervised video summarization. IEEE Transactions on Circuits and Systems for Video Technology, 31(8), 3278-3292.

  7. Yin, Y., Thapliya, R., & Zimmermann, R. (2016). Encoded semantic tree for automatic user profiling applied to personalized video summarization. IEEE Transactions on Circuits and Systems for Video Technology, 28(1), 181-192.

  8. Paul, M., & Salehin, M. M. (2018). Spatial and motion saliency prediction method using eye tracker data for video summarization. IEEE Transactions on Circuits and Systems for Video Technology, 29(6), 1856-1867.

  9. Huang, C., & Wang, H. (2019). A novel key-frames selection framework for comprehensive video summarization. IEEE Transactions on Circuits and Systems for Video Technology, 30(2), 577-589.

  10. Guan, G., Wang, Z., Lu, S., Da Deng, J., & Feng, D. D. (2012). Keypoint-based keyframe selection. IEEE Transactions on circuits and systems for video technology, 23(4), 729-734.

  11. Faghihi, A., Fathollahi, M., & Rajabi, R. (2024). Diagnosis of skin cancer using VGG16 and VGG19 based transfer learning models. Multimedia Tools and Applications, 83(19), 57495-57510.

  12. Shahidi Zandi, M., & Rajabi, R. (2022). Deep learning based framework for Iranian license plate detection and recognition. Multimedia Tools and Applications, 81(11), 15841-15858.

  13. Hong, R., Tang, J., Tan, H. K., Yan, S., Ngo, C., & Chua, T. S. (2009, October). Event driven summarization for web videos. In Proceedings of the first SIGMM workshop on Social media (pp. 43-48).

  14. Ngo, C. W., Ma, Y. F., & Zhang, H. J. (2005). Video summarization and scene detection by graph modeling. IEEE Transactions on circuits and systems for video technology, 15(2), 296-305.

  15. Ma, Y. F., Lu, L., Zhang, H. J., & Li, M. (2002, December). A user attention model for video summarization. In Proceedings of the tenth ACM international conference on Multimedia (pp. 533-542).

  16. Gong, B., Chao, W. L., Grauman, K., & Sha, F. (2014). Diverse sequential subset selection for supervised video summarization. Advances in neural information processing systems, 27.

  17. Lee, Y. J., Ghosh, J., & Grauman, K. (2012, June). Discovering important people and objects for egocentric video summarization. In 2012 IEEE conference on computer vision and pattern recognition (pp. 1346-1353). IEEE.

  18. Liu, D., Hua, G., & Chen, T. (2010). A hierarchical visual model for video object summarization. IEEE transactions on pattern analysis and machine intelligence, 32(12), 2178-2190.

  19. Zhang, Y., Liang, X., Zhang, D., Tan, M., & Xing, E. P. (2020). Unsupervised object-level video summarization with online motion auto-encoder. Pattern Recognition Letters, 130, 376-385.

  20. Zhang, K., Chao, W. L., Sha, F., & Grauman, K. (2016, September). Video summarization with long short-term memory. In European conference on computer vision (pp. 766-782). Cham: Springer International Publishing.

  21. Zhao, B., Li, X., & Lu, X. (2018). Hsa-rnn: Hierarchical structure-adaptive rnn for video summarization. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 7405-7414).

  22. Potapov, D., Douze, M., Harchaoui, Z., & Schmid, C. (2014, September). Category-specific video summarization. In European conference on computer vision (pp. 540-555). Cham: Springer International Publishing.

  23. Lee, S., Sung, J., Yu, Y., & Kim, G. (2018). A memory network approach for story-based temporal summarization of 360 videos. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1410-1419).

  24. Yuan, L., Tay, F. E., Li, P., Zhou, L., & Feng, J. (2019, July). Cycle-SUM: Cycle-consistent adversarial LSTM networks for unsupervised video summarization. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 33, No. 01, pp. 9143-9150).

  25. Ji, Z., Xiong, K., Pang, Y., & Li, X. (2019). Video summarization with attention-based encoder–decoder networks. IEEE Transactions on Circuits and Systems for Video Technology, 30(6), 1709-1717.

  26. Fajtl, J., Sokeh, H. S., Argyriou, V., Monekosso, D., & Remagnino, P. (2018, December). Summarizing videos with attention. In Asian conference on computer vision (pp. 39-54). Cham: Springer International Publishing.

  27. Rochan, M., Ye, L., & Wang, Y. (2018). Video summarization using fully convolutional sequence networks. In Proceedings of the European conference on computer vision (ECCV) (pp. 347-363).

  28. Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.

  29. Liu, Y. T., Li, Y. J., Yang, F. E., Chen, S. F., & Wang, Y. C. F. (2019, September). Learning hierarchical self-attention for video summarization. In 2019 IEEE international conference on image processing (ICIP) (pp. 3377-3381). IEEE.

  30. Larsen, A. B. L., Sønderby, S. K., Larochelle, H., & Winther, O. (2016, June). Autoencoding beyond pixels using a learned similarity metric. In International conference on machine learning (pp. 1558-1566). PMLR.

  31. Gygli, M., Grabner, H., Riemenschneider, H., & Van Gool, L. (2014, September). Creating summaries from user videos. In European conference on computer vision (pp. 505-520). Cham: Springer International Publishing.

  32. Song, Y., Vallmitjana, J., Stent, A., & Jaimes, A. (2015). Tvsum: Summarizing web videos using titles. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 5179-5187).

  33. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., ... & Rabinovich, A. (2015). Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1-9).

  34. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., ... & Fei-Fei, L. (2015). Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3), 211-252.

  35. Rochan, M., & Wang, Y. (2019). Video summarization by learning from unpaired data. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 7902-7911).

  36. Kaufman, D., Levi, G., Hassner, T., & Wolf, L. (2017). Temporal tessellation: A unified approach for video analysis. In Proceedings of the IEEE international conference on computer vision (pp. 94-104).

  37. Huang, S., Li, X., Zhang, Z., Wu, F., & Han, J. (2018). User-ranking video summarization with multi-stage spatio–temporal representation. IEEE Transactions on Image Processing, 28(6), 2654-2664.

  38. Feng, L., Li, Z., Kuang, Z., & Zhang, W. (2018, October). Extractive video summarizer with memory augmented neural networks. In Proceedings of the 26th ACM international conference on Multimedia (pp. 976-983).

  39. Gygli, M., Grabner, H., & Van Gool, L. (2015). Video summarization by learning submodular mixtures of objectives. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3090-3098).

Sustainable Energy and Artificial Intelligence
Volume 1, Issue 4
Autumn 2025
Pages 205-216

  • Receive Date 16 November 2024
  • Revise Date 07 January 2025
  • Accept Date 15 February 2025
  • First Publish Date 12 July 2025

Home

Guide for Authors

Submit Manuscript

Reviewers

Contact Us