CC BY
28
ANALYSIS OF CURRENT VIDEO DATABASES FOR QUALITY
ASSESSMENT
Anastasia Mozhaeva,
The University of Waikato, Hamilton, New Zealand; Moscow Technical University of Communications and Informatics Moscow, Russia, anast.mozhaeva@gmail.com
Elizaveta Vashenko,
Moscow Technical University of Communications and Informatics Moscow, Russia
Vladimir Selivanov,
Moscow Technical University of Communications and Informatics Moscow, Russia
Alexei Potashnikov,
Moscow Technical University of Communications and Informatics Moscow, Russia
Igor Vlasuyk,
Moscow Technical University of Communications and Informatics Moscow, Russia
Lee Streeter,
The University of Waikato, Hamilton, New Zealand
DOI: 10.36724/2072-8735-2022-16-2-48-56
Manuscript received 16 January 2022; Accepted 10 February 2022
Keywords: video quality assessment, subjective testing, video database, video dataset, applied television, mean opinion score (MOS)
The popularity of video streaming has grown significantly over the past few years. Video quality prediction metrics can be used to perform extensive video codec analysis and customize high-quality assurance. Video databases with subjective ratings form an important basis for training video quality metrics, and codecs based on machine learning algorithms. More than three dozen subjective video databases are now available. In this article, modern video databases are presented, analyzed current database and findings methods for improving. For analysis, performance criteria are proposed based on subjective assessments when creating a database of video sequences. At this stage of development, subjective assessments are the most difficult part of creating a database of video sequences, since these assessments are expensive and time-consuming. In addition, subjective experimentation is further complicated by many factors, including viewing distance, a display device, lighting conditions, vision, and mood of the subjects. This information will allow researchers to have a more detailed understanding of the video databases, a new method for collecting subjective data, and can also help in planning future experiments.
Для цитирования:
Можаева А., Ващенко Е., Селиванов В., Поташников А., Власюк И., Стрейтер Л. Анализ современных баз видео данных оценки качества // T-Comm: Телекоммуникации и транспорт. 2022. Том 16. №2. С. 48-56.
For citation:
Mozhaeva A., Vashenko E., Selivanov V., Potashnikov A., Vlasuyk I., Streeter L. (2022) Analysis of current video databases for quality assessment. T-Comm, vol. 16, no.2, pр. 48-56. (in Russian)
I. INTRODUCTION
Video streaming continues to occupy a growing share of Internet bandwidth, and video is expected to account for 82% of Internet traffic by 2022 [1]. With the explosive growth in video traffic, improvements in video encoding technologies are critical for video streaming companies in the coming years. At the present stage of technology development, video coding systems show high-quality and completely satisfactory results. Solutions require video quality issues in streaming video, such as creating video quality assessments, codecs using whole or partial machine learning [2]. However, creating quality metrics and codecs using whole or partial machine learning requires video datasets that accurately reflect the human user experience.
The lack of annotated databases used to be a major hurdle for researchers working on quality assessment algorithms. Even uncompressed video content was hard to find [3]. Currently, there are over 30 publicly available databases and many small datasets that are used by experts for personal research. However, two main problems arise that were absent earlier and were not considered in detail by the scientific community at the current time. First, the problem of choosing the most suitable databases for research, because more than 9 years have passed since the last detailed analysis of video databases, which is a critical aspect in the modern growth of information technology.
The problem also remains in the lack of data to create metrics and parts of video codecs based on machine learning. The open question is how, while creating a large number of new publicly available databases with subjective ratings, the number of which has more than tripled since 2012, the modern industry still lacks training datasets. In this paper, we analyze the 29 most commonly used video datasets to assess encoding quality and generate machine learning-based video quality metrics that accurately reflect human user experience. We also propose a criterions and solution that clearly demonstrates the problem of creating a large-scale dataset of video sequences.
Comparing databases using the same criteria is helpful for model developers, who can make a more informed decision about which databases may be most suited for their specific benchmarking or other needs [3, 4]. In addition, the presented criteria for evaluating video databases explains the problem with a small variety of content, which leads to limitations in the development and evaluation of metrics and codecs using full or partial machine learning effectively. This work will allow researchers to have a more detailed understanding of the video databases and may also help in planning future experiments.
The work is organized as follows. Section II provides an overview of video databases commonly used in the current world, with annotated subjective quality ratings. Section III proposes new scoring criteria for subjective scores, which is then used to compare databases. Section IV reviews the findings of the analysis and discusses methods for improving the database and future work.
II. VIDEO DATABASES
Here we present 29 video databases with the subjectively rated for quality, following previous research [5], that in 2012 provided a comprehensive analysis of the video datasets at the time.
• EPFL / PoliMI Vi eo Quality Assessment Database. Compressed H.264 videos with transmission over an error-prone network [6, 7].
• IRCCyN / IVC 1 80i. Compressed H.264 [8, 9].
• IRCCyN / VC SD. Compressed H.264, with and without transmissionerrors [10, 11].
• IVP Database. Compressed MPEG-2, Dirac wavelet and H.264 codecs, as well as H.264 streams that are affected by packet loss simulation [12].
• LIVE Vi eo Quality Database. Compressed MPEG-2, H.264 compression, the simulated transmission of compressed H.264 bitstreams over error-prone IP wired and wireless networks [13,14].
• MMSP D Video Quality Assessment Database, is the first publicly available 3D video quality database [15, 16].
• MMSP Scala le Video Database (SVD). Test conditions include two scalable video codecs using different spatial and temporal resolutions [17, 18].
• Poly @ NYU Video Quality Databases. Three separate but related tests using video with different frame rates and quantization parameters [19, 20, 21, 22].
• Poly @ NYU Packet Loss (PL) Database Small. Compressed H.264, with packet loss [23].
• VQEG FR-TV Phase I Database. The oldest publicly available quality database. Compressed MPEG-2 compression and transmission, and even includes some analog distortion [24, 25].
• VQEG HDTV Database. Compressed MPEG-2 and H.264, with various types of network disturbances [26, 27].
• AVT-PNATS-UHD (2019). Compressed H.264, High Efficiency Video Coding (HEVC) and VP9 and frame rate variations [28].
• BVI-HD Perceptual Video Quality Database (2018) HD video sequences with frame rates up to 120 Hz [29].
• BVI-HFR High Frame Rate Video Database (2015). Video sequences generated using both original HEVC and HEVC with synthesis mode [30].
• Konstanz Natural Video Database (KoNViD-lk) (2017). 1200 videos with subjective data and attribute evaluation [31].
• LIVE YouTube High Frame Rate (LIVE-YT-HFR) Database (2020). Videos are processed with 5 levels of compression at eachframe rate [32,33].
• LIVE Wild Compressed Video Quality Database (2020). Videos captured with a wide variety of mobile cameras, covering a wide range of content and quality. Most of these videos are distorted with various authentic mixed distortions when captured. H.264 video compression format [34].
• LIVE Mo ile Video Quality Database (2012). Compressed H.264 video with artefacts such as packet loss, frame freeze, and rate adaptation [35, 36, 37].
• ETRI-LIVE Space-Ti e Subsampled Video Quality (STSVQ) Database (2020). Videos created by applying different levels of combined space-time downsampling [38].
• LIVE Ne flix Video Quality of Experience Database (2017). 112 videos of typical adaptive streaming artefacts rated by 55+ people on a mobile device, Figure 1 [39, 40].
Fig. 1. LIVE Netflix Video Quality of Experience Database
Fig. 2. LIVE Video Quality Challenge (VQC) Database
• L VE Video Quality Challenge (VQC) Database (2018). Videos captured using 101 different devices with a wide range of complex, reliable distortion levels. An average of 240 quality ratings was collected for each video through crowdsourcing, Figure 2 [41, 42, 43]:
• MCL-JCV Database (2016). Compressed H.264 / AVC at quality factors (QF) ranging from 1 to 51 [44, 45].
• T M 1080p25 Dataset (2010). Compressed H.264 / AVC and Dirac [46].
• ideoSet (2017). The database includes 3520 sequences, which were evaluated by 800 participants [47].
• LSVQ Data ase. Video quality dataset containing 39,000 distorted real-world videos and 117,000 localized spatiotemporal video patches and 5.5 million human perceptions, 38811 were used for the base with 35 estimates per video sequence [48, 49].
• LIVE-NFLX-II. The database includes 420 videos that were rated by 65 subjects, resulting in 9750 continuous and 9750 retrospective subjective opinions [50].
III. ANALYSIS
There are many criteria that can be used to assess and compare databases [3]. The evaluation of databases is based on three components: quantitative comparisons of original content, testing conditions, and subjective evaluations [51]. Today there are enough solutions to the problem of original content and testing conditions [52]. However, according to subjective assessments, there is still no optimal solution, but subjective assessments are the most valuable and possibly the most difficult part when creating video sequence datasets.
Subjective assessments are costly and time consuming. In addition, subjective experimentation is further complicated by many factors, including viewing distance, display device, lighting conditions, vision, and the mood of the subjects [53]. If the creation of the base takes place in different days, or what is even more problematic - experiments are going on in parallel in several laboratories, it is necessary to strictly observe the same conditions for all participants, the results of which will then be evaluated jointly. In addition, subjective video encoding tests are usually performed by very few experts, who are called the gold standard for worst-case analysis [54].
However, worst-case analysis does not reflect the actual quality for the visual content, and the quantity and quality of perceived distortion levels is individual. Existing databases have very different and often almost incomparable approaches to the collection of subjective data. In order to analyze the collection of subjective data in this section, we will focus on several aspects that we use to evaluate databases:
• Criteria 1. Time efficiency (%). Here we mean the total time spent on creating the entire test base in relation to the total duration of all finished video sequences included in the base along with the original videos.
C1 =
r Ms. ^
V NaH V
where - umber of artifacts, - number of experts for 1 sequence, - number of sequences in database.
• Criteria 2. The processing time of one artifact by the participants in the subjective research.
C2 = ((Nste + tp)N /Nall)60k
where - number of experts for 1 sequence, - time of one sequence (second), is the preparation time [55], - coefficient of converting estimates to a sequence of 10 second long.
A comparison of databases using the time efficiency criterion is presented in Table 1. Please note that another difficult point when comparing existing databases is that not all authors provide complete information about creating a database, empty cells in Table 1. As you can see in Table 1, optimizing for one or more of our criteria does not necessarily lead to a "bettef' database, Figure 3.
9 297 585
Criterion 1
Fig. 3. The visualization of Table 2
Table 2 provides an analysis of the databases by criteria. Criteria 1 in the first column is the real estimates of 28 databases, however, for MCL-JCV and LIVE (VQC) databases, this criterion has very high ratings compared to other databases, which we estimate as an outlier in the analysis, and these databases cannot be included for further rational analysis. Similarly, for Criterion 2 for databases - Poly@NYU (PL), BVI-HD (VQD), KoNViD-1k, LIVE (VQC).
Criteria 1 and Criteria 2 (no emissions), in the remaining columns are presented for 24 databases, Table 3. The visualization
of Table 3 is presented in Figure 4, where databases with acceptable optimal values for Criteria 1 and Criteria 2 the use of in blue. Mean is taken as the threshold value of acceptability. Where Mean:
_ 2x;. 11
Variance:
2 £(x,. - x) CT = --
n -1
where a - standard deviation. As can be seen from Table 1, when used as the threshold value of acceptability - Mean for three criteria of Table 2, only 3 video databases fall into the range of acceptable optimal values.
Fig. 4. The visualization of Table 3, where databases with two acceptable optimal values are indicated in blue, Mean is taken as the threshold value of acceptability
TABLE 1
Database of Number of Number of Time of one Number of Criteria 1 Criteria 2 Criteria 3
sequences artifacts experts sequence (second) experts for 1 sequence
EPFL/PoliMI -1 156 78 40 10 23 12 7 390
EPFL/PoliMI -2 156 78 10 17 9 5 213
IRCCyN/IVC 1080i 216 192 29 10 28 25 6 429
IRCCyN/IVC SD 90 84 25 10 25 23 7 446
IVP 1 128 42 10 35 32 9 750
LIVE VQD 165 150 38 10 38 35 9 842
MMSP 3D 36 30 20 10 17 14 9 361
MMSP 84 16 10 16 15 5 183
Poly@NYU(VQD -1) 66 60 10 22 20 7 403
Poly@NYU(VQD -2) 72 68 10 15 14 5 176
Poly@NYU(VQD -3) 186 180 10 15 15 3 125
Poly@NYU (PL) 46 34 32 2 32 24 50 4190
VQEG FR-TV 340 320 287 10 61 57 12 1841
VQEG HDTV 740 740 120 10 24 24 4 259
AVT-PNATS-UHD 49 4947 121 7 24 24 3 171
BVI-HD (VQD) 416 384 86 5 86 79 18 4044
BVI-HFR 110 88 29 10 29 23 8 589
KoNViD-1k 1200 1200 642 8 114 114 20 5753
LIVE-YT-HFR 480 8 40 39 7.5 1234
LIVE Wild 275 220 10 40 32 8 848
LIVE mobile (VQD) 210 200 38 15 27 26 8 547
ETRI-LIVE-STSVQ 437 10 34 33 6 548
LIVE Netflix 126 112 55 10 44 39 11 1239
LIVE (VQC) 585 585 4776 10 240 240 44 26462
MCL-JCV 1124 120 5 50 585 2 1152
TUM 1080p25 52 48 19 10 19 18 7 338
VideoSet 44880 800 5 128
LSVQ Database 38811 38811 7 35 35 5.6 501
LIVE-NFLX-II 420 65 10 22 21 4 230
TABLE 2
Criteria 1 Criteria 2 Criteria 3
Minimum 2 125
Maximum 50 26462
Mean 10 1938
StdDEv 11 4997
TABLE 3
Criteria 1 (no emissions) Criteria 2 (no emissions) Criteria 3 (no emissions)
Minimum 3 125
Maximum 12 1239
Mean 6.7 491
StdDEv 2 319
Despite the fact that the above provides comprehensive data for the selection of modern video databases to optimal use of experts' time in subjective testing, here we also offer an estimate of the cost of one artifact in a video database. This is a necessary parameter, since the subjective estimates mentioned above are the most expensive part when creating a database, and it is the number of artifacts in the database that is critical when creating codecs and metrics with full or partial use of machine learning.
C3 = ((( Nste )/60 + tp )S / N,) k
where, C3 this is the cost of one artifact in the database, t is the total time of one person when viewing video sequences, is the minimum wage per hour. When calculating, it is also necessary to take into account the error caused by the fact that some databases contain viewing of both artifacts and the original video, while others contain only video with artifacts.
The data on the calculation of this criterion are given in Table 1. In column Criteria 3, we can see the cost of one artefact, the values Russian rubles. However, as can be seen from the Table 1 we have emissions where the cost of artefacts exceeds 1300 rubles. From the information above, we limited the price range from 0 to 1239 rubles. Adding this most visual criterion for the designers of codecs and metrics, we can see that the number of databases, optimal in relation to the cost of the artefact has decreased from 23 to 16, Figure 5.
39
aj LJ
125 682 1239
Criterion 3
Fig. 5. The visualization of Criterion 3 (Table 3), where databases with acceptable optimal values are indicated in blue
IV. DISCUSSION
In this article, we have proposed criteria that can be useful when developing new databases. However, the current biggest problem how creating metrics and codecs using machine learning is the lack of data, or in other words, large-scale databases. To solve this problem by involving people in tests, new approaches to the collection of subjective data are needed, taking into account the maximum possible amount of processing of artifacts with the minimum investment of time for subjective tests.
Here we will look at one of the new approaches. In our previously work, a new device is presented that allows one to collect estimates of the subjective level of quality using the idea of finding an acceptable minimum level of quality for a participant, or, in other words, the threshold of perception [56]. This device optimizes the collection of subjective assessments for this stage in the development of modern telecommunication technologies, and allows you to create video sequences of constant quality. Or other words it creates conditions for creating a database with the maximum number of artifacts and a minimum of subjective experiments. For create a database with this device, it is possible to use a similar time with such databases as LIVE Wild [34], but with a significantly larger number of artifacts. Also, such databases KoNViD-1 [31] will be significantly inferior to the databases created using the device for collecting subjective assessments from [56], in terms of the time spent on subjective tests with the same number of processed artifacts.
A block diagram of the device is shown in Figure 6. First, distorted video sequences with 9 quality levels are created; in modern streaming video, the most demanded threshold levels are at levels from 6 to 10 Mbps, but both low and high levels are necessary for the accuracy of the experiment. The next step will save 9 distorted and one reference sequence. Each clip has an original link and a compressed video sequence with 9 different quality levels. Next comes the selection stage, when a participant can choose a sequence of an acceptable minimum threshold with a smooth change in quality levels. Each participant finds the minimum acceptable threshold for a video sequence for one video, constantly adjusting the quality using a manipulator and overcoming stress.
Fig. 6. The process of measuring the quality of encoded video based on finding an acceptable minimum threshold of perception
The approach proposed in [56] will make it possible to obtain video with a constant quality assessment, which is created by the users themselves and which codec developers should strive for as optimal for users.
Given the analysis of databases presented above, at the present moment in the development of telecommunication technologies, there are opportunities to implement more optimal approaches to collecting subjective assessments for the creation of new databases with various distortions and well-labeled data in a larger volume. New device will make it possible to obtain video with a constant quality assessment, which is created by the users themselves and which codec developers should strive for as optimal for users.
V. CONCLUSION
At the current stage of technology development, there are more than three dozen publicly available video quality databases, as well as a large number of datasets for private testing. This makes it easier to check the quality of algorithms, but is still not enough for generating quality assessment models or video codecs based on full or partial machine learning. The article presents and analyzes modern video databases. Criteria are proposed for analyzing the optimal use of experts' time in subjective testing when creating a database of video sequences, or, in other words, how long it takes to create a database using subjective tests. Also, discussing the method for improving the creation of future video databases.
The list of databases is bound to grow as new applications emerge. The data presented here can be useful for creating a testing methodology that takes into account the maximum possible amount of artifact processing with a minimum investment of time for subjective tests, which is currently not in the public domain.
REFERENCES
1. Cisco Visual Networking Index: Forecast and Methodology 20172022, Feb. 2019, [online] Available: https://www.cisco.eom/c/en/us/ solutions/collateral/serviceprovider/visual-networking-index-vni/white-paper-cll741490.html.J. Clerk Maxwell, A Treatise on Electricity and Magnetism, 3rd ed., vol. 2. Oxford: Clarendon, 1892, pp. 68-73.
2. S.-F. Chang and A. Vetro (2005), "Video adaptation: Concepts technologies and open issues", Proc. of the IEEE, vol. 93, no. 1, pp. 148-158.
3. A. Mozhaeva, L. Streeter, I. Vlasuyk and A. Potashnikov (2021), "Full Reference Video Quality Assessment Metric on Base Human Visual System Consistent with PSNR," 2021 28th Conference of Open InnovationsAssociation (FRUCT),pp. 309-315.
4. A. I. Mozhaeva, I. V. Vlasuyk, A. M. Potashnikov, M. J. Cree and L. Streeter (2021), "The Method and Devices for Research the Parameters of the Human Visual System to Video Quality Assessment," 2021 Systems of Signals Generating and Processing in the Field of on Board Communications,pp. 1-5.
5. S. Winkler (2012), "Analysis of Public Image and Video Databases for Quality Assessment," in IEEEJournal ofSelected Topics in SignalProcessing, vol. 6, no. 6, pp. 616-625, Oct. 2012.
6. F. De Simone et al. (2009), "EPFL-PoliMI video quality assess-mentdatabase," [Online], Available: http://vqa.como.polimi.it/
7. F. De Simone et al. (2009), "Subjective assessment of H.264/AVC video sequences transmitted over a noisy channel," in Proc. Int. Workshop Quality of Multimedia Experience (QoMEX), San Diego, CA, Jul. 29-31.
8. S. Pechard, R. Pepion, and P. Le Callet, "IRCCyN IVC 1080i data- base," 2008 [Online], Available: http://www.irccyn.ec-nantes.fr/spip. php?article541.
9. S. Pechard, R. Pepion, and P. Le Callet (2008), "Suitable methodology in sub- jective video quality assessment: A resolution dependent paradigm," in Proc. Int. Workshop Image Media Quality and its Applicat. (MQA), Kyoto, Japan, Sep. 2008.
10. F. Boulos, W. Chen, B. Parrein, and P. Le Callet, "IRCCyN IVC SD Rol database," 2009 [Online], Available: http://www.ir-ccyn.ec-nantes.fr/spip.php?article551.
11. S. Pechard, R. Pepion, and P. Le Callet (2000), "Region-of-interest intra prediction for H.264/AVC error resilience," in Proc. Int. Conf.ImageProcess. (ICIP), Cairo,Nov. 7-10.
12. F. Zhang, S. Li, L. Ma, Y. C. Wong, and K. N. Ngan (2011), "IVP subjective quality video database," [Online], Available: http://ivp.ee.cuhk. edu.hk/research/database/subjective/
13. K. Seshadrinathan, R. Soundararajan, A. C. Bovik, and L. K. Cormack (2010), "LIVE video quality database," [Online], Available: http://live. ece.utexas.edu/research/quality/live_video.html
14. K. Seshadrinathan, R. Soundararajan, A. C. Bovik, and L. K. Cormack (2010), "Study of subjective and objective quality assessment of video," IEEE Trans. Image Process., vol. 19, no. 6, pp. 1427-1441, Jun. 2010.
15. L. Goldmann et al. (2010), "3D video quality assessment," [Online], Available: http://mmspl.epfl.ch/page38842.html.
16. L. Goldmann, F. De Simone, and T. Ebrahimi (2010), "A comprehensive data- base and subjective evaluation methodology for quality of experience in stereoscopic video," in Proc. SPIE 3D Image Process. (3DIP) andApplicat., San Jose, CA, Jan. 17-21, vol. 7526.
17. J.-S. Lee et al. (2010), "MMSP scalable video database," [Online], Available: http://mmspg.epfl.ch/svd.
18. J.-S. Lee, F. De Simone, and T. Ebrahimi (2011), "Subjective quality eval- uation via paired comparison: Application to scalable video coding," IEEE Trans.Multimedia, vol. 13, no. 5, pp. 882-893, Oct. 2011
19. Y. Wang et al. (2008), "Poly@NYU video quality databases," [Online], Available: http://vision.poly.edu/index.html/index.php?n=Home-Page.QualityAssessmentDatabase
20. Y.-F. Ou, T. Liu, Z. Zhao, Z. Ma, and Y. Wang (2008), "Modeling the impact of frame rate on perceptual quality of video," in Proc. Int. Conf.ImageProcess. (ICIP), SanDiego, CA, Oct. 12-15.
21. Y.-F. Ou, Z. Ma, and Y. Wang (2009), "A novel quality metric for compressed video considering both frame rate and quantization artifacts," in Proc. Int. Workshop Video Process. Quality Metrics (VPQM), Scottsdale, AZ, Jan. 15-16.
22. Y.-F. Ou, Y. Zhou, and Y. Wang (2010), "Perceptual quality of video with frame rate variation: A subjective study," in Proc. Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Dallas, TX, Mar. 14-19, 2010.
23. X. Feng, T. Liu, D. Yang, and Y. Wang (2008), "Saliency based objective quality assessment of decoded video affected by packet losses," in Proc. Int. Conf. Image Process. (ICIP), San Diego, CA, Oct. 1215,2008.
24. "VQEG FR-TV Phase I database," Video Quality Experts Group (VQEG), 2000 [Online], Available: ftp://ftp.crc.ca/crc/vqeg/TestSe-quences.
25. "Final report from the Video Quality Experts Group on the validation of objective models of video quality assessment," VQEG, Apr. 2000 [Online], Available: http://www.vqeg.org.
26. "Report on the validation of video quality models for high definition video content," VQEG, Jun. 2010 [Online], Available: http://www.vqeg.org.
27.'The Consumer Digital Video Library," CDVL, 2010 [Online], Available: http://www.cdvl.org.
28. R. R. Ramachandra Rao, S. Goring, W. Robitza, B. Feiten, and A. Raake (2019), "AVT-VQDB-UHD-1: A large scale video quality database for UHD-1," in Proc. IEEE Int. Symp. Multimedia (ISM), Dec. 2019, pp. 1-8 . Available:
https://ieeexplore.ieee.org/document/8959059.
29. Fan Zhang, Felix Mercer Moss, Roland Baddeley, and David, R. Bull (2018), "BVI-HD: A Video Quality Database for HEVC Compressed and Texture Synthesised Content", IEEE Trans, on Multimedia.
30. Mackin, A. and Zhang, F. and Bull, D. (2015), "A study of subjective video quality at various frame rates", 2015 22nd IEEE International Conference on Image Processing (ICIP).
31. V. Hosu et al. (2017), "The Konstanz natural video database (KoNViD-lk)," 2017 Ninth International Conference on Quality of Multimedia Experience (QoMEX), pp. 1-6, doi: 10.1109/QoMEX.2017.7965673.
32. P. C. Madhusudana, X. Yu, N. Birkbeck, Y. Wang, B. Adsumilli and A. C. Bovik (2020), "Subjective and Objective Quality Assessment of High Frame Rate Videos", submitted to IEEE Transactions on Image Processing, [paper],
33. P. C. Madhusudana, N. Birkbeck, Y. Wang, B. Adsumilli and A. C. Bovik (2020), "Capturing Video Frame Rate Variations through En-tropic Differencing", arXiv preprint arXiv:2006.11424.
34. X. Yu, N. Birkbeck, Y. Wang, C. G. Bampis, B. Adsumilli and A. C. Bovik, "Predicting the Quality of Compressed Videos with PreExisting Distortions", submitted to IEEE Transactions on Circuits and Systemsfor Video Technology, [paper],
35. A. K. Moorthy, L. K. Choi, A. C. Bovik and G. deVeciana (2012), "Video Quality Assessment on Mobile Devices: Subjective, Behavioral and Objective Studies", IEEE Journal of Selected Topics in Signal Processing, to appear in October 2012.
36. A. K. Moorthy, L. K. Choi, G. deVeciana, and A. C. Bovik (2012), "Mobile Video Quality Assessment Database," IEEE ICC Workshop on Realizing Advanced Video Optimized Wireless Networks, Ottawa, Canada, June 10-15,2012.
37. A. K. Moorthy, L. K. Choi, G. deVeciana, and A. C. Bovik (2012), "Subjective Analysis of Video Quality on Mobile Devices," Sixth International Workshop on Video Processing and Quality Metrics forConsumerElectronics (VPQM) (invitedarticle), Scottsdale,Arizona, January 15-16,2012.
38. D. Lee, S. Paul, C. G. Bampis, H. Ko, J. Kim, S. Jeong, B. Homan and A. C. Bovik, "A Subjective and Objective Study of SpaceTime Subsampled Video Quality", submitted to IEEE Transactions on ImageProcessing. [paper],
39. C. G. Bampis, Z. Li, A. K. Moorthy, I. Katsavounidis, A. Aaron, and A. C. Bovik (2017), "Study of Temporal Effects on Subjective Video Quality of Experience," IEEE Trans. Image Process., vol. 26, no. 11, pp. 5217-5231.
40. C. G. Bampis, Z. Li, A. K. Moorthy, I. Katsavounidis, A. Aaron and A. C. Bovik (2016), "LIVE Netflix Video Quality of Experience Database," Online: http://live.ece.utexas.edu/research/LIVE_NFLXStudy/ index.html.
41. Z. Sinno and A.C. Bovik (2019), "Large-Scale Study of Perceptual Video Quality," IEEE Transactions on Image Processing, vol. 28,
no. 2, pp. 612-627, February 2019.
42. Z. Sinno and A.C. Bovik (2018), "Large Scale Subjective Video Quality Study," 2018 IEEE International Conference on Image Processing, Athens, Greece, October 2018.
43. Z. Sinno and A.C. Bovik (2018), "LIVE Video Quality Challenge Database", Online: http://live.ece.utexas.edu/research/LIVEVQC/ index.html.
44. J. Y. Lin, R. Song, T. Liu, H. Wang and C.-C. J. Kuo (2015), "MCL-V: A streaming video quality assessment database", J. Vis. Commun. ImageRepresent., vol. 30, pp. 1-9, Jul. 2015.
45. "MCL-JCV Dataset", [online] Available at: http://mcl.usc.edu/mcl-jcv-dataset/.
46. C. Keimel, J. Habigt, T. Habigt, M. Rothbucher, and K. Diepold (2010), "Visual quality of current coding technologies at high definition IPTV bitrates," in Multimedia Signal Processing (MMSP), 2010 IEEE International Workshop on, pp. 390-393.
47. H. Wang et al.(2017), "VideoSet: A large-scale compressed video quality dataset based on JND measurement", J. Vis. Commun. Image Represent., vol. 46, pp. 292-302, Jul. 2017.
48. Z. Ying, M. Mandai, D. Ghadiyaram and A.C. Bovik (2020), "Patch-VQ: 'Patching Up' the Video Quality Problem," arXiv 2020. [paper]
49. Z. Ying, M. Mandai, D. Ghadiyaram and A.C. Bovik (2020), "LIVE Large-Scale Social Video Quality (LSVQ) Database", Online:https://github.com/baidut/PatchVQ.
50. C. G. Bampis, Z.Li, I. Katsavounidis, TY Huang, C. Ekanadham and A. C. Bovik, "Towards Perceptually Optimized End-to-end Adaptive Video Streaming," submitted to IEEE Transactions on Image Processing.
51. S. Winkler (2012), "Image and video quality resources," [Online], Avail- able: http://stefan.winkler.net/resources.html.
52. Y. Wang, S. Inguva and B. Adsumilli (2019), "YouTube UGC dataset for video compression research", Proc. IEEE 21st Int. Workshop Multimedia SignalProcess. (MMSP), pp. 1-5, Sep. 2019.
53. P. Mohammadu, A. Ebrahimi-Moghadam, S. Shirani (2015), "Subjective and Objective Quality Assessment of Image: A Survey", Majlesi Journal of Electrical Engineering, vol.9(l), Mar 2015, pp.55-83.
54. H. Wang et al. (2016), "MCL-JCV: A JND-based H.264/AVC video quality assessment dataset", Proc. IEEE Int. Conf. Image Process. (ICIP), pp. 1509-1513, Sep. 2016
55. BT-500-11: Methodology for the Subjective Assessment of the Quality ofTelevision Pictures, 2012.
56. A. Mozhaeva, A. Potashnikov, I. Vlasuyk and L. Streeter (2021), "Constant Subjective Quality Database: The Research and Device of Generating Video Sequences of Constant Quality," 2021 International Conference on Engineering Management of Communication and Technology (EMCTECH), 2021, 10.1109/EMCTECH53459.2021.9618977.
АНАЛИЗ СОВРЕМЕННЫХ БАЗ ВИДЕО ДАННЫХ ОЦЕНКИ КАЧЕСТВА
Анастасия Можаева, Университет Вайкато, Гамильтон, Новая Зеландия; Московский технический университет связи и информатики, Москва, Россия, anast.mozhaeva@gmail.com Елизавета Ващенко, Московский технический университет связи и информатики, Москва, Россия Владимир Селиванов, Московский технический университет связи и информатики, Москва, Россия Алексей Поташников, Московский технический университет связи и информатики, Москва, Россия Игорь Власюк, Московский технический университет связи и информатики, Москва, Россия Ли Стрейтер, Университет Вайкато, Гамильтон, Новая Зеландия
Аннотация
Популярность потокового видео значительно выросла за последние несколько лет. Показатели прогнозирования качества видео можно использовать для выполнения обширного анализа видеокодеков и настройки гарантии высокого качества. Базы данных видео с субъективными оценками составляют важную основу для обучения метрик качества видео и кодеков на основе алгоритмов машинного обучения. Сейчас доступно более трех десятков баз данных видео с субъективными оценками. В этой статье представлены и проанализированы современные видео базы. Так же представлен метод улучшения, который можно использовать при создании будущих баз данных. Для анализа предлагаются критерии эффективности, основанные на данных о сборе субъективных оценкок при создании базы данных видеопоследовательностей. На текущем этапе разработки субъективные оценки являются наиболее сложной частью создания базы данных видеопоследовательностей, поскольку эти оценки дороги и требуют значительного времени для создания. Кроме того, субъективное экспериментирование дополнительно осложняется многими факторами, включая расстояние просмотра, устройство отображения, условия освещения, видение и настроение испытуемых. Данная работа позволит исследователям получить более подробное представление о базах данных видео, новом методе сбора субъективных данных, а также может помочь в планировании будущих экспериментов. Так же данная работа будет полезна для специалистов в области сжатия и передачи медиа контента.
Ключевые слова: оценка качества видео, субъективное тестирование, база данных видео, набор видеоданных, прикладное телевидение, средняя оценка мнения (MOS).
Литература
1. Cisco Visual Networking Index: Forecast and Methodology 2017- 2022, Feb. 2019, [online] Available: https://www.cisco.com/c7en/us/solutions/collat-eral/serviceprovider/vis ual-networking-index-vni/white-paper-cll74l490.html.J. Clerk Maxwell, A Treatise on Electricity and Magnetism, 3rd ed., vol. 2. Oxford: Clarendon, 1892, pp. 68-73.
2. S.-F. Chang and A. Vetro. Video adaptation: Concepts technologies and open issues // Proc. of the IEEE, vol. 93, no. 1, 2005, pp. 148-158.
3. A. Mozhaeva, L. Streeter, I. Vlasuyk and A. Potashnikov. Full Reference Video Quality Assessment Metric on Base Human Visual System Consistent with PSNR // 2021 28th Conference of Open Innovations Association (FRUCT), 2021. pp. 309-315.
4. A. I. Mozhaeva, I. V. Vlasuyk, A. M. Potashnikov, M. J. Cree and L. Streeter. The Method and Devices for Research the Parameters of the Human Visual System to Video Quality Assessment // 2021 Systems of Signals Generating and Processing in the Field of on Board Communications, 2021, pp. 1-5.
5. S. Winkler. Analysis of Public Image and Video Databases for Quality Assessment // in IEEE Journal of Selected Topics in Signal Processing, vol. 6, no.
6. pp. 616-625, Oct. 2012.
6. F. De Simone et al. EPFL-PoliMI video quality assessment database, 2009 [Online]. Available: http://vqa.como.polimi.it/
7. F. De Simone et al. Subjective assessment of H.264/AVC video sequences transmitted over a noisy channel // in Proc. Int. Workshop Quality of Multimedia Experience (QoMEX), San Diego, CA, Jul., 2009, pp. 29-31.
8. S. Pechard, R. Pepion, and P. Le Callet. IRCCyN IVC l080i database. 2008 [Online]. Available: http://www.irccyn.ec-nantes.fr/spip. php?article54l
9. S. Pechard, R. Pepion, and P. Le Callet. Suitable methodology in sub- jective video quality assessment: A resolution dependent paradigm // Proc. Int. Workshop Image Media Quality and its Applicat. (IMQA), Kyoto, Japan, Sep. 2008.
10. F. Boulos, W. Chen, B. Parrein, and P. Le Callet. IRCCyN IVC SD RoI database. 2009 [Online]. Available: http://www.ir- ccyn.ec-nantes.fr/spip.php'arti-cle55l
11. S. Pechard, R. Pepion, and P. Le Callet. Region-of-interest intra prediction for H.264/AVC error resilience // Proc. Int. Conf. Image Process. (ICIP), Cairo, Nov. 7-l0, 2000.
12. F. Zhang, S. Li, L. Ma, Y.C. Wong, K.N. Ngan. IVP subjective quality video database. 20ll [Online]. Available: http://ivp.ee.cuhk. edu.hk/research/data-base/subjective/
13. K. Seshadrinathan, R. Soundararajan, A.C. Bovik, L.K. Cormack. LIVE video quality database. 2010 [Online]. Available: http://live. ece.utexas.edu/research/quality/live_video.html
14. K. Seshadrinathan, R. Soundararajan, A.C. Bovik, L.K. Cormack. Study of subjective and objective quality assessment of video. IEEE Trans. Image Process., vol. l9, no. 6, pp. l427-l44l, Jun. 20l0.
15. L. Goldmann et al. 3D video quality assessment. 20l0 [Online]. Available: http://mmspl.epfl.ch/page38842.html
16. L. Goldmann, F. De Simone, T. Ebrahimi. A comprehensive database and subjective evaluation methodology for quality of experience in stereoscopic video," in Proc. SPIE 3D Image Process. (3DIP) and Applicat., San Jose, CA, Jan. l7-2l, 20l0, vol. 7526.
17. J.-S. Lee et al. MMSP scalable video database. 20l0 [Online]. Avail- able: http://mmspg.epfl.ch/svd
18. J.-S. Lee, F. De Simone, T. Ebrahimi. Subjective quality eval- uation via paired comparison: Application to scalable video coding. IEEE Trans. Multimedia, vol. l3, no. 5, pp. 882-893, Oct. 20ll.
19. Y. Wang et al. Poly@NYU video quality databases. 2008 [Online]. Available: http://vision.poly.edu/index.html/index.php?n=Home-Page.QualityAssessmentDatabase
20. Y.-F. Ou, T. Liu, Z. Zhao, Z. Ma, Y. Wang. Modeling the impact of frame rate on perceptual quality of video // Proc. Int. Conf. Image Process. (ICIP), San Diego, CA, Oct. l2-l5, 2008.
21. Y.-F. Ou, Z. Ma, Y. Wang. A novel quality metric for compressed video considering both frame rate and quantization artifacts // Proc. Int. Workshop Video Process. Quality Metrics (VPQM), Scottsdale, AZ, Jan. 15-16, 2009.
22. Y.-F. Ou, Y. Zhou, Y. Wang. Perceptual quality of video with frame rate variation: A subjective study // Proc. Int. Conf. Acoust., Speech, Signal Process. (ICASSP), Dallas, TX, Mar. 14-19, 2010.
23. X. Feng, T. Liu, D. Yang, Y. Wang. Saliency based objective quality assessment of decoded video affected by packet losses // Proc. Int. Conf. Image Process. (ICIP), San Diego, CA, Oct. 12-15, 2008.
24. VQEG FR-TV Phase I database. Video Quality Experts Group (VQEG), 2000 [Online]. Available: ftp://ftp.crc.ca/crc/vqeg/TestSe- quences/
25. Final report from the Video Quality Experts Group on the validation of objective models of video quality assessment. VQEG, Apr. 2000 [Online]. Available: http://www.vqeg.org/
26. Report on the validation of video quality models for high def- inition video content. VQEG, Jun. 2010 [Online]. Available: http://www.vqeg.org/
27. The Consumer Digital Video Library. CDVL, 2010 [Online]. Avail- able: http://www.cdvl.org/
28. R.R. Ramachandra Rao, S. Goring, W. Robitza, B. Feiten, A. Raake. AVT-VQDB-UHD-1: A large scale video quality database for UHD-1 // Proc. IEEE Int. Symp. Multimedia (ISM), Dec. 2019, pp. 1-8. [Online]. Available: https://ieeexplore.ieee.org/document/8959059
29. Fan Zhang, Felix Mercer Moss, Roland Baddeley, David, R. Bull. BVI-HD: A Video Quality Database for HEVC Compressed and Texture Synthesised Content // IEEE Trans. on Multimedia, 2018.
30. A study of subjective video quality at various frame rates. Mackin, A. and Zhang, F. and Bull, D., Image Processing (ICIP), 2015 22nd IEEE International Conference on, 2015.
31. V. Hosu et al. The Konstanz natural video database (KoNViD-1 k). 2017 Ninth International Conference on Quality of Multimedia Experience (QoMEX), 2017, pp. 1-6, doi: I0.II09/QoMEX.20I7.7965673.
32. P.C. Madhusudana, X.Yu, N. Birkbeck, Y. Wang, B. Adsumilli, A. C. Bovik. Subjective and Objective Quality Assessment of High Frame Rate Videos // submitted to IEEE Transactions on Image Processing, 2020 [paper]
33. P.C. Madhusudana, N. Birkbeck, Y. Wang, B. Adsumilli, A.C. Bovik. Capturing Video Frame Rate Variations through Entropic Differencing // arXiv preprint arXiv:2006.ll424, 2020.
34. X.Yu, N. Birkbeck, Y. Wang, C. G. Bampis, B. Adsumilli, A. C. Bovik. Predicting the Quality of Compressed Videos with Pre-Existing Distortions // submitted to IEEE Transactions on Circuits and Systems for Video Technology. [paper]
35. A.K. Moorthy, L.K. Choi, A.C. Bovik, G. deVeciana. Video Quality Assessment on Mobile Devices: Subjective, Behavioral and Objective Studies // IEEE Journal of Selected Topics in Signal Processing, to appear in October 2012.
36. A.K. Moorthy, L.K. Choi, G. deVeciana, A.C. Bovik. Mobile Video Quality Assessment Database. IEEE ICC Workshop on Realizing Advanced Video Optimized Wireless Networks, Ottawa, Canada, June 10-15, 2012.
37. A.K. Moorthy, L.K. Choi, G. deVeciana, A.C. Bovik. Subjective Analysis of Video Quality on Mobile Devices // Sixth International Workshop on Video Processing and Quality Metrics for Consumer Electronics (VPQM) (invited article), Scottsdale, Arizona, January 15-16, 2012.
38. D. Lee, S. Paul, C. G. Bampis, H. Ko, J. Kim, S. Jeong, B. Homan, A. C. Bovik. A Subjective and Objective Study of Space-Time Subsampled Video Quality // submitted to IEEE Transactions on Image Processing. [paper]
39. C.G. Bampis, Z. Li, A.K. Moorthy, I. Katsavounidis, A. Aaron, A.C. Bovik. Study of Temporal Effects on Subjective Video Quality of Experience // IEEE Trans. Image Process., vol. 26, no. 11, pp. 5217-5231, 2017.
40. C.G. Bampis, Z. Li, A.K. Moorthy, I. Katsavounidis, A. Aaron, A.C. Bovik. LIVE Netflix Video Quality of Experience Database // Online: http://live.ece.utexas.edu/research/LIVE_NFLXStudy/index.html, 2016.
41. Z. Sinno, A.C. Bovik. Large-Scale Study of Perceptual Video Quality // IEEE Transactions on Image Processing, vol. 28, no. 2, pp. 612-627, February
2019.
42. Z. Sinno, A.C. Bovik. Large Scale Subjective Video Quality Study // 2018 IEEE International Conference on Image Processing, Athens, Greece, October 2018.
43. Z. Sinno, A.C. Bovik. LIVE Video Quality Challenge Database. Online: http://live.ece.utexas.edu/research/LIVEVQC/index.html, 2018.
44. J.Y. Lin, R. Song, T. Liu, H. Wang and C.-C. J. Kuo. MCL-V: A streaming video quality assessment database // J. Vis. Commun. Image Represent., vol. 30, pp. 1-9, Jul. 2015.
45. MCL-JCV Dataset. [online] Available at: http://mcl.usc.edu/mcl- jcv-dataset/.
46. C. Keimel, J. Habigt, T. Habigt, M. Rothbucher, K. Diepold. Visual quality of current coding technologies at high definition IPTV bitrates // Multimedia Signal Processing (MMSP), 2010 IEEE International Workshop on, 2010, pp. 390-393.
47. H. Wang et all. VideoSet: A large-scale compressed video quality dataset based on JND measurement // J. Vis. Commun. Image Represent., vol. 46, pp. 292-302, Jul. 2017.
48. Z. Ying, M. Mandal, D. Ghadiyaram, A.C. Bovik. Patch-VQ: 'Patching Up' the Video Quality Problem, arXiv 2020. [paper]
49. Z. Ying, M. Mandal, D. Ghadiyaram, A.C. Bovik. LIVE Large-Scale Social Video Quality (LSVQ) Database. Online:https://github.com/baidut/PatchVQ,
2020.
50. C.G. Bampis, Z.Li, I. Katsavounidis, TY Huang, C. Ekanadham, A.C. Bovik. Towards Perceptually Optimized End-to-end Adaptive Video Streaming // submitted to IEEE Transactions on Image Processing
51. S. Winkler. Image and video quality resources, 2012 [Online]. Avail- able: http://stefan.winkler.net/resources.html
52. Y. Wang, S. Inguva, B. Adsumilli. YouTube UGC dataset for video compression research // Proc. IEEE 21st Int. Workshop Multimedia Signal Process. (MMSP), pp. 1-5, Sep. 2019.
53. P. Mohammadu, A. Ebrahimi-Moghadam, S. Shirani. Subjective and Objective Quality Assessment of Image: A Survey // Majlesi Journal of Electrical Engineering, vol.9(l), Mar. 2015, pp. 55-83.
54. H. Wang et al. MCL-JCV: A JND-based H.264/AVC video quality assessment dataset // Proc. IEEE Int. Conf. Image Process. (ICIP), pp. 1509-1513, Sep. 2016.
55. BT-500-ll: Methodology for the Subjective Assessment of the Quality of Television Pictures, 2012.
56. A. Mozhaeva, A. Potashnikov, I. Vlasuyk, L. Streeter. Constant Subjective Quality Database: The Research and Device of Generating Video Sequences of Constant Quality // 2021 International Conference on Engineering Management of Communication and Technology (EMCTECH), 2021, pp. 1-5, doi: l0.ll09/EMCTECH53459.202l.96l8977.
