In signal processing, data compression , source encoding , or bit-rate reduction involves encoding information using fewer bits than original representation. Compression can be lossy or lossless. Lossless compression reduces bits by identifying and eliminating statistical redundancy. No information is lost in lossless compression. Lossy compression reduces bits by removing unnecessary or less important information.
The process of reducing the size of data files is often referred to as data compression. In the context of data transmission, this is called source encoding; encoding is done on the data source before it is stored or sent. Source encoding should not be confused with channel coding, for error detection and correction or line coding, a means for mapping data to signals.
Compression is useful because it reduces the resources needed to store and send data. Computing resources are consumed in the compression process and, usually, in reversing processes (decompression). Data compression is subject to the complexity of space-time complexity. For example, a video compression scheme may require expensive hardware to quickly decompress video for viewing when it is being decompressed, and the option to decompress the full video before viewing it may be uncomfortable or require additional storage. The design of the data compression scheme involves trade-offs among various factors, including the level of compression, the amount of distortion introduced (when using lossy data compression), and the computing resources required to compress and decompress data.
Video Data compression
Lossless
Losingless data compression algorithms typically exploit statistical redundancy to represent data without losing any information, so the process can be reversed. Losingless compression is possible because most real-world data shows statistical redundancy. For example, an image may have an unchanged color area for several pixels; instead of encoding "red pixels, red pixels,..." data can be encoded as "279 red pixels". This is a basic example of run-length encoding; there are many schemes to reduce file size by eliminating redundancy.
The Lempel-Ziv (LZ) compression method is one of the most popular algorithms for storage without loss. DEFLATE is a variation on LZ that is optimized for decompression speed and compression ratio, but compression can be slow. DEFLATE is used in PKZIP, Gzip, and PNG. In the mid-1980s, after work by Terry Welch, the LZW (Lempel-Ziv-Welch) algorithm quickly became the preferred method for most general-purpose compression systems. LZW is used in GIF images, programs like PKZIP, and hardware such as modems. The LZ method uses a table-based compression model in which table entries are replaced for repetitive data strings. For most LZ methods, this table is dynamically generated from the previous data in the input. The table itself is often encoded by Huffman (eg SHRI, LZX). Current LZ-based encoding schemes that perform well are Brotli and LZX. LZX is used in Microsoft CAB format.
The best lossless modern compressors use probabilistic models, such as predictions with partial matching. The Burrows-Wheeler transform can also be seen as an indirect form of statistical modeling.
Grammar-based code classes gain popularity because they can compress highly repetitive very effective inputs, for example, biological data sets of the same or closely related species, large-version documents, internet archives, etc. The basic task of grammar-based code is to build a context-free grammar that generates a single string. Sequitur and Re-Pair are practical grammar compression algorithms for publicly available software.
In further refinements of the direct use of probabilistic modeling, statistical estimates can be coupled to an algorithm called arithmetic coding. Arithmetic coding is a more modern coding technique that uses mathematical calculations from state-to-machine to generate string bits encoded from a series of input data symbols. This can achieve superior compression for other techniques such as the more well known Huffman algorithm. It uses an internal memory state to avoid the need to do one-to-one mapping of individual input symbols for different representations that use a number of integer bits, and it clears the internal memory only after encoding the entire string of data symbols. Arithmetic coding applies very well to adaptive data compression tasks where statistics vary and context-dependent, as they can easily be combined with an adaptive model of the probability distribution of input data. The earliest example of using arithmetic coding is its use as an optional (but not widely used) feature of the JPEG image encoding standard. Since then it has been applied in various other designs including H.263, H.264/MPEG-4 AVC and HEVC for video encoding.
Maps Data compression
Lossy
Lossy data compression is the opposite of lossless data compression. In the late 1980s, digital images became more common, and standards for compressing them appeared. In the early 1990s, lossy compression methods began to be widely used. In this scheme, some loss of information is acceptable. Dropping non-essential details from a data source can save storage space. The lossy data compression scheme is designed by research on how people perceive data in question. For example, the human eye is more sensitive to subtle variations in lighting compared to color variations. JPEG image compression works in part by rounding bits of unimportant information. There is a similar trade off between preserving information and reducing size. A number of popular compression formats exploit these different perceptions, including those used in music, image, and video files.
Weak image compression can be used in digital cameras, to increase storage capacity with reduced image quality. Similarly, DVDs use the lossy MPEG-2 video encoding format for video compression.
In lossy audio compression, the psychoacoustics method is used to remove unheard (or inaudible) audio components. Human speech compression is often done with more specialized techniques; speech coding, or voice coding, is sometimes distinguished as a separate discipline of audio compression . Different audio and speech compression standards are included in the audio encoding format. Voice compression is used in internet telephony, for example, audio compression is used to copy CDs and decoded by an audio player.
Theory
The theoretical background of compression is provided by information theory (which is closely related to algorithmic information theory) to lossless compression and distortion level theory for lossy compression. These fields of study were essentially invented by Claude Shannon, who published fundamental papers on this topic in the late 1940s and early 1950s. The coding theory is also associated with this. The idea of ​​data compression is also strongly associated with statistical inference.
Machine learning
There is a close relationship between machine learning and compression: a system that predicts the posterior probability of a given sequence of its entire history can be used for optimal data compression (by using arithmetic encoding in the output distribution) while the optimal compressor can be used for prediction (by finding symbols which solidifies best, given previous history). This equality has been used as a justification for using data compression as a benchmark for "general intelligence."
Show space vectors
However a new alternative view may show the compression algorithm implicitly mapping a string into an implicit feature space vector, and compression-based comparative measurements calculate similarities in the space of this feature. For each compressor C (.) We define the corresponding vector space, so C (.) Maps the input string x, according to the vector norm || ~ x ||. A complete examination of the space features underlying all compression algorithms is blocked by space; Instead, feature vectors opt to examine three representative lossless compression methods, LZW, LZ77, and PPM.
Data discrepancies
Data compression can be seen as a special case of data difference: The data difference consists of generating the difference provided source and the target , with the patching generating target is given the source and difference, while data compression consists of generating a compressed file that is given the target, and the decompression consists of generating a target given only a compressed file. Thus, one can consider data compression as different data with empty source data, compressed files corresponding to "difference from none." This is the same as considering the absolute entropy (according to data compression) as a special case of relative entropy (according to the data difference) without the initial data.
When someone wants to emphasize the connection, one can use the term differential compression to refer to the data difference.
Usage
Audio
Audio data compression, not to be confused with dynamic range compression, has the potential to reduce transmission bandwidth and audio data storage requirements. Audio compression algorithm is implemented in software as audio codec. Lossy compression audio algorithms provide higher compression with the cost of fidelity and are used in a variety of audio applications. This algorithm is almost all dependent on psychoacoustics to eliminate or reduce the loudness of the sound that is less audible, thus reducing the space needed to store or send it.
In lossy and lossless compression, information redundancy is reduced, using methods such as coding, pattern recognition, and linear prediction to reduce the amount of information used to represent uncompressed data.
The acceptable exchange between loss of audio quality and transmission or storage size depends on the application. For example, a 640MB (CD) compact disc holds about an hour of uncompressed high fidelity music, less than 2 hours of lossless compressed music, or 7 hours of compressed music in MP3 format with moderate bit rates. The digital voice recorder can usually store about 200 hours of clear speech understood in 640 MB.
Lossless audio compression produces a digital data representation that decompresses to precise digital duplicates of the original audio stream, unlike the playback of lossy compression techniques such as Vorbis and MP3. The compression ratio is about 50-60% of the original size, which is similar to that used for generic lossless data compression. Lossless compression can not achieve a high compression ratio because of the complexity of waveforms and rapid changes in the form of sound. Codecs such as FLAC, Shorten, and TTA use linear predictions to estimate the signal spectrum. Many of these algorithms use convolution with filters [-1 1] to slightly whiten or flatten the spectrum, thus allowing traditional lossless compression to work more efficiently. The process is reversed after decompression.
When an audio file is processed, either with further compression or for editing, it is desirable to work from an unchanged original (uncompressed or uncompressed). The processing of files that are accidentally compressed for some purposes usually results in lower end results than creating the same compressed file from an uncompressed original. In addition to sound editing or mixing, lossless audio compression is often used for archive storage, or as a master copy.
A number of lossless audio compression formats exist. Shorten is the initial lossless format. The newer ones include Free Lossless Audio Codec (FLAC), Apple Lossless (ALAC), MPEG-4 ALS, Microsoft Windows Media Audio 9 Lossless (WMA Lossless), Audio Monkey, TTA, and WavPack. See the list of lossless codecs for the complete list.
Some audio formats feature a combination of lossy format and lossless correction; this allows stripping corrections to easily get lost files. These formats include MPEG-4 SLS (Scalable to Lossless), WavPack, and OptimFROG DualStream.
Other formats are associated with different systems, such as:
- Direct Stream Transfers, used in Super Audio CD
- Meridian Lossless Packing, used in DVD-Audio, Dolby TrueHD, Blu-ray, and HD DVD
Lossy audio compression
Lossy audio compression is used in various applications. In addition to the direct application (MP3 player or computer), the digital compressed audio stream is used in most video DVDs, digital television, streaming media on the internet, satellite and cable radios, and terrestrial radio broadcasts. Lossy compression usually achieves much greater compression than lossless compression (data 5 percent to 20 percent of the original stream, not 50 percent to 60 percent), by removing less critical data.
The innovation of lossy audio compression is to use psychoacoustics to recognize that not all data in the audio stream can be perceived by the human auditory system. Most lossy compression reduces perceptual redundancy by first identifying irrelevant sounds, which is a very difficult sound to hear. Common examples include high frequencies or sounds that occur at the same time with louder sounds. The voices are encoded with decreasing accuracy or none at all.
Due to the nature of lossy algorithms, audio quality suffers when files are decompressed and recompressed (digital generation loses). This makes lossy compression unsuitable for storing intermediate results in professional audio engineering applications, such as voice editing and multitrack recording. However, they are very popular with end users (especially MP3s) as megabytes can store music worth one minute with sufficient quality.
Encoding method
To determine what information in irrelevant audio signals, most of the lossy compression algorithms use transformations such as transformed cosmic discrete transforms (MDCT) to change the waveform of time domain samples into transformation domains. Once modified, usually into the frequency domain, component frequency can be allocated bits according to how they sounded. The audibility of the spectral component is calculated using the absolute threshold of the hearing and the principles of simultaneous masking - a phenomenon in which the signal is obscured by another signal separated by frequency - and, in some cases, temporal masking - in which the signal is obscured by another signal separated by time. The same loudness contour can also be used to weigh the perceptual interests of the components. The combination model of the human ear-brain that combines such effects is often called the psychoacoustic model.
Another type of lossy compressor, such as linear predictive coding (LPC) used with speech, is source-based coders. These coders use sound generator models (such as human vocal channels with LPC) to whiten the audio signal (ie, flatten the spectrum) before quantization. LPC can be regarded as a basic perceptual coding technique: the reconstruction of an audio signal using a linear predictor shapes the quantization noise of the coder into the spectrum of the target signal, partially obscuring it.
Lossy format is often used for the distribution of audio streams or interactive applications (such as speech coding for digital transmission on mobile phone networks). In such applications, the data must be decompressed as a data stream, rather than after the entire data stream is transmitted. Not all audio codecs can be used for streaming applications, and for such applications, codecs designed to effectively stream data will usually be selected.
The latent result of the method used to encode and decode data. Some codecs will analyze longer data segments to optimize efficiency, and then encode them in ways that require larger data segments at a time to decode. (Often codecs create segments called "frames" to create discrete data segments for encoding and decoding.) The inherent latency of the encoding algorithm can be important; for example, when there is a two-way data transmission, such as with a telephone conversation, a significant delay can seriously degrade the perceived quality.
Unlike the compression speed, which is proportional to the number of operations required by the algorithm, here latency refers to the number of samples that must be analyzed before the audio blocks are processed. In the case of a minimum, latency is a zero sample (for example, if the coder/decoder simply reduces the number of bits used to measure the signal). Time domain algorithms such as LPC also often have low latency, hence their popularity in speech coding for the phone. In algorithms such as MP3, however, a large number of samples must be analyzed to implement the psychoacoustic model in the frequency domain, and the latency is on the order of 23 ms (46 ms for two-way communication)).
Greeting encoding
Speech encoding is an important category of audio data compression. The perceptual models used to predict what the ear can hear are generally quite different from those used for music. The range of frequencies required to convey sounds from human voices is usually much narrower than needed for music, and sound is usually less complex. As a result, speech can be encoded with high quality using a relatively low bit rate.
If the data to be compressed is analog (as the voltage varies with time), quantization is used to digitize it into numbers (usually integers). This is called an analog-to-digital conversion (A/D). If the integer generated by the quantization of each 8 bits, then the entire range of analog signals is divided into 256 intervals and all signal values ​​in an interval are quantized to the same number. If a 16-bit integer is generated, then the range of the analog signal is divided into 65,536 intervals.
This relationship describes a compromise between high resolution (large number of analog intervals) and high compression (small integers generated). This quantization application is used by some speech compression methods. This is achieved, in general, by some combination of two approaches:
- Only sound encoding can be made by a single human voice.
- Throw more data in the signal - save enough to reconstruct the sound "understandable" rather than the full frequency range of human hearing.
Perhaps the earliest algorithms used in speech coding (and audio data compression in general) are the A-law algorithm and the Âμ-law algorithm.
History
The literary compendium for various audio encoding systems was published in the IEEE Journal of Selected Area in Communication (JSAC), February 1988. Although there were several papers from before that time, this collection documented all the completed and functioning audio variations. coders, almost all use perceptual techniques (ie masking) and some types of frequency analysis and back-end voiced coding. Some of these papers comment on the difficulty of obtaining good, clean digital audio for research purposes. Most, if not most, of the authors in the JSAC edition are also active on the MPEG-1 Audio committee.
The world's first commercial audio automation compression compression system was developed by Oscar Bonello, a professor of engineering at the University of Buenos Aires. In 1983, using the psychoacoustic principle of masking critical bands first published in 1967, he began to develop practical applications based on a recently developed IBM PC computer, and a broadcast automation system was launched in 1987 under the name Audicom. Twenty years later, almost all radio stations in the world use similar technology produced by a number of companies.
Video
Video compression is a practical implementation of source encoding in information theory. In practice, most video codecs are used together with audio compression techniques to store separate but complementary data streams as a combined package using so-called container formats .
Uncompressed video requires a very high data rate. Although lossless video compression codecs work on a compression factor of 5 to 12, the typical MPEG-4 video compression has a compression factor of between 20 and 200.
Encoding theory
Video data can be represented as a series of still picture frames. Such data usually contain an abundance of spatial and temporal redundancy. The video compression algorithm attempts to reduce redundancy and store information more compactly.
Most video compression formats and codecs exploit spatial and temporal redundancy (eg via coding differences with motion compensation). Equations can be coded only by storing the difference between eg. temporally adjacent frames (inter-frame encodings) or spatially adjacent pixels (intra-frame code). Inter-frame compression (temporal delta encode) is one of the most powerful compression techniques. It (re) uses data from one or more earlier or later frames in sequence to describe the current frame. Intra-frame coding, on the other hand, only uses data from within the current frame, effectively being a still image compression. And the intra-frame encoding always uses a lossy compression algorithm.
The special format classes used in camcorders and video editing use a less complex compression scheme that limits their prediction techniques to intra-frame prediction.
Usually video compression also uses lossy compression techniques such as quantization that diminish aspects of data sources that (more or less) are irrelevant to human visual perception by exploiting features of human perception of perception. For example, a small difference in color is harder to see than a change in brightness. The compression algorithm can average colors in similar areas to reduce space, in a manner similar to that used in JPEG image compression. As in all lossy compression, there is a trade-off between video quality, compression and decompression processing costs, and system requirements. Highly compressed videos can display visible or annoying artifacts.
Other methods besides the usual DCT-based transformation formats, such as fractal compression, appropriate search and use of discrete wavelet transforms (DWT), have been the subject of some research, but are typically not used in practical products (except for the use of the wavelet code as a still image encoder without motion compensation). Interest in fractal compression seems to be on the wane, as recent theoretical analysis shows the lack of effectiveness of such methods.
Inter-frame encoding
The inter-frame coding works by comparing each frame in the video with the previous one. Each frame of the video sequence is compared from one frame to the next, and the video compression codec only sends the difference to the reference frame. If the frame contains an area where nothing is moving, the system can only issue a short command that copies a part of the previous frame to the next frame. If parts of the frame move in a simple way, the compressor can issue a (slightly longer) command that tells the decompressor to pan, rotate, lighten, or darken the copy. This longer command is still much shorter than the intraframe compression. Usually the encoder will also send a residual signal that represents a finer difference to the reference image. Using entropy coding, this residual signal has a more concise representation than the full signal. In the video area with more motion, compression must encode more data to keep up with the larger number of pixels that change. Generally during explosions, flames, flocks of animals, and in some panning shots, high frequency detail leads to a decrease in quality or increase in the variable bitrate.
Hybrid-based transformation format
Today, almost all commonly used video compression methods (for example, standards approved by ITU-T or ISO) share the same basic architecture that returned to H.261 standardized in 1988 by the ITU-T. They mostly rely on DCT, applied to neighboring pixel rectangle blocks, and temporal predictions using motion vectors, as well as currently also a loop filtering step.
In the prediction stage, various deduplication and difference-coding techniques are applied that help decorate data and describe new data based on data already sent.
Then a rectangular data block of pixels (residues) is transformed into the frequency domain to facilitate the targeting of irrelevant information in quantization and for some redundancy of spatial reduction. The widely used discrete cosine transformation (DCT) in this regard was introduced by N. Ahmed, T. Natarajan and K. R. Rao in 1974.
In the main lossy processing phase the data is quantified to reduce information irrelevant to human visual perception.
In the last stage, statistical redundancy is largely eliminated by entropy coders that often apply some form of arithmetic coding.
In the additional loop filtering stage, various filters can be applied to the reconstructed image signal. By calculating these filters also inside the encoding loop they can help compression as they can be applied to the reference material before being used in the prediction process and they can be guided using the original signal. The most popular example is a deblocking filter that obscures blocking artifacts from quantization discontinuities on block boundary changes.
History
All of the basic algorithms of today's dominant video codec architecture have been discovered before 1979. In 1950, Lab Bell filed a patent on DPCM which was immediately applied to video coding. Entropy coding began in the 1940s with the introduction of the Shannon-Fano coding that was widely used by Huffman coding developed in 1950; a more modern context-adaptive binary arithmetic encoding (CABAC) was published in the early 1990s. Transform coding (using the Hadamard transformation) was introduced in 1969, the popular discrete cosine transform (DCT) appeared in 1974 in the scientific literature. The ITU-T H.261 standard from 1988 introduced the basic architecture of video compression technology.
Genetics
Genetics compression algorithms are the latest generation of lossless algorithms that compress data (usually a nucleotide sequence) using both conventional compression algorithms and genetic algorithms that are tailored to specific data types. In 2012, a team of scientists from Johns Hopkins University published a genetic compression algorithm that did not use the reference genome for compression. HAPZIPPER is customized for HapMap data and achieves more than 20 times the compression (95% reduction in file size), providing 2- to 4 times better compression and in much faster time than any leading general-purpose compression utility. For this, Chanda, Elhaik, and Bader introduce MAF-based encoding (MAFE), which reduces the heterogeneity of the datasets by sorting SNPs by the frequency of their minor alleles, thus homogenizing the dataset. Other algorithms in 2009 and 2013 (DNAZip and GenomeZip) have a compression ratio of up to 1200 fold - allowing 6 billion basepair of the human genome to be stored in 2.5 megabytes (relative to reference genomes or averaged over many genomes). benchmark in genetics/genomics data compressors, see
Emulation
To emulate a CD-based console such as PlayStation 2, data compression is desired to reduce the large amount of disk space used by ISO. For example, Final Fantasy XII (USA) is normally 2.9 gigabytes. With the right compression, it is reduced to about 90% of that size.
Outlook and potentially unused current
It is estimated that the total amount of data stored on the world storage device can be further compressed with existing compression algorithms with a mean factor of 4.5: 1 remaining. It is estimated that the combined global technology capacity for storing information provides 1,300 exabytes of hardware digits in 2007 , but when the appropriate content is optimally compressed, it represents only 295 exabytes of Shannon information.
See also
References
External links
- Data Compression Basics (Video)
- Video compression 4: 2: 2 10-bit and its benefits
- Why is 10-bit saving bandwidth (even if the content is 8-bit)?
- Which compression technology should be used
- Wiley - Introduction to Compression Theory
- The EBU subjective listening test on the low bitrate audio codec
- Audio Archiving Guide: Music Format (A guide to help users choose the right codec)
- MPEG 1 & amp; 2 video compression intros (pdf format) in Wayback Machine (archived September 28, 2007)
- comparison of wiki hidrogenaudio
- Introduction to Data Compression by Guy E Blelloch from CMU
- Hail HD - 1080p Uncompressed source material for compression testing and research
- Explanation of the lossless signal compression method used by most codecs
- The interactive blind auditory test of audio codec over the internet
- TestVid - 2,000 HD and other uncompressed source video clips for compression testing
- Videsignline - Intro to Video Compression
- Data Trace Reduction Technology
- What is Run length Coding in video compression.
Source of the article : Wikipedia
0 Comments