Don't like ABX testing? How about A-B testing?

[Guest]

Some claim ABX testing 'proves' equipment sounds the same. Others reject the methodology as meaningless as differences are not obvious on all music. I really don't want to get into that discussion again, it has been done too many times.

How about A-B or B-A testing?

It must be possible to, fairly accurately, re-digitize the sound or analogue signal coming from speakers/amplifiers/DACs etc. If that re-digitized signal is subtractively compared with an identical recording from rival equipment (A-B or B-A), we would end up with silence OR sound. If we get silence it proves there is no difference. If we get sound, that is the extra output one audio system is providing. It may of course be desirable extra output or annoying extra output! We would need to listen to it to decide if we wanted it or not.

Does anyone have any experience of such a comparison? Is there a possibility that this could be done in our own homes or is a vast amount of equipment needed?

[Ian]

Some claim ABX testing 'proves' equipment sounds the same. Others reject the methodology as meaningless as differences are not obvious on all music. I really don't want to get into that discussion again, it has been done too many times.

How about A-B or B-A testing?

It must be possible to, fairly accurately, re-digitize the sound or analogue signal coming from speakers/amplifiers/DACs etc. If that re-digitized signal is subtractively compared with an identical recording from rival equipment (A-B or B-A), we would end up with silence OR sound. If we get silence it proves there is no difference. If we get sound, that is the extra output one audio system is providing. It may of course be desirable extra output or annoying extra output! We would need to listen to it to decide if we wanted it or not.

Does anyone have any experience of such a comparison? Is there a possibility that this could be done in our own homes or is a vast amount of equipment needed?

Not sure what you’re getting at here - possibly one of these possibilities?:

There is software that can take a sample of a recording (typically background hiss, hum etc.) and then tries to remove those elements from the rest of the recording, thus supposedly leaving the music without the hiss and hum. However, this method of subtracting part of the audio only really works when you are trying to remove a continuous unchanging sound. It would be nowhere near precise enough to sample even a single event from a piece of music with the aim of using that sample to subtract the original audio of the same event.

In theory you can cancel out a sound by combining it with an out of phase version of itself. But this only works with a sine wave (i.e. a single frequency) If you combine a recording with itself slightly delayed so that a predominant frequency is out of phase - it will just sound horrible.

[MrGongGong]

In theory you can cancel out a sound by combining it with an out of phase version of itself. But this only works with a sine wave (i.e. a single frequency) .

Actually phase cancellation works (not always very well) with non sine wave audio as well.

[Jasmine Bassett]

The technique works well in a purely digital world and I've heard interesting demonstrations of what is "lost" from lossy codecs such as MP3 by subtracting the reconstituted PCM bitstream from the original file. This only really works when the original and coded/decoded versions are matched sample for sample.

The technique is extremely sensitive and I think from a more general perspective, starting from two analogue signals, the requirement to VERY accurately match the two levels and VERY accurately time align the two files before subtracting one from the other would preclude any definitive results. Interpreting any audible results would be even more of a minefield than the interpretation of ABX results.

[David-G]

I someone could explain what ABX means in this context that would be very helpful.

[Ian]

ABX testing is were you are invited to indicate whether X is a sample from A or B. The idea being that if you do it often enough it can give some indication whether there is a discernible difference between A and B.

What I think the OP is referring to here, however, is an objective process by which 2 versions of the same audio (one that has possibly been modified in some way) can be compared by ‘subtracting‘ one from the other to see if anything remains. For example, if you were to subtract a piece of audio from an identical clone nothing should remain.

[Nevalti]

ABX testing is were you are invited to indicate whether X is a sample from A or B. The idea being that if you do it often enough it can give some indication whether there is a discernible difference between A and B.

What I think the OP is referring to here, however, is an objective process by which 2 versions of the same audio (one that has possibly been modified in some way) can be compared by ‘subtracting‘ one from the other to see if anything remains. For example, if you were to subtract a piece of audio from an identical clone nothing should remain.

Thanks for using the word 'objective'. That is indeed what I mean. My hope is that we could objectively compare pieces of equipment rather than just forming a subjective opinion.

Provided we start with the same digital source I see no obvious reason why a fair comparison could not be made, even with complex music. Take two DACs, as some people claim there is no difference, re-digitize the analogue signal from the two competing DACs using the same ADC and will we get precisely the same result and therefore a null signal or will we see and hear a difference because there IS a difference?


To those who mentioned phase cancellation, such as we get in noise cancelling headphones, that is presumably achieved in the analogue domain (?) but the same result could be achieved digitally by a subtractive process so that the signal which remains is unsullied. My hope is to compare two pieces of 'competing' equipment, not to cancel the sound from one piece.

Plenty of software, even free software like 'Audacity', can manipulate the signal and remove selected noises - hum, clicks, etc. but I am not skilled enough to try to set up a detailed comparison.

[MrGongGong]

To those who mentioned phase cancellation, such as we get in noise cancelling headphones, that is presumably achieved in the analogue domain (?).

Not if you are working with digital files

[OldTechie]

Liberty Instruments (http://www.libinst.com/ ) has free software to create difference files.

Have a look at their 2008 AES paper about digital differencing http://www.libinst.com/AES%20Audio%2...ng%20Paper.pdf which explains some of the problems with the approach. It also reminds us that ABX testing never establishes that two signals are indistinguishable from each other. It can only establish that two signals are audibly different with a degree of certainty.

[Nevalti]

Liberty Instruments (http://www.libinst.com/ ) has free software to create difference files.

Have a look at their 2008 AES paper about digital differencing http://www.libinst.com/AES%20Audio%2...ng%20Paper.pdf which explains some of the problems with the approach. It also reminds us that ABX testing never establishes that two signals are indistinguishable from each other. It can only establish that two signals are audibly different with a degree of certainty.

Aha! Thank you very much. So, if I understand correctly, what I am seeking, what I called A-B or B-A, is actually called a 'difference file' and people have been creating them for years for comparison purposes. It is strange that they are not better known, one would have thought it provided a conclusive answer to the age old question - 'Is there any difference or not?'

Do you have any practical experience of using them? If so, do you think I can achieve my aim with that free software? It will be a few days before I can spend the time to examine the paper and the software in full detail but on a quick look through it seems to do most of what I wanted. The paper suggests that mic recordings from a speaker are not practical - but I haven't understood why yet.

[Nevalti]

Purely by coincidence, I have today stumbled over another reference to the same software - http://theaudiocritic.com/plog/index...Id=48&blogId=1

Interestingly, the difference file heard in that particular comparison was actually 'music' rather than 'noise' even though the theoretical difference between the components under test was negligible. Despite the negligible theoretical difference between the components, some reviews have claimed to hear a significant difference between those components. This must be very confusing for the theoreticians who claim that we can't hear differences