Users who purchased the stereo headset adapter for Xbox One are experiencing problems, whether using the official Microsoft headset or other third-party devices. Xbox.com forums feature a four page thread regarding the problem. No current fix is offered.
Microsoft did release a controller firmware update (certainly a console first) to bring in stereo headset compatibility. The process is only a matter of connecting a micro USB cable to the controller and accessing the system menu to select, “update controller.” This process then autocompletes, and conveniently asks for any additional controllers.
This firmware situation could likely be utilized to fix the issue of scratchy audio once Microsoft finds the solution. Hopefully it’s not a physical defect in the adapter itself. Note this would seem to affect all headsets, not only certain brands.