Realistic 3D audio can greatly enhance the sense of presence in a virtual environment. This paper discusses an application 3D audio by the mean of Binaural Spatialization within a larger 3D immersive system, with state of art capturing and rendering modules for visual data. We introduce a framework for capturing, transmitting and rendering 3D audio in such an environment. The framework allows realtime rendering of audio based on the positions of objects in the scene, including animated avatars and on the fly reconstructed humans. We provide a general overview of the application, how audio is integrated in the whole system, and how it can exploit the positions of the objects and the knowledge of room geometry already to render realistic reverberations using HRTFs (head related transfer functions). The network streaming modules used to achieve lip-synchronization, high-quality audio frame reception, are taken into account highlighting how large computational and networking challenges can be addressed efficiently. The proposed architecture as well as the actual implementation represent a first step in adequate networking support for Binaural 3D Audio, useful for tele-presence.