Project: Audio phones

Goal: Submit to MobiSys 2010 (http://www.sigmobile.org/mobisys/2010/)

Important dates: 

Abstracts due: December 4th, 2009. 23:59 EST

          Full papers due: December 11th, 2009. 23:59 EST

          Notification of acceptance: March 3rd, 2010

Project description:

A user needs to have a computer with speaker and microphone, and an audio phone.

Some applications:

     Laptops access Internet with audio phones sending packets back and forth.

               Setup a secure P2P connection for users behind NAT, by use of audio phones (Secure NAT traversal through audio phones for P2P Apps).

       

Timeline (subject to change):

1. Sep. 21 - Oct 5: get familiar with the background
2. Oct. 6 - Oct. 12: test basic framework
3. Oct.13 - Oct. 19: coding and decoding of audio information (IP, port)
4. Oct. 20 - Oct.  26: noise control, redundancy coding, etc
5. Oct .27 - Nov. 2: Finish up implementation
6. Nov. 3 - Nov. 30: Collect implementation results and paper writing.
7. Nov. 24 - Nov. 29: Come out of the basic scheme (can send various files in quiet environment, consider noise and error control if time permits); get port numbers behind NAT. 
8. Nov. 29 - Dec. 1 : Noise and error control
9. Dec.1 - Dec. 4: get the applications working: NAT and internect access.  
10. Dec.4 - Dec. 11: Wrapup and paper writing.

Weekly meetings and progress: 

Nov. 29: our next group meeting will be 3PM in 7th floor conference room .

• Meeting minutes
  • Analyzed/examined different frequency ranges with our application and how accurate we can decode it. We tried 8 frequencies and we are able to decode them easily with 1 second intervals. When the interval gets larger, the ranges overlap and consequently the error rate will be higher.
  • We discussed different applications of the project. It seems that it is more suited toward transmitting small amount of data because of our low data rate.
• Progress made:
  • Fatemah:
    • Implemented the function PointToHz to get the actual frequencies from the relative numbers that the FFT function gives to us.
    • Implemented the basic mechanism to get the actual bit that got transmitted based on the frequency range. The code works as follows: we constantly monitor the frequencies that we get in ranges of time. For example in each 250 miliseconds. We use the frequency range that is repeated most frequently in that time to decide what the actual bit was. Yadi had a good idea of using percentage of frequencies recieved in each range to make the decision since now we have more that 2 frequency ranges.

Nov. 24- Nov. 29

• Next steps:
  •  AudioCapture gets diffrent ranges for inputs with difference frequencies. We need to figure out whether can we get actual frequency ranges rather than ranges like 90, 180. (Fatemah will work on this)
  •  How to increase data rate? (Ashish will work on this. Fatemah should join Ashish)
    • Now we only use 2 frequencies. We need to find out the range of frequency we can use, and divide it into bands. The width of each band depends on the experimental results (it may be different with and without noise).   Make sure each band is wide enough to avoid overlapping.
    • The length of each signal is set to 100ms, try to reduce it and see how it works with different laptops.
    • Try to send various files through the basic scheme and see how it works, what is the ber (bit error rate), what is the performance with existence of noise, etc.Try to send various files through the basic scheme to see how it works, what is the ber (bit error rate), what is the performance with the existence of noise, etc.
  • The NAT application. We need to figure out how to punch the hole to get port numbers to setup P2P connection. Probe some number of ports seems to be applicable. But need to be carefully about upper and lower limits. (Jordan will work on this. Yadi may look for other alternatives)
  • Error control. When there is noise, we may need to retransmit. We may break the original data into chunks, and send checksum at some interval. Need to do some research on how to do this. (Yadi will work on this)
    • find java FEC (forward error correction) library: http://freshmeat.net/projects/javafec/, and an example of using it: http://jamesclarke.info/notes/javafec
  • Check the correctness of current code. (All of us could work on this)
  • Start writing the paper. Should start writing the basic ideas, and provide some guidance for further work (Yadi will start writing)
• TODO:
  • We may need a seperate fft decoder thread and/or multiple buffers, if in current scheme the capture thread can not accomodate the incoming sound data and starts to drop.   
  • Get a clear idea about the other application (internet access).

Week of Nov. 15- Nov. 21

• Meetings on Nov. 15 and Nov. 18: Continue working on the decoding process and understanding the Fourier transform mechanism.
• Progress made:
  • Fatemah: Trying to incorporate the audiocapture and the DFT to get the frequency. 

Week of Nov. 9- Nov. 15

• Meeting on Nov. 9
  • Change in direction: NAT issue is only one application of what we plan to do. We are focusing on communiation of files and infomation by sound through phones without access to internet.
• Meeting on Nov. 11
  • Fatemah and Jordan met, analyzed the frequency analyzer software, and got an idea of where we can extract the frequency from. This is in the Views class and in the ViewUpdate function.
• Progress Made:
  • Fatemah: Continue working on the decoding section.

Week of Nov. 2- Nov. 8

• Group meeting minutes:
  • Action item: Jordan to follow up on the Up&p
  • Action item: Fatemah to continue implementing the decoding algorithm.
  • Action item: Yadi to look at decoding if got a chance.
  • Fatemah suggested having more frequent meetings from next week.
  • Additional meetings will be held Fridays at 1:00 PM.
• Progress made:
  • Fatemah: Found an opensource frequency analyzer that performs what we want. It is written in C++. It gives us the actual frequency in real time. 
    • See http://www.relisoft.com/freeware/freq.html
    • Tested the code with our current sound generator. It is able to detect different frequencies.
    • Looked into code and tried to customize for our need. This should be able to give us what we need.
    • I suggest looking into it as a group to progress faster.  

Week of Oct. 26- Nov. 2

• Group meeting minutes:
  • Action item: Fatemah to implement the decoding algorithm.
  • Action item: Jordan to follow up on the Up&p
• Progress made:
  • Fatemah: Implemented the algorithm in http://jvalentino2.tripod.com/dft/index.html
    • This algorithm gives us a list of frequencies that was seen in the sample sound and their amplitudes based on the Descrete Fourier Transform algorithm. It is mainly used when they want to eliminate noises from the sound (very high or very low amplitudes). However, what we want is to have frequency at a given time.
    • Next step: Get frequency at a given time.

Week of Oct. 20- Oct. 26

• Group meeting minutes:
  • Action item: All to search for how to decode that is generated.
  • Action item: All to search for a way to find the port number assigned to the peer once attempting to make a connection to the initiator.
    • The issue is that when the connection gets refused the socket object is discarded, therefore we are not able to get the port number from this object.
• Progress made:
  • Fatemah: look into how do the decoding and finding the assigned port.
    • Found the algorithm that should be implemented. See the details in: http://jvalentino2.tripod.com/dft/index.html
    • Explanation: You need to break the audio file into it's component samples, and once you have an array of samples, you need to perform a FFT (Fast Fourier Transform : http://en.wikipedia.org/wiki/Fast_fourier_transform ) on it to break the samples down into their respective frequencies.
    • Also useful: http://www.jsresources.org/faq_audio.html#samples_organized, http://www.jsresources.org/faq_audio.html#max_volume
    • Next step: Implementation. Hopefully will be straight forward having this code.
    • Finding the assigned port by the router is not trivial. I do not think the getlocalport() function is giving us the private port that is assigned to us by NAT, rather it is the public port. I looked at few languages socket object and none seem to support this.
  • Jordan: NAT port problem
    • Like Fatemah says, this port is not easy to find.  I am not sure of the best way to go at this point.  getlocalport() does seem to be the right way to go, but only works when a connection can be set up.  The way the Java socket works is it gets local IP, local Port, remote IP, remote Port.  So we might need to try some port prediction algorithm or something.  Also a possibility is bypassing the socket semantics and trying to assemble just a syn packet and reading the errors.

Week of Oct. 12- Oct. 19

• Group meeting minutes:
  • Prilim. algorithm for encoding/decoding using phone: Once the peers call each other, one can have its phone near the speaker to get the encoded IP/Port from the program that we write and send to the other peer. The other peer will have its phone near the microphone to transfer the sound to our program to decode it.
  • Decision: we will use different sound frequencies to encode the IP and Port. For now, we can do a binary translation with one high and one low frequency.
  • Later we can discuss to refine this algorithm.
  • Action items:
    • Jordan: investigate if we can get the port that NAT assigns us from the socket object in Java.
    • All: to work on encoding/decoding the data using sounds. Java has some libraries for sounds.
• Progress made:
  • Yadi found some useful info and sample code about how to capture sound using Java
    • http://www.developer.com/java/other/article.php/1572251/Java-Sound-Getting-Started-Part-1-Playback.htm
    • http://www.developer.com/java/other/article.php/1579071/Java-Sound-Getting-Started-Part-2-Capture-Using-Specified-Mixer.htm
    • http://www.developer.com/java/other/article.php/1565671/Java-Sound-An-Introduction.htm#Discussion%20and%20Sample%20Programs
  • Yadi found a good java sound programmer guide.
    • http://java.sun.com/j2se/1.5.0/docs/guide/sound/programmer_guide/contents.html
  • Fatemah: Wrote a program that converts the port/IP to binary format and generates the corresponding sound. 

    Case '0' Low Frequency

    Case '1' High Frequency

    Case '.' Medium Frequency 

    The frequency, interval for each character to be played, and volume are all parameters that can be changed. Next step: decoding the sound that is generated.

 Week of Oct. 5- Oct. 11

• Group meeting minutes:
  • Basic algorithm: Each peer knows his own public IP address and can give it to the other peer. After that they will each send a message to the others IP address in order to be assigned a port. They can then exchange their port numbers.
  • We assume that peers have some contact method (phone,..)
  • Action items:
    • All group members to search about the best language to use that supports sockets, and is also portable in different platforms
    • Fatemah to write a simple program for the discussed algorithm.
    • All group to look at UP&P and alternate implementations.
    • Yadi to follow up on getting some routers.

• Progress made: 
  • Yadi
    • has set up a wiki page
    • has set up a svn repository for code sharing
    • ...
  • Fatemah
    • Search about programming languages. It seems that Python and Java both have the convinient level of abstraction for socket programming and portability that we would like for our project. For more information please see: http://www.scriptol.com/programming/choose.php, http://catb.org/esr/writings/taoup/html/ch17s05.html
    • Implemented a Java program to get the global and local IP address of a machine behind NAT. It also can get internal gateway of the router.
      • In order to get the global IP address we need to call and external entity such as whatismyip.com to tell us our IP.
      • The program also sends a packet to the peer's IP address in order to punch a hole (which does not succeed because the peer is not waiting for us). However, it is tricky to get the port address that the NAT assigns to us. We cannot use a middle server to tell us the port because we want the port assigned to us while we try to connect to "the other peer's IP address". This is the port from which the peer will connect to us. More investigation underway.
  • Jordan
    • (please post your contributions here)

Week of Sep. 28 - Oct. 4:

• Group meeting canceled.
• Progress made:
  • Fatemah 
    • Read some papers about NAT.
    • Search and found more papers about NAT to be read.
    • Found source code for hole punching: NUTTS See http://nutss.gforge.cis.cornell.edu//

Week of Sep. 21-27:

• Group meeting minutes:
  • We need to set up NAT
  • How about virtual NATs?
  • Think about format of the message. We can go with simple text at first. However, at the end we will need some encryption mechanism. Later we can work  on tunes. Morse code is not suitable because it will be very long for the IP address and port.
  • What we want to be able to send over is basically the IP and port number in order to bypass the mediator's role in getting those and informing the peers.
  • Later on we might want to try different NAT set ups with our approach.   
• Progress made: 
  • Fatemah: Searched and found general information about NAT and Morse code for transmission of IP and Port. We will not be using Morse code since the the message will be long encoding with it.