Python software stack for an HTTP to FTP bridge

Question

To understand this request, I need to explain a bit of a background story.

Background information

We have a device that used to work over UDP and it was reasonably fast. One reason was that any side could initiate data transfer and another reason was that the data was present with less overhead. The decision was made that this device got a new firmware and now uses HTTP and XML as the protocol - and it's slow.

One reason is that the communication is initiated by the device only (polling) and if the server has information to send, it needs to wait until the device asks for data. The polling interval is fixed and does not intelligently adapt whether data was available or not.

The second reason IMHO is that the data is not transmitted to the device in the format it needs. Any discussions to implement our data transfer via HTTP in a different way were not accepted and rejected with the weirdest reasons.

My last chance to convince a few people is to implement a prototype of the necessary preprocessing myself. Here's my idea: I neither change the server implementation nor the device firmware. Instead, I connect a Raspberry Pi with 2 network adapters as a man in the middle which does a) a bit more sophisticated polling and b) converts the data from HTTP into the necessary format of the device and puts it directly into the correct files. For service purposes, the device supports SFTP, which I can enable.

I have calculated a ~50x faster data transfer for a) and a ~600x faster processing for b), so this seems worth a bit of a hobby project for the weekends.

Software stack

I have the Raspberry and I gained a bit of knowledge with Raspbian, so I'd like to use that as the OS. Next, I think Python was recommended to me a few times and I have taken the time to get a bit familiar with it, so this will be my first "real-world" Python project. I don't care about Python versions, in doubt I'd prefer the newer version 3.

Next, I'd need a few libraries to not reinvent the wheel:

• an HTTP client - to get data from the server
• an XML parser - to read the HTTP input data
• an INI file class - to read the initial configuration of the device and write changed configuration data
• a Java object to Python deserializer - to read initial configuration data of the device
• a Python to Java object serializer - to write changed configuration data
• an SFTP client - to transfer the initial configuration from the device and write the changed configuration to the device. This must support authentication by certificates.
• an HTTP server - to handle the requests from the device. For this to work, I only have to provide two different hard-coded answers depending on whether I have sent data via FTP or not.

Optional: To make it perfect, I'd like to make it work with any of our devices and implement an auto-detect feature, i.e. it detects the IP address and HTTP Port of the device. Since no other device is connected to the second Ethernet port, the network adapter could be put into promiscuous mode and sniff for incoming HTTP requests.

• a network library - to turn on promiscuous mode for a network adapter (on device side), distinguish incoming packets by protocol (HTTP for further analysis, simply discard others) and analyze HTTP packets (get source and destination IP address, source and destination TCP port, network mask as well as URL)

Startup (pseudo-code):

set-promiscuous-mode(true)
while (!autodetected):
    wait-packet()
    httpinfo=analyze-packet()
set-promiscuous-mode(false)
configure-network-adapter(httpinfo.IP+1)

start-http-server(httpinfo.destinationport)
start-http-client(httpinfo)

HTTP Server (pseudo-code):

while(listen()):
    if (ftp-data-written):
        send-reload-response()
        ftp-data-written=false
    else:
        send-empty-reponse()

HTTP Client:

certificate=read-crt-file()
ini=ftp-client-get(ini, certificate)
java=ftp-client-get(java, certificate)
while(true):
    response=send-http-request()
    if (is-empty(response)):
        sleep(10)
        continue
    else if (is-ini-data(response)):
        ini=merge-ini-data(ini, response)
        ftp-client-put(ini, certificate)
        ftp-data-written=true
    else if (is-java-data(response)):
        java=merge-java-data(java, response)
        ftp-client-put(java, certificate)
        ftp-data-written=true

Answer 1

You have quite a shopping list. I would suggest taking a look at:

1. the Standard Libraries Documentation - over half of your requirements are in the standard libraries &
2. the Python Package Index PyPi where you will find most, if not all, of the rest.

If there are still some that are missing asking a more targeted question here.