Alex Bligh's blog

•

Apache websockets and tcp / vnc proxy

I’ve put my current work on the Apache vnc/tcp proxy (explanation here) on github, as people were (quite rightly) complaining a tarball was not particularly helpful. I’ve also added initial support for the guacamole protocol, though this is in need of optimisation. What’s on github is actually a clone of self.disconnect’s repo of apache-websocket, with my stuff in the vncproxy directory. It’s rough at the edges at the moment (don’t expect makefiles, substantial documentation etc.)

There is an occasional SEGV with guacamole (after 30 minutes of watching YouTube over it) which is, I think, due to me doing something as yet unknown which is not thread safe.

Comments welcome.

31 responses to “Apache websockets and tcp / vnc proxy”

  1. mike Avatar
    mike

    hello. I have a problem with running http://code.google.com/p/php-websocket-server/
    your example with novnc not try, try it.
    please. help me understand why not start:
    Connecting ..
    ws :/ / 192.168.1.3:9300
    Failed to connect.

    self.disconnect ‘s echo example works Finet

    Could you try and post htpd.conf

    Reply
  2. mike Avatar
    mike

    Hello again.
    Please could you explain to a novice how to run http://code.google.com/p/php-websocket-server/downloads/list or based on it https://github.com/Flynsarmy/PHPWebSocket-Chat with using your tcp proxy plugin.
    I really want to use this websocket chats

    mail to: mgracoder@gmail.com
    send plz your webmoney code to thank for your help

    sorry 4 double post, more posts will not be

    Reply
    1. Alex Bligh Avatar
      Alex Bligh

      Can I suggest first you make sure you are using the most recent version, which is now at:
      https://github.com/abligh/apache-websocket
      which is a clone of self-disconnect’s repo with my stuff in the vncproxy directory. Do not worry, it will also do straight tcp proxying.

      Next, can I suggest you run your Apache with ‘LogLevel Debug’, and look at the error.log, as this is normally helpful in finding out why connections are not established.

      The config you require is going to depend on the set up you are using, and I’m not familiar with PHPWebSocket-Chat. I presume you are using that to provide the client, and are running a straight TCP (non-websocket) server behind Apache?

      Normally the first things to check are:
      a) In the access.log file, do you see a request for the URL? If not, you have got Apache configured wrong outside of my module.
      b) What HTTP response code do you see for that? If it says 404, or indeed anything but (from memory) ‘101 Upgrading’ (or similar) then it isn’t even getting to my module.
      c) If it then closes the connection, this is likely to be for one of the following reasons (established from error.log with loglevel debug)
      (i) Websockets has an internal textual identifier for the ‘Protocol’ used, which depends on the client. It might be (for instance) ‘chat’. You need to specify that correctly in my config.
      (ii) Is the module attempting to open a tcp connection to whatever your back end server is? (again, look at error.log). If this fails, it will close the connection.
      (iii) Websockets can either send data raw or base64 encoded. This doesn’t normally cause an immediate failure to connect, but the setting of the base64 option correctly is vital to data passing correctly, and if the data is garbled one end might be closing the connection.

      Once you have passed step (ii), running tcpdump to examine the session between the websockets module and the back end server is a useful way to identify errors (e.g. broken base64 settings).

      Reply
  3. mike Avatar
    mike

    when I run server.php I get: error.log [error] server reached MaxClients setting, consider raising the MaxClients setting
    hm? maxclients over 1000

    Reply
  4. Alex Bligh Avatar
    Alex Bligh

    Mike: I have no idea what server.php is. It’s certainly not my code! I suggest you ask whoever wrote that.

    Reply
  5. Ali Avatar
    Ali

    Hey Mike! Thanks for this module – makes our lives a lot easier!

    Did you ever see a problem with apache not closing idle connections? Somehow our apache doesn’t manage to understand that websocket connections have long since gone, even stopping the ruby eventmachine that actually responds to these doesn’t make them go away. Now, I’ve tried playing around with the request timeouts, this is what we use now:

    RequestReadTimeout body=30,MinRate=1

    Upping the MinRate either closes the sockets too early or not at all (we’re sending a ping every 30 seconds so I tried upping the MinRate to a value that reflects that).

    This is what I see on the server status page.

    41-6 20834 0/248/698 W 0.34 5934 0 0.0 0.04 0.92 10.0.1.25 xyz GET /websocket HTTP/1.1

    after graceful restart:

    2-4 27408 0/1/1 G 0.01 176266 0 0.0 0.00 0.00 127.0.0.1 xyz GET /websocket HTTP/1.1

    We’ve got tons of connections accumulating throughout the day that never close.

    We’re still using your original tcp_proxy code, so I will try updating it, going through your commit log however I didn’t see anything in that regards so I thought I’d ask you if you’ve ever seen something like this…

    Any help is appreciated :).

    Reply
    1. Alex Bligh Avatar
      Alex Bligh

      I’m not quite clear under what circumstances you are trying to get ‘idle’ connections closed. If you are trying to close truly idle sessions, then why not use the WebSocketTcpProxyTimeout option (may be called something slightly different in the previous version)? That’s exactly what that’s used for. This looks for no bytes passing the TCP session (so ignoring WS keepalives) for n seconds, where n is the parameter. Of course this won’t work if the keepalives filter through to the TCP level as well (i.e. if they generate traffic on the TCP session); in that instance, I’d guess your server should kill them off.

      We use “RequestReadTimeout body=300,minrate=1”, but that’s more to turn OFF the default ubuntu ModTimeout options, rather than to do anything useful.

      But no, in answer to your question, with WebSocketTcpProxyTimeout in use I’ve never seen idle connections piling up.

      Oh, and it’s Alex (not Mike) by the way!

      Alex

      Reply
  6. Ali Avatar
    Ali

    Hey Alex! (sorry about the mixup ;))

    This is very weird. I’m seeing socket connection pile up with no data flowing. I played with the WebSocketTcpProxyTimeout option but alas, no change. It has a default of 30s by the way, so it should have kicked in. Will continue investigating!

    Ali

    Reply
    1. Alex Bligh Avatar
      Alex Bligh

      Ali,

      I suggest you try the latest version. The previous version could (IIRC) block, which would look like a connection that never died.

      Alex

      Reply
  7. Ali Avatar
    Ali

    Hey Alex,

    just updated to the latest version:

    unfortunately this is all I’m getting:

    [Thu Oct 18 18:54:55 2012] [debug] mod_websocket_vnc_proxy.c(1215): [client 10.44.75.203] tcp_proxy_on_connect starting
    [Thu Oct 18 18:54:55 2012] [debug] mod_websocket_vnc_proxy.c(1245): [client 10.44.75.203] tcp_proxy_on_connect protocol correct
    [Thu Oct 18 18:54:55 2012] [debug] mod_websocket_vnc_proxy.c(1319): [client 10.44.75.203] tcp_proxy_on_connect: base64 is 0
    [Thu Oct 18 18:54:55 2012] [debug] mod_websocket_vnc_proxy.c(368): [client 10.44.75.203] tcp_proxy_do_authenticate: key is ”
    [Thu Oct 18 18:54:55 2012] [debug] mod_websocket_vnc_proxy.c(374): [client 10.44.75.203] tcp_proxy_do_authenticate: query_key failed
    [Thu Oct 18 18:54:55 2012] [debug] mod_websocket_vnc_proxy.c(1331): [client 10.44.75.203] tcp_proxy_on_connect: closing connection as authentication / initial data failed

    Do you have any ideas?

    Ali

    Reply
    1. Alex Bligh Avatar
      Alex Bligh

      Ali,

      Are you using the database and key feature? If not, I suggest you ensure you don’t have WebSocketTcpProxyQuery defined.

      I am wondering if it is refusing to play ball because you aren’t sending a key in the URL (that shouldn’t matter if you haven’t got WebSocketTcpProxyQuery defined and if it does matter it is a bug). To test this, if you don’t have WebSocketTcpProxyQuery defined, could you add “&key=foo” to the end of your URL (or “?key=foo” if you have no GET parameters on it already), in which case you should see ‘key=”foo”‘ in the logs.

      Reply
    2. Alex Bligh Avatar
      Alex Bligh

      Actually I think I see the problem.

      Could you move these lines (on line 206):

      So that they are a little further up, just before (as opposed to just after) these lines:

      I think that should fix it for you.

      All my stuff uses the key feature so I may not have tested without it.

      Alex

      Reply
  8. Ali Avatar
    Ali

    Alex,

    I’m not using the TcpProxyQuery, this is what happens when I call:

    new WebSocket(“ws://ubuntu.local/websocket?foo=bar”)

    ==> dashboard/current/log/access.log dashboard/current/log/error.log <==
    [Thu Oct 18 19:09:48 2012] [debug] mod_websocket_vnc_proxy.c(1215): [client 10.44.75.203] tcp_proxy_on_connect starting
    [Thu Oct 18 19:09:48 2012] [debug] mod_websocket_vnc_proxy.c(1245): [client 10.44.75.203] tcp_proxy_on_connect protocol correct
    [Thu Oct 18 19:09:48 2012] [debug] mod_websocket_vnc_proxy.c(1319): [client 10.44.75.203] tcp_proxy_on_connect: base64 is 0
    [Thu Oct 18 19:09:48 2012] [debug] mod_websocket_vnc_proxy.c(368): [client 10.44.75.203] tcp_proxy_do_authenticate: key is ''
    [Thu Oct 18 19:09:48 2012] [debug] mod_websocket_vnc_proxy.c(374): [client 10.44.75.203] tcp_proxy_do_authenticate: query_key failed
    [Thu Oct 18 19:09:48 2012] [debug] mod_websocket_vnc_proxy.c(1331): [client 10.44.75.203] tcp_proxy_on_connect: closing connection as authentication / initial data failed

    seems like it always try to use the TcpProxyQuery code.

    Reply
  9. Ali Avatar
    Ali

    Alex, a sec will change the code and test again! 🙂

    Reply
  10. Ali Avatar
    Ali

    Works! Thanks Alex! It also solved my idle connection problem, will have to keep monitoring it but will be back with an update…

    Reply
    1. Alex Bligh Avatar
      Alex Bligh

      Thanks for testing. I will update the repo.

      Reply
  11. 4llan Avatar
    4llan

    Hi Alex.

    The mod_websocket_vnc_proxy.c was exactly what i’m searching for. But, when I try to compile on Windows, I’m getting various errors.

    There’s something that I have to do? I’m trying to compile under Visual Studio Command Prompt 2010… First error: “C requires that a struct or union has at least one member” and exceeds 100 errors

    Rename to cpp and try to compile > fatal error LNK1107: invalid or corrupt file: cannot read at 0xE31F

    Please, help!

    Reply
    1. Alex Bligh Avatar
      Alex Bligh

      The version at
      https://github.com/abligh/apache-websocket
      in the vncproxy directory is more Windows friendly as I took out all the varargs macros, which are apparently not supported by MSVC, so I suggest you start from there. I don’t have a Windows platform, but would happily accept patches. I would not try renaming to .cpp though – I’d fix whatever gcc-specific thing I’ve used.

      Reply
  12. Dick Gooris Avatar
    Dick Gooris

    Hello Alex,

    My name is Dick Gooris, I am trying to gain some knowledge about websockets in general. I found your initiative for a plugin inside Apache. I built the code and I am using the increment.html as a starting point which works ok in one go.

    Now on my way to make it a bit more exciting, at the client side I make a copy of a video (web-cam) video frame via a canvas frame. These video data frames are now sent towards the web-socket apache module. The function _on_message that test the word ‘reset\n’ does not do anything with the received data.

    Now, after some time, and very randomly in time, the socket connection gets dropped. Looking inside the tcp packets using wireshark I can see that the dis-connect is initiated by the apache module.

    I did quite some experiments like for example, by converting the video data first to a hex format to make it a text transmission. This does not make a difference, eventually the connection drops.

    Also I did some statistical experiments, for example to determine any relationship with the number of octets I and sending from the client. (e.g. to see if it could be a buffer overfow) There seem to be no correlation. Sometimes it runs fine for an half an hour, sometimes the issue occurs after a few seconds.

    Some debugging statements in the main apache module (mod_websocket.c) where a disconnect is handled, does not help me either, since the actual cause is not clear.

    I am using the latest web-socket code version (not touched for two months I think). The server is a 64 bits Suse 12.1 release.

    In your blog, you refer to a segv issue after a 30 min of data stream. Sounds like the opposite direction of the data stream. I am not sure where the segv error is appearing (/var/log/apache ?) But this might not be related anyhow.

    Any idea what the cause of this issue could be, or what I could do to narrow down this issue ?

    Many thanks in advance, and kind regards.

    Dick

    Reply
    1. Alex Bligh Avatar
      Alex Bligh

      Hi Dick,

      Firstly, I strongly suggest you use the latest version of my stuff which is at github. See:
      https://github.com/abligh/apache-websocket
      in the vncproxy directory. This has fixed several issue.

      Secondly, are you seeing a SEGV or are you just seeing a disconnect?

      If you are seeing a SEGV, that would be interesting. I can still (by playing many many many hours of full screen video through it) occasionally get a SEGV. Assistance debugging this would be appreciated!

      If, as I suspect, you are just seeing a disconnect, first set ‘LogLevel Debug’ in your apache configuration. If my module is sending a disconnect ‘deliberately’, this will tell you why. Is your connection completely unidirectional? I think I may require at least some data to flow in the opposite direction (even one byte per second) in order not to close the session.

      Alex

      Reply
  13. Dick Gooris Avatar
    Dick Gooris

    Alex,

    Thanks for the fast response. I will have a look in the vncproxy directory and use that source for sure.

    I do not see the segv, at least, not that I know of. I just do not know where I can see that this occurs. But, if the connections stays up, I will for sure run long duration tests. If I encounter this issue then lets see what I can do to help there, no problem. By the way; does the segment violation dumps the core – for further gdb kind of backtraces ?

    Ok, I will also set the loglevel to debug, hopefully I see more verbose about the cause.

    The connection is sending data in both direction; while sending video frames towards the server, in the meantime the server sends the increment values back to the client. (Ik keep that code in, since that is a very nice example of a pthread implementation within the apache module)

    I will let you know the result. Thanks !

    Dick

    Reply
  14. jaffer Avatar
    jaffer

    Hello,
    Very well done… this is a Cool module/proxy… Thanks for that
    i have query regarding guacamole …
    Why is this guacamole protocol required here ?
    guacamole Does not use websocket…. and what you have written is vnc proxy for websocket …

    Thanks
    Jaffer

    Reply
    1. Alex Bligh Avatar
      Alex Bligh

      Hi Jaffer,

      Yes, Guacamole does use websockets, but the Guacamole client currently requires that Guacamole commands are not split across websocket frames. Actually the requirement may be slightly less strict, such as one frame may not contain more than one Guacamole command, I forget. However, sadly, one cannot treat Guacamole like any other TCP stream protocol, and one needs to understand the protocol itself (or at least where the boundaries of the commands are) in order to correctly frame the reply packets.

      Alex

      Reply
  15. Bo Avatar
    Bo

    Hi

    How easy is it to get up and running with this on a apache on windows(7)
    Nomally modules are just copied to the apache folder – and initialisied from http.conf right ? Is this not the case with this one ?. running http://www.wampserver.com/en/ 64 bit.
    Would love to play around with it, but being frontend’er – I just want to know if its something I’m gonna waste alot of hours on getting up n running, or if it’s doable for a frontend’er 🙂

    Reply
    1. Alex Bligh Avatar
      Alex Bligh

      I have no Windows system to test it on. The github version has some patches in that allegedly help it to compile on Windows (removal of gcc specific constructs), but I’m not sure anyone has yet got it working. Patches / success stories welcome.

      Reply
  16. bon Avatar
    bon

    Hi Alex i am too facing same problem using VNC proxy which you put in http://mail-archives.apache.org/mod_mbox/httpd-users/201209.mbox/%3CDCA0708A7FE3327D08B703AD@Ximines.local&gt;

    Is there any solution available? Apache crashes when we do server->send from worker thread. I created same for RDP, and issue can be repro when we do dragging windows or regress user interaction.

    Reply
    1. Alex Bligh Avatar
      Alex Bligh

      This issue fixes mod_proxy but also some of the bucket brigade code. Arguably that makes the crash less frequent. However, it does not appear to fix it. I would be most interested if there were a solution available. As far as I can tell it’s something deep down within apache as I am pretty sure I’ve got the pool handling right (now) in my module. When you say you see it fail for RDP, is that using RDP completely independently of my module? Or are you using my thing to proxy RDP? I have someone who would probably pay to get this fixed but it needs a core apache dev to fix it I think.

      Reply
  17. bon Avatar
    bon

    Hi Alex,
    Thanks for reply,
    I tweak your VNC proxy plugin to proxy RDP traffic just by changing message type from text to binary(MESSAGE_TYPE_BINARY). As i mentioned screen sharing is working fine, core observed only if user try to do some dragging and regress interaction(play video,animation) . I am using apache 2.4

    Reply
    1. Alex Bligh Avatar
      Alex Bligh

      That’s really useful data. So, whatever it is is not fixed in apache 2.4 (that’s good from my point of view as it will encourage the apache devs to fix it). However, your test doesn’t isolate whether it’s apache or my module. I would make sure you are running with the version from github (particularly the one with this commit in). I suspect the problem is only apparent on multicore CPUs (you might see if you still see it in a single CPU core VM), and arises under some circumstances where a data arrives into the bucket brigade at the same time as its being dequeued. I have a theory it may be actually to do with SSL as I’ve never seen it happen without SSL (though that may be a timing thing). To duplicate this here requires running youtube over VNC for about 96 hours. In the field I hear reports of it more often. If you can replicate this more easily (particularly not requiring an Apache dev to manually make Windows move etc.) I would be really interested. I went to the level of writing an automatic guacamole protocol generator (as the guacomole protocol is nice and simple) which spewed rectangle draw and bitmap drawing commands out, but couldn’t get that to crash it, even simulating network delays. I think if you produced a simple automatically reproducible test case we might get somewhere with this (or if you found an apache dev that could be interested in looking at it).

      Reply
  18. bon Avatar
    bon

    Yes i am using the commit one where you use separate pool for main thread and worker thread(with seperate threadallocator). Also i am using mod_ssl and VM with multiple core. Also i can repro it easily, in case of RDP frequency is on higher side. I am not a apache hardcore dev but need to proxy RDP traffic using latest browser client less hence using websocket.

    Core is generated when doing ->send to browser from worker thread, otherwise ->send work fine.Trace is also similar to one you put in apache support forum as below: Right now using a stripped apache but soon try with unstripped to isolate this issue:

    #0 0x00007f2401fe0757 in kill () from /lib/x86_64-linux-gnu/libc.so.6
    #1
    #2 0x0000000001cbd1c8 in ?? ()
    #3 0x00000000004498d3 in ?? ()
    #4 0x0000000000449d1a in ?? ()
    #5 0x000000000044a799 in ap_core_output_filter ()
    #6 0x00007f24009f16fa in bio_filter_out_pass (outctx=0x1cbbe00) at ssl_engine_io.c:133
    #7 bio_filter_out_flush (bio=) at ssl_engine_io.c:154
    #8 ssl_io_filter_output (f=0x1cbbdd8, bb=0x7f23ec03a580) at ssl_engine_io.c:1601
    #9 0x00007f24009eeb52 in ssl_io_filter_coalesce (f=0x1cbbdb0, bb=0x7f23ec03a580) at ssl_engine_io.c:1558
    #10 0x00007f23faaae990 in mod_websocket_plugin_send (server=, type=128, buffer=0x7f23f13783e0 “g\261}\322\33066C\226No\2016661\343\22326\323!\264\273s\31471v\33623v\370c\35265\254TA\316\200\364\254c\306\377h\303[O\201\263\215\246,iC\324 \\373\242\215\32261\255B\274\311\226\340\363\353\271r\240\366a\271\237\363\210uA-\340\227\264\250b{r\216\343?\200\340joGN(W\36720,\f\304H\376\355Sc}06\221\232\217\227\27160’d\323V\27634^\342&26\362\317\251|\223H\302\301?\\256\316\315\352\317\254\3722634F\214\32520\342\ve\233\23761+F\30364T\264O013135\374\254]30AmV#\274Ky\373\354\252\n\211h,\361vms\276q\26605\310\353\303~7@/”…, buffer_size=) at websocket.c:363
    #11 0x00007f23ff4c02b9 in ew::rdp::proxy_run (thread=, data=0x7f23ec02e530) at ew_rdp.cpp:203
    #12 0x00007f2402543b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
    #13 0x00007f240208a0ed in clone () from /lib/x86_64-linux-gnu/libc.so.6
    #14 0x0000000000000000 in ?? ()

    where ew::rdp::proxy_run is the worker thread.Tried trace with unstripped mod_ssl, seems have to check with unstripped core to get root cause.

    Reply
    1. Alex Bligh Avatar
      Alex Bligh

      That’s interesting. My SEGV is always in the bucket brigade handlers, but perhaps that a feature of lack of symbols in the above. Might be easier to do this by email. My email is alex (at) alex (dot) org (dot) uk

      Reply

Leave a Reply

Discover more from Alex Bligh's blog

Subscribe now to keep reading and get access to the full archive.

Continue reading