From eamerritt at gmail.com Mon Nov 9 10:57:19 2020 From: eamerritt at gmail.com (Ethan Merritt) Date: Mon, 09 Nov 2020 10:57:19 -0800 Subject: [Domterm-discuss] DomTerm question/suggestion Message-ID: <3465449.Z8mJ5ojYR5@stonelion> I have been revisiting the gnuplot + domterm combination for the first time in a couple of years. Both have seen significant additions since I last looked. I am now able to build and use the qtdomterm variant (configure --with-qtwebengine), which works nicely with gnuplot output embedded. There is one area where I hope there is room for improvement - animation/dynamic rotation of 3D plots. As it stands, each gnuplot 'plot' or 'splot' command creates a new output image, i.e. one frame of an animation or one incremental change in the view angle of a 3D rotation. This image is added to the display buffer and the terminal view is scrolled to show it at the top of the window. As the buffer gets larger and larger the process slows down approximately linearly. If I set a rotation loop going to view a surface from all sides, I see approx 5 frames per second for the first 50 frames, 3 fps for the next 50 frames, 1 fps for the third 50 frames, etc. It is soon annoyingly slow. This is benchmarked by repeatedly running a 50-frame loop in a gnuplot session using "set term domterm" in an instance of qtdomterm. Contrast this with running the same animation loop after resetting the buffer and switching to "set term sixel animate" in the same gnuplot session. The "animate" keyword to gnuplot tells it to reset each output image to overwrite the same area in the display buffer (sixel command sequence "\033[H"). Without this keyword the sixel output suffers the same linear decay in display speed as the svg output. I tried adding a "clear-buffer" command sequence "\033[7J" to the loop used in the svg case but this does not work as desired. The timing does become approximately constant, but rather slow. Furthermore it has the unexpected effect of making all output before the final frame invisible. I don't understand why, but this makes the attempted fix useless. AIUI, domterm writes a new html/javascript element for each gnuplot output frame. These pile up in the buffer even though only the most recent one is visible without manual scrolling. The ever-inceasing buffer size makes successive display updates gradually slower. My understanding of how domterm works runs low at this point so I am not sure exactly what might be possible to improve things. Would it be possible to change this so that instead of adding a new element for each new frame, successive frames overwrite or replace the element being displayed? The total buffer size would remain approximately constant and I would expect the display rate to remain independent of the number of frames previously displayed. I.e. behavior would be analogous to that of gnuplot's"sixel animate" output. Separate from this, I wonder if a mechanism already exists or could be added to feed back mouse coordinates from the qtdomterm window to an embedded gnuplot session. Gnuplot provides example javascript routines to handle minimal mousing in svg output displayed directly (not through domterm), but this capability is minimal compared to what the core gnuplot code could do if it saw the mouse coordinates as it does in interactive terminals (qt wxt x11). The DomTerm documentation indicates that it does track mouse locations, but I have not found any hint as to how gnuplot might be able to access that. best regards Ethan From per at bothner.com Mon Nov 9 12:29:17 2020 From: per at bothner.com (Per Bothner) Date: Mon, 9 Nov 2020 12:29:17 -0800 Subject: [Domterm-discuss] DomTerm question/suggestion In-Reply-To: <3465449.Z8mJ5ojYR5@stonelion> References: <3465449.Z8mJ5ojYR5@stonelion> Message-ID: A quick preliminary reply for now - I will be traveling and otherwise busy today. On 11/9/20 10:57 AM, Ethan Merritt wrote: > Contrast this with running the same animation loop after resetting > the buffer and switching to "set term sixel animate" in the same gnuplot > session. The "animate" keyword to gnuplot tells it to reset each > output image to overwrite the same area in the display buffer > (sixel command sequence "\033[H"). Without this keyword the sixel > output suffers the same linear decay in display speed as the svg output. After I did the gnuplot support, I added a feature to replace in-place an existing image. It is used for the Python support: http://domterm.org/Python-tips.html The escape code used is described here: http://domterm.org/Wire-byte-protocol.html Search for "721" or "Replace previously-inserted HTML". Also check the tools/python sub-directory of the DomTermn source tree for how the Python support works. > I tried adding a "clear-buffer" command sequence "\033[7J" to the loop > used in the svg case but this does not work as desired. > The timing does become approximately constant, but rather slow. > Furthermore it has the unexpected effect of making all output before > the final frame invisible. I don't understand why, but this makes > the attempted fix useless. That seems to be the documented effect: "Delete all lines in the buffer ... except the current line." > AIUI, domterm writes a new html/javascript element for each gnuplot > output frame. These pile up in the buffer even though only the > most recent one is visible without manual scrolling. The ever-inceasing > buffer size makes successive display updates gradually slower. Would the 721 escape code mentioned above work? Probably worth an effort. Are you comfortable working on the gnuplot C code? If not, I can try. Another idea: SVG has a mechanism for animation. If gnuplot can make use of these, we could add escape sequences that modify SCG properties such that animation happens. I don't know much about this. > Separate from this, I wonder if a mechanism already exists or > could be added to feed back mouse coordinates from the qtdomterm > window to an embedded gnuplot session. Well, DomTerm does implement the standard xterm mouse-reporting mechanism. It only reports the mouse position in terms of rows and columns. (An image is treated as a single large character in an extra-tall line.) > The DomTerm documentation indicates that it does track mouse > locations, but I have not found any hint as to how gnuplot might > be able to access that. This is mouse-reporting protocol(s) implemented by xterm (and DomTerm): https://invisible-island.net/xterm/ctlseqs/ctlseqs.html#h2-Mouse-Tracking I suspect that isn't enough. Do enough of these other terminal that report mouse position at the pixel level use an escape sequence protocol, or does gnuplot make directly calls to access the mouse? The former should be easy enough (easier if there is documented prior art); the latter would be a problem. (We want this to work over an ssh connection, for example.) -- --Per Bothner per at bothner.com http://per.bothner.com/ From eamerritt at gmail.com Mon Nov 9 13:42:22 2020 From: eamerritt at gmail.com (Ethan Merritt) Date: Mon, 09 Nov 2020 13:42:22 -0800 Subject: [Domterm-discuss] DomTerm question/suggestion In-Reply-To: References: <3465449.Z8mJ5ojYR5@stonelion> Message-ID: <3329977.5Hj0U6d7OM@stonelion> On Monday, 9 November 2020 12:29:17 PST Per Bothner wrote: > A quick preliminary reply for now - I will be traveling and otherwise > busy today. > > On 11/9/20 10:57 AM, Ethan Merritt wrote: > > > Contrast this with running the same animation loop after resetting > > the buffer and switching to "set term sixel animate" in the same gnuplot > > session. The "animate" keyword to gnuplot tells it to reset each > > output image to overwrite the same area in the display buffer > > (sixel command sequence "\033[H"). Without this keyword the sixel > > output suffers the same linear decay in display speed as the svg output. > > After I did the gnuplot support, I added a feature to replace in-place > an existing image. It is used for the Python support: > > http://domterm.org/Python-tips.html > > The escape code used is described here: > http://domterm.org/Wire-byte-protocol.html D'oh! That you for that pointer. For some reason I did not find that protocol description page. I found the "\033[7J" sequence by grepping the DomTerm source for reset-buffer. Now I see that "\033[3J" is the sequence I wanted. With that change I can get domterm animation working like sixel animation. Great! > Search for "721" or "Replace previously-inserted HTML". That might be even better, but I'll have to modify the gnuplot source code to try it. That can be a project for next week. I'll get back to you if I have further questions. > [snip] > Another idea: SVG has a mechanism for animation. If gnuplot can > make use of these, we could add escape sequences that modify > SCG properties such that animation happens. I don't know much about this. That would suffer from the original problem, I think. SVG animation switches amongst previously loaded frames. But I wanted to avoid having all the frames present at one time because it makes the rendering too slow. Now a new thought strikes me, however. Is it slow merely because the buffer is large, or is it slow because even the scrolled offscreen elements consume rendering time? If the latter, it would probably suffice to mark all elements "don't render" except for the one containing the current frame. That is essentially what SVG animation does, as I understand it, so maybe it would in fact be acceptably fast. > > Separate from this, I wonder if a mechanism already exists or > > could be added to feed back mouse coordinates from the qtdomterm > > window to an embedded gnuplot session. > > Well, DomTerm does implement the standard xterm mouse-reporting mechanism. > It only reports the mouse position in terms of rows and columns. > (An image is treated as a single large character in an extra-tall line.) > > > The DomTerm documentation indicates that it does track mouse > > locations, but I have not found any hint as to how gnuplot might > > be able to access that. > > This is mouse-reporting protocol(s) implemented by xterm (and DomTerm): > https://invisible-island.net/xterm/ctlseqs/ctlseqs.html#h2-Mouse-Tracking > > I suspect that isn't enough. Do enough of these other terminal > that report mouse position at the pixel level use an escape sequence > protocol, or does gnuplot make directly calls to access the mouse? Neither. Gnuplot has an event loop that receives mouse events from the window manager and from the relevant graphics subsystem. I imagine that at least for the qt flavor of domterm I could figure out how to enable generation of mouse events from Qt. We already do that for the Qt terminal. The problem would be initial calibration of the mapping from raw mouse pixels to the corresponding x/y offset in pixels from the plot origin. That part requires cooperation between the gnuplot terminal driver and the terminal it talks to. I.e. when gnuplot initializes a new plot, there needs to be some way for the terminal to feed back "I put the lower left corner of your plot at raw mouse coords x0/y0 and the upper right corner at raw mouse coords x1/y1". > The former should be easy enough (easier if there is documented prior > art); the latter would be a problem. (We want this to > work over an ssh connection, for example.) cheers, Ethan From per at bothner.com Mon Nov 9 18:05:29 2020 From: per at bothner.com (Per Bothner) Date: Mon, 9 Nov 2020 18:05:29 -0800 Subject: [Domterm-discuss] DomTerm question/suggestion In-Reply-To: <3329977.5Hj0U6d7OM@stonelion> References: <3465449.Z8mJ5ojYR5@stonelion> <3329977.5Hj0U6d7OM@stonelion> Message-ID: On 11/9/20 1:42 PM, Ethan Merritt wrote: > On Monday, 9 November 2020 12:29:17 PST Per Bothner wrote: > That would suffer from the original problem, I think. > SVG animation switches amongst previously loaded frames. What I mean is that JavaScript can modify an existing SVG structure (DOM) in the browser. For example you could create an SVG image with a circle: The if you modify the cx/cy attributes of the element, the circle would move. We could create an escape sequence that would do that sort of thing. > I imagine that at least for the qt flavor of domterm I could figure out > how to enable generation of mouse events from Qt. That is the wrong way to do it. Better to use JavaScript events. (1) It is more portable - and works for all front-ends, not just Qt. (2) Tying it to Qt doesn't help anyway, since the event still has to be sent from qtdomterm to the domterm background (server) to gnuplot, so it has to be encoded in the character stream, regardless. > The problem would be initial calibration of > the mapping from raw mouse pixels to the corresponding x/y offset in > pixels from the plot origin. One idea is you have a protocol (an escape sequence and/or an element property) that specifies a key (an identifier) and requests that mouse events in that image (or generally within that element) will send an encoding of the event that to the application. The encoded event includes the key, mouse coordinates relative to the element origin, and the button state/action. -- --Per Bothner per at bothner.com http://per.bothner.com/