Web-Based Multi-Screen Apps Including Drag & Drop

I am excited about this topic, since the technology we are going to talk about opens the way for a new generation of web-based apps, which can directly communicate across multiple browser windows without involving a backend.

[side note] The article got pretty long. In case you got only a short amount of time, take a look at the video in 2., read the highlight in 10. and then decide if you want to read all of it.

Content

1. Introduction
2. The demo app
3. Are we using the HTML5 drag & drop API?
4. How can we communicate between browser windows?
5. How is the browser support for SharedWorkers?
6. The neo.mjs framework is a perfect fit!
7. An overview of the demo app code base
8. The drag&drop concept
9. The multi window drag&drop logic
10. Drop the dialog inside the other window
11. Online demo
12. Can we do more?
13. What is coming next for the neo.mjs framework?
14. Final thoughts: Call for action!

Appendix

1. Previous Article: “Expanding Single Page Apps into multiple Browser Windows”
2. Docked browser windows

1. Introduction

In case you want to (for example) create a web based IDE or a banking / trading App which runs on multiple screens at the same time, you were facing several problems.

Of course you can create a native shell (e.g. using GitHub Electron) and show a browser window on each screen in fullscreen mode, but it gets really tricky as soon as multiple windows need to interact with each other.

This starts with simple things: you have a table on your left screen and as soon as you click on a table row, you want to adjust the content of a chart on your right screen.

It does get more complex in case you want to dynamically move content from one screen to another. Example: You have one screen containing a navigation tree and a content view and you want to move the navigation tree into a separate browser window while keeping the functionality in place.

Drag & drop across multiple browser windows is probably the most difficult part. Imagine an app running on 2 screens (one fullscreen browser window on each one) and you can create an in-app dialog and drag it from one screen to another. You know and probably love this feature from most OS desktops, where you can easily drag your programs / views around multiple screens. So why not offer the same functionality for your web apps?

2. The demo app

We are using a simple, yet powerful demo app:

You can open the main window inside your browser and then open a docked window with the button on the top right. You can switch the docking side of the second window dynamically.

We can open an in app dialog with the button on the top left and then drag it into the docked window and drop it there. We can also drag it back from the docked window into the main window.

Let’s take a look at this in action:

3. Are we using the HTML5 drag & drop API?

You might be familiar with this one:

HTML Drag and Drop API

In short: We are not using it.

The API is great for simple use cases, like a file upload button where you want to import a CVS file into your app. It is nice that you can actually drag items from your desktop into your browser window. A plus is that you get a drag proxy which is visible when not hovering the mouse over a browser window. However, the drag proxy can only get customised very little and on drop you only get a data transfer object.

The API is not really intended for app specific complex use cases, like when you want to move a big component tree around.

4. How can we communicate between browser windows?

A: Obviously you can use a backend to handle this part. E.g. 2 browser windows using a socket connection and then you can push changes around. This approach is a nightmare from a performance perspective.

B: A browser window can communicate to popup windows it created (& iFrames) using postMessages:

Window.postMessage()

This approach is already way better than “A”, but it brings a lot of issues developers can struggle with:

1. Where to put the business logic?
2. How to avoid code redundancy?
3. What happens in case there are multiple popups? (Imagine popup1 spawned a new popup2 and you want to communicate between the main window and popup2)
4. Can we keep the same JS instance of a component when moving it to a different window? (Not impossible, BUT…)

C: The smart approach is to use a SharedWorker:

SharedWorker

SharedWorkers use postMessages as well, so each browser window can directly connect to one worker instance and you can set up the communication.

There is no restriction to using popup windows only. You may use “real” browser windows as well.

This is probably closer to the real use case, where you use a native shell and add a WebView (headless browser window) to each screen.

5. How is the browser support for SharedWorkers?

Again: The most common use case for multi window apps are native Shells, so you can pick a browser of your choice (e.g. a headless Chromium).

However, knowing about the support for all browsers feels helpful:
https://developer.mozilla.org/en-US/docs/Web/API/SharedWorker

The support is perfect in Google Chrome. SharedWorkers even support JS modules here.

Since Microsoft Edge switched to the Chromium engine, it is on the same level there.

We have an open ticket for Chrome on Android:
https://bugs.chromium.org/p/chromium/issues/detail?id=154571

SharedWorkers are supported in Firefox (the last time i checked, excluding JS modules).

SharedWorkers are not supported in WebKit (Safari) yet:

149850 - Reinstate support for SharedWorkers

The team is at least thinking about reinstating it, so adding a comment to the ticket will help!

6. The neo.mjs framework is a perfect fit!

For creating this demo app, we do not want to re-invent the wheel, and can solve this use-case with very little code.

The neo.mjs framework & all demos are fully open source based (MIT licensed), so you can use, extend and customise it as you see fit.

You can find the project here:

neomjs/neo

A quick look at the application architecture that comes with it:

The SharedWorkers setup is exactly what we need for our cross window drag&drop demo. We can switch from the normal workers setup to the shared workers setup by changing just one framework config:

useSharedWorkers: true

The way to use the framework stays the same.

So, what are the key benefits inside this demo context?

1. We get the workers setup & communication API out of the box.
2. We get unique DOM ids across all connected browser windows.
3. DOM events are completely de-coupled from the main thread.
4. All components (JS instances) live within the application worker.

7. An overview of the demo app code base

For now the codebase is located inside the apps folder:
https://github.com/neomjs/neo/tree/dev/apps

/sharedDialog & /sharedDialog2 to be precise.

[Side Note] The reason for this layout is due to the way the deployment of the online examples (Github Pages) works right now. It would be nicer to put each example into its own repository and adjust the deployment to pull in each repo on its own. This would also allow us to use different framework versions for each example. On my todo list!

Describing the 2 main views is fairly trivial:
apps/shareddialog2/view/MainContainer.mjs

The code for the first MainContainer is very similar, but also contains the docked window radios:
apps/shareddialog/view/MainContainer.mjs

We have the DemoDialog.mjs view inside app1:
apps/shareddialog/view/DemoDialog.mjs

This file is only included there and does not exist inside our docked window app.

Let us take a look into the second MainContainerController next:
apps/shareddialog2/view/MainContainerController.mjs

If you think about it, this is already amazing!

The docked window app does not contain any logic on its own, so there is zero redundancy.

onCreateDialogButtonClick()

The above method expects the main app to exist, which is fair inside this context. We are triggering a method inside the view controller of the main app directly. We could, as well, fire an event on that main view instance and subscribe to it.

So, the main app MainContainerController contains all of the relevant business logic and it can be done with just 600 lines of code:
apps/shareddialog/view/MainContainerController.mjs

Before we dive deeper into it, let us cover the drag&drop basics first, so that we are on the same page.

8. The drag&drop concept

When we drag an in app dialog, we are not dragging the real DOM node, but a so-called proxyEl.

The proxyEl is supposed to be a more lightweight clone of the real element. For a dialog, it contains the header and an empty body to reduce browser reflows (imagine a complex component tree containing grids / tables).

The dialog class is still work in progress, you can find it here:
src/dialog/Base.mjs

onDragStart() we are fading out the real dialog DOM (reducing the opacity) and we are moving the proxyEl around.

One benefit of this approach is, that we can easily extend & enhance it. E.g. we could implement an ESC key listener to cancel the current drag operation at any point (destroy the proxyEl, fade the dialog back in).

The dialog class is using an instance of draggable.DragZone:
src/draggable/DragZone.mjs

This one can assign the proxyEl movements directly to an optional main thread addon:
src/main/addon/DragDrop.mjs

=> we don’t need to push every drag:move event to the app worker for simple use cases and keeping the movements in main makes sense from a performance perspective.

In short: The dialog class already has the logic in place to drag it around inside the app (browser window) it lives in, so we don’t need to implement this part inside our demo app.

9. The multi window drag&drop logic

As mentioned in 7. already, you can find the full logic here:
apps/shareddialog/view/MainContainerController.mjs

To get started, let us take a look into createDialog()

Both apps have a “Create Dialog” button, clicking it will create a new dialog instance. We could as well just unmount the dialog on close and then mount it again when clicking the button. It depends on your app use case (is it a frequently used dialog you want to keep or something like a login form which (mostly…) only gets used once?).

Enabling or disabling the “Create Dialog” buttons is super easy, since all components live within the app worker. So you can use manager.Component to find instances, even when their DOM lives inside different browser windows:

button.disabled is a framework config, so just assigning a new value to it will trigger a setter and automatically update the UI for you.

We added a dragZoneCreatedlistener into the dialog instance configs, so we can listen to the relevant events:

The DragZone itself will get created onDragStart() inside the dialog class (if it does not exist yet), so we need this custom event hook.

We only do something onDragStart() in case a second browser window exists.

We assume that you can not resize a dialog or browser window during a drag operation (well, maybe with some OS shortcuts, but not with the mouse).

We store the dialog rect, which has the same size as the proxyEl.

We store the dragStartWindowRect . Important: this is the document.body DOMRect of the window where we started a drag, so it can be the main window or the docked window.

We store the size of the window where we did not start the drag inside targetWindowSize .

For drag:move, we are using a rectangle utility class:
src/util/Rectangle.mjs

(We can further enhance this one, when we add floating menus.)

In case we don’t have a proxyEl yet, we will create it (including the current position), otherwise we will move it to the spot which matches our mouse cursor.

We are hiding the proxyEl of the non drag:start window, in case the proxyEl is fully visible inside the drag:start window (we can get side effects otherwise, in case you drag very fast).

The clue here is that in case we started the drag inside the docked window, we can just pretend that it is the main window having a docked window to the other side => re-using the same business logic.

In case we drop the dialog fully inside the drag:start window, we don’t need to do anything, since the dialog DragZone will take care of it.

We need to switch the logic, in case the dialog is fully dropped inside the other window or in case it does get dropped in between both windows.

10. Drop the dialog inside the other window

This is the highlight of this article.

We can basically reduce the code to:

dialog.unmount();

dialog.appName = 'SharedDialog2'; // the name of the other Window App

dialog.mount();

Now you will most likely ask yourself:

“How on earth is this possible?!”

This will bring us back to:

So, what are the [neo.mjs] key benefits inside this demo context?

1. We get the workers setup & communication API out of the box.
2. We get unique DOM ids across all connected browser windows.
3. DOM events are completely de-coupled from the main thread.
4. All components (JS instances) live within the application worker.

We can keep the same dialog JS instance, since it lives within the (shared) app worker.

We can just mount() it inside the other browser window, since we know that there won’t be any conflicting DOM ids.

De-coupled DOM events mean, that the entire business logic of the dialog will still work. After dropping it inside the other window, you can still close it, resize it or drag it again.

The framework knows which app(s) live inside which window, so adjusting the appName is all we need to do here.

11. Online demo

I added the demo(s) to the neo.mjs online examples:
https://neomjs.github.io/pages/

Please open the demos inside a desktop version of Google Chrome.

dist/production/apps/shareddialog/index.html

The dist/prod version runs in Firefox as well, but it is not polished yet. Docked window positions need some adjustments and there are some CSS issues. The drag&drop logic itself works fine.

apps/shareddialog/index.html

The development mode version can only run in Chrome & Edge, since other browsers do not support JS modules inside the shared worker scope yet. In this version, you can see the real code, since it works without any builds.

To inspect SharedWorkers, you need to enter the following URL:

chrome://inspect/#workers

Then click on the app worker and you get a new console window.

Inside the dev mode, you will get the real code (no source maps):

12. Can we do more?

The answer is obviously “yes!”.

1) We can fully polish the demo for Firefox.

2) Making the logic more generic. We could extend dialog.Base or create a new DragZone implementation for the shared workers context. This can make it easier to use.

3) Adding cross window animations.
While it is sweet that we can easily change the animationTarget of dialogs:

It would also be nice if we could keep the anchor node inside a different window. In theory: we start the same animation in both windows at the same time, with different positions (e.g. move the proxyEl from outside the window to the button in one window and moving it from the dialog to a target position outside the visible area on the other window).

4) Dropping dialogs between windows is a very interesting topic.

I started with an approach to figure out which window has the bigger intersection with the dialog and move the dialog there:

To be fair, this feels already sufficient for most use cases.

However, you can drop OS based apps between screens and they stay fully functional. So why not achieve the same for web based apps?

The first idea which comes to mind is to clone the dialog JS instance and have 1 instance inside each window. This approach brings a lot of side effects and custom logic with it. What happens if you drag the real instance? Even worse, what happens if you drag the cloned one? We would need custom logic for every component users can interact with and keep them in sync (e.g. in case you type into a TextField).

Well, with the neo.mjs setup we can do better!

We can just drop the DOM of the dialog into both windows and the logic will still work. My first thought was to keep 1 vdom (new state) object and 2 vnode objects (current state), but this requires to do the delta update calculations twice.

Since the dialogs are supposed to be in sync (except for their position style values), we don’t even need this.

My latest brainstorming idea is to add a flag to the app worker in case it has components (dialogs) which are supposed to live in more than 1 browser window at a time.

If so, we can adjust the appName config of components to support an array of apps. In case we change the vdom of a component living inside a dialog which is mounted more than once, it could just send the delta updates to all relevant main threads.

This still leaves the topic of user generated changes, e.g. when you type into a TextField, we just update the vdom & vnode and there is nothing left todo for single page apps. Or in case you scroll a grid / table, there is no need for a scroll sync.

So there are a couple of spots where the framework would need to trigger new delta updates, ideally without adding irrelevant code for the non-shared workers context.

This is a topic I would deeply enjoy to push further!

5) Once this logic is in place, we could also do things like resizing a dialog so it grows to live within more than 1 browser window.

6) We could support more than 2 windows. E.g. 2 screens on top and 2 screens on the bottom => a dialog could get dragged into 4 windows at the same time. I know, edge case for an edge case… :)

7) We could add a hook into the DragZone, to prevent the drop logic from happening in case a flag is set (e.g. in case we drop a dialog into the other window, there is no need to drop it into the origin window as well (reason for the setTimeout() call).

13. What is coming next for the neo.mjs framework?

Again, you can find the fully MIT licensed project here:

neomjs/neo

My internal roadmap is incredibly long.

Truth-be-told, while working on the drag&drop implementation I had the idea for this tech demo in mind and had the drive to get it done, just to get a feeling if there is actually a need for it.

In case you are working on a business use case where a web based multi screen apps make sense, please definitely add a comment here!

Since the clients I am aware of which are currently using the framework are focussing on SPAs:

I will put my focus on the framework core and adding more components plus further polishing existing ones first.

There are requests for view models & data bindings (VM configs to component configs), which are an epic.

Simplifying component configs => adding public class fields once webpack (Acorn) supports it is very high on my todo list. One of the first breaking changes, so this will become neo.mjs v2.

14. Final Thoughts: Call for action!

Open source projects of this size require a massive amount of work & love.

Not meaning this tech demo on its own, but the entire neo.mjs framework scope.

To get on a sustainable level, they require an active community of contributors or sponsors. Ideally both.

Sadly, neo.mjs is still a hidden gem out in the wild and not enough people have noticed its existence yet.

I think, when the framework got initially released on GitHub (Nov 2019), the concepts were just too far ahead. A lot has changed since then (4700+ commits since the GA). I truly hope that more developers are ready now to give it a try.

In case you want to support the project, just tell a friend about it or e.g. share this article.

This would mean a lot to me!

In case you have the JS skills to contribute, you are very welcome to do so.

I am aware that while using neo.mjs is fairly easy, contributing to it is a different story and requires an expert level.

I am still willing to help getting more devs up to speed!

For now, I will focus more on helping client projects to get their first neo.mjs apps into production to not burn out on the financial side of things. No worries, I will keep pushing the project as good as I can in my spare time.

Best regards & happy coding,
Tobias

Appendix

1. Previous article: “Expanding Single Page Apps into multiple Browser Windows”

This demo app was intentionally kept as simple as possible.

In case you want to see or more complex use case (without cross window drag&drop):

You can find the article here:

Expanding Single Page Apps into multiple Browser Windows

The article is a bit outdated at this point, but the Covid app itself is still an excellent example on how to communicate inside the SharedWorkers context.

2. Docked browser windows

The current implementation is more like a side product, since it felt convenient for testing different drag&drop directions.

However, I already got the feedback that this could be useful for multi window apps in general.

Especially in case your users are working on giant screens.
You know, something like this:

Inside the neo.mjs code base, we have a new (optional) main thread addon to support this:
src/main/addon/WindowPosition.mjs

You can just drop it into your index.html file (mainThreadAddons), as well as into buildScripts/webpack/json/myApps.json.

There are 2 main restrictions:

1. You can only modify a browser window which you created via window.open() , meaning it is limited to popup windows. As a result you can adjust the size & position of your popup windows, but not adjust your main window itself.
2. There is a bug (security feature) which prevents you from programatically moving a popup window onto a different screen. Example: you open the main window on your second screen. Open a docked window. Drag the main window onto your first screen. The docked window will stay on the second screen, even though the positions are correct. In case you manually drag the popup window onto your first screen, it will work again.

I did not dive deeper into these 2 issues, there might be browser flags which allow it.