expansão "limitada" de um grupo de visualização de tabela em árvore quando a tecla Alt é mantida pressionada

Sua ideia de lidar com Behaviora lógica é válida. Mas, infelizmente, isso não ajuda. Como o comportamento da célula é estritamente limitado pela camada de célula, será muito difícil controlar a lógica que ocorre internamente antes de tentar assumir o controle (mesmo com a implementação personalizada de Célula, Pele e Comportamento).

Dito isso, você pode procurar outras maneiras de ajustar isso. Uma delas que me vem à mente é implementar um despachador de eventos personalizado e desviar o processamento de eventos quando as condições desejadas forem atendidas.

A ideia geral desta abordagem é:

• Implemente um despachador de eventos personalizado na célula da tabela e rastreie seus eventos.
• Depois de atingir a condição desejada de mouse pressionado com alt pressionado, você verifica se o evento ocorreu no nó de divulgação.
• Se isso ocorrer no nó de divulgação, desvie/suprima o fluxo de eventos e execute sua lógica personalizada (reduzindo os itens filhos e expandindo o item atual)

Abaixo está o código da célula personalizada conforme a abordagem acima.

class CustomTreeTableCell<T, S> extends TreeTableCell<T, S> {

    public CustomTreeTableCell() {
        // Keep the reference of original dispatcher
        final EventDispatcher orig = getEventDispatcher();

        // Set a new event dispatcher
        setEventDispatcher((event, tail) -> {
            // If the event is mouse pressed and is alt down, then do this logic otherwise proceed with usual orig dispatch.
            if (event instanceof MouseEvent me && me.getEventType() == MouseEvent.MOUSE_PRESSED && me.isAltDown()) {
                Node disclosureNode = getTableRow().getDisclosureNode();
                // Check if disclosure node exists and visible
                if (disclosureNode != null && disclosureNode.isVisible()) {
                    // Check if the mouse press happened on the node by checking the mouse point on node bounds.
                    if (disclosureNode.localToScene(disclosureNode.getLayoutBounds()).contains(me.getSceneX(), me.getSceneY())) {
                        // When you reach this point, you do your logic. ie. collapse all its child items and expand this item.
                        final TreeItem<T> treeItem = getTableRow().getTreeItem();
                        treeItem.getChildren().forEach(this::collapse);
                        treeItem.setExpanded(true);

                        // Ensure to consume the event by returning null;
                        return null;
                    }
                }
            }

            // If it is not your desired event, let the original dispatcher do its stuff.
            return orig.dispatchEvent(event, tail);
        });
    }

    /**
     * Recursive method to collapse all child items and provided item.
     *
     * @param item tree item
     */
    private void collapse(TreeItem<?> item) {
        item.setExpanded(false);
        item.getChildren().forEach(this::collapse);
    }

    @Override
    protected void updateItem(S item, boolean empty) {
        super.updateItem(item, empty);
        setText(empty || item == null ? "" : item.toString());
    }
}

Observação: a abordagem de verificar o ponteiro do mouse nos limites do nó de divulgação não é algo novo. A lógica interna do método TreeTableCellBehaviortambém faz isso handleDisclosureNode.

Abaixo está a demonstração completa com seu exemplo:

import javafx.application.Application;
import javafx.beans.property.SimpleStringProperty;
import javafx.beans.property.StringProperty;
import javafx.event.EventDispatcher;
import javafx.scene.Node;
import javafx.scene.Scene;
import javafx.scene.control.TreeItem;
import javafx.scene.control.TreeTableCell;
import javafx.scene.control.TreeTableColumn;
import javafx.scene.control.TreeTableView;
import javafx.scene.input.MouseEvent;
import javafx.scene.layout.Pane;
import javafx.scene.layout.VBox;
import javafx.stage.Stage;

public class TreeTableView_Demo extends Application {
    @Override
    public void start(Stage stage) {
        Pane rootNode = new VBox();
        Scene scene = new Scene(rootNode, 400, 300);
        stage.setTitle("TreeTableView Demo");
        stage.setScene(scene);

        TreeTableView<Car> treeTableView = new TreeTableView<>();

        TreeTableColumn<Car, String> treeTableColumn1 = new TreeTableColumn<>("Brand");
        treeTableColumn1.setCellFactory(p -> new CustomTreeTableCell<>());
        treeTableColumn1.setPrefWidth(120);

        TreeTableColumn<Car, String> treeTableColumn2 = new TreeTableColumn<>("Model");
        treeTableColumn2.setPrefWidth(120);

        treeTableColumn1.setCellValueFactory(param -> param.getValue().getValue().brandProperty());
        treeTableColumn2.setCellValueFactory(param -> param.getValue().getValue().modelProperty());

        treeTableView.getColumns().add(treeTableColumn1);
        treeTableView.getColumns().add(treeTableColumn2);

        TreeItem<Car> mercedes1 = new TreeItem<>(new Car("Mercedes", "SL500"));
        TreeItem<Car> mercedes2 = new TreeItem<>(new Car("Mercedes", "SL500 AMG"));
        TreeItem<Car> mercedes3 = new TreeItem<>(new Car("Mercedes", "CLA 200"));

        TreeItem<Car> mercedes = new TreeItem<>(new Car("Mercedes", "..."));
        mercedes.getChildren().add(mercedes1);
        mercedes.getChildren().add(mercedes2);

        TreeItem<Car> audi1 = new TreeItem<>(new Car("Audi", "A1"));
        TreeItem<Car> audi2 = new TreeItem<>(new Car("Audi", "A5"));
        TreeItem<Car> audi3 = new TreeItem<>(new Car("Audi", "A7"));

        TreeItem<Car> audi = new TreeItem<>(new Car("Audi", "..."));
        audi.getChildren().add(audi1);
        audi.getChildren().add(audi2);
        audi.getChildren().add(audi3);

        TreeItem<Car> cars = new TreeItem<>(new Car("Cars", "..."));
        cars.getChildren().add(audi);
        cars.getChildren().add(mercedes);

        treeTableView.setRoot(cars);

        rootNode.getChildren().add(treeTableView);

        stage.show();
    }

    public static void main(String[] args) {
        launch();
    }

    class CustomTreeTableCell<T, S> extends TreeTableCell<T, S> {

        public CustomTreeTableCell() {
            // Keep the reference of original dispatcher
            final EventDispatcher orig = getEventDispatcher();
            // Set a new event dispatcher
            setEventDispatcher((event, tail) -> {
                // If the event is mouse pressed and is alt down, then do this logic otherwise proceed with usual orig dispatch.
                if (event instanceof MouseEvent me && me.getEventType() == MouseEvent.MOUSE_PRESSED && me.isAltDown()) {
                    Node disclosureNode = getTableRow().getDisclosureNode();
                    // Check if disclosure node exists and visible
                    if (disclosureNode != null && disclosureNode.isVisible()) {
                        // Check if the mouse press happened on the node by checking the mouse point on node bounds.
                        if (disclosureNode.localToScene(disclosureNode.getLayoutBounds()).contains(me.getSceneX(), me.getSceneY())) {
                            // When you reach this point, you do your logic. ie. collapse all its child items and expand this item.
                            final TreeItem<T> treeItem = getTableRow().getTreeItem();
                            treeItem.getChildren().forEach(this::collapse);
                            treeItem.setExpanded(true);

                            // Ensure to consume the event by returning null;
                            return null;
                        }
                    }
                }

                // If it is not your desired event, let the original dispatcher do its stuff.
                return orig.dispatchEvent(event, tail);
            });
        }

        /**
         * Recursive method to collapse all child items and provided item.
         *
         * @param item tree item
         */
        private void collapse(TreeItem<?> item) {
            item.setExpanded(false);
            item.getChildren().forEach(this::collapse);
        }

        @Override
        protected void updateItem(S item, boolean empty) {
            super.updateItem(item, empty);
            setText(empty || item == null ? "" : item.toString());
        }
    }
}

class Car {
    private StringProperty brand = new SimpleStringProperty();
    private StringProperty model = new SimpleStringProperty();

    public Car(String brand, String model) {
        this.brand.set(brand);
        this.model.set(model);
    }

    public String getBrand() {
        return brand.get();
    }

    public StringProperty brandProperty() {
        return brand;
    }

    public void setBrand(String brand) {
        this.brand.set(brand);
    }

    public String getModel() {
        return model.get();
    }

    public StringProperty modelProperty() {
        return model;
    }

    public void setModel(String model) {
        this.model.set(model);
    }
}

Isso é um pouco complicado, mas... você pode ouvir a tecla Alt sendo pressionada e usá-la quando ocorrer um evento de expansão de ramificação :

import javafx.application.Application;
import javafx.beans.property.SimpleStringProperty;
import javafx.beans.property.StringProperty;
import javafx.scene.Scene;
import javafx.scene.Node;
import javafx.scene.Parent;
import javafx.scene.control.TreeItem;
import javafx.scene.control.TreeTableColumn;
import javafx.scene.control.TreeTableRow;
import javafx.scene.control.TreeTableView;
import javafx.scene.layout.Pane;
import javafx.scene.layout.VBox;
import javafx.stage.Stage;
import javafx.scene.input.KeyEvent;
import javafx.scene.input.KeyCode;

public class TreeTablePartialExpansionDemo extends Application {
    static class Car {
        private StringProperty brand = new SimpleStringProperty();
        private StringProperty model = new SimpleStringProperty();

        public Car(String brand, String model) {
            this.brand.set(brand);
            this.model.set(model);
        }

        public String getBrand() {
            return brand.get();
        }

        public StringProperty brandProperty() {
            return brand;
        }

        public void setBrand(String brand) {
            this.brand.set(brand);
        }

        public String getModel() {
            return model.get();
        }

        public StringProperty modelProperty() {
            return model;
        }

        public void setModel(String model) {
            this.model.set(model);
        }
    }

    private boolean altDown;

    @Override
    public void start(Stage stage) {
        Pane rootNode = new VBox();
        Scene scene = new Scene(rootNode, 400, 300);
        stage.setTitle("TreeTableView Demo");
        stage.setScene(scene);

        TreeTableView<Car> treeTableView = new TreeTableView<>();

        TreeTableColumn<Car, String> treeTableColumn1 = new TreeTableColumn<>("Brand");
        treeTableColumn1.setPrefWidth(120);
        TreeTableColumn<Car, String> treeTableColumn2 = new TreeTableColumn<>("Model");
        treeTableColumn2.setPrefWidth(120);

        treeTableColumn1.setCellValueFactory(param -> param.getValue().getValue().brandProperty());
        treeTableColumn2.setCellValueFactory(param -> param.getValue().getValue().modelProperty());

        treeTableView.getColumns().add(treeTableColumn1);
        treeTableView.getColumns().add(treeTableColumn2);

        TreeItem<Car> mercedes1 = new TreeItem<>(new Car("Mercedes", "SL500"));
        TreeItem<Car> mercedes2 = new TreeItem<>(new Car("Mercedes", "SL500 AMG"));
        TreeItem<Car> mercedes3 = new TreeItem<>(new Car("Mercedes", "CLA 200"));

        TreeItem<Car> mercedes = new TreeItem<>(new Car("Mercedes", "..."));
        mercedes.getChildren().add(mercedes1);
        mercedes.getChildren().add(mercedes2);

        TreeItem<Car> audi1 = new TreeItem<>(new Car("Audi", "A1"));
        TreeItem<Car> audi2 = new TreeItem<>(new Car("Audi", "A5"));
        TreeItem<Car> audi3 = new TreeItem<>(new Car("Audi", "A7"));

        TreeItem<Car> audi = new TreeItem<>(new Car("Audi", "..."));
        audi.getChildren().add(audi1);
        audi.getChildren().add(audi2);
        audi.getChildren().add(audi3);

        TreeItem<Car> cars = new TreeItem<>(new Car("Cars", "..."));
        cars.getChildren().add(audi);
        cars.getChildren().add(mercedes);

        treeTableView.setRoot(cars);
        rootNode.getChildren().add(treeTableView);

        treeTableView.getScene().addEventFilter(KeyEvent.KEY_PRESSED, e -> {
            if (e.getCode() == KeyCode.ALT) {
                altDown = true;
            }
        });
        treeTableView.getScene().addEventFilter(KeyEvent.KEY_RELEASED, e -> {
            if (e.getCode() == KeyCode.ALT) {
                altDown = false;
            }
        });
        treeTableView.getScene().getWindow().focusedProperty().subscribe(
            focused -> altDown &= focused);
        treeTableView.getRoot().addEventFilter(TreeItem.branchExpandedEvent(),
            e -> {
                if (altDown) {
                    e.getTreeItem().getChildren().forEach(
                        item -> item.setExpanded(false));
                }
            });

        stage.show();
    }

    public static void main(String[] args) {
        launch();
    }
}

Adicionei um private boolean altDown;campo, e adicionei isso logo antes de mostrar o Palco:

treeTableView.getScene().addEventFilter(KeyEvent.KEY_PRESSED, e -> {
    if (e.getCode() == KeyCode.ALT) {
        altDown = true;
    }
});
treeTableView.getScene().addEventFilter(KeyEvent.KEY_RELEASED, e -> {
    if (e.getCode() == KeyCode.ALT) {
        altDown = false;
    }
});
treeTableView.getScene().getWindow().focusedProperty().subscribe(
    focused -> altDown &= focused);
treeTableView.getRoot().addEventFilter(TreeItem.branchExpandedEvent(),
    e -> {
        if (altDown) {
            e.getTreeItem().getChildren().forEach(
                item -> item.setExpanded(false));
        }
    });

As três primeiras instruções rastreiam se a tecla Alt está pressionada. A última instrução intercepta eventos de expansão e, se altDownfor verdadeira, colapsa os filhos da raiz.