Generate evolution scripts from which Play runs Ebean's statements

README.md

@@ -57,6 +57,47 @@ The Play Ebean plugin supports several different versions of Play and Ebean.
 
 We also recommend using the payintech fork: https://github.com/payintech/play-ebean
 
+## Testing the evolution scripts on real databases
+
+`EbeanEvolutionScriptDatabasesTest` applies the evolution script that Play Ebean generates with Play's evolutions, uses the models, and reverts the script again. It always runs on H2 and SQLite. On Postgres, MySQL, MariaDB, SQL Server and Oracle, it only runs when these environment variables are set, and is skipped otherwise, like on CI:
+
+- `PLAY_EBEAN_TEST_<DB>_URL`: the JDBC URL
+- `PLAY_EBEAN_TEST_<DB>_USER`
+- `PLAY_EBEAN_TEST_<DB>_PASSWORD`, or `PLAY_EBEAN_TEST_<DB>_PASSWORD_FILE` to read the password from a file
+
+`<DB>` is one of `POSTGRES`, `MYSQL`, `MARIADB`, `SQLSERVER` and `ORACLE`. Use an empty database that you can throw away, because the test creates and drops its tables, and Ebean's stored procedures and history tables.
+
+For example, start the databases with Docker:
+
+```bash
+docker run -d --name play-ebean-test-postgres -e POSTGRES_PASSWORD=test -e POSTGRES_DB=play_ebean_test -p 127.0.0.1:15432:5432 postgres:17
+docker run -d --name play-ebean-test-mysql -e MYSQL_ROOT_PASSWORD=test -e MYSQL_DATABASE=play_ebean_test -p 127.0.0.1:13306:3306 mysql:8.4
+docker run -d --name play-ebean-test-mariadb -e MARIADB_ROOT_PASSWORD=test -e MARIADB_DATABASE=play_ebean_test -p 127.0.0.1:13307:3306 mariadb:11.4
+docker run -d --name play-ebean-test-oracle -e ORACLE_PASSWORD=test -e APP_USER=play_ebean_test -e APP_USER_PASSWORD=test -p 127.0.0.1:11521:1521 gvenzl/oracle-free:23-slim
+# The SQL Server images are only available for x86-64
+docker run -d --name play-ebean-test-sqlserver -e ACCEPT_EULA=Y -e MSSQL_SA_PASSWORD=Test-1234 -p 127.0.0.1:11433:1433 mcr.microsoft.com/mssql/server:2022-latest
+```
+
+When SQL Server has started, create its database:
+
+```bash
+docker exec play-ebean-test-sqlserver /opt/mssql-tools18/bin/sqlcmd -C -S localhost -U sa -P Test-1234 -Q 'CREATE DATABASE play_ebean_test'
+```
+
+Then set the variables for the databases to test, and run the test once the databases are ready (Oracle takes a while on its first start):
+
+```bash
+export PLAY_EBEAN_TEST_POSTGRES_URL='jdbc:postgresql://127.0.0.1:15432/play_ebean_test' PLAY_EBEAN_TEST_POSTGRES_USER=postgres PLAY_EBEAN_TEST_POSTGRES_PASSWORD=test
+export PLAY_EBEAN_TEST_MYSQL_URL='jdbc:mysql://127.0.0.1:13306/play_ebean_test' PLAY_EBEAN_TEST_MYSQL_USER=root PLAY_EBEAN_TEST_MYSQL_PASSWORD=test
+export PLAY_EBEAN_TEST_MARIADB_URL='jdbc:mariadb://127.0.0.1:13307/play_ebean_test' PLAY_EBEAN_TEST_MARIADB_USER=root PLAY_EBEAN_TEST_MARIADB_PASSWORD=test
+export PLAY_EBEAN_TEST_ORACLE_URL='jdbc:oracle:thin:@//127.0.0.1:11521/FREEPDB1' PLAY_EBEAN_TEST_ORACLE_USER=play_ebean_test PLAY_EBEAN_TEST_ORACLE_PASSWORD=test
+export PLAY_EBEAN_TEST_SQLSERVER_URL='jdbc:sqlserver://127.0.0.1:11433;databaseName=play_ebean_test;encrypt=true;trustServerCertificate=true' PLAY_EBEAN_TEST_SQLSERVER_USER=sa PLAY_EBEAN_TEST_SQLSERVER_PASSWORD=Test-1234
+
+sbt --server 'core/testOnly play.db.ebean.EbeanEvolutionScriptDatabasesTest'
+```
+
+With `--server`, sbt starts on its own and sees the variables. Without it, sbt can connect to an sbt server that's already running, which doesn't see them.
+
 ## Releasing a new version
 
 See https://github.com/playframework/.github/blob/main/RELEASING.md

docs/manual/working/javaGuide/main/sql/JavaEbean.md

@@ -72,6 +72,10 @@ Ebean supports one read-only data source per Ebean server. To spread the read-on
 
 In dev and test mode, Play Ebean generates the evolution script `conf/evolutions/<server>/1.sql` of each Ebean server from its models, so that Play's [evolutions](https://www.playframework.com/documentation/latest/Evolutions) can create the database schema. Whenever the models change, Play Ebean updates the script, as long as it starts with the `-- Created by Ebean DDL` comment. To write the evolutions yourself, remove this comment (both lines), then Play Ebean doesn't touch the file anymore.
 
+Some of Ebean's statements contain semicolons, like the stored procedures that Ebean creates on MySQL, MariaDB and SQL Server, or the triggers for [`@History`](https://ebean.io/docs/features/history). Play Ebean splits the DDL like Ebean itself does, and writes such statements between `-- !split-semicolon: never` and `-- !split-semicolon: always` comments, so that Play runs each of them as a whole (see the [evolutions scripts](https://www.playframework.com/documentation/latest/Evolutions#Evolutions-scripts)).
+
+As soon as a production database uses the generated script, remove the comment or turn off generating the script, before you upgrade Ebean or Play Ebean, or run the application in dev mode or its tests again. Write further changes as new evolution scripts (`2.sql` and so on). Otherwise, a dev mode start or a test run can change `1.sql`, e.g. after upgrading Ebean or Play Ebean, and Play can then only apply the changed evolution by reverting the applied one with the down script stored in the database. Removing the comment doesn't change the evolution itself. This especially applies when upgrading to Play Ebean 9, which writes the scripts differently where Ebean's DDL contains statements with semicolons, e.g. on MySQL, MariaDB and SQL Server, on Oracle with sequences, or with `@History`.
+
 You can turn off generating the scripts for all Ebean servers, and override that for single Ebean servers:
 
 ```properties

docs/project/plugins.sbt

@@ -9,7 +9,7 @@ resolvers += Resolver.sonatypeCentralSnapshots
 addSbtPlugin(
   "org.playframework" % "play-docs-sbt-plugin" % sys.props.getOrElse(
     "play.version",
-    "3.1.0-M10-e1f3c2a9-SNAPSHOT"
+    "3.1.0-M10-3968a052-SNAPSHOT"
   )
 )
 

play-ebean/src/main/java/play/db/ebean/EbeanDynamicEvolutions.java

@@ -5,29 +5,42 @@
 package play.db.ebean;
 
 import io.ebean.Database;
+import io.ebean.ddlrunner.DdlDetect;
+import io.ebean.ddlrunner.DdlParser;
 import io.ebeaninternal.api.SpiEbeanServer;
 import io.ebeaninternal.dbmigration.model.CurrentModel;
 import jakarta.inject.Inject;
 import jakarta.inject.Singleton;
 import java.io.File;
 import java.io.IOException;
+import java.io.StringReader;
 import java.nio.charset.StandardCharsets;
 import java.nio.file.Files;
 import java.util.HashMap;
+import java.util.List;
 import java.util.Map;
 import java.util.concurrent.CompletableFuture;
 import java.util.function.Supplier;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
 import play.Environment;
 import play.api.db.evolutions.DynamicEvolutions;
+import play.api.db.evolutions.Evolution;
 import play.api.db.evolutions.Evolutions$;
 import play.api.db.evolutions.EvolutionsConfig;
+import play.api.db.evolutions.UpScript;
 import play.inject.ApplicationLifecycle;
 import play.inject.Injector;
+import scala.jdk.javaapi.CollectionConverters;
 
 /** A Play module that automatically manages Ebean configuration. */
 @Singleton
 public class EbeanDynamicEvolutions extends DynamicEvolutions {
 
+  /** How Ebean resolves {@code ${timestamp}} in the DDL header. */
+  private static final Pattern TIMESTAMP =
+      Pattern.compile("\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}(\\.\\d+)?Z");
+
   private final EbeanConfig config;
   private final Environment environment;
 
@@ -142,16 +155,115 @@ private static String generateScript(SpiEbeanServer spiServer) {
       return null;
     }
 
+    String header = spiServer.config().getDdlHeader();
+
     return "-- Created by Ebean DDL\r\n"
         + "-- To stop Ebean DDL generation, remove this comment (both lines) and start using"
         + " Evolutions\r\n"
         + "\r\n"
         + "-- !Ups\r\n"
         + "\r\n"
-        + ups
+        + evolutionScript(ups, header)
         + "\r\n"
         + "-- !Downs\r\n"
         + "\r\n"
-        + downs;
+        + evolutionScript(downs, header);
+  }
+
+  /**
+   * Like {@link #evolutionScript(String)}, but keeps the DDL header that Ebean puts first as it is,
+   * which, unlike the DDL that Ebean generates, can contain anything. If the DDL doesn't start with
+   * the header, all of it stays as it is.
+   *
+   * @param header the configured header, which Ebean returns with another {@code ${timestamp}} each
+   *     time
+   */
+  static String evolutionScript(String ddl, String header) {
+    if (header == null || header.isEmpty()) {
+      return evolutionScript(ddl);
+    }
+    // Ebean puts the header and a line break first, and resolving its placeholders adds no lines
+    int end = -1;
+    for (int line = 0; line < header.split("\n", -1).length; line++) {
+      end = ddl.indexOf('\n', end + 1);
+      if (end == -1) {
+        return ddl;
+      }
+    }
+    String prefix = ddl.substring(0, end + 1);
+    if (!withoutTimestamps(prefix).equals(withoutTimestamps(header + "\n"))) {
+      return ddl;
+    }
+    return prefix + evolutionScript(ddl.substring(end + 1));
+  }
+
+  private static String withoutTimestamps(String header) {
+    return TIMESTAMP.matcher(header).replaceAll("");
+  }
+
+  /**
+   * Returns Ebean's DDL in a form, from which Play's evolutions run the statements that Ebean
+   * itself would run.
+   *
+   * <p>Ebean's DDL can contain statements with semicolons, like stored procedures or triggers, and
+   * separates those with its own conventions, like {@code delimiter $$} or {@code GO}. Play however
+   * splits on every semicolon. So if Play wouldn't run Ebean's statements, the DDL is split into
+   * Ebean's statements, and those with semicolons are written between {@code !split-semicolon}
+   * comments. Otherwise, the DDL stays as it is.
+   */
+  static String evolutionScript(String ddl) {
+    List<String> statements = ebeanStatements(ddl);
+    if (comparable(playStatements(ddl)).equals(comparable(statements))) {
+      return ddl;
+    }
+    StringBuilder script = new StringBuilder();
+    for (String statement : statements) {
+      if (statement.contains(";")) {
+        script
+            .append("-- !split-semicolon: never\n")
+            .append(statement)
+            .append("\n-- !split-semicolon: always\n;\n\n");
+      } else {
+        script.append(statement).append(";\n\n");
+      }
+    }
+    return script.toString();
+  }
+
+  /** Returns the statements of the DDL that Ebean's DdlRunner would run. */
+  static List<String> ebeanStatements(String ddl) {
+    return new DdlParser(DdlDetect.NONE)
+        .parse(new StringReader(ddl)).stream()
+            // Like DdlRunner, which removes a trailing ; or / from each statement
+            .map(String::trim)
+            .map(
+                sql ->
+                    sql.endsWith(";") || sql.endsWith("/")
+                        ? sql.substring(0, sql.length() - 1)
+                        : sql)
+            .map(String::trim)
+            .filter(sql -> !sql.isEmpty())
+            .collect(Collectors.toList());
+  }
+
+  /** Returns the statements that Play's evolutions would run from the DDL. */
+  static List<String> playStatements(String ddl) {
+    // Play trims the lines of the evolution scripts it reads
+    String sql = ddl.lines().map(String::trim).collect(Collectors.joining("\n"));
+    return CollectionConverters.asJava(new UpScript(new Evolution(1, sql, "")).statements());
+  }
+
+  /** Ignores indentation, empty lines and comment lines, which don't change the statements. */
+  static List<String> comparable(List<String> statements) {
+    return statements.stream()
+        .map(
+            statement ->
+                statement
+                    .lines()
+                    .map(String::trim)
+                    .filter(line -> !line.isEmpty() && !line.startsWith("--"))
+                    .collect(Collectors.joining("\n")))
+        .filter(statement -> !statement.isEmpty())
+        .collect(Collectors.toList());
   }
 }