Release/2.5.0

[qiaoyuang]

All

• Update Kotlin's version to 2.4.21
• Rebuild the SQLite static libraries in libs, which the native tests on Linux and Windows link, from SQLite 3.51.1 with FTS3, FTS4 (with the enhanced query syntax), FTS5, R*Tree, Geopoly, the math functions, dbstat and column metadata enabled, and without debug information, which held paths of the machine that built them

sqllin-dsl

• New API: the opt-in annotation @PlatformDependentSQLiteAPI, which marks the APIs that only work with some SQLite versions or compile-time options, which some platforms SQLlin supports don't have, such as Android's system SQLite at lower API levels. Using one is a compile error unless it is accepted with @OptIn(PlatformDependentSQLiteAPI::class)
• Breaking change: RENAME_COLUMN, which needs SQLite 3.25.0 (Android API 30), and DROP_COLUMN, which needs SQLite 3.35.0 (Android API 34), are marked with @PlatformDependentSQLiteAPI, so code using them has to opt in
• New experimental DSL API: joins of relations. FROM(relation) starts a join, which INNER_JOIN, LEFT_OUTER_JOIN, RIGHT_OUTER_JOIN, FULL_OUTER_JOIN, CROSS_JOIN and the NATURAL joins extend, each with its own ON or USING, as in (FROM(PersonTable) LEFT_OUTER_JOIN BookTable ON (PersonTable.id EQ BookTable.authorId)) SELECT listOf(...). A join is selected from with a projection, a clause function such as WHERE<R>(...), or result columns, which can now select from a join, and the type its rows are read into is checked against the joined relations when the statement is built: every property has to name a column of one of them, of a single one unless USING or a NATURAL join merges them, and has to be nullable where an outer join can leave its column or expression NULL. Columns are written with their relations' names, and an observed query of a join watches all its relations. RIGHT_OUTER_JOIN, FULL_OUTER_JOIN and their NATURAL forms need SQLite 3.39.0 (Android API 34), so they are marked with @PlatformDependentSQLiteAPI
• The joins written as clauses of SELECT, such as table SELECT INNER_JOIN<R>(other) ON (...), which don't check the type their rows are read into, will be removed in the next version after 2.5.0: use the joins of relations instead
• CREATE_INDEX and CREATE_UNIQUE_INDEX document that a function of the table's columns makes an expression index, as lower(email) for case-insensitive lookups or uniqueness, and reject a column or function of another table
• New experimental DSL API: IF NOT EXISTS and IF EXISTS, with CREATE_IF_NOT_EXISTS, CREATE_INDEX_IF_NOT_EXISTS, CREATE_UNIQUE_INDEX_IF_NOT_EXISTS, CREATE_VIEW_IF_NOT_EXISTS, DROP_IF_EXISTS and DROP_INDEX_IF_EXISTS, which do nothing when the table, FTS table, index or view exists, or doesn't, so that a migration step can run more than once
• New experimental DSL API: DROP_INDEX(name), for SQL syntax DROP INDEX, which drops an index that CREATE_INDEX or CREATE_UNIQUE_INDEX created. The guide to modifying the database now describes indexes
• New experimental DSL API: partial indexes. WHERE after CREATE_INDEX, CREATE_UNIQUE_INDEX or their IF_NOT_EXISTS forms indexes only the rows that satisfy a condition, as in UserTable.CREATE_UNIQUE_INDEX("idx_user_email", UserTable.email) WHERE (UserTable.isDeleted IS false). SQLite allows neither parameters nor subqueries there, so the values of the condition are written into the SQL, and a subquery is rejected with an IllegalArgumentException. To support it, these functions now return an IndexStatement, where they returned Unit; this is source-compatible, but a library compiled against an earlier version has to be recompiled
• New experimental DSL API: DISTINCT(element), for aggregate functions of the distinct values, as in count(DISTINCT(author)) for count(DISTINCT book.author). count, avg, sum and group_concat take it, and group_concat separates the distinct values with commas, as SQLite allows no other separator with DISTINCT. What it returns, a Distinct, can't be used anywhere else, which is checked at compile time
• New experimental DSL API: ESCAPE after LIKE, for SQL syntax LIKE pattern ESCAPE character, as in name LIKE "100\\%%" ESCAPE '\\', after which %, _ and the escape character match only themselves. To support it, LIKE now returns a LikeCondition, which is a SelectCondition; this is source-compatible, but a library compiled against an earlier version has to be recompiled
• New experimental DSL API: VACUUM() and VACUUM_INTO(file), for SQL syntax VACUUM and VACUUM INTO, which rebuild the database file into the minimum disk space, or write a vacuumed copy of the database to another file, a consistent backup of a database in use. Both throw an IllegalStateException inside a transaction, where SQLite can't vacuum. VACUUM_INTO needs SQLite 3.27.0 (Android API 30), so it is marked with @PlatformDependentSQLiteAPI
• New experimental DSL API: expressions. +, -, *, /, the unary - and, of integer types, % compute with numbers of the same type, and with values of their type, as in pages + 1, giving a number of that type, and + concatenates strings, SQL's ||. They work in result columns, conditions and ORDER BY. SQLite gives NULL for a division by zero, so / and % can be NULL unless the divisor is a value other than zero. literal(value) makes a value an expression, and CAST(x AS type) converts an expression to another type, named as SQLlin names the columns of that Kotlin type: TINYINT, SMALLINT, INT, BIGINT, FLOAT, DOUBLE, TEXT and BLOB. The values in an expression are written into its SQL, so that it can be part of a view, a trigger or an index, which SQLite doesn't let take parameters. A literal on its own in ORDER BY or GROUP BY is rejected, as SQLite would read it as the number of a result column
• New experimental DSL API: more functions, whose arguments are expressions: coalesce, ifnull and nullif, the scalar max and min of several expressions, hex, quote, unicode, randomblob, zeroblob, trim, ltrim and rtrim with the characters to remove, substr from a position, printf of several values, substr, replace and instr of expressions, round to an integer, the aggregate total, and the date and time functions date, time, datetime, julianday and strftime. A function can only be NULL where its result can: coalesce where all its arguments can, and a date and time function always, but datetime("now"). sign (SQLite 3.35.0, Android API 34), unixepoch (SQLite 3.38.0, Android API 34) and the math functions ceil, floor, trunc, sqrt, exp, ln, log10, log2, pow, mod and pi, which Android's SQLite has at no API level, are marked with @PlatformDependentSQLiteAPI
• New experimental DSL API: more expression syntax: CASE(WHEN(condition) THEN value, ..., ELSE = value), SQL's searched CASE, which is NULL where ELSE is omitted; scalar subqueries, select[R::property], a SELECT of a single column used as a value, which is checked when the statement is built, and whose tables an observed query watches; IS and ISNOT of two expressions, the null-safe comparisons, which do what IS DISTINCT FROM does on every SQLite version; the COLLATE operator with the collations BINARY, NOCASE and RTRIM; and the bitwise operators and, or, inv(), shl and shr of Int and Long
• New experimental DSL API: SET and INSERT of expressions. SET(listOf(expression AS Row::property)) sets columns to expressions, as in table UPDATE SET(listOf((visits + 1) AS Person::visits)) WHERE (...), and INSERT, INSERT_OR_IGNORE and INSERT_OR_REPLACE take a row of expressions, table INSERT listOf(literal("purge") AS Audit::action, datetime("now") AS Audit::at), for SQL syntax INSERT INTO ... VALUES, leaving the other columns to their default values or NULL. AS checks the type of each expression at compile time, and the statement checks, when it is built, that an expression that can be NULL is only written to a nullable column, that it isn't an aggregate function, and that it only reads the columns of the updated table, or none for INSERT, where SQLite allows none
• New experimental DSL API: triggers. CREATE_TRIGGER(name) AFTER DELETE ON PersonTable WHEN (condition) BEGIN { ... }, for SQL syntax CREATE TRIGGER, with BEFORE and AFTER, the events INSERT, DELETE, UPDATE and UPDATE_OF(columns), and CREATE_TRIGGER_IF_NOT_EXISTS. new(column) and old(column) read the row being written, and a row the trigger doesn't have, which SQLite would only report when the trigger fires, is rejected when the statement is built. The body inserts, updates and deletes rows, with expressions, and RAISE(ABORT, message), RAISE(FAIL, message), RAISE(ROLLBACK, message) and RAISE(IGNORE) stop the statement that fired the trigger or skip its row. The values of a trigger are written into its SQL, as SQLite doesn't let a trigger take parameters. A trigger can't be on an FTS table or a view; INSTEAD OF triggers on views are left to another item of the roadmap. DROP_TRIGGER(name) and DROP_TRIGGER_IF_EXISTS(name) drop one. An observed query of an FTS table now counts the writes to any table that has triggers as changes of the FTS table too, as a trigger can write it
• New experimental DSL API: table INSERT DEFAULT_VALUES, for SQL syntax INSERT INTO ... DEFAULT VALUES, which inserts a row of the default values of all columns, with INSERT_OR_IGNORE and INSERT_OR_REPLACE too
• New experimental DSL API: ANALYZE(), ANALYZE(table) and ANALYZE(name), for SQL syntax ANALYZE, which gathers the statistics the query planner chooses indexes with; REINDEX(), REINDEX(table), REINDEX(name) and REINDEX(collation), for REINDEX, which rebuilds indexes; and PRAGMA_OPTIMIZE(), for PRAGMA optimize, which needs SQLite 3.18.0 (Android API 26), so it is marked with @PlatformDependentSQLiteAPI
• New experimental DSL API: the orders of the columns of an index, ASC and DESC, as in PersonTable.CREATE_INDEX("idx_person_age", PersonTable.age to DESC), with CREATE_UNIQUE_INDEX and the IF_NOT_EXISTS forms too
• New experimental DSL API: NULLS FIRST and NULLS LAST in ORDER BY, as in ORDER_BY(age to ASC NULLS LAST), which need SQLite 3.30.0 (Android API 31), so they are marked with @PlatformDependentSQLiteAPI. To support them, ORDER BY takes an Ordering, which OrderByWay is, rather than an OrderByWay; this is source-compatible, but a library compiled against an earlier version has to be recompiled
• New API: deferred = true on @References and @ForeignKeyGroup, which makes a foreign key DEFERRABLE INITIALLY DEFERRED: checked when the transaction commits, rather than after each statement
• New experimental API: the onConflict of @PrimaryKey, @CompositePrimaryKey, @Unique and @CompositeUnique, with the enum OnConflict, for the ON CONFLICT clause of the constraint, as in UNIQUE ON CONFLICT REPLACE. A composite key or group takes the one given on any of its properties, and the processor rejects a key or group whose properties give different ones. ABORT, the default, writes no clause
• CREATE_INDEX and CREATE_UNIQUE_INDEX reject an expression with a subquery or an aggregate function, or a function that doesn't always give the same result for a row, such as random(), datetime('now') or a date and time function with the localtime modifier, and the WHERE of a partial index rejects such a function, where SQLite would only fail once the table has rows
• New experimental DSL API: subqueries. select AS SomeView makes a SELECT a derived table, (SELECT ...) AS name, named after the object generated for a @DBView class, which gives it its name and column properties, though no such view has to exist in the database. A derived table is read like a table, in the FROM clause and in joins, which can join a table with itself through a derived table of its rows. IN takes a SELECT of a single column of values of the same kind, which is checked when the statement is built, and EXISTS(select) checks whether a SELECT returns any row; it can refer to the columns of the query it is in. They also work in the WHERE of UPDATE and DELETE. Parameters are bound in the order the SQL has them, and an observed query watches the tables its subqueries read
• New experimental DSL API: NOT(condition), which negates a condition in parentheses, as in NOT(EXISTS(select))
• New experimental API: the repeatable annotation @Check, which adds a CHECK constraint with an SQL expression every row has to satisfy: on a property, to its column, and on a @DBRow class, to the table. constraintName names it, which SQLite reports when a row fails it; SQLite before 3.34, as on Android below API 34, reports an unnamed one by the table's name. The processor rejects an empty expression, a name that isn't an SQL identifier, and two constraints of a table with the same name, and views and FTS tables reject @Check
• New experimental DSL API: FTS4 and FTS3 full-text search tables, which SQLite has on every platform SQLlin supports, Android's included. A @DBRow class annotated with @Fts4 or @Fts3 gets a generated FtsTable object, which CREATE creates with CREATE VIRTUAL TABLE. Its columns are the class's String properties, and a @PrimaryKey property of type Long or Long? named rowid or docid reads and writes the rowid. @Fts4 takes the tokenizer and its arguments, the prefix lengths to index and the columns not to index, and @Fts3 the tokenizer and its arguments. MATCH searches the table or one of its columns, and snippet, offsets and matchinfo describe the matches. An observed query of an FTS table counts its changes from SQLlin's INSERT, UPDATE and DELETE on it, as a virtual table can't have triggers. withName on an FTS table returns an FTS table, so it is rebuilt as a table is
• New experimental DSL API: views. A class annotated with @DBView gets a generated view object, such as AdultView for Adult, which CREATE_VIEW(AdultView) AS (select) creates from a SELECT of its row type, checked at compile time, and DROP(AdultView) drops. SQLite doesn't let a view take parameters, so the SELECT's values are written into the view's SQL. A view is queried like a table, in joins and compound SELECTs too, and an observed query of it watches the tables it reads, which SQLite lists through EXPLAIN. To keep views from being written to, the new Relation is the base of Table and View: SELECT, joins, the SQL functions and the clause elements take a Relation, while INSERT, UPDATE, DELETE, CREATE and ALTER keep taking a Table. This is source-compatible, but on Kotlin/Native a library compiled against an earlier version has to be recompiled
• New experimental API: Database#observe, which turns a SELECT into a Flow of its results, emitted when it is collected and again whenever a statement run through the database changes a table the SELECT reads, if the results differ. A join or a compound SELECT watches all of its tables. The changes are counted by SQLite itself, through TEMP triggers on the observed tables, so rows changed by foreign key actions count, while rolled back transactions and statements that match no rows don't. Changes made through another Database instance, connection or process aren't seen. Using it needs @OptIn(ExperimentalDSLDatabaseAPI::class)
• kotlinx.coroutines is now an api dependency of sqllin-dsl, as Database#observe returns a Flow
• A Database creates the tracker of observed queries when it observes its first query, so that an app that observes none doesn't reach its code, which code shrinkers then remove: on Kotlin/Native, a release executable of an app that only inserts, queries, updates and deletes is about 418 KB smaller, and with R8, the code of sqllin-dsl it keeps is a quarter smaller
• New experimental DSL API: compound SELECTs, with UNION, UNION_ALL, INTERSECT and EXCEPT as infix operators between two SELECTs of the same result type, as in (table SELECT WHERE(age LT 18)) UNION (table SELECT WHERE(age GTE 65)). The result type is checked at compile time, so the SELECTs can combine the rows of different tables through projections, result columns or joins. Chained operators are evaluated from left to right, as in SQL, and parentheses group them. A SELECT that ends with ORDER BY or LIMIT keeps them to itself, and the compound can be followed by its own ORDER BY and LIMIT. Using it needs @OptIn(ExperimentalDSLDatabaseAPI::class)
• DSLDBConfiguration, Database(DSLDBConfiguration) and the DSL APIs CREATE, CREATE_INDEX, CREATE_UNIQUE_INDEX, DROP, ALTER_ADD_COLUMN, ALTER_RENAME_TABLE_TO, RENAME_COLUMN, DROP_COLUMN and PRAGMA_FOREIGN_KEYS are no longer experimental, so using them no longer needs @OptIn(ExperimentalDSLDatabaseAPI::class)
• Deprecated: the UNION {} and UNION_ALL {} blocks, and the DatabaseScope#beginUnion, createUnionSelectStatement and endUnion functions they use. A block returns the table's row type but reads the rows into its first SELECT's result type, so a union of projections, result columns or joins fails with a ClassCastException when its results are used, and a nested block is flattened into the SQL, which changes its meaning unless it is the block's first statement. These bugs won't be fixed: use the infix operators instead. The blocks will be removed in a future version
• Fix documentation: the guide to rebuilding a table didn't drop the views that read it, so on the JVM, Linux and Windows, where SQLite's legacy_alter_table setting is off by default, renaming the new table failed while a view read the dropped one, with "error in view ...: no such table". The guide now drops the views before dropping the old table and creates them again after the rename, which works on every platform, and explains that only where the setting is off do the foreign keys of other tables follow a renamed table regardless of whether foreign keys are enforced
• Fix: a column given to a function, to GROUP BY or to ORDER BY, or selected into a property with AS, was written without its table's name, so in a join of tables that both have a column of that name, SQLite failed with "ambiguous column name". Such columns are now written with their tables' names, as conditions already were, except in the ORDER BY of a compound SELECT, which can only name the columns of its results
• Fix: AND and OR combined conditions without parentheses. They apply in the order they are written, as Kotlin's infix functions do, so (age EQ 1) OR (age EQ 2) AND (name EQ "x") means (age = 1 OR age = 2) AND name = 'x', but SQL gives AND precedence over OR, so SQLite read it as age = 1 OR (age = 2 AND name = 'x') and returned other rows. A condition that combines others with the other operator is now put in parentheses
• Fix: combining a condition with AND or OR added the other condition's parameters to its own, so a condition kept in a variable and used again, alone or combined, bound parameters its SQL doesn't have, which failed, as with an ArrayIndexOutOfBoundsException on the JVM. Combining conditions now leaves their parameters as they were
• Fix: unsigned values were stored inconsistently. INSERT stored a UByte, UShort or UInt as the bits of a signed number on every platform, UPDATE did so on the JVM, but stored it as its number on Kotlin/Native and Android, so a UByte of 200 read back as 200, but compared, ordered and summed as -56 or as 200, depending on how it was written. They are now always stored as their numbers, which SQLite's 64-bit integers hold, so that SQL compares, orders and sums them as Kotlin does, as sum and max of such a column assumed. A ULong keeps the bits of a Long, as no 64-bit signed integer holds the values above Long.MAX_VALUE. Reading is unchanged and reads the old rows right, but in SQL an old value above the signed range stays negative until it is written again, which one UPDATE per column does: UPDATE t SET c = c + 256 WHERE c < 0 for a UByte column, with 65536 for a UShort one and 4294967296 for a UInt one

sqllin-driver

• Fix: the JVM driver bound a UInt, UShort or UByte parameter as the bits of a signed number, unlike the Kotlin/Native driver, and the Android driver bound one written through execSQL as its string. All of them now bind it as its number, and a ULong as the Long of the same bits
• Fix: on the JVM, Kotlin/Native and Android, a transaction whose COMMIT failed, as one does while a deferred foreign key isn't satisfied, wasn't rolled back, but left open in SQLite: the statements after it ran in it, and the next transaction failed to begin, with "cannot start a transaction within a transaction". It is now rolled back. Android's framework lets go of such a transaction and doesn't pass a ROLLBACK through, so the driver raises one with a TEMP trigger

sqllin-processor

• Fix: the generated createSQL of a table didn't escape ", \ and $, which a Kotlin string literal treats specially, so a @Default value holding one could make the generated code fail to compile, or hold another value
• Breaking change: Fix: reading a column's property in SET {}, as in SET { visits = visits + 1 }, compiled, but its getter only returns a placeholder, such as 0 or "", so the column was set to a value computed from it, here 1. The getter is now deprecated with the level ERROR, so reading the property is a compile error, while assigning it is unchanged

🤖 Generated with Claude Code