Repository navigation
New features for 2.5.0 - #129
Merged
Merged
Conversation
MyMovie bridged a database query to androidx.paging by hand, and got it wrong twice before a test with a real Pager found it: an offset-keyed source invalidated by a mediator needs a prevKey, or a refresh leaves everything above the user's position unreachable; with placeholders off, the anchorPosition is relative to the presented list, not an absolute offset; and offsets have to stay on the page size grid, or neighbouring pages overlap. A source provided by SQLlin, built on observable queries, as Room provides LimitOffsetPagingSource, would spare every consumer that. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Table.UNION {} returned a statement typed by the table's row type, but read
the rows with its first SELECT's deserializer, through unchecked casts. A
union of projections, result columns or joins therefore compiled, and failed
with a ClassCastException wherever its results were used; with members of
different types and the same number of columns, values could also land in
the wrong properties without any error, as SQL matches the columns by
position and SQLlin reads them by name. The SQL was the members' SQL joined
with UNION, so a member's ORDER BY or LIMIT was a syntax error unless it was
the last one, where it applied to the whole union; the union itself could
take neither; and a nested block was flattened, so UNION { a; UNION_ALL { b;
c } } ran as (a UNION b) UNION ALL c. Each of these was checked with sqlite3.
The compound operators are now infix functions between two SELECTs, as in
SQL:
(table SELECT WHERE(age LT 18)) UNION (table SELECT WHERE(age GTE 65))
UNION, UNION_ALL, INTERSECT and EXCEPT take a SelectStatement<R> on both
sides, and SelectStatement is invariant, so mixing result types doesn't
compile; the compound reads its rows into that same R. The SELECTs can come
from any tables, through projections, result columns or joins. They become
part of the compound, removed from the statements of their scope, and their
deferred GROUP BY check runs when they are combined.
Chained operators are evaluated from left to right, as SQLite does. SQLite
has no parentheses in a compound, so a member that would mean something
else is written as a subquery, SELECT * FROM (...): one that ends with ORDER
BY, LIMIT or OFFSET, which then apply to it alone, and a compound on the
right, as in a UNION (b UNION_ALL c), which is then combined as a whole.
FinalSelectStatement.isSimple tells a plain SELECT, which needs none, from
one with OFFSET or a block's union. A CompoundSelectStatement can be followed
by another operator, and by ORDER BY and LIMIT, which apply to all of it.
As new DSL API, the operators, CompoundSelectStatement and the ORDER BY and
LIMIT that follow it require @OptIn(ExperimentalDSLDatabaseAPI::class).
The UNION {} and UNION_ALL {} blocks, and the beginUnion,
createUnionSelectStatement and endUnion functions they need for being
inline, are deprecated and not fixed. Their messages, KDoc and the guide say
why and point to the infix operators. The two blocks suppress the
deprecation of the functions they call, which are deprecated with them.
The guide's UNION section becomes one on compound queries, in English and
Chinese, ending with the deprecated block, its bugs and how to replace it.
The roadmap's UNION item moves to Supported.
Tests: testCompoundSelect covers the four operators, with WHERE parameters
whose order matters; rows of two tables projected into one type; that a
combined SELECT no longer runs alone; members with their own ORDER BY and
LIMIT, which fail without the subquery; the compound's ORDER BY, LIMIT and
OFFSET; left-to-right evaluation against grouping, which return 2 and 1 rows
on the same data; result columns; and rejecting a member that needs GROUP BY.
testUnionSelect keeps covering the deprecated block. Mixed result types and
missing parentheses fail to compile, and the opt-in warning appears without
@OptIn, checked with temporary files, as were the guide's examples. jvmTest
(54) and testAndroidHostTest on API 26 and 37 (108) pass. The native test
sources compile for macosArm64, linuxX64, mingwX64 and watchosArm32 but were
not run, as this machine is an Intel Mac.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
setupNativeConfig got the main compilation with `by compilations.getting`
and created the sqlite3 cinterop with `by main.cinterops.creating`, which
Gradle 9.6 deprecates; the build script reported five warnings for them. They
are now getByName("main") and create("sqlite3"), as the deprecation messages
suggest. The cinterop's local val was never read, so it goes.
The cinterop keeps its name, so its tasks are still cinteropSqlite3<Target>.
Rerun from scratch, cinterop, compilation and test linking pass for
macosArm64 and linuxX64, the driver's jvmTest passes, the sqllin-dsl-test
sources compile for macosArm64, and the script compiles without warnings.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DSLDBConfiguration, Database(DSLDBConfiguration), CREATE, CREATE_INDEX, CREATE_UNIQUE_INDEX, DROP, ALTER_ADD_COLUMN, both ALTER_RENAME_TABLE_TO, both RENAME_COLUMN, DROP_COLUMN and PRAGMA_FOREIGN_KEYS no longer carry @ExperimentalDSLDatabaseAPI, so using them no longer needs an opt-in. The marker stays on the compound SELECT operators added in 2.5.0, which are new. The opt-ins only those APIs needed go: the sample's, and the test class's and test functions', except testCompoundSelect's. The installation guide no longer calls DSLDBConfiguration experimental, and CHANGELOG.md says the opt-in can be removed. jvmTest (54) and testAndroidHostTest on API 26 and 37 (108) pass, the sample and the macosArm64 test sources compile, and nothing reports an opt-in warning. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MyMovie bolted change notification onto SQLlin by hand: its data source
announced every write to a tracker keyed by list, which a PagingSource
listened to. Every write had to remember to announce itself, and a write to
the movies table couldn't say which lists showed those movies, so they kept
their old titles.
Database.observe turns a SELECT into a Flow of its results:
val adults: Flow<List<Person>> = database.observe { PersonTable SELECT WHERE(PersonTable.age GTE 18) }
It emits the results when collected, and again whenever a statement run
through the database changes a table the SELECT reads, if the results
differ. The block runs again to build the SELECT each time. Queries run on
Dispatchers.IO unless another context is given.
Each SELECT now records the tables it reads: its table, the tables of a
join, and those of every member of a compound, carried along the chained
clauses.
The changes are counted by SQLite itself, as Room does. InvalidationTracker
creates a TEMP table of counts and, the first time a table is observed, TEMP
triggers on its INSERT, UPDATE and DELETE that add one to its count. So rows
changed by a foreign key action count though no statement names their
table, an UPDATE or DELETE matching no rows doesn't, and a rolled back
transaction rolls its counts back. The counts only grow, rather than being a
flag read and then cleared, as SQLlin can share a connection between
threads and a change between the read and the clear would be lost. A table
stays tracked until the connection closes, so triggers are never removed
while another observer may install them. On Android the counts are read in a
transaction, as a query outside one may run on a reader connection, which
has no TEMP table. After CREATE, DROP, ALTER or PRAGMA, missing triggers are
created again and every tracked table counts as changed.
Database runs each scope through execute, which reads the counts once the
statements ran, even when one failed, unless the scope only read. observe
tracks the SELECT's tables and notes their counts before running it, under
the same mutex as suspendedScope, so no later change is missed; equal
results aren't emitted again, and several changes in a row may lead to a
single query.
Only changes made through the same Database are seen, not those of another
instance, connection or process. Several SELECTs combine with Flow's
combine, which the guide notes may briefly mix new and old results. The
API is experimental, and kotlinx.coroutines becomes an api dependency of
sqllin-dsl, as the API returns a Flow.
Tests: testObserve covers the first results; an INSERT; no emission for a
change to another table, an UPDATE matching no rows, one leaving the results
equal, or a rolled back transaction; an UPDATE that changes them; a join
watching its second table; and a dropped and recreated table staying
watched. testObserveForeignKeyAction covers rows deleted by ON DELETE
CASCADE. jvmTest (56) and testAndroidHostTest on API 26 and 37 (112) pass, so
the triggers work on both. The native test sources compile for macosArm64,
linuxX64, mingwX64 and watchosArm32 but were not run, as this machine is an
Intel Mac. The guide's examples were compiled with a temporary file.
The guide gains a section on observed queries, in English and Chinese. The
roadmap's last high priority item moves to Supported.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A view is a stored SELECT that queries read like a table. SQLlin had no way to declare, create, query or drop one. A class annotated with @dbview gets a generated view object, AdultView for Adult, with a property per column like a table object, but no SET properties, no CREATE TABLE statement and no constraints. The processor rejects a constraint annotation on it, a class that is both @DBRow and @dbview, and a missing @serializable. CREATE_VIEW(AdultView) AS (PersonTable SELECT WHERE<Adult>(PersonTable.age GTE 18)) // CREATE VIEW adults(name,age) AS SELECT name,age FROM person WHERE age>=18 The SELECT has to read the view's row type, checked at compile time, and is taken from its scope as INSERT INTO ... SELECT does. SQLite doesn't let a view take parameters ("parameters are not allowed in views", checked with sqlite3), so inlineParameters writes the SELECT's values into its SQL as literals: strings with doubled quotes, BLOBs as X'..', Booleans as 1 and 0, NaN as NULL. A ? inside a string literal or a quoted identifier, as in replace(name, "?", ""), isn't a placeholder and stays. DROP(view) drops it with DROP VIEW. To keep views from being written to, Relation is the new base of Table and View. SELECT, joins, the SQL functions and the clause elements take a Relation; INSERT, UPDATE, DELETE, CREATE and ALTER keep taking a Table, so writing to a view doesn't compile. This is source-compatible, and the existing tests compile unchanged, but on Kotlin/Native a library compiled against an earlier version has to be recompiled. An observed query of a view watches the tables it reads. SQLite refuses an AFTER trigger on a view, so InvalidationTracker resolves it: the OpenRead instructions of EXPLAIN SELECT * FROM view name the root pages it opens, which sqlite_master maps back to tables, through views over views and joins alike, as checked with sqlite3. The view API is experimental. The generated view objects opt in, so that querying a view needs no opt-in. Tests: testView creates a view from a SELECT whose literals include 'It''s' and replace(name, "?", ""), queries it, with a WHERE on its columns, observes it through an insert into its table, and drops it. Writing to a view, CREATE on it and a SELECT of another type don't compile, and the processor's three rejections were checked with temporary files. jvmTest (57) and testAndroidHostTest on API 26 and 37 (114) pass, so EXPLAIN works on Android too. The native test sources compile for macosArm64, linuxX64, mingwX64 and watchosArm32 but were not run, as this machine is an Intel Mac. The guide to modifying the database describes views, in English and Chinese, and CREATE VIEW moves to the roadmap's Supported. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
KSP 2.3.12 deprecates KSNode.validate(predicate) in favor of validate(predicate, enableNewFeatures), and the processor reported a warning for each of its two calls. They now pass enableNewFeatures = false, the replacement the deprecation names, which validates exactly as before. The processor compiles without warnings, and the code it generates for the test module compiles. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
UByte, UShort and UInt were stored three different ways. INSERT went through their serializers, which encode the bits as a signed number, on every platform. UPDATE passed the value itself to the driver, which bound it as a signed number on the JVM, as its number on Kotlin/Native, and as its string on Android, which INTEGER affinity turned into its number. Reading truncates to the type's width, so every value read back right, but SQL saw a UByte of 200 as -56 or as 200, depending on how and where it was written: checked on the JVM, WHERE testUByte > 100 and WHERE testUInt > 2000000000 matched neither row after an INSERT or an UPDATE. Comparisons, ORDER BY, and the sum and max of such a column, which 2.4.0 typed as if it held the numbers, were wrong above the signed range, from 128 for a UByte. SQLite's 64-bit integers hold every UByte, UShort and UInt, so they are now stored as their numbers everywhere, which SQL compares, orders and sums as Kotlin does. A ULong keeps the bits of a Long, as before, since no 64-bit signed integer holds the values above Long.MAX_VALUE. - storedValue converts them; InsertValuesEncoder applies it in place of the serializers' signed encoding, as it already did for ByteArray, and SetClause applies it to what UPDATE SET binds. - The JVM driver binds them with setLong(toLong()), as the native driver does, and the Android driver converts them before execSQL, as it already did for queries, so code using sqllin-driver directly agrees across platforms too. Reading is unchanged, and reads rows written before this right. In SQL an old value above the signed range stays negative until it is written again; CHANGELOG.md gives the UPDATE that rewrites a column. testUnsignedValues inserts values above the signed range, reads them back, compares, orders, sums and takes the max of them, then updates them and compares again; it fails with the old code. The jvmTest (58) and testAndroidHostTest on API 26 and 37 (116) of sqllin-dsl-test, and those of sqllin-driver (6, 12), pass. The native test sources compile for macosArm64, linuxX64, mingwX64 and watchosArm32 but were not run, as this machine is an Intel Mac. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rebuild libs/linux/libsqlite3.a and libs/windows/libsqlite3.a, which the native tests on Linux and Windows link, from the SQLite 3.51.1 amalgamation with zig, enabling FTS3, FTS4 with the enhanced query syntax, FTS5, R*Tree, Geopoly, the math functions, dbstat and column metadata. The previous libraries had only FTS5, R*Tree and the math functions, so they couldn't run tests of FTS3 and FTS4. zig cc emits debug information by default, which held paths of the machine that built the previous libraries, made them several times larger, and made the Windows one depend on the name of a temporary file. These are built with -g0, and the Windows one with its object file name set, so that they hold no paths and the same sources and options always give the same bytes. They are about a fifth of their previous size. Add two low-priority items to the roadmap: using a SQLite bundled with SQLlin on Android rather than the system's, and supporting FTS5 and R*Tree, which Android's system SQLite lacks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A @DBRow class annotated with the new @fts4 or @fts3 gets a generated FtsTable object, which CREATE creates with CREATE VIRTUAL TABLE. Its columns are the class's String properties, and a @PrimaryKey Long or Long? property named rowid or docid reads and writes the rowid. @fts4 takes the tokenizer and its arguments, the prefix lengths to index and the columns not to index, and @fts3 the tokenizer and its arguments. The processor rejects constraint annotations, columns of other types, a misnamed or mistyped rowid, and @fts4 or @fts3 without @DBRow. MATCH searches an FTS table or one of its columns, and snippet, offsets and matchinfo describe the matches. The new APIs are experimental. A virtual table can't have triggers, so an observed query of an FTS table counts its changes from SQLlin's INSERT, UPDATE and DELETE on it: the write statements now know their table, and Database#execute passes the tables a scope writes to the invalidation tracker, which keeps their counts in its log table. Which tracked tables are virtual is read again after a schema change, as a table may be created again as the other kind. withName on an FTS table returns an FTS table with the same options, so that an FTS table is rebuilt as a table is; it threw before, as it only recognized CREATE TABLE. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The new repeatable annotation @check adds a CHECK constraint with an SQL expression every row has to satisfy: on a property, to its column, and on a @DBRow class, to the table. constraintName names it, which SQLite reports when a row fails it; SQLite before 3.34, as on Android below API 34, reports an unnamed one by the table's name. The annotation is experimental. SQLite gives a column's constraint the name of the named one before it, so a column's unnamed CHECK constraints come before its named ones, and all of them before the constraint of @references. The processor rejects an empty expression, a name that isn't an SQL identifier and two CHECK constraints of a table with the same name, and views and FTS tables reject @check. The generated createSQL is now escaped as a Kotlin string literal: a CHECK expression or a @default value holding a double quote broke the generated code, and one holding a dollar sign became a string template. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Triggers, upsert's DO UPDATE SET and rebuilding a table with converted data all need UPDATE's SET and INSERT to take expressions, such as visits = visits + 1. Add it to the medium-priority items, before CREATE TRIGGER, which needs it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
select AS SomeView makes a SELECT a derived table, (SELECT ...) AS name, named after the object generated for a @dbview class, which gives it its name and column properties, though no such view has to exist in the database. It is read like a table in the FROM clause and in joins, and a derived table of a table's rows joins the table with itself. IN takes a SELECT of a single column of values of the same kind, which is checked when the statement is built, and EXISTS(select) checks whether a SELECT returns any row, referring to the columns of the query it is in if need be. NOT(condition) negates a condition in parentheses, for NOT EXISTS and NOT IN. These APIs are experimental. A relation now gives how a FROM clause writes it, the parameters that writes, and the tables it reads: a statement binds the parameters of its derived tables first, then those of its clauses, always into a new list, so that a derived table read by several queries keeps its own, and a condition carries the tables of its subqueries to the observed queries. A WHERE after result columns no longer drops the parameters before it, which had none until now. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The infix AND and OR apply in the order they are written, as Kotlin's infix functions do, so (age EQ 1) OR (age EQ 2) AND (name EQ "x") means (age = 1 OR age = 2) AND name = 'x'. They combined the SQL without parentheses, as age = 1 OR age = 2 AND name = 'x', which SQLite reads as age = 1 OR (age = 2 AND name = 'x'), as SQL gives AND precedence over OR, so the query returned other rows, though the KDoc promised parentheses. A condition now knows the operator it combines others with, and is put in parentheses as an operand of the other one, while a chain of the same operator is written as it is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Combining a condition with AND or OR added the other condition's parameters to its own list, so a condition kept in a variable and used again, alone or combined, bound parameters its SQL doesn't have, which failed with an ArrayIndexOutOfBoundsException on the JVM. Combining conditions now puts their parameters in a new list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The SQLite SQLlin runs on depends on the platform: Android's system SQLite is 3.9 at API 24, SQLlin's minimum, and lacks options such as the math functions, Apple's depends on the OS version, and on Linux and Windows it is the SQLite the app links. An API that needs a newer SQLite or a compile-time option fails at runtime where it is missing, so the new opt-in annotation @PlatformDependentSQLiteAPI marks such APIs, and using one is a compile error unless the app opts in. RENAME_COLUMN, which needs SQLite 3.25.0 (Android API 30), and DROP_COLUMN, which needs SQLite 3.35.0 (Android API 34), now carry it, which is a breaking change for code using them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ER BY A column given to a function, to GROUP BY or to ORDER BY, or selected into a property with AS, was written without its table's name, as in count(name), so in a join of tables that both have a column of that name, SQLite failed with "ambiguous column name". Conditions already wrote columns with their tables' names; now these do too, as in count(book.name), except in the ORDER BY of a compound SELECT, which can only name the columns of its results. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FROM(relation) starts a join, which INNER_JOIN, LEFT_OUTER_JOIN, RIGHT_OUTER_JOIN, FULL_OUTER_JOIN, CROSS_JOIN and the NATURAL joins extend, each with its own ON or USING, so that a join can have more than two relations. A join is selected from with a projection, a clause function such as WHERE<R>(...), or result columns, which could only select from a single table until now. The new APIs are experimental, and RIGHT and FULL OUTER JOIN, which need SQLite 3.39.0, are marked with @PlatformDependentSQLiteAPI. The type rows are read into is checked against the joined relations when the statement is built: every property has to name a column of one of them, of a single one unless USING or a NATURAL join merges them, and has to be nullable where an outer join can leave its column or expression NULL, which elements now know from the tables of the columns they read. A relation can only be joined once. What a SELECT reads is now an internal From, a relation or a join, which these checks and the SQL of the columns work on; a single relation keeps its SQL and messages. A join doesn't start with its first relation as the receiver, as in PersonTable INNER_JOIN BookTable, as such an infix function would take over the old CROSS_JOIN(CategoryTable) in the lambda of a table, whose receiver is that table. The old joins, written as clauses of SELECT, are documented to be removed in the next version after 2.5.0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Since columns are written with their tables' names, a function given to CREATE_INDEX or CREATE_UNIQUE_INDEX, as lower(name), became lower(person.name), and the index took what followed its last '.', so it wrote name), a syntax error, where it had made an expression index. SQLite doesn't allow '.' in the expressions of an index anyway. An index now writes a column by its name, and a function without the indexed table's name before its columns, outside its string literals. A column or function of another table is rejected, and the KDoc documents expression indexes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add the SQLite syntax that SQLlin doesn't support yet and the roadmap didn't list, from DROP INDEX to window functions: the items that are both easy and useful as high priority, those that are one of them as medium priority, and the others, especially those that need much change or a design discussion, as low priority. Add the removal of the old joins in the next version after 2.5.0 too. Give every item an ID, Fx, which stays the same when it moves to another priority or is done, and which a new item takes the next number of. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Database created its InvalidationTracker at once, and every scope it ran called the tracker's refresh, so the tracker's code was reachable from any app, which code shrinkers then kept, with the coroutines code its StateFlows need, though only observed queries use it. The first observed query now creates it, under the executive mutex, and a scope refreshes it only once it exists. On Kotlin/Native, a release executable that only inserts, queries, updates and deletes went from 1,752,352 to 1,333,936 bytes, and with R8, the code of sqllin-dsl it keeps went from 44 KB to 33 KB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nothing dropped the indexes that CREATE_INDEX and CREATE_UNIQUE_INDEX create. DROP_INDEX(name) does, with DROP INDEX, as an experimental API. The guide to modifying the database now describes indexes, ordinary, unique and expression ones, which only the KDoc did. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CREATE_IF_NOT_EXISTS, CREATE_INDEX_IF_NOT_EXISTS, CREATE_UNIQUE_INDEX_IF_NOT_EXISTS and CREATE_VIEW_IF_NOT_EXISTS do nothing when what they create exists, and DROP_IF_EXISTS, of a table or a view, and DROP_INDEX_IF_EXISTS when what they drop doesn't, so that a migration step can run more than once. CREATE_IF_NOT_EXISTS creates FTS tables too, as CREATE VIRTUAL TABLE takes IF NOT EXISTS. They are experimental. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
WHERE after CREATE_INDEX, CREATE_UNIQUE_INDEX or their IF_NOT_EXISTS forms makes an index of the rows that satisfy a condition, such as a unique index of the emails of the users that aren't deleted. SQLite allows neither parameters nor subqueries there, so the values of the condition are written into the SQL, its columns without their table's name, and a subquery is rejected. WHERE is experimental. For WHERE to complete it, these functions return an IndexStatement, where they returned Unit, which replaces the statement where it is among those of the scope or the transaction. This is source-compatible, but a library compiled against an earlier version has to be recompiled. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DISTINCT(element) makes count, avg, sum and group_concat take only the distinct values of an element, as count(DISTINCT book.author) does. It returns a Distinct, which only these functions take, so it doesn't compile anywhere else, where SQL doesn't allow it. max and min don't take it, as it doesn't change them, and group_concat separates the distinct values with commas, as SQLite allows only one argument with DISTINCT. It is experimental. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ESCAPE after LIKE gives the pattern an escape character, after which %, _ and the escape character match only themselves, as in name LIKE "100\\%%" ESCAPE '\\', which matches the names that start with 100%. It is experimental. LIKE now returns a LikeCondition, a SelectCondition that ESCAPE can follow, so that it reads as SQL does, without parentheses, and doesn't compile after any other condition. This is source-compatible, but a library compiled against an earlier version has to be recompiled. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
VACUUM() rebuilds the database file into the minimum disk space, as deleting rows doesn't shrink it, and VACUUM_INTO(file) writes a vacuumed copy of the database to another file, a consistent backup of a database in use. Both throw inside a transaction, where SQLite can't vacuum, and are experimental. VACUUM INTO needs SQLite 3.27.0, Android API 30, so it is marked with @PlatformDependentSQLiteAPI. As a change of the schema does, VACUUM makes observed queries query again, as it may change the rowids of the tables without an INTEGER PRIMARY KEY. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reading a column's property in SET {}, as in SET { visits = visits + 1 },
compiled, but its getter only returns a placeholder, as 0 or "", so the
column was set to a value computed from it, here 1, with no error.
The processor now generates the getter deprecated with the level ERROR,
so reading the property doesn't compile, while assigning it is
unchanged. The test of the generated properties now assigns them, and
the compile-time check of their nullability, which reads their types,
suppresses the error. This is a breaking change for code that reads
them, which only got placeholders.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The guide to rebuilding a table renamed the new table after dropping the old one, which fails while a view reads the dropped table on the JVM, Linux and Windows, where SQLite's legacy_alter_table is off by default, as renaming then checks every view: "error in view ...: no such table". Views came with 2.5.0. The guide now drops such views before dropping the old table and creates them again after the rename, which works on every platform, and says that only where the setting is off do the foreign keys of other tables follow a renamed table whether or not foreign keys are enforced. testTableRebuild rebuilds a table a view reads, which fails on the JVM without dropping the view. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first version of triggers leaves out INSTEAD OF triggers, which only views can have, and which would be of no use while the DSL can't write to a view, as a view isn't a Table. F63 keeps them for later, together with INSERT, UPDATE and DELETE on views. Supported now lists the items in the order they were done, the newest first, so F1 to F6 are put in that order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An element now knows the tables its subqueries read, which an observed query watches, whether it always gives the same result for a row, and whether it is a literal on its own, and an expression derives these from its operands. An expression carries no parameters: its values are written into its SQL, so that it can be part of a view, a trigger or an index. literal(value) makes a value an expression; +, -, *, / and the unary - compute with numbers of the same type, % with integers, and + of strings concatenates them; CAST(x AS INT) converts an expression to the types SQLlin names columns with. The new functions take expressions as their arguments: coalesce, ifnull and nullif, the scalar max and min, hex, quote, unicode, randomblob, zeroblob, more forms of trim, substr, replace, instr, printf and round, the aggregate total, and the date and time functions. sign, unixepoch and the math functions, which need a newer SQLite or its math functions, are marked with @PlatformDependentSQLiteAPI. A function can only be NULL where its result can, as a division by zero is. ORDER BY and GROUP BY reject a literal on its own, which SQLite would read as the number of a result column, and an index rejects an expression with a subquery, an aggregate function, or a function that doesn't always give the same result, which SQLite only rejects once the table has rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CASE(WHEN(condition) THEN value, ..., ELSE = value) is SQL's searched CASE, which is NULL where ELSE is omitted. select[R::property] uses a SELECT of a single column as a value, which is checked when the statement is built, and whose tables an observed query watches. IS and ISNOT of two expressions compare them as EQ and NEQ do, NULL equaling NULL, which IS DISTINCT FROM does on newer SQLite only. COLLATE compares and sorts a string by BINARY, NOCASE or RTRIM, and and, or, inv(), shl and shr are the bitwise operators of Int and Long. JSON's -> and ->>, which need SQLite 3.38.0, become their own item of the roadmap, F64, and REGEXP is left out, as the JVM's SQLite has no such function, and Android's only matches whole strings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SET(listOf(expression AS Row::property)) sets columns to expressions,
as visits + 1, and INSERT, INSERT_OR_IGNORE and INSERT_OR_REPLACE take
a row of expressions, for INSERT INTO ... VALUES, leaving the other
columns to their default values. AS checks the type of each expression
at compile time, and the statement checks, when it is built, the
checks of result columns that apply: an expression that can be NULL is
only written to a nullable column, and it isn't an aggregate function
and only reads the updated table, or no table in INSERT, where SQLite
allows none. The error of reading a column in SET {} now points to
these.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CREATE_TRIGGER(name) AFTER UPDATE_OF(column) ON table WHEN (condition)
BEGIN { ... } creates a trigger, with BEFORE and AFTER, the events
INSERT, DELETE, UPDATE and UPDATE_OF, and an IF NOT EXISTS form, and
DROP_TRIGGER drops one. new(column) and old(column) read the row being
written, and a row the trigger doesn't have, which SQLite would only
report when the trigger fires, is rejected when the statement is built.
The body is written as any statements are: the scope collects them in
its own container instead of running them, and only INSERT, UPDATE,
DELETE and RAISE are allowed. The values of a trigger are written into
its SQL, as SQLite doesn't let a trigger take parameters.
A trigger can write an FTS table, which the tracker of observed queries
can't count changes of with triggers, so a write to a table that has
triggers now counts as a change of every tracked FTS table. The guide
to rebuilding a table drops the triggers of other tables that read it,
as it does views, and creates the table's own again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
table INSERT DEFAULT_VALUES inserts a row of the default values of all columns, as INSERT INTO ... DEFAULT VALUES does, with INSERT_OR_IGNORE and INSERT_OR_REPLACE too. A column without a default value takes NULL, which a NOT NULL one rejects. DEFAULT_VALUES is a property of the scope, as the events of triggers are, so it needs no import. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ANALYZE gathers the statistics the query planner chooses indexes with, of the whole database, a table or an index, and REINDEX rebuilds the indexes of the database, a table, an index or a collation. PRAGMA optimize, which needs SQLite 3.18.0, Android API 26, is marked with @PlatformDependentSQLiteAPI. Like VACUUM, they count as changes of the schema for observed queries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CREATE_INDEX, CREATE_UNIQUE_INDEX and their IF_NOT_EXISTS forms take
the columns with the orders they are sorted in, as ORDER_BY does:
CREATE_INDEX("idx_person_age", PersonTable.age to DESC). The first
column is a parameter of its own, so that a call without columns, which
fails, still has a single candidate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ORDER_BY(age to ASC NULLS LAST) puts the NULL values last, as SQL reads, and DESC NULLS FIRST first, which need SQLite 3.30.0, Android API 31, so NULLS is marked with @PlatformDependentSQLiteAPI. ORDER BY now takes an Ordering, which OrderByWay and the new NullsOrdering are, rather than an OrderByWay. Pair and Map are covariant, so this is source-compatible, but a library compiled against an earlier version has to be recompiled. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A COMMIT that fails, as one does while a deferred foreign key isn't satisfied, leaves SQLite's transaction open. No driver rolled it back, so the statements after it ran in it, and the next transaction failed to begin, with "cannot start a transaction within a transaction". The JVM driver even set the connection back to auto-commit over the open transaction. The JVM and Kotlin/Native drivers now roll it back, and throw the error of the COMMIT. Android's framework lets go of such a transaction, and turns a ROLLBACK given to it into its own end of a transaction, which it has none of: it tells a ROLLBACK by its first characters, after a comment too from some API level on. So the Android driver raises the ROLLBACK with a TEMP trigger, whose table and trigger, created in the transaction, are rolled back with it, as tested on API 26 and 37. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
deferred = true on @references and @ForeignKeyGroup makes a foreign key DEFERRABLE INITIALLY DEFERRED: checked when the transaction commits, rather than after each statement, so that a transaction can insert a child before its parent, or rows that refer to each other. A transaction that commits while such a key isn't satisfied fails and is rolled back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The onConflict of @PrimaryKey, @CompositePrimaryKey, @unique and @CompositeUnique, of the new experimental enum OnConflict, writes the ON CONFLICT clause of the constraint, as UNIQUE ON CONFLICT REPLACE. ABORT, the default, writes none, so the tables declared before are created as they were. A composite key or group takes the one given on any of its properties, and the processor rejects one whose properties give different ones. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
F17 gave PRIMARY KEY and UNIQUE their conflict clauses, but not NOT NULL, which a property's type declares, so that no annotation can hold its clause yet. It would let NOT NULL ON CONFLICT REPLACE write the default value instead of NULL, which SQLlin's statements rarely write to a non-null column. F65 keeps it, with the two ways it could be declared, until other syntax makes it of more use, or shows it isn't needed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
New features for 2.5.0, one commit per feature or fix, with roadmap items and two build maintenance commits. Each commit message explains its change in detail,
CHANGELOG.mdhas an entry for each, and the guides insqllin-dsl/doc(English and Chinese) describe the new APIs. Every new DSL API is marked with@ExperimentalDSLDatabaseAPI, and every API that needs a newer SQLite, or a compile-time option of it, with the new error-level@PlatformDependentSQLiteAPI.New features
Queries and tables:
UNION,UNION_ALL,INTERSECTandEXCEPTas infix operators between SELECTs of the same result type, checked at compile time; the oldUNION {}block is deprecated. (8a8a48c)Database#observeturns a SELECT into aFlowof its results, refreshed when a statement writes a table it reads, counted by SQLite through TEMP triggers. Its tracker is only created by the first observed query, so code shrinkers remove it from apps that observe none. (b0394d9, 9db3b62)@DBViewandCREATE_VIEW(view) AS (select). (6c297dd)MATCH,snippet,offsetsandmatchinfo. (e8932b3)@Check. (6daa2d3)IN (select)andEXISTS. (0ab7b37)FROM(relation), of any number of tables, with checked result types and RIGHT and FULL OUTER JOIN; the joins written as clauses of SELECT are documented to be removed in the next version. (9cbb6a9, c9e9819)The high-priority roadmap items:
DROP_INDEX(F1),IF NOT EXISTSandIF EXISTS(F2), DISTINCT in aggregate functions (F3), partial indexes (F4),LIKE ... ESCAPE(F5),VACUUMandVACUUM_INTO(F6). (8d3fb9a, 1e8b1cf, dd05ea7, e4817c8, 46da59d, a8db631)Expressions and triggers, designed together:
+ - * / %, string concatenation,CAST(x AS INT),coalesce,ifnull,nullif, date and time functions and more, with the type of each result checked byAS. (4b24336)CASE, scalar subqueriesselect[R::property], null-safeIS/ISNOT,COLLATEand bitwise operators. (059ae68)SET(listOf((visits + 1) AS Person::visits))andtable INSERT listOf(expression AS Row::property). (dde622b)CREATE_TRIGGER(name) AFTER UPDATE_OF(column) ON table WHEN (condition) BEGIN { ... }, withnew()/old()andRAISE. (e05ccfb)Medium-priority items that turned out easy:
INSERT ... DEFAULT VALUES(F24),ANALYZE,REINDEXandPRAGMA optimize(F28), ASC and DESC on index columns (F20),NULLS FIRST/NULLS LAST(F22), deferred foreign keys (F18), conflict clauses on PRIMARY KEY and UNIQUE (F17). (0d8640d, 95c0807, d456701, b696f57, 453ed59, 8f74c73)Breaking changes
These stop existing code from compiling. Migration is described in
CHANGELOG.md.RENAME_COLUMNandDROP_COLUMNare marked with@PlatformDependentSQLiteAPI, as they need SQLite 3.25.0 and 3.35.0 (Android API 30 and 34), so code using them has to opt in. (f3ab165)SET {}is a compile error (B5):SET { visits = visits + 1 }used to compile and setvisitsto1, as the getter only returns a placeholder. (1a39a29)Several changes are source-compatible but binary-incompatible, so a library compiled against 2.4.0 has to be recompiled:
CREATE_INDEXreturns anIndexStatement,LIKEaLikeCondition, ORDER BY takes anOrdering, and on Kotlin/Native, the DSL takes aRelationwhere it took aTable.Fixes
(a OR b) AND cbecamea OR (b AND c). (e02a03c)sqllin-driver): a transaction whose COMMIT failed, as one does while a deferred foreign key isn't satisfied, was left open on every platform. It is now rolled back; on Android, through a TEMP trigger, as the framework intercepts a ROLLBACK. (b627b6c)createSQLdidn't escape",\and$. (6daa2d3)Other changes
Fx, and the SQL that SQLite has and SQLlin lacks is listed by priority; new items F63 (INSTEAD OF triggers), F64 (JSON) and F65 (NOT NULL conflict clause) keep what this cycle left out. (5859298, d0ed1c2, 8c9b63b, a1aa200, 0f3e880)Testing
sqllin-dsl-test:jvmTest(93 tests) andtestAndroidHostTeston Robolectric API 26 and 37 (186 tests) pass locally.sqllin-driver:jvmTest(7 tests) andtestAndroidHostTest(14 tests) pass.macosArm64andiosSimulatorArm64, and compile forlinuxX64andmingwX64, whose links need a Linux or Windows host. CI on this PR is their first run.SET {}, or a composite constraint whose properties give different conflict resolutions, were verified by compiling temporary files.Left for the next version
UNION {}block (F7).🤖 Generated with Claude Code