Repository navigation
Intermittent errors creating a collection #187
Description
Hi, I am getting intermittent, inconsistent errors creating a collection. Any insight or assistance appreciated. Thank you in advance for your assistance!
Update: The issue seems to go away when I change the shard number from 4 to 2.
Issues
When I first run the below script, when the collection does not exist, I get Error: Error in the response: The operation was cancelled Timeout expired. Despite that, the qdrant_storage/MY_COLLECTION folder exists, and it shows up in the dashboard.
When I run it again, I get
collection exists, deleting...
Error: Error in the response: The operation was cancelled Timeout expired
I have also encountered issues where the client does not think the collection exists (so does not try deleting it), but when it tries to create it, I get errors like
Error in the response: Client specified an invalid argument Wrong input: Can't create collection with name MY_COLLECTION. Collection data already exists at ./storage/collections/MY_COLLECTION
Sometimes, it works fine, but it's unclear what conditions lead to success.
Code
main.rs
use anyhow::{Context, Result};
use qdrant_client::qdrant::{
CreateCollectionBuilder, Datatype, Distance, OptimizersConfigDiffBuilder, VectorParamsBuilder,
};
use qdrant_client::Qdrant;
const QDRANT_CONNECTION_URL: &str = "http://localhost:6334";
#[tokio::main]
async fn main() -> Result<()> {
let client = Qdrant::from_url(QDRANT_CONNECTION_URL)
.build()
.context("Failed to build Qdrant Client")?;
create_collection(&client, "MY_COLLECTION").await?;
Ok(())
}
pub async fn create_collection(client: &Qdrant, name: &str) -> Result<()> {
if client.collection_exists(name).await? {
println!("collection exists, deleting...");
client.delete_collection(name).await?;
}
client
.create_collection(
CreateCollectionBuilder::new(name)
.vectors_config(
VectorParamsBuilder::new(1280, Distance::Cosine)
.datatype(Datatype::Float16)
.on_disk(true),
)
.optimizers_config(OptimizersConfigDiffBuilder::default().indexing_threshold(0))
.shard_number(4),
)
.await?;
Ok(())
}Cargo.toml
[package]
name = "upload-issue"
version = "0.1.0"
edition = "2021"
[dependencies]
anyhow = "1.0.83"
qdrant-client = "1.11.2"
tokio = { version = "1.37.0", features = ["rt-multi-thread"] }
Activity
timvisee commentedon Sep 9, 2024
Is it possible for your storage is slow? If using slow storage, creating more shards on disk would take more time, which might be causing the timeout.
Could you elaborate on your cluster set up a bit? Ideally the following:
The error message you show above is definitely a bug. You should never be able to get into a state where both deleting and creating a collection fails. Hopefully the above gives us a bit more insight.
Note that I don't think this is specifically related to the Rust client, but rather to Qdrant itself.
jlkravitz commentedon Sep 9, 2024
Hi @timvisee It's possible this is related to Qdrant, but I recall being able to run the commands successfully through the API directly. I will explore and confirm that, if that would be helpful. If you'd like me to move this issue to Qdrant, I can do that, too.
Answers below to your questions!
Storage Configuration
Our qdrant data is stored in
/mnt/raid0.• Size: 43.94 TiB
• File System: ext4
• Mount Point: /mnt/raid0
• RAID Level: RAID0 (striping)
• /dev/vdd: 10.74 TiB (Rotational Disk)
• /dev/vde: 13.67 TiB (Rotational Disk)
• /dev/vdf: 14.65 TiB (Rotational Disk)
• /dev/vdg: 4.88 TiB (Rotational Disk)
IOPS
Running a
fiobenchmark gives me 164,000 IOPS (sequential write) and 133,000 IOPS (random write).Full outputs below.
Sequential write benchmark
Random write benchmark
Number of nodes
We just have one node.
jlkravitz commentedon Dec 10, 2024
@timvisee Just following up here. Any thoughts?