optimize xDTD queries and update xDTD database schema

[chunyuma]

This PR was created to resolve the concern reported in issue #2893.

Basically, we did the following modifications:

1. Implement batch queries to minimize xDTD database query frequencies.

2. Fully recover all node properties from the KGX node JSON file.

WARNING:

Please don't merge this PR until we re-build xDTD for the next release of Translator KG, as we have updated the database schema for more node properties, which will crash in the current version of xDTD.

[dkoslicki]

@chunyuma any updates on this?

[chunyuma]

Hi @dkoslicki, as you can see the last two commits before your comment, we have updated the code to do auto-detect all node and edge properties from kgx, and show them all in the xDTD query.

[dkoslicki]

Sorry, I should have been more clear: is it ready to merge? I see some failing tests but don't know if they need to be addressed or not (since the CI/CD tests have been causing issues recently). This PR in particular was talked about during the performance call today, and they would like it to be merged to CI before two weeks from now, so I was trying to ask if we are good to merge it or not

[chunyuma]

Hi @dkoslicki, there's no need to apologize. I should update the status with you after I resolved your concern.

Thanks for letting me know today's performance call discussed this PR. Let me provide more clarifications below:

1. Is the code ready?

Yes, the code itself is ready, but the database is not. As the xDTD database we are currently using still uses the old schema, it is not compatible with the updated code, The CI/CD uses the current xDTD database, but uses the new code. This explains the failing tests. You might notice that the failing tests are all associated with the xDTD.

2. Should we address the failing tests?

There are two options to address these tests:
A. refresh the current xDTD database using the new schema that is compatible with the new code.
Pros: we can address these failing tests immediately and merge this PR to address issue #2893.
Cons: we need to update the xDTD database with the new schema in all relevant servers.

B. we are now re-training xDTD and re-building xDTD database. The new xDTD database will follow the new schema. So, we can wait until the new xDTD is completed by the end of this week.
Pros: we need to need do an extra step to fix these failing tests.
Cons: need to wait a few more days.

Please let me know which option you and the Translator consortium prefer.

[dkoslicki]

Ah, I see. Thanks for the pros/cons list. Since the new xDTD database is running and we will need to put it in place everywhere, let's go ahead and do option B: wait until the new xDTD is complete, then get this new DB everywhere and merge the PR. Go ahead an put it on your TODO list for when the new xDTD is complete